Commit Graph
24 Commits
Author SHA1 Message Date
rootandClaude Opus 5.5 0088c1c9e7 feat(sites): 站点请求失败留样(site_failure_sample,每站最近 5 份)
- 迁移 0077 新表 site_failure_sample:类型(parse_error / challenge)、请求类型、脱敏 URL、状态码、
  关键响应头、错误文案、命中规则、脱敏页面开头(≤ 64 KB);样本几十 KB,不放进每条请求都读的 site.config
- 适配器 onFailureSample 回调接到 runtime.siteAdapter,异步写入并删到最近 5 份,写失败只记日志
- GET /sites/{id}/failure-samples(不带正文)与 /failure-samples/{sampleId}(带正文)

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Cxwgnn2pbzQ7rXv3rq7fei
2026-10-06 15:41:00 +08:00
rootandClaude Opus 5.5 691671a1d2 feat(identity): 识别接入集数解析——映射规则表、规范季集写回解析结果、待确认不建单元并开待处理、规则失效停用、按作品重算
迁移 0066_episode_mapping。

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Cxwgnn2pbzQ7rXv3rq7fei
2026-10-03 18:03:32 +08:00
rootandClaude Opus 5.5 59b3680f35 feat(subtitles): 字幕站(SubHD、字幕库)在线搜索与下载;rar / 7z 解压;验证码与页面解析在沙箱里
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 02:28:29 +08:00
rootandClaude Opus 5.5 8eb8a68295 feat(web): 站点按类型分组、列表模式修样式、下掉自定义 RSS 源与索引器入口;智能体会话分对话 / 自动巡检两栏
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-02 23:56:27 +08:00
rootandClaude Opus 5.5 b8876542b8 feat(sites): 内置公开 BT 站 Nyaa / 蜜柑 / AniBT / AnimeGarden / 萌番组
- 新专用解析器(spiders/public.ts):RSS / Torznab / JSON API,搜索 + 最新列表 + 按资源 ID 取种子或磁力
- 手写定义(definitions/public.ts,public: true),并入内置定义表;定义视图按 public 标为 public_bt
- 添加站点:公开 BT 站可从内置站一键添加(专用适配器、默认定时抓取、默认不进搜索集合),手填 RSS 保留
- 样例取自 2026-10-02 公开接口,截前两条

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-02 21:19:19 +08:00
rootandClaude Opus 5.5 135c50048f feat(downloads): 多下载器路由:默认下载器、站点指定下载器、别名
照 MP(chain/download/submission.py:明确指定 or 站点下载器 or 默认):
- 迁移 0063:downloader_instance.is_default(至多一个,唯一索引)、site.downloader_instance_id
- pickInstance:明确指定 > 站点指定 > 默认 > 第一个可用;自动下载(gates)与手动下载共用
- 设置:下载器名称改称别名,新增「设为默认下载器」;站点编辑新增「下载器」选择
- 测试 downloader-routing.test.ts

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-02 20:32:29 +08:00
rootandClaude Opus 5.5 16083942c0 feat(sites): 出口统一与 Cloudflare 过挑战级联(CF-2)
- 出口:站点请求不再隐式继承 HTTP(S)_PROXY,没开代理明确直连(创建客户端时临时拿掉代理环境变量,Deno 2.9 语义见 egress.ts),
  开了走站点代理;TMDB / Brave / 图片改为显式按环境变量(行为不变)
- CloudflareGate:浏览器代理期直接代抓 → HTTP(可复用时带 cf_clearance)→ 挑战 → 浏览器过挑战(同站 30 分钟一次、单飞)
  → cf_clearance 仅在 UA 跟随浏览器且两端同出口时给 HTTP,重试 1 次记 cfReuse → 仍被挑战则浏览器代抓并进入 6 小时代理期
- 浏览器:solveSiteChallenge(自动放行 ≤15s / 系统发起视觉子流程最多 2 次 / 真验证码转人工待处理)、fetchSiteHtml(profile 锁 + 站点令牌)、
  交还后读回 cf_clearance 继续;site.config.cloudflare 状态契约

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-30 21:26:54 +08:00
rootandClaude Opus 5.5 b1d30bd6ee fix(sites): 观众健康检查误报未登录,健康失败原因细分(HC-1)
观众 2026 版首页没有 <a href=logout>:退出按钮在 userbar 的 data-uploader-actions JSON
里由脚本渲染,旧登录标志 a[href*="logout"] 永远不命中,健康检查报"未登录"而用户数据正常。

- audiences 定义 loginSuccess 改为 #info_block / usercp.php / logout 任一(0.2.1)
- 健康检查判定细分:登录页 / 权限页 / 疑似人机验证 / 找不到登录标志(带选择器),
  原始判定写入 last_health.detail,站点概览技术信息展示
- 首页只是找不到登录标志时,以 24 小时内用户数据解析成功为已登录证据(同 MP)
- 新 SiteError 类 challenge(cf-mitigated: challenge),不算凭据错误;健康状态 challenge
- testkit mock 增加 setHeaderLayout;加观众首页脱敏结构样本的差分测试

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-30 20:38:48 +08:00
rootandClaude Opus 5.5 adc853aa64 feat(sites): site requests follow browser UA, smooth UA switch, CF cookies browser-only (BR-4)
- every outbound site request (adapter list/search/detail/download/health,
  RSS, login, user data) carries the account UA via SourcesRuntime.siteFetch;
  no more Deno default UA
- priority: account manual > pending > global setting > browser identity
  (fallback: same-shape Windows Firefox UA); settings UA default is empty
- migration 0056: existing cookie/password accounts keep their old UA as
  ua_pending until re-login or browser cookie write-back switches cookie+UA
  in one transaction; browser_provider.identity caches the browser identity
- cf_clearance/__cf_bm stripped on browser write-back (names kept only)
- browser sessions get per-site egress (same proxy as HTTP, or direct)
- site account page shows UA source; raw UA in tech info

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-30 03:17:13 +08:00
root 7a24d23bbf Merge branch 'ia2'
Co-Authored-By: Claude Fable 5.1 <[email protected]>

# Conflicts:
#	CHANGELOG.md
#	docs/plan/00-first-version-plan.md
#	packages/agent/src/system-prompt.ts
2026-09-30 00:30:42 +08:00
rootandClaude Opus 5.5 89d2c81de6 feat(sources): site kinds and capabilities (IA-2)
site.kind: pt / public_bt / rss_feed / torznab. Each kind declares the
acquisition modes it may support (search, list, rss, detail, login,
userinfo); a site's effective capabilities = kind ∩ site definition.
torznab only defines TorznabSiteConfig and cannot be added yet.

Migration 0055: generic_rss sites become rss_feed, the rest pt; feeds
without a site are grouped by host into new custom RSS source sites
(cursor window and conditional headers cleared so the next fetch writes
releases). User DB read-only dry-run: 0 orphan feeds.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-30 00:29:56 +08:00
rootandClaude Opus 5.5 fa344a7184 feat(web): token 数以 K 输入显示,已存凭证显示掩码占位(FIX-8)
- 新增 UnitInput 与 kind=tokens:上下文预算、单次 token 上限、自定义模型上下文窗口以 K 输入(允许小数),存储仍为原始 token 数;校验与提示用 K 表述;列表/详情 token 数统一 K/M 简写(formatTokens)
- 新增 SecretInput 与 kind=secret:已保存时显示「已设置 ••••••••」(有服务端掩码时带末 4 位),聚焦直接输入替换,留空保持,「清除」确认后保存生效;替换模型服务、TMDB、网页搜索、下载器、媒体服务器、浏览器服务、站点账号、Passkey、本机 Token 各处
- 统一只写凭证服务端语义:省略或空串 = 不变,null = 清除(contracts secretInput);下载器/媒体服务器新增清除;站点凭证同类更新可留空保持密码/Cookie/API Key/TOTP,TOTP 可 null 清除,不再误删 TOTP 与 UA
- contextBudget 最小值 4096→4000、contextWindow 1024→1000,便于按 K 填写
- 测试:换算与校验、SecretInput 渲染与状态、各接口留空不变/显式清除

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-30 00:10:46 +08:00
root 4c7375b52f Merge branch 'search3'
Co-Authored-By: Claude Fable 5.1 <[email protected]>

# Conflicts:
#	docs/plan/00-first-version-plan.md
2026-09-29 20:52:51 +08:00
rootandClaude Opus 5.5 970296218c feat(search): multi-select quick filters for search results and site releases (SEARCH-3)
- domain release-facets: bucket resolution / codec / release type / site / promotion / HDR /
  release group / season / episode; OR within a dimension, AND across; disjunctive counts
- GET /search/releases and GET /sites/{id}/releases accept qf and return facets; site list
  facets come from local source_release rows only (no site requests), paging stays consistent
- web QuickFilterBar shared by search results and site resources tab: per-dimension popover
  (bottom sheet on phones) with counts, removable tags, clear all, localStorage memory per page
- runbook: quick filter params and bucket definitions

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-29 20:52:29 +08:00
rootandClaude Opus 5.5 9be9813021 feat(sites): structure release detail pages and rework the detail drawer (DETAIL-1)
Adapters (pure functions + tests):
- detail-description: strip HTML/BBCode residue, drop site template blocks and
  ad/signature lines (configurable rules), parse PT info blocks (◎ / 【】 / 『』 /
  「键:」) into infoBlock and body paragraphs
- mediainfo: MediaInfo and BDInfo (quick summary + full report) into a summary
  of container/duration/bitrate/size, video (HDR/DV), audio and subtitle tracks
- file-tree: file list into a flattened directory tree with sizes and
  main/sample/subtitle/extra roles
- formatDetailPage combines them; the server computes it when reading the
  cached detail page, so existing caches benefit without a refetch

Contracts: SiteReleasePage gains infoBlock, paragraphs, media, fileTree.

Web: drawer now reads top-down: meta row (promo/H&R, size, peers, upload time,
IMDb/TMDB/豆瓣 tags) → 资料 → 简介 (collapsible) → 技术规格 (raw folded) →
截图 grid with keyboard lightbox → 文件 tree; inventory overlay moved into the
fixed footer next to 下载.

Fixtures: 6 desensitized samples from the local cache (site.detail and RSS
descriptions) plus synthetic multi-track MediaInfo, BDInfo, bracket blocks and
directory file lists.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-29 20:50:42 +08:00
rootandClaude Opus 5.5 04cffcd1f1 feat(sites): on-demand site detail page content (INT-2)
- SiteReleaseDetail.page: description, MediaInfo/BDInfo, screenshots,
  IMDb / Douban / TMDB ids, file list, tags, uploader, plus a parse probe
  (labels found) for layout-change diagnosis. Generic NexusPHP rules
  (field label -> next sibling) starting from the MovieClaw detail
  selectors; site overrides under umt.detail.page (HHClub: div labels).
  M-Team builds the same structure from descr BBCode + mediainfo.
- Fetched only on demand (drawer open without cache, "复核详情",
  release.inspect); content in kv_cache region site.detail for the
  site detail cache period (>= 24 h); within the window the site is not
  requested again. External ids persist in source_release.raw._detail,
  which list snapshots do not overwrite.
- Screenshots only as proxy paths /api/v1/img/site-detail/{rid}/{i}:
  public-address check on every hop, no site credentials, magic-byte
  image check, 8 MB cap. Poster proxy shares the same helper.
- Identity: site ids count as explicit-id evidence; when TMDB is
  queried, a TMDB id goes straight to details and an IMDb id to
  /find (no title search on a hit). Douban only for local recall.
- Release drawer "站点详情页" section; fixtures and tests for HHClub div
  layout, rowhead layout, BBCode, cache/proxy/identity end to end.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-29 16:21:17 +08:00
rootandClaude Opus 5.5 901b01d1a1 fix(sites): HHClub parsing, resource list layout and request restraint
HHClub (憨憨) was checked read-only against a real account (one detail
request; list, user data and health came from the product's own fetch).

Parsing fixes:
- list titles carried the site's new-torrent marker "[新]" (58 of 100
  rows; the detail page title has none); stripped with a replace filter
- refreshing a detail overwrote seeders / leechers / snatched / category
  name / H&R with null because the HHClub div detail page lacks them;
  detail values now fall back to the list snapshot (mergeDetailOverList),
  and an unknown detail promotion keeps the list promotion
- level "贵宾" is VIP (umt.vipLevels)
- sanitized minimal fixture HHANCLUB_LIST_PAGE_2026 + tests; README marks
  HHClub as verified 2026-09-29
Size, counts, category ids/names, +08:00 time and subtitle were correct.

Request restraint:
- site definitions may carry umt.defaults; merged with settings.sites
  (longer intervals, fewer pages): new settings defaultSnapshotMaxPages 1,
  defaultHealthCheckIntervalSec 6 h, defaultDetailCacheSec 24 h; default
  burst 2. HHClub: snapshot 60 min page 1 only, health 6 h, search
  concurrency 1, detail cache 24 h, 1 token / 3 s burst 2
- periodic / manual snapshots are capped to that page count; scheduled
  health checks skip sites checked within the interval; detail refresh is
  served from cache inside the window; live searches on one site queue
- every rate-limiter token is counted per site per hour
  (site_request_count, migration 0046); GET /sites/{id}/request-stats;
  site overview and resource list show requests in the last 24 hours

Resource list: two-layer rows. Main line: title + spec tags (episodes,
resolution, source, codec, HDR, group, 国语/中字, site tags) + promotion
with time left + H&R. Second line: subtitle (one line, ellipsis) and
fixed-width category / size / seed-leech-snatched / upload time
(relative, absolute on hover) columns, stale badge, inventory overlay;
folds into a card on phones.

Also aligns identity_candidate.normalized_title order in db types with
the migrations (schema drift test).

Takes effect after the stack is rebuilt (migration 0046).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-29 15:18:15 +08:00
rootandClaude Fable 5.1 ac478673bc docs(plan): on-demand site detail page parsing (HHClub details.php)
Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-09-29 15:13:35 +08:00
rootandClaude Opus 5.5 4a92dfd2dc feat(feeds): site-scoped RSS add infers account, parser and poll
Adding a feed from a site page now only asks for the URL, purpose
(discover / track / promotion) and an optional note. POST /feed-sources
with siteId infers the site account (passkey holder first), records the
site definition version as parserVersion, defaults the name to
"<site> RSS" and polling to 30 minutes, and returns non-blocking
warnings when the URL host does not match the site. The dialog shows the
redacted URL, fetches once right after saving and reports the item count
or the error.

The generic feed form on the sources page suggests linking a registered
site when the URL host matches (can be undone). name is now optional.

Takes effect after the stack is rebuilt.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-29 14:43:19 +08:00
rootandClaude Opus 5.5 e2d1e21968 feat(contracts): site center schemas, endpoints and migration 0031
Site health/proxy, SiteAccount login state (challenge, lastLoginAt, lastError,
userData), SiteUserData history table, SiteCredentialsBody by kind,
refresh-login/health-check/user-data/delete endpoints, SitesSettings defaults.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-28 14:40:07 +08:00
rootandClaude Opus 5.5 d3ab54f2bd feat(server): W4-A integration endpoints and contract gaps
- POST /acquisition-plans: manual submit via download.plan -> download.submit
  (same authorization/audit/idempotency path as the agent), slotId or entityId
- read-only /media-assets, /asset-locations, /asset-unit-coverage, /path-bindings,
  /reconcile-runs, /server-mappings, /artifacts, GET /title-aliases/{id};
  unimplemented write endpoints removed from api.ts; registry test now checks both directions
- candidate evaluation upsert per (slot, source_release) + migration 0022 dedupe/unique index
- organization records expose downloadTaskId (+ list filter); inventory adds
  subscribed/candidateReasons; feed sources expose derived possibleGap
- GET /events?after=&limit= JSON replay page for API key callers (same sse_event table)
- preference tag revision history from audit_event; subscriptions/identities/aliases
  lists carry mediaEntityTitle; subscription q/mediaKind search; slot subscriptionId
  filter + currentAsset summary; policy preview samples carry title/reasons
- DecisionSummary, IdentityRevisionList, ReleaseIdentityResult, PolicyPreviewSummary exported from contracts

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-27 21:10:02 +08:00
rootandClaude Opus 5.5 a24bda952b feat(server): W2-B sources, identity resolution, metadata and site workbench
- Sites / accounts / feeds: registry view (/site-definitions), site CRUD with
  enable/disable (history stays queryable, marked site_disabled), accounts and
  feed URLs stored as secrets (only secret_ref / redacted urlDisplay exposed),
  feed enable/disable/reset/fetch; one shared rate limiter + adapter per site.
- Tasks: ingestFeed (FeedFetchState/FeedCursor on feed_source, observation
  feed+guid upsert with digest-change updates, source_release site+id
  aggregation across feeds, auth failure -> account invalid + decision, no
  retry, gap -> feed.gap_suspected -> budgeted backfill snapshot),
  resolveObservation (parseRelease -> local recall via title_alias pg_trgm +
  accepted identities -> TMDB -> resolveIdentity; auto binds entity /
  provisional units / identities / aliases, otherwise identity_candidate pool
  with evidence version and bounded review backoff; manual lock respected),
  refreshMetadata (metadata_field + field_override, user locks win, paths
  untouched), snapshotSite (site_snapshot coverage records).
- Events: feed.due, feed.gap_suspected, release.observed (self-consumed),
  release.resolved, decision.created/updated; scheduler in worker start().
- Identity API: entity detail with redirects, manual mapping correction with
  identity_revision, merge with identity_redirect, pool list/assign/dismiss/
  retry, content unit numbering lock, aliases, external identities.
- Decisions: list/summary (by entity and kind)/detail/resolve; identity
  options executed directly.
- Workbench: /sites/{id}/releases (snapshot first, coalesced live search with
  backoff, origin + freshness) and /sites/{id}/releases/{rid} (volatile field
  recheck times, refresh keeps data on failure).
- Capabilities: catalog.lookup, release.search, release.inspect,
  identity.resolve.
- Migrations 0018-0020: site.adapter api, source_release.resolution_locked,
  site_snapshot, identity_candidate, resolution_attempt, identity_redirect,
  metadata_field, field_override.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-27 20:12:01 +08:00
rootandClaude Fable 5.1 304feea141 chore(deno): migrate toolchain from Node 24 + pnpm to Deno 2.9.7 + TypeScript 7.0.2 (W1.5)
- root deno.json workspace (nodeModulesDir auto), npm versions registered once in imports, deno.lock;
  remove pnpm-lock/pnpm-workspace/root package.json; members keep minimal package.json (name/type/exports)
- tasks: dev, typecheck (tsc 7.0.2 per member + deno check), lint/fmt (biome, useImportExtensions),
  test (vitest via deno), check:tokens, build (web + deno compile), ci, db:migrate/reset, gen:types
- tsconfig base moved to config/ (no root tsconfig.json); deno types -> types/deno.d.ts; web drops baseUrl
- server: Deno.serve entry with graceful shutdown, --role migrate|preflight|probe, WEB_ROOT static SPA hosting;
  drop @hono/node-server, tsx, esbuild build.mjs
- logger: redact child() bindings and serialize err by message/stack; sync stdout destination; tests
- relative imports get explicit extensions (sites/mp-sync paths left for follow-up)
- crash-recovery tests spawn `deno run -A`; tasks tests use unique per-file schemas
- single-container Dockerfile + entrypoint (migrate -> preflight -> api+worker watchdog), docker-compose.yml
- docs: README, runbook, ADR-001 superseded, plan §1 validation row

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-09-27 18:53:06 +08:00
rootandClaude Fable 5.1 cc8b3a7df4 feat(db,contracts): full v0.1 data model migrations and shared zod contracts (W1-1)
- migrations 0002-0012 split by domain (foundation/instances/content/identity/
  sources/assets/policy/execution/agent/ops/organization): 41 tables, uuid pk,
  created_at/updated_at + touch trigger, text+CHECK status sets, pgcrypto/pg_trgm
- key constraints: source_release(site_id, site_release_id), observation(feed_id, guid),
  external_identity accepted partial unique, download_task(instance_id, torrent_hash),
  one active acquisition_plan per slot, one active path_binding per entity/season,
  asset_location root+path uniqueness, organization_record change_seq/change_xid cursor
- secret table (ciphertext/nonce/kdf) + secret_ref columns; outbox dispatch fields (ADR-002)
- src/types.ts generated from the migrated schema (scripts/gen-types.ts), drift-checked in tests;
  int8 -> number, date -> string parsers; migrator accepts migrationTableSchema
- contracts: Row/Create/Update schemas per domain, public organization-record API
  (changeSeq cursor), GlobalSettings, CapabilityDescriptor + 14 capability IO schemas,
  typed SSE events, Api endpoint table
- tests: migration up/down round trip on real PG in an isolated schema, constraint checks,
  contracts<->db column/enum sync, positive/negative parse for every exported schema

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-09-27 18:11:37 +08:00