Commit Graph
6 Commits
Author SHA1 Message Date
rootandClaude Opus 5.5 cabd772243 fix(sites): normalize HHClub RSS items, complete detail page parsing (HH-2)
- RSS: split NexusPHP "[category]name[subtitle][size][uploader]" titles in the
  site adapter layer (category id/name/media type, clean title, subtitle,
  size); snapshot fields win on merge; lazily renormalize old RSS rows and
  re-emit release.observed; unknown list fields show an em dash
- Detail: HHClub section layout (douban info block as description,
  screenshots section, IMDb/TMDB from MediaInfo lines, upload time field,
  on-demand viewfilelist.php once); site CDN screenshots get a Referer
- Site request count now includes site-bound RSS fetches
- Origin view shows "自动化" when idle and "Agent" only after recent
  automation actions (lastAutomationAt); agent step titles no longer get
  squeezed to a sliver by long notes

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-29 17:30:25 +08:00
rootandClaude Opus 5.5 04cffcd1f1 feat(sites): on-demand site detail page content (INT-2)
- SiteReleaseDetail.page: description, MediaInfo/BDInfo, screenshots,
  IMDb / Douban / TMDB ids, file list, tags, uploader, plus a parse probe
  (labels found) for layout-change diagnosis. Generic NexusPHP rules
  (field label -> next sibling) starting from the MovieClaw detail
  selectors; site overrides under umt.detail.page (HHClub: div labels).
  M-Team builds the same structure from descr BBCode + mediainfo.
- Fetched only on demand (drawer open without cache, "复核详情",
  release.inspect); content in kv_cache region site.detail for the
  site detail cache period (>= 24 h); within the window the site is not
  requested again. External ids persist in source_release.raw._detail,
  which list snapshots do not overwrite.
- Screenshots only as proxy paths /api/v1/img/site-detail/{rid}/{i}:
  public-address check on every hop, no site credentials, magic-byte
  image check, 8 MB cap. Poster proxy shares the same helper.
- Identity: site ids count as explicit-id evidence; when TMDB is
  queried, a TMDB id goes straight to details and an IMDb id to
  /find (no title search on a hit). Douban only for local recall.
- Release drawer "站点详情页" section; fixtures and tests for HHClub div
  layout, rowhead layout, BBCode, cache/proxy/identity end to end.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-29 16:21:17 +08:00
rootandClaude Opus 5.5 6e9e8efec2 fix(sites): HHClub parsing, resource list layout and request restraint (re-landed)
Re-lands 901b01d, which was reverted wholesale by 28974d1 (BR-2 committed
from the shared index). The agent UI files 901b01d had rolled back are
left at their AGUI-2 versions; contracts, routes, db types and the runbook
part already reached main through later commits, so this carries the rest.

HHClub (憨憨) was checked read-only against a real account (one detail
request; list, user data and health came from the product's own fetch).

Parsing fixes:
- list titles carried the site's new-torrent marker "[新]" (58 of 100
  rows; the detail page title has none); stripped with a replace filter
- refreshing a detail overwrote seeders / leechers / snatched / category
  name / H&R with null because the HHClub div detail page lacks them;
  detail values now fall back to the list snapshot (mergeDetailOverList),
  and an unknown detail promotion keeps the list promotion
- level "贵宾" is VIP (umt.vipLevels)
- sanitized minimal fixture HHANCLUB_LIST_PAGE_2026 + tests; README marks
  HHClub as verified 2026-09-29
Size, counts, category ids/names, +08:00 time and subtitle were correct.

Request restraint:
- site definitions may carry umt.defaults; merged with settings.sites
  (longer intervals, fewer pages): new settings defaultSnapshotMaxPages 1,
  defaultHealthCheckIntervalSec 6 h, defaultDetailCacheSec 24 h; default
  burst 2. HHClub: snapshot 60 min page 1 only, health 6 h, search
  concurrency 1, detail cache 24 h, 1 token / 3 s burst 2
- periodic / manual snapshots are capped to that page count; scheduled
  health checks skip sites checked within the interval; detail refresh is
  served from cache inside the window; live searches on one site queue
- every rate-limiter token is counted per site per hour
  (site_request_count, migration 0046); GET /sites/{id}/request-stats;
  site overview and resource list show requests in the last 24 hours

Resource list: two-layer rows. Main line: title + spec tags (episodes,
resolution, source, codec, HDR, group, 国语/中字, site tags) + promotion
with time left + H&R. Second line: subtitle (one line, ellipsis) and
fixed-width category / size / seed-leech-snatched / upload time
(relative, absolute on hover) columns, stale badge, inventory overlay;
folds into a card on phones.

Also aligns identity_candidate.normalized_title order in db types with
the migrations (schema drift test).

Takes effect after the stack is rebuilt (migration 0046).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-29 16:00:34 +08:00
rootandClaude Opus 5.5 28974d1e1a feat(browser): unify site credentials with browser profiles (BR-2)
- Inject the site account Cookie (manual + auto-login session, session wins)
  into the camofox profile before the first tab opens; password accounts
  without a session run refresh-login first; API Key sites are skipped.
- Cookie expired after injection -> credential.status=expired; cookie
  accounts go to waiting/login_required ("log in once, applies site-wide").
- Write back site-domain cookies from the browser on handover, session close,
  every 10 min of activity, and from refresh-login when the stored cookie is
  invalid (verified before saving). user_info.cookieOrigin records
  source=browser; source shown as manual/login/browser in the origin view.
- New camofox plugin umt-cookies (GET/POST /umt/sessions/:userId/cookies
  under the access key); profile dir 0700/files 0600 in the desktop image.
- Single entry: sources-identity/service/credentials.ts.
- Real check: HHClub cookie injected into a fresh profile -> home page
  logged in (logout link present), 5 cookies read back.

Requires rebuilding umt-browser-desktop (docker compose build camofox &&
docker compose up -d camofox) to take effect.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-29 15:25:07 +08:00
rootandClaude Opus 5.5 901b01d1a1 fix(sites): HHClub parsing, resource list layout and request restraint
HHClub (憨憨) was checked read-only against a real account (one detail
request; list, user data and health came from the product's own fetch).

Parsing fixes:
- list titles carried the site's new-torrent marker "[新]" (58 of 100
  rows; the detail page title has none); stripped with a replace filter
- refreshing a detail overwrote seeders / leechers / snatched / category
  name / H&R with null because the HHClub div detail page lacks them;
  detail values now fall back to the list snapshot (mergeDetailOverList),
  and an unknown detail promotion keeps the list promotion
- level "贵宾" is VIP (umt.vipLevels)
- sanitized minimal fixture HHANCLUB_LIST_PAGE_2026 + tests; README marks
  HHClub as verified 2026-09-29
Size, counts, category ids/names, +08:00 time and subtitle were correct.

Request restraint:
- site definitions may carry umt.defaults; merged with settings.sites
  (longer intervals, fewer pages): new settings defaultSnapshotMaxPages 1,
  defaultHealthCheckIntervalSec 6 h, defaultDetailCacheSec 24 h; default
  burst 2. HHClub: snapshot 60 min page 1 only, health 6 h, search
  concurrency 1, detail cache 24 h, 1 token / 3 s burst 2
- periodic / manual snapshots are capped to that page count; scheduled
  health checks skip sites checked within the interval; detail refresh is
  served from cache inside the window; live searches on one site queue
- every rate-limiter token is counted per site per hour
  (site_request_count, migration 0046); GET /sites/{id}/request-stats;
  site overview and resource list show requests in the last 24 hours

Resource list: two-layer rows. Main line: title + spec tags (episodes,
resolution, source, codec, HDR, group, 国语/中字, site tags) + promotion
with time left + H&R. Second line: subtitle (one line, ellipsis) and
fixed-width category / size / seed-leech-snatched / upload time
(relative, absolute on hover) columns, stale badge, inventory overlay;
folds into a card on phones.

Also aligns identity_candidate.normalized_title order in db types with
the migrations (schema drift test).

Takes effect after the stack is rebuilt (migration 0046).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-29 15:18:15 +08:00
rootandClaude Opus 5.5 d9f2af2f3c feat(adapters): port MoviePilot open site framework (7 spiders, 20 user parsers) + MovieClaw v0.27.0 site definitions + mp-sync tools (W1-3c)
- sites/spiders: Haidan/HDDolby/RousiPro/SunnyPT/TNode/TorrentLeech/Yema (+mTorrent wrapper), specialized SiteAdapter, indirect download resolved at download time
- sites/user-parsers: SiteParserBase + all 20 MP SiteSchema parsers (lxml XPath via xpath+xmldom), refreshSiteUserData with schema fallback
- sites/definitions: zod SiteDefinition schemas, 23 MovieClaw v0.27.0 (MIT) definitions via scripts/mp-sync/movieclaw-import.mjs, registry getSiteDefinition/listSiteDefinitions, NOTICE
- scripts/mp-sync: ledger.json (upstream sha256), check.mjs, build-ledger.mjs; docs/porting/mp-sync-ledger.md
- license-and-provenance: add copied-mit kind, re-anchor MovieClaw rows to v0.27.0

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-27 18:39:27 +08:00