Commit Graph
28 Commits
Author SHA1 Message Date
rootandClaude Opus 5.5 40dd0e69e4 feat(browser): ship camofox v1.17.0 in compose, verified on a real instance
- compose: camofox service on by default (internal only, volume umt-camofox,
  healthcheck, CAMOFOX_REPLICAS=0 to skip); app depends on it with
  service_started/required=false and auto-registers camofox-env from
  BROWSER_PROVIDER_URL / BROWSER_ACCESS_KEY / BROWSER_ENABLED
- docker/camofox/Dockerfile: builds upstream tag v1.17.0 (389c996); fixes the
  upstream Dockerfile (missing postinstall.js before npm ci, sh -c not
  forwarding SIGTERM so logins were never persisted on restart)
- adapter fixes from the real container: coordinate click via evaluate +
  selector, focus typing with mode=keyboard, password values redacted from
  ARIA snapshots/extract, downloads field names, 410 after browser restart
  treated as lost tab; navigate reopens a tab in the same session
- verified-capability table keyed by upstream version (1.17.0 all ten);
  other versions stay unverified; probe reports auth failures
- settings: browser providers block (create/edit/delete/probe, capability
  table); API POST/PATCH/DELETE /browser/providers, POST .../probe
- scripts/browser-verify: umtbr stack + mock NexusPHP run of REQ 4.7.4 items
- docs: runbook browser §1/§7 (real-instance record), deploy §2.1

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-29 00:05:21 +08:00
rootandClaude Opus 5.5 b4c9b1ead2 fix(system): retired heartbeat rows no longer alarm on status page
- GET /system/health splits processes into current instances (hosts with
  a beat in the last 10 min, newest per host+kind, not stopped) and
  retiredProcesses[]; 45s stale and level only consider current ones
- heartbeat writer marks stopped_at on graceful exit (migration 0043),
  upsert clears it; pure api role now stops its writer on SIGTERM
- scheduled job system.heartbeat_cleanup (maintain, 1h) deletes rows
  without a beat for 24h and samples older than 7 days
- status page: per-role alarms ("io worker 45 秒没有心跳"), collapsible
  retired instances; glossary/runbook updated
- bump all members to 0.2.2, CHANGELOG [0.2.2]

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-28 23:31:21 +08:00
rootandClaude Opus 5.5 946fca5e55 feat(cache): PG kv_cache ported from MP TTLCache, TMDB request and recognize caches
Migration 0042_cache adds UNLOGGED kv_cache. PgCache mirrors MP
runtime/cache.py (region, per-key TTL, TLRU maxsize eviction, @cached as
cachedCall); only the storage moves from memory to PG. TMDB request
responses (MP TMDb.request @cached) and recognize results (MP TmdbCache:
same key, {id:0} negative cache with EMPTY_RESULT_CACHE_TTL, type-conflict
checks, backfill after manual assignment) live in kv_cache, shared by all
processes. Settings cache.tmdbTtlHours / tmdbMaxSize / emptyResultTtlSec
map to META_CACHE_EXPIRE / CONF.tmdb / EMPTY_RESULT_CACHE_TTL. TMDB
details now also store posterPath / backdropPath.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-28 22:26:29 +08:00
rootandClaude Opus 5.5 61f585a707 feat(sites): cross-process site rate limit via PG token bucket
Migration 0041_rate_limit adds rate_limit_bucket. PgTokenBucket (@umt/db)
refills and takes tokens under SELECT ... FOR UPDATE on the DB clock and
polls with jitter within a wait budget; penalize() carries Retry-After.
Site adapters get createSharedSiteRateLimiter (key site:<id>), shared by
api / worker / worker:agent, with an in-process serial queue in front and
a local fallback when the bucket is unavailable.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-28 21:51:32 +08:00
rootandClaude Opus 5.5 c17d4b90bd feat(browser): complete camofox BrowserProvider with tabs, interaction, download capture and control lease
Contracts for browser API and browser.* agent capabilities; adapter implements
close/list/restore sessions, tabs + reconcile, history, click/type/select/scroll/press,
extract, waitForDownload (staged via DownloadSink, sha256), stale-ref guard and
automation/human actor check. testkit mock camofox gains an interactive site
(login redirect, captcha, click-driven page changes, downloads, PNG screenshots).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-28 20:17:46 +08:00
rootandClaude Opus 5.5 abc69886ea feat(contracts,server,web): files G6–G8 fields and identify S-7 contract
- G6: FileRoot.pendingCount/pendingPartial for download roots (same rule as the
  unidentified filter, recursive, cached 5 min, invalidated by files.changed)
- G7: FileEntry.parsedHint from parseReleasePath (file + parent dirs); the web
  subline uses it and no longer parses names itself
- G8: FileIdentifyCandidate.posterUrl (TMDB poster_path via adapter and
  catalog.lookup, local entities from metadata.posterPath); thumbnails in the
  identify panel
- S-7 contract: FilesIdentifyResult.remoteTaskId/remotePending,
  MediaServerInventory.partial, EventType identify.completed with
  IdentifyCompletedPayload; identify panel (useIdentifyLive) and organize wizard
  merge remote results by root + path, buffering events that beat the response

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-28 20:09:29 +08:00
rootandClaude Opus 5.5 d6fca25679 feat(adapters): NexusPHP takelogin login, VIP and seeding page limit for user data
loginNexusPhp (captcha / two-factor / bad credentials / blocked / Cloudflare
classification, TOTP), SiteUserInfo.vip via profile.vip_css or umt.vipLevels,
maxSeedingPages option, MTorrent parser honours apiUrl; testkit mock NexusPHP
site gains takelogin with login guards and user detail pages.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-28 15:18:30 +08:00
rootandClaude Sonnet 5 623aa0d3b7 chore(release): v0.2.0 version and changelog
Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-09-28 13:53:33 +08:00
rootandClaude Opus 5.5 7daf544bd9 fix(fs): fsworker always reports phase completion progress
The throttle could drop the final copy heartbeat when it landed inside the
interval (fsproxy.test saw 6291456 instead of 6291461 under ci load). Phase
completion (bytes >= total) is now sent immediately like phase changes.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-28 00:56:31 +08:00
rootandClaude Opus 5.5 2e3c2098fd docs: S-9 sync READMEs and porting notes with W5 code
- contracts README lists files.ts / auth.ts / browser.ts / redact.ts and auth:"session"
- adapters README fs row: FsProxy and the ./fsworker export
- server README: file-manager module, worker:agent and internal fsworker roles
- moviepilot-rust-transfer §7: compiled-binary fsworker and trash_item done (W5-E)
- 04 S9 row marked fixed (W6-B TOTP entry)

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-28 00:50:29 +08:00
rootandClaude Opus 5.5 2ddaf7bfa6 fix(fs): S-10 hang test hook requires UMT_FS_TEST_HOOKS=1; preflight warns when set
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-28 00:48:45 +08:00
rootandClaude Opus 5.5 47bd04aec0 fix(adapters,security): S-4 rss/browser redaction use the shared contracts key table
- contracts: isSensitiveQueryParam (whole-name match on SENSITIVE_QUERY_PARAMS
  or isSensitiveKey), the single source for query credential names
- rss/redact: drop local SENSITIVE_QUERY_KEYS; param checks go through
  isSensitiveQueryParam (+ caller extras); REDACTED re-exported from contracts
- browser/redact: CREDENTIAL_PARAM built from SENSITIVE_QUERY_PARAMS, JSON keys
  also checked with isSensitiveKey; keeps only its own rules (path passkey,
  cookie names, a11y password fields); remove stray regex statement
- redact-consistency.test: 24 credential spellings (pass_key, c_secure_pass,
  refresh_token, password, uid_key, auth, ...) masked by contracts, rss and
  browser alike; plain params untouched; rss extract/unredact round-trip

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-28 00:45:56 +08:00
rootandClaude Opus 5.5 3641ab9573 feat(server,fs): W5-E production fsworker role, settings auto-investigate mode, replace old file to trash_item
- main.ts: `--role fsworker` dispatched before the rest of the module graph is
  loaded (other imports became dynamic); runs fsWorkerMain from the new
  `@umt/adapters/fsworker` export (~150 ms startup in the compiled binary)
- fsproxy: default worker command picked per runtime (deno CLI -> `deno run
  fsworker.ts`, compiled binary -> `<execPath> --role fsworker`); stats expose
  workerKind; UMT_FS_TEST_HANG_AFTER_BYTES acceptance switch for containers
- compile: --allow-run adds /app/umt-server (spawn self); start task allows deno
- organize: replaced old file moved into the file-manager trash layout
  (<root>/.umt-trash/<day>/<uuid>/<rel>) and registered in trash_item, ledger
  result follows; organize.transfer log carries fsproxy stats
- settings: autoInvestigateMode (investigate/recover) + max recover attempts
- deploy.md: container verification (subprocess self, TIMEOUT -> SIGKILL x3)

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-27 23:50:10 +08:00
rootandClaude Opus 5.5 2e2d56184c feat(fs): W4-E4 fsproxy hang isolation, partial sweep, replace-with-verify, TIMEOUT retry
- adapters/fs: FsProxy + fsworker run each file op in a killable subprocess;
  progress-stall watchdog (FS_HANG_TIMEOUT_MS) for copy/hash, total timeout for
  quick ops; SIGTERM -> SIGKILL -> FsError('TIMEOUT'); stat/checkRoot timeouts
  never read as missing
- copyWithVerify: onExisting conflict/skip/replace (hardlink backup + atomic
  rename, old file to .umt-trash or deleted), sweep stale partials before write,
  document why permissions/owner are not copied
- filesMatch/filesMatchSha256 throw FsError on read failure instead of
  match=false; crossDeviceMove skips redundant re-hash after sha256 copy
- FsErrorCode TIMEOUT + fsErrorClass; organize executeRecord routes fs ops
  through fsproxy, retries TIMEOUT/IO up to FS_TRANSFER_MAX_ATTEMPTS then opens
  a decision without deleting the record; reconcile errors no longer reported
  as EEXIST
- domain: subtitle .default marker (ORGANIZE_DEFAULT_SUBTITLE_LANG) and
  findOtherVersions (same season/episode/part, video only) as upgrade
  disposal candidates
- scripts/bench/fs-bench.ts and benchmark record in porting doc

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-27 23:15:05 +08:00
rootandClaude Opus 5.5 36dc048e89 feat(parser,sites): W5-D port title/zhconv/site parsing gaps found by MP differential
Title recognition (parseRelease / parseReleasePath):
- anime prefix tags (★N月新番, [国漫], 【22年日剧】), category lead bracket, `A / B` cn aliases
- `Name [01]`, `[TV 04]`, `#13`, `名称 12.mkv`, `S01 13-14`, `E13-14` episodes
- lead bracket as title vs group, trailing [Group], YTS.MX/Hall_of_C style groups, 字幕组 suffix
- `[dot.release.name][…]` unbracketing, `Season 2017` as year, leading date not airDate,
  date-only parent dir does not merge as TV
- original disc: Blu-ray + AVC/HEVC/VC-1/MPEG-2 without x26x/H.26x -> BluRay-Disc
- `{[tmdbid=;type=;s=;e=]}` explicit block, bangumi/anilist ids
- bare [1080]/[5.1], DTS-HD MA5.1, £ multi-group, 合集, pinyin title -> cn from subtitle,
  streaming platform table, CHS/CHT/GB/BIG5 in subtitle
zh-map: add 迴滾魷瑯臥駭; document one-to-many chars kept out.
Sites: tbody branch + header-row tightening via movieclaw-import rowSelector (18 definitions),
row-level case/hr/labels include the row itself, subtitle branch (searchType: "subtitles"),
parseSizeMp banker's rounding.
Differential snapshots regenerated; MP test sample sources registered in ledger.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-27 23:09:14 +08:00
rootandClaude Opus 5.5 ceff9cf51f feat(ao): W4-E3 manual organize guard, series completion, engine retry, v0.1.0
- POST /organization-records: sourcePaths must resolve (realpath) under a
  registered library_root or downloader path-mapping host path; `..` and
  symlink escapes rejected (400), dryRun included. Move requires an admin
  session, or an API key with an automation_grant covering the entity or a
  matching one-time approval (consumed on use), else 403. Manual Move from a
  managed download_task with unsatisfied H&R returns 409 and opens a
  decision; never silently downgraded to Copy.
- Series slot completion: total episode count is unknown while units are
  provisional, so the slot is not satisfied and the subscription keeps
  tracking; coverage is registered per actual file (creating provisional
  episode units), and already_in_library files count as verified presence.
- POST /engine/runs/{id}/retry (admin): re-enqueue by original task name and
  raw input with idempotency key retry:<id>:<n>, audited; retry button on
  the run detail page.
- E7 regular series e2e, E8 AT24 tag removal / task disappearance e2e.
- All workspace members and root set to version 0.1.0; CHANGELOG.md.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-27 22:47:11 +08:00
rootandClaude Opus 5.5 591cc78c7d test(porting): differential tests against MoviePilot Rust-accelerated parsers
Snapshot differential tests (known deviations registered, current state passes):
- title/path recognition: 216 samples vs MP MetaInfo/MetaInfoPath
- filter rules: 42 rules x 28 torrents vs MP FilterModule
- RSS/indexer parsing: 15 feeds, 72 indexer samples
- zhconv: 69 samples
Generator: scripts/mp-sync/gen-differential.py; report: docs/porting/mp-rust-logic-parity.md

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-27 22:44:11 +08:00
rootandClaude Opus 5.5 f48e1aac96 fix(security): W4-E1 approvals admin-session only, shared redact table, TMDB key redaction, prod mock gate
- POST /approvals and PATCH /approvals/{id} require a console admin session
  (EndpointDef auth:"session"); API keys incl. admin scope get 403, so an
  organize.execute key can no longer approve its own plan (S1)
- tests for single-use approval checks: expired, revision/payload hash
  mismatch, consumed, revoked, pending/rejected, plan mismatch; re-approve
  after plan change succeeds once and cannot be replayed (S4)
- one redact key table in @umt/contracts shared by server logger and agent
  redactForModel: rsskey, torrent_pass, authkey, downhash, c_secure_pass,
  passkey variants, x-api-key, cookie headers, nested objects/arrays (S2)
- TMDB network errors: message and cause redacted via redactUrl (S3)
- web Agent ?mock=1 only in DEV builds or when pi.provider is fake (S7)
- EndpointDef auth annotations for /auth/*, asserted against
  PUBLIC_AUTH_PATHS (S10); doc drift fixes (S11)

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-27 22:24:44 +08:00
rootandClaude Opus 5.5 f9c20c67dc perf(fs): 1MiB stream chunks, full sha256 via sha256sum subprocess, walk cap
- copy streams use 1MiB highWaterMark
- forced full sha256 uses Deno.Command(sha256sum), falls back to in-process stream;
  method recorded as hashVia and counted
- per-file organize.transfer timing log
- ORGANIZE_WALK_MAX_FILES caps manual path walks; dryRun returns truncated

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-27 22:12:16 +08:00
rootandClaude Opus 5.5 effaa32b70 feat(browser): W4-C BrowserProvider contract and camofox-browser adapter skeleton
- contracts/browser.ts: provider kind, per-capability {supported, verified} flags,
  session/tab/control-lease rows, NavigationBudget, redacted BrowserSnapshot,
  DownloadArtifact with blob/secret refs, BrowserErrorKind
- adapters/browser: BrowserProvider interface, camofox client (health, capabilities
  static + /openapi.json probe, openSession/navigate/snapshot/screenshot; rest
  not_implemented for M2), none provider, redactSnapshot, profile serial lock,
  domain allowlist budget; Bearer access key via BrowserSecretResolver
- testkit/browser: mock camofox server (auth, fixtures with secrets, faults)
- db 0023: browser_provider/session/tab/control_lease with one-active-lease and
  one-live-session-per-profile partial unique indexes; types regenerated
- docs/design/browser-provider.md: contract, camofox OpenAPI 1.17.0 mapping,
  unverified items and M2 prototype checklist; Playwright is not a product browser

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-27 21:39:29 +08:00
rootandClaude Opus 5.5 ef3998767c style(adapters): throw instead of returning never in assertSafeToDeleteData
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-27 19:30:55 +08:00
rootandClaude Opus 5.5 fa9221086b feat(adapters,testkit): guarded qB data deletion (B7) and mock qB file materialization (B8)
adapters: assertSafeToDeleteData({contentPath, libraryRoots, importVerified}) rejects
paths inside/equal to/above any library root, relative or root paths, empty roots and
unverified imports; QbClient.delete({deleteFiles:true}) requires a per-hash
SafeDeleteCheck and throws DownloaderError('unsafe_delete') before sending anything.

testkit: mock qB complete(hash, {files, writeFiles}) writes random-content files under
save_path (opt-in, size-capped, traversal-safe); deleteFiles=true removes them.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-27 19:22:56 +08:00
rootandClaude Fable 5.1 c33034a289 fix(adapters): crossDeviceMove verifies pre-existing target with sha256 before deleting source
With the default hash="size", a target that already existed with the same
size but different content was treated as "already copied" and the source
was deleted (data loss; REQ §2.1 cleanup needs verified import). The
pre-existing-target comparison now always uses sha256; test covers the
same-size/different-content case with default options.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-09-27 19:01:46 +08:00
rootandClaude Fable 5.1 304feea141 chore(deno): migrate toolchain from Node 24 + pnpm to Deno 2.9.7 + TypeScript 7.0.2 (W1.5)
- root deno.json workspace (nodeModulesDir auto), npm versions registered once in imports, deno.lock;
  remove pnpm-lock/pnpm-workspace/root package.json; members keep minimal package.json (name/type/exports)
- tasks: dev, typecheck (tsc 7.0.2 per member + deno check), lint/fmt (biome, useImportExtensions),
  test (vitest via deno), check:tokens, build (web + deno compile), ci, db:migrate/reset, gen:types
- tsconfig base moved to config/ (no root tsconfig.json); deno types -> types/deno.d.ts; web drops baseUrl
- server: Deno.serve entry with graceful shutdown, --role migrate|preflight|probe, WEB_ROOT static SPA hosting;
  drop @hono/node-server, tsx, esbuild build.mjs
- logger: redact child() bindings and serialize err by message/stack; sync stdout destination; tests
- relative imports get explicit extensions (sites/mp-sync paths left for follow-up)
- crash-recovery tests spawn `deno run -A`; tasks tests use unique per-file schemas
- single-container Dockerfile + entrypoint (migrate -> preflight -> api+worker watchdog), docker-compose.yml
- docs: README, runbook, ADR-001 superseded, plan §1 validation row

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-09-27 18:53:06 +08:00
rootandClaude Opus 5.5 d9f2af2f3c feat(adapters): port MoviePilot open site framework (7 spiders, 20 user parsers) + MovieClaw v0.27.0 site definitions + mp-sync tools (W1-3c)
- sites/spiders: Haidan/HDDolby/RousiPro/SunnyPT/TNode/TorrentLeech/Yema (+mTorrent wrapper), specialized SiteAdapter, indirect download resolved at download time
- sites/user-parsers: SiteParserBase + all 20 MP SiteSchema parsers (lxml XPath via xpath+xmldom), refreshSiteUserData with schema fallback
- sites/definitions: zod SiteDefinition schemas, 23 MovieClaw v0.27.0 (MIT) definitions via scripts/mp-sync/movieclaw-import.mjs, registry getSiteDefinition/listSiteDefinitions, NOTICE
- scripts/mp-sync: ledger.json (upstream sha256), check.mjs, build-ledger.mjs; docs/porting/mp-sync-ledger.md
- license-and-provenance: add copied-mit kind, re-anchor MovieClaw rows to v0.27.0

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-27 18:39:27 +08:00
rootandClaude Fable 5.1 a6d049db81 feat(adapters): RSS and PT site adapters ported from MoviePilot v3 + mocks (W1-3b)
- rss: port of MP RssHelper._parse_impl/get_rss_link semantics with safe
  XML (entities off, DTD/ENTITY rejected, size/item caps), ETag/Last-Modified,
  backoff + Retry-After, circuit breaker with manual reset, passkey redaction,
  overlap-window GUID+digest dedupe
- sites/spider: TS port of MP SiteSpider (indexer schema, search URL builder,
  field DSL, filters, Jinja templates via nunjucks, login-page detection);
  audiences definition hand-written as JSON; raw category id preserved;
  shared per-site token bucket
- sites/mteam: port of MP MTorrentSpider (params, result mapping, promo
  factors, genDlToken flow) without embedding the API key in enclosures;
  business code/message checks
- testkit: mock RSS server, mock NexusPHP site, mock M-Team API
- docs: W1-3b migration mapping appended to porting ledger

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-09-27 18:10:34 +08:00
rootandClaude Fable 5.1 ef2e504a94 feat(adapters): qBittorrent, filesystem, TMDB and Emby adapters with testkit mocks (W1-3a)
- qbittorrent: WebUI API v2 client (cookie/API-key auth, add/info/files/filePrio,
  pause/resume with 5.x stop/start fallback, tags, delete with explicit deleteFiles,
  maindata), error classes, idempotent submit with local infohash (own bencode)
  and post-timeout hash/probe-tag lookup; state mapping per MoviePilot
- fs: copyWithVerify (tmp + fsync + size/sha256/sample + no-clobber commit),
  moveSafe (EXDEV surfaced), resumable crossDeviceMove, statSafe/checkRoot with
  root sentinel, budgeted incremental scan with cursor, ffprobe, statfs usage
- tmdb: multi-language search/details/seasons, LRU+TTL cache with coalescing,
  rate-limit backoff, MoviePilot match chain (name normalization, year +-1)
- emby: Emby/Jellyfin items/seasons/episodes/MediaSources, failures -> unknown
- testkit: mock qB / TMDB / Emby servers, torrent builder, temp-tree fixtures
- docs: W1-3a porting table appended to downloader-and-sites ledger

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-09-27 18:07:48 +08:00
umtandClaude Fable 5.1 b22c440139 chore: scaffold monorepo and migrate uhd_web design system (W0-A)
- pnpm workspace: apps/{web,server}, packages/{contracts,db,domain,adapters,tasks,agent,testkit}
- server: Hono + zod-openapi, API Key auth with scopes, redacting pino logger, SSE event bus with replay, worker heartbeat
- db: Kysely + pg, file migrator, 0001_init (settings/api_key/audit_event/outbox_event)
- web: React Router 7 SPA console shell with 12 workspace placeholders, uhd_web tokens/ui/guard scripts
- docs: ADR-001 (Node over Deno), dev runbook, design system rules

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-09-27 17:36:26 +08:00