pixel_pop/UHD_Media_Tool
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refactor(core): migrate to Bun and converge acquisition policies and Agent boundaries

Browse Source
This commit is contained in:
root committed 2026-10-13 14:37:03 +08:00
1 parent 3f1c6d02f6
commit 27c0f27434
268 files changed
+7468 -7852
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.bun-version
+1
View File
@@ -0,0 +1 @@
1.4.3
.dockerignore
+2 -1
View File
@@ -1,4 +1,4 @@
# 构建上下文只需要源码与锁文件;依赖在镜像内由 deno install --frozen 按 deno.lock 重装
# 构建上下文只需要源码与锁文件;依赖在镜像内由 bun install --frozen-lockfile 按 bun.lock 重装
.git
**/node_modules
**/dist
@@ -14,6 +14,7 @@ docs
!.env.example
*.md
!apps/**/README.md
!packages/agent/skills/**/*.md
data
.local
.browser-verify-out
CLAUDE.md
+15 -14
View File
@@ -1,34 +1,35 @@
# UHD Media Tool
影视自动化工具：站点 / RSS 获取 → 识别 → 下载（qB）→ 整理入库 → 刮削，带 Agent。后端 Deno + Hono + Kysely（PostgreSQL），前端 React Router 8 + Vite，同一个 workspace。业务逻辑以 MoviePilot v3 为参考（`docs/porting/`）。
影视自动化工具：站点 / RSS 获取 → 识别 → 下载（qB）→ 整理入库 → 刮削，带 Agent。后端 Bun 1.4.3 + Hono + Kysely（PostgreSQL），前端 React Router 8 + Vite，同一个 workspace。业务逻辑以 MoviePilot v3 为参考（`docs/porting/`）。
面向团队内部使用，目前只有本机测试环境：**避免过度设计和过度防御**，只处理已确认会影响正常使用的问题。回复和文档用中文。
## 常用命令
```sh
deno task dev:up # 开发库(127.0.0.1:55433)+ 迁移到最新;拉代码后先跑
deno task dev # API 4390 + web 5180(4380 是本机部署版,别占)
deno task test:all # 全量测试,逐包跑完汇总;连不上测试库直接失败
deno task typecheck # tsc + deno check
deno task lint # biome;deno task fmt 会写文件,只对自己改的路径用
deno task ci # 提交前完整检查(含 web 构建与编译产物自检)
deno task release # 测试环境发布,见下
bun run dev:up # 开发库(127.0.0.1:55433)+ 迁移到最新;拉代码后先跑
bun run dev # API 4390 + web 5180(4380 是本机部署版,别占)
bun run test:all # 全量测试,逐包跑完汇总;连不上测试库直接失败
bun run typecheck # tsc 7
bun run lint # biome;bun run fmt 会写文件,只对自己改的路径用
bun run ci # 提交前完整检查(含 web 构建与编译产物自检)
bun run release # 测试环境发布,见下
```
REST 端点定义按领域分在 `packages/contracts/src/api/*.ts`，`api.ts` 按顺序合并成 `Api`（顺序即路由注册顺序）。整理服务按阶段分在 `organize-plan / -exec / -finalize / -records / -types.ts`，外部仍从 `organize.ts` 引用。
只改一个包时可在包目录跑 `deno run -A npm:vitest run <文件>`；web 包用 `../../node_modules/.bin/vitest run`（Node）。改迁移后跑 `deno task gen:types` 重新生成 `packages/db/src/types.ts`，contracts 枚举顺序与数据库保持一致（`packages/contracts/test/db-sync.test.ts` 会查）。
只改一个包时可在包目录跑 `bun x --bun vitest run <文件>`；web 包用 `../../node_modules/.bin/vitest run`（Bun）。改迁移后跑 `bun run gen:types` 重新生成 `packages/db/src/types.ts`，contracts 枚举顺序与数据库保持一致（`packages/contracts/test/db-sync.test.ts` 会查）。
## 约定
- **先提交再部署**。发布只走 `deno task release`：工作区不干净直接拒绝；版本取根 `deno.json`（所有成员包同一版本）；`CHANGELOG.md` 必须先有 `## [x.y.z]` 条目；脚本打 tag `vx.y.z`、构建镜像、只替换 app 容器（启动时自动迁移）、核对健康接口版本。测试环境不需要备份、在途任务清单、源码指纹等重流程。
- 工程设计遵循 `docs/runbook/dev.md`「工程设计与适度抽象」:业务服务提供稳定入口,路由与能力共用；及时提取已有稳定的共用机制,执行策略和业务副作用保持显式。状态由对应业务模块维护,配置在模块入口解析；不提前创建通用框架。
- **先提交再部署**。发布只走 `bun run release`：工作区不干净直接拒绝；版本取根 `package.json`（所有成员包同一版本）；`CHANGELOG.md` 必须先有 `## [x.y.z]` 条目；脚本打 tag `vx.y.z`、构建镜像、只替换 app 容器（启动时自动迁移）、核对健康接口版本。测试环境不需要备份、在途任务清单、源码指纹等重流程。
- 功能在分支上做，完成后合回 main；提交按功能拆，别把不相关的改动带进同一个提交。
- 说"检查通过"只认 `deno task test:all` / `deno task ci` 的实际输出；跳过的用例不算通过。
- 解析外部下载 / 上传的文件或字节(种子、字幕、压缩包、图片等)一律走沙箱任务 `runSandbox`(`apps/server/src/sandbox/`,见 ADR-005),不在主进程里直接解析。
- 说"检查通过"只认 `bun run test:all` / `bun run ci` 的实际输出；跳过的用例不算通过。
- 解析外部下载 / 上传的文件或字节(种子、字幕、压缩包、图片等)一律走沙箱任务 `runSandbox`(`apps/server/src/sandbox/`,见 ADR-005 与 ADR-007),不在主进程里直接解析。
- 新增 Agent 能力要登记到某个 Skill（`packages/agent/skills/*/SKILL.md`）和能力地图（`packages/agent/src/prompt/layers/capabilities.ts`），否则 agent 测试会红。
- 界面文案守 `docs/design/glossary.md` 的术语与标点（`deno task check:copy`）；状态色只表示状态（`StatusBadge` 带圆点，别拿来显示数量）。
- 主工作区不留常驻 dev 进程（`--watch` 会改写 `deno.lock`）；构建前确认 `deno.lock` 没被改。
- 界面文案守 `docs/design/glossary.md` 的术语与标点（`bun run check:copy`）；状态色只表示状态（`StatusBadge` 带圆点，别拿来显示数量）。
- 主工作区不留常驻 dev 进程；依赖安装使用 `bun install --frozen-lockfile` 保持锁文件一致。
## 测试
Dockerfile
+13 -13
View File
@@ -1,6 +1,6 @@
# 单容器镜像:一个 deno compile 出的 umt-server 二进制,同时承担 API(含 web 静态 SPA)与 worker。
# 单容器镜像:一个 bun build --compile 出的 umt-server 二进制,同时承担 API(含 web 静态 SPA)与 worker。
# PostgreSQL 不在本镜像内,见 docker-compose.yml。
# 构建:docker build -t uhd-media-tool . 运行时版本锁定 Deno 2.9.7(ADR-004)
# 构建:docker build -t uhd-media-tool . 运行时版本锁定 Bun 1.4.3(ADR-007)
# ffprobe 取自静态构建镜像(单个 static-pie 二进制,约 135 MB;Debian 的 ffmpeg 包要拉 200+ 个依赖、约 450 MB)
ARG FFPROBE_IMAGE=mwader/static-ffmpeg:9.0.2@sha256:7d9bdaaf887f7e6ce6151f67325c344074b5ff1fb75316011c3376503e449a7b
FROM ${FFPROBE_IMAGE} AS ffprobe
@@ -8,29 +8,27 @@ FROM ${FFPROBE_IMAGE} AS ffprobe
# 与 PG 18 服务端同主版本的备份工具,避免 Debian 默认 17 客户端拒绝导出 PG 18。
FROM postgres:18.6-trixie@sha256:5a5a84b19854a9ffaa54082c166ff4ec27473a361e496e5ea167f298f2da9722 AS postgres-tools
FROM denoland/deno:2.9.7@sha256:fa335acdf6b72106eda2cb6a8cb5f4187e7630e357467489db4b2e7352d5e432 AS build
FROM oven/bun:1.4.3 AS build
ARG APT_MIRROR=""
RUN if [ -n "$APT_MIRROR" ]; then sed -i "s|deb.debian.org|$APT_MIRROR|g" /etc/apt/sources.list.d/debian.sources; fi \
&& apt-get update && apt-get install -y --no-install-recommends build-essential libnode-dev && rm -rf /var/lib/apt/lists/*
WORKDIR /src
COPY . .
RUN deno install --frozen
RUN bun install --frozen-lockfile
ARG APP_VERSION=""
ENV APP_VERSION=${APP_VERSION}
RUN deno task check:tokens && cd apps/web && deno task build
# --node-modules-dir=none + --exclude-unused-npm:只嵌入 server 依赖图可达的 npm 包(约 110 MB,而不是整个 node_modules)
# 权限在编译时固定:读写不限路径(库根/下载目录由用户挂载,位置不固定;写保护靠 /app root 只读 + 非 root 运行);
# --allow-run 只放 ffprobe(入库探测)、sha256sum(强制全量哈希走子进程,coreutils 自带)与 /app/umt-server 自身
# (fsproxy 把单个文件操作放进 `umt-server --role fsworker` 一次性子进程,挂死可 SIGKILL 回收;二进制必须装在 /app/umt-server)
# 编译后立即自检:空环境跑 --role preflight,模块图缺件(ERR_MODULE_NOT_FOUND)在构建期就失败,不留到容器启动
RUN cd apps/server && deno task compile && cd /src && sh scripts/check-compiled-binary.sh
RUN bun run check:tokens && cd apps/web && bun run build
RUN cd apps/server && bun run compile && cd /src && sh scripts/check-compiled-binary.sh
FROM debian:trixie-slim@sha256:a99cfc517144bc59b1978475ec53b46ecabec7e43635402ee5b77cc54cd1b20a
# gosu 用于降权;PG 18 客户端来自固定的官方同发行版镜像,这里只安装动态库。
# Deno 自带根证书;健康探测用二进制自身的 fetch。
# 安装系统根证书;健康探测用二进制自身的 fetch。
# 国内网络可 --build-arg APT_MIRROR=mirrors.ustc.edu.cn 换源
ARG APT_MIRROR=""
RUN if [ -n "$APT_MIRROR" ]; then sed -i "s|deb.debian.org|$APT_MIRROR|g" /etc/apt/sources.list.d/debian.sources; fi \
&& apt-get update \
&& apt-get upgrade -y --no-install-recommends \
&& apt-get install -y --no-install-recommends gosu libssl3t64 libgssapi-krb5-2 libldap2 libzstd1 liblz4-1 libreadline8t64 \
&& apt-get install -y --no-install-recommends gosu ca-certificates libssl3t64 libgssapi-krb5-2 libldap2 libzstd1 liblz4-1 libreadline8t64 \
&& rm -rf /var/lib/apt/lists/* \
&& groupadd --system --gid 10001 umt \
&& useradd --system --uid 10001 --gid 10001 --home-dir /data --no-create-home --shell /usr/sbin/nologin umt \
@@ -42,6 +40,8 @@ RUN ldconfig && pg_dump --version && pg_restore --version && psql --version
# /app 全部 root 属主、只读:运行用户(缺省 10001,可用 PUID/PGID 覆盖)改不了二进制与静态产物
COPY --from=ffprobe --chown=root:root --chmod=0555 /ffprobe /usr/local/bin/ffprobe
COPY --from=build --chown=root:root --chmod=0555 /src/dist/umt-server /app/umt-server
COPY --from=build --chown=root:root /src/dist/sandbox-policy.node /app/sandbox-policy.node
COPY --from=build --chown=root:root /src/dist/skills /app/skills
COPY --from=build --chown=root:root /src/apps/web/build/client /app/public
COPY --chown=root:root --chmod=0555 docker/entrypoint.sh /app/entrypoint.sh
COPY --chown=root:root --chmod=0555 scripts/backup.sh scripts/restore.sh /app/scripts/
README.md
+15 -12
View File
@@ -2,7 +2,7 @@
内部使用、Agent 自动化优先的媒体整理入库平台。需求基线见 [media_automation_requirements_architecture_v0.7.md](media_automation_requirements_architecture_v0.7.md)。
当前版本 v0.3.37，本机测试环境已部署；各版变化见 [CHANGELOG.md](CHANGELOG.md)。业务逻辑以 MoviePilot v3 为参考，逐项对照见 [MoviePilot v3 对照](docs/porting/moviepilot-v3-comparison.md)。正在推进的计划与待办见 [docs/plan](docs/plan/README.md)，开发与发布约定见 [CLAUDE.md](CLAUDE.md)。
源码版本以根目录 `package.json` 为准，各版变化见 [CHANGELOG.md](CHANGELOG.md)。业务逻辑以 MoviePilot v3 为参考，逐项对照见 [MoviePilot v3 对照](docs/porting/moviepilot-v3-comparison.md)。正在推进的计划与待办见 [docs/plan](docs/plan/README.md)，开发与发布约定见 [CLAUDE.md](CLAUDE.md)。
## 功能
@@ -25,24 +25,24 @@
## 快速启动
需要 Deno 2.9.7(锁定版本:`curl -fsSL https://deno.land/install.sh | sh -s v2.9.7`)和 Docker。不需要 Node.js / pnpm。
需要 Bun 1.4.3(`curl -fsSL https://bun.sh/install | bash -s "bun-v1.4.3"`)和 Docker。解析沙箱使用 Linux seccomp；本地构建还需要 C 编译器及 Node-API 头文件(Debian/Ubuntu: `apt-get install build-essential libnode-dev`)。
```sh
cp .env.example .env
deno install # 按 deno.lock 安装依赖到 node_modules
deno task dev:up # 开发库 postgres:18.6(端口 55433)+ 迁移到最新;拉代码后也跑一次
deno task dev # API http://localhost:4390 控制台 http://localhost:5180
bun install --frozen-lockfile # 按 bun.lock 安装依赖到 node_modules
bun run dev:up # 开发库 postgres:18.6(端口 55433)+ 迁移到最新;拉代码后也跑一次
bun run dev # API http://localhost:4390 控制台 http://localhost:5180
```
提交前:`deno task ci`(typecheck:tsc 7.0.2 + deno check → biome → test:all(逐包跑完汇总,要求测试库可连)→ check:tokens → check:copy → web 构建 + server 编译 → check:binary)。
提交前:`bun run ci`(typecheck:tsc 7.0.2 → biome → test:all(逐包跑完汇总,要求测试库可连)→ check:tokens → check:copy → web 构建 + server 编译 → check:binary)。
### 前端冒烟(`deno task test:web:smoke`)
### 前端冒烟(`bun run test:web:smoke`)
Playwright(`playwright-core`,用本机 Chromium)登录后逐个打开控制台全部路由(详情页从列表接口取第一条真实 ID),检查页面渲染出来、没有控制台报错 / 未捕获异常 / 4xx·5xx 的 API 请求,并逐页截图。不在 `deno task ci` 里,需要先起好后端与 web、并已初始化管理员:
Playwright(`playwright-core`,用本机 Chromium)登录后逐个打开控制台全部路由(详情页从列表接口取第一条真实 ID),检查页面渲染出来、没有控制台报错 / 未捕获异常 / 4xx·5xx 的 API 请求,并逐页截图。不在 `bun run ci` 里,需要先起好后端与 web、并已初始化管理员:
```sh
deno task dev # 或单容器部署后直接用 4380
SMOKE_BASE_URL=http://127.0.0.1:5180 SMOKE_USER=<管理员> SMOKE_PASSWORD=<密码> deno task test:web:smoke
bun run dev # 或单容器部署后直接用 4380
SMOKE_BASE_URL=http://127.0.0.1:5180 SMOKE_USER=<管理员> SMOKE_PASSWORD=<密码> bun run test:web:smoke
```
可选:`SMOKE_OUT`(截图目录,缺省 `apps/web/test/screenshots/smoke`,已 gitignore)、`SMOKE_CHROMIUM`(浏览器路径,缺省 `/usr/bin/chromium`)、`SMOKE_SCREENSHOTS=0`。全部通过退出 0,有失败 1,前置条件不满足 2。
@@ -77,15 +77,18 @@ docker compose --env-file .env.deploy exec app umt backup # 备份
| `packages/domain` | 纯业务逻辑,无 IO |
| `packages/mapping` | domain ↔ contracts 取值与结构映射(只依赖 domain + contracts) |
| `packages/adapters` | qB / RSS / 站点 / TMDB / Emby / 文件系统适配 |
| `packages/runtime` | Bun 运行时适配与共用子进程协议 |
| `packages/tasks` | 持久任务与 outbox |
| `packages/agent` | Pi 接入与能力注册桥接 |
| `packages/testkit` | mock 服务与测试夹具 |
| `docs/adr` | 架构决策记录 |
| `docs/design` | 设计系统规范 |
| `docs/runbook` | 运行手册 |
| `config/` | tsc 7 用的 tsconfig 基座(根目录不放 tsconfig.json,否则 `deno check` 丢 Deno 全局) |
| `config/` | tsc 7 用的 tsconfig 基座(Node + Bun 类型) |
| `docker/` | 容器入口脚本、compose 环境变量示例 |
| `scripts/` | 备份 / 恢复脚本(镜像内 `/app/scripts/`) |
| `spikes/movieclaw-src/` | MovieClaw 只读源码克隆(不进 git;`scripts/mp-sync/movieclaw-import.mjs` 导入站点定义用) |
技术选型:Deno 2.9.7(锁定)+ TypeScript 7.0.2 strict,deno.json workspace(`nodeModulesDir: auto`,npm 依赖版本统一登记在根 `deno.json` 的 `imports`,`deno.lock` 锁定)。前端 Vite 8 + React Router 8 由 Deno 执行。决策见 [ADR-004](docs/adr/ADR-004-deno-runtime.md)(取代 ADR-001)。
技术选型:Bun 1.4.3(锁定)+ TypeScript 7.0.2 strict,package.json workspace,npm 依赖由 bun.lock 锁定。前端 Vite 8 + React Router 8 由 Bun 执行。决策见 [ADR-007](docs/adr/ADR-007-bun-runtime.md)(取代 ADR-004)。
解析沙箱预加载解析器和 WASM 后,通过 Node-API 原生模块安装 Linux seccomp 系统调用白名单,覆盖全部 Bun 线程。任务执行阶段禁止路径文件访问、网络、执行程序和创建子进程；隔离无法建立时直接失败。超时和 RSS 超限仍由父进程 SIGKILL。生产镜像同时安装编译二进制、沙箱策略模块与内置 skills。
apps/server/README.md
+39 -77
View File
@@ -1,95 +1,57 @@
# @umt/server
服务进程(Deno):Hono HTTP API(前缀 `/api/v1`,OpenAPI 见 `/api/openapi.json`、`/api/docs`)、SSE 事件流与 worker 入口。`--role api|worker|worker:agent|all` 决定本进程承担哪部分,默认 `all` 单进程运行(`worker` 只认领 io 组任务,`worker:agent` 只认领 Agent 组;另有内部角色 `--role fsworker`,由 FsProxy 以子进程方式拉起执行单个文件操作,不要手动启动)。业务规则不写在这里,只做编排、鉴权、审计与协议适配。
服务进程(Bun 1.4.3):Hono HTTP API(`/api/v1`)、SSE 与 worker 入口。默认 `--role all` 单进程运行；部署按 `api`、`worker`(IO)、`worker:agent` 分进程。OpenAPI 见 `/api/openapi.json`、`/api/docs`。业务编排和写入放在模块 service,确定性规则放在 `@umt/domain`。
## 模块骨架(W2)
## 模块骨架
```
```text
src/
main.ts 进程入口:按 --role 装配
context.ts bootstrap():ServerContext、任务注册表(模块 + 空壳兜底)、outbox 路由合并、连接引擎
http/app.ts createApp():系统路由 + 逐个挂 SERVER_MODULES 的 registerRoutes
worker/index.ts startWorker():引擎 worker + OutboxRelay + 各模块 start()
hooks/ 链式钩子注册表与调用入口(ADR-006)
main.ts 按 --role 装配进程
context.ts 装配 ServerContext、实际任务注册表与 outbox 路由
http/app.ts 系统路由 + 模块 registerRoutes
worker/index.ts 任务 worker、OutboxRelay、模块后台循环
authorization/grants.ts 共用计划绑定、持续授权范围与预算检查
sandbox/ 外部字节解析子进程与 seccomp 策略
modules/
types.ts ServerModule / ServerContext / OutboxWriter / SecretStore / SettingsReader(共享,不随子任务改)
types.ts ServerModule / ServerContext 等装配契约
index.ts SERVER_MODULES 清单
auth-settings/ W2-A(provided.ts 提供 secrets / settings 实现)
sources-identity/ W2-B
acquisition-organize/ W2-C
file-manager/ W5-A:/files/*(浏览、mkdir/rename、move/copy/organize 两阶段、回收站、identify/probe/scrape、download),
见 docs/runbook/files.md
agent/ W2-D:capability-registry.ts(authorize / audit / 幂等键 / 操作账本)、会话 API、runAgent、resolveDecision
(见 docs/runbook/agent.md;agent 模块可以 import 其它模块的 service 导出,反向不行)
decision-hooks.ts 待处理事项的实际业务处理器
auth-settings/ 鉴权、密钥与设置
sources-identity/ 站点、资源、识别与元数据
acquisition-organize/ 订阅、收录、下载、整理与刮削
file-manager/ 文件浏览与操作
agent/ Agent 会话、交互与能力调用编排
… 其余模块见 SERVER_MODULES
```
**分工规则:各 W2 子任务只改自己的 `modules/<area>/` 目录,以及 `packages/contracts`、`packages/db` 中本域的文件**(新迁移按编号顺延,开工前在群里认领编号)。`modules/types.ts`、`modules/index.ts`、`context.ts`、`http/app.ts`、`worker/index.ts`、`main.ts` 是共享装配,确需改动时单独提交并通知其他子任务;`http/auth.ts`、`http/sse.ts` 归 W2-A。跨模块只通过 DB 表 + `outbox_event`(事件类型在 contracts/events.ts)+ contracts 类型耦合,不互相 import 对方模块。
跨模块的同步业务操作显式调用 service；可靠的异步后续动作随业务事务写 outbox。路由和能力处理器共用业务入口,service 不依赖 HTTP 路由或 Agent 会话编排。共用语义类型直接复用,有实际取值差异才用 `@umt/mapping`。完整边界、状态归属和抽象约定见 [开发指南](../../docs/runbook/dev.md#工程设计与适度抽象)。
一个模块长这样(`modules/<area>/index.ts` 导出,目录内自由拆 `routes.ts` / `service.ts` / `handlers/*.ts`):
## 模块装配
```ts
import { createRoute } from "@hono/zod-openapi";
import { capabilitySpec, EventType } from "@umt/contracts";
import { defineTask } from "@umt/tasks";
import { requireScope } from "../../http/auth.ts";
import type { ServerModule } from "../types.ts";
`modules/<area>/index.ts` 导出 `ServerModule`,按实际需求实现:
export const sourcesIdentityModule: ServerModule = {
name: "sources-identity",
// 路径相对 /api/v1,已过 API Key 鉴权;每个路由自己声明 scope
registerRoutes(app, ctx) {
app.use("/sites/*", requireScope("read"));
app.openapi(createRoute({ method: "get", path: "/sites", /* … */ }), async (c) => c.json(/* … */));
},
// 同名任务只能由一个模块注册;未注册的 §3.1-8 任务由 @umt/tasks 空壳兜底
registerTasks(registry, ctx) {
registry.register(defineTask({ name: "resolveObservation", version: 1, /* … */ handler: async (t) => {
await ctx.db.transaction().execute(async (trx) => {
// …写业务表,同事务发事件(payload 按 contracts 校验)
await ctx.outbox.emit(trx, EventType.releaseResolved, payload, { idempotencyKey: `release.resolved:${id}` });
});
} }));
},
// 消费别的模块发出的事件:type → 任务;同一 type 只能一个模块认领
outboxRoutes: () => ({ "release.resolved": (ev) => ({ task: "evaluateAcquisition", input: { sourceReleaseId: (ev.payload as { sourceReleaseId: string }).sourceReleaseId } }) }),
// 能力:规格一律取 contracts,处理器调本模块服务
registerCapabilities(registry, ctx) {
registry.register(capabilitySpec("release.search"), async (input, c) => ({ items: [], coverage: [] }));
},
// worker 角色的后台循环(qB 轮询、Feed 调度…);引擎与中继已启动
async start(ctx) {},
async stop() {},
};
```
| 入口 | 职责 |
|---|---|
| `registerRoutes(app, ctx)` | 注册相对 `/api/v1` 的路由,声明 scope、校验输入、调用 service |
| `registerTasks(registry, ctx)` | 用 `defineTask` 注册真实 handler；任务名和版本不能重复,没有空壳兜底 |
| `registerCapabilities(registry, ctx)` | 规格取自 contracts,处理器调用 service |
| `outboxRoutes(ctx)` | 事件 → 任务、任务数组或 null；生命周期事件允许多个订阅方,内部流程事件只有一个认领方 |
| `start(ctx)` / `stop()` | 后台循环与回收；需要恢复的长任务进入持久队列 |
`ServerContext` 字段:`db`(Kysely<DB>)、`config`、`logger`(pino,自动脱敏)、`events`(SSE 总线 `EventBusPort`)、`engine`(TaskEngine:enqueue/signal/cancel/get/list;register* 阶段不要调用)、`outbox`(`emit(trx, type, payload, {idempotencyKey})` + worker 的 `relay`)、`secrets`(SecretStore,W2-A 实现前调用会抛错)、`settings`(读 settings 表并合并缺省)、`capabilities`(CapabilityRegistry)。domain ↔ contracts 的取值转换一律用 `@umt/mapping`。
`ServerContext` 提供 `db`、`config`、`logger`、`events`、`engine`、`outbox`、`secrets`、`settings`、`capabilities`。注册阶段只做装配,不要使用尚未连接的引擎。`all` 连接时顺带迁移引擎；分进程部署先执行 `--role migrate`,再以 `runMigrations:false` 连接。
角色:`all` 单进程时连接引擎会顺带跑引擎迁移;`api` / `worker` 分进程部署时由 `--role migrate` 预先迁移(db + openworkflow),两个进程都以 `runMigrations:false` 连接。
业务写入与 `ctx.outbox.emit(trx, type, payload, { idempotencyKey })` 同事务提交。payload 在 contracts 定义并校验,订阅任务按 ID 回查业务记录、幂等执行。引擎步骤与租约不替代业务表的状态。
## 如何写一个功能模块(ADR-006)
## 新增功能模块
外围功能(补字幕这类)按下面的约定接进来,不做插件系统、不做开关。样板是 `modules/subtitles/`。
以 `modules/subtitles/` 为样板,只接入实际需要的入口:
1. **登记 manifest**:在 `packages/contracts/src/features.ts` 的 `FEATURES` 里加一份(名称、说明、怎么用、订阅的事件、能力、页面入口、插槽、钩子)。它会进智能体系统提示「能力地图」末尾的功能清单;`test/modules.test.ts` 核对模块与清单一一对应。
2. **建模块**:`modules/<id>/index.ts` 导出 `ServerModule`,`name` 与 manifest id 相同,写 `manifest: featureManifest("<id>")`,加进 `modules/index.ts`。
3. **订阅广播事件**(事后通知,异步、至少一次):`outboxRoutes` 里写 `"<事件>": (ev) => 任务 | 任务数组 | null`。生命周期事件(`LIFECYCLE_EVENTS`)可以多个模块同时订阅;payload 用 `parseEventPayload` 取,按 ID 回查业务表;订阅任务要幂等(同一事件可能投递多次)。
```ts
outboxRoutes: (ctx) => ({
"organization.finished": (ev) => {
const p = parseEventPayload("organization.finished", ev.payload);
return p?.outcome === "succeeded" ? { task: "myFollowUp", input: { recordId: p.recordId } } : null;
},
}),
```
4. **注册链式钩子**(同步改写 / 拦截,挂点见 `packages/contracts/src/hooks.ts`):
```ts
registerHooks(hooks) {
hooks.rewrite("organize.rename", (name, c) => (c.mediaType === "movie" ? name.replace(/ - 1080p/, "") : undefined));
hooks.intercept("organize.intercept", (c) => (c.files.some((f) => f.dst.includes("/Sample/")) ? "目标在 Sample 目录" : null));
},
```
钩子可能在业务事务里执行:只做内存里的判断,不发外部请求、不写库;缺省 2 秒超时,抛错 / 超时只跳过自己。
5. **自己发事件**:业务节点在事务里 `ctx.outbox.emit(trx, "<事件>", payload, { idempotencyKey: lifecycleKey("<事件>", id) })`;新事件先在 `contracts/events.ts` 加类型、payload schema,并登记进 `LIFECYCLE_EVENTS`。
6. **前端入口**:在 `apps/web/app/features/<id>.tsx` 用插槽注册(作品页操作区 / 面板区、整理记录行操作、侧栏、设置分区),在 `features/index.ts` 调一次;默认直接显示。
7. **智能体能力**(可选):照常在 contracts 定义能力、在 `registerCapabilities` 挂处理器,并登记 Skill 与能力地图(见仓库根 CLAUDE.md)。
1. 在 `packages/contracts/src/features.ts` 登记 manifest(名称、用法、事件、能力、页面与插槽)。它进入 Agent 功能清单,由 `test/modules.test.ts` 核对。
2. 导出 `ServerModule`,设置 `manifest: featureManifest("<id>")`,加入 `modules/index.ts`。
3. 需要事后动作时订阅生命周期事件,用 `parseEventPayload` 读取；自己发新事件时在 contracts 登记类型、schema 与 `LIFECYCLE_EVENTS`。
4. 需要前端入口时在 `apps/web/app/features/<id>.tsx` 注册插槽并加入 `features/index.ts`,默认直接显示。
5. 新增 Agent 能力时同时登记 Skill 与能力地图,见根目录 `CLAUDE.md`。
同步规则直接在所属 service 实现。目前没有通用链式钩子；`decision-hooks.ts` 只处理已有待处理事项的业务动作。功能默认启用,无需另建插件系统或开关。
开发、测试和容器说明见 [开发手册](../../docs/runbook/dev.md) 与 [部署手册](../../docs/runbook/deploy.md)。
apps/server/deno.json
-12
View File
@@ -1,12 +0,0 @@
{
"name": "@umt/server",
"version": "0.3.44",
"exports": { ".": "./src/main.ts" },
"tasks": {
"dev": "deno run -A --watch --env-file=../../.env src/main.ts",
"start": "deno run --allow-net --allow-env --allow-sys --allow-read --allow-write --allow-run=ffprobe,sha256sum,cat,deno --env-file=../../.env src/main.ts",
"compile": "deno compile --frozen --node-modules-dir=none --exclude-unused-npm --allow-net --allow-env --allow-sys --allow-read --allow-write --allow-run=ffprobe,sha256sum,cat,/app/umt-server --include ../../packages/db/migrations --include ../../packages/agent/skills --output ../../dist/umt-server src/main.ts",
"test": "deno run -A npm:vitest run",
"typecheck": "deno run -A npm:typescript/tsc --noEmit -p tsconfig.json"
}
}
apps/server/package.json
+19
View File
@@ -4,5 +4,24 @@
"type": "module",
"exports": {
".": "./src/main.ts"
},
"version": "0.3.44",
"scripts": {
"dev": "bun --watch --env-file=../../.env src/main.ts",
"start": "bun --env-file=../../.env src/main.ts",
"compile": "bun ../../scripts/compile.ts",
"test": "bun x --bun vitest run",
"typecheck": "bun x --bun tsc --noEmit -p tsconfig.json"
},
"dependencies": {
"@umt/contracts": "workspace:*",
"@umt/adapters": "workspace:*",
"@umt/mapping": "workspace:*",
"@umt/tasks": "workspace:*",
"@umt/agent": "workspace:*",
"@umt/db": "workspace:*",
"@umt/runtime": "workspace:*",
"@umt/domain": "workspace:*",
"@umt/testkit": "workspace:*"
}
}
apps/server/src/assets.d.ts
Vendored
+4
View File
@@ -0,0 +1,4 @@
declare module "*.wasm" {
const path: string;
export default path;
}
apps/server/src/authorization/grants.ts
+174
View File
@@ -0,0 +1,174 @@
import { createHash } from "node:crypto";
import { type DB, type Kysely, sql } from "@umt/db";
// 能力执行共用的计划绑定和持续授权检查;不依赖 Agent 会话、方案选择或审批交互。
/** 能力 → 可覆盖它的授权动作(策略发布生成 policy.<kind> 持续授权;"*" 全部) */
export const GRANT_ACTIONS: Record<string, readonly string[]> = {
"identity.resolve": ["identity.resolve", "policy.identity"],
"download.submit": ["download.submit", "policy.acquisition"],
"organize.execute": ["organize.execute", "policy.acquisition", "policy.organize"],
"download.recheck": ["download.recheck", "policy.automation"],
"task.retry": ["task.retry", "task.*"],
"task.cancel": ["task.cancel", "task.*"],
"policy.apply": ["policy.apply"],
// 集数映射兜底(批次 5):只认专门的持续授权,不随身份识别策略放开
"episode_mapping.apply_rule": ["episode_mapping.apply_rule"],
// VIS-1:坐标操作与视觉子流程和 browser.interact 同一档(操作原站页面),同一条持续授权覆盖
"browser.click_at": ["browser.click_at", "browser.interact"],
"browser.type_at": ["browser.type_at", "browser.interact"],
"browser.scroll": ["browser.scroll", "browser.interact"],
"browser.solve_challenge": ["browser.solve_challenge", "browser.interact"],
};
/** 单次审批绑定的计划类型与输入里的计划 ID 字段 */
export const APPROVAL_PLAN: Record<string, { planType: string; field: string }> = {
"download.submit": { planType: "acquisition_plan", field: "planId" },
"organize.execute": { planType: "organize_plan", field: "planId" },
"policy.apply": { planType: "policy_revision", field: "policyRevisionId" },
};
export interface PlanBinding {
payloadHash: string;
revision: number;
mediaEntityId: string | null;
mediaKind: string | null;
bytes: number | null;
}
const sha = (s: string) => createHash("sha256").update(s, "utf8").digest("hex");
/** 读计划当前的 payload hash / revision / 目标作品(审批绑定与范围校验共用) */
export async function planBinding(db: Kysely<DB>, planType: string, planId: string): Promise<PlanBinding | null> {
if (planType === "acquisition_plan") {
const r = await db
.selectFrom("acquisition_plan as p")
.leftJoin("acquisition_slot as s", "s.id", "p.slot_id")
.leftJoin("media_entity as e", (j) =>
j.onRef("e.id", "=", sql<string>`coalesce(s.media_entity_id, (p.evaluation->>'mediaEntityId')::uuid)`),
)
.select(["p.payload_hash", "p.expected_revision", "e.id as entity_id", "e.kind", "p.reserved_bytes"])
.where("p.id", "=", planId)
.executeTakeFirst();
return r
? {
payloadHash: r.payload_hash,
revision: r.expected_revision,
mediaEntityId: r.entity_id,
mediaKind: r.kind,
bytes: Number(r.reserved_bytes),
}
: null;
}
if (planType === "organize_plan") {
const r = await db
.selectFrom("artifact")
.select(["content_hash", "content"])
.where("id", "=", planId)
.where("kind", "=", "plan")
.executeTakeFirst();
if (!r) return null;
const c = (r.content ?? {}) as { mediaEntityId?: string | null; mediaKind?: string | null };
return {
payloadHash: r.content_hash,
revision: 1,
mediaEntityId: c.mediaEntityId ?? null,
mediaKind: c.mediaKind ?? null,
bytes: null,
};
}
if (planType === "policy_revision") {
const r = await db
.selectFrom("policy_revision")
.select(["revision", "rules", "scope"])
.where("id", "=", planId)
.executeTakeFirst();
return r
? {
payloadHash: sha(JSON.stringify({ rules: r.rules, scope: r.scope })),
revision: r.revision,
mediaEntityId: null,
mediaKind: null,
bytes: null,
}
: null;
}
return null;
}
/**
* 持续授权是否覆盖本能力(不含单次审批与会话内审批):action 覆盖、active 未过期、绑定策略仍发布、
* 对象范围与每日预算。download.plan 用它判定 requiresApproval,authorize 用它放行。
*/
export async function coveringGrant(
db: Kysely<DB>,
capabilityId: string,
target: PlanBinding | null,
now: Date,
): Promise<{ grantId: string } | { reasons: string[] }> {
const d = { id: capabilityId };
const actions = [...(GRANT_ACTIONS[d.id] ?? [d.id]), "*"];
const grants = await db
.selectFrom("automation_grant as g")
.leftJoin("policy_revision as p", "p.id", "g.policy_revision_id")
.select(["g.id", "g.action", "g.scope", "g.budget", "g.policy_revision_id", "p.status as policy_status"])
.where("g.status", "=", "active")
.where("g.action", "in", actions)
.where((eb) => eb.or([eb("g.expires_at", "is", null), eb("g.expires_at", ">", now)]))
.orderBy("g.created_at", "desc")
.execute();
const reasons: string[] = [];
for (const g of grants) {
if (g.policy_revision_id && g.policy_status !== "published") {
reasons.push(`授权 ${g.id} 绑定的策略版本已不是发布状态`);
continue;
}
const scope = (g.scope ?? {}) as { mediaEntityIds?: string[]; mediaKinds?: string[]; capabilities?: string[] };
if (scope.capabilities?.length && !scope.capabilities.includes(d.id)) {
reasons.push(`授权 ${g.id} 不含能力 ${d.id}`);
continue;
}
if (
scope.mediaEntityIds?.length &&
(!target?.mediaEntityId || !scope.mediaEntityIds.includes(target.mediaEntityId))
) {
reasons.push(`授权 ${g.id} 的对象范围不含目标作品`);
continue;
}
if (scope.mediaKinds?.length && target?.mediaKind && !kindCovered(scope.mediaKinds, target.mediaKind)) {
reasons.push(`授权 ${g.id} 的媒体类型范围不含 ${target.mediaKind}`);
continue;
}
const budget = (g.budget ?? {}) as { maxInvocationsPerDay?: number; maxBytesPerDay?: number };
if (budget.maxInvocationsPerDay !== undefined || budget.maxBytesPerDay !== undefined) {
const day = new Date(now);
day.setHours(0, 0, 0, 0);
const used = await sql<{ n: number }>`
select count(*)::int as n from capability_invocation
where grant_id = ${g.id} and status = 'succeeded' and created_at >= ${day}`.execute(db);
if (budget.maxInvocationsPerDay !== undefined && (used.rows[0]?.n ?? 0) >= budget.maxInvocationsPerDay) {
reasons.push(`授权 ${g.id} 今日次数预算已用尽`);
continue;
}
if (budget.maxBytesPerDay !== undefined && d.id === "download.submit") {
const bytes = await sql<{ b: string }>`
select coalesce(sum(p.reserved_bytes), 0)::bigint as b
from capability_invocation ci join acquisition_plan p on p.id::text = ci.input->>'planId'
where ci.grant_id = ${g.id} and ci.status = 'succeeded' and ci.created_at >= ${day}`.execute(db);
if (Number(bytes.rows[0]?.b ?? 0) + (target?.bytes ?? 0) > budget.maxBytesPerDay) {
reasons.push(`授权 ${g.id} 今日字节预算不足`);
continue;
}
}
}
return { grantId: g.id };
}
return { reasons };
}
/** 策略范围用订阅口径(movie / series / variety),作品用 media_entity.kind(movie / tv / variety …) */
function kindCovered(kinds: readonly string[], entityKind: string): boolean {
if (kinds.includes(entityKind)) return true;
if (entityKind !== "movie" && entityKind !== "variety" && kinds.includes("series")) return true;
return false;
}
apps/server/src/config.ts
+4 -4
View File
@@ -55,8 +55,8 @@ export const ConfigSchema = z.object({
/** Agent JSONL 会话目录;缺省 DATA_DIR/agent-sessions(api 与 worker 必须共享同一目录) */
AGENT_SESSION_DIR: z.string().min(1).optional(),
// ── 文件操作挂死隔离(adapters fsproxy,对标 MP fsproxy)──
/** auto:能起 deno 子进程就把单个文件操作放进执行子进程(可 SIGKILL 回收),否则进程内看门狗;inprocess 强制进程内 */
FS_PROXY_MODE: z.enum(["auto", "subprocess", "inprocess"]).default("auto"),
/** auto:能起 bun 子进程就把单个文件操作放进执行子进程(可 SIGKILL 回收),否则进程内看门狗;inprocess 强制进程内 */
FS_PROXY_MODE: z.enum(["auto", "subprocess", "inprocess"]).default("subprocess"),
/** 复制/哈希:进度停滞多久判挂死(ms),超时 SIGTERM→SIGKILL 并以 TIMEOUT 失败 */
FS_HANG_TIMEOUT_MS: z.coerce.number().int().min(1000).default(120_000),
/** stat/根检查/同卷 move 等快操作的总超时(ms) */
@@ -104,7 +104,7 @@ export const ConfigSchema = z.object({
export type Config = z.infer<typeof ConfigSchema>;
/** 从环境变量加载并校验配置;失败时抛出可读错误(不含变量值) */
export function loadConfig(env: Record<string, string | undefined> = Deno.env.toObject()): Config {
export function loadConfig(env: Record<string, string | undefined> = { ...process.env }): Config {
const parsed = ConfigSchema.safeParse(env);
if (!parsed.success) {
const lines = parsed.error.issues.map((i) => ` ${i.path.join(".")}: ${i.message}`);
@@ -117,7 +117,7 @@ export function loadConfig(env: Record<string, string | undefined> = Deno.env.to
export const DATA_SUBDIRS = ["agent-sessions", "backups", "exports", "tmp"] as const;
/** 解析数据目录与会话目录为绝对路径 */
export function dataPaths(c: Pick<Config, "DATA_DIR" | "AGENT_SESSION_DIR">, cwd: string = Deno.cwd()) {
export function dataPaths(c: Pick<Config, "DATA_DIR" | "AGENT_SESSION_DIR">, cwd: string = process.cwd()) {
const dataDir = resolve(cwd, c.DATA_DIR);
return {
dataDir,
apps/server/src/context.ts
+2 -13
View File
@@ -1,7 +1,6 @@
import { type EventType, LIFECYCLE_EVENTS, parseEventPayload } from "@umt/contracts";
import { type DB, type Kysely, sql } from "@umt/db";
import {
createDefaultRegistry,
type LogSink,
OpenWorkflowEngine,
type OutboxRelay,
@@ -12,7 +11,6 @@ import {
} from "@umt/tasks";
import type { Logger } from "pino";
import type { Config } from "./config.ts";
import { HookRegistry } from "./hooks/registry.ts";
import type { EventBusPort } from "./http/sse.ts";
import { assertCapabilityCount, createCapabilityRegistry } from "./modules/agent/capability-registry.ts";
import { createSecretStore, createSettingsReader } from "./modules/auth-settings/provided.ts";
@@ -80,8 +78,8 @@ export function tasksLogSink(logger: Logger): LogSink {
}
/**
* 任务注册表:各模块注册自己的任务;没人注册的 §3.1-8 任务用 @umt/tasks 的空壳兜底
* (outbox 路由到它们时不至于报"未知任务")。同名任务被两个模块注册会在这里报错。
* 任务注册表只收录业务模块的真实实现。同名任务被两个模块注册会在这里报错;
* 没有注册的任务由 TaskRegistry 明确报 UnknownTaskError,不以占位处理器代替。
*/
export function buildTaskRegistry(modules: readonly ServerModule[], ctx: ServerContext): TaskRegistry {
const registry = new TaskRegistry();
@@ -96,7 +94,6 @@ export function buildTaskRegistry(modules: readonly ServerModule[], ctx: ServerC
}
registry.register(...own.all());
}
for (const stub of createDefaultRegistry().all()) if (!registry.has(stub.name)) registry.register(stub);
return registry;
}
@@ -153,13 +150,6 @@ export function collectOutboxRoutes(
return routes;
}
/** 链式钩子注册表:按模块顺序调 registerHooks,执行顺序即注册顺序(ADR-006) */
export function buildHookRegistry(modules: readonly ServerModule[], ctx: ServerContext): HookRegistry {
const hooks = new HookRegistry(ctx.logger?.child({ component: "hooks" }));
for (const m of modules) m.registerHooks?.(hooks.forModule(m.name), ctx);
return hooks;
}
export interface BootstrapOptions {
role: ServerRole;
config: Config;
@@ -204,7 +194,6 @@ export async function bootstrap(opts: BootstrapOptions): Promise<Bootstrapped> {
configureStorage(ctx);
const tasks = buildTaskRegistry(opts.modules, ctx);
for (const m of opts.modules) m.registerCapabilities?.(ctx.capabilities, ctx);
ctx.hooks = buildHookRegistry(opts.modules, ctx);
// 装配了 agent 模块(即完整模块清单)时,首批 14 个能力必须全部挂上处理器
if (opts.modules.some((m) => m.name === "agent")) assertCapabilityCount(ctx.capabilities);
engine.attach(
apps/server/src/egress.ts
+26 -70
View File
@@ -1,75 +1,28 @@
// CF-2 出口统一:每条对外 HTTP 请求走哪条出口由代码显式决定,不再隐式继承容器的 HTTP(S)_PROXY。
//
// Deno 2.9 的语义(2026-09-30 本机实测,见 docs/runbook/sites.md「出口」):
// - `fetch` 缺省客户端与 `Deno.createHttpClient({})` 都会读 HTTP_PROXY / HTTPS_PROXY / ALL_PROXY(大小写都认)与 NO_PROXY;
// 读取发生在**客户端创建时**(缺省客户端在进程第一次 fetch 时创建);
// - `createHttpClient` 没有"不走代理"的选项,`NO_PROXY=*` 也不认;显式 `proxy` 会排在环境变量代理前面,
// 但 NO_PROXY 命中的主机连显式代理也绕过;
// - 所以直连客户端的做法是:创建那一刻把这几个环境变量临时拿掉(同步调用,同一个 isolate 内不会被别的代码插进来),建完立即还原。
// 代理客户端同样在拿掉环境变量后创建,保证只走给定的代理。
//
// 出口表:
// - 站点请求(适配器、RSS、登录、用户数据):站点开"使用代理"且设了代理地址 → 该代理;否则直连(siteClient);
// - TMDB、Brave、图片缓存等外网请求:照旧用环境变量里的代理(行为不变),但改成显式读 env 建客户端(externalFetch)。
const PROXY_ENV = ["HTTP_PROXY", "HTTPS_PROXY", "ALL_PROXY", "http_proxy", "https_proxy", "all_proxy"] as const;
const NO_PROXY_ENV = ["NO_PROXY", "no_proxy"] as const;
/** 进程启动时的代理环境变量快照(外网请求显式使用;之后的临时改动不影响) */
const ENV_SNAPSHOT: Record<string, string | undefined> = (() => {
const out: Record<string, string | undefined> = {};
try {
for (const k of [...PROXY_ENV, ...NO_PROXY_ENV]) out[k] = Deno.env.get(k);
} catch {
// 没有 env 权限:当作没有代理
}
return out;
})();
/** 在"没有代理环境变量"的状态下同步执行 fn(建 HttpClient 用),结束立即还原 */
function withoutProxyEnv<T>(fn: () => T): T {
const saved = new Map<string, string>();
try {
for (const k of [...PROXY_ENV, ...NO_PROXY_ENV]) {
const v = Deno.env.get(k);
if (v !== undefined) {
saved.set(k, v);
Deno.env.delete(k);
}
}
} catch {
// 没有 env 权限时也没有可读的代理变量,直接建
}
try {
return fn();
} finally {
for (const [k, v] of saved) Deno.env.set(k, v);
}
// 每次请求显式指定 Bun 的 proxy,直连使用proxy: false以禁用环境代理。
export interface HttpClient {
proxy: false | { url: string; respectNoProxy: false };
close(): void;
}
let direct: Deno.HttpClient | null = null;
const proxied = new Map<string, Deno.HttpClient>();
/** 明确直连的客户端(不读任何代理环境变量) */
export function directHttpClient(): Deno.HttpClient {
direct ??= withoutProxyEnv(() => Deno.createHttpClient({}));
const direct: HttpClient = { proxy: false, close() {} };
const proxied = new Map<string, HttpClient>();
const ENV_SNAPSHOT = { ...process.env };
export function directHttpClient(): HttpClient {
return direct;
}
/** 只走给定代理的客户端(同一地址复用) */
export function proxyHttpClient(url: string): Deno.HttpClient {
let c = proxied.get(url);
if (!c) {
c = withoutProxyEnv(() => Deno.createHttpClient({ proxy: { url } }));
proxied.set(url, c);
export function proxyHttpClient(url: string): HttpClient {
let client = proxied.get(url);
if (!client) {
client = { proxy: { url, respectNoProxy: false }, close() {} };
proxied.set(url, client);
}
return c;
return client;
}
/** 用指定客户端发请求的 fetch */
export function fetchVia(client: Deno.HttpClient): typeof globalThis.fetch {
export function fetchVia(client: HttpClient): (input: RequestInfo | URL, init?: RequestInit) => Promise<Response> {
return ((input: RequestInfo | URL, init?: RequestInit) =>
globalThis.fetch(input, { ...init, client } as RequestInit)) as typeof globalThis.fetch;
globalThis.fetch(input, { ...init, proxy: client.proxy } as RequestInit)) as (
input: RequestInfo | URL,
init?: RequestInit,
) => Promise<Response>;
}
/** 站点请求的出口:开代理 → 设置里的站点代理地址,没填则用进程的 HTTPS_PROXY / HTTP_PROXY;都没有 → direct */
@@ -98,7 +51,7 @@ export function egressKey(e: SiteEgress): string {
}
}
export function siteFetchFor(e: SiteEgress): typeof globalThis.fetch {
export function siteFetchFor(e: SiteEgress): (input: RequestInfo | URL, init?: RequestInit) => Promise<Response> {
return fetchVia(e.mode === "proxy" && e.proxyUrl ? proxyHttpClient(e.proxyUrl) : directHttpClient());
}
@@ -128,7 +81,10 @@ export function envProxyFor(target: URL, env: Record<string, string | undefined>
}
/** 外网请求用的 fetch:按环境变量显式选代理 / 直连(与之前隐式继承的行为一致) */
export const externalFetch: typeof globalThis.fetch = ((input: RequestInfo | URL, init?: RequestInit) => {
export const externalFetch: (input: RequestInfo | URL, init?: RequestInit) => Promise<Response> = ((
input: RequestInfo | URL,
init?: RequestInit,
) => {
let target: URL | null = null;
try {
target = new URL(input instanceof Request ? input.url : String(input));
@@ -137,5 +93,5 @@ export const externalFetch: typeof globalThis.fetch = ((input: RequestInfo | URL
}
const proxy = target ? envProxyFor(target) : null;
const client = proxy ? proxyHttpClient(proxy) : directHttpClient();
return globalThis.fetch(input, { ...init, client } as RequestInit);
}) as typeof globalThis.fetch;
return fetchVia(client)(input, init);
}) as (input: RequestInfo | URL, init?: RequestInit) => Promise<Response>;
apps/server/src/hooks/index.ts
-44
View File
@@ -1,44 +0,0 @@
import type { HookPoint } from "@umt/contracts";
import type { ServerContext } from "../modules/types.ts";
import type { HookContext, HookValue, InterceptPoint, RewritePoint } from "./points.ts";
import type { HookRejection } from "./registry.ts";
export type * from "./points.ts";
export {
DEFAULT_HOOK_TIMEOUT_MS,
type HookOptions,
type HookRegistrar,
HookRegistry,
type HookRejection,
} from "./registry.ts";
// 业务节点调用钩子的入口。ctx.hooks 由 bootstrap 装配;测试里手搭的上下文没有它,等同于没有注册。
/** 这个挂点有没有钩子(上下文构造代价大时先问一下) */
export function hasHook(ctx: Pick<ServerContext, "hooks">, point: HookPoint): boolean {
return ctx.hooks?.has(point) ?? false;
}
/** 改写:没有钩子时原样返回同一个值;accept 判定单个钩子的结果能不能用(不能用的跳过) */
export async function rewriteHook<P extends RewritePoint>(
ctx: Pick<ServerContext, "hooks">,
point: P,
value: HookValue<P>,
context: HookContext<P> | (() => HookContext<P>),
accept?: (v: HookValue<P>) => boolean,
): Promise<HookValue<P>> {
const h = ctx.hooks;
if (!h?.has(point)) return value;
return h.runRewrite(point, value, typeof context === "function" ? context() : context, accept);
}
/** 拦截:返回第一个拒绝,没有钩子或都放行时返回 null */
export async function interceptHook<P extends InterceptPoint>(
ctx: Pick<ServerContext, "hooks">,
point: P,
context: HookContext<P> | (() => HookContext<P>),
): Promise<HookRejection | null> {
const h = ctx.hooks;
if (!h?.has(point)) return null;
return h.runIntercept(point, typeof context === "function" ? context() : context);
}
apps/server/src/hooks/points.ts
-81
View File
@@ -1,81 +0,0 @@
import type { HOOK_POINTS, HookPoint } from "@umt/contracts";
import type { ParsedRelease } from "@umt/domain";
// 链式钩子挂点的入参 / 返回值类型(挂点清单与语义在 contracts/hooks.ts,ADR-006)。
// rewrite 挂点:value 是要改写的值,context 只读;intercept 挂点只有 context。
/** 资源选择的候选:只能重排或过滤,不能加入新候选(按 sourceReleaseId 映射回原资源) */
export interface SelectionCandidate {
sourceReleaseId: string;
title: string;
parsed: ParsedRelease;
}
export interface HookPointTypes {
"name.recognize": {
value: ParsedRelease;
context: {
title: string;
subtitle: string | null;
/** 来源资源;按名称识别整理文件时为 null */
sourceReleaseId: string | null;
trigger: string;
};
};
"organize.rename": {
/** 目标文件名(不含目录);返回值不能含路径分隔符 */
value: string;
context: {
targetPath: string;
sourcePath: string;
mediaEntityId: string;
mediaType: "movie" | "tv";
season: number | null;
episodes: { start: number; end: number } | null;
};
};
"organize.intercept": {
value: never;
context: {
mediaEntityId: string;
mediaType: "movie" | "tv";
action: string;
trigger: string;
downloadTaskId: string | null;
files: { src: string; dst: string; kind: string; role: "main" | "sidecar" }[];
};
};
"resource.selection": {
value: SelectionCandidate[];
context: { mediaEntityId: string; subscriptionId: string | null; slotId: string | null };
};
"subscription.completion.check": {
value: never;
context: {
subscriptionId: string;
mediaEntityId: string | null;
mediaKind: string;
/** 本次判定完成的原因(slot_satisfied / download_accepted) */
reason: string;
progress: Record<string, unknown>;
};
};
}
type KindOf<P extends HookPoint> = (typeof HOOK_POINTS)[P]["kind"];
export type RewritePoint = { [P in HookPoint]: KindOf<P> extends "rewrite" ? P : never }[HookPoint];
export type InterceptPoint = { [P in HookPoint]: KindOf<P> extends "intercept" ? P : never }[HookPoint];
export type HookValue<P extends HookPoint> = HookPointTypes[P]["value"];
export type HookContext<P extends HookPoint> = Readonly<HookPointTypes[P]["context"]>;
type MaybePromise<T> = T | Promise<T>;
/** 返回新值;返回 undefined 表示不改 */
export type RewriteHook<P extends RewritePoint> = (
value: HookValue<P>,
context: HookContext<P>,
) => MaybePromise<HookValue<P> | undefined>;
/** 返回拒绝原因;返回 null / undefined 表示放行 */
export type InterceptHook<P extends InterceptPoint> = (
context: HookContext<P>,
) => MaybePromise<string | null | undefined>;
apps/server/src/hooks/registry.ts
-126
View File
@@ -1,126 +0,0 @@
import { HOOK_POINTS, type HookPoint } from "@umt/contracts";
import type { Logger } from "pino";
import type { HookContext, HookValue, InterceptHook, InterceptPoint, RewriteHook, RewritePoint } from "./points.ts";
// 链式钩子(ADR-006,参考 MoviePilot ChainEventType):模块启动时注册,业务节点同步调用。
// - 按注册顺序执行(模块顺序 = SERVER_MODULES 顺序,同一模块内按调用顺序);
// - 每个钩子单独限时(缺省 2 秒),超时或抛错只记日志、跳过这个钩子,不影响主流程;
// - 没有注册时 has() 为 false,调用方直接用原值(零开销)。
// 钩子可能在业务事务里被调用:只做内存里的判断与改写,不要发长耗时外部请求、不要写库。
export const DEFAULT_HOOK_TIMEOUT_MS = 2_000;
export interface HookOptions {
/** 单次执行上限,缺省 DEFAULT_HOOK_TIMEOUT_MS */
timeoutMs?: number;
/** 日志里显示的名字,缺省 <模块>#<序号> */
name?: string;
}
export interface HookRegistrar {
rewrite<P extends RewritePoint>(point: P, fn: RewriteHook<P>, opts?: HookOptions): void;
intercept<P extends InterceptPoint>(point: P, fn: InterceptHook<P>, opts?: HookOptions): void;
}
interface Entry {
owner: string;
name: string;
timeoutMs: number;
// biome-ignore lint/suspicious/noExplicitAny: 按挂点分表存,调用时由 run* 的类型参数约束
fn: (...args: any[]) => unknown;
}
const TIMEOUT = Symbol("timeout");
const FAILED = Symbol("failed");
export interface HookRejection {
reason: string;
/** 拒绝的钩子名 */
by: string;
}
export class HookRegistry {
private readonly hooks = new Map<HookPoint, Entry[]>();
constructor(private readonly logger?: Pick<Logger, "warn">) {}
/** 给某个模块的注册入口(owner 进日志) */
forModule(owner: string): HookRegistrar {
const add = (point: HookPoint, kind: "rewrite" | "intercept", fn: Entry["fn"], opts: HookOptions = {}) => {
if (HOOK_POINTS[point].kind !== kind)
throw new Error(`钩子挂点 ${point} 是 ${HOOK_POINTS[point].kind} 类，不能用 ${kind} 注册`);
const list = this.hooks.get(point) ?? [];
list.push({
owner,
name: opts.name ?? `${owner}#${list.filter((e) => e.owner === owner).length + 1}`,
timeoutMs: opts.timeoutMs ?? DEFAULT_HOOK_TIMEOUT_MS,
fn,
});
this.hooks.set(point, list);
};
return {
rewrite: (point, fn, opts) => add(point, "rewrite", fn, opts),
intercept: (point, fn, opts) => add(point, "intercept", fn, opts),
};
}
has(point: HookPoint): boolean {
return (this.hooks.get(point)?.length ?? 0) > 0;
}
/** 已注册的钩子(挂点 → 钩子名),给功能清单与诊断用 */
list(): Record<string, string[]> {
return Object.fromEntries([...this.hooks].map(([p, l]) => [p, l.map((e) => e.name)]));
}
/** 依次改写;某个钩子超时 / 抛错 / 结果不被 accept 接受时跳过它,沿用上一个值 */
async runRewrite<P extends RewritePoint>(
point: P,
value: HookValue<P>,
context: HookContext<P>,
accept?: (v: HookValue<P>) => boolean,
): Promise<HookValue<P>> {
let cur = value;
for (const e of this.hooks.get(point) ?? []) {
const r = await this.call(point, e, [cur, context]);
if (r === undefined || r === TIMEOUT || r === FAILED) continue;
if (accept && !accept(r as HookValue<P>)) {
this.logger?.warn({ point, hook: e.name }, "钩子返回的结果不可用，已跳过");
continue;
}
cur = r as HookValue<P>;
}
return cur;
}
/** 依次判定;第一个返回原因的钩子拒绝,后面的不再执行 */
async runIntercept<P extends InterceptPoint>(point: P, context: HookContext<P>): Promise<HookRejection | null> {
for (const e of this.hooks.get(point) ?? []) {
const r = await this.call(point, e, [context]);
if (typeof r === "string" && r.trim()) return { reason: r.trim(), by: e.name };
}
return null;
}
private async call(point: HookPoint, e: Entry, args: unknown[]): Promise<unknown> {
let timer: ReturnType<typeof setTimeout> | undefined;
try {
const r = await Promise.race([
Promise.resolve().then(() => e.fn(...args)),
new Promise<typeof TIMEOUT>((resolve) => {
timer = setTimeout(() => resolve(TIMEOUT), e.timeoutMs);
}),
]);
if (r === TIMEOUT) this.logger?.warn({ point, hook: e.name, timeoutMs: e.timeoutMs }, "钩子超时，已跳过");
return r;
} catch (err) {
this.logger?.warn(
{ point, hook: e.name, err: { message: err instanceof Error ? err.message : String(err) } },
"钩子出错，已跳过",
);
return FAILED;
} finally {
clearTimeout(timer);
}
}
}
apps/server/src/http/static.ts
+1 -1
View File
@@ -1,5 +1,5 @@
import type { Hono } from "hono";
import { serveStatic } from "hono/deno";
import { serveStatic } from "hono/bun";
/**
* 托管 React Router SPA 构建产物(build/client):存在的文件直接返回,
apps/server/src/logger.ts
+1 -1
View File
@@ -71,7 +71,7 @@ export function createLogger(opts: LoggerOptions = {}): Logger {
pino({ ...options, transport: { target: "pino-pretty", options: { translateTime: "SYS:HH:MM:ss" } } }),
);
}
// 同步写 stdout:进程随后 Deno.exit() 时(迁移/自检子命令、优雅退出)不丢尾部日志
// 同步写 stdout:进程随后 process.exit() 时(迁移/自检子命令、优雅退出)不丢尾部日志
return redactChildBindings(pino(options, pino.destination({ dest: 1, sync: true })));
}
apps/server/src/main.ts
+28 -26
View File
@@ -1,5 +1,7 @@
import "reflect-metadata";
import { parseArgs } from "node:util";
import denoJson from "../deno.json" with { type: "json" };
import { runtimeReadTextFile, serve } from "@umt/runtime";
import packageJson from "../package.json" with { type: "json" };
import type { Config } from "./config.ts";
import type { MappingExport } from "./ops.ts";
@@ -34,7 +36,7 @@ type OneShotRole = (typeof OneShot)[number];
const isOneShot = (r: Role): r is OneShotRole => (OneShot as readonly Role[]).includes(r);
const { values } = parseArgs({
args: Deno.args,
args: process.argv.slice(2),
options: {
role: { type: "string", default: "all" },
out: { type: "string" },
@@ -46,32 +48,32 @@ const { values } = parseArgs({
const role = values.role as Role;
if (!Roles.includes(role)) {
console.error(`--role 必须是 ${Roles.join("|")}`);
Deno.exit(2);
process.exit(2);
}
// fsproxy 执行子进程:必须在加载 db/任务引擎/模块之前分派——每个文件操作起一次,只加载 fs 操作表,启动保持轻量
if (role === "fsworker") {
const { fsWorkerMain } = await import("@umt/adapters/fsworker");
await fsWorkerMain();
Deno.exit(0);
process.exit(0);
}
// 沙箱子进程:同样在加载其余模块前分派,只加载沙箱任务表
if (role === "sandbox") {
const { sandboxWorkerMain } = await import("./sandbox/worker.ts");
await sandboxWorkerMain();
Deno.exit(0);
process.exit(0);
}
// 健康探测:只打本机回环端口,不需要完整配置(HEALTHCHECK 用,镜像里不装 curl)
if (role === "probe") {
const port = Deno.env.get("PORT") ?? "4380";
const port = process.env.PORT ?? "4380";
try {
const res = await fetch(`http://127.0.0.1:${port}/api/v1/health`, { signal: AbortSignal.timeout(3000) });
const body = (await res.json()) as { data?: { status?: string } };
Deno.exit(res.ok && body.data?.status === "ok" ? 0 : 1);
process.exit(res.ok && body.data?.status === "ok" ? 0 : 1);
} catch {
Deno.exit(1);
process.exit(1);
}
}
@@ -98,16 +100,16 @@ try {
config = loadConfig();
} catch (err) {
console.error((err as Error).message);
Deno.exit(2);
process.exit(2);
}
// 运行期写入统一落在 DATA_DIR;Agent 会话目录经环境变量传给 agent 模块(它只读 AGENT_SESSION_DIR)
const paths = dataPaths(config);
Deno.env.set("AGENT_SESSION_DIR", paths.sessionDir);
process.env.AGENT_SESSION_DIR = paths.sessionDir;
const logger = createLogger({
level: config.LOG_LEVEL,
pretty: config.NODE_ENV === "development" && Deno.stdout.isTerminal(),
pretty: config.NODE_ENV === "development" && Boolean(process.stdout.isTTY),
});
const db = createDb({ connectionString: config.DATABASE_URL });
@@ -121,7 +123,7 @@ if (isOneShot(role)) {
} finally {
await db.destroy();
}
Deno.exit(code);
process.exit(code);
}
async function runOneShot(r: OneShotRole): Promise<number> {
@@ -155,9 +157,9 @@ async function runOneShot(r: OneShotRole): Promise<number> {
}
await auditStorageWarnings(db, storage.warnings).catch((err) => logger.warn({ err }, "存储告警写审计失败"));
// 验收开关误设提示(验收 07 S-10):只有 UMT_FS_TEST_HOOKS=1 时才真正生效
if (Deno.env.get("UMT_FS_TEST_HANG_AFTER_BYTES"))
if (process.env.UMT_FS_TEST_HANG_AFTER_BYTES)
logger.warn(
{ active: Deno.env.get("UMT_FS_TEST_HOOKS") === "1" },
{ active: process.env.UMT_FS_TEST_HOOKS === "1" },
"设置了 UMT_FS_TEST_HANG_AFTER_BYTES（fsproxy 挂死验收开关）；生产环境不要设置，生效时所有数据类文件操作都会卡死到被 SIGKILL",
);
for (const p of problems) logger.error({ check: p.check }, p.message);
@@ -175,11 +177,11 @@ async function runOneShot(r: OneShotRole): Promise<number> {
}
case "export-mappings": {
const out = values.out ?? `${paths.exportsDir}/mappings-${new Date().toISOString().replace(/[:.]/g, "-")}.json`;
const data = await exportMappings(db, denoJson.version);
const data = await exportMappings(db, packageJson.version);
await writeMappingExport(out, data);
logger.info({ out, counts: data.counts }, "映射清单已导出");
if (!values.compare) return 0;
const base = JSON.parse(await Deno.readTextFile(values.compare)) as MappingExport;
const base = JSON.parse(await runtimeReadTextFile(values.compare)) as MappingExport;
const diff = diffCounts(base.counts, data.counts);
if (diff.length) {
logger.error({ diff }, "映射清单行数与基准不一致");
@@ -206,7 +208,7 @@ try {
} catch (err) {
logger.error({ err }, "SSE 总线启动失败");
await db.destroy();
Deno.exit(1);
process.exit(1);
}
const stops: Array<() => Promise<void>> = [];
@@ -217,7 +219,7 @@ try {
} catch (err) {
logger.error({ err }, "启动装配失败");
await db.destroy();
Deno.exit(1);
process.exit(1);
}
const { ctx, outbox, engine } = booted;
// 统一元数据缓存:LISTEN 跨进程失效(在线才启用进程内一级缓存)+ 命中统计定期落库
@@ -230,13 +232,13 @@ if (role === "api" || role === "all") {
bus,
db,
role,
version: denoJson.version,
version: packageJson.version,
...(config.WEB_ROOT ? { webRoot: config.WEB_ROOT } : {}),
modules: SERVER_MODULES,
ctx,
});
const ac = new AbortController();
const server = Deno.serve(
const server = serve(
{
port: config.PORT,
hostname: config.HOST,
@@ -250,7 +252,7 @@ if (role === "api" || role === "all") {
app.fetch,
);
stops.push(async () => {
// abort 后 Deno.serve 会断开 SSE 等长连接,finished 在所有连接关闭后 resolve
// abort 后 serve 会断开 SSE 等长连接,finished 在所有连接关闭后 resolve
ac.abort();
await server.finished;
});
@@ -280,18 +282,18 @@ if (role === "worker" || role === "worker:agent" || role === "all") {
}
let stopping = false;
async function shutdown(signal: Deno.Signal) {
async function shutdown(signal: NodeJS.Signals) {
if (stopping) return;
stopping = true;
logger.info({ signal }, "正在退出");
// 兜底:10 秒内没退干净就强退,避免容器 stop 超时被 SIGKILL
const force = setTimeout(() => Deno.exit(1), 10_000);
const force = setTimeout(() => process.exit(1), 10_000);
await Promise.allSettled(stops.map((s) => s()));
await engine.stop().catch((err) => logger.error({ err }, "任务引擎关闭失败"));
await bus.stop().catch(() => {});
await db.destroy();
clearTimeout(force);
Deno.exit(0);
process.exit(0);
}
Deno.addSignalListener("SIGINT", () => void shutdown("SIGINT"));
Deno.addSignalListener("SIGTERM", () => void shutdown("SIGTERM"));
process.on("SIGINT", () => void shutdown("SIGINT"));
process.on("SIGTERM", () => void shutdown("SIGTERM"));
apps/server/src/modules/acquisition-organize/capabilities.ts
+2 -2
View File
@@ -9,11 +9,11 @@ import {
import { capabilitySpec, type DownloadStatus } from "@umt/contracts";
import { sql } from "@umt/db";
import { normalizeTitle, transition } from "@umt/domain";
import { coveringGrant, type PlanBinding, planBinding } from "../agent/authorize.ts";
import { previewPolicy, publishRevision } from "./routes/rules.ts";
import { coveringGrant, type PlanBinding, planBinding } from "../../authorization/grants.ts";
import { evaluateAcquisition, publishedAcquisitionPolicies } from "./services/acquisition.ts";
import { type AoDeps, enqueueTracked, ensureTaskRow, markTask } from "./services/common.ts";
import { type FrozenPlan, planImport } from "./services/organize.ts";
import { previewPolicy, publishRevision } from "./services/policies.ts";
import { assignOrganizeIdentity, OrganizeDecisionError } from "./services/takeover.ts";
import { errMessage, hashJson, jsonb, toApi, toApiList } from "./util.ts";
apps/server/src/modules/acquisition-organize/routes/rule-groups.ts
+10 -20
View File
@@ -16,6 +16,7 @@ import type { ModuleApp } from "../../types.ts";
import { contextOf, parsedOf } from "../services/acquisition.ts";
import type { AoDeps } from "../services/common.ts";
import {
assertRuleGroupIds,
checkAtomPatterns,
loadRuleCatalog,
normalizeGroupInput,
@@ -98,28 +99,15 @@ export async function checkSuppression(trx: Trx, fingerprint: string, source: st
/** 被收录版本 / 订阅显式引用的次数 */
async function referenceCount(trx: Trx, id: string): Promise<number> {
const r = await sql<{ n: number }>`
select (select count(*) from acquisition_slot where rule_group_ids @> ${JSON.stringify([id])}::jsonb)
select (select count(*) from acquisition_slot where (rule_group_ids || coalesce(upgrade_threshold->'ruleGroupIds', '[]'::jsonb)) @> ${JSON.stringify([id])}::jsonb)
+ (select count(*) from subscription where params -> 'ruleGroupIds' @> ${JSON.stringify([id])}::jsonb)
+ (select count(*) from policy_revision p where p.kind = 'acquisition' and p.status in ('draft', 'published')
and exists (select 1 from jsonb_array_elements(coalesce(p.rules->'defaultSlots', '[]'::jsonb) || coalesce(p.rules->'newTitle'->'slots', '[]'::jsonb)) t
where (coalesce(t->'ruleGroupIds', '[]'::jsonb) || coalesce(t->'upgradeThreshold'->'ruleGroupIds', '[]'::jsonb)) @> ${JSON.stringify([id])}::jsonb))
as n`.execute(trx);
return Number(r.rows[0]?.n ?? 0);
}
async function checkIdsExist(exec: Trx | AoDeps["ctx"]["db"], ids: readonly string[]) {
if (!ids.length) return;
const rows = await exec
.selectFrom("rule_group")
.select("id")
.where("id", "in", [...ids])
.where("deleted_at", "is", null)
.execute();
const have = new Set(rows.map((r) => r.id));
const missing = ids.filter((x) => !have.has(x));
if (missing.length) throw new ApiException("bad_request", `规则组不存在：${missing.join(", ")}`);
}
/** 收录版本 / 订阅引用的规则组 id 校验(供 slots 路由复用) */
export const assertRuleGroupIds = checkIdsExist;
const TRACKED_FIELDS = [
"name",
"description",
@@ -204,7 +192,7 @@ export function registerRuleGroupRoutes(app: ModuleApp, d: AoDeps) {
.transaction()
.execute(async (trx) => {
const all = [...body.acquisition, ...body.upgrade, ...body.search];
await checkIdsExist(trx, all);
await assertRuleGroupIds(trx, all);
const rows = await trx
.selectFrom("rule_group")
.select(["id", "kind"])
@@ -528,7 +516,7 @@ export function registerRuleGroupRoutes(app: ModuleApp, d: AoDeps) {
.executeTakeFirst();
if (!cur) notFound("规则组");
const refs = await referenceCount(trx, cur!.id);
if (refs > 0) conflict(`规则组被 ${refs} 个收录版本或订阅引用，先解除引用`);
if (refs > 0) conflict(`规则组被 ${refs} 个收录版本、订阅或策略引用，先解除引用`);
const row = await trx
.updateTable("rule_group")
.set({
@@ -603,7 +591,9 @@ async function kickReevaluate(d: AoDeps, groupId: string) {
const slots = await d.ctx.db
.selectFrom("acquisition_slot")
.select("id")
.where(sql<boolean>`rule_group_ids @> ${JSON.stringify([groupId])}::jsonb`)
.where(
sql<boolean>`(rule_group_ids || coalesce(upgrade_threshold->'ruleGroupIds', '[]'::jsonb)) @> ${JSON.stringify([groupId])}::jsonb`,
)
.limit(50)
.execute();
for (const s of slots)
apps/server/src/modules/acquisition-organize/routes/rules.ts
+7 -129
View File
@@ -7,24 +7,17 @@ import {
type PreferenceTag,
} from "@umt/contracts";
import { type DB, type PreferenceSuppressionRow, type RuleGroupRow, sql, type Transaction } from "@umt/db";
import { evaluateCandidate, type RuleExpr, type RuleLayer, serializeLayers } from "@umt/domain";
import { slotSpecFromRow } from "@umt/mapping";
import { type RuleExpr, type RuleLayer, serializeLayers } from "@umt/domain";
import type { Context } from "hono";
import type { AuthEnv } from "../../../http/auth.ts";
import { ApiException } from "../../../http/errors.ts";
import { titleParser } from "../../sources-identity/service/identify-words-cache.ts";
import type { ModuleApp } from "../../types.ts";
import type { AoDeps, Exec } from "../services/common.ts";
import { normalizeGroupInput } from "../services/rules.ts";
import { previewPolicy, publishRevision } from "../services/policies.ts";
import { assertRuleGroupIds, normalizeGroupInput, upgradeRuleGroupIds } from "../services/rules.ts";
import { actorOf, conflict, errMessage, hashJson, jsonb, notFound, toApi, toApiList } from "../util.ts";
import { mount, paginate } from "./helpers.ts";
import {
assertRuleGroupIds,
checkSuppression,
liftSuppressions,
registerRuleGroupRoutes,
suppressRuleGroup,
} from "./rule-groups.ts";
import { checkSuppression, liftSuppressions, registerRuleGroupRoutes, suppressRuleGroup } from "./rule-groups.ts";
import { registerSubscriptionRoutes } from "./subscriptions.ts";
// 规则 API:收录 Slot、策略版本(草案→发布→回滚,发布即持续授权)、偏好标签(规则组别名)与抑制记录;
@@ -151,7 +144,7 @@ export function registerRuleRoutes(app: ModuleApp, d: AoDeps) {
.where("id", "=", body.mediaEntityId)
.executeTakeFirst();
if (!e) throw new ApiException("bad_request", "mediaEntityId 不存在");
await assertRuleGroupIds(db(), body.ruleGroupIds ?? []);
await assertRuleGroupIds(db(), [...(body.ruleGroupIds ?? []), ...upgradeRuleGroupIds(body.upgradeThreshold)]);
const r = await db()
.insertInto("acquisition_slot")
.values({
@@ -183,7 +176,8 @@ export function registerRuleRoutes(app: ModuleApp, d: AoDeps) {
Api.acquisitionSlots.update,
async (_c, { params, body }) => {
const { expectedRevision, ...b } = body;
if (b.ruleGroupIds !== undefined) await assertRuleGroupIds(db(), b.ruleGroupIds);
if (b.ruleGroupIds !== undefined || b.upgradeThreshold !== undefined)
await assertRuleGroupIds(db(), [...(b.ruleGroupIds ?? []), ...upgradeRuleGroupIds(b.upgradeThreshold)]);
const r = await db()
.updateTable("acquisition_slot")
.set({
@@ -745,119 +739,3 @@ export function registerRuleRoutes(app: ModuleApp, d: AoDeps) {
// 规则组 /rules/*:含 /rules/{id},必须最后注册(静态的 /rules/policies、/rules/suppressions 在前)
registerRuleGroupRoutes(app, d);
}
/** 发布:旧 published → superseded/rolled_back;吊销旧授权;新版本生成持续授权 automation_grant */
export async function publishRevision(
trx: Transaction<DB>,
id: string,
policyKey: string,
by: string,
oldStatus: "superseded" | "rolled_back",
) {
const old = await trx
.selectFrom("policy_revision")
.select(["id"])
.where("policy_key", "=", policyKey)
.where("status", "=", "published")
.execute();
const now = new Date();
for (const o of old as { id: string }[]) {
await trx.updateTable("policy_revision").set({ status: oldStatus }).where("id", "=", o.id).execute();
await trx
.updateTable("automation_grant")
.set({ status: "revoked", revoked_at: now, revoked_by: by })
.where("policy_revision_id", "=", o.id)
.where("status", "=", "active")
.execute();
}
const row = await trx
.updateTable("policy_revision")
.set({ status: "published", published_at: now, published_by: by })
.where("id", "=", id)
.returningAll()
.executeTakeFirstOrThrow();
const rules = (row.rules ?? {}) as { budget?: unknown };
await trx
.insertInto("automation_grant")
.values({
action: `policy.${row.kind}`,
policy_revision_id: row.id,
scope: jsonb(row.scope),
budget: jsonb(rules.budget ?? {}),
granted_by: by,
reason: `发布策略 ${policyKey} r${row.revision}`,
})
.execute();
return row;
}
/** 预览:用规则里的 Slot 模板评估最近已识别的发布(确定性,不产生副作用) */
export async function previewPolicy(d: AoDeps, rules: unknown) {
const r = AcquisitionPolicyRules.safeParse(rules);
const templates = r.success ? [...(r.data.defaultSlots ?? []), ...(r.data.newTitle?.slots ?? [])] : [];
const rels = await d.ctx.db
.selectFrom("source_release")
.select(["id", "title", "subtitle", "size_bytes"])
.where("resolution_status", "=", "resolved")
.orderBy("first_seen_at", "desc")
.limit(200)
.execute();
let matched = 0;
let excluded = 0;
const samples: {
sourceReleaseId: string;
title: string;
sizeBytes: number | null;
template: string;
decision: string;
reasons: string[];
}[] = [];
const parse = await titleParser(d.ctx.db);
for (const rel of rels) {
const parsed = {
...parse(rel.title, rel.subtitle),
...(rel.size_bytes !== null ? { sizeBytes: rel.size_bytes } : {}),
};
for (const t of templates) {
const spec = slotSpecFromRow({
id: "preview",
name: t.name,
hardConditions: t.hardConditions ?? {},
softPreferences: t.softPreferences ?? {},
upgradeThreshold: t.upgradeThreshold ?? {},
completion: t.completion ?? {},
fallback: t.fallback ?? {},
});
const ev = evaluateCandidate(spec, parsed, null);
if (ev.decision === "accept") matched++;
else excluded++;
if (samples.length < 20) {
const reasons =
ev.decision === "reject"
? ev.violations.map((v) => `${v.field}:${v.code}`)
: ev.decision === "unknown_field"
? ev.missing.map((m) => `missing:${m}`)
: "reasons" in ev
? (ev.reasons ?? [])
: [];
samples.push({
sourceReleaseId: rel.id,
title: rel.title,
sizeBytes: rel.size_bytes,
template: t.name,
decision: ev.decision,
reasons,
});
}
}
}
return {
evaluatedAt: new Date().toISOString(),
sampleSize: rels.length,
templates: templates.length,
matched,
excluded,
samples,
valid: r.success,
};
}
apps/server/src/modules/acquisition-organize/routes/subscriptions.ts
+5 -3
View File
@@ -17,7 +17,7 @@ import { ApiException } from "../../../http/errors.ts";
import { emitMediaWatched } from "../../sources-identity/service/lifecycle-events.ts";
import { enqueueWatchedRefresh } from "../../sources-identity/service/watched.ts";
import type { ModuleApp } from "../../types.ts";
import { ensureSubscriptionSlot } from "../services/acquisition.ts";
import { collectionContext, ensureSubscriptionSlot } from "../services/collection-strategy.ts";
import type { AoDeps, Exec } from "../services/common.ts";
import {
buildSubscriptionModels,
@@ -352,6 +352,7 @@ export function registerSubscriptionRoutes(app: ModuleApp, d: AoDeps) {
// pending:TMDB 未就绪(未建档,或剧集没有已接受的 TMDB 身份 → 总集数无来源):照常搜索/评估,不能完成
const ready = await tmdbReady(db(), body.mediaEntityId ?? null, body.mediaKind);
const now = d.ports.now();
const context = await collectionContext(d, body.mediaEntityId ?? null);
const row = await db()
.transaction()
.execute(async (trx) => {
@@ -376,7 +377,7 @@ export function registerSubscriptionRoutes(app: ModuleApp, d: AoDeps) {
.returningAll()
.executeTakeFirstOrThrow();
// 订阅总有自己的收录版本:策略模板,没有就建一个默认版本(质量预设随后由 applyDerived 写入)
await ensureSubscriptionSlot(trx, r);
await ensureSubscriptionSlot(trx, r, context);
await applyDerived(trx, r, norm, now);
// 生命周期广播(ADR-006):订阅已建;关联作品 = 作品进入关注
await emitSubscriptionEvent(d, trx, "subscription.created", r);
@@ -419,6 +420,7 @@ export function registerSubscriptionRoutes(app: ModuleApp, d: AoDeps) {
seasonUnitId: cur.season_unit_id,
excludeId: cur.id,
});
const context = await collectionContext(d, cur.media_entity_id);
const r = await db()
.transaction()
.execute(async (trx) => {
@@ -438,7 +440,7 @@ export function registerSubscriptionRoutes(app: ModuleApp, d: AoDeps) {
.returningAll()
.executeTakeFirstOrThrow();
// 旧订阅没有自己的收录版本:编辑时补建(质量预设 / 升级策略才有地方写)
if (r.status !== "cancelled") await ensureSubscriptionSlot(trx, r);
if (r.status !== "cancelled") await ensureSubscriptionSlot(trx, r, context);
if (norm) await applyDerived(trx, r, norm, d.ports.now(), subParams(cur));
else if (body.status !== undefined) await syncPrivateRuleGroup(trx, r, d.ports.now());
await emitSubscriptionChange(trx, cur, r, Object.keys(body));
apps/server/src/modules/acquisition-organize/services/acquisition.ts
+102 -159
View File
@@ -1,10 +1,4 @@
import {
type AcquisitionPolicyRules,
type AcquisitionPriority,
type CandidateEvaluation,
lifecycleKey,
type SlotTemplate,
} from "@umt/contracts";
import { type AcquisitionPriority, type CandidateEvaluation, lifecycleKey } from "@umt/contracts";
import type { AcquisitionSlotRow, MediaEntityRow, SourceReleaseRow, SubscriptionRow } from "@umt/db";
import { sql } from "@umt/db";
import {
@@ -24,13 +18,22 @@ import {
type SlotEvaluation,
type SlotSpec,
type SlotState,
seasonReleaseKey,
seasonWildcard,
selectAcquisitionPolicy,
transition,
unitsOverlap,
upgradeSpec,
} from "@umt/domain";
import { toContractSlotEvaluation } from "@umt/mapping";
import { hasHook, rewriteHook } from "../../../hooks/index.ts";
import { contextOf, episodeResolutionOf, parsedOf, unitKeysOf } from "../../sources-identity/service/release-facts.ts";
import { hashJson, jsonb, startOfDay } from "../util.ts";
import {
collectionContext,
createSlotsFromTemplates,
ensureSubscriptionSlot,
publishedAcquisitionPolicies,
} from "./collection-strategy.ts";
import {
type AoDeps,
automationBlock,
@@ -43,15 +46,16 @@ import {
loadInstances,
pickInstance,
recordCandidate,
upsertDecision,
} from "./common.ts";
import { autoFillCheck, type ForceFill } from "./evaluate-autofill.ts";
import { emitSubscriptionEvent } from "./lifecycle-events.ts";
import { contextOf, episodeResolutionOf, parsedOf, unitKeysOf } from "./release-facts.ts";
import { entityOriginalLanguage, loadRuleCatalog, type RuleCatalog } from "./rules.ts";
import {
clearDeferral,
clearReleaseDeferrals,
clearSubscriptionDeferrals,
DEFAULT_SLOT_REEVALUATE_MS,
gateRetryDelayMs,
manualInflightPlans,
rangeOf,
@@ -67,10 +71,18 @@ import {
subSeasonOf,
TRACKING_SUB_STATUSES,
} from "./subscriptions.ts";
import { applySlotPreset } from "./subscriptions-params.ts";
export {
createSlotsFromTemplates,
ensureSubscriptionSlot,
publishedAcquisitionPolicies,
} from "./collection-strategy.ts";
import { missingReleaseFields } from "../../sources-identity/service/release-facts-admin.ts";
import { seasonReleaseTarget } from "./season-release.ts";
import { decideVolume, loadVolumes } from "./volumes.ts";
export { contextOf, parsedOf, unitKeysOf } from "./release-facts.ts";
export { contextOf, parsedOf, unitKeysOf } from "../../sources-identity/service/release-facts.ts";
// evaluateAcquisition:release.resolved → 匹配订阅/Slot → 模式 → 确定性评估 → 预算/缓存卷 → 冻结计划并预留。
// 拒绝与被挡住的原因都写成候选评估记录(artifact),供作品详情"为什么选/不选"。
@@ -172,127 +184,30 @@ const firstReleaseDate = (e: MediaEntityRow): string | undefined => {
return e.year ? `${e.year}-01-01` : undefined;
};
/** 已发布的收录策略(kind=acquisition) */
export async function publishedAcquisitionPolicies(exec: Exec) {
const rows = await exec
.selectFrom("policy_revision")
.selectAll()
.where("kind", "=", "acquisition")
.where("status", "=", "published")
.execute();
return rows.map((r) => ({
row: r,
rules: (r.rules ?? {}) as AcquisitionPolicyRules,
scope: (r.scope ?? {}) as { mediaKinds?: string[] },
}));
}
const subKindOf = (k: MediaEntityRow["kind"]): "movie" | "series" | "variety" =>
k === "movie" ? "movie" : k === "variety" ? "variety" : "series";
export async function createSlotsFromTemplates(
exec: Exec,
input: {
entityId: string;
subscriptionId: string | null;
policyRevisionId: string | null;
templates: readonly SlotTemplate[];
},
): Promise<number> {
// 收录版本名在作品内唯一(media_entity_id, content_unit_id, name):旧订阅留下的同名版本不挡新订阅,改名加序号
const taken = new Set(
(
await exec
.selectFrom("acquisition_slot")
.select("name")
.where("media_entity_id", "=", input.entityId)
.where("content_unit_id", "is", null)
.execute()
).map((r) => r.name),
);
let n = 0;
for (const [i, t] of input.templates.entries()) {
let name = t.name;
for (let k = 2; taken.has(name) && k < 1000; k++) name = `${t.name} ${k}`;
taken.add(name);
const row = await exec
.insertInto("acquisition_slot")
.values({
// 同一事务建的收录版本 created_at 相同:按模板顺序给 position,第一个为主版本
position: i,
media_entity_id: input.entityId,
subscription_id: input.subscriptionId,
policy_revision_id: input.policyRevisionId,
name,
hard_conditions: jsonb(t.hardConditions),
soft_preferences: jsonb(t.softPreferences),
fallback: jsonb(t.fallback),
upgrade_threshold: jsonb(t.upgradeThreshold),
completion: jsonb(t.completion),
rule_group_ids: JSON.stringify(t.ruleGroupIds ?? []),
})
.onConflict((oc) => oc.doNothing())
.returning("id")
.executeTakeFirst();
if (row) n++;
}
return n;
}
/** 追踪中的订阅里没有自己收录版本的,逐个补建;返回是否建了 */
async function backfillSubscriptionSlots(
db: AoDeps["ctx"]["db"],
d: AoDeps,
subs: readonly SubscriptionRow[],
slots: readonly Pick<AcquisitionSlotRow, "subscription_id">[],
): Promise<boolean> {
let made = 0;
for (const s of subs)
if (s.media_entity_id && !slots.some((x) => x.subscription_id === s.id)) made += await ensureSlotLocked(db, s);
if (s.media_entity_id && !slots.some((x) => x.subscription_id === s.id)) made += await ensureSlotLocked(d, s);
return made > 0;
}
/** 评估路径上补建:锁订阅行,免得并发的两次评估各建一个 */
function ensureSlotLocked(db: AoDeps["ctx"]["db"], sub: SubscriptionRow): Promise<number> {
return db.transaction().execute(async (trx) => {
async function ensureSlotLocked(d: AoDeps, sub: SubscriptionRow): Promise<number> {
const context = await collectionContext(d, sub.media_entity_id);
return d.ctx.db.transaction().execute(async (trx) => {
await trx.selectFrom("subscription").select("id").where("id", "=", sub.id).forUpdate().execute();
return ensureSubscriptionSlot(trx, sub);
return ensureSubscriptionSlot(trx, sub, context);
});
}
/** 订阅没有收录策略模板时的隐式缺省收录版本:无硬条件,表单的质量预设 / 包含排除随后由订阅参数写入 */
export const DEFAULT_SLOT_TEMPLATE: SlotTemplate = { name: "默认版本" };
/**
* 订阅总有属于自己的收录版本(H1):没有时按收录策略模板(指定的或第一个带 defaultSlots 的已发布策略)建,
* 没有模板就建一个隐式「默认版本」,再把订阅表单的质量预设写进去。作品已有别的订阅(旧季、已取消)的
* 收录版本不算——评估时订阅只用自己的收录版本。返回新建个数(已有则 0)。
*/
export async function ensureSubscriptionSlot(
exec: Exec,
sub: Pick<SubscriptionRow, "id" | "media_entity_id" | "params" | "policy_revision_id">,
): Promise<number> {
if (!sub.media_entity_id) return 0;
const own = await exec
.selectFrom("acquisition_slot")
.select("id")
.where("subscription_id", "=", sub.id)
.executeTakeFirst();
if (own) return 0;
const pols = await publishedAcquisitionPolicies(exec);
const pol =
(sub.policy_revision_id ? pols.find((p) => p.row.id === sub.policy_revision_id) : undefined) ??
pols.find((p) => p.rules.defaultSlots?.length);
const fromPolicy = !!pol?.rules.defaultSlots?.length;
const n = await createSlotsFromTemplates(exec, {
entityId: sub.media_entity_id,
subscriptionId: sub.id,
policyRevisionId: fromPolicy ? pol!.row.id : null,
templates: fromPolicy ? pol!.rules.defaultSlots! : [DEFAULT_SLOT_TEMPLATE],
});
if (n > 0) await applySlotPreset(exec, { subscriptionId: sub.id, prev: {}, next: subParams(sub) });
return n;
}
/**
* 预算用量。并发 = 在跑的下载任务(不含 stalled:无进展任务不占名额,否则卡住几个就把追更全挡住)
* + 已预留但还没建下载任务的计划(刚预留、等补投;防止一轮评估连续预留超过并发上限)。
@@ -392,7 +307,7 @@ async function reevaluate(c: Ctx, input: EvaluateInput): Promise<EvaluateOutcome
sub = sid ? await db.selectFrom("subscription").selectAll().where("id", "=", sid).executeTakeFirst() : undefined;
} else if (input.subscriptionId) {
sub = await db.selectFrom("subscription").selectAll().where("id", "=", input.subscriptionId).executeTakeFirst();
if (sub && (TRACKING_SUB_STATUSES as readonly string[]).includes(sub.status)) await ensureSlotLocked(db, sub);
if (sub && (TRACKING_SUB_STATUSES as readonly string[]).includes(sub.status)) await ensureSlotLocked(c.d, sub);
if (sub)
focus = await db
.selectFrom("acquisition_slot")
@@ -439,13 +354,8 @@ async function reevaluate(c: Ctx, input: EvaluateInput): Promise<EvaluateOutcome
rels.filter((x) => !seen.has(x.id)),
);
const ranked = await rankReleases(c, entity, sub, focus[0], [...deferred, ...fresh]);
const ordered = await selectResources(c, ranked, {
mediaEntityId: entity.id,
subscriptionId: sub?.id ?? null,
slotId: input.acquisitionSlotId ?? null,
});
const scope = input.acquisitionSlotId ? { slotId: input.acquisitionSlotId } : sub ? { subscriptionId: sub.id } : {};
for (const r of ordered) await evaluateRelease(c, r, [], scope);
for (const r of ranked) await evaluateRelease(c, r, [], scope);
return c.out;
}
@@ -513,33 +423,6 @@ async function rankReleases(
})).map((x) => byParsed.get(x.candidate)!);
}
/**
* 链式钩子 resource.selection(ADR-006):在择优排序之后调整顺序或过滤。钩子只能重排 / 删减,
* 返回里不认识的 ID 忽略、重复的只取第一次;没有钩子时原样返回。
*/
export async function selectResources(
c: Ctx,
rels: SourceReleaseRow[],
context: { mediaEntityId: string; subscriptionId: string | null; slotId: string | null },
): Promise<SourceReleaseRow[]> {
if (!hasHook(c.d.ctx, "resource.selection") || rels.length === 0) return rels;
const byId = new Map(rels.map((r) => [r.id, r]));
const picked = await rewriteHook(
c.d.ctx,
"resource.selection",
rels.map((r) => ({ sourceReleaseId: r.id, title: r.title, parsed: parsedOf(r) })),
context,
);
const out: SourceReleaseRow[] = [];
for (const p of picked) {
const r = byId.get(p.sourceReleaseId);
if (!r) continue;
byId.delete(p.sourceReleaseId);
out.push(r);
}
return out;
}
/** 评估范围:只评某个收录版本,或只评某个订阅的收录版本;缺省评作品的全部收录版本 */
interface EvalScope {
slotId?: string | undefined;
@@ -596,10 +479,14 @@ async function evaluateRelease(c: Ctx, rel: SourceReleaseRow, eventUnits: readon
.where("status", "=", "active")
.executeTakeFirst();
if (hasAsset) return skip("not_tracked");
const pol = (await publishedAcquisitionPolicies(db)).find(
(p) =>
p.rules.newTitle?.enabled &&
(!p.rules.newTitle.mediaKinds?.length || p.rules.newTitle.mediaKinds.includes(subKindOf(entity.kind))),
const context = await collectionContext(d, entity.id);
const pol = selectAcquisitionPolicy(
(await publishedAcquisitionPolicies(db)).filter(
(p) =>
p.rules.newTitle?.enabled &&
(!p.rules.newTitle.mediaKinds?.length || p.rules.newTitle.mediaKinds.includes(subKindOf(entity.kind))),
),
{ mediaKind: subKindOf(entity.kind), category: context?.category ?? null },
);
if (!pol?.rules.newTitle) return skip("not_tracked");
const m = determineMode({
@@ -687,7 +574,7 @@ async function evaluateRelease(c: Ctx, rel: SourceReleaseRow, eventUnits: readon
subs.filter((s) => (TRACKING_SUB_STATUSES as readonly string[]).includes(s.status)).map((s) => [s.id, s]),
);
// 一次性补建:旧版本建的订阅可能没有自己的收录版本(H1),首次评估时补上
if (await backfillSubscriptionSlots(db, [...activeSubs.values()], slots))
if (await backfillSubscriptionSlots(d, [...activeSubs.values()], slots))
slots = await db
.selectFrom("acquisition_slot")
.selectAll()
@@ -849,8 +736,8 @@ async function evaluateForSlot(c: Ctx, p: SlotInput): Promise<{ planId: string |
if (isMovie) {
if (inflight.length > 0 || BUSY.includes(slot.status)) return defer("slot_busy", []);
const current = await currentFacts(db, slot.current_asset_id);
if (current && slot.current_asset_id && rulesLaddered(spec)) {
const pri = await rulePriOfAsset(c, spec, slot.current_asset_id, ctx.originalLanguage);
if (current && slot.current_asset_id && rulesLaddered(upgradeSpec(spec))) {
const pri = await rulePriOfAsset(c, upgradeSpec(spec), slot.current_asset_id, ctx.originalLanguage);
if (pri !== undefined) current.pri = pri;
}
ev = evaluateCandidate(spec, parsed, slotSatisfied ? current : null, ctx);
@@ -902,11 +789,25 @@ async function evaluateForSlot(c: Ctx, p: SlotInput): Promise<{ planId: string |
return defer("units_inflight", reservedHit);
} else {
// 都已覆盖:分集洗版,只取能提升的单元(对照 MP get_best_version_interested_episodes)
if (spec.upgrade?.consistency === "season_release") {
const washing = upgradeSpec(spec);
const pri = effectiveLadder(washing, parsed, ctx, ruleChainOf(washing, parsed, ctx)).pri;
ctx.seasonReleaseTarget = await seasonReleaseTarget(
db,
slot,
p.releaseSeason,
washing,
{ ...parsed, pri },
ctx.originalLanguage,
);
}
const unitCurrent = await unitQuality(
db,
slot,
releaseKeys,
rulesLaddered(spec) ? (assetId) => rulePriOfAsset(c, spec, assetId, ctx.originalLanguage) : undefined,
rulesLaddered(upgradeSpec(spec))
? (assetId) => rulePriOfAsset(c, upgradeSpec(spec), assetId, ctx.originalLanguage)
: undefined,
);
const withCurrent = releaseKeys.filter((k) => unitCurrent[k]);
const firstCurrent = withCurrent.length
@@ -928,8 +829,30 @@ async function evaluateForSlot(c: Ctx, p: SlotInput): Promise<{ planId: string |
}
}
const flat = toContractSlotEvaluation(ev);
const evalExtra = { pri: flat.pri, trace: flat.trace };
const evalExtra = {
pri: flat.pri,
trace: flat.trace,
quality: { ...parsed, pri: flat.pri },
...(ctx.seasonReleaseTarget
? { seasonRelease: seasonReleaseKey(ctx.seasonReleaseTarget), season: p.releaseSeason }
: {}),
};
if (!flat.acceptable) {
if (!c.draft && ev.decision === "unknown_field") {
const missing = missingReleaseFields(rel, ev.missing).filter((f) =>
["resolution", "codec", "source", "group", "size", "subtitles", "seeders", "promotion"].includes(f),
);
if (missing.length)
await upsertDecision(d, db, {
kind: "other",
subjectType: "source_release",
subjectId: rel.id,
dedupeKey: `release_facts:${rel.id}`,
title: `资源属性缺失：${rel.title}`,
summary: "查证缺失属性后重新按收录规则评估",
evidence: { sourceReleaseId: rel.id, mediaEntityId: p.entity.id, slotId: slot.id, missing },
});
}
await note(c, {
...base,
decision: ev.decision as "reject" | "unknown_field",
@@ -1032,7 +955,8 @@ async function evaluateForSlot(c: Ctx, p: SlotInput): Promise<{ planId: string |
savePath: gate.savePath,
unitKeys: planUnits,
multiInflight: !isMovie,
...(fill.action === "fill" ? { priority: fill.priority, evaluationExtra: fill.evaluation } : {}),
...(fill.action === "fill" ? { priority: fill.priority } : {}),
evaluationExtra: { ...evalExtra, ...(fill.action === "fill" ? fill.evaluation : {}) },
});
if (r.conflict) return defer("units_inflight", r.conflict, mode);
if (r.retryAt) {
@@ -1305,10 +1229,12 @@ async function reserve(
unitKeys: p.unitKeys,
};
// 升级 = 替换已有资产;电影 Slot 缺口(无资产)的 mode 虽记 upgrade,用途仍是补缺
const purpose = p.mode === "upgrade" && p.slot.current_asset_id ? ("upgrade" as const) : ("fill" as const);
const purpose = p.ev.decision === "upgrade" ? ("upgrade" as const) : ("fill" as const);
const evaluation = jsonb({
...toContractSlotEvaluation(p.ev),
mode: p.mode,
unitKeys: p.unitKeys,
...(purpose === "upgrade" ? { automaticReplacement: true } : {}),
instanceId: p.instanceId,
volumeKey: p.volumeKey,
savePath: p.savePath,
@@ -1373,6 +1299,23 @@ async function reserve(
.executeTakeFirst();
if (!slot) return { planId: null };
const busy = BUSY.includes(slot.status);
if (p.evaluationExtra?.seasonRelease && typeof p.evaluationExtra.season === "number") {
const spec = upgradeSpec(
slotSpecWithRules(
await loadRuleCatalog(trx),
slot,
p.sub,
{ id: slot.media_entity_id, kind: p.sub?.media_kind ?? "series" },
p.rel.site_id,
),
);
const target = await seasonReleaseTarget(trx, slot, p.evaluationExtra.season, spec, {
...parsedOf(p.rel),
pri: "pri" in p.ev ? p.ev.pri : undefined,
});
if (!target || seasonReleaseKey(target) !== p.evaluationExtra.seasonRelease)
return { planId: null, retryAt: new Date(now.getTime() + DEFAULT_SLOT_REEVALUATE_MS) };
}
if (p.multiInflight) {
// Slot×单元预留:在 Slot 行锁内复核与其它在途计划的单元不相交
const others = await slotInflightPlans(trx, slot.id);
apps/server/src/modules/acquisition-organize/services/collection-strategy.ts
+144
View File
@@ -0,0 +1,144 @@
import {
type AcquisitionPolicyRules,
AcquisitionPolicyScope,
type SlotTemplate,
type SubscriptionParams,
} from "@umt/contracts";
import type { SubscriptionRow } from "@umt/db";
import { selectAcquisitionPolicy } from "@umt/domain";
import { ApiException } from "../../../http/errors.ts";
import { jsonb } from "../util.ts";
import { classifyEntity } from "./classification.ts";
import type { AoDeps, Exec } from "./common.ts";
import { applySlotPreset } from "./subscriptions-params.ts";
export async function collectionContext(d: AoDeps, entityId: string | null) {
if (!entityId) return undefined;
const entity = await d.ctx.db.selectFrom("media_entity").selectAll().where("id", "=", entityId).executeTakeFirst();
if (!entity) return undefined;
return {
mediaKind:
entity.kind === "movie"
? ("movie" as const)
: entity.kind === "variety"
? ("variety" as const)
: ("series" as const),
category: (await classifyEntity(d, entity)).category,
};
}
/** 已发布的收录策略(kind=acquisition) */
export async function publishedAcquisitionPolicies(exec: Exec) {
const rows = await exec
.selectFrom("policy_revision")
.selectAll()
.where("kind", "=", "acquisition")
.where("status", "=", "published")
.orderBy("published_at", "desc")
.orderBy("policy_key")
.execute();
return rows.map((r) => ({
row: r,
rules: (r.rules ?? {}) as AcquisitionPolicyRules,
id: r.id,
scope: AcquisitionPolicyScope.parse(r.scope ?? {}),
}));
}
export async function createSlotsFromTemplates(
exec: Exec,
input: {
entityId: string;
subscriptionId: string | null;
policyRevisionId: string | null;
templates: readonly SlotTemplate[];
},
): Promise<number> {
// 收录版本名在作品内唯一(media_entity_id, content_unit_id, name):旧订阅留下的同名版本不挡新订阅,改名加序号
const taken = new Set(
(
await exec
.selectFrom("acquisition_slot")
.select("name")
.where("media_entity_id", "=", input.entityId)
.where("content_unit_id", "is", null)
.execute()
).map((r) => r.name),
);
let n = 0;
for (const [i, t] of input.templates.entries()) {
let name = t.name;
for (let k = 2; taken.has(name) && k < 1000; k++) name = `${t.name} ${k}`;
taken.add(name);
const row = await exec
.insertInto("acquisition_slot")
.values({
// 同一事务建的收录版本 created_at 相同:按模板顺序给 position,第一个为主版本
position: i,
media_entity_id: input.entityId,
subscription_id: input.subscriptionId,
policy_revision_id: input.policyRevisionId,
name,
hard_conditions: jsonb(t.hardConditions),
soft_preferences: jsonb(t.softPreferences),
fallback: jsonb(t.fallback),
upgrade_threshold: jsonb(t.upgradeThreshold),
completion: jsonb(t.completion),
rule_group_ids: JSON.stringify(t.ruleGroupIds ?? []),
})
.onConflict((oc) => oc.doNothing())
.returning("id")
.executeTakeFirst();
if (row) n++;
}
return n;
}
/** 订阅没有收录策略模板时的隐式缺省收录版本:无硬条件,表单的质量预设 / 包含排除随后由订阅参数写入 */
export const DEFAULT_SLOT_TEMPLATE: SlotTemplate = { name: "默认版本" };
/**
* 订阅总有属于自己的收录版本(H1):没有时按收录策略模板(显式指定、分类默认、类型默认或全局默认的已发布策略)建,
* 没有模板就建一个隐式「默认版本」,再把订阅表单的质量预设写进去。作品已有别的订阅(旧季、已取消)的
* 收录版本不算——评估时订阅只用自己的收录版本。返回新建个数(已有则 0)。
*/
export async function ensureSubscriptionSlot(
exec: Exec,
sub: Pick<SubscriptionRow, "id" | "media_entity_id" | "params" | "policy_revision_id">,
context?: { mediaKind: "movie" | "series" | "variety"; category: string | null },
): Promise<number> {
if (!sub.media_entity_id) return 0;
const own = await exec
.selectFrom("acquisition_slot")
.select("id")
.where("subscription_id", "=", sub.id)
.executeTakeFirst();
if (own) return 0;
const pols = await publishedAcquisitionPolicies(exec);
const entity = await exec
.selectFrom("media_entity")
.select("kind")
.where("id", "=", sub.media_entity_id)
.executeTakeFirstOrThrow();
const pol = selectAcquisitionPolicy(
pols.filter((p) => p.rules.defaultSlots?.length),
{
explicitId: sub.policy_revision_id,
mediaKind:
context?.mediaKind ?? (entity.kind === "movie" ? "movie" : entity.kind === "variety" ? "variety" : "series"),
category: context?.category ?? null,
},
);
if (sub.policy_revision_id && !pol)
throw new ApiException("precondition_failed", "所选收录策略尚未发布或没有收录版本");
const fromPolicy = !!pol?.rules.defaultSlots?.length;
const n = await createSlotsFromTemplates(exec, {
entityId: sub.media_entity_id,
subscriptionId: sub.id,
policyRevisionId: fromPolicy ? pol!.row.id : null,
templates: fromPolicy ? pol!.rules.defaultSlots! : [DEFAULT_SLOT_TEMPLATE],
});
if (n > 0)
await applySlotPreset(exec, { subscriptionId: sub.id, prev: {}, next: (sub.params ?? {}) as SubscriptionParams });
return n;
}
apps/server/src/modules/acquisition-organize/services/common.ts
+12 -2
View File
@@ -37,7 +37,11 @@ export async function upsertDecision(d: AoDeps, exec: Exec, input: DecisionInput
${input.subjectType ?? null}, ${input.subjectId ?? null}, ${input.dedupeKey},
${jsonb(input.evidence)}::jsonb, ${JSON.stringify(input.options ?? [])}::jsonb, ${input.taskId ?? null})
on conflict (dedupe_key) where status in ('open', 'investigating', 'awaiting_user')
do update set attempts = decision.attempts + 1, evidence = excluded.evidence, summary = excluded.summary,
do update set attempts = decision.attempts + 1, evidence = coalesce(decision.evidence, '{}'::jsonb) || excluded.evidence ||
case when excluded.dedupe_key like 'release_facts:%' then jsonb_build_object('missing',
(select coalesce(jsonb_agg(distinct field), '[]'::jsonb) from jsonb_array_elements(
coalesce(decision.evidence->'missing', '[]'::jsonb) || coalesce(excluded.evidence->'missing', '[]'::jsonb)) field))
else '{}'::jsonb end, summary = excluded.summary,
subject_type = coalesce(excluded.subject_type, decision.subject_type),
subject_id = coalesce(excluded.subject_id, decision.subject_id),
-- 重新判定时选项随最新证据刷新(如接管识别的候选);没给选项的保留原有(如 Agent 补的"重新校验")
@@ -49,7 +53,13 @@ export async function upsertDecision(d: AoDeps, exec: Exec, input: DecisionInput
await d.ctx.outbox.emit(
exec,
"decision.created",
{ decisionId: row.id, kind: input.kind, severity: input.severity ?? "warning", title: input.title },
{
decisionId: row.id,
kind: input.kind,
severity: input.severity ?? "warning",
title: input.title,
dedupeKey: input.dedupeKey,
},
{ idempotencyKey: `decision.created:${row.id}` },
);
}
apps/server/src/modules/acquisition-organize/services/coverage.ts
+74 -18
View File
@@ -1,5 +1,4 @@
import type {
AcquisitionSlot,
CoverageColumn,
SubscriptionCoverage,
SubscriptionGroup,
@@ -14,15 +13,18 @@ import {
coverageRow,
episodeKey,
parseUnitKey,
type QualityFacts,
resolveAutoFill,
type SlotSpec,
seasonReleaseKey,
selectSeasonRelease,
subscriptionGroup,
type UnitRange,
upgradeSpec,
} from "@umt/domain";
import { slotSpecFromRow } from "@umt/mapping";
import { toApi } from "../util.ts";
import type { Exec } from "./common.ts";
import { defaultChainIds, loadRuleCatalog, type RuleCatalog, resolveRuleChain } from "./rules.ts";
import { INFLIGHT_PLAN_STATUSES, rangeOf, subParams } from "./subscriptions.ts";
import { loadRuleCatalog, type RuleCatalog, resolveRuleChain } from "./rules.ts";
import { INFLIGHT_PLAN_STATUSES, rangeOf, slotSpecWithRules, subParams } from "./subscriptions.ts";
// W9-E 覆盖矩阵与订阅总览(docs/design/subscriptions.md §1、§2.1、§5):
// 一批订阅一次取齐 收录版本 / 资产 / 覆盖 / 分集账本 / 在途计划 / 内容单元 / 延后(固定 8 条查询,与订阅数无关),
@@ -41,6 +43,7 @@ interface SlotModel {
ruleGroupIds: string[];
upgrade: { enabled: boolean; targetPri: number };
requiredEditions: string[];
spec: SlotSpec;
}
interface RowModel {
@@ -85,14 +88,16 @@ const segOf = (u: { segment: string } | undefined) =>
const editionKey = (e: string) => `edition:${e}`;
const ymd = (d: Date) => d.toISOString().slice(0, 10);
/** 订阅有效自动补全:params.autoFill → 规则链上后者优先的 auto_fill → 系统缺省关 */
/** 订阅有效自动补全:params.autoFill → 主版本收录链上后者优先的 auto_fill → 系统缺省关 */
export function subscriptionAutoFill(
cat: RuleCatalog,
sub: Pick<SubscriptionRow, "id" | "params" | "media_kind" | "media_entity_id">,
primarySlot?: Pick<AcquisitionSlotRow, "rule_group_ids">,
): SubscriptionOverviewItem["autoFill"] & { chainIds: string[] } {
const p = subParams(sub);
const chain = resolveRuleChain(cat, {
kind: "acquisition",
slotRuleGroupIds: primarySlot?.rule_group_ids as string[] | undefined,
subRuleGroupIds: p.ruleGroupIds,
subscriptionId: sub.id,
mediaKind: sub.media_kind,
@@ -175,9 +180,10 @@ export async function buildSubscriptionModels(
unit_keys: unknown;
plan_slot_id: string | null;
}>`
select a.id, a.media_entity_id, a.content_edition, a.media_info->'_umt'->'quality' as quality,
select a.id, a.media_entity_id, a.content_edition, coalesce(r.parsed, '{}'::jsonb) || coalesce(a.media_info->'_umt'->'quality', '{}'::jsonb) as quality,
a.media_info->'_umt'->'unitKeys' as unit_keys, p.slot_id as plan_slot_id
from media_asset a
left join source_release r on r.id = a.source_release_id
left join download_task dt on dt.id = a.download_task_id
left join acquisition_plan p on p.id = dt.acquisition_plan_id
where a.status = 'active' and a.parent_asset_id is null
@@ -205,7 +211,15 @@ export async function buildSubscriptionModels(
.leftJoin("download_task as dt", (j) =>
j.onRef("dt.acquisition_plan_id", "=", "p.id").on("dt.removed_at", "is", null),
)
.select(["p.id", "p.slot_id", "p.status", "p.unit_keys", "dt.status as dt_status"])
.leftJoin("source_release as r", "r.id", "p.source_release_id")
.select([
"p.id",
"p.slot_id",
"p.status",
"p.unit_keys",
"dt.status as dt_status",
sql<QualityFacts>`coalesce(p.evaluation->'quality', r.parsed, '{}'::jsonb)`.as("quality"),
])
.where("p.slot_id", "in", slotIds)
.where("p.status", "in", [...INFLIGHT_PLAN_STATUSES])
.execute(),
@@ -251,6 +265,9 @@ export async function buildSubscriptionModels(
? "verifying"
: "downloading";
if (!r.slot_id) continue;
const resolutions = ((slots.find((s) => s.id === r.slot_id)?.hard_conditions ?? {}) as { resolutions?: string[] })
.resolutions;
if (resolutions?.length && !resolutions.includes(r.quality.resolution ?? "")) continue;
const list = inflightBySlot.get(r.slot_id) ?? [];
const prev = list.find((x) => x.id === r.id);
if (!prev) list.push({ id: r.id, stage, unitKeys: r.unit_keys });
@@ -268,10 +285,11 @@ export async function buildSubscriptionModels(
/** 资产集合 → 覆盖事实(单元键 → 资产/质量) */
const coveredOf = (assetSet: Iterable<string>, ledger: typeof ledgerRows): Map<string, CoveredFact> => {
const m = new Map<string, CoveredFact>();
const allowedAssets = new Set(assetSet);
const put = (k: string, f: CoveredFact, override = false) => {
if (override || !m.has(k)) m.set(k, f);
};
for (const id of assetSet) {
for (const id of allowedAssets) {
const a = assetById.get(id);
if (!a) continue;
const q = qualityOf(a.quality);
@@ -290,6 +308,7 @@ export async function buildSubscriptionModels(
}
// 分集账本优先(质量与 pri 按收录版本记)
for (const r of ledger) {
if (!r.media_asset_id || !allowedAssets.has(r.media_asset_id) || !assetById.has(r.media_asset_id)) continue;
const f = { assetId: r.media_asset_id, quality: qualityOf(r.quality), pri: r.pri };
if (r.kind === "episode" && r.episode_number !== null)
put(episodeKey(r.season_number ?? 1, r.episode_number), f, true);
@@ -300,36 +319,49 @@ export async function buildSubscriptionModels(
for (const sub of subs) {
const entity = entities.find((e) => e.id === sub.media_entity_id) ?? null;
const af = subscriptionAutoFill(cat, sub);
const p = subParams(sub);
const explicitGroups = p.ruleGroupIds ?? [];
const subSlots = entity ? slotsOfSub(slotsByEntity.get(entity.id) ?? [], sub) : [];
const af = subscriptionAutoFill(cat, sub, subSlots[0]);
const entitySlotCount = entity ? (slotsByEntity.get(entity.id) ?? []).length : 0;
const slotRuleIds = [...new Set(subSlots.flatMap((s) => (s.rule_group_ids ?? []) as string[]))];
const ruleGroupIds = explicitGroups.length
? explicitGroups
: slotRuleIds.length
? slotRuleIds
: defaultChainIds(cat, "acquisition");
const visibleChainIds = (slot?: AcquisitionSlotRow) =>
resolveRuleChain(cat, {
kind: "acquisition",
slotRuleGroupIds: slot?.rule_group_ids as string[] | undefined,
subRuleGroupIds: p.ruleGroupIds,
subscriptionId: sub.id,
mediaKind: sub.media_kind,
entityId: sub.media_entity_id ?? undefined,
}).ids.filter((gid) => !(cat.groups.get(gid)?.scope as { subscriptionId?: string } | null)?.subscriptionId);
const ruleGroupIds = [...new Set(subSlots.length ? subSlots.flatMap(visibleChainIds) : visibleChainIds())];
const models: SlotModel[] = subSlots.map((s, i) => {
const spec = slotSpecFromRow(toApi<AcquisitionSlot>(s));
const spec = upgradeSpec(slotSpecWithRules(cat, s, sub, entity!));
const cuts = (spec.completion && spec.completion.kind === "movie" ? spec.completion.cuts : undefined) ?? [];
return {
row: s,
primary: i === 0,
autoFill: i === 0 || af.enabled,
ruleGroupIds: (s.rule_group_ids ?? []) as string[],
ruleGroupIds: visibleChainIds(s),
upgrade: { enabled: !!spec.upgrade?.enabled, targetPri: spec.upgrade?.targetPri ?? 100 },
requiredEditions: cuts.length ? cuts.map(String) : ["default"],
spec,
};
});
const assetSetOf = (s: AcquisitionSlotRow): Set<string> => {
const set = new Set<string>();
const resolutions = ((s.hard_conditions ?? {}) as { resolutions?: string[] }).resolutions;
const allowed = (id: string) => {
const a = assetById.get(id);
return !!a && (!resolutions?.length || resolutions.includes((a.quality as QualityFacts)?.resolution ?? ""));
};
for (const a of assets.rows) {
if (!allowed(a.id)) continue;
if (a.plan_slot_id === s.id || a.id === s.current_asset_id) set.add(a.id);
else if (entitySlotCount <= 1 && a.media_entity_id === s.media_entity_id) set.add(a.id);
}
for (const r of ledgerBySlot.get(s.id) ?? []) if (r.media_asset_id) set.add(r.media_asset_id);
for (const r of ledgerBySlot.get(s.id) ?? [])
if (r.media_asset_id && allowed(r.media_asset_id)) set.add(r.media_asset_id);
return set;
};
const entityAssets = entity ? assets.rows.filter((a) => a.media_entity_id === entity.id).map((a) => a.id) : [];
@@ -542,6 +574,30 @@ export async function buildSubscriptionModels(
today,
upgrade: m.upgrade,
});
if (!isMovie && m.spec.upgrade?.consistency === "season_release") {
const targets = new Map<number, string | null>();
for (const season of seasons) {
if (season.seasonNumber === null) continue;
const n = season.seasonNumber;
const versions = r.cells
.filter((c) => parseUnitKey(c.unitKey)?.season === n && c.assetId)
.map((c) => ({
...((assetById.get(c.assetId!)?.quality ?? {}) as QualityFacts),
...(c.pri !== null ? { pri: c.pri } : {}),
}));
const live = inflightRows.find(
(p) => p.slot_id === m.row.id && p.unit_keys.some((k) => parseUnitKey(k)?.season === n),
);
targets.set(n, seasonReleaseKey(live?.quality ?? selectSeasonRelease(m.spec, versions) ?? {}));
}
for (const cell of r.cells) {
const season = parseUnitKey(cell.unitKey)?.season;
if (season === undefined || !cell.assetId) continue;
const target = targets.get(season);
if (!target || seasonReleaseKey((assetById.get(cell.assetId)?.quality ?? {}) as QualityFacts) !== target)
cell.upgradable = true;
}
}
if (isMovie) {
// 电影:该收录版本不要求的内容版本不算缺
for (const c of r.cells)
apps/server/src/modules/acquisition-organize/services/downloads.ts
+25 -6
View File
@@ -1011,6 +1011,19 @@ async function syncInstance(d: AoDeps, topo: InstanceTopology, stats: SyncStats,
continue;
}
await db.transaction().execute(async (trx) => {
const latest = await trx
.selectFrom("download_task")
.selectAll()
.where("id", "=", row.id)
.forUpdate()
.executeTakeFirst();
if (
!latest?.managed ||
latest.removed_at ||
latest.status === "removed" ||
(latest.last_synced_at && latest.last_synced_at > now)
)
return;
if (still) {
// 管理标签被移除:停止后续自动动作,保留历史与资产
await trx
@@ -1018,21 +1031,25 @@ async function syncInstance(d: AoDeps, topo: InstanceTopology, stats: SyncStats,
.set({ managed: false, tags: still.tags, last_synced_at: now, missing_since: null, missing_count: 0 })
.where("id", "=", row.id)
.execute();
} else if (row.missing_count + 1 < MISSING_CONFIRM_ROUNDS) {
} else if (latest.missing_count + 1 < MISSING_CONFIRM_ROUNDS) {
// 列表与按 hash 都查不到:先记缺席(qB 刚重启、任务还没加载完时会短暂返回空 / 部分列表)
await trx
.updateTable("download_task")
.set({ missing_since: row.missing_since ?? now, missing_count: row.missing_count + 1, last_synced_at: now })
.set({
missing_since: latest.missing_since ?? now,
missing_count: latest.missing_count + 1,
last_synced_at: now,
})
.where("id", "=", row.id)
.execute();
} else {
await trx
.updateTable("download_task")
.set({ status: "removed", removed_at: now, last_synced_at: now, missing_count: row.missing_count + 1 })
.set({ status: "removed", removed_at: now, last_synced_at: now, missing_count: latest.missing_count + 1 })
.where("id", "=", row.id)
.execute();
await emitStatus(d, trx, row.id, row.status, "removed", null);
if (row.acquisition_plan_id) await releaseRemovedPlan(d, trx, row.acquisition_plan_id);
await emitStatus(d, trx, row.id, latest.status, "removed", null);
if (latest.acquisition_plan_id) await releaseRemovedPlan(d, trx, latest.acquisition_plan_id);
stats.removed++;
}
});
@@ -1052,7 +1069,9 @@ async function observe(d: AoDeps, seen: DownloadTaskRow, t: QbTorrent, now: Date
.where("id", "=", seen.id)
.forUpdate()
.executeTakeFirst();
if (!row || row.removed_at || row.status === "removed") return false;
// 多轮同步可交错；行锁之外取得的旧 qB 快照不能覆盖新一轮的标签或状态。
if (!row || row.removed_at || row.status === "removed" || (row.last_synced_at && row.last_synced_at > now))
return false;
return observeLocked(d, trx, row, t, now, stats);
});
}
apps/server/src/modules/acquisition-organize/services/evaluate-autofill.ts
+1 -1
View File
@@ -84,7 +84,7 @@ export async function autoFillCheck(
if (!primary || slots.length <= 1 || primary.id === slot.id) return { action: "normal", planUnits };
if (input.force) return { action: "normal", planUnits };
const af = subscriptionAutoFill(input.catalog, sub);
const af = subscriptionAutoFill(input.catalog, sub, primary);
if (!af.enabled)
return {
action: "defer",
apps/server/src/modules/acquisition-organize/services/organize-exec.ts
+6 -21
View File
@@ -20,16 +20,10 @@ import { errMessage, jsonb } from "../util.ts";
import { type AoDeps, upsertDecision } from "./common.ts";
// 入库:冻结整理计划(organization_record + 文件明细)→ 逐文件可恢复执行(操作账本 fencing)→
// 资产/位置/路径绑定同事务登记 → Slot 推进 → Move 后移除 qB 任务记录(不删数据)。
import { finalizeRecord, removeMovedDownload } from "./organize-finalize.ts";
import { finalizeRecord, postprocessRecord, removeMovedDownload } from "./organize-finalize.ts";
import { emitRecord } from "./organize-plan.ts";
import type { FrozenPlan } from "./organize-types.ts";
import {
cleanupReplacement,
quarantineNotVideo,
settleSamePathReplacement,
stagedReplacementBackup,
} from "./replacement.ts";
import { scrapeAfterImport } from "./scrape.ts";
import { stagedReplacementBackup } from "./replacement.ts";
// ── 执行 ──
@@ -86,17 +80,8 @@ export async function executeRecord(
.where("id", "=", recordId)
.executeTakeFirstOrThrow();
if (rec.status !== "running" && rec.status !== "queued") {
await settleSamePathReplacement(d, rec.plan as unknown as FrozenPlan, recordId);
await cleanupReplacement(d, rec.plan as unknown as FrozenPlan, recordId);
await quarantineNotVideo(d, rec.plan as unknown as FrozenPlan, recordId);
const frozen = rec.plan as unknown as FrozenPlan;
// 与 finalizeRecord 同一口径(按设置 / 本次选项);排队按整理记录幂等,重放不会重复刮
if (["succeeded", "partial"].includes(rec.status)) {
await scrapeAfterImport(d, frozen.entityId, recordId, {
force: frozen.metadataMode === "local",
enabled: frozen.scrapeAfterImport,
});
}
await postprocessRecord(d, frozen, recordId, rec.status);
// Move 成功但移除 qB 任务没成功(qB 暂时连不上 / 收尾后进程退出):重放时补做
if (rec.action === "move" && rec.status === "succeeded" && rec.download_task_id) {
const dt = await db
@@ -329,7 +314,7 @@ export async function executeRecord(
mbps: r.durationMs > 0 ? Math.round(((r.bytes ?? pf.size) / 1e6 / r.durationMs) * 1000) : null,
hashVia: r.hashVia ?? null,
resumed: info.resumed,
// 执行方式:subprocess(workerKind script=deno run / self=编译二进制 --role fsworker)或 inprocess 回退
// 执行方式:subprocess(workerKind script=bun / self=编译二进制 --role fsworker)或 inprocess 回退
fsproxy: fsr.proxy.stats(),
},
"organize.transfer",
@@ -515,7 +500,7 @@ export async function trashReplacedOld(
}
await rmdir(dirname(old.path)).catch(() => undefined);
}
const days = Number(Deno.env.get("FILES_TRASH_RETENTION_DAYS") ?? 7) || 7;
const days = Number(process.env.FILES_TRASH_RETENTION_DAYS ?? 7) || 7;
await d.ctx.db
.insertInto("trash_item")
.values({
@@ -569,7 +554,7 @@ export function fsRuntime(d: AoDeps): FsRuntime {
if (hit) return hit;
const rt: FsRuntime = {
proxy: new FsProxy({
mode: c.FS_PROXY_MODE ?? "auto",
mode: c.FS_PROXY_MODE ?? "subprocess",
hangTimeoutMs: c.FS_HANG_TIMEOUT_MS ?? 120_000,
quickTimeoutMs: c.FS_QUICK_TIMEOUT_MS ?? 30_000,
progressIntervalMs: c.FS_PROGRESS_INTERVAL_MS ?? 1000,
apps/server/src/modules/acquisition-organize/services/organize-finalize.ts
+29 -12
View File
@@ -435,7 +435,12 @@ export async function finalizeRecord(d: AoDeps, recordId: string): Promise<Execu
{ type: "verify_passed", complete: false },
{ currentAssetId: verifiedAssets[0]!, reason: "verified" },
);
if (old && old !== verifiedAssets[0] && plan.purpose === "upgrade") {
if (
old &&
old !== verifiedAssets[0] &&
plan.purpose === "upgrade" &&
!frozen.replacement?.assets.some((a) => a.assetId === old)
) {
// 升级:旧版本保留,处置只记待处理,不删
await upsertDecision(d, trx, {
kind: "other",
@@ -525,17 +530,7 @@ export async function finalizeRecord(d: AoDeps, recordId: string): Promise<Execu
};
});
await settleSamePathReplacement(d, frozenPre, recordId);
await cleanupReplacement(d, frozenPre, recordId);
await quarantineNotVideo(d, frozenPre, recordId);
// SCR-1:有资产入库时按设置自动刮削(写 NFO 与图片;排任务,不在这里等)
if (result.status === "succeeded" || result.status === "partial") {
await scrapeAfterImport(d, frozenPre.entityId ?? null, recordId, {
force: frozenPre.metadataMode === "local",
enabled: frozenPre.scrapeAfterImport,
});
}
await postprocessRecord(d, frozenPre, recordId, result.status);
// Move 完成:移除 qB 任务记录,结束下载侧管理(不删数据)
if (result.move && result.dt) await removeMovedDownload(d, result.dt);
return {
@@ -546,6 +541,28 @@ export async function finalizeRecord(d: AoDeps, recordId: string): Promise<Execu
};
}
/**
* 记录终态提交后的文件处置与刮削入队,首次完成和终态重放共用。
* 失败记录也要恢复旧文件、隔离伪视频;部分成功可以刮削。
* Move 后移除 qB 由调用方按完整成功判断,不与文件处置混在一起。
*/
export async function postprocessRecord(
d: AoDeps,
frozen: FrozenPlan,
recordId: string,
status: ExecuteResult["status"],
): Promise<void> {
await settleSamePathReplacement(d, frozen, recordId);
await cleanupReplacement(d, frozen, recordId);
await quarantineNotVideo(d, frozen, recordId);
if (status === "succeeded" || status === "partial") {
await scrapeAfterImport(d, frozen.entityId ?? null, recordId, {
force: frozen.metadataMode === "local",
enabled: frozen.scrapeAfterImport,
});
}
}
/**
* Move 成功后移除 qB 任务(不删数据)并结束纳管。失败只告警:executeRecord 重放已结束记录、
* 整理巡检(recoverStalledRecords)会对"move 成功但下载任务未移除"的记录再试。
apps/server/src/modules/acquisition-organize/services/organize-hooks.ts
-89
View File
@@ -1,89 +0,0 @@
import { basename, dirname, extname, join } from "node:path";
import type { OrganizePlan } from "@umt/domain";
import { hasHook, interceptHook, rewriteHook } from "../../../hooks/index.ts";
import type { ServerContext } from "../../types.ts";
// 整理计划上的链式钩子(ADR-006):organize.rename 改写主视频目标文件名、organize.intercept 拒绝整理。
// 都在计划生成之后、写整理记录之前调用;没有注册钩子时什么都不做。
const BAD_NAME = /[\\/\0]/;
const usableName = (n: unknown): boolean =>
typeof n === "string" && n.trim() !== "" && !BAD_NAME.test(n) && n !== "." && n !== "..";
/**
* organize.rename:逐个主视频把模板生成的目标文件名交给钩子改写,同主干的附属文件(字幕、NFO 等)跟着换主干。
* 只改文件名不改目录;返回带路径分隔符或空名的钩子结果跳过。改名后撞名的项转 hold + duplicate_target(与保留原名同一处理)。
* 改名后的目标是否已存在由执行阶段按冲突处理(计划期的存在性探测在改名之前)。
*/
export async function applyRenameHook(
ctx: Pick<ServerContext, "hooks">,
plan: OrganizePlan,
base: { mediaEntityId: string; mediaType: "movie" | "tv" },
): Promise<number> {
if (!hasHook(ctx, "organize.rename")) return 0;
const renamed = new Map<string, string>();
for (const main of plan.files.filter((f) => f.role === "main")) {
const name = basename(main.dst);
// 返回带路径分隔符或空名的钩子结果跳过(记日志),其余钩子照常
const next = await rewriteHook(
ctx,
"organize.rename",
name,
{
targetPath: main.dst,
sourcePath: main.src,
mediaEntityId: base.mediaEntityId,
mediaType: base.mediaType,
season: main.season ?? null,
episodes: main.episodes ?? null,
},
usableName,
);
if (next === name) continue;
const oldStem = name.slice(0, name.length - extname(name).length);
const newStem = next.slice(0, next.length - extname(next).length);
const to = join(dirname(main.dst), next);
renamed.set(main.dst, to);
main.dst = to;
for (const f of plan.files) {
if (f.mainSrc !== main.src) continue;
const side = basename(f.dst);
if (!side.startsWith(oldStem)) continue;
const nd = join(dirname(f.dst), newStem + side.slice(oldStem.length));
renamed.set(f.dst, nd);
f.dst = nd;
}
}
if (renamed.size === 0) return 0;
for (const c of plan.conflicts) if (c.dst && renamed.has(c.dst)) c.dst = renamed.get(c.dst)!;
for (const pc of plan.preconditions)
if (pc.kind === "target_absent" || pc.kind === "verify_after_copy")
pc.paths = pc.paths.map((p) => renamed.get(p) ?? p);
const byDst = new Map<string, OrganizePlan["files"]>();
for (const f of plan.files) byDst.set(f.dst, [...(byDst.get(f.dst) ?? []), f]);
for (const [dst, fs] of byDst) {
if (fs.length < 2 || plan.conflicts.some((c) => c.kind === "duplicate_target" && c.dst === dst)) continue;
for (const f of fs) f.action = "hold";
plan.conflicts.push({ kind: "duplicate_target", blocking: true, dst, detail: fs.map((f) => f.src).join(" | ") });
}
plan.requiresApproval = plan.conflicts.some((c) => c.blocking);
return renamed.size;
}
/** organize.intercept:返回拒绝原因(含拒绝的钩子名),放行返回 null */
export async function organizeIntercept(
ctx: Pick<ServerContext, "hooks">,
plan: OrganizePlan,
base: {
mediaEntityId: string;
mediaType: "movie" | "tv";
action: string;
trigger: string;
downloadTaskId: string | null;
},
) {
return interceptHook(ctx, "organize.intercept", () => ({
...base,
files: plan.files.map((f) => ({ src: f.src, dst: f.dst, kind: f.kind, role: f.role })),
}));
}
apps/server/src/modules/acquisition-organize/services/organize-plan.ts
+56 -22
View File
@@ -22,6 +22,7 @@ import {
replacementScope,
resolveTargetDirectory,
type SourceFile,
selectAcquisitionPolicy,
sidecarStem,
titleMismatches,
} from "@umt/domain";
@@ -43,9 +44,10 @@ import {
} from "../../sources-identity/service/episode-resolution.ts";
import type { Layout } from "../../sources-identity/service/episode-structure.ts";
import { wordsOption } from "../../sources-identity/service/identify-words-cache.ts";
import { parsedOf, unitKeysOf } from "../../sources-identity/service/release-facts.ts";
import { errMessage, hashJson, jsonb, toApi } from "../util.ts";
import { publishedAcquisitionPolicies, unitKeysOf } from "./acquisition.ts";
import { classifyEntity } from "./classification.ts";
import { collectionContext, publishedAcquisitionPolicies } from "./collection-strategy.ts";
import { type AoDeps, type Exec, hostPathOf, loadInstances, upsertDecision } from "./common.ts";
import { applySlotEvent } from "./downloads.ts";
import { emitOrganizationFinished } from "./lifecycle-events.ts";
@@ -53,7 +55,6 @@ import { effectiveNaming, ensureNamingVersion, loadNamingEntity, namingByVersion
import { resolveOrganizeSource } from "./organize-config.ts";
import { PathGuardError } from "./organize-guard.ts";
import { successfulOrganizeCounts } from "./organize-history.ts";
import { applyRenameHook, organizeIntercept } from "./organize-hooks.ts";
import { planTransfer } from "./organize-transfer.ts";
// 入库:冻结整理计划(organization_record + 文件明细)→ 逐文件可恢复执行(操作账本 fencing)→
// 资产/位置/路径绑定同事务登记 → Slot 推进 → Move 后移除 qB 任务记录(不删数据)。
@@ -77,9 +78,11 @@ import {
type TakeoverIdentify,
takeoverSummary,
} from "./takeover.ts";
import { automaticReplacementAssets } from "./upgrade-replacement.ts";
export const qualityOf = (p: ParsedRelease): QualityFacts => ({
...(p.resolution ? { resolution: p.resolution } : {}),
...(p.videoCodec ? { videoCodec: p.videoCodec } : {}),
...(p.source ? { source: p.source } : {}),
hdr: p.hdr,
...(p.releaseGroup ? { releaseGroup: p.releaseGroup } : {}),
@@ -365,6 +368,9 @@ export async function planImport(d: AoDeps, req: ImportRequest): Promise<PlanRes
mediaEntityId?: string | null;
importMode?: "normal" | "replace";
replacementAssetIds?: string[];
automaticReplacement?: boolean;
quality?: QualityFacts;
unitKeys?: string[];
}
| undefined)
: undefined;
@@ -467,17 +473,25 @@ export async function planImport(d: AoDeps, req: ImportRequest): Promise<PlanRes
const mediaType: "movie" | "tv" = entity.kind === "movie" ? "movie" : "tv";
const inPlace = req.action === "in_place";
const importMode = req.importMode ?? evaluation?.importMode ?? "normal";
let replacementIds = req.replacementAssetIds ?? evaluation?.replacementAssetIds ?? [];
const automaticReplacement = !evaluation?.manual && evaluation?.automaticReplacement && !req.importMode && slotId;
if (automaticReplacement && evaluation.quality) {
const slot = await db
.selectFrom("acquisition_slot")
.selectAll()
.where("id", "=", slotId!)
.executeTakeFirstOrThrow();
replacementIds = await automaticReplacementAssets(db, slot, evaluation.unitKeys ?? [], evaluation.quality);
}
const importMode =
req.importMode ?? evaluation?.importMode ?? (automaticReplacement && replacementIds.length ? "replace" : "normal");
if (inPlace && importMode === "replace") return fail("in_place_cannot_replace", {}, entity.id);
let replacement: FrozenReplacement | undefined;
if (importMode === "replace") {
try {
replacement = await freezeReplacement(
d,
entity.id,
mediaType,
req.replacementAssetIds ?? evaluation?.replacementAssetIds ?? [],
);
replacement = await freezeReplacement(d, entity.id, mediaType, replacementIds);
if (automaticReplacement && evaluation?.quality)
replacement.automatic = { slotId: slotId!, quality: evaluation.quality };
} catch (error) {
return fail("replacement_invalid", { message: errMessage(error) }, entity.id);
}
@@ -531,7 +545,29 @@ export async function planImport(d: AoDeps, req: ImportRequest): Promise<PlanRes
reason: "classification_unavailable",
};
}
const policy = (await publishedAcquisitionPolicies(db))[0];
const slotPolicyId = slotId
? (await db.selectFrom("acquisition_slot").select("policy_revision_id").where("id", "=", slotId).executeTakeFirst())
?.policy_revision_id
: null;
const selectedPolicy = slotPolicyId
? await db.selectFrom("policy_revision").selectAll().where("id", "=", slotPolicyId).executeTakeFirst()
: undefined;
const context = classification
? {
mediaKind:
entity.kind === "movie"
? ("movie" as const)
: entity.kind === "variety"
? ("variety" as const)
: ("series" as const),
category: classification.category,
}
: await collectionContext(d, entity.id);
const policy = selectedPolicy
? { rules: selectedPolicy.rules as import("@umt/contracts").AcquisitionPolicyRules }
: context
? selectAcquisitionPolicy(await publishedAcquisitionPolicies(db), context)
: undefined;
let action: "copy" | "move" | "in_place" =
req.action ?? sourceConfig?.action ?? policy?.rules.organizeMode ?? orgSettings.defaultMode;
const site = rel
@@ -671,6 +707,7 @@ export async function planImport(d: AoDeps, req: ImportRequest): Promise<PlanRes
files.filter((f) => !overrideFor(req.fileOverrides, f.path) && fileKindOf(basename(f.path)) === "video"),
);
const words = await wordsOption(d.ctx.db);
const releaseQuality = rel ? qualityOf(parsedOf(rel)) : undefined;
for (const f of files) {
let parsed = parseReleasePath(segmentsWithin(f.path, sourceRoot), words);
if (parsed.fileKind !== "video") continue;
@@ -722,7 +759,15 @@ export async function planImport(d: AoDeps, req: ImportRequest): Promise<PlanRes
unit === undefined ? parsed : unit ? { ...parsed, season: unit.season, episodes: unit.episodes } : null;
const unitSeason = unit === undefined ? season : unit?.season;
facts[f.path] = {
quality: qualityOf(parsed),
quality: {
...qualityOf(parsed),
...(!parsed.resolution && releaseQuality?.resolution ? { resolution: releaseQuality.resolution } : {}),
...(!parsed.source && releaseQuality?.source
? { source: releaseQuality.source, isEncode: releaseQuality.isEncode }
: {}),
...(!parsed.videoCodec && releaseQuality?.videoCodec ? { videoCodec: releaseQuality.videoCodec } : {}),
...(!parsed.releaseGroup && releaseQuality?.releaseGroup ? { releaseGroup: releaseQuality.releaseGroup } : {}),
},
unitKeys: unitParsed
? unitKeysOf({
...unitParsed,
@@ -942,19 +987,8 @@ export async function planImport(d: AoDeps, req: ImportRequest): Promise<PlanRes
plan.requiresApproval = true;
}
}
// 链式钩子 organize.rename(ADR-006):模板命名时可改写主视频目标文件名
if (!inPlace && !keepNames) await applyRenameHook(d.ctx, plan, { mediaEntityId: entity.id, mediaType });
if (keepNames && !inPlace) keepOriginalNames(plan);
if (req.keepBoth && !inPlace) await keepBothTargets(plan);
// 链式钩子 organize.intercept:返回原因即拒绝本次整理(记为失败)
const veto = await organizeIntercept(d.ctx, plan, {
mediaEntityId: entity.id,
mediaType,
action,
trigger: req.trigger,
downloadTaskId: dt?.id ?? null,
});
if (veto) return fail(`intercepted:${veto.reason}`, { hook: veto.by, reason: veto.reason }, entity.id);
const frozen: FrozenPlan = {
version: 1,
...(req.scrapeAfterImport !== undefined ? { scrapeAfterImport: req.scrapeAfterImport } : {}),
apps/server/src/modules/acquisition-organize/services/policies.ts
+170
View File
@@ -0,0 +1,170 @@
import { AcquisitionPolicyRules, AcquisitionPolicyScope } from "@umt/contracts";
import type { DB, Transaction } from "@umt/db";
import { evaluateCandidate } from "@umt/domain";
import { slotSpecFromRow } from "@umt/mapping";
import { ApiException } from "../../../http/errors.ts";
import { contextOf, parsedOf } from "../../sources-identity/service/release-facts.ts";
import { jsonb } from "../util.ts";
import type { AoDeps } from "./common.ts";
import {
assertRuleGroupIds,
entityOriginalLanguage,
loadRuleCatalog,
mergeHard,
resolveRuleChain,
upgradeRuleGroupIds,
} from "./rules.ts";
/** 发布:旧 published → superseded/rolled_back;吊销旧授权;新版本生成持续授权 automation_grant */
export async function publishRevision(
trx: Transaction<DB>,
id: string,
policyKey: string,
by: string,
oldStatus: "superseded" | "rolled_back",
) {
const current = await trx
.selectFrom("policy_revision")
.select(["kind", "scope", "rules"])
.where("id", "=", id)
.executeTakeFirstOrThrow();
if (current.kind === "acquisition") {
const scope = AcquisitionPolicyScope.safeParse(current.scope);
const rules = AcquisitionPolicyRules.safeParse(current.rules);
if (!scope.success || !rules.success) throw new ApiException("validation_failed", "收录策略范围或规则结构不合法");
await assertRuleGroupIds(
trx,
[...(rules.data.defaultSlots ?? []), ...(rules.data.newTitle?.slots ?? [])].flatMap((t) => [
...(t.ruleGroupIds ?? []),
...upgradeRuleGroupIds(t.upgradeThreshold),
]),
);
}
const old = await trx
.selectFrom("policy_revision")
.select(["id"])
.where("policy_key", "=", policyKey)
.where("status", "=", "published")
.execute();
const now = new Date();
for (const o of old as { id: string }[]) {
await trx.updateTable("policy_revision").set({ status: oldStatus }).where("id", "=", o.id).execute();
await trx
.updateTable("automation_grant")
.set({ status: "revoked", revoked_at: now, revoked_by: by })
.where("policy_revision_id", "=", o.id)
.where("status", "=", "active")
.execute();
}
const row = await trx
.updateTable("policy_revision")
.set({ status: "published", published_at: now, published_by: by })
.where("id", "=", id)
.returningAll()
.executeTakeFirstOrThrow();
const rules = (row.rules ?? {}) as { budget?: unknown };
await trx
.insertInto("automation_grant")
.values({
action: `policy.${row.kind}`,
policy_revision_id: row.id,
scope: jsonb(row.scope),
budget: jsonb(rules.budget ?? {}),
granted_by: by,
reason: `发布策略 ${policyKey} r${row.revision}`,
})
.execute();
return row;
}
/** 预览:用规则里的 Slot 模板评估最近已识别的发布(确定性,不产生副作用) */
export async function previewPolicy(d: AoDeps, rules: unknown) {
const r = AcquisitionPolicyRules.safeParse(rules);
const templates = r.success ? [...(r.data.defaultSlots ?? []), ...(r.data.newTitle?.slots ?? [])] : [];
const rels = await d.ctx.db
.selectFrom("source_release")
.selectAll()
.where("resolution_status", "=", "resolved")
.orderBy("first_seen_at", "desc")
.limit(200)
.execute();
let matched = 0;
let excluded = 0;
const samples: {
sourceReleaseId: string;
title: string;
sizeBytes: number | null;
template: string;
decision: string;
reasons: string[];
}[] = [];
const cat = await loadRuleCatalog(d.ctx.db);
const entityIds = [...new Set(rels.map((r) => r.media_entity_id).filter((id): id is string => !!id))];
const entities = new Map(
(entityIds.length
? await d.ctx.db.selectFrom("media_entity").select(["id", "kind"]).where("id", "in", entityIds).execute()
: []
).map((e) => [e.id, e]),
);
const languages = new Map<string, string | undefined>();
for (const rel of rels) {
const parsed = parsedOf(rel);
const entity = rel.media_entity_id ? entities.get(rel.media_entity_id) : undefined;
const ctx = contextOf(rel);
if (entity) {
if (!languages.has(entity.id)) languages.set(entity.id, await entityOriginalLanguage(d.ctx.db, entity.id));
ctx.originalLanguage = languages.get(entity.id);
}
for (const t of templates) {
const spec = slotSpecFromRow({
id: "preview",
name: t.name,
hardConditions: t.hardConditions ?? {},
softPreferences: t.softPreferences ?? {},
upgradeThreshold: t.upgradeThreshold ?? {},
completion: t.completion ?? {},
fallback: t.fallback ?? {},
});
const chain = resolveRuleChain(cat, {
kind: "acquisition",
slotRuleGroupIds: t.ruleGroupIds,
upgrade: false,
mediaKind: entity?.kind === "movie" ? "movie" : entity?.kind === "variety" ? "variety" : "series",
entityId: entity?.id,
siteId: rel.site_id,
});
spec.hard = mergeHard(spec.hard, chain.hard);
if (chain.groups.length) spec.rules = { groups: chain.groups, atoms: chain.atoms };
const ev = evaluateCandidate(spec, parsed, null, ctx);
if (ev.decision === "accept") matched++;
else excluded++;
if (samples.length < 20) {
const reasons =
ev.decision === "reject"
? ev.violations.map((v) => `${v.field}:${v.code}`)
: ev.decision === "unknown_field"
? ev.missing.map((m) => `missing:${m}`)
: "reasons" in ev
? (ev.reasons ?? [])
: [];
samples.push({
sourceReleaseId: rel.id,
title: rel.title,
sizeBytes: rel.size_bytes,
template: t.name,
decision: ev.decision,
reasons,
});
}
}
}
return {
evaluatedAt: new Date().toISOString(),
sampleSize: rels.length,
templates: templates.length,
matched,
excluded,
samples,
valid: r.success,
};
}
apps/server/src/modules/acquisition-organize/services/replacement.ts
+34 -2
View File
@@ -2,7 +2,13 @@ import { randomUUID } from "node:crypto";
import { basename, dirname, join, relative } from "node:path";
import { DEFAULT_ROOT_SENTINEL, type FileFacts } from "@umt/adapters";
import { sql } from "@umt/db";
import { type ReplacementScope, replacementCovers, replacementScope } from "@umt/domain";
import {
type QualityFacts,
type ReplacementScope,
replacementCovers,
replacementScope,
seasonReleaseKey,
} from "@umt/domain";
import { mkdir, readdir, realpath, rm } from "../../../storage/local-fs.ts";
import {
isRemotePath,
@@ -66,6 +72,7 @@ export interface ReplacementAsset {
export interface FrozenReplacement {
mode: "replace";
assets: ReplacementAsset[];
automatic?: { slotId: string; quality: QualityFacts };
}
export async function freezeReplacement(
@@ -235,6 +242,31 @@ export async function cleanupReplacement(d: AoDeps, frozen: FrozenPlan, recordId
),
];
});
// 分次洗版：本次核验成功后，可以用同保留版本、同套且已核验的文件补齐旧合集覆盖。
const automatic = frozen.replacement.automatic;
const targetKey = automatic ? seasonReleaseKey(automatic.quality) : null;
if (automatic && targetKey && successful.length && frozen.mediaType === "tv") {
const existing = await sql<{ quality: QualityFacts; path: string; season: number; episode: number }>`
select coalesce(r.parsed, '{}'::jsonb) || coalesce(a.media_info->'_umt'->'quality', '{}'::jsonb) as quality, l.relative_path as path, u.season_number as season, u.episode_number as episode
from media_asset a
left join source_release r on r.id = a.source_release_id
join asset_location l on l.media_asset_id = a.id and l.location_kind = 'library' and l.status = 'available'
join asset_unit_coverage c on c.media_asset_id = a.id and c.coverage = 'full'
join content_unit u on u.id = c.content_unit_id
where a.media_entity_id = ${frozen.entityId!} and a.status = 'active'
and a.id not in (${sql.join(frozen.replacement.assets.map((a) => a.assetId))})
and u.season_number is not null and u.episode_number is not null
and (exists (select 1 from slot_unit su where su.slot_id = ${automatic.slotId} and su.media_asset_id = a.id)
or exists (select 1 from download_task dt join acquisition_plan p on p.id = dt.acquisition_plan_id
where dt.id = a.download_task_id and p.slot_id = ${automatic.slotId} and p.status = 'completed'))
and exists (select 1 from organization_file_result f join organization_record rec on rec.id = f.organization_record_id
where f.media_asset_id = a.id and f.role = 'main' and a.verified_at is not null and (f.status = 'succeeded' or (f.status = 'skipped' and f.skip_reason = 'already_in_library')) and rec.status in ('succeeded', 'partial'))`.execute(
db,
);
for (const file of existing.rows)
if (seasonReleaseKey(file.quality ?? {}) === targetKey)
scopes.push(replacementScope(file.path, "tv", { season: file.season, episodes: [file.episode] }));
}
for (const old of frozen.replacement.assets) {
if (successful.some((f) => f.media_asset_id === old.assetId)) continue;
if (!replacementCovers(old.scope, scopes)) {
@@ -595,7 +627,7 @@ export async function quarantineNotVideo(d: AoDeps, frozen: FrozenPlan, recordId
const trash = join(rootReal, ".umt-trash", now.toISOString().slice(0, 10).replace(/-/g, ""), id, rel);
const m = await fsr.move(abs, trash);
if (m.status !== "moved") throw new Error(`移入回收站失败：${m.status}`);
const days = Number(Deno.env.get("FILES_TRASH_RETENTION_DAYS") ?? 7) || 7;
const days = Number(process.env.FILES_TRASH_RETENTION_DAYS ?? 7) || 7;
await db
.insertInto("trash_item")
.values({
apps/server/src/modules/acquisition-organize/services/rules.ts
+22
View File
@@ -384,3 +384,25 @@ export async function previewRuleGroup(
items,
};
}
/** 所有引用入口共享存在性校验，洗版引用与补齐引用使用相同口径。 */
export async function assertRuleGroupIds(exec: Exec, ids: readonly string[]) {
if (!ids.length) return;
const rows = await exec
.selectFrom("rule_group")
.select("id")
.where("id", "in", [...ids])
.where("deleted_at", "is", null)
.execute();
const have = new Set(rows.map((r) => r.id));
const missing = ids.filter((id) => !have.has(id));
if (missing.length) throw new ApiException("bad_request", `规则组不存在：${missing.join(", ")}`);
}
export function upgradeRuleGroupIds(upgrade: unknown): string[] {
const ids = (upgrade as { ruleGroupIds?: unknown } | null)?.ruleGroupIds;
if (ids === undefined) return [];
if (!Array.isArray(ids) || ids.some((id) => typeof id !== "string" || !id))
throw new ApiException("validation_failed", "洗版规则组引用必须是 ID 列表");
return ids;
}
apps/server/src/modules/acquisition-organize/services/season-release.ts
+74
View File
@@ -0,0 +1,74 @@
import type { AcquisitionSlotRow, SourceReleaseRow } from "@umt/db";
import { sql } from "@umt/db";
import {
currentPri,
effectiveLadder,
parseUnitKey,
type QualityFacts,
ruleChainOf,
type SlotSpec,
selectSeasonRelease,
} from "@umt/domain";
import { contextOf, parsedOf } from "../../sources-identity/service/release-facts.ts";
import type { Exec } from "./common.ts";
/** 有成套洗版在途时保持其冻结规格；结束后才允许选择更高档的新套。 */
export async function seasonReleaseTarget(
exec: Exec,
slot: AcquisitionSlotRow,
season: number,
spec: SlotSpec,
candidate?: QualityFacts,
originalLanguage?: string,
): Promise<QualityFacts | undefined> {
const allowed = (q: QualityFacts) =>
!spec.hard.resolutions?.length || (!!q.resolution && spec.hard.resolutions.includes(q.resolution));
const inflight = await exec
.selectFrom("acquisition_plan as p")
.innerJoin("source_release as r", "r.id", "p.source_release_id")
.selectAll("r")
.select(["p.unit_keys as planned_units", "p.evaluation"])
.where("p.slot_id", "=", slot.id)
.where("p.status", "in", ["reserved", "submitted"])
.orderBy("p.created_at")
.orderBy("p.id")
.execute();
const inSeason = (keys: string[]) => keys.some((k) => parseUnitKey(k)?.season === season);
for (const plan of inflight) {
const evaluation = plan.evaluation as { seasonRelease?: string; quality?: QualityFacts };
if (inSeason(plan.planned_units) && evaluation.seasonRelease && evaluation.quality && allowed(evaluation.quality))
return evaluation.quality;
}
const assets = await sql<{ quality: QualityFacts; source_release_id: string | null }>`
select distinct coalesce(a.media_info->'_umt'->'quality', '{}'::jsonb) || coalesce(su.quality, '{}'::jsonb) as quality, a.source_release_id
from media_asset a
join asset_unit_coverage c on c.media_asset_id = a.id and c.coverage = 'full'
join content_unit u on u.id = c.content_unit_id
left join slot_unit su on su.media_asset_id = a.id and su.content_unit_id = u.id and su.slot_id = ${slot.id}
left join download_task dt on dt.id = a.download_task_id
left join acquisition_plan p on p.id = dt.acquisition_plan_id
where a.status = 'active' and u.season_number = ${season}
and (su.slot_id = ${slot.id} or p.slot_id = ${slot.id} or a.id = ${slot.current_asset_id}
or (a.media_entity_id = ${slot.media_entity_id} and (select count(*) from acquisition_slot where media_entity_id = ${slot.media_entity_id}) = 1))`.execute(
exec,
);
const relIds = [...new Set(assets.rows.map((a) => a.source_release_id).filter((id): id is string => !!id))];
const rels = new Map(
(relIds.length ? await exec.selectFrom("source_release").selectAll().where("id", "in", relIds).execute() : []).map(
(r) => [r.id, r],
),
);
const facts = (r: SourceReleaseRow, delivered?: QualityFacts): QualityFacts => {
const parsed = { ...parsedOf(r), ...delivered };
const ctx = { ...contextOf(r), ...(originalLanguage ? { originalLanguage } : {}) };
return { ...parsed, pri: effectiveLadder(spec, parsed, ctx, ruleChainOf(spec, parsed, ctx)).pri };
};
const versions: QualityFacts[] = candidate ? [candidate] : [];
for (const a of assets.rows) {
const rel = a.source_release_id ? rels.get(a.source_release_id) : undefined;
const q = rel ? facts(rel, a.quality) : { ...a.quality, pri: undefined };
versions.push({ ...q, pri: currentPri(spec, q) });
}
for (const r of inflight) if (inSeason(r.planned_units)) versions.push(facts(r));
return selectSeasonRelease(spec, versions.filter(allowed));
}
apps/server/src/modules/acquisition-organize/services/subscriptions-params.ts
+12 -2
View File
@@ -226,7 +226,7 @@ export async function applySlotPreset(
if (sameJson(a, b)) return 0;
const slots = await exec
.selectFrom("acquisition_slot")
.select(["id", "hard_conditions", "soft_preferences", "revision"])
.select(["id", "hard_conditions", "soft_preferences", "revision", "policy_revision_id"])
.where("subscription_id", "=", input.subscriptionId)
.execute();
let n = 0;
@@ -238,7 +238,17 @@ export async function applySlotPreset(
for (const s of slots) {
const curHard = (s.hard_conditions ?? {}) as Record<string, unknown>;
const curSoft = (s.soft_preferences ?? {}) as Record<string, unknown>;
const hard = merge(curHard, a.hard, b.hard);
// 策略模板的分辨率范围是保留版本边界；统一预设只作用于隐式默认版本。
const fromHard = { ...a.hard };
const toHard = { ...b.hard };
if (
curHard.resolutions &&
(s.policy_revision_id || slots.length > 1 || !sameJson(curHard.resolutions, a.hard.resolutions))
) {
delete fromHard.resolutions;
delete toHard.resolutions;
}
const hard = merge(curHard, fromHard, toHard);
const soft = merge(curSoft, a.soft, b.soft);
if (sameJson(hard, curHard) && sameJson(soft, curSoft)) continue;
await exec
apps/server/src/modules/acquisition-organize/services/subscriptions.ts
+98 -30
View File
@@ -22,15 +22,17 @@ import {
type SeasonProgress,
type SlotSpec,
seasonProgress,
seasonReleaseKey,
seasonWildcard,
selectSeasonRelease,
type UnitRange,
upgradeSpec,
} from "@umt/domain";
import { slotSpecFromRow } from "@umt/mapping";
import { interceptHook } from "../../../hooks/index.ts";
import { contextOf, parsedOf, unitKeysOf } from "../../sources-identity/service/release-facts.ts";
import { errMessage, jsonb, toApi } from "../util.ts";
import { type AoDeps, type Exec, upsertDecision } from "./common.ts";
import { emitSubscriptionEvent } from "./lifecycle-events.ts";
import { contextOf, parsedOf, unitKeysOf } from "./release-facts.ts";
import { entityOriginalLanguage, loadRuleCatalog, mergeHard, type RuleCatalog, resolveRuleChain } from "./rules.ts";
// W5-C 订阅业务:季总集数账本、缺集/在途、完成判定与历史、Slot 重评调度、分集质量账本、主动搜索调度。
@@ -212,9 +214,14 @@ export async function applyEpisodeTotal(
/** Slot 已覆盖的内容单元:该 Slot 计划产出资产 + 当前资产 + 分集账本(content_unit id 与合成键两种口径) */
export async function slotCoveredUnits(
exec: Exec,
slot: Pick<AcquisitionSlotRow, "id" | "current_asset_id" | "media_entity_id">,
slot: Pick<AcquisitionSlotRow, "id" | "current_asset_id" | "media_entity_id"> &
Partial<Pick<AcquisitionSlotRow, "hard_conditions">>,
opts: { entityWide?: boolean } = {},
): Promise<{ keys: Set<string>; episodes: Map<number, Set<number>> }> {
const allowed = ((slot.hard_conditions ?? {}) as { resolutions?: string[] }).resolutions ?? [];
const resolutionFilter = allowed.length
? sql`and coalesce(a.media_info->'_umt'->'quality'->>'resolution', r.parsed->>'resolution') in (${sql.join(allowed)})`
: sql``;
const r = await sql<{
unit_id: string;
season_number: number | null;
@@ -224,9 +231,10 @@ export async function slotCoveredUnits(
}>`
with assets as (
select a.id from media_asset a
left join source_release r on r.id = a.source_release_id
left join download_task dt on dt.id = a.download_task_id
left join acquisition_plan p on p.id = dt.acquisition_plan_id
where a.status = 'active' and a.parent_asset_id is null
where a.status = 'active' and a.parent_asset_id is null ${resolutionFilter}
and (p.slot_id = ${slot.id} or a.id = ${slot.current_asset_id}
or a.id in (select media_asset_id from slot_unit where slot_id = ${slot.id})
${opts.entityWide ? sql`or a.media_entity_id = ${slot.media_entity_id}` : sql``})
@@ -237,6 +245,7 @@ export async function slotCoveredUnits(
join content_unit u on u.id = c.content_unit_id
union
select u.id, u.season_number, u.episode_number, u.kind, to_char(u.air_date, 'YYYYMMDD') from slot_unit su
join assets on assets.id = su.media_asset_id
join content_unit u on u.id = su.content_unit_id
where su.slot_id = ${slot.id} and su.media_asset_id is not null`.execute(exec);
const keys = new Set<string>();
@@ -257,9 +266,10 @@ export async function slotCoveredUnits(
const legacy = await sql<{ keys: string[] | null }>`
select array(select jsonb_array_elements_text(coalesce(a.media_info->'_umt'->'unitKeys', '[]'::jsonb))) as keys
from media_asset a
left join source_release r on r.id = a.source_release_id
left join download_task dt on dt.id = a.download_task_id
left join acquisition_plan p on p.id = dt.acquisition_plan_id
where a.status = 'active' and a.parent_asset_id is null
where a.status = 'active' and a.parent_asset_id is null ${resolutionFilter}
and (p.slot_id = ${slot.id} or a.id = ${slot.current_asset_id})`.execute(exec);
for (const row of legacy.rows)
for (const k of row.keys ?? []) {
@@ -376,11 +386,13 @@ export function slotSpecWithRules(
siteId?: string,
): SlotSpec {
const spec = slotSpecFromRow(toApi<AcquisitionSlot>(slot));
if (entity.kind === "movie" && spec.upgrade?.consistency) spec.upgrade = { ...spec.upgrade, consistency: undefined };
const baseHard = spec.hard;
const chain = resolveRuleChain(cat, {
kind: "acquisition",
slotRuleGroupIds: (slot.rule_group_ids ?? []) as string[],
subRuleGroupIds: subParams(sub).ruleGroupIds,
upgrade: !!spec.upgrade?.enabled,
upgrade: false,
mediaKind: ruleMediaKind(entity.kind),
entityId: entity.id,
siteId,
@@ -388,6 +400,22 @@ export function slotSpecWithRules(
});
spec.hard = mergeHard(spec.hard, chain.hard);
if (chain.groups.length) spec.rules = { groups: chain.groups, atoms: chain.atoms };
if (spec.upgrade?.enabled) {
const washing = resolveRuleChain(cat, {
kind: "acquisition",
slotRuleGroupIds: spec.upgrade.ruleGroupIds?.length
? spec.upgrade.ruleGroupIds
: ((slot.rule_group_ids ?? []) as string[]),
subRuleGroupIds: subParams(sub).ruleGroupIds,
upgrade: !spec.upgrade.ruleGroupIds?.length,
mediaKind: ruleMediaKind(entity.kind),
entityId: entity.id,
siteId,
subscriptionId: sub?.id,
});
spec.upgradeHard = mergeHard(baseHard, washing.hard);
spec.upgradeRules = { groups: washing.groups, atoms: washing.atoms };
}
return spec;
}
@@ -396,8 +424,10 @@ export function slotSpecWithRules(
* (同 evaluateCandidate 的 at_ceiling);按来源等级比较的升级没有「最高档」,返回 null(覆盖齐即完成)。
*/
export function upgradeTargetPri(spec: SlotSpec): number | null {
spec = upgradeSpec(spec);
const up = spec.upgrade;
if (!up?.enabled) return null;
if (up.consistency === "season_release") return up.targetPri ?? 100;
const tiered = !!spec.soft?.tiers?.length || !!spec.rules?.groups.some((g) => g.layers.length > 0);
return (up.basis ?? (tiered ? "priority" : "tier")) === "priority" ? (up.targetPri ?? 100) : null;
}
@@ -413,6 +443,8 @@ interface PriVersion {
/** 覆盖的单元:季 → 集号集合;null = 整季(季包通配) */
units: Map<number, Set<number> | null>;
pri: number;
releaseKey: string | null;
quality: QualityFacts;
}
/**
@@ -441,13 +473,13 @@ async function upgradeReached(
kind: sub?.media_kind ?? (input.mediaType === "movie" ? "movie" : "series"),
};
const lang = await entityOriginalLanguage(exec, slot.media_entity_id);
const relPri = (rel: SourceReleaseRow) => {
const spec = slotSpecWithRules(cat, slot, sub, entity, rel.site_id);
const parsed = parsedOf(rel);
const relPri = (rel: SourceReleaseRow, quality?: QualityFacts) => {
const spec = upgradeSpec(slotSpecWithRules(cat, slot, sub, entity, rel.site_id));
const parsed = { ...parsedOf(rel), ...quality };
const ctx = { ...contextOf(rel), ...(lang ? { originalLanguage: lang } : {}) };
return effectiveLadder(spec, parsed, ctx, ruleChainOf(spec, parsed, ctx)).pri;
};
const baseSpec = slotSpecWithRules(cat, slot, sub, entity);
const baseSpec = upgradeSpec(slotSpecWithRules(cat, slot, sub, entity));
const versions: PriVersion[] = [];
const assetIds = [...new Set([...input.newAssets, ...(slot.current_asset_id ? [slot.current_asset_id] : [])])];
const assets = await exec
@@ -515,8 +547,17 @@ async function upgradeReached(
: [];
for (const x of assets) {
const rel = x.source_release_id ? rels.get(x.source_release_id) : undefined;
const quality = {
...(rel ? parsedOf(rel) : {}),
...((x.media_info as { _umt?: { quality?: QualityFacts } })?._umt?.quality ?? {}),
};
if (
baseSpec.hard.resolutions?.length &&
(!quality.resolution || !baseSpec.hard.resolutions.includes(quality.resolution))
)
continue;
const pri =
(rel ? relPri(rel) : undefined) ??
(rel ? relPri(rel, quality) : undefined) ??
evalPri(x.plan_evaluation) ??
currentPri(
baseSpec,
@@ -530,12 +571,18 @@ async function upgradeReached(
set.add(c.episode_number);
units.set(s, set);
}
versions.push({ units, pri });
versions.push({ units, pri, quality: { ...quality, pri }, releaseKey: seasonReleaseKey(quality) });
}
for (const p of plans) {
const rel = p.source_release_id ? rels.get(p.source_release_id) : undefined;
const pri = (rel ? relPri(rel) : undefined) ?? evalPri(p.evaluation);
if (pri === undefined) continue;
const quality: QualityFacts = rel ? parsedOf(rel) : {};
if (
pri === undefined ||
(baseSpec.hard.resolutions?.length &&
(!quality.resolution || !baseSpec.hard.resolutions.includes(quality.resolution)))
)
continue;
const units = new Map<number, Set<number> | null>();
for (const k of p.unit_keys) {
const u = parseUnitKey(k);
@@ -544,9 +591,33 @@ async function upgradeReached(
else if (u.episode !== undefined && units.get(u.season) !== null)
units.set(u.season, (units.get(u.season) ?? new Set<number>()).add(u.episode));
}
versions.push({ units, pri });
versions.push({ units, pri, quality: { ...quality, pri }, releaseKey: seasonReleaseKey(quality) });
}
if (input.mediaType === "movie") return versions.some((v) => v.pri >= input.target);
if (baseSpec.upgrade?.consistency === "season_release") {
for (const [season, eps] of input.covered) {
const targetKey = seasonReleaseKey(
selectSeasonRelease(
baseSpec,
versions.filter((v) => v.units.has(season)).map((v) => v.quality),
) ?? {},
);
if (
!targetKey ||
!eps.every((e) =>
versions.some(
(v) =>
v.releaseKey === targetKey &&
v.pri >= input.target &&
v.units.has(season) &&
(v.units.get(season) === null || v.units.get(season)!.has(e)),
),
)
)
return false;
}
return true;
}
for (const [season, eps] of input.covered)
for (const e of eps) {
const best = Math.max(
@@ -603,16 +674,21 @@ export async function slotComplete(
opts: { includeSubmitted?: boolean } = {},
): Promise<SlotCompletion> {
const completion = (slot.completion ?? {}) as { kind?: string; cuts?: string[] };
const accepted = opts.includeSubmitted
const allowedResolutions = ((slot.hard_conditions ?? {}) as { resolutions?: string[] }).resolutions;
const acceptedRows = opts.includeSubmitted
? await exec
.selectFrom("acquisition_plan as p")
.innerJoin("source_release as r", "r.id", "p.source_release_id")
.select(["p.id as plan_id", "p.unit_keys", "r.parsed", "r.title"])
.selectAll("r")
.select(["p.id as plan_id", "p.unit_keys"])
.where("p.slot_id", "=", slot.id)
.where("p.status", "in", ["submitted", "completed"])
.where("p.status_reason", "!=", "accepted_unconfirmed")
.execute()
: [];
const accepted = acceptedRows.filter(
(r) => !allowedResolutions?.length || allowedResolutions.includes(parsedOf(r).resolution ?? ""),
);
const sub = slot.subscription_id
? ((await exec.selectFrom("subscription").selectAll().where("id", "=", slot.subscription_id).executeTakeFirst()) ??
null)
@@ -735,7 +811,7 @@ export async function afterSlotVerified(
const plan = input.planId
? await trx
.selectFrom("acquisition_plan")
.select(["id", "purpose", "unit_keys"])
.select(["id", "purpose", "unit_keys", "evaluation"])
.where("id", "=", input.planId)
.executeTakeFirst()
: undefined;
@@ -757,7 +833,11 @@ export async function afterSlotVerified(
const planUnits = new Set(plan?.unit_keys ?? []);
for (const r of cov.rows) {
const q = qualityOf(r.media_info);
const pri = priorityLadder(spec, { raw: "", hdr: [], isEncode: false, ...q } as unknown as ParsedRelease).pri;
const plannedPri = (plan?.evaluation as { pri?: unknown })?.pri;
const pri =
typeof plannedPri === "number"
? plannedPri
: priorityLadder(spec, { raw: "", hdr: [], isEncode: false, ...q } as unknown as ParsedRelease).pri;
const inPlan = planUnits.has(r.unit_id) || (r.ep !== null && planUnits.has(episodeKey(r.season ?? 1, r.ep)));
const replace = plan?.purpose === "upgrade" ? inPlan : true;
const values = {
@@ -876,18 +956,6 @@ export async function completeSubscription(
reason: string,
progress: Record<string, unknown> = {},
): Promise<boolean> {
// 链式钩子 subscription.completion.check(ADR-006):追加完成条件,拒绝时订阅继续追踪
const veto = await interceptHook(d.ctx, "subscription.completion.check", () => ({
subscriptionId: sub.id,
mediaEntityId: sub.media_entity_id,
mediaKind: sub.media_kind,
reason,
progress,
}));
if (veto) {
d.ctx.logger.info({ subscriptionId: sub.id, reason, veto }, "订阅完成被钩子拦下，继续追踪");
return false;
}
const done = await trx
.updateTable("subscription")
.set({ status: "completed", status_reason: reason, next_search_at: null })
apps/server/src/modules/acquisition-organize/services/upgrade-replacement.ts
+49
View File
@@ -0,0 +1,49 @@
import type { AcquisitionSlotRow } from "@umt/db";
import { sql } from "@umt/db";
import { episodeKey, type QualityFacts, unitsOverlap } from "@umt/domain";
import type { Exec } from "./common.ts";
/** 自动洗版只选择本保留版本的旧文件；合集可以先冻结，完整覆盖后再回收。 */
export async function automaticReplacementAssets(
exec: Exec,
slot: AcquisitionSlotRow,
units: readonly string[],
candidate: QualityFacts,
): Promise<string[]> {
const rows = await sql<{
id: string;
resolution: string | null;
cut: string | null;
season: number | null;
episode: number | null;
}>`
select distinct a.id, coalesce(a.media_info->'_umt'->'quality'->>'resolution', r.parsed->>'resolution') as resolution,
a.content_edition as cut, u.season_number as season, u.episode_number as episode
from media_asset a
left join source_release r on r.id = a.source_release_id
left join download_task dt on dt.id = a.download_task_id
left join acquisition_plan p on p.id = dt.acquisition_plan_id
left join asset_unit_coverage c on c.media_asset_id = a.id and c.coverage = 'full'
left join content_unit u on u.id = c.content_unit_id
where a.media_entity_id = ${slot.media_entity_id} and a.status = 'active' and a.kind in ('video', 'bluray_dir')
and a.parent_asset_id is null
and (p.slot_id = ${slot.id} or a.id = ${slot.current_asset_id}
or exists (select 1 from slot_unit where slot_id = ${slot.id} and media_asset_id = a.id)
or (select count(*) from acquisition_slot where media_entity_id = ${slot.media_entity_id}) = 1)
and exists (select 1 from asset_location l join library_root lr on lr.id = l.library_root_id
where l.media_asset_id = a.id and l.location_kind = 'library' and l.status = 'available' and lr.management_mode = 'managed')
and not exists (select 1 from asset_location l join library_root lr on lr.id = l.library_root_id
where l.media_asset_id = a.id and l.location_kind = 'library' and l.status = 'available' and lr.management_mode <> 'managed')`.execute(
exec,
);
const allowed = ((slot.hard_conditions ?? {}) as { resolutions?: string[] }).resolutions;
const selected = new Set<string>();
for (const r of rows.rows) {
if (allowed?.length && (!r.resolution || !allowed.includes(r.resolution))) continue;
if (!units.length) {
if ((r.cut ?? "default") === (candidate.cut ?? "default")) selected.add(r.id);
} else if (r.season !== null && r.episode !== null && unitsOverlap(units, [episodeKey(r.season, r.episode)]).length)
selected.add(r.id);
}
return [...selected];
}
apps/server/src/modules/agent/authorize.ts
+11 -171
View File
@@ -1,7 +1,7 @@
import { createHash } from "node:crypto";
import type { AuthorizationChecker, AuthorizationDecision } from "@umt/agent";
import type { CapabilitySpec } from "@umt/contracts";
import { type DB, type Kysely, sql } from "@umt/db";
import { APPROVAL_PLAN, coveringGrant, type PlanBinding, planBinding } from "../../authorization/grants.ts";
import type { SettingsReader } from "../types.ts";
import { inlineApproval, optionApproval, proposeFirstBlock } from "./interaction.ts";
import { investigationDecisionId } from "./runner.ts";
@@ -15,172 +15,9 @@ import { exceedsAncestors } from "./subagents.ts";
// 模型参数里的"用户已同意"不会被读取;输入里的 approvalId 也只是指针,必须在库里核对。
// 控制台管理员从 Web 手动调用(source=web 且 actor=user:*)本身就是人工授权;API Key / Agent / 后台任务一律要授权记录。
/** 能力 → 可覆盖它的授权动作(策略发布生成 policy.<kind> 持续授权;"*" 全部) */
export const GRANT_ACTIONS: Record<string, readonly string[]> = {
"identity.resolve": ["identity.resolve", "policy.identity"],
"download.submit": ["download.submit", "policy.acquisition"],
"organize.execute": ["organize.execute", "policy.acquisition", "policy.organize"],
"download.recheck": ["download.recheck", "policy.automation"],
"task.retry": ["task.retry", "task.*"],
"task.cancel": ["task.cancel", "task.*"],
"policy.apply": ["policy.apply"],
// 集数映射兜底(批次 5):只认专门的持续授权,不随身份识别策略放开
"episode_mapping.apply_rule": ["episode_mapping.apply_rule"],
// VIS-1:坐标操作与视觉子流程和 browser.interact 同一档(操作原站页面),同一条持续授权覆盖
"browser.click_at": ["browser.click_at", "browser.interact"],
"browser.type_at": ["browser.type_at", "browser.interact"],
"browser.scroll": ["browser.scroll", "browser.interact"],
"browser.solve_challenge": ["browser.solve_challenge", "browser.interact"],
};
/** 视觉子流程内由 solve_challenge 授权覆盖的执行类坐标动作 */
const CHALLENGE_STEP_CAPS = new Set(["browser.click_at", "browser.type_at", "browser.scroll"]);
/** 单次审批绑定的计划类型与输入里的计划 ID 字段 */
export const APPROVAL_PLAN: Record<string, { planType: string; field: string }> = {
"download.submit": { planType: "acquisition_plan", field: "planId" },
"organize.execute": { planType: "organize_plan", field: "planId" },
"policy.apply": { planType: "policy_revision", field: "policyRevisionId" },
};
export interface PlanBinding {
payloadHash: string;
revision: number;
mediaEntityId: string | null;
mediaKind: string | null;
bytes: number | null;
}
const sha = (s: string) => createHash("sha256").update(s, "utf8").digest("hex");
/** 读计划当前的 payload hash / revision / 目标作品(审批绑定与范围校验共用) */
export async function planBinding(db: Kysely<DB>, planType: string, planId: string): Promise<PlanBinding | null> {
if (planType === "acquisition_plan") {
const r = await db
.selectFrom("acquisition_plan as p")
.leftJoin("acquisition_slot as s", "s.id", "p.slot_id")
.leftJoin("media_entity as e", (j) =>
j.onRef("e.id", "=", sql<string>`coalesce(s.media_entity_id, (p.evaluation->>'mediaEntityId')::uuid)`),
)
.select(["p.payload_hash", "p.expected_revision", "e.id as entity_id", "e.kind", "p.reserved_bytes"])
.where("p.id", "=", planId)
.executeTakeFirst();
return r
? {
payloadHash: r.payload_hash,
revision: r.expected_revision,
mediaEntityId: r.entity_id,
mediaKind: r.kind,
bytes: Number(r.reserved_bytes),
}
: null;
}
if (planType === "organize_plan") {
const r = await db
.selectFrom("artifact")
.select(["content_hash", "content"])
.where("id", "=", planId)
.where("kind", "=", "plan")
.executeTakeFirst();
if (!r) return null;
const c = (r.content ?? {}) as { mediaEntityId?: string | null; mediaKind?: string | null };
return {
payloadHash: r.content_hash,
revision: 1,
mediaEntityId: c.mediaEntityId ?? null,
mediaKind: c.mediaKind ?? null,
bytes: null,
};
}
if (planType === "policy_revision") {
const r = await db
.selectFrom("policy_revision")
.select(["revision", "rules", "scope"])
.where("id", "=", planId)
.executeTakeFirst();
return r
? {
payloadHash: sha(JSON.stringify({ rules: r.rules, scope: r.scope })),
revision: r.revision,
mediaEntityId: null,
mediaKind: null,
bytes: null,
}
: null;
}
return null;
}
/**
* 持续授权是否覆盖本能力(不含单次审批与会话内审批):action 覆盖、active 未过期、绑定策略仍发布、
* 对象范围与每日预算。download.plan 用它判定 requiresApproval,authorize 用它放行。
*/
export async function coveringGrant(
db: Kysely<DB>,
capabilityId: string,
target: PlanBinding | null,
now: Date,
): Promise<{ grantId: string } | { reasons: string[] }> {
const d = { id: capabilityId };
const actions = [...(GRANT_ACTIONS[d.id] ?? [d.id]), "*"];
const grants = await db
.selectFrom("automation_grant as g")
.leftJoin("policy_revision as p", "p.id", "g.policy_revision_id")
.select(["g.id", "g.action", "g.scope", "g.budget", "g.policy_revision_id", "p.status as policy_status"])
.where("g.status", "=", "active")
.where("g.action", "in", actions)
.where((eb) => eb.or([eb("g.expires_at", "is", null), eb("g.expires_at", ">", now)]))
.orderBy("g.created_at", "desc")
.execute();
const reasons: string[] = [];
for (const g of grants) {
if (g.policy_revision_id && g.policy_status !== "published") {
reasons.push(`授权 ${g.id} 绑定的策略版本已不是发布状态`);
continue;
}
const scope = (g.scope ?? {}) as { mediaEntityIds?: string[]; mediaKinds?: string[]; capabilities?: string[] };
if (scope.capabilities?.length && !scope.capabilities.includes(d.id)) {
reasons.push(`授权 ${g.id} 不含能力 ${d.id}`);
continue;
}
if (
scope.mediaEntityIds?.length &&
(!target?.mediaEntityId || !scope.mediaEntityIds.includes(target.mediaEntityId))
) {
reasons.push(`授权 ${g.id} 的对象范围不含目标作品`);
continue;
}
if (scope.mediaKinds?.length && target?.mediaKind && !kindCovered(scope.mediaKinds, target.mediaKind)) {
reasons.push(`授权 ${g.id} 的媒体类型范围不含 ${target.mediaKind}`);
continue;
}
const budget = (g.budget ?? {}) as { maxInvocationsPerDay?: number; maxBytesPerDay?: number };
if (budget.maxInvocationsPerDay !== undefined || budget.maxBytesPerDay !== undefined) {
const day = new Date(now);
day.setHours(0, 0, 0, 0);
const used = await sql<{ n: number }>`
select count(*)::int as n from capability_invocation
where grant_id = ${g.id} and status = 'succeeded' and created_at >= ${day}`.execute(db);
if (budget.maxInvocationsPerDay !== undefined && (used.rows[0]?.n ?? 0) >= budget.maxInvocationsPerDay) {
reasons.push(`授权 ${g.id} 今日次数预算已用尽`);
continue;
}
if (budget.maxBytesPerDay !== undefined && d.id === "download.submit") {
const bytes = await sql<{ b: string }>`
select coalesce(sum(p.reserved_bytes), 0)::bigint as b
from capability_invocation ci join acquisition_plan p on p.id::text = ci.input->>'planId'
where ci.grant_id = ${g.id} and ci.status = 'succeeded' and ci.created_at >= ${day}`.execute(db);
if (Number(bytes.rows[0]?.b ?? 0) + (target?.bytes ?? 0) > budget.maxBytesPerDay) {
reasons.push(`授权 ${g.id} 今日字节预算不足`);
continue;
}
}
}
return { grantId: g.id };
}
return { reasons };
}
export function createAuthorizer(deps: {
db: Kysely<DB>;
settings: SettingsReader;
@@ -284,6 +121,16 @@ export function createAuthorizer(deps: {
// 集数映射:持续授权的对象范围按事项的作品核对
if (d.id === "episode_mapping.apply_rule" && typeof inp.decisionId === "string")
target = await episodeTarget(db, inp.decisionId);
if (d.id === "release.apply_facts" && typeof inp.sourceReleaseId === "string") {
const release = await db
.selectFrom("source_release as r")
.innerJoin("media_entity as e", "e.id", "r.media_entity_id")
.select(["e.id", "e.kind"])
.where("r.id", "=", inp.sourceReleaseId)
.executeTakeFirst();
if (!release) return { allowed: false, reason: "资源尚未关联作品，无法核对属性补全授权范围" };
target = { payloadHash: "", revision: 0, mediaEntityId: release.id, mediaKind: release.kind, bytes: null };
}
// 2) 持续授权
const covered = await coveringGrant(db, d.id, target, now());
@@ -330,10 +177,3 @@ async function episodeTarget(db: Kysely<DB>, decisionId: string): Promise<PlanBi
}
const isUuid = (s: string) => /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i.test(s);
/** 策略范围用订阅口径(movie / series / variety),作品用 media_entity.kind(movie / tv / variety …) */
function kindCovered(kinds: readonly string[], entityKind: string): boolean {
if (kinds.includes(entityKind)) return true;
if (entityKind !== "movie" && entityKind !== "variety" && kinds.includes("series")) return true;
return false;
}
apps/server/src/modules/agent/capabilities-download.ts
+1 -1
View File
@@ -1,8 +1,8 @@
import { CapabilityError, type CapabilityRegistry } from "@umt/agent";
import { capabilitySpec, type DownloadStatus } from "@umt/contracts";
import { planBinding } from "../../authorization/grants.ts";
import { downloaderClientFor } from "../acquisition-organize/ports.ts";
import type { ServerContext } from "../types.ts";
import { planBinding } from "./authorize.ts";
import type { AgentModuleOptions } from "./runner.ts";
// download.recheck(REQ §8.4.2 / §10.7 授权内恢复):对已纳管的下载任务发起 qB 强制重新校验或恢复暂停。
apps/server/src/modules/agent/capabilities-metadata.ts
+6 -2
View File
@@ -37,7 +37,7 @@ export interface MetadataCapabilityOptions {
/** 模型调用(测试注入 fake);返回 null = 按规则抽取 */
complete?: EnrichDeps["complete"];
/** 读 robots.txt 用的 fetch / DNS 解析(测试注入 mock) */
fetch?: typeof fetch;
fetch?: (input: RequestInfo | URL, init?: RequestInit) => Promise<Response>;
resolveHost?: HostResolver;
}
@@ -51,7 +51,11 @@ const ROBOTS_TTL_SEC = 86400;
export async function robotsAllowedFor(
ctx: Pick<ServerContext, "db">,
url: string,
opts: { fetch?: typeof fetch; resolveHost?: HostResolver; signal?: AbortSignal } = {},
opts: {
fetch?: (input: RequestInfo | URL, init?: RequestInit) => Promise<Response>;
resolveHost?: HostResolver;
signal?: AbortSignal;
} = {},
): Promise<boolean> {
let origin: string;
try {
apps/server/src/modules/agent/interaction.ts
+1 -1
View File
@@ -21,8 +21,8 @@ import {
type KnownCapabilityId,
} from "@umt/contracts";
import { type AgentRunRow, type DB, type Kysely, sql } from "@umt/db";
import { APPROVAL_PLAN, type PlanBinding, planBinding } from "../../authorization/grants.ts";
import type { ServerContext } from "../types.ts";
import { APPROVAL_PLAN, type PlanBinding, planBinding } from "./authorize.ts";
import {
AwaitSuspendedError,
type AwaitSuspension,
apps/server/src/modules/agent/investigate.ts
+57 -16
View File
@@ -1,7 +1,8 @@
import type { DecisionOption, DownloadStatus } from "@umt/contracts";
import { type DecisionRow, sql } from "@umt/db";
import { GRANT_ACTIONS } from "../../authorization/grants.ts";
import { missingReleaseFields } from "../sources-identity/service/release-facts-admin.ts";
import type { ServerContext } from "../types.ts";
import { GRANT_ACTIONS } from "./authorize.ts";
import {
type DriveResult,
INVESTIGATE_CAPABILITIES,
@@ -138,6 +139,8 @@ async function relatedTask(ctx: ServerContext, d: DecisionRow) {
}
function suggestionFor(d: DecisionRow, reason: string): string {
if (d.dedupe_key.startsWith("release_facts:"))
return `${reason}。可查看资源详情补齐属性，或为 release.apply_facts 设置持续授权。`;
if (d.dedupe_key.startsWith("organize_identity:")) return `${reason}。可在整理记录里指定作品或重新整理。`;
if (d.kind === "identity_ambiguous")
return `${reason}。可在媒体库「未识别」里手动指定作品，或为 identity.resolve 发放持续授权。`;
@@ -183,7 +186,8 @@ export async function autoInvestigate(
const d = await ctx.db.selectFrom("decision").selectAll().where("id", "=", decisionId).executeTakeFirst();
if (d?.dedupe_key.startsWith("organize_identity:")) return identifyOrganize(ctx, d, opts.manual === true);
if (d?.status !== "open") return { runId: null, created: false };
if (!(KINDS as readonly string[]).includes(d.kind)) return { runId: null, created: false };
const releaseFacts = d.dedupe_key.startsWith("release_facts:");
if (!(KINDS as readonly string[]).includes(d.kind) && !releaseFacts) return { runId: null, created: false };
// Agent 自己引起的事项不再触发 Agent(防止递归自触发)
if (d.subject_type === "agent_run") return { runId: null, created: false };
const s = await ctx.settings.get("agent");
@@ -228,10 +232,12 @@ export async function autoInvestigate(
);
return { runId: null, created: false, skipped: "attempts_exhausted" };
}
const grants = await recoveryGrants(ctx, ACTIONS_FOR[d.kind] ?? []);
const grants = await recoveryGrants(ctx, releaseFacts ? ["release.apply_facts"] : (ACTIONS_FOR[d.kind] ?? []));
if (grants.length === 0) {
mode = "investigate";
degraded = { reason: `没有覆盖 ${(ACTIONS_FOR[d.kind] ?? []).join(" / ")} 的有效持续授权` };
degraded = {
reason: `没有覆盖 ${(releaseFacts ? ["release.apply_facts"] : (ACTIONS_FOR[d.kind] ?? [])).join(" / ")} 的有效持续授权`,
};
}
}
@@ -252,23 +258,41 @@ export async function autoInvestigate(
d.subject_type && d.subject_id ? `对象：${d.subject_type} ${d.subject_id}。` : "",
dt ? `下载任务 ${dt.id}「${dt.name ?? ""}」当前 ${dt.status}（qB 状态 ${dt.raw_state ?? "未知"}）。` : "",
taskId ? `关联任务 ${taskId}。` : "",
ep
? `作品 ${String(ep.mediaEntityId ?? d.subject_id ?? "")}，来源季 ${ep.season ?? "未知"}，发布组 ${ep.releaseGroup ?? "无"}。` +
releaseFacts
? `资源 ${d.subject_id}，缺失属性 ${JSON.stringify((d.evidence as { missing?: string[] })?.missing ?? [])}。先用 release.inspect(refreshDetail:true) 看详情、MediaInfo 和文件证据；` +
(mode === "recover"
? "先用 episode_mapping.inspect（decisionId）看上下文，再用 episode_mapping.simulate 试一行作品识别词（如 S02 @发布组 >> S01 EP+12）；" +
"模拟里事项样本全部解决、且没有让已对上的资源变成待确认时，才用 episode_mapping.apply_rule 把这一行写进作品识别词一次（带理由）；" +
"拿不准就不写，只把模拟通过的那一行留作建议。不逐集处理，是否解决由系统核实。"
: "只做只读调查：用 episode_mapping.inspect（decisionId）看上下文，用 episode_mapping.simulate（带 decisionId 和理由）试一行作品识别词；" +
"模拟通过的那一行会作为智能体建议留在事项上，由用户确认；不写规则，拿不准就只说明原因。")
: mode === "recover"
? "在授权范围内做有界恢复：先用 task.inspect 等只读能力查证，再按需调用 download.recheck（强制重新校验 / 恢复暂停）、" +
"task.retry（重试失败任务）或 identity.resolve，每种最多一次；不换源、不删除、不清理；恢复结果由系统核实。"
: "只做只读调查：查证据，给出结论与处理建议；不提交下载，不改文件。",
? `有证据时用 release.apply_facts(decisionId:${d.id},sourceReleaseId:${d.subject_id}) 只补缺失事实，原规则自动重评；`
: "只调查并给出有出处的补全建议；") +
"不修改收录范围、保留目标或组合优先级，不直接选资源替代规则判断。"
: ep
? `作品 ${String(ep.mediaEntityId ?? d.subject_id ?? "")}，来源季 ${ep.season ?? "未知"}，发布组 ${ep.releaseGroup ?? "无"}。` +
(mode === "recover"
? "先用 episode_mapping.inspect（decisionId）看上下文，再用 episode_mapping.simulate 试一行作品识别词（如 S02 @发布组 >> S01 EP+12）；" +
"模拟里事项样本全部解决、且没有让已对上的资源变成待确认时，才用 episode_mapping.apply_rule 把这一行写进作品识别词一次（带理由）；" +
"拿不准就不写，只把模拟通过的那一行留作建议。不逐集处理，是否解决由系统核实。"
: "只做只读调查：用 episode_mapping.inspect（decisionId）看上下文，用 episode_mapping.simulate（带 decisionId 和理由）试一行作品识别词；" +
"模拟通过的那一行会作为智能体建议留在事项上，由用户确认；不写规则，拿不准就只说明原因。")
: mode === "recover"
? "在授权范围内做有界恢复：先用 task.inspect 等只读能力查证，再按需调用 download.recheck（强制重新校验 / 恢复暂停）、" +
"task.retry（重试失败任务）或 identity.resolve，每种最多一次；不换源、不删除、不清理；恢复结果由系统核实。"
: "只做只读调查：查证据，给出结论与处理建议；不提交下载，不改文件。",
].join("");
const { run, created } = await startRun(ctx, {
goal,
mode: mode === "recover" ? "execute" : "plan_only",
capabilityAllowlist: mode === "recover" ? [...RECOVER_CAPABILITIES] : [...INVESTIGATE_CAPABILITIES],
capabilityAllowlist: releaseFacts
? [
"release.inspect",
"catalog.lookup",
"inventory.query",
"decision.list",
"web.search",
"web.fetch",
...(mode === "recover" ? ["release.apply_facts"] : []),
]
: mode === "recover"
? [...RECOVER_CAPABILITIES]
: [...INVESTIGATE_CAPABILITIES],
trigger: "event",
createdBy: mode === "recover" ? RECOVER_ACTOR : AUTO_ACTOR,
dedupeKey: `investigate:${d.id}`,
@@ -460,6 +484,23 @@ export async function checkRecovery(ctx: ServerContext, actions: RecoveryAction[
: undefined;
facts[`source_release:${id}`] = r ? { resolutionStatus: r.resolution_status } : null;
states.push(r?.resolution_status === "resolved" && r.media_entity_id ? "recovered" : "pending");
} else if (a.capabilityId === "release.apply_facts") {
const id = String(a.input.decisionId ?? "");
const decision = await ctx.db.selectFrom("decision").selectAll().where("id", "=", id).executeTakeFirst();
const release = decision?.subject_id
? await ctx.db.selectFrom("source_release").selectAll().where("id", "=", decision.subject_id).executeTakeFirst()
: undefined;
const remaining = release
? missingReleaseFields(release, (decision?.evidence as { missing?: string[] })?.missing ?? [])
: [];
facts[`decision:${id}`] = { status: decision?.status ?? null, remaining, releaseId: release?.id ?? null };
states.push(
!decision || !release
? "failed"
: decision.status === "resolved" && remaining.length === 0
? "recovered"
: "pending",
);
} else if (a.capabilityId === "organize.assign_identity") {
// 指定作品后事项随即关闭、重新整理已入队;事项关闭即视为恢复(整理结果看整理记录)
const id = String(a.input.decisionId ?? "");
apps/server/src/modules/agent/routes.ts
+1 -1
View File
@@ -7,12 +7,12 @@ import {
type KnownCapabilityId,
} from "@umt/contracts";
import { type AgentRunRow, sql } from "@umt/db";
import { planBinding } from "../../authorization/grants.ts";
import { hasScope } from "../../http/auth.ts";
import { ApiException } from "../../http/errors.ts";
import { planManualAcquisition } from "../acquisition-organize/services/manual-acquisition.ts";
import { InternalDispatcher } from "../search/dispatch.ts";
import type { ModuleApp, ServerContext } from "../types.ts";
import { planBinding } from "./authorize.ts";
import { cancelTask, retryTask, taskToApi } from "./capabilities.ts";
import { route } from "./http.ts";
import { afterApprovalDecided, answerOptions } from "./interaction.ts";
apps/server/src/modules/agent/runner.ts
+3 -4
View File
@@ -91,6 +91,7 @@ export const RECOVER_CAPABILITIES = [
"task.retry",
"download.recheck",
"episode_mapping.apply_rule",
"release.apply_facts",
] as const;
/**
* 整理没识别出作品时的智能体识别:查作品(TMDB / 媒体库 / 网页)+ 指定作品并重新整理。
@@ -116,6 +117,7 @@ export function investigationDecisionId(dedupeKey: string | null | undefined): s
}
/** 被视为"恢复动作"的能力(结果需系统核实) */
export const RECOVERY_ACTIONS = [
"release.apply_facts",
"task.retry",
"download.recheck",
"identity.resolve",
@@ -158,10 +160,7 @@ export function scopeFor(mode: "execute" | "plan_only", allowlist?: readonly str
*/
export function sessionDirOf(env: AgentEnv, configured?: string | null): string {
return resolve(
env.opts.sessionDir ??
configured ??
Deno.env.get("AGENT_SESSION_DIR") ??
join(Deno.cwd(), "data", "agent-sessions"),
env.opts.sessionDir ?? configured ?? process.env.AGENT_SESSION_DIR ?? join(process.cwd(), "data", "agent-sessions"),
);
}
apps/server/src/modules/agent/tasks.ts
+5 -1
View File
@@ -266,7 +266,11 @@ export async function executeResolution(ctx: ServerContext, decisionId: string)
export const agentOutboxRoutes: Record<string, OutboxRouteFn> = {
"decision.created": (ev) => {
const p = ev.payload as EventPayload<"decision.created">;
if (!["identity_ambiguous", "download_failed", "download_stalled", "episode_mapping"].includes(p.kind)) return null;
if (
!["identity_ambiguous", "download_failed", "download_stalled", "episode_mapping"].includes(p.kind) &&
!p.dedupeKey?.startsWith("release_facts:")
)
return null;
return { task: "resolveDecision", input: { decisionId: p.decisionId, trigger: "created" } };
},
"decision.updated": (ev) => {
apps/server/src/modules/browser/keepalive.ts
+2 -1
View File
@@ -1,3 +1,4 @@
import { runtimeUnrefTimer } from "@umt/runtime";
// AG-4 浏览器会话保活:camofox 的标签页空闲 TAB_INACTIVITY_MS(缺省 5 分钟)没有 API 调用就回收,
// 会话空闲 SESSION_TIMEOUT_MS 回收,全部会话没了之后 BROWSER_IDLE_TIMEOUT_MS 退出浏览器。
// 桌面模式里人的键鼠直接进 Selkies,不经 API;原站视图的截图轮询也不算标签页活动——接管期间页面会被关掉。
@@ -151,7 +152,7 @@ export class BrowserKeepalive {
this.timer = setInterval(() => void this.tick(), this.opts.intervalMs);
// 不拖住进程退出(测试、优雅停机)
try {
Deno.unrefTimer(this.timer as unknown as number);
runtimeUnrefTimer(this.timer as unknown as number);
} catch {
/* 非 Deno 运行时 */
}
apps/server/src/modules/browser/service.ts
+2 -2
View File
@@ -98,7 +98,7 @@ export type Actor = {
export interface BrowserModuleOptions {
/** camofox 请求用的 fetch(测试指向 mock) */
fetch?: typeof globalThis.fetch;
fetch?: (input: RequestInfo | URL, init?: RequestInit) => Promise<Response>;
/** 站点限流器所在的运行时(与快照/实时搜索共享) */
runtime?: (ctx: ServerContext) => SourcesRuntime;
now?: () => Date;
@@ -122,7 +122,7 @@ export interface BrowserModuleOptions {
export interface ProviderDeps {
secrets: BrowserSecretResolver;
sink: DownloadSink;
fetch?: typeof globalThis.fetch;
fetch?: (input: RequestInfo | URL, init?: RequestInit) => Promise<Response>;
downloadPollMs?: number;
}
apps/server/src/modules/cache/images.ts
Vendored
+26 -11
View File
@@ -1,7 +1,7 @@
import { createHash } from "node:crypto";
import { promises as fs } from "node:fs";
import { dirname, extname, join } from "node:path";
import { TMDB_IMAGE_SIZES } from "@umt/contracts";
import { externalImageReferer, TMDB_IMAGE_SIZES } from "@umt/contracts";
// TMDB 图片代理与文件缓存,移植 MP app/application/image.py(ImageHelper)+ api/endpoints/system.py(fetch_image):
// - 缓存路径 = SecurityUtils.sanitize_url_path(url):URL 路径去掉开头 "/" 后百分号编码,超过 120 字符换成
@@ -63,14 +63,14 @@ interface ImageFile {
export interface ImageCacheOptions {
/** DATA_DIR/cache/images */
dir: string;
fetch?: typeof globalThis.fetch;
fetch?: (input: RequestInfo | URL, init?: RequestInit) => Promise<Response>;
timeoutMs?: number;
/** 上游基址(测试指向 mock);缺省 https://image.tmdb.org/t/p */
upstreamBase?: string;
}
export class ImageCache {
private readonly fetchImpl: typeof globalThis.fetch;
private readonly fetchImpl: (input: RequestInfo | URL, init?: RequestInit) => Promise<Response>;
constructor(private readonly opts: ImageCacheOptions) {
this.fetchImpl = opts.fetch ?? globalThis.fetch;
@@ -94,8 +94,18 @@ export class ImageCache {
/** fetch_image:取图(可读缓存)→ ETag / 304 */
async serve(size: string, file: string, useCache: boolean, ifNoneMatch: string | null): Promise<Response> {
const url = this.upstream(size, file);
const { content, mime } = await this.fetchImage(url, useCache);
return this.respond(await this.fetchImage(this.upstream(size, file), useCache), ifNoneMatch);
}
/** 豆瓣 / Bangumi 封面(白名单域名):带来源站 Referer 取图,缓存按域名分目录 */
async serveExternal(url: string, useCache: boolean, ifNoneMatch: string | null): Promise<Response> {
const referer = externalImageReferer(url);
if (!referer) throw new ImageProxyError(400, "只代理豆瓣与 Bangumi 的图片");
const host = new URL(url).hostname.toLowerCase();
return this.respond(await this.fetchImage(url, useCache, { referer, cacheDir: join("ext", host) }), ifNoneMatch);
}
private respond({ content, mime }: { content: Uint8Array; mime: string }, ifNoneMatch: string | null): Response {
const etag = createHash("md5").update(content).digest("hex");
const headers = new Headers({
etag,
@@ -116,8 +126,13 @@ export class ImageCache {
}
/** ImageHelper.fetch_image_with_mime_type */
private async fetchImage(url: string, useCache: boolean): Promise<{ content: Uint8Array; mime: string }> {
const path = join(this.opts.dir, imageCachePath(url));
private async fetchImage(
url: string,
useCache: boolean,
ext: { referer: string; cacheDir: string } | null = null,
): Promise<{ content: Uint8Array; mime: string }> {
const path = join(this.opts.dir, ext?.cacheDir ?? "", imageCachePath(url));
const what = ext ? "图片" : "TMDB 图片";
if (useCache) {
const cached = await fs.readFile(path).catch(() => null);
const mime = cached?.length ? imageMimeType(cached) : null;
@@ -130,19 +145,19 @@ export class ImageCache {
try {
res = await this.fetchImpl(url, {
signal: AbortSignal.timeout(this.opts.timeoutMs ?? 20_000),
headers: { accept: "image/avif,image/webp,image/apng,*/*" },
headers: { accept: "image/avif,image/webp,image/apng,*/*", ...(ext ? { referer: ext.referer } : {}) },
});
} catch (e) {
const reason = (e as Error).name === "TimeoutError" ? "超时" : (e as Error).message;
throw new ImageProxyError(502, `拉取 TMDB 图片失败：${reason}`);
throw new ImageProxyError(502, `拉取${what}失败：${reason}`);
}
if (res.status !== 200) {
await res.body?.cancel().catch(() => {});
throw new ImageProxyError(502, `TMDB 图片返回 HTTP ${res.status}`);
throw new ImageProxyError(502, `${what}返回 HTTP ${res.status}`);
}
const content = new Uint8Array(await res.arrayBuffer());
const mime = imageMimeType(content);
if (!mime) throw new ImageProxyError(502, "TMDB 返回的不是图片");
if (!mime) throw new ImageProxyError(502, `${what}地址返回的不是图片`);
await fs.mkdir(dirname(path), { recursive: true });
const tmp = `${path}.${process.pid}.tmp`;
await fs.writeFile(tmp, content);
apps/server/src/modules/cache/index.ts
Vendored
+13 -1
View File
@@ -38,7 +38,7 @@ import { ImageCache, ImageProxyError } from "./images.ts";
*/
export interface CacheModuleOptions {
/** 图片上游 fetch(测试注入) */
imageFetch?: typeof globalThis.fetch;
imageFetch?: (input: RequestInfo | URL, init?: RequestInit) => Promise<Response>;
/** 图片上游基址(测试指向 mock) */
imageUpstreamBase?: string;
/** 图片缓存目录;缺省 DATA_DIR/cache/images */
@@ -130,6 +130,18 @@ export function createCacheModule(opts: CacheModuleOptions = {}): ServerModule &
}
});
// 豆瓣 / Bangumi 封面:浏览器直连豆瓣图床拿不到(不带 Referer 回 418),经这里带 Referer 取;总是读写磁盘缓存
app.get("/img/ext", requireScope("read"), async (c) => {
try {
return await images(ctx).serveExternal(c.req.query("u") ?? "", true, c.req.header("if-none-match") ?? null);
} catch (e) {
if (e instanceof ImageProxyError) {
return c.json(errorBody(e.status === 400 ? "bad_request" : "upstream_unavailable", e.message), e.status);
}
throw e;
}
});
route(app, Api.cache.tmdb, async ({ query }) => {
const cache = sharedCache(ctx);
const policy = await tmdbCachePolicy(ctx);
apps/server/src/modules/desktop/desktop.ts
+4 -4
View File
@@ -1,5 +1,6 @@
import { createHmac } from "node:crypto";
import type { DB, Kysely } from "@umt/db";
import { upgradeWebSocket } from "@umt/runtime";
import type { Context } from "hono";
import type { AuthEnv } from "../../http/auth.ts";
import { ApiException } from "../../http/errors.ts";
@@ -61,7 +62,7 @@ export interface DesktopDeps {
checkSession(sessionId: string, holder: string): Promise<SessionCheck>;
/** 控制台会话是否仍有效(WS 长连接期间复查) */
consoleSessionValid(sessionRowId: string): Promise<boolean>;
fetch?: typeof fetch;
fetch?: (input: RequestInfo | URL, init?: RequestInit) => Promise<Response>;
/** 角色复查间隔 */
roleCheckMs?: number;
/** 桌面连接建立(AG-4):返回连接断开时调用的函数。用于连接期间保活浏览器会话 */
@@ -325,11 +326,10 @@ function bridgeWebSocket(req: Request, target: string, role: DesktopRole, o: Bri
.split(",")
.map((s) => s.trim())
.filter(Boolean);
const { socket: client, response } = Deno.upgradeWebSocket(req, protocols[0] ? { protocol: protocols[0] } : {});
const { socket: client, response } = upgradeWebSocket(req, protocols[0] ? { protocol: protocols[0] } : {});
const onClose = o.onOpen?.();
client.binaryType = "arraybuffer";
// Deno 的客户端 WebSocket 支持自定义请求头;不带 Origin(Selkies 对无 Origin 的非浏览器客户端放行)
// (tsc 按 DOM 类型只认 protocols 参数;Deno 运行时接受 { headers, protocols })
// Bun 的客户端 WebSocket 支持自定义认证头;不带 Origin(Selkies 对无 Origin 的客户端放行)
const wsOptions = { headers: { authorization: o.auth(role) }, ...(protocols.length ? { protocols } : {}) };
const upstream = new WebSocket(target, wsOptions as unknown as string[]);
upstream.binaryType = "arraybuffer";
apps/server/src/modules/desktop/index.ts
+1 -1
View File
@@ -17,7 +17,7 @@ export function createDesktopModule(): ServerModule {
const r = registerDesktopRoutes(app, {
upstream: desktopUpstream({
providerUrl: ctx.config?.BROWSER_PROVIDER_URL,
desktopUrl: Deno.env.get("BROWSER_DESKTOP_URL") || undefined,
desktopUrl: process.env.BROWSER_DESKTOP_URL || undefined,
}),
accessKey: ctx.config?.BROWSER_ACCESS_KEY ?? null,
...dbSessionChecks(ctx.db),
apps/server/src/modules/file-manager/common.ts
+2 -2
View File
@@ -54,7 +54,7 @@ export interface FmDeps {
}
const envNum = (k: string, dflt: number) => {
const v = Number(Deno.env.get(k));
const v = Number(process.env[k]);
return Number.isFinite(v) && v > 0 ? v : dflt;
};
@@ -66,7 +66,7 @@ export function defaultOptions(ctx: ServerContext): FmOptions {
trashCrossMaxBytes: envNum("FILES_TRASH_CROSS_MAX_BYTES", 2 * 1024 ** 3),
planTtlMs: envNum("FILES_PLAN_TTL_SEC", 1800) * 1000,
identifyMaxVideos: 200,
allowPermanentDelete: Deno.env.get("FILES_ALLOW_PERMANENT_DELETE") === "true",
allowPermanentDelete: process.env.FILES_ALLOW_PERMANENT_DELETE === "true",
healthTimeoutMs: 2000,
purgeIntervalMs: 24 * 3600_000,
backgroundLoops: true,
apps/server/src/modules/file-manager/download.ts
+2 -1
View File
@@ -1,5 +1,6 @@
import { basename, extname } from "node:path";
import type { FilesDownloadQuery } from "@umt/contracts";
import { runtimeOpen } from "@umt/runtime";
import type { Principal } from "../../http/auth.ts";
import { remote, remoteError } from "../../storage/vfs.ts";
import { FilesError, type FmDeps, isAdminSession, recheck, resolveIn } from "./common.ts";
@@ -57,7 +58,7 @@ export async function downloadFile(
throw new FilesError("forbidden", `只能下载 NFO、字幕、图片、文本和 .torrent，这个文件是 ${kind}`);
const real = await recheck(r.abs, r.rootReal);
if (r.def.storageId) return downloadRemote(d, principal, q, r.rel, real, name, kind, pk);
const file = await Deno.open(real, { read: true });
const file = await runtimeOpen(real, { read: true });
let size: number;
try {
const st = await file.stat();
apps/server/src/modules/file-manager/probe.ts
+2 -1
View File
@@ -9,6 +9,7 @@ import type {
FilesScrapeBatchResult,
} from "@umt/contracts";
import { sql } from "@umt/db";
import { runtimeRename } from "@umt/runtime";
import type { Principal } from "../../http/auth.ts";
import { mkdir, readFile, writeFile } from "../../storage/local-fs.ts";
import { vlstat } from "../../storage/vfs.ts";
@@ -84,7 +85,7 @@ export async function probeFiles(d: FmDeps, body: FilesProbeRequest): Promise<Fi
await mkdir(cacheDir, { recursive: true }).catch(() => undefined);
const tmp = `${cacheFile}.${randomUUID()}.tmp`;
await writeFile(tmp, JSON.stringify({ at: Date.now(), result }))
.then(() => Deno.rename(tmp, cacheFile))
.then(() => runtimeRename(tmp, cacheFile))
.catch(() => undefined);
}
items.push({
apps/server/src/modules/mcp/index.ts
+2 -2
View File
@@ -1,5 +1,5 @@
import { featureManifest } from "@umt/contracts";
import denoJson from "../../../deno.json" with { type: "json" };
import packageJson from "../../../package.json" with { type: "json" };
import type { ServerModule } from "../types.ts";
import { createMcpConfigStore } from "./config-store.ts";
import { registerMcpRoutes } from "./routes.ts";
@@ -20,7 +20,7 @@ export function createMcpModule(opts: { version?: string } = {}): ServerModule &
registerMcpRoutes(app, ctx, {
table,
config: createMcpConfigStore(() => ctx.db),
version: opts.version ?? denoJson.version,
version: opts.version ?? packageJson.version,
});
},
registerTasks() {},
apps/server/src/modules/mcp/sessions.ts
+2 -1
View File
@@ -3,6 +3,7 @@ import type { Server } from "@modelcontextprotocol/sdk/server/index.js";
// @ts-types="@modelcontextprotocol/sdk/server/webStandardStreamableHttp"
import type { WebStandardStreamableHTTPServerTransport } from "@modelcontextprotocol/sdk/server/webStandardStreamableHttp.js";
import type { McpSessionInfo } from "@umt/contracts";
import { runtimeUnrefTimer } from "@umt/runtime";
// MCP 会话表(进程内):会话与建立它的 API Key 绑定,换 Key 复用会话一律拒绝。
// 空闲超时回收;每个 Key 的会话数与每分钟工具调用数有上限。
@@ -128,6 +129,6 @@ export class McpSessionTable {
this.timer = setInterval(() => void this.sweep(), 60_000);
// 不因为回收定时器挂住进程退出
(this.timer as unknown as { unref?: () => void }).unref?.();
if (typeof Deno !== "undefined") Deno.unrefTimer(this.timer as unknown as number);
runtimeUnrefTimer(this.timer as unknown as number);
}
}
apps/server/src/modules/mcp/stats.ts
+1 -1
View File
@@ -13,7 +13,7 @@ import {
} from "@umt/contracts";
import { type DB, type Kysely, sql } from "@umt/db";
import { z } from "zod";
import { GRANT_ACTIONS } from "../agent/authorize.ts";
import { GRANT_ACTIONS } from "../../authorization/grants.ts";
import type { SettingsReader } from "../types.ts";
// 能力调用统计与能力清单。统计口径是 audit_event(registry.invoke 每次调用都写一条,含只读能力与拒绝);
apps/server/src/modules/pi/oauth.ts
+1 -1
View File
@@ -48,7 +48,7 @@ interface Session {
/** manual_code 挂起的解析函数 */
paste: ((v: string) => void) | null;
pasteWaiters: (() => void)[];
firstEvent: PromiseWithResolvers<void>;
firstEvent: ReturnType<typeof Promise.withResolvers<void>>;
done: Promise<void>;
timer: ReturnType<typeof setTimeout>;
}
apps/server/src/modules/sources-identity/capabilities.ts
+13 -2
View File
@@ -6,7 +6,9 @@ import { registerEpisodeMappingCapabilities } from "./capabilities-episode.ts";
import { registerSiteCapabilities } from "./capabilities-sites.ts";
import { toApi, toApiList } from "./rows.ts";
import type { SourcesRuntime } from "./runtime.ts";
import { loadDetailPage } from "./service/detail-page.ts";
import { assignReleaseIdentity } from "./service/identity-admin.ts";
import { applyReleaseFacts } from "./service/release-facts-admin.ts";
import { resolveRelease } from "./service/resolve.ts";
import { refreshReleaseDetail } from "./service/snapshot.ts";
import { liveSearch } from "./service/workbench.ts";
@@ -113,7 +115,7 @@ export function registerSourcesCapabilities(registry: CapabilityRegistry, rt: ()
registry.register(capabilitySpec("release.inspect"), async (input) => {
const r = rt();
if (input.refreshDetail) await refreshReleaseDetail(r, input.sourceReleaseId);
const refreshError = input.refreshDetail ? await refreshReleaseDetail(r, input.sourceReleaseId) : null;
const row = await r.ctx.db
.selectFrom("source_release")
.selectAll()
@@ -128,9 +130,18 @@ export function registerSourcesCapabilities(registry: CapabilityRegistry, rt: ()
.where("media_entity_id", "=", row.media_entity_id)
.execute()
: [];
return { release: toApi(row), inventory: toApiList(inventory) } as never;
return {
release: toApi(row),
inventory: toApiList(inventory),
page: refreshError ? null : await loadDetailPage(r, input.sourceReleaseId),
refreshError,
} as never;
});
registry.register(capabilitySpec("release.apply_facts"), (input, ctx) =>
applyReleaseFacts(rt().ctx, input, ctx.agentRunId),
);
registry.register(capabilitySpec("identity.resolve"), async (input, ctx) => {
const r = rt();
let srId = input.sourceReleaseId ?? null;
apps/server/src/modules/sources-identity/runtime.ts
+12 -9
View File
@@ -28,7 +28,7 @@ import { readBrowserUserAgent, resolveUserAgent, type UserAgentDefaults } from "
export interface RuntimeOverrides {
/** 站点与 RSS 请求使用的 fetch(测试指向 mock) */
fetch?: typeof globalThis.fetch;
fetch?: (input: RequestInfo | URL, init?: RequestInit) => Promise<Response>;
/** TMDB 客户端工厂;返回 null 表示未配置(只做本地召回) */
tmdb?: (() => TmdbClient | null) | undefined;
now?: () => Date;
@@ -39,7 +39,7 @@ export interface RuntimeOverrides {
/** false = 站点限流只用进程内令牌桶(缺省走 PG 跨进程令牌桶 rate_limit_bucket) */
sharedRateLimit?: boolean;
/** 豆瓣 / bgm.tv 请求用的 fetch;null = 不在线查(测试缺省不出外网) */
metaFetch?: typeof globalThis.fetch | null;
metaFetch?: ((input: RequestInfo | URL, init?: RequestInit) => Promise<Response>) | null;
}
interface CachedAdapter {
@@ -78,9 +78,9 @@ export class SourcesRuntime {
* 未配置时回退环境变量 TMDB_API_KEY / TMDB_ACCESS_TOKEN / TMDB_BASE_URL。设置变化(引用/地址)时重建。
*/
/** 豆瓣 / bgm.tv 请求:注入优先;测试进程(VITEST)缺省不出外网;否则按环境变量代理出外网 */
metaFetch(): typeof globalThis.fetch | null {
metaFetch(): ((input: RequestInfo | URL, init?: RequestInit) => Promise<Response>) | null {
if (this.overrides.metaFetch !== undefined) return this.overrides.metaFetch;
return Deno.env.get("VITEST") ? null : externalFetch;
return process.env.VITEST ? null : externalFetch;
}
async tmdb(): Promise<TmdbClient | null> {
@@ -110,7 +110,7 @@ export class SourcesRuntime {
if (!cfg) return null;
let n = 0;
const base = cfg.fetch ?? externalFetch;
const counted: typeof fetch = (input, init) => {
const counted: (input: RequestInfo | URL, init?: RequestInit) => Promise<Response> = (input, init) => {
n++;
return base(input, init);
};
@@ -122,7 +122,7 @@ export class SourcesRuntime {
private async tmdbConfig(): Promise<{ sig: string; cfg: Parameters<typeof createTmdbClient>[0] | null }> {
const meta = await this.ctx.settings.get("metadata").catch(() => null);
const policy = await tmdbCachePolicy(this.ctx);
const env = (k: string) => Deno.env.get(k);
const env = (k: string) => process.env[k];
const ref = meta?.tmdbApiKeySecretRef ?? null;
const baseUrl = meta?.tmdbBaseUrl ?? env("TMDB_BASE_URL");
const sig = JSON.stringify([
@@ -189,7 +189,7 @@ export class SourcesRuntime {
site: Pick<SiteRow, "proxy"> & Partial<Pick<SiteRow, "id">>,
defaults: SitesSettings = this.siteDefaultsCache,
ua?: string | (() => Promise<string>),
): typeof globalThis.fetch {
): (input: RequestInfo | URL, init?: RequestInit) => Promise<Response> {
const base = this.transportFetch(site, defaults);
const egress = this.egressKeyOf(site, defaults);
const siteId = site.id;
@@ -214,7 +214,7 @@ export class SourcesRuntime {
return base(input, { ...init, headers: h, ...(signal ? { signal } : {}) });
},
);
}) as typeof globalThis.fetch;
}) as (input: RequestInfo | URL, init?: RequestInit) => Promise<Response>;
}
/** 站点 HTTP 出口键(direct / proxy:<主机>),与浏览器会话的出口同一口径 */
@@ -223,7 +223,10 @@ export class SourcesRuntime {
}
/** 不加 UA 的底层 fetch(测试注入 / 按站点出口:代理或明确直连) */
private transportFetch(site: Pick<SiteRow, "proxy">, defaults: SitesSettings): typeof globalThis.fetch {
private transportFetch(
site: Pick<SiteRow, "proxy">,
defaults: SitesSettings,
): (input: RequestInfo | URL, init?: RequestInit) => Promise<Response> {
if (this.overrides.fetch) return this.overrides.fetch;
return siteFetchFor(siteEgress(site.proxy, defaults.proxyUrl));
}
apps/server/src/modules/sources-identity/service/image-proxy.ts
+1 -1
View File
@@ -21,7 +21,7 @@ export function imageMime(b: Uint8Array): string | null {
}
export async function fetchPublicImage(
doFetch: typeof fetch,
doFetch: (input: RequestInfo | URL, init?: RequestInit) => Promise<Response>,
url: string,
opts: { maxBytes: number; label: string; referer?: string },
): Promise<ImageFetchResult> {
apps/server/src/modules/sources-identity/service/metadata-prewarm.ts
+2 -2
View File
@@ -26,9 +26,9 @@ export interface PrewarmResult {
}
/** 图片预热用的 fetch:测试注入优先;测试进程没注入就不出外网 */
function imageFetch(rt: SourcesRuntime): typeof globalThis.fetch | null {
function imageFetch(rt: SourcesRuntime): ((input: RequestInfo | URL, init?: RequestInit) => Promise<Response>) | null {
if (rt.overrides.fetch) return rt.overrides.fetch;
return Deno.env.get("VITEST") ? null : externalFetch;
return process.env.VITEST ? null : externalFetch;
}
export async function prewarmMetadata(rt: SourcesRuntime, mediaEntityId: string): Promise<PrewarmResult> {
apps/server/src/modules/sources-identity/service/release-facts-admin.ts
+138
View File
@@ -0,0 +1,138 @@
import type { ReleaseApplyFactsInput, ReleaseStaticFacts } from "@umt/contracts";
import type { z } from "zod";
import { ApiException } from "../../../http/errors.ts";
import { investigationDecisionId } from "../../agent/runner.ts";
import type { ServerContext } from "../../types.ts";
import { contextOf, parsedOf } from "./release-facts.ts";
export function missingReleaseFields(row: Parameters<typeof parsedOf>[0], fields: readonly string[]): string[] {
const p = parsedOf(row);
const c = contextOf(row);
const values: Record<string, unknown> = {
resolution: p.resolution,
codec: p.videoCodec,
group: p.releaseGroup,
source: p.source,
size: p.sizeBytes,
subtitles: p.subtitles,
seeders: c.seeders,
promotion: c.downloadVolumeFactor,
site: c.siteId,
};
return fields.filter((f) => values[f] === undefined || values[f] === null);
}
/** 只填事项正在等待的事实；不接受策略、排名、身份或季集字段。 */
export async function applyReleaseFacts(
ctx: ServerContext,
input: z.infer<typeof ReleaseApplyFactsInput>,
runId?: string,
) {
if (runId) {
const run = await ctx.db
.selectFrom("agent_run")
.select(["dedupe_key", "trigger"])
.where("id", "=", runId)
.executeTakeFirstOrThrow();
if (run.trigger !== "user" && investigationDecisionId(run.dedupe_key) !== input.decisionId)
throw new ApiException("forbidden", "只能补全当前调查事项的资源属性");
}
return ctx.db.transaction().execute(async (trx) => {
const decision = await trx
.selectFrom("decision")
.selectAll()
.where("id", "=", input.decisionId)
.forUpdate()
.executeTakeFirst();
if (
decision?.subject_type !== "source_release" ||
decision.subject_id !== input.sourceReleaseId ||
decision.dedupe_key !== `release_facts:${input.sourceReleaseId}`
)
throw new ApiException("bad_request", "资源属性事项与本次资源不一致");
const row = await trx
.selectFrom("source_release")
.selectAll()
.where("id", "=", input.sourceReleaseId)
.forUpdate()
.executeTakeFirstOrThrow();
const fields = ((decision.evidence ?? {}) as { missing?: string[] }).missing ?? [];
if (decision.status === "resolved") return { resolved: true, remaining: missingReleaseFields(row, fields) };
if (!["open", "investigating", "awaiting_user"].includes(decision.status))
throw new ApiException("precondition_failed", "该资源属性事项已关闭");
const allowed: Record<keyof ReleaseStaticFacts, string> = {
resolution: "resolution",
videoCodec: "codec",
source: "source",
releaseGroup: "group",
};
const missing = new Set(missingReleaseFields(row, fields));
const raw = (row.raw ?? {}) as Record<string, unknown>;
const old = (raw._facts ?? {}) as { values?: ReleaseStaticFacts; evidence?: Record<string, unknown> };
const facts: ReleaseStaticFacts = {};
for (const key of Object.keys(input.facts) as (keyof ReleaseStaticFacts)[]) {
if (missing.has(allowed[key])) Object.assign(facts, { [key]: input.facts[key] });
else if (!fields.includes(allowed[key]) || old.values?.[key] !== input.facts[key])
throw new ApiException("bad_request", `只能补全事项里仍未知的属性：${key}`);
}
const updated = {
...raw,
_facts: {
values: { ...old.values, ...facts },
evidence: {
...old.evidence,
...Object.fromEntries(
Object.keys(facts).map((field) => [
field,
{
text: input.evidence,
url: input.evidenceUrl ?? null,
runId: runId ?? null,
at: new Date().toISOString(),
},
]),
),
},
},
};
await trx
.updateTable("source_release")
.set({ raw: JSON.stringify(updated) })
.where("id", "=", row.id)
.execute();
const remaining = missingReleaseFields({ ...row, raw: updated }, fields);
if (remaining.length === 0) {
await trx
.updateTable("decision")
.set({
status: "resolved",
resolution: JSON.stringify({ factsApplied: true, evidence: input.evidence }),
resolved_at: new Date(),
})
.where("id", "=", decision.id)
.execute();
await ctx.outbox.emit(
trx,
"decision.updated",
{ decisionId: decision.id, status: "resolved" },
{ idempotencyKey: `decision.updated:${decision.id}:facts` },
);
}
if (row.media_entity_id)
await ctx.outbox.emit(
trx,
"release.resolved",
{
sourceReleaseId: row.id,
observationId: null,
mediaEntityId: row.media_entity_id,
contentUnitIds: row.content_unit_id ? [row.content_unit_id] : [],
evidenceLevel: "needs_review",
},
{
idempotencyKey: `release.resolved:${row.id}:facts:${decision.id}:${JSON.stringify(updated._facts.values)}:${JSON.stringify([...remaining].sort())}`,
},
);
return { resolved: remaining.length === 0, remaining };
});
}
apps/server/src/modules/acquisition-organize/services/release-facts.ts → apps/server/src/modules/sources-identity/service/release-facts.ts
+10
View File
@@ -1,3 +1,4 @@
import { ReleaseStaticFacts } from "@umt/contracts";
import type { SourceReleaseRow } from "@umt/db";
import { type EvaluationContext, type ParsedRelease, parseRelease } from "@umt/domain";
import { fromParsedColumns } from "@umt/mapping";
@@ -29,6 +30,15 @@ export function parsedOf(r: SourceReleaseRow): ParsedRelease {
p = { ...rest, season: c.season, episodes: { ...c.episodes }, seasonInferred: false };
}
}
const facts = ReleaseStaticFacts.safeParse((r.raw as { _facts?: { values?: unknown } } | null)?._facts?.values).data;
if (facts)
p = {
...p,
...(!p.resolution && facts.resolution ? { resolution: facts.resolution } : {}),
...(!p.videoCodec && facts.videoCodec ? { videoCodec: facts.videoCodec } : {}),
...(!p.source && facts.source ? { source: facts.source, isEncode: facts.source === "BluRay" } : {}),
...(!p.releaseGroup && facts.releaseGroup ? { releaseGroup: { name: facts.releaseGroup, source: "known" } } : {}),
};
return r.size_bytes !== null ? { ...p, sizeBytes: r.size_bytes } : p;
}
apps/server/src/modules/sources-identity/service/resolve.ts
+2 -20
View File
@@ -20,7 +20,6 @@ import {
toParsedColumns,
toReleaseResolutionStatus,
} from "@umt/mapping";
import { rewriteHook } from "../../../hooks/index.ts";
import { jsonb } from "../rows.ts";
import type { SourcesRuntime } from "../runtime.ts";
import { addSiteAliases, type SiteAliasOutcome } from "./alias-guard.ts";
@@ -484,18 +483,7 @@ export async function resolveRelease(rt: SourcesRuntime, input: ResolveInput): P
mediaEntityId: release.media_entity_id,
};
}
// 链式钩子 name.recognize(ADR-006):可改写解析结果;没有钩子时原样
const parsed = await rewriteHook(
ctx,
"name.recognize",
await parseTitle(ctx.db, release.title, release.subtitle),
() => ({
title: release.title,
subtitle: release.subtitle ?? null,
sourceReleaseId: release.id,
trigger: input.trigger ?? (input.force ? "review" : "observed"),
}),
);
const parsed = await parseTitle(ctx.db, release.title, release.subtitle);
const cols = toParsedColumns(parsed);
// 站点给出的外部 ID(列表 / 详情页的 IMDb、豆瓣、TMDB 链接)作为显式 ID 证据参与打分
const siteIds = siteExplicitIds(release.raw);
@@ -1184,13 +1172,7 @@ export async function identifyParsed(
const { ctx } = rt;
const none = { mediaEntityId: null, contentUnitIds: [] as string[], candidates: [], created: false };
const trigger = opts.trigger ?? "takeover";
// 链式钩子 name.recognize(ADR-006):整理 / 接管按名称识别时同样可改写解析结果
const parsed = await rewriteHook(ctx, "name.recognize", input, () => ({
title: input.raw,
subtitle: null,
sourceReleaseId: null,
trigger,
}));
const parsed = input;
if (!parsed.title || parsed.auxiliaryOnly) return { outcome: "unparseable", reason: "unparseable_title", ...none };
const local = await localCandidates(ctx.db, parsed, null);
const lookup = await lookupReason(rt, parsed, trigger, local.length > 0, null);
apps/server/src/modules/system/heartbeat.ts
+8 -7
View File
@@ -1,8 +1,9 @@
import { hostname } from "node:os";
import type { ProcessKind } from "@umt/contracts";
import { type DB, type Kysely, sql } from "@umt/db";
import { runtimeHostname, runtimeUnrefTimer } from "@umt/runtime";
import type { Logger } from "pino";
import serverDeno from "../../../deno.json" with { type: "json" };
import serverPackage from "../../../package.json" with { type: "json" };
import { createProcessCpuMeter } from "./resources.ts";
// 进程心跳写入(W9-I):每个常驻进程每 15s upsert process_heartbeat 一行 + 追加一条采样。
@@ -19,7 +20,7 @@ export const HEARTBEAT_ROW_RETENTION_MS = 24 * 3600_000;
/** 当前实例判定窗口:最近这段时间内有心跳的 host 视为当前在跑的容器 */
export const CURRENT_HOST_WINDOW_MS = 10 * 60_000;
export const SERVER_VERSION: string = serverDeno.version;
export const SERVER_VERSION: string = serverPackage.version;
/** 事件循环延迟采样器:每 intervalMs 测一次 setInterval 实际触发偏差;window 在每次心跳读取后重置 */
export function createLagSampler(intervalMs = 1000) {
@@ -36,7 +37,7 @@ export function createLagSampler(intervalMs = 1000) {
if (lag > lifetimeMax) lifetimeMax = lag;
}, intervalMs);
try {
Deno.unrefTimer(timer);
runtimeUnrefTimer(timer);
} catch {
// 非 Deno 运行时(vitest 走 deno,理论上不会到这里)
}
@@ -80,7 +81,7 @@ export function instanceIdOf(kind: ProcessKind, host: string, pid: number) {
function memory(): Record<string, number> {
try {
const m = Deno.memoryUsage();
const m = process.memoryUsage();
return { rssBytes: m.rss, heapUsedBytes: m.heapUsed };
} catch {
return {};
@@ -159,7 +160,7 @@ export interface HeartbeatWriter {
/** 起一个周期写入器:立即写一次,之后每 intervalMs 写一次;stop 时把本行标 stopped_at */
export function startHeartbeatWriter(o: HeartbeatWriterOptions): HeartbeatWriter {
const host = o.host ?? safeHostname();
const pid = o.pid ?? Deno.pid;
const pid = o.pid ?? process.pid;
const startedAt = o.startedAt ?? new Date();
const version = o.version ?? SERVER_VERSION;
const instanceId = instanceIdOf(o.kind, host, pid);
@@ -214,7 +215,7 @@ export function startHeartbeatWriter(o: HeartbeatWriterOptions): HeartbeatWriter
void beat();
const timer = setInterval(() => void beat(), intervalMs);
try {
Deno.unrefTimer(timer);
runtimeUnrefTimer(timer);
} catch {
// ignore
}
@@ -237,7 +238,7 @@ export function startHeartbeatWriter(o: HeartbeatWriterOptions): HeartbeatWriter
export function safeHostname(): string {
try {
return Deno.hostname();
return runtimeHostname();
} catch {
return hostname();
}
apps/server/src/modules/system/index.ts
+1 -1
View File
@@ -121,7 +121,7 @@ export function createSystemModule(opts: SystemModuleOptions = {}): ServerModule
if (opts.resourceSampler) sampler = opts.resourceSampler;
else if (opts.resourceSampler !== false && !sampler) sampler = createResourceSampler();
const selfKind = ctx.role === "all" ? "all" : ctx.role === "api" ? "api" : null;
const selfId = selfKind ? instanceIdOf(selfKind, safeHostname(), Deno.pid) : undefined;
const selfId = selfKind ? instanceIdOf(selfKind, safeHostname(), process.pid) : undefined;
const paths = (): StoragePaths => {
if (opts.storagePaths) return opts.storagePaths;
const p = dataPaths(ctx.config);
apps/server/src/modules/system/resources.ts
+2 -1
View File
@@ -1,6 +1,7 @@
import { availableParallelism, cpus } from "node:os";
import process from "node:process";
import type { ResourceSample, ResourceScope, SystemResources } from "@umt/contracts";
import { runtimeUnrefTimer } from "@umt/runtime";
import { readProcText } from "../../proc-read.ts";
// 资源监控(界面第二轮 V 线,参考 MP 仪表盘 CPU / 内存 / 网络组件):api 进程每 5 秒读一次 /proc 与 cgroup v2,
@@ -366,7 +367,7 @@ export function createResourceSampler(
void tick();
timer = setInterval(() => void tick(), intervalMs);
try {
Deno.unrefTimer(timer);
runtimeUnrefTimer(timer);
} catch {
// ignore
}
apps/server/src/modules/types.ts
+5 -14
View File
@@ -12,13 +12,11 @@ import type { DB, Kysely, Transaction } from "@umt/db";
import type { OutboxRelay, OutboxRouteFn, TaskEngine, TaskRegistry } from "@umt/tasks";
import type { Logger } from "pino";
import type { Config } from "../config.ts";
import type { HookRegistrar, HookRegistry } from "../hooks/registry.ts";
import type { AuthEnv } from "../http/auth.ts";
import type { EventBusPort } from "../http/sse.ts";
// 服务端模块骨架(W1 审查 B6;任务书 §5)。
// 每个 W2 子任务是一个 ServerModule,只改 modules/<自己的目录>/ 与 contracts/db 中本域的文件。
// 跨模块只通过 DB 表 + outbox_event(事件类型在 contracts/events.ts)+ contracts 类型耦合,不互相 import。
// 服务端模块装配。同步协作使用明确的业务服务导出,事后通知走 outbox;
// 路由和能力处理器依赖服务,业务服务不反向依赖 HTTP 路由或 Agent 会话。
export type ServerRole = "api" | "worker" | "worker:agent" | "all";
@@ -49,8 +47,8 @@ export interface OutboxPort extends OutboxWriter {
}
/**
* 凭证存取(secret 表 + SECRET_MASTER_KEY 加密)。W2-A 在 modules/auth-settings/provided.ts 实现,
* 当前占位实现一律抛"未实现"。其它模块只存/传 secret_ref,需要明文时现场 resolve,不缓存、不打日志。
* 凭证存取(secret 表 + SECRET_MASTER_KEY 加密),由 modules/auth-settings/provided.ts 实现。
* 其它模块只存/传 secret_ref,需要明文时现场 resolve,不缓存、不打日志。
*/
export interface SecretStore {
resolve(secretRef: string): Promise<string>;
@@ -80,11 +78,6 @@ export interface ServerContext {
settings: SettingsReader;
/** 能力注册中心(Web/Pi/CLI/MCP 共用一条 invoke 路径) */
capabilities: CapabilityRegistry;
/**
* 链式钩子(ADR-006):bootstrap 装配,模块在 registerHooks 里注册;业务节点经 hooks/index.ts 的
* rewriteHook / interceptHook 调用。测试里手搭的上下文可以没有(等同于没有注册)。
*/
hooks?: HookRegistry;
}
export interface ServerModule {
@@ -94,7 +87,7 @@ export interface ServerModule {
registerRoutes(app: ModuleApp, ctx: ServerContext): void;
/**
* 所有角色:注册持久任务处理器(api 角色也要注册,enqueue 时按定义校验输入)。
* 同名任务只能由一个模块注册;没人注册的 §3.1-8 任务由 @umt/tasks 的空壳兜底。
* 同名任务只能由一个模块注册;未注册的任务明确报错。
*/
registerTasks(registry: TaskRegistry, ctx: ServerContext): void;
/**
@@ -104,8 +97,6 @@ export interface ServerModule {
outboxRoutes?(ctx: ServerContext): Record<string, OutboxRouteFn>;
/** 外围功能的说明(功能清单只给智能体看,不做开关);取 contracts 的 featureManifest(id) */
manifest?: FeatureManifest;
/** 所有角色:注册链式钩子(同步、按注册顺序、可改写或拦截;见 hooks/registry.ts) */
registerHooks?(hooks: HookRegistrar, ctx: ServerContext): void;
/** 所有角色:用 contracts 的 capabilitySpec(id) 注册能力处理器 */
registerCapabilities?(registry: CapabilityRegistry, ctx: ServerContext): void;
/** worker / all 角色:启动轮询器、定时器等后台循环(引擎与中继已启动) */
apps/server/src/modules/websearch/service.ts
+1 -1
View File
@@ -28,7 +28,7 @@ import type { ServerContext } from "../types.ts";
export interface WebSearchModuleOptions {
/** 搜索与网页读取用的 fetch(测试注入 mock) */
fetch?: typeof fetch;
fetch?: (input: RequestInfo | URL, init?: RequestInit) => Promise<Response>;
/** Brave 基址(测试指向 mock) */
braveBaseUrl?: string;
/** DNS 解析(测试注入);缺省系统解析 */
apps/server/src/ops.ts
+4 -3
View File
@@ -1,6 +1,7 @@
import { dirname } from "node:path";
import type { DB, Kysely } from "@umt/db";
import { sql } from "@umt/db";
import { runtimeMkdir, runtimeRename, runtimeWriteTextFile } from "@umt/runtime";
// 运维一次性子命令(备份/恢复脚本调用):映射清单导出、恢复后打开只读对账。
// 映射清单是"路径不携带完整身份"前提下的关键资产(需求 §8.5):数据库之外再留一份可读 JSON,
@@ -112,10 +113,10 @@ export async function exportMappings(db: Kysely<DB>, appVersion: string): Promis
/** 写 JSON(先写临时文件再改名,避免半截文件) */
export async function writeMappingExport(path: string, data: MappingExport): Promise<void> {
await Deno.mkdir(dirname(path), { recursive: true });
await runtimeMkdir(dirname(path), { recursive: true });
const tmp = `${path}.partial`;
await Deno.writeTextFile(tmp, `${JSON.stringify(data, null, 2)}\n`);
await Deno.rename(tmp, path);
await runtimeWriteTextFile(tmp, `${JSON.stringify(data, null, 2)}\n`);
await runtimeRename(tmp, path);
}
/** 两份清单逐表比行数,返回不一致的表 */
apps/server/src/preflight.ts
+14 -9
View File
@@ -1,6 +1,7 @@
import { join } from "node:path";
import type { DB, Kysely } from "@umt/db";
import { sql } from "@umt/db";
import { runtimeMkdir, runtimeReadDir, runtimeRemove, runtimeStat, runtimeWriteTextFile } from "@umt/runtime";
import type { Config } from "./config.ts";
export interface PreflightProblem {
@@ -19,7 +20,11 @@ export interface PreflightDeps {
const PLACEHOLDER = /change[-_]?me|replace[-_]?me|dev[-_]?only|example/i;
/** 启动自检:返回问题列表,空数组表示通过。不打印变量值。 */
export async function runPreflight({ config, db, uid = Deno.uid() }: PreflightDeps): Promise<PreflightProblem[]> {
export async function runPreflight({
config,
db,
uid = process.getuid?.(),
}: PreflightDeps): Promise<PreflightProblem[]> {
const problems: PreflightProblem[] = [];
const prod = config.NODE_ENV === "production";
@@ -36,7 +41,7 @@ export async function runPreflight({ config, db, uid = Deno.uid() }: PreflightDe
}
if (config.WEB_ROOT) {
try {
await Deno.stat(join(config.WEB_ROOT, "index.html"));
await runtimeStat(join(config.WEB_ROOT, "index.html"));
} catch {
problems.push({ check: "web", message: `WEB_ROOT 下没有 index.html: ${config.WEB_ROOT}` });
}
@@ -66,9 +71,9 @@ const errText = (e: unknown) => (e as { code?: string }).code ?? (e as Error).me
/** 在目录里写一个探针文件再删掉;失败抛错 */
async function probeWrite(dir: string): Promise<void> {
const probe = join(dir, `.umt-write-probe-${Deno.pid}-${crypto.randomUUID().slice(0, 8)}`);
await Deno.writeTextFile(probe, "probe\n");
await Deno.remove(probe);
const probe = join(dir, `.umt-write-probe-${process.pid}-${crypto.randomUUID().slice(0, 8)}`);
await runtimeWriteTextFile(probe, "probe\n");
await runtimeRemove(probe);
}
/** 读出已启用的库根与下载器路径映射在本进程视角下的路径 */
@@ -109,7 +114,7 @@ export async function checkStorage(opts: {
if (!dirs.includes(opts.sessionDir)) dirs.push(opts.sessionDir);
for (const dir of dirs) {
try {
await Deno.mkdir(dir, { recursive: true });
await runtimeMkdir(dir, { recursive: true });
await probeWrite(dir);
} catch (e) {
// copy-allow: 括号里是系统错误原文
@@ -118,9 +123,9 @@ export async function checkStorage(opts: {
}
for (const m of opts.mounts) {
try {
const st = await Deno.stat(m.path);
const st = await runtimeStat(m.path);
if (!st.isDirectory) throw Object.assign(new Error("不是目录"), { code: "ENOTDIR" });
for await (const _ of Deno.readDir(m.path)) break;
for await (const _ of await runtimeReadDir(m.path)) break;
} catch (e) {
warnings.push({ ...m, reason: `不可读：${errText(e)}` });
continue;
@@ -146,7 +151,7 @@ export async function auditStorageWarnings(db: Kysely<DB>, warnings: StorageWarn
action: "preflight.path_unavailable",
target_type: w.kind,
target_id: w.id,
payload: JSON.stringify({ name: w.name, path: w.path, reason: w.reason, uid: Deno.uid() }),
payload: JSON.stringify({ name: w.name, path: w.path, reason: w.reason, uid: process.getuid?.() }),
})),
)
.execute();
apps/server/src/proc-read.ts
+4 -24
View File
@@ -1,29 +1,9 @@
// 读 /proc 与 /sys 的伪文件。
// Deno 2 把这两处当特权路径:不带 --allow-all 时 Deno.readTextFile 一律 NotCapable(--allow-read 及按路径授权都不行)。
// 开发 / 测试用 -A 能直接读;编译产物按最小权限运行,只能借 `cat` 子进程读(编译与 start 任务的 --allow-run 已含 cat)。
// 第一次遇到 NotCapable 后记住,后续直接走 cat。
let viaCat = false;
async function catText(path: string): Promise<string | null> {
import { readFile } from "node:fs/promises";
/** Linux 资源监控和子进程 RSS;权限不足或进程已退出时返回 null。 */
export async function readProcText(path: string): Promise<string | null> {
try {
const o = await new Deno.Command("cat", { args: [path], stdout: "piped", stderr: "null" }).output();
return o.success ? new TextDecoder().decode(o.stdout) : null;
return await readFile(path, "utf8");
} catch {
return null;
}
}
/** 读不到(不存在、无权限、cat 不可用)返回 null */
export async function readProcText(path: string): Promise<string | null> {
if (viaCat) return await catText(path);
try {
return await Deno.readTextFile(path);
} catch (e) {
if (e instanceof Deno.errors.NotCapable) {
viaCat = true;
return await catText(path);
}
return null;
}
}
apps/server/src/sandbox/archives.ts
+12 -2
View File
@@ -1,8 +1,10 @@
// 压缩包解压(沙箱内):zip 用 fflate,rar 用 node-unrar-js(WASM),7z 用 7z-wasm。
// WASM 库要读自己包里的 .wasm 文件:prepareArchives() 在撤销权限前预热(载入内存),之后只在内存里解压。
import * as SevenZipModule from "7z-wasm";
import sevenWasmPath from "7z-wasm/7zz.wasm" with { type: "file" };
import { unzipSync } from "fflate";
import { createExtractorFromData } from "node-unrar-js";
import unrarWasmPath from "node-unrar-js/dist/js/unrar.wasm" with { type: "file" };
/** 用到的 7z-wasm 接口(Emscripten 实例:内存文件系统 + callMain) */
interface Seven {
@@ -17,6 +19,7 @@ interface Seven {
}
// 7z-wasm 是 CJS 包:tsc 与 deno check 对默认导出的理解不同,运行时取 default 或模块本身
const SevenZip = ((SevenZipModule as unknown as { default?: unknown }).default ?? SevenZipModule) as unknown as (opts: {
wasmBinary: Uint8Array;
print: (s: string) => void;
printErr: (s: string) => void;
}) => Promise<Seven>;
@@ -27,11 +30,18 @@ export async function prepareArchives(): Promise<void> {
if (warmed) return;
warmed = true;
try {
await createExtractorFromData({ data: new Uint8Array([0x52, 0x61, 0x72, 0x21]).buffer });
await createExtractorFromData({
data: new Uint8Array([0x52, 0x61, 0x72, 0x21]).buffer,
wasmBinary: await Bun.file(unrarWasmPath).arrayBuffer(),
});
} catch {
// 只为载入 WASM,内容无效报错是预期的
}
seven = await SevenZip({ print: () => {}, printErr: () => {} });
seven = await SevenZip({
wasmBinary: new Uint8Array(await Bun.file(sevenWasmPath).arrayBuffer()),
print: () => {},
printErr: () => {},
});
}
export type AcceptEntry = (name: string, size: number) => boolean;
apps/server/src/sandbox/client.ts
+52 -99
View File
@@ -1,10 +1,11 @@
// 沙箱调用(docs/adr/ADR-005-sandbox.md):每次一个子进程,权限全部撤销;超时 SIGKILL,子进程实际内存(RSS)超限 SIGKILL。
// 起子进程方式同 fsproxy:deno CLI → `deno run worker.ts`(开发 / 测试);编译二进制 → `<自身> --role sandbox`。
// 起不来(权限 / 运行时不支持)时退回进程内执行并记日志:仍有超时,但卡死的同步代码无法收回(只在本机调试时出现)。
// 解析沙箱:预加载后安装 Linux seccomp(ADR-007),父进程实施超时与 RSS 强杀。
// Bun CLI 启动 worker.ts;编译产物启动自身 --role sandbox。建立隔离失败直接拒绝。
import { basename } from "node:path";
import { fileURLToPath } from "node:url";
import { WorkerConnection, WorkerTransportError } from "@umt/runtime/worker";
import { readProcText } from "../proc-read.ts";
import { prepareJobs, SANDBOX_JOBS, type SandboxInput, type SandboxJobName, type SandboxOutput } from "./jobs.ts";
import type { SandboxInput, SandboxJobName, SandboxOutput } from "./jobs.ts";
import { SANDBOX_MESSAGE_BYTES } from "./limits.ts";
export type SandboxErrorKind = "timeout" | "memory" | "crash" | "job" | "spawn";
@@ -27,37 +28,23 @@ export interface SandboxOptions {
}
const WORKER_PATH = fileURLToPath(new URL("./worker.ts", import.meta.url));
let inProcessReason: string | null = null;
let spawnFailureReason: string | null = null;
const stats = { spawned: 0, timeouts: 0, memoryKills: 0, crashes: 0 };
export function sandboxStats() {
return { ...stats, inProcessReason };
return { ...stats, spawnFailureReason };
}
function command(memoryMb: number): { cmd: string; args: string[] } {
const exe = Deno.execPath();
if (/^deno(\.exe)?$/.test(basename(exe)))
return {
cmd: exe,
args: [
"run",
"-A",
"--quiet",
"--no-prompt", // V8 堆上限只作兜底(设为两倍):主要靠下面的 RSS 看门狗,编译产物不认 V8 参数,两种模式一致
`--v8-flags=--max-old-space-size=${memoryMb * 2}`,
WORKER_PATH,
],
};
return { cmd: exe, args: ["--role", "sandbox"] };
function command(): { cmd: string; args: string[] } {
const exe = process.execPath;
return /^bun(\.exe)?$/.test(basename(exe))
? { cmd: exe, args: ["--no-env-file", "--config=/dev/null", WORKER_PATH] }
: { cmd: exe, args: ["--role", "sandbox"] };
}
/**
* 子进程环境:清空,只留 PATH。编译产物启动时要按 PATH 解析 --allow-run 里的命令名(ffprobe、cat…),
* 没有 PATH 会在进程入口报「no path to search」直接退出,所有沙箱任务都失败(2026-10 Bangumi 热门、字幕验证码即此)。
* 子进程随后撤销全部权限,PATH 不放大能力。
*/
/** 清空继承环境,只保留 PATH;CLI 禁用 dotenv,编译产物禁用自动配置加载。 */
export function sandboxEnv(): Record<string, string> {
return { PATH: Deno.env.get("PATH") || "/usr/local/bin:/usr/bin:/bin" };
return { PATH: process.env.PATH || "/usr/local/bin:/usr/bin:/bin" };
}
/** 子进程实际内存(KB);读不到返回 null */
@@ -75,95 +62,61 @@ export async function runSandbox<J extends SandboxJobName>(
const timeoutMs = opts.timeoutMs ?? 20_000;
const memoryMb = opts.memoryMb ?? 512;
opts.signal?.throwIfAborted();
if (inProcessReason) return runInProcess(job, input, timeoutMs) as Promise<SandboxOutput<J>>;
const { cmd, args } = command(memoryMb);
let child: Deno.ChildProcess;
const { cmd, args } = command();
let connection: WorkerConnection;
try {
child = new Deno.Command(cmd, {
args,
stdin: "piped",
stdout: "piped",
stderr: "piped",
env: sandboxEnv(),
clearEnv: true,
}).spawn();
} catch (e) {
inProcessReason = (e as Error).message;
console.warn(`[sandbox] 起不来子进程，退回进程内执行：${inProcessReason}`);
return runInProcess(job, input, timeoutMs) as Promise<SandboxOutput<J>>;
connection = new WorkerConnection({ cmd, args }, { values: sandboxEnv(), clear: true });
} catch (error) {
spawnFailureReason = (error as Error).message;
throw new SandboxError("spawn", `沙箱子进程无法启动：${spawnFailureReason}`);
}
stats.spawned += 1;
let killedFor: SandboxErrorKind | null = null;
const kill = (why: SandboxErrorKind) => {
const kill = (kind: SandboxErrorKind, message: string) => {
if (killedFor) return;
killedFor = why;
try {
child.kill("SIGKILL");
} catch {
// 已退出
}
killedFor = kind;
connection.terminate(new SandboxError(kind, message));
};
const timer = setTimeout(() => kill("timeout"), timeoutMs);
const timer = setTimeout(() => kill("timeout", `沙箱任务 ${job} 超过 ${timeoutMs} ms 未完成，已强制结束`), timeoutMs);
const watch = setInterval(async () => {
const kb = await rssKb(child.pid);
if (kb !== null && kb > memoryMb * 1024) kill("memory");
const kb = await rssKb(connection.pid);
if (kb !== null && kb > memoryMb * 1024) kill("memory", `沙箱任务 ${job} 内存超过 ${memoryMb} MB，已强制结束`);
}, 100);
const onAbort = () => kill("crash");
const onAbort = () => kill("crash", "沙箱任务已取消");
opts.signal?.addEventListener("abort", onAbort, { once: true });
try {
const w = child.stdin.getWriter();
await w.write(new TextEncoder().encode(JSON.stringify({ job, input }))).catch(() => undefined);
await w.close().catch(() => undefined);
const [status, stdout, stderr] = await Promise.all([
child.status,
new Response(child.stdout).text(),
new Response(child.stderr).text(),
]);
if (killedFor === "timeout") {
stats.timeouts += 1;
throw new SandboxError("timeout", `沙箱任务 ${job} 超过 ${timeoutMs} ms 未完成，已强制结束`);
const pending = connection.request(
{ id: "1", task: job, input },
{
maxRequestBytes: SANDBOX_MESSAGE_BYTES,
maxResponseBytes: SANDBOX_MESSAGE_BYTES,
},
);
if (opts.signal?.aborted) onAbort();
const reply = await pending;
if (reply.type === "error") throw new SandboxError("job", reply.error.message);
return reply.result as SandboxOutput<J>;
} catch (error) {
opts.signal?.throwIfAborted();
if (error instanceof SandboxError) {
if (error.kind === "timeout") stats.timeouts += 1;
if (error.kind === "memory") stats.memoryKills += 1;
throw error;
}
if (killedFor === "memory" || /out of memory|heap limit/i.test(stderr)) {
if (error instanceof WorkerTransportError && error.kind === "spawn") {
spawnFailureReason = (error.cause as Error)?.message ?? error.message;
throw new SandboxError("spawn", `沙箱子进程无法启动：${spawnFailureReason}`);
}
if (error instanceof WorkerTransportError && error.kind === "size") throw new SandboxError("job", error.message);
if (/out of memory|heap limit/i.test((error as Error).message)) {
stats.memoryKills += 1;
throw new SandboxError("memory", `沙箱任务 ${job} 内存超过 ${memoryMb} MB，已强制结束`);
}
opts.signal?.throwIfAborted();
const line = stdout.trim().split("\n").pop() ?? "";
let msg: { ok: boolean; result?: unknown; error?: string };
try {
msg = JSON.parse(line);
} catch {
stats.crashes += 1;
throw new SandboxError("crash", `沙箱任务 ${job} 异常退出（code ${status.code}）：${stderr.trim().slice(-300)}`);
}
if (!msg.ok) throw new SandboxError("job", msg.error ?? "沙箱任务失败");
return msg.result as SandboxOutput<J>;
stats.crashes += 1;
throw new SandboxError("crash", (error as Error).message);
} finally {
clearTimeout(timer);
clearInterval(watch);
opts.signal?.removeEventListener("abort", onAbort);
}
}
async function runInProcess<J extends SandboxJobName>(
job: J,
input: SandboxInput<J>,
timeoutMs: number,
): Promise<unknown> {
await prepareJobs();
const fn = SANDBOX_JOBS[job] as (i: SandboxInput<J>) => unknown;
let timer: ReturnType<typeof setTimeout> | undefined;
try {
return await Promise.race([
Promise.resolve().then(() => fn(input)),
new Promise<never>((_, reject) => {
timer = setTimeout(() => reject(new SandboxError("timeout", `沙箱任务 ${job} 超时（进程内）`)), timeoutMs);
}),
]);
} catch (e) {
if (e instanceof SandboxError) throw e;
throw new SandboxError("job", `${(e as Error).name}: ${(e as Error).message}`);
} finally {
clearTimeout(timer);
}
}
apps/server/src/sandbox/jobs.ts
+22 -2
View File
@@ -1,8 +1,12 @@
import { execFileSync } from "node:child_process";
import { readFile, writeFile } from "node:fs/promises";
import { connect } from "node:net";
// 沙箱任务表(docs/adr/ADR-005-sandbox.md):只放「外部不可信字节 → 结构化数据」的纯函数。
// 子进程加载完本表后撤销全部权限(文件 / 网络 / 环境变量 / 子进程),所以任务里不能有任何 I/O;
// 输入输出必须能 JSON 化(二进制用 base64)。新增任务:在这里加一项,调用方用 runSandbox(名字, 输入)。
import { parseBangumiBrowser, parseTorrent } from "@umt/adapters";
import { decodeSubtitle, detectSubtitleLanguage, subtitleEpisode, subtitleExtOf } from "@umt/domain";
import { runtimeReadTextFileSync } from "@umt/runtime";
import { buildPreview } from "../modules/file-manager/preview-build.ts";
import { PAGE_PARSERS, type PageKind } from "../modules/subtitles/online/parse.ts";
import { archiveKindOf, extractArchive, prepareArchives } from "./archives.ts";
@@ -96,14 +100,30 @@ function torrentParse(input: { base64: string }) {
}
/** 测试用:按指令卡死 / 吃内存 / 抛错,验证父进程的超时强杀与内存上限 */
function selfTest(input: { mode: "ok" | "hang" | "memory" | "throw" | "io" }) {
async function selfTest(input: {
mode: "ok" | "hang" | "memory" | "throw" | "io" | "async-io" | "write" | "net" | "exec" | "env";
}) {
if (input.mode === "hang") for (;;) {}
if (input.mode === "memory") {
const keep: Uint8Array[] = [];
for (;;) keep.push(new Uint8Array(16 * 1024 * 1024).fill(1));
}
if (input.mode === "throw") throw new Error("任务自己抛的错");
if (input.mode === "io") return { read: Deno.readTextFileSync("/etc/hostname") };
if (input.mode === "env") return { keys: Object.keys(process.env) };
if (input.mode === "async-io") return { read: await readFile("/etc/hostname", "utf8") };
if (input.mode === "write") return await writeFile("/tmp/umt-sandbox-denied", "denied");
if (input.mode === "net") {
return await new Promise((resolve, reject) => {
const socket = connect({ host: "127.0.0.1", port: 9 });
socket.once("error", reject);
socket.once("connect", () => {
socket.destroy();
resolve({ connected: true });
});
});
}
if (input.mode === "exec") return { stdout: execFileSync("/bin/echo", ["escaped"], { encoding: "utf8" }) };
if (input.mode === "io") return { read: runtimeReadTextFileSync("/etc/hostname") };
return { ok: true };
}
apps/server/src/sandbox/limits.ts
+2
View File
@@ -0,0 +1,2 @@
/** 单条请求/响应包括 base64 的上限；任务自己的解压条目和文件大小限制另计。 */
export const SANDBOX_MESSAGE_BYTES = 128 * 1024 * 1024;
apps/server/src/sandbox/native/policy.c
+107
View File
@@ -0,0 +1,107 @@
#define _GNU_SOURCE
#include <node_api.h>
#include <errno.h>
#include <stddef.h>
#include <stdint.h>
#include <fcntl.h>
#include <sys/socket.h>
#include <linux/audit.h>
#include <linux/filter.h>
#include <linux/seccomp.h>
#include <sched.h>
#include <sys/prctl.h>
#include <sys/syscall.h>
#include <unistd.h>
#if defined(__x86_64__)
#define SANDBOX_ARCH AUDIT_ARCH_X86_64
#elif defined(__aarch64__)
#define SANDBOX_ARCH AUDIT_ARCH_AARCH64
#else
#error Unsupported sandbox architecture
#endif
#define ALLOW(n) BPF_JUMP(BPF_JMP|BPF_JEQ|BPF_K, __NR_##n, 0, 1), BPF_STMT(BPF_RET|BPF_K, SECCOMP_RET_ALLOW)
/* Default-deny policy: no path-based IO, sockets, exec, ptrace, mounts,
* io_uring, or child processes. Apply to every existing runtime thread (TSYNC).
* New JS engine threads inherit the policy; only CLONE_THREAD is allowed. */
static napi_value lockdown(napi_env env, napi_callback_info info) {
struct sock_filter filter[] = {
BPF_STMT(BPF_LD|BPF_W|BPF_ABS, offsetof(struct seccomp_data, arch)),
BPF_JUMP(BPF_JMP|BPF_JEQ|BPF_K, SANDBOX_ARCH, 1, 0),
BPF_STMT(BPF_RET|BPF_K, SECCOMP_RET_KILL_PROCESS),
BPF_STMT(BPF_LD|BPF_W|BPF_ABS, offsetof(struct seccomp_data, nr)),
ALLOW(read), ALLOW(write), ALLOW(readv), ALLOW(writev), ALLOW(close),
ALLOW(fstat), ALLOW(lseek), ALLOW(fcntl),
ALLOW(mmap), ALLOW(mprotect), ALLOW(munmap), ALLOW(mremap), ALLOW(brk), ALLOW(madvise),
ALLOW(rt_sigaction), ALLOW(rt_sigprocmask), ALLOW(rt_sigreturn), ALLOW(sigaltstack),
ALLOW(futex), ALLOW(sched_yield), ALLOW(sched_getaffinity),
ALLOW(clock_gettime), ALLOW(clock_nanosleep), ALLOW(nanosleep), ALLOW(gettimeofday),
ALLOW(getpid), ALLOW(gettid), ALLOW(getuid), ALLOW(geteuid), ALLOW(getgid), ALLOW(getegid),
ALLOW(getrandom), ALLOW(getrusage), ALLOW(prlimit64),
ALLOW(epoll_create1), ALLOW(epoll_ctl), ALLOW(epoll_pwait),
#ifdef __NR_epoll_wait
ALLOW(epoll_wait),
#endif
#ifdef __NR_epoll_pwait2
ALLOW(epoll_pwait2),
#endif
ALLOW(eventfd2), ALLOW(pipe2), ALLOW(set_robust_list), ALLOW(rseq),
#ifdef __NR_arch_prctl
ALLOW(arch_prctl),
#endif
ALLOW(exit), ALLOW(exit_group),
/* clone3 cannot be safely inspected through its pointer argument. ENOSYS
* asks libc to use clone, whose thread flag we can inspect. */
#ifdef __NR_clone3
BPF_JUMP(BPF_JMP|BPF_JEQ|BPF_K, __NR_clone3, 0, 1),
BPF_STMT(BPF_RET|BPF_K, SECCOMP_RET_ERRNO|ENOSYS),
#endif
BPF_JUMP(BPF_JMP|BPF_JEQ|BPF_K, __NR_clone, 0, 3),
BPF_STMT(BPF_LD|BPF_W|BPF_ABS, offsetof(struct seccomp_data, args[0])),
BPF_JUMP(BPF_JMP|BPF_JSET|BPF_K, CLONE_THREAD, 0, 1),
BPF_STMT(BPF_RET|BPF_K, SECCOMP_RET_ALLOW),
BPF_STMT(BPF_RET|BPF_K, SECCOMP_RET_ERRNO|EPERM),
};
struct sock_fprog program = { .len = sizeof(filter)/sizeof(filter[0]), .filter = filter };
if (prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0) != 0 ||
syscall(SYS_seccomp, SECCOMP_SET_MODE_FILTER, SECCOMP_FILTER_FLAG_TSYNC, &program) != 0) {
napi_throw_error(env, "SANDBOX_POLICY", "Cannot install sandbox seccomp policy");
return NULL;
}
/* Verify the installed kernel policy directly. Compiled Bun's network
* wrappers can normalize EPERM to ECONNREFUSED, so JS error strings alone
* cannot prove that the underlying syscall was blocked. */
errno = 0;
long fd = syscall(SYS_socket, AF_INET, SOCK_STREAM, 0);
if (fd >= 0 || errno != EPERM) {
if (fd >= 0) close((int)fd);
napi_throw_error(env, "SANDBOX_POLICY", "Sandbox did not deny socket creation");
return NULL;
}
errno = 0;
fd = syscall(SYS_openat, AT_FDCWD, "/__umt_sandbox_policy_probe__", O_RDONLY, 0);
if (fd >= 0 || errno != EPERM) {
if (fd >= 0) close((int)fd);
napi_throw_error(env, "SANDBOX_POLICY", "Sandbox did not deny file access");
return NULL;
}
errno = 0;
char *argv[] = { "/__umt_sandbox_policy_probe__", NULL };
char *envp[] = { NULL };
syscall(SYS_execve, argv[0], argv, envp);
if (errno != EPERM) {
napi_throw_error(env, "SANDBOX_POLICY", "Sandbox did not deny execve");
return NULL;
}
napi_value out;
napi_get_undefined(env, &out);
return out;
}
static napi_value init(napi_env env, napi_value exports) {
napi_value fn;
napi_create_function(env, "lockdown", NAPI_AUTO_LENGTH, lockdown, NULL, &fn);
napi_set_named_property(env, exports, "lockdown", fn);
return exports;
}
NAPI_MODULE(NODE_GYP_MODULE_NAME, init)
apps/server/src/sandbox/policy.ts
+14
View File
@@ -0,0 +1,14 @@
import { createRequire } from "node:module";
import { dirname, resolve } from "node:path";
import { fileURLToPath } from "node:url";
/** Load before lockdown; compiled deployments install the policy next to the binary. */
export function installSandboxPolicy(): void {
if (process.platform !== "linux") throw new Error("解析沙箱需要 Linux seccomp");
const compiled = !/^bun(?:\.exe)?$/.test(process.execPath.split("/").at(-1) ?? "");
const path = compiled
? resolve(dirname(process.execPath), "sandbox-policy.node")
: fileURLToPath(new URL("../../../../dist/sandbox-policy.node", import.meta.url));
const policy = createRequire(import.meta.url)(path) as { lockdown(): void };
policy.lockdown();
}
apps/server/src/sandbox/worker.ts
+22 -24
View File
@@ -1,33 +1,31 @@
// 沙箱子进程入口:先加载任务表,再撤销全部权限,然后从 stdin 读一条 {job, input},把 {ok, result | error} 写到 stdout。
// 开发 / 测试由 deno run 直接跑本文件;编译二进制以 `<自身> --role sandbox` 拉起(main.ts 在加载其余模块前分派)。
import { decodeWorkerRequest, readWorkerRequestText, writeWorkerMessage } from "@umt/runtime/worker";
import { prepareJobs, SANDBOX_JOBS } from "./jobs.ts";
import { SANDBOX_MESSAGE_BYTES } from "./limits.ts";
import { installSandboxPolicy } from "./policy.ts";
const PERMISSIONS = ["read", "write", "net", "env", "run", "ffi", "sys"] as const;
async function writeAll(bytes: Uint8Array) {
let off = 0;
while (off < bytes.length) off += await Deno.stdout.write(bytes.subarray(off));
}
// 预加载可信任务表和 WASM,有界读取 stdin,安装策略后才 JSON.parse 与处理输入。
export async function sandboxWorkerMain(): Promise<void> {
for (const key of Object.keys(process.env)) if (key !== "PATH") delete process.env[key];
await prepareJobs();
for (const name of PERMISSIONS) {
try {
Deno.permissions.revokeSync({ name } as Deno.PermissionDescriptor);
} catch {
// 运行时不认识的权限名:跳过
}
}
let out: unknown;
const requestText = await readWorkerRequestText(SANDBOX_MESSAGE_BYTES);
installSandboxPolicy();
const req = decodeWorkerRequest(requestText);
try {
const req = JSON.parse(await new Response(Deno.stdin.readable).text()) as { job: string; input: unknown };
const fn = (SANDBOX_JOBS as Record<string, (i: unknown) => unknown>)[req.job];
if (!fn) throw new Error(`没有这个沙箱任务：${req.job}`);
out = { ok: true, result: await fn(req.input) };
} catch (e) {
out = { ok: false, error: `${(e as Error).name}: ${(e as Error).message}`.slice(0, 2000) };
const fn = (SANDBOX_JOBS as Record<string, (i: unknown) => unknown>)[req.task];
if (!Object.hasOwn(SANDBOX_JOBS, req.task) || typeof fn !== "function")
throw new Error(`没有这个沙箱任务：${req.task}`);
const result = await fn(req.input);
writeWorkerMessage({ id: req.id, type: "result", result }, SANDBOX_MESSAGE_BYTES);
} catch (error) {
writeWorkerMessage(
{
id: req.id,
type: "error",
error: { code: "JOB", message: `${(error as Error).name}: ${(error as Error).message}`.slice(0, 2000) },
},
SANDBOX_MESSAGE_BYTES,
);
}
await writeAll(new TextEncoder().encode(`${JSON.stringify(out)}\n`));
}
if (import.meta.main) await sandboxWorkerMain();
apps/server/src/storage/local-fs.ts
+2 -1
View File
@@ -5,7 +5,8 @@ import { assertLocalPaths } from "@umt/adapters";
// 参数是 Alist 虚拟路径(/@alist/…)就抛 FsError("REMOTE_PATH_ON_LOCAL_FS"),不在本机建出 /@alist 目录或写进文件。
// 读类函数(stat / readdir / readFile …)原样转出:对虚拟路径只会 ENOENT。
export * from "node:fs/promises";
// 显式导出避免 Bun 对内置模块 export * 的编译问题。
export const { stat, lstat, readdir, readFile, realpath } = fsp;
type P = Parameters<typeof fsp.mkdir>[0];
const s = (p: unknown) => (typeof p === "string" ? p : p instanceof URL ? p.pathname : undefined);
apps/server/src/storage/vfs.ts
+1 -1
View File
@@ -250,7 +250,7 @@ export async function vreadFile(p: string, maxBytes = 4 * 1024 * 1024): Promise<
/** 本机临时目录(远端写入的本地中转) */
export function relayDir(): string {
return join(storageDataDir() ?? Deno.env.get("DATA_DIR") ?? tmpdir(), "cache", "storage-relay");
return join(storageDataDir() ?? process.env.DATA_DIR ?? tmpdir(), "cache", "storage-relay");
}
/**
apps/server/src/worker/lanes.ts
+2 -1
View File
@@ -1,4 +1,5 @@
import { DEFAULT_WORKER_LANES, WORKER_LANE_MAX, type WorkerLaneSettings } from "@umt/contracts";
import { runtimeUnrefTimer } from "@umt/runtime";
import { laneOf, type QueueCounts, type TaskDefinition, type TaskLane, type WorkerHandle } from "@umt/tasks";
import type { Logger } from "pino";
import type { Config } from "../config.ts";
@@ -167,7 +168,7 @@ export class LaneController {
if (!this.timer) {
this.timer = setInterval(() => void this.refresh(), this.o.refreshMs ?? 5000);
try {
Deno.unrefTimer(this.timer);
runtimeUnrefTimer(this.timer);
} catch {
// 非 Deno 运行时
}
apps/server/test/acquisition-organize/automatic-upgrade-replacement.test.ts
+258
View File
@@ -0,0 +1,258 @@
import { existsSync } from "node:fs";
import { readFile } from "node:fs/promises";
import { createMockQb, createTempTree, type MockQb, makeTorrent, type TempTree } from "@umt/testkit";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { evaluateAcquisition } from "../../src/modules/acquisition-organize/services/acquisition.ts";
import { submitDownload, syncDownloads } from "../../src/modules/acquisition-organize/services/downloads.ts";
import { executeRecord, planImport } from "../../src/modules/acquisition-organize/services/organize.ts";
import { applyEpisodeTotal } from "../../src/modules/acquisition-organize/services/subscriptions.ts";
import { createHarness, fakeDisk, type Harness, pgAvailable, seedBase, seedEntity, seedRelease } from "./harness.ts";
const hasPg = await pgAvailable();
const SIZE = 300_000;
describe.skipIf(!hasPg)("自动洗版物理替换(真实 PG、qB协议与临时文件)", () => {
let h: Harness;
let qb: MockQb;
let lib: TempTree;
let cache: TempTree;
let base: Awaited<ReturnType<typeof seedBase>>;
let groupId: string;
const disk = fakeDisk(100_000_000_000);
const torrents = new Map<string, Uint8Array>();
let pieceSeed = 0;
beforeAll(async () => {
qb = await createMockQb({ addVisibleAfterPolls: 0 });
lib = await createTempTree({}, { sentinel: true, prefix: "umt-auto-up-lib-" });
cache = await createTempTree({}, { prefix: "umt-auto-up-cache-" });
disk.state.watch = [cache.root];
disk.setPct(10);
h = await createHarness("auto_upgrade_replace", {
diskUsage: disk.fn,
torrentSource: async ({ release }) => ({ kind: "file", bytes: torrents.get(release.id)! }),
});
base = await seedBase(h.db, { libraryPath: lib.root, cachePath: cache.root, qbUrl: qb.url });
await h.setSettings("naming", {
movie: "{{title}} ({{year}})/{{title}} ({{year}}) - {{videoFormat}}{{fileExt}}",
tv: "{{title}} ({{year}})/Season {{season}}/{{title}} - {{season_episode}} - {{videoFormat}}{{fileExt}}",
});
const g = await h.api("POST", "/rules", { name: "自动洗版优先REMUX", ruleString: "REMUX > WEBDL" });
expect(g.status, JSON.stringify(g.body)).toBe(201);
groupId = g.body.data.id;
}, 60_000);
afterAll(async () => {
await h?.close();
await qb?.close();
await lib?.cleanup();
await cache?.cleanup();
});
async function tracked(kind: "movie" | "series", title: string, versions: { name: string; resolutions: string[] }[]) {
const entityId = await seedEntity(h.db, { kind, title, year: 2024 });
const sub = await h.db
.insertInto("subscription")
.values({
media_entity_id: entityId,
media_kind: kind,
mode: "once",
season_scope: "current_season",
params: JSON.stringify({ autoFill: "on" }),
})
.returning("id")
.executeTakeFirstOrThrow();
const slotIds: string[] = [];
for (const [position, v] of versions.entries()) {
const slot = await h.db
.insertInto("acquisition_slot")
.values({
media_entity_id: entityId,
subscription_id: sub.id,
name: v.name,
position,
hard_conditions: JSON.stringify({ resolutions: v.resolutions }),
soft_preferences: JSON.stringify({ groups: ["GOOD", "OTHER", "LOW"] }),
upgrade_threshold: JSON.stringify({
enabled: true,
basis: "priority",
mode: kind === "movie" ? "movie" : "per_unit",
...(kind === "series" ? { consistency: "season_release" } : {}),
}),
rule_group_ids: JSON.stringify([groupId]),
})
.returning("id")
.executeTakeFirstOrThrow();
slotIds.push(slot.id);
}
if (kind === "series")
await h.db
.transaction()
.execute((trx) => applyEpisodeTotal(trx, { entityId, season: 1, fresh: 2, now: new Date() }));
return { entityId, slotIds };
}
async function release(entityId: string, title: string) {
const t = makeTorrent({ name: `${title}.mkv`, length: SIZE, pieceSeed: ++pieceSeed });
const id = await seedRelease(h.db, { siteId: base.siteId, entityId, title, size: SIZE, hash: t.hash });
torrents.set(id, t.bytes);
return { id, hash: t.hash };
}
async function acquire(entityId: string, title: string, expectedSlot: string, purpose: "fill" | "upgrade") {
const rel = await release(entityId, title);
const ev = await evaluateAcquisition(h.d, { sourceReleaseId: rel.id });
expect(ev.plans, JSON.stringify(ev)).toHaveLength(1);
const planId = ev.plans[0]!;
const plan = await h.db
.selectFrom("acquisition_plan")
.selectAll()
.where("id", "=", planId)
.executeTakeFirstOrThrow();
expect(plan).toMatchObject({ slot_id: expectedSlot, purpose });
if (purpose === "upgrade") expect(plan.evaluation).toMatchObject({ automaticReplacement: true });
const submitted = await submitDownload(h.d, planId);
expect(submitted.status, JSON.stringify(submitted)).toBe("submitted");
await qb.complete(rel.hash, { writeFiles: true });
expect((await syncDownloads(h.d)).completed).toContain(submitted.downloadTaskId);
const imported = await planImport(h.d, {
downloadTaskId: submitted.downloadTaskId!,
trigger: "auto",
actorType: "system",
});
expect(imported.status, JSON.stringify(imported)).toBe("running");
expect((await executeRecord(h.d, imported.recordId!)).status).toBe("succeeded");
const result = await h.db
.selectFrom("organization_file_result")
.select("media_asset_id")
.where("organization_record_id", "=", imported.recordId!)
.where("role", "=", "main")
.where("status", "=", "succeeded")
.executeTakeFirstOrThrow();
const location = await h.db
.selectFrom("asset_location")
.select("relative_path")
.where("media_asset_id", "=", result.media_asset_id!)
.where("location_kind", "=", "library")
.where("status", "=", "available")
.executeTakeFirstOrThrow();
const bytes = await readFile(lib.path(location.relative_path));
expect(bytes).toHaveLength(SIZE);
return { assetId: result.media_asset_id!, path: location.relative_path, bytes, recordId: imported.recordId! };
}
async function expectKept(file: Awaited<ReturnType<typeof acquire>>) {
const a = await h.db
.selectFrom("media_asset")
.select("status")
.where("id", "=", file.assetId)
.executeTakeFirstOrThrow();
expect(a.status).toBe("active");
expect((await readFile(lib.path(file.path))).equals(file.bytes)).toBe(true);
expect(
await h.db.selectFrom("trash_item").select("id").where("original_path", "=", file.path).execute(),
).toHaveLength(0);
}
async function expectTrashed(file: Awaited<ReturnType<typeof acquire>>, samePath = false) {
const a = await h.db
.selectFrom("media_asset")
.select("status")
.where("id", "=", file.assetId)
.executeTakeFirstOrThrow();
expect(a.status).toBe("retired");
const oldLocation = await h.db
.selectFrom("asset_location")
.select("status")
.where("media_asset_id", "=", file.assetId)
.where("location_kind", "=", "library")
.executeTakeFirstOrThrow();
expect(oldLocation.status).toBe("removed");
if (!samePath) expect(existsSync(lib.path(file.path))).toBe(false);
const items = await h.db
.selectFrom("trash_item")
.select("trash_path")
.where("original_path", "=", file.path)
.execute();
expect(items).toHaveLength(1);
expect((await readFile(items[0]!.trash_path)).equals(file.bytes)).toBe(true);
}
it("电影低档自动入库后升级替换同路径旧文件并回收,另一保留分辨率保持原字节", async () => {
const f = await tracked("movie", "Auto Movie", [
{ name: "HD", resolutions: ["1080p"] },
{ name: "UHD", resolutions: ["2160p"] },
]);
const uhd = await acquire(f.entityId, "Auto.Movie.2024.2160p.BluRay.REMUX.HEVC-GOOD", f.slotIds[1]!, "fill");
const lower = await acquire(f.entityId, "Auto.Movie.2024.1080p.WEB-DL.x264-LOW", f.slotIds[0]!, "fill");
const higher = await acquire(f.entityId, "Auto.Movie.2024.1080p.BluRay.REMUX.AVC-GOOD", f.slotIds[0]!, "upgrade");
expect(higher.path).toBe(lower.path);
expect(higher.assetId).not.toBe(lower.assetId);
expect(higher.bytes.equals(lower.bytes)).toBe(false);
await expectTrashed(lower, true);
await expectKept(uhd);
expect((await readFile(lib.path(higher.path))).equals(higher.bytes)).toBe(true);
await executeRecord(h.d, higher.recordId);
await expectTrashed(lower, true);
await expectKept(uhd);
}, 60_000);
it("旧单文件E1-E2在同套新E1入库时保留,新E2补齐后自动回收旧合集", async () => {
const f = await tracked("series", "Auto Merged", [{ name: "单保留", resolutions: ["720p", "1080p"] }]);
const old = await acquire(f.entityId, "Auto.Merged.S01E01-E02.720p.WEB-DL.x264-LOW", f.slotIds[0]!, "fill");
const first = await acquire(f.entityId, "Auto.Merged.S01E01.1080p.BluRay.REMUX.AVC-GOOD", f.slotIds[0]!, "upgrade");
await expectKept(old);
const second = await acquire(
f.entityId,
"Auto.Merged.S01E02.1080p.BluRay.REMUX.AVC-GOOD",
f.slotIds[0]!,
"upgrade",
);
await expectTrashed(old);
await expectKept(first);
await expectKept(second);
await executeRecord(h.d, second.recordId);
await expectTrashed(old);
}, 60_000);
it("不同组或另一保留分辨率的已入库E2不能凑齐新E1的合集覆盖", async () => {
const f = await tracked("series", "Auto Isolated", [
{ name: "HD", resolutions: ["1080p"] },
{ name: "UHD", resolutions: ["2160p"] },
]);
const old = await acquire(f.entityId, "Auto.Isolated.S01E01-E02.1080p.WEB-DL.x264-LOW", f.slotIds[0]!, "fill");
const otherResolution = await acquire(
f.entityId,
"Auto.Isolated.S01E02.2160p.BluRay.REMUX.AVC-GOOD",
f.slotIds[1]!,
"fill",
);
const otherGroup = await acquire(
f.entityId,
"Auto.Isolated.S01E02.1080p.BluRay.REMUX.AVC-OTHER",
f.slotIds[0]!,
"upgrade",
);
await expectKept(old);
const first = await acquire(
f.entityId,
"Auto.Isolated.S01E01.1080p.BluRay.REMUX.AVC-GOOD",
f.slotIds[0]!,
"upgrade",
);
await expectKept(old);
await expectKept(otherGroup);
await expectKept(otherResolution);
const second = await acquire(
f.entityId,
"Auto.Isolated.S01E02.1080p.BluRay.REMUX.AVC-GOOD",
f.slotIds[0]!,
"upgrade",
);
await expectTrashed(old);
await expectTrashed(otherGroup, true);
await expectKept(first);
await expectKept(otherResolution);
expect((await readFile(lib.path(second.path))).equals(second.bytes)).toBe(true);
}, 60_000);
});
apps/server/test/acquisition-organize/collection-strategy.test.ts
+334
View File
@@ -0,0 +1,334 @@
import { randomUUID } from "node:crypto";
import type { AcquisitionPolicyScope, SlotTemplate, SubscriptionParams } from "@umt/contracts";
import type { AcquisitionSlotRow } from "@umt/db";
import { type QualityFacts, type SlotSoftPreferences, type SlotSpec, seasonReleaseKey } from "@umt/domain";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { ensureSubscriptionSlot } from "../../src/modules/acquisition-organize/services/collection-strategy.ts";
import { seasonReleaseTarget } from "../../src/modules/acquisition-organize/services/season-release.ts";
import { slotComplete } from "../../src/modules/acquisition-organize/services/subscriptions.ts";
import { applySlotPreset } from "../../src/modules/acquisition-organize/services/subscriptions-params.ts";
import { createHarness, type Harness, pgAvailable, seedEntity, seedRelease } from "./harness.ts";
const hasPg = await pgAvailable();
describe.skipIf(!hasPg)("收录策略与成套洗版边界(真实 PG)", () => {
let h: Harness;
let siteId: string;
beforeAll(async () => {
h = await createHarness("collection_strategy");
siteId = (
await h.db
.insertInto("site")
.values({
key: "collection-test",
name: "收录策略测试站",
adapter: "nexusphp",
base_url: "https://pt.example.test",
})
.returning("id")
.executeTakeFirstOrThrow()
).id;
});
afterAll(async () => {
await h?.close();
});
async function policy(scope: AcquisitionPolicyScope, slots: SlotTemplate[], publishedAt = "2026-01-01T00:00:00Z") {
return (
await h.db
.insertInto("policy_revision")
.values({
policy_key: randomUUID(),
revision: 1,
kind: "acquisition",
status: "published",
title: "收录策略",
scope: JSON.stringify(scope),
rules: JSON.stringify({ defaultSlots: slots }),
published_at: new Date(publishedAt),
})
.returning("id")
.executeTakeFirstOrThrow()
).id;
}
async function subscription(
kind: "movie" | "series",
policyId: string | null = null,
params: SubscriptionParams = {},
) {
const entityId = await seedEntity(h.db, { kind, title: `Collection ${randomUUID()}`, year: 2026 });
const sub = await h.db
.insertInto("subscription")
.values({
media_entity_id: entityId,
media_kind: kind,
mode: "once",
season_scope: "current_season",
policy_revision_id: policyId,
params: JSON.stringify(params),
})
.returningAll()
.executeTakeFirstOrThrow();
return { entityId, sub };
}
const slotsOf = (subId: string) =>
h.db.selectFrom("acquisition_slot").selectAll().where("subscription_id", "=", subId).orderBy("position").execute();
async function series(opts: { resolutions?: string[]; wash?: boolean; soft?: SlotSoftPreferences } = {}) {
const { entityId, sub } = await subscription("series");
await h.db
.insertInto("content_unit")
.values({ media_entity_id: entityId, kind: "season", season_number: 1, status: "confirmed", episode_total: 2 })
.execute();
const eps = new Map<number, string>();
for (const ep of [1, 2]) {
const unit = await h.db
.insertInto("content_unit")
.values({
media_entity_id: entityId,
kind: "episode",
season_number: 1,
episode_number: ep,
status: "confirmed",
})
.returning("id")
.executeTakeFirstOrThrow();
eps.set(ep, unit.id);
}
const slot = await h.db
.insertInto("acquisition_slot")
.values({
media_entity_id: entityId,
subscription_id: sub.id,
name: "保留版本",
hard_conditions: JSON.stringify({ resolutions: opts.resolutions ?? ["1080p"] }),
soft_preferences: JSON.stringify(opts.soft ?? {}),
upgrade_threshold: JSON.stringify(
opts.wash === false
? {}
: { enabled: true, basis: "priority", consistency: "season_release", targetPri: 100 },
),
})
.returningAll()
.executeTakeFirstOrThrow();
return { entityId, sub, slot, eps };
}
const quality = (group: string, resolution: QualityFacts["resolution"] = "1080p"): QualityFacts => ({
resolution,
source: "WEB-DL",
videoCodec: "H.264",
releaseGroup: { name: group, source: "suffix" },
hdr: [],
pri: 100,
});
async function asset(f: Awaited<ReturnType<typeof series>>, episodes: number[], q: QualityFacts, ledger = true) {
const a = await h.db
.insertInto("media_asset")
.values({
media_entity_id: f.entityId,
kind: "video",
status: "active",
media_info: JSON.stringify({ _umt: { quality: q } }),
})
.returning("id")
.executeTakeFirstOrThrow();
for (const ep of episodes) {
await h.db
.insertInto("asset_unit_coverage")
.values({ media_asset_id: a.id, content_unit_id: f.eps.get(ep)!, coverage: "full" })
.execute();
if (ledger)
await h.db
.insertInto("slot_unit")
.values({
slot_id: f.slot.id,
content_unit_id: f.eps.get(ep)!,
media_asset_id: a.id,
quality: JSON.stringify(q),
pri: q.pri,
})
.onConflict((oc) =>
oc
.columns(["slot_id", "content_unit_id"])
.doUpdateSet({ media_asset_id: a.id, quality: JSON.stringify(q), pri: q.pri }),
)
.execute();
}
return a.id;
}
async function plan(
slot: AcquisitionSlotRow,
q: QualityFacts,
units: string[],
status: "submitted" | "reserved" = "submitted",
) {
const relId = await seedRelease(h.db, {
siteId,
entityId: slot.media_entity_id,
title: `Collection.S01.${q.resolution}.WEB-DL.x264-${q.releaseGroup!.name}`,
size: 1_000_000,
hash: randomUUID().replaceAll("-", "").padEnd(40, "0"),
});
const p = await h.db
.insertInto("acquisition_plan")
.values({
slot_id: slot.id,
source_release_id: relId,
status,
status_reason: status === "submitted" ? "download_accepted" : null,
purpose: "upgrade",
unit_keys: units,
payload_hash: randomUUID(),
idempotency_key: randomUUID(),
evaluation: JSON.stringify({ pri: q.pri, quality: q, season: 1, seasonRelease: seasonReleaseKey(q) }),
})
.returning("id")
.executeTakeFirstOrThrow();
return { planId: p.id, relId };
}
it("显式选择优先;分类按 movie/tv 匹配,未命中分类时使用媒体类型默认模板", async () => {
const allId = await policy({}, [{ name: "通用" }], "2026-06-01T00:00:00Z");
const movieId = await policy({ mediaKinds: ["movie"] }, [{ name: "电影默认" }]);
const tvId = await policy({ mediaKinds: ["series"] }, [{ name: "剧集默认" }]);
const movieCategoryId = await policy({ categories: { movie: ["精选"] } }, [{ name: "电影精选" }]);
const tvCategoryId = await policy({ categories: { tv: ["精选"] } }, [{ name: "剧集精选" }]);
for (const [kind, category, explicitId, expectedId, name] of [
["series", "精选", allId, allId, "通用"],
["movie", "精选", null, movieCategoryId, "电影精选"],
["series", "精选", null, tvCategoryId, "剧集精选"],
["movie", "其他", null, movieId, "电影默认"],
["series", "其他", null, tvId, "剧集默认"],
] as const) {
const { sub } = await subscription(kind, explicitId);
expect(await ensureSubscriptionSlot(h.db, sub, { mediaKind: kind, category })).toBe(1);
expect(await slotsOf(sub.id)).toMatchObject([{ name, policy_revision_id: expectedId, position: 0 }]);
expect(await ensureSubscriptionSlot(h.db, sub, { mediaKind: kind, category })).toBe(0);
expect(await slotsOf(sub.id)).toHaveLength(1);
}
});
it("显式选择没有模板的策略时报告错误,不静默回退到分类或通用策略", async () => {
const policyId = await policy({}, []);
const { sub } = await subscription("series", policyId);
await expect(ensureSubscriptionSlot(h.db, sub, { mediaKind: "series", category: "精选" })).rejects.toThrow(
"所选收录策略",
);
expect(await slotsOf(sub.id)).toHaveLength(0);
});
it("双分辨率模板保留各自范围,统一预设即使与某模板相同也不能覆盖它", async () => {
const policyId = await policy({}, [
{ name: "HD", hardConditions: { resolutions: ["1080p"] } },
{ name: "UHD", hardConditions: { resolutions: ["2160p"] } },
]);
const { sub } = await subscription("series", policyId, { resolution: "1080p" });
expect(await ensureSubscriptionSlot(h.db, sub)).toBe(2);
const expected = [
{ name: "HD", hard_conditions: { resolutions: ["1080p"] } },
{ name: "UHD", hard_conditions: { resolutions: ["2160p"] } },
];
expect(await slotsOf(sub.id)).toMatchObject(expected);
await applySlotPreset(h.db, {
subscriptionId: sub.id,
prev: { resolution: "1080p" },
next: { resolution: "720p" },
});
expect(await slotsOf(sub.id)).toMatchObject(expected);
});
it("单保留4K版本时,作品已有完整720p不能填补4K覆盖", async () => {
const f = await series({ resolutions: ["2160p"], wash: false });
await asset(f, [1, 2], quality("LOW", "720p"), false);
expect(await slotComplete(h.db, f.slot, "tv", [])).toMatchObject({ complete: false, atTarget: false });
await asset(f, [1, 2], quality("UHD", "2160p"));
expect(await slotComplete(h.db, f.slot, "tv", [])).toMatchObject({ complete: true, atTarget: true });
});
it("已接收的旧720p计划同样不能填补改为4K后的范围", async () => {
const f = await series({ resolutions: ["2160p"], wash: false });
await plan(f.slot, quality("LOW", "720p"), ["s1e1", "s1e2"]);
expect(await slotComplete(h.db, f.slot, "tv", [], { includeSubmitted: true })).toMatchObject({
complete: false,
atTarget: false,
});
await plan(f.slot, quality("UHD", "2160p"), ["s1e1", "s1e2"]);
expect(await slotComplete(h.db, f.slot, "tv", [], { includeSubmitted: true })).toMatchObject({
complete: true,
atTarget: true,
});
});
it("HD/UHD分别完成,另一个保留分辨率的资产不能借来填缺", async () => {
const f = await series({ resolutions: ["1080p"], wash: false });
const uhd = await h.db
.insertInto("acquisition_slot")
.values({
media_entity_id: f.entityId,
subscription_id: f.sub.id,
name: "UHD",
hard_conditions: JSON.stringify({ resolutions: ["2160p"] }),
})
.returningAll()
.executeTakeFirstOrThrow();
await asset(f, [1, 2], quality("HD"));
expect(await slotComplete(h.db, f.slot, "tv", [])).toMatchObject({ complete: true });
expect(await slotComplete(h.db, uhd, "tv", [])).toMatchObject({ complete: false });
await asset({ ...f, slot: uhd }, [1, 2], quality("UHD", "2160p"));
expect(await slotComplete(h.db, uhd, "tv", [])).toMatchObject({ complete: true });
expect(await slotComplete(h.db, f.slot, "tv", [])).toMatchObject({ complete: true });
});
it("各集pri100但组不同仍未到成套目标;同组的分集资源合齐后到目标", async () => {
const f = await series({ soft: { groups: ["B", "A"] } });
await asset(f, [1], quality("B"));
await asset(f, [2], quality("A"));
expect(await slotComplete(h.db, f.slot, "tv", [])).toMatchObject({ complete: true, atTarget: false });
await asset(f, [2], quality("B"));
expect(await slotComplete(h.db, f.slot, "tv", [])).toMatchObject({ complete: true, atTarget: true });
});
it("完整A不能盖过更优但只有部分集的B,成套洗版继续以B为目标", async () => {
const f = await series({ soft: { groups: ["B", "A"] } });
await asset(f, [1, 2], quality("A"));
expect(await slotComplete(h.db, f.slot, "tv", [])).toMatchObject({ complete: true, atTarget: true });
await asset(f, [1], quality("B"));
expect(await slotComplete(h.db, f.slot, "tv", [])).toMatchObject({ complete: true, atTarget: false });
await asset(f, [2], quality("B"));
expect(await slotComplete(h.db, f.slot, "tv", [])).toMatchObject({ complete: true, atTarget: true });
});
it("在途季包冻结版本事实,发布属性修改不漂移锚点,失败后才释放", async () => {
const f = await series({ soft: { groups: ["C", "A", "B"] } });
const frozen = quality("A");
const p = await plan(f.slot, frozen, ["s1*"], "reserved");
const spec: SlotSpec = {
name: "成套目标",
hard: { resolutions: ["1080p"] },
soft: { groups: ["C", "A", "B"] },
upgrade: { enabled: true, basis: "priority", consistency: "season_release" },
};
expect(seasonReleaseKey((await seasonReleaseTarget(h.db, f.slot, 1, spec, quality("B")))!)).toBe(
seasonReleaseKey(frozen),
);
await h.db
.updateTable("source_release")
.set({ title: "Collection.S01.1080p.WEB-DL.x264-C", parsed: JSON.stringify({}), parser_version: null })
.where("id", "=", p.relId)
.execute();
expect(seasonReleaseKey((await seasonReleaseTarget(h.db, f.slot, 1, spec, quality("B")))!)).toBe(
seasonReleaseKey(frozen),
);
await h.db.updateTable("acquisition_plan").set({ status: "failed" }).where("id", "=", p.planId).execute();
expect(seasonReleaseKey((await seasonReleaseTarget(h.db, f.slot, 1, spec, quality("B")))!)).toBe(
seasonReleaseKey(quality("B")),
);
});
});
apps/server/test/acquisition-organize/download-observation-order.test.ts
+186
View File
@@ -0,0 +1,186 @@
import { createQbClient } from "@umt/adapters";
import { createMockQb, createTempTree, type MockQb, type TempTree } from "@umt/testkit";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { MISSING_CONFIRM_ROUNDS, syncDownloads } from "../../src/modules/acquisition-organize/services/downloads.ts";
import { createHarness, fakeDisk, type Harness, pgAvailable, seedBase } from "./harness.ts";
const hasPg = await pgAvailable();
function deferred() {
let resolve!: () => void;
const promise = new Promise<void>((done) => {
resolve = done;
});
return { promise, resolve };
}
describe.skipIf(!hasPg)("下载观测顺序(真实 PG + 受控 qB 响应)", () => {
let h: Harness;
let qb: MockQb;
let lib: TempTree;
let cache: TempTree;
let base: Awaited<ReturnType<typeof seedBase>>;
let tick = 0;
let hashIndex = 0;
const epoch = Date.parse("2026-10-13T00:00:00Z");
let pending: {
method: "list" | "findByHash";
hash?: string;
captured: ReturnType<typeof deferred>;
resume: ReturnType<typeof deferred>;
} | null = null;
beforeAll(async () => {
qb = await createMockQb();
lib = await createTempTree({}, { sentinel: true });
cache = await createTempTree({});
h = await createHarness("download_order", { diskUsage: fakeDisk(10_000_000_000).fn });
base = await seedBase(h.db, { libraryPath: lib.root, cachePath: cache.root, qbUrl: qb.url });
h.d.ports.now = () => new Date(epoch + tick);
const original = h.d.ports.downloader;
h.d.ports.downloader = async (inst) => {
const client = await original(inst);
const hold = async <T>(method: "list" | "findByHash", read: () => Promise<T>, hash?: string): Promise<T> => {
const gate = pending?.method === method && (!pending.hash || pending.hash === hash) ? pending : null;
if (gate) pending = null;
const snapshot = await read();
if (gate) {
gate.captured.resolve();
await gate.resume.promise;
}
return snapshot;
};
return {
...client,
list: (input) => hold("list", () => client.list(input)),
findByHash: (hash) => hold("findByHash", () => client.findByHash(hash), hash),
};
};
});
afterAll(async () => {
pending?.resume.resolve();
await h?.close();
await qb?.close();
await lib?.cleanup();
await cache?.cleanup();
});
const nextTime = () => new Date(epoch + ++tick);
const row = (id: string) =>
h.db.selectFrom("download_task").selectAll().where("id", "=", id).executeTakeFirstOrThrow();
const pauseNext = (method: "list" | "findByHash", hash?: string) => {
const gate = { method, hash, captured: deferred(), resume: deferred() };
pending = gate;
return gate;
};
const seedTorrent = (hash: string) => {
qb.seed({ hash, name: `Observation.${hash}.mkv`, size: 300_000, tags: ["umt"], save_path: cache.root });
qb.setProgress(hash, 0.25);
};
async function task() {
const hash = (++hashIndex).toString(16).padStart(40, "0");
seedTorrent(hash);
nextTime();
await syncDownloads(h.d);
const r = await h.db
.selectFrom("download_task")
.selectAll()
.where("torrent_hash", "=", hash)
.executeTakeFirstOrThrow();
expect(r).toMatchObject({ managed: true, status: "downloading", missing_count: 0 });
return { hash, id: r.id };
}
it("移除标签的新观测先提交时,旧有标签列表晚到不能复活managed", async () => {
const t = await task();
const gate = pauseNext("list");
nextTime();
const older = syncDownloads(h.d);
try {
await gate.captured.promise;
qb.get(t.hash)!.tags = [];
const newerAt = nextTime();
await syncDownloads(h.d);
expect(await row(t.id)).toMatchObject({ managed: false, tags: [], last_synced_at: newerAt });
gate.resume.resolve();
await older;
expect(await row(t.id)).toMatchObject({ managed: false, tags: [], removed_at: null, last_synced_at: newerAt });
} finally {
gate.resume.resolve();
await older;
}
});
it("旧缺失响应晚到不能把已经重新观测到的任务判移除或恢复旧missing_count", async () => {
const t = await task();
await h.db
.updateTable("download_task")
.set({ missing_count: MISSING_CONFIRM_ROUNDS - 1, missing_since: new Date(epoch) })
.where("id", "=", t.id)
.execute();
const external = createQbClient({
instanceId: base.instanceId,
baseUrl: qb.url,
username: "admin",
password: "adminadmin",
});
await external.delete([t.hash], { deleteFiles: false });
const gate = pauseNext("findByHash", t.hash);
nextTime();
const older = syncDownloads(h.d);
try {
await gate.captured.promise;
seedTorrent(t.hash);
const newerAt = nextTime();
await syncDownloads(h.d);
expect(await row(t.id)).toMatchObject({
managed: true,
status: "downloading",
missing_count: 0,
last_synced_at: newerAt,
});
gate.resume.resolve();
await older;
expect(await row(t.id)).toMatchObject({
managed: true,
status: "downloading",
removed_at: null,
missing_count: 0,
missing_since: null,
last_synced_at: newerAt,
});
} finally {
gate.resume.resolve();
await older;
}
});
it("重新贴标签的新观测可恢复管理,旧无标签响应晚到不能撤销恢复", async () => {
const t = await task();
qb.get(t.hash)!.tags = [];
const gate = pauseNext("findByHash", t.hash);
nextTime();
const older = syncDownloads(h.d);
try {
await gate.captured.promise;
qb.get(t.hash)!.tags = ["umt"];
const newerAt = nextTime();
await syncDownloads(h.d);
gate.resume.resolve();
await older;
expect(await row(t.id)).toMatchObject({ managed: true, tags: ["umt"], last_synced_at: newerAt });
qb.get(t.hash)!.tags = [];
nextTime();
await syncDownloads(h.d);
expect(await row(t.id)).toMatchObject({ managed: false });
qb.get(t.hash)!.tags = ["umt"];
const restoredAt = nextTime();
await syncDownloads(h.d);
expect(await row(t.id)).toMatchObject({ managed: true, tags: ["umt"], last_synced_at: restoredAt });
} finally {
gate.resume.resolve();
await older;
}
});
});
apps/server/test/acquisition-organize/organize-recovery.test.ts
+17
View File
@@ -262,6 +262,23 @@ describe.skipIf(!hasPg)("整理卡死回收(真实 PG + mock qB + 临时目录)"
const dec = await decision(`organize_excluded:${plan.recordId}`);
expect(dec).toMatchObject({ status: "open", subject_id: plan.recordId });
expect(JSON.stringify(dec!.evidence)).toContain("excluded:episode_unresolved");
// 终态重放也不能把部分成功当成整包成功,且刮削记录不重复。
const scrapes = () =>
h.db.selectFrom("scrape_record").select("id").where("organization_record_id", "=", plan.recordId!).execute();
const beforeReplay = await scrapes();
expect(beforeReplay).toHaveLength(1);
expect((await executeRecord(h.d, plan.recordId!)).status).toBe("partial");
expect(existsSync(join(cache.root, name, ani(40)))).toBe(true);
expect(qb.get(hash)).toBeDefined();
expect(qb.callsTo("torrents/delete")).toHaveLength(0);
expect(
await h.db
.selectFrom("download_task")
.select(["managed", "removed_at"])
.where("id", "=", dtId)
.executeTakeFirst(),
).toEqual({ managed: true, removed_at: null });
expect(await scrapes()).toEqual(beforeReplay);
// partial 不在补移除范围内
await h.db
.updateTable("organization_record")
apps/server/test/acquisition-organize/policy-combinations.test.ts
+318
View File
@@ -0,0 +1,318 @@
import { randomUUID } from "node:crypto";
import type { AcquisitionPolicyRules, PolicyRevision, RuleGroup } from "@umt/contracts";
import { afterAll, beforeAll, beforeEach, describe, expect, it } from "vitest";
import { evaluateAcquisition } from "../../src/modules/acquisition-organize/services/acquisition.ts";
import { loadRuleCatalog } from "../../src/modules/acquisition-organize/services/rules.ts";
import { slotSpecWithRules } from "../../src/modules/acquisition-organize/services/subscriptions.ts";
import { createHarness, fakeDisk, type Harness, pgAvailable, seedBase, seedEntity, seedRelease } from "./harness.ts";
// 实际 PG + 路由/评估入口：模板引用、补齐与洗版组合，以及发布/删除时的引用保护。
const pg = await pgAvailable();
describe.skipIf(!pg)("收录策略与规则组合", () => {
let h: Harness;
let base: Awaited<ReturnType<typeof seedBase>>;
let n = 0;
const disk = fakeDisk(100_000_000_000);
beforeAll(async () => {
disk.setPct(10);
h = await createHarness("policy_combinations", { diskUsage: disk.fn });
base = await seedBase(h.db, {
libraryPath: "/tmp/umt-policy-combinations-library",
cachePath: "/tmp/umt-policy-combinations-cache",
qbUrl: "http://127.0.0.1:9",
});
});
beforeEach(async () => {
const defaults = await h.api("PUT", "/rules/defaults", { acquisition: [], upgrade: [], search: [] });
expect(defaults.status).toBe(200);
await h.db
.updateTable("policy_revision")
.set({ status: "archived" })
.where("status", "in", ["draft", "published"])
.execute();
await h.db.updateTable("acquisition_plan").set({ status: "cancelled" }).where("status", "=", "reserved").execute();
});
afterAll(async () => {
await h?.close();
});
async function group(ruleString = "H265", kind: "acquisition" | "upgrade" = "acquisition") {
const response = await h.api("POST", "/rules", { name: `组合规则-${++n}`, kind, ruleString });
expect(response.status, JSON.stringify(response.body)).toBe(201);
return response.body.data as RuleGroup;
}
async function draft(rules: AcquisitionPolicyRules, policyKey = `combinations-${++n}`) {
const response = await h.api("POST", "/policy-revisions", {
policyKey,
kind: "acquisition",
title: "组合策略",
rules,
});
expect(response.status, JSON.stringify(response.body)).toBe(201);
return response.body.data as PolicyRevision;
}
async function publish(policy: PolicyRevision) {
const response = await h.api("POST", `/policy-revisions/${policy.id}/publish`, {
expectedRevision: policy.revision,
});
expect(response.status, JSON.stringify(response.body)).toBe(200);
return response.body.data as PolicyRevision;
}
const release = (entityId: string, title: string) =>
seedRelease(h.db, {
siteId: base.siteId,
entityId,
title,
size: 300_000_000,
hash: (++n).toString(16).padStart(40, "0"),
});
it("策略预览执行模板的 H265 规则组，硬条件相同的 H264 资源被排除", async () => {
const filter = await group();
const entityId = await seedEntity(h.db, { kind: "movie", title: "Preview Film", year: 2024 });
const h265 = await release(entityId, "Preview.Film.2024.1080p.WEB-DL.H265-GRP");
const h264 = await release(entityId, "Preview.Film.2024.1080p.WEB-DL.H264-GRP");
const policy = await draft({
defaultSlots: [{ name: "只要 H265", hardConditions: { resolutions: ["1080p"] }, ruleGroupIds: [filter.id] }],
});
const response = await h.api("POST", `/policy-revisions/${policy.id}/preview`);
expect(response.status, JSON.stringify(response.body)).toBe(200);
const preview = response.body.data.preview as {
matched: number;
excluded: number;
samples: { sourceReleaseId: string; decision: string; reasons: string[] }[];
};
expect(preview).toMatchObject({ matched: 1, excluded: 1 });
expect(preview.samples.find((sample) => sample.sourceReleaseId === h265)).toMatchObject({ decision: "accept" });
expect(preview.samples.find((sample) => sample.sourceReleaseId === h264)).toMatchObject({ decision: "reject" });
expect(await h.db.selectFrom("acquisition_plan").select("id").execute()).toHaveLength(0);
});
it("新订阅允许先补 WEB：即使模板开启升级，也不把 REMUX 洗版默认链叠到补齐规则", async () => {
const fill = await group("H265");
const wash = await group("REMUX", "upgrade");
expect(
(await h.api("PUT", "/rules/defaults", { acquisition: [fill.id], upgrade: [wash.id], search: [] })).status,
).toBe(200);
await publish(
await draft({
defaultSlots: [
{
name: "先补齐再洗版",
hardConditions: { resolutions: ["1080p"] },
upgradeThreshold: { enabled: true, mode: "movie", basis: "priority" },
},
],
}),
);
const entityId = await seedEntity(h.db, { kind: "movie", title: "Fill Film", year: 2024 });
const subscription = await h.api("POST", "/subscriptions", { mediaEntityId: entityId, mediaKind: "movie" });
expect(subscription.status, JSON.stringify(subscription.body)).toBe(201);
const slot = await h.db
.selectFrom("acquisition_slot")
.selectAll()
.where("subscription_id", "=", subscription.body.data.id)
.executeTakeFirstOrThrow();
expect(slot.upgrade_threshold).toMatchObject({ enabled: true });
const resource = await release(entityId, "Fill.Film.2024.1080p.WEB-DL.H265-GRP");
const result = await evaluateAcquisition(h.d, { sourceReleaseId: resource });
expect(result.plans, JSON.stringify(result)).toHaveLength(1);
const plan = await h.db
.selectFrom("acquisition_plan")
.selectAll()
.where("id", "=", result.plans[0]!)
.executeTakeFirstOrThrow();
expect(plan).toMatchObject({ slot_id: slot.id, purpose: "fill", status: "reserved" });
const spec = slotSpecWithRules(
await loadRuleCatalog(h.db),
slot,
null,
{ id: entityId, kind: "movie" },
base.siteId,
);
expect(spec.rules?.groups.map((g) => g.id)).toEqual([fill.id]);
expect(spec.upgradeRules?.groups.map((g) => g.id)).toEqual([fill.id, wash.id]);
});
it("规则组被收录版本的洗版引用时不能删；通过版本更新移除引用后可删", async () => {
const wash = await group("REMUX", "upgrade");
const entityId = await seedEntity(h.db, { kind: "movie", title: "Slot Reference", year: 2024 });
const created = await h.api("POST", "/acquisition-slots", {
mediaEntityId: entityId,
name: "洗版引用",
upgradeThreshold: { enabled: true, ruleGroupIds: [wash.id] },
});
expect(created.status, JSON.stringify(created.body)).toBe(201);
expect((await h.api("DELETE", `/rules/${wash.id}`)).status).toBe(409);
expect(
(await h.db.selectFrom("rule_group").select("deleted_at").where("id", "=", wash.id).executeTakeFirstOrThrow())
.deleted_at,
).toBeNull();
const updated = await h.api("PATCH", `/acquisition-slots/${created.body.data.id}`, {
expectedRevision: created.body.data.revision,
upgradeThreshold: { enabled: true, ruleGroupIds: [] },
});
expect(updated.status, JSON.stringify(updated.body)).toBe(200);
expect((await h.api("DELETE", `/rules/${wash.id}`)).status).toBe(200);
expect(
(await h.db.selectFrom("rule_group").select("deleted_at").where("id", "=", wash.id).executeTakeFirstOrThrow())
.deleted_at,
).not.toBeNull();
});
it.each([
{ label: "草案 defaultSlots 补齐", published: false, newTitle: false, wash: false },
{ label: "已发布 defaultSlots 洗版", published: true, newTitle: false, wash: true },
{ label: "草案 newTitle 洗版", published: false, newTitle: true, wash: true },
{ label: "已发布 newTitle 补齐", published: true, newTitle: true, wash: false },
])("$label 引用阻止删除，解除有效策略引用后可删", async ({ published, newTitle, wash }) => {
const referenced = await group(wash ? "REMUX" : "H265", wash ? "upgrade" : "acquisition");
const template = {
name: "引用模板",
...(wash
? { upgradeThreshold: { enabled: true, ruleGroupIds: [referenced.id] } }
: { ruleGroupIds: [referenced.id] }),
};
const rules: AcquisitionPolicyRules = newTitle
? { newTitle: { enabled: true, slots: [template] } }
: { defaultSlots: [template] };
const policy = await draft(rules);
if (published) await publish(policy);
expect((await h.api("DELETE", `/rules/${referenced.id}`)).status).toBe(409);
if (published) {
await publish(await draft({}, policy.policyKey));
expect(
(
await h.db
.selectFrom("policy_revision")
.select("status")
.where("id", "=", policy.id)
.executeTakeFirstOrThrow()
).status,
).toBe("superseded");
} else {
const updated = await h.api("PATCH", `/policy-revisions/${policy.id}`, { rules: {} });
expect(updated.status, JSON.stringify(updated.body)).toBe(200);
}
expect((await h.api("DELETE", `/rules/${referenced.id}`)).status).toBe(200);
});
it.each(["fill", "wash"] as const)("发布拒绝不存在的模板 %s 规则组，旧发布和授权保留", async (phase) => {
const existing = await publish(await draft({}));
const missing = randomUUID();
const rules: AcquisitionPolicyRules =
phase === "fill"
? { defaultSlots: [{ name: "坏补齐引用", ruleGroupIds: [missing] }] }
: {
newTitle: {
enabled: true,
slots: [{ name: "坏洗版引用", upgradeThreshold: { enabled: true, ruleGroupIds: [missing] } }],
},
};
const invalid = await draft(rules, existing.policyKey);
const response = await h.api("POST", `/policy-revisions/${invalid.id}/publish`, {
expectedRevision: invalid.revision,
});
expect(response.status, JSON.stringify(response.body)).toBe(400);
expect(response.body.error.message).toContain("规则组不存在");
expect(response.body.error.message).toContain(missing);
const policies = await h.db
.selectFrom("policy_revision")
.select(["id", "status"])
.where("id", "in", [existing.id, invalid.id])
.execute();
expect(policies).toEqual(
expect.arrayContaining([
{ id: existing.id, status: "published" },
{ id: invalid.id, status: "draft" },
]),
);
expect(
await h.db
.selectFrom("automation_grant")
.select(["status"])
.where("policy_revision_id", "=", existing.id)
.execute(),
).toEqual([{ status: "active" }]);
expect(
await h.db.selectFrom("automation_grant").select("id").where("policy_revision_id", "=", invalid.id).execute(),
).toHaveLength(0);
});
it("电影忽略季级一致性，仍能从已有 WEB 资产生成 REMUX 升级计划", async () => {
const entityId = await seedEntity(h.db, { kind: "movie", title: "Movie Upgrade", year: 2024 });
const subscription = await h.api("POST", "/subscriptions", { mediaEntityId: entityId, mediaKind: "movie" });
expect(subscription.status, JSON.stringify(subscription.body)).toBe(201);
const slot = await h.db
.selectFrom("acquisition_slot")
.selectAll()
.where("subscription_id", "=", subscription.body.data.id)
.executeTakeFirstOrThrow();
const update = await h.api("PATCH", `/acquisition-slots/${slot.id}`, {
expectedRevision: slot.revision,
hardConditions: { resolutions: ["1080p"] },
upgradeThreshold: { enabled: true, mode: "movie", basis: "tier", consistency: "season_release" },
});
expect(update.status, JSON.stringify(update.body)).toBe(200);
const asset = await h.db
.insertInto("media_asset")
.values({
media_entity_id: entityId,
kind: "video",
size_bytes: 300_000_000,
media_info: JSON.stringify({
_umt: {
quality: {
resolution: "1080p",
source: "WEB-DL",
videoCodec: "H.264",
releaseGroup: { name: "OLD", source: "suffix" },
hdr: [],
},
},
}),
})
.returning("id")
.executeTakeFirstOrThrow();
await h.db
.updateTable("acquisition_slot")
.set({ status: "satisfied", current_asset_id: asset.id })
.where("id", "=", slot.id)
.execute();
const persisted = await h.db
.selectFrom("acquisition_slot")
.selectAll()
.where("id", "=", slot.id)
.executeTakeFirstOrThrow();
expect(persisted.upgrade_threshold).toMatchObject({ enabled: true, consistency: "season_release" });
const spec = slotSpecWithRules(
await loadRuleCatalog(h.db),
persisted,
null,
{ id: entityId, kind: "movie" },
base.siteId,
);
expect(spec.upgrade).toMatchObject({ enabled: true, mode: "movie" });
expect(spec.upgrade?.consistency).toBeUndefined();
const resource = await release(entityId, "Movie.Upgrade.2024.1080p.BluRay.REMUX.H264-NEW");
const result = await evaluateAcquisition(h.d, { sourceReleaseId: resource });
expect(result.plans, JSON.stringify(result)).toHaveLength(1);
const plan = await h.db
.selectFrom("acquisition_plan")
.selectAll()
.where("id", "=", result.plans[0]!)
.executeTakeFirstOrThrow();
expect(plan).toMatchObject({
slot_id: slot.id,
purpose: "upgrade",
evaluation: { automaticReplacement: true, replacesAssetId: asset.id },
});
});
});
apps/server/test/auth-settings/rotate-master-key.test.ts
+1 -1
View File
@@ -79,7 +79,7 @@ describe.skipIf(!hasPg)("主密钥轮换(临时库)", () => {
const main = fileURLToPath(new URL("../../src/main.ts", import.meta.url));
const url = `${PG_URL}${PG_URL.includes("?") ? "&" : "?"}options=${encodeURIComponent(`-c search_path=${h.schema},public`)}`;
const run = (extra: Record<string, string>) =>
spawnSync("deno", ["run", "-A", main, "--role", "rotate-master-key"], {
spawnSync(process.execPath, [main, "--role", "rotate-master-key"], {
env: {
PATH: process.env.PATH ?? "",
HOME: process.env.HOME ?? "",
apps/server/test/auth-settings/sse-pg.test.ts
+1 -1
View File
@@ -13,7 +13,7 @@ const hasPg = await pgAvailable();
const SERVER_DIR = fileURLToPath(new URL("../..", import.meta.url));
function proc(args: string[]) {
const child = spawn(process.execPath, ["run", "-A", "test/fixtures/sse-bus-proc.ts", ...args], {
const child = spawn(process.execPath, ["test/fixtures/sse-bus-proc.ts", ...args], {
cwd: SERVER_DIR,
env: { ...process.env, DATABASE_URL: PG_URL },
stdio: ["ignore", "pipe", "pipe"],
apps/server/test/authorization/grants.test.ts
+130
View File
@@ -0,0 +1,130 @@
import { randomUUID } from "node:crypto";
import { afterAll, beforeAll, beforeEach, describe, expect, it } from "vitest";
import { coveringGrant, type PlanBinding, planBinding } from "../../src/authorization/grants.ts";
import { createTestSchema, pgAvailable, type TestSchema, uniqueSchema } from "../support/pg.ts";
const hasPg = await pgAvailable();
const now = new Date("2026-01-15T12:00:00Z");
const target: PlanBinding = {
payloadHash: "plan-hash",
revision: 1,
mediaEntityId: randomUUID(),
mediaKind: "tv",
bytes: 100,
};
describe.skipIf(!hasPg)("共用计划绑定与持续授权(真实 PG)", () => {
let schema: TestSchema;
beforeAll(async () => {
schema = await createTestSchema(uniqueSchema("server_test_grants"));
});
afterAll(async () => {
await schema?.drop();
});
beforeEach(async () => {
await schema.db.deleteFrom("capability_invocation").execute();
await schema.db.deleteFrom("automation_grant").execute();
await schema.db.deleteFrom("policy_revision").execute();
await schema.db.deleteFrom("artifact").execute();
});
const grant = (over: Record<string, unknown> = {}) =>
schema.db
.insertInto("automation_grant")
.values({ action: "download.submit", granted_by: "user:test", ...over })
.returning("id")
.executeTakeFirstOrThrow();
const check = (plan = target) => coveringGrant(schema.db, "download.submit", plan, now);
it("整理计划绑定读取当前内容与 hash,计划不存在时返回 null", async () => {
const p = await schema.db
.insertInto("artifact")
.values({
kind: "plan",
title: "整理计划",
content_hash: target.payloadHash,
content: JSON.stringify({ mediaEntityId: target.mediaEntityId, mediaKind: target.mediaKind }),
})
.returning("id")
.executeTakeFirstOrThrow();
expect(await planBinding(schema.db, "organize_plan", p.id)).toEqual({ ...target, bytes: null });
await schema.db.updateTable("artifact").set({ content_hash: "new-hash" }).where("id", "=", p.id).execute();
expect((await planBinding(schema.db, "organize_plan", p.id))?.payloadHash).toBe("new-hash");
expect(await planBinding(schema.db, "organize_plan", randomUUID())).toBeNull();
});
it("授权范围同时检查能力、作品和媒体类型,series 范围覆盖 tv", async () => {
const g = await grant({
scope: JSON.stringify({
capabilities: ["download.submit"],
mediaEntityIds: [target.mediaEntityId],
mediaKinds: ["series"],
}),
});
expect(await check()).toEqual({ grantId: g.id });
expect(await check({ ...target, mediaEntityId: randomUUID() })).toMatchObject({ reasons: [expect.any(String)] });
expect(await check({ ...target, mediaKind: "movie" })).toMatchObject({ reasons: [expect.any(String)] });
await schema.db
.updateTable("automation_grant")
.set({ scope: JSON.stringify({ capabilities: ["organize.execute"] }) })
.where("id", "=", g.id)
.execute();
expect(await check()).toMatchObject({ reasons: [expect.any(String)] });
});
it.each(["expired", "revoked"] as const)("%s 授权不放行", async (state) => {
await grant(
state === "expired" ? { expires_at: new Date(now.getTime() - 1) } : { status: "revoked", revoked_at: now },
);
expect(await check()).toEqual({ reasons: [] });
});
it("策略被替代后,绑定该策略的持续授权不再放行", async () => {
const p = await schema.db
.insertInto("policy_revision")
.values({
policy_key: "test-acquisition",
revision: 1,
kind: "acquisition",
title: "测试策略",
status: "published",
})
.returning("id")
.executeTakeFirstOrThrow();
const g = await grant({ action: "policy.acquisition", policy_revision_id: p.id });
expect(await check()).toEqual({ grantId: g.id });
await schema.db.updateTable("policy_revision").set({ status: "superseded" }).where("id", "=", p.id).execute();
expect(await check()).toMatchObject({ reasons: [expect.stringContaining("已不是发布状态")] });
});
it("每日次数只统计成功调用,达到预算后拒绝", async () => {
const g = await grant({ budget: JSON.stringify({ maxInvocationsPerDay: 2 }) });
const invoke = (status: "failed" | "succeeded") =>
schema.db
.insertInto("capability_invocation")
.values({
idempotency_key: randomUUID(),
capability_id: "download.submit",
capability_version: 1,
actor_id: "user:test",
source: "web",
status,
grant_id: g.id,
created_at: now,
})
.execute();
await invoke("succeeded");
await invoke("failed");
expect(await check()).toEqual({ grantId: g.id });
await invoke("succeeded");
expect(await check()).toMatchObject({ reasons: [expect.stringContaining("次数预算已用尽")] });
});
it("提交下载时把本次预留字节计入预算", async () => {
const g = await grant({ budget: JSON.stringify({ maxBytesPerDay: 100 }) });
expect(await check()).toEqual({ grantId: g.id });
expect(await check({ ...target, bytes: 101 })).toMatchObject({
reasons: [expect.stringContaining("字节预算不足")],
});
});
});
apps/server/test/desktop/desktop.test.ts
+1 -1
View File
@@ -52,7 +52,7 @@ function harness(
status: 200,
headers: { "content-type": "text/html", "www-authenticate": "Basic", "x-frame-options": "DENY" },
});
}) as typeof fetch,
}) as (input: RequestInfo | URL, init?: RequestInit) => Promise<Response>,
});
const root = new OpenAPIHono<AuthEnv>();
root.onError((err, c) => {
apps/server/test/e2e/agent-prompt-eval.real.test.ts
+1 -1
View File
@@ -14,7 +14,7 @@ import { createE2E, type E2E, pgAvailable, runEvents, runRow, waitFor } from "./
// 运行(凭证只经环境变量传入,不写进代码与报告):
// UMT_EVAL_REAL=1 UMT_EVAL_API_KEY=… UMT_EVAL_PROVIDER=opencode-go UMT_EVAL_MODEL=deepseek-v4.1-flash \
// UMT_EVAL_LEGACY_PROMPT=/tmp/legacy-system-prompt.ts UMT_EVAL_CASES=health,acquire,why UMT_EVAL_OUT=/tmp/eval.json \
// DATABASE_URL=… deno run -A npm:vitest run test/e2e/agent-prompt-eval.real.test.ts
// DATABASE_URL=… bun --bun x vitest run test/e2e/agent-prompt-eval.real.test.ts
// UMT_EVAL_LEGACY_PROMPT 用 `git show main:packages/agent/src/system-prompt.ts > /tmp/legacy-system-prompt.ts` 生成;
// UMT_EVAL_PROMPTS=legacy,layered 控制跑哪几组(缺省两组都跑)。
apps/server/test/e2e/crash-recovery.test.ts
+1 -1
View File
@@ -29,7 +29,7 @@ describe.skipIf(!hasPg)("E4 崩溃恢复(worker 子进程 kill -9)", () => {
const children: ChildProcess[] = [];
const spawnWorker = (): Promise<ChildProcess> => {
const child = spawn(process.execPath, ["run", "-A", WORKER], {
const child = spawn(process.execPath, [WORKER], {
cwd: SERVER_DIR,
env: {
...process.env,
apps/server/test/e2e/fixtures/worker.ts
+8 -8
View File
@@ -11,17 +11,17 @@ import { startWorker } from "../../../src/worker/index.ts";
import { buildModules } from "./modules.ts";
import { crashScript } from "./scripts.ts";
const url = Deno.env.get("DATABASE_URL") ?? "postgres://umt:umt@localhost:55433/umt";
const biz = Deno.env.get("BIZ_SCHEMA")!;
const ow = Deno.env.get("OW_SCHEMA")!;
const sessionDir = Deno.env.get("SESSION_DIR")!;
const url = process.env.DATABASE_URL ?? "postgres://umt:umt@localhost:55433/umt";
const biz = process.env.BIZ_SCHEMA!;
const ow = process.env.OW_SCHEMA!;
const sessionDir = process.env.SESSION_DIR!;
await import("@umt/db"); // 注册 int8 解析器
const db = new Kysely<DB>({
dialect: new PostgresDialect({
pool: new pg.Pool({ connectionString: url, max: 8, options: `-c search_path=${biz},public` }),
}),
});
const logger = createLogger({ level: Deno.env.get("E2E_LOG_LEVEL") === "debug" ? "debug" : "warn" });
const logger = createLogger({ level: process.env.E2E_LOG_LEVEL === "debug" ? "debug" : "warn" });
const config = ConfigSchema.parse({
DATABASE_URL: url,
SECRET_MASTER_KEY: "e2e-master-key-0123456789",
@@ -43,10 +43,10 @@ const worker = await startWorker({
agentConcurrency: 2,
},
});
console.log(`worker ready pid=${Deno.pid}`);
Deno.addSignalListener("SIGTERM", async () => {
console.log(`worker ready pid=${process.pid}`);
process.on("SIGTERM", async () => {
await worker.stop().catch(() => {});
await booted.engine.stop().catch(() => {});
await db.destroy().catch(() => {});
Deno.exit(0);
process.exit(0);
});
apps/server/test/e2e/websearch.test.ts
+1 -1
View File
@@ -52,7 +52,7 @@ const mockFetch = (async (input: string | URL | Request, init?: RequestInit) =>
if (url.href === NOTICE_URL)
return new Response(NOTICE_HTML, { headers: { "content-type": "text/html; charset=utf-8" } });
return new Response("not found", { status: 404 });
}) as typeof fetch;
}) as (input: RequestInfo | URL, init?: RequestInit) => Promise<Response>;
const resolveHost = async (host: string) => (host.endsWith("example.org") ? ["93.184.216.34"] : ["203.0.114.10"]);
apps/server/test/fixtures/sse-bus-proc.ts
Vendored
+5 -5
View File
@@ -1,14 +1,14 @@
// SSE 多进程测试用子进程:sub 模式订阅并逐行输出收到的事件 JSON,收满 n 条退出;pub 模式发布 n 条后退出。
// 用法:deno run -A test/fixtures/sse-bus-proc.ts <sub|pub> <schema> <channel> <n>
// 用法:bun test/fixtures/sse-bus-proc.ts <sub|pub> <schema> <channel> <n>
import type { DB } from "@umt/db";
import { Kysely, PostgresDialect } from "kysely";
import pg from "pg";
import { PgEventBus } from "../../src/http/sse-pg.ts";
import { createLogger } from "../../src/logger.ts";
const [mode, schema, channel, nRaw] = Deno.args;
const [mode, schema, channel, nRaw] = process.argv.slice(2);
const n = Number(nRaw);
const url = Deno.env.get("DATABASE_URL") ?? "postgres://umt:umt@localhost:55433/umt";
const url = process.env.DATABASE_URL ?? "postgres://umt:umt@localhost:55433/umt";
const db = new Kysely<DB>({
dialect: new PostgresDialect({
pool: new pg.Pool({ connectionString: url, max: 2, options: `-c search_path=${schema},public` }),
@@ -32,12 +32,12 @@ if (mode === "sub") {
console.log("READY");
const timeout = setTimeout(() => {
console.error("timeout");
Deno.exit(3);
process.exit(3);
}, 20_000);
await done;
clearTimeout(timeout);
} else {
for (let i = 0; i < n; i++) await bus.publish("test.ping", { i, pid: Deno.pid });
for (let i = 0; i < n; i++) await bus.publish("test.ping", { i, pid: process.pid });
}
await bus.stop();
await db.destroy();
Loaded 100 of 268 files, more files were not shown because too many files have changed in this diff. Show more