test(porting): differential tests against MoviePilot Rust-accelerated parsers
Snapshot differential tests (known deviations registered, current state passes): - title/path recognition: 216 samples vs MP MetaInfo/MetaInfoPath - filter rules: 42 rules x 28 torrents vs MP FilterModule - RSS/indexer parsing: 15 feeds, 72 indexer samples - zhconv: 69 samples Generator: scripts/mp-sync/gen-differential.py; report: docs/porting/mp-rust-logic-parity.md Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
rootandClaude Opus 5.5 committed
1 parent
b86f75c011
commit
591cc78c7d
44 files changed
+32206
No files matched your search
@@ -0,0 +1,222 @@
|
||||
# MoviePilot Rust 加速函数:产品逻辑迁移核对(差分测试)
|
||||
|
||||
> 上游 `/opt/moviepilot-plus/upstream/MoviePilot` @ `e6820a8`,`moviepilot_rust` 0.3.3。函数清单见 `moviepilot-rust-transfer.md` §1。
|
||||
> 方法:用 MP 自己的 venv 调用上游实现生成期望值(`scripts/mp-sync/gen-differential.py`),我们的 TS 实现跑同样的输入,逐字段比较。已知差异写进快照的 `deviations`,测试断言"实际差异集合等于登记集合"。所以现在测试是通过的;以后补迁消掉一条差异,测试会失败,提醒删掉对应登记。
|
||||
> 差异分类:**一致** / **intentional**(故意偏离,README 或 porting 文档有记录)/ **missing**(漏迁)/ **our_bug**(疑似我们的 bug)/ **mp_bug**(疑似 MP 的缺陷,我们的结果更好)/ **representation**(只是表示方式不同)/ **not_expressible**(Slot 表达不了)。
|
||||
|
||||
## 0. 结论
|
||||
|
||||
| Rust 函数 | 业务链路 | 差分规模 | 一致率 | missing / our_bug / mp_bug |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `parse_metainfo_fast` / `parse_metainfo_path_fast` | 搜索/订阅/RSS 的标题识别;整理规划(MetaInfoPath);过滤器按集算体积 | 216 条(标题 186、路径 30)× 18 字段 | 字段级 92.9%;扣掉 intentional/representation/mp_bug 后 97.5%;整条完全一致 84 条(38.9%),产品等价 177 条(81.9%) | 14 条 / 27 条 / 28 条样本 |
|
||||
| `find_metainfo_fast` | 标题中的显式 ID 标签 | 11 条带标签 | 8/11 | 2 / 0 / 1 |
|
||||
| `parse_filter_rule_fast` / `filter_torrents(_with_trace)_fast` | 搜索、订阅匹配、洗版、工作流过滤、下载排序 | 43 条规则;28 种子 × 23 条可完整表达规则 = 644 case | 95.5%(615/644);另有 3 条只能近似表达的规则,84 case 里 31 处不一致,全是能力缺口 | 26 / 14 / 7(另 not_expressible 13) |
|
||||
| `parse_rss_items_fast` | 订阅刷新(站点 RSS 模式)、工作流 fetch_rss、插件 | 15 份 Feed / 36 条,245 字段 | 95.5%(234/245) | 0 / 0 / 4(另 intentional 12) |
|
||||
| `parse_indexer_torrents_fast` / `parse_indexer_subtitles_fast` | 站点搜索/浏览(SiteSpider.parse);字幕搜索 | 72 样本 / 127 行,2108 字段 | 98.1%(2069/2108) | 1(字幕分支)/ 4 / 35 |
|
||||
| `zhconv_fast` | 识别查询名简体化、中文别名判定、音乐元数据 | 69 条 | 84.1%(58/69) | 10 / 0 / 1 |
|
||||
| `jieba_cut` | 仅用于"整理历史按标题模糊搜索"(含 Agent 工具) | 静态对照 | — | 不需要迁 |
|
||||
|
||||
MP 自身两条路径也不一致:标题识别的 Python 实现和 Rust 快路径在 4 条样本上输出不同(`meta-mp.json#rustMismatch`:T-06、A-03、Z-06、Z-08);索引解析在 **40/72** 个样本上不同(`indexer.json` 的 `mp_rust`);RSS 的 Atom/RDF 命名空间也不同(Python 0 条,Rust 有结果)。过滤器的 Python 和 Rust 交叉核对 0 处不一致。我们的对照基准一律用 **Python 参考实现**。
|
||||
|
||||
## 1. 产物与复现
|
||||
|
||||
| 文件 | 内容 |
|
||||
| --- | --- |
|
||||
| `scripts/mp-sync/gen-differential.py {meta,zhconv,filter,rss,indexer,all}` | 生成器,必须用 `$MP_ROOT/.venv/bin/python` 运行。只读调用上游,不复制上游源码 |
|
||||
| `packages/domain/test/differential/meta-{samples,mp,deviations}.json` + `meta-map.ts` + `meta.test.ts` | 标题识别。`meta-map.ts` 把双方输出规范化到同一形式(名称忽略大小写和标点、来源归到 REMUX/BluRay/WEB-DL/…、HDR 集合、编码族、音频去分隔符、发布组集合) |
|
||||
| `packages/domain/test/differential/filter-cases.json` + `filter.test.ts` | 过滤规则 |
|
||||
| `packages/domain/test/differential/zhconv-*.json` + `zhconv.test.ts` | 繁简转换 |
|
||||
| `packages/adapters/test/differential/{rss,indexer}.{json,test.ts}`、`diff-util.ts`、`inputs/**`、`export-inputs.ts` | RSS 和索引解析(§4、§5)。生成前先跑 `deno run -A packages/adapters/test/differential/export-inputs.ts` 导出 testkit fixture;生成器把时区固定为 UTC,并冻结 dateparser 的 now,重跑结果逐字节相同 |
|
||||
|
||||
样本来源:我们已有的 107 条(`release-parser.test.ts` 91 条标题加 16 条路径)、MP `tests/cases/meta.py` 的 76 条(只用标题字符串作输入语料,属于事实数据;这份语料的来源**待登记进 ledger**)、补充 33 条(综艺、动漫、多集、合集、国配、特效字幕、显式 ID)。
|
||||
|
||||
## 2. parse_metainfo_fast / parse_metainfo_path_fast(标题与路径识别)
|
||||
|
||||
**用途**:`MetaInfo()` 用于种子标题、站点副标题和文件名,在搜索、订阅匹配、RSS 刷新、过滤器按集算体积(`total_episode`)时调用。`MetaInfoPath()` 用于整理链路(`transfer/request.py _build_path_meta`)。Rust 只是快路径,返回 None 时回退 Python。
|
||||
|
||||
**分组结果**(216 条):
|
||||
|
||||
| 来源 | 条数 | 完全一致 | 产品等价 | 说明 |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| 我们已有样本 | 107 | 46 | 106 | 差异几乎都是 README 登记的故意偏离 |
|
||||
| MP 自带用例 | 76 | 29 | 43 | **our_bug 65 处里有 57 处在这里**,主要是动漫/字幕组命名 |
|
||||
| 补充 | 33 | 9 | 28 | — |
|
||||
|
||||
按字段看一致率:resource_effect 和 edition 100%;part 99.1%;year 98.6%;video_encode 96.8%;audio_encode 96.3%;begin_episode 89.8%;type 88.9%;begin_season 84.3%;name 81.9%;resource_team 72.7%(59 处差异中 52 处是故意兜底)。`customization` 双方都没配置,不参与比较。
|
||||
|
||||
### 2.1 our_bug(27 条样本,按根因归并)
|
||||
|
||||
| # | 输入(节选) | MP | 我们 | 建议 |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| B1 | `【爪爪字幕组】★7月新番[欢迎来到…教室 第二季/…]`(MP-03/17/18/20/32/45/56/63) | cn=`欢迎来到实力至上主义的教室` | cn=`★7月新番欢迎来到…` / `国漫寻剑` / `22年日剧据…` | 剥离动漫前缀标签:`★N月新番★`、`N年日剧`、`国漫`、`新番` |
|
||||
| B2 | `[桜都字幕组] 异世界归来的舅舅 / Isekai Ojisan [01][1080p]`(MP-44/45、Z-07/27) | E=1,T=TV | en=`Isekai Ojisan 01`,无集号 | 名称后紧跟的 `[NN]` / `[NN][分辨率]` 识别为集号(我们已支持 `[Group][Name][05]`,不支持 `Name [05]`) |
|
||||
| B3 | `[秋叶原冥途战争][Akiba Maid Sensou][2022]…[LeagueWEB]`(MP-05/13/27/57) | grp=LeagueWEB | grp=`秋叶原冥途战争` | 开头方括号含中文或能解析成片名时不当作组;末尾 `[已知组]` 优先 |
|
||||
| B4 | `[Caligula.The.Ultimate.Cut.2023.2160p…-BHYS@OurBits][DIY中字原盘][…].iso`(MP-74) | 正常解析出 en、y、2160p、UHD BluRay、HEVC、DTS-HD MA 5.1 | 名称取成 `DIY`,技术字段全丢 | 迁 MP 的 `BRACKET_DOT_TITLE_RE`:方括号内是"点分隔发布名 + 年份 + 资源词"时去括号保留内容 |
|
||||
| B5 | `The Long Season 2017 2160p…`(MP-01) | en=`The Long Season` y=2017 | 第 2017 季,年份被吞 | `Season` 后的数字 ≥1900 时按年份处理(MP 限制 Season 后 <3 位) |
|
||||
| B6 | `2022.8.2.Twelve.Monkeys.1995…`(MP-52),父目录 `…Collection.20240501.FRDS`(MP-73) | 电影 | 按 airDate 判成 TV | 标题**开头**的日期、以及和年份同时出现的日期,不当作播出日期;父目录的 airDate 不参与类型合并 |
|
||||
| B7 | `[YTS.MX] …`(MP-76)、`[Hall_of_C] 诛仙 …`(MP-60) | 名称干净 | 组名混进英文名 | 开头方括号里是单个 token(含 `.`/`_`)时当作组名剥离 |
|
||||
| B8 | `组长女儿与照料专员 / 组长女儿与保姆 …`(MP-21);`传颂之物/Utawarerumono/うたわれるもの`(MP-55) | 取其一 | 两个中文名被拼接,或三段没拆开 | `/` 分隔的多名称按语言拆分,中文取一个 |
|
||||
| B9 | `哆啦A梦：大雄的宇宙小战争`(MP-06) | cn 含 A | `A` 被拆成英文名 | 中文词中夹的单个拉丁字母并入中文名 |
|
||||
| B10 | `[Gal to Kyouryuu][02][BDRIP]`(MP-24) | en=`Gal To Kyouryuu` | en=`BDRIP` | 资源词不能当名称 |
|
||||
| B11 | `DTS-HD MA5.1`(MP-48)、`E13-14`(MP-51)、`【极影字幕社】…GB`(MP-63) | `DTS-HD MA 5.1`、E13-14、组=极影字幕社 | `DTSHD`、只有 E13、组缺失且 GB 进了名称 | 各补一条词元规则 |
|
||||
|
||||
### 2.2 missing(14 条样本)
|
||||
|
||||
| # | 输入 | MP | 我们 | 建议 |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| M1 | `{[tmdbid=94997;type=tv;s=1;e=3-4]}House.of.the.Dragon…`(Z-31) | tmdb:94997,S1,E3-4,TV | 整块进名称 | 迁 `{[k=v;…]}` 显式身份块(也见 §3) |
|
||||
| M2 | `[bangumiid=400602]`(Z-32) | bangumi:400602 | 无 | ExplicitId 增加 bangumi/anilist |
|
||||
| M3 | `#13`(MP-25)、`[TV 04]`(MP-57)、`西部世界 12.mkv`(MP-29)、`S01 13-14`(MP-50) | 能识别集号 | 集号为空 | 补充 4 种集号写法 |
|
||||
| M4 | `Mai Xiang S01 …`,副标题 `麦香 \| 全36集`(MP-64) | cn=麦香 | 无中文名 | 迁"英文名是拼音时从副标题第一个词取中文名" |
|
||||
| M5 | `[1080]`(MP-05)、`[5.1]`(MP-12) | 1080p、5.1 | 空 | 补方括号内的裸分辨率和裸声道 |
|
||||
| M6 | `iP`(T-13)、`B-Global`(MP-10) | BBC iPlayer、B-Global | 空 | 流媒体平台表补齐(MP 有约 270 条) |
|
||||
| M7 | `cXcY@FRDS`(MP-49/54)、`指环王三部曲合集`(Z-12) | 两个组都识别;去掉"合集" | 只有 FRDS;保留"合集" | 支持 `￡` 分隔符;"合集/三部曲合集"列为干扰词 |
|
||||
|
||||
### 2.3 intentional(与 README 偏离表一致,125 处)
|
||||
|
||||
未知发布组兜底 52 条;第 N 期记为 `issue` 12 条;有集号无季号时推定第 1 季 23 条;播出日期 → airDate 且判为 TV 6 条;路径合并(文件无剧集信号、目录是剧集)3 条;辅助词字幕文件(P-07);附加内容目录(P-05);`CD1`/`D02` 作为 part;`第八季 全12期` 保留季号;先导片。**每一条都能在 `packages/domain/README.md` 的偏离表或 `moviepilot-metadata-recognition.md` 里找到依据,没有发现未经记录的故意偏离。**
|
||||
|
||||
### 2.4 mp_bug(28 条样本,保留我们的行为)
|
||||
|
||||
- MetaAnime(anitopy 路径)不输出来源和编码(15 条)。
|
||||
- 不认 `1x05`、`S01D02`、`Season.2`、`BD50`、`{doubanid=}`。
|
||||
- 丢中文名(MP-13/44/55、Z-07)。
|
||||
- 把 `[Sub]` 当成名称。
|
||||
- `AC3￡` 没切开,音频丢失。
|
||||
- 把 `7³` 拼进音频。
|
||||
- `AVC-8bit` 的结果 Python 和 Rust 不一致(Z-06)。
|
||||
|
||||
representation 共 10 处:`name` 的取法(我们有中文名就取中文,MP 只在中文名全是汉字时才取);`HMAX` 映射成 Max 还是 HBO Max;`2Audio(s)`;`flac_ac3` 记成一条还是两条。
|
||||
|
||||
## 3. find_metainfo_fast(显式 ID 标签)
|
||||
|
||||
**用途**:从标题里剥离 `[tmdbid=…]`、`{tmdb-…}`、`{[k=v;…]}`、`[bangumiid=…]`,得到 media_source/media_id 以及季、集、类型。在 `_prepare_meta_input` 里调用。差分并入 §2 的 `media_id` 字段:8 条 Emby 风格的 TMDB 标签(`[tmdbid=]`、`[tmdbid-]`、`[tmdb=]`、`{tmdb=}`、`{tmdb-}`)**全部一致**。不一致的 3 条:M1、M2 是漏迁;`{doubanid=}` 是 mp_bug,我们能认。另外我们多支持 imdb 和 tvdb,MP 不认。**建议**:把 `{[…]}` 块(含 `type/s/e/g`)和 bangumi 补上。按不变量 1,它们仍只作为证据加一级,不直接确认身份。
|
||||
|
||||
## 4. parse_rss_items_fast(RSS/Atom)
|
||||
|
||||
**用途**:`RssHelper._parse_impl`(`application/rss.py`)先走 Rust(最多 1001 条,截到 1000),返回 None 时回退 lxml。调用方:`TorrentsChain.rss(domain)`(订阅刷新的站点 RSS 模式)、工作流 `fetch_rss`、插件。解析失败时触发 `__renew_rss_url` 重新生成地址。
|
||||
|
||||
**MP 字段语义**:title 取第一个后代 `title` 的文本(不 strip,为空就跳过);description 取 `description|summary`;link 取文本,没有就取 href;enclosure 取 `@url`,缺失回退到 link;**size 只取 `enclosure@length` 的纯数字**(description 里的大小、Torznab/nyaa 的 size 都不读);pubDate/published/updated 用 `dateutil.parse` 后 `astimezone(None)`(不带时区的按宿主本地时区);`dc:creator` 输出为 nickname;不输出 guid 和 category。促销、HR、做种这些字段 RSS 路径都没有。
|
||||
|
||||
**差分**:15 份输入(testkit 2 份、手写 13 份),MP Python 解析出 36 条,逐字段比较标题、描述、链接、enclosure、大小、pubdate、作者,共 245 个字段,一致 234 个。登记 16 条(`packages/adapters/test/differential/rss.json#deviations`):
|
||||
|
||||
| 输入 | MP | 我们 | 分类 | 建议 |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| 带 xmlns 的标准 Atom、RSS 1.0(RDF) | Python 0 条(Rust 有) | 3 条 / 1 条 | mp_bug | 保持 |
|
||||
| 纯文本"RSS 已过期"提示 | 在"不以 `<` 开头"处就提前返回,过期判断走不到,续期永远不触发 | `FeedError('expired')` | mp_bug | 保持;续期逻辑我们自己做 |
|
||||
| 条目内 `<source><title>` 排在前面 | 取到来源标题 | 取直接子元素 `title` | mp_bug | 保持 |
|
||||
| enclosure 带 passkey | 原样 | `[REDACTED]` | intentional(SA5) | — |
|
||||
| 带 DTD | `[]` | 整份拒绝 `forbidden_dtd` | intentional | — |
|
||||
| 值里有换行缩进、`length=" 2048 "` | 保留空白,大小为 0 | trim 后大小为 2048 | intentional | 在 `rss/README` 补登记 |
|
||||
| ` ` 实体 | 在此截断,标题变成 "Nbsp" | 保留字面量 | intentional | P3:补 HTML 命名实体解码表 |
|
||||
| 未闭合标签 | 容错恢复 2 条 | 1 条 | intentional | — |
|
||||
| pubDate `… CST`、纯日期 `2026-09-26` | 按宿主本地时区解释 | 按美国中部 -6 / UTC 午夜 | intentional(两边都可能错) | P2:不带时区的时间按站点 `timezone` 解释 |
|
||||
|
||||
RFC822、ISO、无秒、星期写错这几类两边一致;中文日期、"3 小时前"、纯数字两边都不解析。
|
||||
|
||||
## 5. parse_indexer_torrents_fast / parse_indexer_subtitles_fast(站点列表页)
|
||||
|
||||
**用途**:`SiteSpider.parse`(`modules/indexer/spider/__init__.py`)先走 Rust,返回 None 时回退 PyQuery 的 `get_info`。链路是 `IndexerModule.search_torrents`(关键词搜索)和 `refresh_torrents`(浏览 `browse.list/fields`);字幕分支 `search_type=="subtitles"` 走 `get_subtitle_info` 和 `__fill_subtitle_ids`,由 `/api/search` 字幕接口经 `SearchChain.search_subtitles_*` 调用。
|
||||
|
||||
**MP 字段语义**:size 用 `parse_size`(二进制单位,Python 银行家舍入);做种、下载、完成数取 `/` 前的部分,去掉逗号,不是纯数字时为 0;促销 case 按声明顺序取第一个命中(另有 selector 模式,用 `int()`);freedate 是原样文本;HR 配置了就输出 bool;category 投影成电影/电视剧/音乐/未知;pubdate 是站点墙上时间;imdbid 用 `re_search`;labels 是列表。
|
||||
|
||||
**能否独立调用**:`SiteSpider(indexer).parse(html)` 和 `RssHelper` 注入假 HTTP 端口后可以离线调用。**但 MP 的内置站点定义调不出来**:它们在 Cython `app/application/site/sites.*.so` 和加密的 `user.sites.v3.bin` 里,未认证时 `get_indexer()` 返回 `{}`,`SitesHelper()` 还依赖数据库事务执行器。所以双方共用我们的 `definitions/*.json`(与 MP 同一 schema,来自 MovieClaw)作为输入,**MP 真实站点的选择器只能静态对照**。
|
||||
|
||||
**差分**:72 个样本(21 个 NexusPHP 定义 × 3 个列表页、2 个登录页、3 个手写 DSL 配置、1 个字幕页),MP 解析出 127 行,比较 title、description、size、seeders、peers、grabs、促销系数、freedate、HR、分类、pubdate、链接、imdbid、labels 共 2108 个字段,一致 2069 个。登记 41 条(`indexer.json#deviations`):
|
||||
|
||||
| 输入 | MP | 我们 | 分类 | 建议 |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| hdarea、hddolby、hdfans 等 9 个定义的列表选择器命中表头行 | Jinja 把 None 渲染成标题 "None" | 空串 | mp_bug ×32 | 两边都会产出垃圾行:**收紧这 9 个定义** |
|
||||
| 相对时间 "2天"、"3小时前" | 用 dateparser 按当前时刻换算,结果随抓取时间漂移 | 不写 pubdate | mp_bug | 保持(生成时冻结了 now) |
|
||||
| 促销 selector 取到 "0.5" | `int()` 报错,整行丢失 | 保留 | intentional(SA4) | — |
|
||||
| `"2.5 B"` | 2(银行家舍入) | 3 | our_bug | P3:10 分钟 |
|
||||
| 行元素自身带 `free_bg`/`sticky` class | PyQuery 查询包含行自身,得到 dl=0、hr=true、labels | `.find` 只查后代 | our_bug ×3 | **P1:先决定跟 Python 还是 Rust**(MP 的 Rust 和我们一样),再改 `spider/spider.ts` 或在 README 登记 |
|
||||
| 字幕页 | 解析出 3 条 | 没有这个分支 | missing | P1:补字幕分支,期望数据已在 `indexer.json` |
|
||||
|
||||
**静态发现(差分之外)**:`table.torrents > tr:has(...)` 这种写法在显式带 `<tbody>` 的页面上,我们和 MP Python 都取到 0 行,MP Rust 能取到 5 行。受影响的定义有 agsvpt、chdbits、hdhome、keepfrds、ourbits、pter、pthome、ssd、tjupt。建议在 `scripts/mp-sync/movieclaw-overrides/` 统一补 `, table.torrents > tbody > tr` 这一支(audiences 已经这样写)。
|
||||
|
||||
## 6. parse_filter_rule_fast / filter_torrents(_with_trace)_fast(过滤规则)
|
||||
|
||||
**调用点**:`FilterModule.filter_torrents(rule_groups, torrents, mediainfo)`。先走 Rust,它返回 None 时回退 Python(规则解析在 `app/application/rules.py`,基于 pyparsing)。
|
||||
|
||||
- **搜索**:`chain/search/result.py`,按站点过滤。规则组用 `SearchFilterRuleGroups`。
|
||||
- **订阅匹配 / RSS 刷新**:`chain/subscribe/match.py`,每个种子单独过滤。规则组依次取 `subscribe.filter_groups`、`BestVersionFilterRuleGroups`(洗版)、`SubscribeFilterRuleGroups`。洗版时 `pri_order <= current_priority` 的种子会被拒;`current_priority == 100` 表示已完成。
|
||||
- **其他调用**:订阅搜索、工作流 filter 动作、规则测试接口、下载排序(把 pri_order 当排序键)。
|
||||
- **规则组选取**:按 `media_type` 和 `category` 选组;多个组之间是 AND 关系。
|
||||
|
||||
**语法**:用 `>` 切出优先层,第 1 层 pri_order=100,依次递减。层内运算优先级为 `!` > `&` > `|`,可以加括号。规则名不存在时视为不匹配。单条规则的判定顺序是:tmdb 条件命中就直接放行;然后在 `title+description+labels` 上跑 include/exclude 正则;接着检查 `size_range`(单位 MB,**按每集算**,即总大小除以 MetaInfo 得到的 `total_episode`)、`seeders`、`downloadvolumefactor`、`publish_time`(延迟 N 分钟)。
|
||||
|
||||
**内置关键字 → Slot**:
|
||||
|
||||
| 能表达 ✔ | 近似 ~ | 不能 ✗ |
|
||||
| --- | --- | --- |
|
||||
| `4K/1080P/720P` → `hard.resolutions`;`BLU/REMUX/WEBDL` 及其取反 → `sources` 允许列表加 `unknownPolicy`;`BLURAY & !BLU` → `requireEncode` | `CNSUB` → `requiredSubtitles:["zh"]`(缺华语片由 tmdb 放行这条);体积区间用总量而非每集;`A > B` 用多个 Slot 或 `rankForSlot`(按来源等级优先,不按层级) | `UHD`、`H265/H264`(只有软偏好)、`HDR/DOLBY/SDR`(只有软偏好)、`ATMOS/60FPS/3D`、`SPECSUB`(特效字幕)、`CNVOI/HKVOI`(国配/粤配;我们把"国语"当音轨词剔掉)、`GZ`(官方标签)、`FREE`、做种数、`publish_time`、跨字段 OR、音乐的 13 条规则 |
|
||||
|
||||
**43 条规则样例的结论**:✔ 22、~ 6、✗ 15。完整表格见 `filter-cases.json#rules`,其中 `verdict` 和 `note` 是逐条结论。有代表性的几条:
|
||||
|
||||
- `4K & !REMUX > 1080P & WEBDL` ✔,用两层 Slot 表达。
|
||||
- `SPECSUB & CNSUB & !BLU` ~,SPECSUB 那部分丢了。
|
||||
- `!DOLBY & !REMUX` ~,只能表达 `!REMUX`。
|
||||
- `CNSUB | 4K` ✗,属于跨字段 OR。
|
||||
- `FREE`、`SEED5`、`GZ` ✗,这些是站点元数据。
|
||||
|
||||
**差异**(不一致共 61 处已登记,完整在 `filter-cases.json#deviations`):
|
||||
|
||||
| 输入 | MP | 我们 | 分类 | 建议 |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `2160p UHD Blu-ray HEVC … -BeyondHD`、`1080p Blu-ray AVC DTS-HD MA` | BLU(原盘) | BluRay 压制(isEncode) | our_bug ×12 | Blu-ray 加 AVC/HEVC/VC-1、且没有 x26x 时,判为 `BluRay-Disc` |
|
||||
| 副标题 `CHS/ENG` | 中字 | 不识别 | our_bug ×2 | 字幕词元补 CHS/CHT/GB/BIG5 |
|
||||
| `S03 全10集 60G` 配体积规则 | 按每集 6G 判 | 按总量 60G 判 | missing ×8 | Slot 加 `sizeBasis: per_episode` |
|
||||
| 华语片 + CNSUB | tmdb 放行 18 个种子 | 拒绝 | missing | 允许条件依赖媒体上下文 |
|
||||
| `2160p.HDR.HEVC-NOSRC`(没有来源) | 误命中 BLU | 来源未知 | mp_bug ×5 | 保持我们的做法 |
|
||||
| 副标题只有 `国语` | 算作中字 | 当作音轨 | mp_bug ×2 | 保持;以后另加 audioLang 条件 |
|
||||
| 单 Slot 的 `4K > 1080P > 720P` | 4K 总排在前面 | 先比来源等级,共 44 个逆序对 | intentional | 要复现 MP 的语义,就用多层 Slot 或优先级阶梯 |
|
||||
|
||||
**trace 对比**:MP 的 `filter_torrents_with_trace_fast` 返回 `([(index, pri_order)], [str])`,trace 是中文自由文本,只写进 debug 日志。例如 `种子 X 不包含任何项 [...]`、`大小 5G 不在范围 10000-50000MB`、`做种人数 3 小于 5`、`规则 X 不存在`。它不区分"字段未知"和"不满足";OR 分支里失败的原子也会被打印出来。我们的 `SlotEvaluation` 是结构化的:`violations[{field, code, expected, actual}]` 会列出全部违规,`unknown_field` 单独给出,accept 时带 `tier/softRank/reasons`。**缺两样**:命中的是第几层(对应 pri_order),以及逐层的失败原因。
|
||||
|
||||
## 7. zhconv_fast(繁简转换)
|
||||
|
||||
**用途**:`app/foundation/text.py convert()` 直接调用 `moviepilot_rust.zhconv_fast`,**没有 Python 回退**,原生扩展是硬依赖。调用点:
|
||||
|
||||
- TMDB/豆瓣/IMDb 识别时,把 `cn_name` 简体化后加入查询名(`themoviedb/__init__.py:285` 等)。
|
||||
- TMDB 中文别名的"是否已是简体"判定(`tmdbapi.py:981`)。
|
||||
- MetaAnime 分类标签判定。
|
||||
- 音乐元数据简体化。
|
||||
|
||||
用的是 MediaWiki 词表,**有词组级转换**。
|
||||
|
||||
**差分**:69 条(动漫、港台片名、一对多字)中 58 条一致。
|
||||
|
||||
- missing 10 条:其中 6 条是字表缺字(迴、滾、魷、瑯、臥、駭);4 条是一对多字(乾杯、瞭解、頭髮、理髮),我们刻意不收,MP 靠词组表处理。
|
||||
- mp_bug 1 条:`慶餘年` 转成了 `庆馀年`,官方简体片名是 `庆余年`。
|
||||
|
||||
**我们的替代**:`identity/zh-map.ts` 大约 930 对单字映射,只在 `normalizeTitle` 里用于召回比较。召回本来就会再经过 pg_trgm 相似度和别名表,一两个字不同不会直接失败。
|
||||
**建议**:暂不引入 `opencc-js`(约 1 MB 词典,MIT 许可,需要按需加载)。先把字表补到覆盖 TMDB 中文标题的常用繁体字(把本差分的 missing 清零,约 0.5 小时)。等到要把"简体化后的查询名"作为 TMDB 查询词时(对应 MP `_prepare_search_names`)再评估 opencc。它能提供词组转换和台湾/香港用词转换(如 蜘蛛人 → 蜘蛛侠),代价是包体积和冷启动。
|
||||
|
||||
## 8. jieba_cut(中文分词)
|
||||
|
||||
**用途**:只有两处,都是"整理历史按标题模糊搜索":`application/history.py:712` 用 `"%".join(jieba_cut(title))` 生成 SQL LIKE 模式,另一处是 Agent 工具 `query_transfer_history`。**不参与识别、匹配和过滤。**同样没有 Python 回退。
|
||||
**我们的替代**:资产和别名检索用 pg_trgm(`0004_content`、`0006_sources` 迁移)加 `normalizeTitle`。三元组相似度天然不依赖分词,对中文短标题的召回不比"分词后 LIKE"差。
|
||||
**结论**:**不引入 `jieba-wasm`**。它的代价是 wasm 约 4 MB、词典加载 100 ms 级,而收益只落在历史搜索这一个次要功能上。以后如果要做全文检索,优先考虑 PG 的 `zhparser` 或 `pg_bigm`,而不是在应用层分词。
|
||||
|
||||
## 9. 补迁清单(按优先级)
|
||||
|
||||
| 优先级 | 项 | 涉及文件 | 工作量 | 消除的登记 |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| P0 | 原盘判定:Blu-ray 加 AVC/HEVC/VC-1 且无 x26x → `BluRay-Disc`;`BDMV/BD25/50/66/100`、`COMPLETE BLURAY` | `release-parser/parse.ts` | 0.5 天 | filter our_bug ×12 |
|
||||
| P0 | 动漫/字幕组命名:前缀标签剥离(B1)、`Name [NN]` 集号(B2)、开头方括号是片名还是组名(B3、B7)、`/` 多名称拆分(B8) | `release-parser/parse.ts`、`release-groups.ts` | 1.5 天 | meta our_bug 约 40 处 |
|
||||
| P0 | `[点分隔发布名][…]` 去括号(B4);`Season 2017` 按年份处理(B5);开头日期和目录日期不当 airDate(B6) | `parse.ts`、`path.ts` | 0.5 天 | meta our_bug 约 15 处 |
|
||||
| P1 | Slot 硬条件扩展:`excludeSources`、hdr require/exclude、codec 硬条件、`sizeBasis: per_episode`、`requireUhd` | `slot/types.ts`、`slot/evaluate.ts` | 1 天 | filter missing ×8,规则可表达数 22 → 约 30 |
|
||||
| P1 | 字幕和音轨语言:CHS/CHT/GB/BIG5 识别为中字;国语、粤语作为 `audioLanguages`;特效字幕标记 | `parse.ts`、`types.ts`、Slot | 1 天 | filter our_bug ×2,SPECSUB/CNVOI 可表达 |
|
||||
| P1 | 显式身份块 `{[tmdbid;type;s;e;g]}`,以及 bangumi/anilist | `parse.ts`(`explicitIds`) | 0.5 天 | meta missing ×2 |
|
||||
| P1 | 优先级阶梯:`SlotSpec.tiers[]`,评估结果带 `level`,洗版时按 level 比较 | `slot/*`、server `acquisition.ts` | 1–1.5 天 | filter R05 intentional |
|
||||
| P2 | 集号写法 `#13`、`[TV 04]`、`名称 12.mkv`、`S01 13-14`;裸 `[1080]`、`[5.1]`;`MA5.1`、`E13-14` | `parse.ts` | 0.5 天 | meta missing/our_bug 约 10 处 |
|
||||
| P2 | 站点元数据条件:minSeeders、freeOnly、publishDelay、labels(需要改 `evaluateCandidate` 的签名) | `slot/*`、server | 1 天 | FREE/SEED/GZ 可表达 |
|
||||
| P2 | 流媒体平台表补齐;`￡` 多组分隔;"合集"作干扰词;拼音英文名从副标题取中文名 | `parse.ts`、`release-groups.ts` | 0.5 天 | meta missing 约 6 处 |
|
||||
| P1 | 站点定义:9 个定义补 `tbody` 分支,9 个定义收紧表头 | `movieclaw-overrides/*.json` | 0.5 天 | indexer mp_bug ×32 的根因 |
|
||||
| P1 | 选择器是否包含行元素自身的语义(case/hr/labels) | `sites/spider/spider.ts` | 0.5 天 | indexer our_bug ×3 |
|
||||
| P1 | 字幕页解析分支(`get_subtitle_info`、`__fill_subtitle_ids`) | `spider/spider.ts`、`spider/schema.ts`、adapter 接口 | 1–1.5 天 | indexer missing ×1 |
|
||||
| P2 | RSS 中不带时区的时间按站点时区解释 | `rss/parse.ts` | 0.5 天 | rss intentional ×2 |
|
||||
| P3 | zh-map 补字(§7);`parseSizeMp` 改银行家舍入;RSS 补 HTML 命名实体 | `identity/zh-map.ts`、`sites/spider`、`rss/parse.ts` | 2 小时 | zhconv missing ×6、indexer our_bug ×1 |
|
||||
|
||||
**不迁**:MetaAnime 不输出来源和编码的行为(mp_bug);期号当集号、日期删除、未知组置空(与 README 偏离表一致);jieba。
|
||||
@@ -0,0 +1,89 @@
|
||||
/**
|
||||
* MP 差分测试共用工具:读取 scripts/mp-sync/gen-differential.py 生成的期望 JSON,登记偏差集合与实际偏差集合比对。
|
||||
* 偏差 key:`<sampleId>.status` / `<sampleId>.count` / `<sampleId>#<行号>.<字段>`。
|
||||
*/
|
||||
import { readFileSync, writeFileSync } from "node:fs";
|
||||
import { dirname, join } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
export const HERE = dirname(fileURLToPath(import.meta.url));
|
||||
|
||||
export type DeviationKind = "intentional" | "missing" | "our_bug" | "mp_bug";
|
||||
export interface Deviation {
|
||||
kind: DeviationKind;
|
||||
note: string;
|
||||
/** 仅在条件成立时才出现的偏差;local_tz_not_utc = 宿主本地时区偏移非 0 */
|
||||
condition?: "local_tz_not_utc";
|
||||
}
|
||||
|
||||
function conditionHolds(d: Deviation): boolean {
|
||||
if (d.condition === "local_tz_not_utc") {
|
||||
return new Date("2026-09-26T00:00:00").getTimezoneOffset() !== 0;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
export interface DifferentialDoc<S> {
|
||||
kind: string;
|
||||
generatedFrom: Record<string, string>;
|
||||
fields: string[];
|
||||
samples: S[];
|
||||
deviations: Record<string, Deviation>;
|
||||
}
|
||||
|
||||
export function loadDoc<S>(name: string): DifferentialDoc<S> {
|
||||
return JSON.parse(readFileSync(join(HERE, name), "utf8")) as DifferentialDoc<S>;
|
||||
}
|
||||
|
||||
export function readInput(rel: string): string {
|
||||
return readFileSync(join(HERE, rel), "utf8");
|
||||
}
|
||||
|
||||
export interface Mismatch {
|
||||
key: string;
|
||||
mp: unknown;
|
||||
ours: unknown;
|
||||
}
|
||||
|
||||
/** null / undefined 视为同一"缺省";其余按 JSON 结构比较 */
|
||||
export function sameValue(a: unknown, b: unknown): boolean {
|
||||
if (a == null && b == null) return true;
|
||||
return JSON.stringify(a) === JSON.stringify(b);
|
||||
}
|
||||
|
||||
export class FieldCollector {
|
||||
readonly mismatches: Mismatch[] = [];
|
||||
compared = 0;
|
||||
equal = 0;
|
||||
check(key: string, mp: unknown, ours: unknown): void {
|
||||
this.compared++;
|
||||
if (sameValue(mp, ours)) this.equal++;
|
||||
else this.mismatches.push({ key, mp, ours });
|
||||
}
|
||||
}
|
||||
|
||||
/** 实际不一致集合 vs 登记集合:返回两个方向的差集(均为空才通过)。 */
|
||||
export function diffAgainstRegistry(
|
||||
actual: Mismatch[],
|
||||
registry: Record<string, Deviation>,
|
||||
): { unregistered: Mismatch[]; stale: string[] } {
|
||||
const actualKeys = new Set(actual.map((m) => m.key));
|
||||
return {
|
||||
unregistered: actual.filter((m) => !(m.key in registry)),
|
||||
stale: Object.entries(registry)
|
||||
.filter(([k, d]) => !actualKeys.has(k) && conditionHolds(d))
|
||||
.map(([k]) => k),
|
||||
};
|
||||
}
|
||||
|
||||
/** 设置环境变量 DIFF_DUMP_DIR 时把实际不一致写成 <name>.mismatches.json(整理 deviations 用) */
|
||||
export function dumpMismatches(name: string, ms: Mismatch[], stats?: { compared: number; equal: number }): void {
|
||||
const dir = process.env.DIFF_DUMP_DIR;
|
||||
if (dir) {
|
||||
writeFileSync(join(dir, `${name}.mismatches.json`), `${JSON.stringify({ stats, mismatches: ms }, null, 2)}\n`);
|
||||
}
|
||||
}
|
||||
|
||||
export function formatMismatches(ms: Mismatch[]): string {
|
||||
return ms.map((m) => `${m.key}\n mp: ${JSON.stringify(m.mp)}\n ours: ${JSON.stringify(m.ours)}`).join("\n");
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
/**
|
||||
* 把 @umt/testkit 的 mock RSS / NexusPHP fixture 导出成文件,供 MP 差分生成脚本(scripts/mp-sync/gen-differential.py rss|indexer)读取。
|
||||
* 运行:deno run -A packages/adapters/test/differential/export-inputs.ts
|
||||
* 只写 `inputs/rss/tk-*.xml` 与 `inputs/indexer/tk-*.html.txt`;手写样本(hw-*)不受影响。
|
||||
*/
|
||||
import { mkdirSync, writeFileSync } from "node:fs";
|
||||
import { dirname, join } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import {
|
||||
AUDIENCES_FIXTURE_RELEASES,
|
||||
CHDBITS_LIST_PAGE,
|
||||
CHDBITS_LOGIN_PAGE,
|
||||
createMockNexusPhpSite,
|
||||
HHANCLUB_LIST_PAGE,
|
||||
type MockFeedItem,
|
||||
renderAtom,
|
||||
renderRss,
|
||||
} from "@umt/testkit";
|
||||
|
||||
const here = dirname(fileURLToPath(import.meta.url));
|
||||
const rssDir = join(here, "inputs", "rss");
|
||||
const idxDir = join(here, "inputs", "indexer");
|
||||
mkdirSync(rssDir, { recursive: true });
|
||||
mkdirSync(idxDir, { recursive: true });
|
||||
|
||||
const items: MockFeedItem[] = [
|
||||
{
|
||||
guid: "audiences-812345",
|
||||
title: "Dune.Part.Two.2024.2160p.UHD.BluRay.REMUX.DV.HDR.HEVC.TrueHD.Atmos.7.1-FraMeSToR",
|
||||
link: "https://audiences.me/details.php?id=812345",
|
||||
enclosureUrl: "https://audiences.me/download.php?id=812345&passkey=0123456789abcdef",
|
||||
size: 84203386470,
|
||||
pubDate: new Date(Date.UTC(2026, 8, 26, 13, 14, 3)),
|
||||
categories: ["电影", "401"],
|
||||
description: "沙丘2 / 沙丘：第二部 | <b>杜比视界</b> 国英双语 中字",
|
||||
},
|
||||
{
|
||||
guid: "audiences-812346",
|
||||
title: "Keep.Running.S12E01.20240426.2160p.WEB-DL.H265.AAC-LelveTV & Friends",
|
||||
link: "https://audiences.me/details.php?id=812346",
|
||||
enclosureUrl: "https://audiences.me/download.php?id=812346",
|
||||
size: 5497558138,
|
||||
pubDate: new Date(Date.UTC(2026, 8, 26, 12, 0, 0)),
|
||||
},
|
||||
{
|
||||
guid: "audiences-812347",
|
||||
title: "The.Bear.S03.2160p.DSNP.WEB-DL.DDP5.1.Atmos.DV.H.265-FLUX",
|
||||
link: "https://audiences.me/details.php?id=812347",
|
||||
},
|
||||
];
|
||||
writeFileSync(join(rssDir, "tk-rss-basic.xml"), renderRss("观众 Torrents", items));
|
||||
writeFileSync(join(rssDir, "tk-atom-basic.xml"), renderAtom("观众 Atom", items));
|
||||
|
||||
const site = await createMockNexusPhpSite({
|
||||
releases: AUDIENCES_FIXTURE_RELEASES,
|
||||
});
|
||||
try {
|
||||
const res = await fetch(`${site.baseUrl}/torrents.php?page=0`, {
|
||||
headers: { cookie: site.cookie },
|
||||
});
|
||||
writeFileSync(join(idxDir, "tk-audiences-list.html.txt"), await res.text());
|
||||
site.setLoggedIn(false);
|
||||
const login = await fetch(`${site.baseUrl}/torrents.php`, {
|
||||
headers: { cookie: site.cookie },
|
||||
});
|
||||
writeFileSync(join(idxDir, "tk-audiences-login.html.txt"), await login.text());
|
||||
} finally {
|
||||
await site.close();
|
||||
}
|
||||
writeFileSync(join(idxDir, "tk-hhanclub-list.html.txt"), HHANCLUB_LIST_PAGE);
|
||||
writeFileSync(join(idxDir, "tk-chdbits-list.html.txt"), CHDBITS_LIST_PAGE);
|
||||
writeFileSync(join(idxDir, "tk-chdbits-login.html.txt"), CHDBITS_LOGIN_PAGE);
|
||||
console.log("exported to", here);
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,123 @@
|
||||
/**
|
||||
* 站点索引解析差分:MP `SiteSpider.parse`(PyQuery 回退)vs 本项目 `SiteSpider.parse`。
|
||||
* 同一 indexer 配置(本项目 definitions/*.json 或内联配置)+ 同一 HTML;期望由 scripts/mp-sync/gen-differential.py rss|indexer 生成(indexer.json)。
|
||||
*/
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { BUILTIN_SITE_DEFINITIONS, type IndexerDefinition, SiteSpider } from "../../src/sites/index.ts";
|
||||
import {
|
||||
diffAgainstRegistry,
|
||||
dumpMismatches,
|
||||
FieldCollector,
|
||||
formatMismatches,
|
||||
loadDoc,
|
||||
readInput,
|
||||
} from "./diff-util.ts";
|
||||
|
||||
interface IndexerSample {
|
||||
id: string;
|
||||
definition:
|
||||
| { source: "definition"; id: string }
|
||||
| {
|
||||
source: "inline";
|
||||
config: unknown;
|
||||
};
|
||||
input: { file?: string; inline?: string };
|
||||
query: { keyword: string | null; search_type: "torrents" | "subtitles" };
|
||||
mp: {
|
||||
is_error?: boolean;
|
||||
torrents?: Array<Record<string, unknown>>;
|
||||
status?: string;
|
||||
error?: string;
|
||||
};
|
||||
}
|
||||
|
||||
const doc = loadDoc<IndexerSample>("indexer.json");
|
||||
|
||||
function definitionOf(s: IndexerSample): IndexerDefinition {
|
||||
if (s.definition.source === "definition") {
|
||||
return BUILTIN_SITE_DEFINITIONS[s.definition.id] as IndexerDefinition;
|
||||
}
|
||||
return s.definition.config as IndexerDefinition;
|
||||
}
|
||||
|
||||
/**
|
||||
* 行对齐:条数相同按下标;条数不同(MP 整行丢弃等)且双方都有 page_url 时按 page_url 配对,
|
||||
* 避免一处丢行导致后续所有行错位。行号 i 取 MP 侧下标。
|
||||
*/
|
||||
function alignRows(
|
||||
mp: Array<Record<string, unknown>>,
|
||||
ours: Array<Record<string, unknown>>,
|
||||
): Array<[number, Record<string, unknown>, Record<string, unknown>]> {
|
||||
const byUrl = mp.length !== ours.length && [...mp, ...ours].every((r) => typeof r.page_url === "string");
|
||||
if (!byUrl) {
|
||||
return mp.slice(0, Math.min(mp.length, ours.length)).map((m, i) => [i, m, ours[i]!]);
|
||||
}
|
||||
const index = new Map(ours.map((o) => [o.page_url as string, o]));
|
||||
const out: Array<[number, Record<string, unknown>, Record<string, unknown>]> = [];
|
||||
mp.forEach((m, i) => {
|
||||
const o = index.get(m.page_url as string);
|
||||
if (o) out.push([i, m, o]);
|
||||
});
|
||||
return out;
|
||||
}
|
||||
|
||||
describe("站点索引差分(MP SiteSpider Python 回退)", () => {
|
||||
const c = new FieldCollector();
|
||||
for (const s of doc.samples) {
|
||||
if (s.mp.status === "error") {
|
||||
throw new Error(`${s.id}: MP 运行失败 ${s.mp.error}`);
|
||||
}
|
||||
if (s.query.search_type === "subtitles") {
|
||||
// 本项目 SiteSpider 未移植字幕分支(parse_indexer_subtitles)
|
||||
c.mismatches.push({
|
||||
key: `${s.id}.status`,
|
||||
mp: `ok:${s.mp.torrents?.length ?? 0}`,
|
||||
ours: "unsupported",
|
||||
});
|
||||
continue;
|
||||
}
|
||||
const html = s.input.file ? readInput(s.input.file) : (s.input.inline ?? "");
|
||||
const spider = new SiteSpider(definitionOf(s), s.query.keyword ? { keyword: s.query.keyword } : {});
|
||||
const ours = spider.parse(html);
|
||||
const mpRows = s.mp.torrents ?? [];
|
||||
if (Boolean(s.mp.is_error) !== ours.isError) {
|
||||
c.mismatches.push({
|
||||
key: `${s.id}.status`,
|
||||
mp: s.mp.is_error,
|
||||
ours: ours.isError,
|
||||
});
|
||||
}
|
||||
if (mpRows.length !== ours.torrents.length) {
|
||||
c.mismatches.push({
|
||||
key: `${s.id}.count`,
|
||||
mp: mpRows.length,
|
||||
ours: ours.torrents.length,
|
||||
});
|
||||
}
|
||||
for (const [i, m, o] of alignRows(mpRows, ours.torrents as Array<Record<string, unknown>>)) {
|
||||
for (const f of doc.fields) c.check(`${s.id}#${i}.${f}`, m[f], o[f]);
|
||||
}
|
||||
}
|
||||
|
||||
it("有样本且记录了生成来源", () => {
|
||||
expect(doc.samples.length).toBeGreaterThan(10);
|
||||
expect(doc.generatedFrom.commit).toMatch(/^[0-9a-f]{40}$/);
|
||||
});
|
||||
|
||||
it("实际不一致集合 == 登记的 deviations", () => {
|
||||
dumpMismatches("indexer", c.mismatches, {
|
||||
compared: c.compared,
|
||||
equal: c.equal,
|
||||
});
|
||||
const { unregistered, stale } = diffAgainstRegistry(c.mismatches, doc.deviations);
|
||||
expect(formatMismatches(unregistered), "未登记的不一致").toBe("");
|
||||
expect(stale.join("\n"), "登记了但已不再出现的偏差(请从 indexer.json deviations 删除)").toBe("");
|
||||
});
|
||||
|
||||
it("登记项分类合法", () => {
|
||||
for (const [key, d] of Object.entries(doc.deviations)) {
|
||||
expect(["intentional", "missing", "our_bug", "mp_bug"], key).toContain(d.kind);
|
||||
expect(d.note.length, key).toBeGreaterThan(0);
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,49 @@
|
||||
{
|
||||
"id": "unit-case",
|
||||
"name": "Unit(case)",
|
||||
"domain": "https://pt.example.com/",
|
||||
"search": { "paths": [{ "path": "torrents.php" }] },
|
||||
"category": { "movie": [{ "id": "401" }], "tv": [{ "id": "402" }] },
|
||||
"torrents": {
|
||||
"list": { "selector": "table.torrents > tr:has(\"table.torrentname\")" },
|
||||
"fields": {
|
||||
"title_default": { "selector": "a[href*=\"details.php?id=\"]" },
|
||||
"title_optional": { "selector": "a[title][href*=\"details.php?id=\"]", "attribute": "title" },
|
||||
"title": {
|
||||
"text": "{% if fields['title_optional'] %}{{ fields['title_optional'] }}{% else %}{{ fields['title_default'] }}{% endif %}"
|
||||
},
|
||||
"details": { "selector": "a[href*=\"details.php?id=\"]", "attribute": "href" },
|
||||
"download": { "selector": "a[href*=\"download.php?id=\"]", "attribute": "href" },
|
||||
"imdbid": {
|
||||
"selector": "a[href*=\"imdb.com/title/tt\"]",
|
||||
"attribute": "href",
|
||||
"filters": [{ "name": "re_search", "args": ["tt\\d+", 0] }]
|
||||
},
|
||||
"date_elapsed": { "selector": "td:nth-child(4) > span" },
|
||||
"date_added": { "selector": "td:nth-child(4) > span", "attribute": "title" },
|
||||
"date": {
|
||||
"text": "{% if fields['date_elapsed'] or fields['date_added'] %}{{ fields['date_added'] if fields['date_added'] else fields['date_elapsed'] }}{% else %}now{% endif %}",
|
||||
"filters": [{ "name": "dateparse", "args": "%Y-%m-%d %H:%M:%S" }]
|
||||
},
|
||||
"size": { "selector": "td:nth-child(5)" },
|
||||
"seeders": { "selector": "td:nth-child(6)" },
|
||||
"leechers": { "selector": "td:nth-child(7)" },
|
||||
"grabs": { "selector": "td:nth-child(8)" },
|
||||
"downloadvolumefactor": { "case": { "img.pro_free": 0, "img.pro_50pctdown2up": 0.5, "*": 1 } },
|
||||
"uploadvolumefactor": { "case": { "img.pro_50pctdown2up": 2, "img.pro_2up": 2, "*": 1 } },
|
||||
"description": { "selector": "font.subtitle", "remove": "span, a" },
|
||||
"labels": { "selector": "span.label" },
|
||||
"freedate": {
|
||||
"selector": "img[class^=\"pro_\"]",
|
||||
"attribute": "title",
|
||||
"filters": [{ "name": "re_search", "args": ["\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}", 0] }]
|
||||
},
|
||||
"hr": { "selector": "img.hitandrun" },
|
||||
"category": {
|
||||
"selector": "a[href*=\"?cat=\"]",
|
||||
"attribute": "href",
|
||||
"filters": [{ "name": "querystring", "args": "cat" }]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
<!DOCTYPE html>
|
||||
<html><head><title>Unit :: 种子</title></head><body>
|
||||
<a href="logout.php">退出</a>
|
||||
<table class="torrents">
|
||||
<tr><td class="colhead">类型</td><td class="colhead">标题</td><td class="colhead">评论</td><td class="colhead">时间</td><td class="colhead">大小</td><td class="colhead">种子</td><td class="colhead">下载</td><td class="colhead">完成</td></tr>
|
||||
<tr>
|
||||
<td><a href="?cat=402">TV</a></td>
|
||||
<td>
|
||||
<table class="torrentname">
|
||||
<tr>
|
||||
<td class="embedded">
|
||||
<a href="details.php?id=100" title="Optional.Title">Default.Title</a>
|
||||
<a href="download.php?id=100">DL</a>
|
||||
<a href="https://www.imdb.com/title/tt1234567/">IMDb</a>
|
||||
<font class="subtitle">Main description <span>remove</span><a>link</a></font>
|
||||
<span class="label">FREE</span>
|
||||
<img class="hitandrun" />
|
||||
<img class="pro_free" title="Free until 2026-10-01 12:00:00" />
|
||||
<span class="factor">x0.5</span><span class="upfactor">2.0X</span>
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
</td>
|
||||
<td></td>
|
||||
<td><span title="2025-05-01 12:13:14">1 hour ago</span></td>
|
||||
<td>1.5 GB</td>
|
||||
<td>1,234</td>
|
||||
<td>5/7</td>
|
||||
<td>9</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><a href="?cat=401&x=1">Movie</a></td>
|
||||
<td>
|
||||
<table class="torrentname">
|
||||
<tr>
|
||||
<td class="embedded">
|
||||
<a href="//pt.example.com/details.php?id=101"><b>No.Title.Attr.2026</b></a>
|
||||
<a href="magnet:?xt=urn:btih:0123456789abcdef0123456789abcdef01234567">magnet-download.php?id=101</a>
|
||||
<font class="subtitle">第一行<br/>第二行 多空格</font>
|
||||
<span class="label">国语</span><span class="label"></span><span class="label">中字</span>
|
||||
<img class="pro_50pctdown2up" title="2026-10-02 08:00:00" />
|
||||
<span class="factor">50%</span>
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
</td>
|
||||
<td>0</td>
|
||||
<td><span>3小时前</span></td>
|
||||
<td>1,234.56 MiB</td>
|
||||
<td>-</td>
|
||||
<td>0</td>
|
||||
<td></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><a href="?cat=499">Other</a></td>
|
||||
<td>
|
||||
<table class="torrentname">
|
||||
<tr>
|
||||
<td class="embedded">
|
||||
<a href="/details.php?id=102" title="Slash.Link.Title">Slash</a>
|
||||
<a href="pt.example.com/download.php?id=102">DL</a>
|
||||
<font class="subtitle"> 前后空格 </font>
|
||||
<img class="pro_2up" />
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
</td>
|
||||
<td>2</td>
|
||||
<td>2026/09/26 21:14</td>
|
||||
<td>850 KB</td>
|
||||
<td>12 / 30</td>
|
||||
<td>abc</td>
|
||||
<td>1,000</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><a href="?cat=401">Movie</a></td>
|
||||
<td>
|
||||
<table class="torrentname">
|
||||
<tr>
|
||||
<td class="embedded">
|
||||
<a href="details.php?id=103" title="Tiny.Size">Tiny</a>
|
||||
<a href="/download.php?id=103&passkey=abc">DL</a>
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
</td>
|
||||
<td>0</td>
|
||||
<td>2026-09-26 21:14:03</td>
|
||||
<td>2.5 B</td>
|
||||
<td>0</td>
|
||||
<td>0</td>
|
||||
<td>0</td>
|
||||
</tr>
|
||||
</table>
|
||||
</body></html>
|
||||
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"id": "unit-selector",
|
||||
"name": "Unit(selector)",
|
||||
"domain": "https://pt.example.com",
|
||||
"media_type": "movie",
|
||||
"search": { "paths": [{ "path": "torrents.php" }] },
|
||||
"torrents": {
|
||||
"list": { "selector": "table.torrents tr:has(table.torrentname)" },
|
||||
"fields": {
|
||||
"title": {
|
||||
"selector": "a[href*=\"details.php\"]",
|
||||
"filters": [{ "name": "replace", "args": [".", " "] }, { "name": "appendleft", "args": "[U] " }]
|
||||
},
|
||||
"description": { "selector": "font.subtitle", "contents": -1 },
|
||||
"details": {
|
||||
"selector": "a[href*=\"details.php\"]",
|
||||
"attribute": "href",
|
||||
"filters": [{ "name": "split", "args": ["?", -1] }, { "name": "appendleft", "args": "details.php?" }]
|
||||
},
|
||||
"download": { "selector": "a[href*=\"download.php\"]", "attribute": "href" },
|
||||
"date_added": { "selector": "td:nth-child(4)" },
|
||||
"size": { "selector": "td", "index": 4 },
|
||||
"seeders": { "selector": "td:nth-child(6)", "filters": [{ "name": "re_search", "args": ["(\\d+)\\s*/", 1] }] },
|
||||
"leechers": { "selector": "td:nth-child(7)" },
|
||||
"grabs": { "selector": "td:nth-child(8)", "filters": [{ "name": "strip" }] },
|
||||
"downloadvolumefactor": { "selector": "span.factor" },
|
||||
"uploadvolumefactor": { "selector": "span.upfactor" },
|
||||
"labels": { "selector": "span.label", "attribute": "class" },
|
||||
"hr": { "selector": "img.nonexistent" }
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
<html><head><title>Unit :: 种子</title></head><body>
|
||||
<table class="torrents">
|
||||
<tr class="sticky free_bg"><td><a href="details.php?id=200" title="Row.Class.Free">Row</a></td><td>1 GB</td></tr>
|
||||
<tr class="normal"><td><a href="details.php?id=201" title="Row.Plain">Row</a></td><td>2 GB</td></tr>
|
||||
</table>
|
||||
</body></html>
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"id": "unit-self",
|
||||
"name": "Unit(self)",
|
||||
"domain": "https://pt.example.com/",
|
||||
"search": { "paths": [{ "path": "torrents.php" }] },
|
||||
"torrents": {
|
||||
"list": { "selector": "table.torrents > tr" },
|
||||
"fields": {
|
||||
"title": { "selector": "a[title]", "attribute": "title" },
|
||||
"details": { "selector": "a[title]", "attribute": "href" },
|
||||
"size": { "selector": "td:nth-child(2)" },
|
||||
"downloadvolumefactor": { "case": { "tr.free_bg": 0, "*": 1 } },
|
||||
"hr": { "selector": "tr.sticky" },
|
||||
"labels": { "selector": "tr.sticky", "attribute": "class" }
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,75 @@
|
||||
<!DOCTYPE html><html><head><meta charset="utf-8"><title>观众 :: 种子</title></head>
|
||||
<body><div id="info_block"><a href="userdetails.php?id=10001">mockuser</a>
|
||||
<span class="site-userbar__compact-metric--uploaded">12.5 TB</span>
|
||||
<span class="site-userbar__compact-metric--downloaded">3.2 TB</span>
|
||||
<a href="logout.php">退出</a></div><div id="outer"><p align="center"><a href="?page=0"><b>1</b></a></p>
|
||||
<table class="torrents torrents-table" cellspacing="0" cellpadding="5" width="100%"><tbody><tr><td class="colhead">类型</td><td class="colhead">标题</td><td class="colhead">评论</td><td class="colhead">存活时间</td><td class="colhead">大小</td><td class="colhead">种子</td><td class="colhead">下载</td><td class="colhead">完成</td><td class="colhead">发布者</td></tr><tr>
|
||||
<td class="rowfollow nowrap" valign="middle"><a href="?cat=401"><img class="c_401" src="pic/cattrans.gif" alt="电影" title="电影"></a></td>
|
||||
<td class="rowfollow" width="100%" align="left"><table class="torrentname" width="100%"><tr>
|
||||
<td class="embedded"><a title="Dune.Part.Two.2024.2160p.UHD.BluRay.REMUX.DV.HDR.HEVC.TrueHD.Atmos.7.1-FraMeSToR" href="details.php?id=812345&hit=1"><b>Dune.Part.Two.2024.2160p.UHD.BluRay.REMUX.DV.HDR.HEVC.TrueHD.Atmos.7.1-FraMeSToR</b></a> <img class="pro_free" src="pic/trans.gif" alt="Free" title="2026-10-01 12:00:00"><br><span class="torrent-subtitle-line">沙丘2 / 沙丘：第二部 | 杜比视界 国英双语 中字</span></td>
|
||||
<td class="embedded" width="20"><a href="download.php?id=812345"><img class="download" src="pic/trans.gif" alt="download"></a></td>
|
||||
</tr></table></td>
|
||||
<td class="rowfollow"><a href="comment.php?id=812345">0</a></td>
|
||||
<td class="rowfollow nowrap"><span title="2026-09-26 21:14:03">3小时前</span></td>
|
||||
<td class="rowfollow">78.42 GB</td>
|
||||
<td class="rowfollow" align="center"><b><a href="details.php?id=812345&dllist=1#seeders">1,204</a></b></td>
|
||||
<td class="rowfollow" align="center"><b><a href="details.php?id=812345&dllist=1#leechers">17</a></b></td>
|
||||
<td class="rowfollow"><a href="viewsnatches.php?id=812345"><b>3,388</b></a></td>
|
||||
<td class="rowfollow"><a href="userdetails.php?id=2">uploader1</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="rowfollow nowrap" valign="middle"><a href="?cat=403"><img class="c_403" src="pic/cattrans.gif" alt="综艺" title="综艺"></a></td>
|
||||
<td class="rowfollow" width="100%" align="left"><table class="torrentname" width="100%"><tr>
|
||||
<td class="embedded"><a title="Keep.Running.S12E01.20240426.2160p.WEB-DL.H265.AAC-LelveTV" href="details.php?id=812346&hit=1"><b>Keep.Running.S12E01.20240426.2160p.WEB-DL.H265.AAC-LelveTV</b></a> <img class="hitandrun" src="pic/trans.gif" alt="H&R" title="H&R"><br><span class="torrent-subtitle-line">奔跑吧 第十二季 第1期</span></td>
|
||||
<td class="embedded" width="20"><a href="download.php?id=812346"><img class="download" src="pic/trans.gif" alt="download"></a></td>
|
||||
</tr></table></td>
|
||||
<td class="rowfollow"><a href="comment.php?id=812346">0</a></td>
|
||||
<td class="rowfollow nowrap"><span title="2026-09-26 20:00:00">2026-09-26 20:00:00</span></td>
|
||||
<td class="rowfollow">5.12 GB</td>
|
||||
<td class="rowfollow" align="center"><b><a href="details.php?id=812346&dllist=1#seeders">88</a></b></td>
|
||||
<td class="rowfollow" align="center"><b><a href="details.php?id=812346&dllist=1#leechers">2</a></b></td>
|
||||
<td class="rowfollow"><a href="viewsnatches.php?id=812346"><b>410</b></a></td>
|
||||
<td class="rowfollow"><a href="userdetails.php?id=2">anonymous</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="rowfollow nowrap" valign="middle"><a href="?cat=402"><img class="c_402" src="pic/cattrans.gif" alt="剧集" title="剧集"></a></td>
|
||||
<td class="rowfollow" width="100%" align="left"><table class="torrentname" width="100%"><tr>
|
||||
<td class="embedded"><a title="The.Bear.S03.2160p.DSNP.WEB-DL.DDP5.1.Atmos.DV.H.265-FLUX" href="details.php?id=812347&hit=1"><b>The.Bear.S03.2160p.DSNP.WEB-DL.DDP5.1.Atmos.DV.H.265-FLUX</b></a> <img class="pro_50pctdown2up" src="pic/trans.gif" alt="2X 50%" title="2026-09-29 08:30:00"><br><span class="torrent-subtitle-line">熊家餐馆 第三季</span></td>
|
||||
<td class="embedded" width="20"><a href="download.php?id=812347"><img class="download" src="pic/trans.gif" alt="download"></a></td>
|
||||
</tr></table></td>
|
||||
<td class="rowfollow"><a href="comment.php?id=812347">0</a></td>
|
||||
<td class="rowfollow nowrap"><span title="2026-09-25 08:30:00">2026-09-25 08:30:00</span></td>
|
||||
<td class="rowfollow">1,234.56 MiB</td>
|
||||
<td class="rowfollow" align="center"><b><a href="details.php?id=812347&dllist=1#seeders">35</a></b></td>
|
||||
<td class="rowfollow" align="center"><b><a href="details.php?id=812347&dllist=1#leechers">0</a></b></td>
|
||||
<td class="rowfollow"><a href="viewsnatches.php?id=812347"><b>120</b></a></td>
|
||||
<td class="rowfollow"><a href="userdetails.php?id=2">anonymous</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="rowfollow nowrap" valign="middle"><a href="?cat=402"><img class="c_402" src="pic/cattrans.gif" alt="剧集" title="剧集"></a></td>
|
||||
<td class="rowfollow" width="100%" align="left"><table class="torrentname" width="100%"><tr>
|
||||
<td class="embedded"><a href="details.php?id=812348&hit=1"><b>The.Last.of.Us.S02.2160p.MAX.WEB-DL.DDP5.1.Atmos.D…</b></a></td>
|
||||
<td class="embedded" width="20"><a href="download.php?id=812348"><img class="download" src="pic/trans.gif" alt="download"></a></td>
|
||||
</tr></table></td>
|
||||
<td class="rowfollow"><a href="comment.php?id=812348">0</a></td>
|
||||
<td class="rowfollow nowrap"><span title="2026-09-24 10:00:00">2026-09-24 10:00:00</span></td>
|
||||
<td class="rowfollow">2.1 TB</td>
|
||||
<td class="rowfollow" align="center"><b><a href="details.php?id=812348&dllist=1#seeders">12</a></b></td>
|
||||
<td class="rowfollow" align="center"><b><a href="details.php?id=812348&dllist=1#leechers">1</a></b></td>
|
||||
<td class="rowfollow"><a href="viewsnatches.php?id=812348"><b>44</b></a></td>
|
||||
<td class="rowfollow"><a href="userdetails.php?id=2">anonymous</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="rowfollow nowrap" valign="middle"><a href="?cat=406"><img class="c_406" src="pic/cattrans.gif" alt="纪录片" title="纪录片"></a></td>
|
||||
<td class="rowfollow" width="100%" align="left"><table class="torrentname" width="100%"><tr>
|
||||
<td class="embedded"><a title="Planet.Earth.III.2023.2160p.BluRay.REMUX.HEVC.DTS-HD.MA.5.1-GRP" href="details.php?id=812349&hit=1"><b>Planet.Earth.III.2023.2160p.BluRay.REMUX.HEVC.DTS-HD.MA.5.1-GRP</b></a> <img class="pro_2up" src="pic/trans.gif" alt="2X" title=""><br><span class="torrent-subtitle-line">地球脉动 第三季</span></td>
|
||||
<td class="embedded" width="20"><a href="download.php?id=812349"><img class="download" src="pic/trans.gif" alt="download"></a></td>
|
||||
</tr></table></td>
|
||||
<td class="rowfollow"><a href="comment.php?id=812349">0</a></td>
|
||||
<td class="rowfollow nowrap"><span title="2026-09-23 10:00:00">2026-09-23 10:00:00</span></td>
|
||||
<td class="rowfollow">850 KB</td>
|
||||
<td class="rowfollow" align="center"><b><a href="details.php?id=812349&dllist=1#seeders">5</a></b></td>
|
||||
<td class="rowfollow" align="center"><b><a href="details.php?id=812349&dllist=1#leechers">0</a></b></td>
|
||||
<td class="rowfollow"><a href="viewsnatches.php?id=812349"><b>9</b></a></td>
|
||||
<td class="rowfollow"><a href="userdetails.php?id=2">anonymous</a></td>
|
||||
</tr></tbody></table></div></body></html>
|
||||
@@ -0,0 +1,3 @@
|
||||
<!DOCTYPE html><html><head><meta charset="utf-8"><title>观众 :: 登录</title></head>
|
||||
<body><form method="post" action="takelogin.php"><p>你需要启用cookies才能登录</p>
|
||||
<input type="text" name="username"><input type="password" name="password"><input type="submit" value="登录"></form></body></html>
|
||||
@@ -0,0 +1,35 @@
|
||||
<!DOCTYPE html><html><head><title>CHDBits :: 种子</title></head><body>
|
||||
<table id="info_block"><tr><td><a href="userdetails.php?id=42" class="VIP_Name">carol</a> <a href="logout.php">退出</a>
|
||||
<font class="color_uploaded">上传量：</font> 25.052 TB <font class="color_downloaded">下载量：</font> 1.2 TB</td></tr></table>
|
||||
<table class="torrents" cellspacing="0" cellpadding="5" width="100%">
|
||||
<tr><td class="colhead">类型</td><td class="colhead">标题</td><td class="colhead">评论</td><td class="colhead">时间</td><td class="colhead">大小</td><td class="colhead">种子</td><td class="colhead">下载</td><td class="colhead">完成</td><td class="colhead">发布者</td></tr>
|
||||
<tr>
|
||||
<td class="rowfollow nowrap"><a href="?cat=402"><img class="c_tvseries" src="pic/cattrans.gif" alt="TV Series"/></a></td>
|
||||
<td class="rowfollow"><table class="torrentname" width="100%"><tr>
|
||||
<td class="embedded"><a title="The.Last.of.Us.S02.2025.1080p.BluRay.x264.DTS-HD.MA.5.1-CHD" href="details.php?id=555001&hit=1"><b>The.Last.of.Us.S02.2025.1080p.BluRay...</b></a>
|
||||
<img class="pro_free2up" src="pic/trans.gif" alt="2X Free"/> <span title="2026-09-30 12:00:00">[限时]</span><br/><font class="subtitle">最后生还者 第二季 <span class="tags">中字</span></font></td>
|
||||
<td class="embedded" width="20"><a href="download.php?id=555001"><img class="download" alt="download"/></a></td>
|
||||
</tr></table></td>
|
||||
<td class="rowfollow">3</td>
|
||||
<td class="rowfollow nowrap"><span title="2026-09-25 08:00:00">2天</span></td>
|
||||
<td class="rowfollow">45.20<br/>GB</td>
|
||||
<td class="rowfollow"><b><a href="details.php?id=555001#peers">42</a></b></td>
|
||||
<td class="rowfollow">3</td>
|
||||
<td class="rowfollow"><a href="viewsnatches.php?id=555001"><b>100</b></a></td>
|
||||
<td class="rowfollow"><a href="userdetails.php?id=1">uploader</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="rowfollow nowrap"><a href="?cat=401"><img class="c_movie" src="pic/cattrans.gif" alt="Movies"/></a></td>
|
||||
<td class="rowfollow"><table class="torrentname" width="100%"><tr>
|
||||
<td class="embedded"><a title="Oppenheimer.2023.2160p.UHD.BluRay.REMUX-CHD" href="details.php?id=555002&hit=1"><b>Oppenheimer.2023...</b></a><br/><font class="subtitle">奥本海默</font></td>
|
||||
<td class="embedded" width="20"><a href="download.php?id=555002"><img class="download" alt="download"/></a></td>
|
||||
</tr></table></td>
|
||||
<td class="rowfollow">0</td>
|
||||
<td class="rowfollow nowrap"><span title="2026-09-01 10:00:00">26天</span></td>
|
||||
<td class="rowfollow">80.00<br/>GB</td>
|
||||
<td class="rowfollow">7</td>
|
||||
<td class="rowfollow">0</td>
|
||||
<td class="rowfollow">55</td>
|
||||
<td class="rowfollow"><a href="userdetails.php?id=1">uploader</a></td>
|
||||
</tr>
|
||||
</table></body></html>
|
||||
@@ -0,0 +1,3 @@
|
||||
<!DOCTYPE html><html><head><title>CHDBits :: 登录</title></head><body>
|
||||
<form method="post" action="takelogin.php"><input name="username"/><input type="password" name="password"/></form>
|
||||
<p>未登录!</p></body></html>
|
||||
@@ -0,0 +1,33 @@
|
||||
<!DOCTYPE html><html><head><title>HHCLUB :: 种子</title></head><body>
|
||||
<div id="user-info-panel"><a href="userdetails.php?id=7" class="EliteUser_Name">bob</a> <a href="logout.php">退出</a></div>
|
||||
<div class="torrent-table">
|
||||
<div class="torrent-table-sub-info">
|
||||
<div class="torrent-cat"><a href="?cat[]=403"><img src="pic/cattrans.gif" alt="综艺"/></a></div>
|
||||
<div class="torrent-title"><a class="torrent-info-text-name" href="details.php?id=90001&hit=1">Go.Fighting.S10E01.2026.2160p.WEB-DL.H265.AAC-HHWEB</a>
|
||||
<span class="promotion-tag promotion-tag-free">免费</span></div>
|
||||
<div class="torrent-info-text-small_name">极限挑战 第十季 第1期</div>
|
||||
<div class="torrent-manage"><a href="download.php?id=90001">下载</a></div>
|
||||
<div class="torrent-info-text-added"><span title="2026-09-20 20:30:00">6天23时</span></div>
|
||||
<div class="torrent-info-text-size">6.12 GB</div>
|
||||
<div class="torrent-info-text-seeders">88</div>
|
||||
<div class="torrent-info-text-leechers">5</div>
|
||||
<div class="torrent-info-text-finished">1,024</div>
|
||||
<div class="torrent-info-text-author">anon</div>
|
||||
<div class="torrent-info-hr">30</div>
|
||||
</div>
|
||||
<div class="torrent-table-sub-info">
|
||||
<div class="torrent-cat"><a href="?cat[]=401"><img src="pic/cattrans.gif" alt="电影"/></a></div>
|
||||
<div class="torrent-title"><a class="torrent-info-text-name" href="details.php?id=90002&hit=1">Dune.Part.Two.2024.2160p.UHD.BluRay.REMUX.DV.HDR.HEVC.TrueHD.7.1.Atmos-HHWEB</a>
|
||||
<span class="promotion-tag promotion-tag-50pctdown2up">50%</span></div>
|
||||
<div class="torrent-info-text-small_name">沙丘2</div>
|
||||
<div class="torrent-manage"><a href="download.php?id=90002">下载</a></div>
|
||||
<div class="torrent-info-text-added">2026-09-01 08:00:00</div>
|
||||
<div class="torrent-info-text-size">78.3 GB</div>
|
||||
<div class="torrent-info-text-seeders">12</div>
|
||||
<div class="torrent-info-text-leechers">0</div>
|
||||
<div class="torrent-info-text-finished">300</div>
|
||||
<div class="torrent-info-text-author">anon</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="pagination"><a href="?page=0">1-100</a><a href="?page=3">301-400</a></div>
|
||||
</body></html>
|
||||
@@ -0,0 +1,5 @@
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<feed><title>Atom without namespace</title>
|
||||
<entry><id>tag:e.x,2026:1</id><title>Atom.Entry.One</title><link rel="alternate" href="https://e.x/details?id=1"/><link rel="enclosure" href="https://e.x/dl?id=1" length="4096"/><updated>2026-09-26T13:14:03Z</updated><summary>atom summary</summary></entry>
|
||||
<entry><id>tag:e.x,2026:2</id><title type="html"><b>Atom.Entry.Two</b></title><link href="https://e.x/details?id=2"/><published>2026-09-26T21:14:03+08:00</published><updated>2026-09-27T00:00:00Z</updated><content>content only</content></entry>
|
||||
</feed>
|
||||
@@ -0,0 +1 @@
|
||||
RSS Link has expired, You need to get a new one!
|
||||
@@ -0,0 +1,5 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns="http://purl.org/rss/1.0/" xmlns:dc="http://purl.org/dc/elements/1.1/">
|
||||
<channel rdf:about="https://e.x/"><title>RDF feed</title><link>https://e.x/</link></channel>
|
||||
<item rdf:about="https://e.x/1"><title>RDF.Item.One</title><link>https://e.x/1</link><description>rdf desc</description><dc:date>2026-09-26T13:14:03Z</dc:date><dc:creator>alice</dc:creator></item>
|
||||
</rdf:RDF>
|
||||
@@ -0,0 +1,5 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<rss version="2.0"><channel><title>broken</title>
|
||||
<item><title>Good.One</title><link>https://e.x/1</link></item>
|
||||
<item><title>Unclosed<link>https://e.x/2</link></item>
|
||||
</channel>
|
||||
@@ -0,0 +1,5 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<rss version="2.0" xmlns:dc="http://purl.org/dc/elements/1.1/"><channel><title>CDATA feed</title>
|
||||
<item><title><![CDATA[[BDMV] 奥本海默 Oppenheimer 2023 <UHD> & more]]></title><link>https://pt.example.com/details.php?id=1</link><description><![CDATA[<p>Size: 78.42 GB</p><img src="x.jpg"/>]]></description><enclosure url="https://pt.example.com/download.php?id=1&passkey=abcdef0123456789" length="84203386470" type="application/x-bittorrent"/><pubDate>Sat, 26 Sep 2026 21:14:03 +0800</pubDate><dc:creator>豆瓣用户A</dc:creator><guid>pt-1</guid></item>
|
||||
<item><title>Plain & Escaped <Title> 中文</title><link>https://pt.example.com/details.php?id=2&hit=1</link><description>desc & more</description><enclosure url="https://pt.example.com/download.php?id=2" length="1024"/><pubDate>Sat, 26 Sep 2026 13:14:03 GMT</pubDate></item>
|
||||
</channel></rss>
|
||||
@@ -0,0 +1,5 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE rss [<!ENTITY brand "EvilCorp">]>
|
||||
<rss version="2.0"><channel><title>dtd</title>
|
||||
<item><title>&brand; Title</title><link>https://e.x/1</link></item>
|
||||
</channel></rss>
|
||||
@@ -0,0 +1,5 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<rss version="2.0"><channel><title>entities</title>
|
||||
<item><title>Nbsp Title©2026</title><link>https://e.x/1</link></item>
|
||||
<item><title>Normal.After.Bad</title><link>https://e.x/2</link></item>
|
||||
</channel></rss>
|
||||
@@ -0,0 +1,8 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<rss version="2.0"><channel><title>fallback</title>
|
||||
<item><title>Only.Link.Item</title><link>https://pt.example.com/download.php?id=10</link></item>
|
||||
<item><title>No.Link.No.Enclosure</title><description>skip me</description></item>
|
||||
<item><title></title><link>https://pt.example.com/details.php?id=11</link></item>
|
||||
<item><link>https://pt.example.com/details.php?id=12</link></item>
|
||||
<item><title>Enclosure.Only</title><enclosure url="magnet:?xt=urn:btih:0123456789abcdef0123456789abcdef01234567&dn=x" length="0"/></item>
|
||||
</channel></rss>
|
||||
@@ -0,0 +1,5 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<rss version="2.0" xmlns:media="http://search.yahoo.com/mrss/"><channel><title>nested</title>
|
||||
<item><source url="https://e.x/src"><title>Source.Title.First</title></source><title>Real.Title</title><link>https://e.x/1</link><media:title>Media.Title</media:title></item>
|
||||
<item><title>Two.Enclosures</title><enclosure url="https://e.x/a.torrent" length="100"/><enclosure url="https://e.x/b.torrent" length="200"/></item>
|
||||
</channel></rss>
|
||||
@@ -0,0 +1,5 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<rss xmlns:atom="http://www.w3.org/2005/Atom" xmlns:nyaa="https://nyaa.si/xmlns/nyaa" version="2.0"><channel><title>Nyaa-like</title>
|
||||
<atom:link href="https://nyaa.example/?page=rss" rel="self" type="application/rss+xml"/>
|
||||
<item><title>[Group] Show - 01 [1080p].mkv</title><link>https://nyaa.example/download/1.torrent</link><guid isPermaLink="true">https://nyaa.example/view/1</guid><pubDate>Sat, 26 Sep 2026 13:14:03 -0000</pubDate><nyaa:seeders>12</nyaa:seeders><nyaa:leechers>3</nyaa:leechers><nyaa:size>1.2 GiB</nyaa:size><nyaa:infoHash>0123456789abcdef0123456789abcdef01234567</nyaa:infoHash><description><![CDATA[<a href="https://nyaa.example/view/1">#1</a> | 1.2 GiB]]></description></item>
|
||||
</channel></rss>
|
||||
@@ -0,0 +1,16 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<rss version="2.0"><channel><title>pubdate formats</title>
|
||||
<item><title>rfc822-gmt</title><link>https://e.x/1</link><pubDate>Sat, 26 Sep 2026 13:14:03 GMT</pubDate></item>
|
||||
<item><title>rfc822-offset</title><link>https://e.x/2</link><pubDate>Sat, 26 Sep 2026 21:14:03 +0800</pubDate></item>
|
||||
<item><title>rfc822-no-seconds</title><link>https://e.x/3</link><pubDate>Sat, 26 Sep 2026 21:14 +0800</pubDate></item>
|
||||
<item><title>rfc822-cst</title><link>https://e.x/4</link><pubDate>Sat, 26 Sep 2026 21:14:03 CST</pubDate></item>
|
||||
<item><title>iso-z</title><link>https://e.x/5</link><pubDate>2026-09-26T13:14:03Z</pubDate></item>
|
||||
<item><title>iso-offset</title><link>https://e.x/6</link><pubDate>2026-09-26T21:14:03+08:00</pubDate></item>
|
||||
<item><title>naive-space</title><link>https://e.x/7</link><pubDate>2026-09-26 21:14:03</pubDate></item>
|
||||
<item><title>date-only</title><link>https://e.x/8</link><pubDate>2026-09-26</pubDate></item>
|
||||
<item><title>chinese</title><link>https://e.x/9</link><pubDate>2026年09月26日 21:14:03</pubDate></item>
|
||||
<item><title>garbage</title><link>https://e.x/10</link><pubDate>3小时前</pubDate></item>
|
||||
<item><title>no-weekday</title><link>https://e.x/11</link><pubDate>26 Sep 2026 13:14:03 +0000</pubDate></item>
|
||||
<item><title>wrong-weekday</title><link>https://e.x/12</link><pubDate>Mon, 26 Sep 2026 13:14:03 GMT</pubDate></item>
|
||||
<item><title>epoch-like</title><link>https://e.x/13</link><pubDate>1790428443</pubDate></item>
|
||||
</channel></rss>
|
||||
@@ -0,0 +1,9 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<rss version="2.0"><channel><title>size</title>
|
||||
<item><title>Size.In.Description</title><link>https://pt.example.com/d?id=20</link><description>[Size: 1.50 GB] [Seeders: 12]</description></item>
|
||||
<item><title>Length.NonDigit</title><enclosure url="https://pt.example.com/dl?id=21" length="1.5GB"/></item>
|
||||
<item><title>Length.Empty</title><enclosure url="https://pt.example.com/dl?id=22" length=""/></item>
|
||||
<item><title>Length.Missing</title><enclosure url="https://pt.example.com/dl?id=23"/></item>
|
||||
<item><title>Length.Big</title><enclosure url="https://pt.example.com/dl?id=24" length="2199023255552"/></item>
|
||||
<item><title>Length.Negative</title><enclosure url="https://pt.example.com/dl?id=25" length="-1"/></item>
|
||||
</channel></rss>
|
||||
@@ -0,0 +1,19 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<rss version="2.0"><channel><title>ws</title>
|
||||
<item>
|
||||
<title>
|
||||
Spaced.Title.2026.1080p.WEB-DL
|
||||
</title>
|
||||
<link>
|
||||
https://pt.example.com/details.php?id=3
|
||||
</link>
|
||||
<description>
|
||||
line one
|
||||
line two
|
||||
</description>
|
||||
<enclosure url=" https://pt.example.com/download.php?id=3 " length=" 2048 "/>
|
||||
<pubDate>
|
||||
Sat, 26 Sep 2026 13:14:03 GMT
|
||||
</pubDate>
|
||||
</item>
|
||||
</channel></rss>
|
||||
@@ -0,0 +1,6 @@
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<feed xmlns="http://www.w3.org/2005/Atom"><title>观众 Atom</title>
|
||||
<entry><id>audiences-812345</id><title>Dune.Part.Two.2024.2160p.UHD.BluRay.REMUX.DV.HDR.HEVC.TrueHD.Atmos.7.1-FraMeSToR</title><link rel="alternate" href="https://audiences.me/details.php?id=812345"/><link rel="enclosure" href="https://audiences.me/download.php?id=812345&passkey=0123456789abcdef" length="84203386470"/><published>2026-09-26T13:14:03.000Z</published><category term="电影"/><category term="401"/><summary>沙丘2 / 沙丘：第二部 | <b>杜比视界</b> 国英双语 中字</summary></entry>
|
||||
<entry><id>audiences-812346</id><title>Keep.Running.S12E01.20240426.2160p.WEB-DL.H265.AAC-LelveTV & Friends</title><link rel="alternate" href="https://audiences.me/details.php?id=812346"/><link rel="enclosure" href="https://audiences.me/download.php?id=812346" length="5497558138"/><published>2026-09-26T12:00:00.000Z</published></entry>
|
||||
<entry><id>audiences-812347</id><title>The.Bear.S03.2160p.DSNP.WEB-DL.DDP5.1.Atmos.DV.H.265-FLUX</title><link rel="alternate" href="https://audiences.me/details.php?id=812347"/></entry>
|
||||
</feed>
|
||||
@@ -0,0 +1,6 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<rss version="2.0"><channel><title>观众 Torrents</title>
|
||||
<item><title>Dune.Part.Two.2024.2160p.UHD.BluRay.REMUX.DV.HDR.HEVC.TrueHD.Atmos.7.1-FraMeSToR</title><guid isPermaLink="false">audiences-812345</guid><link>https://audiences.me/details.php?id=812345</link><enclosure url="https://audiences.me/download.php?id=812345&passkey=0123456789abcdef" length="84203386470" type="application/x-bittorrent"/><pubDate>Sat, 26 Sep 2026 13:14:03 GMT</pubDate><category>电影</category><category>401</category><description><![CDATA[沙丘2 / 沙丘：第二部 | <b>杜比视界</b> 国英双语 中字]]></description></item>
|
||||
<item><title>Keep.Running.S12E01.20240426.2160p.WEB-DL.H265.AAC-LelveTV & Friends</title><guid isPermaLink="false">audiences-812346</guid><link>https://audiences.me/details.php?id=812346</link><enclosure url="https://audiences.me/download.php?id=812346" length="5497558138" type="application/x-bittorrent"/><pubDate>Sat, 26 Sep 2026 12:00:00 GMT</pubDate></item>
|
||||
<item><title>The.Bear.S03.2160p.DSNP.WEB-DL.DDP5.1.Atmos.DV.H.265-FLUX</title><guid isPermaLink="false">audiences-812347</guid><link>https://audiences.me/details.php?id=812347</link></item>
|
||||
</channel></rss>
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,121 @@
|
||||
/**
|
||||
* RSS 解析差分:MP `RssHelper._parse_impl`(lxml 回退)vs 本项目 `parseFeed`。
|
||||
* 期望由 scripts/mp-sync/gen-differential.py rss|indexer 生成(rss.json);不一致必须登记在 rss.json `deviations`。
|
||||
*/
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { FeedError, type FeedItem, parseFeed } from "../../src/rss/index.ts";
|
||||
import {
|
||||
diffAgainstRegistry,
|
||||
dumpMismatches,
|
||||
FieldCollector,
|
||||
formatMismatches,
|
||||
loadDoc,
|
||||
readInput,
|
||||
} from "./diff-util.ts";
|
||||
|
||||
interface MpPubdate {
|
||||
kind: "absent" | "unparsed" | "aware" | "naive";
|
||||
text?: string;
|
||||
wall?: string;
|
||||
utc?: string;
|
||||
}
|
||||
interface MpRssItem {
|
||||
title: string;
|
||||
description: string;
|
||||
link: string;
|
||||
enclosure: string;
|
||||
size: number;
|
||||
nickname?: string;
|
||||
pubdate: MpPubdate;
|
||||
}
|
||||
interface RssSample {
|
||||
id: string;
|
||||
input: { file: string; sha256: string };
|
||||
mp: { status: "ok" | "false_error" | "none_expired"; items: MpRssItem[] };
|
||||
}
|
||||
|
||||
const doc = loadDoc<RssSample>("rss.json");
|
||||
|
||||
/** MP 字段 → 可与本项目比较的值:空串即缺省;pubdate 统一为 UTC ISO */
|
||||
function mpPubdate(p: MpPubdate): string | undefined {
|
||||
if (p.kind === "aware") return p.utc;
|
||||
// naive:MP astimezone(None) 以宿主本地时区解释墙上时间;JS 对无偏移的日期时间同样按本地解释
|
||||
if (p.kind === "naive") return new Date(p.wall!).toISOString();
|
||||
return undefined;
|
||||
}
|
||||
const blank = (s: string | null | undefined) => (s === "" || s == null ? undefined : s);
|
||||
|
||||
function runOurs(xml: string): { status: string; items: FeedItem[] } {
|
||||
try {
|
||||
return {
|
||||
status: "ok",
|
||||
items: parseFeed(xml, { fetchedAt: new Date("2026-09-27T00:00:00Z") }).items,
|
||||
};
|
||||
} catch (e) {
|
||||
if (e instanceof FeedError) return { status: `error:${e.kind}`, items: [] };
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
|
||||
function statusEquivalent(mp: RssSample["mp"]["status"], ours: string): boolean {
|
||||
if (mp === "ok") return ours === "ok";
|
||||
if (mp === "none_expired") return ours === "error:expired";
|
||||
return ours.startsWith("error:") && ours !== "error:expired";
|
||||
}
|
||||
|
||||
describe("RSS 差分(MP RssHelper Python 回退)", () => {
|
||||
const c = new FieldCollector();
|
||||
for (const s of doc.samples) {
|
||||
const ours = runOurs(readInput(s.input.file));
|
||||
if (!statusEquivalent(s.mp.status, ours.status)) {
|
||||
c.mismatches.push({
|
||||
key: `${s.id}.status`,
|
||||
mp: s.mp.status,
|
||||
ours: ours.status,
|
||||
});
|
||||
}
|
||||
if (s.mp.status !== "ok" || ours.status !== "ok") continue;
|
||||
if (s.mp.items.length !== ours.items.length) {
|
||||
c.mismatches.push({
|
||||
key: `${s.id}.count`,
|
||||
mp: s.mp.items.length,
|
||||
ours: ours.items.length,
|
||||
});
|
||||
}
|
||||
const n = Math.min(s.mp.items.length, ours.items.length);
|
||||
for (let i = 0; i < n; i++) {
|
||||
const m = s.mp.items[i]!;
|
||||
const o = ours.items[i]!;
|
||||
const k = (f: string) => `${s.id}#${i}.${f}`;
|
||||
c.check(k("title"), m.title, o.title);
|
||||
c.check(k("description"), blank(m.description), blank(o.description));
|
||||
c.check(k("link"), blank(m.link), blank(o.link));
|
||||
c.check(k("enclosure"), blank(m.enclosure), blank(o.enclosureUrl));
|
||||
c.check(k("size"), m.size, o.size);
|
||||
c.check(k("pubdate"), mpPubdate(m.pubdate), o.publishedAt);
|
||||
c.check(k("nickname"), blank(m.nickname), blank(o.nickname));
|
||||
}
|
||||
}
|
||||
|
||||
it("有样本且记录了生成来源", () => {
|
||||
expect(doc.samples.length).toBeGreaterThan(10);
|
||||
expect(doc.generatedFrom.commit).toMatch(/^[0-9a-f]{40}$/);
|
||||
});
|
||||
|
||||
it("实际不一致集合 == 登记的 deviations", () => {
|
||||
dumpMismatches("rss", c.mismatches, {
|
||||
compared: c.compared,
|
||||
equal: c.equal,
|
||||
});
|
||||
const { unregistered, stale } = diffAgainstRegistry(c.mismatches, doc.deviations);
|
||||
expect(formatMismatches(unregistered), "未登记的不一致").toBe("");
|
||||
expect(stale.join("\n"), "登记了但已不再出现的偏差(请从 rss.json deviations 删除)").toBe("");
|
||||
});
|
||||
|
||||
it("登记项分类合法", () => {
|
||||
for (const [key, d] of Object.entries(doc.deviations)) {
|
||||
expect(["intentional", "missing", "our_bug", "mp_bug"], key).toContain(d.kind);
|
||||
expect(d.note.length, key).toBeGreaterThan(0);
|
||||
}
|
||||
});
|
||||
});
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,207 @@
|
||||
/**
|
||||
* MoviePilot 过滤规则 vs Slot 差分测试。
|
||||
*
|
||||
* filter-cases.json 由 MP 的 Python 回退实现生成(见 generatedFrom)。这里把可表达的规则
|
||||
* 手工翻译成 SlotSpec(每个 `>` 优先级层级一个 Slot),用 parseRelease + evaluateCandidate
|
||||
* 求"首个 accept 的层级",与 MP 的 level 比对。
|
||||
*
|
||||
* 断言:不一致集合 == deviations 登记集合。补迁消除差异后测试会失败,提醒删登记。
|
||||
*/
|
||||
import { readFileSync } from "node:fs";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
evaluateCandidate,
|
||||
type ParsedRelease,
|
||||
parseRelease,
|
||||
type ReleaseSource,
|
||||
rankForSlot,
|
||||
type SlotSpec,
|
||||
} from "../../src/index.ts";
|
||||
|
||||
interface Torrent {
|
||||
id: string;
|
||||
title: string;
|
||||
description: string;
|
||||
size: number;
|
||||
seeders: number;
|
||||
}
|
||||
interface Rule {
|
||||
id: string;
|
||||
rule: string;
|
||||
verdict: "full" | "partial" | "none";
|
||||
}
|
||||
interface Case {
|
||||
caseId: string;
|
||||
ruleId: string;
|
||||
torrentId: string;
|
||||
mp: { matched: boolean; priOrder: number | null; level: number | null };
|
||||
}
|
||||
type DeviationKind = "intentional" | "missing" | "our_bug" | "mp_bug" | "not_expressible";
|
||||
interface Deviation {
|
||||
kind: DeviationKind;
|
||||
note: string;
|
||||
pairs?: string[];
|
||||
}
|
||||
interface Fixture {
|
||||
generatedFrom: { commit: string; rustCrossCheck: { mismatches: unknown[] } };
|
||||
torrents: Torrent[];
|
||||
rules: Rule[];
|
||||
cases: Case[];
|
||||
deviations: Record<string, Deviation>;
|
||||
}
|
||||
|
||||
const fx = JSON.parse(readFileSync(new URL("./filter-cases.json", import.meta.url), "utf8")) as Fixture;
|
||||
|
||||
// ---------------------------------------------------------------- MP 关键字 → Slot 片段
|
||||
const MB = 1024 * 1024; // MP size_range 单位 MB = 1024*1024 字节
|
||||
const ALL: ReleaseSource[] = ["REMUX", "BluRay", "BluRay-Disc", "WEB-DL", "WEBRip", "HDTV", "HDRip", "DVD", "DVDRip"];
|
||||
const without = (...xs: ReleaseSource[]) => ALL.filter((s) => !xs.includes(s));
|
||||
|
||||
type Frag = Pick<SlotSpec, "hard"> & { unknown?: Partial<Record<string, "reject" | "accept">> };
|
||||
const RES_4K: Frag = { hard: { resolutions: ["2160p"] }, unknown: { resolution: "reject" } };
|
||||
const RES_1080: Frag = { hard: { resolutions: ["1080p", "1080i"] }, unknown: { resolution: "reject" } };
|
||||
const RES_720: Frag = { hard: { resolutions: ["720p"] }, unknown: { resolution: "reject" } };
|
||||
/** 正向来源条件:来源未知 = MP 正则不命中 = 拒绝 */
|
||||
const srcIs = (...s: ReleaseSource[]): Frag => ({ hard: { sources: s }, unknown: { source: "reject" } });
|
||||
/** 取反来源条件:来源未知 = MP 正则不命中 = !X 为真 = 放行 */
|
||||
const srcNot = (...s: ReleaseSource[]): Frag => ({ hard: { sources: without(...s) }, unknown: { source: "accept" } });
|
||||
const CNSUB: Frag = { hard: { requiredSubtitles: ["zh"] }, unknown: { subtitles: "reject" } };
|
||||
const size = (min?: number, max?: number): Frag => ({
|
||||
hard: {
|
||||
minSizeBytes: min !== undefined ? min * MB : undefined,
|
||||
maxSizeBytes: max !== undefined ? max * MB : undefined,
|
||||
},
|
||||
});
|
||||
|
||||
/** AND 合并片段(同字段取交集由调用方保证不冲突) */
|
||||
function and(...fs: Frag[]): SlotSpec {
|
||||
const hard: SlotSpec["hard"] = {};
|
||||
const unknown: Record<string, "reject" | "accept"> = {};
|
||||
for (const f of fs) {
|
||||
for (const [k, v] of Object.entries(f.hard)) if (v !== undefined) (hard as Record<string, unknown>)[k] = v;
|
||||
Object.assign(unknown, f.unknown);
|
||||
}
|
||||
return { name: "mp", hard, unknownPolicy: { ...unknown } };
|
||||
}
|
||||
|
||||
/** 每条规则 → 各优先级层级的 SlotSpec(verdict=none 的规则不翻译) */
|
||||
const TRANSLATE: Record<string, SlotSpec[]> = {
|
||||
R01: [and(RES_4K)],
|
||||
R02: [and(RES_1080)],
|
||||
R03: [and(RES_720)],
|
||||
R04: [and({ hard: { resolutions: ["2160p", "1080p", "1080i"] }, unknown: { resolution: "reject" } })],
|
||||
R05: [and(RES_4K), and(RES_1080), and(RES_720)],
|
||||
R06: [and(srcIs("BluRay-Disc"))],
|
||||
R07: [and(srcNot("BluRay-Disc"))],
|
||||
R08: [and(srcIs("REMUX"))],
|
||||
R09: [and(srcNot("REMUX"))],
|
||||
R10: [and(srcIs("WEB-DL", "WEBRip"))],
|
||||
R11: [and(srcNot("BluRay-Disc", "REMUX"))],
|
||||
R12: [and(RES_4K, srcNot("REMUX")), and(RES_1080, srcIs("WEB-DL", "WEBRip"))],
|
||||
R13: [and(CNSUB)],
|
||||
R14: [and(CNSUB, RES_4K, srcNot("BluRay-Disc"))],
|
||||
R15: [and(CNSUB, RES_1080), and(RES_1080)],
|
||||
R16: [and(size(2000, 30000))],
|
||||
R17: [and(size(undefined, 8000))],
|
||||
R18: [and(RES_4K, size(2000, 30000))],
|
||||
R19: [and(srcNot("BluRay-Disc", "REMUX", "WEB-DL", "WEBRip"))],
|
||||
R20: [and({ hard: { sources: ["BluRay"], requireEncode: true }, unknown: { source: "reject" } })],
|
||||
R21: [and(RES_4K, srcNot("BluRay-Disc", "REMUX")), and(RES_1080, srcNot("BluRay-Disc", "REMUX"))],
|
||||
R22: [
|
||||
and(
|
||||
{ hard: { resolutions: ["2160p", "1080p", "1080i"] }, unknown: { resolution: "reject" } },
|
||||
srcIs("WEB-DL", "WEBRip"),
|
||||
),
|
||||
],
|
||||
// partial:近似翻译
|
||||
R23: [and(CNSUB)], // 无 TMDB 原语种放行
|
||||
R24: [and(RES_4K)], // 无 !DOLBY
|
||||
R25: [and(CNSUB, srcNot("BluRay-Disc"))], // 无 SPECSUB
|
||||
R26: [and(size(20000))],
|
||||
};
|
||||
|
||||
const parsed = new Map<string, ParsedRelease>(
|
||||
fx.torrents.map((t) => [
|
||||
t.id,
|
||||
{ ...parseRelease(t.title, t.description ? { subtitle: t.description } : {}), sizeBytes: t.size },
|
||||
]),
|
||||
);
|
||||
|
||||
function ourLevel(ruleId: string, torrentId: string): number | null {
|
||||
const levels = TRANSLATE[ruleId]!;
|
||||
const r = parsed.get(torrentId)!;
|
||||
for (let i = 0; i < levels.length; i++) if (evaluateCandidate(levels[i]!, r).decision === "accept") return i + 1;
|
||||
return null;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- 排序语义:单 Slot + rankForSlot vs MP 优先级
|
||||
/** MP `4K > 1080P > 720P` 用单个 Slot(允许三种分辨率)+ rankForSlot 表达时的逆序对 */
|
||||
const RANK_CHECKS: Record<string, { rule: string; spec: SlotSpec }> = {
|
||||
"R05~rank": {
|
||||
rule: "R05",
|
||||
spec: and({ hard: { resolutions: ["2160p", "1080p", "1080i", "720p"] }, unknown: { resolution: "reject" } }),
|
||||
},
|
||||
};
|
||||
|
||||
function rankInversions(key: string): string[] {
|
||||
const { rule, spec } = RANK_CHECKS[key]!;
|
||||
const level = new Map(fx.cases.filter((c) => c.ruleId === rule).map((c) => [c.torrentId, c.mp.level]));
|
||||
const ranked = rankForSlot(
|
||||
spec,
|
||||
fx.torrents.map((t) => ({ ...parsed.get(t.id)!, raw: t.id })),
|
||||
).filter((e) => e.evaluation.decision === "accept");
|
||||
const order = ranked.map((e) => e.candidate.raw);
|
||||
const out: string[] = [];
|
||||
for (let i = 0; i < order.length; i++)
|
||||
for (let j = i + 1; j < order.length; j++) {
|
||||
const a = level.get(order[i]!);
|
||||
const b = level.get(order[j]!);
|
||||
// 我们把 a 排在 b 前,MP 却认为 b 的层级更优
|
||||
if (a != null && b != null && b < a) out.push(`${order[j]}>${order[i]}`);
|
||||
}
|
||||
return out.sort();
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- 断言
|
||||
describe("MP 过滤规则差分(filter-cases.json)", () => {
|
||||
it("fixture 完整:≥20 种子,Python/Rust 一致,每条可表达规则都有翻译", () => {
|
||||
expect(fx.torrents.length).toBeGreaterThanOrEqual(20);
|
||||
expect(fx.generatedFrom.rustCrossCheck.mismatches).toEqual([]);
|
||||
const translatable = fx.rules.filter((r) => r.verdict !== "none").map((r) => r.id);
|
||||
expect(Object.keys(TRANSLATE).sort()).toEqual(translatable.sort());
|
||||
});
|
||||
|
||||
it("不一致集合 == deviations 登记集合", () => {
|
||||
const actual: Record<string, { mp: number | null; ours: number | null }> = {};
|
||||
for (const c of fx.cases) {
|
||||
if (!TRANSLATE[c.ruleId]) continue;
|
||||
const ours = ourLevel(c.ruleId, c.torrentId);
|
||||
if (ours !== c.mp.level) actual[c.caseId] = { mp: c.mp.level, ours };
|
||||
}
|
||||
const registered = Object.keys(fx.deviations).filter((k) => !k.includes("~"));
|
||||
expect(Object.keys(actual).sort()).toEqual(registered.sort());
|
||||
for (const k of registered)
|
||||
expect(["intentional", "missing", "our_bug", "mp_bug", "not_expressible"]).toContain(fx.deviations[k]!.kind);
|
||||
});
|
||||
|
||||
it("单 Slot + rankForSlot 的排序逆序对 == 登记", () => {
|
||||
for (const key of Object.keys(RANK_CHECKS)) {
|
||||
const inv = rankInversions(key);
|
||||
const dev = fx.deviations[key];
|
||||
if (inv.length === 0) expect(dev).toBeUndefined();
|
||||
else expect(dev?.pairs?.slice().sort()).toEqual(inv);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
/** 生成/更新登记时用:UMT_FILTER_DUMP=1 deno task test 打印当前不一致 */
|
||||
if (process.env.UMT_FILTER_DUMP) {
|
||||
const rows: Record<string, unknown> = {};
|
||||
for (const c of fx.cases) {
|
||||
if (!TRANSLATE[c.ruleId]) continue;
|
||||
const ours = ourLevel(c.ruleId, c.torrentId);
|
||||
if (ours !== c.mp.level) rows[c.caseId] = { mp: c.mp.level, ours };
|
||||
}
|
||||
for (const key of Object.keys(RANK_CHECKS)) rows[key] = rankInversions(key);
|
||||
console.log(`UMT_FILTER_DUMP ${JSON.stringify(rows)}`);
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,233 @@
|
||||
/**
|
||||
* 标题识别差分:把 MP MetaInfo 输出与我们的 ParsedRelease 映射到同一"规范形",逐字段比较。
|
||||
* 规范化只抹平表示差异(大小写、分隔符、同义写法),不抹平语义差异;语义差异登记在 meta-deviations.json。
|
||||
*/
|
||||
import { readFileSync } from "node:fs";
|
||||
import { type ParsedRelease, parseRelease, parseReleasePath } from "../../src/release-parser/index.ts";
|
||||
|
||||
export interface MetaSample {
|
||||
id: string;
|
||||
kind: "title" | "path";
|
||||
input: string;
|
||||
subtitle?: string;
|
||||
source: string;
|
||||
ourNote?: string;
|
||||
}
|
||||
|
||||
export type MpMeta = Record<string, string | number | null> & { error?: string };
|
||||
|
||||
export type DeviationKind = "intentional" | "missing" | "our_bug" | "mp_bug" | "representation";
|
||||
export interface Deviation {
|
||||
kind: DeviationKind;
|
||||
mp: string | null;
|
||||
ours: string | null;
|
||||
note: string;
|
||||
}
|
||||
|
||||
/** 参与比较的字段(media_id = find_metainfo 显式 ID 标签;edition = resource_type + resource_effect,由这两项覆盖;customization 双方均未配置) */
|
||||
export const FIELDS = [
|
||||
"name",
|
||||
"cn_name",
|
||||
"en_name",
|
||||
"year",
|
||||
"type",
|
||||
"begin_season",
|
||||
"end_season",
|
||||
"begin_episode",
|
||||
"end_episode",
|
||||
"resource_pix",
|
||||
"resource_type",
|
||||
"resource_effect",
|
||||
"video_encode",
|
||||
"audio_encode",
|
||||
"resource_team",
|
||||
"part",
|
||||
"web_source",
|
||||
"media_id",
|
||||
] as const;
|
||||
export type Field = (typeof FIELDS)[number];
|
||||
export type Canon = Record<Field, string | null>;
|
||||
|
||||
const readJson = <T>(name: string): T => JSON.parse(readFileSync(new URL(`./${name}`, import.meta.url), "utf8")) as T;
|
||||
|
||||
export function loadMeta() {
|
||||
const samples = readJson<{ samples: MetaSample[] }>("meta-samples.json").samples;
|
||||
const mp = readJson<{ python: Record<string, MpMeta>; rustMismatch: Record<string, unknown> }>("meta-mp.json");
|
||||
const deviations = readJson<{ deviations: Record<string, Partial<Record<Field, Deviation>>> }>(
|
||||
"meta-deviations.json",
|
||||
).deviations;
|
||||
return { samples, mp, deviations };
|
||||
}
|
||||
|
||||
const str = (v: unknown): string | null => (v === null || v === undefined || v === "" ? null : String(v));
|
||||
/** 名称:忽略大小写与标点/空白(MP 英文名做 Title Case,我们保留原大小写) */
|
||||
const canonName = (v: unknown) => {
|
||||
const s = str(v);
|
||||
return s === null
|
||||
? null
|
||||
: s
|
||||
.normalize("NFKC")
|
||||
.toLowerCase()
|
||||
.replace(/[\s._\-:：·'’!！?？,，&]+/g, "") || null;
|
||||
};
|
||||
|
||||
function canonSource(v: string | null): string | null {
|
||||
if (!v) return null;
|
||||
const u = v.toUpperCase();
|
||||
const cls = /REMUX/.test(u)
|
||||
? "REMUX"
|
||||
: /BLURAY|BDRIP|\bBD\b|BLU/.test(u)
|
||||
? "BluRay"
|
||||
: /WEB-?DL|^WEB$|\bWEB\b(?!RIP)/.test(u)
|
||||
? "WEB-DL"
|
||||
: /WEBRIP/.test(u)
|
||||
? "WEBRip"
|
||||
: /HDTV|UHDTV/.test(u)
|
||||
? "HDTV"
|
||||
: /HDRIP/.test(u)
|
||||
? "HDRip"
|
||||
: /DVD/.test(u)
|
||||
? "DVD"
|
||||
: /HDDVD/.test(u)
|
||||
? "HDDVD"
|
||||
: null;
|
||||
const uhd = /\bUHD\b/.test(u.replace(/UHDTV/g, "")) ? "UHD " : "";
|
||||
return `${uhd}${cls ?? `?${u}`}`.trim();
|
||||
}
|
||||
|
||||
const HDR_CANON: Record<string, string> = {
|
||||
HDR: "HDR10",
|
||||
HDR10: "HDR10",
|
||||
"HDR10+": "HDR10+",
|
||||
HDR10PLUS: "HDR10+",
|
||||
HDR10P: "HDR10+",
|
||||
DV: "DV",
|
||||
DOVI: "DV",
|
||||
HLG: "HLG",
|
||||
HDRVIVID: "HDR Vivid",
|
||||
VIVID: "HDR Vivid",
|
||||
"HDR VIVID": "HDR Vivid",
|
||||
};
|
||||
function canonHdrList(tokens: string[]): string | null {
|
||||
const set = new Set<string>();
|
||||
for (const t of tokens) {
|
||||
const k = t.toUpperCase().replace(/\s+/g, "");
|
||||
set.add(HDR_CANON[k] ?? HDR_CANON[t.toUpperCase()] ?? `?${t}`);
|
||||
}
|
||||
return set.size ? [...set].sort().join(",") : null;
|
||||
}
|
||||
|
||||
function canonCodec(v: string | null): string | null {
|
||||
if (!v) return null;
|
||||
const u = v.toUpperCase().replace(/[\s.]/g, "");
|
||||
if (/265|HEVC/.test(u)) return "H.265";
|
||||
if (/264|AVC/.test(u)) return "H.264";
|
||||
if (/AV1/.test(u)) return "AV1";
|
||||
if (/VC-?1/.test(u)) return "VC-1";
|
||||
if (/MPEG-?2/.test(u)) return "MPEG-2";
|
||||
if (/XVID/.test(u)) return "XVID";
|
||||
if (/DIVX/.test(u)) return "DIVX";
|
||||
return `?${u}`;
|
||||
}
|
||||
|
||||
const canonAudio = (v: string | null) =>
|
||||
v
|
||||
? v
|
||||
.toUpperCase()
|
||||
.replace(/DD\+/g, "DDP")
|
||||
.replace(/[\s\-_.]/g, "") || null
|
||||
: null;
|
||||
|
||||
const canonGroups = (v: string | null) =>
|
||||
v
|
||||
? v
|
||||
.split(/[@&]/)
|
||||
.map((g) => g.trim().toLowerCase())
|
||||
.filter(Boolean)
|
||||
.sort()
|
||||
.join("@")
|
||||
: null;
|
||||
|
||||
function canonPix(v: string | null): string | null {
|
||||
if (!v) return null;
|
||||
const u = v.toLowerCase();
|
||||
return ({ "4k": "2160p", "8k": "4320p", "2k": "1440p" } as Record<string, string>)[u] ?? u;
|
||||
}
|
||||
|
||||
const canonPart = (v: string | null) => (v ? v.toUpperCase().replace(/[\s._-]/g, "") : null);
|
||||
const canonType = (v: string | null) =>
|
||||
v === "电视剧" ? "tv" : v === "电影" ? "movie" : v === "未知" || v === null ? "unknown" : v;
|
||||
|
||||
export function canonMp(m: MpMeta): Canon {
|
||||
const effect = str(m.resource_effect);
|
||||
const hdrTokens = effect ? effect.split(/\s+/).filter((t) => /^(HDR|DV|DOVI|HLG|VIVID|HDRVIVID)/i.test(t)) : [];
|
||||
return {
|
||||
name: canonName(m.name),
|
||||
cn_name: canonName(m.cn_name),
|
||||
en_name: canonName(m.en_name),
|
||||
year: str(m.year),
|
||||
type: canonType(str(m.type)),
|
||||
begin_season: str(m.begin_season),
|
||||
end_season: str(m.end_season),
|
||||
begin_episode: str(m.begin_episode),
|
||||
end_episode: str(m.end_episode),
|
||||
resource_pix: canonPix(str(m.resource_pix)),
|
||||
resource_type: canonSource(str(m.resource_type)),
|
||||
resource_effect: canonHdrList(hdrTokens),
|
||||
video_encode: canonCodec(str(m.video_encode)),
|
||||
audio_encode: canonAudio(str(m.audio_encode)),
|
||||
resource_team: canonGroups(str(m.resource_team)),
|
||||
part: canonPart(str(m.part)),
|
||||
web_source: canonName(m.web_source),
|
||||
media_id: m.media_id ? `${m.media_source}:${m.media_id}` : null,
|
||||
};
|
||||
}
|
||||
|
||||
const PROVIDER: Record<string, string> = { tmdb: "themoviedb", douban: "douban", imdb: "imdb", tvdb: "tvdb" };
|
||||
|
||||
export function oursParse(s: MetaSample): ParsedRelease {
|
||||
return s.kind === "path"
|
||||
? parseReleasePath(s.input)
|
||||
: parseRelease(s.input, s.subtitle ? { subtitle: s.subtitle } : {});
|
||||
}
|
||||
|
||||
export function canonOurs(r: ParsedRelease): Canon {
|
||||
const src = r.source
|
||||
? `${r.uhd ? "UHD " : ""}${r.source === "BluRay-Disc" ? "BluRay" : r.source === "DVDRip" ? "DVD" : r.source}`
|
||||
: r.uhd
|
||||
? "UHD ?"
|
||||
: null;
|
||||
const eps = r.episodes;
|
||||
return {
|
||||
name: canonName(r.title),
|
||||
cn_name: canonName(r.titleCn),
|
||||
en_name: canonName(r.titleEn),
|
||||
year: str(r.year),
|
||||
type: r.mediaTypeHint,
|
||||
begin_season: str(r.season),
|
||||
end_season: str(r.seasonEnd),
|
||||
begin_episode: str(eps?.start),
|
||||
end_episode: eps && eps.end !== eps.start ? String(eps.end) : null,
|
||||
resource_pix: canonPix(str(r.resolution)),
|
||||
resource_type: src,
|
||||
resource_effect: r.hdr.length ? canonHdrList(r.hdr) : null,
|
||||
video_encode: canonCodec(str(r.videoCodec)),
|
||||
audio_encode: r.audio.length ? canonAudio(r.audio.join(" ")) : null,
|
||||
resource_team: canonGroups(str(r.releaseGroup?.name)),
|
||||
part: canonPart(str(r.part)),
|
||||
web_source: canonName(r.webSource),
|
||||
media_id: r.explicitIds[0] ? `${PROVIDER[r.explicitIds[0].provider]}:${r.explicitIds[0].id}` : null,
|
||||
};
|
||||
}
|
||||
|
||||
/** 逐字段差异(规范形不等) */
|
||||
export function diffSample(
|
||||
s: MetaSample,
|
||||
mp: MpMeta,
|
||||
): Partial<Record<Field, { mp: string | null; ours: string | null }>> {
|
||||
const a = canonMp(mp);
|
||||
const b = canonOurs(oursParse(s));
|
||||
const out: Partial<Record<Field, { mp: string | null; ours: string | null }>> = {};
|
||||
for (const f of FIELDS) if (a[f] !== b[f]) out[f] = { mp: a[f], ours: b[f] };
|
||||
return out;
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,35 @@
|
||||
/**
|
||||
* 标题识别差分测试:MP MetaInfo/MetaInfoPath(Python 参考实现)vs parseRelease/parseReleasePath。
|
||||
* 期望输出由 `scripts/mp-sync/gen-differential.py meta` 生成(meta-mp.json),已知差异登记在 meta-deviations.json。
|
||||
* 断言:实际差异集合 == 登记集合,且规范形取值与快照一致。补迁某条后测试会失败,提示删除对应登记。
|
||||
*/
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { diffSample, loadMeta } from "./meta-map.ts";
|
||||
|
||||
const { samples, mp, deviations } = loadMeta();
|
||||
|
||||
describe("标题识别差分(MP MetaInfo ↔ parseRelease)", () => {
|
||||
it("样本 ≥ 150 且每条都有 MP 输出", () => {
|
||||
expect(samples.length).toBeGreaterThanOrEqual(150);
|
||||
for (const s of samples) {
|
||||
expect(mp.python[s.id], s.id).toBeDefined();
|
||||
expect(mp.python[s.id]?.error, s.id).toBeUndefined();
|
||||
}
|
||||
});
|
||||
|
||||
it("登记项都对应存在的样本", () => {
|
||||
const ids = new Set(samples.map((s) => s.id));
|
||||
for (const id of Object.keys(deviations)) expect(ids.has(id), id).toBe(true);
|
||||
});
|
||||
|
||||
for (const s of samples) {
|
||||
it(`${s.id} ${s.input}`, () => {
|
||||
const actual = diffSample(s, mp.python[s.id]!);
|
||||
const registered = deviations[s.id] ?? {};
|
||||
const snapshot = Object.fromEntries(
|
||||
Object.entries(registered).map(([f, d]) => [f, { mp: d!.mp, ours: d!.ours }]),
|
||||
);
|
||||
expect(actual, `${s.id} 差异与 meta-deviations.json 登记不一致`).toEqual(snapshot);
|
||||
});
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"note": "繁→简差分已知差异:key=输入。kind 同 meta-deviations.json。",
|
||||
"deviations": {
|
||||
"咒術迴戰": { "kind": "missing", "mp": "咒术回战", "ours": "咒术迴战", "note": "字表缺 迴→回" },
|
||||
"孤獨搖滾": { "kind": "missing", "mp": "孤独摇滚", "ours": "孤独摇滾", "note": "字表缺 滾→滚" },
|
||||
"魷魚遊戲": { "kind": "missing", "mp": "鱿鱼游戏", "ours": "魷鱼游戏", "note": "字表缺 魷→鱿" },
|
||||
"瑯琊榜": { "kind": "missing", "mp": "琅琊榜", "ours": "瑯琊榜", "note": "字表缺 瑯→琅(异体)" },
|
||||
"慶餘年": {
|
||||
"kind": "mp_bug",
|
||||
"mp": "庆馀年",
|
||||
"ours": "庆余年",
|
||||
"note": "MediaWiki 词表把 餘 转成 馀,官方简体片名为 庆余年"
|
||||
},
|
||||
"臥虎藏龍": { "kind": "missing", "mp": "卧虎藏龙", "ours": "臥虎藏龙", "note": "字表缺 臥→卧" },
|
||||
"駭客任務": { "kind": "missing", "mp": "骇客任务", "ours": "駭客任务", "note": "字表缺 駭→骇" },
|
||||
"乾杯": { "kind": "missing", "mp": "干杯", "ours": "乾杯", "note": "一对多字 乾 刻意不收;MP 有词组表" },
|
||||
"瞭解": { "kind": "missing", "mp": "了解", "ours": "瞭解", "note": "一对多字 瞭 刻意不收;MP 有词组表" },
|
||||
"頭髮": { "kind": "missing", "mp": "头发", "ours": "头髮", "note": "髮→发 未收(发/發 合并字)" },
|
||||
"理髮": { "kind": "missing", "mp": "理发", "ours": "理髮", "note": "髮→发 未收" }
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,78 @@
|
||||
{
|
||||
"generatedFrom": {
|
||||
"repo": "MoviePilot",
|
||||
"commit": "e6820a8",
|
||||
"entry": "moviepilot_rust.zhconv_fast(text, 'zh-hans')"
|
||||
},
|
||||
"hans": {
|
||||
"葬送的芙莉蓮": "葬送的芙莉莲",
|
||||
"處刑少女的生存之道": "处刑少女的生存之道",
|
||||
"夏日重現": "夏日重现",
|
||||
"間諜家家酒": "间谍家家酒",
|
||||
"進擊的巨人": "进击的巨人",
|
||||
"鬼滅之刃": "鬼灭之刃",
|
||||
"咒術迴戰": "咒术回战",
|
||||
"我的英雄學院": "我的英雄学院",
|
||||
"藥師少女的獨語": "药师少女的独语",
|
||||
"無職轉生": "无职转生",
|
||||
"關於我轉生變成史萊姆這檔事": "关于我转生变成史莱姆这档事",
|
||||
"輝夜姬想讓人告白": "辉夜姬想让人告白",
|
||||
"排球少年!!": "排球少年!!",
|
||||
"鏈鋸人": "链锯人",
|
||||
"孤獨搖滾": "孤独摇滚",
|
||||
"黑暗榮耀": "黑暗荣耀",
|
||||
"與神同行": "与神同行",
|
||||
"魷魚遊戲": "鱿鱼游戏",
|
||||
"機智醫生生活": "机智医生生活",
|
||||
"請回答1988": "请回答1988",
|
||||
"陽光先生": "阳光先生",
|
||||
"鬼怪": "鬼怪",
|
||||
"來自星星的你": "来自星星的你",
|
||||
"後宮甄嬛傳": "后宫甄嬛传",
|
||||
"瑯琊榜": "琅琊榜",
|
||||
"慶餘年": "庆馀年",
|
||||
"琅琊榜之風起長林": "琅琊榜之风起长林",
|
||||
"甄嬛傳": "甄嬛传",
|
||||
"雍正王朝": "雍正王朝",
|
||||
"乾隆王朝": "乾隆王朝",
|
||||
"康熙王朝": "康熙王朝",
|
||||
"大明王朝1566": "大明王朝1566",
|
||||
"頭文字D": "头文字D",
|
||||
"無間道": "无间道",
|
||||
"少林足球": "少林足球",
|
||||
"功夫": "功夫",
|
||||
"臥虎藏龍": "卧虎藏龙",
|
||||
"英雄本色": "英雄本色",
|
||||
"倩女幽魂": "倩女幽魂",
|
||||
"喋血雙雄": "喋血双雄",
|
||||
"阿飛正傳": "阿飞正传",
|
||||
"花樣年華": "花样年华",
|
||||
"重慶森林": "重庆森林",
|
||||
"復仇者聯盟：終局之戰": "复仇者联盟：终局之战",
|
||||
"蜘蛛人：無家日": "蜘蛛人：无家日",
|
||||
"星際效應": "星际效应",
|
||||
"全面啟動": "全面启动",
|
||||
"駭客任務": "骇客任务",
|
||||
"魔戒首部曲：魔戒現身": "魔戒首部曲：魔戒现身",
|
||||
"神隱少女": "神隐少女",
|
||||
"霍爾的移動城堡": "霍尔的移动城堡",
|
||||
"乾杯": "干杯",
|
||||
"著作": "著作",
|
||||
"瞭解": "了解",
|
||||
"頭髮": "头发",
|
||||
"理髮": "理发",
|
||||
"發現": "发现",
|
||||
"後來": "后来",
|
||||
"皇后": "皇后",
|
||||
"範圍": "范围",
|
||||
"姓范": "姓范",
|
||||
"裡面": "里面",
|
||||
"計畫": "计画",
|
||||
"週末": "周末",
|
||||
"臺灣": "台湾",
|
||||
"台灣": "台湾",
|
||||
"麵包": "面包",
|
||||
"鬱金香": "郁金香",
|
||||
"舞台劇": "舞台剧"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,74 @@
|
||||
{
|
||||
"note": "繁→简差分样本(影视标题/人名/常见一对多字)。期望由 MP moviepilot_rust.zhconv_fast(text, 'zh-hans') 生成",
|
||||
"samples": [
|
||||
"葬送的芙莉蓮",
|
||||
"處刑少女的生存之道",
|
||||
"夏日重現",
|
||||
"間諜家家酒",
|
||||
"進擊的巨人",
|
||||
"鬼滅之刃",
|
||||
"咒術迴戰",
|
||||
"我的英雄學院",
|
||||
"藥師少女的獨語",
|
||||
"無職轉生",
|
||||
"關於我轉生變成史萊姆這檔事",
|
||||
"輝夜姬想讓人告白",
|
||||
"排球少年!!",
|
||||
"鏈鋸人",
|
||||
"孤獨搖滾",
|
||||
"黑暗榮耀",
|
||||
"與神同行",
|
||||
"魷魚遊戲",
|
||||
"機智醫生生活",
|
||||
"請回答1988",
|
||||
"陽光先生",
|
||||
"鬼怪",
|
||||
"來自星星的你",
|
||||
"後宮甄嬛傳",
|
||||
"瑯琊榜",
|
||||
"慶餘年",
|
||||
"琅琊榜之風起長林",
|
||||
"甄嬛傳",
|
||||
"雍正王朝",
|
||||
"乾隆王朝",
|
||||
"康熙王朝",
|
||||
"大明王朝1566",
|
||||
"頭文字D",
|
||||
"無間道",
|
||||
"少林足球",
|
||||
"功夫",
|
||||
"臥虎藏龍",
|
||||
"英雄本色",
|
||||
"倩女幽魂",
|
||||
"喋血雙雄",
|
||||
"阿飛正傳",
|
||||
"花樣年華",
|
||||
"重慶森林",
|
||||
"復仇者聯盟：終局之戰",
|
||||
"蜘蛛人：無家日",
|
||||
"星際效應",
|
||||
"全面啟動",
|
||||
"駭客任務",
|
||||
"魔戒首部曲：魔戒現身",
|
||||
"神隱少女",
|
||||
"霍爾的移動城堡",
|
||||
"乾杯",
|
||||
"著作",
|
||||
"瞭解",
|
||||
"頭髮",
|
||||
"理髮",
|
||||
"發現",
|
||||
"後來",
|
||||
"皇后",
|
||||
"範圍",
|
||||
"姓范",
|
||||
"裡面",
|
||||
"計畫",
|
||||
"週末",
|
||||
"臺灣",
|
||||
"台灣",
|
||||
"麵包",
|
||||
"鬱金香",
|
||||
"舞台劇"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
/**
|
||||
* 繁→简差分:MP moviepilot_rust.zhconv_fast(zh-hans) vs identity/toSimplified。
|
||||
* 期望由 `scripts/mp-sync/gen-differential.py zhconv` 生成;已知差异登记在 zhconv-deviations.json。
|
||||
*/
|
||||
import { readFileSync } from "node:fs";
|
||||
import { expect, it } from "vitest";
|
||||
import { toSimplified } from "../../src/identity/index.ts";
|
||||
|
||||
const read = (n: string) => JSON.parse(readFileSync(new URL(`./${n}`, import.meta.url), "utf8"));
|
||||
const mp: Record<string, string> = read("zhconv-mp.json").hans;
|
||||
const dev: Record<string, { mp: string; ours: string }> = read("zhconv-deviations.json").deviations;
|
||||
|
||||
it("toSimplified 与 MP zhconv 的差异集合 == 登记集合", () => {
|
||||
const actual: Record<string, { mp: string; ours: string }> = {};
|
||||
for (const [src, want] of Object.entries(mp)) {
|
||||
const ours = toSimplified(src);
|
||||
if (ours !== want) actual[src] = { mp: want, ours };
|
||||
}
|
||||
const snapshot = Object.fromEntries(Object.entries(dev).map(([k, d]) => [k, { mp: d.mp, ours: d.ours }]));
|
||||
expect(actual).toEqual(snapshot);
|
||||
expect(Object.keys(mp).length).toBeGreaterThanOrEqual(60);
|
||||
});
|
||||
@@ -0,0 +1,760 @@
|
||||
#!/usr/bin/env python3
|
||||
"""用 MoviePilot 的实现生成差分测试的期望输出(只读调用上游,不复制源码)。
|
||||
|
||||
用法(必须用 MP 自己的 venv 解释器,工作目录无要求):
|
||||
|
||||
MP_ROOT=/opt/moviepilot-plus/upstream/MoviePilot \
|
||||
$MP_ROOT/.venv/bin/python scripts/mp-sync/gen-differential.py meta
|
||||
... gen-differential.py filter # 过滤规则
|
||||
... gen-differential.py rss # RSS / 站点索引解析
|
||||
... gen-differential.py all
|
||||
|
||||
meta:读 packages/domain/test/differential/meta-samples.json,分别以
|
||||
accelerator=None(Python 参考实现)与 accelerator=rust(moviepilot_rust 快路径)
|
||||
调用 MetaInfo / MetaInfoPath,写 meta-mp.json。两条路径不一致的样本记在 rustMismatch。
|
||||
|
||||
生成后执行 `deno task fmt`(biome 统一 JSON 格式),再跑差分测试;新出现的差异需人工分类后
|
||||
登记进对应 *-deviations.json / deviations 字段(kind 见 docs/porting/mp-rust-logic-parity.md)。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import glob
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
from typing import Any, Optional
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
MP_ROOT = Path(os.environ.setdefault("MP_ROOT", "/opt/moviepilot-plus/upstream/MoviePilot"))
|
||||
DOMAIN_DIFF = REPO / "packages/domain/test/differential"
|
||||
ADAPTERS_DIFF = REPO / "packages/adapters/test/differential"
|
||||
|
||||
|
||||
def _bootstrap_mp() -> str:
|
||||
"""把 MP 放进 sys.path,隔离 CONFIG_DIR,返回上游提交号。"""
|
||||
os.environ.setdefault("CONFIG_DIR", tempfile.mkdtemp(prefix="mp-diff-cfg-"))
|
||||
if str(MP_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(MP_ROOT))
|
||||
os.chdir(MP_ROOT)
|
||||
try:
|
||||
return subprocess.check_output(["git", "-C", str(MP_ROOT), "rev-parse", "--short", "HEAD"], text=True).strip()
|
||||
except Exception: # noqa: BLE001
|
||||
return "unknown"
|
||||
|
||||
|
||||
META_FIELDS = (
|
||||
"name", "cn_name", "en_name", "year", "type", "begin_season", "end_season", "total_season",
|
||||
"begin_episode", "end_episode", "total_episode", "resource_pix", "resource_type", "resource_effect",
|
||||
"video_encode", "audio_encode", "resource_team", "edition", "part", "customization", "web_source",
|
||||
"fps", "video_bit", "media_source", "media_id",
|
||||
)
|
||||
|
||||
|
||||
def _meta_to_dict(meta) -> dict:
|
||||
out = {}
|
||||
for key in META_FIELDS:
|
||||
value = getattr(meta, key, None)
|
||||
if key == "type":
|
||||
value = value.value if value is not None else None
|
||||
elif key == "media_source" and value is not None:
|
||||
value = getattr(value, "value", str(value))
|
||||
if value == "":
|
||||
value = None
|
||||
out[key] = value
|
||||
out["class"] = type(meta).__name__
|
||||
return out
|
||||
|
||||
|
||||
def gen_meta() -> None:
|
||||
commit = _bootstrap_mp()
|
||||
from pathlib import Path as P
|
||||
|
||||
from app.adapters.system import rust
|
||||
from app.domain.meta.runtime import configure_recognition_runtime
|
||||
from app.domain.metainfo import MetaInfo, MetaInfoPath, clear_rust_parse_options_cache
|
||||
from app.runtime.config import settings
|
||||
|
||||
samples = json.loads((DOMAIN_DIFF / "meta-samples.json").read_text("utf-8"))["samples"]
|
||||
|
||||
def run(accelerator) -> dict:
|
||||
configure_recognition_runtime(
|
||||
media_extensions_provider=lambda: (*settings.RMT_MEDIAEXT, *settings.RMT_SUBEXT, *settings.RMT_AUDIOEXT),
|
||||
audio_extensions_provider=lambda: settings.RMT_AUDIOEXT,
|
||||
accelerator=accelerator,
|
||||
)
|
||||
clear_rust_parse_options_cache()
|
||||
res = {}
|
||||
for s in samples:
|
||||
try:
|
||||
if s["kind"] == "path":
|
||||
meta = MetaInfoPath(P(s["input"]))
|
||||
else:
|
||||
meta = MetaInfo(title=s["input"], subtitle=s.get("subtitle"))
|
||||
res[s["id"]] = _meta_to_dict(meta)
|
||||
except Exception as err: # noqa: BLE001
|
||||
res[s["id"]] = {"error": f"{type(err).__name__}: {err}"}
|
||||
return res
|
||||
|
||||
python_out = run(None)
|
||||
rust_out = run(rust) if rust.is_available() else {}
|
||||
mismatch = {}
|
||||
for sid, py in python_out.items():
|
||||
rs = rust_out.get(sid)
|
||||
if rs is None:
|
||||
continue
|
||||
diff = {k: {"python": py.get(k), "rust": rs.get(k)} for k in py if py.get(k) != rs.get(k)}
|
||||
if diff:
|
||||
mismatch[sid] = diff
|
||||
out = {
|
||||
"generatedFrom": {"repo": "MoviePilot", "commit": commit, "entry": "app.domain.metainfo.MetaInfo/MetaInfoPath",
|
||||
"rust": rust.status() if rust.is_available() else None},
|
||||
"python": python_out,
|
||||
"rustMismatch": mismatch,
|
||||
}
|
||||
(DOMAIN_DIFF / "meta-mp.json").write_text(json.dumps(out, ensure_ascii=False, indent=1) + "\n", "utf-8")
|
||||
print(f"meta: {len(python_out)} samples, rust≠python: {len(mismatch)}")
|
||||
|
||||
|
||||
def gen_zhconv() -> None:
|
||||
"""繁→简:moviepilot_rust.zhconv_fast 无 Python 回退(app/foundation/text.py 直接调用原生扩展)。"""
|
||||
commit = _bootstrap_mp()
|
||||
import moviepilot_rust
|
||||
|
||||
samples = json.loads((DOMAIN_DIFF / "zhconv-samples.json").read_text("utf-8"))["samples"]
|
||||
out = {
|
||||
"generatedFrom": {"repo": "MoviePilot", "commit": commit, "entry": "moviepilot_rust.zhconv_fast(text, 'zh-hans')"},
|
||||
"hans": {s: moviepilot_rust.zhconv_fast(s, "zh-hans") for s in samples},
|
||||
}
|
||||
(DOMAIN_DIFF / "zhconv-mp.json").write_text(json.dumps(out, ensure_ascii=False, indent=1) + "\n", "utf-8")
|
||||
print(f"zhconv: {len(samples)} samples")
|
||||
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- filter:过滤规则
|
||||
# 用 MP 的 Python 回退实现(FilterModule,rust_accel 关闭)对 TORRENTS x RULES 求匹配与优先级,
|
||||
# 并用 Rust filter_torrents_fast 交叉核对。无需数据库:规则组/规则集直接注入。
|
||||
# 已知差异登记(deviations)不由本脚本生成,重新生成时从已有 filter-cases.json 保留。
|
||||
|
||||
GiB = 1024 ** 3
|
||||
MiB = 1024 ** 2
|
||||
|
||||
# ---------------------------------------------------------------- 样本
|
||||
# size 为站点报告的总大小(字节);labels 为站点标签
|
||||
TORRENTS: list[dict[str, Any]] = [
|
||||
{"id": "t01", "title": "Dune.Part.Two.2024.2160p.UHD.BluRay.Remux.DV.HDR.HEVC.TrueHD.Atmos.7.1-FraMeSToR", "description": "", "size": 70 * GiB, "seeders": 20},
|
||||
{"id": "t02", "title": "Dune.Part.Two.2024.2160p.BluRay.x265.10bit.DV.HDR-CHD", "description": "沙丘2 中字 特效", "size": 25 * GiB, "seeders": 12},
|
||||
{"id": "t03", "title": "Dune Part Two 2024 2160p UHD Blu-ray HEVC DV TrueHD 7.1-BeyondHD", "description": "", "size": 90 * GiB, "seeders": 4},
|
||||
{"id": "t04", "title": "Dune.Part.Two.2024.1080p.WEB-DL.H264.DDP5.1-HHWEB", "description": "沙丘2 | 国语 中字", "size": 8 * GiB, "seeders": 50},
|
||||
{"id": "t05", "title": "Dune.Part.Two.2024.1080p.BluRay.x264.DTS-HD.MA.5.1-WiKi", "description": "简繁英字幕", "size": 15 * GiB, "seeders": 9},
|
||||
{"id": "t06", "title": "Dune.Part.Two.2024.720p.WEBRip.x264-GRP", "description": "", "size": 2 * GiB, "seeders": 1},
|
||||
{"id": "t07", "title": "Dune.Part.Two.2024.2160p.WEB-DL.DDP5.1.Atmos.DV.HDR.H.265-FLUX", "description": "", "size": 20 * GiB, "seeders": 30},
|
||||
{"id": "t08", "title": "Movie.2022.2160p.HDR.HEVC-NOSRC", "description": "", "size": 18 * GiB, "seeders": 2},
|
||||
{"id": "t09", "title": "The.Bear.S03.2160p.DSNP.WEB-DL.DDP5.1.HDR.H.265-NTb", "description": "熊家餐馆 第三季 全10集 中字", "size": 60 * GiB, "seeders": 15},
|
||||
{"id": "t10", "title": "The.Bear.S03E01.1080p.WEB-DL.H264-NTb", "description": "", "size": int(2.5 * GiB), "seeders": 8},
|
||||
{"id": "t11", "title": "Movie.2023.1080p.BluRay.REMUX.AVC.DTS-HD.MA.5.1-EPSiLON", "description": "", "size": 30 * GiB, "seeders": 6},
|
||||
{"id": "t12", "title": "Movie 2023 1080p Blu-ray AVC DTS-HD MA 5.1-AAA", "description": "", "size": 40 * GiB, "seeders": 3},
|
||||
{"id": "t13", "title": "Movie.2023.COMPLETE.BLURAY-GRP", "description": "", "size": 45 * GiB, "seeders": 2},
|
||||
{"id": "t14", "title": "Movie.2023.1080i.HDTV.H264-GRP", "description": "", "size": 10 * GiB, "seeders": 1},
|
||||
{"id": "t15", "title": "[Nekomoe kissaten] Frieren - 01 [WebRip 1080p HEVC-10bit AAC ASSx2].mkv", "description": "简繁日内封", "size": 1 * GiB, "seeders": 40},
|
||||
{"id": "t16", "title": "The.Wandering.Earth.II.2023.2160p.WEB-DL.H265.HDR.DDP5.1-HDSWEB", "description": "流浪地球2 国语配音 中文字幕", "size": 12 * GiB, "seeders": 25},
|
||||
{"id": "t17", "title": "Oppenheimer.2023.IMAX.2160p.BluRay.x265.10bit.HDR.DTS-HD.MA.5.1-SWTYBLZ", "description": "奥本海默 CHS/ENG", "size": 30 * GiB, "seeders": 7},
|
||||
{"id": "t18", "title": "Movie.2022.3840x2160.WEB-DL.HDR10.HEVC-GRP", "description": "", "size": 15 * GiB, "seeders": 5},
|
||||
{"id": "t19", "title": "Movie.2022.1920x1080.WEB-DL.H264-GRP", "description": "", "size": 6 * GiB, "seeders": 5},
|
||||
{"id": "t20", "title": "Movie.2022.4K.WEB-DL.H265-GRP", "description": "", "size": 14 * GiB, "seeders": 5},
|
||||
{"id": "t21", "title": "Movie.2022.1080p.WEB-DL.x264 [GB]", "description": "", "size": 4 * GiB, "seeders": 5},
|
||||
{"id": "t22", "title": "Movie.2022.1080p.WEB-DL.H264.AAC 39.23 GB", "description": "", "size": 4 * GiB, "seeders": 5},
|
||||
{"id": "t23", "title": "Some.Show.S01.1080p.BluRay.x264-GRP", "description": "全12集 简体中字", "size": 36 * GiB, "seeders": 3},
|
||||
{"id": "t24", "title": "Movie.2021.DVDRip.x264-GRP", "description": "", "size": int(1.4 * GiB), "seeders": 2},
|
||||
{"id": "t25", "title": "Movie 2021 1080p 3D BluRay Half-SBS x264-GRP", "description": "", "size": 8 * GiB, "seeders": 2},
|
||||
{"id": "t26", "title": "Movie.2021.2160p.WEB-DL.60fps.HEVC-GRP", "description": "60帧 国语", "size": 10 * GiB, "seeders": 4},
|
||||
{"id": "t27", "title": "Movie.2020.2160p.Remux.HEVC.DoVi-GRP", "description": "", "size": 50 * GiB, "seeders": 3},
|
||||
{"id": "t28", "title": "Movie.2020.1080p.NF.WEB-DL.DDP5.1.H.264-CHDWEB", "description": "官方 中字", "labels": ["官方"], "size": 5 * GiB, "seeders": 10, "downloadvolumefactor": 0},
|
||||
]
|
||||
|
||||
# 自定义规则(对应 MP「自定义规则」页:include/exclude 为单个正则字符串,size_range 单位 MB)
|
||||
CUSTOM_RULES: dict[str, dict[str, Any]] = {
|
||||
"SIZEMOVIE": {"size_range": "2000-30000"},
|
||||
"SIZESMALL": {"size_range": "<8000"},
|
||||
"SIZEBIG": {"size_range": ">20000"},
|
||||
"SEED5": {"seeders": "5"},
|
||||
}
|
||||
|
||||
# verdict: full=可完整表达 / partial=近似表达(测试按近似翻译比对,差异登记 not_expressible) / none=不可表达(只生成 MP 结果)
|
||||
RULES: list[dict[str, Any]] = [
|
||||
{"id": "R01", "rule": "4K", "verdict": "full"},
|
||||
{"id": "R02", "rule": "1080P", "verdict": "full"},
|
||||
{"id": "R03", "rule": "720P", "verdict": "full"},
|
||||
{"id": "R04", "rule": "4K | 1080P", "verdict": "full"},
|
||||
{"id": "R05", "rule": "4K > 1080P > 720P", "verdict": "full"},
|
||||
{"id": "R06", "rule": "BLU", "verdict": "full"},
|
||||
{"id": "R07", "rule": "!BLU", "verdict": "full"},
|
||||
{"id": "R08", "rule": "REMUX", "verdict": "full"},
|
||||
{"id": "R09", "rule": "!REMUX", "verdict": "full"},
|
||||
{"id": "R10", "rule": "WEBDL", "verdict": "full"},
|
||||
{"id": "R11", "rule": "!BLU & !REMUX", "verdict": "full"},
|
||||
{"id": "R12", "rule": "4K & !REMUX > 1080P & WEBDL", "verdict": "full"},
|
||||
{"id": "R13", "rule": "CNSUB", "verdict": "full"},
|
||||
{"id": "R14", "rule": "CNSUB & 4K & !BLU", "verdict": "full"},
|
||||
{"id": "R15", "rule": "CNSUB & 1080P > 1080P", "verdict": "full"},
|
||||
{"id": "R16", "rule": "SIZEMOVIE", "verdict": "full"},
|
||||
{"id": "R17", "rule": "SIZESMALL", "verdict": "full"},
|
||||
{"id": "R18", "rule": "4K & SIZEMOVIE", "verdict": "full"},
|
||||
{"id": "R19", "rule": "!BLU & !REMUX & !WEBDL", "verdict": "full"},
|
||||
{"id": "R20", "rule": "BLURAY & !REMUX & !BLU", "verdict": "full"},
|
||||
{"id": "R21", "rule": "4K & !BLU & !REMUX > 1080P & !BLU & !REMUX", "verdict": "full"},
|
||||
{"id": "R22", "rule": "(4K | 1080P) & WEBDL", "verdict": "full"},
|
||||
{"id": "R23", "rule": "CNSUB", "verdict": "partial", "mediainfo": {"original_language": "zh"},
|
||||
"note": "tmdb.original_language=zh 时 CNSUB 对任何种子直接放行"},
|
||||
{"id": "R24", "rule": "4K & !DOLBY", "verdict": "partial", "note": "硬条件无 HDR 维度,近似为 4K"},
|
||||
{"id": "R25", "rule": "SPECSUB & CNSUB & !BLU", "verdict": "partial", "note": "无特效字幕维度,近似为 CNSUB & !BLU"},
|
||||
{"id": "R26", "rule": "SIZEBIG", "verdict": "full"},
|
||||
# ---- 以下不可表达,只记录 MP 行为供报告/将来补迁
|
||||
{"id": "N01", "rule": "H265", "verdict": "none", "note": "编码只有软偏好 soft.codecs"},
|
||||
{"id": "N02", "rule": "!DOLBY & !REMUX", "verdict": "none", "note": "HDR 只有软偏好"},
|
||||
{"id": "N03", "rule": "HDR", "verdict": "none"},
|
||||
{"id": "N04", "rule": "SPECSUB", "verdict": "none"},
|
||||
{"id": "N05", "rule": "CNVOI", "verdict": "none", "note": "无音轨语言字段"},
|
||||
{"id": "N06", "rule": "60FPS", "verdict": "none"},
|
||||
{"id": "N07", "rule": "!3D", "verdict": "none"},
|
||||
{"id": "N08", "rule": "ATMOS", "verdict": "none"},
|
||||
{"id": "N09", "rule": "GZ", "verdict": "none", "note": "站点标签"},
|
||||
{"id": "N10", "rule": "FREE", "verdict": "none", "note": "站点促销"},
|
||||
{"id": "N11", "rule": "SEED5", "verdict": "none", "note": "做种数"},
|
||||
{"id": "N12", "rule": "CNSUB | 4K", "verdict": "none", "note": "跨字段 OR"},
|
||||
{"id": "N13", "rule": "!(4K & REMUX)", "verdict": "none", "note": "跨字段 NOT(AND) = OR"},
|
||||
{"id": "N14", "rule": "UHD", "verdict": "none", "note": "ParsedRelease.uhd 有,但硬条件没有"},
|
||||
{"id": "N15", "rule": "4K & H265 > 4K", "verdict": "none", "note": "level1 需编码硬条件"},
|
||||
{"id": "N16", "rule": "NOSUCHRULE | 4K", "verdict": "none", "note": "未知规则名视为不匹配"},
|
||||
]
|
||||
|
||||
|
||||
def _filter_mp_root(mp_root: Optional[str]) -> Path:
|
||||
root = mp_root or os.environ.get("MP_ROOT") or "/opt/moviepilot-plus/upstream/MoviePilot"
|
||||
return Path(root).resolve()
|
||||
|
||||
|
||||
def _filter_mp_commit(root: Path) -> str:
|
||||
try:
|
||||
return subprocess.check_output(["git", "-C", str(root), "rev-parse", "--short", "HEAD"], text=True).strip()
|
||||
except Exception: # noqa: BLE001
|
||||
return "unknown"
|
||||
|
||||
|
||||
def gen_filter(out_path: str, mp_root: Optional[str] = None) -> dict:
|
||||
root = _filter_mp_root(mp_root)
|
||||
if str(root) not in sys.path:
|
||||
sys.path.insert(0, str(root))
|
||||
old_cwd = os.getcwd()
|
||||
os.chdir(root)
|
||||
try:
|
||||
from unittest.mock import patch
|
||||
|
||||
from app.adapters.system import rust as rust_accel
|
||||
from app.application.rules import BUILTIN_RULE_SET
|
||||
from app.domain.context import TorrentInfo
|
||||
from app.modules.filter import FilterModule
|
||||
finally:
|
||||
os.chdir(old_cwd)
|
||||
|
||||
rule_set = {**BUILTIN_RULE_SET, **CUSTOM_RULES}
|
||||
|
||||
class _RuleHelper:
|
||||
def __init__(self, groups):
|
||||
self._groups = groups
|
||||
|
||||
def get_rule_group_by_media(self, media=None, group_names=None): # noqa: ARG002
|
||||
return self._groups
|
||||
|
||||
def mk(t: dict) -> Any:
|
||||
return TorrentInfo(
|
||||
site_name="S", title=t["title"], description=t.get("description") or "",
|
||||
size=t["size"], seeders=t.get("seeders") or 0, labels=t.get("labels") or [],
|
||||
downloadvolumefactor=t.get("downloadvolumefactor", 1.0),
|
||||
)
|
||||
|
||||
def run(rule: dict, python: bool) -> list[Optional[int]]:
|
||||
module = FilterModule()
|
||||
module.rulehelper = _RuleHelper([SimpleNamespace(name=rule["id"], rule_string=rule["rule"])])
|
||||
module.rule_set = rule_set
|
||||
mi = SimpleNamespace(**rule["mediainfo"]) if rule.get("mediainfo") else None
|
||||
out: list[Optional[int]] = []
|
||||
for t in TORRENTS: # 逐条过滤,和订阅匹配链路(单种子列表)一致
|
||||
ti = mk(t)
|
||||
if python:
|
||||
with patch("app.modules.filter.rust_accel.is_enabled", return_value=False):
|
||||
res = module.filter_torrents(rule_groups=[rule["id"]], torrent_list=[ti], mediainfo=mi)
|
||||
else:
|
||||
res = module.filter_torrents(rule_groups=[rule["id"]], torrent_list=[ti], mediainfo=mi)
|
||||
out.append(res[0].pri_order if res else None)
|
||||
return out
|
||||
|
||||
rust_ok = rust_accel.is_enabled()
|
||||
cases = []
|
||||
rust_mismatch = []
|
||||
for rule in RULES:
|
||||
py = run(rule, python=True)
|
||||
rs = run(rule, python=False) if rust_ok else py
|
||||
for t, p, r in zip(TORRENTS, py, rs):
|
||||
cid = f"{rule['id']}:{t['id']}"
|
||||
if p != r:
|
||||
rust_mismatch.append({"caseId": cid, "python": p, "rust": r})
|
||||
cases.append({
|
||||
"caseId": cid, "ruleId": rule["id"], "torrentId": t["id"],
|
||||
"mp": {"matched": p is not None, "priOrder": p, "level": (101 - p) if p is not None else None},
|
||||
})
|
||||
|
||||
out = Path(out_path)
|
||||
deviations: dict = {}
|
||||
if out.exists():
|
||||
try:
|
||||
deviations = json.loads(out.read_text("utf-8")).get("deviations", {})
|
||||
except Exception: # noqa: BLE001
|
||||
deviations = {}
|
||||
doc = {
|
||||
"generatedFrom": {
|
||||
"repo": "MoviePilot", "commit": _filter_mp_commit(root),
|
||||
"engine": "python-fallback (app.modules.filter FilterModule, rust_accel disabled)",
|
||||
"rustCrossCheck": {"available": rust_ok, "mismatches": rust_mismatch},
|
||||
"generator": "scripts/mp-sync/gen-differential.py filter",
|
||||
"semantics": "level = 1-based index of first matching '>' tier; priOrder = 101 - level (MP pri_order)",
|
||||
},
|
||||
"customRules": CUSTOM_RULES,
|
||||
"torrents": TORRENTS,
|
||||
"rules": RULES,
|
||||
"cases": cases,
|
||||
"deviations": deviations,
|
||||
}
|
||||
out.parent.mkdir(parents=True, exist_ok=True)
|
||||
out.write_text(json.dumps(doc, ensure_ascii=False, indent=1) + "\n", "utf-8")
|
||||
return doc
|
||||
|
||||
|
||||
def gen_filter_cmd() -> None:
|
||||
d = gen_filter(str(DOMAIN_DIFF / "filter-cases.json"), str(MP_ROOT))
|
||||
print(f"filter: cases={len(d['cases'])} rust≠python={len(d['generatedFrom']['rustCrossCheck']['mismatches'])}")
|
||||
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- rss / indexer:RSS 与站点列表页解析
|
||||
# MoviePilot RSS / 站点索引解析 差分期望生成器。
|
||||
#
|
||||
# 用 MoviePilot(MP_ROOT,须用其 .venv 的 python 运行)的 **Python 回退路径** 解析本项目的 fixture,
|
||||
# 把输入、MP 输出(同时附带 Rust 快路径输出作参考)写成 JSON,供
|
||||
# `packages/adapters/test/differential/{rss,indexer}.test.ts` 与本项目解析器逐字段比对。
|
||||
#
|
||||
# - RSS:`app/application/rss.py RssHelper._parse_impl`(注入假 HTTP/解析端口,解析端口返回 None 强制走
|
||||
# lxml 回退;`parse_rss_items_fast` 另行直接调用记录到 `mp_rust`)。
|
||||
# - 站点索引:`app/modules/indexer/spider/__init__.py SiteSpider.parse`(把 `rust_accel.parse_indexer_torrents/
|
||||
# parse_indexer_subtitles` 打桩为 None 强制走 PyQuery 回退;再恢复原函数跑一次记录 `mp_rust`)。
|
||||
# MP 自带站点定义在加密资源 `user.sites.v3.bin`(Cython `SitesHelper`),未认证用户 `get_indexer()` 返回 {},
|
||||
# `get_indexsites()` 只含 id/name/url,因此 indexer 配置改用本项目 `definitions/*.json`(同 MP schema)。
|
||||
# 输入先由 `deno run -A packages/adapters/test/differential/export-inputs.ts` 导出 tk-* fixture。
|
||||
|
||||
UMT_ROOT = REPO
|
||||
DEFAULT_OUT = UMT_ROOT / "packages/adapters/test/differential"
|
||||
DEFINITIONS_DIR = UMT_ROOT / "packages/adapters/src/sites/definitions"
|
||||
|
||||
_BOOTSTRAPPED = False
|
||||
|
||||
|
||||
def _mp_root() -> Path:
|
||||
root = os.environ.get("MP_ROOT")
|
||||
if not root:
|
||||
raise SystemExit("需要环境变量 MP_ROOT(MoviePilot 源码根目录)")
|
||||
return Path(root).resolve()
|
||||
|
||||
|
||||
def _bootstrap() -> Path:
|
||||
"""隔离 CONFIG_DIR 并建表(复用 MP 测试引导),固定进程时区为 UTC,返回 MP_ROOT。"""
|
||||
global _BOOTSTRAPPED
|
||||
root = _mp_root()
|
||||
if _BOOTSTRAPPED:
|
||||
return root
|
||||
# 时区固定为 UTC:RSS 的 astimezone(None) 与站点 normalize_datetime 结果与宿主时区无关
|
||||
os.environ["TZ"] = "UTC"
|
||||
time.tzset()
|
||||
if str(root) not in sys.path:
|
||||
sys.path.insert(0, str(root))
|
||||
cwd = os.getcwd()
|
||||
os.chdir(root)
|
||||
try:
|
||||
from app.testing.bootstrap import prepare_backend # noqa: WPS433
|
||||
|
||||
prepare_backend()
|
||||
# 同 MP tests/conftest.py configure_plugin_system_services:装配类型化配置快照与 runtime setting
|
||||
from app.application.configuration import (
|
||||
RuntimeConfiguration,
|
||||
RuntimeSettingsService,
|
||||
configure_runtime_configuration,
|
||||
configure_runtime_settings,
|
||||
)
|
||||
from app.runtime.config import settings
|
||||
from app.runtime.settings import configure_runtime_setting_provider
|
||||
from app.startup.composition.configuration import (
|
||||
build_api_runtime_config,
|
||||
build_chain_runtime_config,
|
||||
build_scheduler_runtime_config,
|
||||
)
|
||||
|
||||
configure_runtime_configuration(RuntimeConfiguration(
|
||||
api=lambda: build_api_runtime_config(settings),
|
||||
scheduler=lambda: build_scheduler_runtime_config(settings),
|
||||
chain=lambda: build_chain_runtime_config(settings),
|
||||
))
|
||||
configure_runtime_settings(RuntimeSettingsService(settings))
|
||||
configure_runtime_setting_provider(lambda key: getattr(settings, key))
|
||||
_freeze_dateparser_now()
|
||||
finally:
|
||||
os.chdir(cwd)
|
||||
_BOOTSTRAPPED = True
|
||||
return root
|
||||
|
||||
|
||||
# MP normalize_datetime 用 dateparser.parse(value) 无 settings,相对时间("3小时前")按当前时刻换算;
|
||||
# 为使期望文件可复现,冻结其相对基准。
|
||||
FROZEN_NOW = "2026-09-27T12:00:00"
|
||||
|
||||
|
||||
def _freeze_dateparser_now() -> None:
|
||||
import datetime as _dt
|
||||
|
||||
import dateparser as _real
|
||||
from app.foundation import temporal
|
||||
|
||||
base = _dt.datetime.fromisoformat(FROZEN_NOW)
|
||||
|
||||
class _Shim:
|
||||
@staticmethod
|
||||
def parse(value, *args, **kwargs):
|
||||
settings = dict(kwargs.pop("settings", None) or {})
|
||||
settings.setdefault("RELATIVE_BASE", base)
|
||||
return _real.parse(value, *args, settings=settings, **kwargs)
|
||||
|
||||
temporal.dateparser = _Shim()
|
||||
|
||||
|
||||
def _mp_commit(root: Path) -> str:
|
||||
try:
|
||||
return subprocess.check_output(["git", "-C", str(root), "rev-parse", "HEAD"], text=True).strip()
|
||||
except Exception: # noqa: BLE001
|
||||
return "unknown"
|
||||
|
||||
|
||||
def _sha256(data: bytes) -> str:
|
||||
return hashlib.sha256(data).hexdigest()
|
||||
|
||||
|
||||
def _load_existing(path: Path) -> dict:
|
||||
if path.exists():
|
||||
try:
|
||||
return json.loads(path.read_text("utf-8"))
|
||||
except Exception: # noqa: BLE001
|
||||
return {}
|
||||
return {}
|
||||
|
||||
|
||||
def _jsonable(v: Any) -> Any:
|
||||
import datetime as _dt
|
||||
|
||||
if isinstance(v, _dt.datetime):
|
||||
return v.isoformat()
|
||||
if isinstance(v, dict):
|
||||
return {k: _jsonable(x) for k, x in v.items()}
|
||||
if isinstance(v, (list, tuple)):
|
||||
return [_jsonable(x) for x in v]
|
||||
return v
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- RSS
|
||||
|
||||
|
||||
def _run_mp_rss_python(xml_text: str) -> dict:
|
||||
"""走 RssHelper._parse_impl 的 lxml 回退分支;返回 {status, items}。"""
|
||||
import datetime as _dt
|
||||
|
||||
from app.adapters.network.http import RequestUtils
|
||||
from app.application.rss import RssHelper, configure_rss_ports, reset_rss_ports
|
||||
|
||||
raw = xml_text.encode("utf-8")
|
||||
|
||||
class _Http:
|
||||
def get(self, **_kw):
|
||||
return SimpleNamespace(status_code=200, content=raw, text=xml_text,
|
||||
apparent_encoding="utf-8", encoding="utf-8", headers={})
|
||||
|
||||
def post(self, **_kw): # pragma: no cover
|
||||
raise AssertionError("不应 POST")
|
||||
|
||||
@staticmethod
|
||||
def decode_xml(response, **kw):
|
||||
return RequestUtils.get_decoded_xml_content(response, **kw)
|
||||
|
||||
class _Browser:
|
||||
@staticmethod
|
||||
def render(**_kw): # pragma: no cover
|
||||
raise AssertionError("不应访问浏览器")
|
||||
|
||||
class _Parser:
|
||||
@staticmethod
|
||||
def parse(_content, _max):
|
||||
return None # 关闭 Rust,强制 Python 回退
|
||||
|
||||
calls: list = []
|
||||
original = RssHelper._parse_publish_time
|
||||
|
||||
def _spy(value):
|
||||
dt = original(value)
|
||||
calls.append({"text": value, "naive": bool(dt is not None and dt.tzinfo is None),
|
||||
"wall": dt.replace(tzinfo=None).isoformat() if dt is not None else None})
|
||||
return dt
|
||||
|
||||
RssHelper._parse_publish_time = staticmethod(_spy)
|
||||
configure_rss_ports(http=_Http(), browser=_Browser(), parser=_Parser())
|
||||
try:
|
||||
result = RssHelper().parse("https://differential.invalid/rss")
|
||||
finally:
|
||||
reset_rss_ports()
|
||||
RssHelper._parse_publish_time = staticmethod(original)
|
||||
|
||||
if result is None:
|
||||
return {"status": "none_expired", "items": []}
|
||||
if result is False:
|
||||
return {"status": "false_error", "items": []}
|
||||
items = []
|
||||
for it in result:
|
||||
out = {k: it.get(k) for k in ("title", "description", "link", "enclosure", "size")}
|
||||
if "nickname" in it:
|
||||
out["nickname"] = it["nickname"]
|
||||
pd = it.get("pubdate")
|
||||
if pd == "":
|
||||
out["pubdate"] = {"kind": "absent"}
|
||||
else:
|
||||
call = calls.pop(0) if calls else {"text": None, "naive": False, "wall": None}
|
||||
if pd is None:
|
||||
out["pubdate"] = {"kind": "unparsed", "text": call["text"]}
|
||||
else:
|
||||
utc = pd.astimezone(_dt.timezone.utc)
|
||||
out["pubdate"] = {
|
||||
"kind": "naive" if call["naive"] else "aware",
|
||||
"text": call["text"],
|
||||
# naive:MP 以宿主本地时区解释(astimezone(None));wall 为墙上时间
|
||||
"wall": call["wall"],
|
||||
"utc": utc.strftime("%Y-%m-%dT%H:%M:%S.") + f"{utc.microsecond // 1000:03d}Z",
|
||||
}
|
||||
items.append(out)
|
||||
return {"status": "ok", "items": items}
|
||||
|
||||
|
||||
def _run_mp_rss_rust(xml_text: str) -> Optional[dict]:
|
||||
try:
|
||||
import moviepilot_rust # type: ignore
|
||||
except Exception as err: # noqa: BLE001
|
||||
return {"status": "unavailable", "error": str(err)}
|
||||
stripped = xml_text.strip()
|
||||
if not stripped or not stripped.startswith("<"):
|
||||
return {"status": "not_called", "note": "RssHelper.__parse_with_rust 对非 < 开头内容直接返回 None"}
|
||||
try:
|
||||
return {"status": "ok", "items": _jsonable(moviepilot_rust.parse_rss_items_fast(xml_text, 1001))}
|
||||
except BaseException as err: # noqa: BLE001
|
||||
return {"status": "error", "error": f"{type(err).__name__}: {err}"}
|
||||
|
||||
|
||||
def gen_rss(out_dir: Optional[str] = None) -> Path:
|
||||
"""生成 <out_dir>/rss.json;返回文件路径。"""
|
||||
root = _bootstrap()
|
||||
out = Path(out_dir).resolve() if out_dir else DEFAULT_OUT
|
||||
inputs = sorted(glob.glob(str(out / "inputs/rss/*.xml")) + glob.glob(str(out / "inputs/rss/*.txt")))
|
||||
target = out / "rss.json"
|
||||
existing = _load_existing(target)
|
||||
samples = []
|
||||
for f in inputs:
|
||||
p = Path(f)
|
||||
data = p.read_bytes()
|
||||
text = data.decode("utf-8")
|
||||
samples.append({
|
||||
"id": p.stem,
|
||||
"input": {"file": f"inputs/rss/{p.name}", "sha256": _sha256(data)},
|
||||
"mp": _run_mp_rss_python(text),
|
||||
"mp_rust": _run_mp_rss_rust(text),
|
||||
})
|
||||
doc = {
|
||||
"kind": "rss",
|
||||
"generatedFrom": {
|
||||
"repo": "MoviePilot", "commit": _mp_commit(root),
|
||||
"function": "app/application/rss.py RssHelper._parse_impl (lxml fallback; parser port -> None)",
|
||||
"rust": "moviepilot_rust.parse_rss_items_fast (reference only)",
|
||||
"tz": "UTC", "generator": "gen-differential.py rss",
|
||||
},
|
||||
"fields": ["title", "description", "link", "enclosure", "size", "pubdate", "nickname"],
|
||||
"samples": samples,
|
||||
"deviations": existing.get("deviations", {}),
|
||||
}
|
||||
target.write_text(json.dumps(doc, ensure_ascii=False, indent=2) + "\n", "utf-8")
|
||||
return target
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- 站点索引
|
||||
|
||||
|
||||
def _run_mp_spider(indexer: dict, html: str, keyword: Optional[str], search_type: str, use_rust: bool) -> dict:
|
||||
from app.adapters.system import rust as rust_accel
|
||||
from app.modules.indexer.spider import SiteSpider
|
||||
|
||||
orig_t, orig_s = rust_accel.parse_indexer_torrents, rust_accel.parse_indexer_subtitles
|
||||
if not use_rust:
|
||||
rust_accel.parse_indexer_torrents = lambda **_kw: None
|
||||
rust_accel.parse_indexer_subtitles = lambda **_kw: None
|
||||
try:
|
||||
spider = SiteSpider(indexer, keyword=keyword, search_type=search_type)
|
||||
rows = spider.parse(html)
|
||||
return {"is_error": bool(spider.is_error), "torrents": _jsonable(rows)}
|
||||
except BaseException as err: # noqa: BLE001
|
||||
return {"status": "error", "error": f"{type(err).__name__}: {err}"}
|
||||
finally:
|
||||
rust_accel.parse_indexer_torrents, rust_accel.parse_indexer_subtitles = orig_t, orig_s
|
||||
|
||||
|
||||
SUBTITLE_INDEXER = {
|
||||
# 取自 MP tests/test_rust_accel.py::test_rust_subtitle_parser_is_several_times_faster_than_python
|
||||
"id": "hhanclub-subtitles", "name": "憨憨(字幕)", "domain": "https://hhanclub.net/", "public": False,
|
||||
"subtitles": {
|
||||
"list": {"selector": "#subtitles-table > div"},
|
||||
"fields": {
|
||||
"language_icon": {"selector": "div:nth-child(1) img", "attribute": "src"},
|
||||
"title": {"selector": 'div:nth-child(2) a[href*="downloadsubs.php"]'},
|
||||
"download": {"selector": 'div:nth-child(2) a[href*="downloadsubs.php"]', "attribute": "href"},
|
||||
"size": {"selector": "div:nth-child(3)"},
|
||||
"date_added": {"selector": "div:nth-child(4) span", "attribute": "title"},
|
||||
"date_elapsed": {"selector": "div:nth-child(4) span"},
|
||||
"grabs": {"defualt_value": 0},
|
||||
"uploader": {"selector": 'div:nth-child(2) a[href*="userdetails.php"]'},
|
||||
"report": {"selector": 'div:nth-child(5) a[href*="report.php"]', "attribute": "href"},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _subtitle_html() -> str:
|
||||
rows = []
|
||||
for i in range(3):
|
||||
rows.append(f"""<div class="grid"><div><img src="pic/flag/china.gif"></div><div>
|
||||
<a href="downloadsubs.php?torrentid={i}&subid={i + 1000}">Example Show S01E0{i + 1} 1080p WEB-DL CHS</a>
|
||||
<a href="https://hhanclub.net/userdetails.php?id={i}"><b>tester{i}</b></a></div>
|
||||
<div><div>111.99 KB</div></div><div><span title="2026-04-21 20:54:37">1月18天</span></div>
|
||||
<div><a href="report.php?subtitle={i + 1000}">举报</a></div></div>""")
|
||||
return f'<div id="subtitles-table">{"".join(rows)}</div>'
|
||||
|
||||
|
||||
def gen_indexer(out_dir: Optional[str] = None) -> Path:
|
||||
"""生成 <out_dir>/indexer.json;返回文件路径。"""
|
||||
root = _bootstrap()
|
||||
out = Path(out_dir).resolve() if out_dir else DEFAULT_OUT
|
||||
target = out / "indexer.json"
|
||||
existing = _load_existing(target)
|
||||
idx_dir = out / "inputs/indexer"
|
||||
|
||||
defs = {}
|
||||
for f in sorted(DEFINITIONS_DIR.glob("*.json")):
|
||||
d = json.loads(f.read_text("utf-8"))
|
||||
if (d.get("umt") or {}).get("kind") == "nexusphp":
|
||||
defs[d["id"]] = (f, d)
|
||||
|
||||
def page(name: str) -> tuple[str, dict]:
|
||||
data = (idx_dir / name).read_bytes()
|
||||
return data.decode("utf-8"), {"file": f"inputs/indexer/{name}", "sha256": _sha256(data)}
|
||||
|
||||
runs: list[tuple[str, dict, dict, str, dict, Optional[str], str]] = []
|
||||
list_pages = ["tk-audiences-list.html.txt", "tk-hhanclub-list.html.txt", "tk-chdbits-list.html.txt"]
|
||||
for site_id, (f, d) in defs.items():
|
||||
for pg in list_pages:
|
||||
sid = f"{site_id}@{pg.removesuffix(".html.txt")}"
|
||||
runs.append((sid, d, {"source": "definition", "id": site_id,
|
||||
"file": str(f.relative_to(UMT_ROOT)), "sha256": _sha256(f.read_bytes())},
|
||||
pg, {}, "x", "torrents"))
|
||||
for site_id, pg in (("audiences", "tk-audiences-login.html.txt"), ("chdbits", "tk-chdbits-login.html.txt")):
|
||||
f, d = defs[site_id]
|
||||
runs.append((f"{site_id}@{pg.removesuffix(".html.txt")}", d, {"source": "definition", "id": site_id,
|
||||
"file": str(f.relative_to(UMT_ROOT)),
|
||||
"sha256": _sha256(f.read_bytes())},
|
||||
pg, {}, "x", "torrents"))
|
||||
for cfg in sorted(idx_dir.glob("*.indexer.json")):
|
||||
d = json.loads(cfg.read_text("utf-8"))
|
||||
name = cfg.name.replace(".indexer.json", "")
|
||||
# 同名 <name>.html.txt 优先,否则用共享的 hw-dsl-rows.html.txt(.txt 后缀避免 biome 按 HTML 做 a11y lint)
|
||||
pg = f"{name}.html.txt" if (idx_dir / f"{name}.html.txt").exists() else "hw-dsl-rows.html.txt"
|
||||
runs.append((f"{name}@{pg.removesuffix(".html.txt")}", d, {"source": "inline", "file": f"inputs/indexer/{cfg.name}",
|
||||
"config": d}, pg, {}, None, "torrents"))
|
||||
|
||||
samples = []
|
||||
for sid, indexer, def_meta, pg, _extra, keyword, stype in runs:
|
||||
html, meta = page(pg)
|
||||
samples.append({
|
||||
"id": sid,
|
||||
"definition": def_meta,
|
||||
"input": meta,
|
||||
"query": {"keyword": keyword, "search_type": stype},
|
||||
"mp": _run_mp_spider(indexer, html, keyword, stype, use_rust=False),
|
||||
"mp_rust": _run_mp_spider(indexer, html, keyword, stype, use_rust=True),
|
||||
})
|
||||
# 字幕:本项目未移植(静态登记),仍记录 MP 输出作为将来补迁的期望
|
||||
sub_html = _subtitle_html()
|
||||
samples.append({
|
||||
"id": "hhanclub-subtitles@inline",
|
||||
"definition": {"source": "inline", "config": SUBTITLE_INDEXER},
|
||||
"input": {"inline": sub_html, "sha256": _sha256(sub_html.encode())},
|
||||
"query": {"keyword": "Example Show", "search_type": "subtitles"},
|
||||
"mp": _run_mp_spider(SUBTITLE_INDEXER, sub_html, "Example Show", "subtitles", use_rust=False),
|
||||
"mp_rust": _run_mp_spider(SUBTITLE_INDEXER, sub_html, "Example Show", "subtitles", use_rust=True),
|
||||
})
|
||||
doc = {
|
||||
"kind": "indexer",
|
||||
"generatedFrom": {
|
||||
"repo": "MoviePilot", "commit": _mp_commit(root),
|
||||
"function": "app/modules/indexer/spider/__init__.py SiteSpider.parse (PyQuery fallback)",
|
||||
"rust": "moviepilot_rust.parse_indexer_torrents_fast / parse_indexer_subtitles_fast (reference only)",
|
||||
"definitions": "packages/adapters/src/sites/definitions/*.json (MP 内置定义加密且需认证,不可用)",
|
||||
"tz": "UTC", "dateparserNow": FROZEN_NOW, "generator": "gen-differential.py indexer",
|
||||
},
|
||||
"fields": ["title", "description", "page_url", "enclosure", "size", "seeders", "peers", "grabs",
|
||||
"downloadvolumefactor", "uploadvolumefactor", "freedate", "hit_and_run", "category",
|
||||
"pubdate", "date_elapsed", "imdbid", "labels"],
|
||||
"samples": samples,
|
||||
"deviations": existing.get("deviations", {}),
|
||||
}
|
||||
target.write_text(json.dumps(doc, ensure_ascii=False, indent=2) + "\n", "utf-8")
|
||||
return target
|
||||
|
||||
|
||||
def gen_rss_cmd() -> None:
|
||||
print("rss: wrote", gen_rss(None))
|
||||
|
||||
|
||||
def gen_indexer_cmd() -> None:
|
||||
print("indexer: wrote", gen_indexer(None))
|
||||
|
||||
|
||||
COMMANDS = {"meta": gen_meta, "zhconv": gen_zhconv, "filter": gen_filter_cmd, "rss": gen_rss_cmd, "indexer": gen_indexer_cmd}
|
||||
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
cmd = argv[1] if len(argv) > 1 else "all"
|
||||
if cmd == "all":
|
||||
# MP 运行时配置是进程级全局状态,每个子命令独立进程执行
|
||||
for name in COMMANDS:
|
||||
subprocess.run([sys.executable, __file__, name], check=True)
|
||||
return 0
|
||||
if cmd not in COMMANDS:
|
||||
print(f"unknown command {cmd}; choose from {', '.join(COMMANDS)} / all", file=sys.stderr)
|
||||
return 2
|
||||
COMMANDS[cmd]()
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv))
|
||||
Reference in new issue
Block a user