feat(adapters): port MoviePilot open site framework (7 spiders, 20 user parsers) + MovieClaw v0.27.0 site definitions + mp-sync tools (W1-3c)
- sites/spiders: Haidan/HDDolby/RousiPro/SunnyPT/TNode/TorrentLeech/Yema (+mTorrent wrapper), specialized SiteAdapter, indirect download resolved at download time - sites/user-parsers: SiteParserBase + all 20 MP SiteSchema parsers (lxml XPath via xpath+xmldom), refreshSiteUserData with schema fallback - sites/definitions: zod SiteDefinition schemas, 23 MovieClaw v0.27.0 (MIT) definitions via scripts/mp-sync/movieclaw-import.mjs, registry getSiteDefinition/listSiteDefinitions, NOTICE - scripts/mp-sync: ledger.json (upstream sha256), check.mjs, build-ledger.mjs; docs/porting/mp-sync-ledger.md - license-and-provenance: add copied-mit kind, re-anchor MovieClaw rows to v0.27.0 Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
1 parent
cf85481d95
commit
d9f2af2f3c
166 files changed
+16577
-186
No files matched your search
@@ -662,3 +662,36 @@ MC 注册表 `MC:src/movieclaw_tracker/registry.py:172` `_load_site_yaml` 读取
|
||||
### 10.4 许可提示
|
||||
|
||||
本节模块是对 MP(GPL-3.0)实现的逐函数 TS 移植,性质上比 `behavior-ref` 更接近衍生作品。项目目前为 internal-only 使用;若将来对外分发,需按 `license-and-provenance.md` §2 重新评估(GPL-3.0 同许可与源码提供义务)。
|
||||
|
||||
## 11. W1-3c 实际迁移对照(MP 开源站点框架全量 + MovieClaw 站点定义)
|
||||
|
||||
逐文件对照总表与跟随更新流程见 `mp-sync-ledger.md`(`scripts/mp-sync/ledger.json` 由源码 `@mp-source` 标签生成)。本节只记结构性决定。
|
||||
|
||||
### 11.1 专用 spider(`packages/adapters/src/sites/spiders/`)
|
||||
|
||||
| MP | 本项目 | 说明 |
|
||||
| --- | --- | --- |
|
||||
| `IndexerModule.SPIDER_PARSER_CLASSES` / `_SPECIALIZED_SEARCH_ARGUMENTS` | `spiders/index.ts` `SPECIALIZED_SPIDERS`、`SpecializedSpider.search(ctx, {keyword, mtype, cat, page})` | 键名与 MP 相同(Haidan/HDDolby/RousiPro/SunnyPT/TNodeSpider/TorrentLeech/Yema/mTorrent) |
|
||||
| 各 spider `__get_params` / `__parse_result` / `__process_response` / 系数函数 | `spiders/<name>.ts` 同名纯函数 | 常量表逐字保留;结果为 MP 字段(`SpiderTorrent`)+ `category_id`、`site_release_id`、`*_utc` 扩展 |
|
||||
| `(True, [])` 错误返回 | `SiteError` | 401/403 → `auth_invalid`,缺凭据 → `auth_missing`,业务错误按文案分类 |
|
||||
| `[base64 请求配置]url` enclosure + `submission._resolve_indirect_download_url` | `spiders/common.ts` `resolveIndirectDownloadUrl` / `downloadTorrentFile` | 下载时现场换直链,结果约束到可信 API 基址,不带凭据下载并校验 .torrent |
|
||||
| `TNodeSpider.__get_token`(`@cached` 24h)、`SunnyPTSpider._category_cache`(1h) | 适配器实例内 TTL 缓存 | |
|
||||
|
||||
### 11.2 站点用户信息解析器(`packages/adapters/src/sites/user-parsers/`)
|
||||
|
||||
| MP | 本项目 |
|
||||
| --- | --- |
|
||||
| `parser/__init__.py` `SiteSchema` / `SiteParserBase` | `types.ts` `SiteSchema` / `base.ts` `SiteParserBase`(`parse` 流程、`_get_page_content` 请求模式、`_parse_logged_in`、`to_dict`) |
|
||||
| `parser/<name>.py` 20 个类 | `user-parsers/<kebab-name>.ts` 同名类 |
|
||||
| `IndexerModule.refresh_userdata`(schema 回退) | `registry.ts` `refreshSiteUserData` |
|
||||
| `SiteUserData` | `SiteUserInfo`(camelCase,`raw` = `to_dict()`,不含凭据) |
|
||||
| lxml XPath / BeautifulSoup | `dom.ts`(xpath + xmldom)/ cheerio |
|
||||
| `SiteUtils.is_logged_in`、`under_challenge`、`parse_int/parse_float`、`normalize_datetime`、`format_timestamp`、`extract_domain` | `common.ts` 同名函数 |
|
||||
|
||||
偏离:翻页上限 50;网络/HTTP 失败另记 `http_errors`;写操作(标记已读)受 `fetchMessages` 门控;`normalize_datetime` 只处理绝对时间。
|
||||
|
||||
### 11.3 站点定义
|
||||
|
||||
- MP 的站点定义不可得(加密),定义 schema 按 `sites.pyi` 与 spider 读取的键整理为 zod(`definitions/schema.ts`),加载即校验。
|
||||
- 选择器值:用户 2026-09-27 确认复制 MovieClaw **v0.27.0**(MIT)资产,23 个站点经 `scripts/mp-sync/movieclaw-import.mjs` 转换(ledger CL-012/CL-013,`copied-mit`,NOTICE 见 `definitions/NOTICE-MovieClaw.md`)。映射规则按 `movieclaw-assessment.md` §4;`page_offset -1 → browse.start 0`;`promo_*_rules → case` 表(补 `"*": 1`);`X::attr(a) → {selector, attribute}`;分类取 `\d+`;MovieClaw 的 documentary/anime/game/av/other 只进 `categoryNames`(原始 ID 保留)。
|
||||
- 观众/馒头保留 W1-3b 手写值(覆盖层),HHClub/CHD 覆盖层补站点分类名;4 个主力站启用,其余 19 站 `enabled:false`(未实测)。
|
||||
Reference in new issue
Block a user