diff --git a/docs/porting/license-and-provenance.md b/docs/porting/license-and-provenance.md index a5f3b0e7..bd584146 100644 --- a/docs/porting/license-and-provenance.md +++ b/docs/porting/license-and-provenance.md @@ -40,8 +40,9 @@ | 引用事实性常量：协议字段名（qB WebUI API、M-Team API、TMDB API）、站点公开的 HTML 结构、行业通用词表（`WEB-DL`、`REMUX`、`DDP5.1`） | 照搬上游**整理过的数据表**：`RELEASE_GROUPS` 站点到组名的正则表（`app/domain/meta/releasegroup.py:26`）、`StreamingPlatforms` 平台码表（`app/domain/meta/streamingplatform.py`，约 270 条）、`_name_nostring_re` 干扰词表。这类表要**自行编制**，并在 ledger 里写明来源（例如公开的 scene/P2P 命名规范、站点官方规则页） | | 上游测试里的**标题字符串**（事实数据）可以作为输入语料 | 整份拷贝 `tests/cases/meta.py` 这类测试文件；期望值必须由我们自己按 REQ 判定，不是照抄上游的 target | | MovieClaw：阅读 YAML 的**字段结构**，参考其字段设计 | 在没有书面授权时拷贝 MovieClaw main 的配置或代码（附条件许可）；把 main 的内容当作 MIT 使用 | +| MovieClaw **tag `v0.27.0`（`c79bb1b0`，MIT）** 的资产可以原样复制或转换（`kind=copied-mit`），前提是保留 MIT 全文与 `Copyright (c) 2026 MovieClaw`（`packages/adapters/src/sites/definitions/NOTICE-MovieClaw.md`），每个产物写明来源提交与路径（站点定义 JSON 的 `provenance` 字段），且只经 `git show v0.27.0:` 读取（用户 2026-09-27 确认，依据 `movieclaw-assessment.md` §1） | 复制 `v0.27.0` 之后（`251f1988` 起附条件许可）的任何内容；复制 MC 仓库内自带独立许可的第三方子组件（`skill-creator` Apache-2.0、`liquid-glass`）而不遵守其各自许可 | -**对每个 PR 的要求**：只要实现参考了上游，PR 描述里就必须写"参考：`docs/porting/<文件>#<节>`"，并在 §3 ledger 里补一行，登记状态为 `behavior-ref`。只有真的复制了代码或数据时，才登记为 `copied`；按目前的边界，**不应出现 `copied`**。 +**对每个 PR 的要求**：只要实现参考了上游，PR 描述里就必须写"参考：`docs/porting/<文件>#<节>`"，并在 §3 ledger 里补一行，登记状态为 `behavior-ref`。只有真的复制了代码或数据时，才登记为 `copied`；按目前的边界，**不应出现 `copied`**。例外是 `copied-mit`：复制 MIT 许可的资产（目前仅 MovieClaw v0.27.0），必须 `notice_required=是` 并附 NOTICE。 ### 2.1 分发时的注意事项（首期不分发，先写在这里） @@ -59,7 +60,7 @@ | --- | --- | | `id` | `CL-<序号>` | | `our_module` | 本项目的文件或函数，例如 `packages/domain/src/title/parse.ts#parseReleaseTitle` | -| `kind` | `behavior-ref`（只参考行为）/ `data-ref`（参考了数据结构，内容自行编制）/ `copied`（复制，原则上禁止）/ `dependency`（npm 依赖） | +| `kind` | `behavior-ref`（只参考行为）/ `data-ref`（参考了数据结构，内容自行编制）/ `copied`（复制，原则上禁止）/ `copied-mit`（复制 MIT 资产，须保留 NOTICE，见 §2）/ `dependency`（npm 依赖） | | `upstream` | 仓库 URL | | `commit_or_version` | 精确提交或 tag，不能写 `main` | | `upstream_paths` | 文件:函数:行号 | @@ -82,9 +83,11 @@ | CL-006 | `packages/domain/src/organize/*`、`packages/adapters/src/fs/*`、`packages/tasks/src/verify-and-import.ts` | behavior-ref | MoviePilot | e6820a8 | 见 `moviepilot-transfer.md` §2 各表 | GPL-3.0 | transfer §2-§4 | 见 transfer §3 的必须偏离表 | 否 | internal-only | | CL-007 | `packages/adapters/src/qbittorrent/*` | behavior-ref | MoviePilot | e6820a8 | `app/modules/qbittorrent/__init__.py`、`qbittorrent.py`（行号见 downloader-and-sites 文档） | GPL-3.0 | downloader-and-sites §qB | 标签接管，删数据前做引用检查 | 否 | internal-only | | CL-008 | `packages/adapters/src/sites/nexusphp/*` | behavior-ref | MoviePilot | e6820a8 | `app/modules/indexer/spider/*`、`app/modules/indexer/parser/nexus_php.py`、`nexus_audiences.py` | GPL-3.0 | downloader-and-sites §站点 | 站点定义自行编制；不使用闭源 SitesHelper | 否 | internal-only | -| CL-009 | `packages/adapters/src/sites/nexusphp/configs/audiences.ts` | data-ref | MovieClaw | fc9eb72 | `src/movieclaw_tracker/sites/configs/audiences.yaml` | MovieClaw License（Apache-2.0 加非商业条件） | downloader-and-sites §MovieClaw | **只参考字段结构**；选择器值以我们自己抓取的观众页面为准重新编写 | 否（未复制） | internal-only | -| CL-010 | `packages/adapters/src/sites/mteam/*` | behavior-ref + data-ref | MovieClaw + MoviePilot | fc9eb72 / e6820a8 | MC：`sites/configs/mteam.yaml`、`sites/custom/mteam.py`；MP：`app/modules/indexer/parser/mtorrent.py` | MC 附条件 / GPL-3.0 | downloader-and-sites §M-Team | API 字段以 M-Team 公开 API 为准，按 API Key 模式实现 | 否 | internal-only | +| CL-009（retired → CL-012） | `packages/adapters/src/sites/nexusphp/configs/audiences.ts` | data-ref | MovieClaw | fc9eb72 | `src/movieclaw_tracker/sites/configs/audiences.yaml` | MovieClaw License（Apache-2.0 加非商业条件） | downloader-and-sites §MovieClaw | **只参考字段结构**；选择器值以我们自己抓取的观众页面为准重新编写 | 否（未复制） | internal-only | +| CL-010（retired → CL-013） | `packages/adapters/src/sites/mteam/*` | behavior-ref + data-ref | MovieClaw + MoviePilot | fc9eb72 / e6820a8 | MC：`sites/configs/mteam.yaml`、`sites/custom/mteam.py`；MP：`app/modules/indexer/parser/mtorrent.py` | MC 附条件 / GPL-3.0 | downloader-and-sites §M-Team | API 字段以 M-Team 公开 API 为准，按 API Key 模式实现 | 否 | internal-only | | CL-011 | `packages/testkit/fixtures/title-parity.json` | data-ref | MoviePilot | e6820a8 | 输入语料参考 `tests/cases/meta.py`；**期望值自行判定** | GPL-3.0（原测试文件） | metadata-recognition §9 | 只取标题字符串，必要时脱敏 | 否 | internal-only | +| CL-012 | `packages/adapters/src/sites/definitions/*.json`（23 站）、`definitions/manifest.generated.ts`、`spider/mc-selector.ts`、`scripts/mp-sync/movieclaw-import.mjs` | copied-mit | MovieClaw | **v0.27.0**（`c79bb1b0`） | `src/movieclaw_tracker/sites/configs/*.yaml`；`selectors.py` `NexusPHPSelectors` 默认值；`frameworks/nexusphp.py` `_css_text` | MIT | downloader-and-sites §W1-3c | YAML 转为 MP indexer schema + `umt`；观众/馒头/HHClub/CHD 由 `scripts/mp-sync/movieclaw-overrides/*.json` 覆盖；除 4 主力站外 `enabled:false` | 是（`definitions/NOTICE-MovieClaw.md`） | internal-only | +| CL-013 | `packages/adapters/src/sites/mteam/*` | behavior-ref + copied-mit | MovieClaw + MoviePilot | **v0.27.0**（`c79bb1b0`）/ e6820a8 | MC：`sites/configs/mteam.yaml`（→ `definitions/mteam.json`）、`sites/custom/mteam.py`（字段对照）；MP：`app/modules/indexer/spider/mtorrent.py`、`parser/mtorrent.py` | MIT / GPL-3.0 | downloader-and-sites §M-Team | 定义由 MC 转换 + W1-3b 值覆盖；API 行为按 MP | 是（MC 部分） | internal-only | > 新增 npm 依赖（例如 anitopy 的等价实现、`re2`、`fast-xml-parser`）按 `kind=dependency` 登记，填锁定版本和 SPDX。 @@ -92,7 +95,7 @@ 1. ledger 以本文件 §3.2 为唯一登记处，按 id 追加，不删除旧行；撤销时把行标记为 `retired`。 2. 上游提交升级时，**新增一行**，写新的提交号，不覆盖旧行。对照文档里的行号也要按新提交重新核对。 -3. `copied` 行出现时，CI 必须失败。建议在 `scripts/` 里加一个检查：ledger 中 `kind=copied` 的行数必须为 0。 +3. `copied` 行出现时，CI 必须失败。建议在 `scripts/` 里加一个检查：ledger 中 `kind=copied` 的行数必须为 0。`copied-mit` 不在此列，但必须有 NOTICE 文件且 `notice=是`。 4. 审核人检查三件事：PR 里有没有"参考："链接；ledger 有没有对应行；diff 中有没有大段和上游高度相似的代码（可以用 `jscpd` 这类工具，把上游目录作为外部语料做一次抽样检查）。 --- diff --git a/docs/porting/moviepilot-downloader-and-sites.md b/docs/porting/moviepilot-downloader-and-sites.md index 048953db..a85698be 100644 --- a/docs/porting/moviepilot-downloader-and-sites.md +++ b/docs/porting/moviepilot-downloader-and-sites.md @@ -662,3 +662,36 @@ MC 注册表 `MC:src/movieclaw_tracker/registry.py:172` `_load_site_yaml` 读取 ### 10.4 许可提示 本节模块是对 MP(GPL-3.0)实现的逐函数 TS 移植,性质上比 `behavior-ref` 更接近衍生作品。项目目前为 internal-only 使用;若将来对外分发,需按 `license-and-provenance.md` §2 重新评估(GPL-3.0 同许可与源码提供义务)。 + +## 11. W1-3c 实际迁移对照(MP 开源站点框架全量 + MovieClaw 站点定义) + +逐文件对照总表与跟随更新流程见 `mp-sync-ledger.md`(`scripts/mp-sync/ledger.json` 由源码 `@mp-source` 标签生成)。本节只记结构性决定。 + +### 11.1 专用 spider(`packages/adapters/src/sites/spiders/`) + +| MP | 本项目 | 说明 | +| --- | --- | --- | +| `IndexerModule.SPIDER_PARSER_CLASSES` / `_SPECIALIZED_SEARCH_ARGUMENTS` | `spiders/index.ts` `SPECIALIZED_SPIDERS`、`SpecializedSpider.search(ctx, {keyword, mtype, cat, page})` | 键名与 MP 相同(Haidan/HDDolby/RousiPro/SunnyPT/TNodeSpider/TorrentLeech/Yema/mTorrent) | +| 各 spider `__get_params` / `__parse_result` / `__process_response` / 系数函数 | `spiders/.ts` 同名纯函数 | 常量表逐字保留;结果为 MP 字段(`SpiderTorrent`)+ `category_id`、`site_release_id`、`*_utc` 扩展 | +| `(True, [])` 错误返回 | `SiteError` | 401/403 → `auth_invalid`,缺凭据 → `auth_missing`,业务错误按文案分类 | +| `[base64 请求配置]url` enclosure + `submission._resolve_indirect_download_url` | `spiders/common.ts` `resolveIndirectDownloadUrl` / `downloadTorrentFile` | 下载时现场换直链,结果约束到可信 API 基址,不带凭据下载并校验 .torrent | +| `TNodeSpider.__get_token`(`@cached` 24h)、`SunnyPTSpider._category_cache`(1h) | 适配器实例内 TTL 缓存 | | + +### 11.2 站点用户信息解析器(`packages/adapters/src/sites/user-parsers/`) + +| MP | 本项目 | +| --- | --- | +| `parser/__init__.py` `SiteSchema` / `SiteParserBase` | `types.ts` `SiteSchema` / `base.ts` `SiteParserBase`(`parse` 流程、`_get_page_content` 请求模式、`_parse_logged_in`、`to_dict`) | +| `parser/.py` 20 个类 | `user-parsers/.ts` 同名类 | +| `IndexerModule.refresh_userdata`(schema 回退) | `registry.ts` `refreshSiteUserData` | +| `SiteUserData` | `SiteUserInfo`(camelCase,`raw` = `to_dict()`,不含凭据) | +| lxml XPath / BeautifulSoup | `dom.ts`(xpath + xmldom)/ cheerio | +| `SiteUtils.is_logged_in`、`under_challenge`、`parse_int/parse_float`、`normalize_datetime`、`format_timestamp`、`extract_domain` | `common.ts` 同名函数 | + +偏离:翻页上限 50;网络/HTTP 失败另记 `http_errors`;写操作(标记已读)受 `fetchMessages` 门控;`normalize_datetime` 只处理绝对时间。 + +### 11.3 站点定义 + +- MP 的站点定义不可得(加密),定义 schema 按 `sites.pyi` 与 spider 读取的键整理为 zod(`definitions/schema.ts`),加载即校验。 +- 选择器值:用户 2026-09-27 确认复制 MovieClaw **v0.27.0**(MIT)资产,23 个站点经 `scripts/mp-sync/movieclaw-import.mjs` 转换(ledger CL-012/CL-013,`copied-mit`,NOTICE 见 `definitions/NOTICE-MovieClaw.md`)。映射规则按 `movieclaw-assessment.md` §4;`page_offset -1 → browse.start 0`;`promo_*_rules → case` 表(补 `"*": 1`);`X::attr(a) → {selector, attribute}`;分类取 `\d+`;MovieClaw 的 documentary/anime/game/av/other 只进 `categoryNames`(原始 ID 保留)。 +- 观众/馒头保留 W1-3b 手写值(覆盖层),HHClub/CHD 覆盖层补站点分类名;4 个主力站启用,其余 19 站 `enabled:false`(未实测)。 diff --git a/docs/porting/mp-sync-ledger.md b/docs/porting/mp-sync-ledger.md new file mode 100644 index 00000000..9d68fa73 --- /dev/null +++ b/docs/porting/mp-sync-ledger.md @@ -0,0 +1,99 @@ +# MoviePilot 跟随移植账本(mp-sync) + +> **许可结论(引自 `license-and-provenance.md`)**:MoviePilot 为 GPLv3;本目录登记的站点框架文件是用户 2026-09-27 明确要求"照抄 MP v3 实现迁移"的数据来源适配部分,每个移植文件头保留 `SPDX-License-Identifier: GPL-3.0-only` 与来源声明;MP 的闭源/加密部分(`sites.*.so`、`user.sites.v3.bin`)**不解密、不参考**,站点定义改由 MovieClaw **v0.27.0(MIT,`copied-mit`,见 `packages/adapters/src/sites/definitions/NOTICE-MovieClaw.md`)** 转换 + 手写覆盖。MovieClaw v0.27.0 之后改为附条件许可,**不跟随其后续提交**。 + +## 1. 机制 + +- **登记来源**:每个移植文件头写 `@mp-source <上游路径> @ <提交>`(一个文件可多行);sites 目录外或 W1-3a/b 早期文件在 `scripts/mp-sync/ledger.manual.json` 人工登记。 +- **ledger**:`scripts/mp-sync/ledger.json`(`build-ledger.mjs` 生成)按上游文件聚合:`upstream`、移植时 `commit`、该文件在该提交的内容 `sha256`、本项目文件 `ours[]`。 +- **监视目录**:`app/modules/indexer/spider/`、`app/modules/indexer/parser/`,出现 ledger 未收录的新文件即提示(MP 新增站点 spider/解析器)。 + +## 2. 用法 + +MP 本地仓库默认 `/opt/moviepilot-plus/upstream/MoviePilot`(浅克隆,当前 `e6820a8`)。以下命令 Node 24 或 `deno run -A` 均可(不依赖根 package.json / deno.json 任务)。 + +```bash +# 1) 取上游新提交(浅克隆) +git -C /opt/moviepilot-plus/upstream/MoviePilot fetch --depth=1 origin <新提交或 v3> +# 2) 查差异:已变更 / 已删除 / 监视目录新增,附逐文件 git diff 命令;有变化退出码 1 +node scripts/mp-sync/check.mjs --to <新提交> # 或 --mp <路径> --json +# 3) 按 diff 增量移植 TS 文件,把文件头 @mp-source 的提交号改为新提交(新文件加标签) +# 4) 重建 ledger 与下表 +node scripts/mp-sync/build-ledger.mjs --commit <新提交> +# 5) 跑适配器测试 +cd packages/adapters && deno run -A npm:vitest run src/sites +``` + +MovieClaw 站点定义(与 MP 无关,但同属站点框架跟随工具): + +```bash +node scripts/mp-sync/movieclaw-import.mjs # 从 v0.27.0 重新转换 23 个站点定义(覆盖层:scripts/mp-sync/movieclaw-overrides/) +node scripts/mp-sync/movieclaw-import.mjs --check # 只比较,有差异退出码 1 +``` + +## 3. 当前对照总表 + + +共 49 个上游文件(`adapters/` = `packages/adapters/src/`)。 + +| 上游文件 | 本项目文件 | 移植提交 | sha256(前 12 位) | 来源 | +| --- | --- | --- | --- | --- | +| `app/adapters/network/cloudflare.py` | `adapters/sites/user-parsers/common.ts` | `e6820a8` | `a23a2c44da05` | tag | +| `app/application/mediaserver.py` | `adapters/emby/client.ts` | `e6820a8` | `f034f0be8ba6` | manual | +| `app/application/rss.py` | `adapters/rss/fetch.ts`
`adapters/rss/parse.ts`
`adapters/rss/rss-link.ts` | `e6820a8` | `8ea70f712512` | manual | +| `app/application/site/sites.pyi` | `adapters/sites/definitions/schema.ts`
`adapters/sites/registry.ts`
`adapters/sites/spider/schema.ts` | `e6820a8` | `8fea9a239354` | manual | +| `app/chain/download/submission.py` | `adapters/sites/mteam/adapter.ts`
`adapters/sites/spider/adapter.ts`
`adapters/sites/spiders/common.ts` | `e6820a8` | `f7cb870e47d6` | manual+tag | +| `app/domain/site.py` | `adapters/sites/user-parsers/common.ts` | `e6820a8` | `650c795a4def` | tag | +| `app/foundation/dom.py` | `adapters/sites/user-parsers/dom.ts` | `e6820a8` | `6d6accf2cef1` | tag | +| `app/foundation/size.py` | `adapters/sites/spider/spider.ts` | `e6820a8` | `7b0b8f50588c` | manual | +| `app/foundation/temporal.py` | `adapters/sites/spiders/common.ts`
`adapters/sites/user-parsers/common.ts` | `e6820a8` | `74941f460e40` | tag | +| `app/foundation/text.py` | `adapters/sites/user-parsers/common.ts` | `e6820a8` | `a88c75dc8431` | tag | +| `app/modules/__init__.py` | `adapters/qbittorrent/paths.ts` | `e6820a8` | `2edada0ea591` | manual | +| `app/modules/_base/downloader.py` | `adapters/qbittorrent/torrent.ts` | `e6820a8` | `84bd8946c8b2` | manual | +| `app/modules/emby/emby.py` | `adapters/emby/client.ts` | `e6820a8` | `e6468ea34ef2` | manual | +| `app/modules/filemanager/storages/local.py` | `adapters/fs/copy.ts`
`adapters/fs/move.ts` | `e6820a8` | `806a1970ca8e` | manual | +| `app/modules/filemanager/transhandler.py` | `adapters/fs/move.ts` | `e6820a8` | `13e2bba491ac` | manual | +| `app/modules/indexer/__init__.py` | `adapters/sites/registry.ts`
`adapters/sites/spider/adapter.ts`
`adapters/sites/spiders/adapter.ts`
`adapters/sites/spiders/index.ts`
`adapters/sites/spiders/map.ts`
`adapters/sites/spiders/types.ts`
`adapters/sites/user-parsers/registry.ts` | `e6820a8` | `e9c5d15fdaa5` | manual+tag | +| `app/modules/indexer/parser/__init__.py` | `adapters/sites/user-parsers/base.ts`
`adapters/sites/user-parsers/types.ts` | `e6820a8` | `0e6c4215b713` | tag | +| `app/modules/indexer/parser/bitpt.py` | `adapters/sites/user-parsers/bitpt.ts` | `e6820a8` | `871d25e570f1` | tag | +| `app/modules/indexer/parser/discuz.py` | `adapters/sites/user-parsers/discuz.ts` | `e6820a8` | `12277ba955ce` | tag | +| `app/modules/indexer/parser/file_list.py` | `adapters/sites/user-parsers/file-list.ts` | `e6820a8` | `f92e65ed7327` | tag | +| `app/modules/indexer/parser/gazelle.py` | `adapters/sites/user-parsers/gazelle.ts` | `e6820a8` | `c10e5246724b` | tag | +| `app/modules/indexer/parser/hddolby.py` | `adapters/sites/user-parsers/hddolby.ts` | `e6820a8` | `a4336d92d603` | tag | +| `app/modules/indexer/parser/ipt.py` | `adapters/sites/user-parsers/ipt.ts` | `e6820a8` | `2b9d04d76606` | tag | +| `app/modules/indexer/parser/mtorrent.py` | `adapters/sites/user-parsers/mtorrent.ts` | `e6820a8` | `c5d5a3c6f9f5` | tag | +| `app/modules/indexer/parser/nexus_audiences.py` | `adapters/sites/user-parsers/nexus-audiences.ts` | `e6820a8` | `1a2d104db769` | tag | +| `app/modules/indexer/parser/nexus_hhanclub.py` | `adapters/sites/user-parsers/nexus-hhanclub.ts` | `e6820a8` | `8850619069d4` | tag | +| `app/modules/indexer/parser/nexus_php.py` | `adapters/sites/user-parsers/nexus-php.ts` | `e6820a8` | `39303ae96d9c` | tag | +| `app/modules/indexer/parser/nexus_project.py` | `adapters/sites/user-parsers/nexus-project.ts` | `e6820a8` | `2b5508cfb2f8` | tag | +| `app/modules/indexer/parser/nexus_rabbit.py` | `adapters/sites/user-parsers/nexus-rabbit.ts` | `e6820a8` | `09149b89a7c7` | tag | +| `app/modules/indexer/parser/rousi.py` | `adapters/sites/user-parsers/rousi.ts` | `e6820a8` | `1b1ea93de084` | tag | +| `app/modules/indexer/parser/small_horse.py` | `adapters/sites/user-parsers/small-horse.ts` | `e6820a8` | `b73b5de97737` | tag | +| `app/modules/indexer/parser/sunnypt.py` | `adapters/sites/user-parsers/sunnypt.ts` | `e6820a8` | `937f8453fa2b` | tag | +| `app/modules/indexer/parser/tnode.py` | `adapters/sites/user-parsers/tnode.ts` | `e6820a8` | `c9cb809fc4bf` | tag | +| `app/modules/indexer/parser/torrent_leech.py` | `adapters/sites/user-parsers/torrent-leech.ts` | `e6820a8` | `72d1473a3bba` | tag | +| `app/modules/indexer/parser/unit3d.py` | `adapters/sites/user-parsers/unit3d.ts` | `e6820a8` | `9a4d579deca5` | tag | +| `app/modules/indexer/parser/yema.py` | `adapters/sites/user-parsers/yema.ts` | `e6820a8` | `0f98fd580bad` | tag | +| `app/modules/indexer/parser/zhixing.py` | `adapters/sites/user-parsers/zhixing.ts` | `e6820a8` | `90c8954e7387` | tag | +| `app/modules/indexer/spider/__init__.py` | `adapters/sites/spider/map.ts`
`adapters/sites/spider/schema.ts`
`adapters/sites/spider/spider.ts`
`adapters/sites/spider/template.ts` | `e6820a8` | `2513d962fa43` | manual | +| `app/modules/indexer/spider/haidan.py` | `adapters/sites/spiders/haidan.ts` | `e6820a8` | `75eb802f19f0` | tag | +| `app/modules/indexer/spider/hddolby.py` | `adapters/sites/spiders/hddolby.ts` | `e6820a8` | `46a50bd8d49f` | tag | +| `app/modules/indexer/spider/mtorrent.py` | `adapters/sites/mteam/adapter.ts`
`adapters/sites/mteam/client.ts`
`adapters/sites/mteam/spider.ts`
`adapters/sites/spiders/mtorrent.ts` | `e6820a8` | `a8ab2b822cb2` | manual+tag | +| `app/modules/indexer/spider/rousi.py` | `adapters/sites/spiders/rousi.ts` | `e6820a8` | `aa13c386a560` | tag | +| `app/modules/indexer/spider/sunnypt.py` | `adapters/sites/spiders/sunnypt.ts` | `e6820a8` | `735b0d721e65` | tag | +| `app/modules/indexer/spider/tnode.py` | `adapters/sites/spiders/tnode.ts` | `e6820a8` | `a3634a268dba` | tag | +| `app/modules/indexer/spider/torrentleech.py` | `adapters/sites/spiders/torrentleech.ts` | `e6820a8` | `c965f2bcaea4` | tag | +| `app/modules/indexer/spider/yema.py` | `adapters/sites/spiders/yema.ts` | `e6820a8` | `560b62c1ff10` | tag | +| `app/modules/qbittorrent/qbittorrent.py` | `adapters/qbittorrent/client.ts`
`adapters/qbittorrent/state.ts` | `e6820a8` | `ab02e97e986f` | manual | +| `app/modules/themoviedb/tmdbapi.py` | `adapters/tmdb/client.ts`
`adapters/tmdb/match.ts` | `e6820a8` | `787553a6f57d` | manual | +| `app/schemas/site.py` | `adapters/sites/user-parsers/types.ts` | `e6820a8` | `e7952ed40b7b` | tag | + + +## 4. 移植时的统一偏离(每个文件头另有细节) + +1. **凭据不进候选/结果**:MP 两段式 enclosure(`[base64(含 apikey 的请求配置)]url`)不生成,下载时由 adapter 现场换直链;用户信息 `raw` 不含 cookie/apikey/token。 +2. **错误显式化**:MP spider 的 `(is_error, [])` 改为抛 `SiteError{kind}`;用户信息解析保留 MP 的 `err_msg` 语义,另记脱敏的 `http_errors`。 +3. **翻页上限**:做种/消息翻页最多 50 页(MP 无上限)。 +4. **XPath**:lxml → `xpath` + `@xmldom/xmldom`(经 htmlparser2 解析,不插入 tbody),见 `user-parsers/dom.ts`。 +5. **dateparser**:只实现绝对时间格式,相对时间原样返回(待核验)。 +6. **写操作门控**:读取站内信后标记已读(MTorrent/Rousi/NexusRabbit)只在 `fetchMessages=true` 时发生(MP 由全局 `SITE_MESSAGE` 控制)。 diff --git a/packages/adapters/src/sites/README.md b/packages/adapters/src/sites/README.md index e9ed448d..133b53b3 100644 --- a/packages/adapters/src/sites/README.md +++ b/packages/adapters/src/sites/README.md @@ -52,9 +52,51 @@ PT 站点适配器,按 MoviePilot v3(提交 e6820a8)迁移:通用配置驱动 sp 9. `categoryNames` 为自行编制的粗分类名;默认网页域 `tp.m-team.cc` 需按账号实际域名配置。 10. H&R:API 未提供,恒为 `unknown`。 -## M2 预留:站点用户数据解析器(首版未实现) +## W1-3c:MP 开源站点框架全量移植 + MovieClaw 站点定义 -- `app/modules/indexer/parser/nexus_php.py`(NexusPHP 通用:上传/下载/分享率/做种) -- `app/modules/indexer/parser/nexus_audiences.py`(观众新版 userbar;未读私信首页不能带 page 参数) -- `app/modules/indexer/parser/nexus_hhanclub.py`(HHClub) -- `app/modules/indexer/parser/mtorrent.py`(M-Team:`api/member/profile`、`api/tracker/myPeerStatus` 等) +跟随更新工具与对照总表见 `docs/porting/mp-sync-ledger.md`(`scripts/mp-sync/`)。每个移植文件头有 `SPDX-License-Identifier: GPL-3.0-only`、`@mp-source <上游路径> @ e6820a8` 和函数对照。 + +### 专用 spider(`spiders/`,MP `app/modules/indexer/spider/*.py`) + +| parser 键(MP `SPIDER_PARSER_CLASSES`) | 文件 | 凭据 | 下载 | +| --- | --- | --- | --- | +| `Haidan` | `spiders/haidan.ts` | cookie | 列表直链(仅内存),raw 脱敏 | +| `HDDolby` | `spiders/hddolby.ts` | apiKey(+cookie) | `download.php?id=&downhash=`(仅内存) | +| `RousiPro` | `spiders/rousi.ts` | apiKey(Bearer) | 现场请求详情接口取 `data.download_url` | +| `SunnyPT` | `spiders/sunnypt.ts` | apiKey | 现场 `download-token` 换直链 | +| `TNodeSpider` | `spiders/tnode.ts` | cookie + 首页 CSRF(实例内缓存 24h) | 直链 | +| `TorrentLeech` | `spiders/torrentleech.ts` | cookie | 直链;中文关键词按 MP 返回空 | +| `Yema` | `spiders/yema.ts` | apiKey(Authorization) | 现场 `generateDownloadKey` 换 token | +| `mTorrent` | `spiders/mtorrent.ts`(包装 `mteam/*`) | apiKey | genDlToken | + +`createSpecializedAdapterFor(def, deps)` 实现统一 `SiteAdapter`(list = 空关键词搜索;`detail` 为 `unsupported`);定义 `umt.kind = "api"`,schema 见 `definitions/schema.ts` `SpecializedSiteDefinitionSchema`。MP 的 `[base64(含 apikey 的请求配置)]url` 两段式 enclosure 不生成,下载时按同一配置现场换直链并校验 .torrent。`mteam/spider.ts` 已与 `mtorrent.py` 逐条核对(修正:纯数字时间戳按 +08:00 转换)。 + +### 站点用户信息解析器(`user-parsers/`,MP `app/modules/indexer/parser/*.py`) + +20 个 `SiteSchema` 全部移植:NexusPhp、NexusAudiences、NexusHhanclub、NexusProject、NexusRabbit、Gazelle、Unit3d、DiscuzX、IPTorrents、FileList、Small Horse、Zhixing、Bitpt、TNode、TorrentLeech、HDDolby、RousiPro、SunnyPT、Yema、MTorrent。 + +- 基类 `SiteParserBase`(`base.ts`)与 MP 相同的获取顺序:首页登录判定 → `_parseSitePage` → 基础信息 → 详情 → 未读消息(`fetchMessages`)→ 流量 → 做种分页(上限 50 页);cookie/apikey 两种请求模式;200/500/403 视为有内容;Cloudflare 挑战页返回空。 +- XPath 用 `xpath` + `@xmldom/xmldom`(`dom.ts`,htmlparser2 解析,不插入 tbody),表达式与 MP 逐字一致。 +- 入口 `refreshSiteUserData({name, url, schema, cookie|apikey|token, fetch?, limiter?})` → `SiteUserInfo{schema, domain, userId, username, userLevel, joinAt, upload, download, ratio, bonus, seeding, seedingSize, leeching, leechingSize, seedingInfo, messageUnread, messageUnreadContents, hrCount?, errMsg, raw}`;声明 schema 取不到 userid 时按 MP 顺序回退到其他 schema。 +- 写操作:MTorrent / Rousi / NexusRabbit 读消息后标记已读,只在 `fetchMessages: true` 时发生。 + +### 站点定义体系(`definitions/`) + +- 23 个站点 JSON 由 `scripts/mp-sync/movieclaw-import.mjs` 从 **MovieClaw v0.27.0(`c79bb1b0`,MIT)** 转换(见 `definitions/NOTICE-MovieClaw.md`),每个带 `provenance`;手工覆盖层 `scripts/mp-sync/movieclaw-overrides/*.json`(观众、馒头沿用 W1-3b 值,HHClub/CHD 补分类名)。**不要直接改 JSON**,改覆盖层后重跑脚本。 +- 加载时 zod 校验(`definitions/schema.ts`:`IndexerDefinitionSchema` / `MteamDefinitionSchema` / `SpecializedSiteDefinitionSchema`,按 `umt.kind` 分派);注册表 `getSiteDefinition(id)`、`listSiteDefinitions({enabledOnly})`、`createSiteAdapter(def, deps)`。 +- 新增字段:`enabled`、`schema`(MP 用户信息解析器)、`provenance`、`umt.detail.fields`(MovieClaw 逐字段详情选择器,支持 HHClub div 详情页)、`umt.profile`(MovieClaw 资料选择器,备用)、`umt.login`、`umt.authSupported`、`umt.customClass`。选择器扩展语法 `::attr(x)` / `::text` / `::next_sibling_text` 由 `spider/mc-selector.ts` 求值。 + +| 状态 | 站点 | +| --- | --- | +| 启用(主力,仍须真实账号核验) | audiences 观众、hhanclub 憨憨、chdbits 彩虹岛、mteam 馒头 | +| `enabled:false`,**未实测** | agsvpt、hdarea、hddolby、hdfans、hdhome、hdsky、hdtime、hdvideo、keepfrds、nicept、ourbits、piggo、pter、pthome、pttime、soulvoice、ssd、tjupt、ttg | + +未实测站点只经 schema 校验,选择器是 MovieClaw 作者在 2026-08/09 的实测值,未经本项目核验。TTG 需要自定义适配器(中文分类、`/t/` 路径、做种/下载同列),通用 spider 结果不可靠;OurBits 的 `cat[]` 分类参数已由定义表达;hddolby 在 MovieClaw 中按 HTML 站点配置,MP 则用 API spider(`HDDolby`),两条路径待账号确认后二选一。 + +### 待真实站点核验(W1-3c 新增) + +1. HHClub:`div.torrent-table-sub-info` 卡片结构、`?cat[]=` 分类链接(若 URL 编码为 `cat%5B%5D=` 则 `\d+` 会取错,需改 querystring 过滤)、`promotion-tag-*` 新旧类名、`div.torrent-info-hr`、详情页 Tailwind 选择器、分页 0 起。 +2. CHDBits:域名 `ptchdbits.co`、`font.subtitle`、`td.embedded > span[title]` 促销截止、`a[href='logout.php']`。 +3. 专用 spider 的 JSON 字段均按 MP 读取的键编写 fixture,真实响应待核;HDDolby `hr` 字段语义、`page_number` 起点;TorrentLeech 时区;Yema `tagList` 类型;直链缺失时的回退地址(`TODO(verify)`)。 +4. 用户信息解析器 fixture 均为最小自编 HTML/JSON;`normalizeDatetime` 不处理相对时间(MP 用 dateparser)。 +5. 馒头 `mode=adult` 独立索引(MovieClaw custom/mteam.py)未实现。 diff --git a/packages/adapters/src/sites/definitions/NOTICE-MovieClaw.md b/packages/adapters/src/sites/definitions/NOTICE-MovieClaw.md new file mode 100644 index 00000000..fc04fb9f --- /dev/null +++ b/packages/adapters/src/sites/definitions/NOTICE-MovieClaw.md @@ -0,0 +1,38 @@ +# 第三方声明:MovieClaw(MIT) + +本目录下的站点定义 JSON(`provenance.source = "movieclaw"`)由 `scripts/mp-sync/movieclaw-import.mjs` +从 MovieClaw **tag `v0.27.0`(提交 `c79bb1b065c826823fc1c15bcb99b3c282c3fce0`)** 的 +`src/movieclaw_tracker/sites/configs/*.yaml` 与 `src/movieclaw_tracker/selectors.py`(`NexusPHPSelectors` 默认值) +转换而来;`packages/adapters/src/sites/spider/mc-selector.ts` 的选择器语义对照 `frameworks/nexusphp.py` `_css_text`。 +该版本以 MIT 许可发布(MovieClaw 自 `251f1988` 起改为附条件许可,**新许可下的内容未被使用**)。 + +- 上游仓库:https://github.com/movieclaw/movieclaw +- 许可类别:`copied-mit`(见 `docs/porting/license-and-provenance.md`) +- 本项目的修改:字段改写为 MoviePilot indexer schema + `umt` 扩展;默认值展开;分类名与观众/馒头定义由 + `scripts/mp-sync/movieclaw-overrides/*.json` 覆盖。 + +以下为 MovieClaw v0.27.0 `LICENSE` 全文(原样): + +``` +MIT License + +Copyright (c) 2026 MovieClaw + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. +``` diff --git a/packages/adapters/src/sites/definitions/agsvpt.json b/packages/adapters/src/sites/definitions/agsvpt.json new file mode 100644 index 00000000..c7255666 --- /dev/null +++ b/packages/adapters/src/sites/definitions/agsvpt.json @@ -0,0 +1,252 @@ +{ + "id": "agsvpt", + "name": "AGSVPT", + "enabled": false, + "schema": "NexusPhp", + "provenance": { + "source": "movieclaw", + "ref": "v0.27.0", + "commit": "c79bb1b0", + "license": "MIT", + "file": "src/movieclaw_tracker/sites/configs/agsvpt.yaml" + }, + "domain": "https://www.agsvpt.com/", + "encoding": "UTF-8", + "public": false, + "proxy": false, + "language": "zh", + "result_num": 100, + "timeout": 30, + "search": { + "paths": [ + { + "path": "torrents.php", + "type": "all" + } + ], + "params": { + "search": "{keyword}" + } + }, + "batch": { + "delimiter": " ", + "space_replace": "_" + }, + "browse": { + "path": "torrents.php?page={page}", + "start": 0 + }, + "category": { + "movie": [ + { + "id": "401", + "desc": "电影" + } + ], + "tv": [ + { + "id": "402", + "desc": "剧集" + }, + { + "id": "403", + "desc": "剧集" + }, + { + "id": "407", + "desc": "剧集" + }, + { + "id": "419", + "desc": "剧集" + } + ], + "music": [ + { + "id": "406", + "desc": "音乐" + }, + { + "id": "408", + "desc": "音乐" + }, + { + "id": "411", + "desc": "音乐" + } + ] + }, + "torrents": { + "list": { + "selector": "table.torrents > tr:has(table.torrentname)" + }, + "fields": { + "title_default": { + "selector": "a[href*='details.php']" + }, + "title_optional": { + "selector": "a[href*='details.php'][title]", + "attribute": "title" + }, + "title": { + "text": "{% if fields['title_optional'] %}{{ fields['title_optional'] }}{% else %}{{ fields['title_default'] }}{% endif %}" + }, + "details": { + "selector": "a[href*='details.php']", + "attribute": "href" + }, + "download": { + "selector": "a[href*='download.php']", + "attribute": "href" + }, + "imdbid": { + "selector": "a[href*=\"imdb.com/title/tt\"]", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["tt\\d+", 0] + } + ] + }, + "date_elapsed": { + "selector": "td:nth-child(4)" + }, + "date_added": { + "selector": "td:nth-child(4) > span", + "attribute": "title" + }, + "date": { + "text": "{% if fields['date_elapsed'] or fields['date_added'] %}{{ fields['date_added'] if fields['date_added'] else fields['date_elapsed'] }}{% else %}now{% endif %}", + "filters": [ + { + "name": "dateparse", + "args": "%Y-%m-%d %H:%M:%S" + } + ] + }, + "size": { + "selector": "td:nth-child(5)" + }, + "seeders": { + "selector": "td:nth-child(6)" + }, + "leechers": { + "selector": "td:nth-child(7)" + }, + "grabs": { + "selector": "td:nth-child(8)" + }, + "downloadvolumefactor": { + "case": { + "img.pro_free": 0, + "img.pro_free2up": 0, + "img.pro_50pctdown": 0.5, + "img.pro_50pctdown2up": 0.5, + "img.pro_30pctdown": 0.3, + "*": 1 + } + }, + "uploadvolumefactor": { + "case": { + "img.pro_free2up": 2, + "img.pro_50pctdown2up": 2, + "img.pro_2up": 2, + "*": 1 + } + }, + "description": { + "selector": "td.embedded > span" + }, + "category": { + "selector": "a[href^='?cat=']", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["\\d+", 0] + } + ] + } + } + }, + "umt": { + "version": "agsvpt@mc-v0.27.0", + "kind": "nexusphp", + "displayName": "AGSVPT", + "timezone": "+08:00", + "rateLimit": { + "minIntervalMs": 3000, + "burst": 1 + }, + "capabilities": { + "list": true, + "search": true, + "detail": true, + "download": true, + "rss": false + }, + "categoryNames": { + "401": "电影", + "402": "剧集", + "403": "剧集", + "404": "纪录片", + "405": "动漫", + "406": "音乐", + "407": "剧集", + "408": "音乐", + "411": "音乐", + "413": "游戏", + "415": "其他", + "419": "剧集" + }, + "loginSuccess": "a[href*='logout']", + "emptyMarkers": ["没有种子", "没有结果", "Nothing found"], + "pagination": "a[href*=\"page=\"]", + "detailPath": "details.php", + "downloadPath": "download.php", + "login": { + "path": "login.php", + "usernameField": "username", + "passwordField": "password", + "errorCss": "td.text" + }, + "profile": { + "path": "userdetails.php", + "uid_css": "a[class*='User_Name']::attr(href)", + "username_css": "a[class*='User_Name']", + "class_css": "a[class*='User_Name']::attr(class)", + "uploaded_css": "font.color_uploaded::next_sibling_text", + "downloaded_css": "font.color_downloaded::next_sibling_text", + "ratio_css": "font.color_ratio::next_sibling_text", + "bonus_css": "td.rowhead:contains('魔力') + td::text", + "join_date_css": "td.rowhead:contains('加入日期') + td::text", + "seeding_css": "img[alt='Torrents seeding']::next_sibling_text", + "leeching_css": "img[alt='Torrents leeching']::next_sibling_text" + }, + "detail": { + "path": "details.php", + "title": "#top", + "rowLabel": "td.rowhead", + "rowValue": "td.rowfollow", + "labels": { + "subtitle": ["副标题", "Small Description"], + "basicInfo": ["基本信息", "Basic Info"], + "peers": ["同伴", "Peers"] + }, + "description": "#kdescr", + "headerScope": "#top", + "notFoundMarkers": ["没有该ID的种子", "No torrent with ID"], + "fields": { + "title": "#top::text", + "subtitle": "td.rowhead:contains('副标题') + td::text", + "size": "td.rowhead:contains('基本信息') + td.rowfollow > b::next_sibling_text", + "description": "#kdescr", + "download": "a[href*='download.php']::attr(href)", + "imdb": "a[href*='imdb.com']::attr(href)", + "douban": "a[href*='douban.com']::attr(href)", + "fileList": "td.rowfollow > ul > li::text" + } + } + } +} diff --git a/packages/adapters/src/sites/definitions/audiences.json b/packages/adapters/src/sites/definitions/audiences.json index 68b25216..35e49702 100644 --- a/packages/adapters/src/sites/definitions/audiences.json +++ b/packages/adapters/src/sites/definitions/audiences.json @@ -1,7 +1,16 @@ { - "_provenance": "按 MoviePilot SiteSpider indexer schema 手写;字段结构参考 MovieClaw audiences.yaml(data-ref,未复制);选择器值按 NexusPHP 公开页面惯例编写,仅对 testkit mock 验证,待真实页面核验。见 ../README.md", "id": "audiences", "name": "观众", + "enabled": true, + "schema": "NexusAudiences", + "provenance": { + "source": "movieclaw", + "ref": "v0.27.0", + "commit": "c79bb1b0", + "license": "MIT", + "file": "src/movieclaw_tracker/sites/configs/audiences.yaml", + "overrides": "scripts/mp-sync/movieclaw-overrides/audiences.json" + }, "domain": "https://audiences.me/", "encoding": "UTF-8", "public": false, @@ -10,14 +19,45 @@ "result_num": 100, "timeout": 30, "search": { - "paths": [{ "path": "torrents.php", "type": "all" }], - "params": { "search": "{keyword}", "search_area": 4 } + "paths": [ + { + "path": "torrents.php", + "type": "all" + } + ], + "params": { + "search": "{keyword}", + "search_area": 4 + } + }, + "batch": { + "delimiter": " ", + "space_replace": "_" + }, + "browse": { + "path": "torrents.php?page={page}", + "start": 0 }, - "batch": { "delimiter": " ", "space_replace": "_" }, - "browse": { "path": "torrents.php?page={page}", "start": 0 }, "category": { - "movie": [{ "id": "401", "cat": "Movies", "desc": "电影" }], - "tv": [{ "id": "402", "cat": "TV Series", "desc": "剧集" }, { "id": "403", "cat": "TV Shows", "desc": "综艺" }], + "movie": [ + { + "id": "401", + "cat": "Movies", + "desc": "电影" + } + ], + "tv": [ + { + "id": "402", + "cat": "TV Series", + "desc": "剧集" + }, + { + "id": "403", + "cat": "TV Shows", + "desc": "综艺" + } + ], "music": [] }, "torrents": { @@ -25,28 +65,62 @@ "selector": "table.torrents > tbody > tr:has(table.torrentname), table.torrents > tr:has(table.torrentname)" }, "fields": { - "title_default": { "selector": "a[href*=\"details.php?id=\"]" }, - "title_optional": { "selector": "a[title][href*=\"details.php?id=\"]", "attribute": "title" }, + "title_default": { + "selector": "a[href*=\"details.php?id=\"]" + }, + "title_optional": { + "selector": "a[title][href*=\"details.php?id=\"]", + "attribute": "title" + }, "title": { "text": "{% if fields['title_optional'] %}{{ fields['title_optional'] }}{% else %}{{ fields['title_default'] }}{% endif %}" }, - "details": { "selector": "a[href*=\"details.php?id=\"]", "attribute": "href" }, - "download": { "selector": "a[href*=\"download.php?id=\"]", "attribute": "href" }, + "details": { + "selector": "a[href*=\"details.php?id=\"]", + "attribute": "href" + }, + "download": { + "selector": "a[href*=\"download.php?id=\"]", + "attribute": "href" + }, "imdbid": { "selector": "a[href*=\"imdb.com/title/tt\"]", "attribute": "href", - "filters": [{ "name": "re_search", "args": ["tt\\d+", 0] }] + "filters": [ + { + "name": "re_search", + "args": ["tt\\d+", 0] + } + ] + }, + "date_elapsed": { + "selector": "td:nth-child(4) > span" + }, + "date_added": { + "selector": "td:nth-child(4) > span", + "attribute": "title" }, - "date_elapsed": { "selector": "td:nth-child(4) > span" }, - "date_added": { "selector": "td:nth-child(4) > span", "attribute": "title" }, "date": { "text": "{% if fields['date_elapsed'] or fields['date_added'] %}{{ fields['date_added'] if fields['date_added'] else fields['date_elapsed'] }}{% else %}now{% endif %}", - "filters": [{ "name": "dateparse", "args": "%Y-%m-%d %H:%M:%S" }] + "filters": [ + { + "name": "dateparse", + "args": "%Y-%m-%d %H:%M:%S" + } + ] + }, + "size": { + "selector": "td:nth-child(5)" + }, + "seeders": { + "selector": "td:nth-child(6)" + }, + "leechers": { + "selector": "td:nth-child(7)" + }, + "grabs": { + "selector": "td:nth-child(8)" }, - "size": { "selector": "td:nth-child(5)" }, - "seeders": { "selector": "td:nth-child(6)" }, - "leechers": { "selector": "td:nth-child(7)" }, - "grabs": { "selector": "td:nth-child(8)" }, "downloadvolumefactor": { "case": { "img.pro_free": 0, @@ -65,17 +139,31 @@ "*": 1 } }, - "freedate": { - "selector": "img[class^=\"pro_\"]", - "attribute": "title", - "filters": [{ "name": "re_search", "args": ["\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}", 0] }] + "description": { + "selector": "span.torrent-subtitle-line" }, - "description": { "selector": "span.torrent-subtitle-line" }, - "labels": { "selector": "span.tags" }, "category": { "selector": "a[href*=\"?cat=\"]", "attribute": "href", - "filters": [{ "name": "querystring", "args": "cat" }] + "filters": [ + { + "name": "querystring", + "args": "cat" + } + ] + }, + "freedate": { + "selector": "img[class^=\"pro_\"]", + "attribute": "title", + "filters": [ + { + "name": "re_search", + "args": ["\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}", 0] + } + ] + }, + "labels": { + "selector": "span.tags" } } }, @@ -84,14 +172,47 @@ "kind": "nexusphp", "displayName": "观众", "timezone": "+08:00", - "rateLimit": { "minIntervalMs": 3000, "burst": 1 }, - "capabilities": { "list": true, "search": true, "detail": true, "download": true, "rss": true }, - "categoryNames": { "401": "电影", "402": "剧集", "403": "综艺", "406": "纪录片" }, + "rateLimit": { + "minIntervalMs": 3000, + "burst": 1 + }, + "capabilities": { + "list": true, + "search": true, + "detail": true, + "download": true, + "rss": true + }, + "categoryNames": { + "401": "电影", + "402": "剧集", + "403": "综艺", + "406": "纪录片" + }, "loginSuccess": "a[href*=\"logout\"]", "emptyMarkers": ["没有种子", "没有结果", "Nothing found"], "pagination": "a[href*=\"page=\"]", "detailPath": "details.php", "downloadPath": "download.php", + "login": { + "path": "login.php", + "usernameField": "username", + "passwordField": "password", + "errorCss": "td.text" + }, + "profile": { + "path": "userdetails.php", + "uid_css": "a[href*='userdetails.php?id=']::attr(href)", + "username_css": "a[href*='userdetails.php?id=']", + "class_css": "td.rowhead:contains('等级') + td::text", + "uploaded_css": "span.site-userbar__compact-metric--uploaded", + "downloaded_css": "span.site-userbar__compact-metric--downloaded", + "ratio_css": "td.rowhead:contains('分享率') + td::text", + "bonus_css": "td.rowhead:contains('魔力') + td::text", + "join_date_css": "td.rowhead:contains('加入日期') + td::text", + "seeding_css": "td.rowhead:contains('当前做种') + td::text", + "leeching_css": "td.rowhead:contains('当前下载') + td::text" + }, "detail": { "path": "details.php", "title": "h1#top", diff --git a/packages/adapters/src/sites/definitions/chdbits.json b/packages/adapters/src/sites/definitions/chdbits.json new file mode 100644 index 00000000..061a20fb --- /dev/null +++ b/packages/adapters/src/sites/definitions/chdbits.json @@ -0,0 +1,252 @@ +{ + "id": "chdbits", + "name": "彩虹岛", + "enabled": true, + "schema": "NexusPhp", + "provenance": { + "source": "movieclaw", + "ref": "v0.27.0", + "commit": "c79bb1b0", + "license": "MIT", + "file": "src/movieclaw_tracker/sites/configs/chdbits.yaml", + "overrides": "scripts/mp-sync/movieclaw-overrides/chdbits.json" + }, + "domain": "https://ptchdbits.co/", + "encoding": "UTF-8", + "public": false, + "proxy": false, + "language": "zh", + "result_num": 100, + "timeout": 30, + "search": { + "paths": [ + { + "path": "torrents.php", + "type": "all" + } + ], + "params": { + "search": "{keyword}" + } + }, + "batch": { + "delimiter": " ", + "space_replace": "_" + }, + "browse": { + "path": "torrents.php?page={page}", + "start": 0 + }, + "category": { + "movie": [ + { + "id": "401", + "desc": "电影" + } + ], + "tv": [ + { + "id": "402", + "desc": "剧集" + }, + { + "id": "403", + "desc": "综艺" + }, + { + "id": "407", + "desc": "体育" + } + ], + "music": [ + { + "id": "406", + "desc": "MV" + }, + { + "id": "408", + "desc": "HQ Audio" + } + ] + }, + "torrents": { + "list": { + "selector": "table.torrents > tr:has(table.torrentname)" + }, + "fields": { + "title_default": { + "selector": "a[href*='details.php']" + }, + "title_optional": { + "selector": "a[href*='details.php'][title]", + "attribute": "title" + }, + "title": { + "text": "{% if fields['title_optional'] %}{{ fields['title_optional'] }}{% else %}{{ fields['title_default'] }}{% endif %}" + }, + "details": { + "selector": "a[href*='details.php']", + "attribute": "href" + }, + "download": { + "selector": "a[href*='download.php']", + "attribute": "href" + }, + "imdbid": { + "selector": "a[href*=\"imdb.com/title/tt\"]", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["tt\\d+", 0] + } + ] + }, + "date_elapsed": { + "selector": "td:nth-child(4)" + }, + "date_added": { + "selector": "td:nth-child(4) > span", + "attribute": "title" + }, + "date": { + "text": "{% if fields['date_elapsed'] or fields['date_added'] %}{{ fields['date_added'] if fields['date_added'] else fields['date_elapsed'] }}{% else %}now{% endif %}", + "filters": [ + { + "name": "dateparse", + "args": "%Y-%m-%d %H:%M:%S" + } + ] + }, + "size": { + "selector": "td:nth-child(5)" + }, + "seeders": { + "selector": "td:nth-child(6)" + }, + "leechers": { + "selector": "td:nth-child(7)" + }, + "grabs": { + "selector": "td:nth-child(8)" + }, + "downloadvolumefactor": { + "case": { + "img.pro_free": 0, + "img.pro_free2up": 0, + "img.pro_50pctdown": 0.5, + "img.pro_50pctdown2up": 0.5, + "img.pro_30pctdown": 0.3, + "*": 1 + } + }, + "uploadvolumefactor": { + "case": { + "img.pro_free2up": 2, + "img.pro_50pctdown2up": 2, + "img.pro_2up": 2, + "*": 1 + } + }, + "description": { + "selector": "font.subtitle" + }, + "category": { + "selector": "a[href^='?cat=']", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["\\d+", 0] + } + ] + }, + "freedate": { + "selector": "td.embedded > span[title]", + "attribute": "title", + "filters": [ + { + "name": "re_search", + "args": ["\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}", 0] + } + ] + } + } + }, + "umt": { + "version": "chdbits@mc-v0.27.0", + "kind": "nexusphp", + "displayName": "彩虹岛", + "timezone": "+08:00", + "rateLimit": { + "minIntervalMs": 3000, + "burst": 1 + }, + "capabilities": { + "list": true, + "search": true, + "detail": true, + "download": true, + "rss": false + }, + "categoryNames": { + "401": "电影", + "402": "剧集", + "403": "综艺", + "404": "纪录片", + "405": "动漫", + "406": "MV", + "407": "体育", + "408": "HQ Audio", + "409": "其他" + }, + "loginSuccess": "a[href='logout.php']", + "emptyMarkers": ["没有种子", "没有结果", "Nothing found"], + "pagination": "a[href*=\"page=\"]", + "detailPath": "details.php", + "downloadPath": "download.php", + "login": { + "path": "login.php", + "usernameField": "username", + "passwordField": "password", + "errorCss": "td.text" + }, + "profile": { + "path": "userdetails.php", + "username_css": "a[href^='userdetails.php?id=']", + "class_css": "td.rowhead:contains('等级') + td::text", + "uploaded_css": "font.color_uploaded::next_sibling_text", + "downloaded_css": "font.color_downloaded::next_sibling_text", + "ratio_css": "font.color_ratio::next_sibling_text", + "bonus_css": "td.rowhead:contains('魔力') + td::text", + "join_date_css": "td.rowhead:contains('加入日期') + td::text", + "seeding_css": "img[alt='Torrents seeding']::next_sibling_text", + "leeching_css": "img[alt='Torrents leeching']::next_sibling_text", + "vip_css": "a[class^='VIP']" + }, + "detail": { + "path": "details.php", + "title": "#top", + "rowLabel": "td.rowhead", + "rowValue": "td.rowfollow", + "labels": { + "subtitle": ["副标题", "Small Description"], + "basicInfo": ["基本信息", "Basic Info"], + "peers": ["同伴", "Peers"] + }, + "description": "#kdescr", + "headerScope": "#top", + "notFoundMarkers": ["没有该ID的种子", "No torrent with ID"], + "fields": { + "title": "#top::text", + "subtitle": "td.rowhead:contains('副标题') + td::text", + "size": "td.rowhead:contains('大小') + td::text", + "description": "#kdescr", + "download": "a[href*='download.php']::attr(href)", + "imdb": "a[href*='imdb.com']::attr(href)", + "douban": "a[href*='douban.com']::attr(href)", + "fileList": "td.rowfollow > ul > li::text" + } + } + } +} diff --git a/packages/adapters/src/sites/definitions/definitions.test.ts b/packages/adapters/src/sites/definitions/definitions.test.ts new file mode 100644 index 00000000..6c9e2dc0 --- /dev/null +++ b/packages/adapters/src/sites/definitions/definitions.test.ts @@ -0,0 +1,156 @@ +import { CHDBITS_LIST_PAGE, CHDBITS_LOGIN_PAGE, HHANCLUB_DETAIL_PAGE, HHANCLUB_LIST_PAGE } from "@umt/testkit"; +import { describe, expect, it } from "vitest"; +import { getSiteDefinition, listSiteDefinitions } from "../registry.ts"; +import { parseDetailPage } from "../spider/detail.ts"; +import { spiderTorrentToRelease } from "../spider/map.ts"; +import type { IndexerDefinition } from "../spider/schema.ts"; +import { SiteSpider } from "../spider/spider.ts"; +import { + BUILTIN_SITE_DEFINITIONS, + chdbitsDefinition, + hhanclubDefinition, + loadSiteDefinition, + parseIndexerDefinition, +} from "./index.ts"; +import { MOVIECLAW_DEFINITION_JSON } from "./manifest.generated.ts"; + +const OBSERVED = "2026-09-27T00:00:00.000Z"; +const PRIMARY = ["audiences", "chdbits", "hhanclub", "mteam"]; + +describe("站点定义:加载时 zod 校验", () => { + const ids = Object.keys(MOVIECLAW_DEFINITION_JSON).sort(); + + it("MovieClaw v0.27.0 全部 23 站转换结果均在注册表中", () => { + expect(ids).toHaveLength(23); + expect(listSiteDefinitions().map((d) => d.id)).toEqual(ids); + }); + + it.each(ids)("%s:schema 校验通过,provenance 指向 MovieClaw v0.27.0 MIT", (id) => { + const def = loadSiteDefinition(MOVIECLAW_DEFINITION_JSON[id]); + expect(def.id).toBe(id); + expect(def.provenance).toMatchObject({ source: "movieclaw", ref: "v0.27.0", commit: "c79bb1b0", license: "MIT" }); + expect(def.provenance?.file).toBe(`src/movieclaw_tracker/sites/configs/${id}.yaml`); + expect(def.enabled).toBe(PRIMARY.includes(id)); + expect(getSiteDefinition(id)).toBe(BUILTIN_SITE_DEFINITIONS[id]); + }); + + it("enabledOnly 只列 4 个主力站", () => { + expect(listSiteDefinitions({ enabledOnly: true }).map((d) => d.id)).toEqual(PRIMARY); + }); + + it("非法定义被拒绝并指出字段路径", () => { + const bad = structuredClone(MOVIECLAW_DEFINITION_JSON.chdbits) as Record; + (bad as { domain: string }).domain = "https://ptchdbits.co"; // 缺结尾 / + delete (bad as { umt: Record }).umt.timezone; + expect(() => parseIndexerDefinition(bad)).toThrow(/chdbits.*domain.*umt\.timezone|chdbits.*umt\.timezone.*domain/); + expect(() => loadSiteDefinition({ id: "x", umt: { kind: "gopher" } })).toThrow(/umt\.kind 未知/); + const typo = structuredClone(MOVIECLAW_DEFINITION_JSON.hhanclub) as { + torrents: { fields: Record }; + }; + typo.torrents.fields.size = { selectr: "div" }; + expect(() => parseIndexerDefinition(typo)).toThrow(/torrents\.fields\.size/); + }); + + it("观众:W1-3b 手写值优先(覆盖层),MovieClaw 值补空", () => { + const a = getSiteDefinition("audiences") as IndexerDefinition; + expect(a.search?.params).toEqual({ search: "{keyword}", search_area: 4 }); // W1-3b + expect(a.torrents.fields.category?.filters).toEqual([{ name: "querystring", args: "cat" }]); // W1-3b + expect(a.umt.profile?.uploaded_css).toBe("span.site-userbar__compact-metric--uploaded"); // MovieClaw 补空 + expect(a.schema).toBe("NexusAudiences"); + expect(a.provenance?.overrides).toBe("scripts/mp-sync/movieclaw-overrides/audiences.json"); + }); + + it("OurBits 的 cat[] 分类参数由定义表达;TTG 需自定义适配器 → 禁用", () => { + const ob = getSiteDefinition("ourbits") as IndexerDefinition; + expect(ob.category?.param).toBe("cat[]"); + const url = new URL(new SiteSpider(ob, { keyword: "x", mtype: "movie" }).getSearchUrl()); + expect(url.searchParams.getAll("cat[]").length).toBeGreaterThan(0); + const ttg = getSiteDefinition("ttg") as IndexerDefinition; + expect(ttg.umt.customClass).toContain("TTGSite"); + expect(ttg.enabled).toBe(false); + }); +}); + +describe("HHClub 定义(div 卡片模板)", () => { + const parse = (html: string) => { + const r = new SiteSpider(hhanclubDefinition, {}).parse(html); + return r.torrents.map((t) => spiderTorrentToRelease(t, hhanclubDefinition, OBSERVED)!); + }; + + it("列表:标题/副标题/综艺原始分类 403/大小/人数/发布时间/免费 + H&R", () => { + const [a, b] = parse(HHANCLUB_LIST_PAGE); + expect(a).toMatchObject({ + siteId: "hhanclub", + siteReleaseId: "90001", + title: "Go.Fighting.S10E01.2026.2160p.WEB-DL.H265.AAC-HHWEB", + subtitle: "极限挑战 第十季 第1期", + category: { id: "403", name: "综艺" }, + sizeBytes: Math.round(6.12 * 1024 ** 3), + seeders: 88, + leechers: 5, + snatched: 1024, + promotion: { type: "free", downloadFactor: 0, uploadFactor: 1 }, + hr: true, + uploadedAt: "2026-09-20T12:30:00.000Z", + detailUrl: "https://hhanclub.net/details.php?id=90001&hit=1", + }); + expect(b).toMatchObject({ + siteReleaseId: "90002", + category: { id: "401", name: "电影" }, + promotion: { type: "2x50%", downloadFactor: 0.5, uploadFactor: 2 }, + hr: false, + }); + }); + + it("浏览从 page=0 起(MovieClaw page_offset -1);mtype=tv 选 402/403/407/412", () => { + expect(new SiteSpider(hhanclubDefinition, { page: 0 }).getSearchUrl()).toBe( + "https://hhanclub.net/torrents.php?page=0", + ); + const u = new URL(new SiteSpider(hhanclubDefinition, { keyword: "极限挑战", mtype: "tv" }).getSearchUrl()); + expect(["cat402", "cat403", "cat407", "cat412"].every((k) => u.searchParams.get(k) === "1")).toBe(true); + expect(u.searchParams.get("search")).toBe("极限挑战"); + }); + + it("详情页:逐字段选择器(Tailwind div 结构)取标题/副标题/大小/IMDb/豆瓣", () => { + const { kind, detail } = parseDetailPage(HHANCLUB_DETAIL_PAGE, hhanclubDefinition, "90002", OBSERVED); + expect(kind).toBe("ok"); + expect(detail).toMatchObject({ + title: "Dune.Part.Two.2024.2160p.UHD.BluRay.REMUX-HHWEB", + subtitle: "沙丘2 | 中字", + sizeBytes: Math.round(78.3 * 1024 ** 3), + imdbId: "tt15239678", + doubanId: "35575567", + }); + }); +}); + +describe("CHDBits 定义(标准 NexusPHP)", () => { + it("列表:title 属性全名、font.subtitle、2x 免费 + 截止时间、cat 链接分类", () => { + const r = new SiteSpider(chdbitsDefinition, {}).parse(CHDBITS_LIST_PAGE); + const [a, b] = r.torrents.map((t) => spiderTorrentToRelease(t, chdbitsDefinition, OBSERVED)!); + expect(a).toMatchObject({ + siteReleaseId: "555001", + title: "The.Last.of.Us.S02.2025.1080p.BluRay.x264.DTS-HD.MA.5.1-CHD", + category: { id: "402", name: "剧集" }, + sizeBytes: Math.round(45.2 * 1024 ** 3), + seeders: 42, + leechers: 3, + snatched: 100, + promotion: { type: "2xfree", downloadFactor: 0, uploadFactor: 2, expiresAt: "2026-09-30T04:00:00.000Z" }, + hr: "unknown", + uploadedAt: "2026-09-25T00:00:00.000Z", + }); + expect(a!.subtitle).toContain("最后生还者 第二季"); + expect(b).toMatchObject({ + siteReleaseId: "555002", + category: { id: "401", name: "电影" }, + promotion: { type: "none", downloadFactor: 1, uploadFactor: 1, expiresAt: null }, + }); + }); + + it("登录页识别为登录/权限页,不产生结果", () => { + const r = new SiteSpider(chdbitsDefinition, {}).parse(CHDBITS_LOGIN_PAGE); + expect(r.loginPage).toBe(true); + expect(r.torrents).toEqual([]); + }); +}); diff --git a/packages/adapters/src/sites/definitions/hdarea.json b/packages/adapters/src/sites/definitions/hdarea.json new file mode 100644 index 00000000..9c8bd574 --- /dev/null +++ b/packages/adapters/src/sites/definitions/hdarea.json @@ -0,0 +1,279 @@ +{ + "id": "hdarea", + "name": "高清视界 HDArea", + "enabled": false, + "schema": "NexusPhp", + "provenance": { + "source": "movieclaw", + "ref": "v0.27.0", + "commit": "c79bb1b0", + "license": "MIT", + "file": "src/movieclaw_tracker/sites/configs/hdarea.yaml" + }, + "domain": "https://www.hdarea.club/", + "encoding": "UTF-8", + "public": false, + "proxy": false, + "language": "zh", + "result_num": 100, + "timeout": 30, + "search": { + "paths": [ + { + "path": "torrents.php", + "type": "all" + } + ], + "params": { + "search": "{keyword}" + } + }, + "batch": { + "delimiter": " ", + "space_replace": "_" + }, + "browse": { + "path": "torrents.php?page={page}", + "start": 0 + }, + "category": { + "movie": [ + { + "id": "300", + "desc": "电影" + }, + { + "id": "401", + "desc": "电影" + }, + { + "id": "410", + "desc": "电影" + }, + { + "id": "411", + "desc": "电影" + }, + { + "id": "412", + "desc": "电影" + }, + { + "id": "413", + "desc": "电影" + }, + { + "id": "414", + "desc": "电影" + }, + { + "id": "415", + "desc": "电影" + }, + { + "id": "416", + "desc": "电影" + }, + { + "id": "417", + "desc": "电影" + } + ], + "tv": [ + { + "id": "402", + "desc": "剧集" + }, + { + "id": "403", + "desc": "剧集" + } + ], + "music": [ + { + "id": "406", + "desc": "音乐" + }, + { + "id": "408", + "desc": "音乐" + } + ] + }, + "torrents": { + "list": { + "selector": "table.torrents > tr, table.torrents > tbody > tr" + }, + "fields": { + "title_default": { + "selector": "table.torrentname a b" + }, + "title_optional": { + "selector": "table.torrentname a b[title]", + "attribute": "title" + }, + "title": { + "text": "{% if fields['title_optional'] %}{{ fields['title_optional'] }}{% else %}{{ fields['title_default'] }}{% endif %}" + }, + "details": { + "selector": "a[href*='details.php']", + "attribute": "href" + }, + "download": { + "selector": "a[href*='download.php']", + "attribute": "href" + }, + "imdbid": { + "selector": "a[href*=\"imdb.com/title/tt\"]", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["tt\\d+", 0] + } + ] + }, + "date_elapsed": { + "selector": "td:nth-child(5)" + }, + "date_added": { + "selector": "td:nth-child(5) > span", + "attribute": "title" + }, + "date": { + "text": "{% if fields['date_elapsed'] or fields['date_added'] %}{{ fields['date_added'] if fields['date_added'] else fields['date_elapsed'] }}{% else %}now{% endif %}", + "filters": [ + { + "name": "dateparse", + "args": "%Y-%m-%d %H:%M:%S" + } + ] + }, + "size": { + "selector": "td:nth-child(6)" + }, + "seeders": { + "selector": "td:nth-child(7)" + }, + "leechers": { + "selector": "td:nth-child(8)" + }, + "grabs": { + "selector": "td:nth-child(9)" + }, + "downloadvolumefactor": { + "case": { + "img.pro_free": 0, + "img.pro_free2up": 0, + "*": 1 + } + }, + "uploadvolumefactor": { + "case": { + "img.pro_free2up": 2, + "*": 1 + } + }, + "description": { + "selector": "td.embedded > div[title]", + "attribute": "title" + }, + "category": { + "selector": "td:nth-child(1) a[href*='cat=']", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["\\d+", 0] + } + ] + } + } + }, + "umt": { + "version": "hdarea@mc-v0.27.0", + "kind": "nexusphp", + "displayName": "高清视界 HDArea", + "timezone": "+08:00", + "rateLimit": { + "minIntervalMs": 60000, + "burst": 1 + }, + "capabilities": { + "list": true, + "search": true, + "detail": true, + "download": true, + "rss": false + }, + "categoryNames": { + "300": "电影", + "401": "电影", + "402": "剧集", + "403": "剧集", + "404": "纪录片", + "405": "动漫", + "406": "音乐", + "407": "其他", + "408": "音乐", + "409": "其他", + "410": "电影", + "411": "电影", + "412": "电影", + "413": "电影", + "414": "电影", + "415": "电影", + "416": "电影", + "417": "电影" + }, + "loginSuccess": "a[href*='logout']", + "emptyMarkers": ["没有种子", "没有结果", "Nothing found"], + "pagination": "a[href*=\"page=\"]", + "detailPath": "details.php", + "downloadPath": "download.php", + "authSupported": ["cookie", "credential"], + "login": { + "path": "login.php", + "usernameField": "username", + "passwordField": "password", + "errorCss": "td.text" + }, + "profile": { + "path": "userdetails.php", + "uid_css": "a[class*='_Name']::attr(href)", + "username_css": "a[class*='_Name'] b", + "class_css": "a[class*='_Name']::attr(class)", + "uploaded_css": "font.color_uploaded::next_sibling_text", + "downloaded_css": "font.color_downloaded::next_sibling_text", + "ratio_css": "font.color_ratio::next_sibling_text", + "bonus_css": "font.color_bonus", + "join_date_css": "td.rowhead:contains('加入日期') + td::text", + "seeding_css": "img.arrowup::next_sibling_text", + "leeching_css": "img.arrowdown::next_sibling_text" + }, + "detail": { + "path": "details.php", + "title": "h1#top", + "rowLabel": "td.rowhead", + "rowValue": "td.rowfollow", + "labels": { + "subtitle": ["副标题", "Small Description"], + "basicInfo": ["基本信息", "Basic Info"], + "peers": ["同伴", "Peers"] + }, + "description": "#kdescr", + "headerScope": "h1#top", + "notFoundMarkers": ["没有该ID的种子", "No torrent with ID"], + "fields": { + "title": "h1#top", + "subtitle": "td.rowhead:contains('副标题') + td::text", + "size": "td.rowhead:contains('基本信息') + td.rowfollow > b::next_sibling_text", + "description": "#kdescr", + "download": "a[href*='download.php']::attr(href)", + "imdb": "a[href*='imdb.com']::attr(href)", + "douban": "a[href*='douban.com']::attr(href)", + "fileList": "td.rowfollow > ul > li::text" + } + } + } +} diff --git a/packages/adapters/src/sites/definitions/hddolby.json b/packages/adapters/src/sites/definitions/hddolby.json new file mode 100644 index 00000000..fdb3cec0 --- /dev/null +++ b/packages/adapters/src/sites/definitions/hddolby.json @@ -0,0 +1,236 @@ +{ + "id": "hddolby", + "name": "高清杜比 HDDolby", + "enabled": false, + "schema": "HDDolby", + "provenance": { + "source": "movieclaw", + "ref": "v0.27.0", + "commit": "c79bb1b0", + "license": "MIT", + "file": "src/movieclaw_tracker/sites/configs/hddolby.yaml" + }, + "domain": "https://www.hddolby.com/", + "encoding": "UTF-8", + "public": false, + "proxy": false, + "language": "zh", + "result_num": 100, + "timeout": 30, + "search": { + "paths": [ + { + "path": "torrents.php", + "type": "all" + } + ], + "params": { + "search": "{keyword}" + } + }, + "batch": { + "delimiter": " ", + "space_replace": "_" + }, + "browse": { + "path": "torrents.php?page={page}", + "start": 0 + }, + "category": { + "movie": [ + { + "id": "401", + "desc": "电影" + } + ], + "tv": [ + { + "id": "402", + "desc": "剧集" + }, + { + "id": "403", + "desc": "剧集" + } + ], + "music": [ + { + "id": "406", + "desc": "音乐" + }, + { + "id": "408", + "desc": "音乐" + } + ] + }, + "torrents": { + "list": { + "selector": "table.torrents > tr, table.torrents > tbody > tr" + }, + "fields": { + "title_default": { + "selector": "table.torrentname a b" + }, + "title_optional": { + "selector": "table.torrentname a b[title]", + "attribute": "title" + }, + "title": { + "text": "{% if fields['title_optional'] %}{{ fields['title_optional'] }}{% else %}{{ fields['title_default'] }}{% endif %}" + }, + "details": { + "selector": "a[href*='details.php']", + "attribute": "href" + }, + "download": { + "selector": "a[href*='download.php']", + "attribute": "href" + }, + "imdbid": { + "selector": "a[href*=\"imdb.com/title/tt\"]", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["tt\\d+", 0] + } + ] + }, + "date_elapsed": { + "selector": "td:nth-child(4)" + }, + "date_added": { + "selector": "td:nth-child(4) > span", + "attribute": "title" + }, + "date": { + "text": "{% if fields['date_elapsed'] or fields['date_added'] %}{{ fields['date_added'] if fields['date_added'] else fields['date_elapsed'] }}{% else %}now{% endif %}", + "filters": [ + { + "name": "dateparse", + "args": "%Y-%m-%d %H:%M:%S" + } + ] + }, + "size": { + "selector": "td:nth-child(5)" + }, + "seeders": { + "selector": "td:nth-child(6)" + }, + "leechers": { + "selector": "td:nth-child(7)" + }, + "grabs": { + "selector": "td:nth-child(8)" + }, + "downloadvolumefactor": { + "case": { + "img.pro_free": 0, + "img.pro_free2up": 0, + "img.pro_50pctdown": 0.5, + "*": 1 + } + }, + "uploadvolumefactor": { + "case": { + "img.pro_free2up": 2, + "*": 1 + } + }, + "description": { + "selector": "td.embedded span[style*='float:left']" + }, + "category": { + "selector": "td:nth-child(1) a[href*='cat=']", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["\\d+", 0] + } + ] + } + } + }, + "umt": { + "version": "hddolby@mc-v0.27.0", + "kind": "nexusphp", + "displayName": "高清杜比 HDDolby", + "timezone": "+08:00", + "rateLimit": { + "minIntervalMs": 5000, + "burst": 1 + }, + "capabilities": { + "list": true, + "search": true, + "detail": true, + "download": true, + "rss": false + }, + "categoryNames": { + "401": "电影", + "402": "剧集", + "403": "剧集", + "404": "纪录片", + "405": "动漫", + "406": "音乐", + "407": "其他", + "408": "音乐", + "409": "其他", + "410": "游戏", + "411": "其他" + }, + "loginSuccess": "a[href*='logout']", + "emptyMarkers": ["没有种子", "没有结果", "Nothing found"], + "pagination": "a[href*=\"page=\"]", + "detailPath": "details.php", + "downloadPath": "download.php", + "authSupported": ["cookie", "credential"], + "login": { + "path": "login.php", + "usernameField": "username", + "passwordField": "password", + "errorCss": "td.text" + }, + "profile": { + "path": "userdetails.php", + "uid_css": "a[class*='_Name']::attr(href)", + "username_css": "a[class*='_Name'] b", + "class_css": "a[class*='_Name']::attr(class)", + "uploaded_css": "font.color_uploaded::next_sibling_text", + "downloaded_css": "font.color_downloaded::next_sibling_text", + "ratio_css": "font.color_ratio::next_sibling_text", + "bonus_css": "font.color_bonus", + "join_date_css": "td.rowhead:contains('加入日期') + td::text", + "seeding_css": "img.arrowup::next_sibling_text", + "leeching_css": "img.arrowdown::next_sibling_text" + }, + "detail": { + "path": "details.php", + "title": "h1#top", + "rowLabel": "td.rowhead", + "rowValue": "td.rowfollow", + "labels": { + "subtitle": ["副标题", "Small Description"], + "basicInfo": ["基本信息", "Basic Info"], + "peers": ["同伴", "Peers"] + }, + "description": "#kdescr", + "headerScope": "h1#top", + "notFoundMarkers": ["没有该ID的种子", "No torrent with ID"], + "fields": { + "title": "h1#top", + "subtitle": "td.rowhead:contains('副标题') + td::text", + "size": "td.rowhead:contains('基本信息') + td.rowfollow > b::next_sibling_text", + "description": "#kdescr", + "download": "a[href*='download.php']::attr(href)", + "imdb": "a[href*='imdb.com']::attr(href)", + "douban": "a[href*='douban.com']::attr(href)", + "fileList": "td.rowfollow > ul > li::text" + } + } + } +} diff --git a/packages/adapters/src/sites/definitions/hdfans.json b/packages/adapters/src/sites/definitions/hdfans.json new file mode 100644 index 00000000..f41b600d --- /dev/null +++ b/packages/adapters/src/sites/definitions/hdfans.json @@ -0,0 +1,245 @@ +{ + "id": "hdfans", + "name": "HDFans", + "enabled": false, + "schema": "NexusPhp", + "provenance": { + "source": "movieclaw", + "ref": "v0.27.0", + "commit": "c79bb1b0", + "license": "MIT", + "file": "src/movieclaw_tracker/sites/configs/hdfans.yaml" + }, + "domain": "https://hdfans.org/", + "encoding": "UTF-8", + "public": false, + "proxy": false, + "language": "zh", + "result_num": 100, + "timeout": 30, + "search": { + "paths": [ + { + "path": "torrents.php", + "type": "all" + } + ], + "params": { + "search": "{keyword}" + } + }, + "batch": { + "delimiter": " ", + "space_replace": "_" + }, + "browse": { + "path": "torrents.php?page={page}", + "start": 0 + }, + "category": { + "movie": [ + { + "id": "401", + "desc": "电影" + } + ], + "tv": [ + { + "id": "402", + "desc": "剧集" + }, + { + "id": "416", + "desc": "剧集" + } + ], + "music": [ + { + "id": "406", + "desc": "音乐" + }, + { + "id": "407", + "desc": "音乐" + }, + { + "id": "408", + "desc": "音乐" + } + ] + }, + "torrents": { + "list": { + "selector": "table.torrents > tr, table.torrents > tbody > tr" + }, + "fields": { + "title_default": { + "selector": "table.torrentname a b" + }, + "title_optional": { + "selector": "table.torrentname a b[title]", + "attribute": "title" + }, + "title": { + "text": "{% if fields['title_optional'] %}{{ fields['title_optional'] }}{% else %}{{ fields['title_default'] }}{% endif %}" + }, + "details": { + "selector": "a[href*='details.php']", + "attribute": "href" + }, + "download": { + "selector": "a[href*='download.php']", + "attribute": "href" + }, + "imdbid": { + "selector": "a[href*=\"imdb.com/title/tt\"]", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["tt\\d+", 0] + } + ] + }, + "date_elapsed": { + "selector": "td:nth-child(4)" + }, + "date_added": { + "selector": "td:nth-child(4) > span", + "attribute": "title" + }, + "date": { + "text": "{% if fields['date_elapsed'] or fields['date_added'] %}{{ fields['date_added'] if fields['date_added'] else fields['date_elapsed'] }}{% else %}now{% endif %}", + "filters": [ + { + "name": "dateparse", + "args": "%Y-%m-%d %H:%M:%S" + } + ] + }, + "size": { + "selector": "td:nth-child(5)" + }, + "seeders": { + "selector": "td:nth-child(6)" + }, + "leechers": { + "selector": "td:nth-child(7)" + }, + "grabs": { + "selector": "td:nth-child(8)" + }, + "downloadvolumefactor": { + "case": { + "img.pro_free": 0, + "img.pro_free2up": 0, + "img.pro_50pctdown": 0.5, + "*": 1 + } + }, + "uploadvolumefactor": { + "case": { + "img.pro_free2up": 2, + "*": 1 + } + }, + "description": { + "selector": "td.embedded br" + }, + "category": { + "selector": "td:nth-child(1) a[href*='cat=']", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["\\d+", 0] + } + ] + } + } + }, + "umt": { + "version": "hdfans@mc-v0.27.0", + "kind": "nexusphp", + "displayName": "HDFans", + "timezone": "+08:00", + "rateLimit": { + "minIntervalMs": 5000, + "burst": 1 + }, + "capabilities": { + "list": true, + "search": true, + "detail": true, + "download": true, + "rss": false + }, + "categoryNames": { + "401": "电影", + "402": "剧集", + "403": "纪录片", + "404": "其他", + "405": "其他", + "406": "音乐", + "407": "音乐", + "408": "音乐", + "409": "其他", + "410": "其他", + "416": "剧集", + "417": "动漫", + "418": "其他", + "419": "其他", + "421": "游戏", + "423": "其他" + }, + "loginSuccess": "a[href*='logout']", + "emptyMarkers": ["没有种子", "没有结果", "Nothing found"], + "pagination": "a[href*=\"page=\"]", + "detailPath": "details.php", + "downloadPath": "download.php", + "authSupported": ["cookie", "credential"], + "login": { + "path": "login.php", + "usernameField": "username", + "passwordField": "password", + "errorCss": "td.text" + }, + "profile": { + "path": "userdetails.php", + "uid_css": "a[class*='_Name']::attr(href)", + "username_css": "a[class*='_Name'] b", + "class_css": "a[class*='_Name']::attr(class)", + "uploaded_css": "font.color_uploaded::next_sibling_text", + "downloaded_css": "font.color_downloaded::next_sibling_text", + "ratio_css": "font.color_ratio::next_sibling_text", + "bonus_css": "font.color_bonus", + "join_date_css": "td.rowhead:contains('加入日期') + td::text", + "seeding_css": "img.arrowup::next_sibling_text", + "leeching_css": "img.arrowdown::next_sibling_text" + }, + "detail": { + "path": "details.php", + "title": "h1#top", + "rowLabel": "td.rowhead", + "rowValue": "td.rowfollow", + "labels": { + "subtitle": ["副标题", "Small Description"], + "basicInfo": ["基本信息", "Basic Info"], + "peers": ["同伴", "Peers"] + }, + "description": "#kdescr", + "headerScope": "h1#top", + "notFoundMarkers": ["没有该ID的种子", "No torrent with ID"], + "fields": { + "title": "h1#top", + "subtitle": "td.rowhead:contains('副标题') + td::text", + "size": "td.rowhead:contains('基本信息') + td.rowfollow > b::next_sibling_text", + "description": "#kdescr", + "download": "a[href*='download.php']::attr(href)", + "imdb": "a[href*='imdb.com']::attr(href)", + "douban": "a[href*='douban.com']::attr(href)", + "fileList": "td.rowfollow > ul > li::text" + } + } + } +} diff --git a/packages/adapters/src/sites/definitions/hdhome.json b/packages/adapters/src/sites/definitions/hdhome.json new file mode 100644 index 00000000..42ae14f9 --- /dev/null +++ b/packages/adapters/src/sites/definitions/hdhome.json @@ -0,0 +1,434 @@ +{ + "id": "hdhome", + "name": "高清家园", + "enabled": false, + "schema": "NexusPhp", + "provenance": { + "source": "movieclaw", + "ref": "v0.27.0", + "commit": "c79bb1b0", + "license": "MIT", + "file": "src/movieclaw_tracker/sites/configs/hdhome.yaml" + }, + "domain": "https://hdhome.org/", + "encoding": "UTF-8", + "public": false, + "proxy": false, + "language": "zh", + "result_num": 100, + "timeout": 30, + "search": { + "paths": [ + { + "path": "torrents.php", + "type": "all" + } + ], + "params": { + "search": "{keyword}" + } + }, + "batch": { + "delimiter": " ", + "space_replace": "_" + }, + "browse": { + "path": "torrents.php?page={page}", + "start": 0 + }, + "category": { + "movie": [ + { + "id": "411", + "desc": "电影" + }, + { + "id": "412", + "desc": "电影" + }, + { + "id": "413", + "desc": "电影" + }, + { + "id": "414", + "desc": "电影" + }, + { + "id": "415", + "desc": "电影" + }, + { + "id": "416", + "desc": "电影" + }, + { + "id": "450", + "desc": "电影" + }, + { + "id": "499", + "desc": "电影" + }, + { + "id": "505", + "desc": "电影" + }, + { + "id": "506", + "desc": "电影" + }, + { + "id": "518", + "desc": "电影" + } + ], + "tv": [ + { + "id": "425", + "desc": "剧集" + }, + { + "id": "426", + "desc": "剧集" + }, + { + "id": "427", + "desc": "剧集" + }, + { + "id": "428", + "desc": "剧集" + }, + { + "id": "429", + "desc": "剧集" + }, + { + "id": "430", + "desc": "剧集" + }, + { + "id": "431", + "desc": "剧集" + }, + { + "id": "432", + "desc": "剧集" + }, + { + "id": "433", + "desc": "剧集" + }, + { + "id": "434", + "desc": "剧集" + }, + { + "id": "435", + "desc": "剧集" + }, + { + "id": "436", + "desc": "剧集" + }, + { + "id": "437", + "desc": "剧集" + }, + { + "id": "438", + "desc": "剧集" + }, + { + "id": "442", + "desc": "剧集" + }, + { + "id": "443", + "desc": "剧集" + }, + { + "id": "452", + "desc": "剧集" + }, + { + "id": "453", + "desc": "剧集" + }, + { + "id": "502", + "desc": "剧集" + }, + { + "id": "504", + "desc": "剧集" + }, + { + "id": "511", + "desc": "剧集" + }, + { + "id": "523", + "desc": "剧集" + }, + { + "id": "526", + "desc": "剧集" + } + ], + "music": [ + { + "id": "439", + "desc": "音乐" + }, + { + "id": "440", + "desc": "音乐" + }, + { + "id": "441", + "desc": "音乐" + }, + { + "id": "503", + "desc": "音乐" + }, + { + "id": "423", + "desc": "音乐" + }, + { + "id": "424", + "desc": "音乐" + } + ] + }, + "torrents": { + "list": { + "selector": "table.torrents > tr:has(table.torrentname)" + }, + "fields": { + "title_default": { + "selector": "a[href*='details.php']" + }, + "title_optional": { + "selector": "a[href*='details.php'][title]", + "attribute": "title" + }, + "title": { + "text": "{% if fields['title_optional'] %}{{ fields['title_optional'] }}{% else %}{{ fields['title_default'] }}{% endif %}" + }, + "details": { + "selector": "a[href*='details.php']", + "attribute": "href" + }, + "download": { + "selector": "a[href*='download.php']", + "attribute": "href" + }, + "imdbid": { + "selector": "a[href*=\"imdb.com/title/tt\"]", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["tt\\d+", 0] + } + ] + }, + "date_elapsed": { + "selector": "td:nth-child(4)" + }, + "date_added": { + "selector": "td:nth-child(4) > span", + "attribute": "title" + }, + "date": { + "text": "{% if fields['date_elapsed'] or fields['date_added'] %}{{ fields['date_added'] if fields['date_added'] else fields['date_elapsed'] }}{% else %}now{% endif %}", + "filters": [ + { + "name": "dateparse", + "args": "%Y-%m-%d %H:%M:%S" + } + ] + }, + "size": { + "selector": "td:nth-child(5)" + }, + "seeders": { + "selector": "td:nth-child(6)" + }, + "leechers": { + "selector": "td:nth-child(7)" + }, + "grabs": { + "selector": "td:nth-child(8)" + }, + "downloadvolumefactor": { + "case": { + "img.pro_free": 0, + "img.pro_free2up": 0, + "img.pro_50pctdown": 0.5, + "img.pro_50pctdown2up": 0.5, + "img.pro_30pctdown": 0.3, + "*": 1 + } + }, + "uploadvolumefactor": { + "case": { + "img.pro_free2up": 2, + "img.pro_50pctdown2up": 2, + "img.pro_2up": 2, + "*": 1 + } + }, + "description": { + "selector": "td.embedded > span:not([class])" + }, + "category": { + "selector": "a[href^='?cat=']", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["\\d+", 0] + } + ] + }, + "hr": { + "selector": "img.hitandrun" + } + } + }, + "umt": { + "version": "hdhome@mc-v0.27.0", + "kind": "nexusphp", + "displayName": "高清家园", + "timezone": "+08:00", + "rateLimit": { + "minIntervalMs": 3000, + "burst": 1 + }, + "capabilities": { + "list": true, + "search": true, + "detail": true, + "download": true, + "rss": false + }, + "categoryNames": { + "409": "其他", + "411": "电影", + "412": "电影", + "413": "电影", + "414": "电影", + "415": "电影", + "416": "电影", + "417": "纪录片", + "418": "纪录片", + "419": "纪录片", + "420": "纪录片", + "421": "纪录片", + "422": "纪录片", + "423": "音乐", + "424": "音乐", + "425": "剧集", + "426": "剧集", + "427": "剧集", + "428": "剧集", + "429": "剧集", + "430": "剧集", + "431": "剧集", + "432": "剧集", + "433": "剧集", + "434": "剧集", + "435": "剧集", + "436": "剧集", + "437": "剧集", + "438": "剧集", + "439": "音乐", + "440": "音乐", + "441": "音乐", + "442": "剧集", + "443": "剧集", + "444": "动漫", + "445": "动漫", + "446": "动漫", + "447": "动漫", + "448": "动漫", + "449": "动漫", + "450": "电影", + "451": "纪录片", + "452": "剧集", + "453": "剧集", + "454": "动漫", + "499": "电影", + "500": "纪录片", + "501": "动漫", + "502": "剧集", + "503": "音乐", + "504": "剧集", + "505": "电影", + "506": "电影", + "507": "纪录片", + "508": "纪录片", + "509": "动漫", + "510": "动漫", + "511": "剧集", + "518": "电影", + "523": "剧集", + "526": "剧集", + "529": "纪录片", + "531": "动漫" + }, + "loginSuccess": "a[href*='logout']", + "emptyMarkers": ["没有种子", "没有结果", "Nothing found"], + "pagination": "a[href*=\"page=\"]", + "detailPath": "details.php", + "downloadPath": "download.php", + "login": { + "path": "login.php", + "usernameField": "username", + "passwordField": "password", + "errorCss": "td.text" + }, + "profile": { + "path": "userdetails.php", + "uid_css": "a[href*='userdetails.php?id=']::attr(href)", + "username_css": "a[href*='userdetails.php?id=']", + "class_css": "a[href*='userdetails.php?id=']::attr(class)", + "uploaded_css": "font.color_uploaded::next_sibling_text", + "downloaded_css": "font.color_downloaded::next_sibling_text", + "ratio_css": "font.color_ratio::next_sibling_text", + "bonus_css": "font.color_bonus::next_sibling_text", + "join_date_css": "td.rowhead:contains('加入日期') + td::text", + "seeding_css": "img[alt='Torrents seeding'] + *", + "leeching_css": "img[alt='Torrents leeching'] + *" + }, + "detail": { + "path": "details.php", + "title": "#top", + "rowLabel": "td.rowhead", + "rowValue": "td.rowfollow", + "labels": { + "subtitle": ["副标题", "Small Description"], + "basicInfo": ["基本信息", "Basic Info"], + "peers": ["同伴", "Peers"] + }, + "description": "#kdescr", + "headerScope": "#top", + "notFoundMarkers": ["没有该ID的种子", "No torrent with ID"], + "fields": { + "title": "#top::text", + "subtitle": "td.rowhead:contains('副标题') + td::text", + "size": "td.rowhead:contains('大小') + td::text", + "description": "#kdescr", + "download": "a[href*='download.php']::attr(href)", + "imdb": "a[href*='imdb.com']::attr(href)", + "douban": "a[href*='douban.com']::attr(href)", + "fileList": "td.rowfollow > ul > li::text" + } + } + } +} diff --git a/packages/adapters/src/sites/definitions/hdsky.json b/packages/adapters/src/sites/definitions/hdsky.json new file mode 100644 index 00000000..e3e4e00a --- /dev/null +++ b/packages/adapters/src/sites/definitions/hdsky.json @@ -0,0 +1,276 @@ +{ + "id": "hdsky", + "name": "HDSky", + "enabled": false, + "schema": "NexusPhp", + "provenance": { + "source": "movieclaw", + "ref": "v0.27.0", + "commit": "c79bb1b0", + "license": "MIT", + "file": "src/movieclaw_tracker/sites/configs/hdsky.yaml" + }, + "domain": "https://hdsky.me/", + "encoding": "UTF-8", + "public": false, + "proxy": false, + "language": "zh", + "result_num": 100, + "timeout": 30, + "search": { + "paths": [ + { + "path": "torrents.php", + "type": "all" + } + ], + "params": { + "search": "{keyword}" + } + }, + "batch": { + "delimiter": " ", + "space_replace": "_" + }, + "browse": { + "path": "torrents.php?page={page}", + "start": 0 + }, + "category": { + "movie": [ + { + "id": "401", + "desc": "电影" + }, + { + "id": "410", + "desc": "电影" + } + ], + "tv": [ + { + "id": "402", + "desc": "剧集" + }, + { + "id": "403", + "desc": "剧集" + }, + { + "id": "407", + "desc": "剧集" + }, + { + "id": "411", + "desc": "剧集" + }, + { + "id": "412", + "desc": "剧集" + }, + { + "id": "413", + "desc": "剧集" + }, + { + "id": "414", + "desc": "剧集" + }, + { + "id": "415", + "desc": "剧集" + }, + { + "id": "416", + "desc": "剧集" + } + ], + "music": [ + { + "id": "406", + "desc": "音乐" + }, + { + "id": "408", + "desc": "音乐" + } + ] + }, + "torrents": { + "list": { + "selector": "table.torrents.progresstable tr" + }, + "fields": { + "title_default": { + "selector": "table.torrentname a[href*='details.php'] b span" + }, + "title_optional": { + "selector": "table.torrentname a[href*='details.php'] b span[title]", + "attribute": "title" + }, + "title": { + "text": "{% if fields['title_optional'] %}{{ fields['title_optional'] }}{% else %}{{ fields['title_default'] }}{% endif %}" + }, + "details": { + "selector": "table.torrentname a[href*='details.php']", + "attribute": "href" + }, + "download": { + "selector": "form[action*='download.php']", + "attribute": "action" + }, + "imdbid": { + "selector": "a[href*=\"imdb.com/title/tt\"]", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["tt\\d+", 0] + } + ] + }, + "date_elapsed": { + "selector": "td:nth-child(4)" + }, + "date_added": { + "selector": "td:nth-child(4) span", + "attribute": "title" + }, + "date": { + "text": "{% if fields['date_elapsed'] or fields['date_added'] %}{{ fields['date_added'] if fields['date_added'] else fields['date_elapsed'] }}{% else %}now{% endif %}", + "filters": [ + { + "name": "dateparse", + "args": "%Y-%m-%d %H:%M:%S" + } + ] + }, + "size": { + "selector": "td:nth-child(5)" + }, + "seeders": { + "selector": "a[href*='dllist=1#seeders']" + }, + "leechers": { + "selector": "a[href*='dllist=1#leechers']" + }, + "grabs": { + "selector": "a[href*='viewsnatches.php'] b, td:nth-child(8)" + }, + "downloadvolumefactor": { + "case": { + "img.pro_free": 0, + "img.pro_free2up": 0, + "img.pro_50pctdown": 0.5, + "img.pro_50pctdown2up": 0.5, + "img.pro_30pctdown": 0.3, + "*": 1 + } + }, + "uploadvolumefactor": { + "case": { + "img.pro_2up": 2, + "img.pro_free2up": 2, + "img.pro_50pctdown2up": 2, + "*": 1 + } + }, + "description": { + "selector": "table.torrentname br + span" + }, + "category": { + "selector": "td:first-child a[href*='cat=']", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["\\d+", 0] + } + ] + } + } + }, + "umt": { + "version": "hdsky@mc-v0.27.0", + "kind": "nexusphp", + "displayName": "HDSky", + "timezone": "+08:00", + "rateLimit": { + "minIntervalMs": 3000, + "burst": 1 + }, + "capabilities": { + "list": true, + "search": true, + "detail": true, + "download": true, + "rss": false + }, + "categoryNames": { + "401": "电影", + "402": "剧集", + "403": "剧集", + "404": "纪录片", + "405": "动漫", + "406": "音乐", + "407": "剧集", + "408": "音乐", + "409": "其他", + "410": "电影", + "411": "剧集", + "412": "剧集", + "413": "剧集", + "414": "剧集", + "415": "剧集", + "416": "剧集" + }, + "loginSuccess": "a[href='logout.php']", + "emptyMarkers": ["没有种子", "没有结果", "Nothing found"], + "pagination": "a[href*=\"page=\"]", + "detailPath": "details.php", + "downloadPath": "download.php", + "login": { + "path": "login.php", + "usernameField": "username", + "passwordField": "password", + "errorCss": "td.text" + }, + "profile": { + "path": "userdetails.php", + "uid_css": "a[class*='_Name']::attr(href)", + "username_css": "a[class*='_Name'] b", + "class_css": "a[class*='_Name']::attr(class)", + "uploaded_css": "font.color_uploaded::next_sibling_text", + "downloaded_css": "font.color_downloaded::next_sibling_text", + "ratio_css": "font.color_ratio::next_sibling_text", + "bonus_css": "font.color_bonus", + "join_date_css": "td.rowhead:contains('加入日期') + td::text", + "seeding_css": "img.arrowup::next_sibling_text", + "leeching_css": "img.arrowdown::next_sibling_text" + }, + "detail": { + "path": "details.php", + "title": "h1#top", + "rowLabel": "td.rowhead", + "rowValue": "td.rowfollow", + "labels": { + "subtitle": ["副标题", "Small Description"], + "basicInfo": ["基本信息", "Basic Info"], + "peers": ["同伴", "Peers"] + }, + "description": "#kdescr", + "headerScope": "h1#top", + "notFoundMarkers": ["没有该ID的种子", "No torrent with ID"], + "fields": { + "title": "h1#top", + "subtitle": "td.rowhead:contains('副标题') + td.rowfollow", + "size": "td.rowhead:contains('基本信息') + td.rowfollow", + "description": "#kdescr", + "download": "form[action*='download.php']::attr(action)", + "imdb": "a[href*='imdb.com']::attr(href)", + "douban": "a[href*='douban.com']::attr(href)", + "fileList": "td.rowfollow > ul > li::text" + } + } + } +} diff --git a/packages/adapters/src/sites/definitions/hdtime.json b/packages/adapters/src/sites/definitions/hdtime.json new file mode 100644 index 00000000..1234afda --- /dev/null +++ b/packages/adapters/src/sites/definitions/hdtime.json @@ -0,0 +1,239 @@ +{ + "id": "hdtime", + "name": "HDTime", + "enabled": false, + "schema": "NexusPhp", + "provenance": { + "source": "movieclaw", + "ref": "v0.27.0", + "commit": "c79bb1b0", + "license": "MIT", + "file": "src/movieclaw_tracker/sites/configs/hdtime.yaml" + }, + "domain": "https://hdtime.org/", + "encoding": "UTF-8", + "public": false, + "proxy": false, + "language": "zh", + "result_num": 100, + "timeout": 30, + "search": { + "paths": [ + { + "path": "torrents.php", + "type": "all" + } + ], + "params": { + "search": "{keyword}" + } + }, + "batch": { + "delimiter": " ", + "space_replace": "_" + }, + "browse": { + "path": "torrents.php?page={page}", + "start": 0 + }, + "category": { + "movie": [ + { + "id": "401", + "desc": "电影" + }, + { + "id": "424", + "desc": "电影" + } + ], + "tv": [ + { + "id": "402", + "desc": "剧集" + }, + { + "id": "403", + "desc": "剧集" + } + ], + "music": [ + { + "id": "406", + "desc": "音乐" + }, + { + "id": "408", + "desc": "音乐" + } + ] + }, + "torrents": { + "list": { + "selector": "table.torrents > tr, table.torrents > tbody > tr" + }, + "fields": { + "title_default": { + "selector": "table.torrentname a b" + }, + "title_optional": { + "selector": "table.torrentname a b[title]", + "attribute": "title" + }, + "title": { + "text": "{% if fields['title_optional'] %}{{ fields['title_optional'] }}{% else %}{{ fields['title_default'] }}{% endif %}" + }, + "details": { + "selector": "a[href*='details.php']", + "attribute": "href" + }, + "download": { + "selector": "a[href*='download.php']", + "attribute": "href" + }, + "imdbid": { + "selector": "a[href*=\"imdb.com/title/tt\"]", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["tt\\d+", 0] + } + ] + }, + "date_elapsed": { + "selector": "td:nth-child(4)" + }, + "date_added": { + "selector": "td:nth-child(4) > span", + "attribute": "title" + }, + "date": { + "text": "{% if fields['date_elapsed'] or fields['date_added'] %}{{ fields['date_added'] if fields['date_added'] else fields['date_elapsed'] }}{% else %}now{% endif %}", + "filters": [ + { + "name": "dateparse", + "args": "%Y-%m-%d %H:%M:%S" + } + ] + }, + "size": { + "selector": "td:nth-child(5)" + }, + "seeders": { + "selector": "td:nth-child(6)" + }, + "leechers": { + "selector": "td:nth-child(7)" + }, + "grabs": { + "selector": "td:nth-child(8)" + }, + "downloadvolumefactor": { + "case": { + "img.pro_free": 0, + "*": 1 + } + }, + "uploadvolumefactor": { + "case": { + "*": 1 + } + }, + "description": { + "selector": "td.embedded br" + }, + "category": { + "selector": "td:nth-child(1) a[href*='cat=']", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["\\d+", 0] + } + ] + } + } + }, + "umt": { + "version": "hdtime@mc-v0.27.0", + "kind": "nexusphp", + "displayName": "HDTime", + "timezone": "+08:00", + "rateLimit": { + "minIntervalMs": 5000, + "burst": 1 + }, + "capabilities": { + "list": true, + "search": true, + "detail": true, + "download": true, + "rss": false + }, + "categoryNames": { + "401": "电影", + "402": "剧集", + "403": "剧集", + "404": "纪录片", + "405": "动漫", + "406": "音乐", + "407": "其他", + "408": "音乐", + "409": "其他", + "410": "游戏", + "411": "其他", + "414": "其他", + "424": "电影" + }, + "loginSuccess": "a[href*='logout']", + "emptyMarkers": ["没有种子", "没有结果", "Nothing found"], + "pagination": "a[href*=\"page=\"]", + "detailPath": "details.php", + "downloadPath": "download.php", + "authSupported": ["cookie", "credential"], + "login": { + "path": "login.php", + "usernameField": "username", + "passwordField": "password", + "errorCss": "td.text" + }, + "profile": { + "path": "userdetails.php", + "uid_css": "a[class*='_Name']::attr(href)", + "username_css": "a[class*='_Name'] b", + "class_css": "a[class*='_Name']::attr(class)", + "uploaded_css": "font.color_uploaded::next_sibling_text", + "downloaded_css": "font.color_downloaded::next_sibling_text", + "ratio_css": "font.color_ratio::next_sibling_text", + "bonus_css": "font.color_bonus", + "join_date_css": "td.rowhead:contains('加入日期') + td::text", + "seeding_css": "img.arrowup::next_sibling_text", + "leeching_css": "img.arrowdown::next_sibling_text" + }, + "detail": { + "path": "details.php", + "title": "h1#top", + "rowLabel": "td.rowhead", + "rowValue": "td.rowfollow", + "labels": { + "subtitle": ["副标题", "Small Description"], + "basicInfo": ["基本信息", "Basic Info"], + "peers": ["同伴", "Peers"] + }, + "description": "#kdescr", + "headerScope": "h1#top", + "notFoundMarkers": ["没有该ID的种子", "No torrent with ID"], + "fields": { + "title": "h1#top", + "subtitle": "td.rowhead:contains('副标题') + td::text", + "size": "td.rowhead:contains('基本信息') + td.rowfollow > b::next_sibling_text", + "description": "#kdescr", + "download": "a[href*='download.php']::attr(href)", + "imdb": "a[href*='imdb.com']::attr(href)", + "douban": "a[href*='douban.com']::attr(href)", + "fileList": "td.rowfollow > ul > li::text" + } + } + } +} diff --git a/packages/adapters/src/sites/definitions/hdvideo.json b/packages/adapters/src/sites/definitions/hdvideo.json new file mode 100644 index 00000000..7863143f --- /dev/null +++ b/packages/adapters/src/sites/definitions/hdvideo.json @@ -0,0 +1,240 @@ +{ + "id": "hdvideo", + "name": "HDVIDEO", + "enabled": false, + "schema": "NexusPhp", + "provenance": { + "source": "movieclaw", + "ref": "v0.27.0", + "commit": "c79bb1b0", + "license": "MIT", + "file": "src/movieclaw_tracker/sites/configs/hdvideo.yaml" + }, + "domain": "https://hdvideo.top/", + "encoding": "UTF-8", + "public": false, + "proxy": false, + "language": "zh", + "result_num": 100, + "timeout": 30, + "search": { + "paths": [ + { + "path": "torrents.php", + "type": "all" + } + ], + "params": { + "search": "{keyword}" + } + }, + "batch": { + "delimiter": " ", + "space_replace": "_" + }, + "browse": { + "path": "torrents.php?page={page}", + "start": 0 + }, + "category": { + "movie": [ + { + "id": "401", + "desc": "电影" + } + ], + "tv": [ + { + "id": "402", + "desc": "剧集" + }, + { + "id": "403", + "desc": "剧集" + }, + { + "id": "407", + "desc": "剧集" + } + ], + "music": [ + { + "id": "406", + "desc": "音乐" + }, + { + "id": "408", + "desc": "音乐" + } + ] + }, + "torrents": { + "list": { + "selector": "table.torrents tr" + }, + "fields": { + "title_default": { + "selector": "span.tt-name a b" + }, + "title_optional": { + "selector": "span.tt-name a b[title]", + "attribute": "title" + }, + "title": { + "text": "{% if fields['title_optional'] %}{{ fields['title_optional'] }}{% else %}{{ fields['title_default'] }}{% endif %}" + }, + "details": { + "selector": "span.tt-name a", + "attribute": "href" + }, + "download": { + "selector": "a[href*='download.php']", + "attribute": "href" + }, + "imdbid": { + "selector": "a[href*=\"imdb.com/title/tt\"]", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["tt\\d+", 0] + } + ] + }, + "date_elapsed": { + "selector": "td:nth-child(5)" + }, + "date_added": { + "selector": "td:nth-child(5) span", + "attribute": "title" + }, + "date": { + "text": "{% if fields['date_elapsed'] or fields['date_added'] %}{{ fields['date_added'] if fields['date_added'] else fields['date_elapsed'] }}{% else %}now{% endif %}", + "filters": [ + { + "name": "dateparse", + "args": "%Y-%m-%d %H:%M:%S" + } + ] + }, + "size": { + "selector": "td:nth-child(6)" + }, + "seeders": { + "selector": "a[href*='dllist=1#seeders']" + }, + "leechers": { + "selector": "a[href*='dllist=1#leechers'], td:nth-child(8)" + }, + "grabs": { + "selector": "a[href*='viewsnatches.php'] b" + }, + "downloadvolumefactor": { + "case": { + "img.pro_free": 0, + "img.pro_free2up": 0, + "img.pro_50pctdown": 0.5, + "img.pro_50pctdown2up": 0.5, + "img.pro_30pctdown": 0.3, + "*": 1 + } + }, + "uploadvolumefactor": { + "case": { + "img.pro_2up": 2, + "img.pro_free2up": 2, + "img.pro_50pctdown2up": 2, + "*": 1 + } + }, + "description": { + "selector": "span.tt-sub" + }, + "category": { + "selector": "td:nth-child(2) a[href*='cat=']", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["\\d+", 0] + } + ] + } + } + }, + "umt": { + "version": "hdvideo@mc-v0.27.0", + "kind": "nexusphp", + "displayName": "HDVIDEO", + "timezone": "+08:00", + "rateLimit": { + "minIntervalMs": 3000, + "burst": 1 + }, + "capabilities": { + "list": true, + "search": true, + "detail": true, + "download": true, + "rss": false + }, + "categoryNames": { + "401": "电影", + "402": "剧集", + "403": "剧集", + "404": "纪录片", + "405": "动漫", + "406": "音乐", + "407": "剧集", + "408": "音乐" + }, + "loginSuccess": "a[href='logout.php']", + "emptyMarkers": ["没有种子", "没有结果", "Nothing found"], + "pagination": "a[href*=\"page=\"]", + "detailPath": "details.php", + "downloadPath": "download.php", + "login": { + "path": "login.php", + "usernameField": "username", + "passwordField": "password", + "errorCss": "td.text" + }, + "profile": { + "path": "userdetails.php", + "uid_css": "a[class*='_Name']::attr(href)", + "username_css": "a[class*='_Name'] b", + "class_css": "a[class*='_Name']::attr(class)", + "uploaded_css": "font.color_uploaded::next_sibling_text", + "downloaded_css": "font.color_downloaded::next_sibling_text", + "ratio_css": "font.color_ratio::next_sibling_text", + "bonus_css": "font.color_bonus", + "join_date_css": "td.rowhead:contains('加入日期') + td::text", + "seeding_css": "img.arrowup::next_sibling_text", + "leeching_css": "img.arrowdown::next_sibling_text" + }, + "detail": { + "path": "details.php", + "title": "h1#top", + "rowLabel": "td.rowhead", + "rowValue": "td.rowfollow", + "labels": { + "subtitle": ["副标题", "Small Description"], + "basicInfo": ["基本信息", "Basic Info"], + "peers": ["同伴", "Peers"] + }, + "description": "#kdescr", + "headerScope": "h1#top", + "notFoundMarkers": ["没有该ID的种子", "No torrent with ID"], + "fields": { + "title": "h1#top", + "subtitle": "td.rowhead:contains('副标题') + td.rowfollow", + "size": "td.rowhead:contains('基本信息') + td.rowfollow", + "description": "#kdescr", + "download": "a[href*='download.php']::attr(href)", + "imdb": "a[href*='imdb.com']::attr(href)", + "douban": "a[href*='douban.com']::attr(href)", + "fileList": "td.rowfollow > ul > li::text" + } + } + } +} diff --git a/packages/adapters/src/sites/definitions/hhanclub.json b/packages/adapters/src/sites/definitions/hhanclub.json new file mode 100644 index 00000000..5567acaa --- /dev/null +++ b/packages/adapters/src/sites/definitions/hhanclub.json @@ -0,0 +1,254 @@ +{ + "id": "hhanclub", + "name": "憨憨", + "enabled": true, + "schema": "NexusHhanclub", + "provenance": { + "source": "movieclaw", + "ref": "v0.27.0", + "commit": "c79bb1b0", + "license": "MIT", + "file": "src/movieclaw_tracker/sites/configs/hhanclub.yaml", + "overrides": "scripts/mp-sync/movieclaw-overrides/hhanclub.json" + }, + "domain": "https://hhanclub.net/", + "encoding": "UTF-8", + "public": false, + "proxy": false, + "language": "zh", + "result_num": 100, + "timeout": 30, + "search": { + "paths": [ + { + "path": "torrents.php", + "type": "all" + } + ], + "params": { + "search": "{keyword}" + } + }, + "batch": { + "delimiter": " ", + "space_replace": "_" + }, + "browse": { + "path": "torrents.php?page={page}", + "start": 0 + }, + "category": { + "movie": [ + { + "id": "401", + "desc": "电影" + } + ], + "tv": [ + { + "id": "402", + "desc": "电视剧" + }, + { + "id": "403", + "desc": "综艺" + }, + { + "id": "407", + "desc": "体育" + }, + { + "id": "412", + "desc": "短剧" + } + ], + "music": [ + { + "id": "410", + "desc": "音乐" + }, + { + "id": "411", + "desc": "MV" + } + ] + }, + "torrents": { + "list": { + "selector": "div.torrent-table-sub-info" + }, + "fields": { + "title_default": { + "selector": "div.torrent-title a.torrent-info-text-name" + }, + "title_optional": { + "selector": "div.torrent-title a.torrent-info-text-name[title]", + "attribute": "title" + }, + "title": { + "text": "{% if fields['title_optional'] %}{{ fields['title_optional'] }}{% else %}{{ fields['title_default'] }}{% endif %}" + }, + "details": { + "selector": "div.torrent-title a[href*='details.php']", + "attribute": "href" + }, + "download": { + "selector": "div.torrent-manage a[href*='download.php']", + "attribute": "href" + }, + "imdbid": { + "selector": "a[href*=\"imdb.com/title/tt\"]", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["tt\\d+", 0] + } + ] + }, + "date_elapsed": { + "selector": "div.torrent-info-text-added" + }, + "date_added": { + "selector": "div.torrent-info-text-added > span", + "attribute": "title" + }, + "date": { + "text": "{% if fields['date_elapsed'] or fields['date_added'] %}{{ fields['date_added'] if fields['date_added'] else fields['date_elapsed'] }}{% else %}now{% endif %}", + "filters": [ + { + "name": "dateparse", + "args": "%Y-%m-%d %H:%M:%S" + } + ] + }, + "size": { + "selector": "div.torrent-info-text-size" + }, + "seeders": { + "selector": "div.torrent-info-text-seeders" + }, + "leechers": { + "selector": "div.torrent-info-text-leechers" + }, + "grabs": { + "selector": "div.torrent-info-text-finished" + }, + "downloadvolumefactor": { + "case": { + "span.promotion-tag-free": 0, + "span.promotion-tag-free2up": 0, + "span.promotion-tag-2xfree": 0, + "span.promotion-tag-50": 0.5, + "span.promotion-tag-50pctdown": 0.5, + "span.promotion-tag-50pctdown2up": 0.5, + "span.promotion-tag-30": 0.3, + "span.promotion-tag-30pctdown": 0.3, + "*": 1 + } + }, + "uploadvolumefactor": { + "case": { + "span.promotion-tag-free2up": 2, + "span.promotion-tag-50pctdown2up": 2, + "span.promotion-tag-2up": 2, + "span.promotion-tag-2xfree": 2, + "*": 1 + } + }, + "description": { + "selector": "div.torrent-info-text-small_name" + }, + "category": { + "selector": "div.torrent-cat a", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["\\d+", 0] + } + ] + }, + "hr": { + "selector": "div.torrent-info-hr" + } + } + }, + "umt": { + "version": "hhanclub@mc-v0.27.0", + "kind": "nexusphp", + "displayName": "憨憨", + "timezone": "+08:00", + "rateLimit": { + "minIntervalMs": 3000, + "burst": 1 + }, + "capabilities": { + "list": true, + "search": true, + "detail": true, + "download": true, + "rss": false + }, + "categoryNames": { + "401": "电影", + "402": "电视剧", + "403": "综艺", + "404": "纪录片", + "405": "动漫", + "407": "体育", + "409": "其他", + "410": "音乐", + "411": "MV", + "412": "短剧" + }, + "loginSuccess": "a[href='logout.php']", + "emptyMarkers": ["没有种子", "没有结果", "Nothing found"], + "pagination": "a[href*=\"page=\"]", + "detailPath": "details.php", + "downloadPath": "download.php", + "login": { + "path": "login.php", + "usernameField": "username", + "passwordField": "password", + "errorCss": "td.text" + }, + "profile": { + "path": "userdetails.php", + "uid_css": "a[class*='_Name']::attr(href)", + "username_css": "a[class*='_Name']", + "class_css": "a[class*='_Name']::attr(class)", + "uploaded_css": "img[alt='上传']::next_sibling_text", + "downloaded_css": "img[alt='下载']::next_sibling_text", + "ratio_css": "img[alt='分享率'] + div", + "bonus_css": "img[alt='憨豆'] + a div", + "join_date_css": "td.rowhead:contains('加入日期') + td::text", + "seeding_css": "img[alt='做种数']::next_sibling_text", + "leeching_css": "img[alt='下载数']::next_sibling_text" + }, + "detail": { + "path": "details.php", + "title": "div.font-bold.leading-6:contains('标题'):not(:contains('副标题')) + div.font-bold.leading-6", + "rowLabel": "td.rowhead", + "rowValue": "td.rowfollow", + "labels": { + "subtitle": ["副标题", "Small Description"], + "basicInfo": ["基本信息", "Basic Info"], + "peers": ["同伴", "Peers"] + }, + "description": "#kdescr", + "headerScope": "div.font-bold.leading-6:contains('标题'):not(:contains('副标题')) + div.font-bold.leading-6", + "notFoundMarkers": ["没有该ID的种子", "No torrent with ID"], + "fields": { + "title": "div.font-bold.leading-6:contains('标题'):not(:contains('副标题')) + div.font-bold.leading-6", + "subtitle": "div.font-bold.leading-6:contains('副标题') + div.font-bold.leading-6", + "size": "span.font-bold:contains('大小') + span", + "description": "#kdescr", + "download": "a[href*='download.php']::attr(href)", + "imdb": "a[href*='imdb.com']::attr(href)", + "douban": "a[href*='douban.com']::attr(href)", + "fileList": "td.rowfollow > ul > li::text" + } + } + } +} diff --git a/packages/adapters/src/sites/definitions/index.ts b/packages/adapters/src/sites/definitions/index.ts index 4c629331..72d4c2e0 100644 --- a/packages/adapters/src/sites/definitions/index.ts +++ b/packages/adapters/src/sites/definitions/index.ts @@ -1,16 +1,48 @@ -import type { IndexerDefinition } from "../spider/schema"; -import audiencesJson from "./audiences.json"; +/** + * 站点定义加载与校验。 + * - `*.json`:由 scripts/mp-sync/movieclaw-import.mjs 从 MovieClaw v0.27.0(MIT,见 NOTICE-MovieClaw.md)转换, + * 手工覆盖层在 scripts/mp-sync/movieclaw-overrides/;不要直接手改 JSON(重跑脚本会覆盖)。 + * - 加载时一律经 zod schema(./schema.ts)校验,失败即抛错。 + */ +import type { MteamDefinition } from "../mteam/definition.ts"; +import type { IndexerDefinition } from "../spider/schema.ts"; +import type { SpecializedSiteDefinition } from "../spiders/types.ts"; +import { MOVIECLAW_DEFINITION_JSON } from "./manifest.generated.ts"; +import { parseIndexerDefinition, parseMteamDefinition, parseSpecializedSiteDefinition } from "./schema.ts"; -/** 站点定义最小校验(随产品发布的定义文件;加站只需新增 JSON 并在此登记)。 */ -export function loadIndexerDefinition(json: unknown): IndexerDefinition { - const d = json as Partial; - if (!d || typeof d !== "object") throw new Error("站点定义必须是对象"); - for (const k of ["id", "name", "domain"] as const) { - if (typeof d[k] !== "string" || !d[k]) throw new Error(`站点定义缺少 ${k}`); - } - if (!d.torrents?.list?.selector || !d.torrents.fields) throw new Error(`站点定义 ${d.id} 缺少 torrents.list/fields`); - if (!d.umt?.version || !d.umt.timezone || !d.umt.rateLimit) throw new Error(`站点定义 ${d.id} 缺少 umt 扩展`); - return d as IndexerDefinition; +export type SiteDefinition = IndexerDefinition | MteamDefinition | SpecializedSiteDefinition; + +/** 按 `umt.kind` 分派校验 */ +export function loadSiteDefinition(json: unknown): SiteDefinition { + const kind = (json as { umt?: { kind?: unknown } } | null)?.umt?.kind; + if (kind === "mteam") return parseMteamDefinition(json); + if (kind === "nexusphp") return parseIndexerDefinition(json); + if (kind === "api") return parseSpecializedSiteDefinition(json); + throw new Error(`站点定义 ${(json as { id?: unknown })?.id ?? ""} 的 umt.kind 未知:${String(kind)}`); } -export const audiencesDefinition: IndexerDefinition = loadIndexerDefinition(audiencesJson); +/** 兼容 W1-3b 导出名:NexusPHP 定义校验 */ +export function loadIndexerDefinition(json: unknown): IndexerDefinition { + return parseIndexerDefinition(json); +} + +/** 随产品发布的全部定义(已校验),键为站点 id */ +export const BUILTIN_SITE_DEFINITIONS: Readonly> = Object.freeze( + Object.fromEntries(Object.entries(MOVIECLAW_DEFINITION_JSON).map(([id, json]) => [id, loadSiteDefinition(json)])), +); + +export const audiencesDefinition = BUILTIN_SITE_DEFINITIONS.audiences as IndexerDefinition; +export const hhanclubDefinition = BUILTIN_SITE_DEFINITIONS.hhanclub as IndexerDefinition; +export const chdbitsDefinition = BUILTIN_SITE_DEFINITIONS.chdbits as IndexerDefinition; + +export { + DefinitionProvenanceSchema, + IndexerDefinitionSchema, + MteamDefinitionSchema, + parseIndexerDefinition, + parseMteamDefinition, + parseSpecializedSiteDefinition, + SpecializedSiteDefinitionSchema, + SpiderFieldSelectorSchema, + SpiderFilterSchema, +} from "./schema.ts"; diff --git a/packages/adapters/src/sites/definitions/keepfrds.json b/packages/adapters/src/sites/definitions/keepfrds.json new file mode 100644 index 00000000..9e588edb --- /dev/null +++ b/packages/adapters/src/sites/definitions/keepfrds.json @@ -0,0 +1,261 @@ +{ + "id": "keepfrds", + "name": "朋友", + "enabled": false, + "schema": "NexusPhp", + "provenance": { + "source": "movieclaw", + "ref": "v0.27.0", + "commit": "c79bb1b0", + "license": "MIT", + "file": "src/movieclaw_tracker/sites/configs/keepfrds.yaml" + }, + "domain": "https://pt.keepfrds.com/", + "encoding": "UTF-8", + "public": false, + "proxy": false, + "language": "zh", + "result_num": 100, + "timeout": 30, + "search": { + "paths": [ + { + "path": "torrents.php", + "type": "all" + } + ], + "params": { + "search": "{keyword}" + } + }, + "batch": { + "delimiter": " ", + "space_replace": "_" + }, + "browse": { + "path": "torrents.php?page={page}", + "start": 0 + }, + "category": { + "movie": [ + { + "id": "401", + "desc": "电影" + } + ], + "tv": [ + { + "id": "402", + "desc": "剧集" + }, + { + "id": "403", + "desc": "剧集" + }, + { + "id": "407", + "desc": "剧集" + } + ], + "music": [ + { + "id": "406", + "desc": "音乐" + }, + { + "id": "408", + "desc": "音乐" + }, + { + "id": "420", + "desc": "音乐" + }, + { + "id": "421", + "desc": "音乐" + } + ] + }, + "torrents": { + "list": { + "selector": "table.torrents > tr:has(table.torrentname)" + }, + "fields": { + "title_default": { + "selector": "a[href*='details.php']" + }, + "title_optional": { + "selector": "a[href*='details.php'][title]", + "attribute": "title" + }, + "title": { + "text": "{% if fields['title_optional'] %}{{ fields['title_optional'] }}{% else %}{{ fields['title_default'] }}{% endif %}" + }, + "details": { + "selector": "a[href*='details.php']", + "attribute": "href" + }, + "download": { + "selector": "a[href*='download.php']", + "attribute": "href" + }, + "imdbid": { + "selector": "a[href*=\"imdb.com/title/tt\"]", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["tt\\d+", 0] + } + ] + }, + "date_elapsed": { + "selector": "td:nth-child(4)" + }, + "date_added": { + "selector": "td:nth-child(4) > span", + "attribute": "title" + }, + "date": { + "text": "{% if fields['date_elapsed'] or fields['date_added'] %}{{ fields['date_added'] if fields['date_added'] else fields['date_elapsed'] }}{% else %}now{% endif %}", + "filters": [ + { + "name": "dateparse", + "args": "%Y-%m-%d %H:%M:%S" + } + ] + }, + "size": { + "selector": "td:nth-child(5)" + }, + "seeders": { + "selector": "td:nth-child(6)" + }, + "leechers": { + "selector": "td:nth-child(7)" + }, + "grabs": { + "selector": "td:nth-child(8)" + }, + "downloadvolumefactor": { + "case": { + "img.pro_free": 0, + "img.pro_free2up": 0, + "img.pro_nl": 0, + "img.pro_50pctdown": 0.5, + "img.pro_50pctdown2up": 0.5, + "img.pro_30pctdown": 0.3, + "*": 1 + } + }, + "uploadvolumefactor": { + "case": { + "img.pro_nl": 0, + "img.pro_free2up": 2, + "img.pro_50pctdown2up": 2, + "img.pro_2up": 2, + "*": 1 + } + }, + "description": { + "selector": "td.embedded > br" + }, + "category": { + "selector": "a[href^='?cat=']", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["\\d+", 0] + } + ] + } + } + }, + "umt": { + "version": "keepfrds@mc-v0.27.0", + "kind": "nexusphp", + "displayName": "朋友", + "timezone": "+08:00", + "rateLimit": { + "minIntervalMs": 3000, + "burst": 1 + }, + "capabilities": { + "list": true, + "search": true, + "detail": true, + "download": true, + "rss": false + }, + "categoryNames": { + "300": "其他", + "401": "电影", + "402": "剧集", + "403": "剧集", + "404": "纪录片", + "405": "动漫", + "406": "音乐", + "407": "剧集", + "408": "音乐", + "409": "其他", + "414": "其他", + "415": "其他", + "419": "成人", + "420": "音乐", + "421": "音乐", + "422": "其他", + "423": "游戏", + "424": "其他", + "425": "其他" + }, + "loginSuccess": "a[href*='logout']", + "emptyMarkers": ["没有种子", "没有结果", "Nothing found"], + "pagination": "a[href*=\"page=\"]", + "detailPath": "details.php", + "downloadPath": "download.php", + "login": { + "path": "login.php", + "usernameField": "username", + "passwordField": "password", + "errorCss": "td.text" + }, + "profile": { + "path": "userdetails.php", + "uid_css": "a[href*='userdetails.php?id=']::attr(href)", + "username_css": "a[href*='userdetails.php?id=']", + "class_css": "a[href*='userdetails.php?id=']::attr(class)", + "uploaded_css": "span.uploaded", + "downloaded_css": "span.downloaded", + "ratio_css": "td.rowhead:contains('分享率') + td::text", + "bonus_css": "td.rowhead:contains('魔力') + td::text", + "join_date_css": "td.rowhead:contains('加入日期') + td::text", + "seeding_css": "td.rowhead:contains('当前做种') + td::text", + "leeching_css": "td.rowhead:contains('当前下载') + td::text" + }, + "detail": { + "path": "details.php", + "title": "#top", + "rowLabel": "td.rowhead", + "rowValue": "td.rowfollow", + "labels": { + "subtitle": ["副标题", "Small Description"], + "basicInfo": ["基本信息", "Basic Info"], + "peers": ["同伴", "Peers"] + }, + "description": "#kdescr", + "headerScope": "#top", + "notFoundMarkers": ["没有该ID的种子", "No torrent with ID"], + "fields": { + "title": "#top::text", + "subtitle": "td.rowhead:contains('副标题') + td::text", + "size": "td.rowhead:contains('大小') + td::text", + "description": "#kdescr", + "download": "a[href*='download.php']::attr(href)", + "imdb": "a[href*='imdb.com']::attr(href)", + "douban": "a[href*='douban.com']::attr(href)", + "fileList": "td.rowfollow > ul > li::text" + } + } + } +} diff --git a/packages/adapters/src/sites/definitions/manifest.generated.ts b/packages/adapters/src/sites/definitions/manifest.generated.ts new file mode 100644 index 00000000..54552197 --- /dev/null +++ b/packages/adapters/src/sites/definitions/manifest.generated.ts @@ -0,0 +1,51 @@ +// 由 scripts/mp-sync/movieclaw-import.mjs 生成,请勿手改(MovieClaw v0.27.0 c79bb1b0,MIT) +import agsvptJson from "./agsvpt.json" with { type: "json" }; +import audiencesJson from "./audiences.json" with { type: "json" }; +import chdbitsJson from "./chdbits.json" with { type: "json" }; +import hdareaJson from "./hdarea.json" with { type: "json" }; +import hddolbyJson from "./hddolby.json" with { type: "json" }; +import hdfansJson from "./hdfans.json" with { type: "json" }; +import hdhomeJson from "./hdhome.json" with { type: "json" }; +import hdskyJson from "./hdsky.json" with { type: "json" }; +import hdtimeJson from "./hdtime.json" with { type: "json" }; +import hdvideoJson from "./hdvideo.json" with { type: "json" }; +import hhanclubJson from "./hhanclub.json" with { type: "json" }; +import keepfrdsJson from "./keepfrds.json" with { type: "json" }; +import mteamJson from "./mteam.json" with { type: "json" }; +import niceptJson from "./nicept.json" with { type: "json" }; +import ourbitsJson from "./ourbits.json" with { type: "json" }; +import piggoJson from "./piggo.json" with { type: "json" }; +import pterJson from "./pter.json" with { type: "json" }; +import pthomeJson from "./pthome.json" with { type: "json" }; +import pttimeJson from "./pttime.json" with { type: "json" }; +import soulvoiceJson from "./soulvoice.json" with { type: "json" }; +import ssdJson from "./ssd.json" with { type: "json" }; +import tjuptJson from "./tjupt.json" with { type: "json" }; +import ttgJson from "./ttg.json" with { type: "json" }; + +/** MovieClaw 转换来的站点定义原始 JSON(未校验;经 loadSiteDefinition 校验后使用) */ +export const MOVIECLAW_DEFINITION_JSON: Record = { + agsvpt: agsvptJson, + audiences: audiencesJson, + chdbits: chdbitsJson, + hdarea: hdareaJson, + hddolby: hddolbyJson, + hdfans: hdfansJson, + hdhome: hdhomeJson, + hdsky: hdskyJson, + hdtime: hdtimeJson, + hdvideo: hdvideoJson, + hhanclub: hhanclubJson, + keepfrds: keepfrdsJson, + mteam: mteamJson, + nicept: niceptJson, + ourbits: ourbitsJson, + piggo: piggoJson, + pter: pterJson, + pthome: pthomeJson, + pttime: pttimeJson, + soulvoice: soulvoiceJson, + ssd: ssdJson, + tjupt: tjuptJson, + ttg: ttgJson, +}; diff --git a/packages/adapters/src/sites/definitions/mteam.json b/packages/adapters/src/sites/definitions/mteam.json new file mode 100644 index 00000000..b0e7b302 --- /dev/null +++ b/packages/adapters/src/sites/definitions/mteam.json @@ -0,0 +1,72 @@ +{ + "id": "mteam", + "name": "馒头", + "enabled": true, + "schema": "MTorrent", + "provenance": { + "source": "movieclaw", + "ref": "v0.27.0", + "commit": "c79bb1b0", + "license": "MIT", + "file": "src/movieclaw_tracker/sites/configs/mteam.yaml", + "overrides": "scripts/mp-sync/movieclaw-overrides/mteam.json" + }, + "domain": "https://tp.m-team.cc/", + "parser": "mTorrent", + "timeout": 15, + "umt": { + "version": "mteam@0.2.0-unverified", + "kind": "mteam", + "displayName": "馒头", + "apiBaseUrl": "https://api.m-team.cc", + "timezone": "+08:00", + "rateLimit": { + "minIntervalMs": 1000, + "burst": 1 + }, + "capabilities": { + "list": true, + "search": true, + "detail": true, + "download": true, + "rss": false + }, + "categoryNames": { + "401": "电影", + "402": "剧集", + "403": "综艺", + "404": "纪录", + "405": "动画", + "406": "音乐", + "407": "体育", + "408": "音乐", + "409": "其他", + "410": "成人", + "411": "成人", + "412": "成人", + "413": "成人", + "419": "电影", + "420": "电影", + "421": "电影", + "422": "其他", + "423": "游戏", + "424": "成人", + "425": "成人", + "426": "成人", + "427": "其他", + "429": "成人", + "430": "成人", + "431": "成人", + "432": "成人", + "433": "成人", + "434": "音乐", + "435": "剧集", + "436": "成人", + "437": "成人", + "438": "剧集", + "439": "电影" + }, + "authSupported": ["apikey"], + "customClass": "movieclaw_tracker.sites.custom.mteam.MTeamSite" + } +} diff --git a/packages/adapters/src/sites/definitions/nicept.json b/packages/adapters/src/sites/definitions/nicept.json new file mode 100644 index 00000000..e8aa074d --- /dev/null +++ b/packages/adapters/src/sites/definitions/nicept.json @@ -0,0 +1,215 @@ +{ + "id": "nicept", + "name": "NicePT", + "enabled": false, + "schema": "NexusPhp", + "provenance": { + "source": "movieclaw", + "ref": "v0.27.0", + "commit": "c79bb1b0", + "license": "MIT", + "file": "src/movieclaw_tracker/sites/configs/nicept.yaml" + }, + "domain": "https://www.nicept.net/", + "encoding": "UTF-8", + "public": false, + "proxy": false, + "language": "zh", + "result_num": 100, + "timeout": 30, + "search": { + "paths": [ + { + "path": "torrents.php", + "type": "all" + } + ], + "params": { + "search": "{keyword}" + } + }, + "batch": { + "delimiter": " ", + "space_replace": "_" + }, + "browse": { + "path": "torrents.php?page={page}", + "start": 0 + }, + "category": { + "movie": [], + "tv": [], + "music": [] + }, + "torrents": { + "list": { + "selector": "table.torrents > tr, table.torrents > tbody > tr" + }, + "fields": { + "title_default": { + "selector": "table.torrentname a b" + }, + "title_optional": { + "selector": "table.torrentname a b[title]", + "attribute": "title" + }, + "title": { + "text": "{% if fields['title_optional'] %}{{ fields['title_optional'] }}{% else %}{{ fields['title_default'] }}{% endif %}" + }, + "details": { + "selector": "a[href*='details.php']", + "attribute": "href" + }, + "download": { + "selector": "a[href*='download.php']", + "attribute": "href" + }, + "imdbid": { + "selector": "a[href*=\"imdb.com/title/tt\"]", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["tt\\d+", 0] + } + ] + }, + "date_elapsed": { + "selector": "td:nth-child(4)" + }, + "date_added": { + "selector": "td:nth-child(4) > span", + "attribute": "title" + }, + "date": { + "text": "{% if fields['date_elapsed'] or fields['date_added'] %}{{ fields['date_added'] if fields['date_added'] else fields['date_elapsed'] }}{% else %}now{% endif %}", + "filters": [ + { + "name": "dateparse", + "args": "%Y-%m-%d %H:%M:%S" + } + ] + }, + "size": { + "selector": "td:nth-child(5)" + }, + "seeders": { + "selector": "td:nth-child(6)" + }, + "leechers": { + "selector": "td:nth-child(7)" + }, + "grabs": { + "selector": "td:nth-child(8)" + }, + "downloadvolumefactor": { + "case": { + "img.pro_free": 0, + "img.pro_50pctdown": 0.5, + "img.pro_50pctdown2up": 0.5, + "img.pro_30pctdown": 0.3, + "*": 1 + } + }, + "uploadvolumefactor": { + "case": { + "img.pro_50pctdown2up": 2, + "img.pro_2up": 2, + "*": 1 + } + }, + "description": { + "selector": "td.embedded br" + }, + "category": { + "selector": "td:nth-child(1) a[href*='cat=']", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["\\d+", 0] + } + ] + }, + "hr": { + "selector": "img.hitandrun" + } + } + }, + "umt": { + "version": "nicept@mc-v0.27.0", + "kind": "nexusphp", + "displayName": "NicePT", + "timezone": "+08:00", + "rateLimit": { + "minIntervalMs": 5000, + "burst": 1 + }, + "capabilities": { + "list": true, + "search": true, + "detail": true, + "download": true, + "rss": false + }, + "categoryNames": { + "401": "成人", + "402": "成人", + "403": "成人", + "404": "成人", + "500": "成人", + "501": "成人", + "503": "成人", + "504": "成人" + }, + "loginSuccess": "a[href*='logout']", + "emptyMarkers": ["没有种子", "没有结果", "Nothing found"], + "pagination": "a[href*=\"page=\"]", + "detailPath": "details.php", + "downloadPath": "download.php", + "authSupported": ["cookie", "credential"], + "login": { + "path": "login.php", + "usernameField": "username", + "passwordField": "password", + "errorCss": "td.text" + }, + "profile": { + "path": "userdetails.php", + "uid_css": "a[class*='_Name']::attr(href)", + "username_css": "a[class*='_Name'] b", + "class_css": "a[class*='_Name']::attr(class)", + "uploaded_css": "font.color_uploaded::next_sibling_text", + "downloaded_css": "font.color_downloaded::next_sibling_text", + "ratio_css": "font.color_ratio::next_sibling_text", + "bonus_css": "font.color_bonus", + "join_date_css": "td.rowhead:contains('加入日期') + td::text", + "seeding_css": "img.arrowup::next_sibling_text", + "leeching_css": "img.arrowdown::next_sibling_text" + }, + "detail": { + "path": "details.php", + "title": "h1#top", + "rowLabel": "td.rowhead", + "rowValue": "td.rowfollow", + "labels": { + "subtitle": ["副标题", "Small Description"], + "basicInfo": ["基本信息", "Basic Info"], + "peers": ["同伴", "Peers"] + }, + "description": "#kdescr", + "headerScope": "h1#top", + "notFoundMarkers": ["没有该ID的种子", "No torrent with ID"], + "fields": { + "title": "h1#top", + "subtitle": "td.rowhead:contains('副標題') + td::text", + "size": "td.rowhead:contains('基本資訊') + td.rowfollow > b::next_sibling_text", + "description": "#kdescr", + "download": "a[href*='download.php']::attr(href)", + "imdb": "a[href*='imdb.com']::attr(href)", + "douban": "a[href*='douban.com']::attr(href)", + "fileList": "td.rowfollow > ul > li::text" + } + } + } +} diff --git a/packages/adapters/src/sites/definitions/ourbits.json b/packages/adapters/src/sites/definitions/ourbits.json new file mode 100644 index 00000000..ce183a84 --- /dev/null +++ b/packages/adapters/src/sites/definitions/ourbits.json @@ -0,0 +1,261 @@ +{ + "id": "ourbits", + "name": "OurBits", + "enabled": false, + "schema": "NexusPhp", + "provenance": { + "source": "movieclaw", + "ref": "v0.27.0", + "commit": "c79bb1b0", + "license": "MIT", + "file": "src/movieclaw_tracker/sites/configs/ourbits.yaml" + }, + "domain": "https://ourbits.club/", + "encoding": "UTF-8", + "public": false, + "proxy": false, + "language": "zh", + "result_num": 100, + "timeout": 30, + "search": { + "paths": [ + { + "path": "torrents.php", + "type": "all" + } + ], + "params": { + "search": "{keyword}" + } + }, + "batch": { + "delimiter": " ", + "space_replace": "_" + }, + "browse": { + "path": "torrents.php?page={page}", + "start": 0 + }, + "category": { + "movie": [ + { + "id": "401", + "desc": "电影" + }, + { + "id": "402", + "desc": "电影" + } + ], + "tv": [ + { + "id": "405", + "desc": "剧集" + }, + { + "id": "412", + "desc": "剧集" + }, + { + "id": "413", + "desc": "剧集" + }, + { + "id": "415", + "desc": "剧集" + } + ], + "music": [ + { + "id": "416", + "desc": "音乐" + }, + { + "id": "414", + "desc": "音乐" + }, + { + "id": "419", + "desc": "音乐" + } + ], + "param": "cat[]" + }, + "torrents": { + "list": { + "selector": "table.torrents > tr:has(table.torrentname)" + }, + "fields": { + "title_default": { + "selector": "a[href*='details.php']" + }, + "title_optional": { + "selector": "a[href*='details.php'][title]", + "attribute": "title" + }, + "title": { + "text": "{% if fields['title_optional'] %}{{ fields['title_optional'] }}{% else %}{{ fields['title_default'] }}{% endif %}" + }, + "details": { + "selector": "a[href*='details.php']", + "attribute": "href" + }, + "download": { + "selector": "a[href*='download.php']", + "attribute": "href" + }, + "imdbid": { + "selector": "a[href*=\"imdb.com/title/tt\"]", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["tt\\d+", 0] + } + ] + }, + "date_elapsed": { + "selector": "td:nth-child(4)" + }, + "date_added": { + "selector": "td:nth-child(4) > span", + "attribute": "title" + }, + "date": { + "text": "{% if fields['date_elapsed'] or fields['date_added'] %}{{ fields['date_added'] if fields['date_added'] else fields['date_elapsed'] }}{% else %}now{% endif %}", + "filters": [ + { + "name": "dateparse", + "args": "%Y-%m-%d %H:%M:%S" + } + ] + }, + "size": { + "selector": "td:nth-child(5)" + }, + "seeders": { + "selector": "td:nth-child(6)" + }, + "leechers": { + "selector": "td:nth-child(7)" + }, + "grabs": { + "selector": "td:nth-child(8)" + }, + "downloadvolumefactor": { + "case": { + "img.pro_free": 0, + "img.pro_free2up": 0, + "img.pro_50pctdown": 0.5, + "img.pro_50pctdown2up": 0.5, + "img.pro_30pctdown": 0.3, + "*": 1 + } + }, + "uploadvolumefactor": { + "case": { + "img.pro_free2up": 2, + "img.pro_50pctdown2up": 2, + "img.pro_2up": 2, + "*": 1 + } + }, + "description": { + "selector": "td.embedded > div" + }, + "category": { + "selector": "a[href^='?cat=']", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["\\d+", 0] + } + ] + }, + "hr": { + "selector": "img.hitandrun" + } + } + }, + "umt": { + "version": "ourbits@mc-v0.27.0", + "kind": "nexusphp", + "displayName": "OurBits", + "timezone": "+08:00", + "rateLimit": { + "minIntervalMs": 3000, + "burst": 1 + }, + "capabilities": { + "list": true, + "search": true, + "detail": true, + "download": true, + "rss": false + }, + "categoryNames": { + "401": "电影", + "402": "电影", + "405": "剧集", + "410": "纪录片", + "411": "动漫", + "412": "剧集", + "413": "剧集", + "414": "音乐", + "415": "剧集", + "416": "音乐", + "419": "音乐" + }, + "loginSuccess": "a[href*='logout']", + "emptyMarkers": ["没有种子", "没有结果", "Nothing found"], + "pagination": "a[href*=\"page=\"]", + "detailPath": "details.php", + "downloadPath": "download.php", + "authSupported": ["cookie"], + "login": { + "path": "login.php", + "usernameField": "username", + "passwordField": "password", + "errorCss": "td.text" + }, + "profile": { + "path": "userdetails.php", + "uid_css": "a[href*='userdetails.php?id=']::attr(href)", + "username_css": "a[href*='userdetails.php?id=']", + "class_css": "a[href*='userdetails.php?id=']::attr(class)", + "uploaded_css": "font.color_uploaded::next_sibling_text", + "downloaded_css": "font.color_downloaded::next_sibling_text", + "ratio_css": "td.rowhead:contains('分享率') + td::text", + "bonus_css": "td.rowhead:contains('魔力') + td::text", + "join_date_css": "td.rowhead:contains('加入日期') + td::text", + "seeding_css": "img[alt='Torrents seeding'] + *", + "leeching_css": "img[alt='Torrents leeching'] + *" + }, + "detail": { + "path": "details.php", + "title": "#top", + "rowLabel": "td.rowhead", + "rowValue": "td.rowfollow", + "labels": { + "subtitle": ["副标题", "Small Description"], + "basicInfo": ["基本信息", "Basic Info"], + "peers": ["同伴", "Peers"] + }, + "description": "#kdescr", + "headerScope": "#top", + "notFoundMarkers": ["没有该ID的种子", "No torrent with ID"], + "fields": { + "title": "#top::text", + "subtitle": "td.rowhead:contains('副标题') + td::text", + "size": "td.rowhead:contains('大小') + td::text", + "description": "#kdescr", + "download": "a[href*='download.php']::attr(href)", + "imdb": "a[href*='imdb.com']::attr(href)", + "douban": "a[href*='douban.com']::attr(href)", + "fileList": "td.rowfollow > ul > li::text" + } + }, + "customClass": "movieclaw_tracker.sites.custom.ourbits.OurBitsSite" + } +} diff --git a/packages/adapters/src/sites/definitions/piggo.json b/packages/adapters/src/sites/definitions/piggo.json new file mode 100644 index 00000000..b2f4d43f --- /dev/null +++ b/packages/adapters/src/sites/definitions/piggo.json @@ -0,0 +1,233 @@ +{ + "id": "piggo", + "name": "猪猪网 Piggo", + "enabled": false, + "schema": "NexusPhp", + "provenance": { + "source": "movieclaw", + "ref": "v0.27.0", + "commit": "c79bb1b0", + "license": "MIT", + "file": "src/movieclaw_tracker/sites/configs/piggo.yaml" + }, + "domain": "https://piggo.me/", + "encoding": "UTF-8", + "public": false, + "proxy": false, + "language": "zh", + "result_num": 100, + "timeout": 30, + "search": { + "paths": [ + { + "path": "torrents.php", + "type": "all" + } + ], + "params": { + "search": "{keyword}" + } + }, + "batch": { + "delimiter": " ", + "space_replace": "_" + }, + "browse": { + "path": "torrents.php?page={page}", + "start": 0 + }, + "category": { + "movie": [ + { + "id": "401", + "desc": "电影" + } + ], + "tv": [ + { + "id": "402", + "desc": "剧集" + }, + { + "id": "403", + "desc": "剧集" + } + ], + "music": [ + { + "id": "406", + "desc": "音乐" + }, + { + "id": "408", + "desc": "音乐" + } + ] + }, + "torrents": { + "list": { + "selector": "table.torrents > tr, table.torrents > tbody > tr" + }, + "fields": { + "title_default": { + "selector": "table.torrentname a b" + }, + "title_optional": { + "selector": "table.torrentname a b[title]", + "attribute": "title" + }, + "title": { + "text": "{% if fields['title_optional'] %}{{ fields['title_optional'] }}{% else %}{{ fields['title_default'] }}{% endif %}" + }, + "details": { + "selector": "a[href*='details.php']", + "attribute": "href" + }, + "download": { + "selector": "a[href*='download.php']", + "attribute": "href" + }, + "imdbid": { + "selector": "a[href*=\"imdb.com/title/tt\"]", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["tt\\d+", 0] + } + ] + }, + "date_elapsed": { + "selector": "td:nth-child(4)" + }, + "date_added": { + "selector": "td:nth-child(4) > span", + "attribute": "title" + }, + "date": { + "text": "{% if fields['date_elapsed'] or fields['date_added'] %}{{ fields['date_added'] if fields['date_added'] else fields['date_elapsed'] }}{% else %}now{% endif %}", + "filters": [ + { + "name": "dateparse", + "args": "%Y-%m-%d %H:%M:%S" + } + ] + }, + "size": { + "selector": "td:nth-child(5)" + }, + "seeders": { + "selector": "td:nth-child(6)" + }, + "leechers": { + "selector": "td:nth-child(7)" + }, + "grabs": { + "selector": "td:nth-child(8)" + }, + "downloadvolumefactor": { + "case": { + "img.pro_free": 0, + "img.pro_50pctdown": 0.5, + "*": 1 + } + }, + "uploadvolumefactor": { + "case": { + "*": 1 + } + }, + "description": { + "selector": "td.embedded br" + }, + "category": { + "selector": "td:nth-child(1) a[href*='cat=']", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["\\d+", 0] + } + ] + } + } + }, + "umt": { + "version": "piggo@mc-v0.27.0", + "kind": "nexusphp", + "displayName": "猪猪网 Piggo", + "timezone": "+08:00", + "rateLimit": { + "minIntervalMs": 8000, + "burst": 1 + }, + "capabilities": { + "list": true, + "search": true, + "detail": true, + "download": true, + "rss": false + }, + "categoryNames": { + "401": "电影", + "402": "剧集", + "403": "剧集", + "404": "纪录片", + "405": "动漫", + "406": "音乐", + "407": "其他", + "408": "音乐", + "409": "其他", + "912": "其他" + }, + "loginSuccess": "a[href*='logout']", + "emptyMarkers": ["没有种子", "没有结果", "Nothing found"], + "pagination": "a[href*=\"page=\"]", + "detailPath": "details.php", + "downloadPath": "download.php", + "authSupported": ["cookie", "credential"], + "login": { + "path": "login.php", + "usernameField": "username", + "passwordField": "password", + "errorCss": "td.text" + }, + "profile": { + "path": "userdetails.php", + "uid_css": "a[class*='_Name']::attr(href)", + "username_css": "a[class*='_Name'] b", + "class_css": "a[class*='_Name']::attr(class)", + "uploaded_css": "font.color_uploaded::next_sibling_text", + "downloaded_css": "font.color_downloaded::next_sibling_text", + "ratio_css": "font.color_ratio::next_sibling_text", + "bonus_css": "font.color_bonus", + "join_date_css": "td.rowhead:contains('加入日期') + td::text", + "seeding_css": "img.arrowup::next_sibling_text", + "leeching_css": "img.arrowdown::next_sibling_text" + }, + "detail": { + "path": "details.php", + "title": "h1#top", + "rowLabel": "td.rowhead", + "rowValue": "td.rowfollow", + "labels": { + "subtitle": ["副标题", "Small Description"], + "basicInfo": ["基本信息", "Basic Info"], + "peers": ["同伴", "Peers"] + }, + "description": "#kdescr", + "headerScope": "h1#top", + "notFoundMarkers": ["没有该ID的种子", "No torrent with ID"], + "fields": { + "title": "h1#top", + "subtitle": "td.rowhead:contains('副标题') + td::text", + "size": "td.rowhead:contains('基本信息') + td.rowfollow > b::next_sibling_text", + "description": "#kdescr", + "download": "a[href*='download.php']::attr(href)", + "imdb": "a[href*='imdb.com']::attr(href)", + "douban": "a[href*='douban.com']::attr(href)", + "fileList": "td.rowfollow > ul > li::text" + } + } + } +} diff --git a/packages/adapters/src/sites/definitions/pter.json b/packages/adapters/src/sites/definitions/pter.json new file mode 100644 index 00000000..21bd7a5d --- /dev/null +++ b/packages/adapters/src/sites/definitions/pter.json @@ -0,0 +1,233 @@ +{ + "id": "pter", + "name": "猫站", + "enabled": false, + "schema": "NexusPhp", + "provenance": { + "source": "movieclaw", + "ref": "v0.27.0", + "commit": "c79bb1b0", + "license": "MIT", + "file": "src/movieclaw_tracker/sites/configs/pter.yaml" + }, + "domain": "https://pterclub.net/", + "encoding": "UTF-8", + "public": false, + "proxy": false, + "language": "zh", + "result_num": 100, + "timeout": 30, + "search": { + "paths": [ + { + "path": "torrents.php", + "type": "all" + } + ], + "params": { + "search": "{keyword}" + } + }, + "batch": { + "delimiter": " ", + "space_replace": "_" + }, + "browse": { + "path": "torrents.php?page={page}", + "start": 0 + }, + "category": { + "movie": [ + { + "id": "401", + "desc": "电影" + } + ], + "tv": [ + { + "id": "404", + "desc": "剧集" + }, + { + "id": "405", + "desc": "剧集" + } + ], + "music": [ + { + "id": "406", + "desc": "音乐" + }, + { + "id": "418", + "desc": "音乐" + } + ] + }, + "torrents": { + "list": { + "selector": "table.torrents > tr:has(table.torrentname)" + }, + "fields": { + "title_default": { + "selector": "a[href*='details.php']" + }, + "title_optional": { + "selector": "a[href*='details.php'][title]", + "attribute": "title" + }, + "title": { + "text": "{% if fields['title_optional'] %}{{ fields['title_optional'] }}{% else %}{{ fields['title_default'] }}{% endif %}" + }, + "details": { + "selector": "a[href*='details.php']", + "attribute": "href" + }, + "download": { + "selector": "a[href*='download.php']", + "attribute": "href" + }, + "imdbid": { + "selector": "a[href*=\"imdb.com/title/tt\"]", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["tt\\d+", 0] + } + ] + }, + "date_elapsed": { + "selector": "td:nth-child(4)" + }, + "date_added": { + "selector": "td:nth-child(4) > span", + "attribute": "title" + }, + "date": { + "text": "{% if fields['date_elapsed'] or fields['date_added'] %}{{ fields['date_added'] if fields['date_added'] else fields['date_elapsed'] }}{% else %}now{% endif %}", + "filters": [ + { + "name": "dateparse", + "args": "%Y-%m-%d %H:%M:%S" + } + ] + }, + "size": { + "selector": "td:nth-child(5)" + }, + "seeders": { + "selector": "td:nth-child(6)" + }, + "leechers": { + "selector": "td:nth-child(7)" + }, + "grabs": { + "selector": "td:nth-child(8)" + }, + "downloadvolumefactor": { + "case": { + "img.pro_free": 0, + "img.pro_free2up": 0, + "img.pro_50pctdown": 0.5, + "img.pro_50pctdown2up": 0.5, + "img.pro_30pctdown": 0.3, + "*": 1 + } + }, + "uploadvolumefactor": { + "case": { + "img.pro_free2up": 2, + "img.pro_50pctdown2up": 2, + "img.pro_2up": 2, + "*": 1 + } + }, + "category": { + "selector": "a[href^='?cat=']", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["\\d+", 0] + } + ] + } + } + }, + "umt": { + "version": "pter@mc-v0.27.0", + "kind": "nexusphp", + "displayName": "猫站", + "timezone": "+08:00", + "rateLimit": { + "minIntervalMs": 3000, + "burst": 1 + }, + "capabilities": { + "list": true, + "search": true, + "detail": true, + "download": true, + "rss": false + }, + "categoryNames": { + "401": "电影", + "402": "纪录片", + "403": "动漫", + "404": "剧集", + "405": "剧集", + "406": "音乐", + "409": "游戏", + "418": "音乐" + }, + "loginSuccess": "a[href*='logout']", + "emptyMarkers": ["没有种子", "没有结果", "Nothing found"], + "pagination": "a[href*=\"page=\"]", + "detailPath": "details.php", + "downloadPath": "download.php", + "login": { + "path": "login.php", + "usernameField": "username", + "passwordField": "password", + "errorCss": "td.text" + }, + "profile": { + "path": "userdetails.php", + "uid_css": "a[href*='userdetails.php?id=']::attr(href)", + "username_css": "a[href*='userdetails.php?id=']", + "class_css": "a[href*='userdetails.php?id=']::attr(class)", + "uploaded_css": "font.color_uploaded::next_sibling_text", + "downloaded_css": "font.color_downloaded::next_sibling_text", + "ratio_css": "font.color_ratio::next_sibling_text", + "bonus_css": "td.rowhead:contains('魔力') + td::text", + "join_date_css": "td.rowhead:contains('加入日期') + td::text", + "seeding_css": "img[alt='Torrents seeding'] + *", + "leeching_css": "img[alt='Torrents leeching'] + *" + }, + "detail": { + "path": "details.php", + "title": "#top", + "rowLabel": "td.rowhead", + "rowValue": "td.rowfollow", + "labels": { + "subtitle": ["副标题", "Small Description"], + "basicInfo": ["基本信息", "Basic Info"], + "peers": ["同伴", "Peers"] + }, + "description": "#kdescr", + "headerScope": "#top", + "notFoundMarkers": ["没有该ID的种子", "No torrent with ID"], + "fields": { + "title": "#top::text", + "subtitle": "td.rowhead:contains('副标题') + td::text", + "size": "td.rowhead:contains('大小') + td::text", + "description": "#kdescr", + "download": "a[href*='download.php']::attr(href)", + "imdb": "a[href*='imdb.com']::attr(href)", + "douban": "a[href*='douban.com']::attr(href)", + "fileList": "td.rowfollow > ul > li::text" + } + } + } +} diff --git a/packages/adapters/src/sites/definitions/pthome.json b/packages/adapters/src/sites/definitions/pthome.json new file mode 100644 index 00000000..cddc74ba --- /dev/null +++ b/packages/adapters/src/sites/definitions/pthome.json @@ -0,0 +1,239 @@ +{ + "id": "pthome", + "name": "铂金家", + "enabled": false, + "schema": "NexusPhp", + "provenance": { + "source": "movieclaw", + "ref": "v0.27.0", + "commit": "c79bb1b0", + "license": "MIT", + "file": "src/movieclaw_tracker/sites/configs/pthome.yaml" + }, + "domain": "https://pthome.net/", + "encoding": "UTF-8", + "public": false, + "proxy": false, + "language": "zh", + "result_num": 100, + "timeout": 30, + "search": { + "paths": [ + { + "path": "torrents.php", + "type": "all" + } + ], + "params": { + "search": "{keyword}" + } + }, + "batch": { + "delimiter": " ", + "space_replace": "_" + }, + "browse": { + "path": "torrents.php?page={page}", + "start": 0 + }, + "category": { + "movie": [ + { + "id": "401", + "desc": "电影" + } + ], + "tv": [ + { + "id": "402", + "desc": "剧集" + }, + { + "id": "403", + "desc": "剧集" + }, + { + "id": "407", + "desc": "剧集" + } + ], + "music": [ + { + "id": "408", + "desc": "音乐" + } + ] + }, + "torrents": { + "list": { + "selector": "table.torrents > tr:has(table.torrentname)" + }, + "fields": { + "title_default": { + "selector": "a[href*='details.php']" + }, + "title_optional": { + "selector": "a[href*='details.php'][title]", + "attribute": "title" + }, + "title": { + "text": "{% if fields['title_optional'] %}{{ fields['title_optional'] }}{% else %}{{ fields['title_default'] }}{% endif %}" + }, + "details": { + "selector": "a[href*='details.php']", + "attribute": "href" + }, + "download": { + "selector": "a[href*='download.php']", + "attribute": "href" + }, + "imdbid": { + "selector": "a[href*=\"imdb.com/title/tt\"]", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["tt\\d+", 0] + } + ] + }, + "date_elapsed": { + "selector": "td:nth-child(4)" + }, + "date_added": { + "selector": "td:nth-child(4) > span", + "attribute": "title" + }, + "date": { + "text": "{% if fields['date_elapsed'] or fields['date_added'] %}{{ fields['date_added'] if fields['date_added'] else fields['date_elapsed'] }}{% else %}now{% endif %}", + "filters": [ + { + "name": "dateparse", + "args": "%Y-%m-%d %H:%M:%S" + } + ] + }, + "size": { + "selector": "td:nth-child(5)" + }, + "seeders": { + "selector": "td:nth-child(6)" + }, + "leechers": { + "selector": "td:nth-child(7)" + }, + "grabs": { + "selector": "td:nth-child(8)" + }, + "downloadvolumefactor": { + "case": { + "img.pro_free": 0, + "img.pro_free2up": 0, + "img.pro_50pctdown": 0.5, + "img.pro_50pctdown2up": 0.5, + "img.pro_30pctdown": 0.3, + "*": 1 + } + }, + "uploadvolumefactor": { + "case": { + "img.pro_free2up": 2, + "img.pro_50pctdown2up": 2, + "img.pro_2up": 2, + "*": 1 + } + }, + "description": { + "selector": "td.embedded > span:not([class])" + }, + "category": { + "selector": "a[href^='?cat=']", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["\\d+", 0] + } + ] + } + } + }, + "umt": { + "version": "pthome@mc-v0.27.0", + "kind": "nexusphp", + "displayName": "铂金家", + "timezone": "+08:00", + "rateLimit": { + "minIntervalMs": 3000, + "burst": 1 + }, + "capabilities": { + "list": true, + "search": true, + "detail": true, + "download": true, + "rss": false + }, + "categoryNames": { + "401": "电影", + "402": "剧集", + "403": "剧集", + "404": "纪录片", + "405": "动漫", + "407": "剧集", + "408": "音乐", + "409": "其他", + "410": "游戏", + "411": "其他", + "412": "其他" + }, + "loginSuccess": "a[href*='logout']", + "emptyMarkers": ["没有种子", "没有结果", "Nothing found"], + "pagination": "a[href*=\"page=\"]", + "detailPath": "details.php", + "downloadPath": "download.php", + "login": { + "path": "login.php", + "usernameField": "username", + "passwordField": "password", + "errorCss": "td.text" + }, + "profile": { + "path": "userdetails.php", + "uid_css": "a[href*='userdetails.php?id=']::attr(href)", + "username_css": "a[href*='userdetails.php?id=']", + "class_css": "a[href*='userdetails.php?id=']::attr(class)", + "uploaded_css": "font.color_uploaded::next_sibling_text", + "downloaded_css": "font.color_downloaded::next_sibling_text", + "ratio_css": "font.color_ratio::next_sibling_text", + "bonus_css": "font.color_bonus::next_sibling_text", + "join_date_css": "td.rowhead:contains('加入日期') + td::text", + "seeding_css": "img[alt='Torrents seeding'] + *", + "leeching_css": "img[alt='Torrents leeching'] + *" + }, + "detail": { + "path": "details.php", + "title": "#top", + "rowLabel": "td.rowhead", + "rowValue": "td.rowfollow", + "labels": { + "subtitle": ["副标题", "Small Description"], + "basicInfo": ["基本信息", "Basic Info"], + "peers": ["同伴", "Peers"] + }, + "description": "#kdescr", + "headerScope": "#top", + "notFoundMarkers": ["没有该ID的种子", "No torrent with ID"], + "fields": { + "title": "#top::text", + "subtitle": "td.rowhead:contains('副标题') + td::text", + "size": "td.rowhead:contains('大小') + td::text", + "description": "#kdescr", + "download": "a[href*='download.php']::attr(href)", + "imdb": "a[href*='imdb.com']::attr(href)", + "douban": "a[href*='douban.com']::attr(href)", + "fileList": "td.rowfollow > ul > li::text" + } + } + } +} diff --git a/packages/adapters/src/sites/definitions/pttime.json b/packages/adapters/src/sites/definitions/pttime.json new file mode 100644 index 00000000..1930b149 --- /dev/null +++ b/packages/adapters/src/sites/definitions/pttime.json @@ -0,0 +1,240 @@ +{ + "id": "pttime", + "name": "PTTime", + "enabled": false, + "schema": "NexusPhp", + "provenance": { + "source": "movieclaw", + "ref": "v0.27.0", + "commit": "c79bb1b0", + "license": "MIT", + "file": "src/movieclaw_tracker/sites/configs/pttime.yaml" + }, + "domain": "https://www.pttime.org/", + "encoding": "UTF-8", + "public": false, + "proxy": false, + "language": "zh", + "result_num": 100, + "timeout": 30, + "search": { + "paths": [ + { + "path": "torrents.php", + "type": "all" + } + ], + "params": { + "search": "{keyword}" + } + }, + "batch": { + "delimiter": " ", + "space_replace": "_" + }, + "browse": { + "path": "torrents.php?page={page}", + "start": 0 + }, + "category": { + "movie": [ + { + "id": "401", + "desc": "电影" + } + ], + "tv": [ + { + "id": "402", + "desc": "剧集" + }, + { + "id": "403", + "desc": "剧集" + } + ], + "music": [ + { + "id": "408", + "desc": "音乐" + } + ] + }, + "torrents": { + "list": { + "selector": "table.torrents > tr, table.torrents > tbody > tr" + }, + "fields": { + "title_default": { + "selector": "table.torrentname a b" + }, + "title_optional": { + "selector": "table.torrentname a b[title]", + "attribute": "title" + }, + "title": { + "text": "{% if fields['title_optional'] %}{{ fields['title_optional'] }}{% else %}{{ fields['title_default'] }}{% endif %}" + }, + "details": { + "selector": "a[href*='details.php']", + "attribute": "href" + }, + "download": { + "selector": "a[href*='download.php']", + "attribute": "href" + }, + "imdbid": { + "selector": "a[href*=\"imdb.com/title/tt\"]", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["tt\\d+", 0] + } + ] + }, + "date_elapsed": { + "selector": "td:nth-child(5)" + }, + "date_added": { + "selector": "td:nth-child(5) > span", + "attribute": "title" + }, + "date": { + "text": "{% if fields['date_elapsed'] or fields['date_added'] %}{{ fields['date_added'] if fields['date_added'] else fields['date_elapsed'] }}{% else %}now{% endif %}", + "filters": [ + { + "name": "dateparse", + "args": "%Y-%m-%d %H:%M:%S" + } + ] + }, + "size": { + "selector": "td:nth-child(6)" + }, + "seeders": { + "selector": "td:nth-child(7)" + }, + "leechers": { + "selector": "td:nth-child(8)" + }, + "grabs": { + "selector": "td:nth-child(9)" + }, + "downloadvolumefactor": { + "case": { + "font.promotion.free": 0, + "font.promotion.twoupfree": 0, + "font.promotion.halfdown": 0.5, + "*": 1 + } + }, + "uploadvolumefactor": { + "case": { + "font.promotion.twoupfree": 2, + "font.promotion.twoup": 2, + "*": 1 + } + }, + "description": { + "selector": "td.embedded font[title]", + "attribute": "title" + }, + "category": { + "selector": "td:nth-child(1) a[href*='cat=']", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["\\d+", 0] + } + ] + } + } + }, + "umt": { + "version": "pttime@mc-v0.27.0", + "kind": "nexusphp", + "displayName": "PTTime", + "timezone": "+08:00", + "rateLimit": { + "minIntervalMs": 5000, + "burst": 1 + }, + "capabilities": { + "list": true, + "search": true, + "detail": true, + "download": true, + "rss": false + }, + "categoryNames": { + "100": "其他", + "401": "电影", + "402": "剧集", + "403": "剧集", + "404": "纪录片", + "405": "其他", + "406": "动漫", + "407": "其他", + "408": "音乐", + "409": "其他", + "411": "其他", + "412": "其他", + "420": "其他", + "421": "游戏", + "422": "其他", + "423": "其他", + "430": "其他" + }, + "loginSuccess": "a[href*='logout']", + "emptyMarkers": ["没有种子", "没有结果", "Nothing found"], + "pagination": "a[href*=\"page=\"]", + "detailPath": "details.php", + "downloadPath": "download.php", + "authSupported": ["cookie", "credential"], + "login": { + "path": "login.php", + "usernameField": "username", + "passwordField": "password", + "errorCss": "td.text" + }, + "profile": { + "path": "userdetails.php", + "uid_css": "#info_block span.medium a[href*='userdetails.php']::attr(href)", + "username_css": "#info_block span.medium a[href*='userdetails.php'] b", + "class_css": "td.rowhead:contains('用户等级') + td b::text", + "uploaded_css": "#info_block font:contains('上传:')::next_sibling_text", + "downloaded_css": "#info_block font:contains('下载:')::next_sibling_text", + "ratio_css": "#info_block font:contains('分享率:')::next_sibling_text", + "bonus_css": "#info_block a[href='mybonus.php']::next_sibling_text", + "join_date_css": "td.rowhead:contains('加入日期') + td::text", + "seeding_css": "#info_block font[title='当前做种']::next_sibling_text", + "leeching_css": "#info_block font[title='当前下载']::next_sibling_text" + }, + "detail": { + "path": "details.php", + "title": "h1#top", + "rowLabel": "td.rowhead", + "rowValue": "td.rowfollow", + "labels": { + "subtitle": ["副标题", "Small Description"], + "basicInfo": ["基本信息", "Basic Info"], + "peers": ["同伴", "Peers"] + }, + "description": "#kdescr", + "headerScope": "h1#top", + "notFoundMarkers": ["没有该ID的种子", "No torrent with ID"], + "fields": { + "title": "h1#top", + "subtitle": "td.rowhead:contains('副标题') + td::text", + "size": "td.rowhead:contains('基本信息') + td.rowfollow > b::next_sibling_text", + "description": "#kdescr", + "download": "a[href*='download.php']::attr(href)", + "imdb": "a[href*='imdb.com']::attr(href)", + "douban": "a[href*='douban.com']::attr(href)", + "fileList": "td.rowfollow > ul > li::text" + } + } + } +} diff --git a/packages/adapters/src/sites/definitions/schema.ts b/packages/adapters/src/sites/definitions/schema.ts new file mode 100644 index 00000000..7bdb052d --- /dev/null +++ b/packages/adapters/src/sites/definitions/schema.ts @@ -0,0 +1,248 @@ +/** + * 站点定义 zod schema(加载时校验)。 + * + * 键名按 MoviePilot e6820a8 `app/application/site/sites.pyi` `SitesHelper.get_indexer(domain) -> dict` 与 + * `app/modules/indexer/spider/__init__.py` `SiteSpider` / 专用 spider 读取的 indexer dict 键整理 + * (id、name、domain、encoding、public、proxy、language、parser、result_num、timeout、media_type、search、batch、 + * browse、category、torrents),MP 的真实定义在加密 user.sites.v3.bin 中不可读; + * `enabled` / `schema`(MP 用户信息解析器 SiteSchema)/ `provenance` / `umt` 为本项目扩展。 + * TS 类型仍以 spider/schema.ts `IndexerDefinition`、mteam/definition.ts `MteamDefinition` 为准, + * 本文件保证 JSON 与之一致(见 schema.test.ts)。 + */ +import { z } from "zod"; +import type { MteamDefinition } from "../mteam/definition.ts"; +import type { IndexerDefinition } from "../spider/schema.ts"; +import type { SpecializedSiteDefinition } from "../spiders/types.ts"; +import { SiteSchema } from "../user-parsers/types.ts"; + +const SITE_SCHEMA_VALUES = Object.values(SiteSchema) as [string, ...string[]]; + +export const DefinitionProvenanceSchema = z.strictObject({ + source: z.enum(["movieclaw", "handwritten"]), + ref: z.string().optional(), + commit: z.string().optional(), + license: z.string().optional(), + file: z.string().optional(), + overrides: z.string().optional(), + note: z.string().optional(), +}); + +export const SpiderFilterSchema = z.discriminatedUnion("name", [ + z.strictObject({ name: z.literal("re_search"), args: z.tuple([z.string(), z.number().int()]) }), + z.strictObject({ name: z.literal("split"), args: z.tuple([z.string(), z.number().int()]) }), + z.strictObject({ name: z.literal("replace"), args: z.tuple([z.string(), z.string()]) }), + z.strictObject({ name: z.literal("dateparse"), args: z.string() }), + z.strictObject({ name: z.literal("strip"), args: z.unknown().optional() }), + z.strictObject({ name: z.literal("appendleft"), args: z.string() }), + z.strictObject({ name: z.literal("querystring"), args: z.string() }), +]); + +export const SpiderFieldSelectorSchema = z.strictObject({ + selector: z.string().min(1).optional(), + selectors: z.string().min(1).optional(), + attribute: z.string().min(1).optional(), + index: z.number().int().optional(), + contents: z.number().int().optional(), + remove: z.string().optional(), + filters: z.array(SpiderFilterSchema).optional(), + text: z.string().optional(), + case: z.record(z.string(), z.number()).optional(), +}); + +const CategoryItemSchema = z.strictObject({ + id: z.union([z.string(), z.number()]), + cat: z.string().optional(), + desc: z.string().optional(), + param: z.string().optional(), + value: z.union([z.string(), z.number()]).optional(), + params: z.record(z.string(), z.union([z.string(), z.number()])).optional(), +}); + +const CategoryConfigSchema = z.strictObject({ + movie: z.array(CategoryItemSchema).optional(), + tv: z.array(CategoryItemSchema).optional(), + music: z.array(CategoryItemSchema).optional(), + field: z.string().optional(), + delimiter: z.string().optional(), + param: z.string().optional(), +}); + +const CapabilitiesSchema = z.strictObject({ + list: z.boolean(), + search: z.boolean(), + detail: z.boolean(), + download: z.boolean(), + rss: z.boolean(), +}); +const RateLimitSchema = z.strictObject({ minIntervalMs: z.number().int().min(0), burst: z.number().int().min(1) }); +const TimezoneSchema = z.string().regex(/^[+-]\d{2}:\d{2}$/, "时区须为 ±HH:MM 偏移"); +const PairSchema = z.tuple([z.string(), z.string()]); + +const UmtNexusSchema = z.strictObject({ + version: z.string().min(1), + kind: z.literal("nexusphp"), + displayName: z.string().optional(), + customClass: z.string().optional(), + hrSeedHours: z.number().positive().optional(), + authSupported: z.array(z.enum(["cookie", "credential", "apikey"])).optional(), + timezone: TimezoneSchema, + rateLimit: RateLimitSchema, + capabilities: CapabilitiesSchema, + categoryNames: z.record(z.string(), z.string()).optional(), + loginSuccess: z.string().optional(), + emptyMarkers: z.array(z.string()).optional(), + pagination: z.string().optional(), + login: z + .strictObject({ + path: z.string(), + usernameField: z.string(), + passwordField: z.string(), + errorCss: z.string(), + extraFormData: z.array(PairSchema).optional(), + selectDefaults: z.array(PairSchema).optional(), + }) + .optional(), + profile: z.record(z.string(), z.string()).optional(), + detail: z + .strictObject({ + path: z.string(), + title: z.string(), + rowLabel: z.string(), + rowValue: z.string(), + labels: z.strictObject({ + subtitle: z.array(z.string()), + basicInfo: z.array(z.string()), + peers: z.array(z.string()), + }), + description: z.string(), + headerScope: z.string(), + notFoundMarkers: z.array(z.string()), + fields: z + .strictObject({ + title: z.string().optional(), + subtitle: z.string().optional(), + size: z.string().optional(), + description: z.string().optional(), + download: z.string().optional(), + imdb: z.string().optional(), + douban: z.string().optional(), + fileList: z.string().optional(), + }) + .optional(), + }) + .optional(), + downloadPath: z.string().min(1), + detailPath: z.string().min(1), +}); + +const BaseDefinitionShape = { + id: z.string().regex(/^[a-z0-9_]+$/, "id 只能是小写字母/数字/下划线"), + name: z.string().min(1), + enabled: z.boolean().optional(), + schema: z.enum(SITE_SCHEMA_VALUES).optional(), + provenance: DefinitionProvenanceSchema.optional(), + domain: z + .string() + .url() + .refine((s) => s.endsWith("/"), "domain 须以 / 结尾"), + timeout: z.number().positive().optional(), + proxy: z.boolean().optional(), +}; + +const FieldsSchema = z.record(z.string(), SpiderFieldSelectorSchema); + +export const IndexerDefinitionSchema = z.strictObject({ + ...BaseDefinitionShape, + encoding: z.string().optional(), + public: z.boolean().optional(), + language: z.string().optional(), + parser: z.string().optional(), + result_num: z.number().int().positive().optional(), + media_type: z.string().optional(), + search: z + .strictObject({ + paths: z.array(z.strictObject({ path: z.string(), type: z.enum(["all", "movie", "tv", "music"]).optional() })), + params: z.record(z.string(), z.union([z.string(), z.number()])).optional(), + imdbid_format: z.string().optional(), + result_media_type: z.literal("requested").optional(), + }) + .optional(), + batch: z.strictObject({ delimiter: z.string().optional(), space_replace: z.string().optional() }).optional(), + browse: z + .strictObject({ + path: z.string(), + start: z.number().int().optional(), + list: z.strictObject({ selector: z.string() }).optional(), + fields: FieldsSchema.optional(), + }) + .optional(), + category: CategoryConfigSchema.optional(), + torrents: z.strictObject({ + list: z.strictObject({ selector: z.string().min(1) }), + fields: FieldsSchema.refine((f) => Boolean(f.title || f.title_default), "torrents.fields 缺少 title"), + }), + umt: UmtNexusSchema, +}); + +export const MteamDefinitionSchema = z.strictObject({ + ...BaseDefinitionShape, + parser: z.literal("mTorrent"), + timeout: z.number().positive(), + umt: z.strictObject({ + version: z.string().min(1), + kind: z.literal("mteam"), + displayName: z.string(), + apiBaseUrl: z.string().url().optional(), + timezone: TimezoneSchema, + rateLimit: RateLimitSchema, + capabilities: CapabilitiesSchema, + categoryNames: z.record(z.string(), z.string()), + authSupported: z.array(z.enum(["cookie", "credential", "apikey"])).optional(), + customClass: z.string().optional(), + }), +}); + +/** MP 专用 spider(`SPIDER_PARSER_CLASSES` 中除 mTorrent 外的 7 个)站点定义 */ +export const SpecializedSiteDefinitionSchema = z.strictObject({ + ...BaseDefinitionShape, + parser: z.enum(["Haidan", "HDDolby", "RousiPro", "SunnyPT", "TNodeSpider", "TorrentLeech", "Yema"]), + public: z.boolean().optional(), + category: CategoryConfigSchema.optional(), + api_url: z.string().url().optional(), + umt: z.strictObject({ + version: z.string().min(1), + kind: z.literal("api"), + displayName: z.string(), + timezone: TimezoneSchema, + rateLimit: RateLimitSchema, + capabilities: CapabilitiesSchema, + categoryNames: z.record(z.string(), z.string()).optional(), + apiBaseUrl: z.string().url().optional(), + }), +}); + +function formatIssues(id: unknown, e: z.ZodError): string { + const issues = e.issues.map((i) => `${i.path.join(".") || ""}: ${i.message}`).join("; "); + return `站点定义 ${typeof id === "string" ? id : ""} 校验失败:${issues}`; +} + +/** NexusPHP(通用 SiteSpider)定义校验 */ +export function parseIndexerDefinition(json: unknown): IndexerDefinition { + const r = IndexerDefinitionSchema.safeParse(json); + if (!r.success) throw new Error(formatIssues((json as { id?: unknown })?.id, r.error)); + return r.data as IndexerDefinition; +} + +/** 馒头(M-Team API)定义校验 */ +export function parseMteamDefinition(json: unknown): MteamDefinition { + const r = MteamDefinitionSchema.safeParse(json); + if (!r.success) throw new Error(formatIssues((json as { id?: unknown })?.id, r.error)); + return r.data as MteamDefinition; +} + +/** 专用 API spider 定义校验 */ +export function parseSpecializedSiteDefinition(json: unknown): SpecializedSiteDefinition { + const r = SpecializedSiteDefinitionSchema.safeParse(json); + if (!r.success) throw new Error(formatIssues((json as { id?: unknown })?.id, r.error)); + return r.data as SpecializedSiteDefinition; +} diff --git a/packages/adapters/src/sites/definitions/soulvoice.json b/packages/adapters/src/sites/definitions/soulvoice.json new file mode 100644 index 00000000..cb2b755c --- /dev/null +++ b/packages/adapters/src/sites/definitions/soulvoice.json @@ -0,0 +1,238 @@ +{ + "id": "soulvoice", + "name": "聆音", + "enabled": false, + "schema": "NexusPhp", + "provenance": { + "source": "movieclaw", + "ref": "v0.27.0", + "commit": "c79bb1b0", + "license": "MIT", + "file": "src/movieclaw_tracker/sites/configs/soulvoice.yaml" + }, + "domain": "https://pt.soulvoice.club/", + "encoding": "UTF-8", + "public": false, + "proxy": false, + "language": "zh", + "result_num": 100, + "timeout": 30, + "search": { + "paths": [ + { + "path": "torrents.php", + "type": "all" + } + ], + "params": { + "search": "{keyword}" + } + }, + "batch": { + "delimiter": " ", + "space_replace": "_" + }, + "browse": { + "path": "torrents.php?page={page}", + "start": 0 + }, + "category": { + "movie": [ + { + "id": "401", + "desc": "电影" + }, + { + "id": "424", + "desc": "电影" + } + ], + "tv": [ + { + "id": "402", + "desc": "剧集" + }, + { + "id": "403", + "desc": "剧集" + } + ], + "music": [ + { + "id": "406", + "desc": "音乐" + }, + { + "id": "408", + "desc": "音乐" + } + ] + }, + "torrents": { + "list": { + "selector": "table.torrents > tr, table.torrents > tbody > tr" + }, + "fields": { + "title_default": { + "selector": "table.torrentname a b" + }, + "title_optional": { + "selector": "table.torrentname a b[title]", + "attribute": "title" + }, + "title": { + "text": "{% if fields['title_optional'] %}{{ fields['title_optional'] }}{% else %}{{ fields['title_default'] }}{% endif %}" + }, + "details": { + "selector": "a[href*='details.php']", + "attribute": "href" + }, + "download": { + "selector": "a[href*='download.php']", + "attribute": "href" + }, + "imdbid": { + "selector": "a[href*=\"imdb.com/title/tt\"]", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["tt\\d+", 0] + } + ] + }, + "date_elapsed": { + "selector": "td:nth-child(4)" + }, + "date_added": { + "selector": "td:nth-child(4) > span", + "attribute": "title" + }, + "date": { + "text": "{% if fields['date_elapsed'] or fields['date_added'] %}{{ fields['date_added'] if fields['date_added'] else fields['date_elapsed'] }}{% else %}now{% endif %}", + "filters": [ + { + "name": "dateparse", + "args": "%Y-%m-%d %H:%M:%S" + } + ] + }, + "size": { + "selector": "td:nth-child(5)" + }, + "seeders": { + "selector": "td:nth-child(6)" + }, + "leechers": { + "selector": "td:nth-child(7)" + }, + "grabs": { + "selector": "td:nth-child(8)" + }, + "downloadvolumefactor": { + "case": { + "img.pro_free": 0, + "*": 1 + } + }, + "uploadvolumefactor": { + "case": { + "*": 1 + } + }, + "description": { + "selector": "td.embedded br" + }, + "category": { + "selector": "td:nth-child(1) a[href*='cat=']", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["\\d+", 0] + } + ] + } + } + }, + "umt": { + "version": "soulvoice@mc-v0.27.0", + "kind": "nexusphp", + "displayName": "聆音", + "timezone": "+08:00", + "rateLimit": { + "minIntervalMs": 5000, + "burst": 1 + }, + "capabilities": { + "list": true, + "search": true, + "detail": true, + "download": true, + "rss": false + }, + "categoryNames": { + "401": "电影", + "402": "剧集", + "403": "剧集", + "404": "纪录片", + "405": "动漫", + "406": "音乐", + "407": "其他", + "408": "音乐", + "409": "其他", + "424": "电影", + "499": "其他", + "500": "其他" + }, + "loginSuccess": "a[href*='logout']", + "emptyMarkers": ["没有种子", "没有结果", "Nothing found"], + "pagination": "a[href*=\"page=\"]", + "detailPath": "details.php", + "downloadPath": "download.php", + "authSupported": ["cookie", "credential"], + "login": { + "path": "login.php", + "usernameField": "username", + "passwordField": "password", + "errorCss": "td.text" + }, + "profile": { + "path": "userdetails.php", + "uid_css": "a[class*='_Name']::attr(href)", + "username_css": "a[class*='_Name'] b", + "class_css": "a[class*='_Name']::attr(class)", + "uploaded_css": "font.color_uploaded::next_sibling_text", + "downloaded_css": "font.color_downloaded::next_sibling_text", + "ratio_css": "font.color_ratio::next_sibling_text", + "bonus_css": "font.color_bonus", + "join_date_css": "td.rowhead:contains('加入日期') + td::text", + "seeding_css": "img.arrowup::next_sibling_text", + "leeching_css": "img.arrowdown::next_sibling_text" + }, + "detail": { + "path": "details.php", + "title": "h1#top", + "rowLabel": "td.rowhead", + "rowValue": "td.rowfollow", + "labels": { + "subtitle": ["副标题", "Small Description"], + "basicInfo": ["基本信息", "Basic Info"], + "peers": ["同伴", "Peers"] + }, + "description": "#kdescr", + "headerScope": "h1#top", + "notFoundMarkers": ["没有该ID的种子", "No torrent with ID"], + "fields": { + "title": "h1#top", + "subtitle": "td.rowhead:contains('副标题') + td::text", + "size": "td.rowhead:contains('基本信息') + td.rowfollow > b::next_sibling_text", + "description": "#kdescr", + "download": "a[href*='download.php']::attr(href)", + "imdb": "a[href*='imdb.com']::attr(href)", + "douban": "a[href*='douban.com']::attr(href)", + "fileList": "td.rowfollow > ul > li::text" + } + } + } +} diff --git a/packages/adapters/src/sites/definitions/ssd.json b/packages/adapters/src/sites/definitions/ssd.json new file mode 100644 index 00000000..7bf77aec --- /dev/null +++ b/packages/adapters/src/sites/definitions/ssd.json @@ -0,0 +1,252 @@ +{ + "id": "ssd", + "name": "不可说", + "enabled": false, + "schema": "NexusPhp", + "provenance": { + "source": "movieclaw", + "ref": "v0.27.0", + "commit": "c79bb1b0", + "license": "MIT", + "file": "src/movieclaw_tracker/sites/configs/ssd.yaml" + }, + "domain": "https://springsunday.net/", + "encoding": "UTF-8", + "public": false, + "proxy": false, + "language": "zh", + "result_num": 100, + "timeout": 30, + "search": { + "paths": [ + { + "path": "torrents.php", + "type": "all" + } + ], + "params": { + "search": "{keyword}" + } + }, + "batch": { + "delimiter": " ", + "space_replace": "_" + }, + "browse": { + "path": "torrents.php?page={page}", + "start": 0 + }, + "category": { + "movie": [ + { + "id": "501", + "desc": "电影" + } + ], + "tv": [ + { + "id": "502", + "desc": "剧集" + }, + { + "id": "505", + "desc": "剧集" + }, + { + "id": "506", + "desc": "剧集" + } + ], + "music": [ + { + "id": "507", + "desc": "音乐" + }, + { + "id": "508", + "desc": "音乐" + } + ] + }, + "torrents": { + "list": { + "selector": "table.torrents > tr:has(table.torrentname)" + }, + "fields": { + "title_default": { + "selector": "a[href*='details.php']" + }, + "title_optional": { + "selector": "a[href*='details.php'][title]", + "attribute": "title" + }, + "title": { + "text": "{% if fields['title_optional'] %}{{ fields['title_optional'] }}{% else %}{{ fields['title_default'] }}{% endif %}" + }, + "details": { + "selector": "a[href*='details.php']", + "attribute": "href" + }, + "download": { + "selector": "a[href*='download.php']", + "attribute": "href" + }, + "imdbid": { + "selector": "a[href*=\"imdb.com/title/tt\"]", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["tt\\d+", 0] + } + ] + }, + "date_elapsed": { + "selector": "td.rowfollow:nth-last-child(6)" + }, + "date_added": { + "selector": "td.rowfollow:nth-last-child(6) > span[title]", + "attribute": "title" + }, + "date": { + "text": "{% if fields['date_elapsed'] or fields['date_added'] %}{{ fields['date_added'] if fields['date_added'] else fields['date_elapsed'] }}{% else %}now{% endif %}", + "filters": [ + { + "name": "dateparse", + "args": "%Y-%m-%d %H:%M:%S" + } + ] + }, + "size": { + "selector": "td.rowfollow:nth-last-child(5)" + }, + "seeders": { + "selector": "td.rowfollow:nth-last-child(4)" + }, + "leechers": { + "selector": "td.rowfollow:nth-last-child(3)" + }, + "grabs": { + "selector": "td.rowfollow:nth-last-child(2)" + }, + "downloadvolumefactor": { + "case": { + "span.torrent-pro-free": 0, + "span.torrent-pro-free2up": 0, + "span.torrent-pro-50pctdown": 0.5, + "span.torrent-pro-50pctdown2up": 0.5, + "span.torrent-pro-30pctdown": 0.3, + "*": 1 + } + }, + "uploadvolumefactor": { + "case": { + "span.torrent-pro-free2up": 2, + "span.torrent-pro-50pctdown2up": 2, + "span.torrent-pro-2up": 2, + "*": 1 + } + }, + "description": { + "selector": "div.torrent-smalldescr > span:not([class])" + }, + "category": { + "selector": "a[href^='?cat=']", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["\\d+", 0] + } + ] + }, + "freedate": { + "selector": "span.torrent-pro-free ~ span > span[title]", + "attribute": "title", + "filters": [ + { + "name": "re_search", + "args": ["\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}", 0] + } + ] + } + } + }, + "umt": { + "version": "ssd@mc-v0.27.0", + "kind": "nexusphp", + "displayName": "不可说", + "timezone": "+08:00", + "rateLimit": { + "minIntervalMs": 3000, + "burst": 1 + }, + "capabilities": { + "list": true, + "search": true, + "detail": true, + "download": true, + "rss": false + }, + "categoryNames": { + "501": "电影", + "502": "剧集", + "503": "纪录片", + "504": "动漫", + "505": "剧集", + "506": "剧集", + "507": "音乐", + "508": "音乐", + "509": "其他" + }, + "loginSuccess": "a[href^='logout.php']", + "emptyMarkers": ["没有种子", "没有结果", "Nothing found"], + "pagination": "a[href*=\"page=\"]", + "detailPath": "details.php", + "downloadPath": "download.php", + "login": { + "path": "login.php", + "usernameField": "username", + "passwordField": "password", + "errorCss": "td.text" + }, + "profile": { + "path": "userdetails.php", + "uid_css": "a[href^='userdetails.php?id=']::attr(href)", + "username_css": "a[href^='userdetails.php?id=']", + "class_css": "a[href^='userdetails.php?id=']::attr(class)", + "uploaded_css": "font.color_uploaded::next_sibling_text", + "downloaded_css": "font.color_downloaded::next_sibling_text", + "ratio_css": "font.color_ratio::next_sibling_text", + "bonus_css": "td.rowhead:contains('魔力') + td::text", + "join_date_css": "td.rowhead:contains('加入日期') + td::text", + "seeding_css": "img[alt='Torrents seeding'] + *", + "leeching_css": "img[alt='Torrents leeching'] + *", + "vip_css": "a[class^='VIP']" + }, + "detail": { + "path": "details.php", + "title": "#top", + "rowLabel": "td.rowhead", + "rowValue": "td.rowfollow", + "labels": { + "subtitle": ["副标题", "Small Description"], + "basicInfo": ["基本信息", "Basic Info"], + "peers": ["同伴", "Peers"] + }, + "description": "#kdescr", + "headerScope": "#top", + "notFoundMarkers": ["没有该ID的种子", "No torrent with ID"], + "fields": { + "title": "#top::text", + "subtitle": "td.rowhead:contains('副标题') + td::text", + "size": "td.rowhead:contains('大小') + td::text", + "description": "#kdescr", + "download": "a[href*='download.php']::attr(href)", + "imdb": "a[href*='imdb.com']::attr(href)", + "douban": "a[href*='douban.com']::attr(href)", + "fileList": "td.rowfollow > ul > li::text" + } + } + } +} diff --git a/packages/adapters/src/sites/definitions/tjupt.json b/packages/adapters/src/sites/definitions/tjupt.json new file mode 100644 index 00000000..40fb745f --- /dev/null +++ b/packages/adapters/src/sites/definitions/tjupt.json @@ -0,0 +1,198 @@ +{ + "id": "tjupt", + "name": "北洋园", + "enabled": false, + "schema": "NexusPhp", + "provenance": { + "source": "movieclaw", + "ref": "v0.27.0", + "commit": "c79bb1b0", + "license": "MIT", + "file": "src/movieclaw_tracker/sites/configs/tjupt.yaml" + }, + "domain": "https://tjupt.org/", + "encoding": "UTF-8", + "public": false, + "proxy": false, + "language": "zh", + "result_num": 100, + "timeout": 30, + "search": { + "paths": [ + { + "path": "torrents.php", + "type": "all" + } + ], + "params": { + "search": "{keyword}" + } + }, + "batch": { + "delimiter": " ", + "space_replace": "_" + }, + "browse": { + "path": "torrents.php?page={page}", + "start": 1 + }, + "category": { + "movie": [], + "tv": [], + "music": [] + }, + "torrents": { + "list": { + "selector": "table.torrents > tr:not(:first-child)" + }, + "fields": { + "title_default": { + "selector": ":scope > td:nth-child(2) a[href*='details.php']" + }, + "title_optional": { + "selector": ":scope > td:nth-child(2) a[href*='details.php'][title]", + "attribute": "title" + }, + "title": { + "text": "{% if fields['title_optional'] %}{{ fields['title_optional'] }}{% else %}{{ fields['title_default'] }}{% endif %}" + }, + "details": { + "selector": "a[href*='details.php']", + "attribute": "href" + }, + "download": { + "selector": "a[href*='download.php']", + "attribute": "href" + }, + "imdbid": { + "selector": "a[href*=\"imdb.com/title/tt\"]", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["tt\\d+", 0] + } + ] + }, + "date_added": { + "selector": ":scope > td:nth-child(4)" + }, + "date": { + "text": "{% if fields['date_elapsed'] or fields['date_added'] %}{{ fields['date_added'] if fields['date_added'] else fields['date_elapsed'] }}{% else %}now{% endif %}", + "filters": [ + { + "name": "dateparse", + "args": "%Y-%m-%d %H:%M:%S" + } + ] + }, + "size": { + "selector": ":scope > td:nth-child(5)" + }, + "seeders": { + "selector": ":scope > td:nth-child(6)" + }, + "leechers": { + "selector": ":scope > td:nth-child(7)" + }, + "grabs": { + "selector": ":scope > td:nth-child(8)" + }, + "downloadvolumefactor": { + "case": { + "img.pro_free": 0, + "img.pro_free2up": 0, + "img.pro_50pctdown": 0.5, + "img.pro_50pctdown2up": 0.5, + "img.pro_30pctdown": 0.3, + "*": 1 + } + }, + "uploadvolumefactor": { + "case": { + "img.pro_free2up": 2, + "img.pro_50pctdown2up": 2, + "img.pro_2up": 2, + "*": 1 + } + }, + "description": { + "selector": "td.embedded > span" + }, + "category": { + "selector": ":scope > td:nth-child(1) a", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["\\d+", 0] + } + ] + } + } + }, + "umt": { + "version": "tjupt@mc-v0.27.0", + "kind": "nexusphp", + "displayName": "北洋园", + "timezone": "+08:00", + "rateLimit": { + "minIntervalMs": 5000, + "burst": 1 + }, + "capabilities": { + "list": true, + "search": true, + "detail": true, + "download": true, + "rss": false + }, + "categoryNames": {}, + "loginSuccess": "a[href*='logout.php']", + "emptyMarkers": ["没有种子", "没有结果", "Nothing found"], + "pagination": "a[href*=\"page=\"]", + "detailPath": "details.php", + "downloadPath": "download.php", + "authSupported": ["cookie", "credential"], + "login": { + "path": "login.php", + "usernameField": "username", + "passwordField": "password", + "errorCss": "td.text", + "selectDefaults": [["logout", "7"]] + }, + "profile": { + "path": "userdetails.php", + "uid_css": "a.User_Name::attr(href)", + "username_css": "a.User_Name b", + "class_css": "td.rowhead:contains('等级') + td img::attr(title)", + "uploaded_css": "td.rowhead:contains('上传量') + td::text", + "bonus_css": "td.rowhead:contains('魔力值') + td", + "join_date_css": "td.rowhead:contains('加入日期') + td::text" + }, + "detail": { + "path": "details.php", + "title": "#top", + "rowLabel": "td.rowhead", + "rowValue": "td.rowfollow", + "labels": { + "subtitle": ["副标题", "Small Description"], + "basicInfo": ["基本信息", "Basic Info"], + "peers": ["同伴", "Peers"] + }, + "description": "#kdescr", + "headerScope": "#top", + "notFoundMarkers": ["没有该ID的种子", "No torrent with ID"], + "fields": { + "title": "#top::text", + "subtitle": "td.rowhead:contains('副标题') + td::text", + "size": "td.rowhead:contains('大小') + td::text", + "description": "#kdescr", + "download": "a[href*='download.php']::attr(href)", + "imdb": "a[href*='imdb.com']::attr(href)", + "douban": "a[href*='douban.com']::attr(href)", + "fileList": "td.rowfollow > ul > li::text" + } + } + } +} diff --git a/packages/adapters/src/sites/definitions/ttg.json b/packages/adapters/src/sites/definitions/ttg.json new file mode 100644 index 00000000..9b2e6480 --- /dev/null +++ b/packages/adapters/src/sites/definitions/ttg.json @@ -0,0 +1,340 @@ +{ + "id": "ttg", + "name": "听听歌", + "enabled": false, + "schema": "NexusPhp", + "provenance": { + "source": "movieclaw", + "ref": "v0.27.0", + "commit": "c79bb1b0", + "license": "MIT", + "file": "src/movieclaw_tracker/sites/configs/ttg.yaml" + }, + "domain": "https://totheglory.im/", + "encoding": "UTF-8", + "public": false, + "proxy": false, + "language": "zh", + "result_num": 100, + "timeout": 30, + "search": { + "paths": [ + { + "path": "browse.php", + "type": "all" + } + ], + "params": { + "search": "{keyword}" + } + }, + "batch": { + "delimiter": " ", + "space_replace": "_" + }, + "browse": { + "path": "browse.php?page={page}", + "start": 1 + }, + "category": { + "movie": [ + { + "id": "电影DVDRip", + "desc": "电影" + }, + { + "id": "电影720p", + "desc": "电影" + }, + { + "id": "电影1080i/p", + "desc": "电影" + }, + { + "id": "BluRay原盘", + "desc": "电影" + }, + { + "id": "影视2160p", + "desc": "电影" + }, + { + "id": "UHD原盘", + "desc": "电影" + }, + { + "id": "iPhone/iPad视频", + "desc": "电影" + } + ], + "tv": [ + { + "id": "欧美剧720p", + "desc": "剧集" + }, + { + "id": "欧美剧1080i/p", + "desc": "剧集" + }, + { + "id": "高清日剧", + "desc": "剧集" + }, + { + "id": "大陆港台剧720p", + "desc": "剧集" + }, + { + "id": "高清韩剧", + "desc": "剧集" + }, + { + "id": "大陆港台剧1080i/p", + "desc": "剧集" + }, + { + "id": "欧美剧包", + "desc": "剧集" + }, + { + "id": "日剧包", + "desc": "剧集" + }, + { + "id": "华语剧包", + "desc": "剧集" + }, + { + "id": "韩剧包", + "desc": "剧集" + }, + { + "id": "高清体育节目", + "desc": "剧集" + }, + { + "id": "韩国综艺", + "desc": "剧集" + }, + { + "id": "日本综艺", + "desc": "剧集" + }, + { + "id": "高清综艺", + "desc": "剧集" + } + ], + "music": [ + { + "id": "(电影原声&Game)OST", + "desc": "音乐" + }, + { + "id": "无损音乐FLAC&APE", + "desc": "音乐" + }, + { + "id": "MV&演唱会", + "desc": "音乐" + }, + { + "id": "补充音轨", + "desc": "音乐" + } + ] + }, + "torrents": { + "list": { + "selector": "table#torrent_table > tr[id]" + }, + "fields": { + "title_default": { + "selector": "a[href*='details.php']" + }, + "title_optional": { + "selector": "a[href*='details.php'][title]", + "attribute": "title" + }, + "title": { + "text": "{% if fields['title_optional'] %}{{ fields['title_optional'] }}{% else %}{{ fields['title_default'] }}{% endif %}" + }, + "details": { + "selector": "a[href*='details.php']", + "attribute": "href" + }, + "download": { + "selector": "a[href*='download.php']", + "attribute": "href" + }, + "imdbid": { + "selector": "a[href*=\"imdb.com/title/tt\"]", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["tt\\d+", 0] + } + ] + }, + "date_added": { + "selector": "td:nth-child(5)" + }, + "date": { + "text": "{% if fields['date_elapsed'] or fields['date_added'] %}{{ fields['date_added'] if fields['date_added'] else fields['date_elapsed'] }}{% else %}now{% endif %}", + "filters": [ + { + "name": "dateparse", + "args": "%Y-%m-%d %H:%M:%S" + } + ] + }, + "size": { + "selector": "td:nth-child(5)" + }, + "seeders": { + "selector": "td:nth-child(6)" + }, + "leechers": { + "selector": "td:nth-child(7)" + }, + "grabs": { + "selector": "td:nth-child(8)" + }, + "downloadvolumefactor": { + "case": { + "img[alt=\"free\"]": 0, + "img[alt=\"50%\"]": 0.5, + "img[alt=\"30%\"]": 0.3, + "*": 1 + } + }, + "uploadvolumefactor": { + "case": { + "img[alt=\"200%\"]": 2, + "*": 1 + } + }, + "description": { + "selector": "td.embedded > span" + }, + "category": { + "selector": "td:nth-child(1) img", + "attribute": "class", + "filters": [ + { + "name": "re_search", + "args": ["\\d+", 0] + } + ] + }, + "freedate": { + "selector": "span[onclick]", + "attribute": "onclick", + "filters": [ + { + "name": "re_search", + "args": ["\\d+年\\d+月\\d+日\\d+点\\d+分", 0] + } + ] + } + } + }, + "umt": { + "version": "ttg@mc-v0.27.0", + "kind": "nexusphp", + "displayName": "听听歌", + "timezone": "+08:00", + "rateLimit": { + "minIntervalMs": 3000, + "burst": 1 + }, + "capabilities": { + "list": true, + "search": true, + "detail": true, + "download": true, + "rss": false + }, + "categoryNames": { + "电影DVDRip": "电影", + "电影720p": "电影", + "电影1080i/p": "电影", + "BluRay原盘": "电影", + "影视2160p": "电影", + "UHD原盘": "电影", + "iPhone/iPad视频": "电影", + "纪录片720p": "纪录片", + "纪录片1080i/p": "纪录片", + "纪录片BluRay原盘": "纪录片", + "欧美剧720p": "剧集", + "欧美剧1080i/p": "剧集", + "高清日剧": "剧集", + "大陆港台剧720p": "剧集", + "高清韩剧": "剧集", + "大陆港台剧1080i/p": "剧集", + "欧美剧包": "剧集", + "日剧包": "剧集", + "华语剧包": "剧集", + "韩剧包": "剧集", + "高清体育节目": "剧集", + "韩国综艺": "剧集", + "日本综艺": "剧集", + "高清综艺": "剧集", + "(电影原声&Game)OST": "音乐", + "无损音乐FLAC&APE": "音乐", + "MV&演唱会": "音乐", + "补充音轨": "音乐", + "高清动漫": "动漫", + "MiniVideo": "其他" + }, + "loginSuccess": "a[href='/logout.php']", + "emptyMarkers": ["没有种子", "没有结果", "Nothing found"], + "pagination": "a[href*=\"page=\"]", + "detailPath": "details.php", + "downloadPath": "download.php", + "login": { + "path": "login.php", + "usernameField": "username", + "passwordField": "password", + "errorCss": "td.text" + }, + "profile": { + "path": "userdetails.php", + "username_css": "h1::text", + "class_css": "td.rowhead:contains('等级') + td::text", + "uploaded_css": "td.rowhead:contains('上传量') + td::text", + "downloaded_css": "td.rowhead:contains('下载量') + td::text", + "ratio_css": "td.rowhead:contains('分享率') + td::text", + "bonus_css": "td.rowhead:contains('魔力') + td::text", + "join_date_css": "td.rowhead:contains('加入日期') + td::text", + "seeding_css": "td.rowhead:contains('当前做种') + td::text", + "leeching_css": "td.rowhead:contains('当前下载') + td::text" + }, + "detail": { + "path": "details.php", + "title": "#top", + "rowLabel": "td.rowhead", + "rowValue": "td.rowfollow", + "labels": { + "subtitle": ["副标题", "Small Description"], + "basicInfo": ["基本信息", "Basic Info"], + "peers": ["同伴", "Peers"] + }, + "description": "#kdescr", + "headerScope": "#top", + "notFoundMarkers": ["没有该ID的种子", "No torrent with ID"], + "fields": { + "title": "#top::text", + "subtitle": "td.rowhead:contains('副标题') + td::text", + "size": "td.rowhead:contains('大小') + td::text", + "description": "#kdescr", + "download": "a[href*='download.php']::attr(href)", + "imdb": "a[href*='imdb.com']::attr(href)", + "douban": "a[href*='douban.com']::attr(href)", + "fileList": "td.rowfollow > ul > li::text" + } + }, + "customClass": "movieclaw_tracker.sites.custom.ttg.TTGSite" + } +} diff --git a/packages/adapters/src/sites/deps.ts b/packages/adapters/src/sites/deps.ts index 86f4fda0..57bd3ed9 100644 --- a/packages/adapters/src/sites/deps.ts +++ b/packages/adapters/src/sites/deps.ts @@ -1,6 +1,6 @@ -import type { SiteHttpContext } from "./http"; -import { createSiteRateLimiter, type SiteRateLimiter } from "./rate-limit"; -import type { SecretResolver } from "./types"; +import type { SiteHttpContext } from "./http.ts"; +import { createSiteRateLimiter, type SiteRateLimiter } from "./rate-limit.ts"; +import type { SecretResolver } from "./types.ts"; export interface SiteAdapterDeps { secrets: SecretResolver; diff --git a/packages/adapters/src/sites/http.ts b/packages/adapters/src/sites/http.ts index 8072e5b5..33c88156 100644 --- a/packages/adapters/src/sites/http.ts +++ b/packages/adapters/src/sites/http.ts @@ -1,6 +1,6 @@ -import { redactText } from "../rss/redact"; -import type { SiteRateLimiter } from "./rate-limit"; -import { SiteError } from "./types"; +import { redactText } from "../rss/redact.ts"; +import type { SiteRateLimiter } from "./rate-limit.ts"; +import { SiteError } from "./types.ts"; export interface SiteHttpContext { siteId: string; diff --git a/packages/adapters/src/sites/index.ts b/packages/adapters/src/sites/index.ts index e04b66a7..57cccdc8 100644 --- a/packages/adapters/src/sites/index.ts +++ b/packages/adapters/src/sites/index.ts @@ -1,8 +1,24 @@ -export { audiencesDefinition, loadIndexerDefinition } from "./definitions"; -export type { SiteAdapterDeps } from "./deps"; -export { createMteamAdapter, mteamTorrentToRelease } from "./mteam/adapter"; -export { type MteamApiPath, mteamPost } from "./mteam/client"; -export { type MteamDefinition, mteamApiBase, mteamDefinition } from "./mteam/definition"; +export { + audiencesDefinition, + BUILTIN_SITE_DEFINITIONS, + chdbitsDefinition, + DefinitionProvenanceSchema, + hhanclubDefinition, + IndexerDefinitionSchema, + loadIndexerDefinition, + loadSiteDefinition, + MteamDefinitionSchema, + parseIndexerDefinition, + parseMteamDefinition, + parseSpecializedSiteDefinition, + SpecializedSiteDefinitionSchema, + SpiderFieldSelectorSchema, + SpiderFilterSchema, +} from "./definitions/index.ts"; +export type { SiteAdapterDeps } from "./deps.ts"; +export { createMteamAdapter, mteamTorrentToRelease } from "./mteam/adapter.ts"; +export { type MteamApiPath, mteamPost } from "./mteam/client.ts"; +export { type MteamDefinition, mteamApiBase, mteamDefinition } from "./mteam/definition.ts"; export { findImdbid, getDownloadVolumeFactor, @@ -10,13 +26,22 @@ export { type MteamTorrentJson, mteamSearchParams, parseMteamResult, -} from "./mteam/spider"; -export { createSiteRateLimiter, type RateLimiterOptions, type SiteRateLimiter } from "./rate-limit"; -export { createSiteAdapter, getSiteAdapter, SITE_DEFINITIONS, type SiteDefinition } from "./registry"; -export { clearSearchText, createSpiderSiteAdapter } from "./spider/adapter"; -export { parseDetailPage } from "./spider/detail"; -export { promotionFromFactors, spiderTorrentToRelease } from "./spider/map"; +} from "./mteam/spider.ts"; +export { createSiteRateLimiter, type RateLimiterOptions, type SiteRateLimiter } from "./rate-limit.ts"; +export { + createSiteAdapter, + getSiteAdapter, + getSiteDefinition, + listSiteDefinitions, + SITE_DEFINITIONS, + type SiteDefinition, +} from "./registry.ts"; +export { clearSearchText, createSpiderSiteAdapter } from "./spider/adapter.ts"; +export { parseDetailPage } from "./spider/detail.ts"; +export { promotionFromFactors, spiderTorrentToRelease } from "./spider/map.ts"; +export { mcSelect } from "./spider/mc-selector.ts"; export type { + DefinitionProvenance, IndexerDefinition, SpiderCategoryConfig, SpiderCategoryItem, @@ -24,7 +49,7 @@ export type { SpiderFields, SpiderFilter, UmtSiteExtension, -} from "./spider/schema"; +} from "./spider/schema.ts"; export { combineUrl, type MediaKind, @@ -35,7 +60,8 @@ export { type SiteSpiderQuery, type SpiderTorrent, selectMediaCategories, -} from "./spider/spider"; +} from "./spider/spider.ts"; +export * from "./spiders/index.ts"; export { type DownloadTarget, type HitAndRun, @@ -57,5 +83,6 @@ export { type SiteRelease, type SiteReleaseDetail, staticSecretResolver, -} from "./types"; -export { parseCount, parseSiteTime } from "./util"; +} from "./types.ts"; +export * from "./user-parsers/index.ts"; +export { parseCount, parseSiteTime } from "./util.ts"; diff --git a/packages/adapters/src/sites/mteam/adapter.ts b/packages/adapters/src/sites/mteam/adapter.ts index 218f3d91..eab984e0 100644 --- a/packages/adapters/src/sites/mteam/adapter.ts +++ b/packages/adapters/src/sites/mteam/adapter.ts @@ -9,12 +9,12 @@ * - `app/modules/indexer/parser/mtorrent.py`(api/member/profile)→ `healthCheck` * 详情 `/api/torrent/detail` 为本项目扩展(MP spider 无详情调用,字段结构参考 MovieClaw mteam.py,data-ref)。 */ -import { redactUrl } from "../../rss/redact"; -import { buildHttpContext, type SiteAdapterDeps } from "../deps"; -import { readBody, siteFetch } from "../http"; -import { healthFromError } from "../spider/adapter"; -import { promotionFromFactors } from "../spider/map"; -import type { SpiderTorrent } from "../spider/spider"; +import { redactUrl } from "../../rss/redact.ts"; +import { buildHttpContext, type SiteAdapterDeps } from "../deps.ts"; +import { readBody, siteFetch } from "../http.ts"; +import { healthFromError } from "../spider/adapter.ts"; +import { promotionFromFactors } from "../spider/map.ts"; +import type { SpiderTorrent } from "../spider/spider.ts"; import { type DownloadTarget, type SiteAdapter, @@ -24,11 +24,11 @@ import { type SitePage, type SiteRelease, type SiteReleaseDetail, -} from "../types"; -import { extractDoubanId, inspectTorrent, parseSiteTime } from "../util"; -import { mteamPost } from "./client"; -import { type MteamDefinition, mteamApiBase } from "./definition"; -import { type MteamTorrentJson, mteamSearchParams, parseMteamResult } from "./spider"; +} from "../types.ts"; +import { extractDoubanId, inspectTorrent, parseSiteTime } from "../util.ts"; +import { mteamPost } from "./client.ts"; +import { type MteamDefinition, mteamApiBase } from "./definition.ts"; +import { type MteamTorrentJson, mteamSearchParams, parseMteamResult } from "./spider.ts"; interface MteamSearchData { data?: MteamTorrentJson[] | null; diff --git a/packages/adapters/src/sites/mteam/client.ts b/packages/adapters/src/sites/mteam/client.ts index 3fd03895..8cf57fad 100644 --- a/packages/adapters/src/sites/mteam/client.ts +++ b/packages/adapters/src/sites/mteam/client.ts @@ -2,9 +2,9 @@ * M-Team API 调用。对应 MP `MTorrentSpider.search/__process_response`(MP 只看 HTTP 200); * 业务 code/message 检查与凭据/解析错误分层参考 MovieClaw `_post_api`(behavior-ref,迁移账本 MT2)。 */ -import { redactText } from "../../rss/redact"; -import { readText, type SiteHttpContext, siteFetch } from "../http"; -import { SiteError } from "../types"; +import { redactText } from "../../rss/redact.ts"; +import { readText, type SiteHttpContext, siteFetch } from "../http.ts"; +import { SiteError } from "../types.ts"; export type MteamApiPath = | "/api/torrent/search" diff --git a/packages/adapters/src/sites/mteam/definition.ts b/packages/adapters/src/sites/mteam/definition.ts index 94cfb87b..8b814357 100644 --- a/packages/adapters/src/sites/mteam/definition.ts +++ b/packages/adapters/src/sites/mteam/definition.ts @@ -1,4 +1,7 @@ -import type { SiteCapabilities } from "../types"; +import mteamJson from "../definitions/mteam.json" with { type: "json" }; +import { parseMteamDefinition } from "../definitions/schema.ts"; +import type { DefinitionProvenance } from "../spider/schema.ts"; +import type { SiteCapabilities } from "../types.ts"; /** * 馒头站点定义:键名沿用 MP indexer dict(id/name/domain/parser/timeout), @@ -7,6 +10,10 @@ import type { SiteCapabilities } from "../types"; export interface MteamDefinition { id: string; name: string; + enabled?: boolean; + /** MP 用户信息解析器 schema(MTorrent) */ + schema?: string; + provenance?: DefinitionProvenance; /** 网页域(MP `_url`,详情页 = `${domain}detail/${id}`) */ domain: string; parser: "mTorrent"; @@ -22,41 +29,16 @@ export interface MteamDefinition { capabilities: SiteCapabilities; /** 原始分类 ID → 名称(自行编制,待核验) */ categoryNames: Record; + authSupported?: string[]; + customClass?: string; }; } -export const mteamDefinition: MteamDefinition = { - id: "mteam", - name: "馒头", - domain: "https://tp.m-team.cc/", - parser: "mTorrent", - timeout: 15, - umt: { - version: "mteam@0.2.0-unverified", - kind: "mteam", - displayName: "馒头", - timezone: "+08:00", - rateLimit: { minIntervalMs: 1000, burst: 1 }, - capabilities: { list: true, search: true, detail: true, download: true, rss: false }, - categoryNames: { - "401": "电影", - "419": "电影", - "420": "电影", - "421": "电影", - "439": "电影", - "402": "剧集", - "403": "综艺", - "435": "剧集", - "438": "剧集", - "404": "纪录", - "405": "动画", - "407": "体育", - "406": "音乐", - "408": "音乐", - "434": "音乐", - }, - }, -}; +/** + * 馒头定义:definitions/mteam.json(由 scripts/mp-sync/movieclaw-import.mjs 从 MovieClaw v0.27.0 mteam.yaml 转换, + * W1-3b 原值经 movieclaw-overrides/mteam.json 覆盖),加载时 zod 校验。 + */ +export const mteamDefinition: MteamDefinition = parseMteamDefinition(mteamJson); /** MP `site_rules.extract_domain` 的简化:取主机名最后两级。 */ export function rootDomain(url: string): string { diff --git a/packages/adapters/src/sites/mteam/mteam.test.ts b/packages/adapters/src/sites/mteam/mteam.test.ts index a410dac4..86cefafe 100644 --- a/packages/adapters/src/sites/mteam/mteam.test.ts +++ b/packages/adapters/src/sites/mteam/mteam.test.ts @@ -9,7 +9,7 @@ import { SiteError, type SiteRateLimiter, staticSecretResolver, -} from "../index"; +} from "../index.ts"; let api: MockMteamApi | undefined; afterEach(async () => { diff --git a/packages/adapters/src/sites/mteam/spider.ts b/packages/adapters/src/sites/mteam/spider.ts index f9b3980c..412b0949 100644 --- a/packages/adapters/src/sites/mteam/spider.ts +++ b/packages/adapters/src/sites/mteam/spider.ts @@ -14,7 +14,7 @@ * 2. 额外输出 `category_id`(原始分类 ID),`category` 仍是 MP 的媒体类型投影。 * 3. 额外保留 `status` 原文到 raw,便于核验。 */ -import type { MediaKind, MpMediaType, SpiderTorrent } from "../spider/spider"; +import type { MediaKind, MpMediaType, SpiderTorrent } from "../spider/spider.ts"; export const MOVIE_CATEGORY = ["401", "419", "420", "421", "439", "405", "404"]; export const TV_CATEGORY = ["403", "402", "435", "438", "404", "405"]; @@ -87,10 +87,18 @@ export function getUploadVolumeFactor(discount: string | null | undefined): numb return discount ? (map[discount] ?? 1) : 1; } -/** MP `time_tools.format_timestamp`:非数字字符串原样返回,Unix 秒转本地时间(此处按站点时区交给调用方)。 */ -function formatTimestamp(v: string | null | undefined): string | undefined { +/** + * MP `time_tools.format_timestamp`:非纯数字字符串原样返回;Unix 秒 → 本地墙钟 "YYYY-MM-DD HH:MM:SS"。 + * MP 用进程本地时区(容器默认 Asia/Shanghai);这里固定 +08:00,与 adapter 按 `umt.timezone`(+08:00)转 UTC 配套。 + */ +function formatTimestamp(v: string | number | null | undefined): string | undefined { if (v == null) return undefined; - return String(v); + if (typeof v === "string" && !/^\d+$/.test(v)) return v; + const n = Number(v); + if (!Number.isFinite(n)) return String(v); + const d = new Date(Math.trunc(n) * 1000 + 8 * 3600_000); + const p = (x: number) => String(x).padStart(2, "0"); + return `${d.getUTCFullYear()}-${p(d.getUTCMonth() + 1)}-${p(d.getUTCDate())} ${p(d.getUTCHours())}:${p(d.getUTCMinutes())}:${p(d.getUTCSeconds())}`; } function toInt(v: unknown): number { diff --git a/packages/adapters/src/sites/rate-limit.test.ts b/packages/adapters/src/sites/rate-limit.test.ts index 5f5ea1ba..a92be361 100644 --- a/packages/adapters/src/sites/rate-limit.test.ts +++ b/packages/adapters/src/sites/rate-limit.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; -import { createSiteRateLimiter } from "./rate-limit"; -import { parseSizeMp as parseSize } from "./spider/spider"; -import { parseCount, parseSiteTime } from "./util"; +import { createSiteRateLimiter } from "./rate-limit.ts"; +import { parseSizeMp as parseSize } from "./spider/spider.ts"; +import { parseCount, parseSiteTime } from "./util.ts"; describe("站点令牌桶", () => { it("burst 内不等待,之后按间隔补充;penalize 顺延;并发按顺序排队", async () => { diff --git a/packages/adapters/src/sites/registry.ts b/packages/adapters/src/sites/registry.ts index 0caa4a8b..8ac23cc3 100644 --- a/packages/adapters/src/sites/registry.ts +++ b/packages/adapters/src/sites/registry.ts @@ -2,30 +2,49 @@ * 随产品发布的站点注册表(迁移账本 SA1)。 * 对应 MP `SitesHelper.get_indexers()/get_indexer(domain)`(闭源,sites.pyi)与 * `IndexerModule` 按 `indexer.parser` 分派专用解析器(`mTorrent` → MTorrentSpider,其余 → SiteSpider)。 - * 加站:新增 definitions/.json 并在此登记。 + * 加站:MovieClaw 来源重跑 scripts/mp-sync/movieclaw-import.mjs;其他来源新增 JSON 并登记到 definitions/。 */ -import { audiencesDefinition } from "./definitions"; -import type { SiteAdapterDeps } from "./deps"; -import { createMteamAdapter } from "./mteam/adapter"; -import { type MteamDefinition, mteamDefinition } from "./mteam/definition"; -import { createSpiderSiteAdapter } from "./spider/adapter"; -import type { IndexerDefinition } from "./spider/schema"; -import type { SiteAdapter } from "./types"; +import { BUILTIN_SITE_DEFINITIONS, type SiteDefinition } from "./definitions/index.ts"; +import type { SiteAdapterDeps } from "./deps.ts"; +import { createMteamAdapter } from "./mteam/adapter.ts"; +import type { MteamDefinition } from "./mteam/definition.ts"; +import { createSpiderSiteAdapter } from "./spider/adapter.ts"; +import type { IndexerDefinition } from "./spider/schema.ts"; +import { createSpecializedAdapterFor } from "./spiders/index.ts"; +import type { SpecializedSiteDefinition } from "./spiders/types.ts"; +import type { SiteAdapter } from "./types.ts"; -export type SiteDefinition = IndexerDefinition | MteamDefinition; +export type { SiteDefinition }; -export const SITE_DEFINITIONS: Record = { - audiences: audiencesDefinition, - mteam: mteamDefinition, -}; +/** 全部内置定义(含 enabled:false 的未实测站点) */ +export const SITE_DEFINITIONS: Readonly> = BUILTIN_SITE_DEFINITIONS; -function isMteam(def: SiteDefinition): def is MteamDefinition { - return (def as MteamDefinition).parser === "mTorrent"; +/** MP `SitesHelper.get_indexer`:按 id 取定义;未知返回 undefined */ +export function getSiteDefinition(id: string): SiteDefinition | undefined { + return SITE_DEFINITIONS[id]; } +/** MP `SitesHelper.get_indexers`:列出定义;默认含未启用站点,`enabledOnly` 只列 enabled !== false */ +export function listSiteDefinitions(opts: { enabledOnly?: boolean } = {}): SiteDefinition[] { + return Object.values(SITE_DEFINITIONS) + .filter((d) => !opts.enabledOnly || d.enabled !== false) + .sort((a, b) => a.id.localeCompare(b.id)); +} + +/** + * MP `IndexerModule` 按 `indexer.parser` 分派:mTorrent → 馒头适配器;SPIDER_PARSER_CLASSES 其余键(umt.kind=api) + * → 专用 spider 适配器;其余 → 通用 SiteSpider。 + */ export function createSiteAdapter(def: SiteDefinition, deps: SiteAdapterDeps): SiteAdapter { - return isMteam(def) ? createMteamAdapter(def, deps) : createSpiderSiteAdapter(def, deps); + switch (def.umt.kind) { + case "mteam": + return createMteamAdapter(def as MteamDefinition, deps); + case "api": + return createSpecializedAdapterFor(def as SpecializedSiteDefinition, deps); + default: + return createSpiderSiteAdapter(def as IndexerDefinition, deps); + } } export function getSiteAdapter(siteId: string, deps: SiteAdapterDeps): SiteAdapter { diff --git a/packages/adapters/src/sites/spider/adapter.ts b/packages/adapters/src/sites/spider/adapter.ts index 360cef24..32116d8e 100644 --- a/packages/adapters/src/sites/spider/adapter.ts +++ b/packages/adapters/src/sites/spider/adapter.ts @@ -11,9 +11,9 @@ * 本项目调整:登录页不再"返回空列表",而是抛 `auth_expired`(REQ §4.7.1 不展示空列表冒充无内容); * 3xx 不自动跟随(避免把 cookie 带到站外,并把跳转 login 识别为失效);凭证经 SecretResolver 注入。 */ -import { redactText } from "../../rss/redact"; -import { buildHttpContext, type SiteAdapterDeps } from "../deps"; -import { readBody, readText, siteFetch } from "../http"; +import { redactText } from "../../rss/redact.ts"; +import { buildHttpContext, type SiteAdapterDeps } from "../deps.ts"; +import { readBody, readText, siteFetch } from "../http.ts"; import { type DownloadTarget, type SiteAdapter, @@ -23,12 +23,12 @@ import { type SiteListFilters, type SitePage, type SiteReleaseDetail, -} from "../types"; -import { inspectTorrent } from "../util"; -import { parseDetailPage } from "./detail"; -import { spiderTorrentToRelease } from "./map"; -import type { IndexerDefinition } from "./schema"; -import { loadHtml, normalizeSelector, SiteSpider } from "./spider"; +} from "../types.ts"; +import { inspectTorrent } from "../util.ts"; +import { parseDetailPage } from "./detail.ts"; +import { spiderTorrentToRelease } from "./map.ts"; +import type { IndexerDefinition } from "./schema.ts"; +import { loadHtml, normalizeSelector, SiteSpider } from "./spider.ts"; /** MP `text_tools.remove_punctuation(text, replacement=" ", allow_space=True)` */ export function clearSearchText(text: string): string { diff --git a/packages/adapters/src/sites/spider/detail.ts b/packages/adapters/src/sites/spider/detail.ts index 940fa3fa..17965b47 100644 --- a/packages/adapters/src/sites/spider/detail.ts +++ b/packages/adapters/src/sites/spider/detail.ts @@ -4,11 +4,12 @@ * 促销/H&R 复用站点定义中 downloadvolumefactor / uploadvolumefactor / freedate / hr 字段(与列表同语义)。 */ import type { CheerioAPI } from "cheerio"; -import type { SiteReleaseDetail } from "../types"; -import { extractDoubanId, extractImdbId, parseCount, parseSiteTime } from "../util"; -import { categoryName, promotionFromFactors } from "./map"; -import type { IndexerDefinition } from "./schema"; -import { loadHtml, parseSizeMp, SiteSpider } from "./spider"; +import type { SiteReleaseDetail } from "../types.ts"; +import { extractDoubanId, extractImdbId, parseCount, parseSiteTime } from "../util.ts"; +import { categoryName, promotionFromFactors } from "./map.ts"; +import { mcSelect } from "./mc-selector.ts"; +import type { IndexerDefinition } from "./schema.ts"; +import { loadHtml, parseSizeMp, SiteSpider } from "./spider.ts"; export type DetailPageKind = "ok" | "login" | "not_found" | "unknown"; @@ -38,15 +39,23 @@ export function parseDetailPage( return found as ReturnType | null; }; - const title = h1 - .contents() - .filter((_, n) => n.type === "text") - .text() + // 逐字段选择器(MovieClaw detail_*_css)优先于 rowhead 表格模型 + const f = conf.fields ?? {}; + const title = ( + f.title + ? (mcSelect($, null, f.title) ?? "") + : h1 + .contents() + .filter((_, n) => n.type === "text") + .text() + ) .replace(/\s+/g, " ") .trim(); - const subtitle = row(conf.labels.subtitle)?.text().trim(); + const subtitle = f.subtitle ? (mcSelect($, null, f.subtitle) ?? undefined) : row(conf.labels.subtitle)?.text().trim(); const basicText = row(conf.labels.basicInfo)?.text().replace(/\s+/g, " ") ?? ""; - const sizeText = /(?:大小|Size)\s*[:：]\s*([\d.,]+\s*[KMGTP]?i?B)/i.exec(basicText)?.[1] ?? null; + const sizeText = f.size + ? (/([\d.,]+\s*[KMGTP]?i?B)/i.exec(mcSelect($, null, f.size) ?? "")?.[1] ?? null) + : (/(?:大小|Size)\s*[:：]\s*([\d.,]+\s*[KMGTP]?i?B)/i.exec(basicText)?.[1] ?? null); const catText = /(?:类型|Type)\s*[:：]\s*(\S+)/i.exec(basicText)?.[1] ?? null; const catId = catText ? (Object.entries(def.umt.categoryNames ?? {}).find(([, name]) => name === catText)?.[0] ?? null) @@ -102,9 +111,9 @@ export function parseDetailPage( }; if (subtitle) detail.subtitle = subtitle; if (descText) detail.descriptionText = descText.slice(0, 20_000); - const imdb = extractImdbId(descHtml) ?? extractImdbId(descText); + const imdb = extractImdbId(mcSelect($, null, f.imdb)) ?? extractImdbId(descHtml) ?? extractImdbId(descText); if (imdb) detail.imdbId = imdb; - const douban = extractDoubanId(descHtml); + const douban = extractDoubanId(mcSelect($, null, f.douban)) ?? extractDoubanId(descHtml); if (douban) detail.doubanId = douban; return { kind: "ok", detail }; } diff --git a/packages/adapters/src/sites/spider/map.ts b/packages/adapters/src/sites/spider/map.ts index 6303f0cb..1215b9b1 100644 --- a/packages/adapters/src/sites/spider/map.ts +++ b/packages/adapters/src/sites/spider/map.ts @@ -1,8 +1,8 @@ -import { redactUrl } from "../../rss/redact"; -import type { PromotionType, SitePromotion, SiteRelease } from "../types"; -import { parseSiteTime } from "../util"; -import type { IndexerDefinition } from "./schema"; -import type { SpiderTorrent } from "./spider"; +import { redactUrl } from "../../rss/redact.ts"; +import type { PromotionType, SitePromotion, SiteRelease } from "../types.ts"; +import { parseSiteTime } from "../util.ts"; +import type { IndexerDefinition } from "./schema.ts"; +import type { SpiderTorrent } from "./spider.ts"; /** * MP 系数 → 统一促销类型。MP 只有 downloadvolumefactor/uploadvolumefactor/freedate 三个字段, diff --git a/packages/adapters/src/sites/spider/mc-selector.ts b/packages/adapters/src/sites/spider/mc-selector.ts new file mode 100644 index 00000000..e2fc1888 --- /dev/null +++ b/packages/adapters/src/sites/spider/mc-selector.ts @@ -0,0 +1,52 @@ +/** + * MovieClaw 选择器语法求值(parsel/cssselect 扩展伪元素),用于 MovieClaw v0.27.0 转换来的 + * `umt.detail.fields` / `umt.profile` 选择器。语义对照 MovieClaw `NexusPHPSite._css_text`(MIT,见 + * ../definitions/NOTICE-MovieClaw.md): + * - 空选择器 → null + * - `X::next_sibling_text` → X 首个命中元素之后第一个文本兄弟节点(XPath `following-sibling::text()[1]`),trim + * - `X::attr(a)` → 首个命中元素的属性 a;`X::text` → 首个命中元素的直接文本节点(parsel `::text` 取第一个) + * - 普通选择器 → 所有命中元素的全部后代文本,逐段 trim 后以空格连接 + * `:contains()` 由 cheerio-select 原生支持。 + */ +import type { CheerioAPI } from "cheerio"; + +type Scope = ReturnType; + +export function mcSelect($: CheerioAPI, scope: Scope | null, css: string | null | undefined): string | null { + if (!css) return null; + const find = (sel: string) => (scope ? scope.find(sel) : $(sel)); + if (css.endsWith("::next_sibling_text")) { + const el = find(css.slice(0, -"::next_sibling_text".length)).first(); + if (el.length === 0) return null; + let n = el[0]!.next; + while (n && n.type !== "text") n = n.next; + const t = n && n.type === "text" ? (n as unknown as { data: string }).data.trim() : ""; + return t || null; + } + const attr = /::attr\(([^)]+)\)$/.exec(css); + if (attr) { + const v = find(css.slice(0, attr.index)).first().attr(attr[1]!.trim()); + return v?.trim() || null; + } + if (css.endsWith("::text")) { + const el = find(css.slice(0, -"::text".length)).first(); + const t = el + .contents() + .filter((_, n) => n.type === "text") + .first() + .text() + .trim(); + return t || null; + } + const texts: string[] = []; + find(css).each((_, el) => { + const walk = (n: { type: string; data?: string; children?: unknown[] }) => { + if (n.type === "text") { + const t = (n.data ?? "").trim(); + if (t) texts.push(t); + } else for (const c of (n.children ?? []) as Array) walk(c); + }; + walk(el as unknown as { type: string }); + }); + return texts.length ? texts.join(" ") : null; +} diff --git a/packages/adapters/src/sites/spider/schema.ts b/packages/adapters/src/sites/spider/schema.ts index 3f261701..d4402534 100644 --- a/packages/adapters/src/sites/spider/schema.ts +++ b/packages/adapters/src/sites/spider/schema.ts @@ -63,8 +63,25 @@ export interface SpiderFields { [name: string]: SpiderFieldSelector | undefined; } +/** 定义来源(MovieClaw v0.27.0 MIT 转换 / 手写) */ +export interface DefinitionProvenance { + source: "movieclaw" | "handwritten"; + ref?: string; + commit?: string; + license?: string; + file?: string; + /** 本项目手工覆盖层(深合并,优先于来源值) */ + overrides?: string; + note?: string; +} + export interface IndexerDefinition { id: string; + /** 是否随产品启用(未实测站点为 false) */ + enabled?: boolean; + /** MP 用户信息解析器 schema(`SiteSchema` 值,如 NexusPhp / NexusHhanclub) */ + schema?: string; + provenance?: DefinitionProvenance; name: string; domain: string; encoding?: string; @@ -95,6 +112,26 @@ export interface UmtSiteExtension { /** 定义版本,入 site_snapshot */ version: string; kind: "nexusphp"; + /** 需要自定义代码的站点(MovieClaw custom_class,如 TTG);通用 spider 结果不可靠 */ + customClass?: string; + /** H&R 考核做种小时数(MovieClaw hr_seed_hours;暂无站点配置) */ + hrSeedHours?: number; + /** 支持的凭据方式(MovieClaw auth.supported:cookie / credential / apikey) */ + authSupported?: string[]; + /** 登录表单配置(MovieClaw login_*;M2 账号密码登录用) */ + login?: { + path: string; + usernameField: string; + passwordField: string; + errorCss: string; + extraFormData?: Array<[string, string]>; + selectDefaults?: Array<[string, string]>; + }; + /** + * 用户资料选择器(MovieClaw profile_*,选择器语法见 spider/mc-selector.ts:支持 ::attr(x) / ::text / ::next_sibling_text)。 + * 主路径仍是 MP 用户信息解析器(`schema`),此处为备用数据,M2 使用。 + */ + profile?: Record; displayName?: string; /** 站点时间时区偏移,如 "+08:00";pubdate/freedate 按此转 UTC */ timezone: string; @@ -119,6 +156,13 @@ export interface UmtSiteExtension { /** 促销/HR 标记所在范围 */ headerScope: string; notFoundMarkers: string[]; + /** + * 逐字段选择器(MovieClaw detail_*_css,mc-selector 语法)。配置时优先于 rowhead 表格模型, + * 用于 HHClub 这类 div 详情页。 + */ + fields?: Partial< + Record<"title" | "subtitle" | "size" | "description" | "download" | "imdb" | "douban" | "fileList", string> + >; }; /** 下载路径(按资源 ID 拼接) */ downloadPath: string; diff --git a/packages/adapters/src/sites/spider/spider.test.ts b/packages/adapters/src/sites/spider/spider.test.ts index f157f3d0..215267e1 100644 --- a/packages/adapters/src/sites/spider/spider.test.ts +++ b/packages/adapters/src/sites/spider/spider.test.ts @@ -9,7 +9,7 @@ import { SiteError, SiteSpider, staticSecretResolver, -} from "../index"; +} from "../index.ts"; let site: MockNexusPhpSite | undefined; afterEach(async () => { diff --git a/packages/adapters/src/sites/spider/spider.ts b/packages/adapters/src/sites/spider/spider.ts index e9eb08ed..7bb69b36 100644 --- a/packages/adapters/src/sites/spider/spider.ts +++ b/packages/adapters/src/sites/spider/spider.ts @@ -19,8 +19,8 @@ * 3. 不发请求:HTTP/Cookie/UA 由 adapter 注入,本类只负责 URL 构造与 HTML 解析(便于限流与脱敏统一)。 */ import { type CheerioAPI, load } from "cheerio"; -import type { IndexerDefinition, SpiderFieldSelector, SpiderFields, SpiderFilter, SpiderListConfig } from "./schema"; -import { renderTemplate } from "./template"; +import type { IndexerDefinition, SpiderFieldSelector, SpiderFields, SpiderFilter, SpiderListConfig } from "./schema.ts"; +import { renderTemplate } from "./template.ts"; type Sel = ReturnType; diff --git a/packages/adapters/src/sites/spiders/adapter.ts b/packages/adapters/src/sites/spiders/adapter.ts new file mode 100644 index 00000000..797495e7 --- /dev/null +++ b/packages/adapters/src/sites/spiders/adapter.ts @@ -0,0 +1,123 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/modules/indexer/__init__.py @ e6820a8 + * + * 专用 spider 站点适配器:对应 MP `IndexerModule.search_torrents/refresh_torrents` 按 `indexer.parser` + * 分派到 `SPIDER_PARSER_CLASSES` 并以 `_SPECIALIZED_SEARCH_ARGUMENTS` 传参; + * 下载对应 `submission.download_torrent`(两段式换票由 spider.resolveDownload 现场完成)。 + * + * 与 MP 的差异:MP `search` 失败返回 `(True, [])`;本项目抛 SiteError(凭据/限流/解析分类), + * 健康检查即一次空关键词列表。list 对外页码 1 起;`pageSize=null` 的站(Haidan、TorrentLeech 关键词)只有第 1 页。 + */ +import { buildHttpContext, type SiteAdapterDeps } from "../deps.ts"; +import { healthFromError } from "../spider/adapter.ts"; +import { + type DownloadTarget, + type SiteAdapter, + type SiteCredentialKind, + SiteError, + type SiteHealth, + type SiteListFilters, + type SitePage, +} from "../types.ts"; +import { specializedTorrentToRelease } from "./map.ts"; +import type { SpecializedSiteDefinition, SpecializedSpider, SpiderContext } from "./types.ts"; + +export function createSpecializedSiteAdapter( + input: SpecializedSiteDefinition, + spider: SpecializedSpider, + deps: SiteAdapterDeps, +): SiteAdapter { + const def: SpecializedSiteDefinition = deps.baseUrl + ? { + ...input, + domain: deps.baseUrl.endsWith("/") ? deps.baseUrl : `${deps.baseUrl}/`, + umt: { ...input.umt, apiBaseUrl: deps.baseUrl.replace(/\/+$/, "") }, + } + : input; + const http = buildHttpContext( + def.id, + { ...(def.timeout ? { timeoutMs: def.timeout * 1000 } : {}), ...deps }, + def.umt.rateLimit, + ); + const cache = new Map(); + /** 最近搜索结果的直链(仅内存,供 downloadTarget 复用;含 passkey 类参数,不外泄) */ + const enclosures = new Map(); + + function ctx(signal?: AbortSignal): SpiderContext { + const c: SpiderContext = { + def, + http, + cache, + async secret(kind: SiteCredentialKind, required = false) { + const v = await deps.secrets.resolve({ siteId: def.id, kind }); + if (!v && required) { + throw new SiteError( + def.id, + "auth_missing", + `${def.name} 未配置${kind === "cookie" ? " Cookie" : " API Key"}`, + ); + } + return v; + }, + }; + if (deps.userAgent) c.userAgent = deps.userAgent; + if (signal) c.signal = signal; + return c; + } + + async function doSearch(page: number, filters: SiteListFilters, keyword: string | null, signal?: AbortSignal) { + const p = Math.max(1, page); + const fetchedAt = http.now().toISOString(); + const size = spider.pageSize(keyword); + if (size === null && p > 1) { + return { items: [], page: p, totalPages: 1, totalPagesEstimated: true, fetchedAt } satisfies SitePage; + } + const q: Parameters[1] = { keyword, page: p - 1 }; + if (filters.mtype) q.mtype = filters.mtype; + if (filters.categoryIds?.length) q.cat = filters.categoryIds.join(","); + const rows = await spider.search(ctx(signal), q); + let items = rows.flatMap((t) => { + const r = specializedTorrentToRelease(t, def, fetchedAt); + if (r && t.enclosure) enclosures.set(r.siteReleaseId, t.enclosure); + return r ? [r] : []; + }); + if (filters.categoryIds?.length) { + const allow = new Set(filters.categoryIds.map(String)); + items = items.filter((r) => r.category.id === null || allow.has(r.category.id)); + } + if (filters.freeOnly) items = items.filter((r) => r.promotion.downloadFactor === 0); + return { items, page: p, totalPages: null, totalPagesEstimated: false, fetchedAt } satisfies SitePage; + } + + return { + id: def.id, + kind: "api", + displayName: def.umt.displayName, + capabilities: def.umt.capabilities, + definitionVersion: def.umt.version, + list(page, filters = {}, opts = {}) { + return doSearch(page, filters, null, opts.signal); + }, + search(query, filters = {}, opts = {}) { + return doSearch(filters.page ?? 1, filters, query.trim() || null, opts.signal); + }, + async detail() { + throw new SiteError(def.id, "unsupported", `${def.name}:专用 API spider 不提供详情接口`); + }, + async downloadTarget(siteReleaseId, opts = {}): Promise { + return spider.resolveDownload(ctx(opts.signal), siteReleaseId, enclosures.get(String(siteReleaseId))); + }, + async healthCheck(opts = {}): Promise { + const started = Date.now(); + const checkedAt = http.now().toISOString(); + try { + await spider.search(ctx(opts.signal), { keyword: null, page: 0 }); + return { siteId: def.id, status: "ok", ok: true, checkedAt, latencyMs: Date.now() - started }; + } catch (e) { + return healthFromError(def.id, e, checkedAt); + } + }, + }; +} diff --git a/packages/adapters/src/sites/spiders/common.ts b/packages/adapters/src/sites/spiders/common.ts new file mode 100644 index 00000000..6b5f8c6c --- /dev/null +++ b/packages/adapters/src/sites/spiders/common.ts @@ -0,0 +1,371 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/chain/download/submission.py @ e6820a8 + * @mp-source app/foundation/temporal.py @ e6820a8 + * + * 专用 spider 共用 IO 与工具: + * - `submission._resolve_indirect_download_url` → `resolveIndirectDownloadUrl`(配置对象在内存中构造,不再 base64 编进 enclosure) + * - `submission._normalize_indirect_download_url` → `normalizeIndirectDownloadUrl` + * - `submission.download_torrent`(带/不带 cookie 下载并校验)→ `downloadTorrentFile` + * - `temporal.format_timestamp` → `formatTimestampTz`;`urllib.parse.quote` → `pyQuote` + * - 各 spider `__process_response` 的错误分支 → `apiRequest`(HTTP 401/403 → auth_invalid)+ `businessError` + */ +import { redactText, redactUrl } from "../../rss/redact.ts"; +import { readBody, readText, siteFetch } from "../http.ts"; +import { type DownloadTarget, SiteError, type SiteErrorKind } from "../types.ts"; +import { inspectTorrent } from "../util.ts"; +import type { SpiderContext } from "./types.ts"; + +/** 下载链接中需要脱敏的额外参数名(HDDolby downhash 等效 passkey) */ +export const EXTRA_SECRET_KEYS = ["downhash", "token", "credential", "sign", "key"]; + +const RATE_KEYWORDS = ["頻繁", "频繁", "too many", "rate limit", "稍后"]; +const NOT_FOUND_KEYWORDS = ["不存在", "not found", "not exist"]; +const AUTH_KEYWORDS = [ + "停用", + "封禁", + "禁用", + "banned", + "disabled", + "無效", + "无效", + "invalid", + "unauthorized", + "未授權", + "未授权", + "未登录", + "登录", + "api key", + "apikey", + "token", + "permission", + "权限", + "forbidden", +]; + +/** 业务错误文案 → SiteError 类别(默认 parse_error) */ +export function classifyMessage(message: string | null | undefined): SiteErrorKind { + const lower = String(message ?? "").toLowerCase(); + if (RATE_KEYWORDS.some((k) => lower.includes(k))) return "rate_limited"; + if (NOT_FOUND_KEYWORDS.some((k) => lower.includes(k))) return "not_found"; + if (AUTH_KEYWORDS.some((k) => lower.includes(k.toLowerCase()))) return "auth_invalid"; + return "parse_error"; +} + +export function businessError(ctx: SpiderContext, what: string, message: unknown): SiteError { + const msg = message == null || message === "" ? "未知错误" : String(message).slice(0, 200); + return new SiteError(ctx.def.id, classifyMessage(msg), redactText(`${ctx.def.name} ${what}失败:${msg}`)); +} + +export interface ApiRequestInit { + method?: "GET" | "POST"; + headers?: Record; + query?: Record; + json?: unknown; + /** 原样 query 串(已编码,不含 ?) */ + rawQuery?: string; +} + +export function withQuery(url: string, init: Pick): string { + const parts: string[] = []; + if (init.rawQuery) parts.push(init.rawQuery); + if (init.query) { + const qs = new URLSearchParams(); + for (const [k, v] of Object.entries(init.query)) if (v !== undefined && v !== null) qs.append(k, String(v)); + const s = qs.toString(); + if (s) parts.push(s); + } + if (!parts.length) return url; + return `${url}${url.includes("?") ? "&" : "?"}${parts.join("&")}`; +} + +/** 不跟随跳转(请求头可能带 cookie/key);跳 login 视为登录失效 */ +function checkRedirect(ctx: SpiderContext, res: Response, safe: string): void { + if (res.status < 300 || res.status >= 400) return; + void res.body?.cancel().catch(() => {}); + const loc = res.headers.get("location") ?? ""; + if (/login/i.test(loc)) + throw new SiteError(ctx.def.id, "auth_expired", `${ctx.def.name} 请求 ${safe}:跳转登录页,登录已失效`); + throw new SiteError(ctx.def.id, "network", `${ctx.def.name} 请求 ${safe}:意外跳转 HTTP ${res.status}`, { + httpStatus: res.status, + }); +} + +/** 发请求并解析 JSON;HTTP 401/403 → auth_invalid,404 → not_found,其它非 2xx → network,非 JSON → parse_error/site_maintenance */ +export async function apiRequest(ctx: SpiderContext, url: string, init: ApiRequestInit = {}): Promise { + const headers: Record = { ...(init.headers ?? {}) }; + if (ctx.userAgent && !Object.keys(headers).some((k) => k.toLowerCase() === "user-agent")) + headers["User-Agent"] = ctx.userAgent; + let body: string | undefined; + if (init.json !== undefined) { + body = JSON.stringify(init.json); + if (!Object.keys(headers).some((k) => k.toLowerCase() === "content-type")) + headers["Content-Type"] = "application/json"; + } + const full = withQuery(url, init); + const safe = redactUrl(full, EXTRA_SECRET_KEYS); + const res = await siteFetch(ctx.http, full, { + method: init.method ?? (body !== undefined ? "POST" : "GET"), + headers, + ...(body !== undefined ? { body } : {}), + redirect: "manual", + ...(ctx.signal ? { signal: ctx.signal } : {}), + }); + checkRedirect(ctx, res, safe); + if (res.status === 401 || res.status === 403) { + await res.body?.cancel().catch(() => {}); + throw new SiteError( + ctx.def.id, + "auth_invalid", + `${ctx.def.name} 请求 ${safe}:HTTP ${res.status},凭据无效或无权限`, + { + httpStatus: res.status, + }, + ); + } + if (!res.ok) { + await res.body?.cancel().catch(() => {}); + throw new SiteError( + ctx.def.id, + res.status === 404 ? "not_found" : "network", + `${ctx.def.name} 请求 ${safe}:HTTP ${res.status}`, + { + httpStatus: res.status, + }, + ); + } + const text = await readText(ctx.http, res); + try { + return JSON.parse(text) as unknown; + } catch { + const maintenance = /維護|维护|maintenance/i.test(text); + throw new SiteError( + ctx.def.id, + maintenance ? "site_maintenance" : "parse_error", + `${ctx.def.name} 请求 ${safe}:响应不是有效 JSON${maintenance ? "(站点维护)" : ""}`, + { httpStatus: res.status }, + ); + } +} + +/** 发请求取文本(TNode 首页取 CSRF token) */ +export async function textRequest(ctx: SpiderContext, url: string, headers: Record): Promise { + const h = { ...headers }; + if (ctx.userAgent) h["User-Agent"] = ctx.userAgent; + const res = await siteFetch(ctx.http, url, { + method: "GET", + headers: h, + redirect: "manual", + ...(ctx.signal ? { signal: ctx.signal } : {}), + }); + checkRedirect(ctx, res, redactUrl(url)); + if (res.status !== 200) { + await res.body?.cancel().catch(() => {}); + throw new SiteError( + ctx.def.id, + res.status === 401 || res.status === 403 ? "auth_invalid" : "network", + `${ctx.def.name} 请求 ${redactUrl(url)}:HTTP ${res.status}`, + { httpStatus: res.status }, + ); + } + return readText(ctx.http, res); +} + +export function isPlainObject(v: unknown): v is Record { + return v != null && typeof v === "object" && !Array.isArray(v); +} + +// ---- 时间 ---- + +const pad = (n: number) => String(n).padStart(2, "0"); + +function offsetMinutes(tz: string): number { + const m = /^([+-])(\d{2}):?(\d{2})$/.exec(tz.trim()); + if (!m) return 0; + return (m[1] === "-" ? -1 : 1) * (Number(m[2]) * 60 + Number(m[3])); +} + +/** + * MP `format_timestamp`:非纯数字字符串原样返回;Unix 秒 → 本地墙钟 "YYYY-MM-DD HH:MM:SS"。 + * MP 用进程本地时区,这里用站点时区(`umt.timezone`),随后 map 再按同一时区转 UTC,结果等价。 + */ +export function formatTimestampTz(timestamp: unknown, tz: string): string | undefined { + if (timestamp == null || timestamp === "") return undefined; + if (typeof timestamp === "string" && !/^\d+$/.test(timestamp)) return timestamp; + const n = Number(timestamp); + if (!Number.isFinite(n)) return String(timestamp); + const d = new Date(Math.trunc(n) * 1000 + offsetMinutes(tz) * 60_000); + return `${d.getUTCFullYear()}-${pad(d.getUTCMonth() + 1)}-${pad(d.getUTCDate())} ${pad(d.getUTCHours())}:${pad(d.getUTCMinutes())}:${pad(d.getUTCSeconds())}`; +} + +/** 带显式时区(Z / ±HH:MM)的 ISO 时间 → UTC ISO;无时区返回 undefined */ +export function isoWithZoneToUtc(v: unknown): string | undefined { + if (typeof v !== "string") return undefined; + if (!/^\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}(:\d{2}(\.\d+)?)?\s*(Z|[+-]\d{2}:?\d{2})$/i.test(v.trim())) return undefined; + const t = Date.parse(v.trim().replace(" ", "T")); + return Number.isNaN(t) ? undefined : new Date(t).toISOString(); +} + +/** Python `urllib.parse.quote`(safe='/') */ +export function pyQuote(s: string): string { + return encodeURIComponent(s) + .replace(/[!'()*]/g, (c) => `%${c.charCodeAt(0).toString(16).toUpperCase()}`) + .replace(/%2F/g, "/"); +} + +/** Python `int(x or 0)`;非法值 0(MP 会抛异常丢整页,这里保守取 0) */ +export function toInt(v: unknown): number { + if (typeof v === "number") return Number.isFinite(v) ? Math.trunc(v) : 0; + const n = Number.parseInt(String(v ?? "0") || "0", 10); + return Number.isFinite(n) ? n : 0; +} + +/** Python `float(x)`;非法值回退 */ +export function toFloat(v: unknown, fallback = 1): number { + const n = typeof v === "number" ? v : Number(v); + return Number.isFinite(n) ? n : fallback; +} + +// ---- 缓存 ---- + +export function cacheGet(ctx: SpiderContext, key: string): T | undefined { + const hit = ctx.cache.get(key); + if (!hit) return undefined; + if (hit.expiresAt <= ctx.http.now().getTime()) { + ctx.cache.delete(key); + return undefined; + } + return hit.value as T; +} + +export function cacheSet(ctx: SpiderContext, key: string, value: unknown, ttlMs: number): void { + ctx.cache.set(key, { value, expiresAt: ctx.http.now().getTime() + ttlMs }); +} + +// ---- 下载 ---- + +/** MP 两段式下载的请求配置(原本 base64 编进 enclosure;这里只在内存中构造) */ +export interface IndirectRequestConfig { + method: "get" | "post"; + header: Record; + /** requests params → query string */ + params?: Record; + /** 成功标志键(Yema: "success") */ + success?: string; + /** 结果字段路径,点分(如 "data.download_url") */ + result?: string; + result_base_url?: string; + result_path?: string; + result_query_param?: string; +} + +/** MP `_normalize_indirect_download_url`:把换票结果约束到可信 API 来源 */ +export function normalizeIndirectDownloadUrl(url: string, baseUrl?: string): string { + if (!url || !baseUrl) return url; + let base: URL; + try { + base = new URL(baseUrl); + } catch { + return url; + } + let u: URL; + try { + u = new URL(url); + } catch { + return new URL(url, `${baseUrl.replace(/\/+$/, "")}/`).toString(); + } + u.protocol = base.protocol; + u.host = base.host; + return u.toString(); +} + +/** MP `_resolve_indirect_download_url`:换票 → 临时直链(失败抛 SiteError,MP 返回 None) */ +export async function resolveIndirectDownloadUrl( + ctx: SpiderContext, + requestUrl: string, + cfg: IndirectRequestConfig, +): Promise { + const data = await apiRequest(ctx, requestUrl, { + method: cfg.method === "get" ? "GET" : "POST", + headers: cfg.header, + ...(cfg.params ? { query: cfg.params } : {}), + }); + const fail = (why: string): never => { + const msg = isPlainObject(data) ? (data.message ?? data.msg ?? data.errorMessage) : undefined; + throw new SiteError( + ctx.def.id, + msg ? classifyMessage(String(msg)) : "parse_error", + redactText(`${ctx.def.name} 换取下载地址失败:${msg ? String(msg).slice(0, 200) : why}`), + ); + }; + if (!cfg.result) return typeof data === "string" ? data : fail("响应不是字符串"); + if (cfg.success && !(isPlainObject(data) && data[cfg.success])) fail("success 标志为假"); + let cur: unknown = data; + for (const key of cfg.result.split(".")) { + if (!isPlainObject(cur)) fail(`缺少字段 ${cfg.result}`); + cur = (cur as Record)[key]; + if (!cur) fail(`字段 ${cfg.result} 为空`); + } + if (cfg.result_path && cfg.result_query_param) { + const u = new URL(cfg.result_path.replace(/^\/+/, ""), `${String(cfg.result_base_url).replace(/\/+$/, "")}/`); + return `${u.toString()}?${new URLSearchParams({ [cfg.result_query_param]: String(cur) }).toString()}`; + } + if (typeof cur !== "string") return fail("下载地址不是字符串"); + return normalizeIndirectDownloadUrl(cur, cfg.result_base_url); +} + +/** + * MP `download_torrent`:GET 种子并校验。cookie 为空表示"涉及解析地址的不使用 Cookie 下载"。 + * 带 cookie 时不跟随跳转(避免 cookie 外泄;跳 login 视为失效;跳磁力原样返回)。 + */ +export async function downloadTorrentFile( + ctx: SpiderContext, + url: string, + opts: { cookie?: string | null; headers?: Record } = {}, +): Promise { + const headers: Record = { ...(opts.headers ?? {}) }; + if (ctx.userAgent) headers["User-Agent"] = ctx.userAgent; + if (opts.cookie) headers.cookie = opts.cookie; + const safe = redactUrl(url, EXTRA_SECRET_KEYS); + const res = await siteFetch(ctx.http, url, { + method: "GET", + headers, + ...(opts.cookie ? { redirect: "manual" as const } : {}), + ...(ctx.signal ? { signal: ctx.signal } : {}), + }); + const verifiedAt = ctx.http.now().toISOString(); + if (res.status >= 300 && res.status < 400) { + await res.body?.cancel().catch(() => {}); + const loc = res.headers.get("location") ?? ""; + if (loc.startsWith("magnet:")) return { kind: "magnet", uri: loc, verifiedAt }; + if (/login/i.test(loc)) throw new SiteError(ctx.def.id, "auth_expired", `下载 ${safe}:跳转登录页,Cookie 已失效`); + throw new SiteError(ctx.def.id, "network", `下载 ${safe}:意外跳转 HTTP ${res.status}`, { httpStatus: res.status }); + } + if (res.status === 401 || res.status === 403) { + await res.body?.cancel().catch(() => {}); + throw new SiteError(ctx.def.id, "auth_invalid", `下载 ${safe}:HTTP ${res.status}`, { httpStatus: res.status }); + } + if (!res.ok) { + await res.body?.cancel().catch(() => {}); + throw new SiteError(ctx.def.id, res.status === 404 ? "not_found" : "network", `下载 ${safe}:HTTP ${res.status}`, { + httpStatus: res.status, + }); + } + const bytes = await readBody(ctx.http, res); + const info = inspectTorrent(bytes); + if (!info) throw new SiteError(ctx.def.id, "parse_error", `下载 ${safe}:返回内容不是 .torrent`); + return { kind: "torrent", bytes, infoHash: info.infoHash, name: info.name, fileName: null, verifiedAt }; +} + +/** 站点前端地址(以 / 结尾) */ +export function siteBase(def: { domain: string }): string { + return def.domain.endsWith("/") ? def.domain : `${def.domain}/`; +} + +/** 提取资源 ID:纯数字或 URL 末段数字 */ +export function checkReleaseId(siteId: string, id: string): string { + const m = /(\d+)\/?$/.exec(String(id).trim()); + if (!m) throw new SiteError(siteId, "not_found", "资源 ID 非法"); + return m[1]!; +} diff --git a/packages/adapters/src/sites/spiders/haidan.test.ts b/packages/adapters/src/sites/spiders/haidan.test.ts new file mode 100644 index 00000000..426a8bb7 --- /dev/null +++ b/packages/adapters/src/sites/spiders/haidan.test.ts @@ -0,0 +1,58 @@ +import { createMockFetch, HAIDAN_SEARCH_ERROR, HAIDAN_SEARCH_OK } from "@umt/testkit"; +import { describe, expect, it } from "vitest"; +import { haidanSearchQuery, haidanSpider } from "./haidan.ts"; +import { adapterWith, expectSiteError, testDef, withTorrent } from "./testing.ts"; + +const def = testDef("Haidan", { domain: "https://www.haidan.video/" }); + +describe("HaiDanSpider(MP haidan.py)", () => { + it("__get_params:isapi/search_area/空关键词/分类逗号拼接", () => { + expect(haidanSearchQuery(null)).toBe("isapi=1&search_area=0&search=&search_mode=0&cat="); + expect(haidanSearchQuery("tt1160419", "tv")).toBe( + "isapi=1&search_area=4&search=tt1160419&search_mode=0&cat=402%2C403%2C404%2C405", + ); + }); + + it("列表解析:字段映射、分类投影、促销系数表、passkey 脱敏", async () => { + const m = createMockFetch({ "/torrents.php": { body: HAIDAN_SEARCH_OK } }); + const page = await adapterWith(def, haidanSpider, m.fetch).search("Dune", { mtype: "movie" }); + expect(m.requests[0]!.headers.cookie).toBe("c=COOKIESECRET"); + expect(m.requests[0]!.url.searchParams.get("cat")).toBe("401,404,405"); + const by = Object.fromEntries(page.items.map((i) => [i.siteReleaseId, i])); + expect(by["30001"]).toMatchObject({ + title: "Dune.Part.Two.2024.2160p.UHD.BluRay.REMUX.HDR.HEVC.Atmos-HDH", + subtitle: "沙丘2 | 4K 原盘 REMUX", + category: { id: "401" }, + sizeBytes: 76543210987, + seeders: 88, + leechers: 3, + snatched: 412, + promotion: { type: "free", downloadFactor: 0, uploadFactor: 1 }, + hr: "unknown", + uploadedAt: "2024-09-20T00:00:00.000Z", + detailUrl: "https://www.haidan.video/details.php?group_id=9001&torrent_id=30001", + raw: expect.objectContaining({ category: "电影" }), + }); + expect(by["30002"]!.promotion).toMatchObject({ type: "2x50%", downloadFactor: 0.5, uploadFactor: 2 }); + expect(by["30002"]!.raw.category).toBe("电视剧"); + expect(by["30003"]!.raw.category).toBe("音乐"); + expect(JSON.stringify(page)).not.toContain("HDPASSKEY123"); + expect(page.totalPages).toBeNull(); + // 不翻页:第 2 页直接为空 + expect((await adapterWith(def, haidanSpider, m.fetch).list(2)).items).toEqual([]); + }); + + it("业务错误 / 缺 Cookie / 下载沿用缓存直链带 Cookie", async () => { + const bad = createMockFetch({ "/torrents.php": { body: HAIDAN_SEARCH_ERROR } }); + await expectSiteError(adapterWith(def, haidanSpider, bad.fetch).list(1), "auth_invalid"); + await expectSiteError(adapterWith(def, haidanSpider, bad.fetch, {}).list(1), "auth_missing"); + const m = createMockFetch({ "/torrents.php": { body: HAIDAN_SEARCH_OK } }); + const t = withTorrent(m.fetch, (u) => u.pathname === "/download.php"); + const a = adapterWith(def, haidanSpider, t.fetch); + await a.list(1); + const target = await a.downloadTarget("30001"); + expect(target).toMatchObject({ kind: "torrent", infoHash: t.infoHash }); + expect(t.hits[0]!.url.searchParams.get("passkey")).toBe("HDPASSKEY123"); + expect(t.hits[0]!.headers.get("cookie")).toBe("c=COOKIESECRET"); + }); +}); diff --git a/packages/adapters/src/sites/spiders/haidan.ts b/packages/adapters/src/sites/spiders/haidan.ts new file mode 100644 index 00000000..4b9d08e0 --- /dev/null +++ b/packages/adapters/src/sites/spiders/haidan.ts @@ -0,0 +1,141 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/modules/indexer/spider/haidan.py @ e6820a8 + * + * 海胆 haidan.video(parser "Haidan",Cookie + `torrents.php?isapi=1` JSON)。函数对照 `HaiDanSpider`: + * - `_movie_category/_tv_category/_music_category/_dl_state/_up_state` → 同名常量 + * - `get_search_page_size` → `pageSize`(恒 null:不翻页) + * - `__get_params`(含 `__dict_to_query`)→ `haidanSearchQuery` + * - `__parse_result` → `parseHaidanResult`;`__process_response` → `processHaidanResponse` + * - `search/async_search` → `haidanSpider.search` + * - `__get_downloadvolumefactor/__get_uploadvolumefactor` → `haidanDownloadVolumeFactor/haidanUploadVolumeFactor` + * 下载:MP 用结果里的 `url` 字段(可能含 passkey)带 Cookie 下载;缓存缺失时回退 NexusPHP `download.php?id=`(TODO(verify))。 + */ +import type { MpMediaType } from "../spider/spider.ts"; +import { SiteError } from "../types.ts"; +import { + apiRequest, + businessError, + downloadTorrentFile, + formatTimestampTz, + isPlainObject, + siteBase, + toInt, +} from "./common.ts"; +import type { SpecializedSiteDefinition, SpecializedSpider, SpecializedTorrent, SpiderSearchQuery } from "./types.ts"; + +export const HAIDAN_MOVIE_CATEGORY = ["401", "404", "405"]; +export const HAIDAN_TV_CATEGORY = ["402", "403", "404", "405"]; +/** 音乐搜索同时覆盖高品质音频和音乐视频/演唱会分区 */ +export const HAIDAN_MUSIC_CATEGORY = ["406", "408"]; + +/** 促销状态 1-普通,2-免费,3-2X,4-2X免费,5-50%,6-2X50%,7-30% */ +export const HAIDAN_DL_STATE: Record = { "1": 1, "2": 0, "3": 1, "4": 0, "5": 0.5, "6": 0.5, "7": 0.3 }; +export const HAIDAN_UP_STATE: Record = { "1": 1, "2": 1, "3": 2, "4": 2, "5": 1, "6": 2, "7": 1 }; + +export function haidanDownloadVolumeFactor(discount: unknown): number { + return discount ? (HAIDAN_DL_STATE[String(discount)] ?? 1) : 1; +} + +export function haidanUploadVolumeFactor(discount: unknown): number { + return discount ? (HAIDAN_UP_STATE[String(discount)] ?? 1) : 1; +} + +/** MP `__get_params`:返回已编码的 query 串(列表转逗号分隔;空关键词显式传空串) */ +export function haidanSearchQuery(keyword: string | null | undefined, mtype?: SpiderSearchQuery["mtype"]): string { + let categories: string[]; + if (!mtype) categories = []; + else if (mtype === "tv") categories = HAIDAN_TV_CATEGORY; + else if (mtype === "music") categories = HAIDAN_MUSIC_CATEGORY; + else categories = HAIDAN_MOVIE_CATEGORY; + // 0-标题 1-简介(较慢)3-发种用户名 4-IMDb + const searchArea = keyword?.startsWith("tt") ? "4" : "0"; + return new URLSearchParams({ + isapi: "1", + search_area: searchArea, + search: keyword || "", + search_mode: "0", // 0-与 1-或 2-精准 + cat: categories.join(","), + }).toString(); +} + +export interface HaidanItem { + group_id?: string | number; + category?: string | number | null; + name?: string; + small_descr?: string; + url?: string; + added?: string | number; + size?: string | number; + seeders?: string | number; + leechers?: string | number; + times_completed?: string | number; + sp_state?: string | number; + [k: string]: unknown; +} + +/** MP `__parse_result`(data 为 {tid: item} 字典) */ +export function parseHaidanResult( + result: { data?: Record | null }, + def: SpecializedSiteDefinition, +) { + const torrents: SpecializedTorrent[] = []; + const base = siteBase(def); + for (const [tid, item] of Object.entries(result.data ?? {})) { + // 分类字段在每条种子内,接口顶层没有 category 字段 + const categoryValue = String(item.category || ""); + let category: MpMediaType; + if (HAIDAN_MUSIC_CATEGORY.includes(categoryValue)) category = "音乐"; + else if (HAIDAN_TV_CATEGORY.includes(categoryValue) && !HAIDAN_MOVIE_CATEGORY.includes(categoryValue)) + category = "电视剧"; + else if (HAIDAN_MOVIE_CATEGORY.includes(categoryValue)) category = "电影"; + else category = "未知"; + const t: SpecializedTorrent = { + title: item.name, + description: item.small_descr, + size: toInt(item.size), + seeders: toInt(item.seeders), + peers: toInt(item.leechers), + grabs: toInt(item.times_completed), + downloadvolumefactor: haidanDownloadVolumeFactor(item.sp_state), + uploadvolumefactor: haidanUploadVolumeFactor(item.sp_state), + page_url: `${base}details.php?group_id=${item.group_id}&torrent_id=${tid}`, + labels: [], + category, + site_release_id: tid, + }; + if (item.url) t.enclosure = String(item.url); + const pubdate = formatTimestampTz(item.added, def.umt.timezone); + if (pubdate) t.pubdate = pubdate; + if (categoryValue) t.category_id = categoryValue; + torrents.push(t); + } + return torrents; +} + +export const haidanSpider: SpecializedSpider = { + parser: "Haidan", + credential: "cookie", + pageSize: () => null, + async search(ctx, q) { + const cookie = await ctx.secret("cookie", true); + const data = await apiRequest(ctx, `${siteBase(ctx.def)}torrents.php`, { + headers: { cookie: cookie! }, + rawQuery: haidanSearchQuery(q.keyword, q.mtype), + }); + // MP __process_response:code != 0 → 失败 + if (!isPlainObject(data) || data.code !== 0) { + throw businessError(ctx, "搜索", isPlainObject(data) ? data.msg : "响应结构无效"); + } + return parseHaidanResult(data as { data?: Record }, ctx.def); + }, + async resolveDownload(ctx, siteReleaseId, enclosure) { + const cookie = await ctx.secret("cookie", true); + const id = /(\d+)\/?$/.exec(String(siteReleaseId))?.[1]; + if (!id) throw new SiteError(ctx.def.id, "not_found", "资源 ID 非法"); + // TODO(verify):无缓存直链时回退 download.php?id= + const url = enclosure ?? `${siteBase(ctx.def)}download.php?id=${id}`; + return downloadTorrentFile(ctx, url, { cookie }); + }, +}; diff --git a/packages/adapters/src/sites/spiders/hddolby.test.ts b/packages/adapters/src/sites/spiders/hddolby.test.ts new file mode 100644 index 00000000..1d3c853e --- /dev/null +++ b/packages/adapters/src/sites/spiders/hddolby.test.ts @@ -0,0 +1,61 @@ +import { createMockFetch, HDDOLBY_SEARCH_ERROR, HDDOLBY_SEARCH_OK } from "@umt/testkit"; +import { describe, expect, it } from "vitest"; +import { hddolbyApiBase, hddolbySearchParams, hddolbySpider } from "./hddolby.ts"; +import { adapterWith, expectSiteError, testDef, withTorrent } from "./testing.ts"; + +const def = testDef("HDDolby", { domain: "https://www.hddolby.com/" }); + +describe("HddolbySpider(MP hddolby.py)", () => { + it("__get_params 与 API 地址:page_number 为 0 起、无类型时 movie+tv 去重升序", () => { + expect(hddolbyApiBase(def)).toBe("https://api.hddolby.com"); + expect(hddolbySearchParams("Lost")).toEqual({ + keyword: "Lost", + page_number: 0, + page_size: 100, + categories: [401, 402, 403, 404, 405], + visible: 1, + }); + expect(hddolbySearchParams(null, "music", 2).categories).toEqual([406, 408]); + }); + + it("列表解析:促销类型表、标签表、分类优先级(405 归电视剧)、downhash 脱敏", async () => { + const m = createMockFetch({ "POST https://api.hddolby.com/api/v1/torrent/search": { body: HDDOLBY_SEARCH_OK } }); + const page = await adapterWith(def, hddolbySpider, m.fetch).list(1); + const req = m.requests[0]!; + expect(req.headers["x-api-key"]).toBe("APIKEYSECRET"); + expect(req.headers.cookie).toBe("c=COOKIESECRET"); + expect(JSON.parse(req.body!)).toMatchObject({ keyword: null, page_number: 0 }); + const [a, b, c] = page.items; + expect(a).toMatchObject({ + siteReleaseId: "120202", + title: "[DBY] Lost S06 2010 Complete 1080p Netflix WEB-DL AVC DDP5.1-DBTV", + category: { id: "402" }, + promotion: { type: "free", downloadFactor: 0, uploadFactor: 1, expiresAt: "2026-09-30T04:00:00.000Z" }, + tags: ["官方", "中文字幕"], + uploadedAt: "2025-02-18T11:47:56.000Z", + detailUrl: "https://www.hddolby.com/details.php?id=120202&hit=1", + hr: "unknown", + }); + expect(a!.raw.category).toBe("电视剧"); + // promotion_time_type 6 → 2x(下载 1 上传 2),0000 截止无效 + expect(b!.promotion).toEqual({ type: "2x", downloadFactor: 1, uploadFactor: 2, expiresAt: null }); + expect(b!.raw.category).toBe("电影"); + expect(c!.raw.category).toBe("音乐"); + expect(c!.tags).toEqual([]); + expect(JSON.stringify(page)).not.toContain("DOWNHASHSECRET"); + const t = withTorrent(m.fetch, (u) => u.pathname === "/download.php"); + const ad = adapterWith(def, hddolbySpider, t.fetch); + await ad.list(1); + await ad.downloadTarget("120202"); + expect(t.hits[0]!.url.searchParams.get("downhash")).toBe("DOWNHASHSECRET"); + }); + + it("error 字段 / HTTP 401 / 缺 API Key", async () => { + const m = createMockFetch({ "/api/v1/torrent/search": { body: HDDOLBY_SEARCH_ERROR } }); + const e = await expectSiteError(adapterWith(def, hddolbySpider, m.fetch).list(1), "auth_invalid"); + expect(e.message).toContain("invalid api key"); + const h = createMockFetch({ "/api/v1/torrent/search": { status: 401, body: "no" } }); + await expectSiteError(adapterWith(def, hddolbySpider, h.fetch).list(1), "auth_invalid"); + await expectSiteError(adapterWith(def, hddolbySpider, m.fetch, { cookie: "c=1" }).list(1), "auth_missing"); + }); +}); diff --git a/packages/adapters/src/sites/spiders/hddolby.ts b/packages/adapters/src/sites/spiders/hddolby.ts new file mode 100644 index 00000000..603dd0a8 --- /dev/null +++ b/packages/adapters/src/sites/spiders/hddolby.ts @@ -0,0 +1,171 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/modules/indexer/spider/hddolby.py @ e6820a8 + * + * HDDolby(parser "HDDolby",`https://api.<域>/api/v1/torrent/search`,x-api-key + Cookie)。函数对照 `HddolbySpider`: + * - `_movie_category/_tv_category/_music_category/_labels` → 同名常量 + * - `get_search_page_size` → `pageSize`(100) + * - `__get_params` → `hddolbySearchParams`(注意 MP 的 page_number 直接传 0 起页码,原样保留) + * - `__parse_result` → `parseHddolbyResult`;`__process_response` → `hddolbySpider.search` 内的错误分支 + * - `__get_downloadvolumefactor/__get_uploadvolumefactor` → `hddolbyDownloadVolumeFactor/hddolbyUploadVolumeFactor` + * - `__get_download_url` → `hddolbyDownloadUrl`(含 downhash,raw/downloadUrl 中脱敏) + * 差异:API 返回的 `hr` 字段 MP 未映射,这里同样不映射(H&R = unknown,待核验语义)。 + */ +import type { MpMediaType } from "../spider/spider.ts"; +import { extractDomain } from "../user-parsers/common.ts"; +import { apiRequest, businessError, checkReleaseId, downloadTorrentFile, isPlainObject, siteBase } from "./common.ts"; +import type { SpecializedSiteDefinition, SpecializedSpider, SpecializedTorrent, SpiderSearchQuery } from "./types.ts"; + +export const HDDOLBY_MOVIE_CATEGORY = [401, 405]; +export const HDDOLBY_TV_CATEGORY = [402, 403, 404, 405]; +/** 音乐搜索同时覆盖高品质音频和音乐视频/演唱会分区 */ +export const HDDOLBY_MUSIC_CATEGORY = [406, 408]; + +export const HDDOLBY_LABELS: Record = { + gf: "官方", + gy: "国语", + yy: "粤语", + ja: "日语", + ko: "韩语", + zz: "中文字幕", + jz: "禁转", + xz: "限转", + diy: "DIY", + sf: "首发", + yq: "应求", + m0: "零魔", + yc: "原创", + gz: "官字", + db: "Dolby Vision", + hdr10: "HDR10", + hdrm: "HDR10+", + tx: "特效", + lz: "连载", + wj: "完结", + hdrv: "HDR Vivid", + hlg: "HLG", + hq: "高码率", + hfr: "高帧率", +}; + +export function hddolbyDownloadVolumeFactor(discount: unknown): number { + const map: Record = { 2: 0, 5: 0.5, 6: 1, 7: 0.3 }; + return discount ? (map[Number(discount)] ?? 1) : 1; +} + +export function hddolbyUploadVolumeFactor(discount: unknown): number { + const map: Record = { 3: 2, 4: 2, 6: 2 }; + return discount ? (map[Number(discount)] ?? 1) : 1; +} + +export function hddolbyApiBase(def: SpecializedSiteDefinition): string { + return (def.umt.apiBaseUrl ?? `https://api.${extractDomain(def.domain)}`).replace(/\/+$/, ""); +} + +/** MP `__get_params` */ +export function hddolbySearchParams(keyword: string | null | undefined, mtype?: SpiderSearchQuery["mtype"], page = 0) { + let categories: number[]; + if (mtype === "tv") categories = HDDOLBY_TV_CATEGORY; + else if (mtype === "movie") categories = HDDOLBY_MOVIE_CATEGORY; + else if (mtype === "music") categories = HDDOLBY_MUSIC_CATEGORY; + // list(set(movie + tv)):CPython 小整数集合的迭代顺序即升序 + else categories = [...new Set([...HDDOLBY_MOVIE_CATEGORY, ...HDDOLBY_TV_CATEGORY])].sort((a, b) => a - b); + return { keyword: keyword ?? null, page_number: page, page_size: 100, categories, visible: 1 }; +} + +/** MP `__get_download_url` */ +export function hddolbyDownloadUrl(def: SpecializedSiteDefinition, id: unknown, downhash: unknown): string { + return `${siteBase(def)}download.php?id=${id}&downhash=${downhash}`; +} + +export interface HddolbyItem { + id: number | string; + promotion_time_type?: number; + promotion_until?: string; + category?: number; + name?: string; + small_descr?: string; + times_completed?: number; + size?: number; + added?: string; + seeders?: number; + leechers?: number; + downhash?: string; + tags?: string | null; + [k: string]: unknown; +} + +/** MP `__parse_result` */ +export function parseHddolbyResult(results: HddolbyItem[] | null | undefined, def: SpecializedSiteDefinition) { + const torrents: SpecializedTorrent[] = []; + if (!results?.length) return torrents; + for (const result of results) { + const cv = result.category; + let category: MpMediaType; + if (cv != null && HDDOLBY_MUSIC_CATEGORY.includes(cv)) category = "音乐"; + else if (cv != null && HDDOLBY_TV_CATEGORY.includes(cv)) category = "电视剧"; + else if (cv != null && HDDOLBY_MOVIE_CATEGORY.includes(cv)) category = "电影"; + else category = "未知"; + const labels: string[] = []; + for (const labelId of String(result.tags ?? "").split(";")) { + const l = HDDOLBY_LABELS[labelId]; + if (l !== undefined) labels.push(l); + } + const t: SpecializedTorrent = { + title: result.name, + description: result.small_descr, + enclosure: hddolbyDownloadUrl(def, result.id, result.downhash), + size: result.size, + seeders: result.seeders, + peers: result.leechers, + grabs: result.times_completed, + downloadvolumefactor: hddolbyDownloadVolumeFactor(result.promotion_time_type), + uploadvolumefactor: hddolbyUploadVolumeFactor(result.promotion_time_type), + page_url: `${siteBase(def)}details.php?id=${result.id}&hit=1`, + labels, + category, + site_release_id: String(result.id), + }; + if (result.added) t.pubdate = result.added; + if (result.promotion_until) t.freedate = result.promotion_until; + if (cv != null) t.category_id = String(cv); + torrents.push(t); + } + return torrents; +} + +export const hddolbySpider: SpecializedSpider = { + parser: "HDDolby", + credential: "apiKey", + pageSize: () => 100, + async search(ctx, q) { + const apikey = await ctx.secret("apiKey", true); + const cookie = await ctx.secret("cookie"); + const headers: Record = { + "Content-Type": "application/json", + Accept: "application/json, text/plain, */*", + "x-api-key": apikey!, + Referer: siteBase(ctx.def), + }; + if (cookie) headers.cookie = cookie; + const data = await apiRequest(ctx, `${hddolbyApiBase(ctx.def)}/api/v1/torrent/search`, { + method: "POST", + headers, + json: hddolbySearchParams(q.keyword, q.mtype, q.page ?? 0), + }); + if (!isPlainObject(data)) throw businessError(ctx, "搜索", "响应结构无效"); + if (data.error) { + const err = data.error; + throw businessError(ctx, "搜索", isPlainObject(err) ? err.message : err); + } + return parseHddolbyResult(data.data as HddolbyItem[], ctx.def); + }, + async resolveDownload(ctx, siteReleaseId, enclosure) { + const cookie = await ctx.secret("cookie"); + const id = checkReleaseId(ctx.def.id, siteReleaseId); + // TODO(verify):无 downhash 时是否仅凭 Cookie 可下载 + const url = enclosure ?? `${siteBase(ctx.def)}download.php?id=${id}`; + return downloadTorrentFile(ctx, url, { cookie }); + }, +}; diff --git a/packages/adapters/src/sites/spiders/index.ts b/packages/adapters/src/sites/spiders/index.ts new file mode 100644 index 00000000..5d7d9dd4 --- /dev/null +++ b/packages/adapters/src/sites/spiders/index.ts @@ -0,0 +1,70 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/modules/indexer/__init__.py @ e6820a8 + * + * 专用 spider 汇总:`SPECIALIZED_SPIDERS` = MP `SPIDER_PARSER_CLASSES`。 + */ +import type { SiteAdapterDeps } from "../deps.ts"; +import type { SiteAdapter } from "../types.ts"; +import { createSpecializedSiteAdapter } from "./adapter.ts"; +import { haidanSpider } from "./haidan.ts"; +import { hddolbySpider } from "./hddolby.ts"; +import { mtorrentSpider } from "./mtorrent.ts"; +import { rousiSpider } from "./rousi.ts"; +import { sunnyptSpider } from "./sunnypt.ts"; +import { tnodeSpider } from "./tnode.ts"; +import { torrentleechSpider } from "./torrentleech.ts"; +import type { SpecializedParser, SpecializedSiteDefinition, SpecializedSpider } from "./types.ts"; +import { yemaSpider } from "./yema.ts"; + +export const SPECIALIZED_SPIDERS: Record = { + TNodeSpider: tnodeSpider, + TorrentLeech: torrentleechSpider, + mTorrent: mtorrentSpider, + Yema: yemaSpider, + Haidan: haidanSpider, + HDDolby: hddolbySpider, + RousiPro: rousiSpider, + SunnyPT: sunnyptSpider, +}; + +export function isSpecializedParser(v: unknown): v is SpecializedParser { + return typeof v === "string" && Object.hasOwn(SPECIALIZED_SPIDERS, v); +} + +/** 按定义的 parser 键选择专用 spider 创建适配器 */ +export function createSpecializedAdapterFor(def: SpecializedSiteDefinition, deps: SiteAdapterDeps): SiteAdapter { + const spider = SPECIALIZED_SPIDERS[def.parser]; + if (!spider) throw new Error(`未知专用 spider:${def.parser}`); + return createSpecializedSiteAdapter(def, spider, deps); +} + +export { createSpecializedSiteAdapter } from "./adapter.ts"; +export { + apiRequest, + classifyMessage, + downloadTorrentFile, + formatTimestampTz, + type IndirectRequestConfig, + normalizeIndirectDownloadUrl, + pyQuote, + resolveIndirectDownloadUrl, +} from "./common.ts"; +export * from "./haidan.ts"; +export * from "./hddolby.ts"; +export { releaseIdFromPageUrl, specializedTorrentToRelease } from "./map.ts"; +export * from "./mtorrent.ts"; +export * from "./rousi.ts"; +export * from "./sunnypt.ts"; +export * from "./tnode.ts"; +export * from "./torrentleech.ts"; +export type { + SpecializedParser, + SpecializedSiteDefinition, + SpecializedSpider, + SpecializedTorrent, + SpiderContext, + SpiderSearchQuery, +} from "./types.ts"; +export * from "./yema.ts"; diff --git a/packages/adapters/src/sites/spiders/map.ts b/packages/adapters/src/sites/spiders/map.ts new file mode 100644 index 00000000..1d2c6b24 --- /dev/null +++ b/packages/adapters/src/sites/spiders/map.ts @@ -0,0 +1,82 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/modules/indexer/__init__.py @ e6820a8 + * + * 专用 spider 输出(MP TorrentInfo 字段)→ 统一 `SiteRelease`。 + * 促销类型由 MP 系数组合推出(复用 spider/map.ts `promotionFromFactors`);H&R 未提供时为 unknown; + * raw 为完整 MP 字段,enclosure 脱敏(HDDolby downhash、Haidan passkey 等)。 + */ +import { redactUrl } from "../../rss/redact.ts"; +import { promotionFromFactors } from "../spider/map.ts"; +import type { SiteRelease } from "../types.ts"; +import { parseSiteTime } from "../util.ts"; +import { EXTRA_SECRET_KEYS } from "./common.ts"; +import type { SpecializedSiteDefinition, SpecializedTorrent } from "./types.ts"; + +function num(v: unknown): number | null { + if (v == null || v === "") return null; + const n = typeof v === "number" ? v : Number(v); + return Number.isFinite(n) ? n : null; +} + +function factor(v: unknown): number | undefined { + const n = num(v); + return n === null ? undefined : n; +} + +/** 从详情链接取资源 ID:torrent_id= → id= → 末段数字(含 #/torrent/detail/1/ 形式) */ +export function releaseIdFromPageUrl(url: string | undefined): string | null { + if (!url) return null; + return ( + /[?&]torrent_id=(\d+)/.exec(url)?.[1] ?? + /[?&]id=(\d+)/.exec(url)?.[1] ?? + /\/(\d+)\/?(?:[?#].*)?$/.exec(url)?.[1] ?? + null + ); +} + +export function specializedCategoryName(def: SpecializedSiteDefinition, id: string | undefined): string | null { + if (!id) return null; + const named = def.umt.categoryNames?.[id]; + if (named) return named; + for (const k of ["movie", "tv", "music"] as const) { + const hit = (def.category?.[k] ?? []).find((c) => String(c.id) === id); + if (hit?.desc) return hit.desc; + } + return null; +} + +export function specializedTorrentToRelease( + t: SpecializedTorrent, + def: SpecializedSiteDefinition, + observedAt: string, +): SiteRelease | null { + const id = t.site_release_id ?? releaseIdFromPageUrl(t.page_url); + if (!id || !t.title) return null; + const tz = def.umt.timezone; + const freedate = t.freedate_utc ?? (t.freedate ? parseSiteTime(t.freedate, tz) : null); + const raw: Record = { ...t }; + if (t.enclosure) raw.enclosure = redactUrl(t.enclosure, EXTRA_SECRET_KEYS); + const release: SiteRelease = { + siteId: def.id, + siteReleaseId: id, + title: String(t.title).trim(), + category: { id: t.category_id ?? null, name: specializedCategoryName(def, t.category_id) }, + sizeBytes: num(t.size), + seeders: num(t.seeders), + leechers: num(t.peers), + snatched: num(t.grabs), + promotion: promotionFromFactors(factor(t.downloadvolumefactor), factor(t.uploadvolumefactor), freedate), + hr: t.hit_and_run === undefined ? "unknown" : t.hit_and_run, + uploadedAt: t.pubdate_utc ?? parseSiteTime(t.pubdate, tz), + detailUrl: t.page_url ?? `${def.domain}`, + tags: t.labels ?? [], + raw, + observedAt, + }; + if (t.enclosure) release.downloadUrl = redactUrl(t.enclosure, EXTRA_SECRET_KEYS); + if (t.description) release.subtitle = String(t.description).trim(); + if (t.imdbid) release.imdbId = t.imdbid; + return release; +} diff --git a/packages/adapters/src/sites/spiders/mtorrent.ts b/packages/adapters/src/sites/spiders/mtorrent.ts new file mode 100644 index 00000000..9d3594eb --- /dev/null +++ b/packages/adapters/src/sites/spiders/mtorrent.ts @@ -0,0 +1,57 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/modules/indexer/spider/mtorrent.py @ e6820a8 + * + * `mTorrent` 键的专用 spider 形态,复用 W1-3b 已移植的 mteam/spider.ts(`__get_params`/`__parse_result`) + * 与 mteam/client.ts(`__process_response` + 业务码检查),供 `SPECIALIZED_SPIDERS` 统一分派。 + * 完整适配(详情、商城免费、healthCheck=profile)仍以 `createMteamAdapter` 为准。 + */ +import { mteamPost } from "../mteam/client.ts"; +import { rootDomain } from "../mteam/definition.ts"; +import { type MteamTorrentJson, mteamSearchParams, parseMteamResult } from "../mteam/spider.ts"; +import { SiteError } from "../types.ts"; +import { checkReleaseId, downloadTorrentFile, siteBase } from "./common.ts"; +import type { SpecializedSiteDefinition, SpecializedSpider, SpecializedTorrent } from "./types.ts"; + +export function mtorrentApiBase(def: SpecializedSiteDefinition): string { + return (def.umt.apiBaseUrl ?? `https://api.${rootDomain(def.domain)}`).replace(/\/+$/, ""); +} + +export const mtorrentSpider: SpecializedSpider = { + parser: "mTorrent", + credential: "apiKey", + pageSize: () => 100, + async search(ctx, q) { + const apikey = (await ctx.secret("apiKey", true))!; + const params = mteamSearchParams(q.keyword ?? null, q.mtype, q.page ?? 0); + const data = await mteamPost<{ data?: MteamTorrentJson[] | null }>( + ctx.http, + mtorrentApiBase(ctx.def), + apikey, + "/api/torrent/search", + { json: params }, + ctx.signal, + ); + const src = data?.data ?? []; + return parseMteamResult(src, siteBase(ctx.def)).map( + (t, i): SpecializedTorrent => ({ ...t, site_release_id: String(src[i]!.id) }), + ); + }, + async resolveDownload(ctx, siteReleaseId) { + const apikey = (await ctx.secret("apiKey", true))!; + const id = checkReleaseId(ctx.def.id, siteReleaseId); + const url = await mteamPost( + ctx.http, + mtorrentApiBase(ctx.def), + apikey, + "/api/torrent/genDlToken", + { query: { id } }, + ctx.signal, + ); + if (!url || typeof url !== "string") + throw new SiteError(ctx.def.id, "rate_limited", `genDlToken ${id}:未返回下载链接`); + // 涉及解析地址的不使用 Cookie 下载 + return downloadTorrentFile(ctx, url); + }, +}; diff --git a/packages/adapters/src/sites/spiders/rousi.test.ts b/packages/adapters/src/sites/spiders/rousi.test.ts new file mode 100644 index 00000000..322b5cf6 --- /dev/null +++ b/packages/adapters/src/sites/spiders/rousi.test.ts @@ -0,0 +1,54 @@ +import { createMockFetch, ROUSI_DETAIL_OK, ROUSI_SEARCH_ERROR, ROUSI_SEARCH_OK } from "@umt/testkit"; +import { describe, expect, it } from "vitest"; +import { rousiCategoryNamesByIds, rousiSearchParams, rousiSpider } from "./rousi.ts"; +import { adapterWith, expectSiteError, testDef, withTorrent } from "./testing.ts"; + +const def = testDef("RousiPro", { domain: "https://rousi.pro/" }); + +describe("RousiSpider(MP rousi.py)", () => { + it("__get_params:页码 +1、cat 映射只取第一个、否则按 mtype", () => { + expect(rousiCategoryNamesByIds("3, 6,99")).toEqual(["documentary", "variety"]); + expect(rousiSearchParams({ keyword: "x", cat: "6,1", page: 0 })).toEqual({ + page: 1, + page_size: 100, + keyword: "x", + category: "variety", + }); + expect(rousiSearchParams({ mtype: "tv", page: 2 })).toEqual({ page: 3, page_size: 100, category: "tv" }); + }); + + it("列表解析:Bearer、促销 is_active、带时区时间转 UTC、分类 slug", async () => { + const m = createMockFetch({ "GET https://rousi.pro/api/v1/torrents": { body: ROUSI_SEARCH_OK } }); + const page = await adapterWith(def, rousiSpider, m.fetch).search("bear", { mtype: "tv" }); + expect(m.requests[0]!.headers.authorization).toBe("Bearer APIKEYSECRET"); + expect(m.requests[0]!.url.searchParams.get("category")).toBe("tv"); + const [a, b, c] = page.items; + expect(a).toMatchObject({ + siteReleaseId: "501", + category: { id: "tv" }, + sizeBytes: 50000000000, + snatched: 77, + promotion: { type: "2xfree", downloadFactor: 0, uploadFactor: 2, expiresAt: "2026-09-27T16:00:00.000Z" }, + uploadedAt: "2026-09-20T02:00:00.000Z", + detailUrl: "https://rousi.pro/torrents/501", + }); + expect(a!.raw.category).toBe("电视剧"); + // 未激活促销 → 1/1 + expect(b!.promotion).toMatchObject({ type: "none", downloadFactor: 1, uploadFactor: 1 }); + expect(b!.uploadedAt).toBe("2026-09-19T02:00:00.000Z"); + expect(c!.raw.category).toBe("未知"); + }); + + it("业务错误 / 缺 Key / 下载经详情接口换取 download_url,直链不带凭据", async () => { + const bad = createMockFetch({ "/api/v1/torrents": { body: ROUSI_SEARCH_ERROR } }); + await expectSiteError(adapterWith(def, rousiSpider, bad.fetch).list(1), "auth_invalid"); + await expectSiteError(adapterWith(def, rousiSpider, bad.fetch, {}).list(1), "auth_missing"); + const m = createMockFetch({ "GET https://rousi.pro/api/v1/torrents/501": { body: ROUSI_DETAIL_OK } }); + const t = withTorrent(m.fetch, (u) => u.host === "dl.rousi.invalid"); + const target = await adapterWith(def, rousiSpider, t.fetch).downloadTarget("501"); + expect(target).toMatchObject({ kind: "torrent", infoHash: t.infoHash }); + expect(m.requests[0]!.headers.authorization).toBe("Bearer APIKEYSECRET"); + expect(t.hits[0]!.headers.get("authorization")).toBeNull(); + expect(t.hits[0]!.headers.get("cookie")).toBeNull(); + }); +}); diff --git a/packages/adapters/src/sites/spiders/rousi.ts b/packages/adapters/src/sites/spiders/rousi.ts new file mode 100644 index 00000000..b1c3b72c --- /dev/null +++ b/packages/adapters/src/sites/spiders/rousi.ts @@ -0,0 +1,175 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/modules/indexer/spider/rousi.py @ e6820a8 + * + * Rousi.pro PeerGo 兼容 API(parser "RousiPro",Bearer 个人 API Key,需 torrent:read + torrent:download)。 + * 函数对照 `RousiSpider`: + * - `_movie_category/_tv_category/_music_category` → `ROUSI_*_CATEGORY` + * - `get_search_page_size` → `pageSize`(100) + * - `__get_params` → `rousiSearchParams`;`__get_category_names_by_ids` → `rousiCategoryNamesByIds` + * - `__process_response` → `rousiSpider.search` 错误分支;`__parse_result` → `parseRousiResult` + * - `__get_download_url`(GET 详情接口,result=data.download_url)→ `rousiSpider.resolveDownload`(现场换票,不生成含 key 的 enclosure) + */ +import type { MpMediaType } from "../spider/spider.ts"; +import { extractDomain, normalizeDatetime } from "../user-parsers/common.ts"; +import { + apiRequest, + businessError, + checkReleaseId, + downloadTorrentFile, + isoWithZoneToUtc, + isPlainObject, + resolveIndirectDownloadUrl, + toFloat, + toInt, +} from "./common.ts"; +import type { SpecializedSiteDefinition, SpecializedSpider, SpecializedTorrent, SpiderSearchQuery } from "./types.ts"; + +/** API 不支持多分类搜索,每次只使用一个分类 */ +export const ROUSI_MOVIE_CATEGORY = "movie"; +export const ROUSI_TV_CATEGORY = "tv"; +export const ROUSI_MUSIC_CATEGORY = "music"; + +const ROUSI_ID_TO_NAME: Record = { + "1": "movie", + "2": "tv", + "3": "documentary", + "4": "animation", + "5": "music", + "6": "variety", +}; + +/** 站点 origin(MP `https://%s` % extract_domain(domain)) */ +export function rousiBase(def: SpecializedSiteDefinition): string { + return (def.umt.apiBaseUrl ?? `https://${extractDomain(def.domain)}`).replace(/\/+$/, ""); +} + +/** MP `__get_category_names_by_ids` */ +export function rousiCategoryNamesByIds(cat: string | null | undefined): string[] | null { + if (!cat) return null; + const ids = cat + .split(",") + .map((c) => c.trim()) + .filter(Boolean); + const names = ids.filter((id) => id in ROUSI_ID_TO_NAME).map((id) => ROUSI_ID_TO_NAME[id]!); + return names.length ? names : null; +} + +/** MP `__get_params` */ +export function rousiSearchParams(q: SpiderSearchQuery): Record { + const params: Record = { page: Number(q.page ?? 0) + 1, page_size: 100 }; + if (q.keyword) params.keyword = q.keyword; + if (q.cat) { + const names = rousiCategoryNamesByIds(q.cat); + if (names) params.category = names[0]!; + } else if (q.mtype) { + if (q.mtype === "movie") params.category = ROUSI_MOVIE_CATEGORY; + else if (q.mtype === "tv") params.category = ROUSI_TV_CATEGORY; + else if (q.mtype === "music") params.category = ROUSI_MUSIC_CATEGORY; + } + return params; +} + +export interface RousiItem { + id: number | string; + title?: string; + subtitle?: string; + category?: string | { slug?: string; name?: string } | null; + size?: number | string; + seeders?: number | string; + leechers?: number | string; + downloads?: number | string; + created_at?: string; + promotion?: { is_active?: boolean; down_multiplier?: number; up_multiplier?: number; until?: string } | null; + [k: string]: unknown; +} + +/** MP `__parse_result` */ +export function parseRousiResult(results: RousiItem[] | null | undefined, def: SpecializedSiteDefinition) { + const torrents: SpecializedTorrent[] = []; + if (!results?.length) return torrents; + const base = rousiBase(def); + for (const result of results) { + const rawCat = result.category; + let catVal: string | null | undefined = null; + let category: MpMediaType = "未知"; + if (rawCat && typeof rawCat === "object") catVal = rawCat.slug || rawCat.name; + else if (typeof rawCat === "string") catVal = rawCat; + if (catVal) { + catVal = String(catVal).toLowerCase(); + if (catVal === ROUSI_MOVIE_CATEGORY) category = "电影"; + else if (catVal === ROUSI_TV_CATEGORY) category = "电视剧"; + else if (catVal === ROUSI_MUSIC_CATEGORY) category = "音乐"; + } + // API 后端已处理全站促销优先级,直接使用返回的 promotion 数据 + let dl = 1.0; + let ul = 1.0; + let freedate: string | undefined; + let freedateUtc: string | undefined; + const promotion = result.promotion; + if (promotion?.is_active) { + dl = toFloat(promotion.down_multiplier ?? 1.0); + ul = toFloat(promotion.up_multiplier ?? 1.0); + if (promotion.until) { + freedate = normalizeDatetime(promotion.until) ?? undefined; + freedateUtc = isoWithZoneToUtc(promotion.until); + } + } + const id = result.id; + const t: SpecializedTorrent = { + title: result.title, + description: result.subtitle, + size: toInt(result.size), + seeders: toInt(result.seeders), + peers: toInt(result.leechers), + grabs: toInt(result.downloads), + downloadvolumefactor: dl, + uploadvolumefactor: ul, + page_url: `${base}/torrents/${id}`, + labels: [], + category, + site_release_id: String(id), + }; + const pubdate = normalizeDatetime(result.created_at); + if (pubdate) t.pubdate = pubdate; + const pubUtc = isoWithZoneToUtc(result.created_at); + if (pubUtc) t.pubdate_utc = pubUtc; + if (freedate) t.freedate = freedate; + if (freedateUtc) t.freedate_utc = freedateUtc; + if (catVal) t.category_id = catVal; + torrents.push(t); + } + return torrents; +} + +export const rousiSpider: SpecializedSpider = { + parser: "RousiPro", + credential: "apiKey", + pageSize: () => 100, + async search(ctx, q) { + const apikey = await ctx.secret("apiKey", true); + const data = await apiRequest(ctx, `${rousiBase(ctx.def)}/api/v1/torrents`, { + method: "GET", + headers: { Authorization: `Bearer ${apikey}`, Accept: "application/json" }, + query: rousiSearchParams(q), + }); + if (!isPlainObject(data)) throw businessError(ctx, "搜索", "搜索响应结构无效"); + if (data.code === 0) { + const d = isPlainObject(data.data) ? data.data : {}; + return parseRousiResult((d.torrents as RousiItem[]) ?? [], ctx.def); + } + throw businessError(ctx, "搜索", data.message); + }, + async resolveDownload(ctx, siteReleaseId) { + const apikey = await ctx.secret("apiKey", true); + const id = checkReleaseId(ctx.def.id, siteReleaseId); + // MP __get_download_url:method=get, cookie=False, Bearer, result=data.download_url + const url = await resolveIndirectDownloadUrl(ctx, `${rousiBase(ctx.def)}/api/v1/torrents/${id}`, { + method: "get", + header: { Authorization: `Bearer ${apikey}`, Accept: "application/json" }, + result: "data.download_url", + }); + return downloadTorrentFile(ctx, url); + }, +}; diff --git a/packages/adapters/src/sites/spiders/sunnypt.test.ts b/packages/adapters/src/sites/spiders/sunnypt.test.ts new file mode 100644 index 00000000..368f28de --- /dev/null +++ b/packages/adapters/src/sites/spiders/sunnypt.test.ts @@ -0,0 +1,71 @@ +import { + createMockFetch, + SUNNYPT_CATEGORIES_OK, + SUNNYPT_DOWNLOAD_TOKEN_OK, + SUNNYPT_SEARCH_ERROR, + SUNNYPT_SEARCH_OK, +} from "@umt/testkit"; +import { describe, expect, it } from "vitest"; +import { sunnyptBuildParams, sunnyptParseCategoryItems, sunnyptSpider } from "./sunnypt.ts"; +import { adapterWith, expectSiteError, testDef, withTorrent } from "./testing.ts"; + +const def = testDef("SunnyPT", { + domain: "https://sunnypt.top/", + api_url: "https://api.sunnypt.top/api/v1/mp", + category: { movie: [{ id: 401 }], tv: [{ id: 402 }] }, +}); + +describe("SunnyPTSpider(MP sunnypt.py)", () => { + it("_parse_category_items / _build_params", () => { + const map = sunnyptParseCategoryItems(SUNNYPT_CATEGORIES_OK.data); + expect(map).toEqual({ movie: ["401", "405"], tv: ["402", "405"], music: ["406"] }); + expect(sunnyptBuildParams({ keyword: "x", mtype: "tv", page: 1 }, map)).toEqual({ + page: 2, + page_size: 100, + sort: "created_at", + order: "desc", + keyword: "x", + media_type: "tv", + categories: "402,405", + }); + }); + + it("列表解析:分类接口缓存、促销、H&R 布尔、标签", async () => { + const m = createMockFetch({ + "GET https://api.sunnypt.top/api/v1/mp/categories": { body: SUNNYPT_CATEGORIES_OK }, + "GET https://api.sunnypt.top/api/v1/mp/torrents": { body: SUNNYPT_SEARCH_OK }, + }); + const a = adapterWith(def, sunnyptSpider, m.fetch); + const page = await a.search("arcane", { mtype: "tv" }); + await a.search("arcane", { mtype: "tv" }); + expect(m.requests.filter((r) => r.url.pathname.endsWith("/categories"))).toHaveLength(1); + expect(m.requests[0]!.headers["x-api-key"]).toBe("APIKEYSECRET"); + expect(m.requests[1]!.url.searchParams.get("categories")).toBe("402,405"); + const [x, y] = page.items; + expect(x).toMatchObject({ + siteReleaseId: "7001", + category: { id: "402" }, + promotion: { type: "free", downloadFactor: 0, uploadFactor: 1, expiresAt: "2026-09-30T15:59:59.000Z" }, + hr: true, + tags: ["官方", "中字"], + imdbId: "tt11126994", + detailUrl: "https://sunnypt.top/details.php?id=7001", + }); + expect(y).toMatchObject({ hr: false, promotion: { type: "none" }, sizeBytes: 9000000000 }); + }); + + it("业务错误 / 分类接口失败回退配置 / 下载 token 相对地址按 API 基址补全", async () => { + const bad = createMockFetch({ "/api/v1/mp/torrents": { body: SUNNYPT_SEARCH_ERROR } }); + await expectSiteError(adapterWith(def, sunnyptSpider, bad.fetch).list(1), "auth_invalid"); + const fb = createMockFetch({ + "/api/v1/mp/categories": { status: 500, body: "x" }, + "/api/v1/mp/torrents": { body: SUNNYPT_SEARCH_OK }, + }); + await adapterWith(def, sunnyptSpider, fb.fetch).search("x", { mtype: "movie" }); + expect(fb.requests.at(-1)!.url.searchParams.get("categories")).toBe("401"); + const d = createMockFetch({ "POST /api/v1/mp/torrents/7001/download-token": { body: SUNNYPT_DOWNLOAD_TOKEN_OK } }); + const t = withTorrent(d.fetch, (u) => u.pathname.endsWith("/download")); + await adapterWith(def, sunnyptSpider, t.fetch).downloadTarget("7001"); + expect(t.hits[0]!.url.toString()).toBe("https://api.sunnypt.top/api/v1/mp/torrents/7001/download?token=TOK"); + }); +}); diff --git a/packages/adapters/src/sites/spiders/sunnypt.ts b/packages/adapters/src/sites/spiders/sunnypt.ts new file mode 100644 index 00000000..c235e6d0 --- /dev/null +++ b/packages/adapters/src/sites/spiders/sunnypt.ts @@ -0,0 +1,225 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/modules/indexer/spider/sunnypt.py @ e6820a8 + * + * SunnyPT MoviePilot API(parser "SunnyPT",X-API-Key)。函数对照 `SunnyPTSpider`: + * - `_size/_category_cache_ttl` → `SUNNYPT_PAGE_SIZE/SUNNYPT_CATEGORY_CACHE_TTL_MS` + * - `_parse_configured_categories` → `sunnyptConfiguredCategories`;`_parse_category_items` → `sunnyptParseCategoryItems` + * - `_get_cached_categories/_set_cached_categories`(类级缓存)→ 适配器实例缓存 `ctx.cache` + * - `_request_headers` → `sunnyptHeaders`;`_response_data` → `sunnyptResponseData` + * - `_load_categories/_async_load_categories` → `sunnyptLoadCategories` + * - `_media_type_value` → `mtype` 直用;`_build_params` → `sunnyptBuildParams`;`_parse_result` → `parseSunnyptResult` + * - `_process_search_response` + `search` → `sunnyptSpider.search` + * - `_build_download_url`(POST download-token,result=data.download_url,result_base_url=api_url)→ `resolveDownload` + * 差异:分类接口失败时与 MP 一样回退站点配置分类,但凭据错误(HTTP 401/403)直接抛出,不再静默。 + */ +import type { MpMediaType } from "../spider/spider.ts"; +import { SiteError } from "../types.ts"; +import { normalizeDatetime } from "../user-parsers/common.ts"; +import { + apiRequest, + businessError, + cacheGet, + cacheSet, + checkReleaseId, + downloadTorrentFile, + isoWithZoneToUtc, + isPlainObject, + resolveIndirectDownloadUrl, + toFloat, + toInt, +} from "./common.ts"; +import type { + SpecializedSiteDefinition, + SpecializedSpider, + SpecializedTorrent, + SpiderContext, + SpiderSearchQuery, +} from "./types.ts"; + +export const SUNNYPT_PAGE_SIZE = 100; +export const SUNNYPT_CATEGORY_CACHE_TTL_MS = 3600 * 1000; + +export type SunnyptCategoryMap = { movie: string[]; tv: string[]; music: string[] }; + +export function sunnyptApiUrl(def: SpecializedSiteDefinition): string { + return String(def.umt.apiBaseUrl ?? def.api_url ?? "https://api.sunnypt.top/api/v1/mp").replace(/\/+$/, ""); +} + +/** MP `_parse_configured_categories` */ +export function sunnyptConfiguredCategories(config: SpecializedSiteDefinition["category"]): SunnyptCategoryMap { + const map: SunnyptCategoryMap = { movie: [], tv: [], music: [] }; + for (const k of ["movie", "tv", "music"] as const) { + for (const item of config?.[k] ?? []) { + const id = typeof item === "object" ? item.id : item; + if (id != null) map[k].push(String(id)); + } + } + return map; +} + +/** MP `_parse_category_items` */ +export function sunnyptParseCategoryItems(items: unknown[]): SunnyptCategoryMap { + const map: SunnyptCategoryMap = { movie: [], tv: [], music: [] }; + for (const item of items ?? []) { + if (!isPlainObject(item) || item.id == null) continue; + const id = String(item.id); + for (const mt of (item.media_types as unknown[]) ?? []) { + if (mt === "movie" || mt === "tv" || mt === "music") { + if (!map[mt].includes(id)) map[mt].push(id); + } + } + } + return map; +} + +function sunnyptHeaders(apikey: string): Record { + return { Accept: "application/json", "X-API-Key": apikey }; +} + +/** MP `_response_data`:code 不为 "0" → 失败 */ +function sunnyptResponseData(ctx: SpiderContext, payload: unknown, operation: string): unknown { + if (!isPlainObject(payload)) throw businessError(ctx, operation, "响应结构无效"); + if (String(payload.code) !== "0") throw businessError(ctx, operation, payload.msg || "未知错误"); + return payload.data; +} + +/** MP `_load_categories`:读取并缓存分类映射,失败回退站点配置 */ +export async function sunnyptLoadCategories(ctx: SpiderContext, apikey: string): Promise { + const api = sunnyptApiUrl(ctx.def); + const cached = cacheGet(ctx, `sunnypt:categories:${api}`); + if (cached) return cached; + const configured = sunnyptConfiguredCategories(ctx.def.category); + let items: unknown; + try { + items = sunnyptResponseData( + ctx, + await apiRequest(ctx, `${api}/categories`, { method: "GET", headers: sunnyptHeaders(apikey) }), + "获取分类", + ); + } catch (e) { + if (e instanceof SiteError && e.isAuthError) throw e; + return configured; + } + const map = Array.isArray(items) ? sunnyptParseCategoryItems(items) : null; + if (!map || !Object.values(map).some((v) => v.length)) return configured; + cacheSet(ctx, `sunnypt:categories:${api}`, map, SUNNYPT_CATEGORY_CACHE_TTL_MS); + return map; +} + +/** MP `_build_params` */ +export function sunnyptBuildParams(q: SpiderSearchQuery, categoryMap: SunnyptCategoryMap) { + const params: Record = { + page: Number(q.page || 0) + 1, + page_size: SUNNYPT_PAGE_SIZE, + sort: "created_at", + order: "desc", + }; + if (q.keyword) params.keyword = q.keyword; + const apiMediaType = q.mtype ?? null; + if (apiMediaType) params.media_type = apiMediaType; + let categories = q.cat; + if (!categories && apiMediaType) categories = (categoryMap[apiMediaType] ?? []).join(","); + if (categories) params.categories = categories; + return params; +} + +export interface SunnyptItem { + id: number | string; + title?: string; + subtitle?: string; + media_type?: string; + category_id?: number | string; + promotion?: { is_active?: boolean; down_multiplier?: number; up_multiplier?: number; until?: string } | null; + created_at?: string; + size?: number | string; + seeders?: number | string; + leechers?: number | string; + completed?: number | string; + details_url?: string; + imdb_id?: string; + tags?: string[]; + hit_and_run?: boolean | number; + [k: string]: unknown; +} + +/** MP `_parse_result` */ +export function parseSunnyptResult(results: unknown[] | null | undefined): SpecializedTorrent[] { + const torrents: SpecializedTorrent[] = []; + for (const r of results ?? []) { + if (!isPlainObject(r)) continue; + const result = r as SunnyptItem; + let category: MpMediaType; + if (result.media_type === "movie") category = "电影"; + else if (result.media_type === "tv") category = "电视剧"; + else if (result.media_type === "music") category = "音乐"; + else category = "未知"; + const promotion = result.promotion ?? {}; + const active = Boolean(promotion.is_active); + const t: SpecializedTorrent = { + title: result.title, + description: result.subtitle, + size: toInt(result.size), + seeders: toInt(result.seeders), + peers: toInt(result.leechers), + grabs: toInt(result.completed), + downloadvolumefactor: active ? toFloat(promotion.down_multiplier ?? 1.0) : 1.0, + uploadvolumefactor: active ? toFloat(promotion.up_multiplier ?? 1.0) : 1.0, + labels: result.tags ?? [], + hit_and_run: Boolean(result.hit_and_run), + category, + site_release_id: String(result.id), + }; + if (result.details_url) t.page_url = result.details_url; + if (result.imdb_id) t.imdbid = result.imdb_id; + if (active && promotion.until) { + const fd = normalizeDatetime(promotion.until); + if (fd) t.freedate = fd; + const fu = isoWithZoneToUtc(promotion.until); + if (fu) t.freedate_utc = fu; + } + const pubdate = normalizeDatetime(result.created_at); + if (pubdate) t.pubdate = pubdate; + const pu = isoWithZoneToUtc(result.created_at); + if (pu) t.pubdate_utc = pu; + if (result.category_id != null) t.category_id = String(result.category_id); + torrents.push(t); + } + return torrents; +} + +export const sunnyptSpider: SpecializedSpider = { + parser: "SunnyPT", + credential: "apiKey", + pageSize: () => SUNNYPT_PAGE_SIZE, + async search(ctx, q) { + const apikey = (await ctx.secret("apiKey", true))!; + const categoryMap = + q.mtype && !q.cat ? await sunnyptLoadCategories(ctx, apikey) : sunnyptConfiguredCategories(ctx.def.category); + const data = sunnyptResponseData( + ctx, + await apiRequest(ctx, `${sunnyptApiUrl(ctx.def)}/torrents`, { + method: "GET", + headers: sunnyptHeaders(apikey), + query: sunnyptBuildParams(q, categoryMap), + }), + "搜索", + ); + if (!isPlainObject(data)) throw businessError(ctx, "搜索", "data 结构无效"); + return parseSunnyptResult((data.items as unknown[]) ?? []); + }, + async resolveDownload(ctx, siteReleaseId) { + const apikey = (await ctx.secret("apiKey", true))!; + const id = checkReleaseId(ctx.def.id, siteReleaseId); + const api = sunnyptApiUrl(ctx.def); + // MP _build_download_url:POST download-token,result=data.download_url,result_base_url=api_url + const url = await resolveIndirectDownloadUrl(ctx, `${api}/torrents/${id}/download-token`, { + method: "post", + header: { "X-API-Key": apikey, Accept: "application/json" }, + result: "data.download_url", + result_base_url: api, + }); + return downloadTorrentFile(ctx, url); + }, +}; diff --git a/packages/adapters/src/sites/spiders/testing.ts b/packages/adapters/src/sites/spiders/testing.ts new file mode 100644 index 00000000..97d1b2dc --- /dev/null +++ b/packages/adapters/src/sites/spiders/testing.ts @@ -0,0 +1,74 @@ +/** 测试辅助(非公开 API):构造适配器、在 mock fetch 前挂二进制 .torrent 路由。 */ +import { buildFakeTorrent } from "@umt/testkit"; +import { createSiteRateLimiter } from "../rate-limit.ts"; +import { type SiteAdapter, type SiteCredentialKind, SiteError, staticSecretResolver } from "../types.ts"; +import { createSpecializedSiteAdapter } from "./adapter.ts"; +import type { SpecializedSiteDefinition, SpecializedSpider } from "./types.ts"; + +export function testDef( + parser: SpecializedSiteDefinition["parser"], + over: Partial = {}, +): SpecializedSiteDefinition { + return { + id: "demo", + name: "Demo", + domain: "https://pt.example.com/", + parser, + ...over, + umt: { + version: "demo@0.0.0-test", + kind: "api", + displayName: "Demo", + timezone: "+08:00", + rateLimit: { minIntervalMs: 0, burst: 10 }, + capabilities: { list: true, search: true, detail: false, download: true, rss: false }, + ...(over.umt ?? {}), + }, + }; +} + +export function adapterWith( + def: SpecializedSiteDefinition, + spider: SpecializedSpider, + fetchImpl: typeof globalThis.fetch, + secrets: Partial> = { cookie: "c=COOKIESECRET", apiKey: "APIKEYSECRET" }, +): SiteAdapter { + return createSpecializedSiteAdapter(def, spider, { + fetch: fetchImpl, + secrets: staticSecretResolver({ [def.id]: secrets }), + limiter: createSiteRateLimiter({ minIntervalMs: 0 }), + }); +} + +/** 包一层 fetch:命中 predicate 的请求返回合法 .torrent 字节 */ +export function withTorrent( + inner: typeof globalThis.fetch, + predicate: (url: URL) => boolean, + name = "demo", +): { fetch: typeof globalThis.fetch; infoHash: string; hits: Array<{ url: URL; headers: Headers }> } { + const t = buildFakeTorrent({ name }); + const hits: Array<{ url: URL; headers: Headers }> = []; + const f = (async (input: string | URL | Request, init?: RequestInit) => { + const url = new URL(typeof input === "string" ? input : input instanceof URL ? input.href : input.url); + if (predicate(url)) { + hits.push({ url, headers: new Headers(init?.headers) }); + return new Response(t.bytes as Uint8Array, { + status: 200, + headers: { "content-type": "application/x-bittorrent" }, + }); + } + return inner(input, init); + }) as typeof globalThis.fetch; + return { fetch: f, infoHash: t.infoHash, hits }; +} + +export async function expectSiteError(p: Promise, kind: string): Promise { + try { + await p; + } catch (e) { + if (!(e instanceof SiteError)) throw e; + if (e.kind !== kind) throw new Error(`expected SiteError ${kind}, got ${e.kind}: ${e.message}`); + return e; + } + throw new Error(`expected SiteError ${kind}`); +} diff --git a/packages/adapters/src/sites/spiders/tnode.test.ts b/packages/adapters/src/sites/spiders/tnode.test.ts new file mode 100644 index 00000000..a457b10c --- /dev/null +++ b/packages/adapters/src/sites/spiders/tnode.test.ts @@ -0,0 +1,52 @@ +import { createMockFetch, TNODE_HOME_HTML, TNODE_HOME_NO_TOKEN, TNODE_SEARCH_OK } from "@umt/testkit"; +import { describe, expect, it } from "vitest"; +import { adapterWith, expectSiteError, testDef, withTorrent } from "./testing.ts"; +import { parseTnodeToken, tnodeSearchParams, tnodeSpider } from "./tnode.ts"; + +const def = testDef("TNodeSpider", { domain: "https://tnode.example/" }); + +describe("TNodeSpider(MP tnode.py)", () => { + it("__get_params / token 解析", () => { + expect(tnodeSearchParams("tt0923811", 1)).toMatchObject({ + page: 2, + size: 100, + type: "imdbid", + keyword: "tt0923811", + }); + expect(tnodeSearchParams(null)).toMatchObject({ type: "title", keyword: "", category: [501, 502, 503, 504] }); + expect(parseTnodeToken(TNODE_HOME_HTML)).toBe("CSRF-TOKEN-XYZ"); + }); + + it("列表解析:CSRF token 缓存、系数直取、时间戳转 UTC", async () => { + const m = createMockFetch({ + "GET /": { body: TNODE_HOME_HTML }, + "POST /api/torrent/advancedSearch": { body: TNODE_SEARCH_OK }, + }); + const a = adapterWith(def, tnodeSpider, m.fetch); + const page = await a.search("eva"); + await a.list(1); + expect(m.requests.filter((r) => r.url.pathname === "/")).toHaveLength(1); + expect(m.requests[1]!.headers["x-csrf-token"]).toBe("CSRF-TOKEN-XYZ"); + const [x, y] = page.items; + expect(x).toMatchObject({ + siteReleaseId: "3301", + promotion: { type: "free", downloadFactor: 0, uploadFactor: 1 }, + uploadedAt: "2024-09-20T00:00:00.000Z", + imdbId: "tt0923811", + detailUrl: "https://tnode.example/torrent/info/3301", + hr: "unknown", + }); + expect(y!.promotion.type).toBe("2x50%"); + const t = withTorrent(m.fetch, (u) => u.pathname.startsWith("/api/torrent/download/")); + await adapterWith(def, tnodeSpider, t.fetch).downloadTarget("3301"); + expect(t.hits[0]!.headers.get("cookie")).toBe("c=COOKIESECRET"); + }); + + it("取不到 token → auth_expired;缺 Cookie → auth_missing;搜索 HTTP 403 → auth_invalid", async () => { + const m = createMockFetch({ "GET /": { body: TNODE_HOME_NO_TOKEN } }); + await expectSiteError(adapterWith(def, tnodeSpider, m.fetch).list(1), "auth_expired"); + await expectSiteError(adapterWith(def, tnodeSpider, m.fetch, {}).list(1), "auth_missing"); + const f = createMockFetch({ "GET /": { body: TNODE_HOME_HTML }, "/api/torrent/advancedSearch": { status: 403 } }); + await expectSiteError(adapterWith(def, tnodeSpider, f.fetch).list(1), "auth_invalid"); + }); +}); diff --git a/packages/adapters/src/sites/spiders/tnode.ts b/packages/adapters/src/sites/spiders/tnode.ts new file mode 100644 index 00000000..23b62ad3 --- /dev/null +++ b/packages/adapters/src/sites/spiders/tnode.ts @@ -0,0 +1,127 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/modules/indexer/spider/tnode.py @ e6820a8 + * + * TNode(parser "TNodeSpider",Cookie + 首页 CSRF token + `api/torrent/advancedSearch`)。函数对照 `TNodeSpider`: + * - `get_search_page_size` → `pageSize`(100) + * - `__get_token/__async_get_token`(@cached 24h)→ `tnodeToken`(适配器实例缓存,TTL 24h) + * - `__parse_token_response` → `parseTnodeToken` + * - `__get_params` → `tnodeSearchParams`;`__parse_result` → `parseTnodeResult`;`__process_response` → `tnodeSpider.search` + * - `_downloadurl/_pageurl` → `resolveDownload`(带 Cookie 下载)/ page_url + * 差异:取不到 token 时 MP 返回 `(True, [])`;这里抛 auth_expired(首页取不到 token 基本意味着 Cookie 失效)。 + */ +import { SiteError } from "../types.ts"; +import { + apiRequest, + cacheGet, + cacheSet, + checkReleaseId, + downloadTorrentFile, + formatTimestampTz, + isPlainObject, + siteBase, + textRequest, +} from "./common.ts"; +import type { SpecializedSiteDefinition, SpecializedSpider, SpecializedTorrent, SpiderContext } from "./types.ts"; + +export const TNODE_TOKEN_TTL_MS = 24 * 3600 * 1000; + +/** MP `__parse_token_response` */ +export function parseTnodeToken(html: string): string | null { + return //.exec(html)?.[1] ?? null; +} + +async function tnodeToken(ctx: SpiderContext, cookie: string): Promise { + const key = `tnode:token:${ctx.def.domain}`; + const cached = cacheGet(ctx, key); + if (cached) return cached; + const token = parseTnodeToken(await textRequest(ctx, siteBase(ctx.def), { cookie })); + if (!token) + throw new SiteError(ctx.def.id, "auth_expired", `${ctx.def.name} 未获取到 token,无法搜索(Cookie 可能失效)`); + cacheSet(ctx, key, token, TNODE_TOKEN_TTL_MS); + return token; +} + +/** MP `__get_params` */ +export function tnodeSearchParams(keyword: string | null | undefined, page = 0) { + return { + page: Number(page) + 1, + size: 100, + type: keyword?.startsWith("tt") ? "imdbid" : "title", + keyword: keyword || "", + sorter: "id", + order: "desc", + tags: [], + category: [501, 502, 503, 504], + medium: [], + videoCoding: [], + audioCoding: [], + resolution: [], + group: [], + }; +} + +export interface TnodeItem { + id: number | string; + title?: string; + subtitle?: string; + upload_time?: number | string; + size?: number; + seeding?: number; + leeching?: number; + complete?: number; + downloadRate?: number; + uploadRate?: number; + imdb?: string; + [k: string]: unknown; +} + +/** MP `__parse_result` */ +export function parseTnodeResult(results: TnodeItem[] | null | undefined, def: SpecializedSiteDefinition) { + const torrents: SpecializedTorrent[] = []; + if (!results?.length) return torrents; + const base = siteBase(def); + for (const result of results) { + const t: SpecializedTorrent = { + title: result.title, + description: result.subtitle, + enclosure: `${base}api/torrent/download/${result.id}`, + size: result.size, + seeders: result.seeding, + peers: result.leeching, + grabs: result.complete, + page_url: `${base}torrent/info/${result.id}`, + site_release_id: String(result.id), + }; + if (result.downloadRate != null) t.downloadvolumefactor = result.downloadRate; + if (result.uploadRate != null) t.uploadvolumefactor = result.uploadRate; + if (result.imdb) t.imdbid = result.imdb; + const pubdate = formatTimestampTz(result.upload_time, def.umt.timezone); + if (pubdate) t.pubdate = pubdate; + torrents.push(t); + } + return torrents; +} + +export const tnodeSpider: SpecializedSpider = { + parser: "TNodeSpider", + credential: "cookie", + pageSize: () => 100, + async search(ctx, q) { + const cookie = (await ctx.secret("cookie", true))!; + const token = await tnodeToken(ctx, cookie); + const data = await apiRequest(ctx, `${siteBase(ctx.def)}api/torrent/advancedSearch`, { + method: "POST", + headers: { "X-CSRF-TOKEN": token, "Content-Type": "application/json; charset=utf-8", cookie }, + json: tnodeSearchParams(q.keyword, q.page ?? 0), + }); + const d = isPlainObject(data) && isPlainObject(data.data) ? data.data : {}; + return parseTnodeResult((d.torrents as TnodeItem[]) ?? [], ctx.def); + }, + async resolveDownload(ctx, siteReleaseId, enclosure) { + const cookie = await ctx.secret("cookie", true); + const id = checkReleaseId(ctx.def.id, siteReleaseId); + return downloadTorrentFile(ctx, enclosure ?? `${siteBase(ctx.def)}api/torrent/download/${id}`, { cookie }); + }, +}; diff --git a/packages/adapters/src/sites/spiders/torrentleech.test.ts b/packages/adapters/src/sites/spiders/torrentleech.test.ts new file mode 100644 index 00000000..15efc9d0 --- /dev/null +++ b/packages/adapters/src/sites/spiders/torrentleech.test.ts @@ -0,0 +1,64 @@ +import { createMockFetch, TORRENTLEECH_SEARCH_OK } from "@umt/testkit"; +import { describe, expect, it } from "vitest"; +import { adapterWith, expectSiteError, testDef, withTorrent } from "./testing.ts"; +import { torrentleechSearchUrl, torrentleechSpider } from "./torrentleech.ts"; + +const def = testDef("TorrentLeech", { + domain: "https://www.torrentleech.org/", + category: { movie: [{ id: 47 }, { id: 8 }], tv: [{ id: 32 }] }, + umt: { + version: "tl@test", + kind: "api", + displayName: "TL", + timezone: "+00:00", + rateLimit: { minIntervalMs: 0, burst: 10 }, + capabilities: { list: true, search: true, detail: false, download: true, rss: false }, + }, +}); + +describe("TorrentLeech(MP torrentleech.py)", () => { + it("__get_search_url:分类 exact 查询与 quote 编码;浏览页码 +1", async () => { + expect(torrentleechSearchUrl(def, "The Matrix (1999)", "movie")).toBe( + "https://www.torrentleech.org/torrents/browse/list/categories/47,8/exact/1/query/The%20Matrix%20%281999%29", + ); + expect(torrentleechSearchUrl(def, "a/b")).toBe("https://www.torrentleech.org/torrents/browse/list/query/a/b"); + const m = createMockFetch({ "/torrents/browse/list/page/3": { body: TORRENTLEECH_SEARCH_OK } }); + await adapterWith(def, torrentleechSpider, m.fetch).list(3); + expect(m.requests[0]!.url.pathname).toBe("/torrents/browse/list/page/3"); + }); + + it("列表解析:download_multiplier、请求分类投影、UTC 时间", async () => { + const m = createMockFetch({ + "/torrents/browse/list/categories/47,8/exact/1/query/Matrix": { body: TORRENTLEECH_SEARCH_OK }, + }); + const page = await adapterWith(def, torrentleechSpider, m.fetch).search("Matrix", { mtype: "movie" }); + const [a, b] = page.items; + expect(a).toMatchObject({ + siteReleaseId: "240001", + promotion: { type: "free", downloadFactor: 0, uploadFactor: 1 }, + uploadedAt: "2026-09-20T08:00:00.000Z", + imdbId: "tt0133093", + category: { id: "47" }, + detailUrl: "https://www.torrentleech.org/torrent/240001", + downloadUrl: "https://www.torrentleech.org/download/240001/The.Matrix.1999.2160p.UHD.BluRay.x265-TL.torrent", + }); + expect(a!.raw.category).toBe("电影"); + expect(b!.raw.category).toBe("未知"); + expect(b!.promotion.type).toBe("none"); + // 关键词搜索不翻页 + expect((await adapterWith(def, torrentleechSpider, m.fetch).search("Matrix", { page: 2 })).items).toEqual([]); + }); + + it("中文关键词返回空且不发请求;缺 Cookie;跳登录页 → auth_expired;下载带 Cookie", async () => { + const m = createMockFetch({ + "/torrents/browse/list/page/1": { status: 302, headers: { location: "/user/account/login" } }, + }); + expect((await adapterWith(def, torrentleechSpider, m.fetch).search("黑客帝国")).items).toEqual([]); + expect(m.requests).toHaveLength(0); + await expectSiteError(adapterWith(def, torrentleechSpider, m.fetch, {}).list(1), "auth_missing"); + await expectSiteError(adapterWith(def, torrentleechSpider, m.fetch).list(1), "auth_expired"); + const t = withTorrent(m.fetch, (u) => u.pathname.startsWith("/download/")); + await adapterWith(def, torrentleechSpider, t.fetch).downloadTarget("240001"); + expect(t.hits[0]!.headers.get("cookie")).toBe("c=COOKIESECRET"); + }); +}); diff --git a/packages/adapters/src/sites/spiders/torrentleech.ts b/packages/adapters/src/sites/spiders/torrentleech.ts new file mode 100644 index 00000000..3147238d --- /dev/null +++ b/packages/adapters/src/sites/spiders/torrentleech.ts @@ -0,0 +1,121 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/modules/indexer/spider/torrentleech.py @ e6820a8 + * + * TorrentLeech JSON 搜索接口(parser "TorrentLeech",Cookie)。函数对照 `TorrentLeech`: + * - `get_search_page_size` → `pageSize`(关键词搜索 null,浏览 100) + * - `__category_ids` → `torrentleechCategoryIds`;`__get_search_url` → `torrentleechSearchUrl` + * - `_browseurl` → `torrentleechBrowseUrl`;`__parse_result` → `parseTorrentleechResult` + * - `__process_response` + `search` → `torrentleechSpider.search`(含中文关键词直接返回空,MP 行为) + * 差异:MP 只在请求了媒体类型时写 category;这里另外总是输出原始 `category_id`。 + */ +import type { MediaKind } from "../spider/spider.ts"; +import { containsChinese } from "../user-parsers/common.ts"; +import { + apiRequest, + checkReleaseId, + downloadTorrentFile, + formatTimestampTz, + isPlainObject, + pyQuote, + siteBase, +} from "./common.ts"; +import type { SpecializedSiteDefinition, SpecializedSpider, SpecializedTorrent } from "./types.ts"; + +const MEDIA_VALUE = { movie: "电影", tv: "电视剧", music: "音乐" } as const; + +/** MP `__category_ids` */ +export function torrentleechCategoryIds(def: SpecializedSiteDefinition, mtype?: MediaKind): string[] { + if (!mtype) return []; + return (def.category?.[mtype] ?? []).filter((i) => i && i.id != null).map((i) => String(i.id)); +} + +/** MP `__get_search_url` */ +export function torrentleechSearchUrl(def: SpecializedSiteDefinition, keyword: string, mtype?: MediaKind): string { + const domain = siteBase(def); + const encoded = pyQuote(keyword); + const ids = torrentleechCategoryIds(def, mtype); + if (ids.length) return `${domain}torrents/browse/list/categories/${ids.join(",")}/exact/1/query/${encoded}`; + return `${domain}torrents/browse/list/query/${encoded}`; +} + +export function torrentleechBrowseUrl(def: SpecializedSiteDefinition, page = 0): string { + return `${siteBase(def)}torrents/browse/list/page/${Number(page) + 1}`; +} + +export interface TorrentleechItem { + fid: number | string; + filename?: string; + name?: string; + addedTimestamp?: string | number; + size?: number; + seeders?: number; + leechers?: number; + completed?: number; + download_multiplier?: number; + imdbID?: string; + categoryID?: number | string; + [k: string]: unknown; +} + +/** MP `__parse_result` */ +export function parseTorrentleechResult( + results: TorrentleechItem[] | null | undefined, + def: SpecializedSiteDefinition, + mtype?: MediaKind, +) { + const torrents: SpecializedTorrent[] = []; + if (!results?.length) return torrents; + const domain = siteBase(def); + const requested = new Set(torrentleechCategoryIds(def, mtype)); + for (const result of results) { + const t: SpecializedTorrent = { + title: result.name, + enclosure: `${domain}download/${result.fid}/${result.filename}`, + size: result.size, + seeders: result.seeders, + peers: result.leechers, + grabs: result.completed, + uploadvolumefactor: 1, + page_url: `${domain}torrent/${result.fid}`, + site_release_id: String(result.fid), + }; + if (result.download_multiplier != null) t.downloadvolumefactor = result.download_multiplier; + if (result.imdbID) t.imdbid = result.imdbID; + const pubdate = formatTimestampTz(result.addedTimestamp, def.umt.timezone); + if (pubdate) t.pubdate = pubdate; + if (requested.size && mtype) { + t.category = requested.has(String(result.categoryID)) ? MEDIA_VALUE[mtype] : "未知"; + } + if (result.categoryID != null) t.category_id = String(result.categoryID); + torrents.push(t); + } + return torrents; +} + +export const torrentleechSpider: SpecializedSpider = { + parser: "TorrentLeech", + credential: "cookie", + pageSize: (keyword) => (keyword ? null : 100), + async search(ctx, q) { + // 不支持中文(MP 返回 (True, [])) + if (q.keyword && containsChinese(q.keyword)) return []; + const cookie = (await ctx.secret("cookie", true))!; + const url = q.keyword + ? torrentleechSearchUrl(ctx.def, q.keyword, q.mtype) + : torrentleechBrowseUrl(ctx.def, q.page ?? 0); + const data = await apiRequest(ctx, url, { + method: "GET", + headers: { "Content-Type": "application/json; charset=utf-8", cookie }, + }); + const list = isPlainObject(data) ? (data.torrentList as TorrentleechItem[]) : []; + return parseTorrentleechResult(list ?? [], ctx.def, q.mtype); + }, + async resolveDownload(ctx, siteReleaseId, enclosure) { + const cookie = await ctx.secret("cookie", true); + const id = checkReleaseId(ctx.def.id, siteReleaseId); + // TODO(verify):无缓存直链时文件名段是否可任意 + return downloadTorrentFile(ctx, enclosure ?? `${siteBase(ctx.def)}download/${id}/${id}.torrent`, { cookie }); + }, +}; diff --git a/packages/adapters/src/sites/spiders/types.ts b/packages/adapters/src/sites/spiders/types.ts new file mode 100644 index 00000000..f1bc9057 --- /dev/null +++ b/packages/adapters/src/sites/spiders/types.ts @@ -0,0 +1,100 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/modules/indexer/__init__.py @ e6820a8 + * + * 专用 spider(API 型站点)共用类型。 + * - `SpecializedParser` = MP `SPIDER_PARSER_CLASSES` 的键;`SPECIALIZED_SEARCH_ARGUMENTS` 的参数集合 → `SpiderSearchQuery` + * - `SpecializedSiteDefinition` = 这些 spider 的 `__init__(indexer)` 读取的 indexer dict 键 + 本项目 `umt` 扩展 + */ +import type { SiteHttpContext } from "../http.ts"; +import type { SpiderCategoryConfig } from "../spider/schema.ts"; +import type { MediaKind, SpiderTorrent } from "../spider/spider.ts"; +import type { DownloadTarget, SiteCapabilities, SiteCredentialKind } from "../types.ts"; + +export type SpecializedParser = + | "Haidan" + | "HDDolby" + | "RousiPro" + | "SunnyPT" + | "TNodeSpider" + | "TorrentLeech" + | "Yema" + | "mTorrent"; + +export interface SpecializedSiteDefinition { + id: string; + name: string; + enabled?: boolean; + /** MP 用户信息解析器 schema(SiteSchema 值) */ + schema?: string; + provenance?: import("../spider/schema.ts").DefinitionProvenance; + /** 站点前端地址(MP indexer['domain'],以 / 结尾) */ + domain: string; + parser: SpecializedParser; + timeout?: number; + proxy?: boolean; + public?: boolean; + /** 分类组(TorrentLeech 按媒体类型取分类 ID;SunnyPT 分类接口失败时兜底) */ + category?: SpiderCategoryConfig; + /** 独立 API 地址(MP indexer['api_url'],SunnyPT) */ + api_url?: string; + /** 本项目扩展(MP 无对应) */ + umt: { + version: string; + kind: "api"; + displayName: string; + /** 站点时间时区偏移(pubdate/freedate 无时区时按此转 UTC) */ + timezone: string; + rateLimit: { minIntervalMs: number; burst: number }; + capabilities: SiteCapabilities; + categoryNames?: Record; + /** 覆盖 API 基址(测试指向 mock,或站点换域) */ + apiBaseUrl?: string; + }; +} + +/** + * 专用 spider 输出:MP TorrentInfo 字段 + 本项目扩展: + * - `site_release_id`:站点资源 ID(MP 只在 page_url/enclosure 里隐含) + * - `pubdate_utc` / `freedate_utc`:源数据自带时区时的精确 UTC(MP normalize_datetime 会丢时区) + */ +export interface SpecializedTorrent extends SpiderTorrent { + site_release_id?: string; + pubdate_utc?: string; + freedate_utc?: string; +} + +/** MP `_SPECIALIZED_SEARCH_ARGUMENTS` 的并集;page 为 MP 内部 0 起页码 */ +export interface SpiderSearchQuery { + keyword?: string | null; + mtype?: MediaKind; + /** 逗号分隔的站点分类 ID(RousiPro/SunnyPT) */ + cat?: string; + page?: number; +} + +export interface SpiderContext { + def: SpecializedSiteDefinition; + http: SiteHttpContext; + /** 取凭据;required 时缺失抛 SiteError('auth_missing') */ + secret(kind: SiteCredentialKind, required?: boolean): Promise; + /** 实例内缓存(TNode CSRF token、SunnyPT 分类);值带过期时间 */ + cache: Map; + userAgent?: string; + signal?: AbortSignal; +} + +export interface SpecializedSpider { + parser: SpecializedParser; + /** 必需凭据(MP search 开头的检查) */ + credential: SiteCredentialKind; + /** MP `get_search_page_size`:null 表示不支持翻页 */ + pageSize(keyword?: string | null): number | null; + search(ctx: SpiderContext, query: SpiderSearchQuery): Promise; + /** + * 下载前现场取种子(MP `__get_download_url` + submission `_resolve_indirect_download_url` + `download_torrent`)。 + * `enclosure` 为最近一次搜索结果里的直链(仅内存,不落库);缺失时按站点规则构造。 + */ + resolveDownload(ctx: SpiderContext, siteReleaseId: string, enclosure?: string): Promise; +} diff --git a/packages/adapters/src/sites/spiders/yema.test.ts b/packages/adapters/src/sites/spiders/yema.test.ts new file mode 100644 index 00000000..e2610935 --- /dev/null +++ b/packages/adapters/src/sites/spiders/yema.test.ts @@ -0,0 +1,62 @@ +import { + createMockFetch, + YEMA_DOWNLOAD_KEY_OK, + YEMA_SEARCH_ERROR, + YEMA_SEARCH_MOVIE, + YEMA_SEARCH_TV_A, +} from "@umt/testkit"; +import { describe, expect, it } from "vitest"; +import { adapterWith, expectSiteError, testDef, withTorrent } from "./testing.ts"; +import { yemaSpider } from "./yema.ts"; + +const def = testDef("Yema", { domain: "https://www.yemapt.org/" }); +const URL_SEARCH = "POST /openApi/torrent/fetchOpenTorrentList.json"; + +describe("YemaSpider(MP yema.py)", () => { + it("列表解析:促销枚举、标签表(未知 ID 丢弃)、H&R、非整数 ID 跳过", async () => { + const m = createMockFetch({ [URL_SEARCH]: { body: YEMA_SEARCH_MOVIE } }); + const page = await adapterWith(def, yemaSpider, m.fetch).search("past", { mtype: "movie" }); + expect(m.requests[0]!.headers.authorization).toBe("APIKEYSECRET"); + expect(JSON.parse(m.requests[0]!.body!)).toEqual({ + pageParam: { current: 1, pageSize: 40 }, + sorter: {}, + keyword: "past", + categoryId: 4, + }); + expect(page.items).toHaveLength(1); + expect(page.items[0]).toMatchObject({ + siteReleaseId: "88001", + promotion: { type: "2xfree", downloadFactor: 0, uploadFactor: 2, expiresAt: "2026-09-28T16:00:00.000Z" }, + tags: ["官方", "中字"], + hr: true, + detailUrl: "https://www.yemapt.org/#/torrent/detail/88001/", + category: { id: "4" }, + }); + }); + + it("电视剧按 6 个分类逐个查询,部分失败仍合并去重", async () => { + let n = 0; + const m = createMockFetch({ + [URL_SEARCH]: () => { + n++; + return n === 2 ? { body: { success: false, errorMessage: "系统繁忙" } } : { body: YEMA_SEARCH_TV_A }; + }, + }); + const page = await adapterWith(def, yemaSpider, m.fetch).list(1, { mtype: "tv" }); + expect(m.requests.map((r) => JSON.parse(r.body!).categoryId)).toEqual([5, 6, 13, 14, 15, 17]); + expect(page.items.map((i) => i.siteReleaseId)).toEqual(["88002"]); + expect(page.items[0]!.promotion).toMatchObject({ type: "50%", downloadFactor: 0.5, uploadFactor: 1.5 }); + }); + + it("全部失败才报错;缺 AuthKey;下载 key 拼成 download1?token=", async () => { + const bad = createMockFetch({ [URL_SEARCH]: { body: YEMA_SEARCH_ERROR } }); + await expectSiteError(adapterWith(def, yemaSpider, bad.fetch).list(1), "auth_invalid"); + await expectSiteError(adapterWith(def, yemaSpider, bad.fetch, {}).list(1), "auth_missing"); + const d = createMockFetch({ "POST /openApi/torrent/generateDownloadKey.json": { body: YEMA_DOWNLOAD_KEY_OK } }); + const t = withTorrent(d.fetch, (u) => u.pathname === "/api/torrent/download1"); + await adapterWith(def, yemaSpider, t.fetch).downloadTarget("88001"); + expect(d.requests[0]!.url.searchParams.get("id")).toBe("88001"); + expect(t.hits[0]!.url.toString()).toBe("https://www.yemapt.org/api/torrent/download1?token=DLKEY-123"); + expect(t.hits[0]!.headers.get("authorization")).toBeNull(); + }); +}); diff --git a/packages/adapters/src/sites/spiders/yema.ts b/packages/adapters/src/sites/spiders/yema.ts new file mode 100644 index 00000000..db9471ea --- /dev/null +++ b/packages/adapters/src/sites/spiders/yema.ts @@ -0,0 +1,226 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/modules/indexer/spider/yema.py @ e6820a8 + * + * YemaPT 开放 API(parser "Yema",Authorization: AuthKey)。函数对照 `YemaSpider`: + * - `_size/_movie_category/_tv_category/_music_category/_labels` → `YEMA_*` + * - `_request_headers` → `yemaHeaders`;`_build_params` → `yemaBuildParams` + * - `_search_category_ids` → `yemaSearchCategoryIds`;`_merge_search_results` → `yemaMergeSearchResults` + * - `_parse_result` → `parseYemaResult`;`_process_search_response` → `yemaProcessSearchResponse` + * - `_prepare_search_requests/_finalize_search_responses/search` → `yemaSpider.search`(全部分类失败才算失败) + * - `_download_factor/_upload_factor` → `yemaDownloadFactor/yemaUploadFactor` + * - `_build_download_url`(POST generateDownloadKey.json?id=,success/result=data → api/torrent/download1?token=)→ `resolveDownload` + */ +import type { MpMediaType } from "../spider/spider.ts"; +import { SiteError } from "../types.ts"; +import { normalizeDatetime } from "../user-parsers/common.ts"; +import { + apiRequest, + businessError, + checkReleaseId, + downloadTorrentFile, + isoWithZoneToUtc, + isPlainObject, + resolveIndirectDownloadUrl, +} from "./common.ts"; +import type { + SpecializedSiteDefinition, + SpecializedSpider, + SpecializedTorrent, + SpiderContext, + SpiderSearchQuery, +} from "./types.ts"; + +/** YemaPT 开放 API 使用更大的分页容量时会返回空结果 */ +export const YEMA_PAGE_SIZE = 40; +export const YEMA_MOVIE_CATEGORY = [4]; +export const YEMA_TV_CATEGORY = [5, 6, 13, 14, 15, 17]; +export const YEMA_MUSIC_CATEGORY = [8, 16]; +export const YEMA_LABELS: Record = { + "1": "禁转", + "2": "首发", + "3": "官方", + "4": "自制", + "5": "国语", + "6": "中字", + "7": "粤语", + "8": "英字", + "9": "HDR10", + "10": "杜比视界", + "11": "分集", + "12": "完结", +}; + +export function yemaSiteUrl(def: SpecializedSiteDefinition): string { + return String(def.umt.apiBaseUrl ?? def.domain ?? "https://www.yemapt.org/").replace(/\/+$/, ""); +} + +function yemaHeaders(apikey: string): Record { + return { Authorization: apikey, "Content-Type": "application/json", Accept: "application/json, text/plain, */*" }; +} + +/** MP `_build_params` */ +export function yemaBuildParams( + keyword: string | null | undefined, + page: number | undefined, + categoryId?: number | null, +) { + const params: Record = { + pageParam: { current: Number(page || 0) + 1, pageSize: YEMA_PAGE_SIZE }, + sorter: {}, + }; + if (keyword) params.keyword = keyword; + if (categoryId != null) params.categoryId = categoryId; + return params; +} + +/** MP `_search_category_ids`:分类参数不支持数组,逐个查询 */ +export function yemaSearchCategoryIds(mtype?: SpiderSearchQuery["mtype"]): Array { + if (mtype === "movie") return [...YEMA_MOVIE_CATEGORY]; + if (mtype === "tv") return [...YEMA_TV_CATEGORY]; + if (mtype === "music") return [...YEMA_MUSIC_CATEGORY]; + return [null]; +} + +/** MP `_merge_search_results`:按详情链接合并去重,保持首次出现顺序 */ +export function yemaMergeSearchResults(groups: SpecializedTorrent[][]): SpecializedTorrent[] { + const merged: SpecializedTorrent[] = []; + const seen = new Set(); + for (const results of groups) { + for (const t of results) { + const identity = t.page_url || t.enclosure; + if (identity && seen.has(identity)) continue; + if (identity) seen.add(identity); + merged.push(t); + } + } + return merged; +} + +export function yemaDownloadFactor(promotion: string): number { + return ({ free: 0, half: 0.5, none: 1 } as Record)[promotion] ?? 1; +} + +export function yemaUploadFactor(promotion: string): number { + return ({ none: 1, one_half: 1.5, double_upload: 2 } as Record)[promotion] ?? 1; +} + +export interface YemaItem { + id: number; + showName?: string; + shortDesc?: string; + categoryId?: number; + tagList?: string[]; + listingTime?: string; + fileSize?: number; + seedNum?: number; + leechNum?: number; + completedNum?: number; + downloadPromotion?: string; + uploadPromotion?: string; + downloadPromotionEndTime?: string; + hrPunishEnable?: boolean; + [k: string]: unknown; +} + +/** MP `_parse_result` */ +export function parseYemaResult(results: unknown[] | null | undefined, def: SpecializedSiteDefinition) { + const torrents: SpecializedTorrent[] = []; + const site = yemaSiteUrl(def); + for (const r of results ?? []) { + if (!isPlainObject(r)) continue; + const result = r as YemaItem; + const cv = result.categoryId; + let category: MpMediaType; + if (cv != null && YEMA_TV_CATEGORY.includes(cv)) category = "电视剧"; + else if (cv != null && YEMA_MOVIE_CATEGORY.includes(cv)) category = "电影"; + else if (cv != null && YEMA_MUSIC_CATEGORY.includes(cv)) category = "音乐"; + else category = "未知"; + // MP:label_id in self._labels(键为字符串) + const labels = (result.tagList ?? []) + .filter((id) => typeof id === "string" && id in YEMA_LABELS) + .map((id) => YEMA_LABELS[id]!); + const id = result.id; + if (typeof id !== "number" || !Number.isInteger(id)) continue; + const t: SpecializedTorrent = { + title: result.showName, + description: result.shortDesc, + size: result.fileSize, + seeders: result.seedNum, + peers: result.leechNum, + grabs: result.completedNum, + downloadvolumefactor: yemaDownloadFactor(String(result.downloadPromotion || "")), + uploadvolumefactor: yemaUploadFactor(String(result.uploadPromotion || "")), + page_url: `${site}/#/torrent/detail/${id}/`, + labels, + hit_and_run: Boolean(result.hrPunishEnable), + category, + site_release_id: String(id), + }; + const pubdate = normalizeDatetime(result.listingTime); + if (pubdate) t.pubdate = pubdate; + const pu = isoWithZoneToUtc(result.listingTime); + if (pu) t.pubdate_utc = pu; + const fd = normalizeDatetime(result.downloadPromotionEndTime); + if (fd) t.freedate = fd; + const fu = isoWithZoneToUtc(result.downloadPromotionEndTime); + if (fu) t.freedate_utc = fu; + if (cv != null) t.category_id = String(cv); + torrents.push(t); + } + return torrents; +} + +/** MP `_process_search_response` */ +export function yemaProcessSearchResponse(ctx: SpiderContext, payload: unknown): SpecializedTorrent[] { + if (!isPlainObject(payload)) throw businessError(ctx, "搜索", "搜索响应结构无效"); + if (!payload.success) throw businessError(ctx, "搜索", payload.errorMessage || "未知错误"); + if (!Array.isArray(payload.data)) throw businessError(ctx, "搜索", "搜索响应 data 不是数组"); + return parseYemaResult(payload.data, ctx.def); +} + +export const yemaSpider: SpecializedSpider = { + parser: "Yema", + credential: "apiKey", + pageSize: () => YEMA_PAGE_SIZE, + async search(ctx, q) { + const apikey = (await ctx.secret("apiKey", true))!; + const url = `${yemaSiteUrl(ctx.def)}/openApi/torrent/fetchOpenTorrentList.json`; + const groups: SpecializedTorrent[][] = []; + const errors: unknown[] = []; + for (const categoryId of yemaSearchCategoryIds(q.mtype)) { + try { + const payload = await apiRequest(ctx, url, { + method: "POST", + headers: yemaHeaders(apikey), + json: yemaBuildParams(q.keyword, q.page, categoryId), + }); + groups.push(yemaProcessSearchResponse(ctx, payload)); + } catch (e) { + // 凭据错误所有分类都会失败,直接抛出 + if (e instanceof SiteError && e.isAuthError) throw e; + errors.push(e); + } + } + // MP:all(errors) 才算失败 + if (!groups.length && errors.length) throw errors[0]; + return yemaMergeSearchResults(groups); + }, + async resolveDownload(ctx, siteReleaseId) { + const apikey = (await ctx.secret("apiKey", true))!; + const id = checkReleaseId(ctx.def.id, siteReleaseId); + const site = yemaSiteUrl(ctx.def); + const url = await resolveIndirectDownloadUrl(ctx, `${site}/openApi/torrent/generateDownloadKey.json`, { + method: "post", + header: { Authorization: apikey, Accept: "application/json" }, + params: { id: Number(id) }, + success: "success", + result: "data", + result_base_url: site, + result_path: "api/torrent/download1", + result_query_param: "token", + }); + return downloadTorrentFile(ctx, url); + }, +}; diff --git a/packages/adapters/src/sites/types.ts b/packages/adapters/src/sites/types.ts index 1cea0954..7c21b60a 100644 --- a/packages/adapters/src/sites/types.ts +++ b/packages/adapters/src/sites/types.ts @@ -1,6 +1,7 @@ /** 站点适配器自有类型(不依赖 @umt/contracts;W2 负责映射到 source_release / site_snapshot)。 */ -export type SiteKind = "nexusphp" | "mteam"; +/** api = 专用 API spider(spiders/*,Haidan/HDDolby/RousiPro/SunnyPT/TNode/TorrentLeech/Yema) */ +export type SiteKind = "nexusphp" | "mteam" | "api"; export interface SiteCapabilities { list: boolean; diff --git a/packages/adapters/src/sites/user-parsers/base.ts b/packages/adapters/src/sites/user-parsers/base.ts new file mode 100644 index 00000000..4b36518e --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/base.ts @@ -0,0 +1,448 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/modules/indexer/parser/__init__.py @ e6820a8 + * + * 站点用户数据解析器基类。 + * + * 迁移自 MoviePilot e6820a8(GPLv3)`app/modules/indexer/parser/__init__.py` `SiteParserBase`, + * 函数一一对应(Python 私有 `_x_y` → TS `_xY`): + * - `parse` → `parse`(页面获取顺序:首页/登录判定 → `_parse_site_page` → 基础信息 → 详情 → 未读消息 → 流量 → 做种分页) + * - `_pase_unread_msgs` → `_paseUnreadMsgs`(保留 MP 拼写)、`_parse_seeding_pages` → `_parseSeedingPages` + * - `_prepare_html_text` → `prepareHtmlText`、`num_filesize` → `numFilesize` + * - `_get_page_content` → `_getPageContent`(cookie/apikey 两种请求模式、JSON Accept、200/500/403 视为有内容、Cloudflare 挑战返回空) + * - `_parse_logged_in` → `_parseLoggedIn`、`clear` / `to_dict` → `clear` / `toDict` + * - 抽象方法 `_parse_site_page/_parse_user_base_info/_parse_user_traffic_info/_parse_user_torrent_seeding_info/ + * _parse_user_detail_info/_parse_message_unread_links/_parse_message_content` → 同名 camelCase + * + * 与 MP 的差异(安全与健壮性): + * 1. 凭据经构造参数注入,只放请求头;`toDict()`/`raw` 不含 cookie/apikey/token。 + * 2. 做种页与消息页翻页设上限(`MAX_PAGES`=50),防止站点返回自引用"下一页"导致死循环(MP 无上限)。 + * 3. 网络异常与非 200/500/403 状态:与 MP 一样返回空串继续,另把脱敏后的原因记在 `httpErrors`(入 raw)。 + * 4. 响应解码:按 Content-Type / `` 选 TextDecoder(MP `get_decoded_html_content` 用 chardet 探测)。 + */ +import { redactText } from "../../rss/redact.ts"; +import { parseSizeMp } from "../spider/spider.ts"; +import { extractDomain, isLoggedIn, underChallenge } from "./common.ts"; +import type { SiteSchemaValue, SiteUnreadMessage, SiteUserInfo, SiteUserParserOptions } from "./types.ts"; + +export type Params = Record; +export type Headers_ = Record; +/** 做种解析返回:下一页地址;null/undefined/false 表示结束("" 表示同地址再取一页,MTorrent 用法) */ +export type NextPage = string | null | undefined | false; + +const MAX_PAGES = 50; + +/** Python `urljoin` */ +export function urljoin(base: string, path: string): string { + try { + return new URL(path, base).toString(); + } catch { + return path; + } +} + +function detectCharset(contentType: string | null, bytes: Uint8Array): string { + const fromHeader = /charset=["']?([\w-]+)/i.exec(contentType ?? "")?.[1]; + if (fromHeader) return fromHeader.toLowerCase(); + const head = new TextDecoder("latin1").decode(bytes.subarray(0, 2048)); + const fromMeta = /]+charset=["']?([\w-]+)/i.exec(head)?.[1]; + return (fromMeta ?? "utf-8").toLowerCase(); +} + +function decode(bytes: Uint8Array, charset: string): string { + try { + return new TextDecoder(charset).decode(bytes); + } catch { + return new TextDecoder("utf-8").decode(bytes); + } +} + +export abstract class SiteParserBase { + /** 站点模版(MP `schema`) */ + static schema: SiteSchemaValue | null = null; + /** 请求模式 cookie/apikey(MP `request_mode`) */ + requestMode: "cookie" | "apikey" = "cookie"; + + // 站点信息 + apikey: string | null; + token: string | null; + protected _siteName: string; + protected _siteUrl: string; + protected _siteDomain: string; + protected _baseUrl: string; + protected _siteCookie: string | null; + protected _apiUrl: string | null; + protected _ua: string | null; + protected _indexHtml: string | Record = ""; + protected readonly _opts: SiteUserParserOptions; + + // 用户信息 + username: string | null = null; + userid: string | null = null; + userLevel: string | null = null; + joinAt: string | null = null; + bonus = 0.0; + // 流量信息 + upload = 0; + download = 0; + ratio = 0; + // 做种信息 + seeding = 0; + leeching = 0; + seedingSize = 0; + leechingSize = 0; + uploaded = 0; + completed = 0; + incomplete = 0; + uploadedSize = 0; + completedSize = 0; + incompleteSize = 0; + /** 做种人数, 种子大小 */ + seedingInfo: Array<[number, number]> = []; + // 未读消息 + messageUnread = 0; + messageUnreadContents: Array<[string | null, string | null, string | null]> = []; + messageReadForce = false; + // 全局附加请求头 + protected _additionHeaders: Headers_ | null = null; + // 用户基础信息 + protected _userBasicPage: string | null = null; + protected _userBasicParams: Params | null = null; + protected _userBasicMethod: string | null = null; + protected _userBasicHeaders: Headers_ | null = null; + // 用户详情 + protected _userDetailPage: string | null = "userdetails.php?id="; + protected _userDetailParams: Params | null = null; + protected _userDetailHeaders: Headers_ | null = null; + // 流量 + protected _userTrafficPage: string | null = "index.php"; + protected _userTrafficParams: Params | null = null; + protected _userTrafficHeaders: Headers_ | null = null; + // 未读消息 + protected _userMailUnreadPage: string | null = "messages.php?action=viewmailbox&box=1&unread=yes"; + protected _sysMailUnreadPage: string | null = "messages.php?action=viewmailbox&box=-2&unread=yes"; + protected _mailUnreadParams: Params | null = null; + protected _mailUnreadHeaders: Headers_ | null = null; + protected _mailContentParams: Params | null = null; + protected _mailContentHeaders: Headers_ | null = null; + // 做种 + protected _torrentSeedingPage: string | null = "getusertorrentlistajax.php?userid="; + protected _torrentSeedingParams: Params | null = null; + protected _torrentSeedingHeaders: Headers_ | null = null; + /** 错误信息(MP `err_msg`) */ + errMsg: string | null = null; + /** 本项目扩展:请求失败原因(已脱敏) */ + httpErrors: string[] = []; + + constructor(opts: SiteUserParserOptions) { + this._opts = opts; + this.apikey = opts.apikey ?? null; + this.token = opts.token ?? null; + this._siteName = opts.siteName; + this._siteUrl = opts.url; + const u = new URL(opts.url); + this._siteDomain = u.host; + this._baseUrl = `${u.protocol}//${u.host}`; + this._siteCookie = opts.cookie ?? null; + this._apiUrl = opts.apiUrl ?? null; + this._ua = opts.ua ?? null; + } + + get schema(): SiteSchemaValue { + return (this.constructor as typeof SiteParserBase).schema!; + } + + /** MP `site_schema` */ + siteSchema(): SiteSchemaValue { + return this.schema; + } + + /** MP `num_filesize` = `size_tools.parse_size` */ + static numFilesize(text: string | number | null | undefined): number { + return parseSizeMp(text); + } + numFilesize(text: string | number | null | undefined): number { + return parseSizeMp(text); + } + + /** MP `parse` */ + async parse(): Promise { + if (this.requestMode === "apikey") { + if (!this.apikey && !this.token) { + this.errMsg = this.errMsg ?? "未设置cookie 或 apikey/token"; + return; + } + this._indexHtml = {}; + } else { + this._indexHtml = await this._getPageContent(this._siteUrl); + if (!(await this._parseLoggedIn(this._indexHtml))) return; + } + await this._parseSitePage(this._indexHtml as string); + if (this._userBasicPage) { + await this._parseUserBaseInfo( + await this._getPageContent( + urljoin(this._baseUrl, this._userBasicPage), + this._userBasicParams, + this._userBasicHeaders, + this._userBasicMethod, + ), + ); + } else { + await this._parseUserBaseInfo(this._indexHtml as string); + } + if (this._userDetailPage) { + await this._parseUserDetailInfo( + await this._getPageContent( + urljoin(this._baseUrl, this._userDetailPage), + this._userDetailParams, + this._userDetailHeaders, + ), + ); + } + if (this._opts.fetchMessages) await this._paseUnreadMsgs(); + if (this._userTrafficPage) { + await this._parseUserTrafficInfo( + await this._getPageContent( + urljoin(this._baseUrl, this._userTrafficPage), + this._userTrafficParams, + this._userTrafficHeaders, + ), + ); + } + await this._parseSeedingPages(); + } + + /** MP `_pase_unread_msgs` */ + protected async _paseUnreadMsgs(): Promise { + const unreadMsgLinks: string[] = []; + if (this.messageUnread > 0 || this.messageReadForce) { + const links = new Set([this._userMailUnreadPage, this._sysMailUnreadPage]); + for (const link of links) { + if (!link) continue; + const msgLinks: string[] = []; + let nextPage = await this._parseMessageUnreadLinks( + await this._getPageContent(urljoin(this._baseUrl, link), this._mailUnreadParams, this._mailUnreadHeaders), + msgLinks, + ); + let guard = 0; + while (nextPage && guard++ < MAX_PAGES) { + nextPage = await this._parseMessageUnreadLinks( + await this._getPageContent( + urljoin(this._baseUrl, nextPage), + this._mailUnreadParams, + this._mailUnreadHeaders, + ), + msgLinks, + ); + } + unreadMsgLinks.push(...msgLinks); + } + } + if (unreadMsgLinks.length && !this.messageUnread) this.messageUnread = unreadMsgLinks.length; + for (const msgLink of unreadMsgLinks) { + const [head, date, content] = await this._parseMessageContent( + await this._getPageContent(urljoin(this._baseUrl, msgLink), this._mailContentParams, this._mailContentHeaders), + ); + this.messageUnreadContents.push([head, date, content]); + } + } + + /** MP `_parse_seeding_pages` */ + protected async _parseSeedingPages(): Promise { + if (!this._torrentSeedingPage) return; + let nextPage: NextPage = await this._parseUserTorrentSeedingInfo( + await this._getPageContent( + urljoin(this._baseUrl, this._torrentSeedingPage), + this._torrentSeedingParams, + this._torrentSeedingHeaders, + ), + ); + let guard = 0; + while (nextPage !== null && nextPage !== undefined && nextPage !== false && guard++ < MAX_PAGES) { + nextPage = await this._parseUserTorrentSeedingInfo( + await this._getPageContent( + urljoin(urljoin(this._baseUrl, this._torrentSeedingPage), nextPage), + this._torrentSeedingParams, + this._torrentSeedingHeaders, + ), + true, + ); + } + } + + /** MP `_prepare_html_text`:去掉 `\d+px` 与 `#\d+` 干扰 */ + static prepareHtmlText(htmlText: string): string { + return String(htmlText ?? "") + .replace(/\d+px/g, "") + .replace(/#\d+/g, ""); + } + protected _prepareHtmlText(htmlText: string): string { + return SiteParserBase.prepareHtmlText(htmlText); + } + + /** MP `_get_page_content` */ + protected async _getPageContent( + url: string, + params?: Params | null, + headers?: Headers_ | null, + method?: string | null, + ): Promise { + let reqHeaders: Headers_ | null = null; + if (this._ua || headers || this._additionHeaders) { + reqHeaders = this.requestMode === "apikey" ? {} : { "User-Agent": `${this._ua}` }; + if (headers) Object.assign(reqHeaders, headers); + else reqHeaders["Content-Type"] = "application/x-www-form-urlencoded; charset=UTF-8"; + if (this._additionHeaders) Object.assign(reqHeaders, this._additionHeaders); + } + const h = new Headers(); + for (const [k, v] of Object.entries(reqHeaders ?? {})) if (v != null) h.set(k, String(v)); + if (this.requestMode !== "apikey" && this._siteCookie) h.set("cookie", this._siteCookie); + + let init: RequestInit; + if (method === "post" || (params && Object.keys(params).length > 0)) { + if (reqHeaders?.["Content-Type"] === "application/json") { + init = { method: "POST", headers: h, body: JSON.stringify(params ?? {}) }; + } else { + if (!h.has("content-type")) h.set("content-type", "application/x-www-form-urlencoded; charset=UTF-8"); + const form = new URLSearchParams(); + for (const [k, v] of Object.entries(params ?? {})) { + if (v == null) continue; + if (Array.isArray(v)) for (const x of v) form.append(k, String(x)); + else form.append(k, String(v)); + } + init = { method: "POST", headers: h, body: form.toString() }; + } + } else { + init = { method: "GET", headers: h }; + } + + const fetchImpl = this._opts.fetch ?? globalThis.fetch; + const timeout = AbortSignal.timeout(this._opts.timeoutMs ?? 60_000); + const signal = this._opts.signal ? AbortSignal.any([this._opts.signal, timeout]) : timeout; + let res: Response; + try { + await this._opts.limiter?.acquire(this._opts.signal); + res = await fetchImpl(url, { ...init, signal }); + } catch (e) { + if (this._opts.signal?.aborted) throw e; + this.httpErrors.push(redactText(`${init.method} ${url}:${timeout.aborted ? "超时" : (e as Error).message}`)); + return ""; + } + if (![200, 500, 403].includes(res.status)) { + await res.body?.cancel().catch(() => {}); + this.httpErrors.push(redactText(`${init.method} ${url}:HTTP ${res.status}`)); + return ""; + } + const bytes = new Uint8Array(await res.arrayBuffer()); + if (reqHeaders && String(reqHeaders.Accept ?? "").includes("application/json")) { + try { + return JSON.stringify(JSON.parse(new TextDecoder("utf-8").decode(bytes))); + } catch { + this.httpErrors.push(redactText(`${init.method} ${url}:API 响应 JSON 解析失败`)); + return ""; + } + } + const text = decode(bytes, detectCharset(res.headers.get("content-type"), bytes)); + if (underChallenge(text)) { + this.httpErrors.push(redactText(`${url}:检测到 Cloudflare,请更新 Cookie 和 UA`)); + return ""; + } + return text; + } + + /** MP `_parse_logged_in` */ + protected _parseLoggedIn(htmlText: unknown): boolean | Promise { + const loggedIn = isLoggedIn(String(htmlText ?? "")); + if (!loggedIn) this.errMsg = "未检测到已登陆，请检查cookies是否过期"; + return loggedIn; + } + + protected abstract _parseSitePage(htmlText: string): void | Promise; + protected abstract _parseUserBaseInfo(htmlText: string): void | Promise; + protected abstract _parseUserTrafficInfo(htmlText: string): void | Promise; + protected abstract _parseUserTorrentSeedingInfo(htmlText: string, multiPage?: boolean): NextPage | Promise; + protected abstract _parseUserDetailInfo(htmlText: string): void | Promise; + protected abstract _parseMessageUnreadLinks(htmlText: string, msgLinks: string[]): NextPage | Promise; + protected abstract _parseMessageContent( + htmlText: string, + ): [string | null, string | null, string | null] | Promise<[string | null, string | null, string | null]>; + + /** MP `clear` */ + clear(): void { + this._indexHtml = ""; + this.seedingInfo = []; + this.messageUnreadContents = []; + } + + /** MP `to_dict`:公开字段(snake_case),不含凭据 */ + toDict(): Record { + return { + schema: this.schema, + userid: this.userid, + username: this.username, + user_level: this.userLevel, + join_at: this.joinAt, + bonus: this.bonus, + upload: this.upload, + download: this.download, + ratio: this.ratio, + seeding: this.seeding, + leeching: this.leeching, + seeding_size: this.seedingSize, + leeching_size: this.leechingSize, + uploaded: this.uploaded, + completed: this.completed, + incomplete: this.incomplete, + uploaded_size: this.uploadedSize, + completed_size: this.completedSize, + incomplete_size: this.incompleteSize, + seeding_info: this.seedingInfo.map((x) => [...x]), + message_unread: this.messageUnread, + message_unread_contents: this.messageUnreadContents.map((x) => [...x]), + message_read_force: this.messageReadForce, + err_msg: this.errMsg, + http_errors: [...this.httpErrors], + }; + } + + /** MP `refresh_userdata` 里构造 `SiteUserData` 的部分 */ + toUserInfo(): SiteUserInfo { + const num = (v: unknown) => { + const n = typeof v === "number" ? v : Number(v); + return Number.isFinite(n) ? n : 0; + }; + const messages: SiteUnreadMessage[] = this.messageUnreadContents.map(([head, date, content]) => ({ + head, + date, + content, + })); + return { + schema: this.schema, + domain: extractDomain(this._siteUrl) || null, + userId: this.userid == null || this.userid === "" ? null : String(this.userid), + username: this.username, + userLevel: this.userLevel, + joinAt: this.joinAt, + upload: num(this.upload), + download: num(this.download), + ratio: num(this.ratio), + bonus: num(this.bonus), + seeding: num(this.seeding), + seedingSize: num(this.seedingSize), + leeching: num(this.leeching), + leechingSize: num(this.leechingSize), + seedingInfo: this.seedingInfo.map(([a, b]) => [num(a), num(b)] as [number, number]), + messageUnread: num(this.messageUnread), + messageUnreadContents: messages, + errMsg: this.errMsg, + raw: this.toDict(), + }; + } +} + +export type SiteParserClass = (new ( + opts: SiteUserParserOptions, +) => SiteParserBase) & { + schema: SiteSchemaValue | null; +}; diff --git a/packages/adapters/src/sites/user-parsers/bitpt.test.ts b/packages/adapters/src/sites/user-parsers/bitpt.test.ts new file mode 100644 index 00000000..ea6105b9 --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/bitpt.test.ts @@ -0,0 +1,47 @@ +import { + BITPT_USER_DETAILS, + BITPT_USER_INDEX, + BITPT_USER_SEEDING_EMPTY, + BITPT_USER_SEEDING_P1, + createMockFetch, +} from "@umt/testkit"; +import { describe, expect, it } from "vitest"; +import { BitptSiteUserInfo } from "./bitpt.ts"; +import { runParser } from "./testing.ts"; + +const GiB = 1024 ** 3; + +describe("BitptSiteUserInfo(MP bitpt.py)", () => { + it("#userinfotop 登录判定 → frmtable 键值 → browse.php myseed 分页", async () => { + const m = createMockFetch({ + "/": { body: BITPT_USER_INDEX }, + "/userdetails.php?uid=55": { body: BITPT_USER_DETAILS }, + "/browse.php?p=1": { body: BITPT_USER_SEEDING_P1 }, + "/browse.php?p=2": { body: BITPT_USER_SEEDING_EMPTY }, + }); + const info = await runParser(BitptSiteUserInfo, { fetch: m.fetch, cookie: "bitpt=secret" }); + expect(info).toMatchObject({ + schema: "Bitpt", + userId: "55", + username: "gina", + userLevel: "Elite", + joinAt: "2021-06-07 08:09:10", + upload: 3 * 1024 ** 4, + download: 1024 ** 4, + ratio: 3, + bonus: 2048.5, + seeding: 2, + seedingSize: 6 * GiB, + }); + const seedReq = m.requests.find((r) => r.url.pathname === "/browse.php")!; + expect(seedReq.url.search).toBe("?t=myseed&st=2&d=desc&p=1"); + expect(JSON.stringify(info)).not.toContain("secret"); + }); + + it("未登录(无 #userinfotop):只请求首页", async () => { + const m = createMockFetch({ "/": { body: "login" } }); + const info = await runParser(BitptSiteUserInfo, { fetch: m.fetch }); + expect(info.userId).toBeNull(); + expect(m.requests).toHaveLength(1); + }); +}); diff --git a/packages/adapters/src/sites/user-parsers/bitpt.ts b/packages/adapters/src/sites/user-parsers/bitpt.ts new file mode 100644 index 00000000..9cd23f3b --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/bitpt.ts @@ -0,0 +1,153 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/modules/indexer/parser/bitpt.py @ e6820a8 + * + * 极速之星(bitpt.cn)站点用户信息解析器(原作者 ThedoRap)。 + * + * 函数对照(`BitptSiteUserInfo`):`_parse_site_page` → `_parseSitePage`、`_parse_logged_in` → `_parseLoggedIn` + * (`#userinfotop`)、`_parse_user_base_info` → `_parseUserBaseInfo`(`table.frmtable` 两列键值)、 + * `_parse_user_torrent_seeding_page_info` → `parseUserTorrentSeedingPageInfo`(`table.torrenttable tr.btr td.r`)、 + * `parse` → `parse`(基类流程后取 uid → `userdetails.php?uid=` → `browse.php?t=myseed&st=2&d=desc&p=n` 逐页累加)。 + * BeautifulSoup 以 cheerio 等价实现。 + * + * 与 MP 的偏离:末尾不把字段转字符串(同 zhixing.ts);做种翻页上限 50 页。 + */ +import { load } from "cheerio"; +import { type NextPage, SiteParserBase, urljoin } from "./base.ts"; +import { normalizeDatetime, pyFloat, pyInt } from "./common.ts"; +import { SiteSchema, type SiteSchemaValue } from "./types.ts"; + +const MAX_PAGES = 50; + +export class BitptSiteUserInfo extends SiteParserBase { + static override schema: SiteSchemaValue | null = SiteSchema.Bitpt; + + protected _torrentSeedingBase: string | undefined; + + protected _parseSitePage(_htmlText: string): void { + this._userBasicPage = "userdetails.php?uid={uid}"; + this._userDetailPage = null; + this._userBasicParams = {}; + this._userTrafficPage = null; + this._sysMailUnreadPage = null; + this._userMailUnreadPage = null; + this._mailUnreadParams = {}; + this._torrentSeedingBase = "browse.php"; + this._torrentSeedingParams = { t: "myseed", st: "2", d: "desc" }; + this._torrentSeedingHeaders = {}; + this._additionHeaders = {}; + } + + protected override _parseLoggedIn(htmlText: unknown): boolean { + return load(String(htmlText ?? ""))("#userinfotop").length > 0; + } + + protected _parseUserBaseInfo(htmlText: string): void { + if (!htmlText) return; + const $ = load(htmlText); + const table = $("table.frmtable").first(); + if (!table.length) return; + const info: Record = {}; + table.find("tr").each((_, row) => { + const cells = $(row).find("td"); + if (cells.length === 2) info[cells.eq(0).text().trim()] = cells.eq(1).text().trim(); + }); + this.userid = info.UID ?? null; + this.username = "用户名" in info ? info.用户名!.split(" ")[0]! : null; + this.userLevel = info.用户级别 ?? null; + this.joinAt = "注册时间" in info ? (normalizeDatetime(info.注册时间) ?? null) : null; + this.upload = "上传流量" in info ? this.numFilesize(info.上传流量) : 0; + this.download = "下载流量" in info ? this.numFilesize(info.下载流量) : 0; + this.ratio = "共享率" in info ? pyFloat(info.共享率) : 0; + const bonusMatch = /累计([\d.]+)/.exec(info.星辰 ?? ""); + this.bonus = bonusMatch ? pyFloat(bonusMatch[1]) : 0; + this.messageUnread = 0; + if (this._torrentSeedingBase) { + this.seeding = 0; + this.seedingSize = 0; + } else { + const seedingInfo = $("div").filter( + (_, el) => + $(el).attr("style") === + "margin:0 auto;width:90%;font-size:14px;margin-top:10px;margin-bottom:10px;text-align:center;", + ); + if (seedingInfo.length) { + const links = seedingInfo.first().find("a"); + const text = links.length > 1 ? links.eq(1).text() : ""; + const m = /当前上传的种子\((\d+)个, 共([\d.]+ [KMGT]B)\)/.exec(text); + this.seeding = m ? pyInt(m[1]) : 0; + this.seedingSize = m ? this.numFilesize(m[2]) : 0; + } + } + } + + protected _parseUserTrafficInfo(_htmlText: string): void {} + protected _parseUserDetailInfo(_htmlText: string): void {} + + /** 单页做种:返回 [数量, 大小] */ + parseUserTorrentSeedingPageInfo(htmlText: string): [number, number] { + if (!htmlText) return [0, 0]; + const $ = load(htmlText); + const table = $("table.torrenttable").first(); + if (!table.length) return [0, 0]; + const rows = table.find("tr").toArray(); + if (rows.length <= 1) return [0, 0]; + let count = 0; + let size = 0; + for (const row of rows.slice(1)) { + if (!$(row).hasClass("btr")) continue; + const td = $(row).find("td.r").first(); + if (!td.length) continue; + const a = td.find("a").first(); + const sizeText = a.length ? a.text().trim() : td.text().trim(); + if (sizeText) { + count += 1; + size += this.numFilesize(sizeText); + } + } + return [count, size]; + } + + protected _parseMessageUnreadLinks(_htmlText: string, _msgLinks: string[]): NextPage { + return null; + } + protected _parseMessageContent(_htmlText: string): [string | null, string | null, string | null] { + return [null, null, null]; + } + protected _parseUserTorrentSeedingInfo(_htmlText: string): NextPage { + return null; + } + + override async parse(): Promise { + await super.parse(); + if (this._indexHtml) { + const $ = load(String(this._indexHtml)); + const link = $("a") + .filter((_, el) => /userdetails\.php\?uid=\d+/.test($(el).attr("href") ?? "")) + .first(); + if (link.length) { + const m = /uid=(\d+)/.exec(link.attr("href") ?? ""); + if (m) this.userid = m[1]!; + } + } + if (this.userid && this._userBasicPage) { + const basicUrl = this._userBasicPage.replace("{uid}", this.userid); + this._parseUserBaseInfo(await this._getPageContent(urljoin(this._baseUrl, basicUrl))); + } + if (this._torrentSeedingBase) { + const baseUrl = urljoin(this._baseUrl, this._torrentSeedingBase); + const params: Record = {}; + for (const [k, v] of Object.entries(this._torrentSeedingParams ?? {})) params[k] = String(v); + for (let page = 1; page <= MAX_PAGES; page++) { + params.p = String(page); + const [n, s] = this.parseUserTorrentSeedingPageInfo( + await this._getPageContent(`${baseUrl}?${new URLSearchParams(params).toString()}`), + ); + this.seeding += n; + this.seedingSize += s; + if (n === 0) break; + } + } + } +} diff --git a/packages/adapters/src/sites/user-parsers/common.ts b/packages/adapters/src/sites/user-parsers/common.ts new file mode 100644 index 00000000..37ebba45 --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/common.ts @@ -0,0 +1,211 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/foundation/text.py @ e6820a8 + * @mp-source app/foundation/temporal.py @ e6820a8 + * @mp-source app/domain/site.py @ e6820a8 + * @mp-source app/adapters/network/cloudflare.py @ e6820a8 + * + * 解析器公共工具:MoviePilot e6820a8(GPLv3) + * - `app/foundation/text.py` `parse_int/parse_float/contains_chinese` → `parseIntMp/parseFloatMp/containsChinese` + * - `app/foundation/temporal.py` `normalize_datetime/format_timestamp` → 同名(dateparser 的保守子集) + * - `app/domain/site.py` `SiteUtils.is_logged_in` → `isLoggedIn` + * - `app/adapters/network/cloudflare.py` `under_challenge` → `underChallenge` + * - `app/domain/site.py` `extract_domain` → `extractDomain`(与 mteam/definition.ts `rootDomain` 同义) + */ +import { load } from "cheerio"; +import { hasChildElements, parseHtml } from "./dom.ts"; + +/** MP `parse_int`:去千分位,失败返回 0 */ +export function parseIntMp(value: string | number | null | undefined): number { + if (value == null) return 0; + const s = String(value).trim().replace(/,/g, ""); + if (!/^[+-]?\d+$/.test(s)) return 0; + return Number.parseInt(s, 10); +} + +/** MP `parse_float`:去千分位,失败返回 0 */ +export function parseFloatMp(value: string | number | null | undefined): number { + if (value == null) return 0; + const s = String(value).trim().replace(/,/g, ""); + if (s === "") return 0; + if (!/^[+-]?(\d+\.?\d*|\.\d+)([eE][+-]?\d+)?$/.test(s) && !/^[+-]?(inf|infinity|nan)$/i.test(s)) return 0; + const n = Number(s); + return Number.isFinite(n) ? n : 0; +} + +/** Python `int(str)`(带 strip);非法抛错,与 MP 一致交给调用方 */ +export function pyInt(value: unknown): number { + if (typeof value === "number") return Math.trunc(value); + if (typeof value === "boolean") return value ? 1 : 0; + const s = String(value ?? "").trim(); + if (!/^[+-]?\d+$/.test(s)) throw new Error(`invalid literal for int(): '${s}'`); + return Number.parseInt(s, 10); +} + +/** Python `float(str)`;非法抛错 */ +export function pyFloat(value: unknown): number { + if (typeof value === "number") return value; + const s = String(value ?? "").trim(); + const n = Number(s); + if (s === "" || Number.isNaN(n)) throw new Error(`could not convert string to float: '${s}'`); + return n; +} + +export function containsChinese(value: string | string[] | null | undefined): boolean { + if (!value) return false; + const s = Array.isArray(value) ? value.join(" ") : value; + return /[一-鿿]/.test(s); +} + +const MONTHS: Record = { + jan: 1, + feb: 2, + mar: 3, + apr: 4, + may: 5, + jun: 6, + jul: 7, + aug: 8, + sep: 9, + sept: 9, + oct: 10, + nov: 11, + dec: 12, + january: 1, + february: 2, + march: 3, + april: 4, + june: 6, + july: 7, + august: 8, + september: 9, + october: 10, + november: 11, + december: 12, +}; + +const pad = (n: number | string) => String(Number(n)).padStart(2, "0"); + +/** + * MP `normalize_datetime`(dateparser.parse 后 strftime("%Y-%m-%d %H:%M:%S"),失败原样返回)。 + * 这里只实现确定性的绝对时间格式: + * - `YYYY-MM-DD[ T]HH:MM[:SS][.fff][Z|±HH:MM]`、`YYYY/MM/DD`、`YYYY年MM月DD日`(时区后缀按 dateparser 行为丢弃,保留墙钟时间) + * - `DD Mon YYYY[, HH:MM]`、`Mon DD[,] YYYY [HH:MM]`(英文月份) + * 相对时间("3 天前"/"2 hours ago")dateparser 会以当前时间换算,本实现不做(原样返回,标记待核验)。 + */ +export function normalizeDatetime(value: string | null | undefined): string | null | undefined { + if (!value) return value; + const v = String(value).trim(); + let m = + /^(\d{4})[-/.年](\d{1,2})[-/.月](\d{1,2})日?(?:[ T]+(\d{1,2}):(\d{1,2})(?::(\d{1,2}))?(?:\.\d+)?)?\s*(?:Z|[+-]\d{2}:?\d{2}|UTC|GMT)?$/i.exec( + v, + ); + if (m) return `${m[1]}-${pad(m[2]!)}-${pad(m[3]!)} ${pad(m[4] ?? 0)}:${pad(m[5] ?? 0)}:${pad(m[6] ?? 0)}`; + m = + /^(\d{1,2})(?:st|nd|rd|th)?\s+([A-Za-z]{3,9})\.?,?\s+(\d{4})(?:,?\s+(?:at\s+)?(\d{1,2}):(\d{2})(?::(\d{2}))?\s*(am|pm)?)?$/i.exec( + v, + ); + if (m && MONTHS[m[2]!.toLowerCase()]) { + const h = ampm(m[4], m[7]); + return `${m[3]}-${pad(MONTHS[m[2]!.toLowerCase()]!)}-${pad(m[1]!)} ${pad(h)}:${pad(m[5] ?? 0)}:${pad(m[6] ?? 0)}`; + } + m = + /^([A-Za-z]{3,9})\.?\s+(\d{1,2})(?:st|nd|rd|th)?,?\s+(\d{4})(?:,?\s+(?:at\s+)?(\d{1,2}):(\d{2})(?::(\d{2}))?\s*(am|pm)?)?$/i.exec( + v, + ); + if (m && MONTHS[m[1]!.toLowerCase()]) { + const h = ampm(m[4], m[7]); + return `${m[3]}-${pad(MONTHS[m[1]!.toLowerCase()]!)}-${pad(m[2]!)} ${pad(h)}:${pad(m[5] ?? 0)}:${pad(m[6] ?? 0)}`; + } + return value; +} + +function ampm(h: string | undefined, suffix: string | undefined): number { + let n = Number(h ?? 0); + if (suffix?.toLowerCase() === "pm" && n < 12) n += 12; + if (suffix?.toLowerCase() === "am" && n === 12) n = 0; + return n; +} + +/** + * MP `format_timestamp`:非纯数字字符串原样返回;Unix 秒 → 本地时间 "YYYY-MM-DD HH:MM:SS"。 + * MP 用进程本地时区;这里用显式 `tzOffsetMinutes`(默认 +480,即 Asia/Shanghai,MP 容器默认 TZ)。 + */ +export function formatTimestamp(timestamp: unknown, tzOffsetMinutes = 480): string | undefined { + if (timestamp == null) return undefined; + if (typeof timestamp === "string" && !/^\d+$/.test(timestamp)) return timestamp; + const n = Number(timestamp); + if (!Number.isFinite(n)) return String(timestamp); + const d = new Date(Math.trunc(n) * 1000 + tzOffsetMinutes * 60_000); + return `${d.getUTCFullYear()}-${pad(d.getUTCMonth() + 1)}-${pad(d.getUTCDate())} ${pad(d.getUTCHours())}:${pad(d.getUTCMinutes())}:${pad(d.getUTCSeconds())}`; +} + +/** MP `SiteUtils.is_logged_in` */ +export function isLoggedIn(htmlText: string): boolean { + const html = parseHtml(htmlText); + if (!hasChildElements(html)) return false; + if (html.nodes("//input[@type='password']").length) return false; + const xpaths = [ + '//a[contains(@href, "logout") or contains(@data-url, "logout") or contains(@href, "mybonus") or contains(@onclick, "logout") or contains(@href, "usercp") or contains(@lay-on, "logout")]', + '//form[contains(@action, "logout")]', + '//div[@class="user-info-side"]', + '//a[@id="myitem"]', + ]; + return xpaths.some((x) => html.nodes(x).length > 0); +} + +const CHALLENGE_TITLES = ["Just a moment...", "请稍候…", "DDOS-GUARD"]; +const CHALLENGE_SELECTORS = [ + "#cf-challenge-running", + ".ray_id", + ".attack-box", + "#cf-please-wait", + "#challenge-spinner", + "#trk_jschal_js", + "td.info #js_info", + "div.vc div.text-box h2", +]; + +/** MP `under_challenge`:Cloudflare / DDoS-Guard 挑战页 */ +export function underChallenge(htmlText: string | null | undefined): boolean { + if (!htmlText) return false; + const $ = load(htmlText); + const title = $("title").text().trim().toLowerCase(); + if (CHALLENGE_TITLES.some((t) => t.toLowerCase() === title)) return true; + return CHALLENGE_SELECTORS.some((s) => $(s).length > 0); +} + +const SPECIAL_SITE_DOMAINS = ["u2.dmhy.org", "pt.ecust.pp.ua", "pt.gtkpw.xyz", "pt.gtk.pw"]; + +/** MP `site_rules.extract_domain`:特殊域名表 → netloc 超过 3 级原样 → 否则取最后两级 */ +export function extractDomain(url: string | null | undefined): string { + if (!url) return ""; + for (const d of SPECIAL_SITE_DOMAINS) if (url.includes(d)) return d; + let netloc: string; + if (!url.startsWith("http")) netloc = url; + else { + try { + netloc = new URL(url).host; + } catch { + return ""; + } + } + if (!netloc) return ""; + const labels = netloc.split("."); + if (labels.length > 3) return netloc; + return labels.slice(-2).join("."); +} + +/** Python `re.search(...)`: 返回 match 或 null(显式 helper,方便逐行对照) */ +export function reSearch(pattern: RegExp, text: string | null | undefined): RegExpExecArray | null { + if (text == null) return null; + const re = new RegExp(pattern.source, pattern.flags.replace("g", "")); + return re.exec(text); +} + +/** Python `re.findall`(带分组时返回分组数组) */ +export function reFindall(pattern: RegExp, text: string): string[][] { + const re = new RegExp(pattern.source, pattern.flags.includes("g") ? pattern.flags : `${pattern.flags}g`); + return [...text.matchAll(re)].map((m) => m.slice(1)); +} diff --git a/packages/adapters/src/sites/user-parsers/discuz.test.ts b/packages/adapters/src/sites/user-parsers/discuz.test.ts new file mode 100644 index 00000000..cc16c253 --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/discuz.test.ts @@ -0,0 +1,36 @@ +import { createMockFetch, DISCUZ_USER_INDEX, DISCUZ_USER_SEEDING, DISCUZ_USER_SPACE } from "@umt/testkit"; +import { describe, expect, it } from "vitest"; +import { DiscuzUserInfo } from "./discuz.ts"; +import { runParser } from "./testing.ts"; + +describe("DiscuzUserInfo(MP discuz.py)", () => { + it("&uid= 链接定位用户,空间页取等级/积分/流量,做种表按表头定位列", async () => { + const m = createMockFetch({ + "/": { body: DISCUZ_USER_INDEX }, + "/home.php?uid=42": { body: DISCUZ_USER_SPACE }, + "/forum.php?mod=torrents": { body: DISCUZ_USER_SEEDING }, + }); + const info = await runParser(DiscuzUserInfo, { fetch: m.fetch }); + expect(info).toMatchObject({ + schema: "DiscuzX", + userId: "42", + username: "frank", + userLevel: "Lv3", + joinAt: "2018-02-03 04:05:00", + bonus: 520, + upload: 2 * 1024 ** 4, + download: 500 * 1024 ** 3, + seeding: 1, + seedingInfo: [[6, 8 * 1024 ** 3]], + }); + }); + + it("未登录(无登出链接):errMsg", async () => { + const m = createMockFetch({ + "/": { body: "登录" }, + }); + const info = await runParser(DiscuzUserInfo, { fetch: m.fetch }); + expect(info.errMsg).toContain("未检测到已登陆"); + expect(info.userId).toBeNull(); + }); +}); diff --git a/packages/adapters/src/sites/user-parsers/discuz.ts b/packages/adapters/src/sites/user-parsers/discuz.ts new file mode 100644 index 00000000..c660bc24 --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/discuz.ts @@ -0,0 +1,105 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/modules/indexer/parser/discuz.py @ e6820a8 + * + * Discuz!X 站点用户信息解析器。 + * + * 函数对照(`DiscuzUserInfo`):`_parse_user_base_info` → `_parseUserBaseInfo`、`_parse_site_page` → 空、 + * `_parse_user_detail_info` → `_parseUserDetailInfo`、`_parse_user_torrent_seeding_info` → + * `_parseUserTorrentSeedingInfo`、`_parse_user_traffic_info` / `_parse_message_unread_links` / `_parse_message_content` → 空。 + * XPath 与正则逐字保留(含 MP 把"分享率"括号值写入 bonus 的原行为,随后被"积分"覆盖)。 + */ +import { type NextPage, SiteParserBase } from "./base.ts"; +import { normalizeDatetime, parseFloatMp, parseIntMp, reSearch } from "./common.ts"; +import { hasChildElements, parseHtml } from "./dom.ts"; +import { SiteSchema, type SiteSchemaValue } from "./types.ts"; + +export class DiscuzUserInfo extends SiteParserBase { + static override schema: SiteSchemaValue | null = SiteSchema.DiscuzX; + + protected _parseUserBaseInfo(htmlText: string): void { + const html = parseHtml(this._prepareHtmlText(htmlText)); + if (!html) return; + const userInfo = html.nodes('//a[contains(@href, "&uid=")]'); + if (!userInfo.length) return; + const href = userInfo[0]!.attr("href") ?? ""; + const m = reSearch(/&uid=(\d+)/, href); + if (m?.[0].trim()) { + this.userid = m[1]!; + this._torrentSeedingPage = "forum.php?&mod=torrents&cat_5up=on"; + this._userDetailPage = href; + // lxml `.text`:首个子节点之前的文本 + const firstChild = userInfo[0]!.node.firstChild; + this.username = firstChild && firstChild.nodeType === 3 ? (firstChild.nodeValue ?? "").trim() : ""; + } + } + + protected _parseSitePage(_htmlText: string): void {} + + /** 解析用户额外信息,加入时间,等级 */ + protected _parseUserDetailInfo(htmlText: string): void { + const html = parseHtml(htmlText); + if (!hasChildElements(html)) return; + const levels = html.strings('//a[contains(@href, "usergroup")]/text()'); + if (levels.length) this.userLevel = levels[levels.length - 1]!.trim(); + const joinAt = html.strings('//li[em[text()="注册时间"]]/text()'); + if (joinAt.length) this.joinAt = normalizeDatetime(joinAt[0]!.trim()) ?? null; + const ratioText = html.strings('//li[contains(.//text(), "分享率")]//text()'); + if (ratioText.length) { + const m = reSearch(/\(([\d,.]+)\)/, ratioText[0]); + if (m?.[1]?.trim()) this.bonus = parseFloatMp(m[1]); + } + const bonusText = html.strings('//li[em[text()="积分"]]/text()'); + if (bonusText.length) this.bonus = parseFloatMp(bonusText[0]!.trim()); + const upload = html.strings('//li[em[contains(text(),"上传量")]]/text()'); + if (upload.length) this.upload = this.numFilesize(upload[0]!.trim().split("/").pop()); + const download = html.strings('//li[em[contains(text(),"下载量")]]/text()'); + if (download.length) this.download = this.numFilesize(download[0]!.trim().split("/").pop()); + } + + protected _parseUserTorrentSeedingInfo(htmlText: string): NextPage { + const html = parseHtml(htmlText); + if (!hasChildElements(html)) return null; + let sizeCol = 3; + let seedersCol = 4; + if (html.nodes('//tr[position()=1]/td[.//img[@class="size"] and .//img[@alt="size"]]').length) { + sizeCol = + html.nodes('//tr[position()=1]/td[.//img[@class="size"] and .//img[@alt="size"]]/preceding-sibling::td') + .length + 1; + } + if (html.nodes('//tr[position()=1]/td[.//img[@class="seeders"] and .//img[@alt="seeders"]]').length) { + seedersCol = + html.nodes('//tr[position()=1]/td[.//img[@class="seeders"] and .//img[@alt="seeders"]]/preceding-sibling::td') + .length + 1; + } + let pageSeeding = 0; + let pageSeedingSize = 0; + const pageSeedingInfo: Array<[number, number]> = []; + const sizes = html.nodes(`//tr[position()>1]/td[${sizeCol}]`); + const seeders = html.strings(`//tr[position()>1]/td[${seedersCol}]//text()`); + if (sizes.length && seeders.length) { + pageSeeding = sizes.length; + for (let i = 0; i < sizes.length; i++) { + const size = this.numFilesize(sizes[i]!.text().trim()); + pageSeedingSize += size; + pageSeedingInfo.push([parseIntMp(seeders[i]), size]); + } + } + this.seeding += pageSeeding; + this.seedingSize += pageSeedingSize; + this.seedingInfo.push(...pageSeedingInfo); + const next = html.strings('//a[contains(.//text(), "下一页") or contains(.//text(), "下一頁")]/@href'); + return next.length ? next[next.length - 1]!.trim() : null; + } + + protected _parseUserTrafficInfo(_htmlText: string): void {} + + protected _parseMessageUnreadLinks(_htmlText: string, _msgLinks: string[]): NextPage { + return null; + } + + protected _parseMessageContent(_htmlText: string): [string | null, string | null, string | null] { + return [null, null, null]; + } +} diff --git a/packages/adapters/src/sites/user-parsers/dom.ts b/packages/adapters/src/sites/user-parsers/dom.ts new file mode 100644 index 00000000..d955405b --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/dom.ts @@ -0,0 +1,163 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/foundation/dom.py @ e6820a8 + * + * lxml `etree.HTML(...).xpath(...)` 的 TS 等价层(站点用户信息解析器专用)。 + * + * MP 解析器(app/modules/indexer/parser/*.py,提交 e6820a8,GPLv3)全部用 lxml XPath 1.0 取值; + * 为了让 XPath 表达式能逐字迁移,本层: + * 1. 用 htmlparser2(经 cheerio,HTML 模式)解析——与 libxml2 HTML 解析器一样**不插入 **, + * 标签/属性名小写,实体解码(` ` → U+00A0); + * 2. 把 DOM 复制成 @xmldom/xmldom 文档,交给 `xpath` 包求值(XPath 1.0,与 lxml 同语义); + * 3. 近似 libxml2 的 "blank" 规则:html/head/table/thead/tbody/tfoot/tr/ul/ol/select 下的纯空白文本节点丢弃 + * (影响 `//text()[0]` 类取值)。其他 libxml2 容错修复(自动补

/ 等)未模拟。 + * + * 结果约定同 lxml:元素 → XNode;`@attr` → 属性值字符串;`text()` → 文本字符串;`string(.)`/`count()` → 标量。 + */ +import { DOMImplementation } from "@xmldom/xmldom"; +import { load } from "cheerio"; +import * as xpath from "xpath"; + +type XmlNode = Node; + +/** cheerio/domhandler 节点的结构子集(避免直接依赖 domhandler 类型) */ +interface DhNode { + type: string; + name?: string; + data?: string; + attribs?: Record; + children?: DhNode[]; +} + +const BLANK_DROP_PARENTS = new Set(["html", "head", "table", "thead", "tbody", "tfoot", "tr", "ul", "ol", "select"]); + +function copyInto(doc: Document, parent: XmlNode, src: DhNode, parentName: string): void { + if (src.type === "text") { + const data = src.data ?? ""; + if (data.trim() === "" && BLANK_DROP_PARENTS.has(parentName)) return; + parent.appendChild(doc.createTextNode(data)); + return; + } + if (src.type === "comment") { + parent.appendChild(doc.createComment(src.data ?? "")); + return; + } + if (src.type === "tag" || src.type === "script" || src.type === "style") { + const name = (src.name ?? "div").toLowerCase(); + let el: Element; + try { + el = doc.createElement(name); + } catch { + el = doc.createElement("x-invalid"); + } + for (const [k, v] of Object.entries(src.attribs ?? {})) { + try { + el.setAttribute(k.toLowerCase(), v); + } catch { + // 非法属性名(如 @click、:class)在 lxml 下同样取不到,跳过 + } + } + parent.appendChild(el); + for (const c of src.children ?? []) copyInto(doc, el, c, name); + return; + } + // root / directive(doctype) / cdata:仅递归子节点 + for (const c of src.children ?? []) copyInto(doc, parent, c, parentName); +} + +/** 可 XPath 查询的节点包装(对应 lxml `_Element`)。 */ +export class XNode { + constructor(readonly node: XmlNode) {} + + /** lxml `el.xpath(expr)`:节点集 → (XNode|string)[];标量 → [标量] */ + xpath(expr: string): Array { + const r = xpath.select(expr, this.node); + if (Array.isArray(r)) { + return r.map((n) => { + if (n.nodeType === 1 || n.nodeType === 9) return new XNode(n); + if (n.nodeType === 2) return (n as Attr).value; + return n.nodeValue ?? ""; + }); + } + if (r == null) return []; + return [String(r)]; + } + + /** 只取字符串结果(文本/属性);元素结果按 `string(.)` 转文本 */ + strings(expr: string): string[] { + return this.xpath(expr).map((v) => (typeof v === "string" ? v : v.text())); + } + + /** 只取元素结果 */ + nodes(expr: string): XNode[] { + return this.xpath(expr).filter((v): v is XNode => v instanceof XNode); + } + + /** lxml `el.xpath("string(.)")` */ + text(): string { + return String(xpath.select("string(.)", this.node, true) ?? ""); + } + + /** 标量求值(string()/count()/boolean()) */ + evaluate(expr: string): string | number | boolean | null { + const r = xpath.select(expr, this.node, true); + if (r == null) return null; + if (typeof r === "object") return (r as XmlNode).nodeValue ?? new XNode(r as XmlNode).text(); + return r; + } + + get tagName(): string { + return (this.node as Element).tagName ?? ""; + } + + attr(name: string): string | null { + const el = this.node as Element; + return typeof el.getAttribute === "function" && el.hasAttribute(name) ? el.getAttribute(name) : null; + } + + /** lxml `len(el)`:子元素个数 */ + get childElementCount(): number { + let n = 0; + const cs = this.node.childNodes; + for (let i = 0; i < cs.length; i++) if (cs[i]!.nodeType === 1) n++; + return n; + } +} + +/** + * lxml `etree.HTML(text)`:空文本返回 null(lxml 返回 None);否则返回文档节点。 + * 与 MP `DomUtils.has_child_elements(html)` 搭配使用 → `hasChildElements`。 + */ +export function parseHtml(text: string | null | undefined): XNode | null { + if (text == null || String(text).trim() === "") return null; + const $ = load(String(text), { xml: { xmlMode: false, decodeEntities: true } }); + const doc = new DOMImplementation().createDocument(null, "", null) as unknown as Document; + const root = doc.createElement("html"); + doc.appendChild(root); + const dhRoot = $.root()[0] as unknown as DhNode; + const kids = dhRoot.children ?? []; + // 输入自带 时展开到根,避免 html/html 嵌套(lxml 会合并) + const htmlEl = kids.find((k) => k.type === "tag" && k.name?.toLowerCase() === "html"); + if (htmlEl) { + for (const [k, v] of Object.entries(htmlEl.attribs ?? {})) { + try { + root.setAttribute(k.toLowerCase(), v); + } catch { + /* ignore */ + } + } + for (const k of kids) { + if (k === htmlEl) for (const c of htmlEl.children ?? []) copyInto(doc, root, c, "html"); + else if (k.type !== "text" || (k.data ?? "").trim() !== "") copyInto(doc, root, k, "html"); + } + } else { + for (const k of kids) copyInto(doc, root, k, "html"); + } + return new XNode(root); +} + +/** MP `DomUtils.has_child_elements`:存在且至少一个子元素 */ +export function hasChildElements(el: XNode | null | undefined): el is XNode { + return el != null && el.childElementCount > 0; +} diff --git a/packages/adapters/src/sites/user-parsers/file-list.test.ts b/packages/adapters/src/sites/user-parsers/file-list.test.ts new file mode 100644 index 00000000..047136aa --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/file-list.test.ts @@ -0,0 +1,32 @@ +import { createMockFetch, FILELIST_USER_DETAIL, FILELIST_USER_INDEX, FILELIST_USER_SEEDING } from "@umt/testkit"; +import { describe, expect, it } from "vitest"; +import { FileListSiteUserInfo } from "./file-list.ts"; +import { runParser } from "./testing.ts"; + +describe("FileListSiteUserInfo(MP file_list.py)", () => { + it("详情页流量/分享率/做种/等级/加入日期/积分 + snatchlist 做种人数", async () => { + const m = createMockFetch({ + "/": { body: FILELIST_USER_INDEX }, + "/userdetails.php": { body: FILELIST_USER_DETAIL }, + "/snatchlist.php": { body: FILELIST_USER_SEEDING }, + }); + const info = await runParser(FileListSiteUserInfo, { fetch: m.fetch }); + expect(info).toMatchObject({ + schema: "FileList", + userId: "555", + username: "erin", + upload: 3 * 1024 ** 4, + download: 1024 ** 4, + ratio: 3, + seeding: 42, + seedingSize: Math.round(2.5 * 1024 ** 4), + userLevel: "Elite User", + joinAt: "2017-07-08 09:10:11", + bonus: 1000.5, + seedingInfo: [[4, 1024 ** 3]], + }); + expect(m.requests.find((r) => r.url.pathname === "/snatchlist.php")!.url.search).toBe( + "?id=555&action=torrents&type=seeding", + ); + }); +}); diff --git a/packages/adapters/src/sites/user-parsers/file-list.ts b/packages/adapters/src/sites/user-parsers/file-list.ts new file mode 100644 index 00000000..b269f90f --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/file-list.ts @@ -0,0 +1,95 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/modules/indexer/parser/file_list.py @ e6820a8 + * + * FileList 站点用户数据解析器(MP `FileListSiteUserInfo`)。 + * 函数对照:`_parse_site_page` → `_parseSitePage`、`_parse_user_base_info` → `_parseUserBaseInfo`、 + * `_parse_user_traffic_info`(空)、`_parse_user_detail_info` → `_parseUserDetailInfo`、 + * `_parse_user_torrent_seeding_info` → `_parseUserTorrentSeedingInfo`、消息两函数(不支持)。 + */ +import { type NextPage, SiteParserBase } from "./base.ts"; +import { normalizeDatetime, parseFloatMp, parseIntMp, reSearch } from "./common.ts"; +import { hasChildElements, parseHtml } from "./dom.ts"; +import { SiteSchema, type SiteSchemaValue } from "./types.ts"; + +function at(arr: T[], i: number): T { + const v = arr[i]; + if (v === undefined) throw new Error("IndexError: list index out of range"); + return v; +} + +export class FileListSiteUserInfo extends SiteParserBase { + static override schema: SiteSchemaValue | null = SiteSchema.FileList; + + protected _parseSitePage(htmlText: string): void { + const text = this._prepareHtmlText(htmlText); + const m = reSearch(/userdetails.php\?id=(\d+)/, text); + if (m?.[0].trim()) { + this._userDetailPage = m[0].trim().replace(/^\/+/, ""); + this.userid = m[1]!; + } + // MP 此处无条件拼接(userid 为 None 时得到 "id=None") + this._torrentSeedingPage = `snatchlist.php?id=${this.userid ?? "None"}&action=torrents&type=seeding`; + } + + protected _parseUserBaseInfo(htmlText: string): void { + const html = parseHtml(this._prepareHtmlText(htmlText)); + if (!html) return; + const ret = html.strings(`//a[contains(@href, "userdetails") and contains(@href, "${this.userid}")]//text()`); + if (ret.length) this.username = String(ret[0]); + } + + /** 上传/下载/分享率 [做种数/魔力值](FileList 在详情页解析) */ + protected _parseUserTrafficInfo(_htmlText: string): void {} + + protected _parseUserDetailInfo(htmlText: string): void { + const html = parseHtml(this._prepareHtmlText(htmlText)); + if (!html) return; + const up = html.strings('//table//tr/td[text()="Uploaded"]/following-sibling::td//text()'); + if (up.length) this.upload = this.numFilesize(up[0]); + const down = html.strings('//table//tr/td[text()="Downloaded"]/following-sibling::td//text()'); + if (down.length) this.download = this.numFilesize(down[0]); + const ratio = html.strings('//table//tr/td[text()="Share ratio"]/following-sibling::td//text()'); + const shareRatio = ratio.length ? parseFloatMp(ratio[0]) : 0; + this.ratio = this.download === 0 ? 0 : shareRatio; + + const seed = html.strings('//table//tr/td[text()="Seed bonus"]/following-sibling::td//text()'); + if (seed.length) { + this.seeding = parseIntMp(at(seed, 1)); + this.seedingSize = this.numFilesize(at(seed, 3)); + } + const level = html.strings('//table//tr/td[text()="Class"]/following-sibling::td//text()'); + if (level.length) this.userLevel = level[0]!.trim(); + const joinAt = html.strings('//table//tr/td[contains(text(), "Join")]/following-sibling::td//text()'); + if (joinAt.length) this.joinAt = normalizeDatetime(joinAt[0]!.split("(")[0]!.trim()) ?? null; + const bonus = html.nodes('//a[contains(@href, "shop.php")]'); + if (bonus.length) this.bonus = parseFloatMp(bonus[0]!.text().trim()); + } + + protected _parseUserTorrentSeedingInfo(htmlText: string, _multiPage = false): NextPage { + const html = parseHtml(htmlText); + if (!hasChildElements(html)) return null; + const sizeCol = 6; + const seedersCol = 7; + const pageSeedingInfo: Array<[number, number]> = []; + const sizes = html.nodes(`//table/tr[position()>1]/td[${sizeCol}]`); + const seeders = html.nodes(`//table/tr[position()>1]/td[${seedersCol}]`); + if (sizes.length && seeders.length) { + for (let i = 0; i < sizes.length; i++) { + const size = this.numFilesize(sizes[i]!.text().trim()); + pageSeedingInfo.push([parseIntMp(seeders[i]?.text().trim()), size]); + } + } + this.seedingInfo.push(...pageSeedingInfo); + return null; + } + + protected _parseMessageUnreadLinks(_htmlText: string, _msgLinks: string[]): NextPage { + return null; + } + + protected _parseMessageContent(_htmlText: string): [string | null, string | null, string | null] { + return [null, null, null]; + } +} diff --git a/packages/adapters/src/sites/user-parsers/gazelle.test.ts b/packages/adapters/src/sites/user-parsers/gazelle.test.ts new file mode 100644 index 00000000..2b794aa6 --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/gazelle.test.ts @@ -0,0 +1,50 @@ +import { + createMockFetch, + GAZELLE_LOGIN_PAGE, + GAZELLE_USER_DETAIL, + GAZELLE_USER_INDEX, + GAZELLE_USER_SEEDING, +} from "@umt/testkit"; +import { describe, expect, it } from "vitest"; +import { GazelleSiteUserInfo } from "./gazelle.ts"; +import { runParser } from "./testing.ts"; + +const GB = 1024 ** 3; + +describe("GazelleSiteUserInfo(MP gazelle.py)", () => { + it("首页用户链接/流量/积分 + 详情等级/加入时间 + 做种表", async () => { + const m = createMockFetch({ + "/": { body: GAZELLE_USER_INDEX }, + "/user.php": { body: GAZELLE_USER_DETAIL }, + "/index.php": { body: GAZELLE_USER_INDEX }, + "/torrents.php": { body: GAZELLE_USER_SEEDING }, + }); + const info = await runParser(GazelleSiteUserInfo, { fetch: m.fetch, cookie: "session=secret" }); + expect(info).toMatchObject({ + schema: "Gazelle", + userId: "4242", + username: "bob", + userLevel: "Power User", + joinAt: "2019-05-06 07:08:09", + upload: Math.round(120.5 * GB), + download: Math.round(40.25 * GB), + ratio: Math.round((120.5 / 40.25) * 1000) / 1000, + bonus: 1234.5, + seeding: 2, + seedingSize: 11 * GB, + seedingInfo: [ + [7, 10 * GB], + [2, GB], + ], + }); + expect(m.requests.find((r) => r.url.pathname === "/torrents.php")!.url.search).toBe("?type=seeding&userid=4242"); + expect(JSON.stringify(info)).not.toContain("secret"); + }); + + it("登录页:errMsg 提示未登录,不再请求", async () => { + const m = createMockFetch({ "/": { body: GAZELLE_LOGIN_PAGE } }); + const info = await runParser(GazelleSiteUserInfo, { fetch: m.fetch }); + expect(info.errMsg).toContain("未检测到已登陆"); + expect(m.requests).toHaveLength(1); + }); +}); diff --git a/packages/adapters/src/sites/user-parsers/gazelle.ts b/packages/adapters/src/sites/user-parsers/gazelle.ts new file mode 100644 index 00000000..fa2be55f --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/gazelle.ts @@ -0,0 +1,180 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/modules/indexer/parser/gazelle.py @ e6820a8 + * + * Gazelle 站点用户数据解析器(MP `GazelleSiteUserInfo`)。 + * 函数对照:`_parse_logged_in` → `_parseLoggedIn`、`_parse_user_base_info` → `_parseUserBaseInfo`、 + * `_parse_site_page` → `_parseSitePage`(空)、`_parse_user_detail_info` → `_parseUserDetailInfo`、 + * `_parse_user_torrent_seeding_info` → `_parseUserTorrentSeedingInfo`、`_parse_user_traffic_info`(空)、 + * `_parse_message_unread_links` / `_parse_message_content`(不支持)。XPath 与正则逐字保留。 + */ +import { type NextPage, SiteParserBase } from "./base.ts"; +import { normalizeDatetime, parseFloatMp, parseIntMp, reSearch } from "./common.ts"; +import { hasChildElements, parseHtml } from "./dom.ts"; +import { round3 } from "./nexus-php.ts"; +import { SiteSchema, type SiteSchemaValue } from "./types.ts"; + +export class GazelleSiteUserInfo extends SiteParserBase { + static override schema: SiteSchemaValue | null = SiteSchema.Gazelle; + + /** 兼容不展示退出入口的站点首页 */ + protected override _parseLoggedIn(htmlText: unknown): boolean | Promise { + const html = parseHtml(String(htmlText ?? "")); + if (hasChildElements(html) && html.nodes('//a[contains(@href, "user.php?id=")]').length) return true; + return super._parseLoggedIn(htmlText); + } + + protected _parseUserBaseInfo(htmlText: string): void { + const html = parseHtml(this._prepareHtmlText(htmlText)); + if (!html) return; + let tmps = html.nodes('//a[contains(@href, "user.php?id=") or contains(@href, "user?id=")]'); + if (tmps.length) { + const m = reSearch(/user(?:\.php)?\?id=(\d+)/, tmps[0]!.attr("href")); + if (m?.[0].trim()) { + this.userid = m[1]!; + this._torrentSeedingPage = `torrents.php?type=seeding&userid=${this.userid}`; + this._userDetailPage = `user.php?id=${this.userid}`; + // lxml `el.text`:首个子节点为文本时取之 + const first = tmps[0]!.node.firstChild; + this.username = first && first.nodeType === 3 ? (first.nodeValue ?? "").trim() : null; + } + } + + let vals = html.strings('//*[@id="header-uploaded-value"]/@data-value'); + if (vals.length) this.upload = this.numFilesize(vals[0]); + else { + vals = html.strings('//li[@id="stats_seeding"]/span/text()'); + if (vals.length) this.upload = this.numFilesize(vals[0]); + } + vals = html.strings('//*[@id="header-downloaded-value"]/@data-value'); + if (vals.length) this.download = this.numFilesize(vals[0]); + else { + vals = html.strings('//li[@id="stats_leeching"]/span/text()'); + if (vals.length) this.download = this.numFilesize(vals[0]); + } + this.ratio = this.download <= 0.0 ? 0.0 : round3(this.upload / this.download); + + vals = html.strings('//a[contains(@href, "bonus")]/@data-tooltip'); + if (vals.length) { + const m = reSearch(/([\d,.]+)/, vals[0]); + if (m?.[1]?.trim()) this.bonus = parseFloatMp(m[1]); + } else { + tmps = html.nodes('//a[contains(@href, "bonus")]'); + if (tmps.length) { + const m = reSearch(/([\d,.]+)/, tmps[0]!.text()); + if (m?.[1]?.trim()) this.bonus = parseFloatMp(m[1]); + } + } + } + + protected _parseSitePage(_htmlText: string): void {} + + protected _parseUserDetailInfo(htmlText: string): void { + const html = parseHtml(htmlText); + if (!hasChildElements(html)) return; + // 用户等级 + let levels = html.strings('//*[@id="class-value"]/@data-value'); + if (levels.length) this.userLevel = levels[0]!.trim(); + else { + levels = html.strings('//li[contains(text(), "用户等级")]/text()'); + if (levels.length) { + const part = levels[0]!.split(":")[1]; + if (part === undefined) throw new Error("IndexError: list index out of range"); + this.userLevel = part.trim(); + } + } + // 加入日期 + let joinAt = html.strings('//*[@id="join-date-value"]/@data-value'); + if (joinAt.length) this.joinAt = normalizeDatetime(joinAt[0]!.trim()) ?? null; + else { + joinAt = html.strings( + '//div[contains(@class, "box_userinfo_stats")]//li[contains(text(), "加入时间")]/span/text()', + ); + if (joinAt.length) this.joinAt = normalizeDatetime(joinAt[0]!.trim()) ?? null; + } + // 兼容以文本形式展示上传/下载的站点(如 JPopsuki) + if (!this.upload) { + const up = html.nodes('//li[starts-with(normalize-space(text()), "Uploaded:")]'); + if (up.length) { + const m = reSearch(/([\d.,]+\s*[GMKT]?i?B)/i, up[0]!.text()); + if (m) this.upload = this.numFilesize(m[1]); + } + } + if (!this.download) { + const down = html.nodes('//li[starts-with(normalize-space(text()), "Downloaded:")]'); + if (down.length) { + const m = reSearch(/([\d.,]+\s*[GMKT]?i?B)/i, down[0]!.text()); + if (m) this.download = this.numFilesize(m[1]); + } + } + if (!this.ratio && this.upload && this.download) this.ratio = round3(this.upload / this.download); + } + + protected _parseUserTorrentSeedingInfo(htmlText: string, multiPage = false): NextPage { + const html = parseHtml(htmlText); + if (!hasChildElements(html)) return null; + let sizeCol = 3; + const headCells = html.nodes('//table[contains(@id, "torrent")]//tr[1]/td'); + if (headCells.length) sizeCol = headCells.length - 3; + const seedersCol = sizeCol + 2; + + let pageSeeding = 0; + let pageSeedingSize = 0; + const pageSeedingInfo: Array<[number, number]> = []; + const semanticRows = html.nodes( + '//table[contains(@id, "torrent")]//tr[contains(concat(" ", normalize-space(@class), " "), " TableTorrent-rowTitle ")]', + ); + if (semanticRows.length) { + // GPW 等现代 Gazelle 页面以语义 class 标记列,逐行按 class 配对读取 + for (const row of semanticRows) { + const sizeCells = row.nodes( + './td[contains(concat(" ", normalize-space(@class), " "), " TableTorrent-cellStatSize ")]', + ); + const seederCells = row.nodes( + './td[contains(concat(" ", normalize-space(@class), " "), " TableTorrent-cellStatSeeders ")]', + ); + if (!sizeCells.length || !seederCells.length) continue; + const size = this.numFilesize(sizeCells[0]!.text().trim()); + const seeders = parseIntMp(seederCells[0]!.text().trim()); + pageSeeding += 1; + pageSeedingSize += size; + pageSeedingInfo.push([seeders, size]); + } + } else { + const sizes = html.nodes(`//table[contains(@id, "torrent")]//tr[position()>1]/td[${sizeCol}]`); + const seeders = html.strings(`//table[contains(@id, "torrent")]//tr[position()>1]/td[${seedersCol}]/text()`); + if (sizes.length && seeders.length) { + pageSeeding = sizes.length; + for (let i = 0; i < sizes.length; i++) { + const size = this.numFilesize(sizes[i]!.text().trim()); + pageSeedingSize += size; + pageSeedingInfo.push([parseIntMp(seeders[i]), size]); + } + } + } + if (multiPage) { + this.seeding += pageSeeding; + this.seedingSize += pageSeedingSize; + this.seedingInfo.push(...pageSeedingInfo); + } else { + if (!this.seeding) this.seeding = pageSeeding; + if (!this.seedingSize) this.seedingSize = pageSeedingSize; + if (!this.seedingInfo.length) this.seedingInfo = pageSeedingInfo; + } + const next = html.strings( + '//a[contains(.//text(), "Next") or contains(.//text(), "下一页") or contains(@title, "下一页") or contains(@title, "Next")]/@href', + ); + return next.length ? next[next.length - 1]!.trim() : null; + } + + protected _parseUserTrafficInfo(_htmlText: string): void {} + + protected _parseMessageUnreadLinks(_htmlText: string, _msgLinks: string[]): NextPage { + return null; + } + + protected _parseMessageContent(_htmlText: string): [string | null, string | null, string | null] { + return [null, null, null]; + } +} diff --git a/packages/adapters/src/sites/user-parsers/hddolby.test.ts b/packages/adapters/src/sites/user-parsers/hddolby.test.ts new file mode 100644 index 00000000..8a74ad9d --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/hddolby.test.ts @@ -0,0 +1,55 @@ +import { createMockFetch, HDDOLBY_USER_DATA, HDDOLBY_USER_ERROR, HDDOLBY_USER_PEERS } from "@umt/testkit"; +import { describe, expect, it } from "vitest"; +import { HDDolbySiteUserInfo } from "./hddolby.ts"; +import { runParser } from "./testing.ts"; + +const API = "https://api.hddolby.com"; + +describe("HDDolbySiteUserInfo(MP hddolby.py)", () => { + it("api/v1/user/data + api/v1/user/peers(GET,x-api-key)", async () => { + const m = createMockFetch({ + [`GET ${API}/api/v1/user/data`]: { body: HDDOLBY_USER_DATA }, + [`GET ${API}/api/v1/user/peers`]: { body: HDDOLBY_USER_PEERS }, + }); + const info = await runParser(HDDolbySiteUserInfo, { + fetch: m.fetch, + url: "https://www.hddolby.com/", + apikey: "dolby-key", + }); + expect(info).toMatchObject({ + schema: "HDDolby", + userId: "1", + username: "ivy", + userLevel: "Veteran User", + joinAt: "2019-03-03 15:30:36", + upload: 852071699418, + download: 1885536536, + ratio: Math.round((852071699418 / 1885536536) * 100) / 100, + bonus: 99774.5, + messageUnread: 2, + seeding: 2, + seedingSize: 1500, + seedingInfo: [ + [4, 1000], + [1, 500], + ], + }); + expect(m.requests[0]!.headers["x-api-key"]).toBe("dolby-key"); + expect(JSON.stringify(info)).not.toContain("dolby-key"); + }); + + it("业务错误:不填用户数据;无 cookie 时 fetchMessages 也不读网页消息", async () => { + const m = createMockFetch({ + [`GET ${API}/api/v1/user/data`]: { body: HDDOLBY_USER_ERROR }, + [`GET ${API}/api/v1/user/peers`]: { body: HDDOLBY_USER_ERROR }, + }); + const info = await runParser(HDDolbySiteUserInfo, { + fetch: m.fetch, + url: "https://www.hddolby.com/", + apikey: "k", + fetchMessages: true, + }); + expect(info.userId).toBeNull(); + expect(m.requests.map((r) => r.url.pathname)).toEqual(["/api/v1/user/data", "/api/v1/user/peers"]); + }); +}); diff --git a/packages/adapters/src/sites/user-parsers/hddolby.ts b/packages/adapters/src/sites/user-parsers/hddolby.ts new file mode 100644 index 00000000..8ba6c277 --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/hddolby.ts @@ -0,0 +1,133 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/modules/indexer/parser/hddolby.py @ e6820a8 + * + * HDDolby 用户数据解析器(MP `HDDolbySiteUserInfo`,apikey 模式)。 + * 函数对照:`HDDolby_sysRoleList` → `HDDOLBY_SYS_ROLE_LIST`、`_parse_site_page` → `_parseSitePage`(api.<域>)、 + * `_parse_logged_in`(恒 True)、`_parse_user_base_info` → `_parseUserBaseInfo`(api/v1/user/data)、 + * `_parse_user_torrent_seeding_info` → `_parseUserTorrentSeedingInfo`(api/v1/user/peers)、 + * `_pase_unread_msgs` → `_paseUnreadMsgs`(API 只给未读数,正文借 NexusPHP 解析器用 Cookie 读网页)。 + * 注:`_user_basic_params = {}` 在 Python 中为假值,故 profile/peers 两个接口实际为 GET(与 MP 一致)。 + */ +import { type NextPage, SiteParserBase } from "./base.ts"; +import { extractDomain, pyFloat, pyInt } from "./common.ts"; +import { NexusPhpSiteUserInfo } from "./nexus-php.ts"; +import { SiteSchema, type SiteSchemaValue } from "./types.ts"; + +type Json = Record; + +/** 用户级别字典(MP `HDDolby_sysRoleList`) */ +export const HDDOLBY_SYS_ROLE_LIST: Record = { + "0": "Peasant", + "1": "User", + "2": "Power User", + "3": "Elite User", + "4": "Crazy User", + "5": "Insane User", + "6": "Veteran User", + "7": "Extreme User", + "8": "Ultimate User", + "9": "Nexus Master", + "10": "VIP", + "11": "Retiree", + "12": "Helper", + "13": "Seeder", + "14": "Transferrer", + "15": "Uploader", + "16": "Torrent Manager", + "17": "Forum Moderator", + "18": "Coder", + "19": "Moderator", + "20": "Administrator", + "21": "Sysop", + "22": "Staff Leader", +}; + +export class HDDolbySiteUserInfo extends SiteParserBase { + static override schema: SiteSchemaValue | null = SiteSchema.HDDolby; + override requestMode = "apikey" as const; + + protected _parseSitePage(_htmlText: string): void { + this._baseUrl = `https://api.${extractDomain(this._baseUrl)}`; + this._userTrafficPage = null; + this._userDetailPage = null; + this._userBasicPage = "api/v1/user/data"; + this._userBasicParams = {}; + this._userBasicHeaders = { "Content-Type": "application/json", Accept: "application/json, text/plain, */*" }; + this._sysMailUnreadPage = null; + this._userMailUnreadPage = null; + this._mailUnreadParams = {}; + this._torrentSeedingPage = "api/v1/user/peers"; + this._torrentSeedingParams = {}; + this._torrentSeedingHeaders = { "Content-Type": "application/json", Accept: "application/json, text/plain, */*" }; + this._additionHeaders = { "x-api-key": this.apikey ?? "" }; + } + + protected override _parseLoggedIn(_htmlText: unknown): boolean { + return true; + } + + protected _parseUserBaseInfo(htmlText: string): void { + if (!htmlText) return; + const detail = JSON.parse(htmlText) as Json | null; + if (detail?.status !== 0) return; + const infos = detail.data as Json[] | null; + if (!infos?.length) return; + const u = infos[0]!; + this.userid = u.id == null ? null : String(u.id); + this.username = (u.username as string) ?? null; + this.userLevel = HDDOLBY_SYS_ROLE_LIST[String(u.class || "1")] ?? null; + this.joinAt = (u.added as string) ?? null; + this.upload = pyInt(u.uploaded || "0"); + this.download = pyInt(u.downloaded || "0"); + this.ratio = this.download ? Math.round((this.upload / this.download) * 100) / 100 : 0; + this.bonus = pyFloat(u.seedbonus || "0"); + this.messageUnread = pyInt(u.unread_messages || "0"); + } + + protected _parseUserTrafficInfo(_htmlText: string): void {} + + protected _parseUserDetailInfo(_htmlText: string): void {} + + protected _parseUserTorrentSeedingInfo(htmlText: string, _multiPage = false): NextPage { + if (!htmlText) return null; + const seedingInfo = JSON.parse(htmlText) as Json | null; + if (seedingInfo?.status !== 0) return null; + const torrents = (seedingInfo.data ?? []) as Json[]; + let pageSeedingSize = 0; + const pageSeedingInfo: Array<[number, number]> = []; + for (const info of torrents) { + const size = Number(info.size ?? 0); + const seeder = Number(info.seeders || 1); + pageSeedingSize += size; + pageSeedingInfo.push([seeder, size]); + } + this.seeding += torrents.length; + this.seedingSize += pageSeedingSize; + this.seedingInfo.push(...pageSeedingInfo); + return null; + } + + protected _parseMessageUnreadLinks(_htmlText: string, _msgLinks: string[]): NextPage { + return null; + } + + protected _parseMessageContent(_htmlText: string): [string | null, string | null, string | null] { + return [null, null, null]; + } + + /** API 仅返回未读数量,正文需通过 NexusPHP 网页接口读取(需 Cookie) */ + protected override async _paseUnreadMsgs(): Promise { + if (!this._siteCookie) return; + const nexus = new NexusPhpSiteUserInfo({ ...this._opts, cookie: this._siteCookie }); + nexus.messageUnread = this.messageUnread; + nexus.messageReadForce = this.messageUnread > 0; + await (nexus as unknown as { _paseUnreadMsgs(): Promise })._paseUnreadMsgs(); + this.messageUnreadContents = [...nexus.messageUnreadContents]; + this.httpErrors.push(...nexus.httpErrors); + if (this.messageUnreadContents.length && !this.messageUnread) { + this.messageUnread = this.messageUnreadContents.length; + } + } +} diff --git a/packages/adapters/src/sites/user-parsers/index.ts b/packages/adapters/src/sites/user-parsers/index.ts new file mode 100644 index 00000000..7a00a100 --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/index.ts @@ -0,0 +1,45 @@ +export { SiteParserBase, type SiteParserClass, urljoin } from "./base.ts"; +export { BitptSiteUserInfo } from "./bitpt.ts"; +export { + containsChinese, + extractDomain, + formatTimestamp, + isLoggedIn, + normalizeDatetime, + parseFloatMp, + parseIntMp, + underChallenge, +} from "./common.ts"; +export { DiscuzUserInfo } from "./discuz.ts"; +export { hasChildElements, parseHtml, XNode } from "./dom.ts"; +export { FileListSiteUserInfo } from "./file-list.ts"; +export { GazelleSiteUserInfo } from "./gazelle.ts"; +export { HDDolbySiteUserInfo } from "./hddolby.ts"; +export { IptSiteUserInfo } from "./ipt.ts"; +export { MTorrentSiteUserInfo } from "./mtorrent.ts"; +export { NexusAudiencesSiteUserInfo } from "./nexus-audiences.ts"; +export { NexusHhanclubSiteUserInfo } from "./nexus-hhanclub.ts"; +export { NexusPhpSiteUserInfo } from "./nexus-php.ts"; +export { NexusProjectSiteUserInfo } from "./nexus-project.ts"; +export { NexusRabbitSiteUserInfo } from "./nexus-rabbit.ts"; +export { + getUserParserClass, + refreshSiteUserData, + type SiteUserDataRequest, + USER_PARSER_CLASSES, +} from "./registry.ts"; +export { RousiSiteUserInfo } from "./rousi.ts"; +export { SmallHorseSiteUserInfo } from "./small-horse.ts"; +export { SunnyPTSiteUserInfo } from "./sunnypt.ts"; +export { TNodeSiteUserInfo } from "./tnode.ts"; +export { TorrentLeechSiteUserInfo } from "./torrent-leech.ts"; +export { + SiteSchema, + type SiteSchemaValue, + type SiteUnreadMessage, + type SiteUserInfo, + type SiteUserParserOptions, +} from "./types.ts"; +export { Unit3dSiteUserInfo } from "./unit3d.ts"; +export { YemaSiteUserInfo } from "./yema.ts"; +export { ZhixingSiteUserInfo } from "./zhixing.ts"; diff --git a/packages/adapters/src/sites/user-parsers/ipt.test.ts b/packages/adapters/src/sites/user-parsers/ipt.test.ts new file mode 100644 index 00000000..e62dd2ed --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/ipt.test.ts @@ -0,0 +1,31 @@ +import { createMockFetch, IPT_USER_DETAIL, IPT_USER_INDEX } from "@umt/testkit"; +import { describe, expect, it } from "vitest"; +import { IptSiteUserInfo } from "./ipt.ts"; +import { runParser } from "./testing.ts"; + +describe("IptSiteUserInfo(MP ipt.py)", () => { + it("统计块:分享率/上传/下载/做种/下载中/积分;详情页等级与加入日期", async () => { + const m = createMockFetch({ "/": { body: IPT_USER_INDEX }, "/user.php": { body: IPT_USER_DETAIL } }); + const info = await runParser(IptSiteUserInfo, { fetch: m.fetch }); + expect(info).toMatchObject({ + schema: "IPTorrents", + userId: "777", + username: "dave", + ratio: 2.5, + upload: Math.round(1.2 * 1024 ** 4), + download: 480 * 1024 ** 3, + seeding: 15, + leeching: 3, + bonus: 4321.5, + userLevel: "Power User", + joinAt: "2018-01-02 03:04:05", + }); + expect(m.requests.find((r) => r.url.pathname === "/user.php")!.url.search).toBe("?u=777"); + }); + + it("统计块结构缺列时与 MP 一样抛 IndexError", async () => { + const broken = IPT_USER_INDEX.replace("480.00 GB", ""); + const m = createMockFetch({ "/": { body: broken } }); + await expect(runParser(IptSiteUserInfo, { fetch: m.fetch })).rejects.toThrow(/IndexError/); + }); +}); diff --git a/packages/adapters/src/sites/user-parsers/ipt.ts b/packages/adapters/src/sites/user-parsers/ipt.ts new file mode 100644 index 00000000..66fb6cba --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/ipt.ts @@ -0,0 +1,101 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/modules/indexer/parser/ipt.py @ e6820a8 + * + * IPTorrents 站点用户数据解析器(MP `IptSiteUserInfo`)。 + * 函数对照:`_parse_user_base_info` → `_parseUserBaseInfo`、`_parse_site_page`(空)、 + * `_parse_user_detail_info` → `_parseUserDetailInfo`、`_parse_user_torrent_seeding_info` → `_parseUserTorrentSeedingInfo`、 + * 其余为空实现。XPath 逐字保留;统计块下标越界时与 MP 一样抛 IndexError。 + */ +import { type NextPage, SiteParserBase } from "./base.ts"; +import { normalizeDatetime, parseFloatMp, parseIntMp, reSearch } from "./common.ts"; +import { hasChildElements, parseHtml } from "./dom.ts"; +import { SiteSchema, type SiteSchemaValue } from "./types.ts"; + +/** Python 列表下标(越界抛 IndexError,保持 MP 失败语义) */ +function at(arr: T[], i: number): T { + const v = i < 0 ? arr[arr.length + i] : arr[i]; + if (v === undefined) throw new Error("IndexError: list index out of range"); + return v; +} + +export class IptSiteUserInfo extends SiteParserBase { + static override schema: SiteSchemaValue | null = SiteSchema.Ipt; + + protected _parseUserBaseInfo(htmlText: string): void { + const html = parseHtml(this._prepareHtmlText(htmlText)); + if (!html) return; + const names = html.strings('//a[contains(@href, "/u/")]//text()'); + const ids = html.strings('//a[contains(@href, "/u/")]/@href'); + if (names.length) this.username = String(at(names, -1)); + if (ids.length) { + const m = reSearch(/\/u\/(\d+)/, ids[0]); + if (m?.[0].trim()) { + this.userid = m[1]!; + this._userDetailPage = `user.php?u=${this.userid}`; + this._torrentSeedingPage = `peers?u=${this.userid}`; + } + } + const stats = html.nodes('//div[@class = "stats"]/div/div'); + if (stats.length) { + const s = stats[0]!; + const spans = s.strings("span/text()"); + const links = s.nodes("a"); + this.upload = this.numFilesize(String(at(spans, 1)).trim()); + this.download = this.numFilesize(String(at(spans, 2)).trim()); + this.seeding = parseIntMp(at(at(links, 2).strings("text()"), 0)); + this.leeching = parseIntMp(at(at(links, 2).strings("text()"), 1)); + this.ratio = parseFloatMp(String(at(spans, 0)).trim().replaceAll("-", "0")); + this.bonus = parseFloatMp(at(at(links, 3).strings("text()"), 0)); + } + } + + protected _parseSitePage(_htmlText: string): void {} + + protected _parseUserDetailInfo(htmlText: string): void { + const html = parseHtml(htmlText); + if (!hasChildElements(html)) return; + const levels = html.strings('//tr/th[text()="Class"]/following-sibling::td[1]/text()'); + if (levels.length) this.userLevel = levels[0]!.trim(); + const joinAt = html.strings('//tr/th[text()="Join date"]/following-sibling::td[1]/text()'); + if (joinAt.length) this.joinAt = normalizeDatetime(joinAt[0]!.split(" (")[0]) ?? null; + } + + protected _parseUserTorrentSeedingInfo(htmlText: string, _multiPage = false): NextPage { + const html = parseHtml(htmlText); + if (!hasChildElements(html)) return null; + let seedingEndPos = 3; + if (html.nodes('//tr/td[text() = "Leechers"]').length) { + seedingEndPos = html.nodes('//tr/td[text() = "Leechers"]/../preceding-sibling::tr').length + 1; + seedingEndPos = seedingEndPos - 3; + } + let pageSeeding = 0; + let pageSeedingSize = 0; + const torrents = html.strings('//tr/td[text() = "Seeders"]/../following-sibling::tr/td[position()=6]/text()'); + if (torrents.length) { + pageSeeding = seedingEndPos; + // Python 切片 [:n](n 为负时从尾部截) + for (let perSize of torrents.slice(0, seedingEndPos)) { + if (perSize.includes("(") && perSize.includes(")")) { + perSize = perSize.split("(").at(-1)!; + perSize = perSize.split(")")[0]!; + } + pageSeedingSize += this.numFilesize(perSize); + } + } + this.seeding = pageSeeding; + this.seedingSize = pageSeedingSize; + return null; + } + + protected _parseUserTrafficInfo(_htmlText: string): void {} + + protected _parseMessageUnreadLinks(_htmlText: string, _msgLinks: string[]): NextPage { + return null; + } + + protected _parseMessageContent(_htmlText: string): [string | null, string | null, string | null] { + return [null, null, null]; + } +} diff --git a/packages/adapters/src/sites/user-parsers/mtorrent.test.ts b/packages/adapters/src/sites/user-parsers/mtorrent.test.ts new file mode 100644 index 00000000..12c02a10 --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/mtorrent.test.ts @@ -0,0 +1,91 @@ +import { + createMockFetch, + MTORRENT_USER_KEY_INVALID, + MTORRENT_USER_MESSAGES, + MTORRENT_USER_PEER_STATUS, + MTORRENT_USER_PROFILE, + MTORRENT_USER_SEEDING_PAGE1, + MTORRENT_USER_SEEDING_PAGE2, +} from "@umt/testkit"; +import { describe, expect, it } from "vitest"; +import { MTorrentSiteUserInfo } from "./mtorrent.ts"; +import { runParser } from "./testing.ts"; + +const API = "https://api.m-team.cc"; + +function routes() { + return createMockFetch({ + [`POST ${API}/api/member/profile`]: { body: MTORRENT_USER_PROFILE }, + [`POST ${API}/api/member/getUserTorrentList`]: (req) => ({ + body: JSON.parse(req.body ?? "{}").pageNumber === 1 ? MTORRENT_USER_SEEDING_PAGE1 : MTORRENT_USER_SEEDING_PAGE2, + }), + [`POST ${API}/api/tracker/myPeerStatus`]: { body: MTORRENT_USER_PEER_STATUS }, + [`POST ${API}/api/msg/search`]: { body: MTORRENT_USER_MESSAGES }, + [`POST ${API}/api/msg/markRead`]: { body: { code: "0" } }, + }); +} + +describe("MTorrentSiteUserInfo(MP mtorrent.py)", () => { + it("profile + 做种分页(按 myPeerStatus 总数翻页);默认不读消息、不标记已读", async () => { + const m = routes(); + const info = await runParser(MTorrentSiteUserInfo, { + fetch: m.fetch, + url: "https://kp.m-team.cc/", + apikey: "KEY-secret", + }); + expect(info).toMatchObject({ + schema: "MTorrent", + domain: "m-team.cc", + userId: "31337", + username: "henry", + userLevel: "Veteran User", + joinAt: "2020-10-11 12:13:14", + upload: 3000, + download: 1000, + ratio: 3, + bonus: 4567.8, + seeding: 3, + seedingSize: 600, + seedingInfo: [ + [5, 100], + [2, 200], + [1, 300], + ], + messageUnreadContents: [], + }); + const profile = m.requests[0]!; + expect(profile.url.href).toBe(`${API}/api/member/profile`); + expect(profile.headers["x-api-key"]).toBe("KEY-secret"); + expect(m.requests.filter((r) => r.url.pathname === "/api/member/getUserTorrentList")).toHaveLength(2); + expect(m.requests.some((r) => r.url.pathname === "/api/msg/markRead")).toBe(false); + expect(JSON.stringify(info)).not.toContain("KEY-secret"); + }); + + it("fetchMessages=true:读出未读消息并逐条 markRead(写操作)", async () => { + const m = routes(); + const info = await runParser(MTorrentSiteUserInfo, { + fetch: m.fetch, + url: "https://kp.m-team.cc/", + apikey: "k", + fetchMessages: true, + }); + expect(info.messageUnreadContents).toEqual([{ head: "欢迎", date: "2026-09-01 10:00:00", content: "hello" }]); + // 与 MP 一致:消息直接写入 contents,不经 msg_links,故 messageUnread 不回填 + expect(info.messageUnread).toBe(0); + const mark = m.requests.filter((r) => r.url.pathname === "/api/msg/markRead"); + expect(mark).toHaveLength(1); + expect(mark[0]!.body).toBe("msgId=m1"); + }); + + it("key 无效(业务 code≠0):不填用户数据;未配置 key 时不发请求", async () => { + const m = createMockFetch({ [`POST ${API}/api/member/profile`]: { body: MTORRENT_USER_KEY_INVALID } }); + const info = await runParser(MTorrentSiteUserInfo, { fetch: m.fetch, url: "https://kp.m-team.cc/", apikey: "bad" }); + expect(info.userId).toBeNull(); + expect(info.upload).toBe(0); + + const m2 = createMockFetch({}); + const info2 = await runParser(MTorrentSiteUserInfo, { fetch: m2.fetch, url: "https://kp.m-team.cc/" }); + expect(m2.requests).toHaveLength(0); + expect(info2.errMsg).toBeTruthy(); + }); +}); diff --git a/packages/adapters/src/sites/user-parsers/mtorrent.ts b/packages/adapters/src/sites/user-parsers/mtorrent.ts new file mode 100644 index 00000000..80c90e6e --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/mtorrent.ts @@ -0,0 +1,163 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/modules/indexer/parser/mtorrent.py @ e6820a8 + * + * M-Team(mTorrent API)用户数据解析器(MP `MTorrentSiteUserInfo`,apikey 模式)。 + * 函数对照:`MTeam_sysRoleList` → `MTEAM_SYS_ROLE_LIST`、`_parse_site_page` → `_parseSitePage`(改用 api.<域> 接口)、 + * `_parse_logged_in` → `_parseLoggedIn`(恒 True)、`_parse_user_base_info` → `_parseUserBaseInfo`(api/member/profile)、 + * `_parse_user_torrent_seeding_info` → `_parseUserTorrentSeedingInfo`(api/member/getUserTorrentList 分页, + * 以 api/tracker/myPeerStatus 的 seeder 总数判断是否还有下一页)、 + * `_parse_message_unread_links` → `_parseMessageUnreadLinks`(api/msg/search,直接读出正文)。 + * + * 注意(写操作):与 MP 一样,读取未读消息后会调用 `api/msg/markRead` 把消息标记为已读。 + * 该流程只在 `fetchMessages=true`(MP `SITE_MESSAGE`)时执行,默认不触发。 + * API Key 只放在 `x-api-key` 请求头,不进入结果。 + */ +import { type NextPage, SiteParserBase, urljoin } from "./base.ts"; +import { extractDomain, pyInt } from "./common.ts"; +import { SiteSchema, type SiteSchemaValue } from "./types.ts"; + +type Json = Record; + +/** 用户级别字典(MP `MTeam_sysRoleList`) */ +export const MTEAM_SYS_ROLE_LIST: Record = { + "1": "User", + "2": "Power User", + "3": "Elite User", + "4": "Crazy User", + "5": "Insane User", + "6": "Veteran User", + "7": "Extreme User", + "8": "Ultimate User", + "9": "Nexus Master", + "10": "VIP", + "11": "Retiree", + "12": "Uploader", + "13": "Moderator", + "14": "Administrator", + "15": "Sysop", + "16": "Staff", + "17": "Offer memberStaff", + "18": "Bet memberStaff", +}; + +export class MTorrentSiteUserInfo extends SiteParserBase { + static override schema: SiteSchemaValue | null = SiteSchema.MTorrent; + override requestMode = "apikey" as const; + + protected _parseSitePage(_htmlText: string): void { + // 更换 api 地址 + this._baseUrl = `https://api.${extractDomain(this._baseUrl)}`; + this._userTrafficPage = null; + this._userDetailPage = null; + this._userBasicPage = "api/member/profile"; + this._userBasicParams = { uid: this.userid }; + this._sysMailUnreadPage = null; + this._userMailUnreadPage = "api/msg/search"; + this._mailUnreadParams = { keyword: "", box: "-2", type: "pageNumber", pageSize: 100 }; + this._torrentSeedingPage = "api/member/getUserTorrentList"; + this._torrentSeedingHeaders = { + "Content-Type": "application/json", + Accept: "application/json, text/plain, */*", + }; + this._additionHeaders = { "x-api-key": this.apikey ?? "" }; + } + + /** 暂时跳过检测(与 MP 一致) */ + protected override _parseLoggedIn(_htmlText: unknown): boolean { + return true; + } + + /** 基础信息:合并了流量与详情解析 */ + protected _parseUserBaseInfo(htmlText: string): void { + if (!htmlText) return; + const detail = JSON.parse(htmlText) as Json | null; + if (detail?.code !== "0") return; + const u = (detail.data ?? {}) as Json; + const memberCount = (u.memberCount ?? {}) as Json; + this.userid = u.id == null ? null : String(u.id); + this.username = (u.username as string) ?? null; + this.userLevel = MTEAM_SYS_ROLE_LIST[String(u.role || "1")] ?? null; + this.joinAt = ((u.memberStatus ?? {}) as Json).createdDate as string | null; + this.upload = pyInt(memberCount.uploaded || "0"); + this.download = pyInt(memberCount.downloaded || "0"); + this.ratio = (memberCount.shareRate as number) || 0; + this.bonus = (memberCount.bonus as number) || 0; + this.messageReadForce = true; + this._torrentSeedingParams = { pageNumber: 1, pageSize: 200, type: "SEEDING", userid: this.userid }; + } + + protected _parseUserTrafficInfo(_htmlText: string): void {} + + protected _parseUserDetailInfo(_htmlText: string): void {} + + protected async _parseUserTorrentSeedingInfo(htmlText: string, _multiPage = false): Promise { + if (!htmlText) return null; + const seedingInfo = JSON.parse(htmlText) as Json | null; + if (seedingInfo?.code !== "0") return null; + const seedingData = seedingInfo.data || {}; + if (typeof seedingData !== "object" || Array.isArray(seedingData)) return null; + const torrents = (seedingData as Json).data || []; + if (!Array.isArray(torrents)) return null; + let pageSeeding = 0; + let pageSeedingSize = 0; + const pageSeedingInfo: Array<[number, number]> = []; + for (const info of torrents as unknown[]) { + if (!info || typeof info !== "object" || Array.isArray(info)) continue; + const torrent = (info as Json).torrent; + if (!torrent || typeof torrent !== "object" || Array.isArray(torrent)) continue; + const size = pyInt((torrent as Json).size || "0"); + const seeders = pyInt((torrent as Json).source || "0"); + pageSeeding += 1; + pageSeedingSize += size; + pageSeedingInfo.push([seeders, size]); + } + this.seeding += pageSeeding; + this.seedingSize += pageSeedingSize; + this.seedingInfo.push(...pageSeedingInfo); + + // 查询总做种数 + let seederCount = 0; + try { + const result = await this._getPageContent(urljoin(this._baseUrl, "api/tracker/myPeerStatus"), { + uid: this.userid, + }); + if (result) { + const seederInfo = JSON.parse(result) as Json; + seederCount = pyInt(((seederInfo.data ?? {}) as Json).seeder || 0); + } + } catch { + // MP:记录日志后按 0 处理 + } + if (!seederCount) return null; + if (this.seeding >= seederCount) return null; + // 还有下一页 + (this._torrentSeedingParams as Json).pageNumber = Number((this._torrentSeedingParams as Json).pageNumber) + 1; + return ""; + } + + /** 解析未读消息链接,这里直接读出详情(并标记已读,见文件头) */ + protected async _parseMessageUnreadLinks(htmlText: string, _msgLinks: string[]): Promise { + if (!htmlText) return null; + const messagesInfo = JSON.parse(htmlText) as Json | null; + if (messagesInfo?.code !== "0") return null; + const messages = (((messagesInfo.data ?? {}) as Json).data ?? []) as Json[]; + for (const message of messages) { + if (!message.unread) continue; + const head = message.title as string | null; + const date = message.createdDate as string | null; + const content = message.context as string | null; + if (head && date && content) { + this.messageUnreadContents.push([head, date, content]); + // 设置已读 + await this._getPageContent(urljoin(this._baseUrl, "api/msg/markRead"), { msgId: message.id }); + } + } + return null; + } + + protected _parseMessageContent(_htmlText: string): [string | null, string | null, string | null] { + return [null, null, null]; + } +} diff --git a/packages/adapters/src/sites/user-parsers/nexus-audiences.test.ts b/packages/adapters/src/sites/user-parsers/nexus-audiences.test.ts new file mode 100644 index 00000000..c5d3de27 --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/nexus-audiences.test.ts @@ -0,0 +1,75 @@ +import { + AUDIENCES_USER_DETAILS, + AUDIENCES_USER_INDEX, + AUDIENCES_USER_MAILBOX, + AUDIENCES_USER_MESSAGE_901, + AUDIENCES_USER_SEEDING, + createMockFetch, + NEXUS_PHP_LOGIN_PAGE, +} from "@umt/testkit"; +import { describe, expect, it } from "vitest"; +import { NexusAudiencesSiteUserInfo } from "./nexus-audiences.ts"; +import { runParser } from "./testing.ts"; + +const GiB = 1024 ** 3; +const TiB = 1024 ** 4; + +function routes() { + return createMockFetch({ + "/": { body: AUDIENCES_USER_INDEX }, + "/userdetails.php": { body: AUDIENCES_USER_DETAILS }, + "/getusertorrentlistajax.php": { body: AUDIENCES_USER_SEEDING }, + "/messages.php": { body: AUDIENCES_USER_MAILBOX }, + "/messageviewer.php?id=901": { body: AUDIENCES_USER_MESSAGE_901 }, + }); +} + +describe("NexusAudiencesSiteUserInfo(MP nexus_audiences.py)", () => { + it("userbar data-uploader-* + compact metric + Total 行汇总做种", async () => { + const m = routes(); + const info = await runParser(NexusAudiencesSiteUserInfo, { fetch: m.fetch, cookie: "c_secure=abc" }); + expect(info).toMatchObject({ + schema: "NexusAudiences", + userId: "233", + username: "bob", + userLevel: "Elite User", + joinAt: "2020-05-06 07:08:09", + upload: 2 * TiB, + download: 100 * GiB, + ratio: 20.48, + bonus: 8888.5, + leeching: 1, + seeding: 31, + seedingSize: Math.round(3.5 * TiB), + messageUnread: 2, + messageUnreadContents: [], + }); + const seedReq = m.requests.find((r) => r.url.pathname === "/getusertorrentlistajax.php")!; + expect(seedReq.headers.referer).toBe("https://pt.example.com/userdetails.php?id=233"); + expect(m.requests.some((r) => r.url.pathname === "/messages.php")).toBe(false); + }); + + it("读取未读私信:首页不带 page、详情解析、详情为空用列表预览兜底、达到未读数停止翻页", async () => { + const m = routes(); + const info = await runParser(NexusAudiencesSiteUserInfo, { fetch: m.fetch, fetchMessages: true }); + const mailReqs = m.requests.filter((r) => r.url.pathname === "/messages.php"); + expect(mailReqs.map((r) => r.url.search)).toEqual(["?action=viewmailbox&box=1&unread=yes"]); + expect(info.messageUnreadContents).toEqual([ + { head: "种子被删除", date: "2026-09-01 10:00:00", content: "你的种子 X 因重复已被删除。" }, + { head: "欢迎", date: "2026-09-02 11:00:00", content: "欢迎加入观众" }, + ]); + }); + + it("未登录:只请求首页", async () => { + const m = createMockFetch({ "/": { body: NEXUS_PHP_LOGIN_PAGE } }); + const info = await runParser(NexusAudiencesSiteUserInfo, { fetch: m.fetch }); + expect(info.userId).toBeNull(); + expect(m.requests).toHaveLength(1); + }); + + it("收件箱入口解析:总数/未读、角标、仅未读标记", () => { + expect(NexusAudiencesSiteUserInfo.extractInboxUnreadPair("收件箱 1,749 / 172")).toBe(172); + expect(NexusAudiencesSiteUserInfo.extractInboxUnreadBadge(" 5 ")).toBe(5); + expect(NexusAudiencesSiteUserInfo.extractInboxUnreadBadge("新")).toBeNull(); + }); +}); diff --git a/packages/adapters/src/sites/user-parsers/nexus-audiences.ts b/packages/adapters/src/sites/user-parsers/nexus-audiences.ts new file mode 100644 index 00000000..8ab96bf7 --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/nexus-audiences.ts @@ -0,0 +1,401 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/modules/indexer/parser/nexus_audiences.py @ e6820a8 + * + * 观众(Audiences)站点用户信息解析器,继承 NexusPHP 通用解析。 + * + * 函数对照(`NexusAudiencesSiteUserInfo`): + * `__init__` → constructor、`_parse_message_unread` → `_parseMessageUnread`、 + * `_parse_message_unread_links` → `_parseMessageUnreadLinks`、`_parse_message_content` → `_parseMessageContent`、 + * `_pase_unread_msgs` → `_paseUnreadMsgs`、`__parse_unread_message_list_page` → `parseUnreadMessageListPage`、 + * `__reset_unread_message_parse_state` → `resetUnreadMessageParseState`、`__filter_new_message_links` → `filterNewMessageLinks`、 + * `__parse_table_unread_message_links` → `parseTableUnreadMessageLinks`、`__parse_pm_item_unread_message_links` → + * `parsePmItemUnreadMessageLinks`、`__cache_pm_item_preview` → `cachePmItemPreview`、`__extract_pm_item_text` / + * `__extract_first_text` / `__extract_pm_view_meta` / `__normalize_text` → 同名、`__fill_empty_message_content_from_list` → + * `fillEmptyMessageContentFromList`、`__should_fetch_next_unread_page` / `__has_reached_expected_unread_count` / + * `__is_empty_message_content` / `__build_unread_mailbox_page` / `__build_next_unread_mailbox_page` → 同名、 + * `_parse_user_traffic_info` / `_parse_user_detail_info` → 同名(追加 `__parse_userbar_info`)、 + * `__parse_user_identity` / `__parse_uploader_stats` / `__parse_compact_metric` / `__set_metric_value` / + * `__parse_inbox_unread` / `__extract_inbox_unread_pair` / `__extract_inbox_unread_badge` / + * `__has_inbox_unread_marker` → 同名、`_parse_seeding_pages` → `_parseSeedingPages`(Total 行汇总)。 + * XPath 与正则逐字保留。 + */ +import { urljoin } from "./base.ts"; +import { parseFloatMp, parseIntMp, reSearch } from "./common.ts"; +import { hasChildElements, parseHtml, type XNode } from "./dom.ts"; +import { NexusPhpSiteUserInfo } from "./nexus-php.ts"; +import { SiteSchema, type SiteSchemaValue, type SiteUserParserOptions } from "./types.ts"; + +type Msg = [string | null, string | null, string | null]; + +const UNKNOWN_UNREAD_COUNT = 99999; +const MAX_MAIL_PAGES = 50; + +function normalizeText(text: string | null | undefined): string | null { + if (!text) return null; + const t = text.replaceAll(" ", " ").replace(/\s+/g, " ").trim(); + return t || null; +} + +export class NexusAudiencesSiteUserInfo extends NexusPhpSiteUserInfo { + static override schema: SiteSchemaValue | null = SiteSchema.NexusAudiences; + + private nextMailPage = 1; + private readonly seenUnreadMessageLinks = new Set(); + private readonly messageListPreviews = new Map(); + + constructor(opts: SiteUserParserOptions) { + super(opts); + this._userMailUnreadPage = NexusAudiencesSiteUserInfo.buildUnreadMailboxPage(1); + this._sysMailUnreadPage = null; + } + + /** 解析新版顶部用户栏中的未读消息数 */ + protected override _parseMessageUnread(htmlText: string): void { + const html = parseHtml(htmlText); + if (!hasChildElements(html)) { + super._parseMessageUnread(htmlText); + return; + } + const messageTools = html.nodes( + '//a[contains(@class, "site-userbar__compact-tool") and contains(@href, "messages.php") ' + + 'and (contains(@class, "site-userbar__compact-tool--has-unread") ' + + 'or .//*[contains(@class, "site-userbar__compact-tool-badge--unread")])]' + + '|//a[contains(@href, "messages.php") ' + + 'and (contains(@title, "收件箱") or contains(@aria-label, "收件箱"))]', + ); + for (const link of messageTools) { + const unread = NexusAudiencesSiteUserInfo.parseInboxUnread(link); + if (unread !== null) { + this.messageUnread = unread; + return; + } + } + if (messageTools.length) return; + super._parseMessageUnread(htmlText); + } + + protected override _parseMessageUnreadLinks(htmlText: string, msgLinks: string[]): string | null { + const html = parseHtml(htmlText); + if (!hasChildElements(html)) return null; + const messageLinks = NexusAudiencesSiteUserInfo.parseTableUnreadMessageLinks(html); + messageLinks.push(...this.parsePmItemUnreadMessageLinks(html)); + const newLinks = this.filterNewMessageLinks(messageLinks); + msgLinks.push(...newLinks); + return this.buildNextUnreadMailboxPage(this.shouldFetchNextUnreadPage(newLinks)); + } + + protected override _parseMessageContent(htmlText: string): Msg { + const html = parseHtml(htmlText); + if (hasChildElements(html)) { + const head = NexusAudiencesSiteUserInfo.extractFirstText( + html, + '//*[contains(concat(" ", normalize-space(@class), " "), " pm-hero__title ")]', + ); + const date = NexusAudiencesSiteUserInfo.extractPmViewMeta(html, "日期"); + const content = NexusAudiencesSiteUserInfo.extractFirstText( + html, + '//*[contains(concat(" ", normalize-space(@class), " "), " pm-view__body ")]', + ); + if (!NexusAudiencesSiteUserInfo.isEmptyMessageContent(head, date, content)) return [head, date, content]; + } + return super._parseMessageContent(htmlText); + } + + /** 避免异常分页重复通知和空详情通知 */ + protected override async _paseUnreadMsgs(): Promise { + this.resetUnreadMessageParseState(); + const unreadMsgLinks: string[] = []; + if (this.messageUnread > 0 || this.messageReadForce) { + let nextPage = await this.parseUnreadMessageListPage(this._userMailUnreadPage, unreadMsgLinks); + let guard = 0; + while (nextPage && guard++ < MAX_MAIL_PAGES) { + nextPage = await this.parseUnreadMessageListPage(nextPage, unreadMsgLinks); + } + } + if (this.messageUnread === UNKNOWN_UNREAD_COUNT) this.messageUnread = unreadMsgLinks.length; + else if (unreadMsgLinks.length && !this.messageUnread) this.messageUnread = unreadMsgLinks.length; + for (const msgLink of unreadMsgLinks) { + let [head, date, content] = this._parseMessageContent( + await this._getPageContent(urljoin(this._baseUrl, msgLink), this._mailContentParams, this._mailContentHeaders), + ); + [head, date, content] = this.fillEmptyMessageContentFromList(msgLink, head, date, content); + if (NexusAudiencesSiteUserInfo.isEmptyMessageContent(head, date, content)) continue; + this.messageUnreadContents.push([head, date, content]); + } + } + + private async parseUnreadMessageListPage(link: string | null, unreadMsgLinks: string[]): Promise { + if (!link) return null; + return this._parseMessageUnreadLinks( + await this._getPageContent(urljoin(this._baseUrl, link), this._mailUnreadParams, this._mailUnreadHeaders), + unreadMsgLinks, + ); + } + + private resetUnreadMessageParseState(): void { + this.nextMailPage = 1; + this.seenUnreadMessageLinks.clear(); + this.messageListPreviews.clear(); + } + + private filterNewMessageLinks(messageLinks: string[]): string[] { + const out: string[] = []; + for (const link of messageLinks) { + const key = urljoin(this._baseUrl, link); + if (this.seenUnreadMessageLinks.has(key)) continue; + this.seenUnreadMessageLinks.add(key); + out.push(link); + } + return out; + } + + static parseTableUnreadMessageLinks(html: XNode): string[] { + return html.strings( + '//tr[.//img[contains(concat(" ", normalize-space(@class), " "), " unreadpm ") ' + + 'or @alt="Unread" or @title="未读"]]/td/a[contains(@href, "viewmessage")]/@href', + ); + } + + private parsePmItemUnreadMessageLinks(html: XNode): string[] { + const links: string[] = []; + let rows = html.nodes( + '//*[contains(concat(" ", normalize-space(@class), " "), " pm-item-row ") ' + + 'and contains(concat(" ", normalize-space(@class), " "), " is-unread ")]', + ); + if (!rows.length) { + rows = html.nodes( + '//*[contains(concat(" ", normalize-space(@class), " "), " pm-item-row ") ' + + 'and .//*[contains(concat(" ", normalize-space(@class), " "), " pm-item__status--unread ") ' + + 'or @title="未读"]]', + ); + } + for (const row of rows) { + const rowLinks = row.strings('.//a[contains(@href, "viewmessage")]/@href'); + if (!rowLinks.length) continue; + const link = rowLinks[0]!.trim(); + if (!link) continue; + links.push(link); + this.cachePmItemPreview(link, row); + } + return links; + } + + private cachePmItemPreview(link: string, row: XNode): void { + const f = (cls: string) => + NexusAudiencesSiteUserInfo.extractPmItemText( + row, + `.//*[contains(concat(" ", normalize-space(@class), " "), " ${cls} ")]`, + ); + this.messageListPreviews.set(urljoin(this._baseUrl, link), [ + f("pm-item__subject"), + f("pm-item__time"), + f("pm-item__preview"), + ]); + } + + static extractPmItemText(row: XNode, xp: string): string | null { + const nodes = row.nodes(xp); + if (!nodes.length) return null; + return normalizeText(nodes[0]!.text()); + } + + static extractFirstText(html: XNode, xp: string): string | null { + const nodes = html.nodes(xp); + if (!nodes.length) return null; + return normalizeText(nodes[0]!.text()); + } + + static extractPmViewMeta(html: XNode, label: string): string | null { + const values = html.nodes( + '//*[contains(concat(" ", normalize-space(@class), " "), " pm-view__meta ") ' + + `and .//*[contains(concat(" ", normalize-space(@class), " "), " pm-view__label ") ` + + `and normalize-space()="${label}"]]` + + '//*[contains(concat(" ", normalize-space(@class), " "), " pm-view__value ")]', + ); + if (!values.length) return null; + return normalizeText(values[0]!.text()); + } + + private fillEmptyMessageContentFromList( + msgLink: string, + head: string | null, + date: string | null, + content: string | null, + ): Msg { + const preview = this.messageListPreviews.get(urljoin(this._baseUrl, msgLink)); + if (!preview) return [head, date, content]; + return [head || preview[0], date || preview[1], content || preview[2]]; + } + + private shouldFetchNextUnreadPage(newLinks: string[]): boolean { + if (!newLinks.length) return false; + return !this.hasReachedExpectedUnreadCount(); + } + + private hasReachedExpectedUnreadCount(): boolean { + return ( + !this.messageReadForce && + this.messageUnread > 0 && + this.messageUnread !== UNKNOWN_UNREAD_COUNT && + this.seenUnreadMessageLinks.size >= this.messageUnread + ); + } + + static isEmptyMessageContent(head: string | null, date: string | null, content: string | null): boolean { + return ![head, date, content].some((x) => x != null && String(x).trim() !== ""); + } + + static buildUnreadMailboxPage(box: number): string { + return `messages.php?action=viewmailbox&box=${box}&unread=yes`; + } + + /** 首页不能带 page 参数;之后 page=1,2,... */ + private buildNextUnreadMailboxPage(hasUnread: boolean): string | null { + if (!hasUnread) return null; + const next = this.nextMailPage; + this.nextMailPage += 1; + return `${this._userMailUnreadPage}&page=${next}`; + } + + protected override _parseUserTrafficInfo(htmlText: string): void { + super._parseUserTrafficInfo(htmlText); + this.parseUserbarInfo(htmlText); + } + + protected override _parseUserDetailInfo(htmlText: string): void { + super._parseUserDetailInfo(htmlText); + this.parseUserbarInfo(htmlText); + } + + /** 新版顶部用户栏,覆盖 NexusPHP 通用正则的误判 */ + private parseUserbarInfo(htmlText: string): void { + const html = parseHtml(htmlText); + if (!hasChildElements(html)) return; + for (const node of html.nodes("//*[@data-uploader-url or @data-uploader-stats]")) { + this.parseUserIdentity(node); + this.parseUploaderStats(node.attr("data-uploader-stats")); + } + // data-uploader-stats 不包含分享率,需从 compact metric 的 class 中读取 + this.parseCompactMetric(html, "ratio", "ratio"); + this.parseCompactMetric(html, "uploaded", "upload"); + this.parseCompactMetric(html, "downloaded", "download"); + this.parseCompactMetric(html, "bonus", "bonus"); + this.parseCompactMetric(html, "active", "active"); + } + + private parseUserIdentity(node: XNode): void { + const userUrl = node.attr("data-uploader-url") ?? ""; + const m = reSearch(/userdetails\.php\?id=(\d+)/, userUrl); + if (m?.[1]?.trim()) this.userid = m[1].trim(); + const username = node.attr("data-uploader-label"); + if (username?.trim()) this.username = username.trim(); + const level = node.attr("data-uploader-badge"); + if (level?.trim()) this.userLevel = level.trim(); + } + + private parseUploaderStats(statsText: string | null): void { + if (!statsText) return; + let stats: unknown; + try { + stats = JSON.parse(statsText); + } catch { + return; + } + if (!Array.isArray(stats)) return; + for (const item of stats) { + if (!item || typeof item !== "object" || Array.isArray(item)) continue; + const it = item as Record; + const label = String(it.label || "").replace(/^[ ：:]+|[ ：:]+$/g, ""); + const tone = String(it.tone || "").trim(); + const value = String(it.value || "").trim(); + this.setMetricValue(value, label, tone); + } + } + + private parseCompactMetric(html: XNode, metric: string, field: string): void { + let values = html.strings( + `//*[contains(concat(" ", normalize-space(@class), " "), " site-userbar__compact-metric--${metric} ")]` + + "//span[normalize-space()][last()]/text()", + ); + if (!values.length) { + values = html.strings( + `//*[contains(concat(" ", normalize-space(@class), " "), " site-userbar__compact-metric--${metric} ")]` + + "/text()", + ); + } + if (values.length) this.setMetricValue(values[values.length - 1]!.trim(), null, null, field); + } + + private setMetricValue(value: string, label?: string | null, tone?: string | null, field?: string | null): void { + if (!value) return; + const key = field || tone || label || ""; + if (["uploaded", "上传量", "upload"].includes(key)) this.upload = this.numFilesize(value); + else if (["downloaded", "下载量", "download"].includes(key)) this.download = this.numFilesize(value); + else if (["bonus", "爆米花"].includes(key)) this.bonus = parseFloatMp(value); + else if (key === "ratio") this.ratio = parseFloatMp(value); + else if (["active", "活跃"].includes(key)) { + const m = reSearch(/↑\s*(\d+)\s*\/\s*↓\s*(\d+)/, value); + if (m) { + this.seeding = parseIntMp(m[1]); + this.leeching = parseIntMp(m[2]); + } + } + } + + static parseInboxUnread(link: XNode): number | null { + for (const t of [link.attr("title"), link.attr("aria-label")]) { + const u = NexusAudiencesSiteUserInfo.extractInboxUnreadPair(t); + if (u !== null) return u; + } + for (const t of link.strings('.//*[contains(@class, "site-userbar__compact-tool-badge--unread")]/text()')) { + const u = NexusAudiencesSiteUserInfo.extractInboxUnreadBadge(t); + if (u !== null) return u; + } + if (NexusAudiencesSiteUserInfo.hasInboxUnreadMarker(link)) return UNKNOWN_UNREAD_COUNT; + return null; + } + + /** 总数/未读数格式,如 1749/172 */ + static extractInboxUnreadPair(text: string | null | undefined): number | null { + const t = normalizeText(text); + if (!t) return null; + const m = reSearch(/(?:收件箱\s*)?(\d[\d,]*)\s*\/\s*(\d[\d,]*)/, t); + return m ? parseIntMp(m[2]) : null; + } + + static extractInboxUnreadBadge(text: string | null | undefined): number | null { + const pair = NexusAudiencesSiteUserInfo.extractInboxUnreadPair(text); + if (pair !== null) return pair; + const t = normalizeText(text); + if (!t) return null; + const m = /^(\d[\d,]*)$/.exec(t); + return m ? parseIntMp(m[1]) : null; + } + + static hasInboxUnreadMarker(link: XNode): boolean { + if ((link.attr("class") ?? "").includes("site-userbar__compact-tool--has-unread")) return true; + return link.nodes('.//*[contains(@class, "site-userbar__compact-tool-badge--unread")]').length > 0; + } + + /** 做种页只读 Total 汇总行 */ + protected override async _parseSeedingPages(): Promise { + if (!this._torrentSeedingPage) return; + this._torrentSeedingHeaders = { Referer: urljoin(this._baseUrl, this._userDetailPage ?? "") }; + const htmlText = await this._getPageContent( + urljoin(this._baseUrl, this._torrentSeedingPage), + this._torrentSeedingParams, + this._torrentSeedingHeaders, + ); + if (!htmlText) return; + const html = parseHtml(htmlText); + if (!hasChildElements(html)) return; + const totalRow = html.nodes('//table[@class="table table-bordered"]//tr[td[1][normalize-space()="Total"]]'); + if (!totalRow.length) return; + const count = totalRow[0]!.strings("./td[2]/text()"); + const size = totalRow[0]!.strings("./td[3]/text()"); + this.seeding = count.length ? parseIntMp(count[0]) : 0; + this.seedingSize = size.length ? this.numFilesize(size[0]!.trim()) : 0; + } +} diff --git a/packages/adapters/src/sites/user-parsers/nexus-hhanclub.test.ts b/packages/adapters/src/sites/user-parsers/nexus-hhanclub.test.ts new file mode 100644 index 00000000..28d1651b --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/nexus-hhanclub.test.ts @@ -0,0 +1,27 @@ +import { createMockFetch, HHCLUB_USER_DETAILS, HHCLUB_USER_INDEX, HHCLUB_USER_INDEX_NO_PANEL } from "@umt/testkit"; +import { describe, expect, it } from "vitest"; +import { NexusHhanclubSiteUserInfo } from "./nexus-hhanclub.ts"; +import { runParser } from "./testing.ts"; + +describe("NexusHhanclubSiteUserInfo(MP nexus_hhanclub.py)", () => { + it("#user-info-panel 取上传/下载/分享率,详情取加入日期与 _Name 等级", async () => { + const m = createMockFetch({ "/": { body: HHCLUB_USER_INDEX }, "/userdetails.php": { body: HHCLUB_USER_DETAILS } }); + const info = await runParser(NexusHhanclubSiteUserInfo, { fetch: m.fetch }); + expect(info).toMatchObject({ + schema: "NexusHhanclub", + userId: "4567", + username: "hana", + upload: Math.round(1.2 * 1024 ** 4), + download: 300 * 1024 ** 3, + ratio: 3.5, + bonus: 6789, + joinAt: "2022-02-02 02:02:02", + userLevel: "Power User", + }); + }); + + it("面板缺失(改版):与 MP 一样抛 IndexError", async () => { + const m = createMockFetch({ "/": { body: HHCLUB_USER_INDEX_NO_PANEL } }); + await expect(runParser(NexusHhanclubSiteUserInfo, { fetch: m.fetch })).rejects.toThrow(/IndexError/); + }); +}); diff --git a/packages/adapters/src/sites/user-parsers/nexus-hhanclub.ts b/packages/adapters/src/sites/user-parsers/nexus-hhanclub.ts new file mode 100644 index 00000000..4ff3ecfc --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/nexus-hhanclub.ts @@ -0,0 +1,65 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/modules/indexer/parser/nexus_hhanclub.py @ e6820a8 + * + * 憨憨(HHClub)站点用户信息解析器,继承 NexusPHP 通用解析。 + * + * 函数对照(`NexusHhanclubSiteUserInfo`):`_parse_user_traffic_info` → `_parseUserTrafficInfo`、 + * `_parse_user_detail_info` → `_parseUserDetailInfo`、`_get_user_level` → `_getUserLevel`。XPath 与正则逐字保留。 + * + * 与 MP 一致:`#user-info-panel` 结构缺失时 `xpath(...)[0]` 抛错并中断整站解析(交给 schema 回退)。 + */ + +import { normalizeDatetime, parseFloatMp, reSearch } from "./common.ts"; +import { hasChildElements, parseHtml, type XNode } from "./dom.ts"; +import { NexusPhpSiteUserInfo, round3 } from "./nexus-php.ts"; +import { SiteSchema, type SiteSchemaValue } from "./types.ts"; + +function first(values: string[], expr: string): string { + if (!values.length) throw new Error(`IndexError: list index out of range (${expr})`); + return values[0]!; +} + +export class NexusHhanclubSiteUserInfo extends NexusPhpSiteUserInfo { + static override schema: SiteSchemaValue | null = SiteSchema.NexusHhanclub; + + protected override _parseUserTrafficInfo(htmlText: string): void { + super._parseUserTrafficInfo(htmlText); + const text = this._prepareHtmlText(htmlText); + const html = parseHtml(text); + if (!html) throw new Error("AttributeError: 'NoneType' object has no attribute 'xpath'"); + const x = (e: string) => first(html.strings(e), e); + // 上传、下载、分享率 + const uploadMatch = reSearch( + /[_<>/a-zA-Z-="'\s#;]+([\d,.\s]+[KMGTPI]*B)/, + x('//*[@id="user-info-panel"]/div[2]/div[2]/div[4]/text()'), + ); + const downloadMatch = reSearch( + /[_<>/a-zA-Z-="'\s#;]+([\d,.\s]+[KMGTPI]*B)/, + x('//*[@id="user-info-panel"]/div[2]/div[2]/div[5]/text()'), + ); + const ratioMatch = reSearch( + /分享率\][:：_<>/a-zA-Z-="'\s#;]+([\d,.\s]+)/, + x('//*[@id="user-info-panel"]/div[2]/div[1]/div[1]/div/text()'), + ); + this.upload = uploadMatch ? this.numFilesize(uploadMatch[1]!.trim()) : 0; + this.download = downloadMatch ? this.numFilesize(downloadMatch[1]!.trim()) : 0; + const calcRatio = this.download <= 0.0 ? 0.0 : round3(this.upload / this.download); + this.ratio = ratioMatch?.[1]?.trim() ? parseFloatMp(ratioMatch[1]) : calcRatio; + } + + protected override _parseUserDetailInfo(htmlText: string): void { + super._parseUserDetailInfo(htmlText); + const html = parseHtml(htmlText); + if (!hasChildElements(html)) return; + const joinAt = html.strings('//span[contains(text(), "加入日期")]/following-sibling::span/span/@title'); + if (joinAt.length) this.joinAt = normalizeDatetime(joinAt[0]!.trim()) ?? null; + } + + protected override _getUserLevel(html: XNode): void { + super._getUserLevel(html); + const level = html.strings('//b[contains(@class, "_Name")]/text()'); + if (level.length) this.userLevel = level[0]!; + } +} diff --git a/packages/adapters/src/sites/user-parsers/nexus-php.test.ts b/packages/adapters/src/sites/user-parsers/nexus-php.test.ts new file mode 100644 index 00000000..00806dda --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/nexus-php.test.ts @@ -0,0 +1,71 @@ +import { + createMockFetch, + NEXUS_PHP_CF_CHALLENGE, + NEXUS_PHP_INDEX, + NEXUS_PHP_LOGIN_PAGE, + NEXUS_PHP_SEEDING, + NEXUS_PHP_USERDETAILS, +} from "@umt/testkit"; +import { describe, expect, it } from "vitest"; +import { NexusPhpSiteUserInfo } from "./nexus-php.ts"; +import { runParser } from "./testing.ts"; + +describe("NexusPhpSiteUserInfo(MP nexus_php.py)", () => { + it("首页 + 详情 + 做种页:流量/魔力/等级/加入日期/做种统计", async () => { + const m = createMockFetch({ + "/": { body: NEXUS_PHP_INDEX }, + "/userdetails.php": { body: NEXUS_PHP_USERDETAILS }, + "/getusertorrentlistajax.php": { body: NEXUS_PHP_SEEDING }, + }); + const info = await runParser(NexusPhpSiteUserInfo, { fetch: m.fetch, cookie: "uid=1; pass=secret" }); + expect(info).toMatchObject({ + schema: "NexusPhp", + domain: "example.com", + userId: "10086", + username: "alice", + userLevel: "Veteran User", + joinAt: "2021-03-04 05:06:07", + upload: Math.round(1.5 * 1024 ** 4), + download: Math.round(477.3 * 1024 ** 3), + ratio: 3.217, + bonus: 12345.6, + leeching: 2, + seeding: 2, + seedingSize: 10 * 1024 ** 3 + Math.round(1.5 * 1024 ** 4), + seedingInfo: [ + [5, 10 * 1024 ** 3], + [12, Math.round(1.5 * 1024 ** 4)], + ], + messageUnread: 3, + errMsg: null, + }); + // 请求顺序:首页 → 用户详情 → 做种(流量页已合并到首页解析) + expect(m.requests.map((r) => r.url.pathname)).toEqual(["/", "/userdetails.php", "/getusertorrentlistajax.php"]); + expect(m.requests[2]!.url.search).toBe("?userid=10086&type=seeding"); + expect(m.requests[0]!.headers.cookie).toBe("uid=1; pass=secret"); + expect(JSON.stringify(info)).not.toContain("secret"); + }); + + it("登录页(含密码框):不再继续请求,errMsg 提示 cookie 过期", async () => { + const m = createMockFetch({ "/": { body: NEXUS_PHP_LOGIN_PAGE } }); + const info = await runParser(NexusPhpSiteUserInfo, { fetch: m.fetch }); + expect(info.userId).toBeNull(); + expect(info.errMsg).toContain("未检测到已登陆"); + expect(m.requests).toHaveLength(1); + }); + + it("Cloudflare 挑战页按空内容处理并记录原因", async () => { + const m = createMockFetch({ "/": { body: NEXUS_PHP_CF_CHALLENGE, status: 403 } }); + const info = await runParser(NexusPhpSiteUserInfo, { fetch: m.fetch }); + expect(info.errMsg).toContain("未检测到已登陆"); + expect(info.raw.http_errors).toEqual([expect.stringContaining("Cloudflare")]); + }); + + it("_fixup_next_page_url:补 userid/type,已有 userid 原样,无 userid 停止", () => { + expect(NexusPhpSiteUserInfo.fixupNextPageUrl("getusertorrentlistajax.php?page=1", "7")).toBe( + "getusertorrentlistajax.php?page=1&userid=7&type=seeding", + ); + expect(NexusPhpSiteUserInfo.fixupNextPageUrl("x.php?userid=3&page=2", "7")).toBe("x.php?userid=3&page=2"); + expect(NexusPhpSiteUserInfo.fixupNextPageUrl("x.php?page=2", null)).toBeNull(); + }); +}); diff --git a/packages/adapters/src/sites/user-parsers/nexus-php.ts b/packages/adapters/src/sites/user-parsers/nexus-php.ts new file mode 100644 index 00000000..a1437d78 --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/nexus-php.ts @@ -0,0 +1,361 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/modules/indexer/parser/nexus_php.py @ e6820a8 + * + * NexusPHP 站点用户信息解析器。 + * + * 迁移自 MoviePilot e6820a8(GPLv3)`app/modules/indexer/parser/nexus_php.py` `NexusPhpSiteUserInfo`, + * 函数对照:`_parse_site_page` → `_parseSitePage`、`_parse_message_unread` → `_parseMessageUnread`、 + * `_parse_user_base_info` → `_parseUserBaseInfo`、`_parse_user_traffic_info` → `_parseUserTrafficInfo`、 + * `_parse_ucoin` → `parseUcoin`、`_parse_user_torrent_seeding_info` → `_parseUserTorrentSeedingInfo`、 + * `_fixup_next_page_url` → `fixupNextPageUrl`、`_parse_user_detail_info` → `_parseUserDetailInfo`、 + * `_fixup_torrent_seeding_page` → `_fixupTorrentSeedingPage`、`_get_user_level` → `_getUserLevel`、 + * `_parse_message_unread_links` → `_parseMessageUnreadLinks`、`_parse_message_content` → `_parseMessageContent`、 + * `_fixup_traffic_info` → `_fixupTrafficInfo`。XPath 与正则逐字保留。 + * + * 差异:做种表"大小列/做种人数列"条数不一致时 MP 抛 IndexError 中断整站解析,这里缺失的做种人数记 0。 + */ +import { type NextPage, SiteParserBase } from "./base.ts"; +import { normalizeDatetime, parseFloatMp, parseIntMp, reFindall, reSearch } from "./common.ts"; +import { hasChildElements, parseHtml, type XNode } from "./dom.ts"; +import { SiteSchema } from "./types.ts"; + +/** Python `round(x, 3)` 的近似 */ +export function round3(x: number): number { + return Math.round(x * 1000) / 1000; +} + +export class NexusPhpSiteUserInfo extends SiteParserBase { + static override schema: import("./types.ts").SiteSchemaValue | null = SiteSchema.NexusPhp; + + protected _parseSitePage(htmlText: string): void { + const text = this._prepareHtmlText(htmlText); + const userDetail = reSearch(/userdetails.php\?id=(\d+)/, text); + if (userDetail?.[0].trim()) { + this._userDetailPage = userDetail[0].trim().replace(/^\/+/, ""); + this.userid = userDetail[1]!; + this._torrentSeedingPage = `getusertorrentlistajax.php?userid=${this.userid}&type=seeding`; + } else { + const m = reSearch(/(userdetails)/, text); + if (m?.[0].trim()) { + this._userDetailPage = m[0].trim().replace(/^\/+/, ""); + this.userid = null; + this._torrentSeedingPage = null; + } + } + } + + /** 解析未读短消息数量 */ + protected _parseMessageUnread(htmlText: string): void { + const html = parseHtml(htmlText); + if (!hasChildElements(html)) return; + const messageLabels = html.nodes('//a[@href="messages.php"]/..'); + messageLabels.push(...html.nodes('//a[contains(@href, "messages.php")]/..')); + if (messageLabels.length) { + const messageText = messageLabels[0]!.text(); + const matches = reFindall(/[^Date](信息箱\s*|\((?![^)]*:)|你有 )(\d+)/, messageText); + if (matches.length && matches[matches.length - 1]!.length === 2) { + this.messageUnread = parseIntMp(matches[matches.length - 1]![1]); + } else if (/^\d+$/.test(messageText)) { + this.messageUnread = parseIntMp(messageText); + } + } + } + + protected _parseUserBaseInfo(htmlText: string): void { + // 合并解析,减少额外请求调用 + this._parseUserTrafficInfo(htmlText); + this._userTrafficPage = null; + this._parseMessageUnread(htmlText); + + const html = parseHtml(htmlText); + if (!hasChildElements(html)) return; + let ret = html.strings(`//a[contains(@href, "userdetails") and contains(@href, "${this.userid}")]//b//text()`); + if (ret.length) { + this.username = String(ret[0]); + return; + } + ret = html.strings(`//a[contains(@href, "userdetails") and contains(@href, "${this.userid}")]//text()`); + if (ret.length) this.username = String(ret[0]); + ret = html.strings('//a[contains(@href, "userdetails")]//strong//text()'); + if (ret.length) this.username = String(ret[0]); + } + + protected _parseUserTrafficInfo(htmlText: string): void { + const text = this._prepareHtmlText(htmlText); + const uploadMatch = reSearch(/[^总]上[传傳]量?[:：_<>/a-zA-Z-="'\s#;]+([\d,.\s]+[KMGTPI]*B)/i, text); + this.upload = uploadMatch ? this.numFilesize(uploadMatch[1]!.trim()) : 0; + const downloadMatch = reSearch(/[^总子影力]下[载載]量?[:：_<>/a-zA-Z-="'\s#;]+([\d,.\s]+[KMGTPI]*B)/i, text); + this.download = downloadMatch ? this.numFilesize(downloadMatch[1]!.trim()) : 0; + const ratioMatch = reSearch(/分享率[:：_<>/a-zA-Z-="'\s#;]+([\d,.\s]+)/, text); + const calcRatio = this.download <= 0.0 ? 0.0 : round3(this.upload / this.download); + this.ratio = ratioMatch && ratioMatch[1]!.trim() ? parseFloatMp(ratioMatch[1]) : calcRatio; + const leechingMatch = reSearch(/(Torrents leeching|下载中)[一-龥\D\s]+(\d+)[\s\S]+/a-zA-Z_\-="'\s#;.(使用魔力值豆]+\s*([\d,.]+)[<()&\s]/, text); + if (bonusMatch && bonusMatch[1]!.trim()) { + this.bonus = parseFloatMp(bonusMatch[1]); + return; + } + bonusMatch = reSearch(/[魔力值|\]][[\]:：<>/a-zA-Z_\-="'\s#;]+\s*([\d,.]+|"[\d,.]+")[<>()&\s]/s, text); + if (bonusMatch && bonusMatch[1]!.trim()) { + this.bonus = parseFloatMp(bonusMatch[1]!.replace(/^"+|"+$/g, "")); + } + } + + /** MP `_parse_ucoin`:金/银/铜币统一换算为铜币 */ + static parseUcoin(html: XNode | null): [boolean, number] { + if (hasChildElements(html)) { + let gold: number | null = null; + let silver: number | null = null; + let copper: number | null = null; + const golds = html.strings('//span[@class = "ucoin-symbol ucoin-gold"]//text()'); + if (golds.length) gold = parseFloatMp(golds[golds.length - 1]); + const silvers = html.strings('//span[@class = "ucoin-symbol ucoin-silver"]//text()'); + if (silvers.length) silver = parseFloatMp(silvers[silvers.length - 1]); + const coppers = html.strings('//span[@class = "ucoin-symbol ucoin-copper"]//text()'); + if (coppers.length) copper = parseFloatMp(coppers[coppers.length - 1]); + if (gold || silver || copper) { + return [true, (gold || 0) * 100 * 100 + (silver || 0) * 100 + (copper || 0)]; + } + } + return [false, 0.0]; + } + + protected _parseUserTorrentSeedingInfo(htmlText: string, multiPage = false): NextPage { + const html = parseHtml(String(htmlText ?? "").replaceAll("\\/", "/")); + if (!hasChildElements(html)) return null; + + // 首页存在扩展链接,使用扩展链接 + const seedingUrlText = html.strings('//a[contains(@href,"torrents.php") and contains(@href,"seeding")]/@href'); + if (multiPage === false && seedingUrlText.length && seedingUrlText[0]!.trim()) { + this._torrentSeedingPage = seedingUrlText[0]!.trim(); + return this._torrentSeedingPage; + } + + let sizeCol = 3; + let seedersCol = 4; + const sizeColXpath = + '//tr[position()=1]/td[(img[@class="size"] and img[@alt="size"]) or (text() = "大小") or (a/img[@class="size" and @alt="size"])]'; + if (html.nodes(sizeColXpath).length) { + sizeCol = html.nodes(`${sizeColXpath}/preceding-sibling::td`).length + 1; + } + const seedersColXpath = + '//tr[position()=1]/td[(img[@class="seeders"] and img[@alt="seeders"]) or (text() = "在做种") or (a/img[@class="seeders" and @alt="seeders"])]'; + if (html.nodes(seedersColXpath).length) { + seedersCol = html.nodes(`${seedersColXpath}/preceding-sibling::td`).length + 1; + } + + let pageSeeding = 0; + let pageSeedingSize = 0; + const pageSeedingInfo: Array<[number, number]> = []; + const tableClass = html.nodes('//table[@class="torrents"]').length ? '//table[@class="torrents"]' : ""; + const seedingSizes = html.nodes(`${tableClass}//tr[position()>1]/td[${sizeCol}]`); + let seedingSeeders = html.strings(`${tableClass}//tr[position()>1]/td[${seedersCol}]/b/a/text()`); + if (!seedingSeeders.length) + seedingSeeders = html.strings(`${tableClass}//tr[position()>1]/td[${seedersCol}]//text()`); + if (seedingSizes.length && seedingSeeders.length) { + pageSeeding = seedingSizes.length; + for (let i = 0; i < seedingSizes.length; i++) { + const size = this.numFilesize(seedingSizes[i]!.text().trim()); + const seeders = parseIntMp(seedingSeeders[i]); + pageSeedingSize += size; + pageSeedingInfo.push([seeders, size]); + } + } + this.seeding += pageSeeding; + this.seedingSize += pageSeedingSize; + this.seedingInfo.push(...pageSeedingInfo); + + // 是否存在下页数据 + let nextPage: string | null = null; + const nextPageText = html.strings( + '//a[contains(.//text(), "下一页") or contains(.//text(), "下一頁") or contains(.//text(), ">")]/@href', + ); + // 防止识别到详情页 + while (nextPageText.length) { + nextPage = nextPageText.pop()!.trim(); + if (!nextPage.startsWith("details.php")) break; + nextPage = null; + } + if (nextPage) nextPage = NexusPhpSiteUserInfo.fixupNextPageUrl(nextPage, this.userid); + return nextPage; + } + + /** MP `_fixup_next_page_url`:补齐 userid/type,无法补齐时停止翻页 */ + static fixupNextPageUrl(nextPage: string, userid: string | null): string | null { + const hashIdx = nextPage.indexOf("#"); + const fragment = hashIdx >= 0 ? nextPage.slice(hashIdx) : ""; + const noFrag = hashIdx >= 0 ? nextPage.slice(0, hashIdx) : nextPage; + const qIdx = noFrag.indexOf("?"); + const path = qIdx >= 0 ? noFrag.slice(0, qIdx) : noFrag; + const query = qIdx >= 0 ? noFrag.slice(qIdx + 1) : ""; + // parse_qsl → dict:同名键后者覆盖,保留插入顺序 + const params = new Map(); + for (const [k, v] of new URLSearchParams(query)) params.set(k, v); + if (params.get("userid")) return nextPage; + if (!userid) return null; + params.set("userid", userid); + if (!params.has("type")) params.set("type", "seeding"); + const qs = new URLSearchParams([...params.entries()]).toString(); + return `${path}${qs ? `?${qs}` : ""}${fragment}`; + } + + protected _parseUserDetailInfo(htmlText: string): void { + const html = parseHtml(htmlText); + if (!hasChildElements(html)) return; + this._getUserLevel(html); + this._fixupTrafficInfo(html); + + // 加入日期 + const joinAtText = html.strings( + '//tr/td[text()="加入日期" or text()="注册日期" or *[text()="加入日期"]]/following-sibling::td[1]//text()' + + '|//div/b[text()="加入日期"]/../text()' + + '|//*[@id="outer"]/table/tr/td/div/div[1]/div[2]/div[3]/span[1]/span/@title', + ); + if (joinAtText.length) { + this.joinAt = normalizeDatetime(joinAtText[0]!.split(" (")[0]!.trim()) ?? null; + } + + // 做种体积 & 做种数(seeding 页面获取不到的话,此处再获取一次) + const seedingSizes = html.nodes( + '//tr/td[text()="当前上传"]/following-sibling::td[1]//table[tr[1][td[4 and text()="尺寸"]]]//tr[position()>1]/td[4]', + ); + const seedingSeeders = html.strings( + '//tr/td[text()="当前上传"]/following-sibling::td[1]//table[tr[1][td[5 and text()="做种者"]]]//tr[position()>1]/td[5]//text()', + ); + let tmpSeeding = seedingSizes.length; + let tmpSeedingSize = 0; + const tmpSeedingInfo: Array<[number, number]> = []; + for (let i = 0; i < seedingSizes.length; i++) { + const size = this.numFilesize(seedingSizes[i]!.text().trim()); + const seeders = parseIntMp(seedingSeeders[i]); + tmpSeedingSize += size; + tmpSeedingInfo.push([seeders, size]); + } + if (!this.seedingSize) this.seedingSize = tmpSeedingSize; + if (!this.seeding) this.seeding = tmpSeeding; + if (!this.seedingInfo.length) this.seedingInfo = tmpSeedingInfo; + + const stat = html.strings('//tr/td[text()="做种统计"]/following-sibling::td[1]//text()'); + if (stat.length) { + const seedingMatch = reSearch(/总做种数:\s+(\d+)/i, stat[0]); + const seedingSizeMatch = reSearch(/总做种体积:\s+([\d,.\s]+[KMGTPI]*B)/i, stat[0]); + tmpSeeding = seedingMatch?.[1] ? parseIntMp(seedingMatch[1]) : 0; + tmpSeedingSize = seedingSizeMatch ? this.numFilesize(seedingSizeMatch[1]!.trim()) : 0; + } + if (!this.seedingSize) this.seedingSize = tmpSeedingSize; + if (!this.seeding) this.seeding = tmpSeeding; + + this._fixupTorrentSeedingPage(html); + } + + /** MP `_fixup_torrent_seeding_page` */ + protected _fixupTorrentSeedingPage(html: XNode): void { + let seedingUrlText = html.strings( + '//a[contains(@href,"getusertorrentlist.php") and contains(@href,"seeding")]/@href', + ); + if (seedingUrlText.length) this._torrentSeedingPage = seedingUrlText[0]!.trim(); + // 从JS调用种获取用户ID + seedingUrlText = html.strings( + '//a[contains(@href, "javascript: getusertorrentlistajax") and contains(@href,"seeding")]/@href', + ); + const csrfText = html.strings('//meta[@name="x-csrf"]/@content'); + if (!this._torrentSeedingPage && seedingUrlText.length) { + const userJs = reSearch(/javascript: getusertorrentlistajax\(\s*'(\d+)/, seedingUrlText[0]); + if (userJs?.[1]?.trim()) { + this.userid = userJs[1].trim(); + this._torrentSeedingPage = `getusertorrentlistajax.php?userid=${this.userid}&type=seeding`; + } + } else if (seedingUrlText.length && csrfText.length) { + if (csrfText[0]!.trim()) { + this._torrentSeedingPage = "ajax_getusertorrentlist.php"; + this._torrentSeedingParams = { userid: this.userid, type: "seeding", csrf: csrfText[0]!.trim() }; + } + } + } + + /** MP `_get_user_level`:图片等级取 title,否则取文本 */ + protected _getUserLevel(html: XNode): void { + let levels = html.strings( + '//tr/td[text()="等級" or text()="等级" or *[text()="等级"]]/following-sibling::td[1]/img[1]/@title', + ); + if (levels.length) { + this.userLevel = levels[0]!.trim(); + return; + } + let nodes = html.nodes( + '//tr/td[text()="等級" or text()="等级"]/following-sibling::td[1 and not(img)]' + + '|//tr/td[text()="等級" or text()="等级"]/following-sibling::td[1 and img[not(@title)]]', + ); + if (nodes.length) { + this.userLevel = nodes[0]!.text().trim(); + return; + } + nodes = html.nodes('//tr/td[text()="等級" or text()="等级"]/following-sibling::td[1]'); + if (nodes.length) { + this.userLevel = nodes[0]!.text().trim(); + return; + } + levels = html.strings('//a[contains(@href, "userdetails")]/text()'); + if (!this.userLevel && levels.length) { + for (const t of levels) { + const m = reSearch(/\[(.*)]/, t); + if (m?.[1]?.trim()) { + this.userLevel = m[1].trim(); + break; + } + } + } + } + + protected _parseMessageUnreadLinks(htmlText: string, msgLinks: string[]): NextPage { + const html = parseHtml(htmlText); + if (!hasChildElements(html)) return null; + msgLinks.push(...html.strings('//tr[not(./td/img[@alt="Read"])]/td/a[contains(@href, "viewmessage")]/@href')); + const nextPageText = html.strings('//a[contains(.//text(), "下一页") or contains(.//text(), "下一頁")]/@href'); + return nextPageText.length ? nextPageText[nextPageText.length - 1]!.trim() : null; + } + + protected _parseMessageContent(htmlText: string): [string | null, string | null, string | null] { + const html = parseHtml(htmlText); + if (!hasChildElements(html)) return [null, null, null]; + let head: string | null = null; + const messageHead = html.strings('//h1/text()|//div[@class="layui-card-header"]/span[1]/text()'); + if (messageHead.length) head = messageHead[messageHead.length - 1]!.trim(); + let date: string | null = null; + const messageDate = html.nodes( + '//h1/following-sibling::table[.//tr/td[@class="colhead"]]//tr[2]/td[2]|//div[@class="layui-card-header"]/span[2]/span[2]', + ); + if (messageDate.length) date = messageDate[0]!.text().trim(); + let content: string | null = null; + const messageContent = html.nodes( + '//h1/following-sibling::table[.//tr/td[@class="colhead"]]//tr[3]/td|//div[contains(@class,"layui-card-body")]', + ); + if (messageContent.length) content = messageContent[0]!.text().trim(); + return [head, date, content]; + } + + /** MP `_fixup_traffic_info` */ + protected _fixupTrafficInfo(html: XNode): void { + if (!this.bonus) { + const bonusText = html.strings('//tr/td[text()="魔力值" or text()="猫粮"]/following-sibling::td[1]/text()'); + if (bonusText.length) this.bonus = parseFloatMp(bonusText[0]!.trim()); + } + } +} diff --git a/packages/adapters/src/sites/user-parsers/nexus-project.test.ts b/packages/adapters/src/sites/user-parsers/nexus-project.test.ts new file mode 100644 index 00000000..4ac676f1 --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/nexus-project.test.ts @@ -0,0 +1,30 @@ +import { createMockFetch, NEXUS_PHP_INDEX, NEXUS_PROJECT_USER_INDEX } from "@umt/testkit"; +import { describe, expect, it } from "vitest"; +import { round3 } from "./nexus-php.ts"; +import { NexusProjectSiteUserInfo } from "./nexus-project.ts"; +import { runParser } from "./testing.ts"; + +describe("NexusProjectSiteUserInfo(MP nexus_project.py)", () => { + it("user_detail.php?uid= 变种 + span#user_info 无 B 后缀流量", async () => { + const m = createMockFetch({ "/": { body: NEXUS_PROJECT_USER_INDEX } }); + const info = await runParser(NexusProjectSiteUserInfo, { fetch: m.fetch }); + const up = Math.round(445.11 * 1024 ** 3); + const down = Math.round(29.61 * 1024 ** 3); + expect(info).toMatchObject({ + schema: "NexusProject", + userId: "77", + username: "carol", + upload: up, + download: down, + ratio: round3(up / down), + }); + expect(m.requests.map((r) => r.url.pathname + r.url.search)).toContain("/viewusertorrents.php?id=77&show=seeding"); + expect(m.requests.map((r) => r.url.pathname)).toContain("/user_detail.php"); // 与 MP 一致:正则只取 user_detail.php?uid=,不含 p_user/ 前缀 + }); + + it("标准 NexusPHP 首页:回落通用流量解析", async () => { + const m = createMockFetch({ "/": { body: NEXUS_PHP_INDEX } }); + const info = await runParser(NexusProjectSiteUserInfo, { fetch: m.fetch }); + expect(info).toMatchObject({ userId: "10086", upload: Math.round(1.5 * 1024 ** 4), ratio: 3.217 }); + }); +}); diff --git a/packages/adapters/src/sites/user-parsers/nexus-project.ts b/packages/adapters/src/sites/user-parsers/nexus-project.ts new file mode 100644 index 00000000..bb6e83a2 --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/nexus-project.ts @@ -0,0 +1,58 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/modules/indexer/parser/nexus_project.py @ e6820a8 + * + * NexusProject 站点用户信息解析器,继承 NexusPHP 通用解析。 + * + * 函数对照(`NexusProjectSiteUserInfo`):`_parse_site_page` → `_parseSitePage`(兼容 `user_detail.php?uid=` 变种)、 + * `_parse_user_traffic_info` → `_parseUserTrafficInfo`(兼容 `span#user_info` 无 B 后缀的流量文本)。正则逐字保留。 + */ +import { reSearch } from "./common.ts"; +import { parseHtml } from "./dom.ts"; +import { NexusPhpSiteUserInfo, round3 } from "./nexus-php.ts"; +import { SiteSchema, type SiteSchemaValue } from "./types.ts"; + +export class NexusProjectSiteUserInfo extends NexusPhpSiteUserInfo { + static override schema: SiteSchemaValue | null = SiteSchema.NexusProject; + + protected override _parseSitePage(htmlText: string): void { + const text = this._prepareHtmlText(htmlText); + let userDetail = reSearch(/userdetails\.php\?id=(\d+)/, text); + if (userDetail?.[0].trim()) { + this._userDetailPage = userDetail[0].trim().replace(/^\/+/, ""); + this.userid = userDetail[1]!; + } else { + // 兼容部分 NexusProject 变种站点(如 star-space.net)的 p_user/user_detail.php?uid= 用户定位格式 + userDetail = reSearch(/user_detail\.php\?uid=(\d+)/, text); + if (userDetail?.[0].trim()) { + this._userDetailPage = userDetail[0].trim().replace(/^\/+/, ""); + this.userid = userDetail[1]!; + const uname = reSearch(/user_detail\.php\?uid=\d+[^>]*?>\s*<[^>]*>([^<]+) { + it("首页奶糖余额 + 详情 layui-hares + api/general JSON 做种", async () => { + const m = createMockFetch({ + "/": { body: RABBIT_USER_INDEX }, + "/user.php": { body: RABBIT_USER_DETAILS }, + "POST /api/general": { body: RABBIT_USER_SEEDING_JSON }, + }); + const info = await runParser(NexusRabbitSiteUserInfo, { fetch: m.fetch }); + expect(info).toMatchObject({ + schema: "NexusRabbit", + userId: "5", + username: "dave", + userLevel: "Power User", + joinAt: "2022-01-02 03:04:05", + upload: Math.round(1.5 * 1024 ** 4), + download: 500 * 1024 ** 3, + ratio: 3.072, + bonus: 1024.5, + seeding: 2, + seedingSize: Math.round(3.5 * 1024 ** 3), + seedingInfo: [ + [3, 1024 ** 3], + [7, Math.round(2.5 * 1024 ** 3)], + ], + }); + const api = m.requests.find((r) => r.url.pathname === "/api/general")!; + expect(api.headers["x-requested-with"]).toBe("XMLHttpRequest"); + expect(JSON.parse(api.body!)).toMatchObject({ action: "userTorrentsList", data: { type: "seeding", id: 5 } }); + }); + + it("未登录:errMsg 且不请求 API", async () => { + const m = createMockFetch({ "/": { body: NEXUS_PHP_LOGIN_PAGE } }); + const info = await runParser(NexusRabbitSiteUserInfo, { fetch: m.fetch }); + expect(info.errMsg).toContain("未检测到已登陆"); + expect(m.requests).toHaveLength(1); + }); +}); diff --git a/packages/adapters/src/sites/user-parsers/nexus-rabbit.ts b/packages/adapters/src/sites/user-parsers/nexus-rabbit.ts new file mode 100644 index 00000000..56f48db0 --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/nexus-rabbit.ts @@ -0,0 +1,142 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/modules/indexer/parser/nexus_rabbit.py @ e6820a8 + * + * NexusRabbit(兔子/layui-hares 模板)站点用户信息解析器。 + * + * 函数对照(`NexusRabbitSiteUserInfo`):`_parse_site_page` → `_parseSitePage`(做种/消息改走 `api/general` JSON)、 + * `_parse_user_torrent_seeding_info` → `_parseUserTorrentSeedingInfo`、`_parse_message_unread_links` → + * `_parseMessageUnreadLinks`(直接读出消息并调用 readMessage 标记已读)、`_parse_user_base_info` → `_parseUserBaseInfo` + * (奶糖余额)、`_parse_user_detail_info` → `_parseUserDetailInfo`、`_parse_message_content` / `_parse_user_traffic_info` → 空实现。 + */ +import { type NextPage, SiteParserBase, urljoin } from "./base.ts"; +import { normalizeDatetime, parseFloatMp, pyInt, reSearch } from "./common.ts"; +import { hasChildElements, parseHtml } from "./dom.ts"; +import { SiteSchema, type SiteSchemaValue } from "./types.ts"; + +const JSON_HEADERS = { + "Content-Type": "application/json", + Accept: "application/json, text/plain, */*", + "X-Requested-With": "XMLHttpRequest", +}; + +function removePrefix(s: string, p: string): string { + return s.startsWith(p) ? s.slice(p.length) : s; +} + +export class NexusRabbitSiteUserInfo extends SiteParserBase { + static override schema: SiteSchemaValue | null = SiteSchema.NexusRabbit; + + protected _parseSitePage(htmlText: string): void { + const text = this._prepareHtmlText(htmlText); + const userDetail = reSearch(/user.php\?id=(\d+)/, text); + if (!userDetail?.[0].trim()) return; + this.userid = userDetail[1]!; + this._userDetailPage = `user.php?id=${this.userid}`; + this._userTrafficPage = null; + this._torrentSeedingPage = "api/general"; + this._torrentSeedingParams = { + page: 1, + limit: 5000000, + action: "userTorrentsList", + data: { type: "seeding", id: pyInt(this.userid) }, + }; + // X-Requested-With 必须要加上,不然返回的是空数据 + this._torrentSeedingHeaders = { ...JSON_HEADERS }; + this._userMailUnreadPage = null; + this._sysMailUnreadPage = "api/general"; + this._mailUnreadParams = { page: 1, limit: 5000000, action: "getMessageIn" }; + this._mailUnreadHeaders = { ...JSON_HEADERS }; + } + + protected _parseUserTorrentSeedingInfo(htmlText: string): NextPage { + let torrents: Array>; + try { + const data = (JSON.parse(htmlText) as { data?: unknown }).data; + torrents = (data ?? []) as Array>; + } catch { + return null; + } + let seedingSize = 0; + const seedingInfo: Array<[number, number]> = []; + for (const t of torrents) { + const seeders = pyInt(t.seeders ?? 0); + const size = this.numFilesize(t.size as string); + seedingSize += size; + seedingInfo.push([seeders, size]); + } + this.seeding = torrents.length; + this.seedingSize = seedingSize; + this.seedingInfo = seedingInfo; + return null; + } + + protected async _parseMessageUnreadLinks(htmlText: string, _msgLinks: string[]): Promise { + const unreadIds: unknown[] = []; + let messages: Array>; + try { + messages = ((JSON.parse(htmlText) as { data?: unknown }).data ?? []) as Array>; + } catch { + return null; + } + for (const msg of messages) { + const id = msg.id; + const unread = msg.unread; + if (!(id && unread) || unread === "no") continue; + unreadIds.push(id); + const head = msg.subject as string | null; + const date = msg.added as string | null; + const content = msg.msg as string | null; + if (head && date && content) this.messageUnreadContents.push([head, date, content]); + } + this.messageUnread = unreadIds.length; + if (unreadIds.length) { + await this._getPageContent( + urljoin(this._baseUrl, "api/general?loading=true"), + { action: "readMessage", data: { ids: unreadIds } }, + { ...JSON_HEADERS }, + ); + } + return null; + } + + /** 只有奶糖余额需要在 base 中获取,其它均可以在详情页拿到 */ + protected _parseUserBaseInfo(htmlText: string): void { + const html = parseHtml(htmlText); + if (!hasChildElements(html)) return; + const bonus = html.strings('//div[contains(text(), "奶糖余额")]/following-sibling::div[1]/text()'); + if (bonus.length) this.bonus = parseFloatMp(bonus[0]!.trim()); + } + + protected _parseUserDetailInfo(htmlText: string): void { + const html = parseHtml(htmlText); + if (!hasChildElements(html)) return; + const infos = html.nodes('//div[contains(@class, "layui-hares-user-info-right")]'); + if (!infos.length) return; + const info = infos[0]!; + const username = info.strings('.//span[contains(text(), "用户名")]/a/span/text()'); + if (username.length) this.username = username[0]!.trim(); + const level = info.strings('.//span[contains(text(), "等级")]/b/text()'); + if (level.length) this.userLevel = level[0]!.trim(); + const joinDate = info.strings('.//span[contains(text(), "注册日期")]/text()'); + if (joinDate.length) { + const d = removePrefix(joinDate[0]!.trim().split("\r")[0]!, "注册日期："); + this.joinAt = normalizeDatetime(d) ?? null; + } + const upload = info.strings('.//span[contains(text(), "上传量")]/text()'); + if (upload.length) this.upload = this.numFilesize(removePrefix(upload[0]!.trim(), "上传量：")); + const download = info.strings('.//span[contains(text(), "下载量")]/text()'); + if (download.length) this.download = this.numFilesize(removePrefix(download[0]!.trim(), "下载量：")); + const ratio = info.strings('.//span[contains(text(), "分享率")]/em/text()'); + if (ratio.length) this.ratio = parseFloatMp(ratio[0]!.trim()); + } + + /** 已在 _parseMessageUnreadLinks 内实现 */ + protected _parseMessageContent(_htmlText: string): [string | null, string | null, string | null] { + return [null, null, null]; + } + + /** 已在 _parseUserDetailInfo 内实现 */ + protected _parseUserTrafficInfo(_htmlText: string): void {} +} diff --git a/packages/adapters/src/sites/user-parsers/registry.test.ts b/packages/adapters/src/sites/user-parsers/registry.test.ts new file mode 100644 index 00000000..a345c620 --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/registry.test.ts @@ -0,0 +1,50 @@ +import { createMockFetch, NEXUS_PHP_INDEX, NEXUS_PHP_LOGIN_PAGE, NEXUS_PHP_USERDETAILS } from "@umt/testkit"; +import { describe, expect, it } from "vitest"; +import { getUserParserClass, refreshSiteUserData, USER_PARSER_CLASSES } from "./registry.ts"; +import { SiteSchema } from "./types.ts"; + +describe("用户信息解析器注册表(MP refresh_userdata)", () => { + it("覆盖 MP SiteSchema 全部 20 个值,且一一对应", () => { + const schemas = USER_PARSER_CLASSES.map((c) => c.schema); + expect(new Set(schemas)).toEqual(new Set(Object.values(SiteSchema))); + expect(schemas).toHaveLength(20); + expect(getUserParserClass("Small Horse")?.name).toBe("SmallHorseSiteUserInfo"); + expect(getUserParserClass("nope")).toBeUndefined(); + }); + + it("未知 schema 返回 null", async () => { + const m = createMockFetch({}); + expect( + await refreshSiteUserData({ name: "x", url: "https://x.example.com/", schema: "Nope", fetch: m.fetch }), + ).toBeNull(); + expect(m.requests).toHaveLength(0); + }); + + it("声明 schema 解析不出 userid 时按序回退到能解析的 schema(Unit3d 标错 → NexusPhp)", async () => { + const m = createMockFetch({ + "/": { body: NEXUS_PHP_INDEX }, + "/userdetails.php": { body: NEXUS_PHP_USERDETAILS }, + }); + const info = await refreshSiteUserData({ + name: "demo", + url: "https://pt.example.com/", + schema: SiteSchema.Unit3d, + cookie: "c=1", + fetch: m.fetch, + }); + expect(info).toMatchObject({ userId: "10086", username: "alice", domain: "example.com" }); + expect(info!.schema).not.toBe("Unit3d"); + }); + + it("全部回退失败:返回声明 schema 的结果与 errMsg", async () => { + const m = createMockFetch({ "/": { body: NEXUS_PHP_LOGIN_PAGE } }); + const info = await refreshSiteUserData({ + name: "demo", + url: "https://pt.example.com/", + schema: SiteSchema.NexusPhp, + fetch: m.fetch, + }); + expect(info).toMatchObject({ schema: "NexusPhp", userId: null }); + expect(info!.errMsg).toContain("未检测到已登陆"); + }); +}); diff --git a/packages/adapters/src/sites/user-parsers/registry.ts b/packages/adapters/src/sites/user-parsers/registry.ts new file mode 100644 index 00000000..6915dd05 --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/registry.ts @@ -0,0 +1,114 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/modules/indexer/__init__.py @ e6820a8 + * + * 站点用户信息解析器注册表与刷新入口。 + * - `IndexerModule.__init__` 里 `ModuleHelper.load('app.modules.indexer.parser', filter=有 schema)` → `USER_PARSER_CLASSES` + * (按模块文件名字母序,与 pkgutil 遍历顺序一致;回退时按此顺序尝试) + * - `IndexerModule.refresh_userdata` → `refreshSiteUserData`:按站点声明的 schema 取解析器;解析后 userid 为空且非公开站时, + * 依次尝试其他 schema(兼容 schema 标注错误/变种站),其他 schema 的异常只记录不抛出。 + * + * 与 MP 一致:声明 schema 解析器的异常原样抛出(由调用方记录),只有回退尝试的异常被吞掉。 + * 差异:结果不含凭据;`fetchMessages` 默认 false(MP 由全局 SITE_MESSAGE 控制)。 + */ +import type { SiteParserBase, SiteParserClass } from "./base.ts"; +import { BitptSiteUserInfo } from "./bitpt.ts"; +import { extractDomain } from "./common.ts"; +import { DiscuzUserInfo } from "./discuz.ts"; +import { FileListSiteUserInfo } from "./file-list.ts"; +import { GazelleSiteUserInfo } from "./gazelle.ts"; +import { HDDolbySiteUserInfo } from "./hddolby.ts"; +import { IptSiteUserInfo } from "./ipt.ts"; +import { MTorrentSiteUserInfo } from "./mtorrent.ts"; +import { NexusAudiencesSiteUserInfo } from "./nexus-audiences.ts"; +import { NexusHhanclubSiteUserInfo } from "./nexus-hhanclub.ts"; +import { NexusPhpSiteUserInfo } from "./nexus-php.ts"; +import { NexusProjectSiteUserInfo } from "./nexus-project.ts"; +import { NexusRabbitSiteUserInfo } from "./nexus-rabbit.ts"; +import { RousiSiteUserInfo } from "./rousi.ts"; +import { SmallHorseSiteUserInfo } from "./small-horse.ts"; +import { SunnyPTSiteUserInfo } from "./sunnypt.ts"; +import { TNodeSiteUserInfo } from "./tnode.ts"; +import { TorrentLeechSiteUserInfo } from "./torrent-leech.ts"; +import type { SiteSchemaValue, SiteUserInfo, SiteUserParserOptions } from "./types.ts"; +import { Unit3dSiteUserInfo } from "./unit3d.ts"; +import { YemaSiteUserInfo } from "./yema.ts"; +import { ZhixingSiteUserInfo } from "./zhixing.ts"; + +/** 全部解析器(模块字母序:bitpt, discuz, file_list, gazelle, hddolby, ipt, mtorrent, nexus_*, rousi, …) */ +export const USER_PARSER_CLASSES: readonly SiteParserClass[] = [ + BitptSiteUserInfo, + DiscuzUserInfo, + FileListSiteUserInfo, + GazelleSiteUserInfo, + HDDolbySiteUserInfo, + IptSiteUserInfo, + MTorrentSiteUserInfo, + NexusAudiencesSiteUserInfo, + NexusHhanclubSiteUserInfo, + NexusPhpSiteUserInfo, + NexusProjectSiteUserInfo, + NexusRabbitSiteUserInfo, + RousiSiteUserInfo, + SmallHorseSiteUserInfo, + SunnyPTSiteUserInfo, + TNodeSiteUserInfo, + TorrentLeechSiteUserInfo, + Unit3dSiteUserInfo, + YemaSiteUserInfo, + ZhixingSiteUserInfo, +] as unknown as SiteParserClass[]; + +export function getUserParserClass(schema: string | null | undefined): SiteParserClass | undefined { + if (!schema) return undefined; + return USER_PARSER_CLASSES.find((c) => c.schema === schema); +} + +/** refresh_userdata 的站点入参(MP site dict 子集) */ +export interface SiteUserDataRequest extends Omit { + name: string; + /** 站点声明的解析 schema(SiteSchema 值) */ + schema?: string | null; + /** 公开站:不报"未找到解析器",也不做 schema 回退 */ + public?: boolean; +} + +/** MP `refresh_userdata`:未找到解析器返回 null */ +export async function refreshSiteUserData(site: SiteUserDataRequest): Promise { + const make = (schema: string | null | undefined): SiteParserBase | null => { + const cls = getUserParserClass(schema); + if (!cls) return null; + const { name, schema: _s, public: _p, ...rest } = site; + return new cls({ ...rest, siteName: name }); + }; + let obj = make(site.schema); + if (!obj) return null; + try { + await obj.parse(); + if (!obj.userid && !site.public) { + const tried = new Set([String(site.schema)]); + for (const cls of USER_PARSER_CLASSES) { + const value = cls.schema as SiteSchemaValue | null; + if (!value || tried.has(value)) continue; + tried.add(value); + const alt = make(value); + if (!alt) continue; + try { + await alt.parse(); + } catch { + continue; + } + if (alt.userid) { + obj = alt; + break; + } + } + } + const info = obj.toUserInfo(); + info.domain = extractDomain(site.url) || null; + return info; + } finally { + obj.clear(); + } +} diff --git a/packages/adapters/src/sites/user-parsers/rousi.test.ts b/packages/adapters/src/sites/user-parsers/rousi.test.ts new file mode 100644 index 00000000..66b48311 --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/rousi.test.ts @@ -0,0 +1,35 @@ +import { createMockFetch, ROUSI_USER_ERROR, ROUSI_USER_PROFILE } from "@umt/testkit"; +import { describe, expect, it } from "vitest"; +import { RousiSiteUserInfo } from "./rousi.ts"; +import { runParser } from "./testing.ts"; + +describe("RousiSiteUserInfo(MP rousi.py)", () => { + it("api/v1/profile 单接口:流量/karma/做种与下载中统计", async () => { + const m = createMockFetch({ "GET https://rousi.pro/api/v1/profile": { body: ROUSI_USER_PROFILE } }); + const info = await runParser(RousiSiteUserInfo, { fetch: m.fetch, url: "https://rousi.pro/", apikey: "rk" }); + expect(info).toMatchObject({ + schema: "RousiPro", + userId: "1", + username: "jack", + userLevel: "Lv.5", + joinAt: "2024-01-01 00:00:00", + upload: 1073741824, + download: 536870912, + ratio: 2, + bonus: 1000.5, + seeding: 10, + seedingSize: 10737418240, + leeching: 2, + leechingSize: 2147483648, + }); + expect(m.requests[0]!.headers.authorization).toBe("Bearer rk"); + expect(m.requests).toHaveLength(1); + }); + + it("API 错误:errMsg 为站点 message", async () => { + const m = createMockFetch({ "GET https://rousi.pro/api/v1/profile": { body: ROUSI_USER_ERROR } }); + const info = await runParser(RousiSiteUserInfo, { fetch: m.fetch, url: "https://rousi.pro/", apikey: "bad" }); + expect(info.userId).toBeNull(); + expect(info.errMsg).toBe("API Key 无效"); + }); +}); diff --git a/packages/adapters/src/sites/user-parsers/rousi.ts b/packages/adapters/src/sites/user-parsers/rousi.ts new file mode 100644 index 00000000..6fd811e6 --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/rousi.ts @@ -0,0 +1,154 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/modules/indexer/parser/rousi.py @ e6820a8 + * + * Rousi.pro(PeerGo)用户数据解析器(MP `RousiSiteUserInfo`,apikey 模式)。 + * 函数对照:`_parse_site_page` → `_parseSitePage`(api/v1/profile,Bearer API Key)、`_parse_logged_in`(恒 True)、 + * `_parse_user_base_info` → `_parseUserBaseInfo`(单接口含流量/做种/下载中统计)、 + * `_pase_unread_msgs` → `_paseUnreadMsgs`(网页接口 api/messages,需 token;页码从 0 起,照 MP)。 + * + * 注意(写操作):与 MP 一样,读取未读消息后调用 `POST api/messages/read-all` 标记全部已读; + * 只在 `fetchMessages=true` 时执行。API Key / token 只放 Authorization 头,不进入结果。 + */ +import { redactText } from "../../rss/redact.ts"; +import { type NextPage, SiteParserBase, urljoin } from "./base.ts"; +import { extractDomain, normalizeDatetime, pyFloat, pyInt } from "./common.ts"; +import { SiteSchema, type SiteSchemaValue } from "./types.ts"; + +type Json = Record; + +export class RousiSiteUserInfo extends SiteParserBase { + static override schema: SiteSchemaValue | null = SiteSchema.RousiPro; + override requestMode = "apikey" as const; + + protected _parseSitePage(_htmlText: string): void { + this._baseUrl = `https://${extractDomain(this._siteUrl)}`; + this._userBasicPage = "api/v1/profile"; + this._userBasicParams = {}; + this._userBasicHeaders = { + "Content-Type": "application/json", + Accept: "application/json", + Authorization: `Bearer ${this.apikey}`, + }; + this._userTrafficPage = null; + this._userDetailPage = null; + this._torrentSeedingPage = null; + this._userMailUnreadPage = null; + this._sysMailUnreadPage = null; + } + + protected override _parseLoggedIn(_htmlText: unknown): boolean { + return true; + } + + protected _parseUserBaseInfo(htmlText: string): void { + if (!htmlText) return; + let data: unknown; + try { + data = JSON.parse(htmlText); + } catch { + return; + } + if (!data || typeof data !== "object" || Array.isArray(data)) { + this.errMsg = "用户数据响应结构无效"; + return; + } + const d = data as Json; + if (d.code !== 0) { + this.errMsg = String(d.message || "未知错误"); + return; + } + const u = d.data; + if (!u || typeof u !== "object" || Array.isArray(u)) return; + const info = u as Json; + this.userid = info.id == null ? null : String(info.id); + this.username = (info.username as string) ?? null; + this.userLevel = ((info.level_text || info.role_text) as string) ?? null; + const registeredAt = info.registered_at; + const joinAt = typeof registeredAt === "string" ? normalizeDatetime(registeredAt) : null; + if (joinAt) this.joinAt = joinAt.length >= 19 ? joinAt.slice(0, 19) : joinAt; + this.upload = pyInt(info.uploaded || 0); + this.download = pyInt(info.downloaded || 0); + this.ratio = Math.round(pyFloat(info.ratio || 0) * 100) / 100; + this.bonus = pyFloat(info.karma || 0); + const sl = (info.seeding_leeching_data ?? {}) as Json; + this.seeding = pyInt(sl.seeding_count || 0); + this.seedingSize = pyInt(sl.seeding_size || 0); + this.leeching = pyInt(sl.leeching_count || 0); + this.leechingSize = pyInt(sl.leeching_size || 0); + } + + protected _parseUserTrafficInfo(_htmlText: string): void {} + + protected _parseUserDetailInfo(_htmlText: string): void {} + + protected _parseUserTorrentSeedingInfo(_htmlText: string, _multiPage = false): NextPage { + return null; + } + + protected _parseMessageUnreadLinks(_htmlText: string, _msgLinks: string[]): NextPage { + return null; + } + + protected _parseMessageContent(_htmlText: string): [string | null, string | null, string | null] { + return [null, null, null]; + } + + /** 直接请求(MP 此处绕过 `_get_page_content`,用 RequestUtils 带 query GET / POST) */ + private async _rawJson(method: "GET" | "POST", url: string, headers: Record): Promise { + const fetchImpl = this._opts.fetch ?? globalThis.fetch; + const timeout = AbortSignal.timeout(this._opts.timeoutMs ?? 60_000); + const signal = this._opts.signal ? AbortSignal.any([this._opts.signal, timeout]) : timeout; + try { + await this._opts.limiter?.acquire(this._opts.signal); + const res = await fetchImpl(url, { method, headers, signal }); + if (res.status !== 200) { + await res.body?.cancel().catch(() => {}); + this.httpErrors.push(redactText(`${method} ${url}:HTTP ${res.status}`)); + return null; + } + return (await res.json()) as Json; + } catch (e) { + if (this._opts.signal?.aborted) throw e; + this.httpErrors.push(redactText(`${method} ${url}:${(e as Error).message}`)); + return null; + } + } + + /** 解析所有未读消息标题和内容(网页接口) */ + protected override async _paseUnreadMsgs(): Promise { + if (!this.token) return; + const headers: Record = { + "User-Agent": `${this._ua}`, + Accept: "application/json, text/plain, */*", + Authorization: this.token.startsWith("Bearer ") ? this.token : `Bearer ${this.token}`, + }; + const getMessageList = async (page: number): Promise => { + const qs = new URLSearchParams({ page: String(page), page_size: "100", unread_only: "true" }); + const res = await this._rawJson("GET", `${urljoin(this._baseUrl, "api/messages")}?${qs}`, headers); + if (!res || (res.code ?? -1) !== 0) return { messages: [], total_pages: 0 }; + return (res.data ?? {}) as Json; + }; + let page = 0; + let res = await getMessageList(page); + page += 1; + const messages = [...((res.messages ?? []) as Json[])]; + const totalPages = Number(res.total_pages ?? 0); + while (page < totalPages && page < 50) { + res = await getMessageList(page); + messages.push(...((res.messages ?? []) as Json[])); + page += 1; + } + this.messageUnread = messages.length; + for (const m of messages) { + this.messageUnreadContents.push([ + (m.title as string) ?? null, + normalizeDatetime(m.created_at as string) ?? null, + (m.content as string) ?? null, + ]); + } + // 更新消息为已读(写操作,见文件头) + await this._rawJson("POST", urljoin(this._baseUrl, "api/messages/read-all"), headers); + } +} diff --git a/packages/adapters/src/sites/user-parsers/small-horse.test.ts b/packages/adapters/src/sites/user-parsers/small-horse.test.ts new file mode 100644 index 00000000..c0e3c2bc --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/small-horse.test.ts @@ -0,0 +1,55 @@ +import { + createMockFetch, + SMALL_HORSE_USER_INDEX, + SMALL_HORSE_USER_PAGE, + SMALL_HORSE_USER_SEEDING_P1, + SMALL_HORSE_USER_SEEDING_P2, +} from "@umt/testkit"; +import { describe, expect, it } from "vitest"; +import { SmallHorseSiteUserInfo } from "./small-horse.ts"; +import { runParser } from "./testing.ts"; + +const GiB = 1024 ** 3; + +describe("SmallHorseSiteUserInfo(MP small_horse.py)", () => { + it("stats nobullet 按下标取流量/等级,做种表按分页累加", async () => { + const m = createMockFetch({ + "/": { body: SMALL_HORSE_USER_INDEX }, + "/user.php": { body: SMALL_HORSE_USER_PAGE }, + "/torrents.php": { body: SMALL_HORSE_USER_SEEDING_P1 }, + "/torrents.php?page=2": { body: SMALL_HORSE_USER_SEEDING_P2 }, + }); + const info = await runParser(SmallHorseSiteUserInfo, { fetch: m.fetch }); + expect(info).toMatchObject({ + schema: "Small Horse", + userId: "9", + username: "eve", + joinAt: "2019-01-01 00:00:00", + upload: 1024 ** 4, + download: 200 * GiB, + ratio: 5.12, + bonus: 1234.5, + userLevel: "Power User", + leeching: 3, + seeding: 3, + seedingSize: 16 * GiB, + seedingInfo: [ + [4, 10 * GiB], + [2, 5 * GiB], + [9, GiB], + ], + }); + expect(m.requests.filter((r) => r.url.pathname === "/torrents.php").map((r) => r.url.search)).toEqual([ + "?type=seeding&userid=9", + "?type=seeding&userid=9&page=2", + ]); + }); + + it("统计列表缺项:与 MP 一样抛 IndexError", async () => { + const m = createMockFetch({ + "/": { body: SMALL_HORSE_USER_INDEX }, + "/user.php": { body: '

  • x
  • y
' }, + }); + await expect(runParser(SmallHorseSiteUserInfo, { fetch: m.fetch })).rejects.toThrow(/IndexError/); + }); +}); diff --git a/packages/adapters/src/sites/user-parsers/small-horse.ts b/packages/adapters/src/sites/user-parsers/small-horse.ts new file mode 100644 index 00000000..3c7daf39 --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/small-horse.ts @@ -0,0 +1,112 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/modules/indexer/parser/small_horse.py @ e6820a8 + * + * Small Horse(Gazelle 变种)站点用户信息解析器。 + * + * 函数对照(`SmallHorseSiteUserInfo`):`_parse_site_page` → `_parseSitePage`、`_parse_user_base_info` → + * `_parseUserBaseInfo`、`_parse_user_traffic_info` → `_parseUserTrafficInfo`(`ul.stats nobullet` 按下标取值)、 + * `_parse_user_detail_info` → 空、`_parse_user_torrent_seeding_info` → `_parseUserTorrentSeedingInfo`、 + * `_parse_message_unread_links` / `_parse_message_content` → 空。XPath 逐字保留。 + * + * 与 MP 一致:统计列表下标越界时抛错(MP IndexError)。 + */ +import { type NextPage, SiteParserBase } from "./base.ts"; +import { normalizeDatetime, parseFloatMp, parseIntMp, reSearch } from "./common.ts"; +import { hasChildElements, parseHtml, type XNode } from "./dom.ts"; +import { SiteSchema, type SiteSchemaValue } from "./types.ts"; + +function at(arr: T[], i: number): T { + if (i >= arr.length) throw new Error(`IndexError: list index out of range [${i}]`); + return arr[i]!; +} + +/** Python `str.split(":")[1]` */ +function after(s: string): string { + return at(s.split(":"), 1); +} + +export class SmallHorseSiteUserInfo extends SiteParserBase { + static override schema: SiteSchemaValue | null = SiteSchema.SmallHorse; + + protected _parseSitePage(htmlText: string): void { + const text = this._prepareHtmlText(htmlText); + const userDetail = reSearch(/user.php\?id=(\d+)/, text); + if (userDetail?.[0].trim()) { + this._userDetailPage = userDetail[0].trim().replace(/^\/+/, ""); + this.userid = userDetail[1]!; + this._torrentSeedingPage = `torrents.php?type=seeding&userid=${this.userid}`; + } + this._userTrafficPage = `user.php?id=${this.userid}`; + } + + protected _parseUserBaseInfo(htmlText: string): void { + const html = parseHtml(this._prepareHtmlText(htmlText)); + if (!html) return; + const ret = html.strings('//a[contains(@href, "user.php")]//text()'); + if (ret.length) this.username = String(ret[0]); + } + + /** 上传/下载/分享率 [做种数/魔力值] */ + protected _parseUserTrafficInfo(htmlText: string): void { + const html = parseHtml(this._prepareHtmlText(htmlText)); + if (!html) return; + const tmps = html.nodes('//ul[@class = "stats nobullet"]'); + if (!tmps.length) return; + const li = (n: XNode) => n.nodes("li"); + const stats1 = li(at(tmps, 1)); + if (stats1.length && stats1[0]!.strings("span//text()").length) { + this.joinAt = normalizeDatetime(stats1[0]!.strings("span//text()")[0]!) ?? null; + } + this.upload = this.numFilesize(after(at(at(stats1, 2).strings("text()"), 0)).trim()); + this.download = this.numFilesize(after(at(at(stats1, 3).strings("text()"), 0)).trim()); + const ratioSpan = at(stats1, 4).strings("span//text()"); + if (ratioSpan.length) this.ratio = parseFloatMp(ratioSpan[0]!.replaceAll("∞", "0")); + else this.ratio = parseFloatMp(after(at(at(stats1, 5).strings("text()"), 0))); + this.bonus = parseFloatMp(after(at(at(stats1, 5).strings("text()"), 0))); + this.userLevel = after(at(at(li(at(tmps, 3)), 0).strings("text()"), 0)).trim(); + this.leeching = parseIntMp(after(at(at(li(at(tmps, 4)), 6).strings("text()"), 0)).replaceAll("[", "")); + } + + protected _parseUserDetailInfo(_htmlText: string): void {} + + protected _parseUserTorrentSeedingInfo(htmlText: string): NextPage { + const html = parseHtml(htmlText); + if (!hasChildElements(html)) return null; + const sizeCol = 6; + const seedersCol = 8; + let pageSeeding = 0; + let pageSeedingSize = 0; + const pageSeedingInfo: Array<[number, number]> = []; + const sizes = html.nodes(`//table[@id="torrent_table"]//tr[position()>1]/td[${sizeCol}]`); + const seeders = html.nodes(`//table[@id="torrent_table"]//tr[position()>1]/td[${seedersCol}]`); + if (sizes.length && seeders.length) { + pageSeeding = sizes.length; + for (let i = 0; i < sizes.length; i++) { + const size = this.numFilesize(sizes[i]!.text().trim()); + const s = parseIntMp(seeders[i]?.text().trim()); + pageSeedingSize += size; + pageSeedingInfo.push([s, size]); + } + } + this.seeding += pageSeeding; + this.seedingSize += pageSeedingSize; + this.seedingInfo.push(...pageSeedingInfo); + let nextPage: string | null = null; + const nextPages = html.nodes('//ul[@class="pagination"]/li[contains(@class,"active")]/following-sibling::li'); + if (nextPages.length > 1) { + const pageNum = nextPages[0]!.text().trim(); + if (/^\d+$/.test(pageNum)) nextPage = `${this._torrentSeedingPage}&page=${pageNum}`; + } + return nextPage; + } + + protected _parseMessageUnreadLinks(_htmlText: string, _msgLinks: string[]): NextPage { + return null; + } + + protected _parseMessageContent(_htmlText: string): [string | null, string | null, string | null] { + return [null, null, null]; + } +} diff --git a/packages/adapters/src/sites/user-parsers/sunnypt.test.ts b/packages/adapters/src/sites/user-parsers/sunnypt.test.ts new file mode 100644 index 00000000..7c2bc97a --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/sunnypt.test.ts @@ -0,0 +1,47 @@ +import { createMockFetch, SUNNYPT_USER_ERROR, SUNNYPT_USER_MESSAGES, SUNNYPT_USER_PROFILE } from "@umt/testkit"; +import { describe, expect, it } from "vitest"; +import { SunnyPTSiteUserInfo } from "./sunnypt.ts"; +import { runParser } from "./testing.ts"; + +const API = "https://api.sunnypt.top/api/v1/mp"; + +describe("SunnyPTSiteUserInfo(MP sunnypt.py)", () => { + it("profile + 未读消息分页(不标记已读)", async () => { + const m = createMockFetch({ + [`GET ${API}/profile`]: { body: SUNNYPT_USER_PROFILE }, + [`GET ${API}/messages?unread_only=true&page=1`]: { body: SUNNYPT_USER_MESSAGES }, + }); + const info = await runParser(SunnyPTSiteUserInfo, { + fetch: m.fetch, + url: "https://sunnypt.top/", + apikey: "sk", + fetchMessages: true, + }); + expect(info).toMatchObject({ + schema: "SunnyPT", + userId: "12", + username: "kate", + userLevel: "Elite User", + joinAt: "2023-04-05 06:07:08", + upload: 4000, + download: 2000, + ratio: 2, + bonus: 88.8, + seeding: 7, + seedingSize: 7000, + leeching: 1, + leechingSize: 100, + messageUnread: 1, + messageUnreadContents: [{ head: "通知", date: "2026-09-20 08:00:00", content: "内容" }], + }); + expect(m.requests[0]!.headers["x-api-key"]).toBe("sk"); + expect(m.requests.every((r) => r.method === "GET")).toBe(true); + }); + + it("API 错误:errMsg 为站点 msg", async () => { + const m = createMockFetch({ [`GET ${API}/profile`]: { body: SUNNYPT_USER_ERROR } }); + const info = await runParser(SunnyPTSiteUserInfo, { fetch: m.fetch, url: "https://sunnypt.top/", apikey: "x" }); + expect(info.userId).toBeNull(); + expect(info.errMsg).toBe("API Key 无效"); + }); +}); diff --git a/packages/adapters/src/sites/user-parsers/sunnypt.ts b/packages/adapters/src/sites/user-parsers/sunnypt.ts new file mode 100644 index 00000000..5eff0d3c --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/sunnypt.ts @@ -0,0 +1,123 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/modules/indexer/parser/sunnypt.py @ e6820a8 + * + * SunnyPT MoviePilot API 用户数据解析器(MP `SunnyPTSiteUserInfo`,apikey 模式)。 + * 函数对照:`_parse_site_page` → `_parseSitePage`(api_url,缺省 https://api.sunnypt.top/api/v1/mp)、 + * `_load_api_payload` → `_loadApiPayload`、`_parse_user_base_info` → `_parseUserBaseInfo`(profile)、 + * `_pase_unread_msgs` → `_paseUnreadMsgs`(messages?unread_only=true 分页,不标记已读)、 + * `_parse_message_unread_links` → `_parseMessageUnreadLinks`,其余为空实现。 + * + * 差异:MP 消息元组带第 4 项 `sunnypt-message:` 来源标识;本项目统一消息结构只有 (head, date, content), + * 来源标识未保留。 + */ +import { type NextPage, SiteParserBase, urljoin } from "./base.ts"; +import { normalizeDatetime, pyFloat, pyInt } from "./common.ts"; +import { SiteSchema, type SiteSchemaValue } from "./types.ts"; + +type Json = Record; + +export class SunnyPTSiteUserInfo extends SiteParserBase { + static override schema: SiteSchemaValue | null = SiteSchema.SunnyPT; + override requestMode = "apikey" as const; + + protected _parseSitePage(_htmlText: string): void { + this._baseUrl = `${String(this._apiUrl || "https://api.sunnypt.top/api/v1/mp").replace(/\/+$/, "")}/`; + this._userBasicPage = "profile"; + this._userDetailPage = null; + this._userTrafficPage = null; + this._torrentSeedingPage = null; + this._userMailUnreadPage = "messages"; + this._sysMailUnreadPage = null; + this._additionHeaders = { + Accept: "application/json", + "User-Agent": `${this._ua}`, + "X-API-Key": this.apikey ?? "", + }; + } + + /** 解析并校验 SunnyPT 通用 JSON 响应,成功返回 data */ + protected _loadApiPayload(htmlText: string, operation: string): Json | null { + if (!htmlText) { + this.errMsg = `${operation}失败，未收到 API 响应`; + return null; + } + let payload: unknown; + try { + payload = JSON.parse(htmlText); + } catch { + this.errMsg = `${operation}失败，API 响应不是有效 JSON`; + return null; + } + if (!payload || typeof payload !== "object" || Array.isArray(payload)) { + this.errMsg = `${operation}失败，API 响应结构无效`; + return null; + } + const p = payload as Json; + if (String(p.code) !== "0") { + this.errMsg = (p.msg as string) || `${operation}失败`; + return null; + } + const data = p.data; + return data && typeof data === "object" && !Array.isArray(data) ? (data as Json) : null; + } + + protected _parseUserBaseInfo(htmlText: string): void { + const u = this._loadApiPayload(htmlText, "获取用户信息"); + if (!u) return; + this.userid = u.id == null ? null : String(u.id); + this.username = (u.username as string) ?? null; + this.userLevel = (u.level as string) || String(u.class || ""); + this.joinAt = (normalizeDatetime(u.registered_at as string) as string | null | undefined) ?? null; + this.upload = pyInt(u.uploaded || 0); + this.download = pyInt(u.downloaded || 0); + this.ratio = pyFloat(u.ratio || 0); + this.bonus = pyFloat(u.bonus || 0); + this.seeding = pyInt(u.seeding_count || 0); + this.seedingSize = pyInt(u.seeding_size || 0); + this.leeching = pyInt(u.leeching_count || 0); + this.leechingSize = pyInt(u.leeching_size || 0); + this.messageUnread = pyInt(u.unread_messages || 0); + } + + /** 分页读取未读消息;解析阶段不标记已读 */ + protected override async _paseUnreadMsgs(): Promise { + let page = 1; + for (let guard = 0; guard < 50; guard++) { + const query = new URLSearchParams({ unread_only: "true", page: String(page), page_size: "100" }); + const htmlText = await this._getPageContent(urljoin(this._baseUrl, `messages?${query}`)); + const hasMore = await this._parseMessageUnreadLinks(htmlText, []); + if (!hasMore) return; + page += 1; + } + } + + /** 解析未读消息列表并直接保存消息正文;有下一页返回占位串 */ + protected _parseMessageUnreadLinks(htmlText: string, _msgLinks: string[]): NextPage { + const data = this._loadApiPayload(htmlText, "获取未读消息"); + if (!data) return null; + this.messageUnread = pyInt(data.unread_count || 0); + for (const m of (data.items || []) as unknown[]) { + if (!m || typeof m !== "object" || Array.isArray(m) || !(m as Json).unread) continue; + const msg = m as Json; + const title = msg.title as string | null; + const content = msg.content as string | null; + const createdAt = normalizeDatetime(msg.created_at as string); + if (title && content && createdAt) this.messageUnreadContents.push([title, createdAt, content]); + } + return data.has_more ? "next" : null; + } + + protected _parseUserTrafficInfo(_htmlText: string): void {} + + protected _parseUserDetailInfo(_htmlText: string): void {} + + protected _parseUserTorrentSeedingInfo(_htmlText: string, _multiPage = false): NextPage { + return null; + } + + protected _parseMessageContent(_htmlText: string): [string | null, string | null, string | null] { + return [null, null, null]; + } +} diff --git a/packages/adapters/src/sites/user-parsers/testing.ts b/packages/adapters/src/sites/user-parsers/testing.ts new file mode 100644 index 00000000..09646e94 --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/testing.ts @@ -0,0 +1,12 @@ +/** 测试辅助:构造解析器并跑完整 parse 流程(非公开 API,仅测试使用)。 */ +import type { SiteParserClass } from "./base.ts"; +import type { SiteUserInfo, SiteUserParserOptions } from "./types.ts"; + +export async function runParser( + cls: SiteParserClass, + opts: Partial & { fetch: typeof globalThis.fetch }, +): Promise { + const p = new cls({ siteName: "demo", url: "https://pt.example.com/", ...opts }); + await p.parse(); + return p.toUserInfo(); +} diff --git a/packages/adapters/src/sites/user-parsers/tnode.test.ts b/packages/adapters/src/sites/user-parsers/tnode.test.ts new file mode 100644 index 00000000..7b325463 --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/tnode.test.ts @@ -0,0 +1,46 @@ +import { createMockFetch, TNODE_USER_INDEX, TNODE_USER_MAIN_INFO, TNODE_USER_SEEDING } from "@umt/testkit"; +import { describe, expect, it } from "vitest"; +import { runParser } from "./testing.ts"; +import { TNodeSiteUserInfo } from "./tnode.ts"; + +describe("TNodeSiteUserInfo(MP tnode.py)", () => { + it("首页 csrf → getMainInfo / listTorrentActivity JSON", async () => { + const m = createMockFetch({ + "/": { body: TNODE_USER_INDEX }, + "/api/user/getMainInfo": { body: TNODE_USER_MAIN_INFO }, + "/api/user/listTorrentActivity": { body: TNODE_USER_SEEDING }, + }); + const info = await runParser(TNodeSiteUserInfo, { fetch: m.fetch, ua: "UA/1" }); + expect(info).toMatchObject({ + schema: "TNode", + userId: "99", + username: "gina", + userLevel: "Elite", + joinAt: "2022-01-02 03:04:05", + upload: 2000, + download: 1000, + ratio: 2, + bonus: 12.5, + messageUnread: 3, + seeding: 2, + seedingSize: 150, + seedingInfo: [ + [3, 100], + [1, 50], + ], + }); + const main = m.requests.find((r) => r.url.pathname === "/api/user/getMainInfo")!; + expect(main.headers["x-csrf-token"]).toBe("csrf-abc"); + }); + + it("接口 status≠200:不填用户数据", async () => { + const m = createMockFetch({ + "/": { body: TNODE_USER_INDEX }, + "/api/user/getMainInfo": { body: { status: 401, msg: "unauthorized" } }, + "/api/user/listTorrentActivity": { body: { status: 401 } }, + }); + const info = await runParser(TNodeSiteUserInfo, { fetch: m.fetch }); + expect(info.userId).toBeNull(); + expect(info.seeding).toBe(0); + }); +}); diff --git a/packages/adapters/src/sites/user-parsers/tnode.ts b/packages/adapters/src/sites/user-parsers/tnode.ts new file mode 100644 index 00000000..f8bfbaaa --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/tnode.ts @@ -0,0 +1,97 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/modules/indexer/parser/tnode.py @ e6820a8 + * + * TNode 站点用户数据解析器(MP `TNodeSiteUserInfo`)。 + * 函数对照:`_parse_site_page` → `_parseSitePage`(首页取 x-csrf-token,改走 JSON 接口)、 + * `_parse_logged_in` → `_parseLoggedIn`(恒 True)、`_parse_user_base_info` → `_parseUserBaseInfo`、 + * `_parse_user_detail_info` → `_parseUserDetailInfo`(api/user/getMainInfo)、 + * `_parse_user_torrent_seeding_info` → `_parseUserTorrentSeedingInfo`(api/user/listTorrentActivity)、 + * 消息两函数(不支持)。 + */ +import { type NextPage, SiteParserBase } from "./base.ts"; +import { normalizeDatetime, reSearch } from "./common.ts"; +import { round3 } from "./nexus-php.ts"; +import { SiteSchema, type SiteSchemaValue } from "./types.ts"; + +type Json = Record; + +function tryJson(text: string): unknown { + try { + return JSON.parse(text); + } catch { + return undefined; + } +} + +export class TNodeSiteUserInfo extends SiteParserBase { + static override schema: SiteSchemaValue | null = SiteSchema.TNode; + + protected _parseSitePage(htmlText: string): void { + const text = this._prepareHtmlText(htmlText); + const csrf = reSearch(//, text); + if (csrf) { + this._additionHeaders = { "X-CSRF-TOKEN": csrf[1]! }; + this._userDetailPage = "api/user/getMainInfo"; + this._torrentSeedingPage = "api/user/listTorrentActivity?id=&type=seeding&page=1&size=20000"; + } + } + + /** 暂时跳过检测(与 MP 一致) */ + protected override _parseLoggedIn(_htmlText: unknown): boolean { + return true; + } + + protected _parseUserBaseInfo(_htmlText: string): void { + this.username = this.userid; + } + + protected _parseUserTrafficInfo(_htmlText: string): void {} + + protected _parseUserDetailInfo(htmlText: string): void { + const detail = tryJson(htmlText) as Json | undefined; + if (detail === undefined || detail === null || typeof detail !== "object") return; + if (detail.status !== 200) return; + const u = (detail.data ?? {}) as Json; + this.userid = u.id == null ? null : String(u.id); + this.username = (u.username as string) ?? null; + this.userLevel = (((u.class ?? {}) as Json).name as string) ?? null; + this.joinAt = normalizeDatetime(String(u.regTime ?? 0)) ?? null; + this.upload = Number(u.upload ?? 0); + this.download = Number(u.download ?? 0); + this.ratio = this.download <= 0 ? 0 : round3(this.upload / this.download); + this.bonus = Number(u.bonus ?? 0); + this.messageUnread = Number(u.unreadAdmin ?? 0) + Number(u.unreadInbox ?? 0) + Number(u.unreadSystem ?? 0); + } + + protected _parseUserTorrentSeedingInfo(htmlText: string, _multiPage = false): NextPage { + const info = tryJson(htmlText); + if (info === undefined) return null; + if (!info || typeof info !== "object" || Array.isArray(info)) return null; + const seedingInfo = info as Json; + if (seedingInfo.status !== 200) return null; + const torrents = (((seedingInfo.data ?? {}) as Json).torrents ?? []) as Json[]; + let pageSeedingSize = 0; + const pageSeedingInfo: Array<[number, number]> = []; + for (const t of torrents) { + const size = Number(t.size ?? 0); + const seeders = Number(t.seeding ?? 0); + pageSeedingSize += size; + pageSeedingInfo.push([seeders, size]); + } + this.seeding += torrents.length; + this.seedingSize += pageSeedingSize; + this.seedingInfo.push(...pageSeedingInfo); + return null; + } + + protected _parseMessageUnreadLinks(_htmlText: string, _msgLinks: string[]): NextPage { + return null; + } + + /** 系统/收件箱/管理员信息接口(api/message/listSystem 等)MP 未实现 */ + protected _parseMessageContent(_htmlText: string): [string | null, string | null, string | null] { + return [null, null, null]; + } +} diff --git a/packages/adapters/src/sites/user-parsers/torrent-leech.test.ts b/packages/adapters/src/sites/user-parsers/torrent-leech.test.ts new file mode 100644 index 00000000..a791ec4a --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/torrent-leech.test.ts @@ -0,0 +1,43 @@ +import { + createMockFetch, + TORRENTLEECH_USER_INDEX, + TORRENTLEECH_USER_PROFILE, + TORRENTLEECH_USER_SEEDING, +} from "@umt/testkit"; +import { describe, expect, it } from "vitest"; +import { runParser } from "./testing.ts"; +import { TorrentLeechSiteUserInfo } from "./torrent-leech.ts"; + +describe("TorrentLeechSiteUserInfo(MP torrent_leech.py)", () => { + it("顶栏 onclick 取用户 ID;资料页流量/等级/注册/积分;tbody 做种表", async () => { + const m = createMockFetch({ + "/": { body: TORRENTLEECH_USER_INDEX }, + "/profile/frank/view": { body: TORRENTLEECH_USER_PROFILE }, + "/profile/frank/seeding": { body: TORRENTLEECH_USER_SEEDING }, + }); + const info = await runParser(TorrentLeechSiteUserInfo, { fetch: m.fetch }); + expect(info).toMatchObject({ + schema: "TorrentLeech", + userId: "frank", + username: "frank", + upload: 5 * 1024 ** 4, + download: 1024 ** 4, + ratio: 5, + userLevel: "VIP", + joinAt: "2016-06-07 08:09:10", + bonus: 98765.4, + seeding: 1, + seedingSize: 2 * 1024 ** 3, + seedingInfo: [[11, 2 * 1024 ** 3]], + }); + // 流量页置空,不请求 index.php + expect(m.requests.map((r) => r.url.pathname)).toEqual(["/", "/profile/frank/view", "/profile/frank/seeding"]); + }); + + it("已登录但取不到用户 ID:errMsg=未获取到用户ID,不再请求", async () => { + const m = createMockFetch({ "/": { body: `Logout` } }); + const info = await runParser(TorrentLeechSiteUserInfo, { fetch: m.fetch }); + expect(info.errMsg).toBe("未获取到用户ID"); + expect(m.requests).toHaveLength(1); + }); +}); diff --git a/packages/adapters/src/sites/user-parsers/torrent-leech.ts b/packages/adapters/src/sites/user-parsers/torrent-leech.ts new file mode 100644 index 00000000..3c1a761f --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/torrent-leech.ts @@ -0,0 +1,122 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/modules/indexer/parser/torrent_leech.py @ e6820a8 + * + * TorrentLeech 站点用户信息解析器(MP `TorrentLeechSiteUserInfo`)。 + * 函数对照:`_parse_site_page` → `_parseSitePage`、`_parse_user_base_info` → `_parseUserBaseInfo`、 + * `_parse_user_traffic_info` → `_parseUserTrafficInfo`、`_parse_user_detail_info` → `_parseUserDetailInfo`、 + * `_parse_user_torrent_seeding_info` → `_parseUserTorrentSeedingInfo`、消息两函数(不支持)。 + * 注意:做种表 XPath 使用 `//tbody/tr`,本项目解析器不自动插入 tbody,须站点原文带 tbody(与 lxml 行为一致)。 + */ +import { type NextPage, SiteParserBase } from "./base.ts"; +import { normalizeDatetime, parseFloatMp, parseIntMp, reSearch } from "./common.ts"; +import { hasChildElements, parseHtml } from "./dom.ts"; +import { SiteSchema, type SiteSchemaValue } from "./types.ts"; + +export class TorrentLeechSiteUserInfo extends SiteParserBase { + static override schema: SiteSchemaValue | null = SiteSchema.TorrentLeech; + + /** 解析当前用户 ID 并初始化用户资料页面地址 */ + protected _parseSitePage(htmlText: string): void { + const text = this._prepareHtmlText(htmlText); + const html = parseHtml(text); + let currentUserid: string | null = null; + if (hasChildElements(html)) { + const routes = html.strings( + '//span[contains(concat(" ", normalize-space(@class), " "), " centerTopBar ")]//*[@onclick]/@onclick', + ); + for (const route of routes) { + const m = reSearch(/\/profile\/([^/]+)\/view/, route); + if (m) { + currentUserid = m[1]!.trim(); + break; + } + } + } + const detail = reSearch(/\/profile\/([^/]+)\//, text); + const fallback = detail ? detail[1]!.trim() : null; + this.userid = currentUserid || fallback; + if (!this.userid) { + this.errMsg = "未获取到用户ID"; + this._userDetailPage = null; + this._userTrafficPage = null; + this._torrentSeedingPage = null; + return; + } + this._userDetailPage = `profile/${this.userid}/view`; + this._userTrafficPage = null; + this._torrentSeedingPage = `profile/${this.userid}/seeding`; + } + + protected _parseUserBaseInfo(_htmlText: string): void { + this.username = this.userid; + } + + /** 解析用户资料页中的用户名、流量、等级、注册时间和积分 */ + protected _parseUserTrafficInfo(htmlText: string): void { + const html = parseHtml(this._prepareHtmlText(htmlText)); + if (!hasChildElements(html)) return; + let username = html.strings( + '//div[contains(concat(" ", normalize-space(@class), " "), " profile-username ")]/text()', + ); + if (!username.length) { + username = html.strings( + '//table[contains(@class, "profileViewTable")]//tr/td[normalize-space()="Username"]/following-sibling::td[1]/text()', + ); + } + if (username.length && username[0]!.trim()) this.username = username[0]!.trim(); + const up = html.strings('//div[contains(@class,"profile-uploaded")]//span/text()'); + if (up.length) this.upload = this.numFilesize(up[0]); + const down = html.strings('//div[contains(@class,"profile-downloaded")]//span/text()'); + if (down.length) this.download = this.numFilesize(down[0]); + const ratio = html.strings('//div[contains(@class,"profile-ratio")]//span/text()'); + if (ratio.length) this.ratio = parseFloatMp(ratio[0]!.replaceAll("∞", "0")); + const level = html.strings( + '//table[contains(@class, "profileViewTable")]//tr/td[normalize-space()="Class"]/following-sibling::td[1]/text()', + ); + if (level.length) this.userLevel = level[0]!.trim(); + const joinAt = html.strings( + '//table[contains(@class, "profileViewTable")]//tr/td[normalize-space()="Registration date"]/following-sibling::td[1]/text()', + ); + if (joinAt.length) this.joinAt = normalizeDatetime(joinAt[0]!.trim()) ?? null; + const bonus = html.strings('//span[contains(@class, "total-TL-points")]/text()'); + if (bonus.length) this.bonus = parseFloatMp(bonus[0]!.trim()); + } + + protected _parseUserDetailInfo(htmlText: string): void { + this._parseUserTrafficInfo(htmlText); + } + + protected _parseUserTorrentSeedingInfo(htmlText: string, _multiPage = false): NextPage { + const html = parseHtml(htmlText); + if (!hasChildElements(html)) return null; + const sizeCol = 2; + const seedersCol = 7; + let pageSeeding = 0; + let pageSeedingSize = 0; + const pageSeedingInfo: Array<[number, number]> = []; + const sizes = html.nodes(`//tbody/tr/td[${sizeCol}]`); + const seeders = html.strings(`//tbody/tr/td[${seedersCol}]/text()`); + if (sizes.length && seeders.length) { + pageSeeding = sizes.length; + for (let i = 0; i < sizes.length; i++) { + const size = this.numFilesize(sizes[i]!.text().trim()); + pageSeedingSize += size; + pageSeedingInfo.push([parseIntMp(seeders[i]), size]); + } + } + this.seeding += pageSeeding; + this.seedingSize += pageSeedingSize; + this.seedingInfo.push(...pageSeedingInfo); + return null; + } + + protected _parseMessageUnreadLinks(_htmlText: string, _msgLinks: string[]): NextPage { + return null; + } + + protected _parseMessageContent(_htmlText: string): [string | null, string | null, string | null] { + return [null, null, null]; + } +} diff --git a/packages/adapters/src/sites/user-parsers/types.ts b/packages/adapters/src/sites/user-parsers/types.ts new file mode 100644 index 00000000..b0958d4f --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/types.ts @@ -0,0 +1,96 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/modules/indexer/parser/__init__.py @ e6820a8 + * @mp-source app/schemas/site.py @ e6820a8 + * + * 站点用户信息解析类型。 + * + * 来源:MoviePilot e6820a8(GPLv3)`app/modules/indexer/parser/__init__.py` `SiteSchema`、 + * `app/schemas/site.py` `SiteUserData`、`app/modules/indexer/__init__.py` `refresh_userdata`。 + */ + +/** MP `SiteSchema` 枚举(值与 MP 完全一致,站点定义 `schema` 字段直接写这些值)。 */ +export const SiteSchema = { + DiscuzX: "DiscuzX", + Gazelle: "Gazelle", + Ipt: "IPTorrents", + NexusPhp: "NexusPhp", + NexusProject: "NexusProject", + NexusRabbit: "NexusRabbit", + NexusHhanclub: "NexusHhanclub", + NexusAudiences: "NexusAudiences", + SmallHorse: "Small Horse", + Unit3d: "Unit3d", + TorrentLeech: "TorrentLeech", + FileList: "FileList", + TNode: "TNode", + MTorrent: "MTorrent", + Yema: "Yema", + HDDolby: "HDDolby", + Zhixing: "Zhixing", + Bitpt: "Bitpt", + RousiPro: "RousiPro", + SunnyPT: "SunnyPT", +} as const; +export type SiteSchemaValue = (typeof SiteSchema)[keyof typeof SiteSchema]; + +/** MP `SiteUnreadMessage`/`message_unread_contents` 元素 (head, date, content) */ +export interface SiteUnreadMessage { + head: string | null; + date: string | null; + content: string | null; +} + +/** + * 统一用户信息(MP `SiteUserData` 的 camelCase 投影)。 + * 数值字段沿用 MP 语义:解析不到为 0(MP 默认值),`userLevel/joinAt/username` 为 null。 + * `raw` 保存解析器实例的全部公开字段(MP `to_dict()` 等价,snake_case),不含任何凭据。 + */ +export interface SiteUserInfo { + /** 解析时使用的 schema(可能是回退后的 schema) */ + schema: SiteSchemaValue; + domain: string | null; + userId: string | null; + username: string | null; + userLevel: string | null; + joinAt: string | null; + upload: number; + download: number; + ratio: number; + bonus: number; + seeding: number; + seedingSize: number; + leeching: number; + leechingSize: number; + /** MP `seeding_info`:[做种人数, 种子大小] */ + seedingInfo: Array<[number, number]>; + messageUnread: number; + messageUnreadContents: SiteUnreadMessage[]; + /** H&R 未达标数:MP 解析器不提供,保留字段供站点扩展 */ + hrCount?: number; + /** 解析错误说明(MP `err_msg`,如"未检测到已登陆") */ + errMsg: string | null; + raw: Record; +} + +/** 请求上下文(MP `SiteParserBase.__init__` 参数;凭据只在内存中使用,不进入结果/日志)。 */ +export interface SiteUserParserOptions { + siteName: string; + /** 站点前端地址(MP `url`) */ + url: string; + cookie?: string | null; + apikey?: string | null; + token?: string | null; + ua?: string | null; + /** 站点独立 API Base URL(MP `api_url`,SunnyPT 等) */ + apiUrl?: string | null; + /** 是否读取未读消息正文(MP 全局设置 `SITE_MESSAGE`);默认 false。注意 MTorrent 读取后会标记已读 */ + fetchMessages?: boolean; + fetch?: typeof globalThis.fetch; + /** 单请求超时(MP 固定 60s) */ + timeoutMs?: number; + /** 共享站点限流器(与 list/search 共用) */ + limiter?: { acquire(signal?: AbortSignal): Promise }; + signal?: AbortSignal; +} diff --git a/packages/adapters/src/sites/user-parsers/unit3d.test.ts b/packages/adapters/src/sites/user-parsers/unit3d.test.ts new file mode 100644 index 00000000..61392771 --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/unit3d.test.ts @@ -0,0 +1,45 @@ +import { createMockFetch, UNIT3D_USER_DETAIL, UNIT3D_USER_INDEX, UNIT3D_USER_SEEDING } from "@umt/testkit"; +import { describe, expect, it } from "vitest"; +import { runParser } from "./testing.ts"; +import { Unit3dSiteUserInfo } from "./unit3d.ts"; + +const GiB = 1024 ** 3; + +describe("Unit3dSiteUserInfo(MP unit3d.py)", () => { + it("用户名来自 settings 链接;流量/等级/注册日期/做种列按 thead 定位", async () => { + const m = createMockFetch({ + "/": { body: UNIT3D_USER_INDEX }, + "/users/carol": { body: UNIT3D_USER_DETAIL }, + "/index.php": { body: UNIT3D_USER_INDEX }, + "/users/carol/active": { body: UNIT3D_USER_SEEDING }, + }); + const info = await runParser(Unit3dSiteUserInfo, { fetch: m.fetch }); + expect(info).toMatchObject({ + schema: "Unit3d", + username: "carol", + userLevel: "Elite", + joinAt: "2020-02-03 04:05:06", + bonus: 5678.9, + upload: 2 * 1024 ** 4, + download: 500 * GiB, + ratio: 4.1, + seeding: 2, + seedingSize: 4 * GiB, + seedingInfo: [ + [9, 3 * GiB], + [1, GiB], + ], + }); + expect(m.requests.find((r) => r.url.pathname === "/users/carol/active")!.url.searchParams.get("seeding")).toBe( + "include", + ); + }); + + it("首页取不到内容(HTTP 404):未登录并记录请求失败", async () => { + const m = createMockFetch({}); + const info = await runParser(Unit3dSiteUserInfo, { fetch: m.fetch }); + expect(info.username).toBeNull(); + expect(info.errMsg).toContain("未检测到已登陆"); + expect(info.raw.http_errors).toEqual([expect.stringContaining("HTTP 404")]); + }); +}); diff --git a/packages/adapters/src/sites/user-parsers/unit3d.ts b/packages/adapters/src/sites/user-parsers/unit3d.ts new file mode 100644 index 00000000..8c69d458 --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/unit3d.ts @@ -0,0 +1,110 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/modules/indexer/parser/unit3d.py @ e6820a8 + * + * UNIT3D 站点用户数据解析器(MP `Unit3dSiteUserInfo`)。 + * 函数对照:`_parse_user_base_info` → `_parseUserBaseInfo`、`_parse_site_page`(空)、 + * `_parse_user_detail_info` → `_parseUserDetailInfo`、`_parse_user_torrent_seeding_info` → `_parseUserTorrentSeedingInfo`、 + * `_parse_user_traffic_info` → `_parseUserTrafficInfo`、消息两函数(不支持)。XPath 与正则逐字保留。 + */ +import { type NextPage, SiteParserBase } from "./base.ts"; +import { normalizeDatetime, parseFloatMp, parseIntMp, reSearch } from "./common.ts"; +import { hasChildElements, parseHtml } from "./dom.ts"; +import { SiteSchema, type SiteSchemaValue } from "./types.ts"; + +export class Unit3dSiteUserInfo extends SiteParserBase { + static override schema: SiteSchemaValue | null = SiteSchema.Unit3d; + + protected _parseUserBaseInfo(htmlText: string): void { + const html = parseHtml(this._prepareHtmlText(htmlText)); + if (!html) return; + const hrefs = html.strings('//a[contains(@href, "/users/") and contains(@href, "settings")]/@href'); + if (hrefs.length) { + const m = reSearch(/\/users\/(.+)\/settings/, hrefs[0]); + if (m?.[0].trim()) { + this.username = m[1]!; + this._torrentSeedingPage = `/users/${this.username}/active?perPage=100&client=&seeding=include`; + this._userDetailPage = `/users/${this.username}`; + } + } + const bonus = html.nodes('//a[contains(@href, "bonus/earnings")]'); + if (bonus.length) { + const m = reSearch(/([\d,.]+)/, bonus[0]!.text()); + if (m?.[1]?.trim()) this.bonus = parseFloatMp(m[1]); + } + } + + protected _parseSitePage(_htmlText: string): void {} + + protected _parseUserDetailInfo(htmlText: string): void { + const html = parseHtml(htmlText); + if (!hasChildElements(html)) return; + const levels = html.strings('//div[contains(@class, "content")]//span[contains(@class, "badge-user")]/text()'); + if (levels.length) this.userLevel = levels[0]!.trim(); + const joinAt = html.strings( + '//div[contains(@class, "content")]//h4[contains(text(), "注册日期") or contains(text(), "註冊日期") or contains(text(), "Registration date")]/text()', + ); + if (joinAt.length) { + this.joinAt = + normalizeDatetime( + joinAt[0]!.replaceAll("注册日期", "").replaceAll("註冊日期", "").replaceAll("Registration date", ""), + ) ?? null; + } + } + + protected _parseUserTorrentSeedingInfo(htmlText: string, _multiPage = false): NextPage { + const html = parseHtml(htmlText); + if (!hasChildElements(html)) return null; + let sizeCol = 9; + let seedersCol = 2; + if (html.nodes('//thead//th[contains(@class,"size")]').length) { + sizeCol = html.nodes('//thead//th[contains(@class,"size")][1]/preceding-sibling::th').length + 1; + } + if (html.nodes('//thead//th[contains(@class,"seeders")]').length) { + seedersCol = html.nodes('//thead//th[contains(@class,"seeders")]/preceding-sibling::th').length + 1; + } + let pageSeeding = 0; + let pageSeedingSize = 0; + const pageSeedingInfo: Array<[number, number]> = []; + const sizes = html.nodes(`//tr[position()]/td[${sizeCol}]`); + const seeders = html.nodes(`//tr[position()]/td[${seedersCol}]`); + if (sizes.length && seeders.length) { + pageSeeding = sizes.length; + for (let i = 0; i < sizes.length; i++) { + const size = this.numFilesize(sizes[i]!.text().trim()); + pageSeedingSize += size; + pageSeedingInfo.push([parseIntMp(seeders[i]?.text().trim()), size]); + } + } + this.seeding += pageSeeding; + this.seedingSize += pageSeedingSize; + this.seedingInfo.push(...pageSeedingInfo); + + let nextPage: string | null = null; + const nextPages = html.nodes('//ul[@class="pagination"]/li[contains(@class,"active")]/following-sibling::li'); + if (nextPages.length > 1) { + const pageNum = nextPages[0]!.text().trim(); + if (/^\d+$/.test(pageNum)) nextPage = `${this._torrentSeedingPage}&page=${pageNum}`; + } + return nextPage; + } + + protected _parseUserTrafficInfo(htmlText: string): void { + const text = this._prepareHtmlText(htmlText); + const up = reSearch(/[^总]上[传傳]量?[:：_<>/a-zA-Z-="'\s#;]+([\d,.\s]+[KMGTPI]*B)/i, text); + this.upload = up ? this.numFilesize(up[1]!.trim()) : 0; + const down = reSearch(/[^总子影力]下[载載]量?[:：_<>/a-zA-Z-="'\s#;]+([\d,.\s]+[KMGTPI]*B)/i, text); + this.download = down ? this.numFilesize(down[1]!.trim()) : 0; + const ratio = reSearch(/分享率[:：_<>/a-zA-Z-="'\s#;]+([\d,.\s]+)/, text); + this.ratio = ratio?.[1]?.trim() ? parseFloatMp(ratio[1]) : 0.0; + } + + protected _parseMessageUnreadLinks(_htmlText: string, _msgLinks: string[]): NextPage { + return null; + } + + protected _parseMessageContent(_htmlText: string): [string | null, string | null, string | null] { + return [null, null, null]; + } +} diff --git a/packages/adapters/src/sites/user-parsers/yema.test.ts b/packages/adapters/src/sites/user-parsers/yema.test.ts new file mode 100644 index 00000000..dbd5e604 --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/yema.test.ts @@ -0,0 +1,35 @@ +import { createMockFetch, YEMA_USER_BASIC, YEMA_USER_ERROR } from "@umt/testkit"; +import { describe, expect, it } from "vitest"; +import { runParser } from "./testing.ts"; +import { YemaSiteUserInfo } from "./yema.ts"; + +const URL_ = "https://www.yemapt.org/openApi/user/fetchBasicInfo.json"; + +describe("YemaSiteUserInfo(MP yema.py)", () => { + it("POST fetchBasicInfo(JSON 体,Authorization=AuthKey):促销流量为上传/下载", async () => { + const m = createMockFetch({ [`POST ${URL_}`]: { body: YEMA_USER_BASIC } }); + const info = await runParser(YemaSiteUserInfo, { fetch: m.fetch, url: "https://www.yemapt.org/", apikey: "ak" }); + expect(info).toMatchObject({ + schema: "Yema", + userId: "321", + username: "leo", + userLevel: "3", + joinAt: "2024-05-06 07:08:09", + upload: 3000, + download: 1000, + ratio: 3, + bonus: 12.3, + }); + const req = m.requests[0]!; + expect(req.headers.authorization).toBe("ak"); + expect(req.headers["content-type"]).toBe("application/json"); + expect(req.body).toBe("{}"); + }); + + it("success=false:errMsg 为 errorMessage", async () => { + const m = createMockFetch({ [`POST ${URL_}`]: { body: YEMA_USER_ERROR } }); + const info = await runParser(YemaSiteUserInfo, { fetch: m.fetch, url: "https://www.yemapt.org/", apikey: "x" }); + expect(info.userId).toBeNull(); + expect(info.errMsg).toBe("AuthKey 无效"); + }); +}); diff --git a/packages/adapters/src/sites/user-parsers/yema.ts b/packages/adapters/src/sites/user-parsers/yema.ts new file mode 100644 index 00000000..257a601f --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/yema.ts @@ -0,0 +1,84 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/modules/indexer/parser/yema.py @ e6820a8 + * + * YemaPT 开放 API 用户数据解析器(MP `YemaSiteUserInfo`,apikey 模式)。 + * 函数对照:`_parse_site_page` → `_parseSitePage`(POST openApi/user/fetchBasicInfo.json,Authorization 头为 AuthKey)、 + * `_parse_user_base_info` → `_parseUserBaseInfo`(促销流量作为上传/下载),其余为空实现(开放 API 不提供)。 + */ +import { type NextPage, SiteParserBase } from "./base.ts"; +import { normalizeDatetime, pyFloat, pyInt } from "./common.ts"; +import { SiteSchema, type SiteSchemaValue } from "./types.ts"; + +type Json = Record; + +export class YemaSiteUserInfo extends SiteParserBase { + static override schema: SiteSchemaValue | null = SiteSchema.Yema; + override requestMode = "apikey" as const; + + protected _parseSitePage(_htmlText: string): void { + this._userBasicPage = "openApi/user/fetchBasicInfo.json"; + this._userBasicParams = {}; + this._userBasicMethod = "post"; + this._userDetailPage = null; + this._userTrafficPage = null; + this._torrentSeedingPage = null; + this._sysMailUnreadPage = null; + this._userMailUnreadPage = null; + this._additionHeaders = { + Authorization: this.apikey ?? "", + "Content-Type": "application/json", + Accept: "application/json, text/plain, */*", + "User-Agent": `${this._ua}`, + }; + } + + protected _parseUserBaseInfo(htmlText: string): void { + if (!htmlText) { + this.errMsg = "获取用户信息失败，未收到开放 API 响应"; + return; + } + let payload: unknown; + try { + payload = JSON.parse(htmlText); + } catch { + this.errMsg = "获取用户信息失败，开放 API 响应不是有效 JSON"; + return; + } + if (!payload || typeof payload !== "object" || Array.isArray(payload)) { + this.errMsg = "获取用户信息失败，开放 API 响应结构无效"; + return; + } + const p = payload as Json; + if (!p.success || !p.data || typeof p.data !== "object" || Array.isArray(p.data)) { + this.errMsg = (p.errorMessage as string) || "获取用户信息失败"; + return; + } + const u = p.data as Json; + this.userid = u.id == null ? null : String(u.id); + this.username = (u.name as string) ?? null; + this.userLevel = u.level != null ? String(u.level) : null; + this.joinAt = (normalizeDatetime(u.registerTime as string) as string | null | undefined) ?? null; + this.upload = pyInt(u.promotionUploadSize || 0); + this.download = pyInt(u.promotionDownloadSize || 0); + this.ratio = Math.round((this.upload / (this.download || 1)) * 100) / 100; + this.bonus = pyFloat(u.bonus || 0); + } + + protected _parseUserTrafficInfo(_htmlText: string): void {} + + protected _parseUserDetailInfo(_htmlText: string): void {} + + protected _parseUserTorrentSeedingInfo(_htmlText: string, _multiPage = false): NextPage { + return null; + } + + protected _parseMessageUnreadLinks(_htmlText: string, _msgLinks: string[]): NextPage { + return null; + } + + protected _parseMessageContent(_htmlText: string): [string | null, string | null, string | null] { + return [null, null, null]; + } +} diff --git a/packages/adapters/src/sites/user-parsers/zhixing.test.ts b/packages/adapters/src/sites/user-parsers/zhixing.test.ts new file mode 100644 index 00000000..516c17f1 --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/zhixing.test.ts @@ -0,0 +1,55 @@ +import { + createMockFetch, + ZHIXING_LOGIN_PAGE, + ZHIXING_USER_BASIC, + ZHIXING_USER_INDEX, + ZHIXING_USER_SEEDING_EMPTY, + ZHIXING_USER_SEEDING_P1, +} from "@umt/testkit"; +import { describe, expect, it } from "vitest"; +import { runParser } from "./testing.ts"; +import { ZhixingSiteUserInfo } from "./zhixing.ts"; + +const GiB = 1024 ** 3; + +describe("ZhixingSiteUserInfo(MP zhixing.py)", () => { + it("#um 登录判定 → user/{uid}/ 键值 → seeding/p{n} 逐页累加", async () => { + const m = createMockFetch({ + "/": { body: ZHIXING_USER_INDEX }, + "/user/321/": { body: ZHIXING_USER_BASIC }, + "/user/321/seeding/p1": { body: ZHIXING_USER_SEEDING_P1 }, + "/user/321/seeding/p2": { body: ZHIXING_USER_SEEDING_EMPTY }, + }); + const info = await runParser(ZhixingSiteUserInfo, { fetch: m.fetch }); + expect(info).toMatchObject({ + schema: "Zhixing", + userId: "321", + username: "zed", + userLevel: "Lv2", + joinAt: "2023-01-01 10:00:00", + upload: Math.round(1.5 * 1024 ** 4), + download: 123456, + ratio: 12.5, + bonus: 88.8, + seeding: 2, + seedingSize: 30 * GiB, + }); + }); + + it("做种页全空:回落基础信息的当前保种数量/容量", async () => { + const m = createMockFetch({ + "/": { body: ZHIXING_USER_INDEX }, + "/user/321/": { body: ZHIXING_USER_BASIC }, + "/user/321/seeding/p1": { body: ZHIXING_USER_SEEDING_EMPTY }, + }); + const info = await runParser(ZhixingSiteUserInfo, { fetch: m.fetch }); + expect(info).toMatchObject({ seeding: 7, seedingSize: 70 * GiB }); + }); + + it("未登录(无 #um):只请求首页", async () => { + const m = createMockFetch({ "/": { body: ZHIXING_LOGIN_PAGE } }); + const info = await runParser(ZhixingSiteUserInfo, { fetch: m.fetch }); + expect(info.userId).toBeNull(); + expect(m.requests).toHaveLength(1); + }); +}); diff --git a/packages/adapters/src/sites/user-parsers/zhixing.ts b/packages/adapters/src/sites/user-parsers/zhixing.ts new file mode 100644 index 00000000..24087323 --- /dev/null +++ b/packages/adapters/src/sites/user-parsers/zhixing.ts @@ -0,0 +1,159 @@ +/** + * SPDX-License-Identifier: GPL-3.0-only + * 移植自 MoviePilot(https://github.com/jxxghp/MoviePilot,GPLv3),本文件按 GPLv3 条款提供。 + * @mp-source app/modules/indexer/parser/zhixing.py @ e6820a8 + * + * 知行(pt.zhixing.bjtu.edu.cn)站点用户信息解析器(原作者 ThedoRap)。 + * + * 函数对照(`ZhixingSiteUserInfo`):`_parse_site_page` → `_parseSitePage`、`_parse_logged_in` → `_parseLoggedIn` + * (`#um`)、`_parse_user_base_info` → `_parseUserBaseInfo`(`div.user-details-tabs p` 键值)、 + * `_parse_user_torrent_seeding_page_info` → `parseUserTorrentSeedingPageInfo`、`parse` → `parse`(先走基类流程, + * 再从首页取 uid → `user/{uid}/` → `user/{uid}/seeding/p{n}` 逐页累加,空页结束,无结果回落基础信息)。 + * BeautifulSoup 以 cheerio 等价实现。 + * + * 与 MP 的偏离: + * 1. MP 末尾把全部字段转成字符串(避免其 UI join 报错);这里保持数值类型(`toUserInfo` 语义相同)。 + * 2. 做种翻页设上限 50 页(MP `while True`)。 + */ +import { load } from "cheerio"; +import { type NextPage, SiteParserBase, urljoin } from "./base.ts"; +import { normalizeDatetime, pyFloat, pyInt } from "./common.ts"; +import { SiteSchema, type SiteSchemaValue } from "./types.ts"; + +const MAX_PAGES = 50; + +export class ZhixingSiteUserInfo extends SiteParserBase { + static override schema: SiteSchemaValue | null = SiteSchema.Zhixing; + + protected _torrentSeedingBase: string | undefined; + protected _basicInfo: Record | undefined; + + protected _parseSitePage(_htmlText: string): void { + this._userBasicPage = "user/{uid}/"; + this._userDetailPage = null; + this._userBasicParams = {}; + this._userTrafficPage = null; + this._sysMailUnreadPage = null; + this._userMailUnreadPage = null; + this._mailUnreadParams = {}; + this._torrentSeedingBase = "user/{uid}/seeding"; + this._torrentSeedingParams = {}; + this._torrentSeedingHeaders = {}; + this._additionHeaders = {}; + } + + /** 通过是否存在 #um 用户信息判断登录 */ + protected override _parseLoggedIn(htmlText: unknown): boolean { + return load(String(htmlText ?? ""))("#um").length > 0; + } + + private numFilesizeSafe(s: string | undefined): number { + let v = s; + if (v) { + v = v.trim(); + if (/^\d+(\.\d+)?$/.test(v)) v += " B"; + } + return v ? this.numFilesize(v) : 0; + } + + protected _parseUserBaseInfo(htmlText: string): void { + if (!htmlText) return; + const $ = load(htmlText); + const info: Record = {}; + $("div.user-details-tabs").each((_, tab) => { + $(tab) + .find("p") + .each((_i, p) => { + const text = $(p).text().trim(); + let parts: string[]; + if (text.includes("：")) parts = [text.slice(0, text.indexOf("：")), text.slice(text.indexOf("：") + 1)]; + else if (text.includes(":")) parts = [text.slice(0, text.indexOf(":")), text.slice(text.indexOf(":") + 1)]; + else return; + const key = parts[0]!.trim(); + const valueText = parts[1]!.trim(); + info[key] = valueText.split(/\s*\(/)[0]!.trim().split("查看")[0]!.trim(); + }); + }); + this._basicInfo = info; + this.userid = info.UID ?? null; + this.username = info.用户名 ?? null; + this.userLevel = info.用户组 ?? null; + this.joinAt = "注册时间" in info ? (normalizeDatetime(info.注册时间) ?? null) : null; + this.upload = "上传流量" in info ? this.numFilesizeSafe(info.上传流量) : 0; + this.download = "下载流量" in info ? this.numFilesizeSafe(info.下载流量) : 0; + this.ratio = "共享率" in info ? pyFloat(info.共享率) : 0; + this.bonus = "保种积分" in info ? pyFloat(info.保种积分) : 0.0; + this.messageUnread = 0; // 暂无消息解析 + this.seeding = "当前保种数量" in info ? pyInt(info.当前保种数量) : 0; + this.seedingSize = "当前保种容量" in info ? this.numFilesizeSafe(info.当前保种容量) : 0; + } + + protected _parseUserTrafficInfo(_htmlText: string): void {} + protected _parseUserDetailInfo(_htmlText: string): void {} + + /** 单页做种:返回 [数量, 大小] */ + parseUserTorrentSeedingPageInfo(htmlText: string): [number, number] { + if (!htmlText) return [0, 0]; + const $ = load(htmlText); + let count = 0; + let size = 0; + $("tr") + .filter((_, el) => /^t\d+/.test($(el).attr("id") ?? "")) + .each((_, tr) => { + const td = $(tr).find("td.r").first(); + if (td.length) { + const a = td.find("a").first(); + const sizeText = a.length ? a.text() : td.text().trim(); + count += 1; + size += this.numFilesize(sizeText); + } + }); + return [count, size]; + } + + protected _parseMessageUnreadLinks(_htmlText: string, _msgLinks: string[]): NextPage { + return null; + } + protected _parseMessageContent(_htmlText: string): [string | null, string | null, string | null] { + return [null, null, null]; + } + protected _parseUserTorrentSeedingInfo(_htmlText: string): NextPage { + return null; + } + + override async parse(): Promise { + await super.parse(); + // 先从首页解析 userid + if (this._indexHtml) { + const $ = load(String(this._indexHtml)); + const link = $("a") + .filter((_, el) => /\/user\/\d+\//.test($(el).attr("href") ?? "")) + .first(); + if (link.length) { + const m = /\/user\/(\d+)\//.exec(link.attr("href") ?? ""); + if (m) this.userid = m[1]!; + } + } + if (this.userid) { + if (this._userBasicPage) { + const basicUrl = this._userBasicPage.replace("{uid}", this.userid); + this._parseUserBaseInfo(await this._getPageContent(urljoin(this._baseUrl, basicUrl))); + } + if (this._torrentSeedingBase) { + this.seeding = 0; + this.seedingSize = 0; + const baseUrl = urljoin(this._baseUrl, this._torrentSeedingBase.replace("{uid}", this.userid!)); + for (let page = 1; page <= MAX_PAGES; page++) { + const [n, s] = this.parseUserTorrentSeedingPageInfo(await this._getPageContent(`${baseUrl}/p${page}`)); + this.seeding += n; + this.seedingSize += s; + if (n === 0) break; + } + if (this.seeding === 0 && this._basicInfo) { + this.seeding = pyInt(this._basicInfo.当前保种数量 ?? 0); + this.seedingSize = this.numFilesizeSafe(this._basicInfo.当前保种容量 ?? ""); + } + } + } + } +} diff --git a/packages/testkit/src/sites/fixtures/definitions/chdbits.ts b/packages/testkit/src/sites/fixtures/definitions/chdbits.ts new file mode 100644 index 00000000..16c82ec9 --- /dev/null +++ b/packages/testkit/src/sites/fixtures/definitions/chdbits.ts @@ -0,0 +1,43 @@ +/** + * CHDBits(彩虹岛)列表页 fixture:按 MovieClaw v0.27.0 chdbits.yaml 选择器描述的标准 NexusPHP 结构自行编写 + * (tr > td > table.torrentname、font.subtitle、img.pro_*、td.embedded > span[title] 促销截止),**未对照真实页面**。 + */ +export const CHDBITS_LIST_PAGE = `CHDBits :: 种子 +
carol 退出 +上传量： 25.052 TB 下载量： 1.2 TB
+ + + + + + + + + + + + + + + + + + + + + + + + +
类型标题评论时间大小种子下载完成发布者
TV Series + + +
The.Last.of.Us.S02.2025.1080p.BluRay... + 2X Free [限时]
最后生还者 第二季 中字		download
32天45.20
GB		423100uploader
Movies + + +
Oppenheimer.2023...
奥本海默		download
026天80.00
GB		7055uploader
`; + +export const CHDBITS_LOGIN_PAGE = `CHDBits :: 登录 +
+

未登录!

`; diff --git a/packages/testkit/src/sites/fixtures/definitions/hhanclub.ts b/packages/testkit/src/sites/fixtures/definitions/hhanclub.ts new file mode 100644 index 00000000..31e790b2 --- /dev/null +++ b/packages/testkit/src/sites/fixtures/definitions/hhanclub.ts @@ -0,0 +1,47 @@ +/** + * HHClub(憨憨)列表/详情页 fixture:按 MovieClaw v0.27.0 hhanclub.yaml 选择器所描述的结构自行编写 + * (div 卡片模板 + promotion-tag-* + torrent-info-hr + Tailwind 详情页),**未对照真实页面**。 + */ +export const HHANCLUB_LIST_PAGE = `HHCLUB :: 种子 + +
+
+
综艺
+
Go.Fighting.S10E01.2026.2160p.WEB-DL.H265.AAC-HHWEB + 免费
+
极限挑战 第十季 第1期
+
下载
+
6天23时
+
6.12 GB
+
88
+
5
+
1,024
+
anon
+
30
+
+
+
电影
+
Dune.Part.Two.2024.2160p.UHD.BluRay.REMUX.DV.HDR.HEVC.TrueHD.7.1.Atmos-HHWEB + 50%
+
沙丘2
+
下载
+
2026-09-01 08:00:00
+
78.3 GB
+
12
+
0
+
300
+
anon
+
+
+ +`; + +export const HHANCLUB_DETAIL_PAGE = `HHCLUB :: 种子详情 +退出 +
+
标题
Dune.Part.Two.2024.2160p.UHD.BluRay.REMUX-HHWEB
+
副标题
沙丘2 | 中字
+
大小：78.30 GB
+ +
IMDb 豆瓣 简介……
+
`; diff --git a/packages/testkit/src/sites/fixtures/definitions/index.ts b/packages/testkit/src/sites/fixtures/definitions/index.ts new file mode 100644 index 00000000..3e62d175 --- /dev/null +++ b/packages/testkit/src/sites/fixtures/definitions/index.ts @@ -0,0 +1,2 @@ +export * from "./chdbits.ts"; +export * from "./hhanclub.ts"; diff --git a/packages/testkit/src/sites/fixtures/spiders/haidan.ts b/packages/testkit/src/sites/fixtures/spiders/haidan.ts new file mode 100644 index 00000000..cefc81a8 --- /dev/null +++ b/packages/testkit/src/sites/fixtures/spiders/haidan.ts @@ -0,0 +1,48 @@ +/** 海胆 `torrents.php?isapi=1` 响应样例(自行编写,字段名按 MP haidan.py 读取的键;不代表真实数据)。 */ +export const HAIDAN_SEARCH_OK = { + code: 0, + msg: "success", + data: { + "30001": { + group_id: 9001, + category: "401", + name: "Dune.Part.Two.2024.2160p.UHD.BluRay.REMUX.HDR.HEVC.Atmos-HDH", + small_descr: "沙丘2 | 4K 原盘 REMUX", + url: "https://www.haidan.video/download.php?id=30001&passkey=HDPASSKEY123", + added: 1726790400, + size: "76543210987", + seeders: "88", + leechers: "3", + times_completed: "412", + sp_state: "2", + }, + "30002": { + group_id: 9002, + category: "402", + name: "Shogun.2024.S01.1080p.WEB-DL.DDP5.1.H.264-HDH", + small_descr: "幕府将军 第一季", + url: "https://www.haidan.video/download.php?id=30002&passkey=HDPASSKEY123", + added: "2026-09-20 10:00:00", + size: 21474836480, + seeders: 12, + leechers: 0, + times_completed: 30, + sp_state: "6", + }, + "30003": { + group_id: 9003, + category: "406", + name: "Some.Album.2024.FLAC", + small_descr: "", + url: "https://www.haidan.video/download.php?id=30003&passkey=HDPASSKEY123", + added: 1726790400, + size: 1073741824, + seeders: 5, + leechers: 1, + times_completed: 2, + sp_state: "1", + }, + }, +}; + +export const HAIDAN_SEARCH_ERROR = { code: 1, msg: "请先登录" }; diff --git a/packages/testkit/src/sites/fixtures/spiders/hddolby.ts b/packages/testkit/src/sites/fixtures/spiders/hddolby.ts new file mode 100644 index 00000000..afc1b00a --- /dev/null +++ b/packages/testkit/src/sites/fixtures/spiders/hddolby.ts @@ -0,0 +1,53 @@ +/** HDDolby `/api/v1/torrent/search` 响应样例(自行编写,结构按 MP hddolby.py 注释示例)。 */ +export const HDDOLBY_SEARCH_OK = { + data: [ + { + id: 120202, + promotion_time_type: 2, + promotion_until: "2026-09-30 12:00:00", + category: 402, + leechers: 0, + seeders: 1, + name: "[DBY] Lost S06 2010 Complete 1080p Netflix WEB-DL AVC DDP5.1-DBTV", + small_descr: "lost ", + times_completed: 0, + size: 33665425886, + added: "2025-02-18 19:47:56", + url: 0, + hr: 1, + downhash: "DOWNHASHSECRET", + tags: "gf;zz;xx", + }, + { + id: 120203, + promotion_time_type: 6, + promotion_until: "0000-00-00 00:00:00", + category: 401, + leechers: 2, + seeders: 40, + name: "Movie.2024.2160p.WEB-DL.DV.HDR.H.265-DBTV", + small_descr: "电影", + times_completed: 99, + size: 12345678901, + added: "2026-09-01 08:00:00", + hr: 0, + downhash: "DOWNHASHSECRET2", + tags: "", + }, + { + id: 120204, + promotion_time_type: 0, + category: 406, + name: "Concert.2023.1080p", + size: 1000, + seeders: 1, + leechers: 0, + times_completed: 0, + added: "2026-09-01 08:00:00", + downhash: "X", + tags: null, + }, + ], +}; + +export const HDDOLBY_SEARCH_ERROR = { error: { message: "invalid api key" } }; diff --git a/packages/testkit/src/sites/fixtures/spiders/index.ts b/packages/testkit/src/sites/fixtures/spiders/index.ts new file mode 100644 index 00000000..3369b0db --- /dev/null +++ b/packages/testkit/src/sites/fixtures/spiders/index.ts @@ -0,0 +1,7 @@ +export * from "./haidan.ts"; +export * from "./hddolby.ts"; +export * from "./rousi.ts"; +export * from "./sunnypt.ts"; +export * from "./tnode.ts"; +export * from "./torrentleech.ts"; +export * from "./yema.ts"; diff --git a/packages/testkit/src/sites/fixtures/spiders/rousi.ts b/packages/testkit/src/sites/fixtures/spiders/rousi.ts new file mode 100644 index 00000000..714bb41b --- /dev/null +++ b/packages/testkit/src/sites/fixtures/spiders/rousi.ts @@ -0,0 +1,45 @@ +/** Rousi.pro `/api/v1/torrents` 响应样例(自行编写,字段名按 MP rousi.py)。 */ +export const ROUSI_SEARCH_OK = { + code: 0, + data: { + torrents: [ + { + id: 501, + title: "The.Bear.S03.2160p.DSNP.WEB-DL.DDP5.1.HDR.H.265", + subtitle: "熊家餐馆 第三季", + category: { slug: "tv", name: "TV" }, + size: "50000000000", + seeders: 20, + leechers: 2, + downloads: 77, + created_at: "2026-09-20T02:00:00Z", + promotion: { is_active: true, down_multiplier: 0, up_multiplier: 2, until: "2026-09-28T00:00:00+08:00" }, + }, + { + id: 502, + title: "Oppenheimer.2023.1080p.BluRay.x264", + subtitle: "奥本海默", + category: "movie", + size: 15000000000, + seeders: 50, + leechers: 0, + downloads: 300, + created_at: "2026-09-19 10:00:00", + promotion: { is_active: false, down_multiplier: 0.5 }, + }, + { + id: 503, + title: "Variety.Show.E01", + category: { slug: "variety" }, + size: 1, + seeders: 1, + leechers: 0, + downloads: 0, + created_at: "2026-09-19 10:00:00", + }, + ], + }, +}; + +export const ROUSI_SEARCH_ERROR = { code: 403, message: "permission denied: torrent:read" }; +export const ROUSI_DETAIL_OK = { code: 0, data: { download_url: "https://dl.rousi.invalid/t/501?cap=CAPSECRET" } }; diff --git a/packages/testkit/src/sites/fixtures/spiders/sunnypt.ts b/packages/testkit/src/sites/fixtures/spiders/sunnypt.ts new file mode 100644 index 00000000..f51b24fb --- /dev/null +++ b/packages/testkit/src/sites/fixtures/spiders/sunnypt.ts @@ -0,0 +1,55 @@ +/** SunnyPT MoviePilot API 响应样例(自行编写,字段名按 MP sunnypt.py)。 */ +export const SUNNYPT_CATEGORIES_OK = { + code: 0, + data: [ + { id: 401, name: "电影", media_types: ["movie"] }, + { id: 402, name: "剧集", media_types: ["tv"] }, + { id: 405, name: "动漫", media_types: ["movie", "tv"] }, + { id: 406, name: "音乐", media_types: ["music"] }, + ], +}; + +export const SUNNYPT_SEARCH_OK = { + code: "0", + data: { + items: [ + { + id: 7001, + title: "Arcane.S02.2160p.NF.WEB-DL.DDP5.1.Atmos.DV.H.265", + subtitle: "英雄联盟:双城之战 第二季", + media_type: "tv", + category_id: 402, + promotion: { is_active: true, down_multiplier: 0, up_multiplier: 1, until: "2026-09-30 23:59:59" }, + created_at: "2026-09-18 12:00:00", + size: 60000000000, + seeders: 30, + leechers: 1, + completed: 150, + details_url: "https://sunnypt.top/details.php?id=7001", + imdb_id: "tt11126994", + tags: ["官方", "中字"], + hit_and_run: true, + }, + { + id: 7002, + title: "Inception.2010.1080p.BluRay", + media_type: "movie", + category_id: 401, + promotion: { is_active: false, down_multiplier: 0 }, + created_at: "2026-09-18 12:00:00", + size: "9000000000", + seeders: 3, + leechers: 0, + completed: 10, + details_url: "https://sunnypt.top/details.php?id=7002", + hit_and_run: false, + }, + ], + }, +}; + +export const SUNNYPT_SEARCH_ERROR = { code: 40101, msg: "API Key 无效" }; +export const SUNNYPT_DOWNLOAD_TOKEN_OK = { + code: "0", + data: { download_url: "/api/v1/mp/torrents/7001/download?token=TOK" }, +}; diff --git a/packages/testkit/src/sites/fixtures/spiders/tnode.ts b/packages/testkit/src/sites/fixtures/spiders/tnode.ts new file mode 100644 index 00000000..ebbf90d4 --- /dev/null +++ b/packages/testkit/src/sites/fixtures/spiders/tnode.ts @@ -0,0 +1,37 @@ +/** TNode 首页(含 CSRF meta)与 `api/torrent/advancedSearch` 响应样例(自行编写)。 */ +export const TNODE_HOME_HTML = `TNodeok`; +export const TNODE_HOME_NO_TOKEN = `登录
`; + +export const TNODE_SEARCH_OK = { + status: 200, + data: { + total: 2, + torrents: [ + { + id: 3301, + title: "Evangelion.1.11.2007.1080p.BluRay.x264.FLAC", + subtitle: "新世纪福音战士新剧场版:序", + upload_time: 1726790400, + size: 9876543210, + seeding: 15, + leeching: 1, + complete: 60, + downloadRate: 0, + uploadRate: 1, + imdb: "tt0923811", + }, + { + id: 3302, + title: "Mushishi.S01.1080p.BluRay", + subtitle: "虫师", + upload_time: 1726794000, + size: 40000000000, + seeding: 8, + leeching: 0, + complete: 20, + downloadRate: 0.5, + uploadRate: 2, + }, + ], + }, +}; diff --git a/packages/testkit/src/sites/fixtures/spiders/torrentleech.ts b/packages/testkit/src/sites/fixtures/spiders/torrentleech.ts new file mode 100644 index 00000000..44cde05f --- /dev/null +++ b/packages/testkit/src/sites/fixtures/spiders/torrentleech.ts @@ -0,0 +1,31 @@ +/** TorrentLeech `torrents/browse/list/...` 响应样例(自行编写,字段名按 MP torrentleech.py)。 */ +export const TORRENTLEECH_SEARCH_OK = { + numFound: 2, + torrentList: [ + { + fid: 240001, + filename: "The.Matrix.1999.2160p.UHD.BluRay.x265-TL.torrent", + name: "The.Matrix.1999.2160p.UHD.BluRay.x265-TL", + addedTimestamp: "2026-09-20 08:00:00", + size: 25000000000, + seeders: 120, + leechers: 4, + completed: 999, + download_multiplier: 0, + imdbID: "tt0133093", + categoryID: 47, + }, + { + fid: 240002, + filename: "Show.S01E01.1080p.WEB.h264.torrent", + name: "Show.S01E01.1080p.WEB.h264", + addedTimestamp: "2026-09-20 09:00:00", + size: 2000000000, + seeders: 30, + leechers: 1, + completed: 100, + download_multiplier: 1, + categoryID: 32, + }, + ], +}; diff --git a/packages/testkit/src/sites/fixtures/spiders/yema.ts b/packages/testkit/src/sites/fixtures/spiders/yema.ts new file mode 100644 index 00000000..2d67241a --- /dev/null +++ b/packages/testkit/src/sites/fixtures/spiders/yema.ts @@ -0,0 +1,46 @@ +/** YemaPT 开放 API 响应样例(自行编写,字段名按 MP yema.py)。 */ +export const YEMA_SEARCH_MOVIE = { + success: true, + data: [ + { + id: 88001, + showName: "Past.Lives.2023.1080p.WEB-DL.DDP5.1.H.264", + shortDesc: "过往人生", + categoryId: 4, + tagList: ["3", "6", "99"], + listingTime: "2026-09-21 09:30:00", + fileSize: 8000000000, + seedNum: 25, + leechNum: 2, + completedNum: 80, + downloadPromotion: "free", + uploadPromotion: "double_upload", + downloadPromotionEndTime: "2026-09-29 00:00:00", + hrPunishEnable: true, + }, + { id: "bad-id", showName: "非整数 ID 被跳过" }, + ], +}; + +export const YEMA_SEARCH_TV_A = { + success: true, + data: [ + { + id: 88002, + showName: "Show.S01.1080p", + categoryId: 5, + tagList: [], + listingTime: "2026-09-21 09:30:00", + fileSize: 1, + seedNum: 1, + leechNum: 0, + completedNum: 0, + downloadPromotion: "half", + uploadPromotion: "one_half", + hrPunishEnable: false, + }, + ], +}; + +export const YEMA_SEARCH_ERROR = { success: false, errorMessage: "AuthKey 无效" }; +export const YEMA_DOWNLOAD_KEY_OK = { success: true, data: "DLKEY-123" }; diff --git a/packages/testkit/src/sites/fixtures/user-parsers/bitpt.ts b/packages/testkit/src/sites/fixtures/user-parsers/bitpt.ts new file mode 100644 index 00000000..3b9d357a --- /dev/null +++ b/packages/testkit/src/sites/fixtures/user-parsers/bitpt.ts @@ -0,0 +1,24 @@ +/** + * 极速之星(BitPT)用户信息 fixture(自行编写,不代表真实页面)。 + * 对应解析器:adapters/src/sites/user-parsers/bitpt.ts(MP bitpt.py)。 + */ +export const BITPT_USER_INDEX = ``; + +export const BITPT_USER_DETAILS = ` + + + + + + + + +
UID55
用户名gina (在线)
用户级别Elite
注册时间2021-06-07 08:09:10
上传流量3.00 TB
下载流量1.00 TB
共享率3.000
星辰当前100 累计2048.5
`; + +export const BITPT_USER_SEEDING_P1 = ` + + + +
标题大小
A5.00 GB
B1.00 GB
C9.00 GB
`; + +export const BITPT_USER_SEEDING_EMPTY = `
标题
`; diff --git a/packages/testkit/src/sites/fixtures/user-parsers/discuz.ts b/packages/testkit/src/sites/fixtures/user-parsers/discuz.ts new file mode 100644 index 00000000..fbffccb1 --- /dev/null +++ b/packages/testkit/src/sites/fixtures/user-parsers/discuz.ts @@ -0,0 +1,21 @@ +/** + * Discuz!X PT 插件用户信息 fixture(自行编写,不代表真实页面)。 + * 对应解析器:adapters/src/sites/user-parsers/discuz.ts(MP discuz.py)。 + */ +export const DISCUZ_USER_INDEX = ` +frank +退出 +`; + +export const DISCUZ_USER_SPACE = `
    +
  • 注册时间2018-02-03 04:05
    • +
  • 分享率 (1.5)
    • +
  • 积分520
    • +
  • 上传量1 TB / 2.00 TB
    • +
  • 下载量10 GB / 500.00 GB
    • +
+用户组 Lv3 +`; + +export const DISCUZ_USER_SEEDING = ` +
标题分类sizeseeders
A电影8.00 GB6
`; diff --git a/packages/testkit/src/sites/fixtures/user-parsers/file-list.ts b/packages/testkit/src/sites/fixtures/user-parsers/file-list.ts new file mode 100644 index 00000000..b5d11041 --- /dev/null +++ b/packages/testkit/src/sites/fixtures/user-parsers/file-list.ts @@ -0,0 +1,16 @@ +/** FileList 用户信息 fixture(自行编写的最小结构,不代表真实站点;对应 adapters/src/sites/user-parsers/file-list.ts)。 */ +export const FILELIST_USER_INDEX = `erin Logout`; + +export const FILELIST_USER_DETAIL = ` + + + + + + +
Uploaded3.00 TB
Downloaded1.00 TB
Share ratio3.00
Seed bonusSeeding:42Size:2.50 TB
Class Elite User
Join date2017-07-08 09:10:11 (8 years ago)
1,000.5`; + +export const FILELIST_USER_SEEDING = ` + + +
c1c2c3c4c5SizeSeeders
123451.00 GB4
`; diff --git a/packages/testkit/src/sites/fixtures/user-parsers/gazelle.ts b/packages/testkit/src/sites/fixtures/user-parsers/gazelle.ts new file mode 100644 index 00000000..b46ccb57 --- /dev/null +++ b/packages/testkit/src/sites/fixtures/user-parsers/gazelle.ts @@ -0,0 +1,21 @@ +/** Gazelle 用户信息 fixture(自行编写的最小结构,不代表真实站点;对应 adapters/src/sites/user-parsers/gazelle.ts)。 */ +export const GAZELLE_USER_INDEX = `
`; + +export const GAZELLE_USER_DETAIL = `
    +
  • 用户等级: Power User
    • +
  • 加入时间 2019-05-06 07:08:09
    • +
`; + +export const GAZELLE_USER_SEEDING = ` + + + +
NameFilesTimeSizeSnatchesSeedersLeechers
A11d10.00 GB370
B22d1.00 GB120
`; + +export const GAZELLE_LOGIN_PAGE = `
`; diff --git a/packages/testkit/src/sites/fixtures/user-parsers/hddolby.ts b/packages/testkit/src/sites/fixtures/user-parsers/hddolby.ts new file mode 100644 index 00000000..15b690b3 --- /dev/null +++ b/packages/testkit/src/sites/fixtures/user-parsers/hddolby.ts @@ -0,0 +1,26 @@ +/** HDDolby 用户信息 fixture(自行编写的最小结构,不代表真实站点;对应 adapters/src/sites/user-parsers/hddolby.ts)。 */ +export const HDDOLBY_USER_DATA = { + status: 0, + data: [ + { + id: "1", + username: "ivy", + added: "2019-03-03 15:30:36", + class: "6", + uploaded: "852071699418", + downloaded: "1885536536", + seedbonus: "99774.5", + unread_messages: "2", + }, + ], +}; + +export const HDDOLBY_USER_PEERS = { + status: 0, + data: [ + { size: 1000, seeders: 4 }, + { size: 500, seeders: 0 }, + ], +}; + +export const HDDOLBY_USER_ERROR = { status: 1, error: { message: "invalid api key" } }; diff --git a/packages/testkit/src/sites/fixtures/user-parsers/index.ts b/packages/testkit/src/sites/fixtures/user-parsers/index.ts new file mode 100644 index 00000000..92f40799 --- /dev/null +++ b/packages/testkit/src/sites/fixtures/user-parsers/index.ts @@ -0,0 +1,20 @@ +export * from "./bitpt.ts"; +export * from "./discuz.ts"; +export * from "./file-list.ts"; +export * from "./gazelle.ts"; +export * from "./hddolby.ts"; +export * from "./ipt.ts"; +export * from "./mtorrent.ts"; +export * from "./nexus-audiences.ts"; +export * from "./nexus-hhanclub.ts"; +export * from "./nexus-php.ts"; +export * from "./nexus-project.ts"; +export * from "./nexus-rabbit.ts"; +export * from "./rousi.ts"; +export * from "./small-horse.ts"; +export * from "./sunnypt.ts"; +export * from "./tnode.ts"; +export * from "./torrent-leech.ts"; +export * from "./unit3d.ts"; +export * from "./yema.ts"; +export * from "./zhixing.ts"; diff --git a/packages/testkit/src/sites/fixtures/user-parsers/ipt.ts b/packages/testkit/src/sites/fixtures/user-parsers/ipt.ts new file mode 100644 index 00000000..6fcf9747 --- /dev/null +++ b/packages/testkit/src/sites/fixtures/user-parsers/ipt.ts @@ -0,0 +1,12 @@ +/** IPTorrents 用户信息 fixture(自行编写的最小结构,不代表真实站点;对应 adapters/src/sites/user-parsers/ipt.ts)。 */ +export const IPT_USER_INDEX = `Logout +dave +
+2.5001.20 TB480.00 GB +xy1534321.5 +
`; + +export const IPT_USER_DETAIL = ` + + +
ClassPower User
Join date2018-01-02 03:04:05 (7 years ago)
`; diff --git a/packages/testkit/src/sites/fixtures/user-parsers/mtorrent.ts b/packages/testkit/src/sites/fixtures/user-parsers/mtorrent.ts new file mode 100644 index 00000000..ce048eb3 --- /dev/null +++ b/packages/testkit/src/sites/fixtures/user-parsers/mtorrent.ts @@ -0,0 +1,34 @@ +/** M-Team 用户信息 fixture(自行编写的最小结构,不代表真实站点;对应 adapters/src/sites/user-parsers/mtorrent.ts)。 */ +export const MTORRENT_USER_PROFILE = { + code: "0", + message: "SUCCESS", + data: { + id: "31337", + username: "henry", + role: "6", + memberStatus: { createdDate: "2020-10-11 12:13:14" }, + memberCount: { uploaded: "3000", downloaded: "1000", shareRate: "3.000", bonus: "4567.8" }, + }, +}; + +export const MTORRENT_USER_SEEDING_PAGE1 = { + code: "0", + data: { data: [{ torrent: { size: "100", source: "5" } }, { torrent: { size: "200", source: "2" } }] }, +}; +export const MTORRENT_USER_SEEDING_PAGE2 = { + code: "0", + data: { data: [{ torrent: { size: "300", source: "1" } }] }, +}; +export const MTORRENT_USER_PEER_STATUS = { code: "0", data: { seeder: "3", leecher: "0" } }; + +export const MTORRENT_USER_MESSAGES = { + code: "0", + data: { + data: [ + { id: "m1", unread: true, title: "欢迎", createdDate: "2026-09-01 10:00:00", context: "hello" }, + { id: "m2", unread: false, title: "旧消息", createdDate: "2026-08-01 10:00:00", context: "old" }, + ], + }, +}; + +export const MTORRENT_USER_KEY_INVALID = { code: "1", message: "key無效" }; diff --git a/packages/testkit/src/sites/fixtures/user-parsers/nexus-audiences.ts b/packages/testkit/src/sites/fixtures/user-parsers/nexus-audiences.ts new file mode 100644 index 00000000..82bac1de --- /dev/null +++ b/packages/testkit/src/sites/fixtures/user-parsers/nexus-audiences.ts @@ -0,0 +1,31 @@ +/** + * 观众(Audiences)新版 userbar 用户信息 fixture(自行编写的最小结构,不代表真实页面)。 + * 对应解析器:adapters/src/sites/user-parsers/nexus-audiences.ts(MP nexus_audiences.py)。 + */ +export const AUDIENCES_USER_INDEX = `
+bob +分享率20.48 +信 +退出 +
`; + +export const AUDIENCES_USER_DETAILS = ` + +
加入日期2020-05-06 07:08:09 (5年前)
`; + +export const AUDIENCES_USER_SEEDING = ` + +
类型数量体积
电影202.00 TB
Total313.50 TB
`; + +export const AUDIENCES_USER_MAILBOX = `
+
看 +种子被删除2026-09-01 10:00:00你的种子 X 已被删除
+
看 +欢迎2026-09-02 11:00:00欢迎加入观众
+
已读
+
退出`; + +export const AUDIENCES_USER_MESSAGE_901 = `
+

种子被删除

+
日期2026-09-01 10:00:00
+
你的种子 X 因重复已被删除。
`; diff --git a/packages/testkit/src/sites/fixtures/user-parsers/nexus-hhanclub.ts b/packages/testkit/src/sites/fixtures/user-parsers/nexus-hhanclub.ts new file mode 100644 index 00000000..72b3bae4 --- /dev/null +++ b/packages/testkit/src/sites/fixtures/user-parsers/nexus-hhanclub.ts @@ -0,0 +1,23 @@ +/** + * 憨憨(HHClub)用户信息 fixture(自行编写的最小结构,仿 #user-info-panel 布局;不代表真实页面)。 + * 对应解析器:adapters/src/sites/user-parsers/nexus-hhanclub.ts(MP nexus_hhanclub.py)。 + */ +export const HHCLUB_USER_INDEX = ` +hana 登出 +
+
hana
+
+
[分享率]: 3.5
+
a
b
c
[上传量]: 1.20 TB
[下载量]: 300.00 GB
+
+
+魔力值: 6,789.0 +`; + +export const HHCLUB_USER_DETAILS = ` +
加入日期3 年前
+Power User +`; + +/** 已登录但 #user-info-panel 缺失(改版):MP 在 xpath(...)[0] 抛 IndexError */ +export const HHCLUB_USER_INDEX_NO_PANEL = `hana 登出`; diff --git a/packages/testkit/src/sites/fixtures/user-parsers/nexus-php.ts b/packages/testkit/src/sites/fixtures/user-parsers/nexus-php.ts new file mode 100644 index 00000000..f279870a --- /dev/null +++ b/packages/testkit/src/sites/fixtures/user-parsers/nexus-php.ts @@ -0,0 +1,29 @@ +/** + * NexusPHP 用户信息页 fixture(自行编写的最小结构,仿 NexusPHP 1.x 默认模板;不代表任何真实站点)。 + * 对应解析器:adapters/src/sites/user-parsers/nexus-php.ts(MP nexus_php.py)。 + */ +export const NEXUS_PHP_INDEX = `Demo NexusPHP + +
欢迎回来, alice + [退出] [使用]: 12,345.6 +分享率： 3.217 +上传量： 1.50 TB +下载量： 477.30 GB +当前活动：Torrents seeding12 Torrents leeching2 +inbox 信息箱 3
+`; + +export const NEXUS_PHP_USERDETAILS = ` + + + +
加入日期2021-03-04 05:06:07 (4年前)
等级Veteran User
魔力值12345.6
`; + +export const NEXUS_PHP_SEEDING = ` + + +
类型标题sizeseeders
电影A10.00 GB5
剧集B1.5 TB12
`; + +export const NEXUS_PHP_LOGIN_PAGE = `
`; + +export const NEXUS_PHP_CF_CHALLENGE = `Just a moment...
`; diff --git a/packages/testkit/src/sites/fixtures/user-parsers/nexus-project.ts b/packages/testkit/src/sites/fixtures/user-parsers/nexus-project.ts new file mode 100644 index 00000000..ff23c7d3 --- /dev/null +++ b/packages/testkit/src/sites/fixtures/user-parsers/nexus-project.ts @@ -0,0 +1,8 @@ +/** + * NexusProject 变种(star-space 风格 p_user/user_detail.php?uid=)用户信息 fixture(自行编写,不代表真实页面)。 + * 对应解析器:adapters/src/sites/user-parsers/nexus-project.ts(MP nexus_project.py)。 + */ +export const NEXUS_PROJECT_USER_INDEX = ` +carol 退出 +上传：445.11 G下载：29.61 G +`; diff --git a/packages/testkit/src/sites/fixtures/user-parsers/nexus-rabbit.ts b/packages/testkit/src/sites/fixtures/user-parsers/nexus-rabbit.ts new file mode 100644 index 00000000..d9cf6de2 --- /dev/null +++ b/packages/testkit/src/sites/fixtures/user-parsers/nexus-rabbit.ts @@ -0,0 +1,23 @@ +/** + * NexusRabbit(layui-hares 模板)用户信息 fixture(自行编写,不代表真实页面)。 + * 对应解析器:adapters/src/sites/user-parsers/nexus-rabbit.ts(MP nexus_rabbit.py)。 + */ +export const RABBIT_USER_INDEX = ` +dave 退出 +
奶糖余额
1,024.5
+`; + +export const RABBIT_USER_DETAILS = ``; + +export const RABBIT_USER_SEEDING_JSON = { + data: [ + { seeders: "3", size: "1.00 GB" }, + { seeders: 7, size: "2.50 GB" }, + ], +}; diff --git a/packages/testkit/src/sites/fixtures/user-parsers/rousi.ts b/packages/testkit/src/sites/fixtures/user-parsers/rousi.ts new file mode 100644 index 00000000..e61bc333 --- /dev/null +++ b/packages/testkit/src/sites/fixtures/user-parsers/rousi.ts @@ -0,0 +1,23 @@ +/** Rousi.pro 用户信息 fixture(自行编写的最小结构,不代表真实站点;对应 adapters/src/sites/user-parsers/rousi.ts)。 */ +export const ROUSI_USER_PROFILE = { + code: 0, + message: "success", + data: { + id: 1, + username: "jack", + level_text: "Lv.5", + registered_at: "2024-01-01T00:00:00Z", + uploaded: 1073741824, + downloaded: 536870912, + ratio: 2.0, + karma: 1000.5, + seeding_leeching_data: { + seeding_count: 10, + seeding_size: 10737418240, + leeching_count: 2, + leeching_size: 2147483648, + }, + }, +}; + +export const ROUSI_USER_ERROR = { code: 401, message: "API Key 无效" }; diff --git a/packages/testkit/src/sites/fixtures/user-parsers/small-horse.ts b/packages/testkit/src/sites/fixtures/user-parsers/small-horse.ts new file mode 100644 index 00000000..d84709e7 --- /dev/null +++ b/packages/testkit/src/sites/fixtures/user-parsers/small-horse.ts @@ -0,0 +1,29 @@ +/** + * Small Horse(Gazelle 变种)用户信息 fixture(自行编写,不代表真实页面)。 + * 对应解析器:adapters/src/sites/user-parsers/small-horse.ts(MP small_horse.py)。 + */ +export const SMALL_HORSE_USER_INDEX = `eve Logout`; + +export const SMALL_HORSE_USER_PAGE = ` +
  • Profile
+
    +
  • Joined: 2019-01-01 00:00:00
    • +
  • Last seen: now
    • +
  • Uploaded: 1.00 TB
    • +
  • Downloaded: 200.00 GB
    • +
  • Ratio: 5.12
    • +
  • Bonus: 1234.5
    • +
+
  • Percentile
+
  • Class: Power User
+
  • a
  • b
  • c
  • d
  • e
  • f
  • Leeching: 3
+`; + +const row = (size: string, seeders: number) => + `12345${size}7${seeders}`; + +export const SMALL_HORSE_USER_SEEDING_P1 = `${row("10.00 GB", 4)}${row("5.00 GB", 2)}
h
+
  • 1
  • 2
  • Next
`; + +export const SMALL_HORSE_USER_SEEDING_P2 = `${row("1.00 GB", 9)}
h
+
  • 1
  • 2
`; diff --git a/packages/testkit/src/sites/fixtures/user-parsers/sunnypt.ts b/packages/testkit/src/sites/fixtures/user-parsers/sunnypt.ts new file mode 100644 index 00000000..e3f89664 --- /dev/null +++ b/packages/testkit/src/sites/fixtures/user-parsers/sunnypt.ts @@ -0,0 +1,31 @@ +/** SunnyPT 用户信息 fixture(自行编写的最小结构,不代表真实站点;对应 adapters/src/sites/user-parsers/sunnypt.ts)。 */ +export const SUNNYPT_USER_PROFILE = { + code: 0, + msg: "ok", + data: { + id: 12, + username: "kate", + level: "Elite User", + registered_at: "2023-04-05 06:07:08", + uploaded: 4000, + downloaded: 2000, + ratio: 2, + bonus: 88.8, + seeding_count: 7, + seeding_size: 7000, + leeching_count: 1, + leeching_size: 100, + unread_messages: 1, + }, +}; + +export const SUNNYPT_USER_MESSAGES = { + code: 0, + data: { + unread_count: 1, + has_more: false, + items: [{ id: 5, unread: true, title: "通知", content: "内容", created_at: "2026-09-20 08:00:00" }], + }, +}; + +export const SUNNYPT_USER_ERROR = { code: 403, msg: "API Key 无效" }; diff --git a/packages/testkit/src/sites/fixtures/user-parsers/tnode.ts b/packages/testkit/src/sites/fixtures/user-parsers/tnode.ts new file mode 100644 index 00000000..476b786f --- /dev/null +++ b/packages/testkit/src/sites/fixtures/user-parsers/tnode.ts @@ -0,0 +1,28 @@ +/** TNode 用户信息 fixture(自行编写的最小结构,不代表真实站点;对应 adapters/src/sites/user-parsers/tnode.ts)。 */ +export const TNODE_USER_INDEX = ``; + +export const TNODE_USER_MAIN_INFO = { + status: 200, + data: { + id: 99, + username: "gina", + class: { name: "Elite" }, + regTime: "2022-01-02 03:04:05", + upload: 2000, + download: 1000, + bonus: 12.5, + unreadAdmin: 1, + unreadInbox: 2, + unreadSystem: 0, + }, +}; + +export const TNODE_USER_SEEDING = { + status: 200, + data: { + torrents: [ + { size: 100, seeding: 3 }, + { size: 50, seeding: 1 }, + ], + }, +}; diff --git a/packages/testkit/src/sites/fixtures/user-parsers/torrent-leech.ts b/packages/testkit/src/sites/fixtures/user-parsers/torrent-leech.ts new file mode 100644 index 00000000..0487b4c0 --- /dev/null +++ b/packages/testkit/src/sites/fixtures/user-parsers/torrent-leech.ts @@ -0,0 +1,14 @@ +/** TorrentLeech 用户信息 fixture(自行编写的最小结构,不代表真实站点;对应 adapters/src/sites/user-parsers/torrent-leech.ts)。 */ +export const TORRENTLEECH_USER_INDEX = `frank +Logout`; + +export const TORRENTLEECH_USER_PROFILE = `
frank
+
5.00 TB
+
1.00 TB
+
5.00
+
ClassVIP
Registration date2016-06-07 08:09:10
+98765.4`; + +export const TORRENTLEECH_USER_SEEDING = ` + +
A2.00 GB345611
`; diff --git a/packages/testkit/src/sites/fixtures/user-parsers/unit3d.ts b/packages/testkit/src/sites/fixtures/user-parsers/unit3d.ts new file mode 100644 index 00000000..80061cad --- /dev/null +++ b/packages/testkit/src/sites/fixtures/user-parsers/unit3d.ts @@ -0,0 +1,13 @@ +/** UNIT3D 用户信息 fixture(自行编写的最小结构,不代表真实站点;对应 adapters/src/sites/user-parsers/unit3d.ts)。 */ +export const UNIT3D_USER_INDEX = ``; + +export const UNIT3D_USER_DETAIL = `
Elite +

Registration date 2020-02-03 04:05:06

`; + +export const UNIT3D_USER_SEEDING = ` +
NameSeedersSize
X93.00 GiB
Y11.00 GiB
`; diff --git a/packages/testkit/src/sites/fixtures/user-parsers/yema.ts b/packages/testkit/src/sites/fixtures/user-parsers/yema.ts new file mode 100644 index 00000000..36c6fca3 --- /dev/null +++ b/packages/testkit/src/sites/fixtures/user-parsers/yema.ts @@ -0,0 +1,15 @@ +/** YemaPT 用户信息 fixture(自行编写的最小结构,不代表真实站点;对应 adapters/src/sites/user-parsers/yema.ts)。 */ +export const YEMA_USER_BASIC = { + success: true, + data: { + id: 321, + name: "leo", + level: 3, + registerTime: "2024-05-06 07:08:09", + promotionUploadSize: 3000, + promotionDownloadSize: 1000, + bonus: 12.3, + }, +}; + +export const YEMA_USER_ERROR = { success: false, errorMessage: "AuthKey 无效" }; diff --git a/packages/testkit/src/sites/fixtures/user-parsers/zhixing.ts b/packages/testkit/src/sites/fixtures/user-parsers/zhixing.ts new file mode 100644 index 00000000..9c7fe35e --- /dev/null +++ b/packages/testkit/src/sites/fixtures/user-parsers/zhixing.ts @@ -0,0 +1,22 @@ +/** + * 知行(Zhixing)用户信息 fixture(自行编写,不代表真实页面)。 + * 对应解析器:adapters/src/sites/user-parsers/zhixing.ts(MP zhixing.py)。 + */ +export const ZHIXING_USER_INDEX = ``; + +export const ZHIXING_USER_BASIC = `
+

UID：321

用户名：zed

用户组：Lv2 (查看权限)

+

注册时间：2023-01-01 10:00:00 (2年前)

+

上传流量：1.5 TB

下载流量：123456

+

共享率：12.5

保种积分：88.8

+

当前保种数量：7

当前保种容量：70 GB

+
`; + +export const ZHIXING_USER_SEEDING_P1 = ` + + +
A10 GB
B20 GB
`; + +export const ZHIXING_USER_SEEDING_EMPTY = `
没有种子
`; + +export const ZHIXING_LOGIN_PAGE = `
`; diff --git a/packages/testkit/src/sites/index.ts b/packages/testkit/src/sites/index.ts index 45368b34..10a9724d 100644 --- a/packages/testkit/src/sites/index.ts +++ b/packages/testkit/src/sites/index.ts @@ -1,10 +1,20 @@ +export * from "./fixtures/definitions/index.ts"; +export * from "./fixtures/spiders/index.ts"; +export * from "./fixtures/user-parsers/index.ts"; +export { + createMockFetch, + type MockFetch, + type MockFetchRequest, + type MockRouteHandler, + type MockRouteResponse, +} from "./mock-fetch.ts"; export { createMockMteamApi, type MockMteamApi, type MockMteamFailMode, type MockMteamOptions, -} from "./mock-mteam"; -export { createMockNexusPhpSite, type MockNexusPhpOptions, type MockNexusPhpSite } from "./mock-nexusphp"; -export { type MockMteamTorrent, MTEAM_FIXTURE_TORRENTS } from "./mteam-fixtures"; -export { AUDIENCES_FIXTURE_RELEASES, type MockNexusPromo, type MockNexusRelease } from "./nexusphp-fixtures"; -export { buildFakeTorrent, type FakeTorrent } from "./torrent"; +} from "./mock-mteam.ts"; +export { createMockNexusPhpSite, type MockNexusPhpOptions, type MockNexusPhpSite } from "./mock-nexusphp.ts"; +export { type MockMteamTorrent, MTEAM_FIXTURE_TORRENTS } from "./mteam-fixtures.ts"; +export { AUDIENCES_FIXTURE_RELEASES, type MockNexusPromo, type MockNexusRelease } from "./nexusphp-fixtures.ts"; +export { buildFakeTorrent, type FakeTorrent } from "./torrent.ts"; diff --git a/packages/testkit/src/sites/mock-fetch.ts b/packages/testkit/src/sites/mock-fetch.ts new file mode 100644 index 00000000..4861e7bd --- /dev/null +++ b/packages/testkit/src/sites/mock-fetch.ts @@ -0,0 +1,97 @@ +/** + * 路由表式 fetch 替身(站点用户信息解析器 / API spider 测试用)。 + * 路由键:`METHOD path?query`(query 可省略,省略时匹配任意 query;METHOD 省略为任意方法); + * 值为响应描述或函数。未命中返回 404。记录全部请求(含头与 body)供断言。 + */ + +export interface MockRouteResponse { + status?: number; + body?: string | object; + contentType?: string; + headers?: Record; +} + +export type MockRouteHandler = MockRouteResponse | ((req: MockFetchRequest) => MockRouteResponse); + +export interface MockFetchRequest { + method: string; + url: URL; + headers: Record; + body: string | null; +} + +export interface MockFetch { + fetch: typeof globalThis.fetch; + requests: MockFetchRequest[]; + /** 追加/覆盖路由 */ + route(key: string, handler: MockRouteHandler): void; +} + +function match(key: string, req: MockFetchRequest): boolean { + let method: string | null = null; + let target = key; + const sp = key.indexOf(" "); + if (sp > 0 && /^[A-Z]+$/.test(key.slice(0, sp))) { + method = key.slice(0, sp); + target = key.slice(sp + 1); + } + if (method && method !== req.method) return false; + // 可带主机:"https://api.x.com/path" + let host: string | null = null; + if (/^https?:\/\//.test(target)) { + const u = new URL(target); + host = u.host; + target = u.pathname + u.search; + } + if (host && host !== req.url.host) return false; + const [path, query] = target.split("?"); + if (path !== req.url.pathname) return false; + if (query === undefined) return true; + const want = new URLSearchParams(query); + for (const [k, v] of want) if (req.url.searchParams.get(k) !== v) return false; + return true; +} + +export function createMockFetch(routes: Record = {}): MockFetch { + const table = new Map(Object.entries(routes)); + const requests: MockFetchRequest[] = []; + const fetchImpl = async (input: string | URL | Request, init?: RequestInit): Promise => { + const url = new URL(typeof input === "string" ? input : input instanceof URL ? input.href : input.url); + const headers: Record = {}; + new Headers(init?.headers).forEach((v, k) => { + headers[k] = v; + }); + const req: MockFetchRequest = { + method: (init?.method ?? "GET").toUpperCase(), + url, + headers, + body: typeof init?.body === "string" ? init.body : null, + }; + requests.push(req); + // 后加入的路由优先,便于测试覆盖默认路由;同时更具体(带 query)的键优先 + const keys = [...table.keys()].reverse().sort((a, b) => Number(b.includes("?")) - Number(a.includes("?"))); + for (const key of keys) { + if (!match(key, req)) continue; + const h = table.get(key)!; + const r = typeof h === "function" ? h(req) : h; + const isJson = r.body !== undefined && typeof r.body !== "string"; + const body = r.body === undefined ? "" : isJson ? JSON.stringify(r.body) : (r.body as string); + return new Response(body, { + status: r.status ?? 200, + headers: { + "content-type": r.contentType ?? (isJson ? "application/json" : "text/html; charset=utf-8"), + ...(r.headers ?? {}), + }, + }); + } + return new Response("not found", { status: 404 }); + }; + return { + fetch: fetchImpl as typeof globalThis.fetch, + requests, + route(key, handler) { + table.delete(key); + table.set(key, handler); + }, + }; +} diff --git a/packages/testkit/src/sites/mock-mteam.ts b/packages/testkit/src/sites/mock-mteam.ts index 1edd90af..4d054813 100644 --- a/packages/testkit/src/sites/mock-mteam.ts +++ b/packages/testkit/src/sites/mock-mteam.ts @@ -1,8 +1,8 @@ import { randomBytes } from "node:crypto"; import type { ServerResponse } from "node:http"; -import { type StartedServer, startHttpServer } from "../rss/server"; -import { type MockMteamTorrent, MTEAM_FIXTURE_TORRENTS } from "./mteam-fixtures"; -import { buildFakeTorrent } from "./torrent"; +import { type StartedServer, startHttpServer } from "../rss/server.ts"; +import { type MockMteamTorrent, MTEAM_FIXTURE_TORRENTS } from "./mteam-fixtures.ts"; +import { buildFakeTorrent } from "./torrent.ts"; export type MockMteamFailMode = "banned_key" | "rate_limit" | "http_429" | "non_json" | "no_dl_token"; diff --git a/packages/testkit/src/sites/mock-nexusphp.ts b/packages/testkit/src/sites/mock-nexusphp.ts index 68864d17..62263a84 100644 --- a/packages/testkit/src/sites/mock-nexusphp.ts +++ b/packages/testkit/src/sites/mock-nexusphp.ts @@ -1,7 +1,7 @@ import type { IncomingMessage, ServerResponse } from "node:http"; -import { type StartedServer, startHttpServer } from "../rss/server"; -import { AUDIENCES_FIXTURE_RELEASES, type MockNexusRelease } from "./nexusphp-fixtures"; -import { buildFakeTorrent } from "./torrent"; +import { type StartedServer, startHttpServer } from "../rss/server.ts"; +import { AUDIENCES_FIXTURE_RELEASES, type MockNexusRelease } from "./nexusphp-fixtures.ts"; +import { buildFakeTorrent } from "./torrent.ts"; export interface MockNexusPhpOptions { /** 登录态 Cookie;请求 Cookie 头需包含它 */ diff --git a/scripts/mp-sync/README.md b/scripts/mp-sync/README.md new file mode 100644 index 00000000..6b6ff69b --- /dev/null +++ b/scripts/mp-sync/README.md @@ -0,0 +1,17 @@ +# scripts/mp-sync + +跟随上游移植工具(说明与对照总表:`docs/porting/mp-sync-ledger.md`)。Node 24 或 `deno run -A` 均可运行,不依赖根 package.json / deno.json 任务。 + +| 脚本 | 作用 | +| --- | --- | +| `check.mjs [--mp ] [--to <提交>] [--json] [--ledger <文件>]` | 比较 `ledger.json` 与 MP 目标提交:已变更 / 已删除 / `spider/`、`parser/` 新增文件,输出 `git diff` 命令;有变化退出码 1 | +| `build-ledger.mjs [--mp ] [--commit <提交>]` | 扫描源码 `@mp-source <路径> @ <提交>` 标签 + `ledger.manual.json`,生成 `ledger.json`(含上游文件 sha256)并刷新文档表格 | +| `movieclaw-import.mjs [--repo ] [--ref v0.27.0] [--check]` | MovieClaw v0.27.0(MIT)站点 YAML → `packages/adapters/src/sites/definitions/*.json` + `manifest.generated.ts`;覆盖层 `movieclaw-overrides/.json`;`--check` 只比较 | + +```bash +node scripts/mp-sync/check.mjs --to <新提交> +node scripts/mp-sync/build-ledger.mjs --commit <新提交> +node scripts/mp-sync/movieclaw-import.mjs --check +``` + +MovieClaw v0.27.0 之后为附条件许可:`movieclaw-import.mjs` 校验 ref→提交与 LICENSE=MIT,其他 ref 需 `--allow-ref` 并另行核对许可。 diff --git a/scripts/mp-sync/build-ledger.mjs b/scripts/mp-sync/build-ledger.mjs new file mode 100644 index 00000000..f47e3d05 --- /dev/null +++ b/scripts/mp-sync/build-ledger.mjs @@ -0,0 +1,88 @@ +#!/usr/bin/env node +/** + * 生成 scripts/mp-sync/ledger.json 与 docs/porting/mp-sync-ledger.md 中的对照总表。 + * + * node scripts/mp-sync/build-ledger.mjs [--mp /opt/moviepilot-plus/upstream/MoviePilot] [--commit e6820a8] + * + * 数据来源:源码文件头 `@mp-source <上游路径> @ <提交>` 标签 + ledger.manual.json(未打标签的既有移植)。 + * 每条记录保存移植时的上游提交与该文件内容 sha256,供 check.mjs 比较。 + * 增量移植完成后:把对应源码文件头的提交号改为新提交,再重跑本脚本。 + */ +import { readFileSync, writeFileSync } from "node:fs"; +import { + biomeFormat, + DEFAULT_MP_REPO, + DOC_PATH, + LEDGER_PATH, + MANUAL_PATH, + parseArgs, + resolveCommit, + scanTags, + upstreamHash, + WATCH_DIRS, +} from "./lib.mjs"; + +const args = parseArgs(process.argv.slice(2)); +const repo = args.mp ?? DEFAULT_MP_REPO; +const defaultCommit = resolveCommit(repo, args.commit ?? "e6820a8"); + +const byUpstream = new Map(); +function add(upstream, commitRef, ours, source) { + const commit = resolveCommit(repo, commitRef); + const key = upstream; + const e = byUpstream.get(key) ?? { upstream, commit, sha256: null, ours: [], sources: new Set() }; + if (e.commit !== commit) + throw new Error(`${upstream} 在不同文件中登记了不同提交:${e.commit.slice(0, 7)} / ${commit.slice(0, 7)}`); + for (const o of ours) if (!e.ours.includes(o)) e.ours.push(o); + e.sources.add(source); + byUpstream.set(key, e); +} + +for (const t of scanTags()) add(t.upstream, t.commit, [t.ours], "tag"); +const manual = JSON.parse(readFileSync(MANUAL_PATH, "utf8")); +for (const m of manual.moviepilot ?? []) add(m.upstream, m.commit ?? defaultCommit, m.ours, "manual"); + +const entries = [...byUpstream.values()] + .map((e) => { + const h = upstreamHash(repo, e.commit, e.upstream); + if (!h) throw new Error(`上游 ${e.commit.slice(0, 7)}:${e.upstream} 不存在`); + return { + upstream: e.upstream, + commit: e.commit, + sha256: h, + ours: e.ours.sort(), + source: [...e.sources].sort().join("+"), + }; + }) + .sort((a, b) => a.upstream.localeCompare(b.upstream)); + +const ledger = { + $comment: "由 scripts/mp-sync/build-ledger.mjs 生成;不要手改(改源码文件头 @mp-source 标签或 ledger.manual.json)。", + upstream: { name: "MoviePilot", url: "https://github.com/jxxghp/MoviePilot", branch: "v3", license: "GPL-3.0" }, + watchDirs: WATCH_DIRS, + entries, +}; +writeFileSync(LEDGER_PATH, biomeFormat(`${JSON.stringify(ledger, null, 2)}\n`, "scripts/mp-sync/ledger.json")); + +// 更新文档对照总表(标记之间的内容) +const BEGIN = ""; +const END = ""; +const rows = entries.map( + (e) => + `| \`${e.upstream}\` | ${e.ours.map((o) => `\`${o.replace(/^packages\/adapters\/src\//, "adapters/").replace(/^packages\/testkit\/src\//, "testkit/")}\``).join("
")} | \`${e.commit.slice(0, 7)}\` | \`${e.sha256.slice(0, 12)}\` | ${e.source} |`, +); +const table = [ + BEGIN, + `共 ${entries.length} 个上游文件(\`adapters/\` = \`packages/adapters/src/\`)。`, + "", + "| 上游文件 | 本项目文件 | 移植提交 | sha256(前 12 位) | 来源 |", + "| --- | --- | --- | --- | --- |", + ...rows, + END, +].join("\n"); +const doc = readFileSync(DOC_PATH, "utf8"); +const i = doc.indexOf(BEGIN); +const j = doc.indexOf(END); +if (i < 0 || j < 0) throw new Error(`${DOC_PATH} 缺少表格标记`); +writeFileSync(DOC_PATH, doc.slice(0, i) + table + doc.slice(j + END.length)); +console.log(`ledger.json:${entries.length} 个上游文件(${defaultCommit.slice(0, 7)});文档表格已更新`); diff --git a/scripts/mp-sync/check.mjs b/scripts/mp-sync/check.mjs new file mode 100644 index 00000000..f0fa1db5 --- /dev/null +++ b/scripts/mp-sync/check.mjs @@ -0,0 +1,66 @@ +#!/usr/bin/env node +/** + * 比较 ledger.json 与 MoviePilot 目标提交,列出需要跟随移植的上游变化。 + * + * node scripts/mp-sync/check.mjs [--mp ] [--to <目标提交,默认 HEAD>] [--json] + * + * 输出:已变更 / 已删除 / 未变化 的 ledger 文件,监视目录(spider/、parser/)中 ledger 未收录的新文件, + * 以及逐文件 `git diff` 命令。存在变化时退出码为 1(可用于 CI 或定时提醒)。 + * 浅克隆需要先取目标提交:git -C fetch --depth=1 origin (或 fetch --unshallow)。 + */ +import { readFileSync } from "node:fs"; +import { DEFAULT_MP_REPO, LEDGER_PATH, listUpstreamDir, parseArgs, resolveCommit, upstreamHash } from "./lib.mjs"; + +const args = parseArgs(process.argv.slice(2)); +const repo = args.mp ?? DEFAULT_MP_REPO; +const target = resolveCommit(repo, args.to ?? "HEAD"); +const ledger = JSON.parse(readFileSync(args.ledger ?? LEDGER_PATH, "utf8")); + +const changed = []; +const deleted = []; +const unchanged = []; +for (const e of ledger.entries) { + const h = upstreamHash(repo, target, e.upstream); + if (h === null) deleted.push(e); + else if (h !== e.sha256) changed.push({ ...e, newSha256: h }); + else unchanged.push(e); +} +const known = new Set(ledger.entries.map((e) => e.upstream)); +const added = []; +for (const dir of ledger.watchDirs ?? []) { + for (const f of listUpstreamDir(repo, target, dir)) if (!known.has(f)) added.push(f); +} + +const short = (c) => c.slice(0, 7); +if (args.json) { + console.log(JSON.stringify({ target, changed, deleted, added, unchanged: unchanged.length }, null, 2)); +} else { + console.log(`MoviePilot 目标提交 ${short(target)}(ledger ${ledger.entries.length} 个文件)\n`); + const section = (title, list, fmt) => { + console.log(`## ${title}(${list.length})`); + for (const x of list) console.log(fmt(x)); + console.log(""); + }; + section("已变更", changed, (e) => + [ + `- ${e.upstream} (${short(e.commit)} → ${short(target)})`, + ` 本项目:${e.ours.join(", ")}`, + ` git -C ${repo} diff ${short(e.commit)} ${short(target)} -- ${e.upstream}`, + ].join("\n"), + ); + section("已删除 / 改名", deleted, (e) => + [ + `- ${e.upstream}(本项目:${e.ours.join(", ")})`, + ` git -C ${repo} log --oneline --follow ${short(e.commit)}..${short(target)} -- ${e.upstream}`, + ` git -C ${repo} diff --stat -M ${short(e.commit)} ${short(target)} -- ${e.upstream.replace(/\/[^/]+$/, "/")}`, + ].join("\n"), + ); + section("监视目录新增(ledger 未收录)", added, (f) => `- ${f}\n git -C ${repo} show ${short(target)}:${f}`); + console.log(`未变化:${unchanged.length}`); + if (changed.length || deleted.length || added.length) { + console.log( + "\n处理流程:按 diff 增量移植 → 把对应源码文件头 `@mp-source ... @ <提交>` 改为新提交(新文件新增标签)→ node scripts/mp-sync/build-ledger.mjs --commit <新提交> → 测试。", + ); + } +} +process.exit(changed.length || deleted.length || added.length ? 1 : 0); diff --git a/scripts/mp-sync/ledger.json b/scripts/mp-sync/ledger.json new file mode 100644 index 00000000..99736124 --- /dev/null +++ b/scripts/mp-sync/ledger.json @@ -0,0 +1,385 @@ +{ + "$comment": "由 scripts/mp-sync/build-ledger.mjs 生成;不要手改(改源码文件头 @mp-source 标签或 ledger.manual.json)。", + "upstream": { + "name": "MoviePilot", + "url": "https://github.com/jxxghp/MoviePilot", + "branch": "v3", + "license": "GPL-3.0" + }, + "watchDirs": ["app/modules/indexer/spider", "app/modules/indexer/parser"], + "entries": [ + { + "upstream": "app/adapters/network/cloudflare.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "a23a2c44da051cd36f5dca87fe834a9518fee2019a1088aa2eecd88b3cafcca4", + "ours": ["packages/adapters/src/sites/user-parsers/common.ts"], + "source": "tag" + }, + { + "upstream": "app/application/mediaserver.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "f034f0be8ba656e3b6d8148f467f39010f1e2fb51c21565ffef1b0ae8e088eac", + "ours": ["packages/adapters/src/emby/client.ts"], + "source": "manual" + }, + { + "upstream": "app/application/rss.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "8ea70f712512486dca06a96da2d352f832ef1e1dbb430c815245bf13058f05eb", + "ours": [ + "packages/adapters/src/rss/fetch.ts", + "packages/adapters/src/rss/parse.ts", + "packages/adapters/src/rss/rss-link.ts" + ], + "source": "manual" + }, + { + "upstream": "app/application/site/sites.pyi", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "8fea9a239354f2b7eebeaf912a569ce0fa161e18b9483c49a802d828c5b10c51", + "ours": [ + "packages/adapters/src/sites/definitions/schema.ts", + "packages/adapters/src/sites/registry.ts", + "packages/adapters/src/sites/spider/schema.ts" + ], + "source": "manual" + }, + { + "upstream": "app/chain/download/submission.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "f7cb870e47d644c9a85c6c62e9284f5b3405da09148540bf1eec55479b7212c9", + "ours": [ + "packages/adapters/src/sites/mteam/adapter.ts", + "packages/adapters/src/sites/spider/adapter.ts", + "packages/adapters/src/sites/spiders/common.ts" + ], + "source": "manual+tag" + }, + { + "upstream": "app/domain/site.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "650c795a4def0a11317572c9210aa111c51e564973928d3af3fce8ce2da66107", + "ours": ["packages/adapters/src/sites/user-parsers/common.ts"], + "source": "tag" + }, + { + "upstream": "app/foundation/dom.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "6d6accf2cef132e572aabb15da071ee355a3a6b02957fcb9c2e0c4082f863f4e", + "ours": ["packages/adapters/src/sites/user-parsers/dom.ts"], + "source": "tag" + }, + { + "upstream": "app/foundation/size.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "7b0b8f50588cbf9b827be2cfbe00044f493ea329730de8b61526cf159b5a1600", + "ours": ["packages/adapters/src/sites/spider/spider.ts"], + "source": "manual" + }, + { + "upstream": "app/foundation/temporal.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "74941f460e4069f45cae85636ded26366f80f9869438b2ff0ca092ef5dddb892", + "ours": ["packages/adapters/src/sites/spiders/common.ts", "packages/adapters/src/sites/user-parsers/common.ts"], + "source": "tag" + }, + { + "upstream": "app/foundation/text.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "a88c75dc843131cb4a71e041b59386bea1defd332c77087978be854b56177961", + "ours": ["packages/adapters/src/sites/user-parsers/common.ts"], + "source": "tag" + }, + { + "upstream": "app/modules/__init__.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "2edada0ea59108b0e69913ae631beffa3f85f7d4675502cc790696b26cf47ab6", + "ours": ["packages/adapters/src/qbittorrent/paths.ts"], + "source": "manual" + }, + { + "upstream": "app/modules/_base/downloader.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "84bd8946c8b26dd2bb7caa4983d4407c6a9b28385ff28df947fe412c3d49c3b5", + "ours": ["packages/adapters/src/qbittorrent/torrent.ts"], + "source": "manual" + }, + { + "upstream": "app/modules/emby/emby.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "e6468ea34ef2d6beec7219bcf0d5d309578cc0b65c896b91e1a8b6e65967ed6c", + "ours": ["packages/adapters/src/emby/client.ts"], + "source": "manual" + }, + { + "upstream": "app/modules/filemanager/storages/local.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "806a1970ca8eb1f500d5273f62e0a5eaa834568cdb1af7d7c5dd92fe0c3d1702", + "ours": ["packages/adapters/src/fs/copy.ts", "packages/adapters/src/fs/move.ts"], + "source": "manual" + }, + { + "upstream": "app/modules/filemanager/transhandler.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "13e2bba491acfde28e1f6ed083a639f306a3bbaac585732b994eccae925b9bad", + "ours": ["packages/adapters/src/fs/move.ts"], + "source": "manual" + }, + { + "upstream": "app/modules/indexer/__init__.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "e9c5d15fdaa53205e15210d1f19019b9b82716affe075f6b268f6a044d676641", + "ours": [ + "packages/adapters/src/sites/registry.ts", + "packages/adapters/src/sites/spider/adapter.ts", + "packages/adapters/src/sites/spiders/adapter.ts", + "packages/adapters/src/sites/spiders/index.ts", + "packages/adapters/src/sites/spiders/map.ts", + "packages/adapters/src/sites/spiders/types.ts", + "packages/adapters/src/sites/user-parsers/registry.ts" + ], + "source": "manual+tag" + }, + { + "upstream": "app/modules/indexer/parser/__init__.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "0e6c4215b713493a4efdedfb7f42ff9418ec297adc2567b4f596ebaec8120ea3", + "ours": ["packages/adapters/src/sites/user-parsers/base.ts", "packages/adapters/src/sites/user-parsers/types.ts"], + "source": "tag" + }, + { + "upstream": "app/modules/indexer/parser/bitpt.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "871d25e570f1fadcd8f214d8ae9f3b07653a053337a5e14a487a96c99a8da3a0", + "ours": ["packages/adapters/src/sites/user-parsers/bitpt.ts"], + "source": "tag" + }, + { + "upstream": "app/modules/indexer/parser/discuz.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "12277ba955ce7263c4b1862749443878c0900350ea8d5eb970bb57e1249c279d", + "ours": ["packages/adapters/src/sites/user-parsers/discuz.ts"], + "source": "tag" + }, + { + "upstream": "app/modules/indexer/parser/file_list.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "f92e65ed73278c8e68fdd86927cb307e418564580daf8fded3e7a467d7bea7ac", + "ours": ["packages/adapters/src/sites/user-parsers/file-list.ts"], + "source": "tag" + }, + { + "upstream": "app/modules/indexer/parser/gazelle.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "c10e5246724b51391fff60f90e17022c0fd5fa2a4151ae69d6c4ee9ad1c5f1c6", + "ours": ["packages/adapters/src/sites/user-parsers/gazelle.ts"], + "source": "tag" + }, + { + "upstream": "app/modules/indexer/parser/hddolby.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "a4336d92d603c06953bb957d166f354d7a847eda75d9eaf2bcc96212f6e58c56", + "ours": ["packages/adapters/src/sites/user-parsers/hddolby.ts"], + "source": "tag" + }, + { + "upstream": "app/modules/indexer/parser/ipt.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "2b9d04d76606329f32d7f726b5ae478997cdb7ee55f74ff89ab881c641e58bc0", + "ours": ["packages/adapters/src/sites/user-parsers/ipt.ts"], + "source": "tag" + }, + { + "upstream": "app/modules/indexer/parser/mtorrent.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "c5d5a3c6f9f5fa5a60e3d65426bdc6620e94e861e0cf27f718595afc61b86165", + "ours": ["packages/adapters/src/sites/user-parsers/mtorrent.ts"], + "source": "tag" + }, + { + "upstream": "app/modules/indexer/parser/nexus_audiences.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "1a2d104db7699c8dffbc79546a63dd40108e8fa171f1e7380e886521b0c3698f", + "ours": ["packages/adapters/src/sites/user-parsers/nexus-audiences.ts"], + "source": "tag" + }, + { + "upstream": "app/modules/indexer/parser/nexus_hhanclub.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "8850619069d4b9baaac2ef53fbdf5bac88b365b2e3009455ad40593c963c5ccb", + "ours": ["packages/adapters/src/sites/user-parsers/nexus-hhanclub.ts"], + "source": "tag" + }, + { + "upstream": "app/modules/indexer/parser/nexus_php.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "39303ae96d9cd198d0145e10539357ac17186be0127eaa92ada36ee3746296d2", + "ours": ["packages/adapters/src/sites/user-parsers/nexus-php.ts"], + "source": "tag" + }, + { + "upstream": "app/modules/indexer/parser/nexus_project.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "2b5508cfb2f89efa343515c5b91ae7af95fb9adf4624da6686c4f3b0004f07b0", + "ours": ["packages/adapters/src/sites/user-parsers/nexus-project.ts"], + "source": "tag" + }, + { + "upstream": "app/modules/indexer/parser/nexus_rabbit.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "09149b89a7c76b9880d34deedc7335aba3051a81910d333df237614a0a55a98f", + "ours": ["packages/adapters/src/sites/user-parsers/nexus-rabbit.ts"], + "source": "tag" + }, + { + "upstream": "app/modules/indexer/parser/rousi.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "1b1ea93de084c20b7001fc22994d87b4f99d5c04773ee25cb1482966fefca205", + "ours": ["packages/adapters/src/sites/user-parsers/rousi.ts"], + "source": "tag" + }, + { + "upstream": "app/modules/indexer/parser/small_horse.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "b73b5de97737ca60e46bfecfc580bad6d9c7ee573d01bf6002f0282bcdf229ab", + "ours": ["packages/adapters/src/sites/user-parsers/small-horse.ts"], + "source": "tag" + }, + { + "upstream": "app/modules/indexer/parser/sunnypt.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "937f8453fa2b73d06a690ff91e8939cd1478009b4b731f622d6585d9cdbc28cf", + "ours": ["packages/adapters/src/sites/user-parsers/sunnypt.ts"], + "source": "tag" + }, + { + "upstream": "app/modules/indexer/parser/tnode.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "c9cb809fc4bffc161f2595cbb6db97b919c83d437e3bdc29374e46e3c46ba74e", + "ours": ["packages/adapters/src/sites/user-parsers/tnode.ts"], + "source": "tag" + }, + { + "upstream": "app/modules/indexer/parser/torrent_leech.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "72d1473a3bba5398d1a283995b646f7c93683992c4fa9f6a0d5eb80272754b0b", + "ours": ["packages/adapters/src/sites/user-parsers/torrent-leech.ts"], + "source": "tag" + }, + { + "upstream": "app/modules/indexer/parser/unit3d.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "9a4d579deca593b445e80e560ed79c265e9b95cb631bef91e4dca859d020b912", + "ours": ["packages/adapters/src/sites/user-parsers/unit3d.ts"], + "source": "tag" + }, + { + "upstream": "app/modules/indexer/parser/yema.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "0f98fd580bad06edebaf81730deb5f11ad7c55ac5ae0d9ebc76ae50c5f828017", + "ours": ["packages/adapters/src/sites/user-parsers/yema.ts"], + "source": "tag" + }, + { + "upstream": "app/modules/indexer/parser/zhixing.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "90c8954e73876d9437220bf3a26c8f863d6b4773b54d97ae07321ee0c1b1abc5", + "ours": ["packages/adapters/src/sites/user-parsers/zhixing.ts"], + "source": "tag" + }, + { + "upstream": "app/modules/indexer/spider/__init__.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "2513d962fa437f9bf2ca06f5f908b7de8b9bb7b032f80eff283611af38a8b487", + "ours": [ + "packages/adapters/src/sites/spider/map.ts", + "packages/adapters/src/sites/spider/schema.ts", + "packages/adapters/src/sites/spider/spider.ts", + "packages/adapters/src/sites/spider/template.ts" + ], + "source": "manual" + }, + { + "upstream": "app/modules/indexer/spider/haidan.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "75eb802f19f02cf64ed2b7b981026e47369a2d815c6a666c5a9fd70aa86a598e", + "ours": ["packages/adapters/src/sites/spiders/haidan.ts"], + "source": "tag" + }, + { + "upstream": "app/modules/indexer/spider/hddolby.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "46a50bd8d49f83a21f59b6273b0378d2342feb78e7f2f5b953eb9d1311c38804", + "ours": ["packages/adapters/src/sites/spiders/hddolby.ts"], + "source": "tag" + }, + { + "upstream": "app/modules/indexer/spider/mtorrent.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "a8ab2b822cb24a22c16a8db01e0ef68985435a25b5332d741992f9ff99c10de5", + "ours": [ + "packages/adapters/src/sites/mteam/adapter.ts", + "packages/adapters/src/sites/mteam/client.ts", + "packages/adapters/src/sites/mteam/spider.ts", + "packages/adapters/src/sites/spiders/mtorrent.ts" + ], + "source": "manual+tag" + }, + { + "upstream": "app/modules/indexer/spider/rousi.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "aa13c386a560a97125ac3879b56187c50564d544e925ad9f221bbbefc15e712a", + "ours": ["packages/adapters/src/sites/spiders/rousi.ts"], + "source": "tag" + }, + { + "upstream": "app/modules/indexer/spider/sunnypt.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "735b0d721e65ec4006d1ea309b1815635caa7e69e1897643faaa6ca8b84897b8", + "ours": ["packages/adapters/src/sites/spiders/sunnypt.ts"], + "source": "tag" + }, + { + "upstream": "app/modules/indexer/spider/tnode.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "a3634a268dbad6cfe45a5c70606b94a4c2fe8f9b351d9b7d6c6499f4a8107fe3", + "ours": ["packages/adapters/src/sites/spiders/tnode.ts"], + "source": "tag" + }, + { + "upstream": "app/modules/indexer/spider/torrentleech.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "c965f2bcaea4a50335bcb6e670a9ddb3a96a3f62f59bf3f93deff9cd9e7e28e5", + "ours": ["packages/adapters/src/sites/spiders/torrentleech.ts"], + "source": "tag" + }, + { + "upstream": "app/modules/indexer/spider/yema.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "560b62c1ff102d3efa47ad9dc31ba1a5794eae804fc320dbb06f08b72a004f3f", + "ours": ["packages/adapters/src/sites/spiders/yema.ts"], + "source": "tag" + }, + { + "upstream": "app/modules/qbittorrent/qbittorrent.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "ab02e97e986f5d4d5441419364ee1e04411f9c9b77323c713e059301d9746390", + "ours": ["packages/adapters/src/qbittorrent/client.ts", "packages/adapters/src/qbittorrent/state.ts"], + "source": "manual" + }, + { + "upstream": "app/modules/themoviedb/tmdbapi.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "787553a6f57d8b21e04442d100e098694b9ff7835a2f8d7541b53511606caff8", + "ours": ["packages/adapters/src/tmdb/client.ts", "packages/adapters/src/tmdb/match.ts"], + "source": "manual" + }, + { + "upstream": "app/schemas/site.py", + "commit": "e6820a83ab05e8a00eb0e14adf0f357faa965c7e", + "sha256": "e7952ed40b7b5a66d95fe1b07e1d3915601b47c2936ff10ec6880c8a030d0b8f", + "ours": ["packages/adapters/src/sites/user-parsers/types.ts"], + "source": "tag" + } + ] +} diff --git a/scripts/mp-sync/ledger.manual.json b/scripts/mp-sync/ledger.manual.json new file mode 100644 index 00000000..a4b33e6c --- /dev/null +++ b/scripts/mp-sync/ledger.manual.json @@ -0,0 +1,64 @@ +{ + "$comment": "未在文件头打 @mp-source 标签的既有移植(W1-3a/W1-3b 及 sites 目录外文件)的人工对照;build-ledger.mjs 与源码标签合并生成 ledger.json。", + "moviepilot": [ + { + "upstream": "app/modules/indexer/spider/__init__.py", + "ours": [ + "packages/adapters/src/sites/spider/spider.ts", + "packages/adapters/src/sites/spider/schema.ts", + "packages/adapters/src/sites/spider/map.ts", + "packages/adapters/src/sites/spider/template.ts" + ] + }, + { + "upstream": "app/modules/indexer/spider/mtorrent.py", + "ours": [ + "packages/adapters/src/sites/mteam/spider.ts", + "packages/adapters/src/sites/mteam/adapter.ts", + "packages/adapters/src/sites/mteam/client.ts" + ] + }, + { + "upstream": "app/modules/indexer/__init__.py", + "ours": ["packages/adapters/src/sites/spider/adapter.ts", "packages/adapters/src/sites/registry.ts"] + }, + { + "upstream": "app/chain/download/submission.py", + "ours": ["packages/adapters/src/sites/spider/adapter.ts", "packages/adapters/src/sites/mteam/adapter.ts"] + }, + { + "upstream": "app/application/site/sites.pyi", + "ours": [ + "packages/adapters/src/sites/spider/schema.ts", + "packages/adapters/src/sites/definitions/schema.ts", + "packages/adapters/src/sites/registry.ts" + ] + }, + { "upstream": "app/foundation/size.py", "ours": ["packages/adapters/src/sites/spider/spider.ts"] }, + { + "upstream": "app/application/rss.py", + "ours": [ + "packages/adapters/src/rss/fetch.ts", + "packages/adapters/src/rss/parse.ts", + "packages/adapters/src/rss/rss-link.ts" + ] + }, + { + "upstream": "app/modules/qbittorrent/qbittorrent.py", + "ours": ["packages/adapters/src/qbittorrent/client.ts", "packages/adapters/src/qbittorrent/state.ts"] + }, + { "upstream": "app/modules/_base/downloader.py", "ours": ["packages/adapters/src/qbittorrent/torrent.ts"] }, + { "upstream": "app/modules/__init__.py", "ours": ["packages/adapters/src/qbittorrent/paths.ts"] }, + { + "upstream": "app/modules/themoviedb/tmdbapi.py", + "ours": ["packages/adapters/src/tmdb/client.ts", "packages/adapters/src/tmdb/match.ts"] + }, + { + "upstream": "app/modules/filemanager/storages/local.py", + "ours": ["packages/adapters/src/fs/copy.ts", "packages/adapters/src/fs/move.ts"] + }, + { "upstream": "app/modules/filemanager/transhandler.py", "ours": ["packages/adapters/src/fs/move.ts"] }, + { "upstream": "app/modules/emby/emby.py", "ours": ["packages/adapters/src/emby/client.ts"] }, + { "upstream": "app/application/mediaserver.py", "ours": ["packages/adapters/src/emby/client.ts"] } + ] +} diff --git a/scripts/mp-sync/lib.mjs b/scripts/mp-sync/lib.mjs new file mode 100644 index 00000000..1a8ca907 --- /dev/null +++ b/scripts/mp-sync/lib.mjs @@ -0,0 +1,106 @@ +// mp-sync 公共函数(Node 24 / Deno 2 均可运行,无第三方依赖) +import { execFileSync } from "node:child_process"; +import { createHash } from "node:crypto"; +import { existsSync, readdirSync, readFileSync, statSync } from "node:fs"; +import { dirname, join, relative, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; + +export const ROOT = resolve(dirname(fileURLToPath(import.meta.url)), "../.."); +export const LEDGER_PATH = join(ROOT, "scripts/mp-sync/ledger.json"); +export const MANUAL_PATH = join(ROOT, "scripts/mp-sync/ledger.manual.json"); +export const DOC_PATH = join(ROOT, "docs/porting/mp-sync-ledger.md"); +export const DEFAULT_MP_REPO = "/opt/moviepilot-plus/upstream/MoviePilot"; +/** 需要监视"新增文件"的上游目录(MP 开源站点框架) */ +export const WATCH_DIRS = ["app/modules/indexer/spider", "app/modules/indexer/parser"]; +/** 扫描 @mp-source 标签的本项目目录 */ +export const SCAN_DIRS = ["packages/adapters/src", "packages/testkit/src"]; + +export function parseArgs(argv) { + const out = { _: [] }; + for (let i = 0; i < argv.length; i++) { + const a = argv[i]; + if (a.startsWith("--")) { + const k = a.slice(2); + const v = argv[i + 1] && !argv[i + 1].startsWith("--") ? argv[++i] : true; + out[k] = v; + } else out._.push(a); + } + return out; +} + +export function git(repo, ...args) { + return execFileSync("git", ["-C", repo, ...args], { + encoding: "buffer", + maxBuffer: 256 * 1024 * 1024, + stdio: ["ignore", "pipe", "ignore"], + }); +} + +export function tryGit(repo, ...args) { + try { + return git(repo, ...args); + } catch { + return null; + } +} + +export function sha256(buf) { + return createHash("sha256").update(buf).digest("hex"); +} + +export function resolveCommit(repo, ref) { + const out = tryGit(repo, "rev-parse", "--verify", `${ref}^{commit}`); + if (!out) throw new Error(`上游仓库 ${repo} 中找不到提交 ${ref}(浅克隆需先 git fetch --depth=1 origin )`); + return out.toString().trim(); +} + +/** 上游文件在某提交的内容 sha256;不存在返回 null */ +export function upstreamHash(repo, commit, path) { + const buf = tryGit(repo, "show", `${commit}:${path}`); + return buf ? sha256(buf) : null; +} + +export function listUpstreamDir(repo, commit, dir) { + const out = tryGit(repo, "ls-tree", "--name-only", `${commit}:${dir}`); + if (!out) return []; + return out + .toString() + .split("\n") + .filter((f) => f.endsWith(".py")) + .map((f) => `${dir}/${f}`); +} + +function walk(dir, acc) { + for (const name of readdirSync(dir)) { + if (name === "node_modules") continue; + const p = join(dir, name); + if (statSync(p).isDirectory()) walk(p, acc); + else if (/\.(ts|mjs)$/.test(name)) acc.push(p); + } + return acc; +} + +/** 扫描源码文件头 `@mp-source @ ` 标签 */ +export function scanTags() { + const tags = []; + for (const d of SCAN_DIRS) { + for (const file of walk(join(ROOT, d), [])) { + const text = readFileSync(file, "utf8"); + for (const m of text.matchAll(/@mp-source\s+(\S+)\s+@\s+([0-9a-f]{7,40})/g)) { + tags.push({ upstream: m[1], commit: m[2], ours: relative(ROOT, file) }); + } + } + } + return tags; +} + +/** 用仓库 biome 格式化(与 lint 一致,生成物重跑幂等);不可用时原样返回 */ +export function biomeFormat(text, fakePath) { + const bin = join(ROOT, "node_modules/.bin/biome"); + if (!existsSync(bin)) return text; + try { + return execFileSync(bin, ["format", `--stdin-file-path=${fakePath}`], { input: text, encoding: "utf8", cwd: ROOT }); + } catch { + return text; + } +} diff --git a/scripts/mp-sync/movieclaw-import.mjs b/scripts/mp-sync/movieclaw-import.mjs new file mode 100644 index 00000000..4dde23bc --- /dev/null +++ b/scripts/mp-sync/movieclaw-import.mjs @@ -0,0 +1,468 @@ +#!/usr/bin/env node +/** + * MovieClaw v0.27.0(MIT)站点 YAML → 本项目 SiteDefinition JSON 转换器(可重复执行)。 + * + * 用法: + * node scripts/mp-sync/movieclaw-import.mjs [--repo spikes/movieclaw-src/repo] [--ref v0.27.0] [--check] + * + * - 只用 `git show :` 读取文件(固定 tag,不读工作区,避免混入许可切换后的内容)。 + * - 校验 解析到的提交与 LICENSE 为 MIT;ref 不是 v0.27.0 时要求 --allow-ref(许可需另行核对)。 + * - 输出:packages/adapters/src/sites/definitions/.json 与 manifest.generated.ts。 + * - 手工覆盖层:scripts/mp-sync/movieclaw-overrides/.json 深合并到转换结果之上(覆盖层优先; + * `torrents.fields.` 与数组整体替换)。 + * - `--check`:只比较,不写文件;有差异时退出码 1(CI/跟随更新用)。 + * + * 选择器默认值来自 MovieClaw `movieclaw_tracker/selectors.py` NexusPHPSelectors(MIT,见 + * packages/adapters/src/sites/definitions/NOTICE-MovieClaw.md)。 + */ +import { execFileSync } from "node:child_process"; +import { existsSync, readdirSync, readFileSync, writeFileSync } from "node:fs"; +import { createRequire } from "node:module"; +import { dirname, join, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; + +const ROOT = resolve(dirname(fileURLToPath(import.meta.url)), "../.."); + +/** yaml 解析器:Deno(根 deno.json import map)或 Node(node_modules)均可 */ +async function loadYaml() { + try { + return (await import("yaml")).default; + } catch { + for (const base of ["packages/adapters/package.json", "package.json"]) { + try { + return createRequire(join(ROOT, base))("yaml"); + } catch { + /* next */ + } + } + } + throw new Error("找不到 yaml 包:请用 `deno run -A scripts/mp-sync/movieclaw-import.mjs` 运行"); +} +const YAML = await loadYaml(); + +const args = process.argv.slice(2); +const opt = (name, def) => { + const i = args.indexOf(name); + return i >= 0 ? args[i + 1] : def; +}; +const REPO = resolve(ROOT, opt("--repo", "spikes/movieclaw-src/repo")); +const REF = opt("--ref", "v0.27.0"); +const CHECK = args.includes("--check"); +const EXPECTED = { "v0.27.0": "c79bb1b065c826823fc1c15bcb99b3c282c3fce0" }; +const OUT_DIR = join(ROOT, "packages/adapters/src/sites/definitions"); +const OVERRIDES = join(ROOT, "scripts/mp-sync/movieclaw-overrides"); +const CONFIG_DIR = "src/movieclaw_tracker/sites/configs"; +/** 主力站:默认启用;其余 enabled:false(未实测) */ +const PRIMARY = new Set(["audiences", "hhanclub", "chdbits", "mteam"]); +/** MP 用户信息解析器 schema(app/modules/indexer/parser SiteSchema 值) */ +const MP_SCHEMA = { audiences: "NexusAudiences", hhanclub: "NexusHhanclub", mteam: "MTorrent", hddolby: "HDDolby" }; +const CATEGORY_LABELS = { + movie: "电影", + tv: "剧集", + documentary: "纪录片", + anime: "动漫", + music: "音乐", + game: "游戏", + av: "成人", + other: "其他", +}; +const TZ = { "Asia/Shanghai": "+08:00", "Asia/Hong_Kong": "+08:00", "Asia/Taipei": "+08:00", UTC: "+00:00" }; + +// ---- MovieClaw NexusPHPSelectors 默认值(v0.27.0 selectors.py) ---- +const MC_DEFAULTS = { + login_path: "login.php", + login_username_field: "username", + login_password_field: "password", + login_error_css: "td.text", + login_success_css: "a[href*='logout']", + login_extra_form_data: [], + login_select_defaults: [], + torrent_list_path: "torrents.php", + torrent_row_css: "table.torrents > tr:not(:first-child)", + torrent_id_css: "a[href*='details.php']::attr(href)", + torrent_title_css: "a[href*='details.php']", + torrent_subtitle_css: "td.embedded > span", + torrent_category_css: "td:nth-child(1) img::attr(class)", + torrent_size_css: "td:nth-child(5)", + torrent_seeders_css: "td:nth-child(6)", + torrent_leechers_css: "td:nth-child(7)", + torrent_snatched_css: "td:nth-child(8)", + torrent_time_css: "td:nth-child(4) > span::attr(title)", + torrent_time_fallback_css: "td:nth-child(4)", + torrent_time_fmt: "%Y-%m-%d %H:%M:%S", + torrent_uploader_css: "td:nth-child(9) a", + promo_download_rules: [ + ["img.pro_free", 0.0], + ["img.pro_free2up", 0.0], + ["img.pro_50pctdown", 0.5], + ["img.pro_50pctdown2up", 0.5], + ["img.pro_30pctdown", 0.3], + ], + promo_upload_rules: [ + ["img.pro_free2up", 2.0], + ["img.pro_50pctdown2up", 2.0], + ["img.pro_2up", 2.0], + ], + torrent_hr_css: "", + torrent_promo_deadline_css: "", + torrent_promo_deadline_attr: "title", + torrent_promo_deadline_re: "\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}", + torrent_promo_deadline_fmt: "%Y-%m-%d %H:%M:%S", + torrent_detail_url_css: "a[href*='details.php']::attr(href)", + torrent_download_url_css: "a[href*='download.php']::attr(href)", + page_offset: 0, + search_path: "torrents.php", + search_keyword_param: "search", + detail_title_css: "#top::text", + detail_subtitle_css: "td.rowhead:contains('副标题') + td::text", + detail_description_css: "#kdescr", + detail_download_css: "a[href*='download.php']::attr(href)", + detail_imdb_css: "a[href*='imdb.com']::attr(href)", + detail_douban_css: "a[href*='douban.com']::attr(href)", + detail_size_css: "td.rowhead:contains('大小') + td::text", + detail_file_list_css: "td.rowfollow > ul > li::text", + profile_path: "userdetails.php", + profile_uid_css: "", + profile_username_css: "h1::text", + profile_class_css: "td.rowhead:contains('等级') + td::text", + profile_uploaded_css: "td.rowhead:contains('上传量') + td::text", + profile_downloaded_css: "td.rowhead:contains('下载量') + td::text", + profile_ratio_css: "td.rowhead:contains('分享率') + td::text", + profile_bonus_css: "td.rowhead:contains('魔力') + td::text", + profile_join_date_css: "td.rowhead:contains('加入日期') + td::text", + profile_seeding_css: "td.rowhead:contains('当前做种') + td::text", + profile_leeching_css: "td.rowhead:contains('当前下载') + td::text", + profile_vip_css: "", + download_path: "download.php", +}; + +/** MovieClaw custom_class 的行为差异 → 定义层表达(只收录能用定义表达的部分) */ +const CUSTOM_CLASS_RULES = { + "movieclaw_tracker.sites.custom.ourbits.OurBitsSite": (def) => { + // 分类参数为重复的 cat[](custom/ourbits.py _apply_category_params) + def.category.param = "cat[]"; + return true; // 已完全由定义表达 + }, + "movieclaw_tracker.sites.custom.ttg.TTGSite": () => false, // 分类/ID/做种列均非标准,需自定义适配器 + "movieclaw_tracker.sites.custom.mteam.MTeamSite": () => true, // 由 mteam/* 实现 +}; + +function git(...a) { + return execFileSync("git", ["-C", REPO, ...a], { encoding: "utf8", maxBuffer: 64 * 1024 * 1024 }); +} + +function stripPseudo(css) { + return css.replace(/::(attr\([^)]*\)|text|next_sibling_text)$/, ""); +} + +/** MovieClaw `X::attr(a)` / `X::text` / `X` → SiteSpider 字段选择器 */ +function field(css, extra = {}) { + if (!css) return undefined; + const m = /^(.*)::attr\(([^)]+)\)$/.exec(css); + if (m) return { selector: m[1].trim(), attribute: m[2].trim(), ...extra }; + return { selector: stripPseudo(css).trim(), ...extra }; +} + +function caseTable(rules) { + const out = {}; + for (const [css, factor] of rules) out[css] = Number(factor); + out["*"] = 1; + return out; +} + +function rulesFrom(v, fallback) { + if (v == null) return fallback; + if (Array.isArray(v)) return v; + return Object.entries(v); +} + +function convertNexus(y, file) { + const s = { ...MC_DEFAULTS, ...(y.selectors ?? {}) }; + s.promo_download_rules = rulesFrom(y.selectors?.promo_download_rules, MC_DEFAULTS.promo_download_rules); + s.promo_upload_rules = rulesFrom(y.selectors?.promo_upload_rules, MC_DEFAULTS.promo_upload_rules); + const id = y.site_id; + const tz = TZ[y.timezone ?? "Asia/Shanghai"]; + if (!tz) throw new Error(`${file}: 未知时区 ${y.timezone}`); + const cats = y.categories ?? {}; + const item = (k) => (cats[k] ?? []).map((cid) => ({ id: String(cid), desc: CATEGORY_LABELS[k] })); + const categoryNames = {}; + for (const [k, ids] of Object.entries(cats)) + for (const cid of ids ?? []) categoryNames[String(cid)] ??= CATEGORY_LABELS[k] ?? k; + + const titleCss = stripPseudo(s.torrent_title_css); + const fields = { + title_default: { selector: titleCss }, + title_optional: { selector: `${titleCss}[title]`, attribute: "title" }, + title: { + text: "{% if fields['title_optional'] %}{{ fields['title_optional'] }}{% else %}{{ fields['title_default'] }}{% endif %}", + }, + details: field(s.torrent_detail_url_css), + download: field(s.torrent_download_url_css), + imdbid: { + selector: 'a[href*="imdb.com/title/tt"]', + attribute: "href", + filters: [{ name: "re_search", args: ["tt\\d+", 0] }], + }, + date_elapsed: field(s.torrent_time_fallback_css), + date_added: field(s.torrent_time_css), + date: { + text: "{% if fields['date_elapsed'] or fields['date_added'] %}{{ fields['date_added'] if fields['date_added'] else fields['date_elapsed'] }}{% else %}now{% endif %}", + filters: [{ name: "dateparse", args: s.torrent_time_fmt }], + }, + size: field(s.torrent_size_css), + seeders: field(s.torrent_seeders_css), + leechers: field(s.torrent_leechers_css), + grabs: field(s.torrent_snatched_css), + downloadvolumefactor: { case: caseTable(s.promo_download_rules) }, + uploadvolumefactor: { case: caseTable(s.promo_upload_rules) }, + description: field(s.torrent_subtitle_css), + category: field(s.torrent_category_css, { filters: [{ name: "re_search", args: ["\\d+", 0] }] }), + }; + if (s.torrent_promo_deadline_css) { + fields.freedate = { + selector: stripPseudo(s.torrent_promo_deadline_css), + attribute: s.torrent_promo_deadline_attr || "title", + filters: [{ name: "re_search", args: [s.torrent_promo_deadline_re, 0] }], + }; + } + if (s.torrent_hr_css) fields.hr = { selector: s.torrent_hr_css }; + + const profile = {}; + for (const [k, v] of Object.entries(s)) { + if (k.startsWith("profile_") && typeof v === "string" && v) profile[k.slice(8)] = v; + } + + const def = { + id, + name: y.display_name, + enabled: PRIMARY.has(id), + schema: MP_SCHEMA[id] ?? "NexusPhp", + provenance: { + source: "movieclaw", + ref: REF, + commit: COMMIT.slice(0, 8), + license: "MIT", + file: `${CONFIG_DIR}/${file}`, + }, + domain: `${String(y.base_url).replace(/\/+$/, "")}/`, + encoding: "UTF-8", + public: false, + proxy: false, + language: "zh", + result_num: 100, + timeout: Number(y.http?.timeout ?? 30), + search: { paths: [{ path: s.search_path, type: "all" }], params: { [s.search_keyword_param]: "{keyword}" } }, + batch: { delimiter: " ", space_replace: "_" }, + browse: { path: `${s.torrent_list_path}?page={page}`, start: Number(s.page_offset) === -1 ? 0 : 1 }, + category: { movie: item("movie"), tv: item("tv"), music: item("music") }, + torrents: { list: { selector: s.torrent_row_css }, fields }, + umt: { + version: `${id}@mc-${REF}`, + kind: "nexusphp", + displayName: y.display_name, + timezone: tz, + rateLimit: { minIntervalMs: Math.round(Number(y.min_request_interval ?? 3) * 1000), burst: 1 }, + capabilities: { list: true, search: true, detail: true, download: true, rss: false }, + categoryNames, + loginSuccess: s.login_success_css, + emptyMarkers: ["没有种子", "没有结果", "Nothing found"], + pagination: 'a[href*="page="]', + detailPath: "details.php", + downloadPath: s.download_path, + ...(y.auth?.supported ? { authSupported: y.auth.supported } : {}), + login: { + path: s.login_path, + usernameField: s.login_username_field, + passwordField: s.login_password_field, + errorCss: s.login_error_css, + ...(s.login_extra_form_data?.length ? { extraFormData: pairs(s.login_extra_form_data) } : {}), + ...(s.login_select_defaults && Object.keys(s.login_select_defaults).length + ? { selectDefaults: pairs(s.login_select_defaults) } + : {}), + }, + profile, + detail: { + path: "details.php", + title: stripPseudo(s.detail_title_css), + rowLabel: "td.rowhead", + rowValue: "td.rowfollow", + labels: { + subtitle: ["副标题", "Small Description"], + basicInfo: ["基本信息", "Basic Info"], + peers: ["同伴", "Peers"], + }, + description: s.detail_description_css, + headerScope: stripPseudo(s.detail_title_css), + notFoundMarkers: ["没有该ID的种子", "No torrent with ID"], + fields: { + title: s.detail_title_css, + subtitle: s.detail_subtitle_css, + size: s.detail_size_css, + description: s.detail_description_css, + download: s.detail_download_css, + imdb: s.detail_imdb_css, + douban: s.detail_douban_css, + fileList: s.detail_file_list_css, + }, + }, + }, + }; + if (y.custom_class) { + def.umt.customClass = y.custom_class; + const rule = CUSTOM_CLASS_RULES[y.custom_class]; + if (!rule) throw new Error(`${file}: 未登记的 custom_class ${y.custom_class}`); + if (!rule(def)) def.enabled = false; + } + if (y.hr_seed_hours) def.umt.hrSeedHours = Number(y.hr_seed_hours); + return def; +} + +function pairs(v) { + if (Array.isArray(v)) return v.map((p) => [String(p[0]), String(p[1])]); + return Object.entries(v).map(([k, x]) => [k, String(x)]); +} + +function convertApi(y, file) { + if (y.site_id !== "mteam") throw new Error(`${file}: 暂只支持 mteam 这一 API 站`); + const categoryNames = {}; + for (const [k, ids] of Object.entries(y.categories ?? {})) + for (const cid of ids ?? []) categoryNames[String(cid)] ??= CATEGORY_LABELS[k] ?? k; + return { + id: y.site_id, + name: y.display_name, + enabled: PRIMARY.has(y.site_id), + schema: MP_SCHEMA[y.site_id], + provenance: { + source: "movieclaw", + ref: REF, + commit: COMMIT.slice(0, 8), + license: "MIT", + file: `${CONFIG_DIR}/${file}`, + }, + domain: `${String(y.web_base_url ?? y.base_url).replace(/\/+$/, "")}/`, + parser: "mTorrent", + timeout: Number(y.http?.timeout ?? 15), + umt: { + version: `${y.site_id}@mc-${REF}`, + kind: "mteam", + displayName: y.display_name, + apiBaseUrl: String(y.base_url).replace(/\/+$/, ""), + timezone: TZ[y.timezone ?? "Asia/Shanghai"], + rateLimit: { minIntervalMs: Math.round(Number(y.min_request_interval ?? 3) * 1000), burst: 1 }, + capabilities: { list: true, search: true, detail: true, download: true, rss: false }, + categoryNames, + ...(y.auth?.supported ? { authSupported: y.auth.supported } : {}), + ...(y.custom_class ? { customClass: y.custom_class } : {}), + }, + }; +} + +function isObj(v) { + return v && typeof v === "object" && !Array.isArray(v); +} +function deepMerge(base, over, path = "") { + if (!isObj(base) || !isObj(over)) return over; + const out = { ...base }; + for (const [k, v] of Object.entries(over)) { + const p = path ? `${path}.${k}` : k; + if (v === null) delete out[k]; + else if (/^torrents\.fields\.[^.]+$/.test(p) || /^umt\.detail\.fields$/.test(p)) + out[k] = v; // 字段整体替换 + else out[k] = isObj(v) && isObj(out[k]) ? deepMerge(out[k], v, p) : v; + } + return out; +} + +/** 用仓库的 biome 格式化(与 lint 一致,保证 --check 幂等);biome 不可用时退回 JSON.stringify */ +function biomeFormat(text, fakePath) { + const bins = [join(ROOT, "node_modules/.bin/biome")]; + for (const bin of bins) { + if (!existsSync(bin)) continue; + try { + return execFileSync(bin, ["format", `--stdin-file-path=${fakePath}`], { + input: text, + encoding: "utf8", + cwd: ROOT, + }); + } catch { + /* fallthrough */ + } + } + return text; +} + +/** 输出键顺序固定,方便 diff */ +function stable(v, fakePath) { + return biomeFormat(`${JSON.stringify(v, null, 2)}\n`, fakePath); +} + +// ---- main ---- +const COMMIT = git("rev-parse", `${REF}^{commit}`).trim(); +if (EXPECTED[REF] && EXPECTED[REF] !== COMMIT) + throw new Error(`${REF} 解析到 ${COMMIT},与登记的 ${EXPECTED[REF]} 不符`); +if (!EXPECTED[REF] && !args.includes("--allow-ref")) + throw new Error(`ref ${REF} 未登记;许可需另行核对后加 --allow-ref`); +const license = git("show", `${REF}:LICENSE`); +if (!/^MIT License/m.test(license)) throw new Error(`${REF}:LICENSE 不是 MIT,停止(许可切换后的内容不得复制)`); + +const files = git("ls-tree", "--name-only", `${REF}:${CONFIG_DIR}`) + .split("\n") + .filter((f) => f.endsWith(".yaml") && !f.startsWith("_")); +const ids = []; +let diffs = 0; +for (const file of files) { + const y = YAML.parse(git("show", `${REF}:${CONFIG_DIR}/${file}`)); + let def = y.framework === "api" ? convertApi(y, file) : convertNexus(y, file); + const ovPath = join(OVERRIDES, `${y.site_id}.json`); + if (existsSync(ovPath)) { + const ov = JSON.parse(readFileSync(ovPath, "utf8")); + delete ov.$comment; + def = deepMerge(def, ov); + def.provenance = { ...def.provenance, overrides: `scripts/mp-sync/movieclaw-overrides/${y.site_id}.json` }; + } + // 分类项 desc 以最终 categoryNames 为准(覆盖层可能给出更细的站点分类名) + for (const k of ["movie", "tv", "music"]) { + for (const c of def.category?.[k] ?? []) { + const n = def.umt?.categoryNames?.[String(c.id)]; + if (n) c.desc = n; + } + } + const out = join(OUT_DIR, `${y.site_id}.json`); + const text = stable(def, `packages/adapters/src/sites/definitions/${y.site_id}.json`); + const old = existsSync(out) ? readFileSync(out, "utf8") : null; + if (old !== text) { + diffs++; + if (CHECK) console.log(`差异:${y.site_id}.json`); + else writeFileSync(out, text); + } + ids.push(y.site_id); +} + +const ident = (id) => `${id.replace(/[^A-Za-z0-9_]/g, "_")}Json`; +const manifest = `// 由 scripts/mp-sync/movieclaw-import.mjs 生成,请勿手改(MovieClaw ${REF} ${COMMIT.slice(0, 8)},MIT) +${ids + .sort() + .map((id) => `import ${ident(id)} from "./${id}.json" with { type: "json" };`) + .join("\n")} + +/** MovieClaw 转换来的站点定义原始 JSON(未校验;经 loadSiteDefinition 校验后使用) */ +export const MOVIECLAW_DEFINITION_JSON: Record = { +${ids + .sort() + .map((id) => ` ${JSON.stringify(id)}: ${ident(id)},`) + .join("\n")} +}; +`; +const manifestPath = join(OUT_DIR, "manifest.generated.ts"); +const manifestText = biomeFormat(manifest, "packages/adapters/src/sites/definitions/manifest.generated.ts"); +if ((existsSync(manifestPath) ? readFileSync(manifestPath, "utf8") : null) !== manifestText) { + diffs++; + if (CHECK) console.log("差异:manifest.generated.ts"); + else writeFileSync(manifestPath, manifestText); +} +const stale = readdirSync(OUT_DIR).filter((f) => f.endsWith(".json") && !ids.includes(f.replace(/\.json$/, ""))); +if (stale.length) console.log(`注意:定义目录中存在非 MovieClaw 来源的 JSON(保留不动):${stale.join(", ")}`); +console.log( + `${CHECK ? "检查" : "转换"}完成:${ids.length} 个站点,${diffs} 个文件${CHECK ? "有差异" : "已更新"}(${REF} ${COMMIT.slice(0, 8)})`, +); +if (CHECK && diffs) process.exit(1); diff --git a/scripts/mp-sync/movieclaw-overrides/audiences.json b/scripts/mp-sync/movieclaw-overrides/audiences.json new file mode 100644 index 00000000..25e2eba0 --- /dev/null +++ b/scripts/mp-sync/movieclaw-overrides/audiences.json @@ -0,0 +1,204 @@ +{ + "$comment": "W1-3b 手写观众定义(按 MP indexer schema;仅对 testkit mock 验证)。深合并到 MovieClaw v0.27.0 转换结果之上,本文件的值优先,MovieClaw 值只补空。 umt.detail.fields=null:保留 W1-3b rowhead 详情模型(已对 mock 验证),不采用 MovieClaw 默认逐字段选择器。", + "id": "audiences", + "name": "观众", + "domain": "https://audiences.me/", + "encoding": "UTF-8", + "public": false, + "proxy": false, + "language": "zh", + "result_num": 100, + "timeout": 30, + "search": { + "paths": [ + { + "path": "torrents.php", + "type": "all" + } + ], + "params": { + "search": "{keyword}", + "search_area": 4 + } + }, + "batch": { + "delimiter": " ", + "space_replace": "_" + }, + "browse": { + "path": "torrents.php?page={page}", + "start": 0 + }, + "category": { + "movie": [ + { + "id": "401", + "cat": "Movies", + "desc": "电影" + } + ], + "tv": [ + { + "id": "402", + "cat": "TV Series", + "desc": "剧集" + }, + { + "id": "403", + "cat": "TV Shows", + "desc": "综艺" + } + ], + "music": [] + }, + "torrents": { + "list": { + "selector": "table.torrents > tbody > tr:has(table.torrentname), table.torrents > tr:has(table.torrentname)" + }, + "fields": { + "title_default": { + "selector": "a[href*=\"details.php?id=\"]" + }, + "title_optional": { + "selector": "a[title][href*=\"details.php?id=\"]", + "attribute": "title" + }, + "title": { + "text": "{% if fields['title_optional'] %}{{ fields['title_optional'] }}{% else %}{{ fields['title_default'] }}{% endif %}" + }, + "details": { + "selector": "a[href*=\"details.php?id=\"]", + "attribute": "href" + }, + "download": { + "selector": "a[href*=\"download.php?id=\"]", + "attribute": "href" + }, + "imdbid": { + "selector": "a[href*=\"imdb.com/title/tt\"]", + "attribute": "href", + "filters": [ + { + "name": "re_search", + "args": ["tt\\d+", 0] + } + ] + }, + "date_elapsed": { + "selector": "td:nth-child(4) > span" + }, + "date_added": { + "selector": "td:nth-child(4) > span", + "attribute": "title" + }, + "date": { + "text": "{% if fields['date_elapsed'] or fields['date_added'] %}{{ fields['date_added'] if fields['date_added'] else fields['date_elapsed'] }}{% else %}now{% endif %}", + "filters": [ + { + "name": "dateparse", + "args": "%Y-%m-%d %H:%M:%S" + } + ] + }, + "size": { + "selector": "td:nth-child(5)" + }, + "seeders": { + "selector": "td:nth-child(6)" + }, + "leechers": { + "selector": "td:nth-child(7)" + }, + "grabs": { + "selector": "td:nth-child(8)" + }, + "downloadvolumefactor": { + "case": { + "img.pro_free": 0, + "img.pro_free2up": 0, + "img.pro_50pctdown": 0.5, + "img.pro_50pctdown2up": 0.5, + "img.pro_30pctdown": 0.3, + "*": 1 + } + }, + "uploadvolumefactor": { + "case": { + "img.pro_50pctdown2up": 2, + "img.pro_free2up": 2, + "img.pro_2up": 2, + "*": 1 + } + }, + "freedate": { + "selector": "img[class^=\"pro_\"]", + "attribute": "title", + "filters": [ + { + "name": "re_search", + "args": ["\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}", 0] + } + ] + }, + "description": { + "selector": "span.torrent-subtitle-line" + }, + "labels": { + "selector": "span.tags" + }, + "category": { + "selector": "a[href*=\"?cat=\"]", + "attribute": "href", + "filters": [ + { + "name": "querystring", + "args": "cat" + } + ] + } + } + }, + "umt": { + "version": "audiences@0.2.0-unverified", + "kind": "nexusphp", + "displayName": "观众", + "timezone": "+08:00", + "rateLimit": { + "minIntervalMs": 3000, + "burst": 1 + }, + "capabilities": { + "list": true, + "search": true, + "detail": true, + "download": true, + "rss": true + }, + "categoryNames": { + "401": "电影", + "402": "剧集", + "403": "综艺", + "406": "纪录片" + }, + "loginSuccess": "a[href*=\"logout\"]", + "emptyMarkers": ["没有种子", "没有结果", "Nothing found"], + "pagination": "a[href*=\"page=\"]", + "detailPath": "details.php", + "downloadPath": "download.php", + "detail": { + "path": "details.php", + "title": "h1#top", + "rowLabel": "td.rowhead", + "rowValue": "td.rowfollow", + "labels": { + "subtitle": ["副标题", "Small Description"], + "basicInfo": ["基本信息", "Basic Info"], + "peers": ["同伴", "Peers"] + }, + "description": "#kdescr", + "headerScope": "h1#top", + "notFoundMarkers": ["没有该ID的种子", "No torrent with ID"], + "fields": null + } + } +} diff --git a/scripts/mp-sync/movieclaw-overrides/chdbits.json b/scripts/mp-sync/movieclaw-overrides/chdbits.json new file mode 100644 index 00000000..bc9cd5a6 --- /dev/null +++ b/scripts/mp-sync/movieclaw-overrides/chdbits.json @@ -0,0 +1,16 @@ +{ + "$comment": "CHDBits 分类名取自 MovieClaw chdbits.yaml 注释(TV Series / TV Shows / Sports / Music Videos / HQ Audio)。", + "umt": { + "categoryNames": { + "401": "电影", + "402": "剧集", + "403": "综艺", + "404": "纪录片", + "405": "动漫", + "406": "MV", + "407": "体育", + "408": "HQ Audio", + "409": "其他" + } + } +} diff --git a/scripts/mp-sync/movieclaw-overrides/hhanclub.json b/scripts/mp-sync/movieclaw-overrides/hhanclub.json new file mode 100644 index 00000000..d9564c5e --- /dev/null +++ b/scripts/mp-sync/movieclaw-overrides/hhanclub.json @@ -0,0 +1,17 @@ +{ + "$comment": "HHClub 分类名取自 MovieClaw hhanclub.yaml 注释(导航栏分类表);REQ §4.5 要求综艺/体育/短剧保留原始分类。", + "umt": { + "categoryNames": { + "401": "电影", + "402": "电视剧", + "403": "综艺", + "404": "纪录片", + "405": "动漫", + "407": "体育", + "409": "其他", + "410": "音乐", + "411": "MV", + "412": "短剧" + } + } +} diff --git a/scripts/mp-sync/movieclaw-overrides/mteam.json b/scripts/mp-sync/movieclaw-overrides/mteam.json new file mode 100644 index 00000000..fb655723 --- /dev/null +++ b/scripts/mp-sync/movieclaw-overrides/mteam.json @@ -0,0 +1,26 @@ +{ + "$comment": "W1-3b 馒头定义(mteam/definition.ts 原值,按 MP mtorrent.py)。本文件优先,MovieClaw 值只补空;categoryNames 为本项目编制的粗分类名。", + "name": "馒头", + "timeout": 15, + "umt": { + "version": "mteam@0.2.0-unverified", + "displayName": "馒头", + "categoryNames": { + "401": "电影", + "419": "电影", + "420": "电影", + "421": "电影", + "439": "电影", + "402": "剧集", + "403": "综艺", + "435": "剧集", + "438": "剧集", + "404": "纪录", + "405": "动画", + "407": "体育", + "406": "音乐", + "408": "音乐", + "434": "音乐" + } + } +}