feat(adapters): RSS and PT site adapters ported from MoviePilot v3 + mocks (W1-3b)
- rss: port of MP RssHelper._parse_impl/get_rss_link semantics with safe XML (entities off, DTD/ENTITY rejected, size/item caps), ETag/Last-Modified, backoff + Retry-After, circuit breaker with manual reset, passkey redaction, overlap-window GUID+digest dedupe - sites/spider: TS port of MP SiteSpider (indexer schema, search URL builder, field DSL, filters, Jinja templates via nunjucks, login-page detection); audiences definition hand-written as JSON; raw category id preserved; shared per-site token bucket - sites/mteam: port of MP MTorrentSpider (params, result mapping, promo factors, genDlToken flow) without embedding the API key in enclosures; business code/message checks - testkit: mock RSS server, mock NexusPHP site, mock M-Team API - docs: W1-3b migration mapping appended to porting ledger Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
rootandClaude Fable 5.1 committed
1 parent
ef2e504a94
commit
a6d049db81
47 files changed
+5414
No files matched your search
@@ -608,3 +608,57 @@ MC 注册表 `MC:src/movieclaw_tracker/registry.py:172` `_load_site_yaml` 读取
|
||||
### 文件系统(`packages/adapters/src/fs/`)
|
||||
|
||||
对照 `docs/porting/moviepilot-transfer.md` §2.6:`_write_atomically`/`_partial_path`/`_cleanup_stale_partials` → `copyWithVerify`/`partialPathFor`/`sweepStalePartials`(新增 fsync、校验、no-clobber);`LocalStorage.move` → `moveSafe`(EXDEV 不降级)+ `crossDeviceMove`(三步可重入)。`statSafe`/`scanIncremental`/`ffprobe`/`diskUsage` 为本项目新增。
|
||||
|
||||
## 10. W1-3b 实际迁移对照(RSS / 站点)
|
||||
|
||||
按用户指示"照抄 MoviePilot v3 实现",以 MP 提交 `e6820a8` 为蓝本迁移为 TS。只有安全、数据保真与去重相关的偏离,逐条列在"调整"列。MovieClaw 仍只参考字段结构(data-ref)。
|
||||
|
||||
### 10.1 通用站点 spider(`packages/adapters/src/sites/spider/`)
|
||||
|
||||
| 上游函数 | TS 函数 | 保留 / 调整 |
|
||||
| --- | --- | --- |
|
||||
| `spider/__init__.py select_media_categories` | `selectMediaCategories` | 保留 |
|
||||
| `resolve_category_media_type` | `resolveCategoryMediaType` | 保留(投影值仍写 `category`) |
|
||||
| `SiteSpider.__init__` 读取的 indexer 键 | `IndexerDefinition`(`schema.ts`) | 保留全部键;新增 `umt` 扩展(版本、时区、限流、登录标志、详情页、分类名) |
|
||||
| `SiteSpider.__get_search_url` / `__format_search_word` | `SiteSpider.getSearchUrl` / `formatSearchWord` | 保留(默认参数、search_area 仅 IMDb 保留、batch、typed path、category field/param/catN、browse.start、result_media_type) |
|
||||
| `UrlUtils.combine_url` | `combineUrl` | 保留(doseq) |
|
||||
| `__can_search` | `canSearch` | 保留 |
|
||||
| `get_info` + `__get_title … __get_category` | `getInfo` + 同名私有方法 | 保留字段顺序与缺省值(size/seeders/peers/grabs 缺失为 0,相对时间不写 pubdate,case 顺序首个命中);**调整**:系数 selector 模式浮点(SA4);`__get_category` 额外输出 `category_id` 原值 |
|
||||
| `_safe_query / __remove / __attribute_or_text / __index / __filter_text` | 同名方法 | 保留(re_search/split(含负下标)/replace/dateparse/strip/appendleft/querystring,单个过滤器失败跳过) |
|
||||
| `__is_login_or_permission_page` | `SiteSpider.isLoginOrPermissionPage` | 保留判定;**调整**:adapter 抛 `auth_expired` 而不是返回空列表 |
|
||||
| `parse`(PyQuery 分支) | `SiteSpider.parse` | 保留;htmlparser2 HTML 模式对齐 lxml(不插 tbody);`:has("x")` 去引号;**未移植** Rust 加速与字幕分支 |
|
||||
| jinja2 模板 | `template.ts`(nunjucks) | 保留 Jinja 语法 |
|
||||
| `foundation/size.parse_size` | `parseSizeMp` | 保留(round) |
|
||||
| `IndexerModule.__search_check` / `__clear_search_text` | `search` 前检查 / `clearSearchText` | 保留(英文站不搜中文、去标点) |
|
||||
| `SitesHelper.check(domain)`(闭源) | `createSiteRateLimiter` | 以令牌桶实现(SA6) |
|
||||
| `submission.download_torrent` | `downloadTarget` | 保留(Cookie GET、磁力原样);**调整**:不跟随 3xx、校验 bencode 并计算 infohash、登录页识别 |
|
||||
| — | `detail.ts parseDetailPage` | 新增(MP 通用 spider 无详情页) |
|
||||
| MP 站点定义(加密 `user.sites.v3.bin`) | `definitions/audiences.json` | 按 MP schema 手写,待真实页面核验 |
|
||||
|
||||
### 10.2 M-Team(`packages/adapters/src/sites/mteam/`)
|
||||
|
||||
| 上游函数 | TS 函数 | 保留 / 调整 |
|
||||
| --- | --- | --- |
|
||||
| `mtorrent.py _movie_category/_tv_category/_music_category/_labels` | `MOVIE_CATEGORY/TV_CATEGORY/MUSIC_CATEGORY/LABELS` | 保留 |
|
||||
| `MTorrentSpider.__get_params` | `mteamSearchParams` | 保留(`tt` 前缀改写 IMDb URL、pageNumber=page+1、pageSize 100、visible 1、无 mode);**调整**:显式 `categoryIds` 覆盖 mtype 分类组 |
|
||||
| `__parse_result` | `parseMteamResult` | 保留字段映射、promotionRule 覆盖下载系数、mallSingleFree ONGOING → 0、freedate 取值顺序;**调整**:不生成含 apikey 的 base64 enclosure(MT3);输出 `category_id` |
|
||||
| `__find_imdbid / __get_downloadvolumefactor / __get_uploadvolumefactor` | `findImdbid / getDownloadVolumeFactor / getUploadVolumeFactor` | 保留 |
|
||||
| `search` / `__process_response` | `mteamPost` + adapter `list/search` | 保留 header(x-api-key、UA、JSON);**调整**:检查业务 code/message 区分凭据/限流/不存在(MT2) |
|
||||
| `__get_download_url` + `submission._resolve_indirect_download_url` | adapter `downloadTarget` | 保留(POST genDlToken,id 走 query,取 `data`,不带 Cookie GET);令牌不入库不入日志 |
|
||||
| `parser/mtorrent.py`(api/member/profile) | `healthCheck` | 只用作登录有效性 |
|
||||
| — | `detail`(`/api/torrent/detail`) | 新增,字段结构参考 MovieClaw(data-ref) |
|
||||
|
||||
### 10.3 RSS(`packages/adapters/src/rss/`)
|
||||
|
||||
| 上游函数 | TS 函数 | 保留 / 调整 |
|
||||
| --- | --- | --- |
|
||||
| `RssHelper.MAX_RSS_SIZE / MAX_RSS_ITEMS` | fetcher `maxBytes` / parser `maxItems` 默认值 | 保留 50MB / 1000;大小按解压后字节流式限制 |
|
||||
| `RssHelper._parse_impl` 条目循环 | `parseFeed` / `itemFields` | 保留字段语义(见 rss/README);**调整**:fast-xml-parser 关闭实体处理并预检拒绝 ENTITY/内部 DTD;新增 guid/categories/rawDigest |
|
||||
| `_rss_expired_msg` | `RSS_EXPIRED_MESSAGES` → `FeedError('expired')` | 保留识别,返回值改为错误类型 |
|
||||
| `RssHelper.parse`(GET,非 200 失败) | `createRssFetcher().fetchFeed` | 保留;新增 ETag/Last-Modified、退避 + Retry-After、熔断 + 手动恢复、passkey 脱敏 |
|
||||
| `rss_link_conf` / `_get_site_domain` / `get_rss_link` | `RSS_LINK_CONF` / `rssSiteDomain` / `buildRssLinkRequest` + `extractRssLink` | 只迁移 default 与 audiences.me;xpath 改等价 CSS |
|
||||
| — | `applyFeedItems` / `FeedCursor` | 新增重叠窗口去重(REQ §4.2/§4.3) |
|
||||
|
||||
### 10.4 许可提示
|
||||
|
||||
本节模块是对 MP(GPL-3.0)实现的逐函数 TS 移植,性质上比 `behavior-ref` 更接近衍生作品。项目目前为 internal-only 使用;若将来对外分发,需按 `license-and-provenance.md` §2 重新评估(GPL-3.0 同许可与源码提供义务)。
|
||||
Reference in new issue
Block a user