feat(adapters): RSS and PT site adapters ported from MoviePilot v3 + mocks (W1-3b)
- rss: port of MP RssHelper._parse_impl/get_rss_link semantics with safe XML (entities off, DTD/ENTITY rejected, size/item caps), ETag/Last-Modified, backoff + Retry-After, circuit breaker with manual reset, passkey redaction, overlap-window GUID+digest dedupe - sites/spider: TS port of MP SiteSpider (indexer schema, search URL builder, field DSL, filters, Jinja templates via nunjucks, login-page detection); audiences definition hand-written as JSON; raw category id preserved; shared per-site token bucket - sites/mteam: port of MP MTorrentSpider (params, result mapping, promo factors, genDlToken flow) without embedding the API key in enclosures; business code/message checks - testkit: mock RSS server, mock NexusPHP site, mock M-Team API - docs: W1-3b migration mapping appended to porting ledger Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
1 parent
ef2e504a94
commit
a6d049db81
47 files changed
+5414
No files matched your search
@@ -0,0 +1,36 @@
|
||||
# adapters/rss
|
||||
|
||||
RSS 2.0 / RSS 1.0(RDF) / Atom 抓取与解析,按 MoviePilot e6820a8 `app/application/rss.py`(`RssHelper._parse_impl` / `get_rss_link`)迁移,再叠加任务书的安全、条件请求、退避、熔断、脱敏与去重。**仅对 `@umt/testkit` 的 mock RSS 服务器验证,未连接真实站点 Feed。**
|
||||
|
||||
## 能力
|
||||
|
||||
| 能力 | 入口 | 说明 |
|
||||
| --- | --- | --- |
|
||||
| 安全解析 | `parseFeed(xml, {maxItems, maxChars})` | fast-xml-parser 关闭实体处理;预检拒绝 `<!ENTITY` 与内部 DTD 子集(XXE、billion laughs);只解码 XML 预定义实体与数字引用;条目数超限截断并标 `truncated` |
|
||||
| 抓取 | `createRssFetcher(opts).fetchFeed(url, state)` | 条件请求(ETag / Last-Modified → 304)、解压后字节上限(同时约束 gzip/br 解压量)、超时、指数退避 + Retry-After(秒数或 HTTP-date,取较大者)、熔断(连续 N 次失败 → open,`resetFeedCircuit` 手动恢复,可选冷却后半开探测) |
|
||||
| 状态 | `FeedFetchState` | 纯数据,由调用方持久化(feed_source);`fetchFeed` 不抛业务错误,返回 `ok / not_modified / skipped / error` |
|
||||
| 脱敏 | `redactUrl` / `redactText` / `unredactUrl` / `extractSecretParams` | query 中 passkey/token/authkey/sign/apikey 等替换为 `[REDACTED]`,userinfo 抹除;所有错误消息与 `lastError` 已脱敏 |
|
||||
| 去重 | `applyFeedItems(cursor, items)` | 重叠窗口游标 `FeedCursor{lastGuidSet, digests, lastFetchedAt}`;GUID 新增 → `newItems`,同 GUID 摘要变化 → `updatedItems`;乱序/置顶不产生新条目;与上次窗口零重叠 → `possibleGap`(调度补漏) |
|
||||
|
||||
## 与 MP `RssHelper` 的对应
|
||||
|
||||
| MP | 本项目 |
|
||||
| --- | --- |
|
||||
| `.//item \| .//entry` 全树收集,`MAX_RSS_ITEMS=1000` | 同;超出截断并标 `truncated` |
|
||||
| `MAX_RSS_SIZE=50MB`、非 `<` 开头拒绝、`resolve_entities=False` | 同(按解压后字节流式限制);另外预检拒绝 `<!ENTITY` 与内部 DTD 子集 |
|
||||
| title 为空跳过;description\|summary;link 文本或 href;enclosure@url,缺失回退 link,两者皆无跳过;enclosure@length 纯数字才取;pubDate\|published\|updated;creator → nickname | 同(字段见下) |
|
||||
| 非 200 返回 False;过期提示返回 None | `FeedFetchResult.kind='error'`;过期提示 → `FeedError('expired')` |
|
||||
| `rss_link_conf` / `get_rss_link` | `rss-link.ts`:`buildRssLinkRequest` / `extractRssLink`(只含 default 与 audiences.me) |
|
||||
| —(MP 无) | guid、categories、rawDigest、条件请求、退避、Retry-After、熔断、脱敏、重叠窗口去重 |
|
||||
|
||||
## FeedItem
|
||||
|
||||
`{guid, title, link, enclosureUrl?, enclosureHasSecret?, size, publishedAt?, nickname?, categories[], description?, rawDigest, fetchedAt}`
|
||||
|
||||
- `guid`/`link`/`enclosureUrl` 均为**脱敏形态**,可直接入库与作为去重键;passkey 轮换不改变 GUID 与摘要。
|
||||
- 需要真实下载链接时,由调用方从 secret store 取 passkey,用 `unredactUrl(enclosureUrl, {passkey})` 在内存中还原,结果不得记录。
|
||||
- `publishedAt` 是来源发布时间,`fetchedAt` 是抓取时间,二者不可混用(REQ §4.3)。
|
||||
|
||||
## 默认值
|
||||
|
||||
超时 30s、响应上限 50 MiB、条目上限 1000(同 MP)、退避 60s 起 ×2 封顶 6h、熔断阈值 5、熔断默认只能手动恢复。均可在 `createRssFetcher` 覆盖;真实站点默认值待联调后确定。
|
||||
Reference in new issue
Block a user