feat(adapters): RSS and PT site adapters ported from MoviePilot v3 + mocks (W1-3b)
- rss: port of MP RssHelper._parse_impl/get_rss_link semantics with safe XML (entities off, DTD/ENTITY rejected, size/item caps), ETag/Last-Modified, backoff + Retry-After, circuit breaker with manual reset, passkey redaction, overlap-window GUID+digest dedupe - sites/spider: TS port of MP SiteSpider (indexer schema, search URL builder, field DSL, filters, Jinja templates via nunjucks, login-page detection); audiences definition hand-written as JSON; raw category id preserved; shared per-site token bucket - sites/mteam: port of MP MTorrentSpider (params, result mapping, promo factors, genDlToken flow) without embedding the API key in enclosures; business code/message checks - testkit: mock RSS server, mock NexusPHP site, mock M-Team API - docs: W1-3b migration mapping appended to porting ledger Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
rootandClaude Fable 5.1 committed
1 parent
ef2e504a94
commit
a6d049db81
47 files changed
+5414
No files matched your search
@@ -0,0 +1,36 @@
|
||||
# adapters/rss
|
||||
|
||||
RSS 2.0 / RSS 1.0(RDF) / Atom 抓取与解析,按 MoviePilot e6820a8 `app/application/rss.py`(`RssHelper._parse_impl` / `get_rss_link`)迁移,再叠加任务书的安全、条件请求、退避、熔断、脱敏与去重。**仅对 `@umt/testkit` 的 mock RSS 服务器验证,未连接真实站点 Feed。**
|
||||
|
||||
## 能力
|
||||
|
||||
| 能力 | 入口 | 说明 |
|
||||
| --- | --- | --- |
|
||||
| 安全解析 | `parseFeed(xml, {maxItems, maxChars})` | fast-xml-parser 关闭实体处理;预检拒绝 `<!ENTITY` 与内部 DTD 子集(XXE、billion laughs);只解码 XML 预定义实体与数字引用;条目数超限截断并标 `truncated` |
|
||||
| 抓取 | `createRssFetcher(opts).fetchFeed(url, state)` | 条件请求(ETag / Last-Modified → 304)、解压后字节上限(同时约束 gzip/br 解压量)、超时、指数退避 + Retry-After(秒数或 HTTP-date,取较大者)、熔断(连续 N 次失败 → open,`resetFeedCircuit` 手动恢复,可选冷却后半开探测) |
|
||||
| 状态 | `FeedFetchState` | 纯数据,由调用方持久化(feed_source);`fetchFeed` 不抛业务错误,返回 `ok / not_modified / skipped / error` |
|
||||
| 脱敏 | `redactUrl` / `redactText` / `unredactUrl` / `extractSecretParams` | query 中 passkey/token/authkey/sign/apikey 等替换为 `[REDACTED]`,userinfo 抹除;所有错误消息与 `lastError` 已脱敏 |
|
||||
| 去重 | `applyFeedItems(cursor, items)` | 重叠窗口游标 `FeedCursor{lastGuidSet, digests, lastFetchedAt}`;GUID 新增 → `newItems`,同 GUID 摘要变化 → `updatedItems`;乱序/置顶不产生新条目;与上次窗口零重叠 → `possibleGap`(调度补漏) |
|
||||
|
||||
## 与 MP `RssHelper` 的对应
|
||||
|
||||
| MP | 本项目 |
|
||||
| --- | --- |
|
||||
| `.//item \| .//entry` 全树收集,`MAX_RSS_ITEMS=1000` | 同;超出截断并标 `truncated` |
|
||||
| `MAX_RSS_SIZE=50MB`、非 `<` 开头拒绝、`resolve_entities=False` | 同(按解压后字节流式限制);另外预检拒绝 `<!ENTITY` 与内部 DTD 子集 |
|
||||
| title 为空跳过;description\|summary;link 文本或 href;enclosure@url,缺失回退 link,两者皆无跳过;enclosure@length 纯数字才取;pubDate\|published\|updated;creator → nickname | 同(字段见下) |
|
||||
| 非 200 返回 False;过期提示返回 None | `FeedFetchResult.kind='error'`;过期提示 → `FeedError('expired')` |
|
||||
| `rss_link_conf` / `get_rss_link` | `rss-link.ts`:`buildRssLinkRequest` / `extractRssLink`(只含 default 与 audiences.me) |
|
||||
| —(MP 无) | guid、categories、rawDigest、条件请求、退避、Retry-After、熔断、脱敏、重叠窗口去重 |
|
||||
|
||||
## FeedItem
|
||||
|
||||
`{guid, title, link, enclosureUrl?, enclosureHasSecret?, size, publishedAt?, nickname?, categories[], description?, rawDigest, fetchedAt}`
|
||||
|
||||
- `guid`/`link`/`enclosureUrl` 均为**脱敏形态**,可直接入库与作为去重键;passkey 轮换不改变 GUID 与摘要。
|
||||
- 需要真实下载链接时,由调用方从 secret store 取 passkey,用 `unredactUrl(enclosureUrl, {passkey})` 在内存中还原,结果不得记录。
|
||||
- `publishedAt` 是来源发布时间,`fetchedAt` 是抓取时间,二者不可混用(REQ §4.3)。
|
||||
|
||||
## 默认值
|
||||
|
||||
超时 30s、响应上限 50 MiB、条目上限 1000(同 MP)、退避 60s 起 ×2 封顶 6h、熔断阈值 5、熔断默认只能手动恢复。均可在 `createRssFetcher` 覆盖;真实站点默认值待联调后确定。
|
||||
@@ -0,0 +1,68 @@
|
||||
import type { FeedCursor, FeedItem } from "./types";
|
||||
|
||||
export function emptyFeedCursor(): FeedCursor {
|
||||
return { lastGuidSet: [], digests: {}, lastFetchedAt: null };
|
||||
}
|
||||
|
||||
export interface ApplyFeedResult {
|
||||
/** 窗口内首次出现的 GUID */
|
||||
newItems: FeedItem[];
|
||||
/** 同 GUID 摘要变化(标题修正、促销、体积、置顶变化等),视为更新 */
|
||||
updatedItems: FeedItem[];
|
||||
unchangedCount: number;
|
||||
/**
|
||||
* 疑似缺口:本次条目与上次窗口没有任何重叠,且上次窗口非空。
|
||||
* 说明两次抓取之间可能有条目被挤出 Feed,需要调度有预算的列表/搜索补漏(REQ §4.3)。
|
||||
*/
|
||||
possibleGap: boolean;
|
||||
cursor: FeedCursor;
|
||||
}
|
||||
|
||||
export interface ApplyFeedOptions {
|
||||
/** 游标保留的 GUID 数上限,默认 max(500, 本次条目数 * 3) */
|
||||
windowSize?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* 重叠窗口去重:不依赖条目顺序或递增 ID,乱序/置顶/重复出现都按 GUID+摘要判定。
|
||||
* 游标只是快速过滤;持久去重仍以数据库 (feed, guid) 唯一约束为准。
|
||||
*/
|
||||
export function applyFeedItems(cursor: FeedCursor, items: FeedItem[], opts: ApplyFeedOptions = {}): ApplyFeedResult {
|
||||
const newItems: FeedItem[] = [];
|
||||
const updatedItems: FeedItem[] = [];
|
||||
let unchangedCount = 0;
|
||||
let overlap = false;
|
||||
const digests: Record<string, string> = { ...cursor.digests };
|
||||
const seenThisRound = new Set<string>();
|
||||
for (const item of items) {
|
||||
if (seenThisRound.has(item.guid)) continue; // 同一次抓取内重复 GUID 只取首条
|
||||
seenThisRound.add(item.guid);
|
||||
const prev = digests[item.guid];
|
||||
if (prev === undefined) {
|
||||
newItems.push(item);
|
||||
} else {
|
||||
overlap = true;
|
||||
if (prev !== item.rawDigest) updatedItems.push(item);
|
||||
else unchangedCount++;
|
||||
}
|
||||
digests[item.guid] = item.rawDigest;
|
||||
}
|
||||
const windowSize = opts.windowSize ?? Math.max(500, items.length * 3);
|
||||
const order = [...seenThisRound, ...cursor.lastGuidSet.filter((g) => !seenThisRound.has(g))].slice(0, windowSize);
|
||||
const kept: Record<string, string> = {};
|
||||
for (const g of order) {
|
||||
const d = digests[g];
|
||||
if (d !== undefined) kept[g] = d;
|
||||
}
|
||||
const lastFetchedAt = items.reduce<string | null>(
|
||||
(acc, it) => (acc === null || it.fetchedAt > acc ? it.fetchedAt : acc),
|
||||
cursor.lastFetchedAt,
|
||||
);
|
||||
return {
|
||||
newItems,
|
||||
updatedItems,
|
||||
unchangedCount,
|
||||
possibleGap: cursor.lastGuidSet.length > 0 && items.length > 0 && !overlap,
|
||||
cursor: { lastGuidSet: order, digests: kept, lastFetchedAt },
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,229 @@
|
||||
/**
|
||||
* RSS 抓取。迁移自 MP `app/application/rss.py RssHelper.parse/_parse_impl`(GET + UA + 自定义头,非 200 视为失败,
|
||||
* 大小上限后再解析);条件请求、退避、Retry-After、熔断、脱敏为任务书新增要求(MP 无)。
|
||||
*/
|
||||
import { parseFeed } from "./parse";
|
||||
import { redactText, redactUrl } from "./redact";
|
||||
import { FeedError, type FeedFetchResult, type FeedFetchState } from "./types";
|
||||
|
||||
export interface RssFetcherOptions {
|
||||
fetch?: typeof globalThis.fetch;
|
||||
now?: () => Date;
|
||||
/** 单次请求超时,默认 30s(MP `timeout or 30`) */
|
||||
timeoutMs?: number;
|
||||
/** 解压后响应体上限(字节),默认 50 MiB(MP MAX_RSS_SIZE);同时约束 gzip/br 解压量 */
|
||||
maxBytes?: number;
|
||||
/** 条目上限,默认 1000(MP MAX_RSS_ITEMS) */
|
||||
maxItems?: number;
|
||||
userAgent?: string;
|
||||
/** 指数退避:base * 2^(n-1),上限 max。默认 60s / 6h */
|
||||
backoff?: { baseMs: number; maxMs: number };
|
||||
/** 连续失败达到该次数进入熔断(open),默认 5 */
|
||||
circuitThreshold?: number;
|
||||
/** 熔断冷却后允许一次半开探测;null(默认)= 只能手动 resetFeedCircuit 恢复 */
|
||||
circuitCooldownMs?: number | null;
|
||||
}
|
||||
|
||||
export interface FetchFeedOptions {
|
||||
signal?: AbortSignal;
|
||||
/** 手动刷新:忽略退避窗口(不忽略熔断) */
|
||||
ignoreBackoff?: boolean;
|
||||
}
|
||||
|
||||
export function initialFeedState(): FeedFetchState {
|
||||
return { consecutiveFailures: 0, circuit: "closed" };
|
||||
}
|
||||
|
||||
/** 手动恢复:关闭熔断并清空退避,保留 ETag/Last-Modified。 */
|
||||
export function resetFeedCircuit(state: FeedFetchState): FeedFetchState {
|
||||
const { nextAttemptAt: _n, circuitOpenedAt: _c, ...rest } = state;
|
||||
return { ...rest, consecutiveFailures: 0, circuit: "closed" };
|
||||
}
|
||||
|
||||
export function computeBackoffMs(failures: number, baseMs: number, maxMs: number): number {
|
||||
if (failures <= 0) return 0;
|
||||
return Math.min(maxMs, baseMs * 2 ** Math.min(failures - 1, 30));
|
||||
}
|
||||
|
||||
/** Retry-After:秒数或 HTTP-date;返回毫秒,无效返回 undefined。 */
|
||||
export function parseRetryAfter(value: string | null | undefined, now: Date): number | undefined {
|
||||
if (!value) return undefined;
|
||||
const v = value.trim();
|
||||
if (/^\d+$/.test(v)) return Number(v) * 1000;
|
||||
const t = Date.parse(v);
|
||||
if (Number.isNaN(t)) return undefined;
|
||||
return Math.max(0, t - now.getTime());
|
||||
}
|
||||
|
||||
async function readCapped(res: Response, maxBytes: number): Promise<Uint8Array> {
|
||||
const len = Number(res.headers.get("content-length") ?? "");
|
||||
// content-length 是压缩后长度,只做早拒;真正的上限在流式读取里按解压后字节执行
|
||||
if (Number.isFinite(len) && len > maxBytes) {
|
||||
await res.body?.cancel().catch(() => {});
|
||||
throw new FeedError("too_large", `响应声明长度 ${len} 超过上限 ${maxBytes}`);
|
||||
}
|
||||
if (!res.body) return new Uint8Array();
|
||||
const reader = res.body.getReader();
|
||||
const chunks: Uint8Array[] = [];
|
||||
let total = 0;
|
||||
for (;;) {
|
||||
const { done, value } = await reader.read();
|
||||
if (done) break;
|
||||
total += value.byteLength;
|
||||
if (total > maxBytes) {
|
||||
await reader.cancel().catch(() => {});
|
||||
throw new FeedError("too_large", `响应体(解压后)超过上限 ${maxBytes} 字节`);
|
||||
}
|
||||
chunks.push(value);
|
||||
}
|
||||
const out = new Uint8Array(total);
|
||||
let off = 0;
|
||||
for (const c of chunks) {
|
||||
out.set(c, off);
|
||||
off += c.byteLength;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function decodeBody(bytes: Uint8Array, contentType: string | null): string {
|
||||
const head = new TextDecoder("latin1").decode(bytes.subarray(0, 256));
|
||||
const label =
|
||||
/charset=([\w-]+)/i.exec(contentType ?? "")?.[1] ??
|
||||
/<\?xml[^>]*encoding=["']([\w-]+)["']/i.exec(head)?.[1] ??
|
||||
"utf-8";
|
||||
try {
|
||||
return new TextDecoder(label).decode(bytes);
|
||||
} catch {
|
||||
return new TextDecoder("utf-8").decode(bytes);
|
||||
}
|
||||
}
|
||||
|
||||
export interface RssFetcher {
|
||||
fetchFeed(url: string, state?: FeedFetchState, opts?: FetchFeedOptions): Promise<FeedFetchResult>;
|
||||
}
|
||||
|
||||
export function createRssFetcher(options: RssFetcherOptions = {}): RssFetcher {
|
||||
const doFetch = options.fetch ?? globalThis.fetch;
|
||||
const now = options.now ?? (() => new Date());
|
||||
const timeoutMs = options.timeoutMs ?? 30_000;
|
||||
const maxBytes = options.maxBytes ?? 50 * 1024 * 1024;
|
||||
const backoff = options.backoff ?? { baseMs: 60_000, maxMs: 6 * 3600_000 };
|
||||
const threshold = options.circuitThreshold ?? 5;
|
||||
const cooldown = options.circuitCooldownMs ?? null;
|
||||
|
||||
function fail(state: FeedFetchState, error: FeedError): FeedFetchResult {
|
||||
const at = now();
|
||||
const failures = state.consecutiveFailures + 1;
|
||||
const wait = Math.max(computeBackoffMs(failures, backoff.baseMs, backoff.maxMs), error.retryAfterMs ?? 0);
|
||||
const next: FeedFetchState = {
|
||||
...state,
|
||||
consecutiveFailures: failures,
|
||||
nextAttemptAt: new Date(at.getTime() + wait).toISOString(),
|
||||
lastError: {
|
||||
kind: error.kind,
|
||||
message: error.message,
|
||||
at: at.toISOString(),
|
||||
...(error.httpStatus !== undefined ? { httpStatus: error.httpStatus } : {}),
|
||||
},
|
||||
};
|
||||
if (failures >= threshold) {
|
||||
next.circuit = "open";
|
||||
next.circuitOpenedAt = at.toISOString();
|
||||
}
|
||||
return { kind: "error", error, state: next };
|
||||
}
|
||||
|
||||
async function fetchFeed(
|
||||
url: string,
|
||||
state: FeedFetchState = initialFeedState(),
|
||||
opts: FetchFeedOptions = {},
|
||||
): Promise<FeedFetchResult> {
|
||||
const at = now();
|
||||
if (state.circuit === "open") {
|
||||
const openedAt = state.circuitOpenedAt ? Date.parse(state.circuitOpenedAt) : 0;
|
||||
const halfOpen = cooldown !== null && at.getTime() - openedAt >= cooldown;
|
||||
if (!halfOpen) {
|
||||
return {
|
||||
kind: "skipped",
|
||||
reason: "circuit_open",
|
||||
...(cooldown !== null ? { retryAt: new Date(openedAt + cooldown).toISOString() } : {}),
|
||||
state,
|
||||
};
|
||||
}
|
||||
} else if (!opts.ignoreBackoff && state.nextAttemptAt && Date.parse(state.nextAttemptAt) > at.getTime()) {
|
||||
return { kind: "skipped", reason: "backoff", retryAt: state.nextAttemptAt, state };
|
||||
}
|
||||
|
||||
const safeUrl = redactUrl(url);
|
||||
const headers: Record<string, string> = {
|
||||
Accept: "application/rss+xml, application/atom+xml, application/xml;q=0.9, text/xml;q=0.8, */*;q=0.1",
|
||||
};
|
||||
if (options.userAgent) headers["User-Agent"] = options.userAgent;
|
||||
if (state.etag) headers["If-None-Match"] = state.etag;
|
||||
if (state.lastModified) headers["If-Modified-Since"] = state.lastModified;
|
||||
|
||||
const timeout = AbortSignal.timeout(timeoutMs);
|
||||
const signal = opts.signal ? AbortSignal.any([opts.signal, timeout]) : timeout;
|
||||
let res: Response;
|
||||
try {
|
||||
res = await doFetch(url, { headers, signal, redirect: "follow" });
|
||||
} catch (e) {
|
||||
if (opts.signal?.aborted) throw e;
|
||||
const err = e as Error;
|
||||
const kind = timeout.aborted ? "timeout" : "network";
|
||||
return fail(state, new FeedError(kind, redactText(`请求 ${safeUrl} 失败:${err.message}`)));
|
||||
}
|
||||
|
||||
if (res.status === 304) {
|
||||
await res.body?.cancel().catch(() => {});
|
||||
const next: FeedFetchState = {
|
||||
...state,
|
||||
consecutiveFailures: 0,
|
||||
circuit: "closed",
|
||||
lastSuccessAt: at.toISOString(),
|
||||
};
|
||||
delete next.nextAttemptAt;
|
||||
delete next.circuitOpenedAt;
|
||||
return { kind: "not_modified", state: next };
|
||||
}
|
||||
if (!res.ok) {
|
||||
await res.body?.cancel().catch(() => {});
|
||||
const retryAfterMs = parseRetryAfter(res.headers.get("retry-after"), at);
|
||||
const kind =
|
||||
res.status === 429 || (res.status === 503 && retryAfterMs !== undefined) ? "rate_limited" : "http_error";
|
||||
return fail(
|
||||
state,
|
||||
new FeedError(kind, `请求 ${safeUrl} 返回 HTTP ${res.status}`, {
|
||||
httpStatus: res.status,
|
||||
...(retryAfterMs !== undefined ? { retryAfterMs } : {}),
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
try {
|
||||
const bytes = await readCapped(res, maxBytes);
|
||||
const text = decodeBody(bytes, res.headers.get("content-type"));
|
||||
const feed = parseFeed(text, { maxItems: options.maxItems ?? 1000, maxChars: maxBytes, fetchedAt: at });
|
||||
const next: FeedFetchState = {
|
||||
...state,
|
||||
consecutiveFailures: 0,
|
||||
circuit: "closed",
|
||||
lastSuccessAt: at.toISOString(),
|
||||
};
|
||||
delete next.nextAttemptAt;
|
||||
delete next.circuitOpenedAt;
|
||||
const etag = res.headers.get("etag");
|
||||
const lastModified = res.headers.get("last-modified");
|
||||
if (etag) next.etag = etag;
|
||||
if (lastModified) next.lastModified = lastModified;
|
||||
return { kind: "ok", feed, state: next, httpStatus: res.status };
|
||||
} catch (e) {
|
||||
if (opts.signal?.aborted) throw e;
|
||||
if (e instanceof FeedError) return fail(state, new FeedError(e.kind, redactText(e.message)));
|
||||
const kind = timeout.aborted ? "timeout" : "network";
|
||||
return fail(state, new FeedError(kind, redactText(`读取 ${safeUrl} 失败:${(e as Error).message}`)));
|
||||
}
|
||||
}
|
||||
|
||||
return { fetchFeed };
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
export { type ApplyFeedOptions, type ApplyFeedResult, applyFeedItems, emptyFeedCursor } from "./cursor";
|
||||
export {
|
||||
computeBackoffMs,
|
||||
createRssFetcher,
|
||||
type FetchFeedOptions,
|
||||
initialFeedState,
|
||||
parseRetryAfter,
|
||||
type RssFetcher,
|
||||
type RssFetcherOptions,
|
||||
resetFeedCircuit,
|
||||
} from "./fetch";
|
||||
export {
|
||||
assertNoDtdEntities,
|
||||
decodeXmlEntities,
|
||||
type ParseFeedOptions,
|
||||
parseFeed,
|
||||
RSS_EXPIRED_MESSAGES,
|
||||
} from "./parse";
|
||||
export { extractSecretParams, REDACTED, redactText, redactUrl, SENSITIVE_QUERY_KEYS, unredactUrl } from "./redact";
|
||||
export { buildRssLinkRequest, extractRssLink, RSS_LINK_CONF, type RssLinkConf, rssSiteDomain } from "./rss-link";
|
||||
export {
|
||||
type FeedCursor,
|
||||
FeedError,
|
||||
type FeedErrorKind,
|
||||
type FeedFetchResult,
|
||||
type FeedFetchState,
|
||||
type FeedItem,
|
||||
type ParsedFeed,
|
||||
} from "./types";
|
||||
@@ -0,0 +1,271 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { XMLParser } from "fast-xml-parser";
|
||||
import { redactUrl } from "./redact";
|
||||
import { FeedError, type FeedItem, type ParsedFeed } from "./types";
|
||||
|
||||
export interface ParseFeedOptions {
|
||||
/** 条目上限,超出截断并置 truncated(默认 1000,同 MP) */
|
||||
maxItems?: number;
|
||||
/** 文本长度上限(字符,默认 50 MiB,同 MP) */
|
||||
maxChars?: number;
|
||||
/** 抓取时间,默认 now */
|
||||
fetchedAt?: Date;
|
||||
}
|
||||
|
||||
/** MP RssHelper.MAX_RSS_ITEMS */
|
||||
const DEFAULT_MAX_ITEMS = 1000;
|
||||
/** MP RssHelper.MAX_RSS_SIZE */
|
||||
const DEFAULT_MAX_CHARS = 50 * 1024 * 1024;
|
||||
|
||||
/**
|
||||
* 预检:拒绝内部 DTD 子集与任何 ENTITY 声明(XXE / billion laughs)。
|
||||
* 仅外部 DOCTYPE 声明(如 RSS 0.91 的 PUBLIC 标识)无子集时放行,但解析器同样不会去取。
|
||||
*/
|
||||
export function assertNoDtdEntities(xml: string): void {
|
||||
if (/<!ENTITY/i.test(xml)) throw new FeedError("forbidden_dtd", "XML 含 ENTITY 声明,已拒绝");
|
||||
const doctype = /<!DOCTYPE[^>[]*(\[)?/i.exec(xml);
|
||||
if (doctype?.[1]) throw new FeedError("forbidden_dtd", "XML 含内部 DTD 子集,已拒绝");
|
||||
}
|
||||
|
||||
const parser = new XMLParser({
|
||||
ignoreAttributes: false,
|
||||
attributeNamePrefix: "@_",
|
||||
removeNSPrefix: true,
|
||||
parseTagValue: false,
|
||||
parseAttributeValue: false,
|
||||
trimValues: true,
|
||||
processEntities: false,
|
||||
htmlEntities: false,
|
||||
cdataPropName: "__cdata",
|
||||
ignoreDeclaration: true,
|
||||
ignorePiTags: true,
|
||||
isArray: (name) => ["item", "entry", "category", "link", "enclosure"].includes(name),
|
||||
});
|
||||
|
||||
const XML_ENTITIES: Record<string, string> = { amp: "&", lt: "<", gt: ">", quot: '"', apos: "'" };
|
||||
|
||||
/** 只解码 XML 预定义实体与数字字符引用;不做任何自定义实体展开。 */
|
||||
export function decodeXmlEntities(s: string): string {
|
||||
return s.replace(/&(#x[0-9a-f]+|#[0-9]+|[a-z]+);/gi, (m, body: string) => {
|
||||
if (body[0] === "#") {
|
||||
const cp = body[1] === "x" || body[1] === "X" ? Number.parseInt(body.slice(2), 16) : Number(body.slice(1));
|
||||
return Number.isFinite(cp) && cp > 0 && cp <= 0x10ffff ? String.fromCodePoint(cp) : m;
|
||||
}
|
||||
return XML_ENTITIES[body.toLowerCase()] ?? m;
|
||||
});
|
||||
}
|
||||
|
||||
type XNode = unknown;
|
||||
|
||||
function textOf(node: XNode): string {
|
||||
if (node == null) return "";
|
||||
if (typeof node === "string") return decodeXmlEntities(node);
|
||||
if (typeof node === "number" || typeof node === "boolean") return String(node);
|
||||
if (Array.isArray(node)) return node.map(textOf).join("");
|
||||
if (typeof node === "object") {
|
||||
const o = node as Record<string, unknown>;
|
||||
const cdata = o.__cdata;
|
||||
const text = o["#text"];
|
||||
return `${text != null ? textOf(text) : ""}${cdata != null ? (Array.isArray(cdata) ? cdata.join("") : String(cdata)) : ""}`.trim();
|
||||
}
|
||||
return "";
|
||||
}
|
||||
|
||||
function attr(node: XNode, name: string): string | undefined {
|
||||
if (node && typeof node === "object" && !Array.isArray(node)) {
|
||||
const v = (node as Record<string, unknown>)[`@_${name}`];
|
||||
return v == null ? undefined : decodeXmlEntities(String(v));
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
function asArray(v: XNode): XNode[] {
|
||||
if (v == null) return [];
|
||||
return Array.isArray(v) ? v : [v];
|
||||
}
|
||||
|
||||
function toIso(s: string | undefined): string | undefined {
|
||||
if (!s) return undefined;
|
||||
const t = Date.parse(s);
|
||||
return Number.isNaN(t) ? undefined : new Date(t).toISOString();
|
||||
}
|
||||
|
||||
function looksLikeUrl(s: string): boolean {
|
||||
return /^[a-z][a-z0-9+.-]*:/i.test(s);
|
||||
}
|
||||
|
||||
function sha256(s: string): string {
|
||||
return createHash("sha256").update(s).digest("hex");
|
||||
}
|
||||
|
||||
interface RawFields {
|
||||
guid: string;
|
||||
title: string;
|
||||
link: string;
|
||||
enclosure: string;
|
||||
enclosureType?: string;
|
||||
size: number;
|
||||
published?: string;
|
||||
categories: string[];
|
||||
description?: string;
|
||||
nickname?: string;
|
||||
}
|
||||
|
||||
function buildItem(raw: RawFields, fetchedAt: string): FeedItem {
|
||||
// 摘要基于脱敏后的值(避免 passkey 轮换误判为更新)
|
||||
const enclosureRedacted = raw.enclosure ? redactUrl(raw.enclosure) : undefined;
|
||||
const linkRedacted = raw.link ? redactUrl(raw.link) : null;
|
||||
const guidBase = raw.guid || linkRedacted || enclosureRedacted || "";
|
||||
const guid = guidBase
|
||||
? looksLikeUrl(guidBase)
|
||||
? redactUrl(guidBase)
|
||||
: guidBase
|
||||
: `digest:${sha256(`${raw.title}\u0000${raw.published ?? ""}`).slice(0, 32)}`;
|
||||
const rawDigest = sha256(
|
||||
JSON.stringify([
|
||||
guid,
|
||||
raw.title,
|
||||
linkRedacted,
|
||||
enclosureRedacted ?? null,
|
||||
raw.size,
|
||||
raw.published ?? null,
|
||||
[...raw.categories].sort(),
|
||||
raw.description ?? null,
|
||||
]),
|
||||
);
|
||||
const item: FeedItem = {
|
||||
guid,
|
||||
title: raw.title,
|
||||
link: linkRedacted,
|
||||
size: raw.size,
|
||||
categories: raw.categories,
|
||||
rawDigest,
|
||||
fetchedAt,
|
||||
};
|
||||
if (enclosureRedacted) {
|
||||
item.enclosureUrl = enclosureRedacted;
|
||||
item.enclosureHasSecret = enclosureRedacted !== raw.enclosure;
|
||||
}
|
||||
if (raw.enclosureType) item.enclosureType = raw.enclosureType;
|
||||
const publishedAt = toIso(raw.published);
|
||||
if (publishedAt) item.publishedAt = publishedAt;
|
||||
if (raw.description) item.description = raw.description;
|
||||
if (raw.nickname) item.nickname = raw.nickname;
|
||||
return item;
|
||||
}
|
||||
|
||||
/**
|
||||
* 单条 item/entry → 字段。迁移自 MP `RssHelper._parse_impl` 的条目循环:
|
||||
* title(空则跳过)、description|summary、link(文本或 href)、enclosure@url(缺失回退 link,两者皆无跳过)、
|
||||
* enclosure@length(纯数字才取)、pubDate|published|updated、creator(nickname)。
|
||||
* guid/categories 为本项目扩展(去重与整理视图需要)。
|
||||
*/
|
||||
function itemFields(it: Record<string, XNode>): RawFields | null {
|
||||
const title = textOf(asArray(it.title)[0]);
|
||||
if (!title) return null;
|
||||
const description = textOf(asArray(it.description)[0]) || textOf(asArray(it.summary)[0]);
|
||||
const firstLink = asArray(it.link)[0];
|
||||
const link = textOf(firstLink) || attr(firstLink, "href") || "";
|
||||
const encNode = asArray(it.enclosure)[0];
|
||||
let enclosure = attr(encNode, "url") ?? "";
|
||||
if (!enclosure && !link) return null;
|
||||
if (!enclosure && link) enclosure = link;
|
||||
const lengthAttr = attr(encNode, "length") ?? "0";
|
||||
const size = /^\d+$/.test(lengthAttr) ? Number(lengthAttr) : 0;
|
||||
const published =
|
||||
textOf(asArray(it.pubDate)[0]) || textOf(asArray(it.published)[0]) || textOf(asArray(it.updated)[0]);
|
||||
const nickname = textOf(asArray(it.creator)[0]);
|
||||
const raw: RawFields = {
|
||||
guid: textOf(asArray(it.guid)[0]) || textOf(asArray(it.id)[0]),
|
||||
title,
|
||||
link,
|
||||
enclosure,
|
||||
size,
|
||||
categories: asArray(it.category)
|
||||
.map((c) => attr(c, "term") ?? textOf(c))
|
||||
.filter((c) => c.length > 0),
|
||||
};
|
||||
const encType = attr(encNode, "type");
|
||||
if (encType) raw.enclosureType = encType;
|
||||
if (published) raw.published = published;
|
||||
if (description) raw.description = description;
|
||||
if (nickname) raw.nickname = nickname;
|
||||
return raw;
|
||||
}
|
||||
|
||||
/** MP xpath `.//item | .//entry`:在整棵树中按文档顺序收集条目节点(深度受限)。 */
|
||||
function collectItems(node: XNode, out: Array<Record<string, XNode>>, depth = 0): void {
|
||||
if (depth > 12 || node == null || typeof node !== "object") return;
|
||||
if (Array.isArray(node)) {
|
||||
for (const n of node) collectItems(n, out, depth + 1);
|
||||
return;
|
||||
}
|
||||
for (const [k, v] of Object.entries(node as Record<string, XNode>)) {
|
||||
if (k.startsWith("@_") || k === "#text" || k === "__cdata") continue;
|
||||
if (k === "item" || k === "entry") {
|
||||
for (const it of asArray(v)) if (it && typeof it === "object") out.push(it as Record<string, XNode>);
|
||||
} else {
|
||||
collectItems(v, out, depth + 1);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** MP `_rss_expired_msg`:站点返回的 RSS 过期提示(MP 返回 None,本项目映射为 FeedError expired)。 */
|
||||
export const RSS_EXPIRED_MESSAGES = [
|
||||
"RSS 链接已过期, 您需要获得一个新的!",
|
||||
"RSS Link has expired, You need to get a new one!",
|
||||
"RSS Link has expired, You need to get new!",
|
||||
];
|
||||
|
||||
/**
|
||||
* 安全解析 RSS 2.0 / RSS 1.0(RDF) / Atom。
|
||||
* 迁移自 MP `app/application/rss.py RssHelper._parse_impl`(MAX_RSS_SIZE 50MB、MAX_RSS_ITEMS 1000、
|
||||
* `resolve_entities=False`、非 `<` 开头拒绝、过期提示识别);本项目额外拒绝 DTD 实体声明并返回脱敏字段。
|
||||
*/
|
||||
export function parseFeed(xml: string, opts: ParseFeedOptions = {}): ParsedFeed {
|
||||
const maxChars = opts.maxChars ?? DEFAULT_MAX_CHARS;
|
||||
const maxItems = opts.maxItems ?? DEFAULT_MAX_ITEMS;
|
||||
if (xml.length > maxChars) throw new FeedError("too_large", `Feed 文本 ${xml.length} 字符超过上限 ${maxChars}`);
|
||||
const trimmed = xml.trim();
|
||||
if (!trimmed) throw new FeedError("not_feed", "RSS 内容为空");
|
||||
if (RSS_EXPIRED_MESSAGES.includes(trimmed)) throw new FeedError("expired", "RSS 链接已过期,需要重新获取");
|
||||
if (!trimmed.startsWith("<")) throw new FeedError("not_feed", "RSS 内容不是有效的 XML 格式");
|
||||
assertNoDtdEntities(xml);
|
||||
let doc: Record<string, XNode>;
|
||||
try {
|
||||
doc = parser.parse(xml) as Record<string, XNode>;
|
||||
} catch (e) {
|
||||
throw new FeedError("parse_error", `XML 解析失败:${(e as Error).message.slice(0, 200)}`);
|
||||
}
|
||||
const fetchedAt = (opts.fetchedAt ?? new Date()).toISOString();
|
||||
const rss = doc.rss as Record<string, XNode> | undefined;
|
||||
const rdf = doc.RDF as Record<string, XNode> | undefined;
|
||||
const atom = doc.feed as Record<string, XNode> | undefined;
|
||||
let format: ParsedFeed["format"];
|
||||
let title: string | null;
|
||||
if (rss && typeof rss === "object") {
|
||||
format = "rss";
|
||||
title = textOf((rss.channel as Record<string, XNode> | undefined)?.title) || null;
|
||||
} else if (rdf && typeof rdf === "object") {
|
||||
format = "rdf";
|
||||
title = textOf((rdf.channel as Record<string, XNode> | undefined)?.title) || null;
|
||||
} else if (atom && typeof atom === "object") {
|
||||
format = "atom";
|
||||
title = textOf(atom.title) || null;
|
||||
} else {
|
||||
throw new FeedError("not_feed", "内容不是 RSS/Atom Feed");
|
||||
}
|
||||
const nodes: Array<Record<string, XNode>> = [];
|
||||
collectItems(doc, nodes);
|
||||
const truncated = nodes.length > maxItems;
|
||||
const items: FeedItem[] = [];
|
||||
for (const n of nodes.slice(0, maxItems)) {
|
||||
try {
|
||||
const raw = itemFields(n);
|
||||
if (raw) items.push(buildItem(raw, fetchedAt));
|
||||
} catch {
|
||||
// MP:单条解析失败跳过
|
||||
}
|
||||
}
|
||||
return { format, title, items, truncated };
|
||||
}
|
||||
@@ -0,0 +1,104 @@
|
||||
/**
|
||||
* URL/文本脱敏。含 passkey 的 URL 不得进入日志、错误信息、模型上下文(REQ §4.1)。
|
||||
* 规则:query 中凭据类参数值替换为 [REDACTED],URL userinfo 一并抹除。
|
||||
*/
|
||||
|
||||
export const REDACTED = "[REDACTED]";
|
||||
|
||||
/** 视为凭据的 query 参数名(小写比较)。 */
|
||||
export const SENSITIVE_QUERY_KEYS: ReadonlySet<string> = new Set([
|
||||
"passkey",
|
||||
"token",
|
||||
"authkey",
|
||||
"auth",
|
||||
"torrent_pass",
|
||||
"rsskey",
|
||||
"apikey",
|
||||
"api_key",
|
||||
"access_token",
|
||||
"secret",
|
||||
"sign",
|
||||
"downhash",
|
||||
"credential",
|
||||
"key",
|
||||
]);
|
||||
|
||||
function isSensitiveKey(key: string, extra?: Iterable<string>): boolean {
|
||||
const k = key.toLowerCase();
|
||||
if (SENSITIVE_QUERY_KEYS.has(k)) return true;
|
||||
if (extra) for (const e of extra) if (e.toLowerCase() === k) return true;
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* 把 URL 中的凭据参数替换为 [REDACTED]。输入不是合法绝对 URL 时按原文做正则兜底,不抛错。
|
||||
* 输出是确定性的(同一 URL 永远得到同一结果),可直接用作去重键。
|
||||
*/
|
||||
export function redactUrl(input: string, extraKeys?: Iterable<string>): string {
|
||||
let url: URL;
|
||||
try {
|
||||
url = new URL(input);
|
||||
} catch {
|
||||
return redactText(input, extraKeys);
|
||||
}
|
||||
if (url.username || url.password) {
|
||||
url.username = "";
|
||||
url.password = "";
|
||||
}
|
||||
// 手工重建 query,避免 URLSearchParams 把 [REDACTED] 转义成 %5BREDACTED%5D
|
||||
if (url.search) {
|
||||
const parts = url.search
|
||||
.slice(1)
|
||||
.split("&")
|
||||
.map((pair) => {
|
||||
const eq = pair.indexOf("=");
|
||||
const rawKey = eq === -1 ? pair : pair.slice(0, eq);
|
||||
let key = rawKey;
|
||||
try {
|
||||
key = decodeURIComponent(rawKey.replace(/\+/g, " "));
|
||||
} catch {
|
||||
// 保留原样
|
||||
}
|
||||
return eq !== -1 && isSensitiveKey(key, extraKeys) ? `${rawKey}=${REDACTED}` : pair;
|
||||
});
|
||||
const out = `${url.origin}${url.pathname}?${parts.join("&")}${url.hash}`;
|
||||
return url.protocol === "magnet:" ? `magnet:?${parts.join("&")}` : out;
|
||||
}
|
||||
return url.toString();
|
||||
}
|
||||
|
||||
const QUERY_PARAM_RE = /([?&;])([A-Za-z0-9_.-]+)=([^&#\s"'<>]*)/g;
|
||||
const USERINFO_RE = /\b([a-z][a-z0-9+.-]*:\/\/)[^/\s@:]+(?::[^/\s@]*)?@/gi;
|
||||
|
||||
/** 对任意文本(错误消息、日志行)中出现的 URL 凭据参数与 userinfo 做脱敏。 */
|
||||
export function redactText(text: string, extraKeys?: Iterable<string>): string {
|
||||
return text
|
||||
.replace(USERINFO_RE, "$1")
|
||||
.replace(QUERY_PARAM_RE, (m, sep: string, key: string) =>
|
||||
isSensitiveKey(key, extraKeys) ? `${sep}${key}=${REDACTED}` : m,
|
||||
);
|
||||
}
|
||||
|
||||
/** 从 URL 中取出被脱敏的凭据参数(调用方自行交给 secret store;本模块不持久化)。 */
|
||||
export function extractSecretParams(input: string, extraKeys?: Iterable<string>): Record<string, string> {
|
||||
const out: Record<string, string> = {};
|
||||
try {
|
||||
const url = new URL(input);
|
||||
for (const [k, v] of url.searchParams) if (isSensitiveKey(k, extraKeys)) out[k] = v;
|
||||
} catch {
|
||||
// 非 URL 返回空
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* 把脱敏 URL 中的 [REDACTED] 还原为真实凭据,只在即将发出请求时于内存中调用,结果不得记录。
|
||||
* 找不到对应 secret 的参数保持 [REDACTED],调用方应视为不可用。
|
||||
*/
|
||||
export function unredactUrl(redacted: string, secrets: Record<string, string>): string {
|
||||
const lower = new Map(Object.entries(secrets).map(([k, v]) => [k.toLowerCase(), v]));
|
||||
return redacted.replace(/([?&;])([A-Za-z0-9_.-]+)=\[REDACTED\]/g, (m, sep: string, key: string) => {
|
||||
const v = lower.get(key.toLowerCase());
|
||||
return v === undefined ? m : `${sep}${key}=${encodeURIComponent(v)}`;
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,60 @@
|
||||
/**
|
||||
* 站点 RSS 地址获取。迁移自 MP e6820a8 `app/application/rss.py`:
|
||||
* `RssHelper.rss_link_conf`(仅 default 与 audiences.me 两项,v0.1 范围)、`_get_site_domain`、`get_rss_link`。
|
||||
* MP 用 lxml xpath `//a[@class='faqlink']/@href` 取最后一个;这里用等价 CSS `a.faqlink` + href。
|
||||
* 请求本身由调用方(站点适配器)带 Cookie 发送并走站点限流;本模块只构造请求与解析结果。
|
||||
*/
|
||||
import { load } from "cheerio";
|
||||
|
||||
export interface RssLinkConf {
|
||||
/** 等价于 MP xpath 的 CSS 选择器与属性 */
|
||||
selector: string;
|
||||
attribute: string;
|
||||
url: string;
|
||||
params: Record<string, string | number>;
|
||||
}
|
||||
|
||||
export const RSS_LINK_CONF: Record<string, RssLinkConf> = {
|
||||
default: {
|
||||
selector: "a.faqlink",
|
||||
attribute: "href",
|
||||
url: "getrss.php",
|
||||
params: { inclbookmarked: 0, itemsmalldescr: 1, showrows: 50, search_mode: 1 },
|
||||
},
|
||||
"audiences.me": {
|
||||
selector: "a.faqlink",
|
||||
attribute: "href",
|
||||
url: "getrss.php",
|
||||
params: { inclbookmarked: 0, itemsmalldescr: 1, showrows: 50, search_mode: 1, torrent_type: 1, exp: 180 },
|
||||
},
|
||||
};
|
||||
|
||||
/** MP `_get_site_domain`:匹配配置域名,否则回退最后两级域名。 */
|
||||
export function rssSiteDomain(url: string): string {
|
||||
const host = new URL(url).hostname.toLowerCase();
|
||||
for (const d of Object.keys(RSS_LINK_CONF)) {
|
||||
if (d !== "default" && (host === d || host.endsWith(`.${d}`))) return d;
|
||||
}
|
||||
const parts = host.split(".");
|
||||
return parts.length >= 2 ? parts.slice(-2).join(".") : host;
|
||||
}
|
||||
|
||||
/** 构造 getrss.php 的 POST 请求(表单参数) */
|
||||
export function buildRssLinkRequest(siteUrl: string): { url: string; form: Record<string, string> } {
|
||||
const conf = RSS_LINK_CONF[rssSiteDomain(siteUrl)] ?? RSS_LINK_CONF.default!;
|
||||
return {
|
||||
url: new URL(conf.url, siteUrl.endsWith("/") ? siteUrl : `${siteUrl}/`).toString(),
|
||||
form: Object.fromEntries(Object.entries(conf.params).map(([k, v]) => [k, String(v)])),
|
||||
};
|
||||
}
|
||||
|
||||
/** 从 getrss.php 返回页提取 RSS 地址(MP 取匹配结果的最后一个)。返回值含 passkey,调用方须当作凭据保存。 */
|
||||
export function extractRssLink(siteUrl: string, html: string): string | null {
|
||||
const conf = RSS_LINK_CONF[rssSiteDomain(siteUrl)] ?? RSS_LINK_CONF.default!;
|
||||
const $ = load(html);
|
||||
const links = $(conf.selector)
|
||||
.map((_, el) => $(el).attr(conf.attribute))
|
||||
.get()
|
||||
.filter((x): x is string => Boolean(x));
|
||||
return links.at(-1) ?? null;
|
||||
}
|
||||
@@ -0,0 +1,296 @@
|
||||
import { createMockRssServer, type MockFeedItem, type MockRssServer } from "@umt/testkit";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import {
|
||||
applyFeedItems,
|
||||
buildRssLinkRequest,
|
||||
computeBackoffMs,
|
||||
createRssFetcher,
|
||||
emptyFeedCursor,
|
||||
extractRssLink,
|
||||
FeedError,
|
||||
type FeedFetchResult,
|
||||
initialFeedState,
|
||||
parseFeed,
|
||||
parseRetryAfter,
|
||||
redactText,
|
||||
redactUrl,
|
||||
resetFeedCircuit,
|
||||
unredactUrl,
|
||||
} from "./index";
|
||||
|
||||
const PASSKEY = "abcdef0123456789secret";
|
||||
|
||||
function sample(n: number, host = "https://pt.example"): MockFeedItem[] {
|
||||
return Array.from({ length: n }, (_, i) => ({
|
||||
guid: `${host}/details.php?id=${1000 + i}`,
|
||||
title: `Show.S01E${String(i + 1).padStart(2, "0")}.2160p.WEB-DL.H265-GRP`,
|
||||
link: `${host}/details.php?id=${1000 + i}`,
|
||||
enclosureUrl: `${host}/download.php?id=${1000 + i}&passkey=${PASSKEY}`,
|
||||
size: 1_000_000 * (i + 1),
|
||||
pubDate: new Date(Date.UTC(2026, 8, 20, 12, i)),
|
||||
categories: ["402"],
|
||||
}));
|
||||
}
|
||||
|
||||
function okFeed(r: FeedFetchResult) {
|
||||
if (r.kind !== "ok") throw new Error(`expected ok, got ${r.kind}`);
|
||||
return r.feed;
|
||||
}
|
||||
|
||||
let server: MockRssServer | undefined;
|
||||
afterEach(async () => {
|
||||
await server?.close();
|
||||
server = undefined;
|
||||
});
|
||||
|
||||
describe("parseFeed 安全解析", () => {
|
||||
it("解析 RSS 2.0 字段并脱敏 enclosure 中的 passkey", () => {
|
||||
const xml = `<?xml version="1.0"?><rss version="2.0"><channel><title>T & F</title>
|
||||
<item><title>Dune & Part.Two.2024.2160p</title><guid>https://pt.example/details.php?id=1</guid>
|
||||
<link>https://pt.example/details.php?id=1</link>
|
||||
<enclosure url="https://pt.example/download.php?id=1&passkey=${PASSKEY}" length="84202833838" type="application/x-bittorrent"/>
|
||||
<pubDate>Sat, 26 Sep 2026 13:14:03 GMT</pubDate><category>401</category><category>电影</category></item>
|
||||
</channel></rss>`;
|
||||
const feed = parseFeed(xml, { fetchedAt: new Date("2026-09-27T00:00:00Z") });
|
||||
expect(feed.format).toBe("rss");
|
||||
expect(feed.title).toBe("T & F");
|
||||
const item = feed.items[0]!;
|
||||
expect(item.title).toBe("Dune & Part.Two.2024.2160p");
|
||||
expect(item.guid).toBe("https://pt.example/details.php?id=1");
|
||||
expect(item.enclosureUrl).toBe("https://pt.example/download.php?id=1&passkey=[REDACTED]");
|
||||
expect(item.enclosureHasSecret).toBe(true);
|
||||
expect(item.size).toBe(84202833838);
|
||||
expect(item.publishedAt).toBe("2026-09-26T13:14:03.000Z");
|
||||
expect(item.categories).toEqual(["401", "电影"]);
|
||||
expect(item.rawDigest).toMatch(/^[0-9a-f]{64}$/);
|
||||
expect(item.fetchedAt).toBe("2026-09-27T00:00:00.000Z");
|
||||
expect(JSON.stringify(feed)).not.toContain(PASSKEY);
|
||||
});
|
||||
|
||||
it("解析 Atom(MP 语义:enclosure 缺失回退首个 link)", () => {
|
||||
const xml = `<?xml version="1.0"?><feed xmlns="http://www.w3.org/2005/Atom"><title>A</title>
|
||||
<entry><id>urn:x:1</id><title>X</title><link rel="alternate" href="https://a/1"/>
|
||||
<link rel="enclosure" href="https://a/dl/1?token=zzz" length="42"/><published>2026-09-01T00:00:00Z</published>
|
||||
<category term="tv"/></entry></feed>`;
|
||||
const feed = parseFeed(xml);
|
||||
expect(feed.format).toBe("atom");
|
||||
expect(feed.items[0]).toMatchObject({
|
||||
guid: "urn:x:1",
|
||||
link: "https://a/1",
|
||||
enclosureUrl: "https://a/1",
|
||||
size: 0,
|
||||
categories: ["tv"],
|
||||
});
|
||||
});
|
||||
|
||||
it("拒绝外部实体(XXE)", () => {
|
||||
const xml = `<?xml version="1.0"?><!DOCTYPE rss [<!ENTITY xxe SYSTEM "file:///etc/passwd">]>
|
||||
<rss><channel><item><title>&xxe;</title></item></channel></rss>`;
|
||||
expect(() => parseFeed(xml)).toThrowError(FeedError);
|
||||
try {
|
||||
parseFeed(xml);
|
||||
} catch (e) {
|
||||
expect((e as FeedError).kind).toBe("forbidden_dtd");
|
||||
}
|
||||
});
|
||||
|
||||
it("拒绝实体膨胀(billion laughs)与内部 DTD 子集", () => {
|
||||
const lol = `<?xml version="1.0"?><!DOCTYPE lolz [<!ENTITY lol "lol"><!ENTITY lol2 "&lol;&lol;&lol;">]><rss><channel><title>&lol2;</title></channel></rss>`;
|
||||
expect(() => parseFeed(lol)).toThrow(/ENTITY/);
|
||||
const subset = `<?xml version="1.0"?><!DOCTYPE rss [ <!ELEMENT rss ANY> ]><rss><channel/></rss>`;
|
||||
expect(() => parseFeed(subset)).toThrow(/DTD/);
|
||||
});
|
||||
|
||||
it("条目数超过上限截断,非 Feed 报 not_feed", () => {
|
||||
const xml = `<rss><channel>${Array.from({ length: 20 }, (_, i) => `<item><guid>g${i}</guid><title>t${i}</title><link>https://x/${i}</link></item>`).join("")}</channel></rss>`;
|
||||
const feed = parseFeed(xml, { maxItems: 5 });
|
||||
expect(feed.items).toHaveLength(5);
|
||||
expect(feed.truncated).toBe(true);
|
||||
expect(() => parseFeed("<html><body>hi</body></html>")).toThrow(/不是 RSS/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("MP RssHelper 条目语义", () => {
|
||||
it("无标题跳过、无 enclosure 且无 link 跳过、creator → nickname、过期提示识别", () => {
|
||||
const xml = `<rss xmlns:dc="http://purl.org/dc/elements/1.1/"><channel>
|
||||
<item><title></title><link>https://x/1</link></item>
|
||||
<item><title>no link</title></item>
|
||||
<item><title>ok</title><link>https://x/3</link><dc:creator>豆友</dc:creator><pubDate>Sat, 26 Sep 2026 13:14:03 GMT</pubDate></item>
|
||||
</channel></rss>`;
|
||||
const feed = parseFeed(xml);
|
||||
expect(feed.items).toHaveLength(1);
|
||||
expect(feed.items[0]).toMatchObject({ title: "ok", enclosureUrl: "https://x/3", nickname: "豆友", size: 0 });
|
||||
expect(() => parseFeed("RSS Link has expired, You need to get a new one!")).toThrow(/过期/);
|
||||
expect(() => parseFeed("not xml")).toThrow(/不是有效的 XML/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("站点 RSS 地址获取(MP get_rss_link)", () => {
|
||||
it("观众使用专用参数,默认站点回退 default;取最后一个 faqlink", () => {
|
||||
const req = buildRssLinkRequest("https://audiences.me/");
|
||||
expect(req.url).toBe("https://audiences.me/getrss.php");
|
||||
expect(req.form).toMatchObject({ torrent_type: "1", exp: "180", showrows: "50" });
|
||||
expect(buildRssLinkRequest("https://pt.example.org").form.torrent_type).toBeUndefined();
|
||||
const html = `<a class="faqlink" href="https://a/rss?x=1">1</a><a class="faqlink" href="https://a/torrentrss.php?passkey=${PASSKEY}">2</a>`;
|
||||
expect(extractRssLink("https://audiences.me/", html)).toContain("torrentrss.php");
|
||||
});
|
||||
});
|
||||
|
||||
describe("redact", () => {
|
||||
it("redactUrl/redactText/unredactUrl", () => {
|
||||
const u = `https://user:[email protected]/rss.php?rows=50&passkey=${PASSKEY}&linktype=dl`;
|
||||
const r = redactUrl(u);
|
||||
expect(r).toBe("https://pt.example/rss.php?rows=50&passkey=[REDACTED]&linktype=dl");
|
||||
expect(redactText(`fetch failed for ${u}: ECONNRESET`)).not.toContain(PASSKEY);
|
||||
expect(redactText(`fetch failed for ${u}`)).not.toContain("pw@");
|
||||
expect(unredactUrl(r, { passkey: PASSKEY })).toBe(
|
||||
`https://pt.example/rss.php?rows=50&passkey=${PASSKEY}&linktype=dl`,
|
||||
);
|
||||
expect(redactUrl("https://api.m-team.cc/api/rss/dlv2?sign=abc&t=1&tid=9")).toContain("sign=[REDACTED]");
|
||||
});
|
||||
});
|
||||
|
||||
describe("createRssFetcher + mock RSS", () => {
|
||||
it("ETag/Last-Modified 条件请求命中 304", async () => {
|
||||
server = await createMockRssServer({ items: sample(3), passkey: PASSKEY });
|
||||
const fetcher = createRssFetcher();
|
||||
const r1 = await fetcher.fetchFeed(server.feedUrl);
|
||||
expect(okFeed(r1).items).toHaveLength(3);
|
||||
expect(r1.state.etag).toBeTruthy();
|
||||
const r2 = await fetcher.fetchFeed(server.feedUrl, r1.state);
|
||||
expect(r2.kind).toBe("not_modified");
|
||||
expect(server.notModifiedCount).toBe(1);
|
||||
expect(server.requests[1]!.headers["if-none-match"]).toBe(r1.state.etag);
|
||||
server.addItem(sample(1, "https://other.example")[0]!);
|
||||
const r3 = await fetcher.fetchFeed(server.feedUrl, r2.state);
|
||||
expect(okFeed(r3).items).toHaveLength(4);
|
||||
});
|
||||
|
||||
it("5xx + Retry-After:退避窗口取 max(指数退避, Retry-After),窗口内跳过", async () => {
|
||||
server = await createMockRssServer({ items: sample(2) });
|
||||
let now = new Date("2026-09-27T00:00:00Z");
|
||||
const fetcher = createRssFetcher({ now: () => now, backoff: { baseMs: 10_000, maxMs: 3600_000 } });
|
||||
server.failNext(1, { status: 503, retryAfter: "120" });
|
||||
const r1 = await fetcher.fetchFeed(server.feedUrl);
|
||||
expect(r1.kind).toBe("error");
|
||||
if (r1.kind !== "error") return;
|
||||
expect(r1.error.kind).toBe("rate_limited");
|
||||
expect(r1.error.retryAfterMs).toBe(120_000);
|
||||
expect(r1.state.nextAttemptAt).toBe("2026-09-27T00:02:00.000Z");
|
||||
const r2 = await fetcher.fetchFeed(server.feedUrl, r1.state);
|
||||
expect(r2).toMatchObject({ kind: "skipped", reason: "backoff" });
|
||||
expect(server.requests).toHaveLength(1);
|
||||
now = new Date("2026-09-27T00:02:01Z");
|
||||
const r3 = await fetcher.fetchFeed(server.feedUrl, r2.state);
|
||||
expect(r3.kind).toBe("ok");
|
||||
expect(r3.state.consecutiveFailures).toBe(0);
|
||||
expect(r3.state.nextAttemptAt).toBeUndefined();
|
||||
});
|
||||
|
||||
it("computeBackoffMs 指数增长封顶,parseRetryAfter 支持 HTTP-date", () => {
|
||||
expect([1, 2, 3, 4].map((n) => computeBackoffMs(n, 1000, 5000))).toEqual([1000, 2000, 4000, 5000]);
|
||||
const now = new Date("2026-09-27T00:00:00Z");
|
||||
expect(parseRetryAfter("Sun, 27 Sep 2026 00:00:30 GMT", now)).toBe(30_000);
|
||||
expect(parseRetryAfter("garbage", now)).toBeUndefined();
|
||||
});
|
||||
|
||||
it("连续失败 N 次进入熔断,手动恢复后可继续抓取", async () => {
|
||||
server = await createMockRssServer({ items: sample(1) });
|
||||
const fetcher = createRssFetcher({ circuitThreshold: 3, backoff: { baseMs: 0, maxMs: 0 } });
|
||||
server.failNext(3, { status: 500 });
|
||||
let state = initialFeedState();
|
||||
for (let i = 0; i < 3; i++) state = (await fetcher.fetchFeed(server.feedUrl, state)).state;
|
||||
expect(state.circuit).toBe("open");
|
||||
expect(state.consecutiveFailures).toBe(3);
|
||||
const skipped = await fetcher.fetchFeed(server.feedUrl, state, { ignoreBackoff: true });
|
||||
expect(skipped).toMatchObject({ kind: "skipped", reason: "circuit_open" });
|
||||
expect(server.requests).toHaveLength(3);
|
||||
const ok = await fetcher.fetchFeed(server.feedUrl, resetFeedCircuit(state));
|
||||
expect(ok.kind).toBe("ok");
|
||||
expect(ok.state.circuit).toBe("closed");
|
||||
});
|
||||
|
||||
it("熔断冷却后半开探测", async () => {
|
||||
server = await createMockRssServer({ items: sample(1) });
|
||||
let now = new Date("2026-09-27T00:00:00Z");
|
||||
const fetcher = createRssFetcher({ now: () => now, circuitThreshold: 1, circuitCooldownMs: 60_000 });
|
||||
server.failNext(1, { status: 502 });
|
||||
const r1 = await fetcher.fetchFeed(server.feedUrl);
|
||||
expect(r1.state.circuit).toBe("open");
|
||||
expect((await fetcher.fetchFeed(server.feedUrl, r1.state)).kind).toBe("skipped");
|
||||
now = new Date("2026-09-27T00:01:00Z");
|
||||
expect((await fetcher.fetchFeed(server.feedUrl, r1.state)).kind).toBe("ok");
|
||||
});
|
||||
|
||||
it("错误信息与状态中不含 passkey(HTTP 错误与网络错误)", async () => {
|
||||
server = await createMockRssServer({ items: sample(1), passkey: "the-right-one" });
|
||||
const fetcher = createRssFetcher();
|
||||
const wrong = `${server.baseUrl}/rss.xml?passkey=${PASSKEY}`;
|
||||
const r = await fetcher.fetchFeed(wrong);
|
||||
expect(r.kind).toBe("error");
|
||||
expect(JSON.stringify(r)).not.toContain(PASSKEY);
|
||||
if (r.kind === "error") expect(r.error.message).toContain("[REDACTED]");
|
||||
const dead = `http://127.0.0.1:1/rss.xml?passkey=${PASSKEY}`;
|
||||
const r2 = await fetcher.fetchFeed(dead);
|
||||
expect(r2.kind).toBe("error");
|
||||
expect(JSON.stringify(r2)).not.toContain(PASSKEY);
|
||||
});
|
||||
|
||||
it("响应体超过上限拒绝;恶意 DTD 响应计为失败", async () => {
|
||||
server = await createMockRssServer({ items: sample(1) });
|
||||
server.setRawBody(`<rss><channel><title>${"x".repeat(10_000)}</title></channel></rss>`);
|
||||
const fetcher = createRssFetcher({ maxBytes: 1024 });
|
||||
const r = await fetcher.fetchFeed(server.feedUrl);
|
||||
expect(r.kind === "error" && r.error.kind).toBe("too_large");
|
||||
server.setRawBody(`<!DOCTYPE r [<!ENTITY e SYSTEM "file:///etc/passwd">]><rss><channel/></rss>`);
|
||||
const r2 = await createRssFetcher().fetchFeed(server.feedUrl);
|
||||
expect(r2.kind === "error" && r2.error.kind).toBe("forbidden_dtd");
|
||||
expect(r2.state.consecutiveFailures).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe("重叠窗口去重", () => {
|
||||
it("乱序/置顶不产生新条目,同 GUID 内容变化视为更新", async () => {
|
||||
server = await createMockRssServer({ items: sample(5) });
|
||||
server.setConditional(false);
|
||||
const fetcher = createRssFetcher();
|
||||
const a = applyFeedItems(emptyFeedCursor(), okFeed(await fetcher.fetchFeed(server.feedUrl)).items);
|
||||
expect(a.newItems).toHaveLength(5);
|
||||
server.shuffle(7);
|
||||
server.pinItem("https://pt.example/details.php?id=1003");
|
||||
const b = applyFeedItems(a.cursor, okFeed(await fetcher.fetchFeed(server.feedUrl)).items);
|
||||
expect(b.newItems).toHaveLength(0);
|
||||
expect(b.updatedItems).toHaveLength(0);
|
||||
expect(b.unchangedCount).toBe(5);
|
||||
server.updateItem("https://pt.example/details.php?id=1001", { title: "Show.S01E02.REPACK.2160p.WEB-DL.H265-GRP" });
|
||||
server.addItem({
|
||||
guid: "https://pt.example/details.php?id=2000",
|
||||
title: "New.Movie.2026.1080p",
|
||||
link: "https://pt.example/details.php?id=2000",
|
||||
});
|
||||
const c = applyFeedItems(b.cursor, okFeed(await fetcher.fetchFeed(server.feedUrl)).items);
|
||||
expect(c.newItems.map((i) => i.guid)).toEqual(["https://pt.example/details.php?id=2000"]);
|
||||
expect(c.updatedItems.map((i) => i.title)).toEqual(["Show.S01E02.REPACK.2160p.WEB-DL.H265-GRP"]);
|
||||
expect(c.possibleGap).toBe(false);
|
||||
expect(c.cursor.lastGuidSet).toHaveLength(6);
|
||||
});
|
||||
|
||||
it("与上次窗口零重叠标记疑似缺口;窗口按上限裁剪", () => {
|
||||
const mk = (g: string) => ({
|
||||
guid: g,
|
||||
title: g,
|
||||
link: null,
|
||||
size: 0,
|
||||
categories: [],
|
||||
rawDigest: g,
|
||||
fetchedAt: "2026-09-27T00:00:00.000Z",
|
||||
});
|
||||
const a = applyFeedItems(emptyFeedCursor(), [mk("a"), mk("b"), mk("c")], { windowSize: 2 });
|
||||
expect(a.possibleGap).toBe(false);
|
||||
expect(a.cursor.lastGuidSet).toEqual(["a", "b"]);
|
||||
expect(Object.keys(a.cursor.digests)).toEqual(["a", "b"]);
|
||||
const b = applyFeedItems(a.cursor, [mk("x"), mk("y")]);
|
||||
expect(b.possibleGap).toBe(true);
|
||||
expect(b.cursor.lastFetchedAt).toBe("2026-09-27T00:00:00.000Z");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,92 @@
|
||||
/**
|
||||
* RSS/Atom 适配器自有类型。FeedItem 对应 MP `RssHelper._parse_impl` 返回 dict:
|
||||
* title→title、enclosure→enclosureUrl、size→size、description→description、link→link、pubdate→publishedAt、nickname→nickname;
|
||||
* guid/categories/rawDigest/fetchedAt 为本项目扩展(去重/增量)。
|
||||
*/
|
||||
/** RSS/Atom 适配器自有类型(不依赖 @umt/contracts;W2 负责映射到 observation/feed_source)。 */
|
||||
|
||||
export interface FeedItem {
|
||||
/** Feed 内稳定标识:guid → link → enclosure → 标题+时间摘要;URL 形态已脱敏 */
|
||||
guid: string;
|
||||
title: string;
|
||||
/** 详情页链接(已脱敏) */
|
||||
link: string | null;
|
||||
/** MP `enclosure`:下载链接,缺失时回退 link(已脱敏,凭据参数为 [REDACTED];需要时用 unredactUrl 在内存中还原) */
|
||||
enclosureUrl?: string;
|
||||
/** 原 enclosure 是否含凭据参数 */
|
||||
enclosureHasSecret?: boolean;
|
||||
enclosureType?: string;
|
||||
/** MP `size`:enclosure@length,缺失为 0 */
|
||||
size: number;
|
||||
/** MP `nickname`(dc:creator) */
|
||||
nickname?: string;
|
||||
/** 来源发布时间(UTC ISO),≠ 发现时间 */
|
||||
publishedAt?: string;
|
||||
categories: string[];
|
||||
description?: string;
|
||||
/** 原始条目规范化后的 sha256(同 GUID 摘要变化 = 条目更新) */
|
||||
rawDigest: string;
|
||||
/** 本次抓取时间(UTC ISO) */
|
||||
fetchedAt: string;
|
||||
}
|
||||
|
||||
export interface ParsedFeed {
|
||||
format: "rss" | "atom" | "rdf";
|
||||
title: string | null;
|
||||
items: FeedItem[];
|
||||
/** 条目数超过上限被截断 */
|
||||
truncated: boolean;
|
||||
}
|
||||
|
||||
export type FeedErrorKind =
|
||||
| "forbidden_dtd"
|
||||
| "expired"
|
||||
| "too_large"
|
||||
| "not_feed"
|
||||
| "parse_error"
|
||||
| "http_error"
|
||||
| "rate_limited"
|
||||
| "network"
|
||||
| "timeout";
|
||||
|
||||
/** 所有 message 在构造时已经脱敏。 */
|
||||
export class FeedError extends Error {
|
||||
readonly kind: FeedErrorKind;
|
||||
readonly httpStatus?: number;
|
||||
readonly retryAfterMs?: number;
|
||||
constructor(kind: FeedErrorKind, message: string, opts: { httpStatus?: number; retryAfterMs?: number } = {}) {
|
||||
super(message);
|
||||
this.name = "FeedError";
|
||||
this.kind = kind;
|
||||
if (opts.httpStatus !== undefined) this.httpStatus = opts.httpStatus;
|
||||
if (opts.retryAfterMs !== undefined) this.retryAfterMs = opts.retryAfterMs;
|
||||
}
|
||||
}
|
||||
|
||||
/** 单个 Feed 的抓取状态;纯数据,由调用方持久化(feed_source 列或 JSON)。 */
|
||||
export interface FeedFetchState {
|
||||
etag?: string;
|
||||
lastModified?: string;
|
||||
consecutiveFailures: number;
|
||||
/** 退避截止(UTC ISO),之前的调度直接 skipped */
|
||||
nextAttemptAt?: string;
|
||||
circuit: "closed" | "open";
|
||||
circuitOpenedAt?: string;
|
||||
lastSuccessAt?: string;
|
||||
lastError?: { kind: FeedErrorKind; message: string; at: string; httpStatus?: number };
|
||||
}
|
||||
|
||||
export type FeedFetchResult =
|
||||
| { kind: "ok"; feed: ParsedFeed; state: FeedFetchState; httpStatus: number }
|
||||
| { kind: "not_modified"; state: FeedFetchState }
|
||||
| { kind: "skipped"; reason: "circuit_open" | "backoff"; retryAt?: string; state: FeedFetchState }
|
||||
| { kind: "error"; error: FeedError; state: FeedFetchState };
|
||||
|
||||
/** 重叠窗口游标:不依赖递增 ID,保存最近见过的 GUID 与内容摘要。 */
|
||||
export interface FeedCursor {
|
||||
/** 最近见过的 GUID(新→旧),长度受窗口上限约束 */
|
||||
lastGuidSet: string[];
|
||||
/** guid → rawDigest,用于识别同 GUID 内容变化 */
|
||||
digests: Record<string, string>;
|
||||
lastFetchedAt: string | null;
|
||||
}
|
||||
Reference in new issue
Block a user