fix(sites): HHClub parsing, resource list layout and request restraint (re-landed)

Re-lands 901b01d, which was reverted wholesale by 28974d1 (BR-2 committed from the shared index). The agent UI files 901b01d had rolled back are left at their AGUI-2 versions; contracts, routes, db types and the runbook part already reached main through later commits, so this carries the rest. HHClub (憨憨) was checked read-only against a real account (one detail request; list, user data and health came from the product's own fetch). Parsing fixes: - list titles carried the site's new-torrent marker "[新]" (58 of 100 rows; the detail page title has none); stripped with a replace filter - refreshing a detail overwrote seeders / leechers / snatched / category name / H&R with null because the HHClub div detail page lacks them; detail values now fall back to the list snapshot (mergeDetailOverList), and an unknown detail promotion keeps the list promotion - level "贵宾" is VIP (umt.vipLevels) - sanitized minimal fixture HHANCLUB_LIST_PAGE_2026 + tests; README marks HHClub as verified 2026-09-29 Size, counts, category ids/names, +08:00 time and subtitle were correct. Request restraint: - site definitions may carry umt.defaults; merged with settings.sites (longer intervals, fewer pages): new settings defaultSnapshotMaxPages 1, defaultHealthCheckIntervalSec 6 h, defaultDetailCacheSec 24 h; default burst 2. HHClub: snapshot 60 min page 1 only, health 6 h, search concurrency 1, detail cache 24 h, 1 token / 3 s burst 2 - periodic / manual snapshots are capped to that page count; scheduled health checks skip sites checked within the interval; detail refresh is served from cache inside the window; live searches on one site queue - every rate-limiter token is counted per site per hour (site_request_count, migration 0046); GET /sites/{id}/request-stats; site overview and resource list show requests in the last 24 hours Resource list: two-layer rows. Main line: title + spec tags (episodes, resolution, source, codec, HDR, group, 国语/中字, site tags) + promotion with time left + H&R. Second line: subtitle (one line, ellipsis) and fixed-width category / size / seed-leech-snatched / upload time (relative, absolute on hover) columns, stale badge, inventory overlay; folds into a card on phones. Also aligns identity_candidate.normalized_title order in db types with the migrations (schema drift test). Takes effect after the stack is rebuilt (migration 0046). Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-09-29 16:00:34 +08:00
1 parent 99e335d13c
commit 6e9e8efec2
@@ -88,14 +88,33 @@ PT 站点适配器,按 MoviePilot v3(提交 e6820a8)迁移:通用配置驱动 sp

 | 状态 | 站点 |
 | --- | --- |
-| 启用(主力,仍须真实账号核验) | audiences 观众、hhanclub 憨憨、chdbits 彩虹岛、mteam 馒头 |
+| 启用,**已实测 2026-09-29**(真实账号只读核对) | hhanclub 憨憨 |
+| 启用(主力,仍须真实账号核验) | audiences 观众、chdbits 彩虹岛、mteam 馒头 |
 | `enabled:false`,**未实测** | agsvpt、hdarea、hddolby、hdfans、hdhome、hdsky、hdtime、hdvideo、keepfrds、nicept、ourbits、piggo、pter、pthome、pttime、soulvoice、ssd、tjupt、ttg |

 未实测站点只经 schema 校验,选择器是 MovieClaw 作者在 2026-08/09 的实测值,未经本项目核验。TTG 需要自定义适配器(中文分类、`/t/<id>` 路径、做种/下载同列),通用 spider 结果不可靠;OurBits 的 `cat[]` 分类参数已由定义表达;hddolby 在 MovieClaw 中按 HTML 站点配置,MP 则用 API spider(`HDDolby`),两条路径待账号确认后二选一。

 ### 待真实站点核验(W1-3c 新增)

-1. HHClub:`div.torrent-table-sub-info` 卡片结构、`?cat[]=` 分类链接(若 URL 编码为 `cat%5B%5D=` 则 `\d+` 会取错,需改 querystring 过滤)、`promotion-tag-*` 新旧类名、`div.torrent-info-hr`、详情页 Tailwind 选择器、分页 0 起。
+1. HHClub(2026-09-29 已实测,见下节):仍未见到的只有促销(`promotion-tag-*`)与 H&R(`div.torrent-info-hr`)命中的实例——当次第 1 页 100 条全部无促销、无 H&R,类名与 Jackett 现行定义一致,暂不改。
+
+### HHClub 实测记录(2026-09-29)
+
+真实账号只读核对,对站点共发 1 次详情请求(列表、用户数据、健康检测用添加站点时产品自己抓的结果;下载令牌只核对列表里的下载链接形态,没有下载)。
+
+| 项 | 结果 |
+| --- | --- |
+| 列表 `div.torrent-table-sub-info` 卡片、分类 `?cat[]=` 链接、大小 / 做种 / 下载 / 完成 | 正确,全为数字;分类 401–405/412 原始 ID 与名称都在 |
+| 标题 | **有误,已修**:标题链接文本末尾带新种标记「[新]」(100 条里 58 条;详情页标题没有),`title` 加 `replace` 过滤 |
+| 副标题 | 正确,纯文本,无 HTML 残片 |
+| 发布时间 | 正确:`span[title]` 绝对时间按 `+08:00` 转 UTC,相对时间只进 `date_elapsed` |
+| 详情页 | 标题 / 副标题取到;没有做种、完成、分类名、基本信息行,**刷新详情会用 null 冲掉列表值,已修**(服务端合并,见 `mergeDetailOverList`) |
+| 用户数据(`NexusHhanclub`) | 正确;等级「贵宾」之前 `vip` 为 null,**已加 `umt.vipLevels`** |
+| 健康检测 / 登录标志 `a[href='logout.php']` | 正确 |
+| 下载链接 | 列表给 `download.php?id=…&passkey=…`,入库时 passkey 脱敏 |
+| 标签(官方 / 中字 / 国语等) | 列表没有可用的标签选择器;界面按标题解析与副标题原文生成规格标签 |
+
+访问默认(`umt.defaults`,与 `settings.sites` 取更保守的一方):快照每 60 分钟只抓第 1 页、健康检测 6 小时、用户数据 6 小时、实时搜索并发 1、详情缓存 24 小时;限流 3 秒 1 个令牌、突发 2。fixture:`@umt/testkit` 的 `HHANCLUB_LIST_PAGE_2026`(脱敏最小片段)。
 2. CHDBits:域名 `ptchdbits.co`、`font.subtitle`、`td.embedded > span[title]` 促销截止、`a[href='logout.php']`。
 3. 专用 spider 的 JSON 字段均按 MP 读取的键编写 fixture,真实响应待核;HDDolby `hr` 字段语义、`page_number` 起点;TorrentLeech 时区;Yema `tagList` 类型;直链缺失时的回退地址(`TODO(verify)`)。
 4. 用户信息解析器 fixture 均为最小自编 HTML/JSON;`normalizeDatetime` 不处理相对时间(MP 用 dateparser)。
@@ -1,4 +1,10 @@
-import { CHDBITS_LIST_PAGE, CHDBITS_LOGIN_PAGE, HHANCLUB_DETAIL_PAGE, HHANCLUB_LIST_PAGE } from "@umt/testkit";
+import {
+  CHDBITS_LIST_PAGE,
+  CHDBITS_LOGIN_PAGE,
+  HHANCLUB_DETAIL_PAGE,
+  HHANCLUB_LIST_PAGE,
+  HHANCLUB_LIST_PAGE_2026,
+} from "@umt/testkit";
 import { describe, expect, it } from "vitest";
 import { getSiteDefinition, listSiteDefinitions } from "../registry.ts";
 import { parseDetailPage } from "../spider/detail.ts";
@@ -102,6 +108,44 @@ describe("HHClub 定义(div 卡片模板)", () => {
    });
  });

+  it("2026-09-29 实测结构:去掉新种标记「[新]」,短剧 412、完成数 0、无促销、时间按 +08:00、下载链接 passkey 脱敏", () => {
+    const [a, b] = parse(HHANCLUB_LIST_PAGE_2026);
+    expect(a).toMatchObject({
+      siteReleaseId: "91001",
+      title: "示例短剧 1080p WEB-DL H265 AAC-HHWEB",
+      subtitle: "示例短剧 | 全30集 | 1080p",
+      category: { id: "412", name: "短剧" },
+      sizeBytes: Math.round(185.38 * 1024 ** 2),
+      seeders: 1,
+      leechers: 58,
+      snatched: 0,
+      promotion: { type: "none", downloadFactor: 1, uploadFactor: 1, expiresAt: null },
+      hr: false,
+      uploadedAt: "2026-09-29T06:07:03.000Z",
+      detailUrl: "https://hhanclub.net/details.php?id=91001&hit=1",
+    });
+    expect(a!.downloadUrl).not.toContain("0000000000000000");
+    expect(b).toMatchObject({
+      title: "Example Show S01 2026 2160p WEB-DL H.264 AAC-HHWEB",
+      category: { id: "402", name: "电视剧" },
+      snatched: 187,
+      uploadedAt: "2026-09-29T04:03:28.000Z",
+    });
+  });
+
+  it("站点访问默认:快照只抓 1 页、健康检测 6 小时、详情缓存 24 小时、搜索并发 1;3 秒 1 个令牌、突发 2", () => {
+    expect(hhanclubDefinition.umt.defaults).toEqual({
+      snapshotIntervalSec: 3600,
+      snapshotMaxPages: 1,
+      healthCheckIntervalSec: 21600,
+      userDataIntervalSec: 21600,
+      searchConcurrency: 1,
+      detailCacheSec: 86400,
+    });
+    expect(hhanclubDefinition.umt.rateLimit).toEqual({ minIntervalMs: 3000, burst: 2 });
+    expect(hhanclubDefinition.umt.vipLevels).toContain("贵宾");
+  });
+
  it("浏览从 page=0 起(MovieClaw page_offset -1);mtype=tv 选 402/403/407/412", () => {
    expect(new SiteSpider(hhanclubDefinition, { page: 0 }).getSearchUrl()).toBe(
      "https://hhanclub.net/torrents.php?page=0",
@@ -86,7 +86,17 @@
        "attribute": "title"
      },
      "title": {
-        "text": "{% if fields['title_optional'] %}{{ fields['title_optional'] }}{% else %}{{ fields['title_default'] }}{% endif %}"
+        "text": "{% if fields['title_optional'] %}{{ fields['title_optional'] }}{% else %}{{ fields['title_default'] }}{% endif %}",
+        "filters": [
+          {
+            "name": "replace",
+            "args": ["[新]", ""]
+          },
+          {
+            "name": "replace",
+            "args": ["(新)", ""]
+          }
+        ]
      },
      "details": {
        "selector": "div.torrent-title a[href*='details.php']",
@@ -175,13 +185,13 @@
    }
  },
  "umt": {
-    "version": "[email protected]",
+    "version": "[email protected]+umt.1",
    "kind": "nexusphp",
    "displayName": "憨憨",
    "timezone": "+08:00",
    "rateLimit": {
      "minIntervalMs": 3000,
-      "burst": 1
+      "burst": 2
    },
    "capabilities": {
      "list": true,
@@ -249,6 +259,15 @@
        "douban": "a[href*='douban.com']::attr(href)",
        "fileList": "td.rowfollow > ul > li::text"
      }
+    },
+    "vipLevels": ["贵宾", "VIP"],
+    "defaults": {
+      "snapshotIntervalSec": 3600,
+      "snapshotMaxPages": 1,
+      "healthCheckIntervalSec": 21600,
+      "userDataIntervalSec": 21600,
+      "searchConcurrency": 1,
+      "detailCacheSec": 86400
    }
  }
 }
@@ -74,6 +74,14 @@ const CapabilitiesSchema = z.strictObject({
  download: z.boolean(),
  rss: z.boolean(),
 });
+const AccessDefaultsSchema = z.strictObject({
+  snapshotIntervalSec: z.number().int().min(300).optional(),
+  snapshotMaxPages: z.number().int().min(1).max(10).optional(),
+  healthCheckIntervalSec: z.number().int().min(300).optional(),
+  userDataIntervalSec: z.number().int().min(300).optional(),
+  searchConcurrency: z.number().int().min(1).max(4).optional(),
+  detailCacheSec: z.number().int().min(0).optional(),
+});
 const RateLimitSchema = z.strictObject({ minIntervalMs: z.number().int().min(0), burst: z.number().int().min(1) });
 const TimezoneSchema = z.string().regex(/^[+-]\d{2}:\d{2}$/, "时区须为 ±HH:MM 偏移");
 const PairSchema = z.tuple([z.string(), z.string()]);
@@ -87,6 +95,8 @@ const UmtNexusSchema = z.strictObject({
  authSupported: z.array(z.enum(["cookie", "credential", "apikey"])).optional(),
  timezone: TimezoneSchema,
  rateLimit: RateLimitSchema,
+  /** 站点级访问默认(与 settings.sites 取更保守的一方) */
+  defaults: AccessDefaultsSchema.optional(),
  capabilities: CapabilitiesSchema,
  categoryNames: z.record(z.string(), z.string()).optional(),
  loginSuccess: z.string().optional(),
@@ -121,6 +121,21 @@ export interface IndexerDefinition {
  umt: UmtSiteExtension;
 }

+export interface SiteAccessDefaults {
+  /** 定时快照周期(秒) */
+  snapshotIntervalSec?: number;
+  /** 快照(定时 / 手动)最多抓几页列表 */
+  snapshotMaxPages?: number;
+  /** 健康检测最短间隔(秒) */
+  healthCheckIntervalSec?: number;
+  /** 用户数据刷新间隔(秒;定时任务 sites.user_data 的周期,仅作说明与校验) */
+  userDataIntervalSec?: number;
+  /** 同站实时搜索并发 */
+  searchConcurrency?: number;
+  /** 详情页结果缓存(秒):期内不重复抓详情 */
+  detailCacheSec?: number;
+}
+
 export interface UmtSiteExtension {
  /** 定义版本,入 site_snapshot */
  version: string;
@@ -151,6 +166,11 @@ export interface UmtSiteExtension {
  /** 站点时间时区偏移,如 "+08:00";pubdate/freedate 按此转 UTC */
  timezone: string;
  rateLimit: { minIntervalMs: number; burst: number };
+  /**
+   * 站点级访问默认(本项目扩展):按站点实测给出的保守下限。运行时与 settings.sites 取更保守的一方
+   * (周期 / 缓存取较长,页数 / 并发取较小);站点行自己设了快照周期时以站点行为准。
+   */
+  defaults?: SiteAccessDefaults;
  capabilities: { list: boolean; search: boolean; detail: boolean; download: boolean; rss: boolean };
  /** 原始分类 ID → 站点分类名(列表只有 ID 时补名称;不做投影) */
  categoryNames?: Record<string, string>;
@@ -0,0 +1,25 @@
+import type { Kysely } from "kysely";
+import { dropTables, execDdl } from "../src/migration-helpers.ts";
+
+// 站点请求计数:每次从站点限流器拿到令牌(列表 / 搜索 / 详情 / 下载 / 登录 / 用户数据 / 原站视图)记一次,
+// 按小时聚合,api / worker 多进程共用。站点页展示"最近 24 小时请求次数";保留 7 天,写入时顺手清理。
+
+// biome-ignore lint/suspicious/noExplicitAny: 迁移不依赖应用表类型
+export async function up(db: Kysely<any>): Promise<void> {
+  await execDdl(
+    db,
+    `
+create table site_request_count (
+  site_id uuid not null references site(id) on delete cascade,
+  hour timestamptz not null,
+  count integer not null default 0 check (count >= 0),
+  primary key (site_id, hour)
+);
+`,
+  );
+}
+
+// biome-ignore lint/suspicious/noExplicitAny: 同上
+export async function down(db: Kysely<any>): Promise<void> {
+  await execDdl(db, dropTables(["site_request_count"]));
+}
@@ -121,6 +121,8 @@ const EXPECTED_TABLES = [
  "rate_limit_bucket",
  // C2-A PG 短期缓存(0042,UNLOGGED)
  "kv_cache",
+  // 站点请求计数(0046,按小时聚合)
+  "site_request_count",
 ].sort();

 /** 不走 entityTable() 的表:追加型/引擎自有表,没有 uuid id + created_at/updated_at 三件套 */
@@ -149,6 +151,8 @@ const NON_ENTITY_TABLES = [
  "rate_limit_bucket",
  // 缓存:(region, key) 主键
  "kv_cache",
+  // 站点请求计数:(site_id, hour) 主键
+  "site_request_count",
 ];

 describe.skipIf(skip)("schema migrations (real PG)", () => {
@@ -45,3 +45,38 @@ export const HHANCLUB_DETAIL_PAGE = `<!DOCTYPE html><html><head><title>HHCLUB ::
  <div><a href="download.php?id=90002">下载种子</a></div>
  <div id="kdescr"><a href="https://www.imdb.com/title/tt15239678/">IMDb</a> <a href="https://movie.douban.com/subject/35575567/">豆瓣</a> 简介……</div>
 </div></body></html>`;
+
+/**
+ * 2026-09-29 真实账号核对后补的最小脱敏片段(名称、ID、数值均已替换;不含 passkey 与用户信息)。
+ * 核对到的真实行为:列表标题链接文本末尾带新种标记「[新]」(详情页标题没有);短剧分类 412;
+ * 新种完成数为 0;无促销时没有 promotion-tag 元素;上传时间在 span[title] 里,文本是相对时间。
+ * 下载链接带 passkey(这里用占位值)。
+ */
+export const HHANCLUB_LIST_PAGE_2026 = `<!DOCTYPE html><html><head><title>HHCLUB :: 种子</title></head><body>
+<div id="user-info-panel"><a href="userdetails.php?id=7" class="VIP_Name">bob</a> <a href="logout.php">退出</a></div>
+<div class="torrent-table">
+  <div class="torrent-table-sub-info">
+    <div class="torrent-cat"><a href="?cat[]=412"><img src="pic/cattrans.gif" alt="短剧"/></a></div>
+    <div class="torrent-title"><a class="torrent-info-text-name" href="details.php?id=91001&amp;hit=1">示例短剧 1080p WEB-DL H265 AAC-HHWEB[新]</a></div>
+    <div class="torrent-info-text-small_name">示例短剧 | 全30集 | 1080p</div>
+    <div class="torrent-manage"><a href="download.php?id=91001&amp;passkey=0000000000000000">下载</a></div>
+    <div class="torrent-info-text-added"><span title="2026-09-29 14:07:03">5分钟</span></div>
+    <div class="torrent-info-text-size">185.38 MB</div>
+    <div class="torrent-info-text-seeders">1</div>
+    <div class="torrent-info-text-leechers">58</div>
+    <div class="torrent-info-text-finished">0</div>
+  </div>
+  <div class="torrent-table-sub-info">
+    <div class="torrent-cat"><a href="?cat[]=402"><img src="pic/cattrans.gif" alt="电视剧"/></a></div>
+    <div class="torrent-title"><a class="torrent-info-text-name" href="details.php?id=91002&amp;hit=1">Example Show S01 2026 2160p WEB-DL H.264 AAC-HHWEB</a></div>
+    <div class="torrent-info-text-small_name">示例剧集 | 第12集 | 4K | 类型: 奇幻 [内嵌中字]</div>
+    <div class="torrent-manage"><a href="download.php?id=91002&amp;passkey=0000000000000000">下载</a></div>
+    <div class="torrent-info-text-added"><span title="2026-09-29 12:03:28">2时8分钟</span></div>
+    <div class="torrent-info-text-size">1.46 GB</div>
+    <div class="torrent-info-text-seeders">131</div>
+    <div class="torrent-info-text-leechers">5</div>
+    <div class="torrent-info-text-finished">187</div>
+  </div>
+</div>
+<div class="pagination"><a href="?page=0">1-100</a><a href="?page=1">101-200</a></div>
+</body></html>`;