fix(system): retired heartbeat rows no longer alarm on status page

- GET /system/health splits processes into current instances (hosts with
  a beat in the last 10 min, newest per host+kind, not stopped) and
  retiredProcesses[]; 45s stale and level only consider current ones
- heartbeat writer marks stopped_at on graceful exit (migration 0043),
  upsert clears it; pure api role now stops its writer on SIGTERM
- scheduled job system.heartbeat_cleanup (maintain, 1h) deletes rows
  without a beat for 24h and samples older than 7 days
- status page: per-role alarms ("io worker 45 秒没有心跳"), collapsible
  retired instances; glossary/runbook updated
- bump all members to 0.2.2, CHANGELOG [0.2.2]

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
rootandClaude Opus 5.5 committed 2026-09-28 23:31:21 +08:00
1 parent fa261eb6cd
commit b4c9b1ead2
32 files changed
+747 -91

No files matched your search

+6 -1
View File
@@ -381,8 +381,10 @@ export const SystemProcess = z.object({
lastBeatAt: Timestamp,
/** 距上次心跳秒数 */
beatAgeSec: z.number().nonnegative(),
/** 超过 45s 没心跳 */
/** 超过 45s 没心跳(只对当前实例告警) */
stale: z.boolean(),
/** 进程正常退出时写入;非 null 即已下线 */
stoppedAt: Timestamp.nullable(),
/** 最近一次采样的事件循环延迟 */
eventLoopLagMs: NonNegInt,
/** 最近一个心跳周期(15s)内的最大延迟 */
@@ -434,7 +436,10 @@ export const SystemHealth = z.object({
checkedAt: Timestamp,
/** 处理本请求的 api 进程 */
api: z.object({ version: z.string(), startedAt: Timestamp, uptimeSec: z.number().nonnegative() }),
/** 当前实例:最近 10 分钟内有心跳的 host 上、同 host 同 kind 最新启动且未退出的进程;stale 与告警只看这里 */
processes: z.array(SystemProcess),
/** 已下线实例:已正常退出、所在 host 10 分钟没有心跳(容器已重建)、或被同 host 新进程替换;不计入 level */
retiredProcesses: z.array(SystemProcess),
sse: z.object({ connections: NonNegInt }),
db: z.object({
level: HealthLevel,