fix(tasks): align OutboxRelay with outbox_event status/next_attempt_at/idempotency_key
The relay only looked at published_at and ignored the ADR-002 §2.1 columns added in db 0002_foundation: rows stayed status='pending' forever after dispatch, failures were retried every tick without backoff, and the row-level idempotency_key was never used as the engine enqueue key. - claim status='pending' AND next_attempt_at<=now() - enqueue with idempotency_key (fallback outbox:<id>) - success: status=dispatched, published_at, attempts+1 - failure: attempts+1, exponential backoff (cap 5 min), last_error; dead at maxAttempts - test helper table now mirrors db 0001+0002; tests cover backoff and dead Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
1 parent
608e567edf
commit
31729f2bac
4 files changed
+56
-15
No files matched your search
@@ -31,12 +31,16 @@ export interface OutboxRelayOptions {
|
||||
routes: Record<string, OutboxRouteFn>;
|
||||
batchSize?: number;
|
||||
intervalMs?: number;
|
||||
/** 失败达到该次数后置 status='dead'(默认 10);其间按 2^attempts 秒退避,上限 5 分钟 */
|
||||
maxAttempts?: number;
|
||||
logSink?: LogSink;
|
||||
}
|
||||
|
||||
/**
|
||||
* Outbox 中继:业务事务写 outbox_event,中继在自己的事务里
|
||||
* `SELECT … FOR UPDATE SKIP LOCKED` 认领未投递行 → enqueue(幂等键 = outbox:<行 id>)→ 标记 published_at。
|
||||
* `SELECT … FOR UPDATE SKIP LOCKED` 认领 status='pending' 且 next_attempt_at 已到的行
|
||||
* → enqueue(幂等键 = 行的 idempotency_key,缺省 outbox:<行 id>)→ 标记 dispatched + published_at;
|
||||
* 失败则 attempts+1、退避 next_attempt_at、记 last_error,超过 maxAttempts 置 dead(ADR-002 §2.1)。
|
||||
* 多个中继进程并发时互不重复认领;enqueue 成功但标记前崩溃,重投时由引擎幂等键去重(24h),
|
||||
* 更长的去重由业务唯一约束兜底。
|
||||
*/
|
||||
@@ -61,9 +65,9 @@ export class OutboxRelay {
|
||||
let published = 0;
|
||||
let failed = 0;
|
||||
await this.opts.db.transaction().execute(async (trx) => {
|
||||
const rows = await sql<OutboxEvent>`
|
||||
SELECT id, type, payload, at FROM outbox_event
|
||||
WHERE published_at IS NULL AND type = ANY(${types}::text[])
|
||||
const rows = await sql<OutboxEvent & { idempotency_key: string | null; attempts: number }>`
|
||||
SELECT id, type, payload, at, idempotency_key, attempts FROM outbox_event
|
||||
WHERE status = 'pending' AND next_attempt_at <= now() AND type = ANY(${types}::text[])
|
||||
ORDER BY at, id
|
||||
LIMIT ${this.opts.batchSize ?? 50}
|
||||
FOR UPDATE SKIP LOCKED`.execute(trx);
|
||||
@@ -73,17 +77,25 @@ export class OutboxRelay {
|
||||
if (route) {
|
||||
const payloadTrace = (ev.payload as { traceId?: unknown } | null)?.traceId;
|
||||
await this.opts.engine.enqueue(route.task, route.input, {
|
||||
idempotencyKey: OutboxRelay.idempotencyKey(ev.id),
|
||||
idempotencyKey: ev.idempotency_key ?? OutboxRelay.idempotencyKey(ev.id),
|
||||
traceId: route.traceId ?? (typeof payloadTrace === "string" ? payloadTrace : randomUUID()),
|
||||
...(route.priority === undefined ? {} : { priority: route.priority }),
|
||||
});
|
||||
}
|
||||
await sql`UPDATE outbox_event SET published_at = now() WHERE id = ${ev.id}`.execute(trx);
|
||||
await sql`UPDATE outbox_event SET status = 'dispatched', published_at = now(), attempts = attempts + 1,
|
||||
last_error = NULL WHERE id = ${ev.id}`.execute(trx);
|
||||
published++;
|
||||
} catch (err) {
|
||||
failed++;
|
||||
// 留在未投递状态,下一轮重试
|
||||
this.log.error("outbox dispatch failed", { eventId: ev.id, type: ev.type, err });
|
||||
// 退避后重试;超过上限置 dead(需人工处理)
|
||||
const attempts = ev.attempts + 1;
|
||||
const dead = attempts >= (this.opts.maxAttempts ?? 10);
|
||||
const delayS = Math.min(2 ** attempts, 300);
|
||||
const lastError = JSON.stringify({ message: err instanceof Error ? err.message : String(err) });
|
||||
await sql`UPDATE outbox_event SET attempts = ${attempts}, status = ${dead ? "dead" : "pending"},
|
||||
next_attempt_at = now() + make_interval(secs => ${delayS}), last_error = ${lastError}::jsonb
|
||||
WHERE id = ${ev.id}`.execute(trx);
|
||||
this.log.error("outbox dispatch failed", { eventId: ev.id, type: ev.type, attempts, dead, err });
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
Reference in new issue
Block a user