feat(browser): self-heal sessions after camofox restarts (BR-3)

- Any session operation that hits 404/410 (or finds no open tab) reconciles
  tabs; if none survive it re-injects the account cookies (BR-2), opens a new
  tab under the same userId/sessionKey, returns to the recorded URL (site home
  otherwise), marks old tabs lost and retries the operation once. Failure
  returns 503 browser_reconnect_failed with the reason, with a 10s backoff.
- Restart detection: umt plugin GET /umt/instance (fallback /health fields);
  a changed instance id marks every open session needsReconnect.
- POST /browser/sessions/{id}/reconnect (idempotent); session view exposes
  needsReconnect and reconnect {status, trigger, at, reason, humanNotice}.
- Controller and lease are kept; reconnecting during takeover asks the human
  to double-check the reloaded page.
- Web: screenshot and desktop modes show "reconnecting", auto-reconnect once,
  pause polling, offer a reconnect button with the reason on failure; desktop
  mode rebuilds the Selkies connection after a reconnect.
- Mock camofox restart(), tests, smoke step and runbook section 10.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
rootandClaude Opus 5.5 committed 2026-09-29 17:29:11 +08:00
1 parent da24380870
commit 838aea3ebd
19 files changed
+977 -45

No files matched your search

+9
View File
@@ -2400,6 +2400,15 @@ export const Api = {
body: browser.BrowserDownload,
response: browser.BrowserDownloadResult,
}),
reconnect: ep({
method: "post",
path: "/browser/sessions/{id}/reconnect",
scope: "admin",
summary:
"浏览器服务重启后重新连接(幂等):上游上下文还在就什么都不做;否则注入账号 Cookie、新开标签页回到原地址,控制权不变;失败 503 browser_reconnect_failed",
params: browser.BrowserSessionParams,
response: browser.BrowserSessionView,
}),
acquire: ep({
method: "post",
path: "/browser/sessions/{id}/control/acquire",
+19
View File
@@ -327,6 +327,21 @@ export const BrowserCredentialState = z.object({
});
export type BrowserCredentialState = z.infer<typeof BrowserCredentialState>;
/**
* 浏览器服务重启 / 回收后的自动重连(BR-3,存 recovery_state.reconnect):
* trigger = tab_gone(操作时发现标签页不在)/ provider_restarted(实例标识变了)/ manual(原站视图调用 reconnect);
* humanNotice = 重连时会话正由人工接管(页面已重新加载,请核对)。
*/
export const BrowserReconnectState = z.object({
status: z.enum(["ok", "failed"]),
trigger: z.enum(["tab_gone", "provider_restarted", "manual"]),
at: Timestamp,
/** 失败原因(不含凭据) */
reason: z.string().nullable(),
humanNotice: z.boolean(),
});
export type BrowserReconnectState = z.infer<typeof BrowserReconnectState>;
export const BrowserSessionView = BrowserSession.extend({
tabs: z.array(BrowserTab),
currentTabId: z.string().nullable(),
@@ -337,6 +352,10 @@ export const BrowserSessionView = BrowserSession.extend({
waitingReason: z.string().nullable(),
/** 打开会话时的凭据注入与回写状态(BR-2);旧会话没有 */
credential: BrowserCredentialState.nullable().optional(),
/** 浏览器服务重启过、上游上下文已不在:下一次操作(或 POST .../reconnect)自动重建(BR-3) */
needsReconnect: z.boolean().optional(),
/** 最近一次自动重连的结果 */
reconnect: BrowserReconnectState.nullable().optional(),
});
export type BrowserSessionView = z.infer<typeof BrowserSessionView>;
+3
View File
@@ -35,6 +35,7 @@ export const ApiErrorCode = z.enum([
"rate_limited",
"automation_disabled",
"upstream_unavailable",
"browser_reconnect_failed",
"internal",
]);
export type ApiErrorCode = z.infer<typeof ApiErrorCode>;
@@ -51,6 +52,8 @@ export const API_ERROR_STATUS = {
rate_limited: 429,
automation_disabled: 423,
upstream_unavailable: 502,
/** 浏览器服务重启后自动重连失败(BR-3):浏览器服务暂不可用,稍后重试 */
browser_reconnect_failed: 503,
internal: 500,
} as const satisfies Record<ApiErrorCode, number>;