feat(browser): vision automation loop with screenshots, coordinate actions and bounded challenge solving

- browser.screenshot returns the page as an image content block (≤1280px wide, JPEG quality
  configurable) for vision-capable upstreams; non-vision upstreams get a redacted text snapshot.
  Login pages / password fields are marked sensitive and withheld unless a
  browser.sensitive_screenshot grant exists and allowSensitive is passed.
- browser.click_at / type_at / scroll / wait act in screenshot pixels; the server converts via
  viewport/DPR, requires automation control and an allowlisted host, and writes a thumbnail
  evidence file (DATA_DIR/tmp/evidence) referenced from agent_run_event.
- browser.solve_challenge runs a bounded child agent_run (vision caps only, maxSteps ≤ 12,
  budget), judges success from page facts, stops immediately on 2FA / SMS / device confirmation,
  and otherwise moves the session to waiting with evidence and a takeover link.
- PiAgentRuntime passes modelImage outputs as pi-ai image content when model.input has image;
  the ledger never stores image data; coordinate actions use per-invocation idempotency keys.
- Tests: image codec, runtime image content, E14 e2e (login solved, 2FA stop, step limit,
  sensitive withheld). Real camofox 1.17.0 check on example.com recorded in the runbook.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
rootandClaude Opus 5.5 committed 2026-09-29 01:27:48 +08:00
1 parent 0765264a47
commit c437c6bff6
34 files changed
+2194 -35

No files matched your search

+14 -1
View File
@@ -802,6 +802,16 @@ export function registerSubagentCapabilities(registry: CapabilityRegistry, ctx:
* 授权上限:子会话的每次授权调用都要落在全部祖先会话的能力白名单与类别上限之内(再与自身范围取交集)。
* 返回拒绝原因或 null。
*/
/** browser.solve_challenge 子流程用到的能力(与 challenge.ts 的 VISION_CAPABILITIES 一致) */
const VISION_STEP_CAPS = [
"browser.screenshot",
"browser.click_at",
"browser.type_at",
"browser.scroll",
"browser.wait",
"browser.snapshot",
];
export async function exceedsAncestors(
db: Kysely<DB>,
runId: string,
@@ -811,7 +821,10 @@ export async function exceedsAncestors(
const chain = await ancestors(db, runId);
for (const a of chain) {
const sc = (a.scope ?? {}) as Partial<RunScope>;
if (sc.capabilities?.length && !sc.capabilities.includes(capabilityId))
// 视觉子流程(browser.solve_challenge)的截图 / 坐标动作由白名单里的 solve_challenge 覆盖
const viaChallenge =
VISION_STEP_CAPS.includes(capabilityId) && !!sc.capabilities?.includes("browser.solve_challenge");
if (sc.capabilities?.length && !sc.capabilities.includes(capabilityId) && !viaChallenge)
return `超出父会话 ${a.id} 的能力白名单:${capabilityId}`;
if (sc.allowedKinds && !sc.allowedKinds.includes(kind)) return `父会话 ${a.id} 不允许 ${kind} 类能力`;
}