feat(browser): vision automation loop with screenshots, coordinate actions and bounded challenge solving

- browser.screenshot returns the page as an image content block (≤1280px wide, JPEG quality
  configurable) for vision-capable upstreams; non-vision upstreams get a redacted text snapshot.
  Login pages / password fields are marked sensitive and withheld unless a
  browser.sensitive_screenshot grant exists and allowSensitive is passed.
- browser.click_at / type_at / scroll / wait act in screenshot pixels; the server converts via
  viewport/DPR, requires automation control and an allowlisted host, and writes a thumbnail
  evidence file (DATA_DIR/tmp/evidence) referenced from agent_run_event.
- browser.solve_challenge runs a bounded child agent_run (vision caps only, maxSteps ≤ 12,
  budget), judges success from page facts, stops immediately on 2FA / SMS / device confirmation,
  and otherwise moves the session to waiting with evidence and a takeover link.
- PiAgentRuntime passes modelImage outputs as pi-ai image content when model.input has image;
  the ledger never stores image data; coordinate actions use per-invocation idempotency keys.
- Tests: image codec, runtime image content, E14 e2e (login solved, 2FA stop, step limit,
  sensitive withheld). Real camofox 1.17.0 check on example.com recorded in the runbook.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
rootandClaude Opus 5.5 committed 2026-09-29 01:27:48 +08:00
1 parent 0765264a47
commit c437c6bff6
34 files changed
+2194 -35

No files matched your search

+5 -2
View File
@@ -6,6 +6,7 @@ import { createAcquisitionOrganizeModule } from "../../../src/modules/acquisitio
import { createAgentModule } from "../../../src/modules/agent/index.ts";
import { authSettingsModule } from "../../../src/modules/auth-settings/index.ts";
import { createBrowserModule } from "../../../src/modules/browser/index.ts";
import type { BrowserModuleOptions } from "../../../src/modules/browser/service.ts";
import { createSchedulesModule } from "../../../src/modules/schedules/index.ts";
import { createSourcesIdentityModule } from "../../../src/modules/sources-identity/index.ts";
import type { ServerModule } from "../../../src/modules/types.ts";
@@ -29,6 +30,8 @@ export interface E2EModuleOptions {
syncIntervalMs?: number;
/** 网页搜索的 fetch / DNS 注入(E13);缺省不联网也能注册能力 */
webSearch?: WebSearchModuleOptions;
/** 浏览器模块注入(E14 视觉自动化:证据目录、动作后等待) */
browser?: BrowserModuleOptions;
}
export function buildModules(o: E2EModuleOptions): ServerModule[] {
@@ -69,8 +72,8 @@ export function buildModules(o: E2EModuleOptions): ServerModule[] {
...(o.fakeScripts ? { fakeScripts: o.fakeScripts } : {}),
goalPollSec: 1,
}),
// 浏览器能力(browser.*)随首批能力一起注册;e2e 不配置浏览器服务
createBrowserModule(),
// 浏览器能力(browser.*)随首批能力一起注册;浏览器服务由用例自己登记(E14 指向 mock camofox)
createBrowserModule(o.browser ?? {}),
// 网页搜索(web.search / web.fetch);e2e 注入 mock 搜索服务与网页
createWebSearchModule(o.webSearch ?? {}),
// 定时任务调度器(驱动上面模块登记的周期任务;须在末尾)