feat(browser): vision automation loop with screenshots, coordinate actions and bounded challenge solving
- browser.screenshot returns the page as an image content block (≤1280px wide, JPEG quality configurable) for vision-capable upstreams; non-vision upstreams get a redacted text snapshot. Login pages / password fields are marked sensitive and withheld unless a browser.sensitive_screenshot grant exists and allowSensitive is passed. - browser.click_at / type_at / scroll / wait act in screenshot pixels; the server converts via viewport/DPR, requires automation control and an allowlisted host, and writes a thumbnail evidence file (DATA_DIR/tmp/evidence) referenced from agent_run_event. - browser.solve_challenge runs a bounded child agent_run (vision caps only, maxSteps ≤ 12, budget), judges success from page facts, stops immediately on 2FA / SMS / device confirmation, and otherwise moves the session to waiting with evidence and a takeover link. - PiAgentRuntime passes modelImage outputs as pi-ai image content when model.input has image; the ledger never stores image data; coordinate actions use per-invocation idempotency keys. - Tests: image codec, runtime image content, E14 e2e (login solved, 2FA stop, step limit, sensitive withheld). Real camofox 1.17.0 check on example.com recorded in the runbook. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
1 parent
0765264a47
commit
c437c6bff6
34 files changed
+2194
-35
No files matched your search
@@ -4,6 +4,14 @@ import { registerBrowserCapabilities } from "./capabilities.ts";
|
||||
import { registerBrowserRoutes } from "./routes.ts";
|
||||
import { type BrowserModuleOptions, BrowserService } from "./service.ts";
|
||||
|
||||
/** 进程内:ServerContext → 本模块为它建的 BrowserService(agent 模块的视觉子流程要与能力共用同一实例) */
|
||||
const byCtx = new WeakMap<ServerContext, BrowserService>();
|
||||
|
||||
/** 取浏览器模块为这个上下文建的服务;浏览器模块没装配时返回 null */
|
||||
export function browserServiceFor(ctx: ServerContext): BrowserService | null {
|
||||
return byCtx.get(ctx) ?? null;
|
||||
}
|
||||
|
||||
/**
|
||||
* C1-E 浏览器:BrowserProvider(camofox)之上的站点浏览器会话、原站视图与人工接管、Agent 浏览器能力。
|
||||
* 会话/标签/租约以数据库(0023 四张表)为准;站点限流与 sources-identity 共享;下载产物只进后端暂存核验,
|
||||
@@ -22,6 +30,7 @@ export function createBrowserModule(opts: BrowserModuleOptions = {}): ServerModu
|
||||
if (!s) {
|
||||
s = new BrowserService(ctx, options);
|
||||
services.set(ctx, s);
|
||||
byCtx.set(ctx, s);
|
||||
}
|
||||
return s;
|
||||
};
|
||||
@@ -33,6 +42,7 @@ export function createBrowserModule(opts: BrowserModuleOptions = {}): ServerModu
|
||||
},
|
||||
registerTasks() {},
|
||||
registerCapabilities(registry, ctx) {
|
||||
service(ctx);
|
||||
registerBrowserCapabilities(registry, () => service(ctx));
|
||||
},
|
||||
};
|
||||
|
||||
Reference in new issue
Block a user