feat(browser): vision automation loop with screenshots, coordinate actions and bounded challenge solving
- browser.screenshot returns the page as an image content block (≤1280px wide, JPEG quality configurable) for vision-capable upstreams; non-vision upstreams get a redacted text snapshot. Login pages / password fields are marked sensitive and withheld unless a browser.sensitive_screenshot grant exists and allowSensitive is passed. - browser.click_at / type_at / scroll / wait act in screenshot pixels; the server converts via viewport/DPR, requires automation control and an allowlisted host, and writes a thumbnail evidence file (DATA_DIR/tmp/evidence) referenced from agent_run_event. - browser.solve_challenge runs a bounded child agent_run (vision caps only, maxSteps ≤ 12, budget), judges success from page facts, stops immediately on 2FA / SMS / device confirmation, and otherwise moves the session to waiting with evidence and a takeover link. - PiAgentRuntime passes modelImage outputs as pi-ai image content when model.input has image; the ledger never stores image data; coordinate actions use per-invocation idempotency keys. - Tests: image codec, runtime image content, E14 e2e (login solved, 2FA stop, step limit, sensitive withheld). Real camofox 1.17.0 check on example.com recorded in the runbook. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
1 parent
0765264a47
commit
c437c6bff6
34 files changed
+2194
-35
No files matched your search
@@ -60,6 +60,21 @@ export interface BrowserProvider {
|
||||
sessionInfo(sessionId: string): BrowserSessionInfo | null;
|
||||
/** 等待进行中的操作收尾(接管前调用) */
|
||||
drain(sessionId: string): Promise<void>;
|
||||
/**
|
||||
* 页面度量(VIS-1 坐标换算与敏感判定):视口 CSS 尺寸、DPR、是否有密码框、页面文字样本(只在服务端判定用,不外传)。
|
||||
* 只读,不要求自动化控制权;上游不支持 evaluate 时返回 null。
|
||||
*/
|
||||
pageMetrics(ref: TabRef): Promise<PageMetrics | null>;
|
||||
}
|
||||
|
||||
export interface PageMetrics {
|
||||
viewportWidth: number;
|
||||
viewportHeight: number;
|
||||
devicePixelRatio: number;
|
||||
/** 页面上有 input[type=password] */
|
||||
passwordField: boolean;
|
||||
/** document.body.innerText 前 4000 字(判定二次验证等,不进模型、不落日志) */
|
||||
textSample: string;
|
||||
}
|
||||
|
||||
/** 动作发起方:自动化(Agent/任务)或持有接管租约的人 */
|
||||
|
||||
Reference in new issue
Block a user