feat(browser): vision automation loop with screenshots, coordinate actions and bounded challenge solving
- browser.screenshot returns the page as an image content block (≤1280px wide, JPEG quality configurable) for vision-capable upstreams; non-vision upstreams get a redacted text snapshot. Login pages / password fields are marked sensitive and withheld unless a browser.sensitive_screenshot grant exists and allowSensitive is passed. - browser.click_at / type_at / scroll / wait act in screenshot pixels; the server converts via viewport/DPR, requires automation control and an allowlisted host, and writes a thumbnail evidence file (DATA_DIR/tmp/evidence) referenced from agent_run_event. - browser.solve_challenge runs a bounded child agent_run (vision caps only, maxSteps ≤ 12, budget), judges success from page facts, stops immediately on 2FA / SMS / device confirmation, and otherwise moves the session to waiting with evidence and a takeover link. - PiAgentRuntime passes modelImage outputs as pi-ai image content when model.input has image; the ledger never stores image data; coordinate actions use per-invocation idempotency keys. - Tests: image codec, runtime image content, E14 e2e (login solved, 2FA stop, step limit, sensitive withheld). Real camofox 1.17.0 check on example.com recorded in the runbook. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
1 parent
0765264a47
commit
c437c6bff6
34 files changed
+2194
-35
No files matched your search
@@ -76,6 +76,12 @@ export const CAPABILITY_IDS = [
|
||||
"browser.navigate",
|
||||
"browser.interact",
|
||||
"browser.download",
|
||||
"browser.screenshot",
|
||||
"browser.click_at",
|
||||
"browser.type_at",
|
||||
"browser.scroll",
|
||||
"browser.wait",
|
||||
"browser.solve_challenge",
|
||||
"agent.plan",
|
||||
"agent.propose_options",
|
||||
"web.search",
|
||||
@@ -402,6 +408,75 @@ export const CAPABILITY_CATALOG: Record<KnownCapabilityId, CapabilityDescriptor>
|
||||
false,
|
||||
false,
|
||||
),
|
||||
// VIS-1 视觉自动化闭环:截图(图片内容块)→ 视觉模型 → 坐标操作 → 再截图。坐标以最近一次截图的像素为准,
|
||||
// 服务端按视口 / DPR 换算;动作只在自动化控制权下、站点域名内执行,每步写证据缩略图;
|
||||
// 二次验证 / 短信 / 本人设备确认不自动处理,立即转人工
|
||||
"browser.screenshot": d(
|
||||
"browser.screenshot",
|
||||
"截取原站页面",
|
||||
"截取站点浏览器会话当前页面(缩放到不超过 1280 像素宽,作为图片交给支持视觉的模型);后续 click_at / type_at 的坐标以这张截图的像素为准。登录页或含密码框的截图标为敏感,默认不交给模型",
|
||||
"read",
|
||||
"BrowserScreenshot",
|
||||
"read",
|
||||
"low",
|
||||
true,
|
||||
false,
|
||||
),
|
||||
"browser.click_at": d(
|
||||
"browser.click_at",
|
||||
"按坐标点击",
|
||||
"在最近一次 browser.screenshot 的像素坐标处点击,返回点击后的新截图;只在自动化控制权下、站点域名内执行",
|
||||
"execute",
|
||||
"BrowserClickAt",
|
||||
"admin",
|
||||
"medium",
|
||||
false,
|
||||
false,
|
||||
),
|
||||
"browser.type_at": d(
|
||||
"browser.type_at",
|
||||
"按坐标输入",
|
||||
"先点击截图坐标处的输入框(不给坐标则输入到当前焦点),再输入文本,可选回车提交;返回输入后的新截图。账号密码不要用它输入,用 browser.interact 的 credential",
|
||||
"execute",
|
||||
"BrowserTypeAt",
|
||||
"admin",
|
||||
"medium",
|
||||
false,
|
||||
false,
|
||||
),
|
||||
"browser.scroll": d(
|
||||
"browser.scroll",
|
||||
"滚动页面",
|
||||
"按像素滚动当前页面(dy 正数向下、dx 正数向右),返回滚动后的新截图",
|
||||
"execute",
|
||||
"BrowserScroll",
|
||||
"admin",
|
||||
"low",
|
||||
false,
|
||||
false,
|
||||
),
|
||||
"browser.wait": d(
|
||||
"browser.wait",
|
||||
"等待页面",
|
||||
"等待页面加载或动画完成(最多 5 秒),返回等待后的新截图",
|
||||
"read",
|
||||
"BrowserWait",
|
||||
"read",
|
||||
"low",
|
||||
true,
|
||||
false,
|
||||
),
|
||||
"browser.solve_challenge": d(
|
||||
"browser.solve_challenge",
|
||||
"处理页面挑战",
|
||||
"起一个有界的视觉子流程(只能截图、按坐标点击 / 输入、滚动、等待、读页面),识别并处理页面上的挑战,直到调用方给出的成功条件满足;超出步数或预算、遇到二次验证 / 短信 / 本人设备确认时停止,会话转等待处理并附证据截图,由人在原站视图接管",
|
||||
"execute",
|
||||
"BrowserSolveChallenge",
|
||||
"admin",
|
||||
"medium",
|
||||
false,
|
||||
false,
|
||||
),
|
||||
// AG-2 自动化优先的会话交互:步骤清单与方案选择。只在 Agent 会话里可用(HTTP / MCP 调用直接报错)
|
||||
"agent.plan": d(
|
||||
"agent.plan",
|
||||
|
||||
Reference in new issue
Block a user