feat(browser): vision automation loop with screenshots, coordinate actions and bounded challenge solving
- browser.screenshot returns the page as an image content block (≤1280px wide, JPEG quality configurable) for vision-capable upstreams; non-vision upstreams get a redacted text snapshot. Login pages / password fields are marked sensitive and withheld unless a browser.sensitive_screenshot grant exists and allowSensitive is passed. - browser.click_at / type_at / scroll / wait act in screenshot pixels; the server converts via viewport/DPR, requires automation control and an allowlisted host, and writes a thumbnail evidence file (DATA_DIR/tmp/evidence) referenced from agent_run_event. - browser.solve_challenge runs a bounded child agent_run (vision caps only, maxSteps ≤ 12, budget), judges success from page facts, stops immediately on 2FA / SMS / device confirmation, and otherwise moves the session to waiting with evidence and a takeover link. - PiAgentRuntime passes modelImage outputs as pi-ai image content when model.input has image; the ledger never stores image data; coordinate actions use per-invocation idempotency keys. - Tests: image codec, runtime image content, E14 e2e (login solved, 2FA stop, step limit, sensitive withheld). Real camofox 1.17.0 check on example.com recorded in the runbook. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
1 parent
0765264a47
commit
c437c6bff6
34 files changed
+2194
-35
No files matched your search
@@ -15,8 +15,8 @@ const TiB = 1024 ** 4;
|
||||
const SITE_IDS = ["site.list", "site.health_check", "site.refresh_login", "site.user_data"] as const;
|
||||
|
||||
describe("站点能力规格", () => {
|
||||
it("进入首批能力目录(17 + 4 站点 + 4 浏览器 + 2 会话交互 + 2 网页搜索 = 29);类别 / 风险 / 重放按约定", () => {
|
||||
expect(CAPABILITY_IDS).toHaveLength(29);
|
||||
it("进入首批能力目录(17 + 4 站点 + 10 浏览器 + 2 会话交互 + 2 网页搜索 = 35);类别 / 风险 / 重放按约定", () => {
|
||||
expect(CAPABILITY_IDS).toHaveLength(35);
|
||||
for (const id of SITE_IDS) expect(CAPABILITY_IDS).toContain(id);
|
||||
expect(capabilitySpec("site.list")).toMatchObject({ kind: "read", riskLevel: "low", replay: "safe" });
|
||||
expect(capabilitySpec("site.user_data")).toMatchObject({ kind: "read", riskLevel: "low", replay: "safe" });
|
||||
|
||||
Reference in new issue
Block a user