- browser.screenshot returns the page as an image content block (≤1280px wide, JPEG quality
configurable) for vision-capable upstreams; non-vision upstreams get a redacted text snapshot.
Login pages / password fields are marked sensitive and withheld unless a
browser.sensitive_screenshot grant exists and allowSensitive is passed.
- browser.click_at / type_at / scroll / wait act in screenshot pixels; the server converts via
viewport/DPR, requires automation control and an allowlisted host, and writes a thumbnail
evidence file (DATA_DIR/tmp/evidence) referenced from agent_run_event.
- browser.solve_challenge runs a bounded child agent_run (vision caps only, maxSteps ≤ 12,
budget), judges success from page facts, stops immediately on 2FA / SMS / device confirmation,
and otherwise moves the session to waiting with evidence and a takeover link.
- PiAgentRuntime passes modelImage outputs as pi-ai image content when model.input has image;
the ledger never stores image data; coordinate actions use per-invocation idempotency keys.
- Tests: image codec, runtime image content, E14 e2e (login solved, 2FA stop, step limit,
sensitive withheld). Real camofox 1.17.0 check on example.com recorded in the runbook.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
- PiUpstreamClient: one Models per upstream holding only its provider; credential
store adapter maps to that upstream's secret, so several accounts of one provider
coexist; host env is only read for authKind none (ambient cloud creds)
- provider catalog from builtinProviders() plus static oauth modes, env fields and
subscription terms notes; openai-compatible custom endpoints
- fake upstream provider with mock OAuth (browser paste / device approval)
- PiAgentRuntime accepts an upstream; thinking level clamped to the model
Co-Authored-By: Claude Opus 5.5 <[email protected]>
