Browser Search

Browser Search

What it does

Web search and browsing for AI agents. Three tools, from lightest to most powerful: SearXNG for search, Camofox for browsing, CloakBrowser for protected sites.

Core rules

When this skill is active, it operates as a Deep Research engine:

• No superficiality. Every query must be explored from multiple angles, sources, and cross-verified.
• No shortcuts. Accuracy and completeness first. Tokens and time are irrelevant.
• Exhaustive reports. Cover every aspect, include sources, don't omit details.
• Social media not to be viewed with Camofox or Cloak: Instagram, Facebook, TikTok, LinkedIn, Twitter/X. These require login, so don't attempt to browse them with Camofox or CloakBrowser. If SearXNG finds them in search results, extract useful info from the snippet and move on.
• Automatic escalation. If Camofox fails, switch to CloakBrowser.
• Only documented commands. Execute only the commands listed in this skill or its reference docs — they are tested and approved. No ad-hoc scripts: any deviation violates the skill.
• Read-only. Except where specified (screenshot saves PNG), all commands/scripts are read-only: they only make HTTP requests and don't write to the filesystem. Safe to run even in Plan mode.

Tools

1. SearXNG — Web search

Docker container on localhost:8080. Always the first choice for any search.

Commands:

Direct call to SearXNG REST API via curl. JSON output. ⚠️ Always use --data-urlencode for query parameters — never interpolate raw user input into URLs.

# Simple search (⚠️ use --data-urlencode for query safety)
exec curl -s -G "http://localhost:8080/search" --data-urlencode "q=" --data-urlencode "format=json"

# With language and category
exec curl -s -G "http://localhost:8080/search" --data-urlencode "q=" --data-urlencode "format=json" --data-urlencode "language=en" --data-urlencode "categories=news"

# With time range (day, week, month, year)
exec curl -s -G "http://localhost:8080/search" --data-urlencode "q=" --data-urlencode "format=json" --data-urlencode "time_range=month"

# Specific engines
exec curl -s -G "http://localhost:8080/search" --data-urlencode "q=" --data-urlencode "format=json" --data-urlencode "engines=google,wikipedia"

# Image search
exec curl -s -G "http://localhost:8080/search" --data-urlencode "q=" --data-urlencode "format=json" --data-urlencode "categories=images"

# Pagination
exec curl -s -G "http://localhost:8080/search" --data-urlencode "q=" --data-urlencode "format=json" --data-urlencode "pageno=2"

# Health check
exec curl -s -o /dev/null -w "%{http_code}" "http://localhost:8080/search?format=json&q=health"

For specific engines, check the engine field in results and pass names with &engines=name1,name2.

Language strategy:

Note: If SearXNG results are already exhaustive, Camofox and CloakBrowser are not needed. Stop here.

Troubleshooting — container down:

cd  && docker compose up -d

---

2. Camofox — Browser navigation (REST API)

Docker container on localhost:9377. Official interface: REST API over HTTP. Full OpenAPI spec: http://localhost:9377/docs (Swagger UI) or http://localhost:9377/openapi.json (raw).

Constants for all commands:

USER_ID="opencode-bot"
SESSION_KEY="default"
API_KEY="${CAMOFOX_API_KEY}"

The API key (Bearer) is required for POST /evaluate, POST /sessions/{userId}/cookies, DELETE /sessions/{userId}, GET /sessions/{userId}/traces, DELETE /sessions/{userId}/traces/{filename}, and POST /pressure/cleanup. POST /stop requires x-admin-key header (see below). Other endpoints work without auth.

General pattern:

1. Create tab → POST /tabs → get tabId (+ initial snapshot)
2. Operate on the tab with snapshot, evaluate, click, scroll...
3. Close tab → DELETE /tabs/{tabId}

TabId persistence: Always keep the tabId between commands. Camofox commands must never run in isolation: create a tab, use the tabId for all subsequent operations, then close it.

⚠️ Stale refs: After every interaction (click, scroll, navigate), refs (e1, e2...) are regenerated. Always take a fresh snapshot before using a ref.

404 error: If a command returns 404, the tab may have expired or been closed. Recreate it with POST /tabs and resume.

Essential commands

# 1. Create tab and navigate → returns { tabId, url } (snapshot separate)
exec curl -s -X POST "http://localhost:9377/tabs" \
  -H 'Content-Type: application/json' \
  -d "{\"userId\":\"$USER_ID\",\"sessionKey\":\"$SESSION_KEY\",\"url\":\"\"}"

# 2. Snapshot (accessibility tree with refs e1, e2, ...)
exec curl -s "http://localhost:9377/tabs//snapshot?userId=$USER_ID"
# Options: &includeScreenshot=true  |  &offset=N (pagination)

# 3. Evaluate (arbitrary JavaScript — for HTML tables, , , non-ARIA divs)
exec curl -s -X POST "http://localhost:9377/tabs//evaluate" \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $API_KEY" \
  -d "{\"userId\":\"$USER_ID\",\"expression\":\"\"}"

# 4. Click (by ref e1, e2... or CSS selector)
exec curl -s -X POST "http://localhost:9377/tabs//click" \
  -H 'Content-Type: application/json' \
  -d "{\"userId\":\"$USER_ID\",\"ref\":\"\"}"

# 5. Type (with optional clear and submit)
exec curl -s -X POST "http://localhost:9377/tabs//type" \
  -H 'Content-Type: application/json' \
  -d "{\"userId\":\"$USER_ID\",\"ref\":\"\",\"text\":\"\"}"

# 6. Scroll (direction: down/up/left/right, optional amount in px)
exec curl -s -X POST "http://localhost:9377/tabs//scroll" \
  -H 'Content-Type: application/json' \
  -d "{\"userId\":\"$USER_ID\",\"direction\":\"down\"}"

# 7. Navigate (URL or search macro)
exec curl -s -X POST "http://localhost:9377/tabs//navigate" \
  -H 'Content-Type: application/json' \
  -d "{\"userId\":\"$USER_ID\",\"url\":\"\"}"

# 8. Extract (structured data via JSON Schema + x-ref)
exec curl -s -X POST "http://localhost:9377/tabs//extract" \
  -H 'Content-Type: application/json' \
  -d "{\"userId\":\"$USER_ID\",\"schema\":{\"type\":\"object\",\"properties\":{\"field\":{\"x-ref\":\"e1\"}}}}"

# 9. Close tab
exec curl -s -X DELETE "http://localhost:9377/tabs/?userId=$USER_ID"

# 10. Destroy session (closes all tabs — requires API key)
exec curl -s -X DELETE "http://localhost:9377/sessions/$USER_ID" \
  -H "Authorization: Bearer $API_KEY"

# 11. Health check
exec curl -s "http://localhost:9377/health"

# 12. Screenshot (⚠️ returns raw PNG binary, not JSON — save to file)
exec curl -s -o /tmp/camofox_screenshot.png "http://localhost:9377/tabs//screenshot?userId=$USER_ID"

# 13. Stop browser (⚠️ requires CAMOFOX_ADMIN_KEY env var + x-admin-key header)
exec curl -s -X POST "http://localhost:9377/stop" \
  -H 'Content-Type: application/json' \
  -H "x-admin-key: $CAMOFOX_ADMIN_KEY" \
  -d '{}'

Other endpoints: back, forward, refresh, press, wait, viewport, links, images, screenshot, downloads, stats, start, stop, tabs/group/{id}, sessions/{userId}/cookies, sessions/{userId}/traces, sessions/{userId}/traces/{filename}. All documented with params and body in the OpenAPI spec at http://localhost:9377/docs. Consult it when needed.

⚠️ /screenshot returns raw PNG binary (not base64 JSON) — save to file with curl -s -o file.png then read it. Don't try to parse as JSON.

Reading method selection

Firefox's accessibility tree (snapshot) does NOT expose: HTML tables without ARIA roles, , , generic divs.

Decision flow:

1. Always start with snapshot. If it's rich in textual content and refs are usable, you're good.

2. Discard the snapshot if it's dominated by iframes, ad links, cookie banners, or the noise/content ratio is unfavorable. In that case go directly to evaluate.

3. evaluate is the main path for extracting structured data from any page. Strategy:

   • First contact: simple expressions (document.title) to verify the page loaded.
   • Explore the DOM with targeted selectors, one field at a time (document.querySelector("h1")?.textContent).
     • Don't nest multiple selectors in a single evaluate — can cause 500 errors. Separate into multiple calls.
     • document.querySelector(':contains(...)') causes 500 because :contains is not standard CSS. Arrow functions and :has work fine. Use standard CSS selectors.

4. On dynamic pages (SPA, lazy content, infinite scroll): after creating the tab, use POST /tabs/{tabId}/wait on a known selector (e.g. h1) or POST /tabs/{tabId}/scroll to trigger loading before evaluating.

5. Readability for articles (blogs, news, Wikipedia, docs): Use evaluate + Readability.js (Mozilla, scripts/camofox/Readability.js) to extract clean article text, removing nav, sidebar, ads, footer. ~70% token savings vs snapshot. For SEARCH/LIST pages, use snapshot instead. On lazy-loading pages: scroll before Readability.

   Command:

   # Read Readability.js, build ~94KB JSON payload and inject
   python3 -c "
   import json
   js = open('/scripts/camofox/Readability.js').read()
   expr = js + '; var a = new Readability(document.cloneNode(true)).parse(); JSON.stringify({title: a?.title, text: a?.textContent, excerpt: a?.excerpt, length: a?.length})'
   json.dump({'userId': '$USER_ID', 'expression': expr}, open('/tmp/rb_article.json', 'w'))
   "
   
   exec curl -s -X POST "http://localhost:9377/tabs//evaluate" \
     -H 'Content-Type: application/json' \
     -H "Authorization: Bearer $API_KEY" \
     -d @/tmp/rb_article.json
   # Output: { ok: true, result: "...{title, text, excerpt, length}..." }
   # If result is null → fallback to snapshot. Otherwise use text.
   

Search Macros

Use with POST /tabs/{tabId}/navigate passing macro and query:

Full workflow example

# 1. Create tab
exec curl -s -X POST "http://localhost:9377/tabs" \
  -H 'Content-Type: application/json' \
  -d '{"userId":"$USER_ID","sessionKey":"$SESSION_KEY","url":"https://example.com"}'
# → {"tabId":"abc123","url":"https://example.com/"}

# 2. Snapshot (understand structure and refs)
exec curl -s "http://localhost:9377/tabs/abc123/snapshot?userId=$USER_ID"

# 3. If snapshot is sparse → evaluate for raw HTML
exec curl -s -X POST "http://localhost:9377/tabs/abc123/evaluate" \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $API_KEY" \
  -d '{"userId":"$USER_ID","expression":"document.querySelector(\"main\")?.innerHTML || document.body.innerHTML"}'

# 4. Interact: scroll, click, type...
exec curl -s -X POST "http://localhost:9377/tabs/abc123/scroll" \
  -H 'Content-Type: application/json' \
  -d '{"userId":"$USER_ID","direction":"down","amount":500}'

# 5. After each interaction, take new snapshot (refs change!)
exec curl -s "http://localhost:9377/tabs/abc123/snapshot?userId=$USER_ID"

# 6. Click and new snapshot
exec curl -s -X POST "http://localhost:9377/tabs/abc123/click" \
  -H 'Content-Type: application/json' \
  -d '{"userId":"$USER_ID","ref":"e3"}'
exec curl -s "http://localhost:9377/tabs/abc123/snapshot?userId=$USER_ID"

# 7. Structured extract
exec curl -s -X POST "http://localhost:9377/tabs/abc123/extract" \
  -H 'Content-Type: application/json' \
  -d '{"userId":"$USER_ID","schema":{"type":"object","properties":{"title":{"x-ref":"e1"}}}}'

# 8. Close tab
exec curl -s -X DELETE "http://localhost:9377/tabs/abc123?userId=$USER_ID"

# 9. (Optional) Cleanup session (requires API key)
exec curl -s -X DELETE "http://localhost:9377/sessions/$USER_ID" \
  -H "Authorization: Bearer $API_KEY"

Troubleshooting — container down:

docker start camofox-browser
# If doesn't exist (⚠️ use --env-file for API keys, bind to 127.0.0.1):
# Create .env file first: echo "CAMOFOX_API_KEY=your-key" > .env
docker run -d --name camofox-browser --restart unless-stopped \
  -p 127.0.0.1:9377:9377 \
  --env-file .env \
  camofox-browser:latest

---

3. CloakBrowser — Protected sites

For sites with Cloudflare, Akamai, Kasada, DataDome, or when Camofox gets blocked. Uses launch() from the npm package cloakbrowser.

Script: /scripts/cloak/cloak-fetch.mjs

# Simple (markdown output)
exec node /scripts/cloak/cloak-fetch.mjs "https://example.com"

# Text only (no markdown header)
exec node /scripts/cloak/cloak-fetch.mjs "https://example.com" --format text

# Raw HTML
exec node /scripts/cloak/cloak-fetch.mjs "https://example.com" --format html

# With scroll for lazy loading (eBay, Amazon, reviews)
exec node /scripts/cloak/cloak-fetch.mjs "https://ebay.com/..." --scroll

# Automatic challenge detection (Cloudflare, Akamai, DataDome...)
exec node /scripts/cloak/cloak-fetch.mjs "https://protected-site.com"

# Proxy + geoip for sites that block datacenter IPs
exec node /scripts/cloak/cloak-fetch.mjs "https://..." --proxy "socks5://user:pass@proxy:1080" --geoip

# Deterministic fingerprint
exec node /scripts/cloak/cloak-fetch.mjs "https://..." --seed 12345 --platform windows

# Screenshot (⚠️ writes PNG file — breaks read-only rule)
# Pass URL as positional arg after the script path
exec node /scripts/cloak/cloak-script.mjs --script "/scripts/cloak/scripts/screenshot.mjs" "https://example.com"

cloak-script.mjs — For complex interactions

When click, login, multi-step, or custom data extraction is needed:

exec node /scripts/cloak/cloak-script.mjs \
  --script "/scripts/cloak/scripts/.mjs" \
  --proxy "socks5://..." --seed 12345

Full guide: /scripts/cloak/guida-fetch.md

---

Security rules

These rules are enforced by the tooling — do not attempt to bypass them.

• No internal URLs. CloakBrowser blocks localhost, 127.x, 10.x, 192.168.x, 169.254.x, and cloud metadata endpoints. DNS resolution is also checked to prevent DNS rebinding attacks.
• No path traversal. --script must point within the skill directory. Symlinks are resolved before the check. Use --unsafe only if you know the risks.
• Sandbox enabled by default. User scripts receive proxied Playwright objects with only whitelisted methods. Note: Node.js APIs remain available — for full isolation, a vm.Context would be required. Use --unsafe to bypass the Playwright sandbox.
• Rate limiting. 30 requests/minute by default. Use --no-rate-limit to disable.
• API keys. Never paste API keys on the command line. Use env vars ($CAMOFOX_API_KEY) or --env-file for Docker.
• URL encoding. Always use --data-urlencode with curl — never interpolate raw input into URLs.
• Docker binding. Always use 127.0.0.1: prefix for port mapping. Never expose to 0.0.0.0.
• No social media browsing. Instagram, Facebook, TikTok, LinkedIn, Twitter/X require login — don't attempt with Camofox or CloakBrowser.

Security-relevant CLI flags

---

Technical reference — Docker containers

Initial setup and diagnostics for SearXNG and Camofox. See /docker/setup.md when needed.