Free Web Search Js

free-web-search-js

Pure search — no rewriting, no filtering, noise handled by the AI. Uses real browser automation (Playwright) for Bing CN anti-bot, with HTTP fallback for international queries. Zero API keys required.

Quick start

# Search (auto-fetches top 3 page contents)
node scripts/search.js "silver price"
node scripts/search.js "how to deploy docker" --max=5
node scripts/search.js "xxx" --region=cn
node scripts/search.js "xxx" --fetch=5         # fetch top 5
node scripts/search.js "xxx" --no-fetch        # search only

# Fetch specific URLs
node scripts/fetch.js "https://example.com/page1" "https://example.com/page2"

Architecture

Domestic (CN IP):
  Playwright headless → Bing CN → home page for cookie → search box submit
  → 0 results? fallback to DDG HTML
  → auto-fetch top N page contents

International:
  Pure HTTP → DDG HTML parse
  → fetch fails? fallback to Playwright request
  → DDG empty? fallback to Bing
  → auto-fetch top N page contents

Install

All dependencies are installed upfront by the setup script. The skill never runs npm install or pip install at runtime. This avoids audit flags and supply-chain risks.

Prerequisites

One-command setup

# Linux / macOS
bash scripts/setup.sh

# Windows
powershell -File scripts/setup.ps1

The scripts auto-detect region and use npmmirror.com mirrors inside China for faster downloads.

Manual install

cd skills/free-web-search-js
npm install
npx playwright install chromium    # ~150 MB

# Verify
node scripts/check-env.js

If you already have Chrome/Chromium installed:

export CHROMIUM_EXECUTABLE_PATH="/path/to/chrome"
node scripts/check-env.js

check-env.js will confirm whether the provided executable is usable.

Environment variables

Search engines

Strategy

IP detection

On each search, three probes run in parallel — first responder wins:

Override with --region=cn or --region=intl if your proxy causes misdetection.

Fetch mode

After search, auto-fetches the top N results (default 3).

Tier 1 enhancements:

• JSON API response detection with structured content extraction
• JSON-LD (``) articleBody/description
• __NEXT_DATA__ embedded data extraction
• Meta tag fallback (og:description / description)
• Auto GBK charset detection & conversion

Deduplication

Smart dedup: domain + path stem (ignores www/m subdomains, tracking params, trailing slash, .html suffix). Bing redirect URLs (bing.com/ck/) auto-decoded.

Browser daemon (optional, ~70% faster)

Avoid 1.5s+ browser launch overhead by reusing a persistent Chromium instance:

node scripts/browser-daemon.js            # Start (run with & or as background job)
node scripts/browser-daemon.js --status   # Show status
node scripts/browser-daemon.js --stop     # Stop

The daemon auto-exits after 10 minutes of inactivity (configurable via BROWSER_DAEMON_IDLE_MS). Activity is tracked via a .browser-heartbeat file that search.js and fetch.js touch on each connection. The daemon checks the heartbeat every 60 seconds.

Verification & troubleshooting

cd skills/free-web-search-js
node scripts/check-env.js
node scripts/search.js "OpenClaw" --max 2 --region cn --fetch 0
node scripts/search.js "OpenClaw" --max 2 --region intl --fetch 0

Common issues:

• node_modules not found → run npm install
• Executable doesn't exist ... → run bash scripts/setup.sh or set CHROMIUM_EXECUTABLE_PATH
• browserType.launch ... closed → headless+no-sandbox is default; run node scripts/check-env.js to diagnose
• 0 search results → try --region=cn / --region=intl, or --fetch 0 to test search alone

Privacy & network

• Searches connect to Bing CN / DDG HTML / Sogou directly from the skill's runtime — no intermediate proxy
• The daemon heartbeat file (.browser-heartbeat) and endpoint file (.browser-endpoint) are written to the skill root directory
• No analytics, no telemetry, no third-party logging beyond the search engine's own server logs
• Headed warm-up fetches: Cookie warm-up for Bing uses a headed browser context to obtain session cookies from the search engine home page; no cookies are persisted to disk between sessions

Known limitations

• First domestic search slower: Chromium cold start takes 3–6s; daemon avoids this
• Bing CN instant answers: Weather/calculator cards don't use li.b_algo, return 0 results
• Sogou HTTP unstable: No cookie, easily blocked by anti-crawl
• JS-required pages: HTTP tier returns empty for SPA pages — headed tier catches them
• Region-specific sites: Some domestic sites time out from international IPs
• Proxy interference: IP detection may misclassify behind proxies — use --region to override
• DDG blocked in China: DDG HTML is not reachable from mainland China; domestic strategy avoids it