httrack

Adaptive web scraping in Python that bypasses anti-bot systems and scales from single requests to concurrent crawls.

Query Twitter/X profiles, tweets, follower events, and KOL data through the 6551 REST API.

Post videos, photos, text, and documents to 10 social platforms through a single REST API call.

Stores durable facts in a categorized, plain-markdown vault on disk, alongside your agent's built-in memory.

Trade crypto, manage a multi-chain wallet, and query an AI analyst from one CLI.

Write, debug, and tune Playwright specs with locator strategy, trace diagnosis, and CI-aware timeouts.

Detect and repair partially wiped agent workspaces with integrity checks, signed manifests, guarded restore recipes, bounded local recovery state, and explicit off-box sync. Use when files, scripts, trees, models, or build outputs disappear or lose integrity between turns.

Seven offline mechanisms against slow/stale/zombie/sycophantic agent turns: prompt compaction, request fencing, zombie detection, CAPTCHA triage, anti-sycophancy spine, delivery register, invention quarry. Use when chat feels laggy, reconnects surface old answers, long chats degrade, or the agent caves under contradiction. JSON contracts; state per-agent under ~/.arena_turn; no network, no sudo.

Quota-aware LLM router that squeezes maximum usable AI out of free-tier API keys across Gemini, Mistral, OpenRouter, Kilo and Cerebras plus any OpenAI-compatible endpoint (including local Ollama/llama.cpp/vLLM). Probes every model on every key, measures real quality and real published rate limits, then routes each request to the cheapest model that can do the job — spending abundant capacity first and reserving scarce daily quota for when it is actually needed. Persists cooldowns to disk so a 429 discovered in one process is respected by the next. Use when an agent must make many LLM calls on free keys without hitting rate limits, when "all models failed", or when deciding which of several provider keys to use for a task.

Opt-in, model-neutral guidance for evidence-aware, dignified AI communication, with a compact response contract and offline deterministic text audit. It never injects prompts, edits host configuration, calls networks, reads secrets, or treats heuristics as truth.

Iran Chemical Database — live, dated, auditable, BEST-EFFORT index of chemical offerings in configured public Iranian supplier catalogues (websites + public Telegram channels). HTTrack/WooCommerce-REST/Telegram mirroring → local-only parsing → RDKit/PubChem/CAS-validated PostgreSQL with FastAPI + Streamlit. Fail-closed Iranian-suppliers-only country gate; coverage measured and published, never claimed complete. Installation = software + queued crawl, not a populated dataset. Ships a 1399-molecule CID-unique confirmed-organic seed baseline (v2.22, 2026-08-27: v2.19 primary + live Telegram/WooCommerce/sitemap crawl + 5-model fleet normalization, every new identity PubChem-confirmed). For academic procurement research.

Model-agnostic, agent-agnostic fidelity-first pipeline converting operator-authorized Persian/English RTL lecture PDFs into offline HTML study guides — recall-first dual OCR (PyMuPDF + Tesseract fas+eng PSM ensemble), rendered-page evidence, multi-model correction, session-grounded enrichment (tables/flashcards/quizzes/mnemonics/summaries/scenarios), measured fidelity, QA gates, verified ZIP. v1.5.0 runs on ANY model family through 8 API dialects (OpenAI, Responses, Gemini, Anthropic, Cohere, Ollama, HuggingFace, offline mock) or with no model at all, auto-discovers providers from the host agent's environment, self-heals provider quirks and model retirements, and exposes one deterministic CLI/MCP entrypoint plus cross-model consensus so different agents reproduce the same intended result.