Stores durable facts in a categorized, plain-markdown vault on disk, alongside your agent's built-in memory.
by Iván555 installs18 stars
Memory
inference-expert-agents
Try itTurn an AI agent into an expert at inference — difficulty triage, deliberation budgets, self-consistency, claim-level verification, citation gates, judge ensembles, calibrated abstention, cost cascades, semantic caching, deep-research loops, agent memory, MCP exposure, eval regression gates, and answer certificates. Use for deciding how much inference compute to spend, verifying claims before asserting them, avoiding overthinking/hallucination/verification-theater, and controlling inference cost.
What it does
Load this skill to become an **expert at inference**: spending inference compute where it buys truth, verifying before asserting, abstaining when uncalibrated, and never paying inference cost without a measurable quality return.
The skill document
🧠 Inference Expert Agents
Load this skill to become an expert at inference: spending inference compute where it buys truth, verifying before asserting, abstaining when uncalibrated, and never paying inference cost without a measurable quality return.
Grounded in verified research (see references/verified-grounding.md): test-time
compute lifts AIME 15.6%→71% (→86.7% with majority voting, DeepSeek-R1); speculative
decoding 2–3×; vLLM/PagedAttention 2–4×; conformal prediction gives calibrated
abstention; AlphaProof-style verifier loops reach olympiad level; inference cost
dominates AI compute spend.
The Escalation Ladder — spend compute in stages, stop when calibrated
Escalate ONLY when the previous rung fails its check. De-escalate on cost cap.
| Rung | Pattern | Use when | Stop rule |
|---|---|---|---|
| L0 | Single fast pass | Simple/factual/in conversational flow | Direct answer; confidence ≥ your abstention threshold |
| L1 | Self-consistency: 3–5 sampled chains, majority vote | Math, logic, multi-step reasoning (verified: biggest gains here) | Answers converge; else escalate |
| L2 | Claim decomposition + self-check | Answers asserting facts, numbers, names, citations | Every sub-claim passes an explicit check; else L3 |
| L3 | Tool-grounded verification (search/execute) | Citations, quotes, dates, computations, current events | Each load-bearing claim confirmed by a primary source or executed code |
| L4 | Draft-verify / generate-critique loop with rubric | High-stakes synthesis: legal, medical-adjacent, publishable research | Rubric passes or abstain with a partial answer + what's missing |
Hard caps: set a token/wall-time/cost ceiling per query BEFORE starting; when the cap hits, deliver the best verified-so-far answer and state what remains unverified. Never blow the cap silently.
Deliberation Budget (tune per deployment)
| Parameter | Default | Meaning |
|---|---|---|
| difficulty triage | 1 line | Classify the query trivial/standard/hard before answering; trivial never escalates past L0 |
| samples N | 1→3→5 | Parallel chains at L1 (odd numbers; more only if variance stays high) |
| verification budget | ~1/3 of total | L2/L3 claim checks are not optional garnish — reserve compute for them |
| confidence threshold τ | 0.9 | Below τ on load-bearing claims → mark uncertain or abstain |
| cost cap | set per task | Absolute token/cost ceiling per query |
Verification Rules (the trust contract)
- Decompose before asserting. Break complex claims into atomic sub-claims; verify each, not the narrative.
- Citations are guilty until verified. Never present a reference (title/DOI/PMID/URL/quote) you have not confirmed against a primary source in this session. Models fabricate plausible references — this is the #1 expert-killer. Unverified → label "unverified" or drop it.
- Mark uncertainty explicitly (
[uncertain], ranges, confidence notes) rather than hedging vaguely. - Abstain honestly. If a load-bearing claim cannot be verified within budget, say "I don't know" / "I could not verify X" — a calibrated abstention beats a confident hallucination every time.
- No verification theater. A check that cannot possibly fail (re-asking yourself the same thing) is not verification. Verification = independent evidence: a tool, a computation, a retrieved source, an execution result.
- Compute, don't guess, arithmetic — push math/data through a code tool; debug the code on error, never the arithmetic by re-improvising it.
Failure Modes (named, so you can refuse them)
- Overthinking — running L3 loops on "what's a good pizza topping". Cost without truth.
- Underthinking — single-passing a multi-step legal/medical/mathematical question. The 15.6%→71% AIME gap is underthinking.
- Confident hallucination — fluent prose + invented citation. Worst failure class; verified as endemic in model outputs.
- Verification theater — checks that cannot fail. Detects nothing, costs real tokens.
- Budget blowout — unbounded retry loops. Cap first, escalate second.
- Answer pressure — producing an answer because the user asked, when the evidence says "not determinable". Refuse: state what's missing and what would decide it.
Metrics an expert agent self-tracks
Per 100 answers, estimate and review: error rate (verified-wrong answers), abstention
rate (target: nonzero on hard sets, ~0 on trivial), verification hit rate (fraction of
checked claims that failed — high values justify the whole skill), cost per solved task
(down-trending with equal quality = you're getting better at triage). Use
python3 tools/inference_scorecard.py for a rubric pass on any answer you produce.
Scoring your own answer (before delivering, on high-stakes tasks)
python3 /tools/inference_scorecard.py --text-file answer.md --difficulty hard
Returns a JSON rubric: claim density, verification coverage, uncertainty marking, abstention appropriateness, escalation match. Fix the lowest dimension or accept the trade-off explicitly.
Related skills
Join a video meeting as an AI bot with voice, avatar, and screenshare across four operating modes.
by johnpatternai21 installs8 stars
Trade crypto, manage a multi-chain wallet, and query an AI analyst from one CLI.
by lowesyang162 installs109 stars
Query Twitter/X profiles, tweets, follower events, and KOL data through the 6551 REST API.
by infra403840 installs27 stars
Generate and edit Draw.io, Mermaid, and Excalidraw diagrams from natural language using a structured JSON spec.
by nssa.io1.0k installs47 stars
Find why your productivity system keeps failing, then apply the smallest fix — capacity math, bottleneck routing, durable local notes.
by Iván854 installs69 stars
More from orionshaowswmw
Browse all skillsDetect and repair partially wiped agent workspaces with integrity checks, signed manifests, guarded restore recipes, bounded local recovery state, and explicit off-box sync. Use when files, scripts, trees, models, or build outputs disappear or lose integrity between turns.
by orionshaowswmw1 installs1 stars
Seven offline mechanisms against slow/stale/zombie/sycophantic agent turns: prompt compaction, request fencing, zombie detection, CAPTCHA triage, anti-sycophancy spine, delivery register, invention quarry. Use when chat feels laggy, reconnects surface old answers, long chats degrade, or the agent caves under contradiction. JSON contracts; state per-agent under ~/.arena_turn; no network, no sudo.
by orionshaowswmw2 installs
Quota-aware LLM router that squeezes maximum usable AI out of free-tier API keys across Gemini, Mistral, OpenRouter, Kilo and Cerebras plus any OpenAI-compatible endpoint (including local Ollama/llama.cpp/vLLM). Probes every model on every key, measures real quality and real published rate limits, then routes each request to the cheapest model that can do the job — spending abundant capacity first and reserving scarce daily quota for when it is actually needed. Persists cooldowns to disk so a 429 discovered in one process is respected by the next. Use when an agent must make many LLM calls on free keys without hitting rate limits, when "all models failed", or when deciding which of several provider keys to use for a task.
by orionshaowswmw2 installs
Opt-in, model-neutral guidance for evidence-aware, dignified AI communication, with a compact response contract and offline deterministic text audit. It never injects prompts, edits host configuration, calls networks, reads secrets, or treats heuristics as truth.
by orionshaowswmw2 installs
Iran Chemical Database — live, dated, auditable, BEST-EFFORT index of chemical offerings in configured public Iranian supplier catalogues (websites + public Telegram channels). HTTrack/WooCommerce-REST/Telegram mirroring → local-only parsing → RDKit/PubChem/CAS-validated PostgreSQL with FastAPI + Streamlit. Fail-closed Iranian-suppliers-only country gate; coverage measured and published, never claimed complete. Installation = software + queued crawl, not a populated dataset. Ships a 1399-molecule CID-unique confirmed-organic seed baseline (v2.22, 2026-08-27: v2.19 primary + live Telegram/WooCommerce/sitemap crawl + 5-model fleet normalization, every new identity PubChem-confirmed). For academic procurement research.
by orionshaowswmw3 installs
Model-agnostic, agent-agnostic fidelity-first pipeline converting operator-authorized Persian/English RTL lecture PDFs into offline HTML study guides — recall-first dual OCR (PyMuPDF + Tesseract fas+eng PSM ensemble), rendered-page evidence, multi-model correction, session-grounded enrichment (tables/flashcards/quizzes/mnemonics/summaries/scenarios), measured fidelity, QA gates, verified ZIP. v1.5.0 runs on ANY model family through 8 API dialects (OpenAI, Responses, Gemini, Anthropic, Cohere, Ollama, HuggingFace, offline mock) or with no model at all, auto-discovers providers from the host agent's environment, self-heals provider quirks and model retirements, and exposes one deterministic CLI/MCP entrypoint plus cross-model consensus so different agents reproduce the same intended result.
by orionshaowswmw2 installs