Stores durable facts in a categorized, plain-markdown vault on disk, alongside your agent's built-in memory.
by Iván559 installs18 stars
Documents
Peer Review
Try itMulti-model peer review layer using local LLMs via Ollama to catch errors in cloud model output. Fan-out critiques to 2-3 local models, aggregate flags, synthesize consensus. Use when: validating trade analyses, reviewing agent output quality, testing local model accuracy, checking any high-stakes Claude output before publishing or acting on it. Don't use when: simple fact-checking (just search the web), tasks that don't benefit from multi-model consensus, time-critical decisions where 60s latency is unacceptable, reviewing trivial or low-stakes content. Negative examples: - "Check if this date is correct" → No. Just web search it. - "Review my grocery list" → No. Not worth multi-model inference. - "I need this answer in 5 seconds" → No. Peer review adds 30-60s latency. Edge cases: - Short text (<50 words) → Models may not find meaningful issues. Consider skipping. - Highly technical domain → Local models may lack domain knowledge. Weight flags lower. - Creative writing → Factual revie
What it does
**Hypothesis:** Local LLMs can catch ≥30% of real errors in cloud output with <50% false positive rate.
The skill document
Peer Review — Local LLM Critique Layer
Hypothesis: Local LLMs can catch ≥30% of real errors in cloud output with <50% false positive rate.
Architecture
Cloud Model (Claude) produces analysis
│
▼
┌────────────────────────┐
│ Peer Review Fan-Out │
├────────────────────────┤
│ Drift (Mistral 7B) │──► Critique A
│ Pip (TinyLlama 1.1B) │──► Critique B
│ Lume (Llama 3.1 8B) │──► Critique C
└────────────────────────┘
│
▼
Aggregator (consensus logic)
│
▼
Final: original + flagged issues
Swarm Bot Roles
| Bot | Model | Role | Strengths |
|---|---|---|---|
| Drift 🌊 | Mistral 7B | Methodical analyst | Structured reasoning, catches logical gaps |
| Pip 🐣 | TinyLlama 1.1B | Fast checker | Quick sanity checks, low latency |
| Lume 💡 | Llama 3.1 8B | Deep thinker | Nuanced analysis, catches subtle issues |
Scripts
| Script | Purpose |
|---|---|
scripts/peer-review.sh | Send single input to all models, collect critiques |
scripts/peer-review-batch.sh | Run peer review across a corpus of samples |
scripts/seed-test-corpus.sh | Generate seeded error corpus for testing |
Usage
# Single file review
bash scripts/peer-review.sh [output_dir]
# Batch review
bash scripts/peer-review-batch.sh [results_dir]
# Generate test corpus
bash scripts/seed-test-corpus.sh [count] [output_dir]
Scripts live at workspace/scripts/ — not bundled in skill to avoid duplication.
Critique Prompt Template
You are a skeptical reviewer. Analyze the following text for errors.
For each issue found, output JSON:
{"category": "factual|logical|missing|overconfidence|hallucinated_source",
"quote": "...", "issue": "...", "confidence": 0-100}
If no issues found, output: {"issues": []}
TEXT:
---
{cloud_output}
---
Error Categories
| Category | Description | Example |
|---|---|---|
| factual | Wrong numbers, dates, names | "Bitcoin launched in 2010" |
| logical | Non-sequiturs, unsupported conclusions | "X is rising, therefore Y will fall" |
| missing | Important context omitted | Ignoring a major counterargument |
| overconfidence | Certainty without justification | "This will definitely happen" on 55% event |
| hallucinated_source | Citing nonexistent sources | "According to a 2024 Reuters report..." |
Discord Workflow
- Post analysis to #the-deep (or #swarm-lab)
- Drift, Pip, and Lume respond with independent critiques
- Celeste synthesizes: deduplicates flags, weights by model confidence
- If consensus (≥2 models agree) → flag is high-confidence
- Final output posted with recommendation:
publish|revise|flag_for_human
Success Criteria
| Outcome | TPR | FPR | Decision |
|---|---|---|---|
| Strong pass | ≥50% | <30% | Ship as default layer |
| Pass | ≥30% | <50% | Ship as opt-in layer |
| Marginal | 20–30% | 50–70% | Iterate on prompts, retest |
| Fail | <20% | >70% | Abandon approach |
Scoring Rules
- Flag = true positive if it identifies a real error (even if explanation is imperfect)
- Flag = false positive if flagged content is actually correct
- Duplicate flags across models count once for TPR but inform consensus metrics
Dependencies
- Ollama running locally with models pulled:
mistral:7b,tinyllama:1.1b,llama3.1:8b jqandcurlinstalled- Results stored in
experiments/peer-review-results/
Integration
When peer review passes validation:
- Package as Reef API endpoint:
POST /review - Agents call before publishing any analysis
- Configurable: model selection, consensus threshold, categories
- Log all reviews to
#reef-logswith TPR tracking
Related skills
Find why your productivity system keeps failing, then apply the smallest fix — capacity math, bottleneck routing, durable local notes.
by Iván855 installs69 stars
Site audit, content writing, and competitor analysis for organic search rankings.
by Iván574 installs31 stars
Write, debug, and tune Playwright specs with locator strategy, trace diagnosis, and CI-aware timeouts.
by ivangdavila1.3k installs130 stars
Run Git operations — commits, branches, merges, rebases, conflict resolution, and recovery — with safety rules enforced.
by Iván535 installs31 stars
More from staybased
Browse all skillsDesign specific, quick-win lead magnets that attract qualified prospects, capture emails, and smoothly convert them into paying customers.
by staybased107 installs3 stars
Create targeted, personalized multi-touch cold outreach messages using proven frameworks to engage prospects and boost response rates for B2B and local servi...
by staybased148 installs1 stars
Help salespeople transform objections into opportunities using proven frameworks to clarify value, address concerns, and close more deals effectively.
by staybased44 installs2 stars
Systematically find and evaluate Upwork automation jobs, bid strategically using 2,000+ n8n templates to maximize win rate and delivery speed.
by staybased57 installs3 stars
Guide discovery calls to qualify prospects, uncover real problems, assess fit, and set clear next steps before proposing solutions.
by staybased51 installs
10-dimension weighted scoring framework for prediction market trade evaluation. Enforces disciplined position sizing, circuit breakers, and mandatory counter-arguments. Use when: evaluating prediction market trades, scoring opportunities, deciding position sizes, comparing Polymarket/Kalshi opportunities, running pre-trade checklists. Don't use when: general crypto analysis, DeFi yield farming, non-prediction-market investments, stock/equity analysis, sports betting (different framework needed). Negative examples: - "Should I buy ETH?" → No. This is for prediction markets with binary/discrete outcomes. - "What's the best DeFi yield?" → No. Wrong domain entirely. - "Score this sports bet" → No. Sports betting has different dimensions (injuries, matchups). Edge cases: - Crypto prediction markets (e.g., "Will BTC hit $X?") → YES, use this if on Polymarket/Kalshi. - Multi-outcome markets → Score each outcome separately. - Markets with <$25 liquidity → Auto-fail on Liquidity dimension.
by staybased39 installs3 stars