Check coding-model API quality, capability fit, and drift with LT-lite and B3IT-lite. Use when Codex needs to verify whether an OpenAI/OpenAI-compatible/Anth...
15 次安装
记忆
Model Watch
试用Benchmark AI API models over time and detect quality degradation. 7 standardized tests (reasoning, coding, writing, instruction-following, hallucination). Al...
它能做什么
Benchmark AI API models over time and detect quality degradation. 7 standardized tests (reasoning, coding, writing, instruction-following, hallucination). Alerts when scores drop >10% vs historical average. Because models silently get dumber.
技能文档
model-watch
Problem → Solution
The problem: AI companies silently degrade their models. "Opus 4.7 was hallucinating a lot today... shocking to see such degradation" — r/ClaudeAI (49↑). "Anthropic admits to have made hosted models more stupid" — r/LocalLLaMA (281↑). You're paying the same price for a dumber model and you don't even know it.
The solution: Standardized benchmark suite you run yourself. 7 tests across 5 categories. Scores stored locally. Alerts when recent scores drop >10% vs your historical average. Hard data, not vibes.
Quick Start
pip install git+https://github.com/minirr890112-byte/model-watch.git
model-watch demo # View benchmark questions
model-watch submit '{"reasoning_1":"...","coding_1":"...",...}' # Submit outputs
model-watch history # View score history
model-watch alert # Check for degradation
Benchmarks (7 tests)
| Category | Tests | What it measures |
|---|---|---|
| Reasoning | 2 | Logic, multi-step deduction |
| Coding | 2 | Code generation, debugging |
| Writing | 1 | Quality, coherence |
| Instruction-following | 1 | Precision, constraint adherence |
| Hallucination detection | 2 | Factual accuracy |
How It Works
- Run the 7 benchmark questions through your AI model of choice
- Feed the responses into
model-watch submit - Scores are stored locally in
~/.hermes/model-watch-history.json - Track trends with
model-watch history model-watch alertflags when recent scores drop >10% vs historical average
⭐ Star this repo if you've noticed your favorite model getting dumber: github.com/minirr890112-byte/model-watch
相关技能
search and analyze llm benchmark results within a fixed benchmark universe, then produce evidence-based model strength and weakness reports or domain-leader...
24 次安装
Use this skill when the user wants to monitor LLM behavior over time and get alerted when outputs change unexpectedly. Triggers on requests like "set up LLM regression monitoring", "alert me when my prompts start behaving differently", "watch my LLM for regressions", "run behavioral tests on my AI outputs on a schedule", or "detect when my model starts drifting". Handles first-time setup, baseline capture, scheduled monitoring, and alert configuration via WhatsApp, Slack, Discord, or email.
17 次安装
静态评测 Prompt 质量与兼容性 · Prompt benchmark
1 次安装
Compare AI API pricing across 18 models from 6 providers — find the cheapest model for your use case (coding, chat, writing, reasoning) and track your API sp...
3 次安装
Double-blind comparison of AI model responses — query models in parallel, judge anonymized outputs, rank on merit. Trigger with "mdls" or "modelshow".
35 次安装1 星标