AI 说"我记得"，你敢信吗？测一下就知道。长期记忆评测台 memory-bench——本地一键评测智能体/大模型长期记忆：12 类题型（时序/实体/否定/反事实/跨会话整合），EM/F1 标准评分，零配置开箱即跑；可接真实 LLM 严评（SiliconFlow/DeepSeek），密钥 env 注入不落盘。结果可复现、可对比、可入发布证据。自带安全稳定性 10 维实测全 5.0。适合 Agent 开发者、AI 产品经理、记忆方案选型。"记忆好不好，测了才知道。"
Memory
TinkerClaw Memory Bench
Try itBe one of the first to benchmark your agent's memory — and help shape how AI remembers. Peer-review-grade evaluation (LLM-as-judge, nDCG/MAP/MRR with 95% CIs, ablations) against your live memory system. Runs entirely LOCALLY by default — no memory content leaves your machine. The optional OpenAI judge is opt-in, prints exactly what it would send, redacts secrets first, requires typed consent, and refuses to run unattended. Submitting results is a separate confirmed step that previews every public field, and identifies you only if you pass --contributor. Built for the TinkerClaw fork — github.com/globalcaos/tinkerclaw. See Permissions, Data Flow & Consent.
What it does
Be one of the first to benchmark your agent's memory — and help shape how AI remembers. Peer-review-grade evaluation (LLM-as-judge, nDCG/MAP/MRR with 95% CIs, ablations) against your live memory system. Runs entirely LOCALLY by default — no memory content leaves your machine. The optional OpenAI judge is opt-in, prints exactly what it would send, redacts secrets first, requires typed consent, and refuses to run unattended. Submitting results is a separate confirmed step that previews every public field, and identifies you only if you pass --contributor. Built for the TinkerClaw fork — github.com/globalcaos/tinkerclaw. See Permissions, Data Flow & Consent.
The skill document
Memory Bench
Collect, assess, and submit anonymized memory system statistics for the ENGRAM and CORTEX research papers.
Three-Step Pipeline
1. Assess Retrieval Quality
Run the standard test set (30 queries across 4 types × 3 difficulty levels) with LLM-as-judge:
# Full assessment with GPT-4o-mini judge + ablation (recommended)
python3 scripts/rate.py --queries 30 --judge openai --ablation
# Without OpenAI key: local embedding judge (weaker, marked in output)
python3 scripts/rate.py --queries 30 --judge local --ablation
# Custom test set
python3 scripts/rate.py --testset path/to/queries.json --judge openai
What it measures:
- RAR (Recall Accuracy Ratio), MRR (Mean Reciprocal Rank)
- nDCG@5, MAP@5, Precision@5, Hit Rate
- All metrics include 95% bootstrap confidence intervals
- Ablation: runs with AND without spreading activation to isolate its contribution
Judge methods:
openai— GPT-4o-mini rates each (query, result) pair 1-5. Independent from retrieval system. ~$0.01 per run.local— Embedding cosine similarity. Weaker, marked as such in output. Zero cost.
Standard test set (scripts/testset.json): 30 queries stratified across semantic/episodic/procedural/strategic types and easy/medium/hard difficulty. No lexical overlap with stored memories. All deployments run the same queries for cross-site comparability.
2. Collect Statistics
python3 scripts/collect.py --contributor GITHUB_USER --days 14 --output /tmp/memory-bench-report.json
Collected (anonymized): Memory counts/types/ages, strength/importance histograms, association graph size, hierarchy levels, consolidation history, retrieval metrics (RAR/MRR/nDCG/MAP with CIs), ablation results, judge method, algorithm version, embedding coverage. Instance ID is a random UUID (not reversible).
Never collected: Memory content, queries, file paths, usernames, hostnames.
3. Submit as PR
scripts/submit.sh /tmp/memory-bench-report.json GITHUB_USERNAME
Forks, branches, places report, updates INDEX.json, opens PR. Requires gh CLI.
Validation Protocol
For peer-review-ready data, contributors should:
- Run
rate.py --ablation --judge openai(minimum N=30 queries) - Collect at least 2 reports from the same instance, ≥7 days apart (longitudinal)
- Report the algorithm version (auto-captured from git)
Test Set Format
Custom test sets are JSON arrays:
[
{
"id": "T01",
"query": "...",
"category": "semantic|episodic|procedural|strategic",
"difficulty": "easy|medium|hard"
}
]
Agent Workflow
When asked to submit benchmarks: run rate.py --ablation --judge openai, then collect.py, review summary, then submit.sh. Share the PR link.
Related skills
Scaffold, sanitize, or share an OpenClaw multi-agent memory system with a reusable workspace, memory-lancedb-pro configuration, role prompts, task-board conv...
13 installs
Audit and maintain OpenClaw-style long-term memory. Use for MEMORY.md cleanup, daily-note digestion, duplicate detection, stale-memory review, and promoting...
17 installs
Standardize memory contracts across AI agents so multi-layer memory stays consistent, scoped, and exclusion-safe.
19 installs1 stars
Long-term memory for OpenClaw agents — SQLite hybrid recall (FTS5 + keyword + associative expansion + optional LLM embeddings), raw/curated anchors, session...
Better Every Run: capture explicit /ber corrections, review them, and promote only the lessons that deserve durable memory, skill rules, or evals.
25 installs1 stars