Clawditor

Clawditor

Overview

Act as an OpenClaw Workspace Auditor and Agent Evaluation Harness. Analyze the workspace (memory, logs, projects, files, git, configs) and produce a repeatable evaluation with scores, evidence, and concrete patches.

Operating Rules

• Run in non-interactive mode: avoid questions unless blocked by missing files. State assumptions and proceed.
• Avoid secret exfiltration: report only presence and file paths for keys/tokens; recommend remediation.
• Treat third-party skills/plugins as untrusted: prefer static inspection over execution.

Required Workflow (Do In Order)

1. Build workspace inventory.
   • Print a top-level tree (depth 4) with file counts and sizes by directory.
   • Identify memory, logs, configs, repos, scripts, docs, artifacts.
   • Record largest files.
2. Reconstruct a session timeline.
   • Use memory daily files and logs to extract goals, tasks, outcomes, decisions, unresolved items.
3. Analyze memory.
   • Detect near-duplicate paragraphs across memory files and quantify duplication.
   • Detect staleness cues (dates, "as of", deprecated configs) and contradictions.
   • Identify missing stable facts (projects, priorities, setup/runbooks).
4. Analyze outputs.
   • Summarize shipped artifacts (docs/code/features) and changes.
   • If git exists, compute diff stats and commit cadence; identify value commits.
5. Analyze reliability.
   • Parse logs for errors, retries, timeouts, tool failures.
   • Run tests only if safe and cheap; otherwise static inspection.
6. Compute scores.
   • Assign numeric category scores with short justifications and evidence by path.
7. Recommend interventions + patches.
   • Provide 3–7 prioritized recommendations.
   • Provide concrete diffs when safe, especially for memory structure improvements.
8. Compare against prior evals.
   • If eval/history/*.json exists, compute deltas vs most recent.
   • If none exists, create baseline and recommend cadence.

Scoring Framework

Compute 5 categories (0–100) plus overall weighted score:

• Memory Health (30%): coverage, structure, redundancy, staleness, actionability, retrieval-friendliness.
• Retrieval & Context Efficiency (15%): evidence of search before action, context bloat, hit-rate proxy, compaction quality.
• Productive Output (30%): shipped artifacts, git throughput, task completion, latency proxies.
• Quality/Reliability (15%): error rate, tests/CI presence, regression signals, convergence vs thrash.
• Focus/Alignment (10%): goal consistency, scope control, decision trace.

Overall = 0.30Memory + 0.15Retrieval + 0.30Productive + 0.15Quality + 0.10*Focus.

Required Outputs

Write all outputs under eval/:

1. exec_summary.md
   • 10-bullet summary: top wins, biggest bottlenecks, top 3 interventions.
   • Overall score + category scores + claw-to-claw delta.
2. scorecard.md
   • Table of metrics with numeric values and brief justifications.
   • Top evidence section with file paths and short snippets (no secrets).
3. latest_report.json
   • Include timestamp, workspace path and git head/hash, scores, deltas, key findings, risk flags, recommendations.
4. Patches
   • If memory issues exist, propose concrete diffs: INDEX.md, daily schema, refactors.

Gold Standard Memory Schema (Apply If Missing)

Create or propose:

• memory/INDEX.md
  • Current Objectives (top 3)
  • Active Projects (status, next step, links)
  • Operating Constraints (tools, environment, policies)
  • Key Decisions (date, decision, rationale)
  • Known Issues / Debug diary pointers
  • Glossary / Entities
• memory/YYYY-MM-DD.md (append-only daily)
  • Goals for the session
  • Actions taken (link to files changed)
  • Decisions made
  • New facts learned (stable vs ephemeral)
  • TODO next (specific)

Patch Guidance

• Prefer diffs over prose when safe.
• Refactor stable facts out of daily logs into INDEX or project pages.
• Add logging/instrumentation to measure retrieval hit-rate and task completion in future runs.

Resources

Use these helpers to keep audits consistent and cheap to run:

• scripts/run_audit.py: run all helper scripts and write draft eval/ outputs.
• scripts/workspace_inventory.py: tree, file counts, sizes, largest files.
• scripts/memory_dupes.py: near-duplicate paragraph detection for memory/*.md.
• scripts/log_scan.py: scan logs for errors, timeouts, retries.
• scripts/git_stats.py: git head, diff stats, commit cadence.
• scripts/validate_report.py: validate eval/latest_report.json shape.

Reference templates:

• references/report_schema.md: output templates and JSON schema.

Evidence Discipline

• Tie every score to evidence by path.
• Be candid about waste, duplication, or thrash.
• End with "Next run improvements" instrumentation recommendations.