failure-forensics

Failure Forensics

Overview

When an agent task fails, the default response is to retry — hoping for a different outcome. Failure Forensics rejects that reflex. Instead, the agent performs structured root cause analysis before retrying, treating every failure as evidence to be collected, categorized, and learned from.

The workflow has four phases:

1. Triage — Categorize the failure using the taxonomy in references/failure-taxonomy.md.
2. Timeline Reconstruction — Parse tool-call logs and agent decision points to build a chronological failure timeline. The script scripts/failure_forensics.py automates this from JSON or JSONL log formats.
3. Causal Chain Analysis — Trace the chain of decisions, assumptions, and actions that led from the task kickoff to the failure point. Identify the root cause, not just the proximate symptom.
4. Post-Mortem Report — Generate a structured report from the template in references/post-mortem-template.md and persist it so future sessions can learn.

This skill turns a single failure into a permanent, reusable lesson.

When to Use

• An agent task failed and retrying without understanding why is risky.
• A failure recurs across attempts — you suspect a systemic cause, not bad luck.
• You need an artifact documenting what went wrong for a team review or audit.
• A complex multi-step task partially completed then broke — you need to understand which step is safe to resume from.
• You want to improve agent reliability by building a corpus of past failure patterns.

Don't use for:

• Trivial failures with obvious fixes (typo in a command, missing flag). Fix and move on.
• Live debugging of an actively failing process — use systematic-debugging for that. Run forensics after the process is dead or the task is abandoned.
• Human performance reviews. This skill analyzes agent + tool behavior, not people.

The Forensics Workflow

Phase 1: Triage — Categorize the Failure

Read the full taxonomy in references/failure-taxonomy.md. At a high level, every failure falls into one of six categories:

Record the category — it determines the questions you ask next.

Phase 2: Timeline Reconstruction

Collect the evidence:

1. Tool-call logs. If the agent session logged tool calls (JSON or JSONL with timestamps, tool name, args, result/error), feed them to the analyzer:

   python3 scripts/failure_forensics.py analyze \
     --log session.jsonl \
     --output timeline.md
   

   The script produces a chronological timeline with:

   • Each tool call, its timestamp, duration, and outcome (success/failure)
   • The first failure point flagged
   • Error messages and exit codes extracted
   • A summary of the failure category (inferred from error signatures)

2. Manual reconstruction. If no structured logs exist, reconstruct the timeline from memory of the session. List each decision and action in order. Be honest about uncertainty — mark gaps explicitly.

3. What to capture for each step:

   • Timestamp (or relative ordering)
   • The action or decision taken
   • The intent behind it (what the agent was trying to achieve)
   • The actual outcome
   • Any assumption the agent made

Phase 3: Causal Chain Analysis

This is the core of forensics. You're looking for the causal chain — the sequence where each link made the next failure more likely.

Ask these questions in order:

1. What was the immediate (proximate) cause of failure?

   • The error message, the crash point, the wrong output. This is the symptom.

2. What was the agent doing when it failed?

   • The specific tool call or action. What was the goal of that action?

3. Why did the agent take that action at that point?

   • Trace back one decision. What information did the agent have? What did it assume?

4. Was that assumption valid?

   • Check against logs, file contents, environment state. A failed assumption here is a link in the causal chain.

5. Continue backward until you reach either:

   • A root cause: a decision or state that, if different, would have prevented the entire failure cascade. Stop here.
   • The task kickoff: if no single root cause emerges, the failure is systemic (multiple contributing factors).

6. Look for contributing factors that didn't cause the failure but made it worse or harder to recover from:

   • Missing retry logic
   • Poor error messages that obscured the real problem
   • Timeouts set too aggressively or too loosely
   • Insufficient logging that made diagnosis harder

Anti-pattern: the "five whys" that stops at one. The first "why" almost always produces the symptom, not the cause. Keep going. The root cause is usually 3-5 links back.

Phase 4: Post-Mortem Report

Fill out the template in references/post-mortem-template.md. Key sections:

• Summary — one paragraph, plain language. A reader who wasn't there should understand it.
• Timeline — the reconstructed failure timeline from Phase 2.
• Root Cause — the terminal link of the causal chain from Phase 3, stated plainly.
• Contributing Factors — the rest of the chain.
• Action Items — concrete, assigned, verifiable. "Add retry logic" is bad; "Add exponential backoff retry (max 3 attempts) to the web_extract call in session.py:142" is good.
• Lessons Learned — generalizable insights. These are the durable output.

Save the report. Write it to a persistent location (e.g., a post-mortems/ directory, an issue tracker, or a knowledge base). A post-mortem that isn't saved didn't happen.

Using the Forensics Script

The script scripts/failure_forensics.py has three subcommands:

analyze — Build a failure timeline from logs

# JSONL log (one JSON object per line)
python3 scripts/failure_forensics.py analyze --log session.jsonl --format jsonl

# JSON array
python3 scripts/failure_forensics.py analyze --log session.json --format json

# Write report to file
python3 scripts/failure_forensics.py analyze --log session.jsonl --output report.md

categorize — Classify an error message

python3 scripts/failure_forensics.py categorize --error "ConnectionRefusedError: [Errno 111] Connection refused"
# Output: network

python3 scripts/failure_forensics.py categorize --error "PermissionError: [Errno 13] Permission denied"
# Output: permissions

report — Generate a post-mortem template pre-filled with timeline data

python3 scripts/failure_forensics.py report --log session.jsonl --title "Deploy failure 2024-01-15" --author "agent"

Expected Log Format

The analyzer accepts JSON/JSONL where each entry is a tool call record:

{
  "timestamp": "2024-01-15T10:23:45Z",
  "tool": "terminal",
  "args": {"command": "npm install"},
  "result": {"success": false, "error": "EACCES: permission denied, open '/usr/lib/node_modules'"},
  "duration_ms": 1200
}

Required fields: timestamp (ISO 8601), tool. The script tolerates missing optional fields (result, duration_ms, args).

Common Pitfalls

1. Confusing symptom with cause. "The deploy failed because the build failed" is a symptom. "The build failed because package-lock.json was regenerated with a different Node version than CI" is a cause. Keep digging past the symptom.

2. Stopping at human error. "I made a mistake" is never a root cause. Ask why the mistake was possible — missing validation? Misleading documentation? Fatigue from context switching? Fix the system, not the human.

3. Writing action items that aren't actionable. "Be more careful" is worthless. Every action item must specify what to change, where, and how to verify it works.

4. Skipping the timeline. Without a chronological timeline, causal chain analysis becomes guessing. Build the timeline first, even if it's rough.

5. Not saving the post-mortem. A post-mortem kept in chat history is lost when the session ends. Write it to a durable artifact (file, issue, doc).

6. Blame-oriented language. Post-mortems are blameless by design. Describe what happened, not who messed up. This is especially important when the "who" is an agent — focus on the decision and the information available at the time.

7. Retrying before forensics. The whole point is to analyze before retrying. If you retry first, you lose the original failure state and may introduce changes that mask the real cause.

Verification Checklist

• Failure categorized into one of the six taxonomy categories
• Timeline reconstructed (either via script or manually) with timestamps/ordering
• Causal chain traced backward to a root cause or identified as systemic
• Post-mortem report filled from the template
• Report saved to a durable location
• At least one concrete, verifiable action item identified
• Lessons learned phrased as generalizable insights, not task-specific notes