Agent Tests

Agent Tests 🧪

Stop guessing if your agent still works. Start testing it.

The Problem

Agent behavior changes with every model update, config change, and skill addition. Without tests, you have no way to know if something broke until a user catches it.

Agent Tests gives you a lightweight testing framework for agent behavior.

Quick Start

Add a test case

node skills/agent-tests/agent-tests.js --test --add greet "Say hello" "Hello" "contains:Hello"

List all tests

node skills/agent-tests/agent-tests.js --test --list

Run a specific test

node skills/agent-tests/agent-tests.js --test --run greet

Run all tests

node skills/agent-tests/agent-tests.js --test --run

Run performance benchmark

node skills/agent-tests/agent-tests.js --benchmark greet 10

Runs the test 10 times, reports average/min/max duration and pass rate.

Check regression report

node skills/agent-tests/agent-tests.js --regression

Shows pass rates, recent failures, and performance trends for all tests.

Remove a test

node skills/agent-tests/agent-tests.js --test --remove greet

Status overview

node skills/agent-tests/agent-tests.js --status

Assertion Types

Features

Test Management

• Add/remove tests with name, prompt, and expected output
• Track run count and pass rate per test
• Persistent storage in memory/agent-tests/tests.json

Assertion Engine

• Simple assertion syntax (no test framework needed)
• Supports contains, not_contains, regex, and length checks
• Extensible for custom assertions

Regression Tracking

• Groups results by test name
• Shows pass rate, total runs, and average duration
• Lists recent failures with specific assertion failures
• Flags tests below 80% pass rate with ⚠️

Performance Benchmarking

• Runs tests multiple iterations
• Reports average, min, max duration
• Tracks pass rate across iterations
• Saves benchmark history (last 50 runs per test)

Result History

• Stores last 1000 results per test
• Timestamps for every run
• Actual vs expected output captured

Configuration

Data files stored in: memory/agent-tests/

• tests.json — Test definitions
• results.json — Test results history (last 1000)
• benchmarks.json — Benchmark history (last 50 per test)

Override data directory:

--dir /path/to/data

Agent Protocol

When testing agent behavior:

1. Add tests — --test --add for critical behaviors
2. Run before changes — --test --run before model/config updates
3. Benchmark periodically — --benchmark to track performance drift
4. Check regressions — --regression during heartbeats
5. Remove stale tests — --test --remove for outdated test cases

Limitations

• Test execution simulates output (actual agent output integration needed)
• Assertion engine is simple (regex, contains, length)
• No parallel test execution
• Results limited to 1000 entries

Comparison

Agent Tests gives you behavior testing + regression tracking + benchmarking with zero dependencies.

Design Principles

1. Zero setup — Works immediately, no config needed
2. No dependencies — Pure Node.js, no npm packages
3. Simple assertions — Easy to write, hard to misinterpret
4. Persistent — Tests and results survive restarts
5. Transparent — Every run is logged and reportable