LLM Result Cache

LLM Result Cache

If your agent scores, classifies, or analyzes inputs that recur across different tasks (the same competitor URL showing up across multiple client audits, the same document type, the same recurring question), you're paying for the same LLM call over and over even though nothing changed. This is a ~100-line, dependency-free disk cache that skips the call entirely on a hit.

Built and verified in production: added to a website-audit tool where the same handful of competitor sites kept recurring across many different clients' audits. First call: ~10 seconds (real LLM scoring). Repeat call on the same input: 0.0 seconds, zero API cost.

Why not just use functools.lru_cache?

lru_cache is in-memory only — it resets every time your process restarts, which is most of the time for a script-based agent. This persists to disk, survives restarts, and expires entries after a TTL you choose (so stale data doesn't get served forever — e.g. a competitor's website score from 3 months ago shouldn't still count as "fresh").

Setup

from llm_result_cache import ResultCache, cached

cache = ResultCache("my_cache.json", ttl_seconds=14*24*3600)  # 14 days

# Option A — decorator (simplest)
@cached(cache)
def score_url(url: str) -> dict:
    ...  # your expensive LLM call here, must return a dict

# Option B — manual control
def score_url(url: str) -> dict:
    hit = cache.get(url)
    if hit is not None:
        return hit
    result = {...}  # your expensive LLM call
    cache.set(url, result)
    return result

For a custom cache key (e.g. normalizing a URL, or hashing multiple inputs together), pass key_fn to the decorator:

@cached(cache, key_fn=lambda url, **kw: url.strip().lower().rstrip("/"))
def score_url(url: str) -> dict:
    ...

Design notes

• Bounded size: caps at max_entries (default 500), evicting oldest entries first — won't grow unbounded over a long-running process.
• A corrupted cache file is treated as empty, not fatal — this is a cache, not a source of truth, so losing it is safe. (If you need the opposite behavior — treating unreadable data as a hard stop rather than "start fresh" — see the ad-budget-governor skill, where that distinction matters because it's tracking real money, not just saving an API call.)
• Single-process only. This reads/writes the whole JSON file on every call — fine for occasional caching, not a substitute for Redis/Memcached under real concurrent load.