Redis Store

User preferences and memory live in ~/Clawic/data/redis-store/ (see setup.md on first use, memory-template.md for the file format). If you have data at an old location (~/redis-store/ or ~/clawic/redis-store/), move it to ~/Clawic/data/redis-store/.

When To Use

• Choosing how to model something in Redis: hash vs JSON string vs separate keys, sorted set vs stream, set vs bitmap vs HyperLogLog
• Writing Redis commands, pipelines, Lua scripts or Functions, and getting atomicity right
• Building the standard recipes: cache, session store, job queue, distributed lock, rate limiter, leaderboard, dedup/idempotency key, counter
• Operating a server: maxmemory and eviction, persistence and backups, ACLs, connection limits, replication, Sentinel, Cluster
• A production incident: OOM on writes, latency spike, unresponsive server, keyspace evicted or wiped, failover that lost writes, replica that never syncs
• Migrating: standalone to cluster, version upgrade, a managed provider (ElastiCache, MemoryDB, Redis Cloud, Upstash, Memorystore, Azure) or the Valkey fork
• Not for store-agnostic cache hierarchy design (caching), rate-limit algorithm selection (rate-limiting), or relational modeling (pg, sql)

Quick Reference

Depth on demand, by phase:

• Model — data-types.md which structure wins and what it costs · keys-ttl.md key design, expiry semantics, SCAN, keyspace notifications · patterns.md sessions, leaderboards, counters, dedup, autocomplete, geo · modules.md JSON, Search, vectors, Bloom, TimeSeries
• Build — caching.md cache-aside, invalidation, stampede, client-side caching · queues.md Streams, consumer groups, retries, delayed jobs · pubsub.md fan-out, sharded Pub/Sub, notifications · locks.md distributed locks and fencing · scripting.md atomicity, MULTI, Lua, Functions · testing.md test isolation and CI
• Operate — memory-eviction.md maxmemory, eviction, big keys, fragmentation · persistence.md RDB, AOF, backups, restore drills · connections.md pooling, timeouts, buffers, TLS · security.md ACLs, exposure, command policy · cli.md the redis-cli forensics toolkit · managed-redis.md ElastiCache, MemoryDB, Redis Cloud, Upstash, Memorystore, Azure
• Scale and recover — cluster.md slots, hash tags, resharding · high-availability.md replication, Sentinel, failover, durability · performance.md latency triage, pipelining, command cost · incidents.md symptom-to-cause playbooks · migrations.md upgrades, standalone-to-cluster, provider and Valkey moves

Core Rules

1. Every key gets a TTL or a named owner that deletes it. Redis never reclaims what nobody expires. Budget: memory ≈ keys × (60-100 bytes of key overhead + value size), so 10M TTL-less session keys of 200 bytes cost roughly 3 GB whether or not anyone reads them. Check: redis-cli INFO keyspace reports keys=N,expires=M per db — a large N − M is the leak.
2. Set maxmemory and a policy before production. Without maxmemory Redis grows until the kernel swaps or the OOM killer takes it. Sizing: with fork-based persistence enabled, maxmemory ≤ 55-60% of host RAM (a copy-on-write fork can approach a second copy of the dataset in the worst case); a pure cache with persistence off can go to ~80%. Default policy is noeviction, which turns "full" into failed writes.
3. One command, one atomic unit — MULTI is not a rollback. A runtime error inside EXEC (wrong type, OOM) does not undo the commands that already ran. Prefer a single atomic command (INCR, SET NX, LMOVE), then Lua, then WATCH-and-retry. Check: if your logic reads a value and writes a value derived from it, it needs one of the three.
4. Never run an O(N) command against an unbounded collection. Command execution is serial: one KEYS or HGETALL over 1M elements stalls every client for the duration, and at roughly 10^5-10^6 elements that is tens to hundreds of milliseconds. Replacements: SCAN/HSCAN/SSCAN/ZSCAN with COUNT, UNLINK instead of DEL for big keys, LRANGE with real bounds.
5. Round trips, not Redis, are your latency. Unpipelined cost ≈ n × RTT: 1000 sequential GETs over a 0.5 ms network is 500 ms of wall time while the server spends under 10 ms. Batch with a pipeline (500 commands per flush is a sane default), MGET/HMGET, or a Lua script that does the loop server-side.
6. A lock needs a unique token, a TTL longer than the work, and a compare-and-delete release. TTL ≥ 3× the p99 duration of the critical section, renewed at 1/3 of the TTL by the holder. Releasing with a plain DEL deletes whatever lock exists — including the one the next worker just acquired after your TTL expired. The instance must be noeviction: that mandatory TTL is what makes a lock the first victim under volatile-* (→ locks.md).
7. Replication is asynchronous: an acknowledged write can be lost in failover. The master replies before replicas see the write. WAIT 1 100 blocks until one replica acknowledges (still not consensus — a partitioned master can acknowledge and lose), min-replicas-to-write 1 + min-replicas-max-lag 10 refuses writes when nobody is listening. Choose the data-loss budget explicitly (→ high-availability.md).
8. Persistence is a data-loss budget, not a checkbox. RDB alone loses everything since the last snapshot (default save points fire at 3600s/1 change, 300s/100, 60s/10000). AOF with appendfsync everysec loses about one second and is the default balance; always costs an fsync per write. Both enabled = RDB for fast restore, AOF for the tail — Redis restores from AOF when it is on (→ persistence.md).

Choosing The Data Structure

Pick the smallest structure whose access pattern matches the query you will actually run. Encoding thresholds and per-type memory math: data-types.md.

Latency Triage

Run in order; skipping to step 4 tunes the wrong thing.

1. Separate client-side from server-side: redis-cli --latency -h (round-trip as seen from a client) vs redis-cli --intrinsic-latency 100 on the server box (what the kernel and CPU alone cost). A high intrinsic number means the host, not Redis.
2. SLOWLOG GET 10 — the log records commands over slowlog-log-slower-than, default 10000 microseconds, keeping the last slowlog-max-len 128. Its times exclude network, so an entry here is genuinely a slow command.
3. LATENCY LATEST and LATENCY DOCTOR after setting latency-monitor-threshold 100 (default 0 = disabled). Events named fork, aof-fsync-always, expire-cycle name their own cause.
4. INFO commandstats — sort by usec_per_call, then multiply by calls: a 0.2 ms command called 5k/s consumes a full second of CPU per second of wall time — the entire single thread — and beats any 40 ms outlier as a target.
5. Fork suspicion: INFO stats latest_fork_usec. Fork cost tracks page-table size, on the order of 10-20 ms per GB of RSS on ordinary Linux VMs, and transparent huge pages multiply both the pause and the copy-on-write memory (→ persistence.md).
6. Still unexplained: check swap (used_memory_rss far above used_memory while the host swaps), mem_fragmentation_ratio below 1.0 means swapping, and blocked_clients for a queue of BLPOP/BRPOPLPUSH waiters (→ performance.md).

Expiration And Eviction

Two different mechanisms produce the same symptom, "my key is gone", and have opposite fixes.

• Expiry removes keys that had a TTL. Redis mixes lazy deletion on access with an active cycle that runs at hz (default 10) times per second: it samples 20 keys with a TTL per database, deletes the expired ones, and repeats immediately while more than 25% of the sample was expired. Consequence: a key can outlive its TTL in memory for a moment, but never in reads — Redis never returns an expired value.
• Eviction removes keys because maxmemory was hit, and depends entirely on maxmemory-policy: noeviction (default: writes fail with OOM), allkeys-lru / allkeys-lfu / allkeys-random, volatile-lru / volatile-lfu / volatile-random / volatile-ttl. Diagnose with INFO stats: expired_keys rising is expiry, evicted_keys rising is eviction.
• The trap the volatile-* policies set: they can only evict keys that carry a TTL. A mixed workload where the persistent half has no TTL and the cache half does will refuse writes with OOM even though most of memory is evictable — the eviction candidate pool was empty.
• Locks and queues need a noeviction instance, full stop. Under allkeys-* a lock key or a stream is an ordinary eviction candidate; under volatile-* a lock is the first victim, because the TTL every lock must carry is exactly what puts it in the candidate pool (→ locks.md, queues.md).
• LRU here is sampled, not true LRU: maxmemory-samples (default 5) candidates per eviction; raising it to 10 tracks true LRU closely at a small CPU cost. LFU (allkeys-lfu) counts frequency with a logarithmic counter, tuned by lfu-log-factor (default 10) and decayed by lfu-decay-time (default 1 minute) — better for a cache with a hot, stable working set.
• Replicas do not expire keys on their own: they wait for the master's DEL and meanwhile answer reads as if the key were gone. A replica's DBSIZE can therefore exceed the master's without anything being wrong.

Error Messages

Match on the prefix; message tails change between versions. Full playbooks in incidents.md.

Output Gates

Before emitting Redis commands, a client integration, or a config change:

• Does every key this creates either carry a TTL or have a named deleter?
• Is every read-modify-write one atomic command, one Lua script, or a WATCH retry loop?
• Is every keyspace-wide operation a SCAN loop, never KEYS, and every large delete an UNLINK?
• Under topology: cluster, do all keys of each multi-key command or script share a hash tag?
• Do the locks and queues sit on a noeviction instance (never allkeys-*, never volatile-*), and does the code still survive a restart?
• Are the round trips bounded — pipeline, MGET, or a script — rather than one call per item in a loop?
• Does anything here claim durability the configured persistence_mode and replication cannot deliver?
• Is a destructive command (FLUSHALL, pattern delete, CONFIG SET, failover) gated by destructive_confirm?

Configuration

User-dependent variables. Defaults apply until the user states a preference; store them in ~/Clawic/data/redis-store/config.yaml.

Preference areas — customizable dimensions; a stated preference is recorded in config.yaml and applied from then on:

• Tooling — CLI vs RedisInsight vs a provider console, migration tooling (RIOT, --cluster helpers), benchmark harness
• Thresholds — slowlog threshold worth reporting, big-key size that forces a refactor, fragmentation ratio that triggers a defrag, pipeline batch size, SCAN COUNT
• Conventions — key separator and namespace depth, hash-tag policy, stream and consumer-group naming, cache-key versioning scheme
• Platform — server version, container vs VM, instance memory and cores, network RTT and region, TLS on or off
• Risk posture — run destructive or admin commands directly vs hand back reviewed commands, whether replica reads are acceptable, whether runtime CONFIG SET is allowed at all
• Output format — redis-cli transcripts vs client-library code, whether Lua is welcome, how much of the reasoning to narrate
• Work order — measure-before-change gates, whether to rehearse on a restored copy first, review before any keyspace-wide operation
• Integrations — monitoring stack (INFO scraping, Prometheus exporter, provider metrics), backup destination, alerting targets
• Restrictions — commands disabled by policy or provider (KEYS, FLUSHALL, DEBUG, EVAL), compliance regimes mandating TLS, ACLs or encryption at rest
• Cadence — restore-drill frequency, big-key and TTL audit cycle, failover-drill schedule

Traps

Where Experts Disagree

• Redis as a primary database. One camp treats any AOF-durable Redis as a legitimate system of record; the other allows only regenerable data. The testable boundary: can you name the recovery path when the last N seconds of writes are lost and no other system has them? If not, Redis is a cache with good uptime, not a database.
• Redlock for distributed locking. Multi-node Redlock adds availability but not correctness: its author and its critics have never agreed on the clock and process-pause assumptions it needs. Practical stance: single-node lock plus a short TTL for mutual exclusion that is an optimization; when correctness depends on the lock, you need fencing tokens checked by the resource itself (→ locks.md).
• RDB vs AOF vs both. RDB restores fast and forks rarely; AOF loses less. The disagreement is about the fork cost on large datasets. Boundary: dataset under a few GB and both on is nearly free; tens of GB and the fork pause becomes the argument for RDB-on-a-replica-only.
• Cluster vs a bigger instance. Cluster costs you multi-key operations, transactions across slots, and operational complexity. Most workloads outgrow one node's throughput far later than they think: reach for Cluster when the dataset no longer fits one machine or a single core saturates, not at the first latency complaint.
• Lua in the server vs logic in the app. Scripts buy atomicity and cut round trips; they also block the server and are hard to observe and version. Boundary: short, bounded, key-scoped scripts yes; anything looping over unbounded data, no.
• Redis vs the Valkey fork. After the 2024 license change the ecosystem split; both remain protocol-compatible for the classic command set, and the divergence is in modules, threading and licensing rather than in the fundamentals of this skill (dated detail in managed-redis.md).

Related Skills

More Clawic skills, get them at https://clawic.com/skills/redis-store (install if the user confirms):

• caching — store-agnostic cache strategy: hierarchy, invalidation policy, stampede theory
• rate-limiting — picking the algorithm (token bucket, sliding window) before implementing it here
• distributed-systems — consistency models and partial failure behind the replication and lock tradeoffs
• pg — when the data wants durability, joins, or transactions rather than a memory store
• observability — turning INFO and latency events into alerts that fire before the incident

Part of Clawic, the verified skill library. Get this skill: https://clawic.com/skills/redis-store.