MongoDB

User preferences and memory live in ~/Clawic/data/mongodb/ (see setup.md on first use, memory-template.md for the file format). If you have data at an old location (~/mongodb/ or ~/clawic/mongodb/), move it to ~/Clawic/data/mongodb/.

When To Use

• Designing or reviewing document schemas: embed vs reference, growing arrays, multi-tenant layout, versioned shapes
• A query, pipeline, or dashboard is slow: reading explain plans, designing compound indexes, killing a scan
• Building or debugging aggregation pipelines, $lookup joins, materialized views, window functions
• Connecting an application: connection strings, pooling, timeouts, retries, ODM behavior, serverless clients
• Operating production: replica sets, sharding, write and read concerns, backups, upgrades, security, Atlas
• A live incident: no primary, connection storm, disk full, runaway operation, cache stall, blown oplog window
• Not for SQL or relational databases — normalization instincts from relational design actively mislead here (see Related Skills)

Quick Reference

Depth on demand, by phase:

• Diagnose — slow-queries.md explain plans and triage · errors.md code to cause · incidents.md no primary, disk full, connection storm, cache stall · monitoring.md what to watch and alert on · mongosh.md the shell toolkit
• Design — schema.md embed vs reference and the pattern catalog · indexes.md ESR, index types, lifecycle · aggregation.md pipeline craft · time-series.md measurements over time · search.md text and vector search
• Change — migrations.md online schema change, backfills, bulk loading · transactions.md sessions and retry loops · change-streams.md reacting to writes
• Operate — connections.md drivers, pools, timeouts, ODMs · replication.md replica sets, concerns, oplog, upgrades · sharding.md shard keys and the balancer · tuning.md WiredTiger cache and host settings · backups.md dumps, snapshots, PITR, drills · security.md auth, roles, TLS, injection, encryption · atlas.md the managed platform

Core Rules

1. Read explain("executionStats"), never guess. Healthy ratio: totalDocsExamined / nReturned ≈ 1. Worked example: 1,240,000 examined for 25 returned = 49,600:1 — missing or wrong index (Atlas's Query Targeting alert fires at 1000:1 by default). A ratio near 1 with a slow query is a different bug: look at nReturned itself, then at the sort.
2. 16MB is a ceiling, not a budget. WiredTiger rewrites the whole document on every update: a 5MB document taking a 20-byte $inc still costs a 5MB rewrite in cache. Keep working documents in the KB range; a document you routinely read in full should fit in a single-digit number of 4KB pages.
3. Unbounded arrays are the #1 schema failure. Anything that grows per event goes to its own collection or a time-series collection (MongoDB >=5.0). A multikey index adds one entry per element: a 10,000-element array = 10,000 index entries for one document, paid on every write to that document.
4. Set write and read concern explicitly in the connection string. Implicit defaults changed across versions (→ replication.md); code relying on them silently changes durability semantics on upgrade. Baseline URI: ?retryWrites=true&w=majority.
5. One compound index per query shape, ordered Equality → Sort → Range. Prefix rule: {a: 1, b: 1, c: 1} also serves queries on {a} and {a, b} — delete those redundant single-field indexes. Index intersection exists but the planner rarely picks it; never design for it.
6. Single-document atomicity is the concurrency primitive. Model invariants inside one document before reaching for transactions; when you do use them, stay under 1,000 modified documents and well inside the 60s default transaction lifetime (→ transactions.md).
7. Read-your-own-writes requires primary reads or a causal-consistency session. Secondary lag is usually sub-second but unbounded under load — never assume freshness on a secondary; bound it with maxStalenessSeconds when you read one deliberately.
8. One MongoClient per process, sized from concurrency, not from the default. Total server connections = processes × maxPoolSize (driver default 100): 40 pods × 100 = 4,000 connections ≈ 4GB of mongod RAM before a single document is served. Creating a client per request or per serverless invocation is the same bug at higher speed (→ connections.md).

Query Semantics That Bite

• {tags: "red"} matches docs where tags IS "red" or an array CONTAINING "red" — implicit array traversal, both directions, and you cannot turn it off.
• {qty: {$gte: 5, $lte: 10}} on an array field matches [3, 12]: different elements satisfy each bound. One element must satisfy all conditions → $elemMatch.
• {field: null} matches explicit null AND missing field. Distinguish with {field: {$type: "null"}} vs {field: {$exists: false}}.
• $ne, $nin, $not technically use the index but scan most of it — near-collection-scan cost; restructure with a positive predicate or a status field.
• skip(100000).limit(20) walks 100,020 index entries. Paginate by range on the sort key instead: {_id: {$gt: lastSeenId}} — constant cost per page.
• Anchored case-sensitive regex /^abc/ uses the index; /^abc/i cannot. Case-insensitive lookups need a collation index (→ indexes.md).
• Comparisons cross BSON types in a fixed order rather than erroring: the string "5" is never $gt: 4. Mixed types in one field silently split every range query — enforce the type with a $jsonSchema validator.
• $in with a large array is a range, not an equality, for index-order purposes; a short $in gets exploded into parallel index scans and keeps the sort order (→ indexes.md).
• Dotted paths into arrays are positional-ambiguous: {"items.0.sku": x} targets a position, {"items.sku": x} targets any element. $ and $[] update operators follow the same split.

Consistency Model

• Acknowledged ≠ durable: a w: 1 write vanishes into a rollback file if the primary fails before replication — someone must reconcile those by hand, and nothing in the application will ever know.
• Failovers happen; retryable writes (default on in modern drivers) hide most of them — but updateMany and deleteMany are NOT retryable writes. Wrap multi-doc mutations in idempotent logic or a transaction.
• Causal sessions give read-your-own-writes even across secondaries; use them instead of forcing primary everywhere.
• Read concern local can return writes that later roll back; majority cannot. available on a sharded cluster skips orphan filtering and can return documents twice — never use it for correctness-sensitive reads.
• Arbiters are a durability trap, not a cheap third node: the PSA topology stalls majority writes when one data node is down (→ replication.md).

ObjectId and _id

• 12 bytes: 4-byte unix timestamp + 5-byte random + 3-byte counter. ObjectId.getTimestamp() recovers creation time; sorting by _id approximates insertion order (per-process counter, not a global clock).
• Predictable enough to enumerate — never use as a security token or unguessable URL.
• Monotonic growth makes _id a hotspotting shard key (→ sharding.md).
• _id is immutable: changing it means insert-new + delete-old, which is a migration, not an update (error 66 in errors.md).
• Custom _id is legitimate and free of a second index when the natural key is stable and short ({_id: ":"}). Random UUIDv4 as _id costs B-tree locality on inserts; UUIDv7 or a prefixed key keeps writes clustered.

Error Codes

Codes are stable; message text is not. Match on the code. Full catalog with recovery steps in errors.md.

Configuration

User-dependent variables. Defaults apply until the user states a preference; store them in ~/Clawic/data/mongodb/config.yaml.

Preference areas — customizable dimensions; a stated preference is recorded in config.yaml and applied from then on:

• Tooling — shell vs Compass vs driver REPL, migration framework (plain scripts, migrate-mongo, Mongock), diagram and schema-analysis tooling
• Thresholds — profiler slowms, the examined:returned ratio worth reporting, pool size, replication-lag alarm, index-size budget per collection
• Conventions — collection naming (plural/singular), timestamp field names, soft-delete policy, schema_version usage, index naming, enum-as-string vs code
• Platform — server version, deployment target, storage class, instance memory and core count — affects every number in tuning.md
• Risk posture — whether to run mutations directly or hand back reviewed scripts, whether secondary reads are allowed, how aggressive index changes may be on a live collection
• Output format — snippets only vs snippets plus an explain walkthrough, how much plan detail to narrate, whether to include the rollback path by default
• Integrations — monitoring stack (Atlas metrics, Prometheus exporter, FTDC), backup tooling, search backend (Atlas Search vs external engine)
• Restrictions — compliance regimes that mandate field-level encryption or auditing, collections that must never be touched online, features the platform forbids
• Cadence — restore-drill frequency, index-usage review cycle, oplog-window review, upgrade windows

Output Gates

Before emitting a schema, an index, a pipeline, or a migration:

• Is every array in this schema bounded by something other than optimism?
• Does each new index correspond to a real query shape, in ESR order, with its redundant single-field prefixes removed?
• Was the query run through explain("executionStats"), with examined:returned checked, and not just read?
• Does the pipeline's first stage reach an index, and does no blocking stage sit ahead of the filter?
• Is the write concern explicit, and does it match the data class (telemetry vs money)?
• Is the backfill batched, resumable, and safe to run twice?
• Does the destructive step (drop, unfiltered deleteMany, index drop) ship separately from the code change, after the code that stopped using it?
• For anything user-supplied reaching a query object: is it type-checked so {$ne: null} cannot arrive where a string was expected (→ security.md)?

Traps

Where Experts Disagree

• Embed-first vs reference-first. MongoDB's own guidance is embed-first; teams from relational or microservice backgrounds reference-first for independent lifecycles. Boundary: embed when the child is always read with the parent AND bounded; reference otherwise.
• Transactions. One school treats a multi-document transaction as a schema-design smell (redesign so the invariant fits one document); the other uses them freely since 4.0. Boundary: cross-entity invariants you cannot co-locate (ledger + balance) justify them; convenience joins do not.
• Secondary reads for scale. Often called a myth — every secondary applies every write anyway, so they add read capacity only. Still legitimate for analytics isolation and geo-local latency with a bounded maxStalenessSeconds.
• Schema validators. One camp enforces $jsonSchema at the database because application-only validation always drifts; the other keeps validation in code because a validator change is a live DDL-shaped operation. Testable boundary: more than one writer (services, scripts, humans in the shell) means the database has to hold the rule.
• Aggregation in the database vs the application. Pushing analytics into pipelines competes with the operational workload for the same WiredTiger cache. Boundary: sub-second, index-backed, user-facing aggregations belong in MongoDB; anything scanning history belongs in a materialized collection, an analytics node, or a warehouse (→ aggregation.md).

Related Skills

More Clawic skills, get them at https://clawic.com/skills/mongodb (install if the user confirms):

• pg — PostgreSQL; jump there when the model is genuinely relational or you need cross-row constraints and SQL
• db — general database operations, reliability, and scaling patterns
• database-indexing — index theory beyond MongoDB specifics: write-cost budgets, structures, composite ordering
• vector-databases — comparing Atlas Vector Search against dedicated vector stores by scale and filter needs
• elasticsearch — when text search outgrows Atlas Search: analyzers, relevance tuning, aggregations

Part of Clawic, the verified skill library. Get this skill: https://clawic.com/skills/mongodb.