the-librarian

The Librarian

Lightweight document search with TurboVec quantization. Build semantic search indexes that run on minimal hardware.

Author: RandTrad Consulting — Document Intelligence for SMEs License: MIT — Free for personal and commercial use with attribution Version: 1.2.0 Security: See Security section for safe usage guidelines and vulnerability history

What It Does

• Builds quantized vector indexes from Markdown/text documents
• Supports hybrid search (vector + BM25 keyword matching)
• Optional Flashrank reranking for improved accuracy
• Chunk expansion for surrounding context
• 8-16x smaller indexes than FAISS

When to Use

Accuracy: ~97-98% of FAISS for 4-bit quantization. Top results may occasionally swap ranking.

Quick Start

Prerequisites

# Install BLAS library (required for TurboVec)
sudo apt install libblas3

# Create venv and install dependencies
cd /path/to/the-librarian
python3 -m venv venv
source venv/bin/activate
pip install turbovec numpy requests rank-bm25 flashrank

Build an Index

# Using the wrapper (recommended)
./scripts/librarian build /path/to/documents/ index/my_library

# With options
./scripts/librarian build /path/to/docs/ index/my_library --bits 3 --chunk-size 800

# Direct Python
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libblas.so.3 \
  python scripts/build_index.py --input /path/to/docs/ --output index/my_library

Search

# Pure vector search
./scripts/librarian search "habit formation" index/my_library

# Hybrid (vector + BM25)
./scripts/librarian search "habit formation" index/my_library --hybrid

# Hybrid + rerank (best accuracy)
./scripts/librarian search "habit formation" index/my_library --hybrid --rerank

# With context expansion
./scripts/librarian search "habit formation" index/my_library --hybrid --rerank --expand 1

# JSON output
./scripts/librarian search "habit formation" index/my_library --json

Search Modes

Bit Width Options

File Structure

the-librarian/
├── SKILL.md
├── scripts/
│   ├── librarian           # Wrapper script (handles LD_PRELOAD)
│   ├── build_index.py      # Build quantized index
│   └── search.py           # Search with hybrid + rerank
└── references/
    └── quantization.md     # How TurboVec compression works

Index Files

After building, you'll have:

index/my_library/
├── library.qindex      # TurboVec quantized index
├── chunks.json         # Document chunks with metadata
├── bm25_index.json     # BM25 keyword index (JSON — safe, no code execution risk)
└── stats.json          # Build statistics

Security: BM25 index stored as JSON (not pickle) to prevent arbitrary code execution from untrusted index files. Legacy bm25_index.pkl files are detected but NOT loaded — rebuild indexes to migrate.

Security

Safe Usage Guidelines

• Only load indexes you built yourself or received from a trusted source
• Always use a local Ollama instance (localhost or 127.0.0.1) for sensitive documents — document text is sent to the embedding API over HTTP
• Never share index directories — they contain full document text in chunks.json
• If you receive an index from someone else, ensure it uses bm25_index.json (not .pkl)
• The --api flag can point to any HTTP endpoint — verify the host before running
• If using a remote embedding API, be aware that document text and queries are transmitted over the network

Vulnerability History

v1.1.0 (July 2026):

• Critical fix: pickle.load on bm25_index.pkl removed. Pickle files can execute arbitrary code when loaded. BM25 index now stored as JSON. Legacy pickle files are detected but refused to load.
• High fix: Embedding API URL now checked against localhost. If a non-local API endpoint is configured, a privacy warning is displayed before any data is transmitted.
• Medium fix: SKILL.md now documents that document text and search queries are transmitted to the Ollama HTTP API for embedding generation.

v1.2.0 (July 2026):

• Removed all private infrastructure details (local file paths, API endpoints, credentials paths, phone numbers, cron job IDs) from published skill. Skill is now generic and safe for public distribution.

Reporting Security Issues

Report vulnerabilities to randtradbusiness@gmail.com. Security fixes will be published on ClawHub.

Accuracy Guidance

For critical applications (medical, legal, financial):

Use FAISS instead. The ~2-3% ranking variance in TurboVec is acceptable for personal knowledge bases, parts catalogs, and general document search, but not for applications where missing a result has consequences.

For personal/team use:

TurboVec is ideal. The accuracy difference is negligible for most queries, and the size savings enable deployment on hardware that couldn't run FAISS at all.

Performance Comparison

Note: Both spend ~120-140ms generating embeddings via Ollama. The search difference is minimal.

References

• references/quantization.md - Technical details on how TurboVec compression works

Author

RandTrad Consulting — Document Intelligence consultancy for SMEs

• Website: https://www.randtradconsulting.com
• Contact: randtradbusiness@gmail.com
• Services: AI Training, EU AI Act Compliance, Document RAG Systems

Built by Enda Rochford — RandTrad Consulting

License

MIT License — Free for personal and commercial use with attribution.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files, to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, subject to the following condition:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.