voice-agent-memory

🎙️ Voice Agent Memory — Cross-Call BlueColumn Recall

Give your AI agent a phone number and a memory that never forgets.

This skill connects Twilio telephony → ElevenLabs/Deepgram voice engine → BlueColumn vector memory in one integrated pipeline. Every call is automatically transcribed, stored in BlueColumn, and recalled before the next call from that caller.

Why This Exists

The existing skills in the ecosystem are fragmented:

Architecture

                  ┌─────────────────────────┐
                  │      Twilio Phone #     │
                  │     +1 (929) 828-8689   │
                  └──────────┬──────────────┘
                             │ SIP / Webhook
                             ▼
                  ┌─────────────────────────┐
                  │  ElevenLabs Agent (TTS) │
                  │  or Deepgram Voice Agt  │
                  └──────────┬──────────────┘
                             │ /v1/chat/completions
                             ▼
                  ┌─────────────────────────┐
                  │   Voice Bridge Server   │ ←── port 8013
                  │   (FastAPI + Claude)    │
                  └──┬─────────────────┬────┘
                     │                 │
          ┌──────────▼────┐    ┌───────▼──────────┐
          │  BlueColumn   │    │  Cost Tracking   │
          │  /agent-recall│    │  Transcript Logs │
          │  /agent-store │    │  Call Analytics  │
          └───────────────┘    └──────────────────┘

  CALL FLOW:
  1. Phone rings → Twilio routes to ElevenLabs
  2. ElevenLabs → Voice Bridge (port 8013) with caller ID
  3. Bridge → BlueColumn recall: "what do we know about +12065550123?"
  4. Bridge injects memory into Claude's system prompt
  5. Claude responds → ElevenLabs TTS → Caller hears
  6. After call → Bridge auto-stores transcript → BlueColumn

What Makes This Different

1️⃣ Caller-Based Memory Isolation

Each phone number gets its own memory namespace. When +12065550123 calls:

• BlueColumn recalls every past interaction from that caller
• Agent greets them by name, picks up where they left off
• No cross-contamination between callers

2️⃣ Auto-Transcribe + Auto-Store

Every call transcript is automatically:

1. Truncated/summarized to key info
2. Pushed to BlueColumn /agent-remember with caller phone as title
3. Tagged with voice-call, caller number, and date
4. Available for recall on the NEXT call before the agent says a word

3️⃣ Pre-Call Context Injection

On inbound call, before the agent speaks:

1. Bridge reads caller ID from ElevenLabs metadata
2. Queries BlueColumn: "what do I know about [caller]?"
3. Injects answer into Claude's system prompt
4. Agent starts the call already informed

4️⃣ Real-Time Streaming Recall (Optional)

Mid-call recall queries:

• Agent can ask "what did we discuss about pricing last time?"
• Bridge streams recall results into the conversation context
• Sub-100ms recall means no awkward pauses

---

Setup

Prerequisites

• OpenClaw running (gateway on port 18789)
• Twilio account + phone number (already configured: +1 (929) 828-8689)
• BlueColumn API key (already configured: bc_live_*)
• ElevenLabs account + API key
• Anthropic API key (already configured)
• Cloudflare tunnel or ngrok (for exposing bridge to ElevenLabs)

Step 1: Configure Environment

# Copy template
cp .env.example .env
# Edit with your keys

Step 2: Install Dependencies

cd ~/.openclaw/workspace/skills/voice-agent-memory
pip install -r requirements.txt
# Or individually:
pip install fastapi uvicorn httpx anthropic python-dotenv

Step 3: Start the Bridge Server

python3 bridge/server.py
# Starts on port 8013
# Endpoint: POST /v1/chat/completions

Step 4: Expose with Cloudflare Tunnel

# If using existing tunnel:
cloudflared tunnel --url http://localhost:8013

# Or configure a permanent tunnel in Cloudflare dashboard
# Point a subdomain to localhost:8013

Step 5: Configure ElevenLabs Agent

In ElevenLabs dashboard:

1. Create a Conversational AI Agent
2. Set Custom LLM URL to: https://your-tunnel.ngrok.dev/v1/chat/completions
3. Add auth header: Authorization: Bearer
4. Set your Twilio number as the agent's phone number

Step 6: Connect Twilio

In ElevenLabs agent settings → Phone section:

1. Enter Twilio Account SID and Auth Token
2. Select your Twilio phone number
3. Save — your number is now live with BlueColumn memory

---

How It Works (Detailed)

BlueColumn Integration

The bridge uses three BlueColumn API endpoints:

Base URL: https://xkjkwqbfvkswwdmbtndo.supabase.co/functions/v1

Pre-Call Recall (on inbound ring)

# Before agent speaks, query memory for this caller
POST /agent-recall
{
  "q": "What do I know about caller +12065550123?"
}
# Response:
{
  "answer": "Joe Pagano set up BlueColumn... prefers bullet points...",
  "sources": [...]  # With relevance scores
}

Post-Call Storage (after call ends)

# Store the conversation summary
POST /agent-remember
{
  "text": "Full transcript or summary of the call...",
  "title": "Voice call - +12065550123 - 2026-06-08"
}
# Response:
{
  "session_id": "sess_...",
  "summary": "Discussed pricing plans...",
  "action_items": ["Send proposal by Friday"]
}

Mid-Call Quick Note (during call)

# Agent saves an observation mid-conversation
POST /agent-note
{
  "text": "Caller prefers email over phone for follow-up",
  "tags": ["voice-call", "+12065550123", "preference"]
}

Memory Prompts (Zero→First→Nth Call)

---

Voice Bridge Server

The bridge server (bridge/server.py) handles:

• HTTP Method: POST
• Endpoint: /v1/chat/completions (OpenAI-compatible)
• Auth: Bearer token in Authorization header
• Streaming: SSE-based streaming (ElevenLabs expects this)
• Model routing: Claude Sonnet 4 (via Anthropic API) or OpenClaw gateway

Key Features in the Bridge

1. Parallel recall: Starts BlueColumn recall in a background task while initializing the Claude stream — if recall finishes in <2s, it's injected; if slow, the call proceeds without it
2. Fire-and-forget memory: After each call exchange, the conversation is auto-stored to BlueColumn asynchronously
3. Cost tracking: Logs per-call costs (Twilio + ElevenLabs + Anthropic) to JSONL for analytics
4. Caller identification: Extracts caller ID from ElevenLabs metadata for memory namespacing

---

BlueColumn Audio API (Direct Audio Ingestion)

BlueColumn also accepts audio files directly — useful for uploading call recordings post-hoc:

# Upload a recorded call for memory storage
curl -X POST https://xkjkwqbfvkswwdmbtndo.supabase.co/functions/v1/agent-remember \
  -H "Authorization: Bearer bc_live_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "audio_url": "https://your-storage/call-recording-2026-06-08.mp3",
    "title": "Voice call recording - +12065550123 - 2026-06-08"
  }'

# Or upload via file URL (PDFs also supported)
curl -X POST .../agent-remember \
  -d '{
    "file_url": "https://your-storage/transcript.pdf",
    "title": "Call transcript - +12065550123"
  }'

---

Outbound Calling

In addition to inbound, the bridge supports outbound calls:

# Trigger an outbound call from the agent
curl -X POST http://localhost:8013/call/outbound \
  -H "Authorization: Bearer " \
  -d '{
    "to": "+12065550123",
    "purpose": "Follow up on pricing page deadline",
    "pre_call_recall": true
  }'

The bridge will:

1. Query BlueColumn for context about this caller
2. Initiate the call via Twilio
3. Inject memory into the agent's prompt before the call connects
4. Auto-store the transcript after the call

---

Costs

Estimated per-minute:

---

Files

voice-agent-memory/
├── SKILL.md              ← This file
├── .env.example          ← Config template
├── requirements.txt      ← Python dependencies
├── bridge/
│   ├── server.py         ← FastAPI bridge (port 8013)
│   ├── prompt_builder.py ← System prompt + memory injection
│   └── memory.py         ← BlueColumn recall/store helpers
├── scripts/
│   ├── start.sh          ← Start the bridge server
│   ├── test-recall.sh    ← Test BlueColumn recall for a caller
│   └── test-call.sh      ← Test making an outbound call
└── contacts.json         ← Whitelisted callers

---

Testing

# Test BlueColumn recall for a specific caller
./scripts/test-recall.sh +12065550123

# Test the bridge directly (simulate ElevenLabs)
curl -X POST http://localhost:8013/v1/chat/completions \
  -H "Authorization: Bearer " \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4",
    "messages": [
      {"role": "user", "content": "Hello, this is Joe calling again"}
    ],
    "stream": false
  }'

# Check call logs
tail -f ~/.openclaw/workspace/memory/voice-calls/costs.jsonl

---

License

MIT — built on BlueColumn.ai and OpenClaw.