OATDA Compare Models

OATDA Compare Models

Send one prompt to multiple LLM providers in parallel and get each model's response back. Benchmarking, model evaluation, and "which LLM is best for X" use cases.

API Key Resolution

All commands need the OATDA API key. Resolve it inline for each exec call:

export OATDA_API_KEY="${OATDA_API_KEY:-$(cat ~/.oatda/credentials.json 2>/dev/null | jq -r '.profiles[.defaultProfile].apiKey' 2>/dev/null)}"

If the key is empty or null, tell the user to get one at https://oatda.com and configure it.

Security: Never print the full API key. Only verify existence or show first 8 chars.

Model Mapping

Default comparison set if user doesn't specify: openai/gpt-5, anthropic/claude-sonnet-4-5-20250929, google/gemini-3-pro-preview, deepseek/deepseek-v4-pro.

⚠️ Models update frequently. If a model ID fails, query oatda-list-models for the latest available models.

API Call

Wichtig: Feld heißt modelId (nicht model!), models ist Array-of-objects, max 8.

export OATDA_API_KEY="${OATDA_API_KEY:-$(cat ~/.oatda/credentials.json 2>/dev/null | jq -r '.profiles[.defaultProfile].apiKey' 2>/dev/null)}" && \
curl -s -X POST "https://oatda.com/api/v1/compare" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OATDA_API_KEY" \
  -d '{
    "prompt": "",
    "models": [
      {"provider": "openai", "modelId": "gpt-5"},
      {"provider": "anthropic", "modelId": "claude-sonnet-4-5-20250929"},
      {"provider": "google", "modelId": "gemini-3-pro-preview"}
    ],
    "temperature": 0.7,
    "maxTokens": 1024,
    "stream": false
  }'

Replace `` with the user's prompt (JSON-escape special characters).

Optional parameters:

• temperature: 0 (deterministic) to 2 (creative). Default: 0.7
• maxTokens: Max tokens per model response
• stream: true for SSE stream (default), false for batch JSON
• Per-model override: each entry in models may include its own messages array and service_tier (standard | flex | priority)

Response

Non-stream JSON shape:

{
  "results": [
    {
      "provider": "openai",
      "modelId": "gpt-5",
      "success": true,
      "response": "GPT-5's answer...",
      "tokenUsage": {"prompt_tokens": 30, "completion_tokens": 120, "total_tokens": 150, "cost": 0.0035}
    },
    {
      "provider": "anthropic",
      "modelId": "claude-sonnet-4-5-20250929",
      "success": true,
      "response": "Claude's answer...",
      "tokenUsage": {"prompt_tokens": 30, "completion_tokens": 95, "total_tokens": 125, "cost": 0.0021}
    }
  ]
}

Stream response: SSE events tagged with provider/modelId, one or more per model.

Present results side-by-side so the user can compare quality / cost / token-efficiency:

Comparison: ""
─────────────────────────────────────────────────

[GPT-5]  (cost: $0.0035, 150 tokens)
GPT-5's answer...

[Claude Sonnet 4.5]  (cost: $0.0021, 125 tokens)
Claude's answer...

Errors

Individual model failures inside results[].success: false are NOT API errors — that specific model failed, the API call succeeded. Show the failure inline with the others.

Example

User: "Compare how GPT-5, Claude Sonnet 4.5, and Gemini 3 Pro explain recursion in one sentence."

export OATDA_API_KEY="${OATDA_API_KEY:-$(cat ~/.oatda/credentials.json 2>/dev/null | jq -r '.profiles[.defaultProfile].apiKey' 2>/dev/null)}" && \
curl -s -X POST "https://oatda.com/api/v1/compare" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OATDA_API_KEY" \
  -d '{
    "prompt": "Explain recursion in one sentence.",
    "models": [
      {"provider": "openai", "modelId": "gpt-5"},
      {"provider": "anthropic", "modelId": "claude-sonnet-4-5-20250929"},
      {"provider": "google", "modelId": "gemini-3-pro-preview"}
    ],
    "temperature": 0.7,
    "maxTokens": 256,
    "stream": false
  }'

Tips

• Field name is modelId, not model (stricter schema than /api/v1/llm).
• Max 8 models per request (server-configurable max_compare_models).
• Cost accumulates per model: 4-model comparison ≈ sum of 4 individual calls. Run oatda-check-balance first if budget is tight.
• For a single-model call, use oatda-text-completion instead — simpler, returns direct cost.
• For deeper model discovery (capabilities, pricing, supported_params), run oatda-list-models first.
• NEVER expose the full API key in output.