flickies

flickies

Self-hosted video toolkit — lipsync, face restore, and ffmpeg ops in one container. POST a JSON body, get a video back. Every video endpoint takes the same input/output contract; drive it from curl, the generated Go/Python clients, or point a function-calling LLM at the MCP endpoint.

Lipsync (POST /v1/video/lipsync): drive a face video/image from an audio track. Engines: latentsync-1.5 (ByteDance, Apache-2.0, commercial-safe default, CUDA-only) and wav2lip / wav2lip-gan (Rudrabha, fast/low-VRAM, LRS2 non-commercial — refused unless FLICKIES_ENABLE_NONCOMMERCIAL=1 is set in the server env). restore_face=true chains GFPGAN over the result.

Face restore (POST /v1/video/restore): GFPGAN v1.4 (gfpgan, Apache-2.0) — clean up a Wav2Lip mouth crop or old footage, standalone.

ffmpeg ops (pure CPU, no engine): POST /v1/video/trim, /concat, /transcode (mp4/webm/mov/mkv + gif + fps + codec change), /scale, /mux_audio, /extract_audio, /thumbnail_grid. Metadata: POST /v1/video/info (ffprobe).

Extras: async jobs (async_job=true → 202 + job_id → poll GET /v1/jobs/{job_id}), HMAC-signed webhooks on async completion, server-side file staging, engine load/evict control, an MCP endpoint at /v1/mcp with 11 tools, optional bearer-token auth.

For installation, configuration, and container setup, see references/setup.md.

Security & safety

• Auth is off by default — FLICKIES_AUTH_TOKEN is unset out of the box, so the whole API is open to anyone who can reach the port. Set it for any deployment beyond localhost, pass Authorization: Bearer once it's set, and bind to loopback / behind an authenticating proxy. Never expose an unauthenticated instance on a network.
• File and engine management — the staging and engine endpoints include remove and evict operations. Like any state change, only run them against a resource the current task created, only when the user asked, and not against a shared instance others depend on.

When To Use

• Lipsync a face (video or still image) to a driving audio track — latentsync-1.5 (commercial-safe, CUDA) or wav2lip / wav2lip-gan (non-commercial gate).
• Restore / sharpen faces in a video with GFPGAN, standalone or chained after a Wav2Lip pass (restore_face=true).
• Trim a clip, concatenate several clips, or transcode to another container/codec (incl. animated GIF).
• Change frame rate, re-encode with a specific codec/CRF/preset, scale dimensions, mux an audio track in, extract the audio out, or generate a thumbnail sprite-sheet.
• Probe a video for duration, codec, fps, dimensions, bitrate (/v1/video/info).
• Run any long operation fire-and-forget: submit with async_job=true, poll /v1/jobs/{id}, or receive an HMAC-signed webhook on completion.
• Drive the whole pipeline from a function-calling LLM (Claude, LibreChat, Cursor) via MCP at /v1/mcp.

When NOT To Use

• Real-time / streaming video output — every endpoint is request/response; long jobs go async + poll, not stream.
• latentsync-1.5 on the CPU image — it's CUDA-only (cuda_only: true) and the CPU image refuses to load it. GFPGAN is also CUDA-only in practice. Use the :latest-cuda image for those.
• wav2lip / wav2lip-gan anywhere unless the server was started with FLICKIES_ENABLE_NONCOMMERCIAL=1 — otherwise the request returns 403 NONCOMMERCIAL_GATE_REFUSED. This is a server-side env flag; you cannot flip it per-request.
• Two engines resident at once — one model is resident at a time. A different engine request triggers hot-swap eviction of the current one. If you need two simultaneously, run two containers.
• output_url via the MCP tools — MCP tools only accept output_path (they write under FILES_DIR). Use the REST endpoint if you need presigned-PUT output_url delivery.
• Multipart upload on the video endpoints — only PUT /v1/files/{path} accepts a raw-body upload. Video endpoints take JSON with file_path or file_url.

Setup

The container should already be running. Set the base URL:

export FLICKIES_URL=http://localhost:8000

If the server has FLICKIES_AUTH_TOKEN set, export it too:

export FLICKIES_AUTH_TOKEN=
# every request below then needs: -H "Authorization: Bearer $FLICKIES_AUTH_TOKEN"

Verify: curl $FLICKIES_URL/healthz returns {"status": "ok"} (unversioned, always auth-exempt). For the richer discovery payload — device, ffmpeg version, available/enabled/loaded engines, non-commercial flag — hit GET /v1/health:

curl -s $FLICKIES_URL/v1/health | jq
# { "status": "ok", "version": "...", "device": "cuda", "ffmpeg": "...",
#   "available_engines": [...], "enabled_engines": [...],
#   "loaded_engine": null, "noncommercial_enabled": false }

For install / configuration / env vars / CPU vs CUDA images / engine weights, see references/setup.md.

Quick Start

The input/output contract is uniform across every video endpoint:

• Input — exactly one of file_path (FILES_DIR-relative, staged via /v1/files) or file_url (any HTTP/HTTPS URL the server fetches).
• Output — exactly one of output_path (server writes to FILES_DIR/, response {path, size, ...}; download via GET /v1/files/) or output_url (server PUTs to a presigned/PUT-accepting URL, response {url, size, ...}). In async mode both are optional — the server auto-stages to jobs/{id}.{ext}.

# Probe a video (stage it first, then reference by path).
curl -s -X PUT --data-binary @clip.mp4 \
  -H "Authorization: Bearer $FLICKIES_AUTH_TOKEN" \
  "$FLICKIES_URL/v1/files/uploads/clip.mp4"

curl -s -X POST "$FLICKIES_URL/v1/video/info" \
  -H "Content-Type: application/json" \
  -d '{"file_path": "uploads/clip.mp4"}' | jq

# Trim seconds 5–12, write the result under FILES_DIR, download it.
curl -s -X POST "$FLICKIES_URL/v1/video/trim" \
  -H "Content-Type: application/json" \
  -d '{
        "file_path": "uploads/clip.mp4",
        "start_sec": 5, "end_sec": 12, "precise": true,
        "output_path": "out/clip-trimmed.mp4"
      }' | jq
curl -s "$FLICKIES_URL/v1/files/out/clip-trimmed.mp4" --output clip-trimmed.mp4

# Lipsync a face to an audio track (LatentSync, commercial-safe default, CUDA).
curl -s -X POST "$FLICKIES_URL/v1/video/lipsync" \
  -H "Content-Type: application/json" \
  -d '{
        "face_url": "https://example.com/face.mp4",
        "audio_url": "https://example.com/voice.wav",
        "engine": "latentsync-1.5",
        "output_path": "out/lipsynced.mp4"
      }' | jq

Add -H "Authorization: Bearer $FLICKIES_AUTH_TOKEN" to every call if the server has a token set. /healthz is the only always-exempt route.

API — POST /v1/video/lipsync

Drive a face from an audio track. JSON body.

Request Fields

Response

200 (sync) — one of:

{ "path": "out/lipsynced.mp4", "size": 4823110 }

{ "url": "https://bucket.example.com/out.mp4?...", "size": 4823110 }

StagedOutputResponse may also carry sha256, duration_sec, width, height. 202 (async) — { "job_id": "", "status": "accepted" }.

Error Contract

Error body is always { "code": "UPPER_SNAKE", "message": "...", "details"?: {...} }.

API — POST /v1/video/restore

GFPGAN face restoration on a video. Input via file_path / file_url, output via output_path / output_url (same contract).

Request Fields

Response

200 → StagedOutputResponse or UrlOutputResponse (as lipsync). 202 → JobAcceptedResponse.

Error Contract

400 BAD_REQUEST, 401 UNAUTHORIZED, 422 VALIDATION_FAILED. (GFPGAN carries no non-commercial gate.)

API — ffmpeg ops (POST /v1/video/{trim,concat,transcode,scale,mux_audio,extract_audio,thumbnail_grid})

Pure ffmpeg, CPU. Each takes the standard input/output contract plus op-specific fields. All return StagedOutputResponse xor UrlOutputResponse on 200. All support async_job / webhook_url except where noted (the info-shaped ones — extract_audio, thumbnail_grid — carry no BaseVideoOutputRequest; they take output_path / output_url directly and run sync).

trim — cut [start_sec, end_sec]

concat — join ≥2 videos in order

transcode — universal re-encode

output_format (from the output contract, mp4/webm/mov/mkv) drives the filter graph. Additional:

scale — resize

mux_audio — replace / merge the audio track

extract_audio — pull the audio out

thumbnail_grid — sprite-sheet PNG

Error Contract (ffmpeg ops)

Same envelope. 400 BAD_REQUEST (bad input xor, constraint violation), 422 VALIDATION_FAILED (missing start_sec/end_sec, rows/cols, width/height, arrays under minItems), 401 when auth is on.

API — POST /v1/video/info

ffprobe metadata. Input via file_path / file_url; no output fields.

curl -s -X POST "$FLICKIES_URL/v1/video/info" \
  -H "Content-Type: application/json" \
  -d '{"file_path": "uploads/clip.mp4"}' | jq

Response

{
  "duration_sec": 12.4, "width": 1920, "height": 1080, "fps": 30.0,
  "video_codec": "h264", "audio_codec": "aac", "bitrate": 4200000,
  "container_format": "mov,mp4,m4a,3gp,3g2,mj2", "size_bytes": 6510022
}

audio_codec, bitrate, container_format are nullable. Errors: 400 BAD_REQUEST, 401 UNAUTHORIZED.

Async Job Lifecycle

Any video-producing endpoint (lipsync, restore, and the ffmpeg ops that carry async_job) runs fire-and-forget when you set async_job: true. Lipsync/restore are the ones worth doing async — they're the slow ones.

1. Submit — POST with async_job: true. Output is optional; if you omit both output_path and output_url, the server auto-stages the result to jobs/{job_id}.{ext} under FILES_DIR. Response is 202:

JOB=$(curl -s -X POST "$FLICKIES_URL/v1/video/lipsync" \
  -H "Content-Type: application/json" \
  -d '{
        "face_url": "https://example.com/face.mp4",
        "audio_url": "https://example.com/voice.wav",
        "engine": "latentsync-1.5",
        "async_job": true
      }' | jq -r .job_id)

2. Poll — GET /v1/jobs/{job_id}:

curl -s "$FLICKIES_URL/v1/jobs/$JOB" | jq
# { "job_id": "...", "status": "running", "result": null, "error": null }

status ∈ pending / running / complete / failed / cancelled. On complete, result holds the output payload ({path, size} or {url, size}). On failed, error holds {code, message}.

3. Fetch — when status: complete, download the auto-staged result:

curl -s "$FLICKIES_URL/v1/files/jobs/$JOB.mp4" --output result.mp4

Webhook alternative — pass webhook_url on the async submit and the server POSTs the final job state ({job_id, status, result, error}) to that URL on completion instead of making you poll:

• HMAC-SHA256 over timestamp + "." + body, keyed by the server's FLICKIES_WEBHOOK_SECRET.
• Headers: X-Webhook-Timestamp: , X-Webhook-Signature: t=,v1=.
• Retried on non-2xx / transport error with exponential backoff (30s, 1m, 5m, 30m, 2h, 12h), then dead-lettered to the server log.
• Receiver MUST verify the signature and de-dupe on (timestamp, signature).

GET /v1/jobs/{job_id} returns 404 NOT_FOUND for an unknown id. The queue is in-process — jobs don't survive a container restart.

MCP Endpoint

flickies mounts a Model Context Protocol server at /v1/mcp (streamable-HTTP JSON-RPC, same FastAPI process, same auth middleware). Point a function-calling LLM at it and it drives the pipeline.

Eleven tools mirror the REST surface: list_engines, info, lipsync, restore, transcode, trim, concat, scale, mux_audio, extract_audio, thumbnail_grid. Argument shapes match the REST bodies, with one difference: MCP tools accept output_path only (they write under FILES_DIR; output_url presigned-PUT delivery is REST-only). MCP tools run synchronously — there's no async_job on the MCP side.

Wire it into Claude Code:

claude mcp add --transport http flickies $FLICKIES_URL/v1/mcp
# with auth:
claude mcp add --transport http flickies $FLICKIES_URL/v1/mcp \
  --header "Authorization: Bearer $FLICKIES_AUTH_TOKEN"

The transport requires Accept: application/json, text/event-stream. Raw JSON-RPC over HTTP POST for debugging / non-MCP callers:

# tools/list
curl -s "$FLICKIES_URL/v1/mcp/" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{"jsonrpc": "2.0", "id": 1, "method": "tools/list"}'

# tools/call — lipsync
curl -s "$FLICKIES_URL/v1/mcp/" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{
    "jsonrpc": "2.0", "id": 2, "method": "tools/call",
    "params": {
      "name": "lipsync",
      "arguments": {
        "face_url": "https://example.com/face.mp4",
        "audio_url": "https://example.com/voice.wav",
        "engine": "latentsync-1.5",
        "output_path": "out/lipsynced.mp4"
      }
    }
  }'

The canonical mount path carries a trailing slash (/v1/mcp/); bare /v1/mcp redirects to it. With auth on, every MCP call needs the same Authorization: Bearer header.

Bearer-Token Auth

If FLICKIES_AUTH_TOKEN is set on the server, every route except /healthz (and CORS preflight) requires Authorization: Bearer . Wrong/missing token returns 401 UNAUTHORIZED.

curl -H "Authorization: Bearer $FLICKIES_AUTH_TOKEN" $FLICKIES_URL/v1/engines

With FLICKIES_AUTH_TOKEN unset the API/MCP surface is unauthenticated — anyone who can reach it gets full access. Set the token and bind to loopback / behind an authenticating proxy; for untrusted networks add a reverse proxy doing TLS + rate limiting. See references/setup.md.

Typical Workflows

Lipsync + face restore in one shot

curl -s -X POST "$FLICKIES_URL/v1/video/lipsync" \
  -H "Content-Type: application/json" \
  -d '{
        "face_path": "uploads/portrait.png",
        "audio_path": "uploads/line.wav",
        "engine": "wav2lip-gan",
        "restore_face": true,
        "output_path": "out/talking.mp4"
      }' | jq
# NOTE: wav2lip-gan needs FLICKIES_ENABLE_NONCOMMERCIAL=1 on the server.

Stage once, run several ops off the same file

curl -s -X PUT --data-binary @raw.mp4 "$FLICKIES_URL/v1/files/uploads/raw.mp4"

curl -s -X POST "$FLICKIES_URL/v1/video/scale" \
  -H "Content-Type: application/json" \
  -d '{"file_path":"uploads/raw.mp4","width":1280,"height":720,"output_path":"out/720p.mp4"}' | jq

curl -s -X POST "$FLICKIES_URL/v1/video/transcode" \
  -H "Content-Type: application/json" \
  -d '{"file_path":"uploads/raw.mp4","output_format":"gif","fps":12,"gif_options":{"width":480}}' \
  --fail | jq   # async auto-stages if you omit output_path

Concatenate several clips (mixed sources → re-encode)

curl -s -X POST "$FLICKIES_URL/v1/video/concat" \
  -H "Content-Type: application/json" \
  -d '{
        "inputs_urls": ["https://ex.com/a.mp4","https://ex.com/b.mp4"],
        "precise": true,
        "output_path": "out/joined.mp4"
      }' | jq

Long lipsync, async + poll to completion

FLICKIES_URL=$FLICKIES_URL FLICKIES_AUTH_TOKEN=$FLICKIES_AUTH_TOKEN \
  bash scripts/flickies.sh lipsync \
  '{"face_url":"https://ex.com/f.mp4","audio_url":"https://ex.com/v.wav","engine":"latentsync-1.5"}' \
  result.mp4

See scripts/flickies.sh — submits any endpoint async, polls /v1/jobs/{id} to a terminal state, and downloads the staged result.

Free VRAM after a job

Frees the resident engine from VRAM. Rarely needed — engines hot-swap on demand — so only evict one the current task loaded, and remember a shared instance may have another caller using it.

curl -s -X DELETE "$FLICKIES_URL/v1/engines/latentsync-1.5"   # evict from VRAM (204)

Tips

1. file_url over staging — if the source is already at a URL, pass file_url and skip the upload round-trip.
2. precise=false is the fast path for trim/concat but snaps to keyframes / needs matching codecs. Flip to precise=true when you need frame accuracy or are joining mismatched inputs — it re-encodes.
3. One engine resident at a time — a lipsync request after a restore evicts the restore engine (hot-swap). restore_face=true intentionally chains GFPGAN second, evicting the lipsync model to free VRAM.
4. async_job=true for lipsync/restore — they're slow. Submit, then poll /v1/jobs/{id} or take a webhook. ffmpeg ops are fast enough to run sync.
5. Async output is auto-staged — omit output_path/output_url on an async submit and fetch from jobs/{job_id}.{ext}.
6. wav2lip* is gated at the server — 403 NONCOMMERCIAL_GATE_REFUSED means the operator hasn't set FLICKIES_ENABLE_NONCOMMERCIAL=1. You can't override it per-request; latentsync-1.5 is the ungated default.
7. CPU image can't run latentsync-1.5 or gfpgan — both are CUDA-only. Wav2Lip-CPU works (slow) if the gate is set. Use :latest-cuda for the full engine set.
8. GET /v1/health shows device, loaded_engine, and noncommercial_enabled — check it before you get a surprise 403 or a CPU-refusal.
9. Idempotency-Key — pass an Idempotency-Key header on a POST for safe retries; the server replays the cached response for a repeat (key, method, path).
10. X-Request-Id — send one to correlate logs across hops; the server echoes it back and mints a UUID4 if you don't.