Video Understanding

Video Understanding (Gemini)

Analyze videos using Google Gemini's multimodal video understanding. Supports 1000+ video sources via yt-dlp.

Requirements

• Python 3.10+ with uv
• yt-dlp — brew install yt-dlp / pip install yt-dlp
• ffmpeg — brew install ffmpeg (for merging video+audio streams)
• GEMINI_API_KEY environment variable

Trigger Rule

Use this skill first whenever the user provides a video URL, YouTube link, or video file. The trigger is the video input itself, not whether the user explicitly says "Gemini," "analyze," or "transcribe."

Casual asks like "watch this," "look at this," "what is this?," "you laugh," or "what happens here?" still require this skill.

Gotchas

• If the skill cannot run because GEMINI_API_KEY or another dependency is missing, report that blocker before falling back to local extraction.
• Requires yt-dlp and ffmpeg installed (sanity-check with which yt-dlp / which ffmpeg).
• Gemini File API uploads are reusable for follow-up questions while ACTIVE, but expire after about 48 hours.
• Gemini explicit CachedContent can speed repeated questions, but has model-specific minimum token counts and storage billing.
• The analyze script downloads non-YouTube videos locally first unless a valid cached Gemini file handle is reused.
• YouTube age-restricted or private videos require authentication cookies.

Default Output

Returns structured JSON:

• transcript — Verbatim transcript with [MM:SS] timestamps
• description — Visual description (people, setting, UI, text on screen, flow)
• summary — 2-3 sentence summary
• duration_seconds — Estimated duration
• speakers — Identified speakers

Usage

Analyze a video (structured JSON output)

uv run {baseDir}/scripts/analyze_video.py "https://example.com/video.mp4"

Local video files work too:

uv run {baseDir}/scripts/analyze_video.py "/path/to/video.mp4"

Ask a question (adds "answer" field)

uv run {baseDir}/scripts/analyze_video.py "https://example.com/video.mp4" -q "What product is shown?"

Continue questions against the same video

Use a stable --session-key for the conversation or Slack thread. --reuse-file-cache keeps and reuses an ACTIVE Gemini File API handle instead of re-uploading; stale handles automatically fall back to a fresh upload/download.

uv run {baseDir}/scripts/analyze_video.py "https://example.com/video.mp4" --session-key "slack-thread-123" --reuse-file-cache -q "Summarize this"
uv run {baseDir}/scripts/analyze_video.py "https://example.com/video.mp4" --session-key "slack-thread-123" --reuse-file-cache --continue-chat -q "What happens after the pricing screen?"

For long videos with repeated follow-ups, add --use-context-cache to try Gemini explicit CachedContent. If Gemini rejects caching, the script falls back to File API reuse.

uv run {baseDir}/scripts/analyze_video.py "https://example.com/video.mp4" --session-key "slack-thread-123" --reuse-file-cache --use-context-cache --cache-ttl-seconds 3600 -q "Find all UI errors"

Override prompt entirely

uv run {baseDir}/scripts/analyze_video.py "https://example.com/video.mp4" -p "Custom prompt" --raw

Download only (no analysis)

uv run {baseDir}/scripts/analyze_video.py "https://example.com/video.mp4" --download-only -o video.mp4

Options

How It Works

1. Local video files → Uploaded directly to Gemini File API
2. YouTube URLs → Passed directly to Gemini (no download needed)
3. All other URLs → Downloaded via yt-dlp → uploaded to Gemini File API → poll until processed
4. With --reuse-file-cache, valid Gemini file handles are reused for follow-up questions; expired/deleted handles trigger a fresh upload/download
5. With --use-context-cache, the script tries Gemini CachedContent and falls back to File API reuse if caching is unavailable
6. Gemini analyzes video with structured prompt → returns JSON
7. Temp local files are cleaned up automatically; remote Gemini uploads are deleted unless reuse/cache mode is enabled

Supported Sources

Any URL supported by yt-dlp: Loom, YouTube, TikTok, Vimeo, Twitter/X, Instagram, Dailymotion, Twitch, and 1000+ more.

Tips

• Use -q for targeted questions on top of the full analysis
• Use --session-key plus --reuse-file-cache for Slack-thread follow-ups so Loom/local videos are not re-uploaded every turn
• Use --continue-chat when the follow-up relies on prior wording like "that screen" or "the second speaker"
• Use --purge-cache when done with a sensitive video to delete the remote Gemini handles
• YouTube is fastest (no download step)
• YouTube follow-ups resend the public URL and optional prior context; there is no documented durable YouTube asset handle
• Transient Gemini 503/high-demand errors are retried; set --fallback-model only when that model has quota on the account
• Large videos (10min+) work fine — Gemini File API supports up to 2GB (free) / 20GB (paid)
• The script auto-installs Python dependencies via uv