article-to-video

Article to Video

Convert articles in Word (.docx), PDF (.pdf), Text (.txt), or Markdown (.md) format into narrated videos with AI-generated visuals, TTS voiceover, subtitles, background music, and scene transitions.

When to Use

• User says "把这篇文章转为视频" / "convert this article to video"
• User uploads a .docx/.pdf/.txt/.md file and wants a video version
• User mentions "文章转视频" / "article to video" / "文字转视频"
• User wants to create a narrated video from a document with voiceover

Workflow Overview

The pipeline has 5 stages, each handled by a dedicated Python script:

[Document] → ① parse_doc.py → scenes.json
                              ↓
                   ② generate_tts.py → audio + timing.json
                              ↓
              ③ create_slides.py (or GenerateImage) → images
                              ↓
                ④ assemble_video.py → final MP4 + SRT
                              ↓
                         ⑤ Output: video.mp4 + subtitle.srt

Stage 1: Document Parsing (scripts/parse_doc.py)

Parses the input document into a structured JSON of scenes.

Supported formats:

• .docx → python-docx (paragraphs with heading levels)
• .pdf → pdfplumber (page-by-page text extraction, table detection)
• .txt → direct read, split by blank lines
• .md → markdown library, preserve heading hierarchy

Output format (scenes.json):

{
  "title": "Article Title",
  "scenes": [
    {
      "index": 0,
      "heading": "Section Title",
      "level": 1,
      "narration": "Clean text suitable for TTS reading...",
      "slide_text": "Key points for visual display...",
      "image_prompt": "Description for AI image generation...",
      "char_count": 250
    }
  ],
  "total_chars": 5000,
  "estimated_duration_sec": 1000
}

Usage:

python scripts/parse_doc.py --input article.docx --output scenes.json

Key logic:

• Split by headings (H1/H2) into scenes
• For each scene: extract narration text (clean markdown/HTML), generate slide text (key points), create image prompt
• Estimate duration: Chinese ~4.5 chars/sec, English ~2.8 words/sec
• Handle complex content: tables → text summary, code blocks → formatted text, formulas → description

Content type auto-detection: The parser automatically detects the document's content type using keyword analysis, and stores it in scenes.json as content_type. This drives visual style and BGM selection in later stages.

Override with --content-type technology.

Stage 2: TTS Generation (scripts/generate_tts.py)

Converts narration text to speech with retry and fallback.

Primary engine: edge-tts (free, high quality, multi-voice Chinese support) Fallback chain: edge-tts → gTTS → pyttsx3 (offline)

Key features:

• Retry with exponential backoff on 429 rate limit (2^n seconds, max 3 retries)
• 1-3 second delay between scenes to avoid throttling
• Precise duration measurement via ffprobe/ffmpeg -i after generation
• Caching: text SHA256 hash → skip regeneration if cached
• Output: scene_NN.mp3 + timing.json (with actual measured durations)

Voice Profiles (new): Use --profile to apply a preset combination of voice + rate + pitch + volume:

python scripts/generate_tts.py --scenes scenes.json --outdir ./audio --profile professional

Available voices (Chinese):

• zh-CN-XiaoxiaoNeural - 女声，亲切自然 (default)
• zh-CN-YunxiNeural - 男声，年轻活力
• zh-CN-YunyangNeural - 男声，专业播音
• zh-CN-XiaoyiNeural - 女声，温柔甜美
• zh-CN-YunjianNeural - 男声，体育解说

Stage 3: Visual Generation (scripts/create_slides.py)

Two modes for generating scene visuals:

Mode A - Template slides (default, fast): Renders HTML templates to PNG images using the built-in slide.html template.

• Title slides with gradient backgrounds
• Content slides with bullet points
• Quote/highlight slides
• Configurable color themes and fonts
• Auto-selects theme based on content_type (with optional --theme override)

Mode B - AI image generation (optional, higher quality): Uses the GenerateImage tool or Seedream plugin to create custom illustrations.

• Content-type-aware prompts: Each scene's image_prompt is automatically adapted based on the detected content type (e.g. finance → financial infographic style, technology → futuristic cyberpunk aesthetic)
• In AI mode, the pipeline generates ai_image_requests.json — a manifest of all image generation requests with prompts, output paths, and dimensions
• Template slides are generated as fallback — if AI generation fails, the template slide remains
• The AI agent reads the manifest and calls GenerateImage for each request

AI Image Style Mapping:

Usage (template mode):

python scripts/create_slides.py --scenes scenes.json --outdir ./slides --mode template

Usage (AI mode): The skill instructions below describe how to use GenerateImage for this stage.

Stage 4: Video Assembly (scripts/assemble_video.py)

Combines images + audio into final video using FFmpeg.

Pipeline:

1. For each scene: create video clip from image + audio (with Ken Burns zoom + pan effect)
2. Add crossfade transitions between scenes (xfade filter)
3. Mix background music with narration (amix filter, BGM at -15dB)
4. Burn subtitles into video (optional)
5. Concatenate all scene clips → final MP4

Key features:

• Ken Burns effect: zoompan filter with zoom + pan for cinematic camera movement. Pan direction (right/left/down/up) is deterministically selected per scene via filename hash, ensuring varied but reproducible movement. Pan range is configurable per speed preset (slow: 60px, normal: 80px, fast: 100px).
• Ken Burns speed adapts to content type (slow for finance, fast for tech)
• Scene transitions: xfade filter (fade, dissolve, slideleft)
• BGM auto-selection: picks BGM from assets/bgm// based on content type
• BGM style override: use --bgm-style to force a specific BGM style (e.g. --bgm-style cinematic)
• Style-specific BGM volume: each BGM style has its own volume level (e.g. corporate: -18dB, cinematic: -12dB)
• Random selection: when multiple audio files exist in a style directory, one is randomly selected
• Background music: amix with automatic ducking
• Subtitle burning: SRT → subtitles filter with Chinese font (FontSize=12, no background box, 20px from bottom)
• Extended timeout: FFmpeg operations use 1800s timeout for long videos (20+ min), with fast preset and CRF 24 for subtitle burning and concatenation
• Windows-safe: all temp files use ASCII-only short names
• Disk space check before starting
• Progress tracking via JSON file

Usage:

# BGM auto-selected based on content type (with style-specific volume)
python scripts/assemble_video.py \
  --slides ./slides \
  --audio ./audio \
  --timing ./audio/timing.json \
  --output final_video.mp4 \
  --resolution 1920x1080 \
  --transition fade

# Override BGM style (ignores content type mapping)
python scripts/assemble_video.py \
  --slides ./slides \
  --audio ./audio \
  --timing ./audio/timing.json \
  --output final_video.mp4 \
  --bgm-style cinematic \
  --resolution 1920x1080

# Or specify BGM file manually
python scripts/assemble_video.py \
  --slides ./slides \
  --audio ./audio \
  --timing ./audio/timing.json \
  --output final_video.mp4 \
  --bgm ./assets/bgm/corporate/track01.mp3 \
  --resolution 1920x1080

BGM Library Management (scripts/bgm_manager.py)

Manage background music files in the assets/bgm// directory structure.

BGM Styles:

Commands:

# List all BGM files
python scripts/bgm_manager.py list

# List files in a specific style
python scripts/bgm_manager.py list --style corporate

# Upload (copy) audio file to a style directory
python scripts/bgm_manager.py upload --file music.mp3 --style corporate
python scripts/bgm_manager.py upload --file music.mp3 --style corporate --name track01

# Remove a BGM file
python scripts/bgm_manager.py remove --style corporate --file track01.mp3

# Validate an audio file before uploading
python scripts/bgm_manager.py validate --file music.mp3

# Show BGM configuration summary
python scripts/bgm_manager.py info

Upload validation:

• Supported formats: .mp3, .m4a, .wav
• Minimum file size: 10KB
• Minimum audio duration: 5 seconds (verified via ffprobe)
• ASCII-safe filenames (spaces replaced with underscores)

Stage 5: Output

Final deliverables:

• final_video.mp4 - The complete narrated video
• subtitle.srt - Subtitle file (also burned into video)
• thumbnail.jpg - Video thumbnail (first scene frame)

Complete Usage Example

# Step 1: Parse document (auto-detects content type)
python scripts/parse_doc.py --input article.docx --output scenes.json
# Or manually specify: --content-type technology

# Step 2: Generate TTS audio (use voice profile or individual settings)
python scripts/generate_tts.py --scenes scenes.json --outdir ./audio --profile professional
# Or: --voice zh-CN-XiaoxiaoNeural --rate +0% --pitch +0Hz

# Step 3: Generate slide images (auto-selects theme from content type)
python scripts/create_slides.py --scenes scenes.json --outdir ./slides --mode template
# Or override: --theme dark

# Step 4: Assemble final video (auto-selects BGM from content type)
python scripts/assemble_video.py \
  --slides ./slides \
  --audio ./audio \
  --timing ./audio/timing.json \
  --output final_video.mp4 \
  --resolution 1920x1080

Configuration

All defaults are in scripts/config.py. Key options:

Platform Presets

Important Notes

• Windows paths: All temp files use ASCII-only names (scene_001, scene_002...) to avoid FFmpeg/edge-tts path encoding issues
• FFmpeg: Uses imageio-ffmpeg which bundles a full FFmpeg binary with all codecs (PNG decode, H.264 encode, AAC/MP3 audio). No separate system FFmpeg installation required. The config auto-detects the bundled binary.
• Chinese font: Subtitle burning uses "Microsoft YaHei" font family (must be installed on system). Change in config if unavailable
• Disk space: Video generation needs ~5x document size in temp space. Script checks before starting
• Long documents: Documents >10,000 chars are split into batches. Each batch generates independently, then concatenated
• Breakpoint resume: Progress is saved to progress.json. Re-running skips completed scenes

Error Handling

• TTS 429 → exponential backoff retry, then fallback to gTTS, then pyttsx3
• Image generation failure → fallback to template text card
• FFmpeg error → check disk space, temp file cleanup, retry once
• Document parse error → report format-specific error message with fix suggestion