AI Video Repurpose × Digital Human Dubbing

AI Video Repurpose × Digital Human Dubbing

Core Scenario

Paste a Douyin/TikTok short video link → wait ~3 minutes → get a video that looks like YOU said it

Architecture

Paste link → [Download] → [Transcribe] → [AI Rewrite] → [TTS] → [Digital Human] → Deliver
    yt-dlp       Whisper       LLM        edge-tts     SadTalker

Quick Start

🏃 3-step setup

cd ai-recreator

# 1️⃣ Install dependencies
pip install -r backend/requirements.txt

# 2️⃣ Configure (at minimum set your API Key)
cp backend/.env.example backend/.env  # then edit .env with your key

# 3️⃣ Start
cd backend && uvicorn main:app --host 0.0.0.0 --port 8080

Open http://localhost:8080 in your browser → paste a Douyin link → click "Start" → wait 3 minutes → get your digital human video.

Prerequisites

5-Step Pipeline

Step 1: Download (modules/downloader.py)

• Downloads video & extracts audio via yt-dlp from Douyin/TikTok/Kuaishou
• Auto-detects platform; 120s timeout, 3 retries
• Supports proxy (DOWNLOAD_PROXY env) for cloud server bypass

Step 2: Transcribe (modules/transcriber.py)

• Whisper speech-to-text, default base model
• Model cascade: tiny → base → small → medium → large
• Cached per task to avoid re-transcription

Step 3: AI Rewrite (modules/rewriter.py)

• LLM rewrites short-video copy into natural, personal speech
• Strips influencer buzzwords ("hey guys", "OMG", etc.)
• Supports custom style prompts
• Falls back to rule-based rewrite when no API Key is set

Step 4: TTS Synthesis (modules/tts_engine.py)

• Default edge-tts (free, fast, great Chinese)
• Supports MeloTTS / CosyVoice switching
• Auto-splits long text + ffmpeg concatenation

Step 5: Digital Human (modules/digital_human.py)

• Default SadTalker: single photo → talking-head video
• Falls back to audio+placeholder video when SadTalker unavailable
• Wav2Lip interface reserved

API Reference

Submit a task (URL)

curl -X POST http://localhost:8080/api/tasks \
  -H "Content-Type: application/json" \
  -d '{
    "video_url": "https://www.douyin.com/video/xxxxx",
    "custom_prompt": "Make it more casual and humorous",
    "tts_voice": "zh-CN-XiaoxiaoNeural"
  }'

Upload a video file (skip download)

curl -X POST http://localhost:8080/api/tasks/upload-video \
  -F "file=@/path/to/your/video.mp4" \
  -F "custom_prompt=Use a friendly tone"

Configuration (.env)

File Structure

ai-recreator/
├── README.md                 # Project overview
├── SKILL.md                  # This skill definition
├── skill-card.md             # ClawHub listing card
├── .gitignore                # Git ignore rules
├── cookies.example.txt       # Cookie template (optional)
├── backend/
│   ├── main.py              # FastAPI entry point
│   ├── config.py            # Config management
│   ├── models.py            # Data models
│   ├── task_manager.py      # Task queue manager
│   ├── pipeline.py          # Pipeline orchestrator
│   ├── requirements.txt     # Python dependencies
│   ├── .env.example         # Config template (copy to .env)
│   ├── assets/              # Static assets
│   │   └── default_avatar.png
│   └── modules/
│       ├── file_handler.py  # File upload handling
│       ├── downloader.py    # Video downloader
│       ├── transcriber.py   # Speech-to-text
│       ├── rewriter.py      # AI text rewriting
│       ├── tts_engine.py    # Text-to-speech
│       └── digital_human.py # Digital human generator
├── frontend/
│   ├── index.html           # Single-page Web UI
│   └── static/              # Static assets
├── references/
│   ├── faq.md               # FAQ (20 questions)
│   ├── anti-patterns.md     # Anti-patterns guide
│   ├── quickstart.md        # Quick start guide
│   └── privacy.md           # Privacy & data handling
└── data/
    ├── downloads/           # Download cache
    ├── audio/               # TTS output
    ├── transcripts/         # Transcription results
    ├── rewrites/            # Rewrite results
    ├── output/              # Final videos
    └── temp/                # Temporary files

Capability Boundaries

✅ What it does well

1. Short video repurposing - Extract speech from Douyin/TikTok/Kuaishou links, AI-rewrite into personal expression
2. Digital human talking-head generation - Voice + photo → lip-synced talking-head video
3. Style transformation - Removes influencer lingo, keeps core message
4. Multi-engine TTS - Supports edge-tts / MeloTTS / CosyVoice
5. Real-time progress tracking - Live 5-step pipeline status
6. Graceful degradation - Falls back to rule rewriting (no API Key) or placeholder video (no GPU)

⚠️ Requires user-provided assets

1. Voice cloning - Requires voice sample recordings (future release)
2. HD digital human (>720p) - Needs paid API (HeyGen/D-ID)
3. Lip-sync with original video - Needs Wav2Lip reference video (currently SadTalker single-image mode)
4. Custom digital human avatar - User needs to provide a front-facing half-body photo
5. Non-short-video platform URLs - Platforms other than Douyin/TikTok/Kuaishou need extra testing

❌ Out of scope

1. Batch processing - Single link per submission (submit multiple times for batch)
2. Video editing/cutting - Only extracts audio; no video editing
3. Watermark removal - Operates on audio track only
4. Deepfake/face swap - No face-swapping tech; lip-sync only
5. Live stream processing - No live stream or m3u8 support
6. Original content generation - Not responsible for factual accuracy of rewrites

🔒 Security & Privacy

1. URL filtering - Only http/https allowed; rejects file:// protocol injection
2. Timeout control - Each step has guard timeouts (120s~400s)
3. Graceful degradation - Handles missing API Key / GPU without crashing
4. File cleanup - data/temp/ cleaned after task completion
5. Data isolation - Audio & transcripts stored per task_id
6. API Key security - Read from .env, never logged
7. Lightweight dependencies - All AI capabilities optional; default runs fully local

👤 Audience

Reference Docs

• references/faq.md — 20 FAQs + edge cases
• references/anti-patterns.md — 7 common anti-patterns + fixes
• references/quickstart.md — 10-second quick start
• references/privacy.md — Privacy & data handling

Version History

• v1.0.1 (2026-07-29): Fixed .env startup crash; added DOWNLOAD_PROXY; added --js-runtimes node; added upload-audio/upload-video endpoints; privacy docs & config template
• v1.0.0 (2026-07-28): Initial release. 5-step pipeline, Web UI, REST API, multi-engine TTS, SadTalker digital human, graceful degradation