Video Subtitle Translation & Dubbing

video-subtitle-translation-dubbing

[English] | 简体中文

OpenClaw Skill — Multi-language video subtitle translation and automatic dubbing, producing high-quality multi-track, multi-subtitle videos on demand.

• Name: video-subtitle-translation-dubbing
• Version: 0.1.3
• Entrypoint: translation_dubbing_skill.run
• Manifest: manifest.yaml

Overview

This skill processes input videos (via external subtitles or by auto-extracting embedded subtitle tracks) and produces:

• An independent translated target-language subtitle file (UTF-8 SRT/VTT)
• A muxed .mkv video file

Callers select one of two processing modes via processing_mode:

Quick Start

from translation_dubbing_skill import parse_manifest, run

params = parse_manifest({
    "video_path": "/path/to/input.mp4",
    "subtitle_path": "/path/to/input.en.srt",   # Optional
    "source_language": "en",
    "target_language": "zh-CN",
    "processing_mode": "subtitle_and_dubbing",
    "translation_provider": "llm",
    "translation_endpoint": "https://api.example.com/v1/chat/completions",
    "translation_credential": "sk-...",
    "translation_config": {"model_name": "gpt-4o-mini"},
    "tts_provider": "edge",
    "tts_endpoint": "none",
    "tts_credential": "none",
})

result = await run(params)
print(result.output_video_path, result.output_subtitle_path)

Inputs

For full parameter definitions and defaults, see manifest.yaml.

Outputs

Output Video Track Structure

subtitle_and_dubbing mode:

streams:
  video:  video  (codec copy, preserving resolution/fps/encoding)
  audio:  target (AAC, language=target, default=1, title="Target Dubbing")
          source (copy, language=source, default=0, title="Original Audio")
  subs:   target (SRT, language=target, default=1, title="Target Subtitle")
          source (SRT, language=source, default=0, title="Original Subtitle")

subtitle_only mode:

streams:
  video:  video  (codec copy)
  audio:  source (copy, language=source, default=1)
  subs:   target (SRT, language=target, default=1)
          source (SRT, language=source, default=0)

Progress Reporting

The skill reports progress stage-by-stage via the progress callback injected by the OpenClaw runtime:

parsing → translating → [tts] → muxing → done

• translating stage includes completed / total counts (monotonic non-decreasing)
• tts stage occurs only in subtitle_and_dubbing mode and also reports progress counts
• done stage returns output_video_path / output_subtitle_path in extra

Pluggable Providers

Built-in providers (auto-registered via @register upon module loading):

New providers can be added by implementing the protocol under translation_dubbing_skill.providers.{translation,tts} and decorating with @register(kind, provider_type). No caller code changes required.

Adaptive Scheduler

Translation and TTS requests are driven by AdaptiveScheduler, featuring 3D adaptive tuning:

• Batch Size (batch_size): Number of entries per request
• Payload Size (payload_size): Text length measured in tokens or characters
• Concurrency (concurrency): Number of simultaneous in-flight requests

Uses AIMD strategy: scales up on consecutive successes; scales down on 429 (RateLimitError) with exponential backoff; reduces payload_size and re-slices without penalty on 413 / context window overflow (PayloadTooLargeError); retries with backoff on 5xx / timeouts (TransientError).

Default parameters are specified in manifest.yaml under translation_rate_limit / tts_rate_limit.default.

Error Handling

All exceptions inherit from SkillError, carrying a stage / code / reason / context tuple. Sensitive keys (credential / api_key / authorization) are automatically masked as *** during to_dict() serialization.

Prerequisites

• Python ≥ 3.11
• ffmpeg / ffprobe: Must be available in PATH. Used for subtitle extraction, audio time-stretching, video muxing, and media probing.
• httpx: HTTP client
• pydub: Audio segment manipulation and peak normalization
• PyYAML: Manifest parsing

License

See LICENSE and pyproject.toml in the repository root.