Voice Cloning

Voice cloning

Produce speech in a custom voice, one you either clone from a real recording or design from a text description, then synthesize any text in it. The difference from plain text-to-speech: the voice is bespoke, not picked off a preset list. Two intents live here, clone-from-sample and design-from-description, and they route to different models.

Inputs to collect

• Which intent: clone an existing voice (have a recording) or design a new one (have a description). This picks the model.
• For cloning: a clean reference recording of the target voice. A few seconds is enough (Qwen ~3s, Dia ~5-10s). Mono, low noise, single speaker. Plus a transcript of what's said in it if using the Qwen clone path.
• For designing: a voice description covering gender, age, timbre, accent, pace, and mood (e.g. "warm older male, gentle British accent, unhurried").
• The text to speak, and the target language.
• Consent / likeness: clone only voices you own or have explicit rights to. Refuse cloning a third party's voice without permission.

Models

• Clone from a sample (single speaker): Qwen3-TTS Base (alibaba:qwen@3-tts-1.7b-base). Takes the reference recording in inputs.audio with speech.voice: "clone". Highest-similarity mode (ICL) needs a transcript of the sample; embedding-only mode (settings.xVectorOnly: true) skips the transcript at lower similarity. 10+ languages.
• Clone for dialogue / multi-speaker: Dia 1.6B (runware:dia@1.6b) or Dia2 2B (runware:dia2@2b). Reference voices go in inputs.audios (first = [S1], second = [S2]), supports non-verbal cues. English only. Dia2 streams and is the newer pick.
• Design from a description: Qwen3-TTS VoiceDesign (alibaba:qwen@3-tts-1.7b-voicedesign). The voice is created from a natural-language description in positivePrompt, with speech.voice: "design". No recording needed.
• Premium preset timbres (not cloning): Qwen3-TTS CustomVoice (alibaba:qwen@3-tts-1.7b-customvoice). Nine curated timbres (aiden, serena, vivian, …) plus a style hint. Use it when a polished stock voice with style control beats a clone.

Confirm each model is live and inspect its schema via runware-models + runware-run before calling. Voice/cloning fields differ per model, never hardcode them.

Workflow

1. Resolve the chosen model's schema (runware-run) and confirm the cloning/voice fields and speech.text limit.
2. Upload the reference recording (cloning paths) into inputs.audio (Qwen) or inputs.audios (Dia). For designing, no upload, just the description.
3. Build the request: taskType: "audioInference", the model AIR, speech.text with the lines to speak, plus the voice field for that model.
4. Run asynchronously and poll getResponse to terminal. Audio is a time-based task, do not block a sync call on it.
5. Read the audio URL from the result, then check it before returning (see Quality bar).

Technique

• Reference quality sets the ceiling. A clean, dry, single-speaker sample clones well. Background music, reverb, or two voices in the clip degrade similarity more than any setting can recover.
• Qwen clone, ICL vs embedding. Provide the transcript of the reference for the high-similarity ICL mode. Only fall back to settings.xVectorOnly: true (no transcript) when you cannot transcribe the sample, and expect a looser match.
• Design by stacking traits. For VoiceDesign, write the positivePrompt as ordered attributes: gender and age, then timbre, then accent, then pace and mood. "A bright young female voice, soft timbre, light Spanish accent, quick and upbeat" beats a vague "nice voice."
• Style and emotion ride the prompt. On CustomVoice and VoiceDesign, a style hint in positivePrompt ("Speak with great enthusiasm") shapes delivery on top of the voice. This is the same emotion/delivery lever covered in voiceover.
• Dia speaks dialogue. Tag lines [S1] / [S2] in speech.text and align them to the reference order in inputs.audios. Insert non-verbal cues inline like (laughs), (sighs), (coughs). For turning that into a back-and-forth scene, compose with dialogue-audio.
• One voice, many lines. Once a clone or design reads right, reuse the exact same inputs (same reference, same description, same voice settings) across every line so the character stays one voice and does not drift between clips.

Parameters that matter

• speech.text - the lines to speak. Max 2000 chars on Qwen variants, 3000 on Dia. Split longer scripts.
• speech.voice - "clone" (Qwen Base), "design" (VoiceDesign), a preset id (CustomVoice). Dia has no voice field, the reference audio is the voice.
• inputs.audio (Qwen Base, single string) / inputs.audios (Dia, up to 2) - the reference recording(s) as URL, UUID, or base64.
• settings.transcript + settings.xVectorOnly (Qwen Base) - transcript drives high-similarity ICL, xVectorOnly: true trades similarity for skipping it.
• speech.language - Auto, English, Chinese, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian (Qwen). Dia is English only.
• speech.speed - 0.25 to 4 on Qwen variants (default 1).
• CFGScale / settings.temperature (Dia) - lower temperature is cleaner and more consistent, higher is more expressive but riskier. Confirm exact field names against the live schema, never guess.

Quality bar

• The cloned voice is recognizably the reference speaker, or the designed voice matches every requested trait (gender, age, accent, mood).
• Speech is clean: no clipping, garble, dropped words, or runaway hallucinated tail audio. Retry with a lower temperature or a cleaner reference if it drifts.
• Across a multi-line set the voice stays identical (same inputs reused), not a near-cousin per clip.
• Likeness consent was confirmed for any cloned real person.

Related skills

runware-run, runware-models, runware-prompting; voiceover (single-narrator delivery and emotion control), dialogue-audio (multi-speaker conversations from these voices).