What Makes a Good Audio Prompt for Zero-Shot TTS?
Learning Prompt Selection from Synthesis Outcomes

Arnon Turetzky, Yossi Adi
The Hebrew University of Jerusalem
Paper arXiv Code (coming soon) BibTeX
Bar chart: WER gain over a random prompt for four TTS models. The audio selector gains more than the UTMOS and DNSMOS rules and the best single descriptor, close to probe selection.
Zero-shot TTS clones a voice from a few seconds of audio, and which few seconds you cut from a recording changes how intelligible the result is. Choosing the cleanest-sounding prompt by predicted quality (UTMOS, DNSMOS) misses most of this. Our audio selector, a small network trained on synthesis outcomes, picks the prompt from its audio alone and lowers the word error rate (WER) more, for every model, close to probe selection, which first synthesizes 12 outputs from every candidate. WER gain over a randomly chosen prompt on 400 held-out recordings, in percentage points (higher is better).

Abstract

Zero-shot text-to-speech (TTS) models clone a target voice from a short reference clip, commonly referred to as the prompt. In practice, the prompt has to be cut from a longer recording of the speaker, so which part to use must be chosen. To understand what makes a good prompt, we analyze hundreds of thousands of outputs synthesized by four TTS models from candidate prompts of YouTube recordings. The choice of prompt substantially changes intelligibility, naturalness and speaker similarity. Standard predicted-quality scores capture mainly the naturalness part but miss most of the intelligibility gain. We find that the models largely agree on which prompt yields the highest measured speaker similarity, but much less on which prompt yields intelligible speech. We propose a small network that learns from the outputs a TTS model produced to rank a recording's candidate prompts from their audio alone. Without synthesizing anything at selection time, it picks prompts that lower the word error rate on held-out recordings by a third (Qwen3-TTS) to two fifths (F5-TTS) relative to a randomly chosen prompt, more than predicted quality achieves and nearly as much as synthesizing a dozen test sentences from every candidate first.

Which few seconds should the model hear?

Benchmarks hand every test sentence its prompt. Real use does not: a user has a video, a podcast or a voice message of the speaker, and has to decide which few seconds to give the model. We judge a prompt by the speech it leads to. For over 1,500 YouTube recordings, we cut 30 s of the speaker's voice into 19 candidate prompts (3, 5 and 10 s), synthesize held-out sentences from every candidate with four open TTS models (Qwen3-TTS 0.6B and 1.7B, F5-TTS, Chatterbox), and score every output for intelligibility, naturalness and speaker similarity.

30 s of the speaker's recording best predicted quality our pick 5 s candidate 10 s candidate 19 candidate prompts per recording (10 × 3 s, 6 × 5 s, 3 × 10 s) zero-shot TTS model + a target sentence from the best-predicted-quality prompt misreads words from our pick reads the sentence … for every candidate: 5 sentences × 3 seeds Illustration; the paper reports averages over hundreds of held-out recordings.

What makes a good prompt?

The prompt matters, and not only by chance

Choosing a recording's best 3 s prompt by test syntheses lowers WER by about a third for Qwen3-TTS and by half for F5-TTS, and also improves naturalness and speaker similarity. Part of this is a stable property of the prompt: for a fixed sentence, a sizeable share of the variation between outputs is due to the prompt rather than the seed.

Predicted quality finds the clean prompt, not the intelligible one

Picking the candidate with the best UTMOS or DNSMOS score improves mainly naturalness and recovers only part of the intelligibility gain. It also looks better than it is when the same predictor judges the outputs: by a held-out judge (Audiobox Aesthetics), much of its naturalness gain disappears. When candidates differ in added noise, the quality rule picks the clean version, yet our picks are still more intelligible.

Partly shared, partly model-specific

The four models largely agree on which prompt yields the highest measured speaker similarity, but much less on which yields intelligible speech, which depends on how a model uses its prompt (Chatterbox, which does not read the prompt transcript, agrees with neither). Selection for intelligibility is therefore best learned for each model.

Left: transfer of selectors between TTS models in percent of probe selection's gain. Right: agreement between models, high for speaker similarity and low for WER.
Left: a selector trained on one model's outcomes (rows), applied to another model (columns), in percent of that model's probe-selection gain. Right: do two models prefer the same prompts? Within-recording rank correlation of the candidates' outcomes, corrected for how consistently each model ranks them; above the diagonal speaker similarity, below it WER.

Learning prompt selection from synthesis outcomes

Synthesizing from every candidate tells us which prompt is good, but it is too costly to repeat for every new speaker. We therefore learn to predict it, once per TTS model. A small network hears only the candidates' audio and is trained to rank a recording's candidates in the order of the outcomes the model produced from them. At selection time it scores the candidates and keeps the best one, with no synthesis at all.

candidate 1 candidate 2 candidate 3 … 19 prompt audio only Whisper-base encoder (frozen) 21M parameters small trained network 2 transformer layers + attention pooling compared with the recording 1.4M parameters, 5 averaged score per candidate chosen prompt training only: synthesis outcomes rank the candidates by the WER of the model's outputs (listwise loss)

The network predicts several outcomes of each candidate (mean WER, WER rank within the recording, failure rate, UTMOS, DNSMOS and speaker similarity) and selects by the WER rank, since intelligibility is what predicted quality misses. Adding its UTMOS output with a small weight trades back some predicted naturalness without retraining.

Results

On held-out recordings, the selector lowers WER by 32% (Qwen3-TTS 0.6B) to 43% (F5-TTS) relative to a randomly chosen prompt, without synthesizing anything at selection time. On Qwen3-TTS this matches probe selection, which needs 228 syntheses per recording, and on F5-TTS it gains 2.7 times as much as choosing by UTMOS.

Probe selection's WER gain grows with the number of syntheses per candidate and reaches the audio selector at 12.
Probe selection improves with more syntheses per candidate and reaches the audio selector only at 12, that is 228 syntheses per recording (Qwen3-TTS 0.6B, held-out recordings).
WER gain against naturalness gain by UTMOS and by the held-out Audiobox judge, for the quality rules and the selectors.
Intelligibility against naturalness, judged by UTMOS (left) and by the held-out Audiobox judge (right). The curve adds the selector's UTMOS output with increasing weight. The UTMOS rule's lead in naturalness is large only by UTMOS itself.

Robustness

  • The selector beats the UTMOS rule among candidates of each single prompt length, so it does not just prefer some length.
  • Whisper transcribes the prompts, judges the outputs and provides the encoder. Judged by an unrelated recognizer (wav2vec 2.0) instead, the advantage is even larger.
  • The gain holds with the YouTube captions as prompt transcripts, and on prompts whose Whisper transcript and caption agree.
  • The supervision is what matters: trained on the prompts' own UTMOS or DNSMOS instead of synthesis outcomes, the same network gains half as much or less. A small speech encoder suffices: Whisper-base matches Whisper-large-v3 with a thirtieth of its parameters.

BibTeX

@article{turetzky2026prompt,
  title   = {What Makes a Good Audio Prompt for Zero-Shot {TTS}? Learning Prompt Selection from Synthesis Outcomes},
  author  = {Turetzky, Arnon and Adi, Yossi},
  journal = {arXiv preprint arXiv:2610.40372},
  year    = {2026}
}