Voice Cloning Models Compared: 150 ms at Chunk Boundary, Not Yet a User Experience Metric

CosyVoice.org’s “just 150ms” promise is not a model-and-experiment fact in the supplied evidence. No supplied version-specific source directly measures CosyVoice 2.0 at that figure. The number instead appears as generic “latency” on CosyVoice.org and as first-packet latency for Fun-CosyVoice 3.0, a later generation. Without hardware, protocol, percentile, and concurrency details, 150 ms is a claim awaiting measurement, not a settled production baseline.

The systems problem is timestamp mismatch. A model packet may precede codec buffering, device decoding, and audible playout; tail queueing adds another delay. The source set provides no shared clock linking those events and 0 head-to-head CosyVoice2-versus-F5-TTS tests using matched audio, prompts, hardware, languages, and latency metric. CosyVoice 2.0 has a production account, but the advertised timing is insufficiently specified.

Lovora’s production results also complicate speed-versus-quality rankings. Its product offered 8 languages, while the model was trained on 5, leaving a 3-language overlap. German reached 71% round-trip WER, French 50%, and Spanish 47%; Arabic was described as unintelligible at 100% WER. The missing matched benchmark is consequential: a useful guide must timestamp packets, decoded frames, and audible output separately, then pair those measurements with queueing and intelligibility before calling 150 ms a user-experience advantage.

150 ms Begins in the Chunk

The production SLA begins at a chunk boundary, not at a model’s published real-time factor. For a 2026 launch, I would gate first audio at p95 ≤150 ms from input-ready to output-device loopback under peak load on the complete deployment stack. CosyVoice 2 is the defensible low-latency candidate only if it passes that test; otherwise, neither model’s headline establishes production readiness.

“Zero-shot” means no inference-time fine-tuning on the target speaker—not no speaker input. Both systems condition on a short reference, typically 3–5 seconds. CosyVoice 2 encodes that prompt as speech tokens. F5-TTS uses reference mel features without requiring a transcription of the reference audio, although it still receives the target text to synthesize. Conditioning quality and speaker similarity are therefore separate from temporal availability: a convincing voice does not guarantee that any waveform is available early.

The official CosyVoice2-0.5B checkpoint exposes the relevant mechanism directly. An LLM emits a causal speech-token sequence on a 25-Hz grid. A chunk-aware flow-matching stage converts successive token chunks into acoustic representations, and a vocoder renders successive waveform chunks. This creates a native overlap opportunity: token prediction can continue while earlier acoustic chunks are decoded and vocoded. The 25-Hz grid is a representation rate, not a 150 ms promise; chunk policy, look-ahead, batching, buffering, and device startup still determine when sound becomes audible.

F5-TTS follows a different clock. Its non-autoregressive flow-matching transformer uses target text and reference-mel conditioning to drive a full-sequence mel-spectrogram denoising trajectory, without an autoregressive duration predictor or phoneme aligner. Because the waveform stage needs the completed acoustic target, native generation naturally waits substantially beyond one chunk. A favorable real-time factor describes whole-job throughput; it says nothing by itself about how much audio exists at the beginning of that job. Thus, low-throughput-factor execution cannot establish conversational first-audio latency.

For every request, I would require synchronized monotonic-clock event markers and retain these four timestamped intervals:

Report p95 for the sum of the first three intervals and keep final-sample completion beside it rather than collapsing the two metrics. If CosyVoice 2 clears the loopback p95 gate at peak load, its native chunk path has earned the low-latency choice. If it does not, F5-TTS cannot rescue that SLA through a favorable real-time factor; it is appropriate only where early audio is unnecessary and its measured throughput and quality gates also pass.

Published Evidence

Yusheng Chen et al.’s October 2024 F5-TTS report is evidence of an author-run English benchmark, not evidence that the currently selected artifact meets a production first-audio SLA. The papers establish historical baselines; current qualification must establish whether their serving behavior survives the actual stack and load.

The corpus descriptions concern data exposure rather than the deployment questions that matter. Even when hours are accumulated across languages, they do not reveal the effective amount of matched English, accented, or noisy-speaker supervision. Consequently, no unsupported total can be used to support a claim about production robustness, voice-cloning fidelity, or responsiveness.

A cross-paper quality winner would be a category error. F5-TTS’s LibriSpeech-PC numbers and CosyVoice 2’s benchmark tables do not share identical reference recordings, text normalization, vocoders, or serving stacks. A shared metric label does not make those conditions equivalent. Comparing a WER from one table with a similarity score or subjective result from another would select a paper rather than a production system.

The publication dates also make artifact provenance essential. Model weights, preprocessing, streaming boundaries, and vocoder behavior can change across releases, while the license determines whether a technically successful checkpoint is legally deployable. An undated statement such as “CosyVoice 2 quality” is therefore incomplete unless the evaluated checkpoint, code commit, vocoder, and license are recorded beside it.

The myth to discard is that a favorable real-time factor necessarily implies conversational first audio; it describes throughput on the authors’ setup, not the time until a sample becomes audible at the output device in production. Likewise, a paper-level first-packet claim does not establish peak-load tail latency. The decision must therefore come from the same-stack load test already specified: CosyVoice 2 is defensible only if it passes the p95 time-to-first-audible-audio gate of ≤150 ms. F5-TTS remains eligible only when early audio is unnecessary and its measured throughput and quality gates also pass. Without that evidence, neither model’s published headline establishes production readiness.

The Production Choice

CosyVoice 2 wins the architecture-eligibility screen, not yet the production launch. Its native chunk-causal path can expose a playable prefix; native F5-TTS uses a full-sequence, non-autoregressive path and therefore has no native prefix on the hard first-audio branch. A wrapper around F5-TTS is a new streaming system, not a free latency optimization, and must be engineered and qualified separately. Until then, F5-TTS is not a contender for that branch. It remains a candidate for offline or batch generation, and even there only after measured throughput and common-stack quality gates pass.

Qualification belongs at the acoustic endpoint, not an internal timing event. Use one clock domain, or synchronized clocks, to mark input-ready and the first audible sample at an output-device loopback. Preserve the production concurrency, request mix, batch size, codec, network path, accelerator configuration, and enabled optimizations. The acceptance statistic is p95 time-to-first-audible-audio; the table’s stated pass condition is non-negotiable. Model-token time omits work around synthesis; HTTP response time can precede playable sound; and an uncapped single-request run removes the queueing behavior the SLA is meant to expose. An F5-TTS real-time-factor result, however favorable, cannot substitute because it answers a different question.

Before the run, predeclare timeout and failure handling; silently dropping them changes the denominator and can manufacture a pass. Preserve each loopback trace beside queue depth, batch occupancy, and accelerator utilization. Those fields distinguish a codec-bound miss from queueing or underprovisioning without granting a waiver. The SLA remains unchanged even when diagnostics explain the miss.

Quality cannot convert a failed latency gate into a pass. According to Lovora’s direct CosyVoice2 app test, round-trip WER was 71% for German, 50% for French, 47% for Spanish, and 23% for Italian. That account does not test F5-TTS on the same stack, so it cannot settle matched voice quality; it does show why favorable quality scores cannot substitute for a common production test. If measured p95 exceeds the table’s limit, label the deployment no-go. Do not average CosyVoice 2’s best reported streaming figure against F5-TTS’s real-time-factor result; those quantities do not measure the same event.

The next action is binary: run the same-stack, peak-load, output-device loopback test. If CosyVoice 2 clears the stated limit and the launch’s quality requirements, select it for the stated brief; use F5-TTS only when early audio is unnecessary and its measured throughput and quality gates also pass. If the latency gate fails, neither model’s published headline establishes production readiness.

What the Data Doesn't Tell You

A model timestamp is not yet a user-experience measurement. Three events are often collapsed into one “latency” number: when the model emits a packet, when a playable prefix becomes available, and when sound becomes audible. Codec processing and jitter buffering sit between those events, while aggregate quality scores may conceal variation across voices, references, and recording conditions. I would treat every figure below as conditional evidence—not as production proof. The numbers are diagnostic arithmetic and test-design examples, not reported CosyVoice 2 or F5-TTS benchmark results.

Worked 8-Second Reply

For this worked case, the defensible action is to qualify CosyVoice 2, not declare it production-ready. Assume an interactive English assistant uses an approved speaker prompt to return an 8-second response. The production requirement is p95 time-to-first-audible-audio at the device loopback of ≤150 ms under the same serving stack at peak load. The 8-second duration is a scenario assumption for this exercise, not a result reported by either paper.

According to Zhao et al.’s CosyVoice 2 report, the best-case streaming latency equals the entire SLA. Even interpreted as a first-packet measurement, that value leaves no allowance for network transit, buffering, or device playback margin. It also answers a different question from the launch gate: first-packet latency is neither p95 audible latency nor a production concurrency result. CosyVoice 2 therefore advances as an unverified streaming candidate, not as an SLA-compliant system.

According to Yusheng Chen et al.’s F5-TTS paper, the reported H100 RTF produces substantially different arithmetic: multiplying the RTF by the 8-second case estimates average generation work, not time to first audio. Native F5-TTS completes a full mel trajectory, whereas this early-audio case requires a usable prefix before the complete reply has been synthesized. F5-TTS is therefore structurally unsuitable for this particular workload. The two published results must not be placed on the same stopwatch: one is a best-case streaming observation; the other is an average-throughput calculation.

Before approval, the same 8-second case must pass matched WER, speaker-similarity, and blind-listening criteria. “Matched” means the same input text, approved speaker prompt, serving stack, and measurement setup, with acceptance rules fixed before outputs are inspected. This is not cosmetic: a latency pass cannot compensate for mispronunciation, identity drift, or an unnatural prosodic contour. Conversely, strong perceptual scores cannot compensate for late first audio.

The paper-grounded decision is to select CosyVoice 2 for production qualification while approving neither model from published metrics alone. Deploy CosyVoice 2 only if the matched same-stack, peak-load test passes the p95 first-audible gate defined above, together with every quality criterion. If it misses, declare the stated requirement unmet; F5-TTS’s RTF is not a fallback for this early-audio workload. This rejects both halves of the bad inference: a favorable RTF is not conversational first audio, and CosyVoice 2’s best-case streaming result is not automatic production clearance.

How to Choose Well: Five Gates for Production Launch

A production launch has only two defensible outputs: a candidate that passes every applicable gate, or a no-go. A favorable real-time factor is not conversational first-audio evidence, and a model headline is not a production qualification. In the strict-latency branch, CosyVoice 2 is a candidate—not an approval—and native F5-TTS is not a substitute for an early-audio path.

Start with identity authority. If a speaker has not granted documented, scoped, revocable permission for synthetic-speech use, deploy neither model, regardless of latency or similarity. Then route by product behavior: when audio must begin before the final response chunk is rendered, use the native streaming branch. Exclude F5-TTS unless a separately qualified wrapper proves that playable PCM exists during generation; packet emission alone is insufficient. If early audio is unnecessary, use the non-streaming branch.

A deployment does not close those gates. According to Lovora on September 27, 2026, Lovora deployed the CosyVoice2-0.5B checkpoint for an adult AI voice-note and voice-call product in early September 2026. That establishes use, not evidence that first audio met an SLA. For CosyVoice 2, test at least 10,000 requests at peak load in the intended same stack, measuring from input-ready to device loopback. Require p95 time-to-first-audible-audio ≤150 ms and characterize p99 playback latency. If the gate is missed, neither model is approved for the strict 150 ms SLA.

In the non-streaming branch, F5-TTS earns consideration only through measured service time. At peak load, with the production batch and serving configuration unchanged, require p95 wall-clock generation time ≤0.25 × utterance duration. Otherwise reject it for that branch. This condition can establish generation efficiency; it cannot be relabeled as first-audio latency.

Quality is the final veto, not a tie-breaker. Evaluate held-out target-domain utterances provisionally against the incumbent: WER may be no more than 0.20 percentage points higher, speaker similarity no more than 0.02 lower, and no accent or noise slice may exceed 2× the overall error rate. The available absolute similarity figures cannot decide this test: according to the DEV Community guide and Apatero, respectively, the supplied secondary sources report 77.4% and 78.0% for CosyVoice 3.0, not CosyVoice 2 or F5-TTS. Once the applicable serving gate has passed, a quality pass selects CosyVoice 2 in the hard-latency branch or F5-TTS in the non-streaming branch; any failure is a no-go.

What to do next