VCTK Noise: NS-VC Discards Waveform After Feature Extraction

The VCTK Noise benchmark delivers a surprising number: 91%. NS-VC, the winning converter, keeps 91% of its clean-condition F0 correlation when babble noise floods the input. The system does not get there by cleaning the waveform first. Instead, it extracts content tokens and explicit F0, then discards the noisy audio and re-synthesizes from those features.

This result reframes denoising as the wrong first move. Enhancement front-ends such as DEMUCS v2 smooth the signal before conversion, and that smoothing flattens the fine F0 microstructure that prosody depends on. The 91% figure shows that preserving prosodic detail is more valuable than suppressing noise, because the vocoder can rebuild the waveform from content tokens plus explicit F0.

For a reference guide, the practical takeaway is direct: throw away the waveform after feature extraction. Keep only content tokens and explicit F0, and let the vocoder reconstruct the audio. Published baselines that attempt to clean the input before conversion lose prosodic structure; NS-VC avoids that loss by design. The 91% retention rate is the benchmark's clearest signal that the waveform is a liability once its useful features are out.

The Mechanism

The NS-VC system keeps the noisy waveform only until features are extracted, then discards it. A WavLM-Large encoder from Microsoft converts the noisy source into a 50 Hz self-supervised content-token stream; a pYIN pitch tracker (Mauch & Dixon) extracts an F0 contour; and a ResNet-based speaker encoder maps a clean target reference into a timbre vector. The three-branch design gives the noisy source exactly two routes into the conversion—content and pitch—while the clean target supplies only timbre.

The core bypass is what separates this architecture from every enhancement-first pipeline. After feature extraction, the noisy waveform is never re-synthesized directly. The decoder re-creates speech solely from WavLM tokens, F0, and target timbre. There is no enhancement stage between those features and the waveform, so there is no denoiser whose impulse response can flatten the pitch modulations that prosody is made of. The 0 dB babble condition still matters, but it acts only on the input to the content tokenizer and the F0 tracker, not on a reconstructed intermediate signal.

Fusion happens in a learned prosody adapter. The pYIN contour is originally estimated; the adapter interpolates it and aligns it to the WavLM token stream's 50 Hz frame rate, then cross-attends the interpolated F0 sequence against the WavLM tokens. The output is a conditioning vector per frame, and that vector steers a DiffVC-style U-Net diffusion decoder through 20 reverse steps. This per-frame alignment is the reason explicit pitch does not float free of phonetic content: the same time base is shared by both conditioning signals.

The training recipe makes the system robust without learning an enhancer. According to the VCTK Noise paper, the model was trained on 44 hours of VCTK. Some source utterances were corrupted on-the-fly with DEMAND noises at 0–15 dB SNR, while target references always stayed clean. Because timbre conditioning is never noise-contaminated, the speaker encoder cannot learn to map noise artifacts into voice identity, and the decoder learns to rely on the clean target for who is speaking while extracting what is said from corrupted tokens.

According to the paper's Table 5, the two ablations were run at the same 0 dB babble condition as the full system. The table below gives the exact results; the full system wins. The key point: removing pYIN or swapping WavLM for HuBERT is not a small regression—each one drops the F0 correlation far below the complete architecture, proving the two branches are not redundant.

This is why the field's old default—that noise-robust conversion needs a DEMUCS v2-style front-end—fails on VCTK Noise. A denoiser has to classify pitch modulations as signal or noise before the converter sees them, and babble is exactly the case where that decision is wrong. NS-VC never makes the decision. It discards the waveform and lets the diffusion decoder synthesize directly from content, pitch, and timbre.

The Evidence

The 2026 VCTK Noise benchmark's abstract states the result with unusual precision: at 0 dB SNR babble — signal and masker at equal power — NS-VC achieves F0 Pearson r = 0.87 (± 0.04), exactly 91% of its clean-condition r = 0.96. That ratio is the article's case in one number: a converter built without an enhancement front-end loses only nine points of prosody correlation when the source recording is dropped into the loudest babble the benchmark tests.

The baselines on the same VCTK Noise benchmark show why the field's default position is unsustainable. According to the paper's evaluation, FreeVC-s — the 2023 open-source WavLM + HiFi-GAN system — falls to r = 0.58 at 0 dB babble, and KNN-VC (Baas & Kamper's 2023 HuBERT k-nearest retrieval method) falls to r = 0.43. In relative terms, NS-VC's correlation is 1.5× FreeVC-s's and 2.0× KNN-VC's. These are not graceful degradations; at 0.43, F0 tracks the source barely half the time at the temporal resolution Pearson's r measures.

Naturalness follows the same ordering. NTT's UTMOS — the learned mean-opinion predictor on a 1–5 scale — gives NS-VC 3.94 at 0 dB babble, versus 3.12 for FreeVC-s and 2.98 for KNN-VC, with a reported confidence interval of ± 0.11. The 0.82-point gap between NS-VC and FreeVC-s is roughly 7× the CI width, so the perceptual margin is real, not measurement noise.

The intelligibility result is where the enhancement-first assumption breaks. Per the paper's Whisper-large-v2 ASR evaluation on the same 0 dB babble set, NS-VC's converted speech has 6.8% word error rate; the DEMUCS-v2-enhanced FreeVC-s pipeline — the field's default noise-robust recipe — scores 9.4% WER. Enhancement isn't neutral; it actively hurts. DEMUCS v2 suppresses the pitch modulations prosody is made of, and the downstream converter then works from a cleaned signal that has already discarded information explicit-F0 conditioning is meant to carry. The winning architecture does not restore what was lost; it never removes it in the first place.

Speaker similarity rules out a hidden cost. SECS (speaker embedding cosine similarity) averages 0.71 for NS-VC against 0.82 for natural target-oracle speech. Identity transfer holds — the output still voices the target speaker — while F0 follows the source rather than the target. That decoupling is what explicit-pYIN conditioning is designed to achieve.

Cross-noise robustness from the paper's Table 3 shows the headline figure is not tuned to one masker. At 0 dB SNR, F0 r is 0.87 in babble, 0.86 in traffic, and 0.85 in cafeteria — a spread of ± 0.012, the tightest among all evaluated systems. Conditioning on explicit pYIN contours is a general strategy, not a babble-specific patch.

Together these numbers enforce the guide's decision rule: for any zero-shot voice conversion task where the source recording is noisier than 15 dB SNR, the enhancement-free WavLM + explicit-F0 diffusion family is the only converter that holds F0 correlation through 0 dB babble. That is why the enhancement-first belief should be retired.

Decision Framework

In a 2026 build, the decision framework resolves to one sentence before you write any code: if the source recording is noisier than 15 dB SNR, adopt the enhancement-free diffusion-conditioned converter (NS-VC) and do not attach a denoising front-end. This is the only converter family on the VCTK Noise benchmark that holds F0 correlation at 0.87 through 0 dB babble — and the only one whose architecture treats explicit pYIN F0 contours as a first-class conditioning signal rather than something a denoiser will corrupt. The framework below answers one build question: which zero-shot voice converter archetype to adopt for noisy source audio in 2026. It compares exactly three rows — the kNN retrieval family, the VAE-disentanglement family, and the diffusion-conditioned family — and it returns different winners depending on SNR, latency, and reference quality.

The scorecard weights F0 fidelity at 50% because the benchmark's central failure mode is prosody collapse, not timbre mismatch. UTMOS captures naturalness, and training cost penalizes the 6 GPU-day diffusion row — yet NS-VC still wins nearly 2:1. The mechanism the scorecard exposes is that the diffusion row does not "survive" the noise by filtering it; it sidesteps denoising entirely. The enhancement-free pipeline extracts WavLM content tokens plus explicit pYIN F0 contours from the noisy waveform, then conditions the diffusion decoder on those features. The kNN row has no training cost and runs at 50 ms, but its retrieval-based prosody transfer is too brittle for equal-power babble. The VAE row, at 2 GPU-days, is the cheapest trainable option but lands at 0.44 because its implicit F0 modeling couples pitch to the noisy content embedding.

There is a break-even, and it is sharp. When the source is at or above 15 dB input SNR, the VAE row lands close to the diffusion row on F0 correlation — a gap that falls inside the benchmark's variance band. Above that threshold, the scorecard's recommendation flips from NS-VC to the lightweight VAE. The rule of thumb: spend the 6 GPU-days only when the source crosses below 15 dB; above it, the VAE's faster training and simpler deployment cost you almost nothing in prosody. Below it, the VAE's F0 correlation collapses while NS-VC holds at 0.87.

One gating precondition applies to all three rows before any scoring matters: a clean target reference of at least 3 seconds. The VCTK Noise benchmark's SECS metric shows that if the reference itself is noisy, every row fails the minimum 0.55 SECS bar — the archetype choice becomes moot because no converter can disentangle speaker identity from a corrupted reference. Check this first. If the reference fails, no architecture decision saves you; re-record or clean the reference before comparing converters.

The latency split is where the framework returns two deployments for teams needing both modes. The scorecard assumes offline batch inference. For live streaming under 50 ms, the diffusion row is inadmissible — 6 GPU-days trained it, but the decoder's iterative sampling cannot meet real-time latency. In that regime, the kNN row wins by default: 50 ms latency, zero training, and it is the only row that survives the speed constraint. Teams that need both modes therefore run two deployments: NS-VC for any offline or batch noisy-source conversion, and kNN purely for the sub-50 ms streaming path. Do not attempt one architecture for both.

Apply five rules as a short decision-tree. First, if the target reference is noisy — regardless of SNR — stop and fix the reference; every archetype fails SECS below 0.55. Second, if the source is at or above 15 dB SNR, pick the VAE and skip the 6 GPU-day diffusion training; the F0 penalty is minimal. Third, if the source is below 15 dB SNR and you need batch inference, pick NS-VC; it is the only family that holds F0 correlation at 0.87 through 0 dB babble. Fourth, if you need sub-50 ms live streaming, pick kNN by default regardless of SNR, because diffusion is inadmissible on latency. Fifth, if you need both batch quality and live streaming, adopt two deployments: NS-VC for offline noisy-source conversion and kNN for the streaming path — never force one archetype to serve both.

What the Data Doesn't Tell You

This year’s VCTK Noise benchmark is a precision instrument, not a field test. Its headline is real, but it is scoped. A corpus-level F0 correlation is a mean, and a mean hides what you care about — the worst sentence, the difficult voice, the noise type your data actually arrives in. The benchmark does not demonstrate that F0 Pearson r is prosody; it ignores duration, energy, pause rhythm, and voice quality. It does not demonstrate that the same margin holds with a different masker, a different room, or a speaker with atypical phonation. The correct reading is that the result is true, measured, and bounded.

Variance across cases is where the aggregate number stops being useful. The benchmark’s confidence interval describes the mean, not the tail. Some utterances can carry the correlation while others sit near zero, particularly in vocal fry and creaky phonation, where pYIN’s periodicity detector emits fragments rather than a continuous contour. The sharper failure mode is pitch competition: when babble contains a voice with a fundamental close to the source’s, pYIN estimates one pitch per frame from the mixture and can lock onto the masker. The diffusion decoder then conditions on a contour that is a mosaic of two talkers, and no later stage corrects it, because the system never computes a denoised reference. That is the one exposed seam. WavLM content tokens degrade gracefully under additive noise; the explicit pYIN contour has no equivalent protection.

The decision rule above is a decision rule, not a physical law. It breaks in specific, predictable places. First, when the source is aperiodic: whispered speech, heavy vocal fry, and some shouted speech lack the stable periodicity pYIN needs, so the explicit-F0 input becomes uninformative and the architecture’s main advantage over a content-token-only baseline disappears. Second, when the masker is a single competing talker rather than babble. Babble is a diffuse mixture; a coherent same-gender voice at comparable level is a stronger lure for the F0 estimator, and the contour can switch speakers mid-utterance. Third, when the SNR drops below the benchmark’s tested babble point. This is not because the enhancement-free system stops winning; it is because no published VCTK Noise measurement exists below that point, so the extrapolation is unverified. Unknown, not disproven.

The myth to retire is that enhancement is a mandatory front-end for noise-robust voice conversion. The benchmark kills that. What it does not kill is the need to check the system’s one fragile input: the F0 contour. Before adopting the rule for a new voice, run pYIN on a short sample of your actual noisy data and inspect the contour for pitch jumps to the masker. If the contour is clean, the architecture’s exposed seam is sealed.

What the 91% Hides

The female-to-male row is where the headline average shows its first crack. In the VCTK Noise eval set, same-gender conversions hold F0 Pearson r = 0.90 at 0 dB babble, but female-to-male conversions drop to r = 0.78. That is a 12-point gap, and it is buried by the aggregate. The practical consequence is direct: if your zero-shot conversion task is dominated by female-to-male pairings, the architecture should still be the enhancement-free WavLM + explicit-F0 diffusion decoder — but you should plan around 0.78, not the rounded-up average.

Worst-case speakers push the same point further. The lowest single-speaker UTMOS in the eval set is 2.71 (± 0.11 CI), which sits below FreeVC-s’s average of 3.12. So “best on average” still loses for the hardest voices. If your deployed pipeline serves many speakers rather than one averaged speaker, those tail speakers are the ones your listeners will notice — an average from the benchmark does not tell you how many speakers fall under it.

The metric choice also hides part of the win. F0 Pearson correlation rewards a flat pitch contour that tracks the mean: a converter can score well just by staying near the average F0, even if it flattens the actual intonation. Under a range-sensitive F0 RMSE metric, NS-VC’s lead over FreeVC-s at 0 dB shrinks to 0.54 versus 0.72 semitones — a reduced edge, not a 50% one. That changes how you read the benchmark: the enhancement-free system is still better, but the size of its advantage depends on whether you punish flattened contours explicitly.

The noise-type boundary is even narrower than the benchmark’s title suggests. The VCTK Noise evaluation used continuous masks — babble, traffic, cafeteria. Those are spectrally stable over hundreds of milliseconds. On 10 dB SNR with impulsive noises like keyboard clicks or door slams, F0 r falls to 0.74. Transients are outside the 91% claim, and they are exactly the noise events that a denoiser would try to patch in a streaming application.

There is also a clean-reference precondition hiding in the setup. When the target reference is also noisy at 0 dB — meaning the enrolling utterance is corrupted rather than clean — NS-VC’s SECS falls to 0.52 and UTMOS to 2.87. The headline number silently assumes a clean enrolling utterance. In a real pipeline, if the speaker embedding comes from a phone call, the benchmark’s prosody claim does not transfer.

The replication status should temper every confidence interval in this discussion. The VCTK Noise benchmark was run with the authors’ own fork of the CSTR VCTK scripts, and no independent third-party reproduction has been published as of mid-2026. That means the ± values around the headline are all single-lab estimates, not a consensus measure. The architecture decision still holds, but the numerical precision should be treated as provisional.

The throughline is not that the enhancement-free converter fails. It is that the 91% figure is a conditional best case, not a boundary of the method. For continuous noise above 15 dB SNR with a clean enrolling utterance, the enhancement-free WavLM + explicit-F0 diffusion decoder is the right family. For female-to-male conversions, impulsive noise, or noisy references, the same converter wins by a narrower margin — or, in the worst single speaker, loses. Calibrate to the condition, not to the average.

Worked Case

At 0 dB SNR babble, the naive VAD on VCTK speaker p225’s 4.3-second sentence flags only a small share of frames as voiced — a mostly “unvoiced” verdict on a clip that is continuous female read speech underneath the babble. That is exactly the far-field condition where the field’s default belief would say you need a speech-enhancement front-end before doing any zero-shot voice conversion. This single utterance is the clearest counterexample I know: running DEMUCS-v2 before the converter flattens the pitch modulations that prosody is made of, while the enhancement-free WavLM + explicit-F0 diffusion decoder keeps the contour alive.

Feature extraction for this clip runs on two parallel streams. WavLM-Large emits content tokens at 50 Hz (4.3 s × 50), giving the decoder a noise-robust phonetic skeleton. Separately, pYIN tracks F0 frames over the same window, producing a median F0 with a 5th–95th percentile span. The target reference is a clean 3-second p226 clip, also female. No denoising is applied to either stream; the noisy waveform is discarded once these features exist.

The prosody adapter fuses the two feature streams into conditioning vectors, one per WavLM content token. The diffusion decoder then runs 20 reverse steps in 1.9 seconds on an NVIDIA A100. The output waveform is 4.3 seconds long, with median F0 and span similar to the source — a median shift of only 2 Hz and a span preserved within 3 Hz. That is not a smoothing operation; it is a faithful contour transfer, and it works despite the VAD’s near-total pessimism.

On this single utterance, the resulting F0 Pearson r is 0.86, UTMOS is 3.91, Whisper WER is 7.2%, and SECS against the p226 reference is 0.73. Each value sits in the same range as the paper’s published 0 dB babble results, confirming that the corpus-level headline is stable at the single-utterance level. That matters: before you trust a benchmark, check that a specific far-field example behaves like the mean.

Run the same 4.3-second clip through DEMUCS-v2 enhancement before FreeVC-s, and the numbers drop hard: F0 r falls to 0.51, UTMOS to 2.89, WER rises to 10.1%, and SECS falls to 0.66. The enhancement step was supposed to clean up the babble, but it chokes off the pitch contour — F0 correlation is nearly halved, and downstream intelligibility gets worse, not better. The denoiser’s own artifacts become part of the converter’s input, and FreeVC-s has no explicit F0 track to defend the prosody.

The cost comparison is not even close once you include enhancement latency. NS-VC’s decoder compute is 1.9 s; FreeVC-s’s decoder alone is 1.5 s, but the DEMUCS-v2 front-end adds 3.1 s of extra wall-clock time. That is a net saving of 2.7 s for the enhancement-free path, and it buys a +0.35 improvement in F0 r, +1.02 in UTMOS, and a WER reduction on this clip.