AI Tech Inspire

noisekit: Turn Clean Speech Into Realistic Phone Noise for ASR Benchmarks

Most speech-to-text demos sound great—until they meet a crackly G.711 phone line, a busy call center, and a mic pointed half a room away. That gap between clean benchmarks and messy reality is where accuracy promises go to die. At AI Tech Inspire, we spotted a small CLI that tries to turn that problem into an engineering exercise you can actually run: noisekit, a tool that converts clean annotated speech into realistic degraded audio for apples-to-apples ASR benchmarking.

Why clean benchmarks fail real-world ASR

If you’ve ever tried to pick an STT vendor for a voice agent or support line, you’ve likely run into three blockers:

  • Your production audio is unlabeled, so you can’t compute WER on it without costly annotation and privacy headaches.
  • Public datasets like FLEURS, Common Voice, and LibriSpeech are relatively clean and studio-like—great for research, not much like phone calls.
  • Teams often benchmark on clean corpora, choose a vendor, then discover in production which one actually survives noise, compression, and room reverb.

The result: big swings in quality once you ship. The practical fix is obvious but annoying—create a noisy, still-labeled test set that mirrors your environment. That’s the gap noisekit aims to fill.

What noisekit does

noisekit is a CLI that takes a clean, labeled speech dataset and applies degradations that approximate real production conditions. You end up with a noisy, still-annotated corpus you can run through any STT system and evaluate with WER. The output layout is compatible with Hugging Face’s AudioFolder, so you can drop it directly into a benchmarking pipeline.

“Benchmark on what you actually ship into—not what you wish you shipped into.”

Here’s the gist:

uvx noisekit generate \
  --dataset google/fleurs \
  --config en_us \
  --split test \
  --samples 100 \
  --output ./noisy-fleurs

Feed ./noisy-fleurs to your STT candidates, normalize transcripts, compute WER, and compare. Each sample also gets PESQ, SNR, and NISQA scores in a metadata.jsonl, so you can correlate model failures with measurable signal quality.

Degradations that actually matter (and stack)

The presets read like a checklist of things that clobber call quality in production. You can run them individually or compose them into realistic chains:

  • telecom: Narrowband bandpass + 8-bit bitcrush + 16–32 kbps MP3. This approximates a real phone path instead of a simplistic low-pass filter.
  • noise: Real ambient noise at 5–15 dB SNR. Pulls from a MUSAN noise-only subset automatically, or point to your own --noise-dir for domain-specific sounds (call center, cafe, car, street).
  • reverb: Far-field room response (1–3 m mic distance) via pyroomacoustics—useful if your users aren’t speaking directly into the mic.
  • low_bitrate: Wideband MP3 at 16–32 kbps—handy for streaming over constrained links.
  • clipping: Simulated ADC/mic saturation for those loud talkers and over-eager gain settings.
  • clean_reference: A control track to establish your baseline WER floor.

Crucially, compound chains stack in the same order reality would. For example, noise_telecom applies room noise, then phone codec. That ordering matters: noise pre-compression interacts differently than noise injected post-codec.

Plugging into a practical benchmarking workflow

The design goal is to shrink the distance between “we should test this” and “we have a ranked leaderboard.” A minimal pipeline looks like this:

  1. Use noisekit to generate a noisy subset from a clean labeled corpus (e.g., FLEURS test set).
  2. Iterate over your STT candidates (managed APIs and open-source models in PyTorch or other frameworks), feed each audio file, and collect raw transcripts.
  3. Apply the same text normalization across outputs (punctuation, casing, numbers, abbreviations).
  4. Compute WER and optionally CER per sample and aggregate by degradation type.
  5. Join results with the provided PESQ, SNR, and NISQA from metadata.jsonl for correlation plots and thresholds.

Because the output is AudioFolder-compatible, loading is straightforward:

from datasets import load_dataset

ds = load_dataset("audiofolder", data_dir="./noisy-fleurs")

From there, it’s just orchestration—batch to vendor APIs, or run local decoders on GPU, and log everything to the same experiment tracker.

Why this matters to teams shipping voice

  • Vendor selection with fewer surprises: A lot of models claim parity on clean speech; they diverge sharply under telecom + noise + reverb.
  • Privacy-friendly iteration: Avoid moving real user audio around just to evaluate WER during the RFP phase.
  • Targeted hardening: Use per-scenario WER (e.g., noise_telecom at 10 dB SNR) to guide prompt engineering, VAD tuning, or voice UX changes.
  • Observability hooks: Correlating WER with PESQ/NISQA lets on-call teams define SLOs tied to signal quality thresholds, not just black-box failures.

What’s included under the hood

  • MIT license and a GitHub repo for transparency and modification.
  • Zero-install via uvx: Run the CLI without setting up a dedicated environment.
  • Metadata-first: Each file ships with computed quality metrics and the original transcript—no extra bookkeeping to align audio and text.

What you might still want to simulate

The current presets cover the common killers, but production systems often face a few more gremlins. If you’re benchmarking seriously, consider whether your scenario includes:

  • Network artifacts: Packet loss, jitter, PLC artifacts typical of VoIP.
  • Automatic gain control: AGC pumping, dynamic range compression, and fast attack/slow release behavior.
  • Echo cancellation residue: Far-end echo remnants and double-talk moments.
  • DTMF and background speech: Overlapping talkers and keypad tones during IVR navigation.
  • Device idiosyncrasies: Bluetooth headset codecs (SBC), cheap laptop mics, handset frequency response curves.
  • Truncation and VAD errors: Cut-ins/cut-outs when VAD is too aggressive, common in streaming agents.

Even a handful of these can flip the leaderboard compared to clean conditions.

Metrics: beyond just WER

WER is table stakes, but also watch:

  • CER for languages where character-level errors reveal robustness gaps.
  • Latency and stability in streaming—especially for barge-in and turn-taking in voice agents.
  • Entity error rate when downstream tasks care about names, addresses, or IDs more than filler words.

With PESQ, SNR, and NISQA already captured, you can tie your operational metrics (e.g., agent deflection rate) to concrete signal thresholds. That helps justify investments in acoustic treatment, device recommendations, or codec upgrades.

Try it in 10 minutes

If you’ve got a few hundred clean clips with transcripts, you can produce a useful noisy benchmark quickly:

  1. Generate data with a relevant preset chain:
uvx noisekit generate \
  --dataset google/fleurs \
  --config en_us \
  --split test \
  --samples 200 \
  --output ./noisy-fleurs \
  --preset noise_telecom
  1. Run three STT systems (two managed, one OSS) over the folder and normalize text.
  2. Compute WER/CER by preset and correlate with NISQA. Plot WER vs. SNR and look for crossover points.
  3. Repeat with a different SNR range or add reverb to see what breaks.

A simple tweak—like moving from 10 dB to 5 dB SNR—often reshuffles winners dramatically. This is the sort of due diligence that prevents fire drills after go-live.

Caveats to keep it honest

  • Synthetic isn’t identical to real: Degradations are approximations. If you can, validate a slice on real (properly consented) traffic.
  • Domain noise matters: Bring your own --noise-dir with realistic call-center or in-car sounds to avoid overfitting to MUSAN.
  • Normalization is everything: Make sure every vendor output goes through the same text cleanup. Inconsistent pipelines can swing WER by multiple points.

The bottom line

No single CLI will solve ASR benchmarking, but noisekit does something refreshingly practical: it turns clean, widely available corpora into domain-shaped testbeds you can use today. For teams stuck choosing between shiny demo accuracy and crackly reality, this offers a fast, reproducible middle path.

Curious what degradation conditions your stack has seen that aren’t in the presets yet? Packet loss models? Handset-specific EQs? At AI Tech Inspire, the most interesting stories we hear come from those edge cases—because they’re where tools like this earn their keep.

Recommended Resources

As an Amazon Associate, I earn from qualifying purchases.