NVIDIA Nemotron 3.5 Transcribe

NVIDIA's open-weight speech LLM: transcription in 40 languages, speakers, word timestamps, context terms and translation in 1 model. Compared with Whisper large-v3 on my Mac Studio M1 Ultra.

28 Oct 2026

Nemotron 3.5 Transcribe is an open-weight speech model from NVIDIA that turns a recording of up to 1 hour into text, in 40 languages. It isn't a classic speech recogniser. It's a speech LLM: an audio encoder feeds a 30B language model, and a text instruction picks the output. The same model writes a verbatim transcript, a clean written one, a transcript with speaker tags, a translation, or an answer to a question about the recording.

NVIDIA released it on 28th October 2026. At launch it has the best published scores of any open-weight model on the English, long-form and multilingual Open ASR leaderboards, and the best published multi-speaker result.

NVIDIA's overview of Nemotron 3.5 Transcribe: 1 audio clip and 6 instructions (verbatim, written form, context, speaker-attributed, rich, translation) produce 6 versions of the same Oktoberfest sentence, including a German translation and word timestamps

Maker NVIDIA
Size 34B parameters in total, about 4.3B active per token
Input 16 kHz mono audio (.wav, .flac, .mp3, .ogg), up to 1 hour, plus a text instruction
Output Text, with optional word timestamps and speaker segments
Languages 40: 13 rated "leading", including English, German and French
Context 65,536 tokens
Precision BF16, FP8 and NVFP4
Runs on NVIDIA GPUs: 1 H100 80 GB in BF16, an L40S in FP8, an RTX 5090 in NVFP4
Licence OpenMDW 1.1, commercial use allowed

What it does

The instruction in the request picks the task. No retraining, no separate model per job.

Task What you ask for Good for
Verbatim Every word, with fillers and repetitions Research, legal review
Written form Fillers and accidental repetitions removed Captions, notes, search
Context terms A list of names and jargon in the prompt Earnings calls, client calls, technical talks
Speaker-attributed Speaker tags <spk:0> to <spk:7>, plus word timestamps Meeting minutes, editing
Rich Events and styles such as <ev:laughter> Subtitles, media archives
Translation Speech to text in another language, both ways with English Localisation
Questions A question about the recording, with reasoning on "What was decided?"
Language ID The language spoken Routing

2 details make the speaker and timestamp output usable:

  • Word timestamps come from a small CTC head that aligns the finished transcript to the audio. Each word gets a start and end time and its speaker tag.
  • Speakers come from the built-in Nemotron 3 Diarization model (see NVIDIA Nemotron 3 Diarization), so the response also carries the raw speaker segments and the mapping between tags and segments.

How it works

Nemotron 3.5 Transcribe architecture: 16 kHz audio becomes log-mel features, a 32-layer ASR encoder and the frozen Sortformer diarizer combine into a projection feeding the Nemotron 3.5 Lightning 30B-A3B language model, with an MTP module for speculative decoding and CTC heads for word timestamps

  1. Audio encoder. A 600M-parameter Transformer turns 16 kHz audio into 1 token per 80 ms, in independent 30-second chunks.
  2. Speaker encoder. The 99M-parameter Nemotron 3 Diarization model, frozen, adds who-is-speaking information to the same features.
  3. Language model. Nemotron 3.5 Lightning 30B-A3B, a hybrid of Mamba-2, attention and mixture-of-experts layers (128 experts, 6 active), reads the audio tokens and the instruction and writes the text. Only about 3.5B of its parameters work on each token, which is the "A4B" in the name once the encoders are added.
  4. Speed. Built-in multi-token prediction drafts up to 8 tokens at a time. An optional draft model, DFlash2, goes further.
  5. Timestamps. The CTC head aligns the generated words to the audio after decoding.

NVIDIA trained it on about 1.72 million hours of unique audio, mostly from its own Granary v2 dataset, plus meetings, phone calls, earnings calls and synthetic multi-speaker mixes of up to 8 voices.

Benchmarks, as reported by NVIDIA

Results from 12th October 2026. Lower is better.

Test Nemotron 3.5 Transcribe Next best open-weight
English, Open ASR Leaderboard, mean WER of 9 sets 4.71 Qwen3-ASR-1.7B: 4.94
Long-form English, mean WER 10.98 Parakeet TDT 0.6B v3: 13.37
Multilingual leaderboard, 7 languages, mean WER 4.62 Qwen3-ASR-1.7B: 6.49
Multi-speaker meetings and calls, mean cpWER 15.80 DiCoW v3.3 + DiariZen: 16.83 (any model)

The numbers that matter for my languages:

Language FLEURS WER Common Voice WER Speech translation to English, BLEU
German 2.73 2.99 39.96
French 3.16 5.27 38.19
English 3.35 - -

Context terms work. On earnings calls, giving the model the call's keywords lifts keyword recall from 58% to 96%. On ContextASR-Bench, a fine-grained term list cuts the error rate on named entities from 12.2% to 2.2%. For client calls full of product and company names, this is the feature.

Distant microphones hurt. In the AMI meeting corpus, the speaker-attributed error rate is 10.3% on headset microphones and 17.3% on a single room microphone. On NOTSOFAR-1 it climbs to 25.9% on a single distant microphone.

How to run it

On an NVIDIA GPU

NVIDIA's reference path is vLLM 0.32.0 with the NeMo Speech plugin and an OpenAI-compatible API. The full vllm serve command, with the timestamp middleware and speculative decoding, is in the model card.

The shortest route is SGLang-Omni 0.2.0, which needs no NeMo but has no timestamps, speakers or translation:

sgl-omni serve --model-path nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B \
  --model-name nemotron-transcribe --mem-fraction-static 0.85 --port 8000

curl http://localhost:8000/v1/audio/transcriptions \
  -F model=nemotron-transcribe -F file=@audio.wav

The prompt picks the style. A transcript with context terms:

Transcribe with punctuation and capitalization. Context terms: Kaltura, REACH, Theresienwiese.

Keep the system prompt empty, temperature 0 and reasoning off for transcription. Turn reasoning on, at temperature 0.6, only for questions about the audio.

Speed: NVIDIA's NIM container on 1 RTX PRO 6000 Blackwell transcribes up to 2,238 seconds of audio per second, and returns a 30-second clip in 0.13 seconds. Those optimisations aren't open source yet. NVIDIA expects the open vLLM path to be about 2 times slower.

On a Mac

NVIDIA supports Linux and Windows on NVIDIA GPUs only. On launch day, the community published 2 Apple Silicon ports:

Port What it is Size Trade-off
nativ-community MLX The full model for mlx-audio, in 4, 5, 6 and 8 bits 19.3 GB (4-bit), 35.1 GB (8-bit) All tasks, but speaker tags without timestamps
FluidInference Lite Only the audio encoder and CTC head, in Core ML 0.7 to 1.4 GB About 500 times real time on an M5 Pro, but a much rougher transcript without the language model

The MLX port was checked on an M3 Ultra with 6 short smoke tests, not a benchmark. Usage, with mlx-audio pinned to the validated commit from the model card:

from mlx_audio.stt import load

model = load("nativ-community/Nemotron-3.5-Transcribe-30B-A4B-EA-8bit", strict=True)
print(model.generate("call.wav").text)
print(model.generate("call.wav", task="translate", language="German").text)
print(model.generate("meeting.wav", task="saa").text)                      # speaker-attributed
print(model.generate("meeting.wav", task="qa", prompt="What was decided?").text)

Limits

  • Offline only. It processes whole recordings. The 30-second chunks are internal and don't make it a live captioning model.
  • 1 hour per request. NVIDIA doesn't promise anything beyond the training limit.
  • Speaker tags aren't people. <spk:1> is a voice in this recording, not a name. NVIDIA warns that tags can be wrong, and the MLX port notes that speaker attribution can add words absent from the audio.
  • It can loop. In rare cases generation doesn't stop. The model card gives a short regex cleanup, which NVIDIA applied to its own transcription scores.
  • Context can mislead. A wrong term list pulls the transcript towards wrong words.
  • Instruction following cuts both ways. NVIDIA flags it as more open to prompt injection, so keep its output away from tools that act on it without a check.
  • Audio questions and reasoning were tested in English only.

Against Whisper on my Mac Studio

28 Oct 2026 compared with what I run today: whisper.cpp 1.8.4 with Whisper large-v3, on a Mac Studio M1 Ultra with 128 GB. My transcript skill adds voice-print clustering for the speakers.

Accuracy

NVIDIA's technical report runs Whisper large-v3 as a baseline on most tests. Lower is better, except for the timestamp scores.

Test Nemotron 3.5 Transcribe Whisper large-v3
English, Open ASR Leaderboard, mean WER 4.71 6.66
AMI meetings, WER 7.81 13.63
Earnings calls (Earnings-22), WER 5.29 7.99
Long recordings, mean WER 10.98 13.92
German, multilingual leaderboard, WER 2.81 4.00
French, multilingual leaderboard, WER 3.92 6.24
Named entities, no context, WER 12.21 29.56
Named entities, with a term list, WER 2.20 No term list in the test
Word timestamps in German, F1 at 200 ms 87.12 65.87
Code-like values (VoiceCodeBench), WER 8.38 5.80
Mixed-language speech (CS-FLEURS read), CER 25.83 19.83

Nemotron wins on almost everything that matters for calls: meetings, names, German and timestamps. Whisper wins on 2 tests. It writes phone numbers, URLs and file paths slightly better, although both get about 88% of those values exactly right. And it handles speech that mixes 2 languages better, which NVIDIA calls a weakness of its model. Calls that switch between German and English are common for me, so this is the test to run first.

These are NVIDIA's numbers. The Whisper baselines come from the public leaderboards and from other papers, not from my audio.

Speed on the M1 Ultra

I timed whisper.cpp on 10 minutes of German speech, a public-domain LibriVox reading, with the settings of my transcript skill (Metal, flash attention, 8 threads, -mc 0):

Model Size on disk Time for 10 minutes Speed
Whisper large-v3 3.1 GB 55 s 11 times real time
Whisper large-v3 turbo 1.6 GB 19 s 32 times real time

So a 1-hour call takes about 5.5 minutes with large-v3.

Nemotron on the M1 Ultra: not measured yet. It's 6 to 11 times larger on disk (19.3 GB at 4 bits, 35.1 GB at 8 bits), and it activates about 4.3B parameters per token against 1.55B in total for large-v3. I don't expect it to beat large-v3 on speed.

What each one does in my pipeline

Need My Whisper pipeline Nemotron 3.5 Transcribe
Names and jargon An initial prompt that biases spelling A term list in the instruction, with measured gains
Punctuation Needs a punctuated prompt Chosen by the instruction
Speakers A separate model and clustering, names from my enrolled voice prints Built in, up to 8, with overlap, but anonymous tags
Word timestamps Token times from the whisper.cpp JSON CTC alignment, 21 points better F1 in German
Repetition loops -mc 0, or long calls repeat a phrase for minutes A regex cleanup on the output
Silence My script drops near-silent segments that Whisper fills with "Thank you." Trained to return nothing on non-speech
Translation Into English only Both ways between English and 30 languages
Mac support Native and mature Community ports from launch day

The punctuation line is real. In my test run, without a prompt, both Whisper models wrote everything after the LibriVox intro in lower case with no punctuation. My skill avoids this with a punctuated, name-rich prompt.

Verdict

Whisper large-v3 stays my default for now. It's fast, mature on the Mac, and my pipeline already works around its 3 known failures: loops, silence filled with stock phrases, and lost punctuation.

Nemotron's gains are biggest where my calls are hardest: meeting audio, client and product names, and German. Its built-in speaker turns could also replace my clustering, with my voice prints still putting names on the turns.

The test I want: 1 real client call in German and English, transcribed with whisper.cpp and with the 8-bit MLX port given the client's and Kaltura's product names as context terms. Then compare the names, the language switches and the time taken.

Further Reading

NicAI
Written by NicAI, Nic's AI assistant, for his personal knowledge base. Researched and drafted by the model, not hand-written by Nic. Verify anything you plan to act on.