Riva ASR NIM

Agent: When walking the user through a multi-step workflow, announce each step before presenting it: Step N/M — Step Title (e.g., "Step 1/4 — Set Model Variables").

Purpose

Deploy and run NVIDIA Riva ASR (speech-to-text) NIMs. Supports cloud-hosted inference via build.nvidia.com (no GPU required) and self-hosted deployment on your own GPU using Docker. Covers streaming and offline transcription, word boosting, and performance benchmarking.

Workflow

Choose Option A (cloud) for quick testing without a GPU, or Option B (self-hosted) for production. Self-hosted follows a 4-step process: set model variables → run container → verify health → run inference.

Prerequisites

• Complete riva-nim-setup before self-hosted deployment: NVIDIA Container Toolkit, NGC_API_KEY exported, Docker logged in to nvcr.io
• Cloud-hosted inference: pip install -U nvidia-riva-client and a valid NVIDIA_API_KEY
• Not sure which model to use? Run riva-model-selection first

Instructions

1. Pick a path: cloud inference (Option A) or self-hosted NIM (Option B).
2. For cloud inference (Option A): install nvidia-riva-client, set NVIDIA_API_KEY, pick a function ID, run transcribe_file.py (streaming) or transcribe_file_offline.py (offline) against grpc.nvcf.nvidia.com:443 with --use-ssl.
3. For self-hosted (Option B): look up CONTAINER_ID and NIM_TAGS_SELECTOR from the ASR support matrix, mount a model cache dir with chmod 700, then follow Steps 1–4 below (set vars, run container, verify readiness, run inference).

Option A — Cloud-Hosted Inference (build.nvidia.com)

Setup: pip install -U nvidia-riva-client + clone https://github.com/nvidia-riva/python-clients and cd into it.

Auth: Set NVIDIA_API_KEY from https://build.nvidia.com (different from NGC_API_KEY).

Server: grpc.nvcf.nvidia.com:443 — always pass --use-ssl.

Canonical command (streaming models):

python python-clients/scripts/asr/transcribe_file.py \
    --server grpc.nvcf.nvidia.com:443 --use-ssl \
    --metadata function-id "<FUNCTION_ID>" \
    --metadata "authorization" "Bearer $NVIDIA_API_KEY" \
    --language-code <LANG_CODE> \
    --input-file /path/to/audio.wav

For offline-only models use transcribe_file_offline.py instead. For Whisper translation add --custom-configuration "task:translate".

Word timestamps: All models except Whisper and Canary support --word-time-offsets. Accuracy ranking: CTC > TDT > RNNT.

Note: Both cloud and self-hosted scripts use --input-file, not --audio-file.

Function IDs

If a function-id no longer works, fetch the current one from https://build.nvidia.com/<org>/<model>/api.

Option B — Self-Hosted NIM Deployment

Complete riva-nim-setup first: NVIDIA Container Toolkit, NGC_API_KEY exported, Docker logged in to nvcr.io. Driver and VRAM minimums: see https://docs.nvidia.com/nim/speech/latest/reference/support-matrix/asr.html. Latency and throughput benchmarks: https://docs.nvidia.com/nim/speech/latest/reference/performances/asr/performance.html

Step 1 — Set Model Variables

Get the current CONTAINER_ID and NIM_TAGS_SELECTOR values from the ASR support matrix (includes all models, modes, VRAM, and deployment profiles): https://docs.nvidia.com/nim/speech/latest/reference/support-matrix/asr.html

export CONTAINER_ID=<container-id-from-support-matrix>
export NIM_TAGS_SELECTOR="<selector-from-support-matrix>"

NIM_TAGS_SELECTOR pattern: name=<model-name>,mode=<str|offline|all>[,model_type=<prebuilt|rmir>]

Prebuilt vs RMIR: The NIM auto-detects your GPU on startup. For well-known GPUs (A100, H100, select Blackwell), it pulls a prebuilt model repo — a tarball of TensorRT engines already compiled for that GPU via riva-build + riva-deploy. For unsupported GPUs, it falls back to RMIR (Riva Model Intermediate Representation), which is a portable format that gets compiled into TensorRT engines on your local GPU at first run (slower startup, same runtime performance). You rarely need to set model_type explicitly — omit it and the NIM picks the right one automatically.

Step 2 — Run the Container

export LOCAL_NIM_CACHE=~/.cache/nim
mkdir -p $LOCAL_NIM_CACHE && chmod 700 $LOCAL_NIM_CACHE

docker run -it --rm --name=$CONTAINER_ID \
  --runtime=nvidia \
  --gpus '"device=0"' \
  --shm-size=8GB \
  -e NGC_API_KEY \
  -e NIM_TAGS_SELECTOR \
  -e NIM_HTTP_API_PORT=9000 \
  -e NIM_GRPC_API_PORT=50051 \
  -p 9000:9000 \
  -p 50051:50051 \
  -v $LOCAL_NIM_CACHE:/opt/nim/.cache \
  nvcr.io/nim/nvidia/$CONTAINER_ID:latest

Omit -v $LOCAL_NIM_CACHE:/opt/nim/.cache to skip caching (re-downloads model on every run).

Security note: NGC_API_KEY passed via -e NGC_API_KEY inherits from the shell environment. For production, use Docker secrets or a secrets manager instead of env vars; avoid storing API keys in shell history or plaintext config files.

RMIR Model (Export + Re-run Pattern)

export NIM_EXPORT_PATH=~/nim_export
mkdir -p $NIM_EXPORT_PATH && chmod 700 $NIM_EXPORT_PATH
export NIM_TAGS_SELECTOR="name=parakeet-1-1b-ctc-en-us,mode=str,model_type=rmir"

# Step 1: Export
docker run -it --rm --name=$CONTAINER_ID \
  --runtime=nvidia --gpus '"device=0"' --shm-size=8GB \
  -e NGC_API_KEY -e NIM_TAGS_SELECTOR \
  -e NIM_HTTP_API_PORT=9000 -e NIM_GRPC_API_PORT=50051 \
  -p 9000:9000 -p 50051:50051 \
  -v $NIM_EXPORT_PATH:/opt/nim/export \
  -e NIM_EXPORT_PATH=/opt/nim/export \
  nvcr.io/nim/nvidia/$CONTAINER_ID:latest

# Step 2: Run from export
docker run -it --rm --name=$CONTAINER_ID \
  --runtime=nvidia --gpus '"device=0"' --shm-size=8GB \
  -e NGC_API_KEY -e NIM_TAGS_SELECTOR \
  -e NIM_DISABLE_MODEL_DOWNLOAD=true \
  -e NIM_HTTP_API_PORT=9000 -e NIM_GRPC_API_PORT=50051 \
  -p 9000:9000 -p 50051:50051 \
  -v $NIM_EXPORT_PATH:/opt/nim/export \
  -e NIM_EXPORT_PATH=/opt/nim/export \
  nvcr.io/nim/nvidia/$CONTAINER_ID:latest

Step 3 — Verify Readiness

curl -X GET http://localhost:9000/v1/health/ready
# Expected: {"status":"ready"}

Step 4 — Run Inference

Streaming ASR (Python — gRPC)

python3 python-clients/scripts/asr/transcribe_file.py \
  --server 0.0.0.0:50051 \
  --input-file /path/to/audio.wav \
  --language-code en-US

With diarization and word timestamps:

python3 python-clients/scripts/asr/transcribe_file.py \
  --server 0.0.0.0:50051 \
  --input-file /path/to/audio.wav \
  --language-code en-US \
  --speaker-diarization \
  --word-time-offsets

For real-time microphone streaming:

python3 python-clients/scripts/asr/transcribe_mic.py \
  --server 0.0.0.0:50051

Offline Transcription (Python — gRPC)

python3 python-clients/scripts/asr/transcribe_file_offline.py \
  --server 0.0.0.0:50051 \
  --input-file /path/to/audio.wav \
  --language-code en-US

HTTP API

curl -X POST http://localhost:9000/v1/audio/transcriptions \
  -H "Content-Type: multipart/form-data" \
  -F "file=@/path/to/audio.wav" \
  -F "model=asr"

C++ Client

# Build (requires Bazel in cpp-clients/)
cd cpp-clients
bazel build //riva/clients/asr:riva_asr_client

# Run
./bazel-bin/riva/clients/asr/riva_asr_client \
  --server=0.0.0.0:50051 \
  --audio-file=/path/to/audio.wav

WebSocket (AudioCodes / Telephony)

The websocket-bridge repo provides a Node.js bridge for AudioCodes-compatible telephony integrations:

cd websocket-bridge
npm install
node src/index.js --riva-server=localhost:50051

Port Reference

Examples

These are end-to-end use cases that combine features covered above; they are not repeats of the canonical commands in Option A or Step 4.

Whisper translation — non-English audio → English transcript (cloud):

python python-clients/scripts/asr/transcribe_file_offline.py \
    --server grpc.nvcf.nvidia.com:443 --use-ssl \
    --metadata function-id "b702f636-f60c-4a3d-a6f4-f3568c13bd7d" \
    --metadata authorization "Bearer $NVIDIA_API_KEY" \
    --custom-configuration "task:translate" \
    --input-file spanish_audio.wav

Streaming with diarization and word timestamps (self-hosted):

python3 python-clients/scripts/asr/transcribe_file.py \
  --server 0.0.0.0:50051 --input-file meeting.wav --language-code en-US \
  --speaker-diarization --word-time-offsets

Domain-specific vocabulary via token boosting (CTC, self-hosted):

python3 python-clients/scripts/asr/transcribe_file.py \
  --server 0.0.0.0:50051 --input-file clinical_dictation.wav \
  --language-code en-US \
  --boosted-lm-words 'Abloopar:▁a/b/lo/op/er' \
  --boosted-lm-score 80

Troubleshooting

• Wrong NIM_TAGS_SELECTOR — if the selector doesn't match any available profile, the container exits. Check the support matrix for exact tag values.
• GPU device index — --gpus '"device=0"' targets GPU 0. Adjust for multi-GPU hosts.
• Port 8000 conflict — avoid NIM_HTTP_API_PORT=8000; use 9000 (default).
• Parakeet TDT and word timestamps — only available via gRPC with enable_word_time_offsets=True in the recognition config.

Customization

Word Boosting

Bias the model toward domain-specific words at request time. Pass each word as a separate flag:

python scripts/asr/transcribe_file.py \
  --server 0.0.0.0:50051 \
  --input-file audio.wav \
  --boosted-lm-words 'AntiBERTa' \
  --boosted-lm-words 'Abloopar' \
  --boosted-lm-score 20

Boost score ranges: CTC: 20–100 (negative scores discourage words) | RNNT/TDT: 0.5–2.0 (single score for all words, no negatives, no OOV).

Token Boosting (CTC only)

Maps the tokens the model predicts to the word you want. Use when word boosting alone can't overcome the acoustic model (e.g. multi-syllable OOV terms).

Step 1 — Get the model tokenizer from the running container:

docker exec <container> find /data/models -name "*tokenizer.model" | head -5
docker cp <container>:/data/models/<model-dir>/1/<hash>_tokenizer.model /tmp/tokenizer.model

Step 2 — Generate token mapping with SentencePiece:

import sentencepiece as spm
s = spm.SentencePieceProcessor(model_file='/tmp/tokenizer.model')

word_asr_predicts = "ablooper"       # what the model currently outputs
word_asr_should_predict = "Abloopar" # what you want

tokens = s.encode(word_asr_predicts, out_type=str)
print(word_asr_should_predict + ':' + '/'.join(tokens))
# e.g. Abloopar:▁a/b/lo/op/er

Step 3 — Pass mapping as boosted word:

--boosted-lm-words 'Abloopar:▁a/b/lo/op/er' --boosted-lm-score 80

Important: Always use the actual model tokenizer from inside the container — test tokenizers generate different token sequences and the mapping will silently fail.

Performance Benchmarking (Self-Hosted)

Use riva_streaming_asr_client — a pre-built binary available in PATH inside the NIM container. Do not use the Python script from python-clients; run the binary via docker exec.

A sample LibriSpeech wav file is bundled at /opt/riva/examples/asr_lib/1272-135031-0000.wav inside the container.

Streaming Models

Run at increasing concurrency levels (1, 2, 4, 8, …). Set num_iterations to 3× num_parallel_requests for stable results.

export N=4  # num parallel streams — sweep: 1, 2, 4, 8, ...

docker exec <container_name> riva_streaming_asr_client \
  --riva_uri=0.0.0.0:50051 \
  --language_code=en-US \
  --audio_file=/opt/riva/examples/asr_lib/1272-135031-0000.wav \
  --chunk_duration_ms=160 \
  --simulate_realtime=true \
  --automatic_punctuation=true \
  --num_parallel_requests=$N \
  --num_iterations=$((3 * N)) \
  --word_time_offsets=false \
  --print_transcripts=false \
  --interim_results=false \
  --output_filename=/tmp/output.json

Offline Models

export N=4  # num parallel requests — sweep: 1, 2, 4, 8, ...

docker exec <container_name> riva_streaming_asr_client \
  --riva_uri=0.0.0.0:50051 \
  --language_code=en-US \
  --audio_file=/opt/riva/examples/asr_lib/1272-135031-0000.wav \
  --automatic_punctuation=true \
  --num_parallel_requests=$N \
  --num_iterations=$((3 * N)) \
  --word_time_offsets=false \
  --print_transcripts=false \
  --interim_results=false \
  --output_filename=/tmp/output.json

Note: Omit --chunk_duration_ms and --simulate_realtime for offline models — they process the full audio in one shot, not in streaming chunks.

Key flags:

Output metrics:

Limitations

• x86_64 architecture only — ARM is not supported
• Self-hosted deployment requires an NVIDIA AI Enterprise license
• Cloud-hosted inference requires an active NVIDIA_API_KEY and internet access
• Word/token boosting is advisory — does not guarantee transcription of boosted terms
• TDT models have a silence-based end-of-utterance threshold in streaming mode

Next Steps

• Customize ASR pipeline (VAD, diarization, language model): see riva-pipelines
• Deploy a custom-trained model: see riva-asr-custom
• Check system requirements: see riva-ops