AI Audio and Speech Model Deployment: Whisper, ElevenLabs, Bark GPU Inference

Audio AI spans three distinct inference profiles. Automatic speech recognition (ASR) models like Whisper and Parakeet process audio-to-text with bounded input lengths and near-real-time latency requirements. Text-to-speech (TTS) models like ElevenLabs, Bark, and Coqui generate audio from text with variable output lengths and high quality expectations. Music and audio generation models like MusicGen, AudioLDM, and Stable Audio produce full-bandwidth audio from prompts with compute demands similar to diffusion-based image models but with temporal structure.

The critical distinction is real-time factor (RTF): a model with RTF < 1.0 processes audio faster than real-time. Whisper large-v3 achieves RTF 0.08-0.12 on an H100 for English (8-12x faster than real-time), while Bark TTS runs at RTF 2.5-4.0 on the same hardware. For interactive voice applications, ASR must run at RTF < 0.3 to avoid perceptible lag, while TTS can tolerate RTF up to 1.0 with appropriate buffering.

OpenAI Whisper large-v3 (1.5B parameters) uses a 32-layer encoder with 2x temporal downsampling and a 32-layer autoregressive decoder. The encoder processes 80-channel log-Mel spectrogram features and produces a 1500-token hidden representation for 30-second audio segments. Inference on an H100 with Flash Attention 2 achieves encoder throughput of 375 segments per second (batch size 64, FP16) and decoder throughput of 280 tokens per second per segment. This translates to approximately 60-80 hours of audio transcription per GPU-hour at batch size 64.

The recommended production pattern uses Whisper with a VAD frontend that segments raw audio before feeding chunks to the model. For throughput-optimized serving, deploy Whisper encoder as a stateless batchable endpoint on H100 GPUs with max batch size 64-128 and the decoder on GPU with KV-cache reuse across segments from the same speaker. This bifurcated architecture doubles throughput compared to a monolithic encoder-decoder deployment.

ElevenLabs TTS models are proprietary transformer-based architectures with approximately 800M-1.2B parameters generating audio at 24 kHz using a two-stage pipeline: a text-to-semantic encoder followed by a semantic-to-audio decoder. Production benchmarks show RTF 0.15-0.30 on H100 for Turbo models and RTF 0.40-0.70 for Pro models with multi-voice synthesis.

The critical infrastructure challenge for TTS is the autoregressive audio decoder: generating 24,000 samples per second requires approximately 12 TFLOPS at inference. At RTF 0.5, a 10-second audio clip takes 5 seconds to generate. ElevenLabs solves this with prefix caching: the model caches KV activations for the first 2 seconds of speech, reducing cold-start latency by 60 percent.

Suno Bark is a three-model cascade: a text-to-semantic transformer (300M), a semantic-to-coarse audio transformer (300M), and a coarse-to-fine audio transformer (600M). The cascade design explains its RTF of 2.8-4.0 on H100. Each pass operates on discrete audio codes derived from an EnCodec audio tokenizer. Bark's advantage is its ability to generate non-verbal sounds and prosody variation, but the latency makes it unsuitable for real-time use without aggressive quantization and speculative decoding.

Meta's MusicGen uses a single-stage transformer with EnCodec tokens (50 Hz frame rate, 4 codebooks). A 1.5B model generates 30 seconds of music in 12-15 seconds on an H100 (RTF 0.4-0.5). Stable Audio uses a latent diffusion architecture with a 1D U-Net operating on a compressed audio latent at 46 Hz. For production, the key optimization is using the compressed latent for all processing and only decoding to waveform at the final output stage.

Real-time audio pipelines impose strict latency budgets: end-to-end delay below 300 ms for natural conversation. This requires ASR streaming with partial results and TTS streaming with look-ahead buffering. For Whisper, streaming is achieved through model-level modifications: a simplified decoder with chunked cross-attention over the encoder output for each audio segment.

The GPU architecture for real-time audio typically uses a mid-range GPU (L40S or A10) for ASR and a dedicated H100 for TTS. The ASR GPU processes 500-1000 concurrent streams while the TTS GPU runs a single high-quality voice generation at a time. This asymmetry creates a 1:10 to 1:20 ratio of TTS-to-ASR GPUs in a voice AI deployment.

Audio models benefit from three specific GPU optimizations. First, INT8 quantization: Whisper large-v3 with smooth-quant reduces model size from 5.1 GB to 1.6 GB with WER increase of only 0.3-0.5 percent, enabling cost-effective deployment on L4 instances. Second, CUDA graph capture: the autoregressive decoder loops in both Whisper and TTS models benefit from graph capture, reducing kernel launch overhead by 40-60 percent.

Third, and most impactful, is speculative decoding for the audio decoder. By training a small draft model (40-80M parameters) that predicts the next 4-8 audio tokens in parallel, then verifying against the full model, inference speed improves by 2.0-2.5x on Bark and Coqui XTTS with zero quality degradation. These three optimizations combined can reduce per-character TTS cost by 60-70 percent.