Case Study: AI Coding Assistant Company GPU Infrastructure Requirements

The AI coding assistant market has exploded from 3 major players in 2023 (GitHub Copilot, Tabnine, Amazon CodeWhisperer) to 25+ in 2026, including Cursor (Anysphere), Continue (open source), Codeium, Replit AI, Cody (Sourcegraph), and two dozen open-source and agentic coding tools. Each operates with fundamentally different GPU requirements based on three variables: model size (code completion models ranging from 500M parameters to Mixtral 8x22B), latency budget (100-1,000 ms for inline completion vs. 5-60 seconds for agentic coding), and throughput (50-5,000 requests per second at peak).

The naive assumption-that coding assistants just run CodeLlama 7B or DeepSeek-Coder 6.7B on a few GPUs-is wrong. Production coding assistants must serve sub-200ms completions across thousands of concurrent users while maintaining 99.9 percent availability. This demands carefully designed inference architectures, often combining speculative decoding with small draft models, KV-cache quantization, and multi-region deployment. The GPU requirements scale linearly with request volume but super-linearly with acceptable latency-halving the latency budget from 500ms to 250ms requires 2-3x more GPUs in the serving tier.

A chatbot can tolerate 2-5 second response times and batch requests for efficiency. A code completion assistant must respond in under 300 milliseconds-ideally under 150ms-because developers type at 40-60 words per minute, and any delay breaks flow state. This latency requirement imposes a “throughput tax”: serving code completions at 300ms requires approximately 4-5x more GPU capacity than an equivalent chatbot model running at 3-second response times, because the inference system must over-provision for peak load rather than optimizing for average throughput.

Consider a coding assistant with 100,000 daily active users. If each user makes 150 completion requests per day (a moderate estimate based on Copilot usage data), the system handles 15 million requests per day (174 per second average). At 200ms target latency with a 300MB 6.7B model in FP16 (batch size 1 for lowest latency), a single H100 handles approximately 10 requests per second. That means 18 H100s for base serving-but peak traffic is 3-4x average, requiring 54-72 H100s just for inline completions. Add chat/agentic features, which consume 5-10x more compute per request, and total GPU demand reaches 120-200 H100s for a mid-size coding assistant.

Every major coding assistant uses speculative decoding (or assisted generation) to hit latency targets without sacrificing output quality. The technique pairs a small, fast draft model (typically 100M-1.5B parameters) with the full-size model. The draft model generates 5-20 candidate tokens quickly; the full model verifies them in a single forward pass. When acceptance rates are high (60-85 percent for code completion, where the draft model is trained on the same codebase), speculative decoding delivers 1.5-3x throughput improvement with zero quality degradation. The net effect: a single H100 serving a 7B model with speculative decoding can match the generation throughput of an H100 running a 7B model without it, but at 200-250ms instead of 400-600ms.

Cursor’s published infrastructure notes indicate they use a custom 800M-parameter Transformer trained on code completion data as their draft model, fine-tuned per-organization on private codebases via LoRA. The draft model runs on the same GPU as the primary model but with a fraction of the KV cache size-approximately 15-20 percent of the primary model’s memory footprint. This configuration, combined with FP8 KV-cache quantization, allows both the draft and primary model to fit on a single H100 with a 32K context window, enabling sub-200ms completion latencies for most requests.

Agentic coding assistants-tools that autonomously navigate repositories, edit multiple files, run tests, and iterate-represent a regime shift in GPU requirements. Where inline completion consumes 0.01-0.03 GPU-seconds per request, a single agentic coding session (planning, code generation, test execution, debugging) can consume 30-300 GPU-seconds on a 34B or 70B model. Devin (Cognition) reportedly uses a 70B-class model with Mixture-of-Experts architecture for the planning component, combined with smaller 7B models for file editing and an 8B model for test generation-all orchestrated via a custom agent loop.

The infrastructure implication is stark: 1,000 active agentic coding sessions per hour requires 8-15 H100-equivalent GPUs at 50 percent utilization, versus 1-2 H100s for an equivalent number of inline completion sessions. This 8-15x compute multiplier means that coding assistants adding agentic features must increase their GPU fleet by 3-5x to maintain service quality. The offsetting factor is pricing: agentic coding tools charge $30-60 per month versus $10-20 for basic completion, providing 2-3x higher revenue per user to fund the additional GPU cost. The economics work as long as agentic GPU utilization stays above 40 percent.

Beyond real-time inference, coding assistants require batch processing infrastructure for codebase indexing and repository-level analysis. This includes: code chunking and embedding generation (using models like CodeBERT or Voyage Code-2 at 1-10K files per repository), semantic code search indexes (FAISS or Pinecone vector databases with 50-500K vectors per mid-size codebase), and dependency graph construction (typically CPU-based but memory-intensive for large monorepos). A 20-person engineering team’s codebase (500K lines, 5,000 files) generates approximately 2-5 GB of index data.

The indexing infrastructure is typically separate from the serving infrastructure because the workload patterns differ: indexing is batch-oriented (a few concurrent jobs that saturate 4-8 GPUs for 10-60 minutes per full re-index), while serving is latency-sensitive and continuously loaded. Most coding assistants run indexing on lower-cost A10G or L4 GPUs ($0.60-1.00 per hour) rather than H100s, because embedding generation doesn’t require FP8 tensor cores or high inter-node bandwidth. Companies like Sourcegraph (Cody) report indexing costs of $200-800 per month for a 100-repo deployment, versus $15,000-60,000 per month for inference serving.

Coding assistants with global user bases face a dilemma: GPU availability is concentrated in US data centers (Northern Virginia, Silicon Valley, Dallas), but latency-sensitive code completion requires inference endpoints within 30-50ms of developers. A developer in Tokyo connecting to a us-west-1 endpoint experiences 100-150ms network latency before any model inference, consuming 50-75 percent of the 200ms latency budget. The solutions are multi-region GPU deployment (1-2 H100 nodes per region for local completion, routing complex requests to centralized clusters) and edge-based prefix caching.

Codeium and Tabnine both use a pattern of deploying small H100 clusters (4-16 GPUs) in 4-6 global regions (US West, US East, EU West, EU Central, AP Southeast, AP Northeast) for inline completions, while routing chat and agentic requests to larger US-based clusters (64-256 H100s). The regional clusters use model distillation to run 3-5x smaller models locally (1B-3B parameters instead of 7B-34B), accepting a 1-3 percent accuracy reduction in exchange for 40-70ms inference latency. The combined approach cuts p95 completion latency from 800ms to 220ms for Japanese developers, a 3.6x improvement that directly impacts user retention (Cursor reported a 12 percent increase in weekly active usage after deploying APAC inference endpoints).