LLM Serving & Concurrency Capacity Calculator Production vLLM Meta
Estimate real-world request throughput (RPS, RPM, daily requests), concurrent user streams, and generation latency under interactive SLA targets.
Hardware Deployment Serving Benchmarks
Realistic concurrent streams and request capacities across common inference clusters (800 prompt tokens / 250 output tokens at ≥20 t/s SLA).
| Hardware Setup | Model Deployed | Max Concurrent Streams | Sustained RPM | Est. Daily Requests (75% load) | Bottleneck Profile |
|---|---|---|---|---|---|
| 1x RTX 4090 24GB | Llama 3.3 8B (Q4_K_M) | 12 streams | ~65 RPM | ~70,000 / day | Memory Bandwidth SLA limit |
| 2x RTX 4090 48GB | Llama 3.3 70B (Q4_K_M) | 3 streams | ~16 RPM | ~17,200 / day | Tight VRAM Headroom & Bandwidth |
| Mac Studio 192GB (M2/M3) | Llama 3.3 70B (Q4_K_M) | 4 streams | ~14 RPM | ~15,100 / day | Unified Bandwidth (800 GB/s) |
| 1x A100 80GB SXM4 | Llama 3.3 70B (Q4_K_M) | 8 streams | ~42 RPM | ~45,000 / day | Balanced Enterprise Node |
| 1x H100 80GB SXM5 | Llama 3.3 70B (FP8) | 14 streams | ~85 RPM | ~91,800 / day | High-Throughput Single Node |
| 8x H100 Cluster (640GB) | DeepSeek-R1 (MoE FP8) | 48 streams | ~280 RPM | ~302,400 / day | Inter-GPU NVLink Fabric Overhead |
Prefill vs. Decode Phase Mechanics
LLM inference is split into two phases: Prefill (parallel matrix multiplication over prompt tokens, bound by compute TFLOPs) and Decode (sequential autoregressive token generation, strictly bound by GPU memory bandwidth).
Continuous Batching & PagedAttention
Modern serving engines (vLLM, TGI, TensorRT-LLM) use continuous batching to interleave requests as previous ones finish. This prevents idle GPU time but scales aggregate throughput sub-linearly as batch sizes increase.
KV Cache Memory Sizing
Each active stream requires dedicated VRAM for Key-Value tensors across all layers. When VRAM fills up, the server cannot accept new requests regardless of compute capacity, causing queue backpressure.
Production Concurrency & Latency Benchmarks
Real-world production case studies analyzing the mathematical interplay between Memory Bandwidth, Continuous Batching, and Time-to-First-Token (TTFT).
Sub-500ms Latency on Single RTX 4090
Voice agents demand a strict SLA of ≥30 tokens/sec decode speed to match human cadence without audio jitter. Running Llama 3.3 8B (Q4_K_M ~5.2GB) on an RTX 4090 delivers ~145 t/s for a single user.
Llama 3.3 70B on 2x RTX 4090 (48GB Total)
Deploying a 70B parameter model across two 24GB GPUs leaves only ~5GB of VRAM headroom after loading 43GB of weights. At 1,200 token context batches, VRAM fills up before compute saturates.
DeepSeek-R1 (MoE) on 8x NVIDIA H100 Node
With 26,800 GB/s aggregate SXM5 bandwidth and 640GB HBM3, the 37B active parameters per token allow massive continuous batching over thousands of simultaneous API users.