When a model is too large for one accelerator, its weights normally stay in fixed shards. The chips move activations and partial results—not the whole model—while every generated token makes another pass through the active weights.
Five-way thought experimentThe weights do not shuffle when a request arrives
parallel partial matrix results→ collective communication →one combined activation
With tensor parallelism, each GPU computes a slice of the same layer, then an all-gather, reduce-scatter, or all-reduce combines the pieces. With pipeline parallelism, different GPUs own different groups of layers and the activation moves stage to stage. Real systems commonly combine both.
A dense 1T model is 2 TB at FP16, so an imaginary five-way split still needs about 400 GB of raw weights per GPU. Five 180 GB B200s provide only 900 GB total: not enough for FP16 or FP8, although a 500 GB INT4 checkpoint could fit before KV cache and runtime overhead.
Are five GPUs generating five tokens?
Not for one ordinary autoregressive sequence. Token 101 depends on token 100, so all five shards cooperate on token 101, then all five cooperate again on token 102. Independent requests and sequences can be batched in parallel; speculative decoding can also propose several candidates and verify them together.
Prefill and decode behave differently
PrefillProcess the prompt's many tokens largely in parallel. Matrix-matrix work is compute-heavy, so FLOPS matter.
→
DecodeGenerate one next token per sequence. At small batch sizes, reading the weights repeatedly often makes memory bandwidth the limit.
What are all those FLOPS for?
They accelerate prompt prefill, training, large batches, long-context attention, and multi-token verification. Batching lets one weight read serve many sequences, raising arithmetic per byte until compute becomes the bottleneck. A GPU can therefore have modest tokens/s per user but enormous aggregate tokens/s.
GPU speed has no single number
In NVIDIA's published tests, one H200 running Llama 3.1 70B produced about 51 output tok/s without a draft model and 114–146 tok/s with speculative decoding. An eight-GPU H200 system reached 268 tok/s per user in a separate low-latency test. A B200 throughput test reported 11,253 output tok/s for 70B FP4—but that is aggregate serving throughput, not one reader's stream.
Model, precision, prompt/output length, batch, latency target, software, and quality constraints must match before comparing.
Cerebras: fewer small-chip boundaries
Cerebras maps compute and weight pieces across the SRAM distributed over each wafer. Its published inference design splits models larger than one wafer at layer boundaries: about 20B FP16 parameters fit on one CS-3 and 70B on as few as four. A simple capacity extrapolation puts a dense 1T FP16 model near 50 wafers before overhead, but Cerebras has not published a matched 1T serving topology or benchmark here.
What changes—and what does not
The model pieces still stay mapped and each output token is still sequential. The advantage is locality: within a wafer, 44 GB of SRAM and 21 PB/s of on-wafer memory bandwidth avoid much of the fine-grained GPU-to-GPU traffic. Cerebras reported 2,100 tok/s per user for 70B and 969 tok/s for 405B in vendor benchmarks that also used software techniques such as speculative decoding.
Why go faster than a person can read?
Fast per-user inference shortens reasoning, code generation, research agents, tool loops, and multi-agent workflows that may consume thousands of hidden or intermediate tokens before showing an answer. Voice benefits mainly from low time-to-first-audio and interruption latency. Video generation uses different image or latent tokens, so text tok/s is not a direct video-speed metric.
Aggregate throughput is the economic prize: if a replica emits more useful tokens per second while meeting the same latency target, it can serve more simultaneous users, so fewer replicas—and fewer chips—are needed.
Serving-capacity laboratorySize a hypothetical production fleet
The per-replica throughput is deliberately an input: for a 1T model, it must come from a benchmark using your exact model, precision, sequence lengths, batching policy, and latency SLO.
Raw demand200Koutput tok/s
Replicas required65including headroom
Accelerators1,040chips total
Active-at-peak users60Kat chosen cadence
At 1,000 requests/s and 200 output tokens/request, the fleet must sustain 200,000 output tok/s—or 260,000 with 30% headroom. At 4,000 tok/s per 16-chip replica, that is 65 replicas and 1,040 chips.
Voice caveat: one request per minute is usually too coarse for an active voice session. Plan voice by concurrent sessions, turns per minute, time to first audio, interruption behavior, and audio real-time factor—not request rate alone.
Latency
p50/p95/p99 time to first token, inter-token latency, time per output token, queue time, and end-to-end response time.
Goodput
Requests or output tokens completed within the TTFT and inter-token latency SLO—not merely work the hardware eventually finishes.
Efficiency
Output tok/s per GPU, per rack, and per megawatt; memory-bandwidth utilization; compute utilization; KV-cache occupancy and hit rate.
Economics
The clean endpoint metric is fully loaded cost per million accepted input and output tokens at the target SLO. Include amortized servers, networking, power, cooling, software, operations, idle capacity, and redundancy.
Why not $ / token / second?
That mixes a flow rate with a unit cost. Use tok/s per dollar of provisioned capacity for infrastructure productivity, then $/million delivered tokens for unit economics. Always hold model quality and availability constant.
