1 · First principles2 · Memory hierarchy3 · Historical evolution 4 · Architecture explorer5 · Models larger than memory 6 · Comparison laboratory7 · Business and market lens 8 · Knowledge check9 · Glossary and sources
Interactive lesson · compute architecture

Memory ×
Compute

Why a register, cache, Apple unified memory, NVIDIA HBM, Google TPU, Groq LPU, and Cerebras wafer-scale engine are all answers to the same problem: how to keep arithmetic fed with data.

Scroll to move from intuition to architecture and economics.
01 · First principles

Computation is easy.
Feeding it is hard.

Processors transform data. Memory stores data. In modern systems, the time and energy spent moving a value can exceed the cost of the arithmetic performed on it.

Latency

How long until the first requested byte arrives. Low latency matters for dependent, sequential work.

Bandwidth

How many bytes arrive per second. High bandwidth matters when thousands of units need data continuously.

Capacity

How much can be stored. More capacity is useful, but it is not automatically faster or closer.

The governing tradeoff: storage closest to compute is usually fastest and most expensive per byte; distant storage is larger and cheaper but slower to access.
Storage
large, distant
Memory
active data
Compute
operations
Result
new state
02 · The hierarchy

From the value in hand
to a warehouse away

Click each level. The approximate rankings are conceptual; exact latency and capacity depend on the processor and generation.

Registers are not cache

Registers hold operands being used now and are named or allocated by the instruction set and compiler. A cache automatically keeps copies of data from lower levels.

Cache is usually SRAM

L1, L2, and L3 are normally built from SRAM. SRAM is a memory technology; “cache” is a role and management policy.

HBM is DRAM nearby

HBM dies are stacked and placed beside the accelerator on an advanced package. They are not part of the GPU logic die, but the path is short and extremely wide.

Unified does not mean on-die

Apple unified memory is package-mounted LPDDR shared coherently by CPU, GPU, and accelerators. It reduces copies; it does not turn DRAM into SRAM.

03 · Historical evolution

Every generation pulls
important data closer

Board era

CPU ↔ memory across a bus

Compute and memory were separate packages. As processors accelerated, waiting for DRAM became increasingly painful.

Cache era

SRAM moves onto the processor

Small caches exploit locality and hide much of DRAM latency. Multiple cache levels emerge as one level cannot optimize size and speed simultaneously.

Parallel era

GPUs create enormous appetite

Thousands of parallel threads and vector units increase throughput—but only when memory systems keep them supplied.

Integration era

SoCs and unified memory

CPU, GPU, media, and neural engines share package memory and a coherent address space, reducing copying and improving efficiency.

Packaging era

HBM, interposers, chiplets, and 3D cache

Advanced packages place dense DRAM or stacked SRAM near compute and connect smaller specialized dies with high-speed links.

Wafer-scale era

Cerebras keeps the wafer intact

A distributed sea of compute and SRAM avoids many chip boundaries. Defect tolerance, cooling, power delivery, and compiler mapping make the approach possible.

Emerging

Near-memory and in-memory compute

Research systems perform selected operations beside or within memory arrays, attempting to reduce movement further.

Diagram zooming from a chip package to an accelerator board, server tray, and full rack
A chip's memory design propagates outward into board layout, cooling, power delivery, networking, rack density, and ultimately system economics.
04 · Architecture explorer

Different machines,
different compromises

Select an architecture to see where compute, fast local storage, and bulk memory live.

05 · Large models

What if the weights
do not fit?

A model's raw weight size is parameter count multiplied by bits per parameter. Real deployments also need KV cache, activations, workspace, and runtime overhead.

Weight-size calculator

70B parameters

140 GB

Raw weights only; excludes runtime overhead.

Ways to make it work

Shard the model

Tensor, pipeline, or expert parallelism divides work and weights among accelerators. Communication becomes part of the critical path.

Quantize

Fewer bits per parameter reduce capacity and bandwidth demand, sometimes with a quality or engineering tradeoff.

Stream or offload

Weights can live in CPU memory, external parameter memory, or storage and move into fast memory as required. The risk is starving compute.

Exploit sparsity or MoE

Only some weights or experts activate for a token. Hardware and software must avoid moving inactive data.

Important correction: Cerebras' 44 GB is on-wafer SRAM, not necessarily the total model-storage limit of a full Cerebras deployment. External parameter memory, streaming, quantization, and multiple systems change what can be trained or served.
Dense 1T FP16 thought experiment

Two terabytes of weights must move for every complete pass.

A one-trillion-parameter model at 16 bits per parameter occupies about 2 TB before KV cache, activations, workspace, and runtime overhead.

bandwidth ceiling ≈ memory bandwidth ÷ 2 TB

This is a simplified ceiling, not a token-per-second benchmark. It ignores communication, batching, model fit, kernel efficiency, sparsity, and streaming. In particular, a dense 1T FP16 model does not fit in 44 GB of on-wafer SRAM.

06 · Comparison laboratory

Never compare chips
with one headline number

Choose a dimension. Scores below are qualitative teaching aids, not purchase benchmarks. Real rankings change with model, precision, batch size, latency target, software version, and system scale.

Representative hardware snapshot

Published bandwidth and capacity, with unlike memory tiers labeled

ArchitectureMain memory arrangementRepresentative capacityBandwidthDense 1T FP16 ceiling
Intel Xeon serverDDR5 system memory512 GB–2 TB+ per socket307–410 GB/s≈ 0.18 tok/s
Apple M4 MaxUnified package LPDDR5XUp to 128 GB546 GB/s≈ 0.27 tok/s
NVIDIA B200On-package HBM3e180 GB8.0 TB/s≈ 4.00 tok/s
Google TPU IronwoodOn-package HBM192 GB7.37 TB/s≈ 3.69 tok/s
Groq LPUOn-chip SRAM-centricNot exposed as an HBM-like poolNo single comparable figureNot comparable
Cerebras WSE-3Distributed on-wafer SRAM44 GB on-wafer21 PB/s on-wafer10,500 tok/s mathematical ceiling*

*The Cerebras figure divides on-wafer bandwidth by raw weight size; a dense 1T FP16 model does not fit in the on-wafer SRAM and must be streamed or sharded. These are memory-bandwidth ceilings, not matched workload benchmarks.

Performance

Time to first token, tokens per second, tail latency, training time to target quality, and sustained utilization.

Economics

Cost per million tokens, completed training run, watt, rack, operator, network, cooling, and idle hour.

Software

Framework support, libraries, compilers, debugging, model availability, reliability, and developer familiarity.

07 · Business and market lens

Can a better architecture
become a better business?

Cerebras does not need to replace NVIDIA

A specialist can become valuable by winning a meaningful slice of latency-sensitive inference, sovereign AI, private clusters, training, or scientific computing. The likely future is heterogeneous rather than one universal processor.

Hyperscalers hedge

Google develops TPUs. AWS develops Trainium and Inferentia. Microsoft develops Maia. They may still offer NVIDIA and specialist third-party accelerators when customers demand them.

Evidence to watch

Repeat customers, cloud distribution, recurring utilization, matched-quality benchmarks, model launch cadence, gross margin, customer concentration, and supply-chain capacity.

Four outcomes

Niche, strong specialist, broad platform, or commercial failure. Chip specifications alone cannot select the outcome or justify a market capitalization.

Google specifically: its publicly described core AI stack centers on TPUs and NVIDIA GPUs. There is no strong public evidence that Cerebras is a major part of Google's core production fleet. AWS announced a Cerebras offering/integration path in 2026.
08 · Knowledge check

Test the mental model

09 · Glossary and sources

Keep exploring

Compact glossary

FLOPS · HBM · SRAM · DRAM · KV cache · chiplet · wafer-scale.

Primary sources

Product specifications and availability evolve. Recheck current documentation before engineering, procurement, or investment decisions.