← Back to examples

Shared vs Distributed Memory

The second lens for classifying a parallel machine: how do the processors share data? Either they all reach into one pool of memory (shared), or each keeps its own and they mail each other messages (distributed). This one choice decides your programming tool.

Shared Memory

One pool of RAM — every core sees the same variables
Shared Memory Core 1 Core 2 Core 3 Core 4
Pro: easy — just use threads, they all see the same data. No moving anything.
Con: the one memory is a bottleneck; doesn't scale past tens of cores. (UMA = every core equidistant; NUMA = local fast, remote slow.)
Tool → OpenMP / threads.

Distributed Memory

Each processor has its OWN memory — share by messages
MemMemMemMem CPUCPUCPUCPU Network — messages "here's my result" → sent over the wire
Pro: scales to thousands of nodes — nothing is shared, nothing contends.
Con: you must explicitly move data with messages; more code, more complexity.
Tool → MPI.

The trade-off in one line: shared memory is easy to program but hard to scale; distributed memory is harder to program but scales without limit.

This is exactly vertical scaling (one bigger box = shared) vs horizontal scaling (more boxes = distributed). Every modern supercomputer is hybrid — a cluster of shared-memory nodes, programmed with MPI + OpenMP + CUDA together.