Flynn's Big Idea: Count the Streams

In 1966, Michael Flynn asked a beautifully simple question. Forget the details of any chip — just ask two things:

The 2×2 Matrix

The Four Categories

Example: Classify Six Real Machines

Don't classify by what the machine is. Classify by answering the two questions: how many instruction streams, how many data streams?

Example in Code: SIMD Is Already In Your Laptop

SIMD isn't only a GPU thing. Your CPU has had SIMD units for 25 years. Same C code — the difference is what the compiler emits.

for (i = 0; i < 8; i++)
    c[i] = a[i] + b[i];

// compiles to 8 x addss
// eight instructions,
// one number each

gcc -O3 -mavx2
// same source!

// compiles to 1 x vaddps
// ONE instruction,
// eight numbers at once

Watch: Flynn's Taxonomy & Real Examples

A clean walk through the four categories — a good reinforcement after the matrix.

Quick Check: Name the Class

SISD, SIMD, MISD or MIMD? Ask the two questions before you answer. Shout it out.

SPMD: What Real Programs Do

Nobody hand-writes a different program for each processor. Instead, real parallel code uses one pattern:

if (my_rank == 0):
    read the file, hand out the chunks
else:
    wait for my chunk, then process it

# One program. Rank 0 takes a different path than rank 7.

Worked Example: One Program, Four Ranks

Sum a 400-element array on 4 processors. Here is the entire program — one file, shipped to all four.

rank  = my_id()          // 0, 1, 2 or 3 — the ONLY thing that differs
size  = num_procs()      // 4 everywhere
chunk = 400 / size       // 100 everywhere

start = rank * chunk
end   = start + chunk

partial = sum(a[start:end])

if rank == 0:
    total = partial + receive_from(1, 2, 3)
    print(total)
else:
    send_to(0, partial)

Shared Memory: One Fridge

New lens: forget streams — ask how do processors share data? First answer: they all share one pool of RAM. Core 3 writes x=5, core 7 sees it. Just use threads.

UMA vs NUMA: Every Fridge Equidistant?

Worked Example: What NUMA Actually Costs

A 2-socket server. 32 cores per socket, 256 GB of RAM attached to each socket. Typical measured latencies:

   SOCKET 0                              SOCKET 1
  +-----------+                        +-----------+
  | 32 cores  |<===== interconnect ===>| 32 cores  |
  +-----------+                        +-----------+
       |                                     |
   [256 GB]                              [256 GB]
    ~80 ns  <-- LOCAL                     ~80 ns  <-- LOCAL

  Socket 0 core reaching Socket 1 memory: ~140 ns   (1.75x slower)

Now run the same program, 100 million memory accesses, and vary only where the data sits:

Watch: What Is NUMA?

AKIO TV — a friendly visual of why local and remote memory differ in speed.

Distributed Memory: Own Houses

Each processor has its own private memory. No one can touch another's memory directly — the only way to share is to send a message over a network. That is exactly what MPI does.

Worked Example: When One Box Runs Out

Your app serves 1,000 requests/sec and it's struggling. The tempting fix: get a bigger server. Let's follow that all the way down.

Hybrid: The Modern Reality

Real supercomputers are both: a cluster of nodes on a fast network (distributed), where each node is a shared-memory multicore machine, usually with a GPU. Every modern supercomputer is hybrid.

Example in Depth: One Real Machine, Dissected

An NVIDIA DGX SuperPOD — the class of machine every large AI model is trained on. Let's apply both lenses at every level.

LEVEL 4  32 DGX nodes, joined by InfiniBand         <-- DISTRIBUTED memory, MIMD
             |
LEVEL 3  one DGX node: 2 CPU sockets + 8 GPUs      <-- SHARED memory (NUMA), MIMD
             |
LEVEL 2  one H100 GPU: ~16,900 CUDA cores          <-- its own memory, SIMD
             |
LEVEL 1  one CPU core: an AVX-512 vector unit      <-- SIMD inside SISD

Model → Tool: The Bridge to Unit IV

The payoff: the memory model tells you which tool to use. Classify the machine, then pick the tool. That's the whole workflow.

Your Turn: Classify, Then Choose the Tool

In pairs, 6 minutes. For each system: Flynn class? Memory model? Which tool?

Recap & What's Next

Key Takeaways

• Flynn sorts by streams: SISD, SIMD (GPUs), MISD (rare), MIMD (the common case).
• SPMD = one program, branch by rank — how real MPI/OpenMP code is written (a case of MIMD).
• Shared memory = one pool of RAM (UMA/NUMA); easy, but hits a scaling ceiling.
• Distributed memory = private memory + messages (MPI); harder, but scales forever. Hybrid = both.
• The memory model picks the tool: OpenMP, MPI, or CUDA.

Homework

• Explain "one big computer vs a thousand small computers" (shared vs distributed) to a non-technical friend.
• Is your laptop UMA or NUMA? Confirm a multi-socket server is NUMA.
• Come ready: "If shared memory is so easy, why do supercomputers still use distributed memory between nodes?"