One Core → Many Cores → A Sea of Cores

• Single-core (until ~2005): one core, one instruction stream. Ended at the power wall (Session 01).
• Multicore (~2005→now): a few full, powerful cores — 4, 8, 16. Your laptop. MIMD.
• Many-core (GPU): thousands of simple cores. Weak alone, a monster together.

   Single-core          Multicore              Many-core (GPU)
   ---------            ---------              ---------------
   +-------+            +---++---+             +-+-+-+-+-+-+-+-+
   |       |            | C || C |             +-+-+-+-+-+-+-+-+
   |  ONE  |            +---++---+             +-+-+-+-+-+-+-+-+
   |  big  |            +---++---+             +-+-+-+-+-+-+-+-+
   | core  |            | C || C |             +-+-+-+-+-+-+-+-+
   +-------+            +---++---+             +-+-+-+-+-+-+-+-+
   1 fast core          4-16 fat cores         1000s of tiny cores
   latency              latency, but wider     pure throughput

Homogeneous vs Heterogeneous

CPU vs GPU: Two Philosophies of "Fast"

Worked Example: When the GPU Loses

Two jobs. Both are "obviously" parallel. One of them the GPU loses badly. Guess which before we do the arithmetic.

Job 1 — add two arrays of 100 million floats

Job 2 — multiply two 8192×8192 matrices

SIMT: How a GPU Runs Threads

SIMT = Single Instruction, Multiple Thread. GPU threads are grouped into bundles of 32, called a warp. All 32 run the same instruction, in lockstep, on their own data.

A warp = 32 threads, all running the SAME instruction in lockstep

  instruction:  c[i] = a[i] + b[i]
  +----+----+----+----+--- ... ---+----+
  | T0 | T1 | T2 | T3 |           | T31|   <- 32 threads
  +----+----+----+----+--- ... ---+----+
    same same same same          same     <- one instruction, 32 data items

The Catch: Warp Divergence

What if the 32 threads hit an if/else and want to go different ways? They can't — the warp shares one instruction stream. So the hardware serializes the branch.

Warp hits an if/else -- threads diverge

  Step 1: run doA()     T0  __  T2  __  T4  __  ...   (odd threads idle)
  Step 2: run doB()     __  T1  __  T3  __  T5  ...   (even threads idle)

  Both paths run one after another -> up to 2x slower, half wasted.

Worked Example: Costing an if on a GPU

A warp is 32 threads sharing one instruction pointer. When they disagree about a branch, the hardware runs both sides, masking off the threads that shouldn't be there.

if (condition) { pathA(); }   // 100 cycles
else           { pathB(); }   // 100 cycles

The subtlety that separates a good GPU programmer from a bad one

if (tid % 2 == 0)
    doThis();
else
    doThat();

if (tid / 32 % 2 == 0)
    doThis();
else
    doThat();

Memory Bandwidth & the Roofline

A GPU's thousands of math units are useless if you can't feed them. Its real superpower is memory bandwidth — a firehose of data (~30× a CPU's).

When to Use a GPU vs a CPU

Your Turn: CPU, GPU, or Neither?

In pairs, 6 minutes. Decide — and say which of today's three tests made the decision: enough parallel work, enough arithmetic per byte, or little enough divergence.

Unit II Wrap-Up: The Hardware Story

Recap & What's Next

Key Takeaways

• Evolution: single → multi → many-core; modern systems are heterogeneous.
• CPU = few fat cores, latency, MIMD. GPU = thousands of thin cores, throughput, SIMT.
• A few professors vs a stadium of students — match the work to the chip.
• Warps of 32 run in lockstep; divergence on branches wastes threads.
• Regular + massive + data-parallel → GPU. Branchy + sequential + latency → CPU.

Homework

• Explain to a friend why a gaming laptop needs both a CPU and a GPU (use the analogy).
• Look up one NVIDIA GPU and one CPU: core count & memory bandwidth of each.
• Come ready: "I have a big problem and a parallel machine — how do I design the solution?"