Unit II — Parallel Architecture

Multicore Processors & GPUs

Session 06 • 2311CSC501J — Parallel Processing

What You'll Learn

Single-core → multicore → many-core
CPU vs GPU: latency vs throughput
SIMT, warps, and warp divergence
When to reach for a GPU vs a CPU

"A CPU is a few brilliant professors. A GPU is a stadium full of students. Give the professors the PhD problem; give the stadium the million flashcards."

— The analogy for today

One Core → Many Cores → A Sea of Cores

Single-core (until ~2005): one core, one instruction stream. Ended at the power wall (Session 01).
Multicore (~2005→now): a few full, powerful cores — 4, 8, 16. Your laptop. MIMD.
Many-core (GPU): thousands of simple cores. Weak alone, a monster together.

   Single-core          Multicore              Many-core (GPU)
   ---------            ---------              ---------------
   +-------+            +---++---+             +-+-+-+-+-+-+-+-+
   |       |            | C || C |             +-+-+-+-+-+-+-+-+
   |  ONE  |            +---++---+             +-+-+-+-+-+-+-+-+
   |  big  |            +---++---+             +-+-+-+-+-+-+-+-+
   | core  |            | C || C |             +-+-+-+-+-+-+-+-+
   +-------+            +---++---+             +-+-+-+-+-+-+-+-+
   1 fast core          4-16 fat cores         1000s of tiny cores
   latency              latency, but wider     pure throughput

Homogeneous vs Heterogeneous

Homogeneous

All cores the same kind. A plain multicore CPU: 8 identical cores.

Heterogeneous

Different processing units together: CPU + GPU + accelerators (NPU, video encoder).

Why heterogeneous? Same reason a company hires both senior architects and a large support team: some jobs need one genius; some need a thousand hands. Right tool for the job.

CPU vs GPU: Two Philosophies of "Fast"

CPU — latency

A few powerful cores, big caches, branch prediction. Finish one thread as fast as possible. Silicon spent on being smart.

A few brilliant professors.

GPU — throughput

Thousands of simple cores, huge bandwidth. Finish a million tasks/sec. Silicon spent on being wide.

A stadium full of students.

Dimension	CPU	GPU
Cores	A few (4–16), powerful	Thousands, simple
Optimized for	Latency (one task fast)	Throughput (many tasks/sec)
Silicon spent on	Cache + control logic	Arithmetic units
Bandwidth	~50–100 GB/s	~1–3 TB/s
Flynn (S02)	MIMD	SIMT / SIMD-like

Watch: CPU vs GPU, Visually

NVIDIA / Mythbusters — the CPU paints one dot at a time; the GPU paints the whole picture in a single shot.

The CPU is a few smart painters, one dot at a time — that's latency. The GPU fires all the paint at once — that's throughput, and it's why GPUs run graphics and AI.

SIMT: How a GPU Runs Threads

SIMT = Single Instruction, Multiple Thread. GPU threads are grouped into bundles of 32, called a warp. All 32 run the same instruction, in lockstep, on their own data.

A warp = 32 threads, all running the SAME instruction in lockstep

  instruction:  c[i] = a[i] + b[i]
  +----+----+----+----+--- ... ---+----+
  | T0 | T1 | T2 | T3 |           | T31|   <- 32 threads
  +----+----+----+----+--- ... ---+----+
    same same same same          same     <- one instruction, 32 data items

The trick: the GPU decodes one instruction and applies it to 32 data items. The control logic a CPU duplicates per core is shared across 32 threads — that's how a GPU packs in thousands of cores.

The Catch: Warp Divergence

What if the 32 threads hit an if/else and want to go different ways? They can't — the warp shares one instruction stream. So the hardware serializes the branch.

Warp hits an if/else -- threads diverge

  Step 1: run doA()     T0  __  T2  __  T4  __  ...   (odd threads idle)
  Step 2: run doB()     __  T1  __  T3  __  T5  ...   (even threads idle)

  Both paths run one after another -> up to 2x slower, half wasted.

Instead of 32 threads working, you get 16, then 16. This is warp divergence — a top reason real GPU code is slower than peak.

The lesson: keep threads in a warp doing the same thing. Branchy code? Leave it on the CPU.

Memory Bandwidth & the Roofline

A GPU's thousands of math units are useless if you can't feed them. Its real superpower is memory bandwidth — a firehose of data (~30× a CPU's).

Compute ceiling

How many math ops/sec the chip can do.

Bandwidth ceiling

How fast it moves data in and out of memory.

Your speed is capped by whichever ceiling you hit first (the roofline). GPUs win when the problem is data-parallel and bandwidth-hungry — graphics, matrix math, deep learning. No coincidence all three run on GPUs.

When to Use a GPU vs a CPU

If the work is…	Use the…	Why
Data-parallel (same op, millions of items)	GPU	Built for exactly this
Large & regular (arrays, matrices, images)	GPU	Bandwidth feeds the cores
Branchy / decision-heavy	CPU	Divergence kills the GPU
Sequential (step 2 needs step 1)	CPU	No parallelism to exploit
Latency-sensitive / small	CPU	Transfer cost outweighs the gain

The hidden cost: to use a GPU you must copy data to it and results back. For a small job, that copy costs more than the GPU saves.

SIMD vs MIMD in Real Silicon

Flynn's four boxes from Session 02 weren't exam trivia — they're the blueprint of the two chips in front of you.

Multicore CPU = MIMD

Multiple Instruction, Multiple Data. 8 cores, 8 independent programs on independent data.

GPU = SIMT (SIMD-like)

Within a warp, all 32 threads run the same instruction on different data — textbook SIMD, with per-thread masking.

A GPU isn't purely one thing: one SM is SIMD-ish (warps in lockstep), but a whole GPU running many different kernels is MIMD-ish at the top level.

Unit II Wrap-Up: The Hardware Story

Session 04 — Interconnection Networks

How cores & memory talk. Add cores and communication — not computation — becomes the bottleneck.

Session 05 — Memory Hierarchy & Coherence

Shared memory is easy to say, hard to build. Caches create the coherence problem (MESI) and false sharing.

Session 06 — Multicore & GPUs

The chips: a few fat MIMD CPU cores for latency, or thousands of thin SIMT GPU cores for throughput.

The through-line: parallel hardware is a story about communication and memory, not just cores. Adding cores is easy; making them talk and agree is the hard part.

Recap & What's Next

Key Takeaways

Evolution: single → multi → many-core; modern systems are heterogeneous.
CPU = few fat cores, latency, MIMD. GPU = thousands of thin cores, throughput, SIMT.
A few professors vs a stadium of students — match the work to the chip.
Warps of 32 run in lockstep; divergence on branches wastes threads.
Regular + massive + data-parallel → GPU. Branchy + sequential + latency → CPU.

Homework

Explain to a friend why a gaming laptop needs both a CPU and a GPU (use the analogy).
Look up one NVIDIA GPU and one CPU: core count & memory bandwidth of each.
Come ready: "I have a big problem and a parallel machine — how do I design the solution?"

Next (Unit III): Designing Parallel Algorithms

Foster's PCAM methodology — from hardware to design.