Unit I — Introduction to Parallel Processing

Parallel Programming Models

Session 03 • 2311CSC501J — Parallel Processing • + your first OpenMP program

What You'll Learn

Three ways to split work: data, task, pipeline
The models landscape: OpenMP, MPI, CUDA
OpenMP & the fork-join model
Write your first parallel program — and meet the race condition

"Add one line above a for-loop, and the compiler runs it across all your cores. That's where we start."

— The promise of OpenMP

From Theory to Typing

Session 01

Why the world went parallel. The free lunch ended.

Session 02

How machines are classified. Flynn's taxonomy; shared vs distributed memory.

Session 03 — today

How YOU write parallel code. Your first real program.

Today parallelism stops being a lecture and becomes something you type. We start with OpenMP — the gentlest possible entry: ordinary C with a few #pragma hints.

This is the last session of Unit I.

Three Ways to Split Work

Data

Same operation, lots of data. Split the array; each core does its slice.

100 students grade one page each of the same exam. → OpenMP, GPUs.

Task

Different operations, at once. Thread A downloads, B compresses, C uploads.

One chops, one fries, one plates.

Pipeline

An assembly line of stages. Items flow through stages that run concurrently.

Factory line: weld, paint, seats — all busy on different cars.

A factory line, once full, ships a finished car at the rate of its slowest stage — not the sum of all stages. Your CPU pipelines instructions the same way. Most real systems mix all three. Today's OpenMP work is data parallelism.

The Programming-Models Landscape

Session 02's memory model decides your tool — because it decides how workers communicate.

Memory model	Tool	What the code looks like
Shared memory	Threads / OpenMP	Add `#pragma` hints; all threads share the same variables
Distributed memory	MPI	Explicit `send` / `receive` messages between processes
GPU (SIMD)	CUDA	A "kernel" that runs on every data element at once
Hybrid	MPI + OpenMP + CUDA	Messages between nodes, threads within a node, kernels on the GPU

This course teaches all three — OpenMP, MPI, CUDA — in Unit IV. Today we get our hands on the friendliest one: OpenMP.

What OpenMP Actually Is

Not a new language. Not a big library. Just compiler directives — special comments of the form #pragma omp ... — added to normal C/C++.
The compiler reads them and generates all the thread code for you: creating, scheduling, cleaning up.
If the compiler doesn't understand OpenMP, the pragmas are ignored — your code still runs correctly, just serially. A built-in safety net.

Turn it on with one flag

gcc -fopenmp myprogram.c -o myprogram

Set the team size without recompiling, straight from the shell:

OMP_NUM_THREADS=4 ./myprogram    # a team of 4
OMP_NUM_THREADS=1 ./myprogram    # serial (team of 1)

The Fork-Join Model

Program starts as one thread — the master.
At #pragma omp parallel it forks into a team; all threads run the region.
At the closing brace they join back into one, and the program continues serially.

FORK-JOIN MODEL

                    #pragma omp parallel
                          |
                          v
                     ____ fork ____
                    /      |       \
   master ●========●   thread 1     \
   (serial)         \   thread 2      >  all run AT ONCE
                     \  thread 3     /   (the parallel region)
                      \____ join ___/
                          |
                          v
                     master ●========●   (serial again)

You never wrote a line of thread-creation code. One pragma — the compiler handled fork, scheduling, and join. See it move: examples/04-fork-join.html.

Today's OpenMP Toolkit

Directive / function	What it does
`#pragma omp parallel`	Fork a team; the block runs once per thread
`#pragma omp parallel for`	Fork a team and split the loop's iterations across it
`omp_get_thread_num()`	This thread's id (0, 1, 2, …)
`omp_get_num_threads()`	How many threads are in the team
`reduction(+:sum)`	Private partial `sum` per thread, then a safe combine
`OMP_NUM_THREADS`	Env var: set team size without recompiling

That's the whole toolkit for today. Six things — and you can already write real parallel code.

Your First OpenMP Program

#include <stdio.h>
#include <omp.h>

int main(void) {
    #pragma omp parallel
    {
        int id = omp_get_thread_num();
        int total = omp_get_num_threads();
        printf("Hello from thread %d of %d\n", id, total);
    }
    return 0;
}

Without the pragma: prints once.

With it: the block runs once per thread.

gcc -fopenmp hello.c -o hello
OMP_NUM_THREADS=4 ./hello

No compiler? OnlineGDB · godbolt.org (add -fopenmp) · Colab.

The First Lesson: Order Is Not Yours

A likely run prints:

Hello from thread 2 of 4
Hello from thread 0 of 4
Hello from thread 3 of 4
Hello from thread 1 of 4

The ids are out of order — 2, 0, 3, 1 — and run it again, the order changes. This is not a bug. The threads run genuinely at once; the OS decides who reaches printf first.

Parallel output is non-deterministic. "What order do things happen in?" often has the answer: you don't get to know. That single fact is the source of most parallel bugs — including the next one.

Parallel Sum — the WRONG Way

long sum = 0;
#pragma omp parallel for
for (int i = 0; i < N; i++) {
    sum += a[i];        // RACE: many threads write 'sum' at once
}

Looks fine. It's broken — the answer is too small, and different every run. Why? sum += a[i] is really three steps:

1. Read

load sum from memory

2. Add

compute sum + a[i]

3. Write

store it back to sum

Two threads both read 100, one writes 105, the other writes 103 — clobbering the 105. An update is lost. That's a data race — and it does not exist in serial code.

Parallel Sum — the RIGHT Way

long sum = 0;
#pragma omp parallel for reduction(+:sum)
for (int i = 0; i < N; i++) {
    sum += a[i];        // each thread gets a PRIVATE partial sum
}

reduction(+:sum): every thread gets its own private sum, adds up its own slice with no interference, and OpenMP safely combines them at the end. Correct answer, same every run, still fast.

Heavier alternatives (know they exist)

#pragma omp critical / atomic — let one thread update at a time. Correct, but they serialize the line and kill your speedup.
For a sum, reduction is the right tool: correct and parallel.

Session 01 warned: parallelism trades the speed problem for coordination problems. The race condition is that trade, live.

Recap, Unit I Wrap-Up & What's Next

Key Takeaways

Work splits three ways: data, task, pipeline.
Memory model → tool: shared → OpenMP, distributed → MPI, GPU → CUDA (all three in Unit IV).
OpenMP = #pragma directives on normal C, compiled with -fopenmp; the fork-join model.
Parallel output is non-deterministic; a data race is fixed here by reduction(+:sum).

Unit I in one breath

01 why we went parallel → 02 how machines are classified (Flynn, memory models) → 03 how you program them (OpenMP, and the race condition). Why → how organized → how programmed.

Next: Unit II — Parallel Architecture

The hardware underneath — how all those cores and memories actually talk (interconnection networks).