2311CSC501J

Parallel Processing — Course Materials

A hands-on course on how modern computers run thousands of things at once — from Flynn's taxonomy and cache coherence to writing real parallel programs in OpenMP, MPI, and CUDA. Concepts first, then code.

Essential Resources
Unit I: Introduction to Parallel Processing

Session 01: Why Parallelism?

Easy

The end of the free lunch, concurrency vs parallelism vs multitasking, where speedup comes from

After this session: Explain why every modern chip is multi-core and what parallelism buys you
Slides Examples

Session 02: Flynn's Taxonomy & Memory Models

Easy

SISD, SIMD, MISD, MIMD; shared vs distributed vs hybrid memory

After this session: Classify any machine by Flynn's taxonomy and its memory model
Slides Examples

Session 03: Parallel Programming Models

Medium

Thread, task, and data parallelism + your first OpenMP program

After this session: Write and run a parallel "hello threads" in OpenMP
Slides Examples
Unit II: Parallel Architecture

Session 04: Interconnection Networks

Medium

Bus, crossbar, multistage, hypercube, mesh, torus — how processors talk

After this session: Compare network topologies by cost, latency, and scalability
Slides Examples

Session 05: Memory Hierarchy & Cache Coherence

Medium

The cache coherence problem, MESI protocol, SMP vs AMP

After this session: Explain false sharing and why it destroys parallel performance
Slides Examples

Session 06: Multicore Processors & GPUs

Medium

From one core to thousands; SIMD vs MIMD in real silicon

After this session: Explain why a GPU has thousands of "weak" cores and a CPU has a few "strong" ones
Slides Examples
Unit III: Parallel Algorithms & Design

Session 07: Designing Parallel Algorithms

Medium

Decomposition, granularity, dependency graphs, the PCAM method

After this session: Break a serial problem into parallel tasks and spot dependencies
Slides Examples

Session 08: Measuring Parallel Performance

Hard

Speedup, efficiency, scalability, Amdahl's Law vs Gustafson's Law

After this session: Compute speedup and predict the ceiling with Amdahl's Law
Slides Examples

Session 09: Load Balancing & Case Studies

Hard

Scheduling, parallel search, matrix multiplication, sorting, prime generation

After this session: Choose static vs dynamic scheduling for a given workload
Slides Examples
Unit IV: Programming Models & Tools

Session 10: OpenMP in Depth

Medium

Shared-memory parallelism, directives, reductions, matrix multiplication (Lab 1)

After this session: Parallelize loops with #pragma omp and measure the speedup
Slides Examples

Session 11: MPI — Message Passing

Hard

Distributed-memory programming, send/recv, collectives, parallel sort (Lab 2)

After this session: Write an MPI program that splits work across processes
Slides Examples

Session 12: CUDA & the Modern GPU Stack

Hard

GPU programming, reduction & image processing (Labs 3 & 5), Kubernetes, TensorRT, CUDA-X-AI

After this session: Launch a CUDA kernel and reason about threads, blocks, and grids
Slides Examples
Unit V: Applications of Parallel Computing

Session 13: Scientific Computing & Parallel Databases

Medium

Simulations, parallel databases, data mining at scale

After this session: Explain how a query is parallelized across shards
Slides Examples

Session 14: Big Data & Parallel Processing

Medium

Hadoop, the MapReduce model, Apache Spark

After this session: Trace a word-count through MapReduce and see why Spark is faster
Slides Examples

Session 15: Real-Time Systems & Recent Trends

Medium

Real-time & embedded parallelism, Federated Learning, course wrap-up

After this session: Connect everything learned to where parallel computing is heading
Slides Examples
Browse Complete Resources Library

Hand-picked tools, courses, and papers for learning parallel computing

Prepared by Rajiv Ramakrishnan

gmail@rajiv.kr +91 9566134326 LinkedIn