Unit V — Applications of Parallel Computing

Real-Time Systems & Recent Trends

Session 15 • 2311CSC501J — Parallel Processing • The final session + course wrap-up

What You'll Learn

Real-time & embedded systems: deadlines over throughput
Federated learning — privacy-preserving parallelism
Edge computing & distributed AI training
The whole course, connected into one idea

"Thread → core → GPU → node → cluster → data center is one continuous idea."

— the thesis of this course

When Being On Time IS Being Correct

"A self-driving car that computes the correct braking decision 50 ms too late has computed the wrong decision."

All course we chased throughput. A huge class of systems cares about something else: being on time. A real-time system is one where correctness depends not just on the result, but on when the result arrives.

The mindset flip: Session 01 said parallelism buys throughput, not latency. Real-time systems are the exception — here, guaranteed, predictable latency is the whole point. We now prize determinism over average speed.

Hard vs Soft Real-Time

Type	A late answer is…	Examples
Hard real-time	a failure — the deadline is absolute	ABS brakes, airbags, pacemaker, flight control, robot arm
Soft real-time	degraded quality, but tolerable	video call, game frame rate, live audio, stock ticker

Cars (ADAS)

Camera + radar + LiDAR fused in parallel; perception, planning, control run concurrently — each with its own deadline.

Phones (big.LITTLE)

Big power cores + small efficient cores. Asymmetric multiprocessing (Session 05's AMP) — for battery, not just speed.

RTOS scheduling

Priority-based preemptive scheduling. Optimizes worst-case timing, not average. Predictable beats fast.

Worked Example: Will These Tasks Meet Their Deadlines?

Real-time is the one place in this course where you can prove a system will be fast enough — on paper, before it runs. Three periodic tasks on one processor:

Task	Period T (runs every…)	Execution C (needs…)	Utilisation C/T
Read the brake sensor	50 ms	12 ms	0.24
Update the display	100 ms	20 ms	0.20
Log diagnostics	200 ms	90 ms	0.45
Total utilisation U			0.89

RATE-MONOTONIC (fixed priority: shorter period = higher priority)
    guaranteed schedulable if  U <= n(2^(1/n) - 1)
    for n = 3:                 U <= 3(1.2599 - 1) = 0.779
    we have U = 0.89           -> test FAILS, no guarantee

EARLIEST-DEADLINE-FIRST (dynamic priority: nearest deadline wins)
    guaranteed schedulable if  U <= 1
    we have U = 0.89           -> GUARANTEED to meet every deadline

Same tasks, same processor, same code — and one scheduling policy meets every deadline while the other can't promise it. That's the entire argument for EDF: it uses up to 100% of the processor, where rate-monotonic tops out around 69% as the task count grows.

So why does almost every shipped real-time system use rate-monotonic anyway? Because fixed priorities are simple, predictable and cheap, and because under overload RMS degrades gracefully — the low-priority task misses — while EDF can cascade, with everything missing at once. Engineering is rarely about the optimal algorithm.

One honest caution about the RMS test: it is sufficient, not necessary. U = 0.89 failing the bound means "not guaranteed by this test" — not "definitely misses." A more precise response-time analysis may still prove it fine. In safety-critical work you use the exact test; the bound above is the five-second sanity check.

And the parallel twist that makes this a fitting last topic. On two cores you'd expect the bound to become U ≤ 2. It doesn't. Consider 3 tasks each needing 60 ms every 100 ms: U = 1.8, well under 2 — and yet on 2 cores it is impossible, because no task can run on two cores at once and the third has nowhere to go.

Multiprocessor real-time scheduling is genuinely harder than the single-processor case, and it is still an active research area. A perfect closing note for this course: everything you've learned about parallelism — Amdahl, load balance, dependencies — gets harder the moment "on time" becomes part of "correct."

Recent Trend: Federated Learning

The problem it solves

To train a good ML model — say your phone keyboard's next-word predictor — you need lots of real data. The obvious move: collect everyone's data into one data center and train there.

That's a privacy nightmare. Your messages, a hospital's records, a bank's transactions — nobody wants to ship that to a server, and increasingly the law won't let them (GDPR, India's DPDP Act).

What if we could train a shared model on everyone's data — without the data ever leaving their device?

The core idea: move the model to the data, not the data to the model. Centralize only what the model learned, never the raw data.

One Federated Round

   +---------------------------------------------+
   |            CENTRAL SERVER (model)           |
   +---------------------------------------------+
        |  (1) send current model to devices
        v           v           v           v
   +--------+  +--------+  +--------+  +--------+
   |Phone A |  |Phone B |  |Hospital|  |Phone D |
   | local  |  | local  |  | local  |  | local  |   (2) each trains
   | data   |  | data   |  | data   |  | data   |       LOCALLY on its
   +--------+  +--------+  +--------+  +--------+       OWN private data
        |           |           |           |
        +------ (3) send back ONLY the updates -------+
                    (weight changes / gradients --
                     never the raw data)
                            v
   +---------------------------------------------+
   |   SERVER AGGREGATES updates -> new model    |   (4) Federated
   |   (Federated Averaging: weighted mean)      |       Averaging
   +---------------------------------------------+
                            |
                            +----- repeat for many rounds ---->

The raw data never moves. Only anonymous "lessons learned" flow to the center. Devices train in parallel on their own private shards — then the server reduces the updates into one better model.

It's Your Course, Reassembled

Federated learning step	Course concept it reuses
Send model to N devices	Broadcast / scatter (MPI, Session 11)
Each device trains locally, in parallel	Data parallelism (S01/S06); embarrassingly parallel (S09)
Send back only updates	Communication-minimizing design (Foster's agglomeration, S07)
Server averages the updates	Reduction / all-reduce (S10/S11)

Federated learning is the map → reduce pattern (Session 14) with a privacy twist: map on private data you never see, reduce only the summaries. You already know every piece.

Worked Example: Why You Cannot Centralise the Data

Train a photo model on 1 million phones, each holding about 1,000 photos at 2 MB. Two architectures — and the arithmetic settles it before any privacy argument does.

	Centralise the data	Federated (send the model instead)
What crosses the network	Every photo	A 10 MB model, and 10 MB of updates back
Devices per round	all 1,000,000	a sample of 1,000
Traffic per round	2,000,000 TB = 2 exabytes	20 GB
Total, 1,000 rounds	2 exabytes (once, then stored forever)	~20 TB
Raw photos leaving the device	All of them	None. Ever.
Legal position under GDPR / DPDP	A liability you now own	Nothing personal was collected

Two exabytes. At 100 Gb/s — the fastest link you could realistically buy — that upload takes about five years. Federated learning is not primarily a privacy feature that happens to be efficient. Centralising was never physically possible. The privacy benefit is a gift the physics happened to hand over.

And you have seen this exact move four times already:

Session 04 — computation is cheap, communication is expensive.
Session 12 — keep the data on the GPU, don't round-trip over PCIe.
Session 13 — broadcast the 10 MB table, never shuffle the 1 TB one.
Session 14 — move the code to the data, not the data to the code.

Federated learning is that same principle, pushed to its logical end: a data centre with a billion nodes, where every node is somebody's phone and the data is legally forbidden to move. Different words, one idea — the same one this course has been circling since day one.

The honest caveats, because they're the interesting part: devices go offline mid-round (so you over-sample and drop stragglers — Session 09's load imbalance, with the workers on buses); data is not identically distributed across phones, which genuinely slows convergence; and model updates can leak information about training data, so real deployments add differential privacy noise and secure aggregation on top.

It ships anyway — Gboard's next-word prediction, Apple's on-device models, hospital consortia training on patient data that legally cannot leave the building.

The Honest Catch & Where It's Used

Non-IID data

Every device's data differs. Averaging conflicting updates = heterogeneity / load-imbalance (S09) in disguise.

Stragglers

Phones go offline or are slow. Can't wait for the slowest — the straggler problem (S09). Use whoever reports in time.

Communication cost

Updates over mobile networks are slow. Minimize rounds — the overhead Amdahl (S08) warned eats speedup.

Privacy isn't automatic

Add secure aggregation (server sees only the sum) + differential privacy (add noise) to close the gap.

In the wild: Google Gboard (Android keyboard next-word), healthcare (shared tumor-detection models without sharing patient records), finance (fraud models without exposing transactions).

The Other Frontiers

Edge computing

Do the compute near the data — phone, car, 5G tower — not a distant cloud. Lower latency, better privacy, less bandwidth. Session 14's "move computation to the data," pushed to the edge.

Distributed AI training — three parallelisms at once

Data parallelism

Each GPU gets a different slice of the training data; same model copy.

Model parallelism

Model too big for one GPU — different layers live on different GPUs.

Pipeline parallelism

Factory line: batch 1 at layer 3 while batch 2 is at layer 1.

One arrow of scale: multicore → GPU → clusters → planet-scale. Each step is the same "divide, run at once, combine" at a bigger radius.

The Whole Course, In One Arc

Unit I — Why + Classify + Models

Free lunch over; concurrency vs parallelism; Flynn's SISD/SIMD/MISD/MIMD; shared vs distributed memory; threads & fork-join.

Unit II — Architecture

Interconnects (bus/mesh/torus/hypercube); memory hierarchy, cache coherence (MESI), false sharing; CPUs vs GPUs, SIMT.

Unit III — Design + Measure

Foster's PCAM; Amdahl's Law (the ceiling) & Gustafson; speedup, efficiency; load balancing & work stealing.

Unit IV — Tools

OpenMP (shared, one node) · MPI (distributed, many nodes) · CUDA (the GPU). Labs 1–5.

Unit V — Applications

Scientific computing & parallel databases; big data (MapReduce, Spark); real-time systems & the AI frontier (today).

One Continuous Idea

  thread  ->  core  ->  GPU  ->  node  ->  cluster  ->  data center  ->  the planet
    |         |         |         |          |             |               |
  OpenMP    multi-    CUDA /     MPI      MPI /        MapReduce /       Federated
  thread    core      SIMT      ranks    clusters     Spark             learning

     "Split the work into independent pieces, run them at once,
      keep coordination cheap, and combine the results."
                  ... the SAME idea, at a bigger and bigger scale.

The three questions that never change

1. What can run in parallel?

Decomposition — Session 07.

2. What's the ceiling?

Amdahl / the serial fraction — Session 08.

3. Keep coordination cheap?

Communication, load balance, coherence — S04, S05, S09.

The tools change. These three questions never do.

Exam-Prep Pointers

Highest-value topic: Amdahl's Law, numerically. S = 1 / (f + (1−f)/p), ceiling 1/f. Example: f = 0.1, p = 8 → S = 1/(0.1 + 0.9/8) ≈ 4.7×; ceiling = 10×. Do three practice problems.

Flynn's taxonomy — define all four, one example each.
Memory & coherence — shared vs distributed; MESI; false sharing.
Speedup & efficiency — S = T(1)/T(p); E = S/p; where lost efficiency went.
The three tools & when — OpenMP (one node), MPI (many nodes), CUDA (GPU).
Foster's PCAM; data vs task decomposition; MapReduce word-count.
Federated learning — explain in 3–4 sentences.

Exam-answer formula: for any "explain X" — definition → one-line analogy → real example. It earns most of the marks, and it's exactly how we taught every topic.

That's the Course

In Session 01 I told you the free lunch is over — that someone now has to make software fast on purpose. Fifteen sessions later, that someone is you. You now know:

Why the world went parallel (power wall, more cores).
How to classify any machine (Flynn, memory models).
How the hardware works (interconnects, coherence, CPUs vs GPUs).
How to design & measure (PCAM, Amdahl, speedup, load balancing).
The three real tools and when to use each (OpenMP, MPI, CUDA).
Where it all runs (science, data, real-time, the AI frontier).

You now think in parallel.

Go build things that use all the cores. Thank you — it's been a genuine pleasure.