Parallelism

Three patterns, from the simplest to the most demanding

Pattern	Communication	Examples
Trivial parallelism (embarrassingly parallel)	none: each task runs alone, results are gathered at the end	image conversion, parameter sweeps, Monte Carlo
Tight coupling (MPI)	constant: at each time step, each processor exchanges with its neighbors	atmospheric simulation, fluid dynamics, finite elements
Data parallelism	per batch: the model is replicated on each GPU and synchronized after each batch	LLM training, vision, recommendation

The three patterns differ in the shape of their exchanges:

flowchart TB
  subgraph TRI["Trivial"]
    direction LR
    T1["Task 1"] --> TR["Results<br/>gathered"]
    T2["Task 2"] --> TR
    T3["Task 3"] --> TR
  end
  subgraph MPI["Tight coupling, MPI"]
    direction LR
    M1["P1"] <--> M2["P2"]
    M2 <--> M3["P3"]
    M3 <--> M4["P4"]
  end
  subgraph DAT["Data parallelism"]
    direction LR
    G1["GPU 1<br/>model copy"] <--> SY["Synchronization<br/>after each batch"]
    G2["GPU 2<br/>model copy"] <--> SY
  end
  TRI ~~~ MPI
  MPI ~~~ DAT

The more frequent the communication, the more critical the interconnect becomes and the more a single slow node slows down all the others.

Scaling up

Strong scaling: the problem stays fixed, you add processors and measure the time saved.

Weak scaling: the problem grows along with the number of processors, and you check whether the time stays stable.

Amdahl’s law

Doubling the number of processors rarely halves the time. Part of the work cannot be divided. At large scale, these non-parallel parts and the cost of communications set the limit.

Example: if 5% of the work is sequential, the speedup will never exceed 20, whatever the number of processors.

Who uses HPC

Field	Uses
Weather and climate	global forecasts several times a day, simulations over several decades
Energy and engineering	aerodynamics, crash tests, reservoirs, nuclear safety
Life sciences	molecular screening, genomics, medical imaging
Finance and risk	valuation of complex portfolios, regulatory stress tests
Defense	cryptography, signal processing, geospatial
Public research	astrophysics, materials, shared national infrastructures

In most of these fields, there is no other way to see the work through to the end.

Mark as read