Parallel, Distributed, and Cloud Computing

Modern AI workloads are dominated by one operation performed billions of times: multiplying and adding numbers in matrices, as when a neural network layer transforms an input vector. A CPU core executes such operations mostly one after another, optimized for flexible, branching logic; a GPU instead packs thousands of simpler cores that all execute the same instruction on different pieces of data simultaneously — a pattern called data parallelism. Concretely, multiplying a 1,024-by-1,024 weight matrix by a batch of input vectors decomposes into millions of independent multiply-add operations; a GPU assigns different output entries to different cores and computes them in parallel, while a CPU would work through them largely sequentially. This is why training large models moved from CPUs to GPUs (and specialized accelerators) over the last decade: the underlying arithmetic of deep learning is embarrassingly parallel, and the MIT OpenCourseWare parallel-and-distributed-systems curriculum frames this class of problem as exactly the kind that benefits from many simple processing units working concurrently.

When a single machine — even one with several GPUs — is not enough, work is spread across a cluster of networked machines, which introduces distributed computing rather than just parallel computing. The distinction matters: parallel computing typically means many processors sharing memory and working in tight lock-step, while distributed computing means separate machines, each with its own memory, coordinating over a network. Training a model across a cluster can be split two ways: in data parallelism, each machine holds a full copy of the model but trains on a different slice of the dataset, periodically synchronizing weight updates; in model parallelism, the model itself is too large for one machine's memory, so different layers or partitions of the model live on different machines, and data flows between them. A worked example: splitting ten million training images across 20 machines (500,000 images each) so every machine computes gradients on its shard, then averaging those gradients across the cluster before the next training step — this is data parallelism in practice.

Distributing work across machines introduces problems that do not exist on a single computer: network latency between machines slows down the frequent synchronization that training requires, and any individual machine can fail mid-job. MIT's graduate distributed-systems curriculum centers on exactly these concerns — coordinating multiple independent computers, replicating state, and tolerating the failure of individual nodes without losing the whole computation. In practice this means an AI training cluster needs mechanisms to checkpoint progress periodically (so a crashed node doesn't erase days of training) and to tolerate stragglers (slow machines that would otherwise hold up every other machine waiting to synchronize). Understanding this trade-off — more machines mean more raw compute, but also more communication overhead and more chances for something to fail — is essential before deciding how much parallelism a given workload actually needs. A cluster of 20 machines does not automatically finish a training job 20 times faster than one machine, because time spent exchanging gradient updates over the network eats into the gains; past a certain cluster size for a given model, adding more machines can even slow the job down once communication overhead outweighs the extra compute, which is why practitioners benchmark scaling behavior rather than assuming it is linear.

Getting a distributed training job to actually run reliably requires more than raw hardware — it requires a consistent way to package software so it behaves identically wherever it runs. A container packages an application together with everything it needs to run — code, runtime, system libraries, and settings — as a single lightweight, portable unit that is isolated from other software on the same machine, as defined in Docker's own documentation. Concretely, a data science team can package a training script, its exact Python version, and every library it depends on into one container image; that same image runs identically on a laptop, a lab server, and a cloud GPU instance, eliminating the classic "it worked on my machine" failure. This portability is what makes it practical to move an AI workload between a researcher's laptop and a large cluster without re-installing dependencies by hand at every step.

Running one container is straightforward, but a real training or serving job may need dozens or thousands of containers started, restarted after failures, and scaled up or down as demand changes — a job called orchestration. Kubernetes, the dominant open-source orchestration platform, describes itself as "a portable, extensible, open source platform for managing containerized workloads and services that facilitates both declarative configuration and automation". In practice, an engineer tells Kubernetes the desired end state — for example, "keep 20 copies of this training-worker container running" — and Kubernetes continuously monitors the cluster, restarting or rescheduling containers on healthy machines whenever one fails, without a human manually intervening. This declarative, self-healing approach is what lets a distributed AI workload survive individual machine failures gracefully rather than halting the whole job. Kubernetes also handles scaling: if the number of incoming requests to a deployed model grows, an operator (or an automated rule) can raise the declared number of running copies, and Kubernetes schedules the additional containers onto available cluster machines without anyone manually logging into each one, which is what makes it practical to run a training job across dozens of nodes or serve a model to a fluctuating number of users.

Cloud computing is the layer that supplies the underlying machines, GPUs, and storage on demand rather than requiring an organization to buy and maintain its own hardware. NIST SP 800-145 formally defines cloud computing through five essential characteristics: on-demand self-service, broad network access, resource pooling, rapid elasticity, and measured service. A worked example: instead of purchasing a cluster of GPU servers that would sit idle most of the year, a research group rents GPU instances by the hour from a cloud provider, scales up to hundreds of instances for a two-day training run (rapid elasticity), and pays only for the compute actually consumed (measured service). This convenience introduces new security considerations, since data and code now run on infrastructure managed by a third party; NIST SP 800-144 catalogs risks specific to this shared, outsourced environment — such as multi-tenant data isolation and the loss of direct physical control — and recommends security controls organizations should evaluate before migrating workloads to the cloud.