AI Engineering, MLOps, Applications, and Capstone Research

A model that performs well in a research notebook is not yet a system anyone can rely on. Sculley et al.'s widely cited 2015 paper, "Hidden Technical Debt in Machine Learning Systems," makes a specific, deliberately uncomfortable argument about why: in a real-world ML system, only a small fraction of the overall codebase is the actual machine learning code — the model definition and training loop — while a much larger volume of surrounding infrastructure handles data collection, feature extraction, serving, and monitoring, and it is this surrounding system where maintenance cost accumulates fastest as technical debt. Two specific mechanisms the paper describes are worth carrying forward into any deployment plan. Entanglement, which the paper summarizes with the shorthand CACE — "Changing Anything Changes Everything" — describes how, because ML systems mix input signals together through learned weights rather than isolating them behind clean interfaces, altering one input feature, upstream model, or data source can silently shift behavior throughout the whole system in ways that are hard to trace back to a single cause. Hidden feedback loops describe a related but distinct problem: a system's own outputs can influence the very data it is later trained or evaluated on — for example, a recommender system's picks shaping what users click, which then becomes its next round of training data — creating gradual, hard-to-detect drift that a one-time offline evaluation would never catch.

Paleyes, Urma, and Lawrence's survey of deployment case studies documents how these abstract debt categories show up in practice. Reported challenges cluster around data — data quality problems and a distribution shift between the data a model was trained on and the data it actually encounters in production — and around a persistent gap between offline research metrics, such as accuracy on a fixed held-out test set, and the requirements that actually matter once a system is live, including latency, cost per inference, and robustness to inputs the test set never covered. The survey also treats deployment explicitly as an ongoing process rather than a one-time event, which is why standard MLOps practice pairs a launch with concrete operational habits: versioning data and models together so any prediction can be traced back to exactly what produced it and rolled back if needed; staged rollouts, such as canary releases that expose a new model to a small slice of traffic before a full rollout, or shadow deployments that run a new model silently alongside the current one to compare their outputs before anyone trusts the new one's decisions; and monitoring dashboards that track both system health, like latency and error rate, and model-quality signals, like shifts in the distribution of predictions, since a model can keep returning fast, well-formed answers while quietly becoming wrong.

Training a modern foundation model at scale is not something a single machine can do in reasonable time: the model's parameters and the batch of training data it needs to process together typically exceed what one accelerator's memory and compute can handle, which is why large training runs are distributed across many GPUs or TPUs. Two complementary strategies handle this at a conceptual level. Data parallelism keeps a full copy of the model on every device and splits the training data across them, with each device computing gradients on its own shard before those gradients are combined; this scales well as long as the model itself still fits on one device. Model, or pipeline, parallelism instead splits the model's layers or parameters across multiple devices, which becomes necessary once a model is simply too large for any single accelerator's memory, regardless of how the data is split. Patterson et al.'s 2021 analysis of the energy and carbon costs of training several large neural networks examined the resource side of this scaling directly, and a central point of their analysis is that the resulting footprint is not a fixed cost of scale alone — it depends heavily on choices practitioners can actually influence, including how computationally efficient the model architecture and training procedure are, which hardware is used, and the energy mix of the datacenter where the training run takes place.

The same efficiency logic that governs training compute applies, arguably with even more direct financial consequences, to serving a model in production, since a deployed system is queried repeatedly rather than trained once. Consider a simple illustrative estimate, not tied to any specific published figures, just to make the scaling concrete: suppose a deployed model serves 2,000,000 inference requests per day, and each request costs $0.0015 in compute. Daily compute cost is 2,000,000 × $0.0015 = $3,000, and over a 30-day month that is $3,000 × 30 = $90,000. Now suppose an engineering effort — through techniques like model quantization or distillation, which are themselves active areas of efficient deep learning research — cuts the average cost per request in half, to $0.00075. The same monthly workload now costs $45,000, exactly half. This kind of small-looking per-request change compounds directly at production scale, which is why inference efficiency, and not just training-time efficiency, is treated as a first-class engineering concern once a system is actually serving traffic, and why cost sits alongside accuracy as a genuine product requirement rather than an afterthought.

None of these engineering tradeoffs are made in a vacuum; they are shaped by what the system is actually for. Turning a trained model into a product requires decisions the model alone cannot make: what latency budget the user experience can tolerate, what should happen when the model is uncertain or simply wrong — falling back to a human reviewer, a simpler rule-based system, or an explicit refusal rather than a confident guess — and how the application's domain reshapes the entire pipeline, since a model deployed in healthcare or finance needs far more conservative failure handling, audit trails, and human oversight than one recommending a video. This is also why the fairness, robustness, and transparency considerations that responsible-AI practice raises are not a separate concern layered on top of engineering after a model is otherwise "done": they are inputs to the very same latency, cost, and reliability tradeoffs a team makes at every stage of shipping a real system.

Deployment challenges assume a model that actually works as reported, and that assumption is not always safe. Pineau et al.'s 2021 report from the NeurIPS 2019 reproducibility program describes a concrete, field-wide effort to address this: the authors introduced a machine learning reproducibility checklist that paper submissions were asked to complete, covering practical questions such as whether code is released, whether exact train/test splits and hyperparameters are specified, and how many random seeds or repeated runs a reported result is averaged over, and then studied whether attaching this checklist to the review process changed what authors actually reported. Their findings pointed to specific, fixable causes of irreproducible results that recur across the field — incomplete reporting of hyperparameters and compute budgets, unspecified hardware or software versions that quietly change results, and single-run numbers reported without any indication of variance across seeds. The report treats reproducibility explicitly as an ongoing practice-improvement effort rather than a solved problem, which is a fitting note on which to close a technical arc that began, many modules ago, with reasoning about how confident a claim in AI actually deserves to be.

That arc is worth naming directly, because this module is where it converges. The major opened with the classical, symbolic foundations of AI and with probabilistic reasoning as a way to represent uncertainty formally; it moved through supervised and deep learning as a way to fit flexible models to data at scale; it built language and vision systems on top of those foundations; it introduced reinforcement learning, agents, and robotics as ways for a system to act and not just predict; it arrived at foundation models and the instruction-tuning, preference-learning, retrieval, and tool-use techniques that adapt a pretrained model into something usable and, ideally, aligned with what people actually want; and it has now reached the engineering discipline — MLOps, hardware and compute economics, reproducibility — that decides whether any of that research ever reaches a person who benefits from it. No single technique studied across the major is inherently "the" solution to a real-world problem; each is a tool with known strengths, known failure modes, and a cost profile, and a practitioner's actual job is choosing among them, combining them, and being honest, through monitoring and evaluation, about how that choice performs once it is out of the notebook.

A strong capstone project makes that same set of judgment calls explicitly, on a problem the learner chooses. It should state a clear problem naming a specific user and task, not a vague "use AI for X"; a data plan describing what data the project would need, where it could plausibly come from, and what gaps or biases that data likely has; a model or architecture choice justified against at least one real alternative studied somewhere in this major, with explicit reasoning about the tradeoff rather than a default pick; an evaluation plan naming the metric or metrics that would actually indicate success, including at least one consideration beyond raw accuracy, such as latency, fairness across subgroups, or factuality; a deployment and monitoring plan naming one concrete signal that would be watched after launch and what would trigger a retrain or a rollback; and an explicit ethical or responsible-AI check identifying the single most likely way the system could cause harm and a concrete mitigation for it. The capstone that follows is a design exercise in making exactly these choices, not a requirement to actually train or ship anything.