Data Engineering for AI

A data pipeline, in Google Cloud's framing, fundamentally "moves data from one system to another" by reading from sources and writing to sinks, and successful pipeline design starts not with the technical stages but with defining measurable Service Level Objectives (SLOs) — targets like data freshness, correctness, and processing deadlines — before any implementation begins. Worked example: a pipeline feeding a fraud-detection model might set an SLO of "new transaction data must be available to the model within 15 minutes of occurring, with fewer than 0.1% of records dropped or malformed"; that single sentence then dictates architectural choices — whether the pipeline streams continuously or runs in scheduled batches, how much retry logic ingestion needs, and how aggressively validation must reject bad records before they reach the model.

Ingestion is the first concrete stage: pulling raw data from its source system, whether that is an API, a file drop, a database change stream, or a third-party data feed, and landing it somewhere the rest of the pipeline can process it reliably. Google Cloud's guidance notes that because a pipeline reads from external sources and writes to external sinks, the pipeline's achievable performance is directly limited by those external systems' own scalability and quota limits — a pipeline cannot promise 15-minute freshness if the upstream API it depends on only refreshes its own data hourly, so understanding a source's real update cadence and rate limits is a precondition for setting any SLO honestly rather than aspirationally.

The same guidance discusses the architectural choice between a single pipeline handling all data flows and multiple specialized pipelines, noting that when different data flows carry meaningfully different priorities or deadline requirements, separate pipelines can isolate resources so a slow, low-priority batch job cannot starve a time-critical stream of the compute or network capacity it needs to hit its own SLO. This decomposition mirrors the modular software design covered elsewhere in this domain: just as splitting a monolithic script into independently testable functions contains bugs and eases change, splitting a data platform into pipelines scoped to distinct SLOs contains performance problems and makes each pipeline's behavior easier to reason about on its own.

Once raw data has been ingested, it rarely arrives in the shape a model or downstream analysis needs directly — timestamps in inconsistent formats, missing fields, categorical labels spelled inconsistently across sources — so a transformation stage cleans, reshapes, and standardizes it before storage. Worked example: a pipeline ingesting customer support tickets from three different systems might need to transform each system's own status vocabulary ("closed," "resolved," "done") into one standardized value before the records can be safely combined into a single training set; skipping this step would leave a model treating three descriptions of the same real-world state as three different categories.

Validation is the step that decides whether transformed data is trustworthy enough to store and use, and it is where an engineering discipline explicitly named in this module's essential question — catching bad data before it reaches the model — actually happens: automated checks for missing required fields, values outside plausible ranges, unexpected duplicate records, or a sudden shift in the distribution of an important column compared to prior data. This connects directly to the practice, described in Google's Machine Learning Crash Course, of partitioning a dataset into training, validation, and test sets before model development begins: validated, clean data flowing consistently through a pipeline is the precondition for that split to mean anything, because a validation or test set contaminated with the same ingestion bugs as the training data will not actually reveal how the model performs on trustworthy, representative inputs.

Storage is the pipeline's final stage for a given run, and the right storage model is exactly the deliberate choice covered in the databases module of this domain: cleaned tabular records with a stable schema often belong in a relational or columnar warehouse table queryable by SQL, semi-structured records with variable shape may belong in a NoSQL document store, and text or image data destined for retrieval by semantic similarity may need to be embedded and written into a vector database. A data engineering pipeline, in other words, is not a separate discipline from the storage decisions covered earlier in this module sequence — it is the mechanism that gets clean, validated, well-understood data into whichever of those storage models actually fits the downstream use case.

Even a well-designed, validated pipeline is not fully trustworthy unless it also records where its data came from and what happened to it along the way — this is data provenance, which the W3C's PROV standard defines as "information about entities, activities, and people involved in producing a piece of data or thing, which can be used to form assessments about its quality, reliability or trustworthiness". PROV structures this information around three core concepts: an entity (the object being described, such as a specific version of a training dataset), an activity (a processing step that produces or transforms data, such as the transformation stage from the previous lesson), and an agent (the person, team, or automated system responsible, such as "the nightly ingestion job" or "the data engineering team").

Applied to this module's opening scenario — a model behaving strangely in production and nobody able to say where its training records came from — a pipeline that records provenance at every stage can answer exactly that question: which raw source an entity (a training record) originated from, which activity (which specific run of the ingestion and transformation pipeline, on which date, with which code version) produced its current form, and which agent was responsible for that run. Worked example: if a transformation step's status-vocabulary mapping had a bug for one week, provenance metadata recording the pipeline run and code version associated with every record lets an engineer precisely identify and remediate only the affected records, rather than distrusting the entire dataset or, worse, not being able to tell which records are suspect at all.

This traceability, often called data lineage when described as the path data traveled through a pipeline's stages, is what separates a data pipeline that merely produces output from one that produces defensible, auditable output — a distinction that matters increasingly as AI systems face scrutiny over exactly the question this module opened with: can you actually retrace, step by step, where a given piece of training data came from and how it was handled. Building provenance recording into a pipeline from the start, alongside the SLOs, ingestion, transformation, and validation stages covered earlier in this module, is what turns "we have a data pipeline" into "we have a data pipeline whose outputs we can trust and explain."