Data Preparation and Exploratory Analysis

Real-world datasets are rarely ready to feed directly into a model; Google's Machine Learning Crash Course describes the process of detecting and fixing such problems as "scrubbing," and notes that a program can be written to detect omitted values, duplicate examples, and out-of-range feature values before training begins. Missing data is one of the most common issues: a patient record with a blank age field can either be dropped from the dataset (if missingness is rare) or imputed — filled in with a reasonable estimate, such as the median age of similar patients — so that no information is discarded unnecessarily. The choice matters: dropping too many rows can shrink a dataset past the point of being representative, while naive imputation can quietly distort the very patterns a model is meant to learn, so the method chosen should be documented and justified for the specific dataset at hand and revisited if the proportion of missing values in a column turns out to be unusually high.

Outliers are values that lie far outside the typical range of a variable and can be genuine (a very high but real income) or erroneous (a sensor glitch or data-entry mistake). The NIST/SEMATECH e-Handbook of Statistical Methods treats identifying such unusual observations as a core purpose of exploratory data analysis, using systematic techniques rather than eyeballing individual numbers. A common rule of thumb is the interquartile range (IQR) method: compute the 25th and 75th percentiles of a variable, and flag any value more than 1.5 times the IQR beyond either boundary as a potential outlier. A worked example: in a dataset of sensor temperature readings mostly between 15 and 30 degrees, a single recorded value of 9999 is almost certainly a placeholder for a missing or failed reading rather than a real temperature, and the IQR rule would flag it immediately for investigation rather than allowing it to silently distort any statistic computed from the column, such as a mean.

Duplicate records are another quiet source of error, especially when data is merged from multiple sources — for example, combining a hospital's intake system with its billing system might create two records for the same patient visit with slightly different formatting. Google's crash course material on scrubbing explicitly lists duplicate examples alongside omitted values and out-of-range values as a problem to detect programmatically, since duplicates silently give some observations more influence over a model's learned patterns than others, biasing results toward whatever happened to be entered twice. A worked example: if 200 of ten thousand patient records are accidental duplicates of the same 100 patients, any statistic — like the average length of hospital stay — will be skewed toward those 100 patients' particular characteristics unless the duplicates are identified and removed. Consistency checks — verifying that a supposed unique identifier really is unique, and that categorical fields use consistent spelling — catch this class of error before it reaches a model.

Exploratory data analysis (EDA) is the practice of examining a dataset's structure, distributions, and relationships before formal modeling, and the NIST/SEMATECH e-Handbook presents it as an approach that emphasizes graphical techniques to gain insight into data, rather than relying purely on a small number of summary statistics. A single summary number can hide a great deal: two variables can have the same mean and standard deviation while having entirely different shapes, and a histogram or scatterplot reveals this immediately where a table of statistics would not. This graphical emphasis is deliberate: rather than assuming a dataset fits a particular textbook distribution and computing statistics that presuppose that shape, EDA starts by looking, so that any unusual structure — skew, multiple peaks, or unexpected gaps — is caught before it silently undermines a later modeling choice that assumed the data looked simpler than it actually does. A worked example: a histogram of patient ages that appears bimodal — with peaks around age 30 and age 70, and few patients in between — visually suggests two distinct subpopulations (perhaps maternity patients and geriatric patients) that may need to be modeled or sampled differently, a pattern a single "average age" statistic would completely conceal.

Many variables in real data are skewed — a small number of very large values pull the distribution's tail out, as with income, hospital charges, or word frequencies — and this skew can make patterns harder to see and can violate assumptions that some downstream statistical or modeling techniques rely on. A log transformation (replacing each value with its logarithm) compresses large values proportionally more than small ones, often turning a heavily right-skewed distribution into something closer to symmetric and easier to visualize and model; this is a standard technique covered in both the NIST handbook's discussion of data transformation and in scikit-learn's preprocessing documentation, which provides tools such as power and quantile transformers for exactly this purpose. A worked example: raw household income data, where most households earn under $80,000 but a few earn over $2,000,000, produces a histogram dominated by one enormous bar and a long empty tail; after a log transform, the same data spreads out into a roughly bell-shaped curve that is far easier to inspect for outliers or subgroups.

Beyond individual variables, EDA looks for relationships between variables — patterns that later inform feature engineering and modeling choices. A correlation matrix or a grid of pairwise scatterplots can reveal, for instance, that two supposedly independent sensor readings are almost perfectly correlated (suggesting redundancy) or that a variable expected to predict an outcome shows no visible relationship to it at all. The NIST handbook frames this pattern-discovery step as central to EDA's purpose: uncovering underlying structure, detecting anomalies, and testing assumptions before committing to a particular model. A worked example: plotting hospital length-of-stay against patient age might reveal a clear upward trend for patients over 60 but no relationship at all for younger patients, suggesting that age should be represented as a nonlinear or segmented feature rather than treated as a single linear predictor. Noticing this kind of relationship during EDA, before any model is trained, is what allows a practitioner to make an informed feature-engineering decision later rather than discovering the same limitation only after a linear model has already underperformed.