Feature Engineering and Representation

Machine learning models do not operate on raw spreadsheet rows; they operate on feature vectors, arrays of floating-point numbers, and Google's Machine Learning Crash Course notes that raw dataset values first pass through a process called feature engineering before a model ever sees them, converting a mix of numbers, categories, and text into a consistent numeric representation. A worked example: a house-sales row with square footage of 1,800, 3 bedrooms, and the neighborhood name "Maple Heights" cannot be handed to most algorithms as a mix of numbers and text; square footage and bedroom count can be used directly as numbers, but the neighborhood name must first be converted into some numeric encoding before the row becomes a valid feature vector the model can process. The same row might also include a raw sale date like "2024-03-14," which is similarly unusable as-is until it is engineered into something numerically meaningful, such as days since the start of the dataset or a separate month-of-year feature capturing seasonal effects on sale price.

Even when a feature is already numeric, its scale can distort learning: square footage might range from 500 to 5,000 while bedroom count ranges from 1 to 6, and many algorithms — especially those relying on gradient-based optimization or on measuring distances between points — will let the larger-scale feature dominate simply because its raw numbers are bigger, not because it is more informative. Standardization addresses this by rescaling each feature to have zero mean and unit variance; scikit-learn's preprocessing documentation notes that many estimators "might behave badly if the individual features do not more or less look like standard normally distributed data," motivating tools like its StandardScaler. A worked example: after standardizing, a 1,800-square-foot house and a 3-bedroom house are both expressed on a comparable scale (for instance, as the number of standard deviations each falls from the average house in the dataset), so neither feature mechanically overwhelms the other purely due to its raw numeric range.

Categorical features like a neighborhood name require encoding rather than scaling. One-hot encoding creates a separate binary column for each possible category — a "Maple Heights" column that is 1 for that row and 0 otherwise, and similarly for every other neighborhood — which scikit-learn's preprocessing module supports through tools such as its OneHotEncoder, alongside alternatives like ordinal encoding for categories that have a natural order. One-hot encoding works well for a modest number of categories, but for a feature with thousands of possible values — such as ZIP code or product ID — it produces an unwieldy number of mostly-zero columns; in such high-cardinality cases, practitioners often turn instead to learned embeddings, discussed in the next lesson, which represent each category as a small dense vector learned from data rather than a long sparse one-hot vector. Ordinal encoding is a middle option worth distinguishing from both: it assigns a single integer to each category, which is appropriate for genuinely ordered categories like "small, medium, large," but misleading for an unordered category like neighborhood name, where an arbitrary integer assignment would falsely imply that some neighborhoods are numerically "between" others.

Not every feature that can be constructed should be kept: a model with hundreds of redundant or irrelevant features can be slower to train, harder to interpret, and more prone to fitting noise rather than genuine patterns. Feature selection is the practice of choosing a useful subset of features, and scikit-learn's preprocessing and feature-selection tools support several strategies, including removing features with very low variance (which carry little information) and selecting features most correlated with the target outcome. A worked example: if a house-sales dataset includes both "square footage" and "square meters" as separate columns, the two are perfectly correlated and redundant — keeping both adds no new information while doubling the risk that a model treats the same underlying signal as two independent pieces of evidence, so one would typically be dropped. Low-variance features pose a related problem: a column recording "country" in a dataset where every single row happens to be from the same country carries no discriminative information at all for that dataset, however important country might seem conceptually, and can safely be removed before training.

Beyond selecting among existing features, transformation reshapes a feature's values to be more useful — the log transformation discussed for skewed variables is one example, and scikit-learn also documents polynomial feature expansion, which creates new features by multiplying existing ones together (for instance, square footage times number of bedrooms) to let a model capture interaction effects it otherwise could not represent directly. When a dataset has many correlated numeric features — for example, a sensor array recording 50 highly correlated temperature readings across a factory floor — dimensionality reduction techniques such as principal component analysis (PCA) can compress that redundancy into a much smaller number of new, uncorrelated features (principal components) that still capture most of the original information, a technique covered in Stanford CS229's course materials on unsupervised learning and dimensionality reduction. A worked example: reducing 50 correlated sensor readings down to 3 principal components that together explain 95 percent of the readings' variance lets a model train on 3 features instead of 50, with minimal loss of information, and often makes downstream training noticeably faster besides.

Not all useful representations are hand-designed: deep learning models can learn their own features directly from raw or lightly processed data, an approach sometimes called representation learning. Rather than a person deciding that "neighborhood" should be one-hot encoded, an embedding layer learns a dense vector for each neighborhood automatically during training, positioning neighborhoods with similar sale-price patterns near each other in the learned vector space — conceptually similar to how word embeddings in natural language processing place semantically related words near one another. Google's crash course material on feature vectors notes that this progression — from raw values, to hand-engineered numeric features, toward representations a model learns for itself — is a recurring theme in how machine learning systems are built. The practical trade-off is that hand-engineered features are easier to inspect and reason about, while learned embeddings can capture subtler patterns but are harder to interpret directly, which is why many production systems combine both — using hand-engineered features where transparency matters and learned embeddings for high-cardinality categories where manual encoding would be impractical.