Linear Algebra for AI

A vector is simply an ordered list of numbers, and in AI it is the universal format for representing a single 'thing' as a point or direction in space: a customer might be represented as the 4-dimensional vector [age, income, purchases_last_month, days_since_signup] = [34, 52000, 3, 120], and two customers are considered similar exactly when their vectors are close together by some distance measure. A matrix bundles many such vectors — or, equivalently, defines a specific way of transforming one vector into another — and Gilbert Strang's MIT 18.06 course, one of the most widely used linear algebra courses in the world, organizes the entire subject around exactly this dual view of matrices as both data containers and transformations, alongside systems of equations, vector spaces, determinants, eigenvalues, and positive definite matrices. Concretely, multiplying a matrix by a vector combines rows of the matrix with the vector's entries: given matrix M = [[1, 2], [3, 4]] and vector v = [5, 6], the product Mv is computed row by row — the first output entry is (1×5) + (2×6) = 5 + 12 = 17, and the second is (3×5) + (4×6) = 15 + 24 = 39, so Mv = [17, 39]. Stanford's CS229 linear algebra review, a standard machine-learning reference, works through exactly this kind of matrix-vector product alongside matrix-matrix products, transpose, trace, and rank as the operations every later ML technique is built from.

A tensor generalizes this idea to any number of dimensions: a single number is a 0-dimensional tensor (a scalar), an ordered list is a 1-dimensional tensor (a vector), a grid of numbers is a 2-dimensional tensor (a matrix), and stacking matrices produces 3-, 4-, or higher-dimensional tensors. The generalization is not just notational convenience — it matches how real AI data is actually structured. A single grayscale image is naturally a 2-dimensional tensor (height × width pixel intensities); a single color image adds a third dimension for red, green, and blue channels, making it a 3-dimensional tensor (height × width × channels); and a training batch of many color images stacked together becomes a 4-dimensional tensor (batch × height × width × channels), the exact shape that convolutional neural network libraries expect as input. Every operation introduced for vectors and matrices — addition, scaling by a number, an inner product measuring similarity — extends to tensors of any dimension, which is why the vector and matrix operations covered in a linear algebra course such as MIT 18.06 or the CS229 review remain the computational foundation even for models that technically operate on much higher-dimensional tensors.

Multiplying a vector by a matrix is best understood geometrically as applying a transformation — the matrix takes every point in space and moves it somewhere else, according to a rule that is the same everywhere (a straight line stays a straight line, and the origin stays fixed). Two small 2×2 examples make this concrete. The matrix S = [[2, 0], [0, 1]] stretches space: applied to the vector [3, 4], it produces [2×3 + 0×4, 0×3 + 1×4] = [6, 4] — the x-coordinate has doubled while the y-coordinate is unchanged, so S stretches the plane by a factor of 2 along the x-axis only. The matrix R = [[0, -1], [1, 0]] rotates space: applied to the same vector [3, 4], it produces [0×3 + (-1)×4, 1×3 + 0×4] = [-4, 3], which is exactly [3, 4] rotated 90 degrees counterclockwise around the origin. These examples generalize into a central fact of the subject: every matrix defines some combination of stretching, rotating, reflecting, and shearing space, and the algebraic rules for combining, inverting, and composing these transformations — covered in depth in MIT 18.06's treatment of systems of equations and vector spaces — are what let a neural network's sequence of matrix multiplications be reasoned about layer by layer rather than only as an opaque whole.

For most matrices, a transformation changes the direction of most vectors — but for every matrix there are usually a few special directions that the transformation only stretches or shrinks, without rotating them at all; these are the matrix's eigenvectors, and the amount of stretching along each one is its eigenvalue, formalized by the equation Av = λv, where A is the matrix, v is the eigenvector, and λ (a plain number) is the eigenvalue. Returning to the earlier stretching matrix S = [[2, 0], [0, 1]], the vector [1, 0] satisfies S[1,0] = [2, 0] = 2×[1,0], so [1, 0] is an eigenvector of S with eigenvalue 2; likewise [0, 1] satisfies S[0,1] = [0,1] = 1×[0,1], an eigenvector with eigenvalue 1 — exactly the two directions the matrix was built to stretch by 2 and by 1, respectively, made visible by the eigenvalue computation. Eigenvalues and eigenvectors, covered for both general and symmetric matrices in the CS229 linear algebra review and central to MIT 18.06's units on eigenvalues, similarity, and positive definite matrices, are what let a researcher summarize a large, complicated matrix by its handful of most important directions and scales — the mathematical basis for techniques such as principal component analysis, which finds the eigenvectors of a dataset's covariance matrix to identify the directions of greatest variation in the data.

Matrix decomposition takes this idea further by rewriting an entire matrix as a product of simpler pieces chosen specifically to expose its eigenvalues, eigenvectors, or other structure — the same way factoring an integer into primes exposes its divisibility structure. The CS229 linear algebra review develops this machinery through quadratic forms, positive semidefinite matrices, and 'optimization via eigenvalue decomposition,' showing how decomposing a matrix into its eigenvalues and eigenvectors turns an optimization problem that looks intractable in the original coordinates into one that can be solved coordinate by coordinate in the transformed space. A practical payoff appears in dimensionality reduction: if a dataset's covariance matrix is decomposed and only the handful of eigenvectors with the largest eigenvalues are kept, the original high-dimensional data can be projected onto just those directions, discarding the low-eigenvalue directions that contribute little to the data's actual variation, compressing the representation while preserving most of the structure a model needs. The same decomposition machinery, applied to the weight matrices of a trained neural network, gives researchers one of their few genuinely quantitative tools for inspecting what a model has learned, rather than treating it purely as a black box.