Sean's Blog

Linear Algebra: Matrices Move Space

November 17, 2026
Edit on GitHub

Most people pass linear algebra without learning what a matrix does. They learn to multiply rows by columns, to expand determinants and to solve det(A − λI) = 0. They get the right numbers and no picture. The picture is simple: a matrix moves space (by rotating and stretching vectors). It takes every arrow in the plane and sends it somewhere else, and it keeps straight lines straight while doing so. Once you see that, the rest of the subject turns into questions about motion. How much does it stretch areas? Can you undo it? Which directions does it leave alone?

The same picture runs much of machine learning. A dataset is a matrix. Fitting a line means dropping a perpendicular. Finding the main trends in data means finding the directions a matrix only stretches.

Part 1: vectors, and matrices as motions of the plane.

Part 2: eigenvectors, the directions a matrix doesn't turn, and why they make hard matrices easy.

Part 3: linear regression, PCA and k-means, seen as geometry.

Vectors

A vector is an arrow from the origin. In two dimensions you write it as a pair of numbers, v = (x, y): go x to the right and y up. You can do two things with arrows. Add them (put them tip to tail) and scale them (make them longer, shorter, or flip them around). That is all linear algebra allows, and it is enough.

A sum of scaled vectors, like 2a − 3b, is a linear combination. All the linear combinations of some vectors form their span. Two arrows that don't lie on the same line span the whole plane: every point is some mix of them. Such a pair is a basis. The standard basis uses î = (1, 0) and ĵ = (0, 1), so v = (x, y) really means x·î + y·ĵ. The coordinates of a vector tell you how much of each basis vector to take. Pick a different basis and the same arrow gets different coordinates.

The dot product a · b = a₁b₁ + a₂b₂ measures how much two arrows agree. It equals |a| |b| cos θ, where θ is the angle between them. Same direction: large and positive. Right angle: zero. Opposite: negative. Divide by the two lengths and you get cosine similarity, the score that vector databases use to compare embeddings. The dot product also gives you projection: the shadow of b on the line through a unit vector u has length b · u. Projection comes back three times in the machine learning part.

Matrices Move Space

Here is the key fact. A linear map must respect sums and scaling: A(v + w) = Av + Aw and A(cv) = c·Av. So once you know where î and ĵ land, you know where every vector lands. The vector x·î + y·ĵ has to land on x·(new î) + y·(new ĵ).

A 2 × 2 matrix stores exactly those two landing spots, as its columns:

A = [ab ; cd]   sends   î → (a, c)   and   ĵ → (b, d)

Matrix times vector is then no mystery:

Av = x·(a, c) + y·(b, d) = (ax + by, cx + dy)

The row times column rule from school is this sum, written out. Seen as a picture, the grid of the plane moves, but its lines stay straight, parallel and evenly spaced, and the origin stays put. Drag the tips of î and ĵ below, or pick a preset and slide from "nothing happens" to the full map.

Drag the blue and orange handles. The columns of the matrix are where î and ĵ land.

Matrix
Determinant ad − bc
What it does

The faint grid is the plane before the move. The shaded square is the unit square spanned by î and ĵ. It turns orange when the map flips the plane over.

The Determinant Measures Area

Watch the shaded unit square. It starts with area 1. After the map it becomes a parallelogram with area |ad − bc|. Every other shape grows or shrinks by the same factor, since you can tile any shape with tiny squares. That factor is the determinant:

det A = ad − bc

A negative determinant means the map flips the plane, like a mirror: the F turns into its mirror image. A determinant of zero means the map squashes the plane onto a line or a point. Try the squash preset. Whole lines of points now land on the same spot, so no map can send them back. That is why a matrix has an inverse exactly when its determinant is not zero.

Multiplying Matrices Chains Moves

Do one map, then another. The result is again a linear map, and its matrix is the product. BA means "first A, then B", read right to left as in f(g(x)). The odd rule for multiplying matrices follows from this: column j of BA is B applied to column j of A, which is where A sent the j-th basis vector.

Order matters. Rotate then shear gives a different picture than shear then rotate, so AB ≠ BA in general. The inverse A⁻¹ undoes A, so A⁻¹A = I, the identity map that leaves everything alone.

Solving Equations

Ax = b asks: which input lands on b? If det A ≠ 0, exactly one does, x = A⁻¹b. If A squashes the plane onto a line, two things can happen. If b lies off that line, no input works. If b lies on it, infinitely many do, because a whole line of inputs (the null space) lands on 0, and you can add any of them to a solution. The rank counts the dimensions that survive the map: 2 onto the plane, 1 onto a line, 0 onto a point. A system with a million unknowns has the same three outcomes: one solution, none, or infinitely many.

Eigenvectors: Directions That Don't Turn

Follow a single arrow while a matrix moves it. Most arrows get turned to a new direction and stretched. A few special ones only get stretched. The matrix leaves their direction alone. Those are the eigenvectors, and the stretch factor λ is their eigenvalue:

Av = λv

The equation says that, on this one arrow, the whole matrix acts like multiplying by a single number. Drag v below and look for the spots where v and Av line up.

Drag v (blue) or click anywhere. Av (orange) shows where the matrix sends it. When the two line up, both turn green: you found an eigenvector.

v → Av
Status
Matrix
Eigenvalues

Dashed green lines mark the eigenvector directions. The dashed circle holds every vector as long as v. The orange ellipse is where the matrix sends that circle.

Take A = [[2, 1], [1, 2]]. Its eigenvectors are (1, 1) with λ = 3 and (1, −1) with λ = 1. Check: A(1, 1) = (3, 3) and A(1, −1) = (1, −1). This matrix stretches one diagonal by 3 and leaves the other diagonal as it is. Those two facts describe the whole matrix. Every vector is a mix of the two eigenvectors, and the matrix just scales each part. That is why the circle turns into an ellipse whose axes lie along the eigenvectors.

How do you find them by hand? Av = λv means (A − λI)v = 0. A nonzero v can only land on 0 if A − λI squashes the plane, so its determinant must be zero:

det(A − λI) = λ² − (a + d)λ + (ad − bc) = 0

Now the old recipe has a meaning. You look for the stretch factors that make A − λI flatten some direction, and that flattened direction is the eigenvector. The two roots multiply to the determinant and add up to the trace a + d.

Not every matrix has real eigenvectors. A rotation turns every arrow, so no direction stays put. Its eigenvalues come out as complex numbers, which fits, since complex numbers describe rotations. A shear keeps only one direction: it slides the plane along the x axis. And [[1, 2], [2, 1]] has λ = −1 along (1, −1). A negative eigenvalue flips that direction, so Av points the opposite way.

Symmetric matrices (equal to their own transpose, A = Aᵀ) behave best. The spectral theorem says their eigenvalues are real and their eigenvectors stand at right angles to each other. Compare the symmetric preset with the lopsided one above, whose eigenvectors meet at a slant. This fact will carry PCA later.

Diagonalization: Change the Basis, Keep Only Stretches

If a matrix has enough eigenvectors to form a basis, describe it in that basis. There it only stretches. Put the eigenvectors in the columns of P and the eigenvalues on the diagonal of D. Then the matrix is diagonalizable:

A = P D P⁻¹

Read it right to left. P⁻¹ turns the eigenvectors onto the axes. D stretches each axis by its eigenvalue. P turns everything back. Three easy moves replace one confusing one. Step through them below and compare with applying A directly.

Step through A = P D P⁻¹ one factor at a time. Green and purple are the two eigenvectors.

Stage
Total map so far
D (eigenvalues)
P (eigenvectors as columns)
(1, 1)/√2 and (−1, 1)/√2

Why bother? Because diagonal matrices are easy. Ak = P Dk P⁻¹, and Dk just raises each eigenvalue to the k-th power. Apply A a hundred times and the direction with the largest |λ| wins, since 3100 dwarfs 1100. Start with almost any vector, keep multiplying by A, and it swings toward the top eigenvector. This trick, power iteration, is how Google first computed PageRank. It also explains growth in recurrences: the Fibonacci numbers grow like φn because φ ≈ 1.618 is the largest eigenvalue of [[1, 1], [1, 0]].

SVD: Every Matrix Is Rotate, Stretch, Rotate

What about matrices that aren't square, or lack a full set of eigenvectors? The singular value decomposition works for every matrix:

A = U Σ Vᵀ

Every linear map is a rotation (Vᵀ), then a stretch along the axes (Σ), then another rotation (U). Reflections may sneak into the rotations. The stretch factors σ on the diagonal of Σ are the singular values. You saw this in the eigen demo: the circle always turns into an ellipse, even for the shear and the rotation. The singular values are the half lengths of that ellipse.

Keep only the few largest singular values and you get the best low rank copy of the matrix. This compresses images, fills in missing ratings in recommender systems, and inspired LoRA, which fine tunes large language models by learning a small low rank update instead of the full weight matrix.

Machine Learning Is Geometry

A dataset is a matrix. Put one sample per row and one feature per column, so n houses with d features each give an n × d matrix X. Each row is a point in d dimensional space. Machine learning then asks questions about that cloud of points. Which line runs through it? In which directions does it spread? Where are its clumps? Three classic methods answer these three questions, and each one reuses the tools above.

Linear Regression: Drop a Perpendicular

Linear regression fits a line y = wx + b through points. Which line? Least squares, which Legendre and Gauss worked out around 1805, picks the line with the smallest sum of squared errors. Each error is the vertical gap between a point and the line. Draw each squared error as a real square, and the best line is the one with the least total orange area. Try to beat it.

Drag points, click empty space to add one, double click a point to remove it. Switch to "Your line" and drag its two green handles to try to beat least squares.

Line shown
Its squared error
Least squares error
R² of best fit

For one input the best slope has a short formula, w = cov(x, y) / var(x). The linear algebra view is better, because it works for any number of features. Stack the inputs in X (plus a column of ones for b) and the targets in y. We want Xw = y. With more points than unknowns, no exact solution exists: y lies outside the set of vectors that Xw can reach, the column space of X. So we settle for the closest point we can reach, the projection of y onto the column space. At that point the error vector stands at a right angle to every column, Xᵀ(y − Xw) = 0. Rearranged, these are the normal equations:

XᵀX w = Xᵀy

Fitting a line is dropping a perpendicular, just in n dimensions instead of two. Now press "Add outlier". One far point adds one huge square, so it drags the whole line toward itself. Squares punish big misses hard. When X gets too large to solve directly, gradient descent walks downhill on the same squared error and lands on the same answer.

PCA: Find the Axes of the Cloud

Principal component analysis asks a different question: along which directions does the data spread most? First center the data by subtracting the mean. Then build the covariance matrix:

C = XᵀX / (n − 1)

With two features, C holds the variance of x and of y on its diagonal and their covariance off it. The variance of the data along any unit direction u is uᵀCu. We want the u that makes it largest. C is symmetric, so the spectral theorem applies. Its eigenvectors stand at right angles, and the top one is the best direction. Its eigenvalue is the variance along it. The second eigenvector catches the most spread of what remains, and so on. These are the principal components: a new set of axes that fit the data.

Turn and squeeze the cloud with the sliders. The arrows are the eigenvectors of the covariance matrix, each two standard deviations long.

Covariance matrix C
Eigenvalues (variances)
Variance kept by PC1
PC1 direction

Green: PC1, the direction of most spread. Orange: PC2. The ellipse spans two standard deviations along each.

Projecting onto the top k components gives the best k dimensional summary of the data. It keeps the most variance and moves the points the least, in squared distance. That is how you plot data with a thousand features in two dimensions, strip noise, or shrink inputs before a slower method. In practice you compute it with the SVD of the centered X: the right singular vectors are the components, and the squared singular values divided by n − 1 are the variances. A linear autoencoder trained to compress and rebuild data ends up learning the same subspace.

Turn on the regression line and compare. Regression measures errors straight up and down, because it predicts y from x. PCA measures them at right angles to the line, because it treats both features alike. The two lines agree when the cloud is thin. Make it wider and turn it toward upright, and they part ways: the regression line stays flatter than the cloud.

k-means: Split Space by Distance

k-means looks for clumps. Pick k centers, then repeat two steps:

  1. Assign: give each point to its nearest center. This cuts the plane into regions, one per center, with straight borders between them: a Voronoi diagram.
  2. Update: move each center to the mean of its points.

Neither step can raise the total squared distance from points to their centers, so the loop has to settle. The mean is no accident: of all points in space, the mean has the smallest sum of squared distances to a group. Stuart Lloyd wrote the method down at Bell Labs in 1957.

Step through k-means or let it play. Drag a center to move it, and watch the shaded regions follow.

Iteration
Next step
Total squared distance
Status

Gray points have no cluster yet. Dashed trails show the path of each center.

Press "New centers" a few times, or drag a center into a bad spot. k-means stops at a local best, not always the global one. In practice you run it several times, or pick spread out starting centers (k-means++). It also assumes round clumps of similar size, because "nearest" uses plain distance. For stretched or curved clusters, run PCA first or use spectral clustering, which runs k-means on the eigenvectors of a graph.

The Same Pieces, Stacked

Neural networks stack these pieces. A layer computes σ(Wx + b): a matrix moves the input, and a simple bend σ (such as ReLU, max(0, x)) breaks the straight lines. Without the bend, a hundred layers would collapse into one matrix, since a product of matrices is just another matrix. Training runs on the chain rule, which multiplies one Jacobian matrix per layer. Embeddings, attention and vector search all come down to dot products between long vectors. The geometry of those vectors is a subject of its own, see the structure of neural latent space.

A matrix moves space. Its determinant says how areas change, its eigenvectors say which directions it keeps, and its SVD says that every such move is rotate, stretch, rotate. Least squares drops a perpendicular, PCA finds the axes of an ellipse, and k-means splits space by distance. The rest is the same moves in more dimensions.

References

  1. 3Blue1Brown: Essence of linear algebra
  2. Gilbert Strang: MIT 18.06 Linear Algebra
  3. Immersive Linear Algebra
  4. Deisenroth, Faisal, Ong: Mathematics for Machine Learning
  5. Jonathon Shlens: A Tutorial on Principal Component Analysis
  6. scikit-learn: K-means clustering