Systems of equations, span and singularity

Start away from mathematics entirely. Someone tells you three things about their pets:

• The dog is black.
• The cat is orange.

Two sentences, two facts. You now know something you did not know before, and you know it about both animals. Call that pair complete.

Now a different pair:

• The dog is black.
• The dog is black.

Two sentences, one fact. The second sentence is perfectly true and completely useless — it carries nothing the first did not already carry. Call it redundant.

And a third:

• The dog is black.
• The dog is white.

Two sentences, and now you know less than you did with one, because they cannot both be true. Call them contradictory.

Every idea in this lesson is that distinction, moved into arithmetic and then into geometry. The interesting question is never "can I solve this?" It is "do the things I have been told actually pin the answer down?"

Swap sentences for equations and nothing changes but the notation. Two unknowns, two clues:

x + y = 10
2x + y = 16

The second clue is not a rearrangement of the first, so it genuinely adds something. Subtract and you get x = 6, then y = 4. One answer, and only one.

x + y = 10
2x + 2y = 20

Here the second clue is the first one doubled. It is true, it is useless, and you are left with one real constraint on two unknowns. There are infinitely many pairs that work — (0, 10), (3, 7), (10, 0). You cannot pick one, and the correct answer is to say so.

x + y = 10
2x + 2y = 21

Now the second clue is the first one doubled on the left and not doubled on the right. Nothing satisfies both. No solution exists.

Those are the only three outcomes a system of linear equations can have: exactly one, infinitely many, or none. There is no fourth case, and there is no "two solutions".

An equation in two unknowns is a line. x + y = 10 is the set of every point whose coordinates add to ten, which is a straight line through (10, 0) and (0, 10). Solving a pair of equations means finding the points that lie on both lines at once.

Now the three outcomes are three pictures, and they are the only three arrangements two lines in a plane can take:

• They cross once. One shared point, one solution.
• They are the same line. Every point is shared — infinitely many solutions. This is the redundant clue, drawn.
• They are parallel and distinct. No shared point, no solution. This is the contradiction, drawn.

Three unknowns works the same way one dimension up. Each equation is a plane, and three planes can meet at a single point, along a whole line, across a shared plane, or nowhere at all.

This is what "the geometry of data" means in practice. A table of numbers is not just a table — it is a set of constraints with a shape, and the shape tells you whether the constraints determine anything.

Look again at the two failing pictures. The lines coincide, or the lines are parallel. In both, the lines point the same way — they have the same slope. That shared direction is the failure, and whether the result is "infinitely many" or "none" is decided afterwards, by the constants.

A system whose clues fail to point the answer down to a single point is called singular. One that does is non-singular.

The word is worth taking seriously, because it names a property of the left-hand sides alone. Both of these are singular:

x + y = 10        x + y = 10
2x + 2y = 20      2x + 2y = 21

The first has infinitely many solutions, the second has none, and they have exactly the same coefficients. Singularity does not tell you which of the two failures you have. It tells you that the clues, as clues, do not pin anything down — and that is usually the thing you needed to know.

Geometrically, a singular system is one that has lost a dimension. Two lines that should have cut the plane down to a point instead leave you a whole line, or nothing. Space collapsed.

Since singularity depends only on the coefficients, it is worth writing them down on their own. Take the system, drop the variable names and the constants, and keep the grid:

x + y = 10          [ 1  1 ]
2x + y = 16    ->    [ 2  1 ]

That grid is a matrix, and "the system is singular" and "the matrix is singular" are the same statement. The matrix is the part of the problem that decides whether an answer exists uniquely; the constants only decide what the answer is.

Read a matrix by its rows and each row is one clue. Read it by its columns and each column is one unknown's worth of coefficients. Both readings are useful, and both give the same verdict on singularity — which is a genuinely surprising fact, and one you can lean on.

This is also the point where a dataset and a system of equations stop being different objects. A table of observations, one row per example and one column per feature, is a matrix. Everything below applies to it unchanged.

Treat each row of a matrix as a vector — an arrow from the origin. Two things you are allowed to do with vectors: scale one (stretch it, shrink it, flip it through the origin with a negative scalar) and add two together. Do both, as much as you like, and the set of every point you can reach is called the span of those vectors.

The span of a single non-zero vector is the infinite line through it. The span of two vectors pointing in different directions is the entire plane — from (1, 0) and (0, 1) you can build any point at all. But the span of two vectors pointing along the same line is only that line. Adding a second vector bought you nothing.

That is the whole idea. A set of vectors is linearly dependent when one of them lies in the span of the others, and linearly independent when none of them does. Dependent rows are redundant clues wearing different clothes:

[ 1  0  0 ]
[ 0  1  0 ]
[ 1  1  0 ]   <- row 1 + row 2, exactly

The third row is inside the span of the first two, so it says nothing new, so the matrix is singular. Dependence can be subtler than "one row is a multiple of another" — any combination counts, including averages and differences — but the test never changes: can I build this row out of the others?

This is the vocabulary the rest of machine learning uses for it. Two features that are the same measurement in different units are dependent columns. A feature that is the sum of two others is a dependent column. In every case the matrix is singular and the model has no unique answer to find.

Checking for dependence by hunting for combinations gets old fast. For a square matrix there is a single number that answers it outright, called the determinant. For a two-by-two:

| a  b |
| c  d |    det = ad - bc

The rule that makes it worth knowing:

The determinant is zero exactly when the matrix is singular. Not approximately, not usually — exactly, and in both directions.

So [[1, 1], [2, 2]] has determinant 1×2 − 1×2 = 0 and is singular, while [[1, 1], [2, 1]] has determinant 1×1 − 1×2 = −1 and is not.

The geometry is what makes it stick. Draw the two rows as arrows and complete the parallelogram they form. The determinant is that parallelogram's area, with a sign recording which row is on which side. When the two rows point along the same line the parallelogram is flat — zero area — and that flatness is the collapse this lesson has been circling. A matrix with determinant zero squashes the plane onto a line, and squashed space cannot be un-squashed, which is the same reason the system has no unique solution.

In three dimensions the determinant is the volume of the box the three rows make, and it goes to zero when they flatten into a plane. The pattern holds all the way up.