Rules for matrix multiplication, and the identity
For numbers $(a+b)^{2}=a^{2}+2ab+b^{2}.$ What is $(\mathbf{A}+\mathbf{B})^{2}$ for square matrices?
Addition of matrices behaves exactly like addition of numbers, and Lecture 12 said so. Multiplication does not, so its rules have to be checked rather than assumed.
Recap From Lecture 12: when a product exists, and what it is
If $\mathbf{A}$ is $m\times n$ and $\mathbf{B}$ is $n\times p,$ the product $\mathbf{A}\mathbf{B}$ is the $m\times p$ matrix whose $(i,k)$ entry is row $i$ of $\mathbf{A}$ times column $k$ of $\mathbf{B}{:}$ $$ \begin{aligned} c_{ik}&=a_{i1}b_{1k}+\cdots+a_{in}b_{nk}\\ &=\sum_{j=1}^{n}a_{ij}b_{jk} \end{aligned} $$ (Definition 12.17). The orders must meet in the middle, and the ends give the answer: $(m\times n)(n\times p)$ is $m\times p,$ and the product does not exist when the inner numbers differ (Watch out 12.18). So the order of the factors matters before a single entry is computed.
Let $\mathbf{A},\mathbf{B},\mathbf{C}$ be matrices and $\alpha$ a scalar. Whenever the orders are such that the products written down exist, $$ \begin{array}{l} (\mathbf{A}\mathbf{B})\mathbf{C} = \mathbf{A}(\mathbf{B}\mathbf{C})\,,\\ \quad\text{(associative law)}\\[4pt] \mathbf{A}(\mathbf{B}+\mathbf{C}) = \mathbf{A}\mathbf{B}+\mathbf{A}\mathbf{C}\,,\\ \quad\text{(left distributive law)}\\[4pt] (\mathbf{A}+\mathbf{B})\mathbf{C} = \mathbf{A}\mathbf{C}+\mathbf{B}\mathbf{C}\,,\\ \quad\text{(right distributive law)}\\[4pt] (\alpha\mathbf{A})\mathbf{B} = \mathbf{A}(\alpha\mathbf{B}) = \alpha(\mathbf{A}\mathbf{B})\,. \end{array} $$
Entry by entry. Each rule is proved by comparing corresponding entries. For the associative law, take $\mathbf{A}=(a_{ij})_{m\times n},$ $\mathbf{B}=(b_{jk})_{n\times p},$ $\mathbf{C}=(c_{kl})_{p\times q};$ both sides are then $m\times q,$ and $$ \begin{aligned} \bigl[(\mathbf{A}\mathbf{B})\mathbf{C}\bigr]_{il} &= \sum_{k=1}^{p}\Bigl(\sum_{j=1}^{n} a_{ij}b_{jk}\Bigr)c_{kl}\\ &= \sum_{j=1}^{n} a_{ij}\Bigl(\sum_{k=1}^{p} b_{jk}c_{kl}\Bigr)\\ &= \bigl[\mathbf{A}(\mathbf{B}\mathbf{C})\bigr]_{il}\,, \end{aligned} $$
The same numbers, added in another order. The two sides agree, since both double sums add up the same $np$ numbers $a_{ij}b_{jk}c_{kl},$ in a different order.
Two distributive laws are stated where numbers need only one, because $\mathbf{A}(\mathbf{B}+\mathbf{C})$ and $(\mathbf{B}+\mathbf{C})\mathbf{A}$ are different expressions.
Associativity is what makes $\mathbf{A}\mathbf{B}\mathbf{C}$ unambiguous, so brackets may be dropped; and for a square $\mathbf{A}$ it makes the powers $\mathbf{A}^{2}=\mathbf{A}\mathbf{A},$ $\mathbf{A}^{3}=\mathbf{A}\mathbf{A}\mathbf{A},$ …, $\mathbf{A}^{n}$ well defined without brackets either.
The practical value of Fact 13.1 is that identities can be established by manipulating whole matrices rather than by inspecting $mn$ entries.
Guess first. The example below expands a square with these rules. Answer the question before you read it.
Pick an answer before reading on. Getting it wrong is fine — that is how you find out what you believe.
$\mathbf{A}$ and $\mathbf{B}$ are square, of the same order.
What is $(\mathbf{A}+\mathbf{B})^2$?
- Why is that answer tempting? The school answer. Collecting AB and BA into 2AB assumes AB = BA, which is exactly the rule matrices do not obey. It is the most common algebraic error with matrices.
- Why is that answer tempting? Drops the cross terms entirely — that is not a matrix phenomenon, it is the distributive law being forgotten. The same answer would be wrong for numbers.
- Why is that answer tempting? Hedges in the wrong place. The four-term expansion always holds — the two distributive laws give it with no condition at all. AB = BA is what would be needed to collapse it to option A.
Apply the right distributive law, then the left one twice: $$ \begin{aligned} (\mathbf{A}+\mathbf{B})^2 &= (\mathbf{A}+\mathbf{B})\mathbf{A}\\ &\quad+ (\mathbf{A}+\mathbf{B})\mathbf{B}\\ &= \mathbf{A}^2 + \mathbf{BA}\\ &\quad+ \mathbf{AB} + \mathbf{B}^2\,. \end{aligned} $$
Four terms, and they stay four. Collecting the middle two would need $\mathbf{AB} = \mathbf{BA},$ which is false in general.
Expanding a square
For square $\mathbf{A},\mathbf{B}$ of the same order, expand $(\mathbf{A}+\mathbf{B})^{2}.$
Try it yourself first. Then open one step at a time.
Two distributive laws. For square $\mathbf{A},\mathbf{B}$ of the same order, the two distributive laws give $$ \begin{aligned} (\mathbf{A}+\mathbf{B})^{2} &= (\mathbf{A}+\mathbf{B})\mathbf{A}\\ &\quad+(\mathbf{A}+\mathbf{B})\mathbf{B}\\ &= \mathbf{A}^{2}+\mathbf{B}\mathbf{A}\\ &\quad+\mathbf{A}\mathbf{B}+\mathbf{B}^{2}\,. \end{aligned} $$
Not the school formula. This is not $\mathbf{A}^{2}+2\mathbf{A}\mathbf{B}+\mathbf{B}^{2}{:}$ collecting the two middle terms would require $\mathbf{A}\mathbf{B}=\mathbf{B}\mathbf{A},$ which is exactly what matrices do not give us.
$(\mathbf{A}+\mathbf{B})^{2}=\mathbf{A}^{2}+\mathbf{A}\mathbf{B}+\mathbf{B}\mathbf{A}+\mathbf{B}^{2},$ four terms, with the middle two uncollected.
The identity matrix
Among the rules that survive, one number has a matrix partner.
If $\mathbf{A}$ is $m\times n$ then $\mathbf{A}\mathbf{I}_{n}=\mathbf{A}$ and $\mathbf{I}_{m}\mathbf{A}=\mathbf{A}$. In particular $\mathbf{A}\mathbf{I}=\mathbf{I}\mathbf{A}=\mathbf{A}$ for every square $\mathbf{A}$, and $\mathbf{I}$ is the only matrix with this property.
Neutral. The $(i,k)$ entry of $\mathbf{A}\mathbf{I}_{n}$ is row $i$ of $\mathbf{A}$ dotted with column $k$ of $\mathbf{I}_{n}.$ That column is zero except for a $1$ in position $k,$ so the inner product collapses to the single term $a_{ik}.$
The only one. For uniqueness, suppose $\mathbf{A}\mathbf{E}=\mathbf{A}$ for all $n\times n$ matrices $\mathbf{A};$ taking $\mathbf{A}=\mathbf{I}$ gives $\mathbf{I}\mathbf{E}=\mathbf{I},$ and $\mathbf{I}\mathbf{E}=\mathbf{E},$ so $\mathbf{E}=\mathbf{I}.$
So $\mathbf{I}$ plays the part of the number $1.$ Note that it must be square, and that the two identities in $\mathbf{A}\mathbf{I}_{n}=\mathbf{I}_{m}\mathbf{A}$ have different orders unless $\mathbf{A}$ is square: shape first, here as elsewhere.
Pick an answer before reading on. Getting it wrong is fine — that is how you find out what you believe.
$\mathbf{A}$ is a $2\times3$ matrix. Which products exist and equal $\mathbf{A}$?
- Why is that answer tempting? $\mathbf{A}\mathbf{I}_{2}$ is a $(2\times3)(2\times2)$ product: the inner numbers $3$ and $2$ differ, so it does not exist. One identity cannot stand on both sides of a matrix that is not square.
- Why is that answer tempting? $\mathbf{I}_{3}\mathbf{A}$ is a $(3\times3)(2\times3)$ product, which does not exist. On the left $\mathbf{A}$ needs an identity with as many columns as $\mathbf{A}$ has rows.
- Why is that answer tempting? $\mathbf{I}_{2}\mathbf{A}$ exists too, a $(2\times2)(2\times3)$ product, and it equals $\mathbf{A}$ by Fact 13.4 with $m=2.$
By Fact 13.4 with $m=2$ and $n=3{:}$ $\mathbf{A}\mathbf{I}_{3}=\mathbf{A}$ and $\mathbf{I}_{2}\mathbf{A}=\mathbf{A}.$ The identity on the right has the order of the columns, the one on the left the order of the rows. Only for a square $\mathbf{A}$ is it the same $\mathbf{I}$ on both sides.
Three rules that do not survive
For numbers $ab=0$ forces $a=0$ or $b=0.$ Can two matrices, neither of them zero, multiply to the zero matrix?
The rules above are the ones that survive. Three rules of ordinary algebra do not, and each is worth knowing by name.
A product that vanishes
Take $$ \mathbf{A}=\begin{pmatrix} 3 & 1\\ 6 & 2\end{pmatrix}\,, \qquad \mathbf{B}=\begin{pmatrix} 1 & 2\\ -3 & -6\end{pmatrix}\,. $$ Then $\mathbf{A}\mathbf{B}=\mathbf{0},$ although neither factor is the zero matrix.
Try it yourself first. Then open one step at a time.
The four entries (the app's). Row by column: $$ \begin{aligned} 3\cdot1+1\cdot(-3)&=0\,,\\ 3\cdot2+1\cdot(-6)&=0\,,\\ 6\cdot1+2\cdot(-3)&=0\,,\\ 6\cdot2+2\cdot(-6)&=0\,. \end{aligned} $$ So every entry of $\mathbf{A}\mathbf{B}$ is $0,$ and neither $\mathbf{A}$ nor $\mathbf{B}$ is the zero matrix: failure (2).
No cancellation either. The same pair settles (3) as well: $\mathbf{A}\mathbf{B}=\mathbf{A}\mathbf{0}$ with $\mathbf{A}\ne\mathbf{0},$ and yet $\mathbf{B}\ne\mathbf{0}.$ So from $\mathbf{A}\mathbf{B}=\mathbf{A}\mathbf{C}$ one may cancel nothing.
$\mathbf{A}\mathbf{B}=\mathbf{0}$ with $\mathbf{A}\ne\mathbf{0}$ and $\mathbf{B}\ne\mathbf{0};$ and $\mathbf{A}\mathbf{B}=\mathbf{A}\mathbf{0}$ without $\mathbf{B}=\mathbf{0}.$
Failure (2) is not a curiosity. For numbers, $ab=0$ forces a factor to vanish because every non-zero number can be divided by; the matrix statement fails precisely because some non-zero matrices cannot be divided by. Which ones they are is the subject of Lecture 14, and $\mathbf{A}$ above is one of them.
Pick an answer before reading on. Getting it wrong is fine — that is how you find out what you believe.
What follows?
- Why is that answer tempting? Imports ab = 0 ⟹ a = 0 or b = 0 from the real numbers. That theorem is true for numbers because you can divide by a non-zero number; matrices give you no such division, and the conclusion goes with it.
- Why is that answer tempting? Assumes commutativity. Multiply it out: BA = (15 5; −45 −15), nowhere near zero. Two matrices can annihilate each other in one order and not the other.
- Why is that answer tempting? Looks for an explanation in the entries. There is a real explanation and it is structural, not arithmetic: the rows of B are multiples of each other, and so are the columns of A.
$\mathbf{AB} = \mathbf{0}$, and neither factor is $\mathbf{0}$.
The same example kills cancellation: $\mathbf{AB} = \mathbf{A}\mathbf{0}$ with $\mathbf{A} \ne \mathbf{0}$, and yet $\mathbf{B} \ne \mathbf{0}$. So from $\mathbf{AB} = \mathbf{AC}$ you may cancel nothing.
The transpose
$\mathbf{A}$ is $2\times5$ and $\mathbf{B}$ is $5\times3.$ What order does $(\mathbf{A}\mathbf{B})^{\top}$ have, and which product of transposes is it?
One genuinely new operation, and it is an operation on shape.
Reflecting the array in its main diagonal, in other words. The order is reversed: $m\times n$ becomes $n\times m.$
A matrix, a column, and their transposes
Transpose $$ \mathbf{A}=\begin{pmatrix} 2 & -1 & 3\\ 0 & 4 & 1\end{pmatrix} \quad\text{and}\quad \mathbf{a}=\begin{pmatrix} a_{1}\\ a_{2}\\ a_{3}\end{pmatrix}. $$
Try it yourself first. Then open one step at a time.
Rows become columns. $$ \begin{aligned} \mathbf{A}&=\begin{pmatrix} 2 & -1 & 3\\ 0 & 4 & 1\end{pmatrix}\\ \;\Longrightarrow\; \mathbf{A}^{\top}&=\begin{pmatrix} 2 & 0\\ -1 & 4\\ 3 & 1\end{pmatrix}\,, \end{aligned} $$ and $$ \begin{aligned} \mathbf{a}&=\begin{pmatrix} a_{1}\\ a_{2}\\ a_{3}\end{pmatrix}\\ \;\Longrightarrow\; \mathbf{a}^{\top}&=(a_{1}\;\; a_{2}\;\; a_{3})\,. \end{aligned} $$
The inner product is a matrix product. A column vector transposes to a row vector. If $\mathbf{a},\mathbf{b}$ are $n\times1$ columns, then $\mathbf{a}^{\top}\mathbf{b}$ is a $(1\times n)(n\times 1)$ product: a $1\times1$ matrix, whose single entry is $\mathbf{a}\cdot\mathbf{b}.$ The inner product of Lecture 12 is therefore a matrix product in disguise, and we write $\mathbf{a}^{\top}\mathbf{b}=\mathbf{a}\cdot\mathbf{b}.$
$\mathbf{A}^{\top}$ is the $3\times2$ matrix with rows $(2,0),$ $(-1,4),$ $(3,1);$ $\mathbf{a}^{\top}=(a_{1}\;\;a_{2}\;\;a_{3})$ is a row, and $\mathbf{a}^{\top}\mathbf{b}=\mathbf{a}\cdot\mathbf{b}.$
In economics a quantity vector $\mathbf{x}$ is normally a column and a price vector is written $\mathbf{p}^{\top},$ a row, so that $\mathbf{p}^{\top}\mathbf{x}$ is the bill.
For matrices $\mathbf{A},\mathbf{B}$ and a scalar $\alpha$, whenever the operations written down are defined, $$ \begin{aligned} (\mathbf{A}^{\top})^{\top}&=\mathbf{A}\,,\\ (\mathbf{A}+\mathbf{B})^{\top}&=\mathbf{A}^{\top}+\mathbf{B}^{\top}\,,\\ (\alpha\mathbf{A})^{\top}&=\alpha\mathbf{A}^{\top}\,,\\ (\mathbf{A}\mathbf{B})^{\top}&=\mathbf{B}^{\top}\mathbf{A}^{\top}\,. \end{aligned} $$
Guess first. The last rule has a reason in the orders alone. Try the question before you read the Watch out that gives it.
Pick an answer before reading on. Getting it wrong is fine — that is how you find out what you believe.
$\mathbf{A}$ is $2 \times 5$ and $\mathbf{B}$ is $5 \times 3$.
What is $(\mathbf{AB})^{\top}$?
- Why is that answer tempting? Keeps the original order. Aᵀ is 5 × 2 and Bᵀ is 3 × 5, so AᵀBᵀ is a (5 × 2)(3 × 5) product — it does not exist at all, whatever order you claim for it.
- Why is that answer tempting? That is (Aᵀ)ᵀ = A, a different statement. Transposing AB once does change it — 2 × 3 becomes 3 × 2.
- Why is that answer tempting? Right rule, then takes the order from AB instead of from its transpose. AB is 2 × 3, so (AB)ᵀ is 3 × 2 — and BᵀAᵀ is (3 × 5)(5 × 2), which agrees.
$\mathbf{AB}$ is $2 \times 3,$ so $(\mathbf{AB})^{\top}$ is $3 \times 2.$
The shapes leave exactly one possibility, so the rule does not have to be remembered.
Two symmetric products
For any $m\times n$ matrix $\mathbf{X},$ show that both $\mathbf{X}\mathbf{X}^{\top}$ (of order $m\times m$) and $\mathbf{X}^{\top}\mathbf{X}$ (of order $n\times n$) are symmetric.
Try it yourself first. Then open one step at a time.
Transpose the product. $$ \begin{aligned} (\mathbf{X}\mathbf{X}^{\top})^{\top} &= (\mathbf{X}^{\top})^{\top}\mathbf{X}^{\top}\\ &= \mathbf{X}\mathbf{X}^{\top}\,, \end{aligned} $$ using $(\mathbf{A}\mathbf{B})^{\top}=\mathbf{B}^{\top}\mathbf{A}^{\top}$ and then $(\mathbf{A}^{\top})^{\top}=\mathbf{A}.$
The other one. The argument for $\mathbf{X}^{\top}\mathbf{X}$ is identical. Note that the two products are symmetric but generally of different sizes, and are not each other.
$(\mathbf{X}\mathbf{X}^{\top})^{\top}=\mathbf{X}\mathbf{X}^{\top}$ and $(\mathbf{X}^{\top}\mathbf{X})^{\top}=\mathbf{X}^{\top}\mathbf{X}{:}$ both are symmetric, of orders $m\times m$ and $n\times n.$
Pick an answer before reading on. Getting it wrong is fine — that is how you find out what you believe.
$\mathbf{X}$ is a $3\times2$ matrix with entries that are not all zero. Which of these is certainly a symmetric matrix?
- Why is that answer tempting? A sum needs two matrices of the same order. $\mathbf{X}$ is $3\times2$ and $\mathbf{X}^{\top}$ is $2\times3,$ so the sum does not exist. (For a square $\mathbf{X}$ it would be symmetric.)
- Why is that answer tempting? $\mathbf{X}^{\top}$ is $2\times3,$ not square, and Definition 13.11 speaks only of square matrices.
- Why is that answer tempting? It exists, a $(3\times2)(2\times3)(3\times2)$ product, but it is $3\times2,$ not square, so it cannot be symmetric.
$\mathbf{X}^{\top}\mathbf{X}$ is a $(2\times3)(3\times2)$ product, a $2\times2$ matrix, and by Example 13.12 $$ \begin{aligned} (\mathbf{X}^{\top}\mathbf{X})^{\top} &=\mathbf{X}^{\top}(\mathbf{X}^{\top})^{\top}\\ &=\mathbf{X}^{\top}\mathbf{X}\,. \end{aligned} $$ The other product, $\mathbf{X}\mathbf{X}^{\top},$ which is $3\times3,$ is symmetric too.
Gaussian elimination
For numbers one solves $ax=b$ by dividing by $a.$ Can $\mathbf{A}\mathbf{x}=\mathbf{b}$ be solved without dividing by $\mathbf{A}$?
We now solve $\mathbf{A}\mathbf{x}=\mathbf{b}.$ The method is the one from school — eliminate the unknowns one at a time — made systematic, and it is what a computer does.
Two equation systems are equivalent if they have exactly the same solutions. Three operations turn a system into an equivalent one, and they are the only ones we shall need.
An elementary row operation produces an equivalent system: the set of solutions is unchanged.
Each can be undone. Each operation is reversible by another operation of the same kind — swap back, multiply by $1/\alpha,$ subtract the same multiple. So any solution of the original system solves the new one and conversely.
Why not zero. This is why (2) forbids $\alpha=0{:}$ multiplying an equation by $0$ destroys information and cannot be undone.
Only the coefficients change under these operations; the symbols $x_{1},x_{2},\dots$ do nothing but mark columns. So we drop them.
Elimination, one row operation at a time
The augmented matrix of a system of three equations, and the row operations of the method below, one press at a time.
- Press Next operation until the end, and name each operation before you press: which row changes, and by what? Count them: seven, each one of the three kinds above.
- Watch the leading $1$s become solid as they are fixed: first a staircase going down, then zeros cleared going up.
- Press the button with the system's name to switch to infinitely many and then no solution. The left-hand sides and the operations are the same; one number of $\mathbf{b}$ differs. Compare the last rows. (The next section explains them.)
Step (3) is optional labour: one may stop at the staircase and substitute upwards instead (back substitution). It is the same arithmetic either way. An unknown whose column has no leading entry is a free unknown, and the number of them is the number of degrees of freedom.
One solution
Solve $$ \begin{aligned} x_{1}+ x_{2}+3x_{3} &= 2\\ 2x_{1}+3x_{2}+ x_{3} &= -4\\ 3x_{1}- x_{2}+2x_{3} &= 11\,. \end{aligned} $$
Try it yourself first. Then open one step at a time.
First column. Subtract $2\times$ row $1$ from row $2,$ and $3\times$ row $1$ from row $3{:}$ $$ \begin{aligned} &\left(\begin{array}{ccc|c} 1 & 1 & 3 & 2\\ 2 & 3 & 1 & -4\\ 3 & -1 & 2 & 11 \end{array}\right)\\ \sim{}&\left(\begin{array}{ccc|c} 1 & 1 & 3 & 2\\ 0 & 1 & -5 & -8\\ 0 & -4 & -7 & 5 \end{array}\right) \end{aligned} $$
Second column, and the last leading entry. Then add $4\times$ the new row $2$ to the new row $3;$ then divide by $-27{:}$ $$ \begin{aligned} \sim{}&\left(\begin{array}{ccc|c} 1 & 1 & 3 & 2\\ 0 & 1 & -5 & -8\\ 0 & 0 & -27 & -27 \end{array}\right)\\ \sim{}&\left(\begin{array}{ccc|c} 1 & 1 & 3 & 2\\ 0 & 1 & -5 & -8\\ 0 & 0 & 1 & 1 \end{array}\right)\,. \end{aligned} $$
Back substitution. The staircase is complete, with leading entries in all three columns. Back substitution gives $x_{3}=1,$ then $x_{2}=-8+5=-3,$ then $x_{1}=2-(-3)-3=2.$ The only solution is $(x_{1},x_{2},x_{3})=(2,-3,1).$
Or clear upwards. Clearing above the leading entries instead — row $2+5\times$ row $3,$ row $1-3\times$ row $3,$ then row $1-$ row $2$ — ends at $$ \left(\begin{array}{ccc|c} 1 & 0 & 0 & 2\\ 0 & 1 & 0 & -3\\ 0 & 0 & 1 & 1 \end{array}\right)\,, $$ which is the answer written out.
$(x_{1},x_{2},x_{3})=(2,-3,1),$ the only solution. (Check, the app's: $2-3+3=2,$ $4-9+1=-4,$ $6+3+2=11.$)
Pick an answer before reading on. Getting it wrong is fine — that is how you find out what you believe.
Which of these is not an elementary row operation on a system of three equations?
- Why is that answer tempting? Operation (2) allows any non-zero scalar, fractions and negative numbers included. Multiplying by $-3$ undoes it.
- Why is that answer tempting? That is operation (3), adding the multiple $-4$ of one equation to a different one. Adding $4$ times equation $1$ undoes it.
- Why is that answer tempting? That is operation (1). Interchanging them again undoes it.
Multiplying by $0$ turns the equation into $0=0$ and loses it: the operation cannot be undone, so the new system may have more solutions than the old one. That is why (2) of Definition 13.13 asks for a non-zero scalar, and the reason Fact 13.14 holds for the other three.
No solution, one, or infinitely many
Two systems have the same left-hand sides and differ in one constant. Can one of them have infinitely many solutions and the other none at all?
Example 13.17 ended with a leading entry in every column. That need not happen, and the same three left-hand sides show both ways it can fail.
Infinitely many solutions
Solve $$ \begin{aligned} x_{1}+2x_{2}- x_{3} &= 3\\ 2x_{1}+5x_{2}+ x_{3} &= 7\\ x_{1}+ x_{2}-4x_{3} &= 2\,. \end{aligned} $$
Try it yourself first. Then open one step at a time.
Eliminate. Subtract $2\times$ row $1$ from row $2$ and row $1$ from row $3,$ then add row $2$ to row $3{:}$ $$ \begin{aligned} &\left(\begin{array}{ccc|c} 1 & 2 & -1 & 3\\ 2 & 5 & 1 & 7\\ 1 & 1 & -4 & 2 \end{array}\right)\\ \sim{}&\left(\begin{array}{ccc|c} 1 & 2 & -1 & 3\\ 0 & 1 & 3 & 1\\ 0 & -1 & -3 & -1 \end{array}\right)\\ \sim{}&\left(\begin{array}{ccc|c} 1 & 2 & -1 & 3\\ 0 & 1 & 3 & 1\\ 0 & 0 & 0 & 0 \end{array}\right)\,. \end{aligned} $$
A row of zeros. The third equation has become $0=0{:}$ it carried no information the first two did not already carry, and is dropped. The leading entries sit in columns $1$ and $2,$ so $x_{3}$ is free.
Read off with a parameter. Putting $x_{3}=t,$ the second row gives $x_{2}=1-3t$ and the first gives $x_{1}=3-2(1-3t)+t=1+7t.$ The solutions are $$ \begin{aligned} &(x_{1},x_{2},x_{3})\\ &= (1+7t,\;1-3t,\;t)\,,\\ &t\in{\mathbb R}\,, \end{aligned} $$ one degree of freedom: three unknowns, two independent equations.
$(x_{1},x_{2},x_{3})=(1+7t,\;1-3t,\;t)$ for every real $t{:}$ infinitely many solutions, one degree of freedom.
Guess first. Now change one constant on the right. Answer the question before you read the Watch out below; its second array is exactly that change.
Pick an answer before reading on. Getting it wrong is fine — that is how you find out what you believe.
Two systems with the same coefficient matrix eliminate to $$ \text{(i)}\;\left(\begin{array}{ccc|c} 1 & 2 & -1 & 3\\ 0 & 1 & 3 & 1\\ 0 & 0 & 0 & 0 \end{array}\right) $$ $$ \text{(ii)}\;\left(\begin{array}{ccc|c} 1 & 2 & -1 & 3\\ 0 & 1 & 3 & 1\\ 0 & 0 & 0 & 3 \end{array}\right) $$
How many solutions has each?
- Why is that answer tempting? Assumes the number of solutions is a property of A. It is not: whether solutions exist depends on b. Only uniqueness is settled by A alone — which is exactly what Lecture 14 will measure.
- Why is that answer tempting? Reads the signal backwards. The last row of (i) says 0 = 0, which is true and carries no information; the last row of (ii) says 0 = 3, which is false and destroys the system.
- Why is that answer tempting? A row that is zero all the way across is a redundant equation, not a contradiction — it means one equation was implied by the others. The fatal case needs a non-zero entry in the b column.
(i) last row is $0 = 0$: a repeated equation, dropped. Columns $1$ and $2$ lead, column $3$ does not, so $x_3$ is free — infinitely many solutions.
(ii) last row is $0 = 3$. No numbers satisfy it: no solutions at all.
Same $\mathbf{A}$, one number of $\mathbf{b}$ changed, and the two systems could not differ more.
A system of linear equations has either no solution, exactly one solution, or infinitely many. There is no fourth possibility, and Gaussian elimination decides which occurs:
- a row $(0\;\cdots\;0\mid c)$ with $c\ne0$ — no solution;
- otherwise, a leading entry in every one of the $n$ unknown columns — exactly one solution;
- otherwise — infinitely many, with one degree of freedom for each column without a leading entry.
For two equations in two unknowns this is the picture of two lines in Lecture 1 (Fact 1.7); it holds for any size.
Two consequences are worth stating. A system with fewer equations than unknowns can never have exactly one solution: at most $m$ columns can carry a leading entry, so if $m<n$ some column does not, and the system has either none or infinitely many. And whether a square system has exactly one solution is a property of $\mathbf{A}$ alone, not of $\mathbf{b}$ — which is what a determinant will measure in Lecture 14.
Pick an answer before reading on. Getting it wrong is fine — that is how you find out what you believe.
A system has $2$ equations in $3$ unknowns. What can be said about the number of solutions without solving it?
- Why is that answer tempting? Exactly one needs a leading entry in each of the $3$ columns, and $2$ rows carry at most $2$ leading entries. At least one unknown is always free.
- Why is that answer tempting? Only if it is consistent. $x+y+z=1$ and $x+y+z=2$ eliminate to the row $(0\;0\;0\mid1){:}$ no solution at all.
- Why is that answer tempting? Too few equations means too many solutions, not none. A system fails only when a row reads $0=c$ with $c\ne0,$ and that has nothing to do with counting equations.
At most $m=2$ of the $n=3$ columns carry a leading entry, so at least one unknown is free, and by Fact 13.20 the outcome "exactly one" is ruled out. Which of the other two occurs depends on $\mathbf{b}{:}$ a row $(0\;0\;0\mid c)$ with $c\ne0$ means none, otherwise there are infinitely many.
The shape of the solution set
A finished staircase has four unknowns and only three leading entries. How many solutions — and what do they look like?
When there are infinitely many solutions, they are not a shapeless heap: they have a very definite structure, and it is the same structure every time. We look at one system carefully and then say what is true in general.
Guess first. The question below is the next example. Answer it before you read on.
Pick an answer before reading on. Getting it wrong is fine — that is how you find out what you believe.
A system in four unknowns eliminates to $$\left(\begin{array}{cccc|c} 1 & 0 & 2 & 0 & 3\\ 0 & 1 & -1 & 0 & 5\\ 0 & 0 & 0 & 1 & 2 \end{array}\right)$$
How many solutions?
- Why is that answer tempting? Confuses underdetermined with inconsistent. Too few equations means too many solutions, never none. A system fails only when a row says 0 = c with c ≠ 0, and no row here does.
- Why is that answer tempting? Reads the empty third column as x₃ = 0. It says nothing of the kind — a column with no leading entry is a free unknown, and this is the most expensive misreading in the section.
- Why is that answer tempting? A row with some zeros is still an equation: this one reads x₄ = 2, which is perfectly good information. Only an all-zero left-hand side with a non-zero right-hand side is fatal.
Leading entries sit in columns $1,$ $2$ and $4$ — so $x_1,$ $x_2,$ $x_4$ are determined. Column $3$ has none, so $x_3$ is free: put $x_3 = t$ and $$ \begin{aligned} x_1 &= 3 - 2t, & x_2 &= 5 + t,\\ x_3 &= t, & x_4 &= 2. \end{aligned} $$
Four unknowns, three leading entries: one degree of freedom.
Reading off every solution
A system in the four unknowns $x_{1},x_{2},x_{3},x_{4}$ has augmented matrix $$ \left(\begin{array}{cccc|c} 1 & 0 & 2 & 0 & 3\\ 0 & 1 & -1 & 0 & 5\\ 0 & 0 & 0 & 1 & 2 \end{array}\right)\,, $$ that is, $$ \mathbf{A}= \begin{pmatrix} 1 & 0 & 2 & 0\\ 0 & 1 & -1 & 0\\ 0 & 0 & 0 & 1 \end{pmatrix}\,, \quad \mathbf{b}= \begin{pmatrix} 3\\ 5\\ 2 \end{pmatrix}\,. $$ Find all solutions of $\mathbf{A}\mathbf{x}=\mathbf{b}.$
Try it yourself first. Then open one step at a time.
Already finished. Elimination is already finished: there is a staircase, the leading entries are $1,$ and there are zeros above them. Written out, the three rows say $$ \begin{aligned} x_{1}+2x_{3}&=3\,,\\ x_{2}-x_{3}&=5\,,\\ x_{4}&=2\,. \end{aligned} $$
One free unknown. The leading entries sit in columns $1,$ $2$ and $4;$ column $3$ has none, so $x_{3}$ is free. Putting $x_{3}=t,$ the solutions are $$ \begin{aligned} &(x_{1},x_{2},x_{3},x_{4})\\ &= (3-2t,\;5+t,\;t,\;2)\,,\\ &t\in{\mathbb R}\,, \end{aligned} $$ with one degree of freedom: four unknowns, three leading entries.
A blank column is not a zero. Note that nothing in the matrix says $x_{3}=0.$ An empty column is not an equation; it is the absence of one.
$(x_{1},x_{2},x_{3},x_{4})=(3-2t,\;5+t,\;t,\;2)$ for every real $t{:}$ infinitely many solutions, one degree of freedom.
The recipe has done its job, but it hides why the answer looks as it does. Let us take it apart into two pieces.
Piece one: a single solution. The quickest way to get some solution is to give the free unknown the simplest possible value, $x_{3}=0.$ Then the rows give $x_{1}=3,$ $x_{2}=5,$ $x_{4}=2,$ and $$ \begin{aligned} &\mathbf{s}=\begin{pmatrix} 3\\ 5\\ 0\\ 2 \end{pmatrix} \qquad\text{satisfies}\\ &\mathbf{A}\mathbf{s} =\begin{pmatrix} 3+2\cdot0\\ 5-0\\ 2 \end{pmatrix} =\begin{pmatrix} 3\\ 5\\ 2 \end{pmatrix} =\mathbf{b}\,. \end{aligned} $$ So $\mathbf{s}$ is a solution — one of infinitely many, and not the solution. Such an $\mathbf{s}$ is often called a particular solution.
A homogeneous system always has at least one solution, namely $\mathbf{x}=\mathbf{0},$ the trivial solution. It can never be inconsistent.
Piece two: the homogeneous system. Now forget $\mathbf{b}$ and solve $\mathbf{A}\mathbf{x}=\mathbf{0}.$ No new elimination is needed: row operations applied to a column of zeros leave it a column of zeros, so we simply replace the right-hand column $3,5,2$ by $0,0,0.$ The rows now say $$ \begin{aligned} x_{1}+2x_{3}&=0\,,\\ x_{2}-x_{3}&=0\,,\\ x_{4}&=0\,, \end{aligned} $$ and $x_{3}$ is again free. Any non-zero choice of $x_{3}$ gives a solution other than $\mathbf{0};$ with $x_{3}=-1$ we get $x_{1}=2,$ $x_{2}=-1,$ $x_{4}=0,$ so $$ \begin{aligned} &\mathbf{r}=\begin{pmatrix} 2\\ -1\\ -1\\ 0 \end{pmatrix} \qquad\text{satisfies}\\ &\mathbf{A}\mathbf{r} =\begin{pmatrix} 2+2\cdot(-1)\\ -1-(-1)\\ 0 \end{pmatrix} =\begin{pmatrix} 0\\ 0\\ 0 \end{pmatrix} =\mathbf{0}\,. \end{aligned} $$
Putting the pieces together. Here is the point. For every number $\alpha,$ the rules of Fact 13.1 — a matrix distributes over a sum, and a scalar moves freely through a product — give $$ \begin{aligned} \mathbf{A}(\mathbf{s}+\alpha\mathbf{r}) &= \mathbf{A}\mathbf{s}+\alpha\,\mathbf{A}\mathbf{r}\\ &= \mathbf{b}+\alpha\,\mathbf{0} = \mathbf{b}\,. \end{aligned} $$ So every vector $\mathbf{s}+\alpha\mathbf{r}$ is again a solution of $\mathbf{A}\mathbf{x}=\mathbf{b}.$ Adding a solution of the homogeneous system to a solution does not disturb it, because $\mathbf{A}$ turns what was added into $\mathbf{0}.$ This property of $\mathbf{A}$ — that $\mathbf{A}(\mathbf{u}+\alpha\mathbf{v})=\mathbf{A}\mathbf{u}+\alpha\mathbf{A}\mathbf{v}$ — is what is meant by calling the system linear.
And there are no others. Conversely, let $\mathbf{x}$ be any solution of $\mathbf{A}\mathbf{x}=\mathbf{b}.$ Then $\mathbf{A}(\mathbf{x}-\mathbf{s})=\mathbf{A}\mathbf{x}-\mathbf{A}\mathbf{s} =\mathbf{b}-\mathbf{b}=\mathbf{0},$ so $\mathbf{x}-\mathbf{s}$ solves the homogeneous system. Those solutions we know: with $x_{3}=c$ free they are $(-2c,\,c,\,c,\,0)=-c\,\mathbf{r},$ that is, exactly the multiples of $\mathbf{r}.$ Hence $\mathbf{x}-\mathbf{s}=\alpha\mathbf{r}$ for some $\alpha,$ and $$ \begin{aligned} &\bigl\{\text{solutions of } \mathbf{A}\mathbf{x}=\mathbf{b}\bigr\}\\ &=\bigl\{\,\mathbf{s}+\alpha\mathbf{r} \;:\; \alpha\in{\mathbb R}\,\bigr\}\\ &=\left\{\, \begin{pmatrix} 3\\ 5\\ 0\\ 2 \end{pmatrix} +\alpha\begin{pmatrix} 2\\ -1\\ -1\\ 0 \end{pmatrix} \;:\; \alpha\in{\mathbb R} \right\}\,. \end{aligned} $$ This is the answer of Example 13.21 again, now with $\alpha=-t.$ Nothing depended on the choice $x_{3}=-1{:}$ choosing $x_{3}=1$ gives $-\mathbf{r}$ instead, and the set of all multiples is the same.
Geometrically, the multiples $\alpha\mathbf{r}$ form a line through the origin in ${\mathbb R}^{4},$ and the solutions of $\mathbf{A}\mathbf{x}=\mathbf{b}$ form the parallel line through $\mathbf{s}{:}$ the first line, moved off the origin by one particular solution. Nothing in the argument used the particular numbers.
Suppose $\mathbf{s}$ is one solution of $\mathbf{A}\mathbf{x}=\mathbf{b}$. Then the solutions of $\mathbf{A}\mathbf{x}=\mathbf{b}$ are exactly the vectors $\mathbf{s}+\mathbf{h}$, where $\mathbf{h}$ is a solution of the corresponding homogeneous system $\mathbf{A}\mathbf{x}=\mathbf{0}$. In words: $$ \begin{aligned} &\text{all solutions}\\ &=\text{one particular solution}\\ &\quad+\text{all homogeneous solutions}\,. \end{aligned} $$
Every s + h is a solution. If $\mathbf{A}\mathbf{h}=\mathbf{0}$ then $\mathbf{A}(\mathbf{s}+\mathbf{h})=\mathbf{A}\mathbf{s}+\mathbf{A}\mathbf{h} =\mathbf{b}+\mathbf{0}=\mathbf{b}.$
Every solution is an s + h. Conversely, if $\mathbf{A}\mathbf{x}=\mathbf{b}$ then $\mathbf{h}=\mathbf{x}-\mathbf{s}$ satisfies $\mathbf{A}\mathbf{h}=\mathbf{b}-\mathbf{b}=\mathbf{0},$ and $\mathbf{x}=\mathbf{s}+\mathbf{h}.$
This explains two things we met above. The degrees of freedom belong to the homogeneous system and so to $\mathbf{A}$ alone; $\mathbf{b}$ only decides whether there is a particular solution $\mathbf{s}$ to start from. And a consistent system has exactly one solution precisely when $\mathbf{A}\mathbf{x}=\mathbf{0}$ has only the trivial solution $\mathbf{0}$ — which is again a property of $\mathbf{A},$ not of $\mathbf{b}.$
Pick an answer before reading on. Getting it wrong is fine — that is how you find out what you believe.
$\mathbf{s}=(3,1,0)$ is a solution of $\mathbf{A}\mathbf{x}=\mathbf{b},$ with $\mathbf{b}\ne\mathbf{0},$ and $\mathbf{h}=(1,-2,1)$ is a solution of $\mathbf{A}\mathbf{x}=\mathbf{0}.$ Nothing else is known about $\mathbf{A}.$ Which of these is certainly a solution of $\mathbf{A}\mathbf{x}=\mathbf{b}$?
- Why is that answer tempting? That is $2\mathbf{s},$ and $\mathbf{A}(2\mathbf{s})=2\mathbf{b},$ which is not $\mathbf{b}$ since $\mathbf{b}\ne\mathbf{0}.$ Multiples of a solution solve the homogeneous system only when they started there.
- Why is that answer tempting? That is $\mathbf{h}$ itself, and $\mathbf{A}\mathbf{h}=\mathbf{0},$ not $\mathbf{b}.$ A homogeneous solution has to be added to a particular one.
- Why is that answer tempting? Changing the free-looking last entry of $\mathbf{s}$ is not allowed by anything known: $(3,1,1)-\mathbf{s}=(0,0,1),$ and nothing says $\mathbf{A}(0,0,1)=\mathbf{0}.$
$(5,-3,2)=\mathbf{s}+2\mathbf{h},$ and by Fact 13.23 $$ \begin{aligned} \mathbf{A}(\mathbf{s}+2\mathbf{h}) &=\mathbf{A}\mathbf{s}+2\mathbf{A}\mathbf{h}\\ &=\mathbf{b}+\mathbf{0}=\mathbf{b}\,. \end{aligned} $$ Every $\mathbf{s}+\alpha\mathbf{h}$ is a solution; a vector is certainly one when its difference from $\mathbf{s}$ is known to solve $\mathbf{A}\mathbf{x}=\mathbf{0}.$
What you should take away
- Matrix multiplication is associative and distributive on both sides, and scalars move freely through a product. Brackets in $\mathbf{A}\mathbf{B}\mathbf{C}$ and in powers $\mathbf{A}^{n}$ are therefore unnecessary.
- Three rules of ordinary algebra fail: $\mathbf{A}\mathbf{B}\ne \mathbf{B}\mathbf{A}$; $\mathbf{A}\mathbf{B}=\mathbf{0}$ with neither factor zero; and no cancellation. Consequently $(\mathbf{A}+\mathbf{B})^{2}=\mathbf{A}^{2}+\mathbf{A}\mathbf{B} +\mathbf{B}\mathbf{A}+\mathbf{B}^{2}$, with the middle terms uncollected.
- $\mathbf{I}_{n}$ has ones on the main diagonal and zeros elsewhere, and $\mathbf{A}\mathbf{I}=\mathbf{I}\mathbf{A}=\mathbf{A}$. It is the matrix equivalent of the number $1$, and it is the only one.
- The transpose $\mathbf{A}^{\top}$ turns rows into columns: $\bigl(\mathbf{A}^{\top}\bigr)_{rs}=a_{sr}$, and $m\times n$ becomes $n\times m$. It satisfies $(\mathbf{A}^{\top})^{\top}=\mathbf{A}$, $(\mathbf{A}+\mathbf{B})^{\top}=\mathbf{A}^{\top}+\mathbf{B}^{\top}$ and — with the order reversed — $(\mathbf{A}\mathbf{B})^{\top}=\mathbf{B}^{\top}\mathbf{A}^{\top}$. The shapes force the reversal. A square matrix is symmetric if $\mathbf{A}^{\top}=\mathbf{A}$.
- The three elementary row operations do not change the solution set of a system, and act only on the augmented matrix $(\mathbf{A}\mid\mathbf{b})$.
- Gaussian elimination: make a staircase with leading entries $1$, clear above them, and read off the answer, treating the unknowns without a leading entry as free.
- Every linear system has no solution, one solution, or infinitely many. A row $(0\;\cdots\;0\mid c)$ with $c\ne0$ means none; a leading entry in every unknown's column means one; anything else means infinitely many, one degree of freedom per free column.
- Structure of the solution set: if $\mathbf{s}$ is one solution of $\mathbf{A}\mathbf{x}=\mathbf{b}$, then all solutions are $\mathbf{s}+\mathbf{h}$ with $\mathbf{A}\mathbf{h}=\mathbf{0}$. With one degree of freedom this is a line $\mathbf{s}+\alpha\mathbf{r}$, $\alpha\in{\mathbb R}$.
Think further
Questions without computation. Think on your own or with a classmate before opening the thought.
Discussion question 1 — four orders, one product that does not exist
$\mathbf{A}$ is $2\times3$ and $\mathbf{B}$ is $3\times4.$ What are the orders of $\mathbf{A}\mathbf{B},$ $(\mathbf{A}\mathbf{B})^{\top},$ $\mathbf{B}^{\top}\mathbf{A}^{\top}$ and $\mathbf{A}^{\top}\mathbf{B}^{\top}$?
One way to think about it
$\mathbf{A}\mathbf{B}$ is a $(2\times3)(3\times4)$ product, so it is $2\times4,$ and $(\mathbf{A}\mathbf{B})^{\top}$ is $4\times2.$ $\mathbf{B}^{\top}\mathbf{A}^{\top}$ is $(4\times3)(3\times2),$ that is $4\times2$ — the same order, as Fact 13.9 requires. $\mathbf{A}^{\top}\mathbf{B}^{\top}$ is $(3\times2)(4\times3){:}$ the inner numbers $2$ and $4$ differ, so it does not exist at all (Watch out 13.10).
Discussion question 2 — the difference of two squares
Expand $(\mathbf{A}+\mathbf{B})(\mathbf{A}-\mathbf{B})$ for square $\mathbf{A},\mathbf{B}.$ When does it equal $\mathbf{A}^{2}-\mathbf{B}^{2}$?
One way to think about it
By the two distributive laws, $$ \begin{aligned} &(\mathbf{A}+\mathbf{B})(\mathbf{A}-\mathbf{B})\\ &=\mathbf{A}^{2}-\mathbf{A}\mathbf{B}+\mathbf{B}\mathbf{A}-\mathbf{B}^{2}\,. \end{aligned} $$ This equals $\mathbf{A}^{2}-\mathbf{B}^{2}$ exactly when $\mathbf{B}\mathbf{A}-\mathbf{A}\mathbf{B}=\mathbf{0},$ that is, when $\mathbf{A}\mathbf{B}=\mathbf{B}\mathbf{A}{:}$ precisely when $\mathbf{A}$ and $\mathbf{B}$ commute. For the pair of the first Common-error box it fails.
Discussion question 3 — counting free columns
After elimination a system of $4$ equations in $6$ unknowns has two rows of zeros and no inconsistency. How many degrees of freedom does it have?
One way to think about it
Four. The two rows of zeros are $0=0$ (there is no inconsistency), so two non-zero rows are left, with one leading entry each. Two of the six columns carry a leading entry and $6-2=4$ do not, and each of those is a degree of freedom (Fact 13.20).
Discussion question 4 — which rule holds for all square matrices
Which of the following is true for all square $\mathbf{A}$ and $\mathbf{B}$ of the same order?
(A) $\mathbf{A}\mathbf{B}=\mathbf{0}\Rightarrow \mathbf{A}=\mathbf{0}$ or $\mathbf{B}=\mathbf{0}$;
(B) $\mathbf{A}\mathbf{B}=\mathbf{A}\mathbf{C}$ and $\mathbf{A}\ne\mathbf{0}$ $\Rightarrow \mathbf{B}=\mathbf{C};$
(C) $(\mathbf{A}\mathbf{B})^{\top}=\mathbf{B}^{\top}\mathbf{A}^{\top};$
(D) $(\mathbf{A}+\mathbf{B})^{2} =\mathbf{A}^{2}+2\mathbf{A}\mathbf{B}+\mathbf{B}^{2}.$
One way to think about it
(C) only; it is Fact 13.9. The pair of Example 13.6 refutes (A), since $\mathbf{A}\mathbf{B}=\mathbf{0}$ with neither factor zero, and also (B), with $\mathbf{C}=\mathbf{0}.$ (D) needs $\mathbf{A}\mathbf{B}=\mathbf{B}\mathbf{A}$ (Example 13.2), which fails for the same pair: there $\mathbf{B}\mathbf{A}\ne\mathbf{0}.$