Orthogonality

Your browser lacks required capabilities. Please upgrade it or switch to another to continue.

Loading…

<center>Author: Nadiah Kristensen (nadiah.org)</center>

<center>(1) -- (2) -- (3) -- (4) -- (5) -- (6) -- (end) </center> <h2>Part 1</h2> <nowiki> $$ \begin{equation} A = \begin{bmatrix} 1 & 1 \\ 0 & 1 \\ 0 & 0 \end{bmatrix} \end{equation} $$ </nowiki> Question: What is the columnspace of $A$? <hr> <<linkreplace "Reveal hint" t8n>> Hints To describe a vector space like a columnspace, you describe it in terms of its basis vectors. You write it in the form: <nowiki> $$ \begin{equation} \text{basis}(C(A)) = \left\{ \begin{bmatrix} \cdot \\ \cdot \\ \cdot \end{bmatrix}, \begin{bmatrix} \cdot \\ \cdot \\ \cdot \end{bmatrix}, \ldots \right\} \end{equation} $$ </nowiki> or the form <nowiki> $$ \begin{equation} C(A) = \text{span} \left\{ \begin{bmatrix} \cdot \\ \cdot \\ \cdot \end{bmatrix}, \begin{bmatrix} \cdot \\ \cdot \\ \cdot \end{bmatrix}, \ldots \right\} \end{equation} $$ </nowiki> where each vector is a basis vector of the columnspace of $ A$. <<linkreplace "How do I get the basis vectors of the columnspace of A?" t8n>> The basis vectors of the columnspace of matrix $A$ are the pivot columns of $A$. <<linkreplace "Which are the pivot columns of A?" t8n>> A pivot column is any column with a pivot in it. Usually you would perform a Gaussian elimination to find the pivots; however, $ A$ is already in row-echelon form, so you can read the pivots of $A$ directly. <<linkreplace "Show me step by step how to get the pivots" t8n>> To identify the pivots, walk along each row from the left and highlight the first non-zero element. First row: <nowiki> $$ \begin{equation} \begin{matrix} \rightarrow \\ \phantom{0} \\\phantom{0} \\ \end{matrix} \begin{bmatrix} \color{red} 1 & 1 \\ 0 & 1 \\ 0 & 0 \end{bmatrix} \end{equation} $$ </nowiki> Second row: <nowiki> $$ \begin{equation} \begin{matrix} \phantom{0} \\ \rightarrow \\\phantom{0} \\ \end{matrix} \begin{bmatrix} \color{red} 1 & 1 \\ 0 & \color{red} 1 \\ 0 & 0 \end{bmatrix} \end{equation} $$ </nowiki> Third row -- no pivot: <nowiki> $$ \begin{equation} \begin{matrix} \phantom{0} \\ \phantom{0} \\ \rightarrow \\ \end{matrix} \begin{bmatrix} \color{red} 1 & 1 \\ 0 & \color{red} 1 \\ 0 & 0 \end{bmatrix} \end{equation} $$ </nowiki> Therefore, the pivot columns of $A$ are columns 1 and 2. <<mathjax>><</linkreplace>> <<mathjax>><</linkreplace>> <<mathjax>><</linkreplace>> <<mathjax>><</linkreplace>> <hr> <<linkreplace "I'm ready to answer">> Answer: The columnspace of $A$ is (a) $ \text{col}(A) = \text{span} \left\{ \begin{bmatrix} 1 \\ 1 \end{bmatrix}, \begin{bmatrix} 0 \\ 1 \end{bmatrix}, \begin{bmatrix} 0 \\ 0 \end{bmatrix} \right\} $ [[Choose (a) -> part1_incorrect]] (b) $ \text{col}(A) = \text{span} \left\{ \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix}, \begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix} \right\} $ [[Choose (b) -> part1_correct]] (c) $ \text{col}(A) = \text{span} \left\{ \begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix} \right\} $ [[Choose (c) -> part1_incorrect]] (d) $ \text{col}(A) = \text{span} \left\{ \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix} \right\} $ [[Choose (d) -> part1_incorrect]] <<mathjax>> <</linkreplace>>

<center>(1) -- (2) -- (3) -- (4) -- (5) -- (6) -- (end) </center> <h2>Incorrect.</h2> The columnspace of $ A $ is the space spanned by its pivot columns. Usually, we perform a Gaussian elimination on $A$ to get $A$ into row-echelon form so we can identify the pivots. However, $A$ is already in row-echelon form, so we can read-off the pivots directly <nowiki> $$ \begin{equation} A = \begin{bmatrix} 1 & 1 \\ 0 & 1 \\ 0 & 0 \end{bmatrix} \end{equation} $$ </nowiki> The pivots of $A$ are: (a) $A = \begin{bmatrix} \color{red} 1 & 1 \\ 0 & 1 \\ 0 & 0 \end{bmatrix} $ <<linkreplace "Choose (a)" t8n>> Incorrect. <<run $("#found_pivots_wrong").show()>> <<replace "#find_pivots_right">><</replace>> <</linkreplace>> (b) $A = \begin{bmatrix} \color{red} 1 & 1 \\ 0 & \color{red} 1 \\ 0 & 0 \end{bmatrix} $ <<linkreplace "Choose (b)" t8n>> Correct! <<replace "#find_pivots_wrong">><</replace>> <<run $("#found_pivots").show()>> <</linkreplace>> To identify the pivots, walk along each row from the left and highlight the first non-zero element. First row: <nowiki> $$ \begin{equation} \begin{matrix} \rightarrow \\ \phantom{0} \\\phantom{0} \\ \end{matrix} \begin{bmatrix} \color{red} 1 & 1 \\ 0 & 1 \\ 0 & 0 \end{bmatrix} \end{equation} $$ </nowiki> Second row: <nowiki> $$ \begin{equation} \begin{matrix} \phantom{0} \\ \rightarrow \\\phantom{0} \\ \end{matrix} \begin{bmatrix} \color{red} 1 & 1 \\ 0 & \color{red} 1 \\ 0 & 0 \end{bmatrix} \end{equation} $$ </nowiki> Third row -- no pivot: <nowiki> $$ \begin{equation} \begin{matrix} \phantom{0} \\ \phantom{0} \\ \rightarrow \\ \end{matrix} \begin{bmatrix} \color{red} 1 & 1 \\ 0 & \color{red} 1 \\ 0 & 0 \end{bmatrix} \end{equation} $$ </nowiki> <<linkreplace "Continue" t8n>><<run $("#found_pivots").show()>><</linkreplace>> Therefore, the pivot columns of $A$ are: <div> (a) Column 1. <<linkreplace "Choose (a)" t8n>>Incorrect. The pivot columns are all columns containing a pivot.<</linkreplace>> </div> (b) Columns 1 and 2. <<linkreplace "Choose (b)" t8n>> Correct! <<run $("#correct_choice").show()>> <<replace ".wrong_columns" t8n>><</replace>> <</linkreplace>> <div> (c) Column 2. <<linkreplace "Choose (c)" t8n>>Incorrect. The pivot columns are all columns containing a pivot.<</linkreplace>> </div> The pivot columns of $A$ are: <nowiki> $$ \begin{equation} \left\{ \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix}, \begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix} \right\} \end{equation} $$ </nowiki> The pivot columns are the basis vectors of the columnspace of $A$. Or equivalently, the columnspace is the span of the basis vectors. Therefore, we can describe the columnspace of $A$ as: <div> (a) $ \text{col}(A) = \text{span} \left\{ \begin{bmatrix} 1 \\ 1 \end{bmatrix}, \begin{bmatrix} 0 \\ 1 \end{bmatrix}, \begin{bmatrix} 0 \\ 0 \end{bmatrix} \right\} $ <<linkreplace "Choose (a)" t8n>> Incorrect. The basis vectors of the columnspace are the pivot columns. <</linkreplace>> </div> (b) $ \text{col}(A) = \text{span} \left\{ \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix}, \begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix} \right\} $ <<linkreplace "Choose (b)" t8n>> Correct! <<run $("#desmos").show()>> <</linkreplace>> The span of the pivot columns is all vectors that can be expressed as a linear combination of the pivot columns, which is the plane shown in blue below. <iframe src="https://www.desmos.com/3d/cnbpgoe2h8" width="100%" style="min-height:700px"></iframe> <center>Figure 1: The pivot columns of $A$ (orange vectors) and the columnspace of $ A $ (blue plane).</center> [[CONTINUE to next part -> part2]]

<h2>Correct!</h2> The matrix $A$ is already in row-echelon form. Its pivots are highlighted below <nowiki> $$ \begin{equation} A = \begin{bmatrix} \mathbf{1} & 1 \\ 0 & \mathbf{1} \\ 0 & 0 \end{bmatrix} \end{equation} $$ </nowiki> The columnspace of $A$ is all vectors that can be expressed as a linear combination of the pivot columns, which is the span of the pivot columns. <nowiki>$$ \begin{equation} \text{col}(A) = \text{span} \left\{ \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix}, \begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix} \right\} \end{equation} $$ </nowiki> The columnspace of $A$ is the plane shown in blue below. <iframe src="https://www.desmos.com/3d/cnbpgoe2h8" width="100%" style="min-height:700px"></iframe> <center>Figure 1: The pivot columns of $A$ (orange vectors) and the columnspace of $ A $ (blue plane).</center> [[CONTINUE to next part -> part2]]

<center>(1) -- (2) -- (3) -- (4) -- (5) -- (6) -- (end) </center> <h2>Part 2</h2> Consider <nowiki> $$ \begin{equation} A = \begin{bmatrix} 1 & 1 \\ 0 & 1 \\ 0 & 0 \end{bmatrix} \quad \text{and} \quad \mathbf{b} = \begin{bmatrix} 2 \\ 3 \\ 4 \end{bmatrix} \end{equation} $$ </nowiki> Recall that the columnspace of $ A$ is <nowiki> $$ \begin{equation} \text{col}(A) = \text{span} \left\{ \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix}, \begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix} \right\} \end{equation} $$ </nowiki> Question: How could you show that $\mathbf{b}$ is not in the columnspace of $A$? <hr> <div> (a) Show that $ \mathbf{b} $ cannot be written as a linear combination of the columns of $ A $. <<linkreplace "Choose (a)" t8n>> That's correct, but to show it in an exam, what method would you use? <<mathjax>> <</linkreplace>> </div> <div> (b) Since $ A $ has only two columns, the columnspace of $ A$ is two-dimensional, but $ \mathbf{b} $ has 3 components, so $ \mathbf{b} $ can't be in the columnspace of $ A $. <<linkreplace "Choose (b)" t8n>> Incorrect. Although $ \text{col}(A) $ is two-dimensional, $ \text{col}(A) $ lives in $ \mathbb{R}^3 $ because $ A $ has 3 rows. It is a plane through the origin embedded in 3D space. <<mathjax>> <</linkreplace>> </div> <div> (c) Show that row reducing $ [A \mid b ] $ produces an inconsistent row. <<linkreplace "Choose (c)" t8n>> Correct! To show that $\mathbf{b}$ is not in the columnspace of $A$, you need to show that the system $ A \mathbf{x} = \mathbf{b} $ has no solution. $A$ is already in row-echelon form, so the row-reduced augmented matrix is <nowiki> $$ \begin{equation} \left[ \left. \begin{matrix} 1 & 1 \\ 0 & 1 \\ 0 & 0 \end{matrix} \right| \left. \begin{matrix} 2 \\ 3 \\ 4 \end{matrix} \right. \right] \end{equation} $$ </nowiki> The final row gives the equation <nowiki> $$ \begin{equation} 0 x_1 + 0 x_2 = 4 \end{equation} $$ </nowiki> but $ 0 = 4 $ is impossible. Therefore, the system is inconsistent. Fig. 1 shows that indeed the vector $\mathbf{b}$ (red) is not in the columnspace of $A$ (blue plane). <iframe src="https://www.desmos.com/3d/j0nyokwdn4" width="100%" style="min-height:700px"></iframe> <center> Figure 1: The vector $ \mathbf{b} $ (red) is not in the columnspace of $A $ (blue plane). </center> Notice also that every vector in $ \text{col} (A) $ has a third component 0, since both basis vectors do, but $\mathbf{b} $ has a third component 4. So $\mathbf{b} $ sits above the plane, which is what the row reduction tells us algebraically. [[CONTINUE to next part -> part3]] <<mathjax>> <</linkreplace>> </div>

<center>(1) -- (2) -- (3) -- (4) -- (5) -- (6) -- (end) </center> <h2>Part 3</h2> Consider <nowiki> $$ \begin{equation} A = \begin{bmatrix} 1 & 1 \\ 0 & 1 \\ 0 & 0 \end{bmatrix} \quad \text{and} \quad \mathbf{b} = \begin{bmatrix} 2 \\ 3 \\ 4 \end{bmatrix} \end{equation} $$ </nowiki> Recall that the columnspace of $ A$ is <nowiki> $$ \begin{equation} \text{col}(A) = \text{span} \left\{ \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix}, \begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix} \right\} \end{equation} $$ </nowiki> Question: Which vector in the columnspace of $A$ is closest to $ \mathbf{b} $? <iframe src="https://www.desmos.com/3d/j0nyokwdn4" width="100%" style="min-height:700px"></iframe> <center>Figure 1: The pivot columns of $A$ (orange vectors), the columnspace of $A $ (blue plane), and the vector $ \mathbf{b} $ (red vector).</center> <hr> <<linkreplace "Reveal hint">> Hint Geometrically, $ \text{col}(A) $ is a plane (Fig. 1). Imagine dropping a perpendicular line from $ \mathbf{b} $ down to that plane. Where does it land? <<mathjax>><</linkreplace>> <hr> <div> (a) The column of $ A $ that is closest to $ \mathbf{b} $, which is $ \begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix} $. <<linkreplace "Choose (a)" t8n>> Incorrect. While it's true that $ (1, 1, 0 ) $ is the closest column vector to $ \mathbf{b} $, the columnspace of $ A$ is the entire plane spanned by those columns. There are infinitely many possible vectors on that plane, and one is closer to $ \mathbf{b} $ than any other -- it would be the shadow $ \mathbf{b} $ casts on the plane at high noon. What do we call that vector? Pick the option that names it. <<mathjax>> <</linkreplace>> </div> (b) The projection of $ \mathbf{b} $ onto the columnspace of $ A $. [[Choose (b) -> part3_correct]] <div> (c) The average of the columns of $ A $ . <<linkreplace "Choose (c)" t8n>> Incorrect. The average is just another vector in $ \text{col}(A)$. There's no reason it'd be the one closest to $ \mathbf{b} $. <<mathjax>> <</linkreplace>> </div> <div> (d) $ \mathbf{b} $ itself. <<linkreplace "Choose (d)" t8n>> Incorrect. While it's true that $ \mathbf{b} $ is the vector closest to $ \mathbf{b} $ in all $ \mathbb{R}^3 $, the question restricts us to vectors in the columnspace of $ A$, and we showed in Part 2 that $ \mathbf{b} $ is not in $ \text{col}(A) $. <<mathjax>> <</linkreplace>> </div>

<h2>Correct!</h2> The vector in the columnspace of $A$ that is closest to $ \mathbf{b} $ is the projection of $ \mathbf{b} $ onto the columnspace of $ A $. <iframe src="https://www.desmos.com/3d/ppa5ospwuy" width="100%" style="min-height:700px"></iframe> <center> Figure 1: The pivot columns of $A$ (orange vectors), the columnspace of $A $ (blue plane), and the vector $ \mathbf{b} $ (red vector). The vector in the columnspace of $A $ that is closest to $ \mathbf{b} $ is the projection of $ \mathbf{b} $ onto $ \text{col}(A) $ (purple vector). </center> <<linkreplace "Explain more" t8n>> You were given <nowiki> $$ \begin{equation} A = \begin{bmatrix} 1 & 1 \\ 0 & 1 \\ 0 & 0 \end{bmatrix} \quad \text{and} \quad \mathbf{b} = \begin{bmatrix} 2 \\ 3 \\ 4 \end{bmatrix} \end{equation} $$ </nowiki> and you found that the columnspace of $ A$ is <nowiki> $$ \text{col}(A) = \text{span} \left\{ \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix}, \begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix} \right\} $$ </nowiki> This columnspace is the blue plane in Fig. 1. The vector in the columnspace (blue plane) that is closest to $ \mathbf{b} $ (red vector) is the projection of $ \mathbf{b} $ onto the columnspace (purple vector). You could think of the projection $ \mathbf{p} $ as the "shadow" that $ \mathbf{b} $ casts at high noon onto the blue plane. In fact, what guarantees that the projection is closest is that the vector $ \mathbf{e} = \mathbf{b} - \mathbf{p} $ (shown in grey in Fig. 2 below) is perpendicular to the columnspace. <iframe src="https://www.desmos.com/3d/rjbsdqydcx" width="100%" style="min-height:700px"></iframe> <center> Figure 2: The projection $ \mathbf{p} $ (purple vector) is closest to $ \mathbf{b} $ (red vector) because $ \mathbf{e} = \mathbf{b} - \mathbf{p} $ (grey vector) is perpendicular to the columnspace (blue plane). </center> <<mathjax>><</linkreplace>> [[CONTINUE to next part -> part4]]

<center>(1) -- (2) -- (3) -- (4) -- (5) -- (6) -- (end) </center> <h2>Part 4</h2> Consider <nowiki> $$ \begin{equation} A = \begin{bmatrix} 1 & 1 \\ 0 & 1 \\ 0 & 0 \end{bmatrix} \quad \text{and} \quad \mathbf{b} = \begin{bmatrix} 2 \\ 3 \\ 4 \end{bmatrix} \end{equation} $$ </nowiki> Question: Find the projection of $ \mathbf{b} $ onto the columnspace of $ A $ by solving $ A^T A \hat{\mathbf{x}} = A^T \mathbf{b} $ and $ \mathbf{p} = A\hat{\mathbf{x}} $. <hr> <<linkreplace "Reveal hint" t8n>> Follow these steps to solve: <ol> <li> Find $A^T A$ (gives a $2 \times 2$ matrix)</li> <li> Find $A^T \mathbf{b}$ (gives a $2 \times 1$ vector)</li> <li> Solve $A^T A \hat{\mathbf{x}} = A^T \mathbf{b}$ for $\hat{\mathbf{x}}$: <ol> <li>Because $A^T A$ is a $2 \times 2$ matrix, we can use the formula for the inverse: <nowiki> $$ \begin{equation*} \text{For } A^T A = \underbrace{ \begin{bmatrix} a & b \\ c & d \\ \end{bmatrix} }_{\text{(1)}} , \text{ the inverse } (A^T A)^{-1} = \frac{1}{ad - bc} \begin{bmatrix} d & -b \\ -c & a \\ \end{bmatrix} \end{equation*} $$ </nowiki> </li> <li> Find $\hat{\mathbf{x}}$ directly <nowiki> $$ \begin{equation*} \hat{\mathbf{x}} = \underbrace{(A^T A)^{-1}}_{\text{(3.1)}} \underbrace{A^T \mathbf{b}}_{\text{(2)}} \end{equation*} $$ </nowiki> $ \hat{\mathbf{x}} $ can be thought of as a co-ordinate vector telling us how much of each column vector is required. </li> </ol> </li> <li> Find $ \mathbf{p} $ <nowiki> $$ \begin{equation*} \mathbf{p} = A \underbrace{\hat{\mathbf{x}}}_{\text{(3.2)}} \end{equation*} $$ </nowiki> </li> </ol> <<mathjax>><</linkreplace>> <hr> [[Show me step-by-step -> part4_step_by_step]] <hr> <<linkreplace "I'm ready to answer" t8n>> Answer: The projection of $ \mathbf{b} $ onto the columnspace of $ A $ is (a) $ \mathbf{p} = \begin{bmatrix} 2 \\ 3 \\ 4 \end{bmatrix} $ [[Choose (a) -> part4_incorrect]] (b) $ \mathbf{p} = \begin{bmatrix} 2 \\ 5 \end{bmatrix} $ [[Choose (b) -> part4_incorrect]] (c) $ \mathbf{p} = \begin{bmatrix} 2 \\ 3 \\ 0 \end{bmatrix} $ [[Choose (c) -> part4_correct]] (d) $ \mathbf{p} = \begin{bmatrix} 5 \\ 3 \\ 0 \end{bmatrix} $ [[Choose (d) -> part4_incorrect]] (e) $ \mathbf{p} = \begin{bmatrix} 4 \\ 3 \\ 0 \end{bmatrix} $ [[Choose (e) -> part4_incorrect]] (f) $ \mathbf{p} = \begin{bmatrix} -1 \\ 3 \\ 0 \end{bmatrix} $ [[Choose (f) -> part4_incorrect]] <<mathjax>><</linkreplace>>

<center>(1) -- (2) -- (3) -- (4) -- (5) -- (6) -- (end) </center> <h2>Incorrect</h2> [[Show me step-by-step -> part4_step_by_step]] [[Redo question -> part4]]

<center>(1) -- (2) -- (3) -- (4) -- (5) -- (6) -- (end) </center> <h2>Part 4: Guided solution</h2> Consider <nowiki> $$ \begin{equation} A = \begin{bmatrix} 1 & 1 \\ 0 & 1 \\ 0 & 0 \end{bmatrix} \quad \text{and} \quad \mathbf{b} = \begin{bmatrix} 2 \\ 3 \\ 4 \end{bmatrix} \end{equation} $$ </nowiki> Question: Find the projection of $ \mathbf{b} $ onto the columnspace of $ A $ by solving $ A^T A \hat{\mathbf{x}} = A^T \mathbf{b} $ and $ \mathbf{p} = A\hat{\mathbf{x}} $. <hr> Step 1. Find $A^T A$: <div style="margin-left: 20px"> (a) $ A^T A = \begin{bmatrix} 1 & 2 \\ 1 & 1 \end{bmatrix} $ <<linkreplace "Choose (a)" t8n>> Incorrect. This answer suggests you didn't take the transpose of $ A$ correctly. The rule is $a_{ij} \rightarrow a_{ji} $. You can also remember: first row (read left-to-right) $ \rightarrow $ first column (read top-to-bottom), second row $ \rightarrow $ second column, and so on. <<run $("#show_ATA").show()>> <<replace "#find_ATA_right">><</replace>> <<mathjax>> <</linkreplace>> (b) $ A^T A = \begin{bmatrix} 1 & 1 \\ 1 & 2 \end{bmatrix} $ <<linkreplace "Choose (b)" t8n>> Correct! <<run $("#show_ATA").show()>> <<replace "#find_ATA_wrong">><</replace>> <</linkreplace>> The answer is: <nowiki> $$ \begin{align*} A^T A &= \begin{bmatrix} 1 & 0 & 0 \\ 1 & 1 & 0 \end{bmatrix} \begin{bmatrix} 1 & 1 \\ 0 & 1 \\ 0 & 0 \end{bmatrix} = \begin{bmatrix} 1 & 1 \\ 1 & 2 \end{bmatrix} \end{align*} $$ </nowiki> <<linkreplace "Continue" t8n>><<run $("#step2").show()>><</linkreplace>> </div> Step 2. Find $A^T \mathbf{b}$: <div style="margin-left: 20px"> (a) $ A^T \mathbf{b} = \begin{bmatrix} 2 \\ 5 \end{bmatrix} $ <<linkreplace "Choose (a)" t8n>> Correct! <<run $("#show_ATb").show()>> <<replace "#find_ATb_wrong">><</replace>> <</linkreplace>> (b) $ A^T \mathbf{b} = \begin{bmatrix} 5 \\ 2 \end{bmatrix} $ <<linkreplace "Choose (b)" t8n>> Incorrect. Check the ordering: row 1 of $ A^T $ is $ (1, 0, 0) $ not $ (1, 1, 0) $. <<run $("#show_ATb").show()>> <<replace "#find_ATb_right">><</replace>> <<mathjax>> <</linkreplace>> The answer is: <nowiki> $$ \begin{align*} A^T \mathbf{b} &= \begin{bmatrix} 1 & 0 & 0 \\ 1 & 1 & 0 \end{bmatrix} \begin{bmatrix} 2 \\ 3 \\ 4 \end{bmatrix} = \begin{bmatrix} 2 \\ 5 \end{bmatrix} \end{align*} $$ </nowiki> <<linkreplace "Continue" t8n>><<run $("#step3").show()>><</linkreplace>> </div> Step 3. Solve $A^T A \hat{\mathbf{x}} = A^T \mathbf{b}$ for $\hat{\mathbf{x}}$. <div style="margin-left: 20px"> <<linkreplace "Reveal hint" t8n>> Hint: You'll need to solve $ \hat{\mathbf{x}} = (A^T A)^{-1}(A^T \mathbf{b}) $. You already have $ A^T \mathbf{b} $ above. You can also find $ (A^T A)^{-1} $ from $ A^T A $ above. <<linkreplace "How do I find the inverse?" t8n>> The equation for the inverse of a $ 2 \times 2 $ matrix $ M = \begin{bmatrix} a & b \\ c & d \end{bmatrix} $ is <nowiki> $$ \begin{equation} M^{-1} = \frac{1}{ad - bc} \begin{bmatrix} d & -b \\ -c & a \\ \end{bmatrix} \end{equation} $$ </nowiki> <<mathjax>><</linkreplace>> <<mathjax>><</linkreplace>> (a) $ \hat{\mathbf{x}} = \begin{bmatrix} -1 \\ 3 \end{bmatrix} $ <<linkreplace "Choose (a)" t8n>> Correct! <<run $("#show_x").show()>> <<replace "#find_x_wrong">><</replace>> <</linkreplace>> (b) $ \hat{\mathbf{x}} = \begin{bmatrix} 1 \\ 3 \end{bmatrix} $ <<linkreplace "Choose (b)" t8n>> Incorrect. Check the signs: the inverse formula flips the signs of the off-diagonal entries. <<run $("#show_x").show()>> <<replace "#find_x_right">><</replace>> <<mathjax>> <</linkreplace>> The answer is <nowiki> $$ \begin{equation*} (A^T A)^{-1} = \frac{1}{1 \times 2 - 1^2} \begin{bmatrix} 2 & -1 \\ -1 & 1 \end{bmatrix} = \begin{bmatrix} 2 & -1 \\ -1 & 1 \end{bmatrix} \end{equation*} $$ </nowiki> <nowiki> $$ \begin{equation*} \hat{\mathbf{x}} = (A^T A)^{-1}(A^T \mathbf{b}) = \begin{bmatrix} 2 & -1 \\ -1 & 1 \end{bmatrix} \begin{bmatrix} 2 \\ 5 \end{bmatrix} = \begin{bmatrix} -1 \\ 3 \end{bmatrix} \end{equation*} $$ </nowiki> $ \hat{\mathbf{x}} $ can be thought of as a co-ordinate vector telling us how much of each column vector is required. <<linkreplace "Continue" t8n>><<run $("#step4").show()>><</linkreplace>> </div> Step 4. Find $ \mathbf{p} = A \hat{\mathbf{x}} $ <div style="margin-left: 20px"> (a) $ \mathbf{p} = \begin{bmatrix} 2 \\ 3 \\ 0 \end{bmatrix} $ <<linkreplace "Choose (a)" t8n>> Correct! <<run $("#show_p").show()>> <<replace "#find_p_wrong">><</replace>> <</linkreplace>> (b) $ \mathbf{p} = \begin{bmatrix} -4 \\ 3 \\ 0 \end{bmatrix} $ <<linkreplace "Choose (b)" t8n>> Incorrect. You've made an arithmetic error. <<run $("#show_p").show()>> <<replace "#find_p_right">><</replace>> <<mathjax>> <</linkreplace>> The answer is <nowiki> $$ \begin{equation*} \mathbf{p} = A\hat{\mathbf{x}} = \begin{bmatrix} 1 & 1 \\ 0 & 1 \\ 0 & 0 \end{bmatrix} \begin{bmatrix} -1 \\ 3 \end{bmatrix} = \begin{bmatrix} 2 \\ 3 \\ 0 \end{bmatrix} \end{equation*} $$ </nowiki> <<linkreplace "Continue" t8n>> <<run $("#return_to_part4").hide()>> <<run $("#step5").show()>> <</linkreplace>> </div> $ \mathbf{p} $ is the vector in the columnspace of $ A $ that is closest to $ \mathbf{b} $. $ \hat{\mathbf{x}} $ tells us how much of each column of $ A $ is required to form $ \mathbf{p} $, i.e., <nowiki> $$ \begin{equation*} \mathbf{p} = A\hat{\mathbf{x}} = -1 \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix} + 3 \begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix} \end{equation*} $$ </nowiki> <iframe src="https://www.desmos.com/3d/trtzpurb1l" width="100%" style="min-height:700px"></iframe> <center> Figure 1: The projection $ \mathbf{p} $ is the linear combination of columns of $A $ (orange vectors) given by $ \hat{\mathbf{x}} $. Specifically, $ \mathbf{p} = -1 \times \text{col 1} + 3 \times \text{col 2} $ (blue vectors). </center> [[CONTINUE to next part -> part5]]             [[Return to redo question by myself -> part4]]

<h2>Correct!</h2> <nowiki> $$ \begin{equation} A = \begin{bmatrix} 1 & 1 \\ 0 & 1 \\ 0 & 0 \end{bmatrix} \quad \text{and} \quad \mathbf{b} = \begin{bmatrix} 2 \\ 3 \\ 4 \end{bmatrix} \end{equation} $$ </nowiki> The projection of $ \mathbf{b} $ onto the columnspace of $ A $ is <nowiki> $$ \begin{equation*} \mathbf{p} = \begin{bmatrix} 2 \\ 3 \\ 0 \end{bmatrix} \end{equation*} $$ </nowiki> $ \mathbf{p} $ is the vector in the columnspace of $ A $ that is closest to $ \mathbf{b} $. <<linkreplace "Reveal to discover more" t8n>> In the process of finding the solution, you would have also found <nowiki> $$ \begin{equation*} \hat{\mathbf{x}} = \begin{bmatrix} -1 \\ 3 \end{bmatrix} \end{equation*} $$ </nowiki> and also solved <nowiki> $$ \begin{equation*} \mathbf{p} = A\hat{\mathbf{x}} = \begin{bmatrix} 1 & 1 \\ 0 & 1 \\ 0 & 0 \end{bmatrix} \begin{bmatrix} -1 \\ 3 \end{bmatrix} = \begin{bmatrix} 2 \\ 3 \\ 0 \end{bmatrix} \end{equation*} $$ </nowiki> So $ \hat{\mathbf{x}} $ tells us how much of each column of $ A $ is required to form $ \mathbf{p} $, i.e., <nowiki> $$ \begin{equation*} \mathbf{p} = A\hat{\mathbf{x}} = -1 \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix} + 3 \begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix} \end{equation*} $$ </nowiki> <iframe src="https://www.desmos.com/3d/trtzpurb1l" width="100%" style="min-height:700px"></iframe> <center> Figure 1: The projection $ \mathbf{p} $ is the linear combination of columns of $A $ (orange vectors) given by $ \hat{\mathbf{x}} $. Specifically, $ \mathbf{p} = -1 \times \text{col 1} + 3 \times \text{col 2} $ (blue vectors). </center> <<mathjax>> <</linkreplace>> [[Show full solution -> part4_full_solution]] [[CONTINUE to next part -> part5]]

<center>(1) -- (2) -- (3) -- (4) -- (5) -- (6) -- (end) </center> <h2>Part 4: Full solution</h2> Consider <nowiki> $$ \begin{equation} A = \begin{bmatrix} 1 & 1 \\ 0 & 1 \\ 0 & 0 \end{bmatrix} \quad \text{and} \quad \mathbf{b} = \begin{bmatrix} 2 \\ 3 \\ 4 \end{bmatrix} \end{equation} $$ </nowiki> Question: Find the projection of $ \mathbf{b} $ onto the columnspace of $ A $ by solving $ A^T A \hat{\mathbf{x}} = A^T \mathbf{b} $ and $ \mathbf{p} = A\hat{\mathbf{x}} $. <hr> Step 1. Find $A^T A$: <nowiki> $$ \begin{align*} A^T A &= \begin{bmatrix} 1 & 0 & 0 \\ 1 & 1 & 0 \end{bmatrix} \begin{bmatrix} 1 & 1 \\ 0 & 1 \\ 0 & 0 \end{bmatrix} = \begin{bmatrix} 1 & 1 \\ 1 & 2 \end{bmatrix} \end{align*} $$ </nowiki> Step 2. Find $A^T \mathbf{b}$: <nowiki> $$ \begin{align*} A^T \mathbf{b} &= \begin{bmatrix} 1 & 0 & 0 \\ 1 & 1 & 0 \end{bmatrix} \begin{bmatrix} 2 \\ 3 \\ 4 \end{bmatrix} = \begin{bmatrix} 2 \\ 5 \end{bmatrix} \end{align*} $$ </nowiki> Step 3. Solve $A^T A \hat{\mathbf{x}} = A^T \mathbf{b}$ for $\hat{\mathbf{x}}$. <nowiki> $$ \begin{equation*} (A^T A)^{-1} = \frac{1}{1 \times 2 - 1^2} \begin{bmatrix} 2 & -1 \\ -1 & 1 \end{bmatrix} = \begin{bmatrix} 2 & -1 \\ -1 & 1 \end{bmatrix} \end{equation*} $$ </nowiki> <nowiki> $$ \begin{equation*} \hat{\mathbf{x}} = (A^T A)^{-1}(A^T \mathbf{b}) = \begin{bmatrix} 2 & -1 \\ -1 & 1 \end{bmatrix} \begin{bmatrix} 2 \\ 5 \end{bmatrix} = \begin{bmatrix} -1 \\ 3 \end{bmatrix} \end{equation*} $$ </nowiki> $ \hat{\mathbf{x}} $ can be thought of as a co-ordinate vector telling us how much of each column vector is required. Step 4. Find $ \mathbf{p} = A \hat{\mathbf{x}} $ <nowiki> $$ \begin{equation*} \mathbf{p} = A\hat{\mathbf{x}} = \begin{bmatrix} 1 & 1 \\ 0 & 1 \\ 0 & 0 \end{bmatrix} \begin{bmatrix} -1 \\ 3 \end{bmatrix} = \begin{bmatrix} 2 \\ 3 \\ 0 \end{bmatrix} \end{equation*} $$ </nowiki> $ \mathbf{p} $ is the vector in the columnspace of $ A $ that is closest to $ \mathbf{b} $. $ \hat{\mathbf{x}} $ tells us how much of each column of $ A $ is required to form $ \mathbf{p} $, i.e., <nowiki> $$ \begin{equation*} \mathbf{p} = A\hat{\mathbf{x}} = -1 \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix} + 3 \begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix} \end{equation*} $$ </nowiki> <iframe src="https://www.desmos.com/3d/trtzpurb1l" width="100%" style="min-height:700px"></iframe> <center> Figure 1: The projection $ \mathbf{p} $ is the linear combination of columns of $A $ (orange vectors) given by $ \hat{\mathbf{x}} $. Specifically, $ \mathbf{p} = -1 \times \text{col 1} + 3 \times \text{col 2} $ (blue vectors). </center> [[CONTINUE to next part -> part5]]

<center>(1) -- (2) -- (3) -- (4) -- (5) -- (6) -- (end) </center> <h2>Part 5</h2> Consider <nowiki> $$ \begin{equation} A = \begin{bmatrix} 1 & 1 \\ 0 & 1 \\ 0 & 0 \end{bmatrix} \quad \text{and} \quad \mathbf{b} = \begin{bmatrix} 2 \\ 3 \\ 4 \end{bmatrix} \end{equation} $$ </nowiki> Recall that the projection of $ \mathbf{b} $ onto the columnspace of $ A $ is <nowiki> $$ \begin{equation*} \mathbf{p} = \begin{bmatrix} 2 \\ 3 \\ 0 \end{bmatrix} \end{equation*} $$ </nowiki> Question: Find $ \mathbf{e} = \mathbf{b} - \mathbf{p}$ and show it is perpendicular to the columns of $ A$. <hr> Hint: <<linkreplace "How do I show perpendicularity?" t8n>> To show two vectors are orthogonal, show that their dot product equals zero. <<mathjax>><</linkreplace>> <hr> Answer: <<linkreplace "Reveal." t8n>> <nowiki> $$ \begin{equation*} \mathbf{e} = \mathbf{b} - \mathbf{p} = \begin{bmatrix} 2 \\ 3 \\ 4 \end{bmatrix} - \begin{bmatrix} 2 \\ 3 \\ 0 \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \\ 4 \end{bmatrix} \end{equation*} $$ </nowiki> To verify perpendicularity, we check the dot products of $ \mathbf{e} $ with each column of $ A $ are equal to zero: <ul> <li> $\mathbf{e} \cdot \mathbf{a}_1 = (0, 0, 4) \cdot (1, 0, 0) = 0 \times 1 + 0 \times 0 + 4 \times 0 = 0$ </li> <li> $\mathbf{e} \cdot \mathbf{a}_2 = (0, 0, 4) \cdot (1, 1, 0) = 0 \times 1 + 0 \times 1 + 4 \times 0 = 0$ </li> </ul> as required. Since the columns of $ A $ span $ \text{col}(A) $, showing that $ \mathbf{e} $ is perpendicular to both columns means $ \mathbf{e} $ is perpendicular to every vector in $ \text{col}(A) $, i.e., to the whole columnspace. The fact that $ \mathbf{e} $ is perpendicular to the columnspace of $ A $ is what guarantees that $ \mathbf{p} $ is the closest vector to $ \mathbf{b} $. <iframe src="https://www.desmos.com/3d/rjbsdqydcx" width="100%" style="min-height:700px"></iframe> <center> Figure 1: The projection $ \mathbf{p} $ (purple vector) is closest to $ \mathbf{b} $ (red vector) because $ \mathbf{e} = \mathbf{b} - \mathbf{p} $ (grey vector) is perpendicular to the columnspace (blue plane). </center> [[CONTINUE to next part -> part6]] <<mathjax>><</linkreplace>>

<center>(1) -- (2) -- (3) -- (4) -- (5) -- (6) -- (end) </center> <h2>Part 6</h2> We were given <nowiki> $$ \begin{equation} A = \begin{bmatrix} 1 & 1 \\ 0 & 1 \\ 0 & 0 \end{bmatrix} \quad \text{and} \quad \mathbf{b} = \begin{bmatrix} 2 \\ 3 \\ 4 \end{bmatrix} \end{equation} $$ </nowiki> We found that the projection of $ \mathbf{b} $ onto the columnspace of $ A $ was <nowiki> $$ \begin{equation*} \mathbf{p} = \begin{bmatrix} 2 \\ 3 \\ 0 \end{bmatrix} \end{equation*} $$ </nowiki> and <nowiki> $$ \begin{equation*} \mathbf{e} = \mathbf{b} - \mathbf{p} = \begin{bmatrix} 2 \\ 3 \\ 4 \end{bmatrix} - \begin{bmatrix} 2 \\ 3 \\ 0 \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \\ 4 \end{bmatrix} \end{equation*} $$ </nowiki> Question: Draw a diagram showing $ \mathbf{b} $, $ \mathbf{p} $, and $ \mathbf{e} $ including any important relationships between them. <hr> Hint: <<linkreplace "Reveal" t8n>> What geometric relationship did you prove in Part 5? <<mathjax>><</linkreplace>> <hr> Answer: <div> (a) <img src="assets/Week8_A1_Q4_no_orth.png" style="width: 300px;"> <<linkreplace "Choose" t8n>> Incorrect. You will lose marks if you don't show clearly that $ \mathbf{p} $ and $ \mathbf{e} $ are orthogonal, e.g., with a right-angle symbol. <<mathjax>> <</linkreplace>> </div> <div> (b) <img src="assets/Week8_A1_Q4_orth.png" style="width: 300px;"> <<linkreplace "Choose" t8n>> Correct; this diagram shows that $ \mathbf{p} $ and $ \mathbf{e} $ are orthogonal. [[CONTINUE to next part -> end]] <<mathjax>> <</linkreplace>> </div>

/* <center>(1) -- (2) -- (3) -- (4) -- (5) -- (6) -- (end) </center> */ <h1>Congratulations!</h1> <div style="text-align: center;"> <img src="assets/purrpendicular.jpg" style="width: 300px;"> You have completed the questions. [[Play again -> Start]] <hr> <a href="mailto:nadiah@nadiah.org">Send feedback to: nadiah@nadiah.org</a> </div>

<<widget "mathjax">> <<run requestAnimationFrame(function() { MathJax.typesetPromise([document.querySelector('.passage')]); })>> <</widget>>