Matrix Approaches to Simple Linear Regression Analysis

class: center, middle, title-slide

# Matrix Approaches to Simple Linear Regression Analysis
## AU STAT-615
### Emil Hvitfeldt
### 2021-02-24

---

`$$\require{color}\definecolor{orange}{rgb}{1, 0.603921568627451, 0.301960784313725}$$`
`$$\require{color}\definecolor{blue}{rgb}{0.301960784313725, 0.580392156862745, 1}$$`
`$$\require{color}\definecolor{pink}{rgb}{0.976470588235294, 0.301960784313725, 1}$$`

# Matrices

Matrix algebra is widely used in Mathematics and Statistics alike

It is more or less required to do multiple linear regression as it allows us to express large systems of equations and data in a compact way

---

# Example

We have 2 variables of data, .blue[income] and .orange[age]

$$
\blue{
`\begin{bmatrix}
16,000\\
16,000\\
16,000\\
\end{bmatrix}`
}
\orange{
`\begin{bmatrix}
23\\
47\\
35\\
\end{bmatrix}`
}
$$

---

# Example

These columns can also be seen as being composed of rows (of observations)
$$
`\begin{bmatrix}
\blue{16,000}\\
\orange{16,000}\\
\pink{16,000}\\
\end{bmatrix}`
`\begin{bmatrix}
\blue{23}\\
\orange{47}\\
\pink{35}\\
\end{bmatrix}`
$$

---

# Notation

Usual notation for a matrix

`$$\mathbf{A} = [a_{ij}] \qquad i = 1, 2; \quad j = 1, 2, 3$$`

`$$\mathbf{A} = \begin{bmatrix}
a_{11} & a_{12} & a_{13}\\
a_{21} & a_{22} & a_{23}
\end{bmatrix}$$`

We call this a 2-by-3 matrix or more generally .blue[n]-by-.orange[m] matrix when the matrix has

- .blue[n rows]
- .orange[m columns]

---

# Square Matrix

A matrix is called **square** if the number of rows equals the number of columns

`$$\begin{bmatrix}
a_{11} & a_{12} & a_{13}\\
a_{21} & a_{22} & a_{23}\\
a_{31} & a_{32} & a_{33}
\end{bmatrix}$$`

`$$\begin{bmatrix}
4 & 7\\
3 & 9
\end{bmatrix}$$`

---

# Vector

A vector can be thought of as a matrix with 1 column

A column vector

`$$\mathbf{A} = \begin{bmatrix}
4\\
7\\
10
\end{bmatrix}$$`

or a row vector

`$$\mathbf{A} = \begin{bmatrix}
15 & 25 & 50
\end{bmatrix}$$`

---

# Transpose

A transpose of matrix `$\mathbf{A}$` is denotes as `$\mathbf{A}^T$` or `$\mathbf{A}'$`.

`$$\mathbf{A} = \begin{bmatrix}
4 & 7 \\
3 & 9 \\
1 & 2
\end{bmatrix} 
\rightarrow 
\mathbf{A}^T = \begin{bmatrix}
4 & 3 & 1 \\
7 & 9 & 2
\end{bmatrix}$$`

Can be seen as flipping the row and column indices. Or flipping over the diagonal

---

# Equality of matrices

Let `$\mathbf{A} = \begin{bmatrix}a_{11} & a_{12}\\a_{21} & a_{22}\end{bmatrix}$` and `$\mathbf{B} = \begin{bmatrix}b_{11} & b_{12}\\b_{21} & b_{22}\end{bmatrix}$`

then we say that `$\mathbf{A} = \mathbf{B}$` if and only if

`$$a_{11} = b_{11}, \quad a_{12} = b_{12}, \quad a_{21} = b_{21}, \quad a_{22} = b_{22}$$`

---

# Regression Example

In regression analysis on of the basic matrices is `$\mathbf{Y}$`, that contains thee `$n$` observations on `$\mathbf{Y}$`

`$$\mathbf{Y} = \begin{bmatrix}
Y_1\\
Y_2\\
\vdots\\
Y_n
\end{bmatrix}$$`

---

# Regression Example

Another basic matrix we use in regression is matrix `$\mathbf{X}$`

In simplee linear regression this matrix is

`$$\mathbf{X} = \begin{bmatrix}
\blue{1} & \orange{Y_1}\\
\blue{1} & \orange{Y_2}\\
\blue{\vdots} & \orange{\vdots}\\
\blue{1} & \orange{Y_n}
\end{bmatrix}$$`

- .blue[columns of 1s]
- .orange[n observations of predictor variables]

---

### Matrix addition & Subtraction

Let `$\mathbf{A} = \begin{bmatrix}1 & 4\\2 & 5\\3 & 6\end{bmatrix}$` and `$\mathbf{B} = \begin{bmatrix}1 & 2\\2 & 3\\3 & 4\end{bmatrix}$`

Then we have that

`$$\mathbf{A} + \mathbf{B} = \begin{bmatrix}
1 + 1 & 4 + 2\\
2 + 2 & 5 + 3\\
3 + 3 & 6 + 4
\end{bmatrix} = \begin{bmatrix}
2 & 6\\
4 & 8\\
6 & 10
\end{bmatrix}$$`

and

`$$\mathbf{A} - \mathbf{B} = \begin{bmatrix}
1 - 1 & 4 - 2\\
2 - 2 & 5 - 3\\
3 - 3 & 6 - 4
\end{bmatrix} = \begin{bmatrix}
0 & 2\\
0 & 2\\
0 & 2
\end{bmatrix}$$`

---

# Regression example

`$$Y_i = E\{Y_i\} + \varepsilon_i \quad i = 1, ..., n$$`

This can be written in matrix form when

`$$\mathbf{E\{Y_i\}} = \begin{bmatrix}
E\{Y_1\}\\
E\{Y_2\}\\
\vdots\\
E\{Y_n\}
\end{bmatrix}, \qquad \boldsymbol\varepsilon = \begin{bmatrix}
\varepsilon_1\\
\varepsilon_2\\
\vdots\\
\varepsilon_n
\end{bmatrix}$$`

---

# Regression example

So we get that

`$$\mathbf{Y} = \mathbf{E\{Y_i\}} + \boldsymbol\varepsilon$$`

can written for

`$$\begin{bmatrix}
Y_1\\
Y_2\\
\vdots\\
Y_n
\end{bmatrix} = \begin{bmatrix}
E\{Y_1\}\\
E\{Y_2\}\\
\vdots\\
E\{Y_n\}
\end{bmatrix} + \begin{bmatrix}
\varepsilon_1\\
\varepsilon_2\\
\vdots\\
\varepsilon_n
\end{bmatrix} = \begin{bmatrix}
E\{Y_1\} + \varepsilon_1\\
E\{Y_2\} + \varepsilon_2\\
\vdots\\
E\{Y_n\} + \varepsilon_n
\end{bmatrix}$$`

---

# Matrix Multiplication

Matrix by scalar

let `$\mathbf{A} = \begin{bmatrix}4 & 7\\3 & 9\end{bmatrix}$`, then

`$$4 \cdot \mathbf{A} = \begin{bmatrix}
4 \cdot 4 & 4 \cdot 7\\
4 \cdot 3 & 4 \cdot 9
\end{bmatrix} = \begin{bmatrix}
16 & 28\\
12 & 36
\end{bmatrix}$$`

In general we have that for a scalar `$k$` and matrix `$\mathbf{A}$`

`$$k \cdot \mathbf{A} = \mathbf{A} \cdot k$$`

---

# Matrix Multiplication

matrix by matrix