4.2 Special Matrices: The Identity Matrix

In normal arithmetic, we have the number $1$. The number $1$ is special because multiplying any number by it leaves that number completely unchanged:

$5 \times 1 = 5$ $-12.5 \times 1 = -12.5$

In matrix algebra, we have a similar concept called the Identity Matrix, denoted by the letter $I$. It acts as the number "1" of the matrix world.

1. Defining the Identity Matrix

The Identity matrix is always a square matrix (it has the same number of rows and columns). It features:

• Ones ($1$) along the main diagonal (running from top-left to bottom-right).
• Zeros ($0$) everywhere else.

Depending on the dimension, here is how we write it:

$I_2 = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}$ $I_3 = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix}$

2. Geometric Intuition: The "Do Nothing" Transformation

Let's look at the Identity matrix columns to see what it does geometrically:

• Column 1 is $\begin{bmatrix} 1 \\ 0 \end{bmatrix}$: This means basis vector $\hat{i}$ lands at $(1, 0)$ (it doesn't move).
• Column 2 is $\begin{bmatrix} 0 \\ 1 \end{bmatrix}$: This means basis vector $\hat{j}$ lands at $(0, 1)$ (it doesn't move).

Because the basis vectors do not budge, the Identity transformation leaves the entire coordinate space completely untouched. Every single vector in the universe stays exactly where it was.

3. Algebraic Property: Neutral Element

If you multiply any matrix $A$ by the Identity matrix $I$, the result is exactly $A$.

$A \cdot I = A$ $I \cdot A = A$

Let's test this with a $2 \times 2$ example:

$A = \begin{bmatrix} 3 & 5 \\ 1 & -2 \end{bmatrix}, \quad I = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}$

Let's compute $AI$:

• Row 1 of A $\times$ Col 1 of I: $(3 \times 1) + (5 \times 0) = 3$
• Row 1 of A $\times$ Col 2 of I: $(3 \times 0) + (5 \times 1) = 5$
• Row 2 of A $\times$ Col 1 of I: $(1 \times 1) + (-2 \times 0) = 1$
• Row 2 of A $\times$ Col 2 of I: $(1 \times 0) + (-2 \times 1) = -2$

$AI = \begin{bmatrix} 3 & 5 \\ 1 & -2 \end{bmatrix} = A$

It works exactly as advertised!

4. Machine Learning Application: Initialization & Regularization

Why is the Identity matrix important in Machine Learning?

I. Identity Initialization

In deep learning, particularly for recurrent networks (RNNs), we initialize weight matrices. If we initialize weights to zero, neurons learn nothing. If we initialize them too large, values explode. A common technique is initializing weight matrices close to the Identity Matrix. This guarantees that, initially, information flows through network layers without being distorted or shrunk.

II. Ridge Regression ($L_2$ Regularization)

In linear regression, if features are highly correlated, the matrix $X^T X$ is singular (non-invertible). To fix this, we add a small diagonal term using the identity matrix:

$(X^T X + \lambda I)$

This adds a tiny number ($\lambda$) to the diagonal elements, guaranteeing the matrix is non-singular and invertible, which makes the model numerically stable.

5. Check Your Understanding