Sign in Subscribe
MSA

MSA_Week 3 Moore-Penrose Inverse

Moore-Penrose Inverse...

hrayleung

• 3 min read

1.2.2 Moore-Penrose Inverse

Definition 1.2.2

Let $A$ be an $n \times p$ matrix. If there exists a $p \times n$ matrix $X$ such that

$$ AXA = A, \quad XAX = X \\ (AX)' = AX, \quad (XA)' = XA $$

then $X$ is called the Moore-Penrose inverse of $A$, denoted by $X = A^+$.

Property 1.2.8

  1. If $A$ is invertible, then

$$ A^+ = A^{-1}. $$

  1. If $A$ has full column rank, then

$$ X = (A'A)^{-1} A' $$

is the Moore-Penrose inverse of $A$. In particular, if $a$ is a non-zero column vector, then

$$ a^+ = \frac{a'}{\|a\|^2}. $$

  1. If $A$ has full row rank, then

$$ X = A'(AA')^{-1} $$

is the Moore-Penrose inverse of $A$. In particular, if $a$ is a non-zero row vector, then

$$ a^+ = \frac{a'}{\|a\|^2}. $$

Property 1.2.9

  1. For any matrix $A$, the Moore-Penrose inverse $A^+$ exists and is unique.
  2. Existence
  3. Uniqueness

Proof of Existence

Let $\mathrm{rk}\bigl(A_{n \times p}\bigr) = r.$

  1. Case: $r =0$
    In this case, $A = 0$. Simply take $A^+ = 0$, which trivially satisfies the four Moore-Penrose conditions.
  2. Case: $r>0$
    There exist invertible matrices $P_1$ and $Q_1$ such that

$$ A = P_1 \begin{pmatrix} I_r & 0 \\ 0 & 0 \end{pmatrix} Q_1. $$

We can also write

$$ A = P_1 \begin{pmatrix} I_r & 0 \end{pmatrix} \begin{pmatrix} I_r \\ 0 \end{pmatrix} Q_1 \;\subset\; P \, Q', $$

where

$$ P = P_1 \begin{pmatrix} I_r \\ 0 \end{pmatrix}, \quad Q' = \begin{pmatrix} I_r & 0 \end{pmatrix} Q_1, \quad Q = (Q')'. $$

Both $P$ and $Q$ have full column rank, so $P'P$ and $Q'Q$ are invertible. Define

$$ X = Q \bigl(Q'Q\bigr)^{-1} \bigl(P'P\bigr)^{-1} P'. $$

It can be verified that $X$ satisfies the four Moore-Penrose conditions:

  1. $$ AXA = A $$
  2. $$ XAX = X $$
  3. $$ (AX)' = AX $$
  4. $$ (XA)' = XA $$

Hence, $X$ is the Moore-Penrose inverse of $A$, proving its existence.

Proof of Uniqueness

If $A_1^+$ and $A_2^+$ are two Moore-Penrose inverses of $A$, then:

$$ \begin{aligned} A_1^+ &= A_1^+ \, A \, A_1^+ = A_1^+ \bigl(A A_1^+\bigr) = A_1^+ \bigl(A_1^+\bigr)' \, A' \bigl(A A_2^+\bigr) \\ &= A_1^+ \bigl(A A_1^+\bigr)' \bigl(A A_2^+\bigr)' = A_1^+ \, A \, A_1^+ \, A \, A_2^+ = A_1^+ \, A \, A_2^+. \end{aligned} $$

Similarly,

$$ \begin{aligned} A_2^+ &= A_2^+ \, A \, A_2^+ = \bigl(A_2^+ A\bigr)' \bigl(A_2^+\bigr)' A_2^+ = A' \bigl(A_1^+\bigr)' A' \bigl(A_2^+\bigr)' A_2^+ \\ &= \bigl(A_1 A\bigr)' \bigl(A_2 A\bigr)' A_2^+ = A_1^+ \, A \, A_2^+ \, A \, A_2^+ = A_1^+ \, A \, A_2^+. \end{aligned} $$

Thus $A_1^+ = A_2^+$. Therefore, the Moore-Penrose inverse of $A$ is unique.

Property 1.2.10

$$ \bigl(A^+\bigr)^+ = A. $$

Property 1.2.11

$$ A^+ = A' \bigl(AA'\bigr)^+ = \bigl(A'A\bigr)^+ A'. $$

Property 1.2.12

$$ \bigl(A'A\bigr)^+ = A^+ \bigl(A^+\bigr)'. $$

Property 1.2.13

$$ \bigl(A'\bigr)^+ = \bigl(A^+\bigr)'. $$

In particular, if $A$ is a symmetric matrix, then $A^+$ is also symmetric.

Property 1.2.14

Let $A = P Q'$, where

$$ \mathrm{rk}\bigl(A_{n \times p}\bigr) = \mathrm{rk}\bigl(P_{n \times r}\bigr) = \mathrm{rk}\bigl(Q_{p \times r}\bigr) = r. $$

Then

$$ A^+ = \bigl(Q'\bigr)' \, P^+. $$

Note: In general, for two arbitrary matrices $A$ and $B$, it does not necessarily hold that

$$ (AB)^+ = B^+ \, A^+. $$

Property 1.2.15

If $A$ is a projection matrix, then

$$ A^+ = A. $$

Property 1.2.16

If $A$ is an $n \times n$ symmetric matrix, there exists a decomposition

$$ A = H' \,\Lambda \,H, $$

where $H$ is an orthogonal matrix, and

$$ \Lambda = \mathrm{diag}\bigl(\lambda_1, \lambda_2, \dots, \lambda_n\bigr). $$

Define

$$ \lambda^+ = \begin{cases} \lambda^{-1}, & \lambda \neq 0, \\ 0, & \lambda = 0. \end{cases} $$

Then

$$ A^+ = H' \,\mathrm{diag}\bigl(\lambda_1^+, \lambda_2^+, \dots, \lambda_n^+\bigr) \,H. $$

Property 1.2.17

For any matrix $A$, both $A A^+$ and $A^+ A$ are projection matrices. We denote

$$ P_A = A A^+, \quad P_A = A^+ A. $$

Summary

  1. The Moore-Penrose inverse exists for any matrix $A$ and is unique.
  2. For a non-singular (invertible) matrix $A$, its MP inverse is precisely the usual inverse $A^{-1}$.
  3. If $A $ is a symmetric matrix, then its MP inverse $A^+$ is also symmetric. In fact, if

$$ A = H' \,\mathrm{diag}(\lambda_1, \lambda_2, \dots, \lambda_n)\,H, $$

where $H$ is an orthogonal matrix, then

$$ A^+ = H' \,\mathrm{diag}(\lambda_1^+, \lambda_2^+, \dots, \lambda_n^+) \,H, $$

with

$$ \lambda_i^+ = \begin{cases} \frac{1}{\lambda_i}, & \lambda_i \neq 0,\\ 0, & \lambda_i = 0. \end{cases} $$

To find the MP inverse of a general matrix $A$, one often uses

$$ A^+ = A' \bigl(A A'\bigr)^+ = \bigl(A' A\bigr)^+ A'. $$