MSA_Week2 1.2 Minus Inverse

1.2.1 Minus Inverse

Definition 1.2.1 (Minus Inverse)

Let $A$ be an $n \times p$ matrix. If there exists a $p \times n$ matrix $X$ such that

$$A\,X\,A = A,$$

then $X$ is called a minus inverse of $A$, denoted by $A^-.$

Property 1.2.1

If $A$ is nonsingular (invertible), then $A^-$ is unique and

$$A^- = A^{-1}.$$

Proof. Since $A \times A^{-1} \times A = A$, $A^{-1}$ is a generalized inverse of $A$. If $X$ is another generalized inverse, then $A \times X \times A = A$. Multiplying on the left and right by $A^{-1}$ yields $X = A^{-1}$. Hence uniqueness.

Property 1.2.2

Every matrix $A$ has at least one minus inverse, but it may not be unique (unless $A$ is invertible).

Sketch of construction/proof. Suppose $\mathrm {rank}(A) = r$. There exist nonsingular matrices $P \quad (\text{size } n \times n)$ and $Q \quad (\text{size } p \times p)$ such that

$$A = P \begin{pmatrix} I_r & 0 \\ 0 & 0 \end{pmatrix} Q.$$

Equivalently:

$$A = P \begin{pmatrix} I_r & 0 \\ 0 & 0 \end{pmatrix} Q = \bigl(P\bigr) \Bigl( I_r \quad 0 \Bigr) \begin{pmatrix} I_r \\ 0 \end{pmatrix} \bigl(Q\bigr),$$

where $P$ and $Q$ are invertible, and the middle block has rank $r$.

To construct $A^-$, consider:

$$X = Q^{-1} \begin{pmatrix} I_r & T_{12} \\ T_{21} & T_{22} \end{pmatrix} P^{-1},$$

where $T_{12}, T_{21}, T_{22}$ can be arbitrarily chosen (of compatible dimensions). One checks that $A\,X\,A = A.$ Thus $X$ is a minus inverse of $A$. This shows existence but also shows that in general it is not unique (unless $A$ is invertible).

Property 1.2.3

For any $n \times p$ matrix $A$:

$$\mathrm{rank}(A^-) \;\ge\; \mathrm{rank}(A).$$

Proof. From the same decomposition

$$A = P \begin{pmatrix} I_r & 0 \\ 0 & 0 \end{pmatrix} Q,$$

we take

$$A^- = Q^{-1} \begin{pmatrix} I_r & T_{12}\\ T_{21} & T_{22} \end{pmatrix} P^{-1}.$$

Because

$$\begin{pmatrix} I_r & T_{12}\\ T_{21} & T_{22} \end{pmatrix}$$

has rank at least (r), it follows that

$$\mathrm{rank}(A^-) \;\ge\; r \;=\; \mathrm{rank}(A).$$

Property 1.2.4

For any $n \times p$ matrix $A$:

$$\mathrm{rank}(A) = \mathrm{rank}\bigl(A\,A^-\bigr) = \mathrm{rank}\bigl(A^-\,A\bigr) = \mathrm{tr}\bigl(A\,A^-\bigr) = \mathrm{tr}\bigl(A^-\,A\bigr).$$

Proof. Let $\mathrm{rank}(A) = r$. Then

$$A = P \begin{pmatrix} I_r & 0 \\ 0 & 0 \end{pmatrix} Q, \quad A^- = Q^{-1} \begin{pmatrix} I_r & T_{12}\\ T_{21} & T_{22} \end{pmatrix} P^{-1}.$$

Hence

$$A\,A^- = P \begin{pmatrix} I_r & 0 \\ 0 & 0 \end{pmatrix} Q \,Q^{-1} \begin{pmatrix} I_r & T_{12}\\ T_{21} & T_{22} \end{pmatrix} P^{-1} = P \begin{pmatrix} I_r & T_{12}\\ 0 & 0 \end{pmatrix} P^{-1}.$$

It follows that

$$\mathrm{rank}(A\,A^-) = r.$$

Also,

$$(A\,A^-)^2 = A\,A^-\,A\,A^- = A\,A^-,$$

which implies $A\,A^-$ is idempotent; hence

$$\mathrm{rank}(A\,A^-) = \mathrm{tr}(A\,A^-) = r.$$

A similar argument shows

$$\mathrm{rank}(A^-\,A) = \mathrm{tr}(A^-\,A) = r.$$

In particular:

• If $\mathrm{rank}(A)=p$ (full column rank), then $A^-\,A = I_p$.
• If $\mathrm {rank}(A)=n$ (full row rank), then $A\,A^- = I_n$.

Property 1.2.5

For any $n \times p$ matrix $A$,

$$A'(A' A)^- A' \;=\; A', \quad A\,(A' A)^-\;A' A \;=\; A.$$

(Here $A'$ denotes $A$ transposed.)

Sketch of the proof.

Show

$$A\,x = 0 \;\;\Longleftrightarrow\;\; A'A\,x = 0.$$

Show

$$A\,x = A\,y \;\;\Longleftrightarrow\;\; A'A\,x = A'A\,y.$$

From these, deduce

$$A'(A'A)^-A' = A' \quad\text{and}\quad A\,(A'A)^-\,A'A = A.$$

A key step is “canceling” a factor with proper rank conditions. The result is

$$A\,(A'A)^-A' A = A \quad\text{and}\quad A'(A'A)^-A' = A'.$$

(Additional Note on “Cancelation”)

$$A\,B\,C = 0 \;\;\text{and}\;\; B\,C = 0 \quad\Longleftrightarrow\quad \mathrm{rank}(A\,B) = \mathrm{rank}(B).$$ $$C\,A\,B = 0 \;\;\text{and}\;\; C\,A = 0 \quad\Longleftrightarrow\quad \mathrm{rank}(A\,B) = \mathrm{rank}(A).$$

Property 1.2.6

For any matrix $A$, the matrix

$$A\,(A'A)^-\,A'$$

is a projection matrix (i.e., it is idempotent) and does not depend on the particular choice of the minus inverse $A'A^-$.

1. Independence of the choice

If $(A'A)^{-1}_1 \quad \text{and} \quad (A'A)^{-1}_2$ are two different minus inverses of $A'A$, one can show

$$A' A \,(A'A)^-_1\, A' A \;=\; A' A \;=\; A' A \,(A'A)^-_2\, A' A,$$

which implies

$$A\,(A'A)^-_1\,A' \;=\; A\,(A'A)^-_2\,A'.$$

Hence the product $(A, (A'A)^{-1}, A')$ is indeed independent of which minus inverse is chosen.

2. Symmetry

Since $A'A$ is symmetric, it can be diagonalized by an orthogonal matrix. A suitable choice of $(A'A)^{-1}$ can also be made symmetric, implying that

$$A\,(A'A)^-\,A'$$

is itself symmetric.

3. Idempotence

One checks

$$\bigl(A\,(A'A)^-\,A'\bigr)^2 \;=\; A\,(A'A)^-\, \underbrace{A'\,A}_{\text{common factor}}\, (A'A)^-\, A' \;=\; A\,(A'A)^-\,A'.$$

Hence

$$A\,(A'A)^-\,A'$$

is idempotent; in other words, it is a projection matrix.

Property 1.2.7

For any arbitrary matrix $A$, the matrix

$$A (A'A)^{-} A'$$

is the projection matrix onto the column space $R(A)$. Denote it by

$$P_A \;\equiv\; A (A'A)^{-} A'.$$

1. For every $x \in R(A)$, we have

$$P_A x \;=\; x.$$

1. For every $u \in \mathbb{R}^n$, we have

$$P_A u \;\in\; R(A).$$

Proof.

1. For every $x \in R(A),$ there exists some $y \in \mathbb{R}^p$ such that $x = A y$. Then

$$P_A x \;=\; A (A'A)^{-} A' \,A y \;=\; A y \;=\; x.$$

1. For every $u \in \mathbb{R}^n$,

$$P_A u \;=\; A (A'A)^{-} A' \,A u \;=\; A\bigl((A'A)^{-} A' \,A u\bigr) \;\in\; R(A).$$

Therefore,

$$P_A \;=\; A (A'A)^{-} A'$$

is indeed the projection matrix onto $R(A)$.