MSA_Week4: Vectorization and Kronecker Product

$1.3$ Vectorization and Kronecker Product

Definition $1.3.1$ (Vectorization)

Let

$$ \mathbf{A} = a_{ij} $$

be an

$$ n \times p $$

matrix, expressed as

$$ \mathbf{A} = \bigl(a_1, a_2, \dots, a_p\bigr) = \begin{pmatrix} a_{(1)}' \\[6pt] a_{(2)}' \\[6pt] \vdots \\[6pt] a_{(n)}' \end{pmatrix}. $$

The vectorization of $\mathbf{A}$, denoted by $\mathrm{vec}(\mathbf{A})$, is defined as

$$ \mathrm{vec}(\mathbf{A}) = \begin{pmatrix} a_1 \\[6pt] a_2 \\[6pt] \vdots \\[6pt] a_p \end{pmatrix}. $$

Property $1.3.1$ (Linearity)

Let $c, d$ be real numbers and $\mathbf{A}, \mathbf{B}$ matrices of identical dimensions. Then,

$$ \mathrm{vec}(c\mathbf{A}+d\mathbf{B}) = c\,\mathrm{vec}(\mathbf{A}) + d\,\mathrm{vec}(\mathbf{B}). $$

Property $1.3.2$ (Alternative Representations)

For an $n \times p$ matrix $\mathbf{A}$, we have

$$ \mathbf{A} = \sum_{i,j} a_{ij}\mathbf{E}_{ij} = \sum_{j} a_j e_j' = \sum_{i} e_i a_{(i)}', $$

where $\mathbf{E}_{ij}$ is an $n\times p$ matrix with 1 in position $(i,j)$ and 0 elsewhere, $e_i$ is an $n$-dimensional column vector, and $e_j$ is a $p$-dimensional column vector.

Property $1.3.3$ (Row and Column Representation)

Columns $a_j$ and rows $a_{(i)}'$ of $\mathbf{A}$ satisfy

$$ a_j = \mathbf{A} e_j = \sum_{i}a_{ij} e_i, $$ $$ a_{(i)} = \mathbf{A}' e_i = \sum_{j} a_{ij} e_j. $$

Property $1.3.4$ (Element Representation)

Any element $a_{ij}$ of $\mathbf{A}$ is represented by

$$ a_{ij} = e_i' \mathbf{A} e_j. $$

Property $1.3.5$ (Trace Representation)

Let $\mathbf{A}_{p\times q}, \mathbf{B}_{q\times p}$. Then

$$ \mathrm{tr}(\mathbf{A}\mathbf{B}) = [\mathrm{vec}(\mathbf{A}')]'\,[\mathrm{vec}(\mathbf{B})]. $$

Definition $1.3.2$ (Symmetric Vectorization)

For a symmetric $p\times p$ matrix $\mathbf{A}$, define its symmetric vectorization as

$$ \mathrm{svec}(\mathbf{A}) = (a_{11},a_{21},\dots,a_{p1},a_{22},a_{32},\dots,a_{p2},\dots,a_{pp})'. $$

Kronecker Product

Definition $1.3.3$ (Kronecker Product)

Given matrices $\mathbf{A}=a_{ij}$ with dimensions $n\times p$ and $\mathbf{B}$ with dimensions $m\times q$, define the Kronecker product $\mathbf{A}\otimes \mathbf{B}$ as

$$ \mathbf{A}\otimes\mathbf{B} = \begin{pmatrix} a_{11}\mathbf{B} & \dots & a_{1p}\mathbf{B}\\[6pt] \vdots & \ddots & \vdots\\[6pt] a_{n1}\mathbf{B} & \dots & a_{np}\mathbf{B} \end{pmatrix}, $$

which is an $nm\times pq$ matrix.

Property $1.3.6$ (Scalar Multiplication)

For a scalar $\alpha$, we have

$$ (\alpha \mathbf{A})\otimes\mathbf{B} = \mathbf{A}\otimes(\alpha\mathbf{B}) = \alpha(\mathbf{A}\otimes\mathbf{B}). $$

Property $1.3.7$ (Distributivity)

For matrices $\mathbf{B},\mathbf{C}$ of identical dimensions, we have

$$ \mathbf{A}\otimes(\mathbf{B}+\mathbf{C})=\mathbf{A}\otimes\mathbf{B}+\mathbf{A}\otimes\mathbf{C}, $$ $$ (\mathbf{B}+\mathbf{C})\otimes\mathbf{A}=\mathbf{B}\otimes\mathbf{A}+\mathbf{C}\otimes\mathbf{A}. $$

Property $1.3.8$ (Associativity)

For matrices $\mathbf{A},\mathbf{B},\mathbf{C}$,

$$ (\mathbf{A}\otimes\mathbf{B})\otimes\mathbf{C}=\mathbf{A}\otimes(\mathbf{B}\otimes\mathbf{C}). $$

Property $1.3.9$ (Identity Matrix)

For identity matrices $\mathbf{I}_n,\mathbf{I}_m$,

$$ \mathbf{I}_{mn}=\mathbf{I}_m\otimes\mathbf{I}_n=\mathbf{I}_n\otimes\mathbf{I}_m. $$

Property $1.3.10$ (Transpose)

$$ (\mathbf{A}\otimes\mathbf{B})'=\mathbf{A}'\otimes\mathbf{B}'. $$

Property $1.3.11$ (Multiplication Rule)

If dimensions match, we have

$$ (\mathbf{A}\otimes\mathbf{B})(\mathbf{C}\otimes\mathbf{D})=(\mathbf{A}\mathbf{C})\otimes(\mathbf{B}\mathbf{D}). $$

Property $1.3.12$ (Inverse)

If $\mathbf{A},\mathbf{B}$ are nonsingular square matrices, then

$$ (\mathbf{A}\otimes\mathbf{B})^{-1}=\mathbf{A}^{-1}\otimes\mathbf{B}^{-1}. $$

Property $1.3.13$ (Relation Between Vectorization and Kronecker Product)

For $\mathbf{A}_{n\times m},\mathbf{X}_{m\times p},\mathbf{B}_{p\times q}$,

$$ \mathrm{vec}(\mathbf{A}\mathbf{X}\mathbf{B})=(\mathbf{B}'\otimes\mathbf{A})\,\mathrm{vec}(\mathbf{X}). $$

Property $1.3.14$ (Trace)

For square matrices $\mathbf{A},\mathbf{B}$,

$$ \mathrm{tr}(\mathbf{A}\otimes\mathbf{B})=\mathrm{tr}(\mathbf{A})\cdot\mathrm{tr}(\mathbf{B}). $$

Property $1.3.15$ (Rank-1 Decomposition)

For column vectors $x,y$,

$$ xy'=x\otimes y'=y'\otimes x. $$