Table of Contents¶

Projection and idempotent matrix¶

Highlights: In this lecture, we learn that (1) $\mathbb{R}^m = \mathcal{C}(\mathbf{A}) \oplus \mathcal{N}(\mathbf{A}')$, $\mathbb{R}^n = \mathcal{C}(\mathbf{A}') \oplus \mathcal{N}(\mathbf{A})$; (2) any idempotent matrix $\mathbf{P}$ is a projector into $\mathcal{C}(\mathbf{P})$ along $\mathcal{N}(\mathbf{P}) = \mathcal{C}(\mathbf{I} - \mathbf{P})$; (3) Given $\mathbb{R}^n = \mathcal{S}_1 \oplus \mathcal{S}_2$, know how to construct a projector $\mathbf{P}$ that projects into $\mathcal{S}_1$ along $\mathcal{S}_2$.

Note when many authors say projection or projectors, they actually mean orthogonal projection or orthogonal projectors, which are subject of next lecture. In this course, we differentiate (oblique) projection and orthogonal projection.

Direct sum¶

The sum of two vector spaces $\mathcal{S}_1$ and $\mathcal{S}_2$ of same order is $$ \mathcal{S}_1 + \mathcal{S}_2 = \{\mathbf{x}_1 + \mathbf{x}_2: \mathbf{x}_1 \in \mathcal{S}_1, \mathbf{x}_2 \in \mathcal{S}_2\}. $$

In HW4, we showed that $\mathcal{S}_1 + \mathcal{S}_2$ is a vector space.
Dimension of a sum of subspaces. Let $\mathcal{S}_1$ and $\mathcal{S}_2$ be two subspaces in $\mathbb{R}^n$. Then $$ \text{dim}(\mathcal{S}_1 + \mathcal{S}_2) = \text{dim}(\mathcal{S}_1) + \text{dim}(\mathcal{S}_2) - \text{dim}(\mathcal{S}_1 \cap \mathcal{S}_2). $$

Recall that the sum of two vector spaces $\mathcal{S}_1 + \mathcal{S}_2$ is defined in HW4.

Proof (optional): Idea: extend a basis of $\mathcal{S}_1 \cap \mathcal{S}_2$ to ones for $\mathcal{S}_1$ and $\mathcal{S}_2$. See, e.g., BR p156-157.
Corollaries.
1. Subadditivity of dim function. Let $\mathcal{S}_1$ and $\mathcal{S}_2$ be two subspaces in $\mathbb{R}^n$. Then $$ \text{dim}(\mathcal{S}_1 + \mathcal{S}_2) \le \text{dim}(\mathcal{S}_1) + \text{dim}(\mathcal{S}_2). $$
  
  This is an immediate corollary of the preceding theorem. In HW4, try to give a self-contained proof without using the preceding theorem.
2. Subadditivity of rank function. Let $\mathbf{A}$ and $\mathbf{B}$ be two matrices of the same order. Then
  $$ \text{rank}(\mathbf{A} + \mathbf{B}) \le \text{rank}(\mathbf{A}) + \text{rank}(\mathbf{B}). $$
  
  Proof: First claim trivially follows from the previous result or your own proof in HW4. For the second claim. First note $\mathcal{C}(\mathbf{A} + \mathbf{B}) \subseteq \mathcal{C}(\mathbf{A}) + \mathcal{C}(\mathbf{B})$ (why?). Then \begin{eqnarray*} & & \text{rank}(\mathbf{A} + \mathbf{B}) \\ &=& \text{dim}(\mathcal{C}(\mathbf{A} + \mathbf{B})) \quad \text{(definition of rank)} \\ &\le& \text{dim}(\mathcal{C}(\mathbf{A}) + \mathcal{C}(\mathbf{B})) \quad \text{(monotonicity of dim)} \\ &\le& \text{dim}(\mathcal{C}(\mathbf{A})) + \text{dim}(\mathcal{C}(\mathbf{B})) \quad \text{(subadditivity of dim)} \\ &=& \text{rank}(\mathbf{A}) + \text{rank}(\mathbf{B}). \quad \text{(definition of rank)} \end{eqnarray*} The second inequality follows from the first claim.

Two subspaces $\mathcal{S}_1$ and $\mathcal{S}_2$ in a vector space $\mathcal{V}$ are said to be complementary whenever $$ \mathcal{V} = \mathcal{S}_1 + \mathcal{S}_2 \text{ and } \mathcal{S}_1 \cap \mathcal{S}_2 = \{\mathbf{0}\}. $$ In such cases, we say $\mathcal{V}$ is a direct sum of $\mathcal{S}_1$ and $\mathcal{S}_2$ and denote $\mathcal{V} = \mathcal{S}_1 \oplus \mathcal{S}_2$.
TODO: visualize. $\mathbb{R}^3 = \text{a plane} \oplus \text{a line}$.
Let $\mathcal{S}_1, \mathcal{S}_2$ be two subspaces of same order and $\mathcal{V} = \mathcal{S}_1 + \mathcal{S}_2$. Following statements are equivalent:
1. $\mathcal{V} = \mathcal{S}_1 \oplus \mathcal{S}_2$.
2. $\text{dim}(\mathcal{V}) = \text{dim}(\mathcal{S}_1) + \text{dim}(\mathcal{S}_2)$.
3. Any vector $\mathbf{x} \in \mathcal{V}$ can be uniquely represented as $$ \mathbf{x} = \mathbf{x}_1 + \mathbf{x}_2, \text{ where } \mathbf{x}_1 \in \mathbf{S}_1, \mathbf{x}_2 \in \mathbf{S}_2. $$ We will refer to this as the unique representation or unique decomposition property of direct sums.
  
  Proof (optional): We show that $1 \Rightarrow 2 \Rightarrow 3 \Rightarrow 1$.
  $1 \Rightarrow 2$: By definition of direct sum, we know $\mathcal{S}_1 \cap \mathcal{S}_2 = \{\mathbf{0}\}$. Thus $\text{dim}(\mathcal{S}_1 + \mathcal{S}_2) = \text{dim}(\mathcal{S}_1) + \text{dim}(\mathcal{S}_2)$.
  $2 \Rightarrow 3$: By 2, we know $\text{dim}(\mathcal{S}_1 \cap \mathcal{S}_2) = 0$ so $\mathcal{S}_1 \cap \mathcal{S}_2 = \{\mathbf{0}\}$. Let $\mathbf{x} \in \mathcal{S}_1 + \mathcal{S}_2$ and assume $\mathbf{x}$ can be decomposed in two ways: $\mathbf{x} = \mathbf{u}_1 + \mathbf{u}_2 = \mathbf{v}_1 + \mathbf{v}_2$, where $\mathbf{u}_1, \mathbf{v}_1 \in \mathcal{S}_1$ and $\mathbf{u}_2, \mathbf{v}_2 \in \mathcal{S}_2$. Then $\mathbf{u}_1 - \mathbf{v}_1 = -(\mathbf{u}_2 - \mathbf{v}_2)$, indicating that the vectors $\mathbf{u}_1 - \mathbf{v}_1$ and $\mathbf{u}_2 - \mathbf{v}_2$ belong to both $\mathcal{S}_1$ and $\mathcal{S}_2$ and thus must be $\mathbf{0}$. Therefore $\mathbf{u}_1 = \mathbf{v}_1$ and $\mathbf{u}_2 = \mathbf{v}_2$.
  $3 \Rightarrow 1$: We only need to show $\mathcal{S}_1 \cap \mathcal{S}_2 = \{\mathbf{0}\}$. Let $\mathbf{x} \in \mathcal{S}_1 \cap \mathcal{S}_2$. Decompose $\mathbf{x}$ in two ways: $\mathbf{x} = \mathbf{x} + \mathbf{0} = \mathbf{0} + \mathbf{x}$. Then by the uniqueness part of 3, $\mathbf{x}=\mathbf{0}$. So the only possible element in $\mathcal{S}_1 \cap \mathcal{S}_2$ is $\mathbf{0}$.

An immediate corollary of the previous result is, for a matrix $\mathbf{A} \in \mathbb{R}^{m \times n}$,
1. $\mathbb{R}^m = \mathcal{C}(\mathbf{A}) \oplus \mathcal{N}(\mathbf{A}')$.
2. $\mathbb{R}^n = \mathcal{C}(\mathbf{A}') \oplus \mathcal{N}(\mathbf{A})$.

Projection¶

Let $\mathbb{R}^n = \mathcal{S} \oplus \mathcal{T}$ and $\mathbf{y} = \mathbf{x} + \mathbf{z}$ be the unique representation of $\mathbf{y} \in \mathbb{R}^n$ with $\mathbf{x} \in \mathcal{S}$ and $\mathbf{z} \in \mathcal{T}$. Then
- the vector $\mathbf{x}$ is called the projection of $\mathbf{y}$ into $\mathcal{S}$ along $\mathcal{T}$;
- the vector $\mathbf{z}$ is called the projection of $\mathbf{y}$ into $\mathcal{T}$ along $\mathcal{S}$.
Let $\mathbb{R}^n = \mathcal{S} \oplus \mathcal{T}$. We call a matrix $\mathbf{P} \in \mathbb{R}^{n \times n}$ a projection matrix or projector into $\mathcal{S}$ along $\mathcal{T}$ if, for any $\mathbf{y} \in \mathbb{R}^n$, $\mathbf{P} \mathbf{y}$ is the projection of $\mathbf{y}$ into $\mathcal{S}$ along $\mathcal{T}$.
If $\mathbf{P}$ is a projector into $\mathcal{S}$ along $\mathcal{T}$, then
1. $\mathbf{P} \mathbf{x} = \mathbf{x}$ for all $\mathbf{x} \in \mathcal{S}$.
2. $\mathbf{P} \mathbf{z} = \mathbf{0}$ for all $\mathbf{z} \in \mathcal{T}$.
  
  Proof: For 1, since $\mathbf{x} = \mathbf{x} + \mathbf{0}$, where $\mathbf{x} \in \mathcal{S}$ and $\mathbf{0} \in \mathcal{T}$. By uniqueness of the projection, $\mathbf{P} \mathbf{x}$ must be equal to $\mathbf{x}$. For 2, $\mathbf{z} = \mathbf{0} + \mathbf{z}$ is the unique decomposition. Then $\mathbf{P} \mathbf{z}$ must be $\mathbf{0}$.
A square matrix $\mathbf{P}$ is said to be idempotent if $\mathbf{P}^2 = \mathbf{P}$.
Following statements about a square matrix $\mathbf{P} \in \mathbb{R}^{n \times n}$ are equivalent:
1. $\mathbf{P}$ is a projector.
2. $\mathbf{P}$ and $\mathbf{I} - \mathbf{P}$ are idemponent. That is $\mathbf{P}^2 = \mathbf{P}$ and $(\mathbf{I} - \mathbf{P})^2 = \mathbf{I} - \mathbf{P}$.
3. $\mathcal{N}(\mathbf{P}) = \mathcal{C}(\mathbf{I} - \mathbf{P})$.
4. $\text{rank}(\mathbf{P}) + \text{rank}(\mathbf{I} - \mathbf{P}) = n$.
5. $\mathbb{R}^n = \mathcal{C}(\mathbf{P}) \oplus \mathcal{C}(\mathbf{I} - \mathbf{P})$.
  We can use any of these five statements as a definition of a projector.
  In words, any idempotent matrix $\mathbf{P}$ projects into $\mathcal{C}(\mathbf{P})$ along $\mathcal{N}(\mathbf{P})$ and $\mathbf{I} - \mathbf{P}$ projects into $\mathcal{N}(\mathbf{P})$ along $\mathcal{C}(\mathbf{P})$.
  
  Proof (optional): We show $1 \Rightarrow 2 \Rightarrow 3 \Rightarrow 4 \Rightarrow 5 \Rightarrow 1$.
  $1 \Rightarrow 2$: Suppose $\mathbf{P}$ is a projector into $\mathcal{S}$ along $\mathcal{T}$, where $\mathcal{S} \oplus \mathcal{T} = \mathbb{R}^n$. Then $\mathbf{P}^2 \mathbf{y} = \mathbf{P}(\mathbf{P} \mathbf{y}) = \mathbf{P} \mathbf{y}$ for any $\mathbf{y} \in \mathbb{R}^n$. Now taking $\mathbf{y} = \mathbf{e}_1, \ldots, \mathbf{e}_n$, we showed each column of $\mathbf{P}^2$ is equal to the corresponding column of $\mathbf{P}$. Therefore $\mathbf{P}^2 = \mathbf{P}$. Finally $(\mathbf{I} - \mathbf{P})^2 = \mathbf{I} - 2\mathbf{P} + \mathbf{P}^2 = \mathbf{I} - \mathbf{P}$.
  $2 \Rightarrow 3$: To show $\mathcal{N}(\mathbf{P}) \supseteq \mathcal{C}(\mathbf{I} - \mathbf{P})$, \begin{eqnarray*} & & \mathbf{x} \in \mathcal{C}(\mathbf{I} - \mathbf{P}) \\ &\Rightarrow& \mathbf{x} = (\mathbf{I} - \mathbf{P}) \mathbf{v} \text{ for some } \mathbf{v} \\ &\Rightarrow& \mathbf{P} \mathbf{x} = \mathbf{P} (\mathbf{I} - \mathbf{P}) \mathbf{v} = (\mathbf{P} - \mathbf{P}^2) \mathbf{v} = \mathbf{0}_{n \times n} \mathbf{v} = \mathbf{0} \\ &\Rightarrow& \mathbf{x} \in \mathcal{N}(\mathbf{P}). \end{eqnarray*} To show $\mathcal{N}(\mathbf{P}) \subseteq \mathcal{C}(\mathbf{I} - \mathbf{P})$, \begin{eqnarray*} & & \mathbf{x} \in \mathcal{N}(\mathbf{P}) \\ &\Rightarrow& \mathbf{P} \mathbf{x} = \mathbf{0} \\ &\Rightarrow& \mathbf{x} = \mathbf{x} - \mathbf{P} \mathbf{x} = (\mathbf{I} - \mathbf{P}) \mathbf{x} \\ &\Rightarrow& \mathbf{x} \in \mathcal{C}(\mathbf{I} - \mathbf{P}). \end{eqnarray*} $3 \Rightarrow 4$: By the Rank-Nullity theorem, $$ \text{rank}(\mathbf{P}) = n - \text{nullity}(\mathbf{P}) = n - \text{dim}(\mathcal{N}(\mathbf{P})) = n - \text{dim}(\mathcal{C}(\mathbf{I} - \mathbf{P})) = n - \text{rank}(\mathcal{C}(\mathbf{I} - \mathbf{P})). $$
  $4 \Rightarrow 5$: For any $\mathbf{x} \in \mathbb{R}^n$, $\mathbf{x} = (\mathbf{P} + \mathbf{I} - \mathbf{P}) \mathbf{x} = \mathbf{P} \mathbf{x} + (\mathbf{I} - \mathbf{P}) \mathbf{x}$. Thus $\mathbb{R}^n = \mathcal{C}(\mathbf{P}) + \mathcal{C}(\mathbf{I} - \mathbf{P})$. Then $\mathcal{C}(\mathbf{P})$ and $\mathcal{C}(\mathbf{I} - \mathbf{P})$ must be a direct sum by earlier result (3 equivalent properties of direct sum).
  $5 \Rightarrow 1$: For any $\mathbf{x} \in \mathbb{R}^n$, $\mathbf{x} = (\mathbf{P} + \mathbf{I} - \mathbf{P}) \mathbf{x} = \mathbf{P} \mathbf{x} + (\mathbf{I} - \mathbf{P}) \mathbf{x}$, where $\mathbf{P} \mathbf{x} \in \mathcal{C}(\mathbf{P})$ and $(\mathbf{I} - \mathbf{P}) \mathbf{x} \in \mathcal{C}(\mathbf{I} - \mathbf{P})$. 5 then confirms that $\mathbf{P}$ projects into $\mathcal{C}(\mathbf{P})$ along $\mathcal{C}(\mathbf{I} - \mathbf{P})$.
Let $\mathbf{P} \in \mathbb{R}^{n \times n}$ has rank $r$ and $\mathbf{P} = \mathbf{C} \mathbf{R}$ be a rank factorization of $\mathbf{P}$. Then $\mathbf{P}$ is a projector if and only if $\mathbf{R} \mathbf{C} = \mathbf{I}_r$.

Proof: Since $\mathbf{P} = \mathbf{C} \mathbf{R}$ is a rank factorization, we have $\mathbf{C} \in \mathbb{R}^{n \times r}$ and $\mathbf{R} \in \mathbb{R}^{r \times n}$.
If $\mathbf{R} \mathbf{C} = \mathbf{I}_r$, then $$ \mathbf{P}^2 = \mathbf{C} \mathbf{R} \mathbf{C} \mathbf{R} = \mathbf{C} \mathbf{I}_r \mathbf{R} = \mathbf{C} \mathbf{R} = \mathbf{P}. $$ If $\mathbf{P}$ is a projector, then $\mathbf{P}$ is idemponent. That is $$ \mathbf{C} \mathbf{R} \mathbf{C} \mathbf{R} = \mathbf{C} \mathbf{R}. $$ Since $\mathbf{P} = \mathbf{C} \mathbf{R}$ is a rank factorization, $\text{rank}(\mathbf{C}) = \text{rank}(\mathbf{P}) = r$. $\mathbf{C}$ has full column rank (rows of $\mathbf{C}$ span $\mathbb{R}^r$); thus there exists a left inverse $\mathbf{C}_L$ such that $\mathbf{C}_L \mathbf{C} = \mathbf{I}_r$. Similarly $\mathbf{R}$ having full row rank (columns of $\mathbf{R}$ span $\mathbb{R}^r$) implies that there exists a right inverse $\mathbf{R}_R$ such that $\mathbf{R} \mathbf{R}_R = \mathbf{I}_r$. Pre-multiplying by $\mathbf{C}_L$ and post-multiplying by $\mathbf{R}_R$ on both sides of above equation $$ \mathbf{C}_L \mathbf{C} \mathbf{R} \mathbf{C} \mathbf{R} \mathbf{R}_R = \mathbf{C}_L \mathbf{C} \mathbf{R} \mathbf{R}_R. $$ gives the desired identity $\mathbf{R}\mathbf{C} = \mathbf{I}_r$.
Above result gives us a recipe to construct (oblique) projectors. Suppose $\mathbf{R}^n = \mathcal{S}_1 \oplus \mathcal{S}_2$. How to construct a projector $\mathbf{P}$ that projects into $\mathcal{S}_1$ along $\mathcal{S}_2$?

Solution: Let columns of $\mathbf{B}_1 \in \mathbb{R}^{n \times r}$ be a basis of $\mathcal{S}_1$ and columns of $\mathbf{B}_2 \in \mathbb{R}^{n \times (n-r)}$ be a basis of $\mathcal{S}_2$. Let $\mathbf{P} = \mathbf{B}_1 \mathbf{R}$ is a rank factorization of the desired projector. Then $\mathbf{R} \mathbf{B}_1 = \mathbf{I}_r$ and $\mathbf{R} \mathbf{B}_2 = \mathbf{0}_{r \times (n-r)}$. That is $$ \mathbf{R} (\mathbf{B}_1 \,\, \mathbf{B}_2) = (\mathbf{I}_r \,\, \mathbf{0}_{r \times (n-r)}). $$ We get $\mathbf{R} = (\mathbf{I}_r \,\, \mathbf{0}_{r \times (n-r)}) (\mathbf{B}_1 \,\, \mathbf{B}_2)^{-1}$.
Let $\mathbf{P}$ be a projector, or equivalently idempotent. Then $\text{tr}(\mathbf{P}) = \text{rank}(\mathbf{P})$.

Proof: By HW2 (BV 10.11), $$ \text{tr}(\mathbf{P}) = \text{tr}(\mathbf{C} \mathbf{R}) = \text{tr}(\mathbf{R} \mathbf{C}) = \text{tr}(\mathbf{I}_r) = r = \text{rank}(\mathbf{P}). $$
Example: Homework 5. Find the 3 projectors in the picture.

using LinearAlgebra
n = 5
# this P is symmetric, thus indeed an orthogonal projector
P = (1/n) * ones(n, n)

5×5 Matrix{Float64}:
 0.2  0.2  0.2  0.2  0.2
 0.2  0.2  0.2  0.2  0.2
 0.2  0.2  0.2  0.2  0.2
 0.2  0.2  0.2  0.2  0.2
 0.2  0.2  0.2  0.2  0.2

# idempotent
P^2

5×5 Matrix{Float64}:
 0.2  0.2  0.2  0.2  0.2
 0.2  0.2  0.2  0.2  0.2
 0.2  0.2  0.2  0.2  0.2
 0.2  0.2  0.2  0.2  0.2
 0.2  0.2  0.2  0.2  0.2

tr(P)

1.0

rank(P), rank(I - P)

(1, 4)

# let's try a non-orthogonal projection: P = c * ones(n) * v'
using Random
Random.seed!(216)
v = randn(n)
c = 1 / sum(v)
P = c * ones(n) * v'

5×5 Matrix{Float64}:
 0.892643  1.00408  0.170126  0.226762  -1.29362
 0.892643  1.00408  0.170126  0.226762  -1.29362
 0.892643  1.00408  0.170126  0.226762  -1.29362
 0.892643  1.00408  0.170126  0.226762  -1.29362
 0.892643  1.00408  0.170126  0.226762  -1.29362

P * P

5×5 Matrix{Float64}:
 0.892643  1.00408  0.170126  0.226762  -1.29362
 0.892643  1.00408  0.170126  0.226762  -1.29362
 0.892643  1.00408  0.170126  0.226762  -1.29362
 0.892643  1.00408  0.170126  0.226762  -1.29362
 0.892643  1.00408  0.170126  0.226762  -1.29362

tr(P)

1.0000000000000002

rank(P), rank(I - P)

(1, 4)

Cochran theorem (optional)¶

This theorem is important for deriving F test in linear models.

Let $\mathbf{P}_1, \ldots, \mathbf{P}_k$ be $n \times n$ matrices with $\sum_{i=1}^k \mathbf{P}_i = \mathbf{I}_n$. Then the following statements are equivalent:
1. $\sum_{i=1}^k \text{rank}(\mathbf{P}_i) = n$.
2. $\mathbf{P}_i \mathbf{P}_j = \mathbf{0}_{n \times n}$ for $i \ne j$.
3. $\mathbf{P}_i^2 = \mathbf{P}_i$ for $i=1,\ldots,k$. That is each $\mathbf{P}_i$ is a projector.
  
  Proof: We will prove $1 \Rightarrow 2 \Rightarrow 3 \Rightarrow 1$.
  Proof of $1 \Rightarrow 2$: Let $r_i = \text{rank}(\mathbf{P}_i)$ and $\mathbf{P}_i = \mathbf{S}_i \mathbf{U}_i'$ be a rank factorization of $\mathbf{P}_i$ for $i=1,\ldots,k$, where $\mathbf{S}_i \in \mathbb{R}^{n \times r_i}$ and $\mathbf{U}_i' \in \mathbb{R}^{r_i \times n}$ with $\text{rank}(\mathbf{S}_i) = \text{rank}(\mathbf{U}_i) = r_i$. Then \begin{eqnarray*} & & \sum_{i=1}^k \mathbf{P}_i = \mathbf{I}_n \\ &\Rightarrow& \mathbf{S}_1 \mathbf{U}_1' + \cdots + \mathbf{S}_k \mathbf{U}_k' = \mathbf{I}_n \\ &\Rightarrow& \begin{pmatrix} \mathbf{S}_1 \, \cdots \, \mathbf{S}_k \end{pmatrix} \begin{pmatrix} \mathbf{U}_1' \\ \vdots \\ \mathbf{U}'_k \end{pmatrix} = \mathbf{I}_n \\ &\Rightarrow& \mathbf{S} \mathbf{U}' = \mathbf{I}_n, \end{eqnarray*} where $$ \mathbf{S} = \begin{pmatrix} \mathbf{S}_1 \, \cdots \, \mathbf{S}_k \end{pmatrix}, \quad \mathbf{U}' = \begin{pmatrix} \mathbf{U}_1' \\ \vdots \\ \mathbf{U}'_k \end{pmatrix}. $$ Since $\sum_{i=1}^k r_i = n$, $\mathbf{S}, \mathbf{U}$ must be square and $\mathbf{I}_n = \mathbf{S} \mathbf{U}'$ is a rank factorization. $\mathbf{I}_n$ apparently is idempotent thus we have $$ \mathbf{U}' \mathbf{S} = \mathbf{I}_n = \begin{pmatrix} \mathbf{I}_{r_1} & \cdots & \mathbf{O} \\ \vdots & \ddots & \vdots \\ \mathbf{O} & \cdots & \mathbf{I}_{r_k} \end{pmatrix}. $$ Therefore $\mathbf{U}_i' \mathbf{S}_j = \mathbf{O}$ whenver $i \ne j$, implying $$ \mathbf{P}_i \mathbf{P}_j = \mathbf{S}_i \mathbf{U}_i' \mathbf{S}_j \mathbf{U}_j' = \mathbf{O} $$ whenever $i \ne j$.
  Proof of $2 \Rightarrow 3$: Since $\sum_{i=1}^k \mathbf{P}_i = \mathbf{I}_n$, from 2, $$ \mathbf{P}_j = \mathbf{P}_j \left( \sum_{i=1}^k \mathbf{P}_i \right) = \mathbf{P}_j^2 + \sum_{i \ne j} \mathbf{P}_i \mathbf{P}_j = \mathbf{P}_j^2. $$
  Proof of $3 \Rightarrow 1$: $\mathbf{P}_i$ are idemponent, so $\text{rank}(\mathbf{P}_i) = \text{tr}(\mathbf{P}_i)$ for all $i$. $\sum_{i=1}^k \text{rank}(\mathbf{P}_i) = \sum_{i=1}^k \text{tr}(\mathbf{P}_i) = \text{tr}(\sum_i \mathbf{P}_i) = \text{tr}(\mathbf{I}_n) = n$.