Principal Component Analysis

Concept

Observations can be treated as vectors. PCA changes their coordinate system so the most important variation comes first.

Equation

$x_i\text{'}=x_i-\mu$, then $Cv=\lambda v$. The covariance matrix’s eigenvectors are the principal directions.

Observe

The ellipse is a geometric picture of the directions and relative amount of variation; it is not a confidence interval. PC1 has the largest eigenvalue, so it captures the most variation. PC2 is perpendicular and captures what remains.

From 2D to 1D

For a centered observation $x$ and unit PC1 direction $u_1$, its PCA coordinate is $z=x·u_1$. Keeping only $z$ reduces two dimensions to one. A compact reconstruction is $\hat{x}=\mu +z{u}_1$.

Computing, data & machine learning

Covariance depends on feature scale: a large-unit feature can dominate PCA. In many applications, features are standardized before PCA. PCA is useful for compression, visualization, and preparing data for later models.

Connection to eigenvectors

Previously, $Av=\lambda v$ identified special directions of a matrix. Here the matrix is covariance: its eigenvectors are principal directions and its eigenvalues measure variation along them.

Previous: Eigenvectors & Eigenvalues