Modern datasets can contain hundreds or even thousands of variables. While more data often sounds better, high dimensionality can slow down machine learning models, complicate visualization, and even reduce performance.
This is where Principal Component Analysis (PCA) becomes incredibly powerful.
In this article, we’ll break down:
- What PCA is
- Why dimensionality matters
- How PCA works at a high level
- What PC1 and PC2 actually represent
- Real-world use cases in machine learning
What Is Principal Component Analysis?
Principal Component Analysis (PCA) is a dimensionality reduction technique that transforms large datasets into a smaller set of variables called principal components, while retaining most of the original information.
Instead of working with dozens or hundreds of features, PCA helps us compress that information into fewer dimensions — without losing the structure that matters most.
Why Dimensionality Matters: A Loan Risk Example
Imagine a loan risk management scenario.
Each loan applicant might have dozens of attributes:
- Loan amount
- Credit score
- Annual income
- Debt-to-income ratio
- Employment history
- Age
- And potentially hundreds more
With so many dimensions, it becomes difficult to:
- Visualize the data
- Identify clusters or similarities
- Train models efficiently
Some features are clearly more important than others. For example, credit score is likely more influential than years at a current job when predicting risk.
PCA helps identify which combinations of variables capture the most meaningful variation in the data.
The Visualization Problem
Let’s say we only measure loan amount. We can plot that on a number line. Easy.
Add credit score? Now we can build a 2D scatter plot.
Add annual income? Now we’re in 3D.
Add a fourth variable? Things get messy. Add 10? 100? Impossible to visualize directly.
PCA solves this problem by projecting high-dimensional data into just two or three principal components, making visualization manageable again.
Instead of plotting dozens of variables, we plot:
- PC1 (First Principal Component) on the x-axis
- PC2 (Second Principal Component) on the y-axis
Suddenly, clusters and correlations become visible again.
A Brief History of PCA
PCA isn’t new.
It was first introduced in 1901 by Karl Pearson, long before machine learning existed. However, it gained real popularity once computers became powerful enough to perform large-scale matrix computations.
Today, PCA is widely used in:
- Data preprocessing
- Feature engineering
- Machine learning pipelines
- Statistical analysis
The Curse of Dimensionality
As dimensionality increases, models often suffer from reduced performance. This phenomenon is known as the curse of dimensionality.
More features can:
- Increase computational cost
- Make models harder to generalize
- Increase risk of overfitting
Overfitting Explained
Overfitting occurs when a model performs well on training data but poorly on new, unseen data.
By reducing dimensionality, PCA:
- Simplifies the feature space
- Removes redundant information
- Helps models generalize better
How PCA Works (Without the Heavy Math)
Behind the scenes, PCA involves linear algebra and matrix operations. But at a conceptual level, here’s what it does:
- It finds directions in the data that capture the most variance.
- These directions become the principal components.
- Each principal component is a linear combination of the original variables.
- The components are uncorrelated with each other.
PC1: First Principal Component
- Captures the maximum variance in the dataset.
- Represents the direction where the data varies the most.
- No other component explains more variance than PC1.
PC2: Second Principal Component
- Captures the second-highest variance.
- Must be uncorrelated with PC1 (correlation = 0).
Together, PC1 and PC2 often capture the majority of the useful information in the dataset.
Where Is PCA Used?
1. Image Compression
PCA reduces image dimensionality while preserving key information. This allows:
- Smaller storage sizes
- Faster transmission
- Efficient image representation
2. Data Visualization
High-dimensional datasets can be projected into:
- 2D or 3D scatter plots
This makes it easier to identify clusters, patterns, or anomalies.
3. Noise Filtering
PCA removes noise by focusing only on components that capture meaningful variance, ignoring minor variations that may represent randomness.
4. Healthcare & Medical Diagnostics
PCA has been used to assist in disease diagnosis. For example:
- Reducing multiple tumor attributes in a breast cancer dataset
- Applying logistic regression afterward for classification
In these cases, PCA improves efficiency before a supervised model makes predictions.
PCA and Machine Learning Pipelines
PCA is often used as a preprocessing step before training supervised learning models.
For example:
- Apply PCA to reduce dimensionality.
- Train a classifier (e.g., logistic regression).
- Evaluate performance.
This pipeline:
- Speeds up training
- Reduces overfitting
- Improves model stability
When Should You Use PCA?
Consider PCA if:
- You have a high-dimensional dataset
- Features are correlated
- Training is slow
- Visualization is difficult
- Overfitting is a concern
If your goal is to extract the most informative structure from large datasets, PCA is often an excellent first step.
Not bad for a technique first introduced in 1901.
If you’re working with large, complex datasets, PCA may be exactly what you need to simplify your feature space while preserving what truly matters.
Watch the full video below for a complete walkthrough:
Photo by Steve Johnson on Unsplash