What Is Linear Discriminant Analysis (LDA) and How Does It Work?

Linear Discriminant Analysis (LDA), originally introduced by Ronald A. Fisher in 1936, is a classical statistical method used for classification and dimensionality reduction. It finds a linear combination of features that best separates two or more classes of objects or events. Unlike Principal Component Analysis (PCA), which focuses on variance, LDA explicitly maximizes the separation between known groups.

What It Is

LDA is a supervised technique. Given a dataset with labeled classes, LDA projects the data onto a lower-dimensional space such that the classes are as distinct as possible. It is widely used in pattern recognition, machine learning preprocessing, and quality control for distinguishing between product categories or process states.

The core idea is to find a discriminant function—a linear equation of the input variables—that yields high scores for one class and low scores for another. For multiple classes, LDA produces several discriminant functions (at most the number of classes minus one).

How It Works: Formula and Steps

LDA seeks to maximize the ratio of between-class variance to within-class variance. For a two-class problem, the discriminant function is:

\[
y = \mathbf{w}^T \mathbf{x}
\]

where \(\mathbf{x}\) is the vector of predictor variables, and \(\mathbf{w}\) is the weight vector that maximizes:

\[
J(\mathbf{w}) = \frac{(\mu_1 - \mu_2)^2}{s_1^2 + s_2^2}
\]

Here, \(\mu_1\) and \(\mu_2\) are the class means of the projected data, and \(s_1^2\), \(s_2^2\) are the within-class variances. The solution for \(\mathbf{w}\) is proportional to:

\[
\mathbf{w} \propto \mathbf{S}_W^{-1}(\mathbf{m}_1 - \mathbf{m}_2)
\]

where \(\mathbf{S}_W\) is the pooled within-class scatter matrix, and \(\mathbf{m}_1\), \(\mathbf{m}_2\) are the original class mean vectors.

Steps to apply LDA:

1. Compute the mean vector for each class.
2. Compute the within-class scatter matrix \(\mathbf{S}_W\).
3. Compute the between-class scatter matrix \(\mathbf{S}_B\).
4. Solve the generalized eigenvalue problem \(\mathbf{S}_W^{-1}\mathbf{S}_B \mathbf{v} = \lambda \mathbf{v}\).
5. Select the eigenvectors with the largest eigenvalues to form the discriminant axes.
6. Project new observations onto these axes and classify them by nearest class mean (or using a threshold for two classes).

A Worked Illustrative Example

Example data (illustrative only): Suppose a quality engineer measures two features (hardness and thickness) on 10 units from two production lines, A and B. The within-class scatter matrix is computed as:

\[
\mathbf{S}_W = \begin{bmatrix} 2.0 & 0.5 \\ 0.5 & 1.5 \end{bmatrix}
\]

The difference between class mean vectors is:

\[
\mathbf{m}_A - \mathbf{m}_B = \begin{bmatrix} 1.0 \\ -0.8 \end{bmatrix}
\]

Then the discriminant weight vector is:

\[
\mathbf{w} = \mathbf{S}_W^{-1}(\mathbf{m}_A - \mathbf{m}_B)
\]

First, compute the inverse of \(\mathbf{S}_W\):

\[
\mathbf{S}_W^{-1} = \frac{1}{(2.0)(1.5) - (0.5)^2} \begin{bmatrix} 1.5 & -0.5 \\ -0.5 & 2.0 \end{bmatrix}
= \frac{1}{2.75} \begin{bmatrix} 1.5 & -0.5 \\ -0.5 & 2.0 \end{bmatrix}
\]

Multiplying gives:

\[
\mathbf{w} = \frac{1}{2.75} \begin{bmatrix} 1.5(1.0) + (-0.5)(-0.8) \\ -0.5(1.0) + 2.0(-0.8) \end{bmatrix}
= \frac{1}{2.75} \begin{bmatrix} 1.9 \\ -2.1 \end{bmatrix}
= \begin{bmatrix} 0.691 \\ -0.764 \end{bmatrix}
\]

A new unit with hardness = 5.2 and thickness = 3.1 yields a discriminant score:

\[
y = 0.691(5.2) + (-0.764)(3.1) = 3.593 - 2.368 = 1.225
\]

If the midpoint between the class centroids is 0.9, this unit is assigned to Line A. (All numbers above are fabricated for illustration.)

Common Pitfalls

• Assumes equal covariance matrices: LDA works best when classes share a similar covariance structure. If not, Quadratic Discriminant Analysis (QDA) may be more appropriate.
• Sensitive to outliers: Extreme values can distort the scatter matrices and the resulting discriminant direction.
• Small sample size: When the number of predictors exceeds the sample size, \(\mathbf{S}_W\) becomes singular and cannot be inverted. Regularization or PCA pre‑processing is needed.
• Misinterpreting coefficients: The magnitude of \(\mathbf{w}\) depends on the scale of the variables; standardize predictors if you want to compare variable importance.

---

Ready to apply LDA to your own classification problem? Use our free, interactive tool at https://www.6sq.com/tools/discriminant/ to compute discriminant functions and visualize class separation in seconds.