6SQ ToolsWhat Is Multiple Comparison in ANOVA, and Which Method Should You Use?
When you run an ANOVA and get a significant p-value, the work isn't done. ANOVA tells you that at least one group mean differs, but it doesn't tell you which groups differ. That's where multiple comparison procedures come in—and why they matter more than running many t-tests.
What It Is
Multiple comparison (also called post-hoc comparison or pairwise testing) is a set of statistical procedures used after a significant ANOVA result to identify exactly which group means are significantly different from each other.
The core problem: if you compare every pair of groups with ordinary t-tests, your chance of at least one false positive grows with each test. With 5 groups, you have 10 pairwise comparisons. At α = 0.05 per test, the probability of at least one false "significant" result is about 40%—not 5%. Multiple comparison methods control this family-wise error rate (FWER), keeping the overall chance of any false positive at your chosen α level.
How It Works: Three Standard Methods
Three widely used methods appear in most statistical standards and textbooks:
### 1. Tukey HSD (Honestly Significant Difference)
The Tukey method is the most common for pairwise comparisons when group sizes are equal (or nearly equal). It uses the studentized range distribution (q) rather than the t-distribution.
The minimum difference needed for significance is:
\[
\text{HSD} = q_{\alpha, k, df} \times \sqrt{\frac{MS_W}{n}}
\]
Where:
Any pair whose mean difference exceeds HSD is declared significant. Tukey controls FWER exactly when sample sizes are equal.
### 2. Bonferroni Correction
The simplest and most conservative approach. You divide your desired α by the number of comparisons:
\[
\alpha_{\text{adjusted}} = \frac{\alpha}{k}
\]
Where k = number of pairwise comparisons = \( \frac{m(m-1)}{2} \) for m groups.
For 4 groups, k = 6, so at α = 0.05 you test each pair at 0.05/6 ≈ 0.0083. Bonferroni works with any number of comparisons and unequal sample sizes, but it becomes overly conservative with many groups, increasing the risk of false negatives.
### 3. Scheffé Method
Scheffé is the most conservative of the three and is designed for any possible contrast (not just pairwise). It's appropriate when you plan to test complex comparisons you didn't specify in advance.
The critical value for any contrast is:
\[
\sqrt{(m-1) \times F_{\alpha, (m-1), df}}
\]
Where \( F_{\alpha, (m-1), df} \) is the F critical value. Scheffé guarantees FWER control for all possible contrasts, which is why it's stricter for simple pairwise tests.
A Worked Illustrative Example
Example data (illustrative only): A quality engineer tests tensile strength (MPa) of plastic parts from 4 suppliers. ANOVA gives MS_W = 4.2, error df = 20, and the group means are:
Supplier | Mean (MPa) | n
ANOVA is significant (p < 0.05). Now compare pairs.
Tukey HSD: For k = 4 groups and df = 20, the q critical value at α = 0.05 is q = 3.96 (standard published value).
\[
\text{HSD} = 3.96 \times \sqrt{\frac{4.2}{6}} = 3.96 \times 0.837 = 3.31 \text{ MPa}
\]
Any pair differing by more than 3.31 MPa is significant. The differences:
Bonferroni: With 6 comparisons, adjusted α = 0.05/6 = 0.0083. Using the t-table with df = 20, the critical t at 0.0083/2 ≈ 0.004 (two-tailed) is about t = 3.15. The minimum significant difference is:
\[
3.15 \times \sqrt{4.2 \times \left(\frac{1}{6} + \frac{1}{6}\right)} = 3.15 \times 1.183 = 3.73 \text{ MPa}
\]
Only A vs D (4.7) exceeds 3.73. C vs D (3.5) is no longer significant under Bonferroni.
Scheffé: The critical value uses F with (3, 20) df. At α = 0.05, F = 3.10. The Scheffé critical difference for a pairwise contrast is:
\[
\sqrt{3 \times 3.10} \times \sqrt{4.2 \times \left(\frac{1}{6} + \frac{1}{6}\right)} = 3.05 \times 1.183 = 3.61 \text{ MPa}
\]
Result: only A vs D is significant—same conclusion as Bonferroni here, but Scheffé would be even more conservative for more complex contrasts.
Common Pitfalls
Choosing the Right Tool
For routine pairwise comparisons with balanced data, Tukey HSD is the standard choice. For unequal sample sizes or few comparisons, Bonferroni is simple and safe. For unplanned complex contrasts, Scheffé is the rigorous option.
To avoid calculation errors and get instant results, use the free multiple comparison calculator at 6sq.com/tools/multicomp/. It handles Tukey HSD, Bonferroni, and Scheffé automatically—so you can focus on interpreting the engineering meaning, not the arithmetic.
What It Is
Multiple comparison (also called post-hoc comparison or pairwise testing) is a set of statistical procedures used after a significant ANOVA result to identify exactly which group means are significantly different from each other.
The core problem: if you compare every pair of groups with ordinary t-tests, your chance of at least one false positive grows with each test. With 5 groups, you have 10 pairwise comparisons. At α = 0.05 per test, the probability of at least one false "significant" result is about 40%—not 5%. Multiple comparison methods control this family-wise error rate (FWER), keeping the overall chance of any false positive at your chosen α level.
How It Works: Three Standard Methods
Three widely used methods appear in most statistical standards and textbooks:
### 1. Tukey HSD (Honestly Significant Difference)
The Tukey method is the most common for pairwise comparisons when group sizes are equal (or nearly equal). It uses the studentized range distribution (q) rather than the t-distribution.
The minimum difference needed for significance is:
\[
\text{HSD} = q_{\alpha, k, df} \times \sqrt{\frac{MS_W}{n}}
\]
Where:
- \( q_{\alpha, k, df} \) = critical value from the studentized range table (k = number of groups, df = error degrees of freedom)
- \( MS_W \) = mean square within groups (from ANOVA)
- \( n \) = number of observations per group
Any pair whose mean difference exceeds HSD is declared significant. Tukey controls FWER exactly when sample sizes are equal.
### 2. Bonferroni Correction
The simplest and most conservative approach. You divide your desired α by the number of comparisons:
\[
\alpha_{\text{adjusted}} = \frac{\alpha}{k}
\]
Where k = number of pairwise comparisons = \( \frac{m(m-1)}{2} \) for m groups.
For 4 groups, k = 6, so at α = 0.05 you test each pair at 0.05/6 ≈ 0.0083. Bonferroni works with any number of comparisons and unequal sample sizes, but it becomes overly conservative with many groups, increasing the risk of false negatives.
### 3. Scheffé Method
Scheffé is the most conservative of the three and is designed for any possible contrast (not just pairwise). It's appropriate when you plan to test complex comparisons you didn't specify in advance.
The critical value for any contrast is:
\[
\sqrt{(m-1) \times F_{\alpha, (m-1), df}}
\]
Where \( F_{\alpha, (m-1), df} \) is the F critical value. Scheffé guarantees FWER control for all possible contrasts, which is why it's stricter for simple pairwise tests.
A Worked Illustrative Example
Example data (illustrative only): A quality engineer tests tensile strength (MPa) of plastic parts from 4 suppliers. ANOVA gives MS_W = 4.2, error df = 20, and the group means are:
Supplier | Mean (MPa) | n
- A | 52.3 | 6
- B | 49.8 | 6
- C | 51.1 | 6
- D | 47.6 | 6
ANOVA is significant (p < 0.05). Now compare pairs.
Tukey HSD: For k = 4 groups and df = 20, the q critical value at α = 0.05 is q = 3.96 (standard published value).
\[
\text{HSD} = 3.96 \times \sqrt{\frac{4.2}{6}} = 3.96 \times 0.837 = 3.31 \text{ MPa}
\]
Any pair differing by more than 3.31 MPa is significant. The differences:
- A vs D: 52.3 − 47.6 = 4.7 → significant
- A vs B: 2.5 → not significant
- C vs D: 3.5 → significant
- All others: below 3.31 → not significant
Bonferroni: With 6 comparisons, adjusted α = 0.05/6 = 0.0083. Using the t-table with df = 20, the critical t at 0.0083/2 ≈ 0.004 (two-tailed) is about t = 3.15. The minimum significant difference is:
\[
3.15 \times \sqrt{4.2 \times \left(\frac{1}{6} + \frac{1}{6}\right)} = 3.15 \times 1.183 = 3.73 \text{ MPa}
\]
Only A vs D (4.7) exceeds 3.73. C vs D (3.5) is no longer significant under Bonferroni.
Scheffé: The critical value uses F with (3, 20) df. At α = 0.05, F = 3.10. The Scheffé critical difference for a pairwise contrast is:
\[
\sqrt{3 \times 3.10} \times \sqrt{4.2 \times \left(\frac{1}{6} + \frac{1}{6}\right)} = 3.05 \times 1.183 = 3.61 \text{ MPa}
\]
Result: only A vs D is significant—same conclusion as Bonferroni here, but Scheffé would be even more conservative for more complex contrasts.
Common Pitfalls
- Running multiple t-tests instead of a proper method — inflates false positives dramatically.
- Using Tukey with very unequal sample sizes — Tukey assumes balanced design; use Tukey-Kramer or Bonferroni instead.
- Applying post-hoc tests when ANOVA is not significant — the overall test protects against data snooping; post-hoc tests after a non-significant ANOVA are generally not justified.
- Choosing Scheffé for simple pairwise tests — it's unnecessarily conservative; reserve it for complex, unplanned contrasts.
- Forgetting to state which method you used — conclusions depend heavily on the choice, so always report it.
Choosing the Right Tool
For routine pairwise comparisons with balanced data, Tukey HSD is the standard choice. For unequal sample sizes or few comparisons, Bonferroni is simple and safe. For unplanned complex contrasts, Scheffé is the rigorous option.
To avoid calculation errors and get instant results, use the free multiple comparison calculator at 6sq.com/tools/multicomp/. It handles Tukey HSD, Bonferroni, and Scheffé automatically—so you can focus on interpreting the engineering meaning, not the arithmetic.
No related results found
Invited:
0 replies