6SQ ToolsWhat Is Cohen's Kappa and How Do You Use It to Measure Rater Agreement?
When two inspectors, two measurement systems, or two doctors evaluate the same set of items, they will not always agree. The question is: how much of that agreement is real, and how much could have happened by chance alone? Cohen's Kappa is the standard statistical tool that answers this question by measuring inter-rater agreement while correcting for chance agreement.
What It Is
Cohen's Kappa (κ) is a statistic that measures the agreement between two raters who each classify items into mutually exclusive categories. Unlike a simple percentage of agreement, Kappa accounts for the agreement that would occur purely by random chance. It is widely used in quality management for attribute (go/no-go) gauge studies, in medicine for diagnostic agreement, and in social sciences for coding reliability.
The method is named after Jacob Cohen, who introduced it in 1960. For more than two raters, variations such as Fleiss' Kappa exist, but Cohen's Kappa is the foundational tool for pairwise comparison.
How It Works / Formula
The Kappa coefficient is calculated using the following formula:
\[
\kappa = \frac{P_o - P_e}{1 - P_e}
\]
Where:
The value of κ ranges from −1 to +1:
To interpret the strength of agreement, the widely cited thresholds from Landis and Koch (1977) are used:
Kappa Value | Strength of Agreement
These thresholds are a general guideline from the published literature, not a strict industry standard. In manufacturing, the AIAG MSA manual (Measurement Systems Analysis) recommends that for attribute gauge studies, Kappa values should generally be ≥ 0.75 for the measurement system to be considered acceptable for process control.
A Worked Illustrative Example
Example data (illustrative only).
Suppose two inspectors, A and B, evaluate 100 parts and classify each as either "Accept" or "Reject." Their results are summarized in the table below:
Inspector B: Accept | Inspector B: Reject | Total
Step 1: Calculate observed agreement (\(P_o\))
The raters agree on 45 parts (both Accept) and 30 parts (both Reject).
\[
P_o = \frac{45 + 30}{100} = 0.75
\]
Step 2: Calculate chance agreement (\(P_e\))
The probability that both say "Accept" by chance is the product of their marginal proportions:
\[
P_{\text{Accept}} = \frac{55}{100} \times \frac{60}{100} = 0.55 \times 0.60 = 0.33
\]
The probability that both say "Reject" by chance:
\[
P_{\text{Reject}} = \frac{45}{100} \times \frac{40}{100} = 0.45 \times 0.40 = 0.18
\]
Total chance agreement:
\[
P_e = 0.33 + 0.18 = 0.51
\]
Step 3: Calculate Kappa
\[
\kappa = \frac{0.75 - 0.51}{1 - 0.51} = \frac{0.24}{0.49} \approx 0.49
\]
Interpretation: A Kappa of 0.49 falls in the "Moderate" range per Landis and Koch. In a manufacturing context, this is below the AIAG recommended threshold of 0.75, indicating the inspection system needs improvement before it is used for release decisions.
Common Pitfalls
---
To calculate Cohen's Kappa quickly and correctly, use the free tool at: https://www.6sq.com/tools/kappa/ — it handles the math and gives you the Landis & Koch interpretation automatically.
What It Is
Cohen's Kappa (κ) is a statistic that measures the agreement between two raters who each classify items into mutually exclusive categories. Unlike a simple percentage of agreement, Kappa accounts for the agreement that would occur purely by random chance. It is widely used in quality management for attribute (go/no-go) gauge studies, in medicine for diagnostic agreement, and in social sciences for coding reliability.
The method is named after Jacob Cohen, who introduced it in 1960. For more than two raters, variations such as Fleiss' Kappa exist, but Cohen's Kappa is the foundational tool for pairwise comparison.
How It Works / Formula
The Kappa coefficient is calculated using the following formula:
\[
\kappa = \frac{P_o - P_e}{1 - P_e}
\]
Where:
- \(P_o\) = the observed proportion of agreement (the number of items on which both raters agree, divided by the total number of items).
- \(P_e\) = the expected proportion of agreement by chance, calculated from the marginal totals of the rater classification table.
The value of κ ranges from −1 to +1:
- κ = 1: Perfect agreement.
- κ = 0: Agreement no better than chance.
- κ < 0: Agreement worse than chance (rare in practice).
To interpret the strength of agreement, the widely cited thresholds from Landis and Koch (1977) are used:
Kappa Value | Strength of Agreement
- < 0 | Poor
- 0.00 – 0.20 | Slight
- 0.21 – 0.40 | Fair
- 0.41 – 0.60 | Moderate
- 0.61 – 0.80 | Substantial
- 0.81 – 1.00 | Almost Perfect
These thresholds are a general guideline from the published literature, not a strict industry standard. In manufacturing, the AIAG MSA manual (Measurement Systems Analysis) recommends that for attribute gauge studies, Kappa values should generally be ≥ 0.75 for the measurement system to be considered acceptable for process control.
A Worked Illustrative Example
Example data (illustrative only).
Suppose two inspectors, A and B, evaluate 100 parts and classify each as either "Accept" or "Reject." Their results are summarized in the table below:
Inspector B: Accept | Inspector B: Reject | Total
- Inspector A: Accept | 45 | 10 | 55
- Inspector A: Reject | 15 | 30 | 45
- Total | 60 | 40 | 100
Step 1: Calculate observed agreement (\(P_o\))
The raters agree on 45 parts (both Accept) and 30 parts (both Reject).
\[
P_o = \frac{45 + 30}{100} = 0.75
\]
Step 2: Calculate chance agreement (\(P_e\))
The probability that both say "Accept" by chance is the product of their marginal proportions:
\[
P_{\text{Accept}} = \frac{55}{100} \times \frac{60}{100} = 0.55 \times 0.60 = 0.33
\]
The probability that both say "Reject" by chance:
\[
P_{\text{Reject}} = \frac{45}{100} \times \frac{40}{100} = 0.45 \times 0.40 = 0.18
\]
Total chance agreement:
\[
P_e = 0.33 + 0.18 = 0.51
\]
Step 3: Calculate Kappa
\[
\kappa = \frac{0.75 - 0.51}{1 - 0.51} = \frac{0.24}{0.49} \approx 0.49
\]
Interpretation: A Kappa of 0.49 falls in the "Moderate" range per Landis and Koch. In a manufacturing context, this is below the AIAG recommended threshold of 0.75, indicating the inspection system needs improvement before it is used for release decisions.
Common Pitfalls
- Kappa is affected by prevalence. If one category is very rare (e.g., 95% of parts are Accept), Kappa can be low even when the percentage of agreement is high. Always report both the raw agreement and Kappa.
- Kappa is not a percentage. A Kappa of 0.60 does not mean "60% agreement." It means the observed agreement is 60% of the way from chance to perfect agreement.
- Do not use Kappa for ordinal or continuous data. Kappa is designed for nominal (categorical) classifications. If your categories have a natural order (e.g., minor/major/critical), weighted Kappa is more appropriate.
- The Landis and Koch thresholds are arbitrary cutoffs. They are convenient labels, not statistical significance tests. Always consider the business context and the cost of disagreement.
---
To calculate Cohen's Kappa quickly and correctly, use the free tool at: https://www.6sq.com/tools/kappa/ — it handles the math and gives you the Landis & Koch interpretation automatically.
No related results found
Invited:
0 replies