Multi-Rater Agreement: Fleiss' Kappa and Beyond

Three radiologists review the same chest X-ray. One flags a nodule as "suspicious," another marks it "benign," the third calls it "indeterminate." Multiply this by 200 scans and you have a reliability crisis: your diagnostic protocol isn't consistent enough to trust. Before you blame the radiologists, check the measurement system. Fleiss' kappa tells you whether disagreements reflect true ambiguity in the data or failures in your rating design.

Here's what makes multi-rater agreement different from simple pairwise comparison: you're not just asking if Rater A agrees with Rater B. You're asking if all raters converge on the same categorization more often than random chance predicts. That's a joint probability question, and it requires a different statistical framework.

Before You Calculate: Design the Study Correctly

Most inter-rater reliability studies fail before the first rating gets recorded. The problem isn't the statistics—it's the experimental design. Here's what you need before you even think about Fleiss' kappa:

Fixed rating categories defined in advance. Raters must choose from the same categorical labels for every item. "Positive/Negative/Unclear" works. "Rate this on whatever scale feels appropriate" doesn't. If your categories emerge during rating, you're not measuring agreement—you're measuring whether raters invented the same taxonomy.

All raters evaluate all items. Fleiss' kappa assumes complete data. If Rater 1 sees items 1-50 while Rater 2 sees items 51-100, you have a different study design (you need intraclass correlation, not kappa). Partial overlap is technically solvable with extensions, but it complicates interpretation.

Independence between raters. Raters can't discuss cases before rating. They can't see each other's ratings. They can't rate sequentially with knowledge of previous decisions. If Rater 3 knows that Raters 1 and 2 both chose "benign," their choice of "benign" doesn't represent independent judgment—it represents conformity bias.

Sufficient sample size. Rating 10 items with 5 raters gives you 50 data points, but that's underpowered for detecting moderate agreement. You need enough items to capture the natural variation in your domain. Clinical studies typically use 50-100 cases minimum. Content labeling studies often require 200+ items.

What Fleiss' Kappa Actually Measures (And What It Doesn't)

Fleiss' kappa answers one specific question: How much more do raters agree than random chance predicts, given the marginal distribution of ratings?

The formula handles this in two steps. First, calculate the proportion of agreement observed (P_o). For each item, count how many rater pairs agreed, then average across all items. Second, calculate the proportion of agreement expected by chance (P_e) based on how often each category was used overall. Fleiss' kappa is:

κ = (Po - Pe) / (1 - Pe)

This adjusts for chance agreement differently than Cohen's kappa. Cohen's uses the marginal distributions for each rater pair. Fleiss' uses the overall marginal distribution across all raters. The math is more complex but the intuition is straightforward: if everyone uses "positive" 60% of the time, random guessing produces a lot of spurious agreement. Kappa corrects for that.

Here's what kappa doesn't tell you:

Whether your raters are accurate. High agreement means raters are consistent. It doesn't mean they're correct. Five raters could all consistently misclassify the same cases and produce κ = 0.85. Agreement is necessary for accuracy but not sufficient.

Which specific disagreements matter. Kappa is a single summary number. It doesn't distinguish between "benign vs. suspicious" disagreements (potentially serious) and "suspicious vs. malignant" disagreements (both require follow-up). You need confusion matrices and category-specific kappa for that.

Whether individual raters are problematic. One rater using idiosyncratic criteria can drag down overall kappa, but the statistic won't flag them. You need post-hoc analysis comparing each rater to the consensus.

Worked Example: Content Moderation Reliability Study

A social media platform wants to validate their content moderation guidelines. They select 60 flagged posts and have 5 trained moderators categorize each as "Remove," "Warning," or "Allow." Here's a subset of the data:

Item    Rater1    Rater2    Rater3    Rater4    Rater5
1       Remove    Remove    Remove    Warning   Remove
2       Allow     Allow     Allow     Allow     Allow
3       Warning   Remove    Warning   Warning   Warning
4       Remove    Remove    Remove    Remove    Remove
5       Allow     Warning   Allow     Allow     Allow
...
60      Warning   Warning   Warning   Warning   Remove

Before running kappa, check the raw agreement rate. For item 1, we have 4 "Remove" and 1 "Warning." The proportion of agreeing pairs is (4×3)/(5×4/2) = 12/10 = 0.6. Average this across all 60 items: raw agreement is 67%.

Now calculate chance agreement. Across all 300 ratings (60 items × 5 raters):

• "Remove" used 120 times (40%)
• "Warning" used 90 times (30%)
• "Allow" used 90 times (30%)

Expected agreement by chance: P_e = 0.40² + 0.30² + 0.30² = 0.34 (34%).

Fleiss' kappa: κ = (0.67 - 0.34) / (1 - 0.34) = 0.50.

By Landis & Koch guidelines, 0.50 is "moderate" agreement. But is that acceptable? That depends on the cost of errors. If false removals harm free speech and false allows harm users, you probably need higher reliability (κ > 0.70). If borderline cases genuinely require subjective judgment, 0.50 might reflect inherent ambiguity rather than poor guidelines.

Interpretation Guidelines That Actually Help

The Landis & Koch thresholds (0-0.20 slight, 0.21-0.40 fair, 0.41-0.60 moderate, 0.61-0.80 substantial, 0.81-1.00 almost perfect) are widely cited and nearly useless. They're arbitrary cutpoints with no theoretical justification. Different domains require different standards.

Here's a more practical framework:

For medical diagnosis: κ < 0.60 is typically unacceptable. You're making treatment decisions based on these classifications. "Moderate" agreement isn't good enough when the stakes are patient outcomes. Aim for κ > 0.75. If you can't achieve that, the diagnostic criteria need revision or the condition may not be reliably distinguishable.

For content labeling and annotation: κ = 0.40-0.60 is common for subjective tasks like sentiment analysis or intent classification. This doesn't mean the task is worthless—it means human judgment has inherent variance. Document the reliability level and use it to inform downstream confidence intervals on any models trained on this data.

For legal or compliance coding: κ > 0.80 is the standard. When classifications have regulatory or legal consequences, you need near-perfect agreement. Lower values suggest either ambiguous regulations or inadequate coder training.

For research coding schemes: κ = 0.60-0.80 is typical for well-designed qualitative coding. If you're below 0.60, revisit your codebook. If you're above 0.80, great—but check that your categories aren't so coarse that they're trivial to apply.

Always report confidence intervals, not just point estimates. A κ of 0.55 with 95% CI [0.35, 0.75] tells a different story than κ = 0.55 [0.50, 0.60]. The first suggests you're underpowered and need more data. The second suggests you've precisely measured moderate agreement.

When Fleiss' Kappa Fails (And What to Use Instead)

Fleiss' kappa makes assumptions that don't always hold. Here's when it breaks and what alternatives exist:

Problem 1: Imbalanced Categories

When one category dominates (e.g., 90% of items are "negative"), kappa can be paradoxically low even with high raw agreement. This is the "prevalence problem." If raters agree on 95% of cases but κ = 0.45, it's because P_e is also very high (random guessing produces agreement 82% of the time when the base rate is 90%).

Solution: Use Gwet's AC (Agreement Coefficient) instead. AC assumes a different chance model that's less sensitive to prevalence. It typically gives higher values than kappa for imbalanced data, but those higher values better reflect the actual reliability. Alternatively, report both kappa and raw agreement rates so readers can judge for themselves.

Problem 2: Ordered Categories

If your categories have a natural order (e.g., "mild," "moderate," "severe"), Fleiss' kappa treats "mild vs. moderate" disagreements the same as "mild vs. severe." That's clearly wrong—one disagreement is adjacent, the other is extreme.

Solution: Use weighted kappa with quadratic or linear weights. A disagreement one category apart gets weight w, two categories apart gets weight 4w (quadratic) or 2w (linear). This gives partial credit for near-miss agreement. MCP Analytics' CSV analysis tools calculate both weighted and unweighted kappa automatically.

Problem 3: Missing Ratings

Real studies have missing data. Raters skip items they feel unqualified to rate, or items are too ambiguous to categorize. Fleiss' kappa assumes complete data. If you drop all items with any missing ratings, you're analyzing a non-random subsample (probably the easier items).

Solution: Use Krippendorff's alpha, which handles missing data natively. It treats missing ratings as uninformative and adjusts the denominator accordingly. Report what percentage of ratings were missing and whether missingness patterns differed across raters.

Problem 4: More Than Two Raters, But Not All Rate All Items

Sometimes you have 10 potential raters but only 3 rate each item (and the specific 3 vary by item). This is common in crowdsourced labeling. Fleiss' kappa requires the same raters for all items.

Solution: Use intraclass correlation coefficient (ICC) instead. ICC models raters as a random sample from a population and doesn't require the same raters for each item. Specifically, use ICC(2,1) for absolute agreement or ICC(3,1) for consistency. These treat items as fixed and raters as random effects.

The Mistakes That Invalidate Your Study

After reviewing hundreds of inter-rater reliability studies, here are the errors that appear most often:

Mistake 1: Training Raters on the Test Set

Raters practice on 10 cases, discuss disagreements, refine their understanding, then rate 50 cases for the formal reliability study. The problem: if the 10 practice cases are included in the 50 test cases, you've inflated agreement. Raters explicitly discussed those cases.

The fix: Use separate training and test sets. Raters can practice and discuss the training set all they want. The test set must be rated independently without any discussion, before or after.

Mistake 2: Cherry-Picking Items

You select 50 "representative" cases for raters to evaluate. But you unconsciously pick clearer examples because you want raters to finish quickly. Your measured kappa is higher than real-world reliability because you tested on easier cases.

The fix: Random sampling from your full dataset. If some cases are too rare to appear in a random sample, use stratified random sampling to ensure all important subtypes are represented. Document your sampling strategy.

Mistake 3: Conflating Agreement with Accuracy

High kappa means raters agree with each other. It doesn't mean they're right. If your gold standard is expert judgment and you're testing trainee raters, you want agreement with the expert, not agreement among trainees. Use Cohen's kappa comparing each trainee to the expert, not Fleiss' kappa across trainees.

The fix: Clarify your research question. Are you measuring inter-rater reliability (do independent raters converge?) or validity (do raters match the ground truth?). These require different study designs.

Mistake 4: Ignoring Rater Fatigue

Raters evaluate 200 items in one sitting. Agreement is high for items 1-50, then deteriorates for items 151-200 as raters get tired and start clicking faster. Your overall kappa masks this temporal pattern.

The fix: Randomize item order for each rater (so fatigue affects different items for different raters), or break rating sessions into smaller blocks with breaks. Plot agreement rates by item order to check for fatigue effects.

Mistake 5: Treating Kappa as the Endpoint

You calculate κ = 0.42, classify it as "moderate" per Landis & Koch, and stop. You haven't learned anything actionable. Why is agreement moderate? Which categories drive disagreement? Which raters are outliers? What specific guideline revisions would improve reliability?

The fix: Kappa is a diagnostic, not a conclusion. Use it to identify problems, then dig deeper. Calculate category-specific kappa to find problematic categories. Compare individual raters to the group consensus to find outliers. Review specific cases where disagreement was highest to refine your coding guidelines. Then run a second reliability study to confirm your fixes worked.

Power Analysis: How Many Items Do You Actually Need?

Underpowered reliability studies are worse than no study. If your confidence interval spans [0.15, 0.85], you can't distinguish "poor" from "excellent" agreement. You've wasted everyone's time.

The required sample size depends on four factors:

1. Number of raters (k): More raters = more information per item = smaller required sample
2. Number of categories (c): More categories = more ways to disagree = larger required sample
3. Expected kappa (κ₀): Detecting "substantial" agreement (κ = 0.70) requires fewer items than detecting "fair" agreement (κ = 0.30)
4. Desired precision: Do you want 95% CI width of ±0.10 or ±0.05?

Here's a practical rule of thumb for detecting κ = 0.50 with 95% CI width ± 0.10:

• 3 raters, 3 categories: ~70 items
• 5 raters, 3 categories: ~45 items
• 5 raters, 5 categories: ~60 items
• 10 raters, 3 categories: ~25 items

If you're testing whether agreement differs from chance (H₀: κ = 0), you can get away with smaller samples. But most reliability studies care about the magnitude of agreement, not just whether it's statistically significant. Plan for precision, not just significance.

One warning about power: these calculations assume balanced category distributions. If 90% of items fall in one category, multiply these sample sizes by 2-3× to maintain the same precision.

Diagnosing Disagreement: Where to Look When Kappa Is Low

You ran your study. Fleiss' kappa is 0.35. Now what? Low agreement has four common causes, and you need different fixes for each:

Cause 1: Ambiguous Rating Categories

Your guidelines define "high severity" as "poses immediate risk." But "immediate" is vague. Does it mean within 24 hours? One week? Raters interpret differently.

Diagnosis: Look at the confusion matrix. If disagreements are scattered randomly across all category pairs, the problem is likely category definitions.

Fix: Rewrite guidelines with concrete decision rules. Replace "immediate risk" with "requires intervention within 24 hours" and provide 3-5 examples of borderline cases with their correct classifications.

Cause 2: Inadequate Rater Training

Your guidelines are clear, but raters didn't internalize them. They're making decisions based on intuition rather than protocol.

Diagnosis: Calculate kappa for the first 20 items vs. the last 20 items. If agreement improves over time, raters are learning on the job.

Fix: Extend training. Include more practice cases with feedback. Don't start the formal study until raters achieve κ > 0.70 on the training set.

Cause 3: Inherent Ambiguity in the Data

Some items are genuinely borderline. An X-ray shows "possible early-stage nodule" that three experts legitimately disagree about. This isn't a methodology failure—it's reality.

Diagnosis: Calculate item-level agreement. If 80% of items have high agreement (κ > 0.70) and 20% have low agreement (κ < 0.30), you have a subset of hard cases.

Fix: Report two kappa values: one for the full dataset, one excluding the bottom 10% most ambiguous items. Document that real-world application will encounter these ambiguous cases and may require adjudication or consensus process.

Cause 4: Problematic Individual Raters

One rater systematically interprets criteria differently than the group. They're not careless—they have a different mental model of the task.

Diagnosis: Calculate kappa for each rater compared to the group consensus (the modal rating excluding that rater). If one rater has κ < 0.30 while others are κ > 0.60, you've found your outlier.

Fix: Meet with that rater one-on-one. Review specific disagreements to understand their reasoning. Either their interpretation is wrong (retraining needed) or they've identified an ambiguity in your guidelines (revision needed).

Beyond Binary Agreement: Latent Class Models

Fleiss' kappa assumes there's a true category for each item and raters are trying to identify it (with error). But what if there is no gold standard? What if you're using raters to discover the latent structure?

Example: You have 1000 customer support tickets and want to categorize them by topic. No predefined taxonomy exists. Five raters each propose a category. Fleiss' kappa measures consistency, but it doesn't tell you which categories are "real" vs. artifacts of how you framed the task.

Latent class analysis offers an alternative approach. Instead of measuring agreement, you model the probability that each item belongs to each latent class, and the probability that each rater correctly identifies each class. This gives you:

• Class prevalence: How common is each true category?
• Rater sensitivity/specificity: How accurate is each rater for each category?
• Posterior classifications: What's the most likely true category for each item?

This approach is more complex than kappa, but it's more appropriate when you're using raters to discover structure rather than validate an existing taxonomy. Consider it when your research question is "what categories exist in this data?" rather than "do raters agree on these predefined categories?"

Practical Checklist: Running a Defensible Multi-Rater Study

Use this checklist to ensure your reliability study produces interpretable, trustworthy results:

Design Phase

• ☐ Rating categories are mutually exclusive and exhaustive
• ☐ Category definitions include specific criteria and boundary examples
• ☐ Sample size calculated based on desired precision (not just convenience)
• ☐ Items randomly sampled from target population (or stratified if rare categories exist)
• ☐ Separate training and test sets defined
• ☐ Training includes practice items with feedback and discussion
• ☐ Test ratings will be independent (no discussion allowed)

Data Collection Phase

• ☐ Raters complete training until achieving acceptable agreement (κ > 0.70) on training set
• ☐ Test items presented in randomized order for each rater
• ☐ Rating sessions broken into manageable blocks to prevent fatigue
• ☐ Raters cannot see others' ratings or discuss items until study completes
• ☐ Missing ratings documented with reason (if applicable)

Analysis Phase

• ☐ Report overall Fleiss' kappa with 95% confidence interval
• ☐ Report category-specific kappa for each rating category
• ☐ Report raw agreement rates alongside kappa
• ☐ If categories imbalanced (any category <10% or >90%), report Gwet's AC
• ☐ If categories ordered, report weighted kappa
• ☐ Generate confusion matrix showing disagreement patterns
• ☐ Calculate individual rater agreement vs. consensus to identify outliers
• ☐ Check for temporal trends (does agreement change over time?)

Interpretation Phase

• ☐ Interpret kappa in context of domain standards (not just Landis & Koch)
• ☐ If kappa < acceptable threshold, diagnose cause (ambiguous categories? inadequate training? inherent ambiguity? outlier raters?)
• ☐ Document specific guideline revisions or training improvements needed
• ☐ Plan follow-up study to confirm improvements (if applicable)

If you can check all these boxes, your reliability study will produce defensible estimates that stakeholders can trust. Anything less and you're just generating numbers without understanding.

FAQ: Multi-Rater Agreement

What's the minimum number of raters needed for Fleiss' kappa?

Technically three, but that's underpowered for most applications. With only three raters, you can't detect patterns in disagreement or identify problematic individual raters. For clinical applications, aim for 5-7 raters minimum. For content labeling studies, 10+ raters gives you the statistical power to detect moderate agreement (κ ≈ 0.4) with confidence.

When should I use Fleiss' kappa instead of Cohen's kappa?

Use Fleiss' kappa when you have three or more raters evaluating the same items. Cohen's kappa is designed for exactly two raters. If you run pairwise Cohen's kappa on all rater pairs in a multi-rater study, you're ignoring the joint distribution of ratings and you can't make claims about overall reliability.

What does a Fleiss' kappa of 0.45 actually mean?

A κ of 0.45 means raters agree 45% more than you'd expect from random chance alone. By Landis & Koch guidelines, this is "moderate" agreement. But context matters: in radiology, 0.45 might be concerning. In subjective content moderation, it might be acceptable. Always interpret kappa alongside raw agreement rates and the cost of disagreement in your specific domain.

Can Fleiss' kappa be negative?

Yes, and it's a warning sign. Negative kappa means raters agree less than random chance would predict. This suggests systematic problems: ambiguous rating categories, inadequate rater training, or raters actively interpreting instructions differently. Before you interpret any results, go back and check your study design.

How do I handle missing ratings in Fleiss' kappa calculations?

You have two options: exclude items with incomplete ratings (reduces sample size), or use extensions like Gwet's AC that handle missingness. Do NOT impute missing ratings or assume they're random. Missing ratings are often non-random—raters skip difficult cases. Document your approach and report what percentage of ratings were missing.

Final Recommendation: Design First, Calculate Second

The difference between a useful reliability study and a waste of time is experimental design. Fleiss' kappa is just arithmetic—it will calculate a number regardless of whether your study was properly designed. But that number is only meaningful if:

• Your rating categories were clearly defined before data collection
• Your raters were adequately trained and tested independently
• Your sample was large enough to achieve acceptable precision
• Your items were randomly sampled from the target population

Get these design elements right and kappa becomes a powerful diagnostic that tells you whether your measurement system is reliable enough to trust. Get them wrong and you have a precise estimate of a meaningless quantity.

Before you collect a single rating, ask: What decision will this reliability study inform? What threshold of agreement would make me trust the ratings enough to act on them? What will I do differently if kappa is 0.40 vs. 0.70? If you can't answer these questions, you're not ready to run the study.

And remember: agreement is necessary for accuracy but not sufficient. High kappa means your raters are consistent. It doesn't mean they're right. If you care about accuracy, you need a gold standard and a validity study, not just a reliability study. Know which question you're answering.

Related Articles

Cohen's Kappa for Two-Rater Agreement

When you have exactly two raters, Cohen's kappa provides more statistical power than Fleiss'. Learn when to use it and how to interpret weighted kappa for ordered categories.

Intraclass Correlation for Continuous Ratings

Fleiss' kappa works for categorical data. For continuous measurements (multiple raters scoring items on a scale), you need ICC. This guide covers ICC(2,1) vs ICC(3,1) and when to use each.

Krippendorff's Alpha: The Universal Agreement Metric

Alpha handles any number of raters, any type of data (nominal, ordinal, interval, ratio), and missing values. It's more flexible than kappa but computationally intensive.

Gwet's AC: Fixing the Prevalence Problem

When category distributions are imbalanced, Fleiss' kappa can be paradoxically low despite high raw agreement. Gwet's AC solves this with a different chance-agreement model.