Measuring Inter-Annotator Agreement with Cohen’s Kappa

Cohen’s kappa scores how much two annotators agree on class labels beyond what chance would produce given their own labelling habits. For matched features it takes their two class vectors and returns a number from below zero (worse than chance) to one (perfect). It exists because raw percentage agreement is inflated by class imbalance: on a batch where 92% of features are buildings, two annotators who call everything a building agree 92% of the time and demonstrate no skill at all. Kappa returns 0.0 for that pair, which is the honest answer. This guide computes it over geospatial features, reads the confusion matrix that says why it is low, and covers the two situations where kappa itself misleads.

# Why the Chance Correction Matters Here

Geospatial label distributions are extreme. A building-footprint project is mostly buildings; a land-cover project over farmland is mostly one crop class; an infrastructure project may have one class at 95% and four at one percent each. Under those distributions, raw agreement carries almost no information — it is a restatement of the class prior.

Kappa’s correction is simple. Expected agreement is what you would get if each annotator independently drew labels from their own observed distribution. Observed agreement is what actually happened. Kappa is the fraction of the available improvement that was achieved:

kappa = (p_observed − p_expected) / (1 − p_expected)

When p_expected is 0.85 because one class dominates, an observed 0.92 yields kappa 0.47 — mediocre. When p_expected is 0.30 on a balanced taxonomy, an observed 0.92 yields kappa 0.89 — excellent. Same raw number, opposite verdicts, and the second is the one that reflects skill.

# Step-by-Step Implementation

# Step 1 — Install and Build the Paired Vectors

pip install scikit-learn==1.5.1 geopandas==0.14.4 pandas==2.2.2 numpy==1.26.4

Kappa needs two aligned label vectors, one per annotator, over the same matched features. The matching itself — deciding which of A’s polygons corresponds to which of B’s — is the step described in annotation quality metrics and agreement; this guide starts from its output.

import pandas as pd
import geopandas as gpd

def paired_labels(a: gpd.GeoDataFrame, b: gpd.GeoDataFrame, pairs: pd.DataFrame,
                  class_field: str = "class_name") -> tuple[list[str], list[str]]:
    """Two aligned class vectors over the matched pairs, in a stable order."""
    pairs = pairs.sort_values("ia").reset_index(drop=True)
    ya = [a.loc[int(r.ia), class_field] for r in pairs.itertuples()]
    yb = [b.iloc[int(r.ib)][class_field] for r in pairs.itertuples()]
    return ya, yb

Sorting the pairs is not cosmetic. Kappa itself is order-independent, but the confusion matrix you will read next is easier to diff between batches when the row order is stable.

# Step 2 — Fix the Label Set From the Taxonomy

Passing the labels explicitly, from the taxonomy rather than from the batch, keeps matrices comparable when a rare class happens to be absent.

import json
from sklearn.metrics import cohen_kappa_score, confusion_matrix

def load_taxonomy_labels(path: str) -> list[str]:
    with open(path, encoding="utf-8") as fh:
        return sorted(json.load(fh)["classes"])

def agreement(ya: list[str], yb: list[str], labels: list[str]) -> dict:
    """Kappa, raw agreement and the confusion matrix over a fixed label set."""
    n = len(ya)
    raw = sum(x == y for x, y in zip(ya, yb)) / n if n else float("nan")
    return {
        "n_pairs": n,
        "kappa": float(cohen_kappa_score(ya, yb, labels=labels)) if n else float("nan"),
        "raw_agreement": float(raw),
        "labels": labels,
        "matrix": confusion_matrix(ya, yb, labels=labels).tolist(),
    }

Reporting n_pairs alongside the score is what stops a kappa computed on eleven features being read with the same confidence as one computed on nine hundred.

# Step 3 — Read the Confusion Matrix, Not Just the Number

A low kappa is a symptom. The matrix says which disagreement produced it, and the shape of the answer determines who fixes it.

def dominant_confusion(result: dict, top_k: int = 3) -> list[tuple[str, str, int]]:
    """The largest off-diagonal cells: which class pairs the disagreement lives in."""
    labels, m = result["labels"], result["matrix"]
    cells = [(labels[i], labels[j], m[i][j])
             for i in range(len(labels)) for j in range(len(labels)) if i != j and m[i][j]]
    return sorted(cells, key=lambda c: c[2], reverse=True)[:top_k]

If one pair holds most of the mass — orchard against cropland, say — the taxonomy has not distinguished them on this imagery, and no amount of annotator training will help. If the off-diagonal mass is spread evenly, the disagreement is about care rather than definitions, and it is a training conversation.

# Step 4 — Go Pairwise Across Annotators

With three or more people, the mean of pairwise kappas answers a question Fleiss’ kappa cannot: is one person the outlier?

from itertools import combinations

def pairwise_matrix(labels_by_annotator: dict[str, list[str]], labels: list[str]) -> pd.DataFrame:
    """Kappa for every pair of annotators over their common features."""
    names = sorted(labels_by_annotator)
    out = pd.DataFrame(index=names, columns=names, dtype=float)
    for x, y in combinations(names, 2):
        k = cohen_kappa_score(labels_by_annotator[x], labels_by_annotator[y], labels=labels)
        out.loc[x, y] = out.loc[y, x] = round(float(k), 3)
    return out

An annotator whose row is uniformly low disagrees with everyone, which is one conversation. A matrix where every cell is low is a taxonomy problem affecting the whole team, which is a different one.

# Parameters and Thresholds Reference

# Common Errors and Fixes

ValueError: Number of classes ... does not match Root cause: labels was inferred from one vector rather than passed explicitly. Fix: always pass the full taxonomy list, as in Step 2.

Kappa is nan Root cause: every feature in the batch has the same class in both vectors, so the expected-agreement denominator is zero. Fix: report raw_agreement and n_pairs instead, and note that kappa is undefined here — it is not zero, and reporting it as zero is a mistake that reads as a broken team.

Kappa is negative Root cause: the two annotators systematically disagree — often because one is using an old taxonomy version where two class names swapped meaning. Fix: check the taxonomy version each annotation was made under before concluding anything about people, using the versioned taxonomy described in defining ROI label taxonomies.

Agreement collapses after adding a class Root cause: a new class carved out of an existing one, so features that were unambiguous now have two plausible labels. Fix: expected. Re-adjudicate a sample under the new taxonomy and treat the pre-change kappa as belonging to a different measurement.

# Frequently Asked Questions

# Related

• Annotation Quality Metrics & Inter-Annotator Agreement — the matching step that produces the pairs, and the geometry half of the measurement
• Computing Boundary IoU for Footprint Quality — the delineation score that answers the question kappa deliberately does not
• Defining ROI Label Taxonomies for Aerial Imagery — where a persistently low per-class kappa sends you

This measurement is one part of the Annotation Quality Metrics & Inter-Annotator Agreement topic within Geospatial Annotation Fundamentals & Architecture.