Prioritizing Annotation Tiles by Model Disagreement

When an active learning loop has to choose which of thousands of unlabeled satellite or drone tiles to send to annotators next, single-model uncertainty is not the only signal available. Query-by-committee ranks tiles by how much a set of models disagree with each other: run N model checkpoints — or N augmentation-perturbed passes of one model — over every unlabeled tile, then score each tile by vote entropy and average KL divergence from the committee consensus. Tiles where the committee splits are the most informative to label, because they sit near a decision boundary that new annotations will sharpen. This guide implements that ranking in NumPy and adds a batch selector that avoids wasting budget on spatially adjacent, near-duplicate tiles.

# Why Committee Disagreement Beats Single-Model Uncertainty

A single model’s softmax entropy tells you where that model is unsure, but it conflates two very different situations: genuine class ambiguity, and a model that is confidently wrong. A committee separates them. If five independently trained checkpoints all assign 60% to “building” on a tile, the model family agrees — the tile is only mildly uncertain and probably not worth a label. If three checkpoints say “building” and two say “greenhouse”, the tile sits exactly on a boundary the committee has not resolved, and one human label there corrects several models at once.

This matters for geospatial data specifically because uncertainty is spatially structured. Terrain, sensor geometry, seasonal illumination, and class frequency all vary smoothly across a scene, so disagreement clusters into contiguous patches rather than scattering randomly. A naive top-K selection would hand annotators twenty tiles from the same confusing field. The batch selector below treats spatial redundancy as a first-class constraint, spreading the label budget across distinct regions so each retraining round sees maximally diverse examples. The committee approach also composes cleanly with the confidence scores you already log per prediction — disagreement is a complementary axis, not a replacement.

# Building the Committee

You can form a committee two ways. The first is to snapshot several checkpoints from a single training run — for example the model at epochs 40, 50, 60, and 70, or several runs with different random seeds. The second, cheaper, option is test-time augmentation: take one deployed model and run it several times over each tile with different augmentations (horizontal flip, 90° rotation, small brightness jitter), treating each perturbed pass as a voter. Both yield an array of per-tile class probabilities of shape (n_members, n_classes), which is all the scoring code needs.

Install the single dependency:

pip install numpy==1.26.4

# Step-by-Step Implementation

# Step 1 — Represent Committee Predictions Per Tile

Store predictions as a dictionary keyed by tile id, each value an array of shape (n_members, n_classes). In production these come from your inference workers; here a small helper documents the expected shape and validates it.

from __future__ import annotations
import numpy as np

def as_committee_array(member_probs: list[np.ndarray]) -> np.ndarray:
    """
    Stack a list of per-member class-probability vectors into one array.

    Args:
        member_probs: length-N list, each entry shape (n_classes,) and summing to 1.

    Returns:
        Array of shape (n_members, n_classes), dtype float64.
    """
    arr = np.asarray(member_probs, dtype=np.float64)
    if arr.ndim != 2:
        raise ValueError(f"expected (n_members, n_classes), got shape {arr.shape}")
    row_sums = arr.sum(axis=1)
    if not np.allclose(row_sums, 1.0, atol=1e-3):
        raise ValueError("each member row must be a probability distribution summing to 1")
    return arr

# Step 2 — Compute Vote Entropy

Vote entropy looks only at each member’s hard argmax decision, then measures how split those votes are. A unanimous committee scores 0; an evenly split committee scores the maximum. Normalising by log(n_classes) puts the result in [0, 1].

def vote_entropy(committee: np.ndarray) -> float:
    """
    Normalised entropy of the hard-vote histogram across committee members.

    Args:
        committee: shape (n_members, n_classes) of class probabilities.

    Returns:
        Float in [0, 1]; higher means the committee's argmax votes are more split.
    """
    n_members, n_classes = committee.shape
    votes = committee.argmax(axis=1)                       # one class index per member
    counts = np.bincount(votes, minlength=n_classes)
    p = counts / n_members                                  # vote distribution
    nz = p[p > 0.0]
    entropy = -np.sum(nz * np.log(nz))
    return float(entropy / np.log(n_classes)) if n_classes > 1 else 0.0

# Step 3 — Compute Average KL Divergence from Consensus

Vote entropy ignores confidence: a 51/49 split and a 99/1 split both cast one vote each way. Average KL divergence fixes that by comparing every member’s full distribution against the committee mean (the consensus). This is the soft-disagreement signal, closely related to the entropy and margin scores used for segmentation masks, but measured between models rather than within one.

def mean_kl_disagreement(committee: np.ndarray, eps: float = 1e-12) -> float:
    """
    Average KL divergence of each member's distribution from the consensus mean.

    Args:
        committee: shape (n_members, n_classes) of class probabilities.
        eps: floor added before logs to avoid divide-by-zero.

    Returns:
        Non-negative float; higher means members' soft predictions diverge more.
        Equals the Jensen-Shannon-style consensus dispersion of the committee.
    """
    p = np.clip(committee, eps, 1.0)
    consensus = p.mean(axis=0, keepdims=True)              # shape (1, n_classes)
    kl_per_member = np.sum(p * (np.log(p) - np.log(consensus)), axis=1)
    return float(kl_per_member.mean())

# Step 4 — Combine into One Disagreement Score

Both signals are normalised to [0, 1] and blended with a weight alpha. Because the KL term is unbounded, it is squashed with a saturating transform first so a single wildly over-confident member cannot dominate the ranking.

def disagreement_score(committee: np.ndarray, alpha: float = 0.5) -> float:
    """
    Blend normalised vote entropy and squashed mean-KL into a single score.

    Args:
        committee: shape (n_members, n_classes).
        alpha: weight on vote entropy; (1 - alpha) weights the KL term.

    Returns:
        Float in [0, 1]; higher tiles are more informative to annotate.
    """
    h_vote = vote_entropy(committee)                        # already in [0, 1]
    kl = mean_kl_disagreement(committee)
    kl_norm = kl / (kl + 1.0)                               # saturating map to [0, 1)
    return float(alpha * h_vote + (1.0 - alpha) * kl_norm)

# Step 5 — Batch-Select While De-Duplicating Adjacent Tiles

Ranking alone is not enough — the top of the queue is usually a band of neighbouring tiles that share the same ambiguity. Greedily accept the highest-scoring tiles, but reject any candidate whose tile-grid position lies within min_gap cells of an already-selected tile. Tile positions are expressed as integer (row, col) grid coordinates, so adjacency is a cheap Chebyshev-distance check.

from dataclasses import dataclass

@dataclass
class TileScore:
    tile_id: str
    score: float
    row: int          # tile-grid row index
    col: int          # tile-grid column index

def select_batch(
    scored: list[TileScore],
    batch_size: int,
    min_gap: int = 2,
) -> list[TileScore]:
    """
    Greedily pick the highest-disagreement tiles, skipping any tile within
    `min_gap` grid cells (Chebyshev distance) of an already-selected tile.

    Args:
        scored: per-tile disagreement scores with grid positions.
        batch_size: number of tiles to send to annotators this round.
        min_gap: minimum grid separation between two selected tiles.

    Returns:
        Up to `batch_size` spatially spread-out TileScore objects, score-descending.
    """
    ranked = sorted(scored, key=lambda t: t.score, reverse=True)
    chosen: list[TileScore] = []
    for cand in ranked:
        too_close = any(
            max(abs(cand.row - c.row), abs(cand.col - c.col)) < min_gap
            for c in chosen
        )
        if too_close:
            continue
        chosen.append(cand)
        if len(chosen) == batch_size:
            break
    return chosen

# Step 6 — Wire It Together

The end-to-end pass turns raw committee predictions into an annotation batch:

def prioritize_tiles(
    predictions: dict[str, list[np.ndarray]],
    positions: dict[str, tuple[int, int]],
    batch_size: int = 25,
    alpha: float = 0.5,
    min_gap: int = 2,
) -> list[TileScore]:
    """Score every tile by committee disagreement and return one annotation batch."""
    scored: list[TileScore] = []
    for tile_id, member_probs in predictions.items():
        committee = as_committee_array(member_probs)
        row, col = positions[tile_id]
        scored.append(TileScore(tile_id, disagreement_score(committee, alpha), row, col))
    return select_batch(scored, batch_size, min_gap)

# Choosing Thresholds and Weights

Committee disagreement is a ranking signal first, but a few cut-offs keep the queue sensible:

• Committee size N: 3–5 checkpoints or augmentation passes. Fewer than 3 makes vote entropy too coarse; more than 5 rarely changes the ranking.
• Score floor for queueing: skip tiles scoring below ~0.15. Near-zero disagreement means the committee agrees and a label there teaches little.
• alpha (vote-entropy weight): start at 0.5. Raise toward 0.7 when member probabilities are poorly calibrated so the ranking leans on the calibration-robust vote count; lower toward 0.3 when calibration is trustworthy and you want soft-confidence spread to count more.
• min_gap (spatial exclusion): 2 grid cells for 256 px tiles at sub-metre resolution; increase to 3–4 for larger scenes where autocorrelation extends further.
• Batch size: size it to one annotation shift’s throughput so the loop retrains on a full, diverse batch rather than trickling single tiles.

# Common Errors and Fixes

Every tile scores near the maximum and the ranking is flat Cause: the “committee” is N copies of essentially the same model — checkpoints saved a few steps apart, or augmentations too weak to perturb the output. Fix: widen the committee. Use checkpoints from different seeds or epochs spaced further apart, or stronger augmentations (rotations, brightness shifts) so members make genuinely independent errors.

The batch is a tight cluster of neighbouring tiles despite select_batch Cause: positions holds pixel or geographic coordinates, not integer tile-grid indices, so the Chebyshev check compares the wrong units and never triggers. Fix: pass (row, col) grid indices; convert real-world coordinates to grid cells by integer-dividing by tile size before building the positions dict.

mean_kl_disagreement returns inf or nan Cause: a member emitted a hard 0.0 probability for some class, so log(0) appears in the KL sum. Fix: keep the eps clip in place (np.clip(committee, eps, 1.0)); it floors probabilities before any logarithm. Verify each member row still sums to ~1 after your inference softmax.

Disagreement tracks one over-confident model instead of true ambiguity Cause: an uncalibrated member outputs near-one-hot vectors, skewing the consensus and inflating KL. Fix: temperature-scale each member’s logits before scoring, or raise alpha so the calibration-robust vote-entropy term dominates the blend.

High-disagreement tiles turn out to be cloud, sensor artefacts, or nodata Cause: the committee disagrees because the input is corrupt, not because the class is genuinely ambiguous. Fix: run a quality mask before scoring and drop tiles exceeding a cloud or nodata fraction, so annotation budget is not spent labeling garbage.

# Related

• Uncertainty Sampling for Geospatial Active Learning — the broader topic area covering entropy, margin, and BALD scoring for ranking unlabeled tiles
• Entropy vs Margin Sampling for Segmentation Masks — single-model per-pixel uncertainty scoring that pairs naturally with committee disagreement
• Confidence Scoring for Geospatial Labels — per-annotation uncertainty values that complement disagreement when triaging a queue
• Active Learning & Model Feedback Loops for Geospatial Annotation — how tile prioritization feeds retraining triggers and drift detection across the whole loop

This guide covers one selection strategy within Uncertainty Sampling for Geospatial Active Learning, which is itself part of Active Learning & Model Feedback Loops for Geospatial Annotation.