Debugging Annotation Drift Across Dataset Versions

When two versions of a geospatial annotation dataset produce different model metrics without a deliberate data change, annotation drift is the most likely culprit. Resolve it by computing deterministic spatial and semantic deltas between consecutive version snapshots — using geometry tolerance checks in a shared metric coordinate reference system, attribute alignment, and statistical distribution profiling — before allowing the data to enter training queues. Drift originates from four distinct failure modes: reprojection-induced coordinate shifts, schema mutations between annotation tool releases, label taxonomy revisions from updated annotator guidelines, and precision loss during format serialization. Identifying the mode first determines which threshold and remediation path to apply.

# Why Annotation Drift Breaks Geospatial ML Pipelines

Even a sub-pixel coordinate shift in EPSG:4326 can collapse IoU scores below the acceptance threshold when the model evaluates predictions in a local metric projection. Schema changes that silently remap class IDs between annotation rounds cause training runs to learn from a relabeled dataset without triggering any visible pipeline error. Statistical drift from revised annotator guidelines degrades model recall in underrepresented classes before it appears in aggregate mAP. Without explicit version diffing, each of these failure modes is invisible until model performance regresses in production.

The three practical consequences of undetected drift:

1. Silent model degradation — accuracy drops surface in production before the dataset is flagged as corrupted.
2. Unreproducible training runs — the same DVC pipeline stage produces different outputs when the upstream data has drifted undetected.
3. Wasted rollback effort — without knowing the drift type, engineers revert entire dataset versions when only a subset of features or labels is affected.

# Root Cause Taxonomy for Geospatial Annotation Drift

Annotation drift in production ML pipelines falls into four diagnostic categories. Identifying the category first determines which tool and threshold to apply.

# Six-Step Debugging Workflow

# Step 1 — Lock Immutable Baselines

Never diff live data streams. Export fixed snapshots of both versions and verify SHA-256 checksums before any comparison to confirm the snapshots are stable:

import hashlib
import pathlib


def sha256_file(path: str) -> str:
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(65536), b""):
            h.update(chunk)
    return h.hexdigest()


# Verify before diffing
v1_hash: str = sha256_file("v1_annotations.gpkg")
v2_hash: str = sha256_file("v2_annotations.gpkg")
assert v1_hash != v2_hash, "Files are identical — no diff needed."
print(f"v1: {v1_hash}\nv2: {v2_hash}")

# Step 2 — Validate CRS Alignment

Confirm both versions share the same coordinate reference system before any spatial comparison. A mismatch in EPSG code is the single most common source of false-positive geometric drift because distance thresholds are interpreted in the wrong unit:

import geopandas as gpd
import shapely

v1: gpd.GeoDataFrame = gpd.read_file("v1_annotations.gpkg")
v2: gpd.GeoDataFrame = gpd.read_file("v2_annotations.gpkg")

if v1.crs != v2.crs:
    print(f"CRS mismatch: v1={v1.crs.to_epsg()}, v2={v2.crs.to_epsg()}")
    v2 = v2.to_crs(v1.crs)   # reproject v2 to match v1

# Enforce valid geometries on both before any spatial operation
v1["geometry"] = v1.geometry.apply(shapely.make_valid)
v2["geometry"] = v2.geometry.apply(shapely.make_valid)

If the data is stored in a geographic CRS like EPSG:4326, reproject both to a local metric system (e.g. EPSG:32633 for UTM Zone 33N) before computing distances — otherwise max_distance is interpreted in degrees and produces meaningless results.

# Step 3 — Compute Spatial Deltas with Hausdorff Distance

Match features using a nearest-neighbour spatial join bounded by a project-specific tolerance. Features that exceed the threshold are flagged as geometrically drifted:

# Preserve v2 geometries before the join — sjoin_nearest keeps only the left GDF geometry
v2_geoms: gpd.GeoSeries = v2.geometry.rename("geometry_v2")

matched: gpd.GeoDataFrame = gpd.sjoin_nearest(
    v1, v2,
    max_distance=0.5,      # metres; adjust per GSD (see thresholds table below)
    how="inner",
    suffixes=("_v1", "_v2"),
)

if matched.empty:
    raise ValueError(
        "No spatial matches within tolerance. "
        "Check CRS, tolerance value, or data overlap."
    )

# Re-attach v2 geometries for pair-wise metrics
matched = matched.join(v2_geoms, on="index_right")

matched["hausdorff_dist"] = matched.apply(
    lambda r: shapely.hausdorff_distance(r.geometry, r["geometry_v2"]), axis=1
)
matched["centroid_shift_m"] = matched.geometry.distance(
    matched["geometry_v2"].centroid
)

print(matched[["hausdorff_dist", "centroid_shift_m"]].describe())

# Step 4 — Align Attributes and Detect Label Mutations

Join spatially matched features and compare label columns, confidence scores, and custom metadata. Explicit class ID remapping checks prevent silent training degradation when a tool upgrade silently reassigns integer class IDs:

label_col: str = "class_id"

matched["label_mismatch"] = (
    matched[f"{label_col}_v1"] != matched[f"{label_col}_v2"]
)

mismatch_rate: float = matched["label_mismatch"].mean()
print(f"Label mismatch rate: {mismatch_rate:.1%}")

# Identify which class transitions are most common
transitions = (
    matched[matched["label_mismatch"]]
    .groupby([f"{label_col}_v1", f"{label_col}_v2"])
    .size()
    .reset_index(name="count")
    .sort_values("count", ascending=False)
)
print(transitions.head(10))

# Step 5 — Profile Label Distribution Shift with Jensen-Shannon Distance

Calculate Jensen-Shannon distance on label distributions. A score above 0.10 indicates meaningful distributional shift — likely from guideline revisions rather than a pipeline bug:

from scipy.spatial.distance import jensenshannon
import numpy as np
import pandas as pd

dist_v1: pd.Series = v1[label_col].value_counts(normalize=True).sort_index()
dist_v2: pd.Series = v2[label_col].value_counts(normalize=True).sort_index()

# Align indices to handle classes present in one version but not the other
common_idx = dist_v1.index.union(dist_v2.index)
p: np.ndarray = dist_v1.reindex(common_idx, fill_value=0.0).values
q: np.ndarray = dist_v2.reindex(common_idx, fill_value=0.0).values

jsd: float = float(jensenshannon(p, q, base=2))
print(f"Jensen-Shannon distance: {jsd:.4f}")

# High JSD + low Hausdorff drift = guideline change, not pipeline bug

# Step 6 — Scope the Rollback Using Drift Fingerprints

Cross-reference drift timestamps with CI/CD logs, annotation tool version bumps, and exporter configuration changes. Determine whether the drift is universal or confined to specific tiles or feature classes before committing to a rollback:

HAUSDORFF_THRESHOLD: float = 0.5   # metres for 0.3 m GSD imagery
JSD_THRESHOLD: float = 0.10

drifted_geom = matched[matched["hausdorff_dist"] > HAUSDORFF_THRESHOLD]
drifted_label = matched[matched["label_mismatch"]]
unmatched_count: int = len(v1) - len(matched)

print(f"Features with geometric drift : {len(drifted_geom):,} / {len(matched):,}")
print(f"Features with label drift     : {len(drifted_label):,} / {len(matched):,}")
print(f"Unmatched v1 features         : {unmatched_count:,}")

If drift is isolated to a single tile batch or exporter run, a targeted feature-level correction avoids a full dataset revert. If geometric drift is widespread, escalate to a full version rollback.

# Drift Detection Thresholds by Imagery Resolution

# Common Errors and Fixes

No spatial matches found within tolerance

Root cause: CRS mismatch left unresolved, or max_distance is set in degrees when the active CRS is geographic (EPSG:4326). Fix: reproject both datasets to a local metric CRS (e.g. EPSG:32633) before calling sjoin_nearest.

ValueError: geometry type changed between versions

Root cause: annotation tool update converted polygon features to bounding boxes during export. Fix: audit the schema migration log; re-export v2 from the pinned tool version or apply a geometry-type backfill script that reconstructs polygons from bounding box coordinates.

High Jensen-Shannon distance with low geometric drift

Root cause: annotator guideline revision changed the class distribution without altering feature geometries. Fix: treat this as a label taxonomy issue, not a pipeline bug — version the guidelines alongside the data using DVC pipeline stages so guideline changes produce a traceable commit.

hausdorff_dist is NaN for a subset of rows

Root cause: sjoin_nearest matched a feature to a null or empty geometry in v2. Fix: add a v2.geometry.is_empty | v2.geometry.isna() pre-filter before the join and log all dropped feature IDs to a separate audit file.

This workflow is one diagnostic component of the broader Rollback Strategies for Corrupted Spatial Datasets guide, which covers choosing between partial feature rollback, full version revert, and re-annotation.

Related

• Rollback Strategies for Corrupted Spatial Datasets — choosing between partial feature rollback, full version revert, and re-annotation
• Tracking Annotation Changes with SHA Hashing — building the stable checksum baseline that makes drift detection reliable
• Implementing DVC for Geospatial Training Data — versioning pipelines and data together so drift is always attributable to a specific commit
• Coordinate Reference Systems in Annotation Pipelines — understanding why CRS choices upstream produce geometric drift downstream
• Dataset Versioning & Spatial Data Sync — the parent section covering the full versioning lifecycle