Confidence Scoring for Geospatial Labels

When a pre-labeling model trained on Sentinel-2 imagery is applied to a new SAR dataset, its raw softmax outputs appear confident — but those probabilities are miscalibrated for the new sensor modality. If your pipeline treats every model output above 0.5 as a valid training label, the resulting dataset is silently contaminated: noisy polygons enter the training shard, loss curves stall, and the root cause is invisible until model accuracy collapses on held-out evaluation tiles weeks later.

Confidence scoring addresses this by transforming raw annotation outputs — annotator agreement, geometry quality, and model probabilities — into a composite quality signal that controls training inclusion, active learning prioritization, and QA routing. In production geospatial annotation fundamentals & architecture pipelines, not all labeled polygons, bounding boxes, or raster masks carry equal reliability. A systematic confidence scoring framework isolates high-fidelity training samples, triggers targeted human review, and maintains reproducible dataset versions tied to explicit quality gates.

# Prerequisites & Toolchain Alignment

Confidence scoring sits downstream of geometry validation but upstream of dataset sharding and model training. Before implementing automated scoring, your annotation infrastructure must expose structured metadata alongside geometry.

Required Python packages (pinned):

geopandas==0.14.4
shapely==2.0.4
pyproj==3.6.1
pandas==2.2.2
numpy==1.26.4
scikit-learn==1.4.2

System dependencies: GDAL 3.8+, PROJ 9.3+

Spatial knowledge prerequisites:

• Understanding of coordinate reference systems in annotation pipelines, particularly the distinction between geographic (EPSG:4326) and local projected CRS (e.g., a UTM zone), since IoU calculations on unprojected coordinates produce distorted area measurements proportional to latitude.
• Familiarity with the geometric differences between vector and raster annotation workflows, as pixel-level quantization in raster masks can artificially depress boundary agreement metrics when compared against vector polygons.
• Access to annotation logs or database exports containing per-label metadata: annotator IDs, timestamps, tool versions, and optional model-assisted prediction probabilities.

# Signal Extraction Through QA Routing: Pipeline Architecture

The diagram below shows how raw annotations flow from ingestion through scoring and into training or QA queues. The <defs> element defines the arrowhead marker used on all flow lines.

# Core Scoring Workflow

# Step 1 — Ingest and Normalize Annotations

Load raw annotations into a GeoDataFrame. Strip invalid geometries, enforce a unified CRS, and align attribute schemas. Missing metadata fields should be imputed with neutral baseline values (e.g., confidence=0.5) rather than dropped, preserving dataset completeness.

Geometry validation is non-negotiable at this stage. Self-intersecting polygons or unclosed rings will corrupt downstream spatial joins and IoU threshold calculations. Use shapely’s validity checks to flag or repair topological errors before scoring begins. All geometries must be re-projected to a local metric CRS — never compute areas or overlaps in EPSG:4326 (degrees).

import geopandas as gpd
from shapely.validation import make_valid

def ingest_annotations(
    path: str,
    target_crs: str = "EPSG:32632",  # UTM zone 32N — replace with your region
) -> gpd.GeoDataFrame:
    gdf = gpd.read_file(path)
    if gdf.crs is None:
        raise ValueError("Source file has no CRS defined. Set it before ingestion.")
    gdf = gdf.to_crs(target_crs)
    gdf["geometry"] = gdf["geometry"].apply(
        lambda g: make_valid(g) if g is not None and not g.is_valid else g
    )
    gdf = gdf[gdf.geometry.notna() & ~gdf.geometry.is_empty].copy()
    # Neutral baseline for missing metadata signals
    for col in ("iou_score", "geometry_score", "model_prob"):
        if col not in gdf.columns:
            gdf[col] = 0.5
    return gdf

# Step 2 — Extract Orthogonal Quality Signals

Confidence derived from a single metric captures only one failure mode. Production pipelines combine at least three orthogonal signals.

The diagram below illustrates how each signal type targets a distinct source of annotation noise: annotator disagreement, geometric irregularity, and model uncertainty. All three must be present and normalized before aggregation.

Inter-annotator agreement (IoU): Overlap consistency when multiple labelers annotate the same tile. Measured via pairwise Intersection-over-Union or Dice coefficient on reprojected geometries. Pairs are constructed using a spatial index to avoid O(n²) comparisons.

Geometric consistency score: Boundary regularity, topology validity, and alignment with expected object scales. Highly fragmented or deeply concave polygons often indicate tracing fatigue or ambiguous imagery. A useful proxy is the ratio of polygon area to its convex hull area — compact objects score high, erratic outlines score low.

Model-assisted probability: If using semi-automated labeling via a foundation model or pre-labeling classifier, extract raw softmax or sigmoid probabilities. These provide a strong prior but require calibration — a model predicting 0.92 on a new sensor modality may reflect an empirical accuracy of only 0.74.

When evaluating raster masks alongside vector outputs, account for pixel-level quantization, which can artificially depress boundary agreement metrics. Consult the vector vs. raster annotation workflows trade-off analysis before mixing signal types.

import numpy as np

def geometry_compactness(geom) -> float:
    """Ratio of polygon area to convex hull area. Returns 0.5 for None."""
    if geom is None or geom.is_empty:
        return 0.5
    hull_area = geom.convex_hull.area
    return float(geom.area / hull_area) if hull_area > 0 else 0.5

def pairwise_iou(geom_a, geom_b) -> float:
    intersection = geom_a.intersection(geom_b).area
    union = geom_a.union(geom_b).area
    return float(intersection / union) if union > 0 else 0.0

# Step 3 — Calibrate and Aggregate Composite Scores

Raw signals rarely share the same scale or distribution. Normalize each metric to [0, 1] using min-max scaling or rank-based transformation, then apply domain-informed weights.

Aggregation must use a weighted geometric mean rather than arithmetic mean. A single unreliable metric (e.g., IoU of 0.08) should drag the composite score down more aggressively than a high score on another signal can lift it. This prevents noisy labels from slipping through when one signal is artificially inflated by the pre-labeling model.

import geopandas as gpd
from sklearn.preprocessing import MinMaxScaler

def compute_confidence_scores(
    gdf: gpd.GeoDataFrame,
    signal_cols: list[str] = ["iou_score", "geometry_score", "model_prob"],
    weights: dict[str, float] = {"iou_score": 0.40, "geometry_score": 0.30, "model_prob": 0.30},
) -> gpd.GeoDataFrame:
    """
    Compute composite confidence scores via weighted geometric mean.
    Requires CRS to be set on gdf before calling.
    """
    if gdf.crs is None:
        raise ValueError("GeoDataFrame must have a defined CRS before scoring.")

    for col in signal_cols:
        if col not in gdf.columns:
            gdf[col] = 0.5

    # Normalize signals to [0, 1]
    scaler = MinMaxScaler()
    signal_matrix = gdf[signal_cols].fillna(0.5).values.astype(float)
    normalized = scaler.fit_transform(signal_matrix)

    # Build weight vector — normalized to sum to 1.0
    w = np.array([weights.get(col, 0.0) for col in signal_cols], dtype=float)
    w /= w.sum()

    # Weighted geometric mean via log-space dot product
    epsilon = 1e-6
    log_signals = np.log(normalized + epsilon)
    composite_log = log_signals @ w
    composite_score = np.exp(composite_log)

    gdf = gdf.copy()
    gdf["confidence_score"] = np.clip(composite_score, 0.0, 1.0)
    return gdf

The epsilon guard prevents RuntimeWarning when a normalized signal exactly reaches zero, which can occur when min-max scaling produces a degenerate lower bound.

# Step 4 — Apply QA Routing Thresholds by Confidence Tier

Once composite scores are computed, route labels into three tiers. Thresholds below are starting points and should be swept per-class (see the configuration table in the next section).

Hard-coding a universal threshold across all classes biases your training distribution. Defining ROI label taxonomies for aerial imagery typically reveals that certain classes — shadowed rooftops, transitional vegetation, partially occluded vehicles — inherently produce lower inter-annotator agreement. Maintain a per-class threshold YAML and reload it per pipeline run.

def route_by_confidence(
    gdf: gpd.GeoDataFrame,
    class_thresholds: dict[str, dict[str, float]] | None = None,
    default_high: float = 0.85,
    default_low: float = 0.60,
) -> gpd.GeoDataFrame:
    """
    Assign routing tier per label. class_thresholds maps class_label ->
    {"high": float, "low": float}. Falls back to default_high / default_low.
    """
    tiers = []
    for _, row in gdf.iterrows():
        thresholds = (class_thresholds or {}).get(row.get("class_label", ""), {})
        high = thresholds.get("high", default_high)
        low = thresholds.get("low", default_low)
        score = row["confidence_score"]
        if score >= high:
            tiers.append("training")
        elif score >= low:
            tiers.append("training_weighted")
        else:
            tiers.append("qa_queue")
    gdf = gdf.copy()
    gdf["routing_tier"] = tiers
    return gdf

# Spatial Parameters & Configuration Reference

# Edge Cases & Spatial Gotchas

CRS mismatch at signal extraction time: Computing IoU in EPSG:4326 distorts area calculations, especially above ±50° latitude. All geometries must share a local projected CRS before any overlap computation. Validate CRS consistency with a unit test that checks gdf.crs.is_projected before each scoring run.

Raster mask vs. vector polygon boundary discrepancy: Pixel quantization in raster masks introduces systematic boundary offsets proportional to GSD (ground sampling distance). A 30 cm GSD image annotated in raster format will show boundary disagreements of up to one pixel width (30 cm) even for perfect polygon traces. Do not penalize boundary agreement below the GSD floor.

Over-reliance on model-assisted probability on OOD scenes: A pre-labeling model calibrated on one sensor modality (e.g., Sentinel-2 multispectral) will be overconfident when applied to a new modality (e.g., SAR intensity). Reduce the model_prob weight or apply domain-specific recalibration before scoring new sensor data.

Annotator behavior drift: Confidence distributions shift as annotation teams scale, tools are updated, or labeling guidelines change. Schedule quarterly recalibration runs using recent QA-reviewed samples drawn from the SHA-hashed annotation change log to maintain scoring fidelity over time.

Self-intersecting polygons after tool export: Some annotation tools export geometries that pass UI validation but contain topological errors (butterfly ties, dangling edges). shapely.validation.make_valid() corrects most cases, but verify the repaired geometry area has not changed by more than 1–2% compared to the original.

Static thresholds across imbalanced class distributions: Rare classes have fewer inter-annotator overlap pairs, so their IoU signal has higher variance. Apply class-stratified threshold sweeps that optimize for downstream [email protected] or F1-score per class, not a shared global cutoff.

# Frequently Asked Questions

Why does my composite score cluster near 0.5 for all labels?

This typically indicates missing metadata: if all three signal columns were imputed to the neutral baseline of 0.5, the geometric mean will also be 0.5. Verify that your annotation export includes iou_score, geometry_score, and model_prob before calling the scoring function. Log a warning when more than 20% of rows are imputed.

Why do boundary-dominant classes (roads, coastlines) score low on geometry consistency?

Linear features have low compactness scores by definition — their area-to-convex-hull ratio approaches zero. For elongated classes, replace the compactness signal with a sinuosity or medial-axis deviation measure. Alternatively, exclude those classes from the geometry_score signal and increase the weight of iou_score and model_prob.

Should I use arithmetic or geometric mean to aggregate signals?

Always use weighted geometric mean. Arithmetic mean allows a high score on one signal to mask a critically low score on another. Geometric mean amplifies the penalty for any single low-confidence signal, which prevents noisy labels from slipping through quality gates.

How do I prevent calibration leakage?

Use spatial block cross-validation: divide the annotation extent into a regular grid of tiles (e.g., 5 × 5 km), assign each tile to a fold, and ensure no tile appears in both calibration and test folds. The scikit-learn GroupKFold estimator with a tile-ID group column handles this cleanly. Random train/validation splits leak geographic context due to spatial autocorrelation.

How do I handle per-class thresholds for ambiguous aerial imagery?

Maintain a per-class threshold configuration (e.g., a JSON or YAML mapping of class_id to min_confidence). Visually ambiguous classes like transitional vegetation or shadowed rooftops inherently produce lower inter-annotator agreement; a universal threshold will systematically exclude too many valid labels from these classes.

# Integration & Automation Hooks

Confidence scoring integrates naturally into DVC pipelines for geospatial training data. Define each scoring stage as a DVC stage so that parameter changes (e.g., weight adjustments) automatically trigger pipeline reruns and produce a new dataset version with its own SHA digest — traceable via the SHA-hashing annotation change tracking workflow.

# dvc.yaml — confidence scoring stage
stages:
  score_annotations:
    cmd: python score_pipeline.py --config params.yaml
    deps:
      - annotations/raw/
      - score_pipeline.py
    params:
      - params.yaml:
          - scoring.weights
          - scoring.thresholds
    outs:
      - annotations/scored/gdf_scored.parquet
    metrics:
      - metrics/scoring_summary.json:
          cache: false

For Label Studio exports, parse the JSON export to extract annotator IDs and per-task annotation counts before calling compute_confidence_scores. Each Label Studio task maps to one tile; multiple completions on the same task provide the inter-annotator pairs needed for IoU signal extraction.

To wire into a CI gate, add a GitHub Actions step after each annotation batch export:

# .github/workflows/annotation_qa.yml (excerpt)
- name: Score and validate annotations
  run: |
    python score_pipeline.py --config params.yaml
    python validate_scores.py --min-high-tier-pct 0.60

The validate_scores.py check fails the CI run if fewer than 60% of labels reach the high-confidence tier, surfacing annotation quality regressions before they reach training.

# Validation & Testing

Before promoting a scored dataset to a training shard, run the following assertions to catch silent failures:

import geopandas as gpd
import numpy as np

def validate_scored_gdf(gdf: gpd.GeoDataFrame) -> None:
    # CRS must be projected (metric) before scoring
    assert gdf.crs is not None and gdf.crs.is_projected, (
        f"Expected projected CRS, got: {gdf.crs}"
    )
    # All geometries must be valid after make_valid pass
    invalid = gdf[~gdf.geometry.is_valid]
    assert len(invalid) == 0, (
        f"{len(invalid)} invalid geometries remain after validation pass"
    )
    # confidence_score must be in [0, 1]
    assert gdf["confidence_score"].between(0.0, 1.0).all(), (
        "confidence_score values outside [0, 1] detected"
    )
    # At least one label should reach training tier (sanity check)
    assert (gdf["routing_tier"] == "training").any(), (
        "No labels reached the high-confidence training tier — review thresholds"
    )
    # Score distribution should not be degenerate (all 0.5 signals indicate missing metadata)
    score_std = gdf["confidence_score"].std()
    assert score_std > 0.01, (
        f"Score distribution has near-zero variance ({score_std:.4f}) — check signal extraction"
    )

# Run as a pytest test or as a post-scoring assertion in the pipeline:
# validate_scored_gdf(scored_gdf)

Calibration verification: After applying Platt scaling or isotonic regression, check that the calibration curve (reliability diagram) shows points close to the diagonal across all confidence bins. A systematic deviation in the 0.6–0.8 range is common for models applied to new sensor modalities and indicates the model_prob weight should be reduced.

This workflow is one component of the broader Geospatial Annotation Fundamentals & Architecture pipeline, covering everything from CRS contracts and label taxonomy design through geometry validation and export.

# Related

• Coordinate Reference Systems in Annotation Pipelines — CRS selection, reprojection, and the PROJ/GDAL stack required before spatial scoring
• Calculating IoU Thresholds for Geospatial Object Detection — per-class IoU cutoffs by mission type and GSD range
• Defining ROI Label Taxonomies for Aerial Imagery — class hierarchy design and the class-level threshold decisions that feed into confidence routing
• Vector vs. Raster Annotation Workflows — how annotation modality choice affects boundary agreement signals
• Tracking Annotation Changes with SHA Hashing — version-controlled audit trail for scored datasets