Closing the Loop with Automated Model Retraining

A geospatial detection team hits this wall constantly: annotators keep validating fresh satellite tiles, the reviewed labels pile up in a bucket, and yet the model in production has not changed in three months. Nobody wired the last step. The deployed checkpoint keeps mis-detecting the exact objects the new annotations were created to fix — new construction it reads as bare ground, a solar farm it labels as water — because those corrections never re-entered training. The labeling budget is spent, the model stays stale, and every inference run reproduces the same errors at scale.

Closing that gap is the whole point of an active learning loop: the model tells you which tiles to label, annotators label them, and the loop must then feed those labels back into a new model version safely. This guide covers the return path — how to turn a stream of validated annotations into a retrained checkpoint that only reaches production if it demonstrably does not regress. The hard part is not the training call; it is the promotion gate and the rollback that protect a live model from a bad batch.

# Prerequisites & Pipeline Toolchain

The loop coordinates three subsystems: a versioned data store, a training runtime, and an experiment log. Pin every version so a retraining run months from now reproduces byte-for-byte.

Required Python packages (pinned):

• dvc[s3]==3.51.2 — content-addressed dataset versions and pipeline stages
• torch==2.3.1 — training and fine-tuning runtime
• geopandas==0.14.4 — reading and joining validated annotation geometries
• pyarrow==16.1.0 — GeoParquet I/O for annotation manifests
• mlflow==2.14.1 or wandb==0.17.4 (optional) — experiment tracking and checkpoint registry

Install the core stack with:

pip install "dvc[s3]==3.51.2" torch==2.3.1 geopandas==0.14.4 pyarrow==16.1.0
# optional experiment tracker (pick one)
pip install mlflow==2.14.1

The loop assumes your training data is already under DVC versioning, so that each dataset revision is content-addressed and every model can be traced to the exact tiles it saw. It also assumes annotations arrive pre-normalized to a single coordinate reference system — mixing projections silently shifts geometry and poisons a retraining set. Because distance-sensitive evaluation such as IoU depends on a metric CRS, keep the pipeline on a local UTM zone rather than EPSG:4326.

Baseline checklist before wiring the loop:

• Base training set is DVC-tracked with a resolvable revision hash
• A frozen hold-out evaluation region exists and is excluded from all labeling
• The deployed checkpoint and its hold-out score are recorded as the baseline
• A rollback path can restore the previous checkpoint and dataset revision atomically

# Core Retraining Loop Workflow

The five stages below map one-to-one onto the diagram above. Each returns a value the next stage consumes, so the whole loop can run as a single orchestrated job or as separate pipeline stages.

# Assemble the incremental training set from validated annotations

New annotations arrive as a stream of reviewed tiles. Before they touch training, deduplicate them against the base manifest, drop anything still pending review, and emit a single content-addressed dataset version. Deduplication by tile identity is what keeps a re-reviewed tile from being counted twice and skewing class balance.

from __future__ import annotations

from pathlib import Path

import geopandas as gpd
import pandas as pd


def assemble_increment(
    validated_dir: Path,
    base_manifest: Path,
    replay_fraction: float = 0.15,
    seed: int = 13,
) -> gpd.GeoDataFrame:
    """Build the incremental training manifest from newly validated tiles.

    Returns a GeoDataFrame combining all fresh annotations with a small,
    randomly sampled replay slice of the base set to limit forgetting.
    """
    base: gpd.GeoDataFrame = gpd.read_parquet(base_manifest)

    fresh_parts: list[gpd.GeoDataFrame] = [
        gpd.read_parquet(p) for p in sorted(validated_dir.glob("*.parquet"))
    ]
    if not fresh_parts:
        raise RuntimeError("No validated annotation batches found — nothing to retrain on.")

    fresh: gpd.GeoDataFrame = pd.concat(fresh_parts, ignore_index=True)
    fresh = fresh[fresh["review_status"] == "validated"].copy()

    # Deduplicate: a re-reviewed tile must appear once, keeping the newest label.
    fresh = fresh.sort_values("reviewed_at").drop_duplicates("tile_id", keep="last")

    # Never train on tiles reserved for evaluation.
    fresh = fresh[~fresh["tile_id"].isin(base.loc[base["split"] == "holdout", "tile_id"])]

    replay: gpd.GeoDataFrame = (
        base[base["split"] == "train"]
        .sample(frac=replay_fraction, random_state=seed)
        .copy()
    )

    increment = pd.concat([fresh, replay], ignore_index=True)
    increment.attrs["n_fresh"] = int(len(fresh))
    increment.attrs["n_replay"] = int(len(replay))
    return gpd.GeoDataFrame(increment, crs=base.crs)

The replay_fraction slice is deliberate: fine-tuning purely on new tiles is the fastest route to catastrophic forgetting, covered in the gotchas below. Mixing in a random sample of previously seen tiles anchors the model to its existing competencies.

# Define an evaluation gate

The gate is the safety valve. It compares the candidate’s score on a frozen hold-out set against the deployed baseline and returns a single boolean. Encode the rule as data, not scattered if statements, so the same policy governs every run and can be logged.

from __future__ import annotations

from dataclasses import dataclass


@dataclass(frozen=True)
class EvalGate:
    """Promotion policy for a retraining candidate.

    A candidate passes only if its hold-out metric clears the baseline
    minus a noise tolerance, with no per-class score falling below a floor.
    """

    metric: str = "map_50"
    baseline: float = 0.0
    tolerance: float = 0.004        # accepted noise band around the baseline
    required_gain: float = 0.0      # extra margin the candidate must earn
    per_class_floor: float = 0.30   # no class may drop under this

    def passes(self, overall: float, per_class: dict[str, float]) -> bool:
        clears_overall = overall >= self.baseline - self.tolerance + self.required_gain
        clears_classes = all(score >= self.per_class_floor for score in per_class.values())
        return clears_overall and clears_classes

Freezing the hold-out region is non-negotiable. The assemble_increment function above already refuses any tile whose split is holdout; the gate depends on that region never seeing a fresh label, or the comparison against the baseline becomes meaningless.

# Retrain or fine-tune the candidate

Warm-start from the production checkpoint rather than training from scratch. Fine-tuning on the incremental manifest is faster, cheaper, and — combined with the replay slice — far less destructive to prior knowledge. Log the run so the candidate is reproducible.

from __future__ import annotations

from pathlib import Path

import torch
from torch.utils.data import DataLoader


def fine_tune_candidate(
    model: torch.nn.Module,
    train_loader: DataLoader,
    production_ckpt: Path,
    epochs: int = 4,
    lr: float = 1e-4,
    device: str = "cuda",
) -> torch.nn.Module:
    """Warm-start from production weights and fine-tune on the increment."""
    state = torch.load(production_ckpt, map_location=device)
    model.load_state_dict(state["model"])
    model.to(device).train()

    optimizer = torch.optim.AdamW(model.parameters(), lr=lr)
    for epoch in range(epochs):
        running: float = 0.0
        for images, targets in train_loader:
            images = images.to(device)
            targets = [{k: v.to(device) for k, v in t.items()} for t in targets]

            optimizer.zero_grad()
            loss_dict = model(images, targets)          # detection-style loss dict
            loss = sum(loss_dict.values())
            loss.backward()
            optimizer.step()
            running += float(loss.detach())

        print(f"epoch {epoch + 1}/{epochs}  mean_loss={running / len(train_loader):.4f}")

    return model

If you use an experiment tracker, wrap this call with mlflow.start_run() (or wandb.init()) and log epochs, lr, the dataset revision hash, and the resulting checkpoint as an artifact. The tracker becomes your checkpoint registry and your audit trail when a regression needs diagnosing weeks later.

# Promote the checkpoint only if the gate passes

Evaluate the fine-tuned candidate on the frozen hold-out, feed both the overall and per-class scores to the gate, and swap the production pointer only on a pass. The swap itself must be atomic — write the new checkpoint beside the old and move a symlink, never overwrite in place.

from __future__ import annotations

import os
from pathlib import Path


def promote_if_gated(
    candidate_ckpt: Path,
    production_link: Path,
    gate: EvalGate,
    overall: float,
    per_class: dict[str, float],
) -> bool:
    """Atomically point production at the candidate iff it clears the gate."""
    if not gate.passes(overall, per_class):
        worst = min(per_class, key=per_class.get)
        print(
            f"REJECTED: {gate.metric}={overall:.4f} vs baseline {gate.baseline:.4f} "
            f"(worst class '{worst}'={per_class[worst]:.4f}). Production unchanged."
        )
        return False

    tmp_link = production_link.with_suffix(".swap")
    tmp_link.symlink_to(candidate_ckpt.resolve())
    os.replace(tmp_link, production_link)      # atomic pointer swap
    print(f"PROMOTED: {candidate_ckpt.name} → {production_link} ({gate.metric}={overall:.4f})")
    return True

# Roll back on regression

Rejection at the gate leaves production untouched — that is already a rollback in the pre-deploy sense. The dangerous case is a candidate that clears the gate but degrades on live traffic, caught by a post-deploy monitor. Restore both the checkpoint pointer and the dataset revision so the next loop iteration starts from a coherent known-good state.

from __future__ import annotations

import os
import subprocess
from pathlib import Path


def rollback(
    production_link: Path,
    last_good_ckpt: Path,
    last_good_data_rev: str,
) -> None:
    """Restore the previous checkpoint and pin the dataset to its last good revision."""
    tmp_link = production_link.with_suffix(".swap")
    tmp_link.symlink_to(last_good_ckpt.resolve())
    os.replace(tmp_link, production_link)

    # Return the DVC-tracked data to the revision the good checkpoint trained on.
    subprocess.run(["dvc", "checkout", "--rev", last_good_data_rev], check=True)
    print(f"ROLLED BACK: production → {last_good_ckpt.name}, data → {last_good_data_rev}")

Detailed recovery playbooks for the data side of that restore live in the guide on rollback strategies for corrupted spatial datasets; the loop here reuses the same revision-pinning discipline for the model side.

# Promotion Parameters & Configuration Reference

# Edge Cases & Gotchas

# Catastrophic forgetting on the incremental batch

Fine-tuning a network on a small, homogeneous batch of new tiles will overwrite weights that encoded rare classes, so the model gets better at whatever the fresh batch contains and quietly worse at everything else. The replay_fraction slice is the first defence; the per_class_floor in the gate is the backstop. If forgetting persists, lower the learning rate, freeze the backbone and fine-tune only the detection or segmentation head, or add an elastic-weight-consolidation penalty that anchors important parameters to their production values.

# Label noise from fresh annotations

Newly validated tiles are not automatically trustworthy. A single annotator’s systematic error — a mislabeled class, a consistently loose bounding box — enters training as ground truth and the model dutifully learns it. Before assembling the increment, cross-check fresh labels against model predictions and flag tiles where a high-confidence prediction disagrees sharply with the new label for a second review. Calibrated confidence scores make that disagreement signal meaningful rather than noisy.

# Eval-set contamination via spatial autocorrelation

Adjacent tiles are not independent samples: they share roads, buildings, and terrain that spill across tile edges. A random train/hold-out split therefore places near-copies of hold-out content into training, and the evaluation score climbs for reasons that have nothing to do with real generalization. Split the hold-out by geographic block — an entire region or acquisition scene — and add a buffer so no training tile physically borders a hold-out tile. Freeze that block once and never label inside it.

# Silent metric regression

The most expensive failure is a candidate that clears the overall gate while collapsing on a class that barely registers in the aggregate. A one-point mean-average-precision gain can hide a rare-object recall dropping by half. Guarding the gate with a per-class floor catches most of these, but also track the full per-class table across loop iterations and alert on any class trending down for two consecutive promotions, even when each individual step stays above the floor.

# Integration & Automation Hooks

# DVC pipeline stage

Express the loop as pipeline stages so each dataset revision deterministically produces one candidate, and DVC’s dependency hashing skips retraining when neither data nor code changed. The child guide on triggering retraining from new annotations with DVC walks through the full trigger wiring; the stage skeleton is:

# dvc.yaml
stages:
  assemble_increment:
    cmd: python scripts/assemble_increment.py --validated data/validated/ --base data/base.parquet
    deps:
      - scripts/assemble_increment.py
      - data/validated/
    outs:
      - data/increment.parquet
  train_candidate:
    cmd: python scripts/train_candidate.py --data data/increment.parquet --out models/candidate.pt
    deps:
      - scripts/train_candidate.py
      - data/increment.parquet
    outs:
      - models/candidate.pt
  eval_gate:
    cmd: python scripts/eval_gate.py --candidate models/candidate.pt --holdout data/holdout.parquet
    deps:
      - scripts/eval_gate.py
      - models/candidate.pt
    metrics:
      - reports/gate.json

# CI promotion job

# .github/workflows/retrain_gate.yml
name: retrain promotion gate
on:
  workflow_dispatch:
  schedule:
    - cron: "0 3 * * 1"      # weekly candidate build, gate decides promotion
jobs:
  gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: {python-version: "3.11"}
      - run: pip install "dvc[s3]==3.51.2" torch==2.3.1 geopandas==0.14.4
      - run: dvc repro eval_gate
      - name: Promote or reject
        run: python scripts/promote.py --report reports/gate.json --production models/production.pt

# Validation & Testing

Test the loop’s decision logic in isolation, without a GPU, by driving the gate with synthetic scores. The gate is the component most likely to silently rot, so it deserves the most coverage.

from __future__ import annotations


def test_gate_rejects_per_class_collapse() -> None:
    gate = EvalGate(baseline=0.62, tolerance=0.004, per_class_floor=0.30)
    # Overall improves, but one class collapses below the floor.
    passed = gate.passes(overall=0.64, per_class={"building": 0.71, "solar": 0.18})
    assert passed is False, "gate must reject a rare-class collapse hidden by overall gain"


def test_gate_accepts_within_tolerance() -> None:
    gate = EvalGate(baseline=0.62, tolerance=0.004, per_class_floor=0.30)
    passed = gate.passes(overall=0.618, per_class={"building": 0.70, "solar": 0.44})
    assert passed is True, "a candidate inside the noise band with healthy classes must pass"


def test_increment_excludes_holdout(tmp_path) -> None:
    import geopandas as gpd
    from shapely.geometry import box

    holdout = gpd.GeoDataFrame(
        {"tile_id": ["h1"], "split": ["holdout"], "review_status": ["validated"]},
        geometry=[box(0, 0, 1, 1)], crs="EPSG:32632",
    )
    holdout.to_parquet(tmp_path / "base.parquet")

    batch = gpd.GeoDataFrame(
        {"tile_id": ["h1", "t2"], "split": ["train", "train"],
         "review_status": ["validated", "validated"],
         "reviewed_at": [1, 2]},
        geometry=[box(0, 0, 1, 1), box(2, 2, 3, 3)], crs="EPSG:32632",
    )
    (tmp_path / "validated").mkdir()
    batch.to_parquet(tmp_path / "validated" / "b1.parquet")

    inc = assemble_increment(tmp_path / "validated", tmp_path / "base.parquet", replay_fraction=0.0)
    assert "h1" not in set(inc["tile_id"]), "hold-out tile must never enter the increment"

Run the suite with pytest -q inside the CI gate job so a broken promotion policy fails the build before any checkpoint moves. Pair it with a post-deploy monitor that samples live predictions and triggers rollback when the tracked metric drops more than two standard deviations below the promoted baseline.

# Frequently Asked Questions

# Related

• Triggering Retraining from New Annotations with DVC — the DVC trigger stage that detects validated batches and launches the loop
• Uncertainty Sampling for Geospatial Active Learning — how the loop chooses which tiles to send for labeling in the first place
• Detecting Distribution Drift in Spatial Datasets — the drift signal that fires the retraining trigger before a model decays
• Implementing DVC for Geospatial Training Data — the content-addressed dataset versioning every retraining run depends on
• Rollback Strategies for Corrupted Spatial Datasets — restoring a known-good data revision when a candidate regresses in production

This guide is part of the broader Active Learning & Model Feedback Loops for Geospatial Annotation topic area, which connects tile selection, retraining, and drift detection into one feedback system.