Triggering Retraining from New Annotations with DVC

To retrain automatically when reviewers approve fresh labels, define a DVC pipeline stage whose dependency is the validated-annotations directory. When new annotations land, the content hash of that directory changes; dvc status reports the stage as out of date; and dvc repro — run from a scheduled CI job or a local filesystem watcher — rebuilds the versioned dataset and launches the downstream training stage. Guard the whole thing with a manifest so you only fire above a minimum-new-annotations threshold, and a single corrected tile never burns a GPU-hour. This turns the human review step of an active learning loop into a hands-off retraining event without a bespoke orchestration server.

# Why Hash-Driven Triggers Beat Commit Hooks

The naive approach — retrain on every push to the annotations repo — collapses the moment your team scales. Reviewers touch the annotations directory dozens of times a day: fixing a mislabeled building, adjusting a polygon vertex, adding a note in a sidecar file. Most of those edits are far too small to shift a geospatial detector’s weights, yet each one would enqueue an expensive run and pollute your checkpoint registry with near-identical models you then have to evaluate and prune.

Content hashing solves this at the level DVC already operates on. DVC records a hash of every tracked dependency in dvc.lock; a stage is “changed” only when the recomputed hash of its inputs differs from the recorded one. By pointing the dataset-build stage at the validated-annotations directory, you get a precise, reproducible signal — the same one that survives across machines and CI runners — instead of the noisy “someone committed something” signal a Git hook gives you. Layering a minimum-new-features threshold on top means the pipeline distinguishes a meaningful batch of new labels from a one-line touch, and only the former reaches the training stage.

# The Pipeline: Annotations, Dataset Build, Train

Install the toolchain first, pinning versions so the hash algorithm and CLI flags stay stable across every runner:

pip install dvc==3.55.2 geopandas==0.14.4 pyproj==3.6.1 shapely==2.0.6 pyyaml==6.0.2

Declare three stages in dvc.yaml. The build_dataset stage lists the validated-annotations directory as a dependency, so DVC re-hashes it on every dvc status. The train stage depends on the built dataset, which chains the two: a new dataset version forces a retraining run.

stages:
  validate_annotations:
    cmd: python -m pipeline.validate --src annotations/raw --out annotations/validated
    deps:
      - annotations/raw
      - pipeline/validate.py
    outs:
      - annotations/validated

  build_dataset:
    cmd: python -m pipeline.build_dataset --annotations annotations/validated --out data/dataset
    deps:
      - annotations/validated
      - pipeline/build_dataset.py
    outs:
      - data/dataset
    params:
      - dataset.tile_size
      - dataset.crs

  train:
    cmd: python -m pipeline.train --dataset data/dataset --out models/detector.pt
    deps:
      - data/dataset
      - pipeline/train.py
    params:
      - train.epochs
      - train.lr
    outs:
      - models/detector.pt
    metrics:
      - metrics/eval.json:
          cache: false

The validate_annotations stage matters: it is the boundary between raw reviewer edits and the hash that drives retraining. Only labels that pass geometry and CRS checks land in annotations/validated, so an invalid polygon never changes the dependency hash that fires training. Keep the tile size and target coordinate reference system in params.yaml — for example a metric projection such as EPSG:32633 — so a projection change also invalidates the build without touching the pipeline code.

# The Manifest-Driven Trigger

The trigger is the gate between “something changed” and “spend a GPU-hour.” It reads dvc status in JSON, confirms the build_dataset stage is out of date, then counts how many new features the validated directory holds compared to the last committed manifest. Only when that count clears the threshold does it return True.

from __future__ import annotations

import json
import subprocess
from dataclasses import dataclass
from pathlib import Path

import geopandas as gpd


@dataclass(frozen=True)
class TriggerConfig:
    stage: str = "build_dataset"
    annotations_dir: Path = Path("annotations/validated")
    manifest_path: Path = Path("annotations/manifest.json")
    min_new_features: int = 300
    min_new_fraction: float = 0.03


def stage_is_stale(stage: str) -> bool:
    """Return True if `dvc status` reports the given stage as changed."""
    proc = subprocess.run(
        ["dvc", "status", "--json", stage],
        capture_output=True, text=True, check=True,
    )
    status: dict[str, object] = json.loads(proc.stdout or "{}")
    return stage in status


def count_features(directory: Path) -> int:
    """Total feature count across every GeoJSON in the validated directory."""
    total = 0
    for path in sorted(directory.glob("**/*.geojson")):
        total += len(gpd.read_file(path))
    return total


def load_baseline(manifest_path: Path) -> int:
    if not manifest_path.exists():
        return 0
    data: dict[str, int] = json.loads(manifest_path.read_text())
    return int(data.get("feature_count", 0))


def should_retrain(cfg: TriggerConfig) -> tuple[bool, int]:
    """Decide whether a retraining run is justified.

    Returns (fire, new_feature_count).
    """
    if not stage_is_stale(cfg.stage):
        return False, 0

    current = count_features(cfg.annotations_dir)
    baseline = load_baseline(cfg.manifest_path)
    new = max(current - baseline, 0)

    absolute_ok = new >= cfg.min_new_features
    relative_ok = baseline == 0 or (new / baseline) >= cfg.min_new_fraction
    return (absolute_ok and relative_ok), new


def write_manifest(cfg: TriggerConfig, feature_count: int) -> None:
    """Persist the new baseline after a successful run."""
    cfg.manifest_path.write_text(
        json.dumps({"feature_count": feature_count}, indent=2)
    )

Two conditions must both hold: an absolute floor (min_new_features) and a relative floor (min_new_fraction). The relative check keeps the threshold meaningful as the dataset grows — 300 new features matter on a 2,000-feature set but are noise on a 200,000-feature set. Call write_manifest only after dvc repro succeeds, so a failed or skipped run leaves the baseline untouched and the new labels accumulate toward the next attempt. For local iteration, a watchdog observer on annotations/validated can call should_retrain on every change; the CI path calls the identical function on a schedule, keeping one source of truth.

One subtlety worth designing for early: the manifest stores a scalar feature count, which is enough to gate on volume but blind to which labels changed. If reviewers correct 300 existing polygons rather than adding 300 new ones, the count stays flat while the training signal shifts substantially. When that pattern matters for your workflow, extend the manifest to record a per-file hash map alongside the count, and treat any file whose hash changed as contributing to the “new” tally. That keeps corrections visible to the trigger without abandoning the cheap volume check for the common append-only case, and it composes cleanly with the class-coverage floor described in the threshold table below.

# Running It in CI on a Schedule

A scheduled GitHub Actions job pulls the tracked data, runs the trigger, and reproduces the pipeline only when the trigger fires. This is where hash detection pays off: the runner is stateless, but dvc.lock and the remote cache give it exactly the state it needs to decide.

name: retrain-on-new-annotations
on:
  schedule:
    - cron: "0 3 * * *"   # nightly at 03:00 UTC
  workflow_dispatch: {}

jobs:
  maybe-retrain:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install dvc[s3]==3.55.2 geopandas==0.14.4 shapely==2.0.6 pyproj==3.6.1
      - run: dvc pull annotations/validated.dvc
      - name: Evaluate trigger and reproduce
        run: |
          python - <<'PY'
          import sys
          from pipeline.trigger import TriggerConfig, should_retrain
          fire, new = should_retrain(TriggerConfig())
          print(f"new_features={new} fire={fire}")
          sys.exit(0 if fire else 78)
          PY
      - name: Retrain
        if: success()
        run: |
          dvc repro train
          dvc push
          python -m pipeline.trigger_update   # write_manifest after success

Exit code 78 is a neutral “nothing to do” that many CI systems treat as a non-failure skip; the Retrain step runs dvc repro train, which walks the dependency graph, rebuilds build_dataset because its annotations hash changed, then executes train. dvc push uploads both the new dataset version and the checkpoint so they are reproducible from the committed dvc.lock. This mirrors the snapshot mechanics in using DVC pipelines for automated snapshots, reused here as a retraining trigger rather than an archival one.

# Threshold Reference

Tune these guards to your model’s data appetite and the cost of a run. Start conservative — a threshold too low is worse than too high, because a flood of marginal checkpoints is expensive to evaluate.

The class-coverage floor is worth a note: a batch of 400 new features that are all one class can skew a detector toward that class. Extend should_retrain to inspect the label distribution of the new features and require a minimum spread before firing.

# Common Errors and Fixes

dvc status reports the stage stale on a fresh checkout even though nothing changed Root cause: the annotations directory was pulled with different line endings or file ordering, changing the tree hash. Fix: commit dvc.lock and pull the exact cached version with dvc pull before running the trigger; never re-add the directory with dvc add inside CI.

Retraining fires on a one-line correction to a single tile Root cause: the trigger only checked stage_is_stale and skipped the feature-count gate. Fix: ensure should_retrain returns after the min_new_features and min_new_fraction checks, and confirm the manifest baseline was written on the previous run.

subprocess raises CalledProcessError from dvc status --json Root cause: DVC exits non-zero when the stage name is unknown or the repo has no dvc.lock yet. Fix: run one dvc repro to initialise dvc.lock, and verify the stage name in TriggerConfig matches dvc.yaml exactly.

The manifest baseline drifts and the threshold never triggers again Root cause: write_manifest ran even when dvc repro failed, advancing the baseline past labels that were never trained on. Fix: call write_manifest only in the success branch of the CI job, after dvc push completes.

geopandas count differs between the watcher and CI Root cause: one environment read a stale local copy of the annotations directory. Fix: point both entry points at the DVC-tracked path and run dvc checkout before counting so the working tree matches dvc.lock.

# Related

• Closing the Loop with Automated Model Retraining — the topic area covering checkpoint promotion gates and evaluation guards that this trigger feeds into
• Implementing DVC for Geospatial Training Data — set up DVC remotes, tracking, and cache before wiring a retraining stage on top
• Using DVC Pipelines for Automated Dataset Snapshots — the snapshot pipeline pattern this guide reuses as a retraining trigger
• Active Learning & Model Feedback Loops for Geospatial Annotation — how automated retraining fits the wider loop of uncertainty sampling and drift detection

This guide is part of the broader Closing the Loop with Automated Model Retraining topic area within Active Learning & Model Feedback Loops for Geospatial Annotation.