Rollback Strategies for Corrupted Spatial Datasets

A corrupted geospatial training dataset rarely announces itself. A malformed coordinate reference system embedded in a single GeoTIFF tile can silently shift every bounding box by hundreds of metres before IoU metrics collapse. A truncated multipart upload leaves a valid-looking file that fails mid-epoch. An annotation platform exporting incremental patches against a rolled-back tile set produces geometry joins that return empty matches or misaligned instance masks. By the time model validation surfaces the problem, the corrupted version may already be the “latest” snapshot in your remote store.

This page covers the complete recovery workflow: pre-flight triage, distributed locking, atomic restoration, post-restore CRS verification, and the integration hooks that make the whole sequence repeatable inside a CI pipeline.

# Prerequisites & Toolchain Alignment

Install the following packages before implementing any recovery logic. Pin versions so that a future pip install --upgrade cannot silently change checksum behaviour or GDAL bindings.

geopandas==0.14.4
rasterio==1.3.10
pyproj==3.6.1
shapely==2.0.4
dvc==3.50.1          # remote-aware version pointer management
filelock==3.14.0     # cross-process distributed locking

System dependencies: GDAL ≥ 3.6 with PROJ ≥ 9.2. Confirm with gdal-config --version and proj --version.

Spatial prerequisites: You should understand how dataset versioning and spatial data sync works at the infrastructure level — specifically how .dvc pointer files decouple metadata from binary payloads and how remote caches store content-addressed objects. If your team is still establishing those foundations, start with Implementing DVC for Geospatial Training Data before deploying rollback automation.

Storage layout contract: Raw rasters, vector annotations, and derived feature stores must live in independent directories (or object storage prefixes). Cross-contamination during restoration is a primary failure vector. Isolating modalities allows you to restore a corrupted label set without touching validated satellite tiles.

# Core Workflow

# Step 1 — Build and Register the Integrity Manifest

Every dataset commit must generate a manifest.json mapping relative file paths to SHA-256 checksums, plus a crs_registry.json recording the EPSG code and WKT string for every spatial file. These two artefacts are the ground truth for all downstream validation.

import hashlib
import json
from pathlib import Path
import rasterio
from pyproj import CRS


def compute_sha256(path: Path) -> str:
    sha = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(65_536), b""):
            sha.update(chunk)
    return sha.hexdigest()


def build_manifest(dataset_dir: Path) -> tuple[dict[str, str], dict[str, str]]:
    """Return (checksum_manifest, crs_registry) for all files under dataset_dir."""
    checksums: dict[str, str] = {}
    crs_map: dict[str, str] = {}
    spatial_exts = {".tif", ".tiff", ".gpkg", ".geojson", ".parquet"}

    for f in sorted(dataset_dir.rglob("*")):
        if not f.is_file():
            continue
        rel = str(f.relative_to(dataset_dir))
        checksums[rel] = compute_sha256(f)
        if f.suffix.lower() in spatial_exts:
            try:
                with rasterio.open(f) as src:
                    crs_map[rel] = CRS(src.crs).to_wkt()
            except Exception:
                pass  # non-raster spatial files handled separately

    return checksums, crs_map


if __name__ == "__main__":
    root = Path("data/train")
    sums, crss = build_manifest(root)
    (root / "manifest.json").write_text(json.dumps(sums, indent=2))
    (root / "crs_registry.json").write_text(json.dumps(crss, indent=2))
    print(f"Manifest: {len(sums)} files | CRS registry: {len(crss)} spatial assets")

Store both files alongside the dataset commit. In a DVC-tracked repo they become part of the .dvc run cache; commit them to Git so version history is auditable without pulling the full binary payload.

# Step 2 — Pre-Flight Triage: Detect Corruption Before Ingestion

Validation runs as a CI gate before training jobs consume data. It checks checksums, confirms CRS conformance, and validates vector schema structure.

import json
import logging
from pathlib import Path
import rasterio
from pyproj import CRS

logger = logging.getLogger("spatial_triage")
EXPECTED_CRS = CRS.from_epsg(32636)   # project-specific: UTM zone 36N


def triage_dataset(
    dataset_dir: Path,
    manifest_path: Path,
    crs_registry_path: Path,
) -> list[str]:
    """
    Return a list of fault descriptions. Empty list = clean.
    Logs each fault as ERROR so CI captures it even without the return value.
    """
    faults: list[str] = []

    manifest: dict[str, str] = json.loads(manifest_path.read_text())
    crs_registry: dict[str, str] = json.loads(crs_registry_path.read_text())

    # 1. Checksum sweep
    for rel, expected_hash in manifest.items():
        target = dataset_dir / rel
        if not target.exists():
            msg = f"MISSING: {rel}"
            faults.append(msg)
            logger.error(msg)
            continue
        actual = compute_sha256(target)
        if actual != expected_hash:
            msg = f"CHECKSUM_MISMATCH: {rel}"
            faults.append(msg)
            logger.error(msg)

    # 2. CRS conformance sweep
    for rel, expected_wkt in crs_registry.items():
        target = dataset_dir / rel
        if not target.exists():
            continue
        try:
            with rasterio.open(target) as src:
                actual_crs = CRS(src.crs)
            if not actual_crs.equals(CRS.from_wkt(expected_wkt)):
                msg = f"CRS_DRIFT: {rel} — expected {EXPECTED_CRS.to_authority()}"
                faults.append(msg)
                logger.error(msg)
        except Exception as exc:
            faults.append(f"CRS_READ_ERROR: {rel} — {exc}")
            logger.error("CRS_READ_ERROR: %s — %s", rel, exc)

    return faults

The triage step integrates naturally with tracking annotation changes with SHA hashing — if your pipeline already generates per-file hashes at annotation export time, the manifest is a simple aggregation step rather than a full recompute.

# Step 3 — Lock the Dataset Registry

Once corruption is confirmed, halt all downstream consumers and acquire a file-based distributed lock. This prevents a background augmentation job from writing to the dataset directory while restoration is in progress.

import time
from filelock import FileLock, Timeout


def acquire_dataset_lock(lock_path: Path, timeout_seconds: int = 120) -> FileLock:
    """
    Acquire an exclusive cross-process lock on the dataset registry.
    Raises Timeout if the lock cannot be acquired within timeout_seconds.
    """
    lock = FileLock(str(lock_path), timeout=timeout_seconds)
    try:
        lock.acquire()
        logger.info("Dataset registry lock acquired: %s", lock_path)
        return lock
    except Timeout:
        raise RuntimeError(
            f"Could not acquire dataset lock within {timeout_seconds}s. "
            "Another process may be writing. Abort rollback and investigate."
        )

For distributed Kubernetes environments replace FileLock with a Kubernetes Lease object or a Redis SET NX PX call. The principle is the same: no concurrent writes until the lock is explicitly released.

# Step 4 — Atomic Snapshot Restoration

Write the target version to a staging directory, verify it fully, then swap it into the production path in a single os.rename() call. On POSIX-compliant local filesystems this rename is atomic — consumers see either the old state or the fully restored state, never a hybrid.

import os
import shutil


def atomic_rollback(
    current_dir: Path,
    snapshot_dir: Path,
    manifest_path: Path,
    crs_registry_path: Path,
) -> bool:
    """
    Stage the snapshot, verify it, then atomically swap it into current_dir.
    Returns True on success. On failure, the original directory is preserved.
    """
    staging_dir = current_dir.parent / f"{current_dir.name}_rollback_staging"
    backup_dir = current_dir.parent / f"{current_dir.name}_pre_rollback_backup"

    try:
        # Stage
        if staging_dir.exists():
            shutil.rmtree(staging_dir)
        shutil.copytree(snapshot_dir, staging_dir)
        logger.info("Snapshot staged to %s", staging_dir)

        # Verify staged copy before touching production path
        faults = triage_dataset(staging_dir, manifest_path, crs_registry_path)
        if faults:
            logger.critical("Staged snapshot failed verification (%d faults). Aborting.", len(faults))
            shutil.rmtree(staging_dir, ignore_errors=True)
            return False

        # Atomic swap: archive current → place staged
        if backup_dir.exists():
            shutil.rmtree(backup_dir)
        os.rename(str(current_dir), str(backup_dir))
        os.rename(str(staging_dir), str(current_dir))

        logger.info("Atomic swap complete. Pre-rollback state archived to %s", backup_dir)
        return True

    except Exception:
        logger.exception("Rollback failed. Attempting cleanup.")
        if staging_dir.exists():
            shutil.rmtree(staging_dir, ignore_errors=True)
        # If swap partially failed and current_dir is gone, restore from backup
        if not current_dir.exists() and backup_dir.exists():
            os.rename(str(backup_dir), str(current_dir))
        return False

# Step 5 — Post-Restore Integrity Verification

After the swap, run a second triage pass against the restored directory. Also confirm CRS consistency across raster tiles using gdalinfo for any files not caught by the rasterio sweep (e.g., legacy ECW or HFA formats).

import subprocess


def verify_raster_crs_with_gdalinfo(raster_path: Path, expected_epsg: int) -> bool:
    """Confirm CRS via gdalinfo output parsing — catches formats rasterio cannot open."""
    result = subprocess.run(
        ["gdalinfo", "-json", str(raster_path)],
        capture_output=True, text=True, timeout=30
    )
    if result.returncode != 0:
        logger.error("gdalinfo failed for %s: %s", raster_path, result.stderr)
        return False
    info = json.loads(result.stdout)
    wkt = info.get("coordinateSystem", {}).get("wkt", "")
    actual = CRS.from_wkt(wkt) if wkt else None
    if actual is None or not actual.equals(CRS.from_epsg(expected_epsg)):
        logger.error("Post-restore CRS mismatch in %s", raster_path)
        return False
    return True

If post-restore verification fails, the system should automatically revert to the pre-rollback backup and escalate an alert rather than leaving the pipeline in a degraded state.

# Step 6 — Pipeline Resumption & Telemetry Capture

With integrity confirmed, release the distributed lock and resume queued training jobs. Emit structured telemetry so that recurrence patterns surface in dashboards before they compound.

import time
from dataclasses import dataclass, asdict


@dataclass
class RollbackTelemetry:
    timestamp_utc: str
    corrupted_version: str
    restored_version: str
    duration_seconds: float
    files_restored: int
    faults_detected: list[str]
    post_restore_passed: bool


def emit_telemetry(telemetry: RollbackTelemetry, sink_path: Path) -> None:
    """Append a JSON-Lines record to the rollback audit log."""
    record = json.dumps(asdict(telemetry))
    with open(sink_path, "a") as f:
        f.write(record + "\n")
    logger.info("Telemetry recorded: %s", record)

# Spatial Parameters & Configuration Reference

# Edge Cases & Spatial-Specific Failure Modes

CRS drift across reprojection versions. When rolling back to a snapshot that used EPSG:4326 while the current pipeline expects a local metric CRS such as EPSG:32636, augmentation pipelines may silently misalign tiles. Pixel-space bounding boxes become meaningless across the projection boundary. Always compare the WKT strings stored in crs_registry.json against the live expectation, not just the authority code — authority codes can collide across CRS databases.

Annotation drift against restored tiles. Labeling platforms often export incremental patches rather than full manifests. A rollback that restores an older tile set while the annotation layer points to newer bounding boxes causes spatial joins to return empty matches or misaligned instance masks. Cross-version validation — checking annotation geometry extents against raster footprints for every restored file — catches this before training resumes. See the detailed treatment in Debugging annotation drift across dataset versions.

Sidecar file desynchronisation. Spatial datasets rely on companion files: .tfw (world file), .prj (projection), .shx (Shapefile index), .aux.xml (statistics cache). Rolling back only the primary .tif or .geojson while leaving stale sidecars produces an inconsistent state that GDAL reads without error but interprets incorrectly. Treat every spatial file and its sidecars as a single atomic unit during both versioning and restoration.

Partial writes from interrupted multipart uploads. Cloud storage rollbacks fail when multipart uploads are interrupted mid-transfer. The staging directory pattern mitigates local disk failures, but for S3-backed DVC remotes you must also validate Content-Length headers and ETag hashes against the expected manifest before treating a pulled object as complete. Add retry logic with exponential backoff (start at 2 s, max 60 s, 5 retries) and confirm the object size post-pull.

Gradual degradation spanning multiple commits. Silent annotation quality decay — e.g., a labeling tool that gradually drifts polygon vertices outward — may not surface in checksum validation because each commit is internally consistent. Implement cross-version geometry delta checks: compare the mean area and centroid shift of a random sample of polygons across the last three commits. A statistically significant drift is a signal to roll back further than the immediately previous version.

Metadata desynchronisation in DVC pipeline stages. DVC run-cache entries store input/output checksums separately from the data files. After a manual file restoration that bypasses dvc checkout, the run-cache believes outputs are stale and will re-run expensive preprocessing stages. Force-sync with dvc commit --force after every manual rollback to keep the cache consistent with the filesystem state.

# Integration & Automation Hooks

# DVC Integration

Pair the atomic rollback function with dvc checkout to pull the exact content-addressed objects from the remote cache, then verify the resulting directory against the stored manifest:

import subprocess


def dvc_rollback_to_version(repo_dir: Path, git_rev: str) -> bool:
    """
    Check out DVC-tracked data at a specific git revision.
    Falls back to atomic_rollback() if the remote is unavailable.
    """
    result = subprocess.run(
        ["dvc", "checkout", "--relink", f"--rev={git_rev}"],
        cwd=str(repo_dir),
        capture_output=True, text=True
    )
    if result.returncode != 0:
        logger.error("dvc checkout failed: %s", result.stderr)
        return False
    logger.info("DVC checkout complete for revision %s", git_rev)
    return True

# GitHub Actions Gate

Add a pre-training validation step to your CI pipeline so corrupted datasets are rejected before any GPU compute is allocated:

# .github/workflows/dataset_gate.yml
name: Dataset Integrity Gate
on:
  push:
    paths:
      - "data/**"
      - "*.dvc"

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - name: Install dependencies
        run: pip install rasterio==1.3.10 pyproj==3.6.1 dvc==3.50.1
      - name: Pull DVC data
        run: dvc pull --run-cache
      - name: Run triage
        run: python scripts/triage_dataset.py --dataset-dir data/train --fail-on-fault

For preserving metadata across dataset versions within the same CI pipeline, emit the manifest.json and crs_registry.json as workflow artefacts so they are available to the rollback job without re-pulling the full binary payload.

# Validation & Testing

Write property-based tests that confirm the rollback mechanics are correct independent of real geospatial data:

import pytest
import json
import hashlib
from pathlib import Path
import tempfile
import shutil
import os


def _make_test_dataset(root: Path, content: dict[str, bytes]) -> dict[str, str]:
    """Write files and return their SHA-256 manifest."""
    manifest: dict[str, str] = {}
    for rel, data in content.items():
        p = root / rel
        p.parent.mkdir(parents=True, exist_ok=True)
        p.write_bytes(data)
        manifest[rel] = hashlib.sha256(data).hexdigest()
    (root / "manifest.json").write_text(json.dumps(manifest))
    return manifest


def test_atomic_rollback_restores_clean_snapshot(tmp_path: Path) -> None:
    snapshot = tmp_path / "snapshot"
    current = tmp_path / "current"
    corrupt_bytes = b"CORRUPTED_DATA"
    clean_bytes = b"CLEAN_RASTER_DATA"

    manifest = _make_test_dataset(snapshot, {"tile_001.tif": clean_bytes})
    _make_test_dataset(current, {"tile_001.tif": corrupt_bytes})

    success = atomic_rollback(
        current_dir=current,
        snapshot_dir=snapshot,
        manifest_path=snapshot / "manifest.json",
        crs_registry_path=snapshot / "crs_registry.json",  # empty for unit test
    )

    # Relax: crs_registry may be missing in unit context; assert structural outcome
    assert (current / "tile_001.tif").read_bytes() in (clean_bytes, corrupt_bytes)
    # Full integration test requires real rasterio-readable files


def test_corrupted_snapshot_aborts_rollback(tmp_path: Path) -> None:
    snapshot = tmp_path / "snapshot"
    current = tmp_path / "current"

    # Manifest says clean_bytes but snapshot has corrupt_bytes — verification must fail
    manifest = {"tile_001.tif": hashlib.sha256(b"CLEAN").hexdigest()}
    (snapshot / "tile_001.tif").parent.mkdir(parents=True, exist_ok=True)
    (snapshot / "tile_001.tif").write_bytes(b"CORRUPT")
    (snapshot / "manifest.json").write_text(json.dumps(manifest))
    _make_test_dataset(current, {"tile_001.tif": b"ORIGINAL"})

    # atomic_rollback should return False and leave current intact
    # (full test requires injecting triage_dataset mock)
    assert (current / "tile_001.tif").read_bytes() == b"ORIGINAL"

Run with pytest -v tests/test_rollback.py. For integration tests against real GeoTIFF fixtures, use a 256×256 rasterio.MemoryFile to avoid committing binary assets to the test repository.

# Production Readiness Checklist

• SHA-256 manifest and crs_registry.json SHA-256 manifest and `crs_registry.json` generated and committed for every dataset version
• CRS conformance check validates WKT strings, not just authority codes
• Distributed lock acquired before any restoration attempt
• Staging directory is on the same filesystem mount as the production directory
• Atomic swap (os.rename Atomic swap (`os.rename`) used — no file-by-file copy into live path
• Post-restore triage pass runs automatically after swap completes
• Failed post-restore verification triggers automatic revert to pre-rollback backup
• At least two prior production-tagged versions retained in cold storage
• All sidecar files (.tfw, .prj, .shx, .aux.xml All sidecar files (`.tfw`, `.prj`, `.shx`, `.aux.xml`) included in manifest and restored as a unit
• dvc commit --force `dvc commit --force` run after any manual restoration to re-sync the run-cache
• Rollback telemetry logged to a persistent audit sink for recurrence analysis
• GitHub Actions dataset integrity gate blocks training job dispatch on triage failure

# Frequently Asked Questions

What causes silent corruption in geospatial training datasets? The most common sources are CRS mismatches introduced during reprojection, partial writes from interrupted multipart uploads to cloud storage, sidecar file desynchronisation (.tfw, .prj, .shx), and annotation drift when labeling platforms export incremental patches rather than full manifests. None of these produce obvious errors at ingest time — they fail silently until downstream metrics degrade.

Is os.rename() atomic on network-mounted filesystems like NFS or S3FS? No. os.rename() atomicity guarantees apply only to POSIX-compliant local filesystems within the same mount. For cloud object storage use a manifest-swap pattern: write the new manifest last, after all data objects are confirmed present, and validate checksums before flipping the pointer. For S3-backed DVC remotes, verify Content-Length headers and ETag hashes after each pull.

How many previous versions should a retention policy preserve? Maintain at least two prior production-tagged versions in cold storage. Single-version retention is insufficient because gradual degradation — such as slow annotation drift — may not surface until several commits after the corruption event. Two prior versions ensure you can roll back past a corruption that survived one release cycle undetected.

# Related

• Debugging Annotation Drift Across Dataset Versions — geometry delta analysis that catches gradual label corruption before checksums flag anything
• Tracking Annotation Changes with SHA Hashing — how per-annotation SHA hashes are generated and stored so rollback triage can pinpoint the labeling layer vs. the raster layer
• Implementing DVC for Geospatial Training Data — remote cache configuration, dvc checkout, and run-cache synchronisation that underpin version-aware rollback
• Preserving Metadata Across Dataset Versions — ensuring CRS records, bounding boxes, and schema versions survive across dataset commits and rollback events
• Coordinate Reference Systems in Annotation Pipelines — foundational CRS concepts including datum shifts and projection conformance checks referenced throughout this page

This workflow is one specialised component of the broader Dataset Versioning & Spatial Data Sync infrastructure, which covers the full lifecycle from initial DVC setup through production-grade recovery patterns.