Recovering from a Corrupted COG Export

A Cloud-Optimized GeoTIFF (COG) that fails midway through an export leaves a file that may still open, still preview, and still pass a casual glance — yet break the moment a training loader tries to range-read a tile or a reprojection step reads its geotransform. Recovery follows one disciplined path: diagnose the exact failure with gdalinfo and rio cogeo validate, decide whether the primary pixel data survived, and then either rebuild a valid COG around the intact data or roll back to the last known-good copy tracked in DVC versioning. The one decision that governs everything is whether the corruption is structural — overviews, layout, or georeferencing metadata — or whether the base raster band itself is gone. Structural damage is repairable in place; lost primary data is not, and the only correct response is a versioned restore.

# Why a Silently Corrupted COG Poisons Downstream Training

A COG packs the full-resolution image plus a pyramid of downsampled overviews into a single tiled TIFF, with Image File Directories (IFDs) arranged so that headers precede pixel data. That layout is what lets a remote reader fetch a 512-pixel tile over HTTP range requests without downloading gigabytes. When an export is interrupted — a killed process, a full disk, a dropped network mount, a partial multipart upload — the IFD chain or the tile offsets it references can be truncated. The damage is often invisible: the file still carries a valid magic number and an openable primary header, so a thumbnail renders and gdalinfo prints dimensions.

The failure surfaces later and far from its cause. An annotation loader that reads the highest overview to draw a slippy-map preview works fine, but the training pipeline that range-reads full-resolution tiles hits a byte offset that points past the end of the file. Or the geotransform came back as the identity affine, so every polygon annotation reprojects to the wrong ground coordinates and the model trains on systematically mislocated labels. Because the corruption is structural rather than a crash, nothing throws until deep inside a batch. Catching it at the point of export — and knowing immediately whether to repair or roll back — is what keeps a bad file out of a training run.

The cost of a wrong diagnosis runs in both directions. Treat a data-loss failure as structural and you will happily rewrite a valid-looking COG around zeroed or missing pixels, then ship it downstream where it silently degrades every model that touches it. Treat a merely structural failure as data loss and you roll back further than necessary, discarding a perfectly recoverable export and forcing an expensive re-acquisition or re-processing run. The diagnostic steps below exist to remove that ambiguity: they force the primary band to be read, so the repair-versus-restore decision is driven by evidence rather than by how the file happens to open in a viewer.

# Diagnosing the Corruption with gdalinfo and rio cogeo validate

Install the toolchain with pinned versions so diagnosis is reproducible across machines:

pip install rasterio==1.3.10 rio-cogeo==5.3.0 numpy==1.26.4

Start with gdalinfo. It reports the driver, raster dimensions, the geotransform (the Origin and Pixel Size lines), the CRS, and the overview levels attached to each band. Add -stats to force a read of the primary band — this is the single most useful signal, because a truncated primary IFD or missing tile data throws an I/O error here rather than staying hidden:

gdalinfo -stats suspect_export.tif

Read the output for three things. A geotransform reported as Origin = (0.0, 0.0) with Pixel Size = (1.0, -1.0) is the GDAL default — the real georeferencing was never written. An Overviews: line that is absent from a file that should carry a pyramid means the overview IFDs were stripped or never flushed. An ERROR 1: ... failed to read during -stats means the primary band data itself is unreadable, which is the one symptom you cannot repair in place.

Then run the strict layout check. gdalinfo opens any readable TIFF, but rio cogeo validate enforces the Cloud-Optimized contract specifically — internal tiling, an overview pyramid, and IFD ordering:

rio cogeo validate suspect_export.tif

A response of is_valid: False with a message such as The file is not tiled or The offset of the IFD for overview ... is not greater than the previous tells you the pixels may be fine but the layout is broken, which cog_translate can rewrite. A hard read error during validation points back to lost primary data.

# Choosing Between In-Place Repair and Versioned Rollback

The diagnosis funnels into a single branch. If the primary band reads cleanly and only the structure is wrong — no overviews, a bad geotransform, or an out-of-order layout — rebuild the COG in place and keep the exact original pixels. If the base band throws read errors or the primary IFD is truncated, the imagery is lost and no rewrite can recreate it; restore the last valid copy from version control instead.

# Rebuilding a Valid COG When the Pixels Survive

When the base band reads, the fix is to reassign a correct geotransform if one is missing and rewrite the file through rio-cogeo, which regenerates the overview pyramid and orders the IFDs to spec. The first mention of a georeferencing frame here is worth grounding: the geotransform and its coordinate reference system together map pixel indices to ground coordinates, and a COG that lost either is spatially meaningless even when its pixels are intact.

The helper below detects an invalid (identity or degenerate) geotransform, substitutes a recovered Affine and CRS when supplied, and rewrites a compliant COG:

from pathlib import Path

import rasterio
from rasterio.crs import CRS
from rasterio.transform import Affine
from rio_cogeo.cogeo import cog_translate
from rio_cogeo.profiles import cog_profiles


def geotransform_is_invalid(transform: Affine) -> bool:
    """A default identity transform or zero-scale pixel is unusable for georeferencing."""
    is_identity: bool = transform == Affine.identity()
    has_zero_scale: bool = transform.a == 0.0 or transform.e == 0.0
    return is_identity or has_zero_scale


def rebuild_cog(
    src_path: Path,
    dst_path: Path,
    recovered_transform: Affine | None = None,
    recovered_crs: CRS | None = None,
) -> None:
    """Rewrite a structurally broken COG, restoring georeferencing when supplied."""
    with rasterio.open(src_path) as src:
        profile = src.profile.copy()
        data = src.read()  # raises if the primary band data is truncated

        if geotransform_is_invalid(src.transform):
            if recovered_transform is None or recovered_crs is None:
                raise ValueError(
                    "Geotransform is invalid and no recovery values were provided; "
                    "roll back to a versioned copy instead."
                )
            profile.update(transform=recovered_transform, crs=recovered_crs)

    # Write a temporary plain GeoTIFF with corrected metadata, then translate to COG.
    tmp_path = dst_path.with_suffix(".tmp.tif")
    with rasterio.open(tmp_path, "w", **profile) as tmp:
        tmp.write(data)

    cog_translate(
        tmp_path,
        dst_path,
        cog_profiles.get("deflate"),
        overview_resampling="average",
        in_memory=False,
        quiet=True,
    )
    tmp_path.unlink(missing_ok=True)

The src.read() call is deliberate: if the primary IFD is truncated, it raises here and the rebuild aborts before producing a valid-looking container around missing data. When it succeeds, cog_translate rebuilds the overview pyramid with average resampling and writes IFDs in the order rio cogeo validate expects. If the export’s origin coordinates use a specific [EPSG:32633](https://www.geospatialannotation.com/geospatial-annotation-fundamentals-architecture/coordinate-reference-systems-in-annotation-pipelines/) UTM frame, pass that as recovered_crs so the rewritten file carries the correct projection rather than an unset one.

Confirm the rebuild before releasing it back to the pipeline:

rio cogeo validate rebuilt_export.tif

# Rolling Back to the Last Valid Version

When the primary band cannot be read, no rewrite helps — the pixels are gone. Restore the last known-good asset from version control. If the corrupted file is DVC-tracked, check out the version from the commit before the bad export landed:

# Inspect the history of the tracked COG pointer.
git log --oneline -- data/tiles/scene_0421.tif.dvc

# Restore the pointer to the last good commit, then materialize the data.
git checkout <good_commit_sha> -- data/tiles/scene_0421.tif.dvc
dvc checkout data/tiles/scene_0421.tif.dvc

git checkout rewinds the small .dvc pointer file to the known-good content hash, and dvc checkout pulls the matching COG from the local cache or remote into the workspace. To fetch a specific historical version straight from the remote without touching the working tree — useful when recovering into a clean staging area — use dvc get against the good revision:

dvc get . data/tiles/scene_0421.tif \
  --rev <good_commit_sha> \
  --out recovered/scene_0421.tif

Always re-validate the restored file with rio cogeo validate before returning it to service, then record a fresh content hash so downstream stages can detect future drift — computing stable content hashes for COGs covers hashing that ignores volatile metadata so an identical raster always hashes identically across re-exports.

# Symptom-to-Fix Reference

# Common Errors and Fixes

rasterio.errors.RasterioIOError: ... not recognized as a supported file format Root cause: the export was truncated before the TIFF header magic bytes were written, so the file is not a valid TIFF at all. Fix: the file is unrecoverable in place; roll back to the last DVC-tracked version.

rio cogeo validate prints is_valid: True but tiles still fail remotely Root cause: the local copy is valid but a partial multipart upload left a truncated object on the remote. Fix: re-upload from the validated local file, or dvc checkout to force the cache-verified copy back to the remote.

cog_translate produces a valid COG but every pixel is nodata Root cause: src.read() returned an array of fill values because the base band data was zeroed during a failed write, and the truncation did not raise. Fix: compare against the tracked content hash; if it differs, discard the rebuild and roll back rather than shipping a valid container of empty pixels.

ValueError: Geotransform is invalid and no recovery values were provided Root cause: the georeferencing metadata is gone and no sidecar, STAC item, or sibling tile supplies the correct affine transform. Fix: recover the transform from an intact adjacent tile in the same acquisition, or roll back to the versioned copy that still carries it.

# Related

• Rollback Strategies for Corrupted Spatial Datasets — the broader topic area covering how to detect, isolate, and reverse corruption across a versioned spatial dataset
• Computing Stable Content Hashes for COGs — hash imagery in a way that ignores volatile metadata so silent corruption and drift are detectable
• Implementing DVC for Geospatial Training Data — set up the version control that makes a clean rollback to a known-good COG possible
• Debugging Annotation Drift Across Dataset Versions — a sibling guide on tracing when and how labels diverge between versioned snapshots

This guide is part of the broader Rollback Strategies for Corrupted Spatial Datasets, itself one topic within Dataset Versioning & Spatial Data Sync.