Computing Stable Content Hashes for COGs

A raw file SHA of a Cloud-Optimized GeoTIFF changes on almost every re-export, even when not a single pixel moved. GeoTIFF headers carry volatile bytes — a creation timestamp, a GDAL version tag, TIFF tag ordering, and shifting IFD offsets — so re-running the identical export on a newer library or a different machine rewrites the header and flips the digest. If your versioning layer keys on that file-level SHA, it will report phantom changes on every rebuild and mask real ones. The fix is to hash the decoded pixel array plus a canonical slice of geo-metadata (the affine transform and the CRS), producing a stable content id that stays identical across re-exports of the same imagery.

# Why a File-Level SHA Reports Phantom Changes

The SHA-256 of a byte stream is only as stable as the bytes. A COG is a container: compressed pixel tiles, overview pyramids, and a metadata header (the image file directory, or IFD). The pixel tiles are deterministic for a given codec, but the header is not. gdal_translate stamps a TIFFTAG_SOFTWARE string that names the GDAL version; tag write order can differ between builds; and the byte offsets that point to each tile shift when overview blocks are laid out differently. None of that alters the imagery a model trains on, yet all of it alters the file hash.

This matters most in a content-addressed workflow. When you track datasets with DVC versioning, the tool records each file’s MD5 or SHA to decide what changed and what to re-upload. If a nightly pipeline re-exports rasters from a source archive, a file-level hash flags every raster as modified, triggering needless re-uploads across cloud remotes and burying the one tile that genuinely changed. A content hash that ignores header noise lets the versioning layer see the truth: same pixels, same georeferencing, same identity.

The diagram below contrasts the two hashing strategies on two re-exports of one scene.

# Step-by-Step Implementation

Install the pinned toolchain once. rasterio bundles its own GDAL, so the decoded pixel values are consistent regardless of any system GDAL:

pip install rasterio==1.3.10 numpy==1.26.4

# Step 1 — Read Decoded Pixels Band by Band

Open the COG with rasterio and read one band at a time. Reading with rasterio decodes the compression and returns the stored sample values as a NumPy array, so the codec, tiling scheme, and overview layout drop out of the hash entirely. Iterating band by band keeps memory bounded for large multi-band scenes:

import hashlib
import rasterio
from rasterio.io import DatasetReader


def _hash_pixels(dataset: DatasetReader, digest: "hashlib._Hash") -> None:
    """Feed each band's decoded bytes into the running digest, in index order."""
    for band_index in range(1, dataset.count + 1):
        band = dataset.read(band_index)          # 2D ndarray, dtype preserved
        digest.update(band.tobytes(order="C"))   # exact stored IEEE/int bytes

band.tobytes(order="C") serializes the array in a fixed row-major order so the byte stream never depends on NumPy’s internal memory layout. Because read() returns the values exactly as stored, float32 and int16 rasters are bit-reproducible across machines.

# Step 2 — Hash Pixel Buffers Incrementally

Use a single hashlib.sha256 object and update() it per band rather than concatenating every band into one buffer. This bounds peak memory to a single band and makes the order of bands explicit — a property that becomes part of the content identity:

def _new_digest() -> "hashlib._Hash":
    """A fresh SHA-256 accumulator, domain-separated with a version tag."""
    digest = hashlib.sha256()
    digest.update(b"cog-content-hash-v1\x00")   # guards against format drift
    return digest

The leading version tag (cog-content-hash-v1) domain-separates this scheme, so if you ever change what goes into the hash you can bump the tag and avoid silent collisions with previously computed ids.

# Step 3 — Append Canonicalized Transform and EPSG

Pixels alone are not the whole identity: two rasters can share pixel values while sitting at different map locations. Fold the georeferencing in, but canonicalize it first so formatting noise cannot leak in. Serialize the affine transform to six fixed-precision floats and reduce the CRS to its authority code. The first raster in your pipeline is typically stored in a projected CRS such as EPSG:32633; reducing it to that integer code makes the hash independent of the exact WKT string GDAL happened to emit:

from rasterio.transform import Affine
from rasterio.crs import CRS


def _canonical_geo(transform: Affine, crs: CRS | None) -> bytes:
    """Format the geotransform and CRS into a stable, whitespace-free byte string."""
    coeffs = (transform.a, transform.b, transform.c,
              transform.d, transform.e, transform.f)
    transform_str = ",".join(f"{c:.10g}" for c in coeffs)
    epsg = crs.to_epsg() if crs is not None else None
    crs_str = f"EPSG:{epsg}" if epsg is not None else (crs.to_wkt() if crs else "NONE")
    return f"|transform={transform_str}|crs={crs_str}".encode("utf-8")

Using {c:.10g} fixes the printed precision so 10.0 and 10.000000001 never both appear from equivalent transforms, and falling back to WKT only when a raster lacks an EPSG code keeps unusual custom projections hashable.

# Step 4 — Produce the Hex Digest

Assemble the pieces into one entry point. It opens the file, hashes pixels, appends the metadata block (including band count, dtype, and declared nodata so those are part of the identity too), and returns the hex string:

def content_hash_cog(path: str) -> str:
    """Return a re-export-stable SHA-256 hex digest for a COG's content."""
    digest = _new_digest()
    with rasterio.open(path) as dataset:
        _hash_pixels(dataset, digest)
        meta = (
            f"|bands={dataset.count}"
            f"|dtype={dataset.dtypes[0]}"
            f"|nodata={dataset.nodata}"
        ).encode("utf-8")
        digest.update(meta)
        digest.update(_canonical_geo(dataset.transform, dataset.crs))
    return digest.hexdigest()

# Step 5 — Verify Two Re-exports Match

Prove the property that motivated the whole exercise: re-export the same source imagery and confirm the content hash is identical even though the file SHA differs. This assertion belongs in your test suite so a future GDAL upgrade cannot silently break reproducibility:

import hashlib
from pathlib import Path


def raw_file_sha(path: str) -> str:
    """Plain SHA-256 of the on-disk bytes — the unstable baseline."""
    h = hashlib.sha256()
    h.update(Path(path).read_bytes())
    return h.hexdigest()


def assert_reexport_stable(export_a: str, export_b: str) -> None:
    """export_a and export_b are two COG exports of identical imagery."""
    assert content_hash_cog(export_a) == content_hash_cog(export_b), "content drifted"
    assert raw_file_sha(export_a) != raw_file_sha(export_b), "headers identical?"
    print("content hash stable:", content_hash_cog(export_a))

The content hash now serves as a durable id in a manifest — the same manifest that drives content-addressed sync across cloud remotes, where re-uploading a raster only because its header changed is pure waste.

# What Goes Into the Hash — and What Must Not

Deciding which inputs belong in the digest is the whole design. The table below records the ruling for each candidate input and the reason.

# Common Errors and Fixes

Content hash differs after masking nodata Root cause: the code replaced nodata fill with np.nan or a sentinel before hashing, so the result depends on the masking convention rather than the stored bytes. Fix: hash dataset.read(i).tobytes() on the raw array and fold the declared nodata value in as a separate metadata field, exactly as content_hash_cog does. Never mutate the array before tobytes().

Float rasters hash differently on another machine Root cause: a scale, offset, astype() cast, or resample was applied before hashing, and those operations round differently across BLAS or GDAL builds. Fix: hash the array precisely as read() returns it. read() yields the stored IEEE-754 bytes with no recomputation, so tobytes() is bit-identical everywhere. Move any scaling downstream of the hash.

RGB and BGR versions of one scene collide or diverge unexpectedly Root cause: bands were read in an unstable order — for example by iterating a Python set of indices or by relying on a color-interpretation lookup that varies between files. Fix: always iterate range(1, dataset.count + 1) in ascending order and record bands= in the metadata block. Reordered channels then hash to distinct, reproducible ids.

content_hash_cog raises on a raster with no CRS Root cause: dataset.crs is None for an ungeoreferenced TIFF and to_epsg() cannot be called on it. Fix: the _canonical_geo guard handles this by emitting crs=NONE. If you require georeferencing, validate dataset.crs is not None before hashing and reject the file, which also catches the truncated headers described in recovering from a corrupted COG export.

# Related

• Tracking Annotation Changes with SHA Hashing — the topic area this guide belongs to, covering content-addressed change detection across a versioned annotation pipeline
• Recovering from a Corrupted COG Export — diagnose truncated IFDs and broken overviews that a content hash will also expose as unreadable
• Syncing Annotations Across Cloud Remotes — use stable content ids so a re-export with a fresh header never triggers a needless re-upload
• Implementing DVC for Geospatial Training Data — the versioning layer that keys on file hashes and benefits most from a stable content id
• Coordinate Reference Systems in Annotation Pipelines — why the EPSG authority code, not raw WKT, is the canonical CRS identity to fold into a hash

This guide is one detailed technique within Tracking Annotation Changes with SHA Hashing, which is itself part of Dataset Versioning & Spatial Data Sync.