How to Version Control Large Satellite Imagery Datasets

To version control large satellite imagery datasets, decouple binary rasters from your Git repository. Store code, configuration, and lightweight annotation exports in Git, and use Data Version Control (DVC) to track multi-gigabyte raster files via cryptographic pointers. Convert raw scenes to Cloud-Optimized GeoTIFF (COG) or Zarr format before tracking so that remote storage supports HTTP range requests and chunked access. This keeps the repository lean, preserves full dataset lineage, and enables any team member or CI runner to reproduce a training snapshot exactly.

# Why Standard Git Breaks on Satellite Imagery

Satellite scenes routinely exceed hundreds of gigabytes per acquisition. Git stores a full binary copy of every version of every committed file. Committing raw .tif or .jp2 files causes repository size to compound linearly with dataset evolution, exhausts local disk space on developer machines, and makes CI/CD runners fail with out-of-memory errors during clone.

Git LFS partially mitigates file size but introduces different problems for spatial workloads: it lacks native support for coordinate reference system metadata as a versioned entity, has no concept of chunked raster access, and can generate significant egress costs when pulling historical commits. DVC solves this by committing only a .dvc pointer file — a small YAML containing the file’s SHA-256 hash and storage path — while the binary lives in a scalable remote backend.

# Why Imagery Scale Makes This a Pipeline Bottleneck

At the scale typical in ML workflows — multi-temporal stacks for change detection, multi-sensor fusion campaigns, or high-resolution urban mapping — the naive approach of committing rasters to Git collapses in three predictable ways. First, repository clone time grows proportionally to total historical binary size, breaking CI/CD environment setup. Second, reproducing a past experiment requires reconstructing every dataset version from scratch because there is no content-addressable cache. Third, team members on bandwidth-constrained connections (field offices, overseas contractors) cannot participate in dataset pulls at all.

The combination of DVC pointers in Git and COG-formatted rasters in object storage fixes all three problems: clones stay under a few megabytes regardless of dataset size, any historical version is reproducible by checking out the pointer file and running dvc pull, and remote caches mean that unchanged files are never re-transferred. This same mechanism also integrates cleanly with SHA-based annotation tracking so that raster and vector versions stay in lockstep.

# Step-by-Step Implementation

# Step 1: Convert Raw Imagery to Cloud-Optimized GeoTIFF

Before tracking with DVC, convert each scene to COG format. COG files interleave tile overviews and data internally, so cloud storage can serve HTTP range requests against individual tiles without fetching the full file. The GDAL COG driver handles this:

gdal_translate \
  -of COG \
  -co COMPRESS=DEFLATE \
  -co TILED=YES \
  -co BLOCKXSIZE=512 \
  -co BLOCKYSIZE=512 \
  -co COPY_SRC_OVERVIEWS=YES \
  input_raw.tif \
  output_cog.tif

For time-series stacks or hyperspectral cubes, Zarr is preferable. Zarr stores data as chunked arrays in separate files, so only modified chunks are re-uploaded when a tile changes:

pip install "zarr==2.18.0" "rioxarray==0.15.5"

import rioxarray

ds = rioxarray.open_rasterio("output_cog.tif", chunks={"x": 512, "y": 512})
ds.to_zarr("timeseries.zarr", mode="w")

# Step 2: Initialize DVC in Your Repository

DVC must sit alongside an existing Git repository. Run these commands from the repository root:

pip install "dvc[s3]==3.51.2"   # swap [s3] for [gcs] or [azure] as needed

git init
dvc init

# Commit the DVC configuration files Git needs to track
git add .dvc/config .dvcignore
git commit -m "Initialize DVC"

# Step 3: Configure Remote Storage

Point DVC at a cloud bucket. Use the --local flag to keep credentials out of the shared .dvc/config file:

# Shared config (safe to commit)
dvc remote add -d geospatial-remote s3://your-bucket/dvc-data
dvc remote modify geospatial-remote region us-east-1

# Per-machine credentials (never committed)
dvc remote modify --local geospatial-remote access_key_id YOUR_KEY
dvc remote modify --local geospatial-remote secret_access_key YOUR_SECRET

git add .dvc/config
git commit -m "Add S3 DVC remote"

# Step 4: Track the Imagery Directory

dvc add data/satellite_imagery/

# DVC writes data/satellite_imagery.dvc (pointer) and updates .gitignore
git add data/satellite_imagery.dvc .gitignore
git commit -m "Track satellite imagery v1 with DVC"

The generated .dvc pointer file looks like this:

outs:
- md5: d41d8cd98f00b204e9800998ecf8427e.dir
  size: 4831838208
  nfiles: 47
  path: data/satellite_imagery

# Step 5: Push Binaries and Tag the Release

dvc push                            # transfer rasters to the S3 remote
git tag v1.0-imagery                # immutable snapshot for training run 1
git push origin main --tags

Any teammate or CI runner can now reproduce the exact dataset with:

git checkout v1.0-imagery
dvc pull

# Step 6: Validate COG Structure Before Committing

Run this validation script before calling dvc add. It uses rasterio to assert internal tiling and overview presence — the two properties required for cloud-native chunked reads:

from __future__ import annotations

import subprocess
import sys
from pathlib import Path

import rasterio  # rasterio==1.3.10


def validate_cog(path: Path) -> bool:
    """Return True only if the file is a valid COG with tiling and overviews."""
    try:
        with rasterio.open(path) as src:
            is_tiled: bool = bool(src.profile.get("tiled", False))
            has_overviews: bool = len(src.overviews(1)) > 0
            if not is_tiled:
                print(f"  FAIL {path.name}: not internally tiled", file=sys.stderr)
            if not has_overviews:
                print(f"  FAIL {path.name}: no overviews", file=sys.stderr)
            return is_tiled and has_overviews
    except Exception as exc:
        print(f"  ERROR {path}: {exc}", file=sys.stderr)
        return False


def track_directory(data_dir: Path) -> None:
    """Validate all GeoTIFFs, then add the directory to DVC and commit."""
    if not data_dir.exists():
        raise FileNotFoundError(f"Directory not found: {data_dir}")

    tif_files = list(data_dir.glob("**/*.tif"))
    if not tif_files:
        raise RuntimeError(f"No .tif files found in {data_dir}")

    invalid = [f for f in tif_files if not validate_cog(f)]
    if invalid:
        print(f"{len(invalid)} file(s) failed COG validation — aborting.", file=sys.stderr)
        sys.exit(1)

    print(f"All {len(tif_files)} file(s) passed COG validation.")
    subprocess.run(["dvc", "add", str(data_dir)], check=True)
    subprocess.run(
        ["git", "add", f"{data_dir}.dvc", ".gitignore"], check=True
    )
    subprocess.run(
        ["git", "commit", "-m", f"Track {data_dir.name} with DVC"], check=True
    )
    print(f"Tracked and committed: {data_dir}")


if __name__ == "__main__":
    track_directory(Path("data/satellite_imagery"))

# Spatial Parameters and Format Flags Reference

The first time imagery is ingested, record the source CRS explicitly in a dataset_metadata.json sidecar committed to Git. This prevents silent CRS drift when future contributors add scenes from different acquisition providers. See Preserving Metadata Across Dataset Versions for a schema that captures acquisition timestamp, sensor type, and spatial resolution alongside the CRS.

# Common Errors and Fixes

dvc push hangs or times out on large files : Root cause: default single-threaded upload. Fix: dvc remote modify geospatial-remote jobs 8 to enable parallel multipart transfer.

rasterio.errors.NotGeoreferencedWarning during COG validation : Root cause: the input file lacks a geotransform — the file has no embedded CRS. Fix: run gdal_edit.py -a_srs EPSG:4326 input.tif to embed the projection before conversion.

.dvc pointer file shows md5: null : Root cause: dvc add was run before dvc init completed or the .dvc/ directory is missing. Fix: confirm git status shows .dvc/config tracked, delete the broken .dvc file, and re-run dvc add.

git commit includes gigabyte-scale files instead of pointer : Root cause: .gitignore was not updated by dvc add (possible permissions issue). Fix: manually verify that the imagery directory path appears in .gitignore, then re-stage and commit.

This workflow is one component of the broader Implementing DVC for Geospatial Training Data guide, which covers multi-stage dvc.yaml orchestration, preprocessing locks, and experiment reproduction at scale.

Related

• Implementing DVC for Geospatial Training Data — parent guide: DVC pipeline stages, remote auth, and dvc repro patterns
• Preserving Metadata Across Dataset Versions — keep CRS, geotransform, and acquisition timestamp in sync with binary snapshots
• Tracking Annotation Changes with SHA Hashing — extend content-addressable hashing to GeoJSON and COCO annotation exports
• Dataset Versioning & Spatial Data Sync — section overview covering the full versioning architecture