Configuring DVC Remotes for Geospatial Data

To version terabytes of satellite and drone imagery without stalling your team, configure a DVC remote — on S3, Google Cloud Storage, or Azure Blob — that is tuned for multi-gigabyte Cloud-Optimized GeoTIFFs rather than the small text files DVC ships defaults for. Three settings carry the load: raise jobs so DVC transfers file chunks in parallel, point a shared cache.dir with a reflink or symlink cache.type at fast local storage so huge rasters are never re-hashed or duplicated, and resolve credentials through environment variables or a cloud profile instead of writing keys into tracked config. The entire configuration is two commands — dvc remote add to register the bucket and dvc remote modify to tune it.

# Why Remote Tuning Decides Whether Versioning Scales

DVC’s defaults are built for a repository of code and small artifacts, where a naive copy-and-hash checkout costs nothing. Geospatial training data breaks those assumptions. A single mosaic can exceed 20 GB, an annotated tile set can run to hundreds of thousands of COGs, and a working copy of the dataset may not fit twice on one disk. Left untuned, dvc push opens too few connections and crawls across a link that could saturate, while dvc checkout copies every raster into the workspace and computes a fresh content hash on each one — turning a five-minute pull into an hour.

The fix is to align three layers with the shape of the data. Parallelism (jobs) matches transfer concurrency to the many-large-files profile. A shared cache with reflinks eliminates redundant copies and re-hashing so identical imagery costs disk space once. And credential handling through profiles keeps long-lived keys out of the config that every collaborator clones. Get these right once and the same remote serves an annotation team across sites; get them wrong and every pull is a bottleneck.

# Step-by-Step Remote Configuration

Install a DVC build with the cloud extras you need. The extras pull in the S3, GCS, and Azure drivers:

pip install "dvc[s3]==3.55.2" "dvc[gs]==3.55.2" "dvc[azure]==3.55.2"

Initialise DVC in the repository if you have not already, then add a remote. Every command below assumes you run it from the repository root.

# Step 1 — Add and Tune an S3 Remote

Register the bucket, mark it the default, then modify it for large-raster throughput. jobs sets the number of parallel transfer connections; endpointurl lets the same command target any S3-compatible store such as MinIO or Cloudflare R2:

dvc remote add -d s3store s3://geo-training-datasets/dvc
dvc remote modify s3store jobs 12
dvc remote modify s3store endpointurl https://s3.eu-central-1.amazonaws.com

Set credentials without writing them into the tracked config. Prefer an AWS profile or an instance role; use --local only when a value must live on disk:

export AWS_PROFILE=geo-annotation
dvc remote modify --local s3store profile geo-annotation

# Step 2 — Add a GCS Remote

Google Cloud Storage uses a service-account key. Point DVC at it through GOOGLE_APPLICATION_CREDENTIALS and keep the path in local-only config:

dvc remote add gcsstore gs://geo-training-datasets/dvc
dvc remote modify gcsstore jobs 12
dvc remote modify --local gcsstore credentialpath /secrets/geo-sa.json

# Step 3 — Add an Azure Blob Remote

Azure authenticates with an account name plus a connection string. Store the connection string locally or, better, in the AZURE_STORAGE_CONNECTION_STRING environment variable:

dvc remote add azstore azure://geo-container/dvc
dvc remote modify azstore jobs 12
dvc remote modify azstore account_name geotrainingdata
dvc remote modify --local azstore connection_string "$AZURE_STORAGE_CONNECTION_STRING"

# Step 4 — Configure a Shared Cache for Rasters

The cache is where the real geospatial savings live. Point cache.dir at fast local storage on the same filesystem as your workspace, and set cache.type so checkouts link rather than copy. reflink gives copy-on-write with no re-hashing on filesystems that support it (XFS, Btrfs, APFS); symlink is the portable fallback:

dvc cache dir /data/dvc-cache
dvc config cache.type "reflink,symlink,copy"
dvc config cache.shared group

The comma-separated cache.type list makes DVC try each strategy in order and fall back gracefully. cache.shared group sets group-writable permissions so several users on a shared annotation box reuse one cache instead of each holding a private copy of every COG.

# Step 5 — Verify Connectivity Before a Large Push

Before pushing hundreds of gigabytes, confirm the remote resolves. DVC exposes its configured filesystems through the DVCFileSystem and remote API, so a short Python 3.10+ helper can open the target and list a prefix — surfacing a credential or endpoint error in seconds rather than after a half-finished transfer:

from __future__ import annotations

from dvc.repo import Repo


def verify_remote(remote_name: str, prefix: str = "", limit: int = 10) -> list[str]:
    """List objects under a DVC remote to confirm credentials and endpoint resolve.

    Args:
        remote_name: Name passed to `dvc remote add` (e.g. "s3store").
        prefix: Optional sub-path within the remote to list.
        limit: Maximum number of entries to return.

    Returns:
        A list of object paths found on the remote (may be empty on a fresh bucket).
    """
    repo: Repo = Repo(".")
    fs = repo.cloud.get_remote_odb(remote_name).fs
    base: str = repo.cloud.get_remote_odb(remote_name).path
    target: str = f"{base.rstrip('/')}/{prefix.lstrip('/')}" if prefix else base

    entries: list[str] = []
    for path in fs.find(target, detail=False):
        entries.append(str(path))
        if len(entries) >= limit:
            break
    return entries


if __name__ == "__main__":
    found = verify_remote("s3store")
    print(f"Reachable — {len(found)} object(s) sampled from remote")
    for item in found:
        print(f"  {item}")

If this prints without raising, credentials and endpoint are valid and dvc push will proceed against a live target.

# Recommended Settings for Large Rasters

The table maps each remote to the exact add/modify keys that matter for multi-GB geospatial files, with the reasoning behind each choice.

A practical baseline for an imagery pipeline: jobs 12, cache.type reflink,symlink,copy, credentials from the environment, and the bucket in the same cloud region as your training compute. Preserving the acquisition and CRS metadata alongside these blobs is a separate concern covered in preserving metadata across dataset versions.

# Common Errors and Fixes

Unable to locate credentials / 403 Forbidden on dvc push Root cause: DVC found the remote URL but no valid credentials, usually because a key was expected in the tracked config that was never committed (correctly, since secrets do not belong there). Fix: export AWS_PROFILE, GOOGLE_APPLICATION_CREDENTIALS, or AZURE_STORAGE_CONNECTION_STRING, or set the value with dvc remote modify --local so it lands in .dvc/config.local outside git.

dvc push saturates only a fraction of available bandwidth Root cause: default jobs is too low for a workload of many large rasters, so most of the link sits idle. Fix: dvc remote modify <name> jobs 12 (or higher on a fast link). If one enormous file dominates, the win is bounded by that single multipart upload — split monolithic mosaics into tiles so parallelism has files to spread across.

checkout is slow and the workspace uses double the disk Root cause: cache.type is left at the copy default, so every COG is duplicated into the workspace and re-hashed. Fix: dvc config cache.type "reflink,symlink,copy" and ensure cache.dir is on the same filesystem as the workspace — reflinks and symlinks cannot cross a filesystem boundary and DVC silently falls back to copying when they do.

failed to create symlink / broken links on Windows or a network mount Root cause: symlink support is unavailable or requires elevated privileges, and the cache sits on a different volume than the checkout. Fix: move the cache to the same volume as the workspace, or drop symlink from the cache.type list and rely on reflink,copy. On Windows, enable Developer Mode or run the shell elevated so symlink creation is permitted.

# Related

• Implementing DVC for Geospatial Training Data — the broader guide to initialising DVC, structuring stages, and versioning imagery that this remote configuration plugs into
• Integrating STAC Catalogs with Versioned Datasets — map each versioned snapshot on your remote back to STAC items so acquisition metadata and spatiotemporal queries stay reproducible
• Syncing Annotations Across Cloud Remotes — keep labels consistent once several sites push to the same S3, GCS, or Azure remote, with conflict detection and content-addressed sync
• How to Version Control Large Satellite Imagery Datasets — companion walkthrough for tracking terabyte-scale imagery once the remote is in place

This guide is part of the broader Implementing DVC for Geospatial Training Data topic area within Dataset Versioning & Spatial Data Sync.