Syncing Annotations Across Cloud Remotes

Two annotation teams — one in Lisbon, one in Nairobi — spend a Tuesday labeling building footprints over the same city extent. Both pull the dataset in the morning, both work against a shared S3 bucket, and both run sync at the end of the day. The Nairobi run finishes ninety seconds after Lisbon’s. Because the tool wrote the annotation index with an unconditional PUT to a fixed key, Nairobi’s index overwrites Lisbon’s. No error is raised, no exit code is non-zero, and the bucket reports success. The Lisbon team’s entire day of GeoJSON — several hundred hand-drawn polygons — is now unreferenced garbage in the object store, discovered only a week later when a reviewer notices the count of labeled tiles has gone down.

This failure is not a bug in any one tool. It is the default behaviour of every object store: a write to a key clobbers whatever was there, and there is no built-in notion of “only if you have seen the latest version.” This guide, part of the broader dataset versioning and spatial data sync practice, shows how to make multi-site annotation sync safe by construction — using content-addressed storage so features never overwrite each other, a manifest diff so divergence is detected before any write, an explicit push/pull ordering so partial uploads never orphan a pointer, and a lease plus generation number so concurrent sites serialize instead of racing.

The rest of this guide builds each piece as runnable Python 3.10+ against pinned dependencies. The techniques compose with DVC versioning for large binary assets and with tracking annotation changes with SHA hashing for the per-feature identity that content addressing depends on.

# Prerequisites & Remote Configuration

Content-addressed sync depends on deterministic serialization and a client that can do conditional writes. Pin the toolchain so hashes are reproducible across every site.

Required Python packages (pinned):

pip install dvc==3.51.2 \
            boto3==1.34.128 \
            gcsfs==2024.6.1 \
            geopandas==0.14.4 \
            pyproj==3.6.1 \
            shapely==2.0.6

Remote credentials and endpoints:

• S3: an IAM role or key pair with s3:GetObject, s3:PutObject, and s3:ListBucket; enable bucket versioning so an accidental pointer overwrite stays recoverable.
• GCS: a service-account JSON keyed for roles/storage.objectAdmin on the target bucket, reachable through gcsfs.
• Azure Blob: a SAS token or account key scoped to a single container, used through the adlfs filesystem if you target Azure.

Spatial and versioning prerequisites:

• Every annotation feature carries a stable identifier and a declared coordinate reference system. If features arrive in mixed projections, normalize them first — the first stop for that is coordinate reference systems in annotation pipelines, and the canonical training projection is usually a local UTM zone rather than EPSG:4326.
• A per-feature content hash. This guide computes one inline, but the durable definition lives in the SHA hashing workflow linked above.

Baseline checklist before your first multi-site sync:

• All sites agree on one canonical CRS for stored labels
• The GeoJSON serialization is byte-deterministic (sorted keys, fixed coordinate precision)
• The remote bucket has object versioning enabled for recoverability
• Exactly one object key is designated the mutable manifest pointer
• Every site records the manifest generation number it based its edits on

# Content-Addressed Sync: Hash Features Before Push

The core move is to stop treating the remote as a mutable folder of files and start treating it as an immutable, content-addressed object store. Each annotation feature is serialized to canonical bytes, hashed, and stored under objects/<sha256>. Because the key is the content hash, two sites that independently draw the identical polygon produce the identical object — a harmless deduplicated write. Two sites that draw different polygons produce different keys and can never overwrite each other. All contention collapses onto a single mutable object: the manifest that lists which hashes belong to the current dataset.

Canonicalization is the load-bearing detail. If Site A serializes coordinates to six decimals and Site B to full float64, identical geometry hashes differently and deduplication fails. Fix the precision and sort keys before hashing.

from __future__ import annotations

import hashlib
import json
from typing import Any

COORD_PRECISION = 7  # ~1 cm at the equator; match across all sites


def canonical_feature_bytes(feature: dict[str, Any]) -> bytes:
    """Serialize a GeoJSON feature to deterministic bytes for hashing."""
    def round_coords(obj: Any) -> Any:
        if isinstance(obj, float):
            return round(obj, COORD_PRECISION)
        if isinstance(obj, list):
            return [round_coords(x) for x in obj]
        if isinstance(obj, dict):
            return {k: round_coords(v) for k, v in obj.items()}
        return obj

    normalized = round_coords(feature)
    return json.dumps(
        normalized, sort_keys=True, separators=(",", ":"), ensure_ascii=False
    ).encode("utf-8")


def feature_hash(feature: dict[str, Any]) -> str:
    return hashlib.sha256(canonical_feature_bytes(feature)).hexdigest()

With a stable hash per feature, building the local manifest is a plain map from feature id to content hash plus the object key. The manifest is the only thing a reviewer needs to reconstruct the exact labeled state.

from dataclasses import dataclass, field


@dataclass
class Manifest:
    generation: int
    features: dict[str, str] = field(default_factory=dict)  # feature_id -> sha256

    def object_keys(self) -> set[str]:
        return {f"objects/{h}" for h in self.features.values()}


def build_manifest(
    features: dict[str, dict[str, Any]], generation: int
) -> Manifest:
    """Map each feature id to the content hash of its canonical bytes."""
    return Manifest(
        generation=generation,
        features={fid: feature_hash(feat) for fid, feat in features.items()},
    )

Uploading is now a write-once loop. Skip any object key that already exists — it is, by definition, byte-identical to what you would upload — and push only the missing ones.

import boto3


def push_objects(
    features: dict[str, dict[str, Any]],
    manifest: Manifest,
    bucket: str,
    s3: "boto3.client" = None,
) -> int:
    s3 = s3 or boto3.client("s3")
    uploaded = 0
    for fid, sha in manifest.features.items():
        key = f"objects/{sha}"
        try:
            s3.head_object(Bucket=bucket, Key=key)
            continue  # already present, immutable, identical
        except s3.exceptions.ClientError:
            pass
        body = canonical_feature_bytes(features[fid])
        s3.put_object(Bucket=bucket, Key=key, Body=body)
        uploaded += 1
    return uploaded

# Detect Divergence Between Local and Remote Manifests

Before writing anything mutable, download the remote manifest and diff it against the local one. The diff classifies every feature id as added at your site, removed at your site, or conflicting — present in both but pointing at different content hashes. A conflict means two sites edited the same feature id since the common base and needs reconciliation rather than a blind overwrite.

@dataclass
class ManifestDiff:
    added: dict[str, str]                 # feature_id -> local hash
    removed: dict[str, str]               # feature_id -> remote hash
    conflicting: dict[str, tuple[str, str]]  # id -> (local, remote)
    unchanged: int


def diff_manifests(local: Manifest, remote: Manifest) -> ManifestDiff:
    added, removed, conflicting = {}, {}, {}
    unchanged = 0
    all_ids = set(local.features) | set(remote.features)
    for fid in all_ids:
        lh = local.features.get(fid)
        rh = remote.features.get(fid)
        if lh == rh:
            unchanged += 1
        elif lh is not None and rh is None:
            added[fid] = lh
        elif lh is None and rh is not None:
            removed[fid] = rh
        else:  # both present, hashes differ
            conflicting[fid] = (lh, rh)
    return ManifestDiff(added, removed, conflicting, unchanged)

Because objects are immutable and hashed, a non-empty conflicting set is the only case that ever requires human or three-way-merge attention. The mechanics of resolving those cases — feature-id reconciliation and a last-writer-wins guard — are covered in depth in resolving sync conflicts in distributed annotation.

# Safe Multi-Remote Push/Pull Ordering

Ordering is what prevents a crash from leaving the dataset half-written. The invariant: the manifest pointer must only ever reference objects that already exist on the remote. That forces a strict sequence — pull and reconcile first, upload immutable content next, and commit the manifest pointer last. If the process dies after uploading objects but before committing the pointer, the extra objects are simply unreferenced and harmless; a later garbage-collection pass reclaims them. The reverse ordering — pointer first — would publish a manifest referencing objects that do not yet exist, breaking every reader.

def sync_to_remote(
    local_features: dict[str, dict[str, Any]],
    bucket: str,
    s3: "boto3.client" = None,
) -> Manifest:
    s3 = s3 or boto3.client("s3")

    # 1. Pull the authoritative remote manifest and its generation.
    remote = pull_manifest(bucket, s3)

    # 2. Build the local manifest on top of the remote generation.
    local = build_manifest(local_features, generation=remote.generation)
    diff = diff_manifests(local, remote)
    if diff.conflicting:
        raise SyncConflict(diff.conflicting)  # resolve before proceeding

    # 3. Union the two feature sets and upload MISSING objects first.
    merged_features = {**remote.features, **local.features}
    merged = Manifest(generation=remote.generation, features=merged_features)
    push_objects(local_features, local, bucket, s3)

    # 4. Commit the manifest pointer LAST, guarded by the generation number.
    return commit_manifest(merged, bucket, s3)

Multi-remote fan-out (for example S3 as primary and GCS as a disaster-recovery mirror) reuses the same ordering per remote: fully commit the primary, then replicate the immutable objects and finally the manifest to each mirror. Never commit a mirror’s pointer before its objects have finished transferring, or a failover read will hit dangling references.

# Coordinate Concurrent Sites With a Lease and Generation Number

The manifest pointer is the single mutable object, so it needs a concurrency guard. Two mechanisms work, and you can layer them. The first is a monotonic generation number: a writer records the generation it pulled, and its commit only succeeds if the remote is still at that generation. The second is a short-lived lease that gives one site an exclusive window to commit, which reduces wasted retries under heavy contention.

Object stores expose a conditional-write primitive that makes the generation check atomic. On S3, IfMatch on the pointer’s ETag rejects a write if another site has already advanced it; on GCS, ifGenerationMatch does the same with the object’s native generation number.

import json


class StaleGenerationError(RuntimeError):
    pass


def commit_manifest(
    merged: Manifest, bucket: str, s3: "boto3.client"
) -> Manifest:
    key = "manifest.json"

    # Read the current pointer AND its ETag as the compare-and-swap token.
    head = s3.head_object(Bucket=bucket, Key=key)
    current_etag = head["ETag"]
    current = Manifest(**json.loads(
        s3.get_object(Bucket=bucket, Key=key)["Body"].read()
    ))

    if current.generation != merged.generation:
        raise StaleGenerationError(
            f"remote at gen={current.generation}, expected {merged.generation}"
        )

    merged.generation = current.generation + 1
    body = json.dumps(merged.__dict__, sort_keys=True).encode("utf-8")
    try:
        s3.put_object(
            Bucket=bucket, Key=key, Body=body, IfMatch=current_etag
        )
    except s3.exceptions.ClientError as exc:
        # 412 Precondition Failed => another site committed first.
        raise StaleGenerationError("manifest advanced during commit") from exc
    return merged

When commit_manifest raises StaleGenerationError, the losing site does exactly what the diagram’s Site B does: re-pull the now-newer manifest, re-run the diff and merge, and retry. Wrapping that in a bounded retry loop turns a lost-update race into a serialized sequence with no manual intervention.

import random
import time


def sync_with_retry(
    local_features: dict[str, dict[str, Any]],
    bucket: str,
    max_attempts: int = 5,
) -> Manifest:
    for attempt in range(max_attempts):
        try:
            return sync_to_remote(local_features, bucket)
        except StaleGenerationError:
            backoff = (2 ** attempt) * 0.1 + random.uniform(0, 0.1)
            time.sleep(backoff)  # jittered exponential backoff
    raise RuntimeError(f"sync failed after {max_attempts} attempts")

A lease is worth adding when many sites commit frequently and backoff retries start to dominate. Store a lease.json object holding a holder id and an expiry timestamp; acquire it with the same conditional-write primitive, hold it only for the commit window, and always set an expiry so a crashed holder cannot deadlock the dataset. The generation check remains the real correctness guarantee — the lease is only an optimization that reduces contention.

# Remote Behaviour & Sync Parameters Reference

Different object stores offer different consistency guarantees, and the safe conflict strategy follows from them. This table maps each remote to the primitive you rely on.

# Edge Cases & Gotchas

# Eventual consistency on bucket listings

S3 gives strong read-after-write for a single key, but a LIST of a prefix can lag seconds behind reality. Never decide “which objects exist” from a listing during a sync — resolve existence by exact-key HEAD on the hashes named in the manifest. Treating a listing as authoritative is how a reader concludes a just-uploaded object is missing and aborts a valid sync.

# Partial uploads and orphaned objects

A crash mid-upload leaves some objects/<sha> present and the pointer unadvanced. The strict push-then-pointer ordering makes this benign: the manifest never referenced the incomplete set, so readers are unaffected. Reclaim the orphans with a periodic mark-and-sweep that lists all object keys, subtracts every hash reachable from the current and recent manifests, and deletes the remainder. Keep a grace window so an in-flight sync is not swept.

# Clock skew across sites

Lease expiry and any “latest wins” tie-break must not trust wall-clock timestamps from different machines — a site whose clock is five minutes fast can steal or prematurely expire a lease. Anchor expiry to the remote’s server time (S3 and GCS return authoritative Date headers) or to the monotonic generation number, never to a client clock. The generation number is skew-proof precisely because it is a counter, not a time.

# Large COG transfers stalling the pointer commit

When annotations reference large Cloud-Optimized GeoTIFF imagery, uploading a multi-gigabyte object under the manifest window can stall the whole commit. Keep heavy imagery on a content-addressed binary track managed by DVC and let the annotation manifest reference the COG by its stable content hash instead of re-uploading pixels. Use multipart uploads with retry for the imagery, and commit the small annotation pointer only after the multipart upload reports complete — a half-finished multipart upload has no visible key, so it can never be referenced early.

# Feature-id reuse across sites

If two sites mint the same feature id for genuinely different objects, content addressing correctly stores two distinct hashes, but the diff reports a conflict on that id. Prefix feature ids with a site identifier (or a UUID) at creation time so independent work never collides on the id space in the first place.

# Integration Hooks

# DVC as the binary and cache backend

DVC already stores tracked files in a content-addressed cache and pushes them to a configured remote, which is the same model this guide applies to annotation features. Let DVC own the large raster and vector binaries, and keep the fine-grained per-feature manifest as a small tracked artifact alongside them. A dvc push moves the heavy objects; the manifest commit described above coordinates the label-level pointer.

# dvc.yaml
stages:
  sync_annotations:
    cmd: python scripts/sync_annotations.py --bucket s3://annos --site lisbon
    deps:
      - scripts/sync_annotations.py
      - data/labels/
    outs:
      - data/manifest.json

# SHA hashing as the identity source

The feature_hash used here is one concrete instance of the identity scheme described in tracking annotation changes with SHA hashing. Reuse a single canonicalization routine across both workflows so the hash that tracks a change is the same hash that addresses the object — otherwise a feature can be “changed” by one system and “identical” to the other.

# CI push gate

Wire the sync into CI so a merge that would clobber the remote is caught before it runs. The job pulls the remote manifest, runs diff_manifests, and fails the build if the branch’s base generation is stale relative to the remote.

# .github/workflows/sync_gate.yml
name: Annotation sync gate
on: [push, pull_request]
jobs:
  sync-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: {python-version: "3.11"}
      - run: pip install boto3==1.34.128 geopandas==0.14.4 shapely==2.0.6
      - name: Check manifest divergence
        run: python scripts/check_divergence.py --bucket s3://annos
        env:
          AWS_REGION: eu-west-1

# Validation & Testing

# Manifest diff assertion

The cheapest guard is asserting that a freshly built local manifest, when diffed against the remote you just committed, reports no conflicts and no surprises.

def assert_clean_sync(
    local_features: dict[str, dict[str, Any]], bucket: str
) -> None:
    remote = pull_manifest(bucket)
    local = build_manifest(local_features, generation=remote.generation)
    diff = diff_manifests(local, remote)
    assert not diff.conflicting, f"unexpected conflicts: {list(diff.conflicting)}"
    assert not diff.added, f"local has unpushed features: {list(diff.added)}"
    print(f"Sync clean: {diff.unchanged} features aligned at gen={remote.generation}")

# Roundtrip integrity check

After a commit, re-download the manifest and verify that every referenced object exists and that its stored bytes still hash to the key it lives under. This catches silent truncation, a mismatched precision setting, or an eventual-consistency read that returned a stale body.

import boto3


def verify_roundtrip(bucket: str, s3: "boto3.client" = None) -> None:
    s3 = s3 or boto3.client("s3")
    manifest = pull_manifest(bucket, s3)
    for fid, sha in manifest.features.items():
        key = f"objects/{sha}"
        body = s3.get_object(Bucket=bucket, Key=key)["Body"].read()
        actual = hashlib.sha256(body).hexdigest()
        assert actual == sha, f"corruption on {fid}: {key} hashes to {actual}"
    print(f"Roundtrip verified: {len(manifest.features)} objects intact")

# Concurrency race test

Simulate two sites committing against the same base generation and assert that exactly one succeeds while the other raises StaleGenerationError and then converges on retry. Running this against a real bucket (or a MinIO container in CI) is the only way to confirm your remote actually honors conditional writes — some S3-compatible backends silently ignore IfMatch, which would reintroduce the original clobbering bug.

# Frequently Asked Questions

# Related

• Resolving Sync Conflicts in Distributed Annotation — three-way merge and feature-id reconciliation when the manifest diff reports a real conflict
• Implementing DVC for Geospatial Training Data — the content-addressed binary backend that carries heavy imagery under this sync model
• Tracking Annotation Changes with SHA Hashing — the per-feature identity scheme that content addressing reuses
• Rollback Strategies for Corrupted Spatial Datasets — how versioned pointers let you recover when a sync commits bad state
• Coordinate Reference Systems in Annotation Pipelines — normalizing projections so features hash consistently across sites

This guide is part of the broader Dataset Versioning & Spatial Data Sync practice that keeps geospatial training data reproducible from raw labels through to model input.