Detecting Train/Test Leakage in Tiled Datasets

A split built correctly in March can leak by June without anyone doing anything wrong. A re-tiling pass gives the same ground new tile ids; a scene is re-processed and lands under a new name; an annotator’s export is written to a directory the split manifest never heard of. Each is routine, and each can put near-identical content on both sides of a boundary. This guide gives four tests — spatial adjacency, scene exclusivity, temporal overlap and duplicate content — that catch those four paths, plus the wiring that runs them on every dataset change rather than once when the split was drawn.

# Why the Split Manifest Is Not Enough

The split manifest records an assignment. It is authoritative about which tile is on which side, and it says nothing about whether two tiles on opposite sides are the same ground. Those are different claims, and only the second one is what “held out” means.

Four ways they come apart:

• Adjacency. A new tile lands in a training block and touches a validation block, because the buffer was computed when the dataset was smaller.
• Scene. Two tiles far apart geographically came from one acquisition, sharing sun angle, atmosphere and processing.
• Time. The same ground appears on two dates, one on each side, so the model has seen the buildings it is being evaluated on.
• Duplicate content. The identical tile entered twice under two identifiers, and the hash assignment sent the two copies to different sides.

# Step-by-Step Implementation

# Step 1 — Spatial Adjacency, With the Buffer From the Manifest

pip install geopandas==0.14.4 shapely==2.0.6 pandas==2.2.2 numpy==1.26.4 rasterio==1.3.10

import json
import geopandas as gpd

def load_split_params(manifest_path: str) -> dict:
    """Read the distances the split was actually built with — never hard-code them."""
    with open(manifest_path, encoding="utf-8") as fh:
        m = json.load(fh)
    return {"buffer_m": float(m["buffer_m"]), "block_m": float(m["block_m"]), "crs": m["crs"]}

def check_adjacency(tiles: gpd.GeoDataFrame, buffer_m: float,
                    split_col: str = "split") -> list[str]:
    train = tiles[tiles[split_col] == "train"]
    held = tiles[tiles[split_col].isin(["val", "test"])]
    if train.empty or held.empty:
        return []
    zone = train.geometry.buffer(buffer_m).union_all()
    bleed = held[held.geometry.intersects(zone)]
    return [f"tile {i} is within {buffer_m} m of training data" for i in bleed.index[:20]]

Reading buffer_m from the manifest rather than passing a literal is the difference between a test that verifies the split and one that verifies a number somebody typed. A test asserting 200 m against a split built with 500 m passes a leaking dataset and shows a green check, which is worse than having no test.

# Step 2 — Scene and Temporal Exclusivity

def check_scene(tiles: gpd.GeoDataFrame, split_col: str = "split",
                scene_col: str = "scene_id") -> list[str]:
    train = set(tiles.loc[tiles[split_col] == "train", scene_col])
    held = set(tiles.loc[tiles[split_col].isin(["val", "test"]), scene_col])
    return [f"scene {s} appears in both splits" for s in sorted(train & held)[:20]]

def check_temporal(tiles: gpd.GeoDataFrame, split_col: str = "split",
                   block_col: str = "block", date_col: str = "acquired") -> list[str]:
    """The same block on two dates, one per side, is the same ground seen twice."""
    train = tiles[tiles[split_col] == "train"]
    held = tiles[tiles[split_col].isin(["val", "test"])]
    shared = set(train[block_col]) & set(held[block_col])
    out = []
    for b in sorted(shared)[:20]:
        d_train = set(train.loc[train[block_col] == b, date_col])
        d_held = set(held.loc[held[block_col] == b, date_col])
        if d_train and d_held:
            out.append(f"block {b} is in training on {sorted(d_train)[0]} "
                       f"and held out on {sorted(d_held)[0]}")
    return out

The temporal test is the one most projects skip, and it is the one that bites hardest on change-detection work, where the same ground appearing on two dates is the entire point of the dataset and must therefore be handled deliberately rather than by accident.

# Step 3 — Duplicate Content by Pixel Hash

File bytes change on re-compression; decoded pixels do not. Hash what the model will actually see.

import hashlib
import rasterio
import numpy as np

def tile_content_hash(path: str, downsample: int = 8) -> str:
    """Hash decoded pixels, downsampled for speed. Stable across re-compression."""
    with rasterio.open(path) as src:
        h = hashlib.sha256()
        for band in range(1, src.count + 1):
            arr = src.read(band,
                           out_shape=(src.height // downsample, src.width // downsample),
                           resampling=rasterio.enums.Resampling.average)
            h.update(np.ascontiguousarray(arr).tobytes())
        return h.hexdigest()

def check_duplicates(tiles: gpd.GeoDataFrame, path_col: str = "path",
                     split_col: str = "split") -> list[str]:
    seen: dict[str, tuple[str, str]] = {}
    problems: list[str] = []
    for row in tiles.itertuples():
        digest = tile_content_hash(getattr(row, path_col))
        split = getattr(row, split_col)
        if digest in seen:
            other_id, other_split = seen[digest]
            if other_split != split and {other_split, split} != {"train", "buffer"}:
                problems.append(f"identical pixels in {split} ({row.Index}) "
                                f"and {other_split} ({other_id})")
        else:
            seen[digest] = (str(row.Index), split)
    return problems

Downsampling by eight makes this fast enough to run over a full dataset nightly and still catches genuine duplicates: two tiles that agree at one-eighth resolution across every band are not coincidentally similar.

# Step 4 — Wire All Four Into One Gate

def audit_split(tiles: gpd.GeoDataFrame, manifest_path: str, *,
                with_pixels: bool = False) -> None:
    params = load_split_params(manifest_path)
    problems: list[str] = []
    problems += check_adjacency(tiles, params["buffer_m"])
    problems += check_scene(tiles)
    problems += check_temporal(tiles)
    if with_pixels:
        problems += check_duplicates(tiles)
    if problems:
        raise AssertionError(f"{len(problems)} leakage problem(s):\n  " + "\n  ".join(problems[:20]))

Run the three metadata tests on every pull request that touches the dataset, and the pixel test nightly — the same split between cheap and expensive checks the CI/CD gate already makes for geometry and schema.

# Parameters and Thresholds Reference

# Common Errors and Fixes

The adjacency test passes but production still disagrees with validation Root cause: the buffer was applied to val but the dataset also has a test split that was not included. Fix: treat every non-training split as held out, as check_adjacency does above.

The duplicate test reports thousands of collisions Root cause: downsampling turned large areas of empty ground — sea, desert, cloud — into identical arrays. Fix: skip tiles whose pixel variance is below a threshold before hashing; genuinely featureless tiles are duplicates in a sense that does not matter.

Scene test fails on a dataset with no scene column Root cause: the tiling pass did not carry the acquisition identifier through. Fix: add it at tiling time — the manifest pattern in preserving metadata across dataset versions exists for exactly this.

Temporal test fires on a change-detection dataset Root cause: the same ground on two dates is the intended design. Fix: mark the dataset as multi-temporal in the manifest and pair the dates explicitly, so the test asserts that pairs stay on one side rather than that blocks appear once.

# Frequently Asked Questions

# Related

• Reproducible Train/Validation Splits for Spatial Data — how the split and its manifest are built in the first place
• CI/CD Gates for Annotation Datasets — the gate these tests join, and the cheap-before-expensive ordering it already uses
• Computing Stable Content Hashes for COGs — the decoded-pixel hashing the duplicate test borrows

These tests defend the split built in Reproducible Train/Validation Splits for Spatial Data, part of Dataset Versioning & Spatial Data Sync.