Bootstrapping Labels with a Zero-Shot Model

A zero-shot segmentation model will happily outline every roof, shadow, field boundary and parked car in a tile without having seen a single label from your project. That is genuinely useful for a first batch — tracing is the expensive part of annotation and the model does it for free — provided two rules hold. The proposals must be filtered hard enough that correcting them beats drawing from scratch, and they must arrive without classes, because the model’s vocabulary is not your taxonomy and a confident wrong label costs more than no label. This guide covers the run, the vectorisation, the filters, and the measurement that says whether the assistance is paying for itself.

# Why This Matters at Cold Start

In the cold-start phase there is no project model, so there is no uncertainty to sample on and the only lever is making the first batches cheaper. Zero-shot geometry is that lever. It is also the phase where a bad pre-labelling setup does the most damage: the first batch defines the annotation guide, sets the team’s habits, and becomes the seed everything else is measured against. A pre-label that quietly biases it toward what a general-purpose model finds salient is expensive to unwind.

# Step-by-Step Implementation

# Step 1 — Run on Source Pixels, Tile by Tile

pip install torch==2.3.1 rasterio==1.3.10 shapely==2.0.6 geopandas==0.14.4 numpy==1.26.4

Read a window from the source COG rather than a rendered map tile. The model then sees native values, and the georeferencing is a single affine rather than a chain of inversions through a tiler’s rendering.

import rasterio
from rasterio.windows import Window

def tile_windows(path: str, size: int = 1024, overlap: int = 64):
    """Yield (window, transform) pairs covering the scene with a fixed overlap."""
    with rasterio.open(path) as src:
        step = size - overlap
        for row in range(0, src.height, step):
            for col in range(0, src.width, step):
                w = Window(col, row,
                           min(size, src.width - col),
                           min(size, src.height - row))
                if w.width < size // 2 or w.height < size // 2:
                    continue                       # skip slivers at the far edges
                yield w, rasterio.windows.transform(w, src.transform), src.crs

The overlap matters for the same reason it does in automating pre-labeling with foundation models: an object crossing a window edge is proposed twice, in truncated form, and the duplicates must be merged after georeferencing rather than in pixel space.

# Step 2 — Vectorise, Simplify, Georeference

from rasterio.features import shapes
from shapely.geometry import shape
from shapely.geometry.base import BaseGeometry

def masks_to_polygons(masks, transform, simplify_m: float = 0.3) -> list[BaseGeometry]:
    """Trace each boolean mask to a simplified polygon in the source CRS."""
    out: list[BaseGeometry] = []
    for mask in masks:
        for geom, value in shapes(mask.astype("uint8"), mask=mask, transform=transform):
            if value != 1:
                continue
            poly = shape(geom).simplify(simplify_m).buffer(0)
            if not poly.is_empty and poly.geom_type in ("Polygon", "MultiPolygon"):
                out.append(poly)
    return out

simplify_m should be roughly the ground size of one pixel: enough to remove the staircase that pixel tracing produces, not enough to round off real corners. The trade-off is the one covered in best practices for polygon vs bounding box annotation.

# Step 3 — Filter Hard

Most of what a zero-shot model returns is not an object your project cares about. Two cheap geometric filters remove the bulk of it.

import math

def keep_proposal(poly: BaseGeometry, min_area_m2: float = 20.0,
                  max_elongation: float = 12.0) -> bool:
    """Area and shape filters, both in projected metres."""
    if poly.area < min_area_m2:
        return False                       # correcting it costs more than drawing it
    box = poly.minimum_rotated_rectangle
    xs, ys = box.exterior.coords.xy
    edges = [math.dist((xs[i], ys[i]), (xs[i + 1], ys[i + 1])) for i in range(4)]
    long_, short = max(edges), min(edges)
    if short == 0 or (long_ / short) > max_elongation:
        return False                       # a shadow strip or a field margin, not an object
    return True

The elongation filter earns its place on aerial imagery specifically: shadows, hedgerows and road margins are the commonest false proposals and they are all long and thin, while the built objects most projects care about are not.

# Step 4 — Ship Them Class-Less

from dataclasses import dataclass

@dataclass(frozen=True)
class Proposal:
    tile_id: str
    geometry_wkt: str
    crs: str
    source: str = "zero-shot"
    class_name: str | None = None          # deliberately empty — the human decides

Every annotation platform will happily accept a class on a prediction, and it is worth resisting. The failure it prevents is not the model being wrong occasionally; it is the model being plausibly wrong systematically, so that a whole batch inherits its idea of where warehouse stops and industrial starts.

# Step 5 — Measure Whether It Helped

Pre-labelling is an intervention and deserves a control. Run a slice of the first batch without proposals and compare.

import pandas as pd

def assistance_report(events: pd.DataFrame) -> pd.DataFrame:
    """Accept/adjust/delete rates and median tile time, assisted against control."""
    g = events.groupby("arm")
    return pd.DataFrame({
        "tiles": g["tile_id"].nunique(),
        "median_seconds_per_tile": g["seconds"].median().round(1),
        "accepted": g["action"].apply(lambda s: (s == "accept").mean().round(3)),
        "adjusted": g["action"].apply(lambda s: (s == "adjust").mean().round(3)),
        "deleted": g["action"].apply(lambda s: (s == "delete").mean().round(3)),
    })

Three readings and what each means:

• Median time per tile is not lower than the control. The proposals are not helping; the filters are too loose or the model is a poor fit for this imagery.
• Acceptance above ~90%. Almost certainly nobody is checking. Confirm with an adjudication sample rather than a survey, using the agreement machinery in annotation quality metrics and agreement.
• Deletion above ~40%. The filters are letting through objects the project does not care about. Raise min_area_m2 before touching the model.

# Parameters and Thresholds Reference

# Common Errors and Fixes

Proposals land tens of metres from the objects Root cause: the model ran on rendered map tiles and the coordinates were inverted through the tiler’s Web Mercator rendering. Fix: run on source windows, as in Step 1, and transform with the window’s own affine.

Thousands of proposals per tile Root cause: automatic mode with no filtering — the model has segmented every roof facet and shadow separately. Fix: apply Step 3’s filters before anything reaches a platform, and consider prompting with a detector’s boxes rather than running in automatic mode.

Annotators say the proposals slow them down Root cause: deletion rate is high, so the assistance is net negative. Fix: measure it with the control arm rather than debating it; raise min_area_m2 and re-measure.

Duplicate proposals along window edges Root cause: overlapping windows, deduplicated in pixel space or not at all. Fix: merge after georeferencing, on world-coordinate IoU — the two windows have different pixel origins.

# Frequently Asked Questions

# Related

• Cold-Start Strategies for New Annotation Projects — the phase this fits into, and what to do with the batches it produces
• Automating Pre-Labeling with Foundation Models — the production pipeline for running these models at scale, including tile-boundary deduplication
• Automating Batch Pre-Labeling with SAM and QGIS — the desktop route, when proposals are corrected in a GIS rather than a web queue

Bootstrapping is the first move within Cold-Start Strategies for New Annotation Projects, part of Active Learning & Model Feedback Loops.