Enforcing the COCO JSON Schema in CI

A COCO annotation file that parses as JSON and even passes a shape schema can still crash a training run. Real validity needs three layers that a schema alone cannot express: structural shape (the images, annotations, and categories arrays hold objects with the right field types), referential integrity (every annotation.image_id and category_id resolves to a record that exists), and geometry sanity (each bbox has positive width and height and sits inside its image, and segmentation is non-empty for non-crowd regions). The reliable pattern is a two-stage gate: run jsonschema==4.21.1 for the shape, then a custom-check script for the cross-record invariants, and have the combined tool exit non-zero so continuous integration blocks the merge. This guide builds that gate end to end and wires it into GitHub Actions.

# Why a Shape Schema Is Not Enough

JSON Schema is a language for describing the form of a document — which keys exist, what types they hold, which are required. That catches a bbox stored as a string or a missing categories array, and it should run first because it is cheap and it fails fast. But COCO’s correctness lives in the relationships between records, and those relationships are invisible to a schema.

Consider an export where an annotation carries "category_id": 7 but the categories array only defines ids 1 through 5. The document is perfectly well-shaped; every field has the right type. Yet pycocotools will raise a KeyError the instant it builds its category index, and a training loop that ignores the exception will learn against a phantom class. The same blind spot hides duplicate image_id values, a bbox whose corner falls twenty pixels outside a 512-pixel tile, and an empty segmentation: [] on a polygon annotation that a mask-based loss will silently skip. These are exactly the failures that a validation gate exists to stop, and none of them are shape errors.

This matters more for geospatial data than for consumer photo datasets. Aerial and satellite exports are tiled, stitched, and reprojected by pipelines that touch pixel coordinates directly, so an off-by-one in image dimensions or a bbox clipped at a tile seam is common. When the geotransform and CRS are carried alongside the pixels — as covered in embedding geotransform metadata in COCO exports — a bounds error is not just a bad label, it reprojects to the wrong place on Earth. The gate below treats that geometry as a first-class check rather than an afterthought.

# Building the Validation Gate

Install the single pinned dependency once. The standard library covers JSON parsing and the command-line surface, so the gate stays lightweight enough to run on every push:

pip install jsonschema==4.21.1

# Step 1 — Declare the COCO JSON Schema

Define the shape contract as a Draft 2020-12 schema dict. Require the three top-level arrays and pin the field types inside each object. Keeping additionalProperties permissive lets vendor-specific extensions through while still enforcing the fields COCO consumers depend on:

from typing import Any

COCO_SCHEMA: dict[str, Any] = {
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "type": "object",
    "required": ["images", "annotations", "categories"],
    "properties": {
        "images": {
            "type": "array",
            "items": {
                "type": "object",
                "required": ["id", "width", "height", "file_name"],
                "properties": {
                    "id": {"type": "integer"},
                    "width": {"type": "integer", "minimum": 1},
                    "height": {"type": "integer", "minimum": 1},
                    "file_name": {"type": "string", "minLength": 1},
                },
            },
        },
        "annotations": {
            "type": "array",
            "items": {
                "type": "object",
                "required": ["id", "image_id", "category_id", "bbox"],
                "properties": {
                    "id": {"type": "integer"},
                    "image_id": {"type": "integer"},
                    "category_id": {"type": "integer"},
                    "iscrowd": {"type": "integer", "enum": [0, 1]},
                    "bbox": {
                        "type": "array",
                        "items": {"type": "number"},
                        "minItems": 4,
                        "maxItems": 4,
                    },
                    "segmentation": {"type": ["array", "object"]},
                },
            },
        },
        "categories": {
            "type": "array",
            "items": {
                "type": "object",
                "required": ["id", "name"],
                "properties": {
                    "id": {"type": "integer"},
                    "name": {"type": "string", "minLength": 1},
                },
            },
        },
    },
}

# Step 2 — Validate Structural Shape

Use iter_errors rather than validate so a single run reports every shape violation instead of aborting on the first. Each error is flattened to a readable path so the CI log points straight at the offending record:

from jsonschema import Draft202012Validator

def check_shape(document: dict[str, Any]) -> list[str]:
    """Return every JSON Schema violation as a human-readable string."""
    validator = Draft202012Validator(COCO_SCHEMA)
    errors: list[str] = []
    for err in sorted(validator.iter_errors(document), key=lambda e: e.path):
        location = "/".join(str(p) for p in err.path) or "<root>"
        errors.append(f"shape: {location}: {err.message}")
    return errors

# Step 3 — Check Referential Integrity

Build the id sets once, then walk the annotations. This is the pass that catches the dangling category_id, the missing image, and duplicate ids that a shape schema cannot see:

def check_references(document: dict[str, Any]) -> list[str]:
    """Assert every annotation id edge resolves and ids are unique."""
    errors: list[str] = []
    image_ids: set[int] = set()
    for img in document["images"]:
        if img["id"] in image_ids:
            errors.append(f"ref: duplicate image id {img['id']}")
        image_ids.add(img["id"])

    category_ids: set[int] = {cat["id"] for cat in document["categories"]}

    seen_ann: set[int] = set()
    for ann in document["annotations"]:
        if ann["id"] in seen_ann:
            errors.append(f"ref: duplicate annotation id {ann['id']}")
        seen_ann.add(ann["id"])
        if ann["image_id"] not in image_ids:
            errors.append(
                f"ref: annotation {ann['id']} -> missing image_id {ann['image_id']}"
            )
        if ann["category_id"] not in category_ids:
            errors.append(
                f"ref: annotation {ann['id']} -> missing category_id {ann['category_id']}"
            )
    return errors

# Step 4 — Enforce bbox Bounds and Non-Empty Segmentation

Index images by id, then check each annotation’s geometry against its parent frame. COCO’s bbox is [x, y, width, height] in pixel space; the box must have positive extent and stay inside the image. Require a non-empty segmentation only when the annotation is not a crowd region, because crowd masks are stored as run-length encoding rather than a polygon list:

def check_geometry(document: dict[str, Any]) -> list[str]:
    """Validate bbox bounds and segmentation presence per annotation."""
    errors: list[str] = []
    images_by_id: dict[int, dict[str, Any]] = {
        img["id"]: img for img in document["images"]
    }
    for ann in document["annotations"]:
        img = images_by_id.get(ann["image_id"])
        if img is None:
            continue  # already reported by check_references

        x, y, w, h = ann["bbox"]
        if w <= 0 or h <= 0:
            errors.append(f"bbox: annotation {ann['id']} has non-positive size {w}x{h}")
        if x < 0 or y < 0 or x + w > img["width"] or y + h > img["height"]:
            errors.append(
                f"bbox: annotation {ann['id']} escapes image "
                f"{img['width']}x{img['height']} at [{x},{y},{w},{h}]"
            )

        if ann.get("iscrowd", 0) == 0:
            seg = ann.get("segmentation")
            if not seg:  # None, [], or {} all fail for a non-crowd polygon
                errors.append(f"seg: annotation {ann['id']} has empty segmentation")
    return errors

# Step 5 — Wrap the Checks as a CLI Returning Exit Codes

Aggregate the three passes, print the failures, and translate the result into a process exit code. This is the piece that makes the script a gate rather than a report: CI reads the exit code, so returning 1 on any failure is what turns the job red:

import json
import sys
from pathlib import Path

def validate_coco(path: Path) -> list[str]:
    document = json.loads(path.read_text(encoding="utf-8"))
    errors = check_shape(document)
    if errors:
        return errors  # semantic checks assume a well-shaped document
    return check_references(document) + check_geometry(document)

def main(argv: list[str]) -> int:
    if len(argv) < 2:
        print("usage: validate_coco.py <file.json> [<file.json> ...]")
        return 2
    total = 0
    for arg in argv[1:]:
        problems = validate_coco(Path(arg))
        if problems:
            total += len(problems)
            print(f"FAIL {arg} ({len(problems)} error(s)):")
            for p in problems:
                print(f"  - {p}")
        else:
            print(f"OK   {arg}")
    return 1 if total else 0

if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

# Step 6 — Call the Gate from GitHub Actions

Add a workflow that installs the pinned dependency and runs the validator against every COCO file on each pull request. A non-zero exit fails the step, which blocks the merge:

name: validate-coco
on: [pull_request]

jobs:
  coco-schema:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install jsonschema==4.21.1
      - name: Validate COCO exports
        run: python validate_coco.py annotations/*.json

# Validation Thresholds and Flags

Each check maps to a specific COCO field, a pass condition, and the exit behaviour CI relies on. Treat every row as a hard gate — none of these should be warnings, because a warning that does not change the exit code lets a broken export merge.

This gate sits naturally alongside the geometry and CRS assertions described in CI/CD gates for annotation datasets; run the shape and reference checks first because they are the cheapest, then layer geometry validation on the survivors.

# Common Errors and Fixes

jsonschema.exceptions.ValidationError raised on the first bad record Cause: calling Draft202012Validator(schema).validate(document), which stops at the first violation. Fix: use iter_errors(document) and collect the results so one CI run reports every shape problem at once.

KeyError in pycocotools despite a green schema check Cause: an annotation.category_id or image_id points at an id that no record defines — a referential failure a shape schema cannot detect. Fix: run the check_references pass that builds id sets from images and categories and asserts every annotation edge resolves.

bbox validation passes but masks render outside the tile Cause: bbox treated as [x1, y1, x2, y2] instead of COCO’s [x, y, width, height], so the bounds math is wrong. Fix: unpack as x, y, w, h and test x + w <= image["width"] and y + h <= image["height"].

Crowd annotations wrongly flagged for empty segmentation Cause: requiring a polygon segmentation list on every annotation, including crowd regions stored as RLE. Fix: gate the non-empty segmentation check behind iscrowd == 0 so RLE crowd masks pass.

CI reports failures but the job still turns green Cause: the script prints errors but returns exit code 0. Fix: return 1 from main whenever total errors is non-zero so the runner marks the step failed and blocks the merge.

# Related

• Validating Annotation Export Formats: COCO, YOLO, and GeoJSON — the topic area this gate belongs to, covering schema contracts, geometry checks, and CRS sidecar verification across every export format
• COCO vs GeoParquet for Annotation Export — a decision guide on when COCO’s JSON graph is the right export target versus a columnar spatial format
• CI/CD Gates for Annotation Datasets — wire this validator into a broader gate stack with geometry validation, CRS assertions, and class-balance checks
• Embedding Geotransform Metadata in COCO Exports — attach the affine transform and EPSG code so validated pixel detections can be reprojected to real-world coordinates

This guide is one specialised check within Validating Annotation Export Formats: COCO, YOLO, and GeoJSON, part of the broader Labeling Workflows & Toolchain Integration for Geospatial AI topic area.