Converting Shapefiles to GeoParquet for Annotation

Legacy annotation deliverables almost always arrive as Shapefiles — a fragile bundle of .shp, .shx, .dbf, and .prj files that must stay together, whose attribute names are clipped to ten characters, and whose coordinate reference system lives in a plain-text .prj file that anything can overwrite. Converting to GeoParquet fixes all three problems at once. The reliable path is short: read the Shapefile with geopandas, confirm or assign the CRS from the .prj, repair any truncated or duplicated field names, then write GeoParquet — which embeds the CRS directly as file metadata, removes the 10-character .dbf field-name limit, and collapses the multi-file sidecar cluster into one self-describing file. Finish by validating the roundtrip so you can prove no geometry or projection information was lost.

# Why the Shapefile Sidecar Cluster Fails ML Pipelines

A Shapefile is not one file; it is a minimum of three and usually four, and every consumer must keep them adjacent with identical basenames. Drop the .prj during an scp and the CRS is gone. Zip only the .shp and the dataset is unreadable. That fragility is tolerable for a one-off GIS map but corrosive for an annotation pipeline where thousands of label files move between object storage, a labeling platform, and a training loader on every iteration. The attribute table compounds the problem: it is a dBASE .dbf, so field names are hard-capped at 10 characters. A column you named annotation_confidence is written to disk as annotatio_c, and if you also have annotation_class the two can truncate into a collision that the driver resolves by silently renaming one. Feed that into a training script keyed on column names and the run fails, or worse, trains on the wrong field.

GeoParquet removes each failure mode. It is a single Parquet file whose geometry column is stored as Well-Known Binary and whose CRS is written as PROJJSON into a file-level geo metadata key, so the projection travels inside the bytes that hold the coordinates. Parquet’s schema imposes no practical limit on column-name length, so full annotation field names survive. And because Parquet is columnar and compressed, a loader that needs only geometry and a label reads those columns alone.

# Prerequisites and Environment

Install the pinned toolchain once. pyarrow provides the Parquet writer that geopandas calls under the hood:

pip install geopandas==0.14.4 pyarrow==16.1.0 shapely==2.0.6 pyproj==3.6.1

# Step 1 — Read the Shapefile and Inspect What Arrived

geopandas.read_file resolves the whole sidecar cluster from the single .shp path — it opens the .shx, .dbf, and .prj automatically. Read it, then immediately inspect the CRS and the columns so you know the true state of the data before touching it:

from pathlib import Path
import geopandas as gpd


def load_shapefile(shp_path: Path) -> gpd.GeoDataFrame:
    """Load a Shapefile and its sidecars into a single GeoDataFrame."""
    gdf: gpd.GeoDataFrame = gpd.read_file(shp_path)
    print(f"features:   {len(gdf)}")
    print(f"crs:        {gdf.crs}")
    print(f"geom types: {sorted(gdf.geom_type.unique())}")
    print(f"columns:    {list(gdf.columns)}")
    return gdf


gdf = load_shapefile(Path("annotations/parcels.shp"))

If crs prints None, the .prj is missing or unreadable — handle that in Step 2 before anything else. If geom types shows both Polygon and MultiPolygon, note it for Step 4.

# Step 2 — Confirm or Assign the CRS

When the .prj parsed cleanly, gdf.crs is a pyproj.CRS object and you only need to confirm it is the projection you expect. When it is None, you must assign the correct authority code — use set_crs, which attaches the CRS without moving any coordinates (that is to_crs, which you do not want here because the coordinates are already in the source projection):

from pyproj import CRS


def ensure_crs(gdf: gpd.GeoDataFrame, expected_epsg: int) -> gpd.GeoDataFrame:
    """Confirm the parsed CRS, or assign it if the .prj was missing."""
    if gdf.crs is None:
        # .prj was absent — attach the known CRS without reprojecting.
        gdf = gdf.set_crs(epsg=expected_epsg, allow_override=False)
        print(f"assigned CRS EPSG:{expected_epsg} (was None)")
    else:
        parsed = CRS.from_user_input(gdf.crs)
        if parsed.to_epsg() != expected_epsg:
            raise ValueError(
                f".prj CRS EPSG:{parsed.to_epsg()} != expected EPSG:{expected_epsg}"
            )
        print(f"confirmed CRS EPSG:{parsed.to_epsg()}")
    return gdf


gdf = ensure_crs(gdf, expected_epsg=4326)

Assigning the wrong code when coordinate values disagree with the .prj is its own diagnostic problem; if the numbers look like they belong to a different projection than the file claims, work through fixing legacy Shapefile .prj mismatches before you convert. The first time your annotations pass through code, the authority code should be one you have verified — here [EPSG:4326](https://www.geospatialannotation.com/geospatial-annotation-fundamentals-architecture/coordinate-reference-systems-in-annotation-pipelines/) — not one guessed from the filename.

# Step 3 — Repair Truncated and Duplicate Field Names

The .dbf clipped every field name to 10 characters on write. GeoParquet will happily store the full names, but only if you restore them first. Supply an explicit rename map keyed by the truncated name, and guard against the collisions truncation can create:

def restore_field_names(
    gdf: gpd.GeoDataFrame,
    rename_map: dict[str, str],
) -> gpd.GeoDataFrame:
    """Rename dbf-truncated columns back to their intended full names."""
    present = {old: new for old, new in rename_map.items() if old in gdf.columns}
    renamed = gdf.rename(columns=present)

    duplicates = renamed.columns[renamed.columns.duplicated()].tolist()
    if duplicates:
        raise ValueError(f"rename produced duplicate columns: {duplicates}")

    over_10 = [c for c in renamed.columns if len(c) > 10 and c != "geometry"]
    print(f"restored {len(present)} name(s); {len(over_10)} now exceed 10 chars")
    return renamed


gdf = restore_field_names(
    gdf,
    rename_map={
        "annotatio_c": "annotation_confidence",
        "annotati_1": "annotation_class",
        "reviewer_i": "reviewer_id",
    },
)

# Step 4 — Homogenise Geometry and Write GeoParquet

If Step 1 reported mixed Polygon and MultiPolygon rows, promote everything to MultiPolygon so the geometry column has one homogeneous type that every GeoParquet reader accepts. Then write with to_parquet, which serialises geometry to WKB and embeds the CRS as PROJJSON into the file’s geo metadata:

from shapely.geometry.base import BaseGeometry
from shapely.geometry import MultiPolygon


def to_multi(geom: BaseGeometry) -> BaseGeometry:
    """Promote a bare Polygon to MultiPolygon; leave other types unchanged."""
    if geom.geom_type == "Polygon":
        return MultiPolygon([geom])
    return geom


def write_geoparquet(gdf: gpd.GeoDataFrame, out_path: Path) -> Path:
    """Write a single self-describing GeoParquet file with the CRS embedded."""
    if gdf.crs is None:
        raise ValueError("refusing to write GeoParquet without a CRS")

    homogeneous = gdf.copy()
    homogeneous["geometry"] = homogeneous.geometry.apply(to_multi)

    out_path.parent.mkdir(parents=True, exist_ok=True)
    homogeneous.to_parquet(out_path, index=False, compression="zstd")
    print(f"wrote {out_path} ({out_path.stat().st_size / 1024:.1f} KiB)")
    return out_path


out = write_geoparquet(gdf, Path("annotations/parcels.parquet"))

# Step 5 — Validate the Roundtrip

A conversion you have not verified is a conversion you cannot trust. Read the GeoParquet back and assert two things: every geometry equals its source geometry, and the CRS matches exactly. Use geom_equals for a topological comparison that tolerates the WKB re-encoding, and compare CRS via authority code:

def validate_roundtrip(
    source: gpd.GeoDataFrame,
    parquet_path: Path,
) -> None:
    """Assert geometry and CRS survived the Shapefile to GeoParquet conversion."""
    loaded: gpd.GeoDataFrame = gpd.read_parquet(parquet_path)

    # CRS equality by authority code.
    src_epsg = source.crs.to_epsg()
    dst_epsg = loaded.crs.to_epsg()
    assert src_epsg == dst_epsg, f"CRS changed: {src_epsg} -> {dst_epsg}"

    # Row count and geometry equality.
    assert len(source) == len(loaded), "feature count changed"
    src_geom = source.geometry.apply(to_multi).reset_index(drop=True)
    dst_geom = loaded.geometry.reset_index(drop=True)
    mismatches = int((~src_geom.geom_equals(dst_geom)).sum())
    assert mismatches == 0, f"{mismatches} geometries differ after roundtrip"

    print(f"roundtrip OK: {len(loaded)} features, CRS EPSG:{dst_epsg} preserved")


validate_roundtrip(gdf, out)

When this passes, the GeoParquet is a faithful, self-describing replacement for the four-file bundle and is safe to hand to a training loader or annotation platform.

# Shapefile Limitation to GeoParquet Fix

# Common Errors and Fixes

ValueError: Cannot write GeoDataFrame with CRS set to None Root cause: the .prj was missing when the Shapefile was read, so gdf.crs is None, and to_parquet refuses to write a projection-less file. Fix: call set_crs(epsg=..., allow_override=False) with the verified authority code in Step 2 before writing — never to_crs, which would move coordinates that are already correct.

Column named annotatio_c instead of annotation_confidence in the output Root cause: the name was truncated to 10 characters by the .dbf on the original write and carried straight through the conversion. Fix: apply an explicit rename map keyed on the truncated names before writing, as in Step 3, and assert no duplicates were produced.

Duplicate column names raised during rename Root cause: two long field names truncated to the same 10-character string in the .dbf, so one row of your rename map cannot uniquely recover them. Fix: inspect the raw .dbf field order to disambiguate which physical column is which, rename positionally, then apply the semantic names — do not trust the clipped strings to be unique.

Geometry column contains mixed geometry types on read by a strict GeoParquet consumer Root cause: the source mixed Polygon and MultiPolygon rows, and the writer recorded a heterogeneous geometry type that a strict reader rejects. Fix: promote every feature to its Multi form with the to_multi helper in Step 4 so the column is homogeneous before to_parquet.

pyarrow.lib.ArrowInvalid: GeoParquet metadata not found when reading back Root cause: the file was written with a plain pandas/pyarrow to_parquet on a de-geometried frame, so the geo metadata key was never added. Fix: keep the object as a GeoDataFrame and use gdf.to_parquet (geopandas), which writes the GeoParquet geo metadata; read it with gpd.read_parquet, not pandas.read_parquet.

# Related

• Spatial Data Formats for ML Annotation: COG, GeoParquet, and STAC — the topic area this guide belongs to, comparing raster and vector formats and when each fits an annotation pipeline
• Fixing Legacy Shapefile .prj Mismatches — diagnose and repair a .prj that disagrees with the actual coordinate values before you convert
• Coordinate Reference Systems in Annotation Pipelines — how CRS contracts, datum handling, and reprojection are managed across an entire annotation pipeline
• How to Structure GeoJSON for ML Training Datasets — a complementary vector-label format for cases where human-readable text export matters more than columnar speed

This guide is one focused conversion within Spatial Data Formats for ML Annotation, which is itself part of Geospatial Annotation Fundamentals & Architecture.