Preserving CRS Metadata Through Shapefile Exports

The shapefile is the format most cadastral deliverables are still required in, and it can carry neither a datum realisation unambiguously, nor a coordinate epoch, nor the operation that produced the coordinates. This guide, part of geospatial file formats and CRS metadata, makes a shapefile deliverable defensible anyway: a package rather than a file, with a sidecar that carries what the format drops and a manifest that stops the two being separated.

What the Format Drops, and Where It Goes Instead #

Figure — six things a shapefile drops, and where each one has to be carried instead.

The last two rows are the ones that surprise people who have only met the CRS problems. A shapefile attribute table is a DBF file: names are limited to ten characters and numeric fields have a declared width and precision, so a northing_m column declared with two decimals silently rounds every value written into it. Coordinates carried as attributes are therefore only as good as the field definition, and a package should either widen them deliberately or not carry them at all.

Complete Runnable Implementation #

from __future__ import annotations

import hashlib
import json
from dataclasses import dataclass, asdict
from pathlib import Path

from pyproj import CRS

SHAPEFILE_PARTS = (".shp", ".shx", ".dbf", ".prj", ".cpg")
ENSEMBLE_CODES = {("EPSG", "4326"), ("EPSG", "4269"), ("EPSG", "4258")}


@dataclass(frozen=True)
class DeliveryPackage:
    """A shapefile plus everything the shapefile cannot say."""

    basename: str
    crs: CRS
    epoch: float | None
    operation_id: str
    operation_accuracy_m: float
    grid_checksums: dict[str, str]
    field_name_map: dict[str, str]        # full name -> the 10-char DBF name

    def sidecar(self) -> dict:
        auth = self.crs.to_authority()
        if auth is None:
            raise ValueError("CRS has no authority code; record it explicitly")
        if tuple(auth) in ENSEMBLE_CODES:
            raise ValueError(f"{auth[0]}:{auth[1]} is an ensemble, not a realisation")
        return {
            "basename": self.basename,
            "crs_authority": f"{auth[0]}:{auth[1]}",
            "crs_wkt2": self.crs.to_wkt(version="WKT2_2019"),
            "coordinate_epoch": self.epoch,
            "operation_id": self.operation_id,
            "operation_accuracy_m": self.operation_accuracy_m,
            "grid_checksums": dict(sorted(self.grid_checksums.items())),
            "field_name_map": dict(sorted(self.field_name_map.items())),
        }

    def manifest(self, directory: Path) -> dict:
        """Hash every part of the package, so nothing can be quietly dropped."""
        files = {}
        for suffix in SHAPEFILE_PARTS + (".meta.json",):
            path = directory / f"{self.basename}{suffix}"
            if not path.exists():
                if suffix in (".cpg",):
                    continue                       # optional
                raise FileNotFoundError(f"package is incomplete: {path.name}")
            files[path.name] = hashlib.sha256(path.read_bytes()).hexdigest()
        return {"basename": self.basename, "files": files}

    def write(self, directory: Path) -> None:
        """Write the sidecar and the manifest beside an already-written shapefile."""
        (directory / f"{self.basename}.meta.json").write_text(
            json.dumps(self.sidecar(), indent=2, sort_keys=True)
        )
        (directory / f"{self.basename}.manifest.json").write_text(
            json.dumps(self.manifest(directory), indent=2, sort_keys=True)
        )

Parameter Reference #

Worked Example #

from pathlib import Path
from pyproj import CRS

pkg = DeliveryPackage(
    basename="parcels_2026_08",
    crs=CRS.from_epsg(6339),
    epoch=2020.00,
    operation_id="EPSG:8264",
    operation_accuracy_m=0.015,
    grid_checksums={"us_noaa_conus.tif": "3f9a2c..."},
    field_name_map={"parcel_identifier": "PARCEL_ID",
                    "survey_northing_m": "NORTH_M"},
)
print(pkg.sidecar()["crs_authority"], pkg.sidecar()["coordinate_epoch"])
# EPSG:6339 2020.0
pkg.write(Path("out"))
# FileNotFoundError: package is incomplete: parcels_2026_08.shx

The failure in that last line is the feature. A package assembled without one of its parts is not a deliverable, and finding out at write time is much better than finding out when a recipient cannot open it.

Figure — the parts of a shapefile package; a missing one is an incomplete deliverable.

Validation Check #

def assert_prj_matches_sidecar(directory: Path, pkg: DeliveryPackage) -> None:
    """The .prj and the sidecar must describe the same CRS."""
    prj = CRS.from_wkt((directory / f"{pkg.basename}.prj").read_text())
    assert prj.equals(pkg.crs), (
        "the .prj and the sidecar disagree about the CRS; the file is "
        "self-contradictory and a reader has no way to choose"
    )

Common Mistakes #

Shipping the shapefile without the sidecar. The whole strategy depends on the two travelling together; a .zip of the shapefile parts alone is a deliverable missing its metadata. Include the sidecar and the manifest in the same archive, and name them in whatever transmittal accompanies it.

Coordinates in attributes at DBF default precision. A numeric DBF field declared with two decimals rounds every value silently. Either declare the field with four decimals and enough width for a six-million-metre northing, or leave the coordinates in the geometry where they are float64.

Field names truncated without a map. The DBF ten-character limit silently truncates and, on collision, renames. Without a field-name map, a recipient cannot tell SURVEY_NOR from SURVEY_NOR_1, and the map costs a dictionary in the sidecar.

Field Definitions That Do Not Lose Precision #

The DBF attribute table is where a shapefile quietly discards precision, and the fix is entirely in the field definitions rather than in the writing code. Three rules cover it.

Figure — DBF field widths that hold a survey coordinate, against the defaults that do not.

Size numeric fields from the value range and the precision together. A northing of six million metres to four decimal places needs seven integer digits, a decimal point and four decimals — twelve characters. A field declared as N(10,2) cannot hold it, and what happens next is silent truncation rather than an error.

Keep coordinates in the geometry. Attribute copies exist for convenience, and every one of them is a second source of truth that can drift from the first. If they are required by the recipient’s workflow, declare them at full precision and regenerate them from the geometry rather than carrying them alongside the transformation.

Map field names before writing, not after. The ten-character limit truncates, and on collision the writer appends a numeric suffix — so survey_northing and survey_northerly become SURVEY_NOR and SURVEY_NO1, and neither is recoverable without the map. Deciding the short names deliberately, and recording the mapping in the sidecar, keeps the meaning attached.

FIELD_DEFS = {
    "PARCEL_ID": ("C", 24, 0),
    "EAST_M": ("N", 12, 4),      # 7 integer digits + point + 4 decimals
    "NORTH_M": ("N", 13, 4),     # northings need one more
    "EPOCH": ("N", 9, 2),        # decimal year, e.g. 2020.00
}

Writing that table out explicitly, rather than letting a library infer field widths from the first few values, is the difference between a deliverable that holds millimetres and one that rounds them away in the third row.

Frequently Asked Questions #

Should I ship a GeoPackage as well when a shapefile is mandated? #

Yes, where the recipient will accept it. The GeoPackage is the self-describing artefact and the shapefile is the compatibility one; shipping both means the identity survives even if the sidecar is separated from the shapefile downstream. Name both in the manifest so the pair is verifiable.

What about the .cpg file? #

Write it. It declares the DBF character encoding, and without it a recipient guesses — usually correctly for ASCII names and wrongly for anything with an accent, which in cadastral data means place names and surveyor names. It costs one line and removes a whole category of support question.

Is a zipped package enough to keep the parts together? #

It is the practical answer, and it works as long as the manifest is inside the archive rather than beside it. What defeats it is a recipient who extracts only the .shp, .shx, .dbf and .prj because those are the parts their software recognises — which is why the transmittal should say, in words, that the sidecar is part of the deliverable.

Does the sidecar need a defined schema? #

If the agency publishes one, use it. Otherwise a small, stable JSON object like the one above is enough, and keeping the key names identical across deliveries matters more than the structure being sophisticated: a recipient writes a parser once, and every change costs them.

Does the 2 GB shapefile size limit matter for cadastral data? #

It does at county scale and above. The .shp and .dbf both use 32-bit file offsets, so each is capped near two gigabytes, and the failure mode when the cap is reached is a truncated file rather than an error — which is exactly the kind of silent loss this guide is about. Check the sizes as part of the package validation, and where a dataset approaches the limit, split it on a meaningful boundary and record the split in the manifest rather than letting a writer discover it.

It is worth stating the underlying position plainly: a shapefile is a transport format, not a record. Treat the package — data, sidecar, manifest — as the deliverable, keep an authoritative self-describing copy elsewhere, and the format’s limitations become an inconvenience rather than a loss of information. The judgement to apply is simple: if the package were opened by a stranger in ten years, could they say exactly what these coordinates mean and how they were produced?

Related #

• Geospatial file formats and CRS metadata — the parent topic
• Reading and writing .prj files with Python
• Exporting ISO 19111 metadata for cadastral deliverables
• Generating audit hashes for transformation batches
• Recording coordinate epochs in cadastral deliverables