Pinning PROJ and Grid Versions for Reproducible Builds
Two machines running the same code can produce coordinates that differ by centimetres, and neither is malfunctioning: they have different PROJ releases, different EPSG database versions, or different grid files on disk. This guide, part of testing and CI for coordinate transformation code, pins all three so that a result is a function of the inputs rather than of the machine — which is the minimum a cadastral deliverable has to be able to claim.
The Three Things That Have to Be Pinned
The PROJ library. Minor releases refine series expansions and occasionally change which operation is considered best for a CRS pair. The first moves the last few digits; the second can move a coordinate by centimetres.
Figure — a Python lock file pins two of these by accident and the third not at all.
The EPSG database. It ships separately from the library and defines the operations, their accuracies and their areas of validity. A database update can add a better operation, deprecate one you were relying on, or change a declared accuracy that your tolerance gate compares against.
The grid files. These carry the actual shift values. A re-issued grid changes coordinates directly, and grid availability changes which operation is selectable at all — the mechanism described in fallback routing strategies for missing grid files.
A Python lock file pins none of these. It pins the pyproj wheel, which bundles a PROJ build, so it pins the first two by accident and the third not at all.
Complete Runnable Implementation
from __future__ import annotations
import hashlib
import json
from dataclasses import dataclass, asdict
from pathlib import Path
import pyproj
@dataclass(frozen=True)
class DataDependencies:
"""Everything outside the source tree that changes transformation results."""
proj_version: str
epsg_version: str
proj_data_dir: str
grids: dict[str, str] # filename -> sha256
@classmethod
def capture(cls) -> "DataDependencies":
data_dir = Path(pyproj.datadir.get_data_dir().split(";")[0])
grids: dict[str, str] = {}
for path in sorted(data_dir.rglob("*")):
if path.suffix.lower() not in (".gsb", ".gtx", ".tif", ".las", ".los"):
continue
h = hashlib.sha256()
with path.open("rb") as fh:
for block in iter(lambda: fh.read(1 << 20), b""):
h.update(block)
grids[path.name] = h.hexdigest()
return cls(
proj_version=pyproj.proj_version_str,
epsg_version=pyproj.database.get_database_metadata("EPSG.VERSION"),
proj_data_dir=str(data_dir),
grids=grids,
)
def to_lock(self, path: Path) -> None:
"""Write the lock file. `proj_data_dir` is recorded but not compared."""
payload = asdict(self)
payload.pop("proj_data_dir")
path.write_text(json.dumps(payload, indent=2, sort_keys=True))
def diff(self, locked: dict) -> list[str]:
"""Human-readable differences against a lock file, empty when identical."""
out: list[str] = []
if self.proj_version != locked["proj_version"]:
out.append(f"PROJ {locked['proj_version']} -> {self.proj_version}")
if self.epsg_version != locked["epsg_version"]:
out.append(f"EPSG db {locked['epsg_version']} -> {self.epsg_version}")
for name, digest in locked["grids"].items():
got = self.grids.get(name)
if got is None:
out.append(f"grid MISSING: {name}")
elif got != digest:
out.append(f"grid CHANGED: {name}")
for name in sorted(self.grids.keys() - locked["grids"].keys()):
out.append(f"grid ADDED: {name}")
return out
def verify(lock_path: Path) -> None:
"""Fail fast, before any coordinate is transformed."""
locked = json.loads(lock_path.read_text())
problems = DataDependencies.capture().diff(locked)
if problems:
raise RuntimeError(
"transformation data dependencies differ from the lock file:\n "
+ "\n ".join(problems)
+ "\nRe-baseline deliberately (and re-run the golden vectors) if this "
"change is intended."
)Parameter Reference
| Field | Type | Note |
|---|---|---|
proj_version |
str |
From the library, not from the wheel metadata |
epsg_version |
str |
Ships separately; changes operations and accuracies |
grids |
dict[str, str] |
Filename to SHA-256; the values that move coordinates |
proj_data_dir |
str |
Recorded for diagnosis, deliberately not compared |
verify() |
— | Raises before the first transformation runs |
Worked Example
from pathlib import Path
lock = Path("proj.lock.json")
# Once, deliberately, when the environment is known good:
DataDependencies.capture().to_lock(lock)
# At the start of every run and every CI job:
verify(lock)
# RuntimeError: transformation data dependencies differ from the lock file:
# PROJ 9.3.1 -> 9.4.0
# grid CHANGED: ca_nrc_NA83SCRS.tif
# Re-baseline deliberately (and re-run the golden vectors) if this change is intended.That message is the whole value of the exercise. Without it, the same upgrade produces coordinates a centimetre different from last month’s deliverable and no signal at all.
Figure — verify before the first coordinate, re-baseline as a deliberate commit.
Validation Check
def assert_required_grids_present(lock_path: Path, required: set[str]) -> None:
"""The grids an operation needs must be in the lock, not merely on disk."""
locked = json.loads(lock_path.read_text())
missing = required - locked["grids"].keys()
assert not missing, (
f"operations in this pipeline need grids that are not pinned: "
f"{sorted(missing)} — a run without them silently uses a fallback"
)Common Mistakes
Pinning the Python package and calling it done. The wheel pins a PROJ build; it does not pin the grid files, which are downloaded at first use in a typical setup and are therefore whatever the network provided that day. Grid files are data dependencies and need the same treatment as code dependencies.
Letting the grid cache populate itself on demand. On-demand fetching is convenient and makes the result depend on when the job ran and whether the network was up. Provision the grids explicitly, check them against the lock, and treat a fetch during a production run as a failure rather than a feature.
Re-baselining silently. Regenerating the lock file because CI went red converts a diagnostic into a rubber stamp. Re-baselining is legitimate and should be its own commit, with the golden vectors re-run in the same change and the reason recorded — the discipline described in writing golden vector tests for coordinate transforms.
What a Version Change Actually Does to the Numbers
Knowing the shape of each kind of change makes the diff readable when the lock file finally goes red.
Figure — typical coordinate movement by kind of dependency change.
A PROJ patch release typically moves nothing, or moves the last one or two digits of a projected coordinate — micrometres. A minor release can refine a series expansion, which shows up as a uniform change of a fraction of a millimetre across every projected vector and nothing at all for pure datum shifts.
An EPSG database update changes results only when it changes which operation is selected: a new, more accurate operation appears, or one you were using is deprecated. When it does bite, the change is a jump of centimetres to decimetres confined to the region the operation covers, which is the signature that distinguishes it from a library refinement.
A grid re-issue moves coordinates directly, by whatever the agency changed — usually centimetres, occasionally more in areas that were re-surveyed. It is confined to the grid’s extent and can be spatially structured, which makes it the most disruptive of the three for a dataset that spans a boundary.
A grid disappearing is the most dangerous, because it is not a change in the numbers so much as a change in the method: the operation that needed it drops out of the candidate list and a less accurate fallback runs. The coordinates move by the difference between the two methods, typically decimetres, and nothing in the output says a different method was used. That is the case the lock file exists for.
Frequently Asked Questions
Should the lock file be committed to the repository?
Yes. It is a description of the environment the code is known to behave correctly in, which is exactly the kind of thing version control exists for, and its history becomes the record of when and why the numbers moved. Keep the grid files out of the repository unless they are small; keep their checksums in it always.
How do I pin the environment when the deployment target is not a container?
Record and verify rather than control: capture the lock, run verify at startup, and fail loudly on a mismatch. You cannot stop an operator upgrading a system package, but you can guarantee that a run under a changed environment stops rather than quietly producing different coordinates.
Does this make upgrades harder?
It makes them visible, which is different. The upgrade path becomes: bump the dependency, run the suite, inspect the golden-vector differences, decide whether they are a series refinement or a changed operation, re-baseline, commit. That is more work than not noticing, and considerably less than reconciling two deliverables that disagree by a centimetre for reasons nobody recorded.
What about grids provisioned by a system package manager?
Treat the package version as part of the environment record and still checksum the files, because distributions repackage and patch. The checksum is the ground truth; the package version is useful context for a human reading the diff.
Should the lock cover optional grids that only some regions need?
Yes, and mark them as such. A grid that is only needed for one region still changes operation selection when it is present, so leaving it out of the lock means two machines can legitimately differ. Recording it with a note about which region it serves keeps the set explainable, which matters when somebody later asks why a several-gigabyte download is part of the build.