CoordTransform Cadastral & High-Precision Python

Provisioning PROJ Grids for Offline Environments

A transformation pipeline that downloads its grid files on demand has an accuracy that depends on the network, and a pipeline behind an air gap has no accuracy at all until someone puts the grids there deliberately. This guide, part of fallback routing strategies for missing grid files, provisions grids explicitly: deciding which files an operation actually needs, staging them with verified checksums, and failing loudly when the set on disk is not the set that was pinned.

Which Grids an Operation Needs

The question is answerable before anything runs. For a given CRS pair, the candidate coordinate operations each name the grid files they require, and the ones whose grids are absent are excluded from the usable set — the mechanism described in automating datum fallback chains in pyproj. Enumerating those requirements up front turns provisioning from a guess into a list.

Assembling a pinned grid set from the operations that need it The CRS pairs the pipeline actually transforms between are enumerated first. For each pair the candidate operations are listed, including the ones whose grids are absent, because those name the files that would improve accuracy. Each operation is then accepted or declined deliberately — provisioning a grid changes which operation runs, so the set is a statement about method. The accepted grids are checksummed into the lock, and the decision is recorded alongside. CRS pairs the pipeline uses Candidate operations available AND unavailable Accept or decline each this decides the METHOD Checksum into the lock + record why

Figure — from a CRS pair to a pinned grid set, deciding rather than accumulating.

Two subtleties are worth naming. A grid may be required by an operation you never intend to use, and provisioning it changes which operation is selected — so the provisioned set is part of the pipeline’s behaviour, not an optimisation. And an operation can name several grids, of which only some are needed for your geography; a national pipeline usually wants the whole set anyway, because a partial set produces different results in different regions.

Complete Runnable Implementation 

from __future__ import annotations

import hashlib
import json
from dataclasses import dataclass
from pathlib import Path

from pyproj import CRS
from pyproj.transformer import TransformerGroup


@dataclass(frozen=True)
class GridRequirement:
    """One grid file an operation needs, and whether it is present."""

    filename: str
    operation: str
    accuracy_m: float | None
    available: bool


def required_grids(source: str, target: str,
                   area_of_interest=None) -> list[GridRequirement]:
    """Enumerate the grids the candidate operations for this pair would use."""
    group = TransformerGroup(CRS.from_user_input(source), CRS.from_user_input(target),
                             always_xy=True, area_of_interest=area_of_interest)
    out: list[GridRequirement] = []
    for available, ops in ((True, group.transformers),
                           (False, group.unavailable_operations)):
        for op in ops:
            info = getattr(op, "operations", None) or [op]
            for step in info:
                for grid in getattr(step, "grids", []) or []:
                    out.append(GridRequirement(
                        filename=getattr(grid, "short_name", str(grid)),
                        operation=getattr(op, "description", str(op)),
                        accuracy_m=getattr(op, "accuracy", None),
                        available=available,
                    ))
    return out


def stage(files: dict[str, Path], target_dir: Path, expected: dict[str, str]) -> None:
    """Copy grids into the PROJ data directory, verifying each one.

    Verification happens BEFORE the copy: a grid that fails its checksum must not
    reach the directory PROJ reads, or the next run silently uses it.
    """
    target_dir.mkdir(parents=True, exist_ok=True)
    for name, src in sorted(files.items()):
        want = expected.get(name)
        if want is None:
            raise ValueError(f"{name} is not in the pinned set; add it deliberately")
        digest = hashlib.sha256(src.read_bytes()).hexdigest()
        if digest != want:
            raise ValueError(
                f"{name} checksum {digest[:12]} != pinned {want[:12]}; refusing to "
                f"stage a grid that is not the one this pipeline was validated with"
            )
        (target_dir / name).write_bytes(src.read_bytes())


def audit_data_dir(target_dir: Path, expected: dict[str, str]) -> list[str]:
    """Differences between the directory and the pinned set, empty when identical."""
    problems: list[str] = []
    present = {p.name: p for p in target_dir.iterdir() if p.is_file()}
    for name, want in expected.items():
        path = present.get(name)
        if path is None:
            problems.append(f"missing: {name}")
            continue
        got = hashlib.sha256(path.read_bytes()).hexdigest()
        if got != want:
            problems.append(f"changed: {name}")
    for name in sorted(present.keys() - expected.keys()):
        problems.append(f"unpinned grid present: {name}")
    return problems

Parameter Reference

Name Type Note
source, target str CRS identifiers; the pair whose operations are enumerated
area_of_interest object or None Narrows the candidate list to your geography
expected dict[str, str] Filename to SHA-256 — the pinned set
stage() — Verifies before copying, never after
audit_data_dir() list[str] Includes unpinned files, which change selection

Worked Example 

for req in required_grids("EPSG:4267", "EPSG:6318"):
    mark = "ok " if req.available else "MISSING"
    acc = f"{req.accuracy_m:.3f} m" if req.accuracy_m is not None else "unstated"
    print(f"{mark} {req.filename:<28} {acc}")
# ok      us_noaa_conus.tif            0.150 m
# MISSING us_noaa_alaska.tif           0.500 m

The missing row is the useful one: it names a grid that would extend coverage, together with the accuracy of the operation that needs it, so the decision to provision it is made on evidence rather than on a hunch about which files might be relevant.

Three ways to provision grids, and what each guarantees Three arrangements. A container image with everything baked in pins the library, the database and the grids under one digest and is the strongest guarantee. A lock file plus a verified cache pins the Python side exactly and the native side approximately, and depends on the grid source remaining reachable at provisioning time. Whatever the runner happens to have guarantees nothing, and is the default arrangement whenever no one has chosen otherwise. Pins Guarantee Container image everything image digest Lock + verified cache most of it lock verification Whatever is installed nothing none

Figure — three provisioning arrangements and what each one guarantees.

Validation Check 

def assert_data_dir_matches_lock(target_dir: Path, lock_path: Path) -> None:
    """Run before the first transformation, not after the batch."""
    expected = json.loads(lock_path.read_text())["grids"]
    problems = audit_data_dir(target_dir, expected)
    assert not problems, (
        "the PROJ data directory does not match the pinned set:\n  "
        + "\n  ".join(problems)
    )

Common Mistakes

Leaving on-demand fetching enabled in production. It converts a missing grid into a network request, so the pipeline’s accuracy depends on connectivity and on whatever the remote currently serves. Disable it in production and let a missing grid be an error the provisioning step already prevented.

Accuracy cost when a grid is missing and a fallback runs Bar chart of typical horizontal accuracy in metres for the operation that runs when the intended grid is absent: the NTv2 grid itself at 0.015, a NADCON pair at 0.05, a published Helmert set at 0.30 and a geocentric translation at 1.00. A missing grid does not degrade the result slightly; it changes the method, and the change is invisible in the output. 0.01 0.1 1 m 0.015 NTv2 (intended) 0.05 NADCON 0.3 Helmert 1 translation

Figure — what a missing grid costs, by which fallback engages instead.

Provisioning extra grids “just in case”. An unpinned grid on disk changes which operation is selected, so a machine with a larger grid set produces different — usually better, but different — coordinates than one without it. That is why audit_data_dir reports unpinned files as problems rather than ignoring them.

Verifying after copying. A grid staged and then checked has already been readable by PROJ for the intervening moment, and in a long-running process that is enough for it to be cached. Verify first, copy second.

Building the Pinned Set the First Time

Assembling the initial set is a one-off exercise and worth doing systematically rather than by adding files until the errors stop.

Start from the CRS pairs the pipeline actually transforms between — usually a handful, not the whole registry. For each pair, enumerate the candidate operations and their grid requirements with the function above, including the unavailable ones, because those name the files that would improve accuracy if provisioned.

Then decide, per operation, whether you want it. This is the step that is easy to skip and is the whole point: provisioning a grid changes which operation runs, so the pinned set is a statement about the method, not just about the data. An operation whose accuracy is worse than one you already have is not worth enabling; one that is better is worth the disk.

Finally, record the decision alongside the checksums — which operations the set enables, which it deliberately excludes, and why. Six months later that note is the only thing standing between a colleague and the reasonable-looking conclusion that a missing grid was an oversight.

A useful sanity check at the end: transform a handful of points spread across the working area and confirm that the operation actually selected is the one the pinned set was assembled to enable. It takes a minute, and it catches the case where a grid is present but not found, which produces exactly the same silent fallback as a grid that was never provisioned at all.

Frequently Asked Questions

How large is a realistic grid set?

For one country, usually tens of megabytes; for a global set, several gigabytes. That range is why “just install everything” is a defensible answer for a national pipeline and a poor one for a container image that has to ship. Enumerate what the operations actually need and pin that.

Can I share a grid directory between projects?

You can, and it couples them: a project that adds a grid changes which operation the other project selects. Where two pipelines have to be independently reproducible, give each its own directory and its own pinned set, and treat the disk cost as the price of independence.

What about grids served over a network protocol?

Convenient for exploration and unsuitable for a deliverable pipeline: the result then depends on a remote service’s availability and contents at run time. If network access is the only practical distribution route, fetch once into a verified local cache during provisioning, then run with remote access disabled.

How do I know a grid file is the one the agency published?

The checksum, compared against the agency’s published value where they publish one and against your own first-download value otherwise, recorded at the moment you validated the pipeline. That is the same argument as pinning PROJ and grid versions for reproducible builds makes for the library itself, applied to the data it reads.

How should a container image handle the grid set?

Bake it in and record the image digest, which then pins the library, the database and the grids in one identifier. Mounting the grids at run time is more flexible and gives away the property that makes the image worth using: the guarantee that what ran in CI is what ran in production. Where image size is genuinely prohibitive, mount a read-only volume whose contents are verified against the lock at startup.