Testing and CI for Coordinate Transformation Code

Transformation code fails in a way ordinary software does not: it keeps running, keeps returning coordinates, and the coordinates are wrong by centimetres. No exception is raised, no test times out, and the defect reaches a deliverable. This topic, part of batch transformation and automation, is about the test strategy that catches that class of failure — golden vectors that pin the numbers, property tests that explore the space no fixture covers, pinned data dependencies so a green build stays green for a reason, and a CI configuration that provisions the grid files a real transformation needs.

The premise throughout is that the correctness of a coordinate transformation is not a property of the code alone. It is a property of the code, the PROJ release, the EPSG database version and the grid files present on disk — and a test suite that pins only the first of those is testing a quarter of the system.

What Actually Has to Be Tested #

Four categories, in descending order of how often they catch something real.

Figure — four categories, and the failure each one is the only way to catch.

Numerical agreement. A known input must produce a known output to a stated tolerance. This is what golden vectors do, and it is the only category that catches a silently changed operation.

Invariants. Properties that must hold for every input rather than for one: a round trip closes, a transformation is deterministic across runs, results do not depend on the worker count, row order is preserved. These are what property-based tests explore, and they find the inputs nobody thought to write a fixture for.

Refusals. The pipeline must reject what it cannot do correctly — a missing grid, an out-of-extent coordinate, an absent epoch on a time-dependent frame, an ensemble CRS code where a realisation is required. A test suite that only tests the happy path leaves every guard unverified, and guards are exactly the code that never runs in development.

Environment. The PROJ version, the EPSG database version and the checksums of the grid files are part of the result. A test that asserts them turns “the numbers changed and nobody knows why” into a failing build with a diff.

A Layered Suite #

Figure — runtime by layer: everything except the last should finish before you look away.

The whole suite except the last layer should run in under a few seconds, because a suite that takes minutes is a suite that gets skipped before a deploy. The end-to-end layer earns its cost by exercising the parts that only exist at scale — partitioning, worker counts, audit assembly — and belongs on every merge rather than every save.

Production Implementation: the Environment Assertion #

from __future__ import annotations

import hashlib
from dataclasses import dataclass
from pathlib import Path

import pyproj


@dataclass(frozen=True)
class Environment:
    """The parts of the environment that change transformation results."""

    proj_version: str
    epsg_database_version: str
    grid_checksums: dict[str, str]

    @classmethod
    def capture(cls, grid_dir: Path) -> "Environment":
        """Read the environment as it actually is, not as configuration claims."""
        checksums = {}
        for path in sorted(grid_dir.glob("*")):
            if not path.is_file():
                continue
            digest = hashlib.sha256(path.read_bytes()).hexdigest()
            checksums[path.name] = digest
        return cls(
            proj_version=pyproj.proj_version_str,
            epsg_database_version=pyproj.database.get_database_metadata("EPSG.VERSION"),
            grid_checksums=checksums,
        )

    def assert_matches(self, expected: "Environment") -> None:
        """Fail the build with a diff rather than letting the numbers move silently."""
        problems: list[str] = []
        if self.proj_version != expected.proj_version:
            problems.append(
                f"PROJ {expected.proj_version} -> {self.proj_version}"
            )
        if self.epsg_database_version != expected.epsg_database_version:
            problems.append(
                f"EPSG db {expected.epsg_database_version} -> "
                f"{self.epsg_database_version}"
            )
        for name, digest in expected.grid_checksums.items():
            got = self.grid_checksums.get(name)
            if got is None:
                problems.append(f"grid missing: {name}")
            elif got != digest:
                problems.append(f"grid changed: {name}")
        for name in self.grid_checksums.keys() - expected.grid_checksums.keys():
            problems.append(f"grid added: {name}")
        if problems:
            raise AssertionError(
                "transformation environment changed; re-baseline the golden "
                "vectors deliberately:\n  " + "\n  ".join(problems)
            )

The message matters as much as the assertion. A build that fails with “PROJ 9.3.1 → 9.4.0, grid changed: ntv2_0.gsb” tells the next engineer exactly what to do; one that fails with “expected 0.0231, got 0.0244” sends them looking for a bug in their own code that is not there.

Precision and Tolerances in Tests #

The row worth arguing about is the second. A golden vector should be exact while the environment is pinned, because that is what pinning is for; loosening it to a millimetre “to be safe” hides precisely the silent operation change the vector exists to catch. Loosen the tolerance only when the environment assertion has already told you the environment moved.

Structuring the Suite so Failures Are Legible #

A transformation test suite is read far more often when it is red than when it is green, so its structure should make a failure self-explaining. Three conventions do most of that work.

Name the operation, not the function. A test called test_transform_works tells a reviewer nothing; test_nad27_to_nad83_conus_via_nadcon names the operation whose behaviour changed, which is the first thing anyone needs to know. When a grid is re-issued and four tests go red, their names should immediately say that all four use the same grid.

Separate the layers into separate files or markers, so a run can be scoped. When the environment assertion fails, every golden vector fails with it, and a suite that cannot be told “run only the environment checks” buries the one informative failure under fifty derived ones. Marking the layers also lets CI run the fast three on every push and the end-to-end layer on merge.

Assert one thing per test, and put the context in the message. A golden-vector assertion that reports the operation identifier, the grid checksum and the difference in millimetres turns a failing build into a diagnosis. The environment assertion above does this deliberately; the same discipline applied to the numerical tests is what keeps the suite from being an oracle nobody trusts.

import pytest

pytestmark = pytest.mark.golden          # a marker per layer, so CI can scope runs


@pytest.mark.parametrize("case", GOLDEN_CASES, ids=lambda c: c.name)
def test_golden_vector(case, environment):
    """One case per operation; the id names the operation, not the index."""
    got = case.run()
    delta_mm = abs(got - case.expected) * 1000.0
    assert delta_mm.max() <= case.tolerance_mm, (
        f"{case.name}: {delta_mm.max():.4f} mm > {case.tolerance_mm} mm "
        f"(operation {case.operation_id}, grid {case.grid_sha256[:12]}, "
        f"PROJ {environment.proj_version})"
    )

The environment fixture in that signature is doing quiet work: it makes every numerical test depend on the environment assertion, so a changed PROJ release fails once with an explanation rather than fifty times with arithmetic.

What a Regression Actually Looks Like #

It is worth knowing the shapes, because they are recognisable and each points at a different cause.

Figure — four regression signatures, each pointing at a different cause.

A uniform offset across every vector in one region, of a few centimetres, is a changed operation — usually a grid that was re-issued or a fallback that engaged because a grid went missing. The environment assertion catches it first if it is in place.

A change in the last two or three digits only, across all vectors everywhere, is a PROJ release refining a series expansion. It is not a defect, and the correct response is to re-baseline deliberately and record the version the new baseline belongs to.

A failure in one vector while its neighbours pass is usually a boundary case: a point that has moved across a sub-grid edge, past a grid extent, or onto a cell with a no-data corner. These are the vectors worth having, and they are the ones a fixture set assembled from convenient coordinates will not contain.

A result that differs between runs on the same machine is nondeterminism, and in a pipeline that must be bitwise reproducible it is the most serious of the four — threading in a BLAS backend, dictionary iteration order reaching an output, or a set traversal deciding which of two equally accurate operations was chosen.

Compliance: Tests as Evidence #

For a cadastral pipeline the test suite is part of the audit story rather than a private engineering convenience. Three artefacts are worth publishing alongside a deliverable: the environment record the run executed under, the golden-vector results with their tolerances, and the coverage of the refusal tests. Together they answer the question a reviewer asks when a boundary is challenged years later — not “is your code good” but “can you show that the software that produced this file behaved as documented on inputs whose answers were known”.

Running Real Transformations in CI #

The awkward part of this whole area is that a meaningful test needs PROJ, an EPSG database and grid files, none of which are pure Python and all of which change the answers. Three arrangements are in common use, and they differ in what they guarantee.

A container image with everything pinned is the strongest: PROJ version, EPSG database and grids are all baked in, and the image digest is the single thing that has to be recorded for the environment to be reproducible. It is also the heaviest, and the image has to be rebuilt deliberately when a dependency moves — which is the point rather than a drawback.

A lock file plus a provisioning step pins the Python side exactly and the native side approximately, then downloads grids to a cache keyed by checksum. It is lighter and it depends on the grid source remaining available, so the cache should be treated as a build artefact rather than as a transient.

Whatever the runner happens to have is the arrangement that produces a green build for reasons nobody can state. It is worth naming because it is the default: a CI configuration that installs a geospatial stack without pinning is this arrangement whether or not anyone chose it.

Whichever is used, the environment assertion from earlier belongs at the front of the suite, because it converts a silent drift into a labelled failure. The provisioning mechanics are in pinning PROJ and grid versions for reproducible builds, and the CI-specific checks that guard a cadastral pipeline are collected in continuous integration checks for cadastral pipelines.

Failure Modes #

• Golden vectors generated by the code under test. A fixture produced by running the implementation records what it did, not what it should do. Derive vectors from an independent source — published control, an agency’s worked example, a second implementation.
• Grid files absent in CI. Tests that quietly exercise a fallback operation instead of the intended grid pass while testing something else entirely. Assert the grid is present, as pinning PROJ and grid versions for reproducible builds sets out.
• Only the happy path tested. Every guard — extent, sentinel, epoch, tolerance, ensemble CRS — needs a test that proves it refuses. Guards are the code most likely to be wrong, because they are least often executed.
• Tolerances chosen to make tests pass. A tolerance is a claim about the numerical behaviour of the operation. Deriving it backwards from an observed failure converts the test into a record of current behaviour.
• Nondeterminism accepted. A test that passes intermittently is reporting real nondeterminism — threading, iteration order, an unpinned dependency — and in a pipeline that must be bitwise reproducible it is a defect, not a flaky test.

Testing the Refusals #

The guards in a cadastral pipeline are the code least likely to run in development and most likely to be wrong, so they deserve tests of their own rather than a footnote in the happy-path suite. Five refusals are worth asserting explicitly, and each has a natural fixture that costs a few lines.

A coordinate outside every grid extent must raise rather than return a clamped or extrapolated shift. A cell with a no-data corner must raise rather than average the sentinel in. A time-dependent frame with no epoch must be rejected at ingress rather than defaulted to the current date. An ensemble CRS code where a realisation is required must be refused, because it hides up to two metres. And a batch whose best available operation misses the tolerance must fail the batch rather than emit degraded coordinates with a warning.

import pytest


@pytest.mark.parametrize("case,expected_message", [
    (outside_extent_case, "outside the grid extent"),
    (null_corner_case, "unmodelled"),
    (no_epoch_case, "require a coordinate epoch"),
    (ensemble_crs_case, "ensemble"),
    (tolerance_miss_case, "exceeds"),
])
def test_pipeline_refuses(case, expected_message):
    """Each guard must refuse, and say why in terms the operator can act on."""
    with pytest.raises(ValueError, match=expected_message):
        case.run()

Matching on the message rather than only on the exception type is deliberate. A guard that raises the right exception with an unhelpful message passes a type-only test and still costs an operator an afternoon, and messages drift as code is refactored unless something holds them in place.

What This Buys, Concretely #

The argument for all of this is not general engineering hygiene; it is that four specific failures become impossible to ship unnoticed. A grid quietly missing, so a fallback ran. An operation silently reselected after a database update. A guard that stopped guarding during a refactor. A result that depends on how the work was divided across workers. Each of those produces coordinates that look entirely ordinary, and each is caught by one cheap layer of the suite described above.

Frequently Asked Questions #

How many golden vectors are enough? #

Enough to cover the operations and the geography the pipeline actually serves: one per coordinate operation, plus one near each grid boundary, one in a nested sub-grid, one outside every grid, and one at an extreme of the projection zone. That is usually a dozen or two, and their value comes from their spread rather than their count — twenty vectors clustered near the same origin test one cell of one grid.

Should tests hit the network to fetch grid files? #

No. A test suite that downloads its data depends on a remote service being up and unchanged, which turns an unrelated outage into a red build and a silent grid revision into a mystery. Vendor the grids, check their checksums, and treat an update as a deliberate change with its own commit.

What belongs in CI versus in a pre-commit hook? #

Fast, deterministic and dependency-free checks — linting, unit tests of pure arithmetic — belong in a hook where they cost seconds. Anything needing PROJ, the EPSG database or grid files belongs in CI, where the environment is pinned and reproducible. The distinction is not importance but reproducibility: a hook runs in whatever environment a developer happens to have.

How do I test code that needs a grid file too large to vendor? #

Synthesise one. A small, hand-built grid in the same format, with known values and a known no-data node, tests the reader, the interpolation, the extent logic and the sentinel handling completely — and it can be committed. Keep one real-file test for format compatibility and run it against a vendored subset or a locally provisioned copy.

Related #

• Batch transformation and automation — the parent section
• Writing golden vector tests for coordinate transforms
• Pinning PROJ and grid versions for reproducible builds
• Property-based testing for round-trip transformations
• Generating audit hashes for transformation batches