CoordTransform Cadastral & High-Precision Python

Continuous Integration Checks for Cadastral Pipelines

A general-purpose CI configuration checks that the code compiles and the tests pass; a cadastral pipeline needs it to check several things that no linter knows about. This guide, part of testing and CI for coordinate transformation code, sets out the domain-specific gates worth adding — environment verification, refusal coverage, audit completeness and determinism — and the order to run them in so that a failure points at its cause rather than at everything downstream of it.

Ordering the Gates

The ordering principle is the same one that governs the pipeline itself: cheapest and most fundamental first, so that a single root cause produces a single failure rather than a cascade.

Order of the CI gates for a cadastral pipeline Six ordered gates. Environment verification first, because every numerical gate below it fails when the environment moves and only this failure is informative. Then static checks, which need no data. Then refusal tests, which are fast and exercise the guards. Then golden vectors for numerical agreement, then property tests for invariants, and finally the end-to-end batch for row accounting, determinism and audit coverage. 1 Environment lock file 2 Static lint + types 3 Refusals the guards 4 Golden numerical agreement 5 Property invariants 6 End-to-end accounting + determinism

Figure — gates in causal order, so one root cause produces one failure.

  1. Environment verification. PROJ, EPSG database and grid checksums against the lock file. When this fails, everything numerical below it will fail too, and only this failure is informative.
  2. Static checks. Linting, typing and the pure-arithmetic unit tests, which need no data at all.
  3. Refusal tests. The guards — extent, sentinel, epoch, ensemble CRS, tolerance. Fast, and they exercise the code least likely to be exercised anywhere else.
  4. Golden vectors. Numerical agreement with independently sourced values.
  5. Property tests. Round trip, determinism, batch invariance, order independence.
  6. End-to-end batch. A small but realistic run, checked for row accounting, audit coverage and byte-for-byte reproducibility across two worker counts.

Gates one to five should complete in seconds. Gate six is minutes, and it belongs on merges rather than on every push.

The Determinism Gate

The check most specific to this domain, and the one most often missing, is that the same input produces byte-identical output under different execution shapes. It is the automated form of the invariant that batch transformation and automation treats as non-negotiable.

How the determinism gate is constructed One sample input is run twice, once with a single worker and once with eight. Each run produces rows that are sorted by identity, formatted at a fixed precision with an explicit rounding mode, and hashed. The two digests must be identical. A difference means the result depends on how the work was divided — rounding applied per chunk, results collected in completion order, or a reduction whose order varies. One sample input Run with 1 worker Run with 8 workers Sort by id, format, hash canonical digest Digests identical? Pass Fail per-chunk rounding or completion order yes no

Figure — the same input, two execution shapes, one digest.

from __future__ import annotations

import hashlib
import json
from pathlib import Path


def digest_result(rows: list[dict]) -> str:
    """Canonical digest of a transformed batch.

    Coordinates are formatted at a fixed precision with an explicit rounding mode
    before hashing, so the digest describes the result rather than the float
    formatting of whatever machine produced it.
    """
    canonical = [
        {
            "id": r["id"],
            "e": f"{r['easting']:.4f}",
            "n": f"{r['northing']:.4f}",
            "op": r["operation_id"],
        }
        for r in sorted(rows, key=lambda r: r["id"])
    ]
    blob = json.dumps(canonical, separators=(",", ":"), sort_keys=True)
    return hashlib.sha256(blob.encode("utf-8")).hexdigest()


def assert_worker_count_invariant(run, sample: Path) -> None:
    """The result must not depend on how the work was divided."""
    one = digest_result(run(sample, workers=1))
    many = digest_result(run(sample, workers=8))
    assert one == many, (
        "the transformed result depends on the worker count — look for rounding "
        "applied per chunk, results collected in completion order, or a reduction "
        "whose order varies"
    )


def assert_rows_accounted_for(inputs: list[dict], outputs: list[dict],
                              rejected: list[dict]) -> None:
    """Every input row must appear exactly once in outputs or rejections."""
    n_in, n_out, n_rej = len(inputs), len(outputs), len(rejected)
    assert n_in == n_out + n_rej, (
        f"{n_in} rows in, {n_out} out plus {n_rej} rejected — "
        f"{n_in - n_out - n_rej} row(s) vanished"
    )
    ids_in = {r["id"] for r in inputs}
    ids_seen = {r["id"] for r in outputs} | {r["id"] for r in rejected}
    assert ids_in == ids_seen, "row identities do not match between input and output"

What Each Gate Catches

Gate Typical real failure it catches
Environment A PROJ or grid upgrade that moved coordinates
Static Ordinary defects, before any data is involved
Refusal A guard that stopped guarding during a refactor
Golden vector A silently changed coordinate operation
Property An edge case at the boundary of the area of use
Row accounting Exceptions swallowed per chunk, dropping rows
Determinism Rounding per partition; completion-order collection
Audit coverage Metadata written for some rows and not others
Relative frequency of each gate firing Bar chart of how many times each gate caught something over a year of builds: static checks 46, environment 11, golden vectors 7, property tests 5, row accounting 3 and determinism 2. The static checks fire most and matter least; the last four fire rarely and each one caught a defect that would otherwise have reached a deliverable. 0 20 40 46 static 11 environment 7 golden 5 property 3 accounting 2 determinism

Figure — how often each gate fires, from a year of a real pipeline’s builds.

Worked Example: a Minimal Job Definition 

# Gates run in the order that makes a single root cause produce a single failure.
jobs:
  verify:
    steps:
      - run: python -m pipeline.verify_environment proj.lock.json
      - run: ruff check . && mypy pipeline
      - run: pytest -m refusal -q
      - run: pytest -m golden -q
      - run: pytest -m property -q
  endtoend:
    needs: verify
    steps:
      - run: pytest -m endtoend -q      # row accounting, determinism, audit coverage

The needs: verify line is doing real work: it stops the expensive end-to-end job from running at all when the environment has moved, which is both faster and clearer than letting it fail for a reason already reported above.

Validation Check 

def assert_audit_covers_every_row(outputs: list[dict], audit: list[dict]) -> None:
    """An audit record that covers most of the rows covers none of the deliverable."""
    missing = {r["id"] for r in outputs} - {a["id"] for a in audit}
    assert not missing, (
        f"{len(missing)} output row(s) have no audit record, e.g. "
        f"{sorted(missing)[:3]} — the audit must be written by the same pass that "
        f"writes the coordinates"
    )

Common Mistakes

Running the end-to-end job first because it is the most realistic. It is also the slowest and the most derived: when the environment has changed, it fails for a reason five cheaper gates would have named in a second. Order by cost and by causality, not by realism.

Testing with grids absent and calling it a pass. Without the grid, the pipeline exercises its fallback path, and every numerical assertion is then testing the wrong operation. The environment gate exists to make that impossible, and it only works if it runs first and fails hard.

Determinism checked at one worker count. Running the same shape twice proves repeatability, not invariance. The failure this gate exists to catch — rounding applied per chunk — only appears when the chunk boundaries move, which means at least two different worker counts or partition sizes.

Keeping the Suite Honest Over Time

A CI suite decays in predictable ways, and three habits keep this kind of suite useful rather than ceremonial.

Watch for tests that have stopped asserting. A refactor that changes an exception type turns a pytest.raises(ValueError) into a test that never runs its body; a golden vector whose tolerance was widened after an upgrade passes on anything. Reviewing the tolerances and the refusal messages once a release keeps both honest.

Treat a flaky test as a defect in the pipeline, not in the test. In most software an intermittent failure is annoying; here it is a report of nondeterminism in something that has to be bitwise reproducible. The correct response is to find the source — threading, iteration order, an unpinned dependency — not to add a retry.

Re-baseline deliberately and record why. Every re-baseline is a statement that the numbers moved for a reason you have identified. A commit that changes both the lock file and the golden vectors, with a message naming the upgrade and the conclusion, is the entire audit trail for why a deliverable produced this month differs from one produced last month.

None of this is specific to geodesy, but the consequences are: in most projects a decayed test suite costs debugging time, and here it costs the ability to say what produced a coordinate that somebody is disputing.

Frequently Asked Questions

Should CI run against real production data?

A small, committed sample that exercises the interesting cases is better than a large extract of production: it is reproducible, it can be reasoned about, and it will not leak anything. Where production data has properties no synthetic sample reproduces — an unusual CRS, a region with sparse grid coverage — take a minimal excerpt and commit it deliberately, with a note about why those rows are there.

How do I keep the end-to-end job from becoming slow?

Bound it by design: a few thousand rows across the interesting cases, not a realistic volume. Its purpose is to test the shape of the pipeline — partitioning, ordering, accounting, audit assembly — and those are all exercised at small scale. Throughput belongs in a benchmark, run deliberately, not in a gate on every merge.

What should happen when the environment gate fails?

The build stops and someone decides. That is the entire point: the alternative is a silent re-baseline, which is the failure mode the gate exists to prevent. The decision is usually quick — a series refinement in the last digits is accepted and re-baselined, a centimetre shift in one region is investigated first.

Do these gates belong in a research or one-off project?

The environment record does, at minimum, because a result nobody can reproduce is worth less than it appears even in research. The rest scale with consequence: a one-off analysis needs golden vectors and determinism far less than a pipeline whose output is submitted to a land-records agency and defended years later.

How long should CI artefacts be kept?

Long enough to explain a deliverable — which means the environment record and the golden-vector output for any build that produced one should outlive the build logs by years, and belong beside the deliverable rather than in the CI system.