Compliance Report Generation for Agency Submission

Every transformed cadastral dataset that leaves your pipeline for a receiving authority carries an obligation that the numbers alone cannot discharge: the submission has to prove how it was produced, to what accuracy, and that it has not been altered since. This page — part of Batch Transformation & Automation for Cadastral Coordinate Pipelines — builds the audit-ready deliverable that travels with a transformed batch: the transformation provenance (source and target CRS, the EPSG coordinate-operation code, the grid file and its version), the per-point and batch horizontal RMSE against certified control, a cryptographic audit hash that makes the package tamper-evident, and ISO 19111-style operation metadata — all emitted as one structured report in machine-readable JSON alongside a human-readable summary. The three concerns that make up the bulk of that report each get their own dedicated treatment: the deterministic hashing is built in generating audit hashes for transformation batches, the accuracy gate in the RMSE-to-agency-submission workflow in Python, and the standards-conformant CRS description in exporting ISO 19111 metadata for cadastral deliverables. Here we assemble those parts into a single defensible object.

Figure — the transformed batch, its provenance, accuracy, and hash converge into one report that either clears the tolerance gate and submits, or is returned for rework.

What an Agency Actually Requires #

A cadastral filing authority does not accept a bare list of coordinates. The deliverable has to answer three independent questions, and each maps to a distinct part of the report:

• Traceability. From which realized datum, through which coordinate operation, to which target frame were these points moved? A receiving reviewer must be able to reconstruct the exact pipeline: the EPSG code of the operation (for example the NAD83(2011) to NAD83(CSRS) family), the grid shift file consumed, and — critically — the version of that grid, because a grid revision silently changes results at the millimetre level.
• Accuracy. How well do the transformed coordinates agree with an independent, certified truth? This is the horizontal root-mean-square error against control monuments, reported both per point (so an outlier is visible) and as a single batch figure compared to the agency’s tolerance gate.
• Integrity. Can anyone prove the file was not edited after certification? A cryptographic digest computed over the canonicalised inputs, parameters, and outputs makes any post-hoc byte change detectable.

ISO 19111 — the geographic-information standard for referencing by coordinates — gives the vocabulary for the first of these. Its model separates the coordinate reference systems from the coordinate operation that maps one to the other, and it treats operation accuracy as a first-class attribute of that operation. A compliant report therefore names the source CRS, the target CRS, and the operation between them as three separate, authority-coded objects rather than a single opaque string. The datum-alignment evidence that underpins the accuracy claim is established separately in validating datum alignment with control points; this page consumes that evidence and packages it.

Report Structure and the Accuracy Statistic #

The single number a reviewer looks at first is the batch horizontal RMSE. For $n$ transformed points with easting/northing residuals against their matched control monuments, it is

where each squared term is the full planar separation of a point from its control, so a single point contributes both its easting and northing miss. This is a two-dimensional RMS: dividing the summed squared distances by $n$, not by $2n$. The report stores the per-point residual $r_i = \sqrt{(E_i - E_i^{\text{ctrl}})^2 + (N_i - N_i^{\text{ctrl}})^2}$ so outliers survive into the record, and the aggregate is compared to a tolerance $\tau$ — a 20 mm horizontal gate is typical for cadastral submission. The pass condition is simply $\text{RMSE}_H \le \tau$.

The integrity field is a digest $h = \text{SHA256}(c)$ over a canonical byte stream $c$ that concatenates the serialized inputs, the transformation parameters, and the outputs. Canonicalisation — sorted keys, fixed float formatting, explicit array dtype and endianness — is what makes $h$ reproducible on a reviewer’s machine; that determinism is the whole subject of the audit-hash page and we treat it here as a trusted primitive.

Step-by-Step Implementation #

We build a ComplianceReport dataclass to hold the finished evidence, then a generator that assembles it from a transformed batch. Every step is a runnable, type-hinted unit.

Step 1 — Model the provenance #

Provenance is immutable once recorded, so a frozen dataclass is the right container. The grid version is a mandatory field, not an optional one — omitting it is the single most common reason a submission is bounced.

from __future__ import annotations

import hashlib
import json
import logging
from dataclasses import dataclass, asdict, field
from datetime import datetime, timezone

import numpy as np

logger = logging.getLogger("compliance")


@dataclass(frozen=True)
class TransformationProvenance:
    """Traceability block: how the batch was transformed (ISO 19111 sense)."""
    source_crs: str          # authority code, e.g. 'EPSG:4269' (NAD83)
    target_crs: str          # authority code, e.g. 'EPSG:6318' (NAD83(2011))
    operation_code: str      # EPSG coordinate-operation code, e.g. 'EPSG:1888'
    operation_method: str    # e.g. 'NTv2' or 'Helmert 7-parameter'
    grid_file: str | None    # grid shift filename, or None for a parametric op
    grid_version: str | None # grid revision/edition — mandatory when grid_file set

    def __post_init__(self) -> None:
        # ISO 19111 ties accuracy to a specific operation realisation; a grid
        # without its version is not a reproducible operation.
        if self.grid_file is not None and not self.grid_version:
            raise ValueError(
                f"grid_file {self.grid_file!r} supplied without grid_version"
            )

Step 2 — Compute per-point and batch residuals #

The accuracy block reduces the transformed and control arrays to residuals. We keep the per-point vector so the report can flag which monument failed, and derive the two-dimensional batch RMSE from it.

@dataclass(frozen=True)
class AccuracyStatement:
    """Accuracy block: per-point residuals and the batch RMSE gate result."""
    n_points: int
    per_point_residual_m: list[float]  # planar separation per point (m)
    rmse_h_m: float                    # batch horizontal RMSE (m)
    tolerance_m: float                 # agency gate (m)
    passed: bool


def build_accuracy(
    transformed_en: np.ndarray,   # shape (n, 2): easting, northing (m)
    control_en: np.ndarray,       # shape (n, 2): certified monument E/N (m)
    tolerance_m: float = 0.020,
) -> AccuracyStatement:
    """Compute per-point residuals and the two-dimensional batch RMSE."""
    if transformed_en.shape != control_en.shape:
        raise ValueError("transformed and control arrays must match in shape")
    diff = transformed_en.astype(np.float64) - control_en.astype(np.float64)
    per_point = np.hypot(diff[:, 0], diff[:, 1])           # planar separation
    # Two-dimensional RMS: mean of squared distances, then root (divide by n).
    rmse = float(np.sqrt(np.mean(per_point ** 2)))
    return AccuracyStatement(
        n_points=int(transformed_en.shape[0]),
        per_point_residual_m=[round(float(r), 4) for r in per_point],
        rmse_h_m=round(rmse, 4),
        tolerance_m=tolerance_m,
        passed=rmse <= tolerance_m,
    )

Step 3 — Canonicalise and hash the payload #

The integrity digest must be reproducible, so the payload is serialized with sorted keys and no incidental whitespace, and NumPy arrays are hashed from their raw bytes with the dtype pinned to little-endian float64. Note that the timestamp is deliberately excluded from the hashed stream — it is recorded beside the digest, not inside it, so re-running the pipeline reproduces the same hash.

def _canonical_array_bytes(arr: np.ndarray) -> bytes:
    """Deterministic bytes for a coordinate array: fixed dtype and endianness
    so the digest is identical on any platform."""
    canonical = np.ascontiguousarray(arr, dtype="<f8")  # little-endian float64
    return canonical.tobytes()


def audit_hash(
    inputs: np.ndarray,
    outputs: np.ndarray,
    provenance: TransformationProvenance,
) -> str:
    """SHA-256 over canonical inputs + operation parameters + outputs."""
    h = hashlib.sha256()
    h.update(_canonical_array_bytes(inputs))
    # sort_keys + compact separators = one canonical parameter serialisation.
    params = json.dumps(asdict(provenance), sort_keys=True, separators=(",", ":"))
    h.update(params.encode("utf-8"))
    h.update(_canonical_array_bytes(outputs))
    return h.hexdigest()

Step 4 — Assemble the report #

The ComplianceReport binds the three blocks with the digest and a UTC timestamp, and knows how to emit both the machine-readable JSON and a human-readable summary. The timestamp lives outside the hashed region, keeping the digest reproducible while still recording when the certification ran.

@dataclass(frozen=True)
class ComplianceReport:
    provenance: TransformationProvenance
    accuracy: AccuracyStatement
    audit_sha256: str
    generated_utc: str
    schema: str = "coordtransform.compliance/1"

    def to_json(self) -> str:
        """Machine-readable deliverable; sorted keys for stable diffs."""
        payload = {
            "schema": self.schema,
            "generated_utc": self.generated_utc,
            "audit_sha256": self.audit_sha256,
            "provenance": asdict(self.provenance),
            "accuracy": asdict(self.accuracy),
        }
        return json.dumps(payload, sort_keys=True, indent=2)

    def to_summary(self) -> str:
        """Human-readable one-screen summary for a reviewer."""
        p, a = self.provenance, self.accuracy
        verdict = "PASS" if a.passed else "FAIL"
        grid = f"{p.grid_file} (v{p.grid_version})" if p.grid_file else "n/a"
        return (
            f"Compliance report [{verdict}]\n"
            f"  operation : {p.source_crs} -> {p.target_crs} via {p.operation_code}\n"
            f"  method    : {p.operation_method}   grid: {grid}\n"
            f"  points    : {a.n_points}\n"
            f"  RMSE_H    : {a.rmse_h_m:.4f} m   (tolerance {a.tolerance_m:.3f} m)\n"
            f"  sha256    : {self.audit_sha256}\n"
            f"  generated : {self.generated_utc}"
        )


def generate_report(
    inputs_en: np.ndarray,
    outputs_en: np.ndarray,
    control_en: np.ndarray,
    provenance: TransformationProvenance,
    tolerance_m: float = 0.020,
) -> ComplianceReport:
    """Assemble provenance, accuracy and integrity into one deliverable."""
    accuracy = build_accuracy(outputs_en, control_en, tolerance_m)
    digest = audit_hash(inputs_en, outputs_en, provenance)
    report = ComplianceReport(
        provenance=provenance,
        accuracy=accuracy,
        audit_sha256=digest,
        generated_utc=datetime.now(timezone.utc).isoformat(timespec="seconds"),
    )
    logger.info(
        "compliance_report rmse=%.4f passed=%s sha256=%s",
        accuracy.rmse_h_m, accuracy.passed, digest[:12],
    )
    return report

Parameter and Return-Value Reference #

Worked Example #

Take a small batch transformed from NAD83 (EPSG:4269) to the NAD83(2011) realization (EPSG:6318) via an NTv2 grid, checked against three certified monuments. The residuals are all sub-centimetre, so the batch clears the 20 mm gate and the report certifies as a pass.

inputs = np.array([[500000.000, 5000000.000],
                   [500100.000, 5000100.000],
                   [500200.000, 5000200.000]], dtype=np.float64)
outputs = np.array([[500000.006, 5000000.004],
                    [500100.005, 5000100.007],
                    [500200.008, 5000200.003]], dtype=np.float64)
control = np.array([[500000.008, 5000000.006],
                    [500100.007, 5000100.010],
                    [500200.010, 5000200.007]], dtype=np.float64)

prov = TransformationProvenance(
    source_crs="EPSG:4269", target_crs="EPSG:6318",
    operation_code="EPSG:1888", operation_method="NTv2",
    grid_file="us_noaa_nadcon5.tif", grid_version="2021.1",
)

report = generate_report(inputs, outputs, control, prov, tolerance_m=0.020)
print(report.to_summary())
# Compliance report [PASS]
#   operation : EPSG:4269 -> EPSG:6318 via EPSG:1888
#   method    : NTv2   grid: us_noaa_nadcon5.tif (v2021.1)
#   points    : 3
#   RMSE_H    : 0.0033 m   (tolerance 0.020 m)
#   sha256    : <64 hex chars>
#   generated : <UTC timestamp>

The per-point residuals here are roughly 2.8–4.5 mm, giving a batch RMSE near 3.3 mm — comfortably inside a 20 mm gate. The digest is stable: re-running the same three arrays and provenance produces byte-identical JSON and the same sha256.

Verification and Residual Analysis #

A report is only defensible if it re-verifies independently: the digest must recompute to the stored value, and the JSON must round-trip without altering the accuracy figures. Gate both before the deliverable is released.

def verify_report(report: ComplianceReport,
                  inputs_en: np.ndarray,
                  outputs_en: np.ndarray) -> None:
    """Re-derive the digest and confirm the accuracy gate; raise on mismatch."""
    recomputed = audit_hash(inputs_en, outputs_en, report.provenance)
    if recomputed != report.audit_sha256:
        raise ValueError("audit hash mismatch — payload changed after signing")
    parsed = json.loads(report.to_json())
    assert parsed["accuracy"]["passed"] is report.accuracy.passed
    assert parsed["accuracy"]["rmse_h_m"] == report.accuracy.rmse_h_m
    logger.info("report verified sha256=%s", report.audit_sha256[:12])


verify_report(report, inputs, outputs)  # silent success = defensible package

The round-trip assertion catches the subtle failure where a report serializes a float that does not survive JSON re-parsing; the hash re-derivation catches any edit to the coordinate arrays. Extending this to a monument network rather than three points, and computing normalized residuals to flag a single suspect station, follows the residual discipline in least-squares adjustment for control networks.

Troubleshooting and Gotchas #

Frequently Asked Questions #

Related #

• Generating audit hashes for transformation batches — the deterministic SHA-256 canonicalisation this report depends on.
• RMSE-to-agency-submission workflow in Python — the accuracy gate that decides pass or rework.
• Exporting ISO 19111 metadata for cadastral deliverables — the standards-conformant CRS and operation document.
• Validating datum alignment with control points — the control-network evidence behind the accuracy claim.
• Batch Transformation & Automation for Cadastral Coordinate Pipelines — the parent reference on automating and certifying batch transformations.