CoordTransform Cadastral & High-Precision Python

Thread-Safe pyproj Transformer Caching

Reusing a pyproj.Transformer instead of rebuilding it on every call is an easy speed win until threads enter the picture, at which point sharing one transformer becomes a data-corruption bug; this page — a companion to Concurrent pyproj Transformation Pipelines in Python within the broader Batch Transformation & Automation for Cadastral Coordinate Pipelines practice — shows how to cache transformers safely by keying them on (source_crs, target_crs) and giving each thread its own instance through thread-local storage.

Thread-local Transformer cache keyed by CRS pair Three worker threads each ask a factory for a Transformer for the same source and target CRS pair. The factory keeps a separate cache per thread using thread-local storage, keyed by the normalized CRS pair. Each thread receives and reuses its own Transformer instance, so no PROJ context is ever shared across threads. Thread A Thread B Thread C thread-local factory key = (src, dst) A's own Transformer B's own Transformer C's own Transformer

Figure — one cached Transformer per thread per CRS pair; contexts are never shared.

Why a Transformer cannot be shared across threads

A pyproj.Transformer holds a PROJ context (PJ_CONTEXT) and mutable transient state used while a transform runs. That state is not guarded by a lock inside PROJ for concurrent access from multiple threads, so two threads calling .transform on the same instance at the same time can interleave writes to the shared context and produce coordinates that belong to neither call. The corruption is silent — no exception, just numbers that are subtly off — which is the worst failure mode for a survey deliverable. There are two sound remedies, and the right one depends on the workload. If the threads exist to overlap I/O (streaming grid files, fetching tiles), give each thread its own transformer via thread-local storage so no context is ever contended. If you truly must share a single instance, serialize every call behind a lock — but a lock reintroduces the very serialization you spun up threads to avoid. The choice is bounded by the GIL: for CPU-bound transform math the achievable speedup from PP threads is governed by the serial fraction ss that the GIL forces,

S(P)=1s+1−sP,S(P) = \frac{1}{s + \dfrac{1 - s}{P}},

and since Python-level array handling holds the GIL, ss stays high and threads deliver little on pure compute. Threads pay off when 1−s1 - s is dominated by GIL-releasing waits — disk and network. For genuine multi-core transform throughput, processes are the answer, as built in parallelizing coordinate transforms with multiprocessing.

Complete Runnable Implementation

The factory below stores a per-thread cache in a threading.local() object and normalizes the CRS pair into a stable string key so that CRS objects, EPSG codes, and WKT strings that denote the same system collapse to one entry. A bounded eviction keeps the per-thread cache from growing without limit when many distinct CRS pairs are requested.

from __future__ import annotations

import threading
from collections import OrderedDict

from pyproj import CRS, Transformer

# Each thread gets its own attribute namespace here; nothing is shared.
_local = threading.local()

_MAX_ENTRIES = 32   # bound per-thread cache growth across many CRS pairs


def _normalize(crs: str | int | CRS) -> str:
    """Collapse EPSG codes, CRS objects, and WKT to one stable key so that
    equivalent CRS spellings share a single cache entry (avoids key collisions
    where two spellings of the same CRS would build two Transformers)."""
    return CRS.from_user_input(crs).to_wkt()


def get_transformer(source_crs: str | int | CRS,
                    target_crs: str | int | CRS) -> Transformer:
    """Return a thread-local Transformer for the CRS pair, building and caching
    it on first use in this thread. The returned instance is only ever touched
    by the calling thread, so no PROJ context is shared across threads."""
    cache: OrderedDict[tuple[str, str], Transformer] | None = getattr(
        _local, "cache", None)
    if cache is None:
        cache = OrderedDict()
        _local.cache = cache

    key = (_normalize(source_crs), _normalize(target_crs))
    hit = cache.get(key)
    if hit is not None:
        cache.move_to_end(key)          # LRU: mark as most-recently used
        return hit

    tf = Transformer.from_crs(source_crs, target_crs, always_xy=True)
    cache[key] = tf
    if len(cache) > _MAX_ENTRIES:       # bounded: evict least-recently used
        cache.popitem(last=False)
    return tf

For the rarer case where a single transformer must be shared — say a fixed pair reused by every thread and memory is tight — wrap its use in a lock so calls never interleave:

import numpy as np

_shared_lock = threading.Lock()
_shared_tf = Transformer.from_crs("EPSG:4326", "EPSG:32633", always_xy=True)


def locked_transform(lon: np.ndarray, lat: np.ndarray) -> np.ndarray:
    """Serialize access to one shared Transformer. Correct but non-parallel:
    the lock forces threads to take turns, trading throughput for one instance."""
    with _shared_lock:
        x, y = _shared_tf.transform(lon, lat)
    return np.column_stack([x, y]).astype(np.float64, copy=False)

Parameter Reference

Name Type Units Valid range Notes
source_crs str | int | CRS — any PROJ-parseable CRS normalized to WKT for a collision-free key
target_crs str | int | CRS — any PROJ-parseable CRS e.g. "EPSG:32633"
_MAX_ENTRIES int count ≥ 1 per-thread LRU bound; prevents unbounded growth
return of get_transformer Transformer — — thread-local; safe to reuse within the owning thread only
lon, lat np.ndarray degrees (m,) float64 inputs for the shared-lock variant

Minimal Worked Example

Run several threads that each request a transformer for the same CRS pair and confirm the cache serves one instance per thread while transforming correctly.

import numpy as np
from concurrent.futures import ThreadPoolExecutor

pts = np.array([[13.4050, 52.5200]], dtype=np.float64)   # Berlin (lon, lat)


def task() -> tuple[int, float]:
    tf = get_transformer("EPSG:4326", "EPSG:32633")       # thread-local, cached
    x, y = tf.transform(pts[:, 0], pts[:, 1])
    return id(tf), float(x[0])


with ThreadPoolExecutor(max_workers=3) as pool:
    results = list(pool.map(lambda _: task(), range(6)))

eastings = {round(e, 2) for _, e in results}
print(sorted(eastings))
# [392788.16]   every thread transforms Berlin to the same easting in metres

Validation Check

Assert the transformed easting is correct and that repeated calls within one thread reuse the identical cached object rather than rebuilding it.

import numpy as np

tf1 = get_transformer("EPSG:4326", "EPSG:32633")
tf2 = get_transformer("EPSG:4326", "EPSG:32633")   # same thread, same pair
assert tf1 is tf2, "cache should return the identical instance within a thread"

x, y = tf1.transform(np.array([13.4050]), np.array([52.5200]))
assert abs(float(x[0]) - 392788.16) < 0.05, "Berlin easting drifted"
print("ok — cached instance reused and transform within tolerance")

Common Mistakes

Sharing one Transformer across threads without a lock
Two threads calling .transform on the same instance interleave writes to the PROJ context and return coordinates belonging to neither call — silent corruption, no exception. Give each thread its own instance through threading.local(), or serialize every call behind a lock if a single instance is unavoidable.
Unbounded cache growth
Caching a transformer for every distinct CRS pair a long-running service encounters slowly leaks memory as PROJ contexts pile up. Bound the cache with an LRU policy — an OrderedDict with a maximum size that evicts the least-recently-used entry — so the working set stays fixed.
Key collisions on equivalent CRS spellings
Keying the cache on the raw argument treats 4326, "EPSG:4326", and a WKT string as three different systems, building three transformers for one CRS. Normalize each CRS with CRS.from_user_input(...).to_wkt() before forming the key so equivalent spellings share one entry.