CoordTransform Cadastral & High-Precision Python

Thread-Safe pyproj Transformer Caching

Reusing a pyproj.Transformer instead of rebuilding it on every call is an easy speed win until threads enter the picture, at which point sharing one transformer becomes a data-corruption bug; this page — a companion to Concurrent pyproj Transformation Pipelines in Python within the broader Batch Transformation & Automation for Cadastral Coordinate Pipelines practice — shows how to cache transformers safely by keying them on (source_crs, target_crs) and giving each thread its own instance through thread-local storage.

Thread-local Transformer cache keyed by CRS pair Three worker threads each ask a factory for a Transformer for the same source and target CRS pair. The factory keeps a separate cache per thread using thread-local storage, keyed by the normalized CRS pair. Each thread receives and reuses its own Transformer instance, so no PROJ context is ever shared across threads. Thread A Thread B Thread C thread-local factory key = (src, dst) A's own Transformer B's own Transformer C's own Transformer

Figure — one cached Transformer per thread per CRS pair; contexts are never shared.

Why a Transformer cannot be shared across threads

A pyproj.Transformer holds a PROJ context (PJ_CONTEXT) and mutable transient state used while a transform runs. That state is not guarded by a lock inside PROJ for concurrent access from multiple threads, so two threads calling .transform on the same instance at the same time can interleave writes to the shared context and produce coordinates that belong to neither call. The corruption is silent — no exception, just numbers that are subtly off — which is the worst failure mode for a survey deliverable. There are two sound remedies, and the right one depends on the workload. If the threads exist to overlap I/O (streaming grid files, fetching tiles), give each thread its own transformer via thread-local storage so no context is ever contended. If you truly must share a single instance, serialize every call behind a lock — but a lock reintroduces the very serialization you spun up threads to avoid. The choice is bounded by the GIL: for CPU-bound transform math the achievable speedup from PP threads is governed by the serial fraction ss that the GIL forces,

What a Transformer cache key must contain Five components of the cache key. The source and target CRS, normalised to their authority codes rather than their strings, because two spellings of the same CRS must hit the same entry. The always_xy flag, because it changes the argument order of every call. The area of interest, because it changes which operation PROJ selects. The thread identity, because the cached object cannot cross threads. Omitting the always_xy flag or the area of interest gives a cache that returns a valid Transformer for the wrong operation. source CRS authority code normalise, do not use the string target CRS authority code same always_xy bool changes the argument order area_of_interest bbox or None changes the operation chosen thread id implicit via threading.local, not in the key

Figure — what belongs in the cache key: leave any of it out and the cache returns a wrong answer.

S(P)=1s+1−sP,S(P) = \frac{1}{s + \dfrac{1 - s}{P}},

and since Python-level array handling holds the GIL, ss stays high and threads deliver little on pure compute. Threads pay off when 1−s1 - s is dominated by GIL-releasing waits — disk and network. For genuine multi-core transform throughput, processes are the answer, as built in parallelizing coordinate transforms with multiprocessing.

Complete Runnable Implementation

The factory below stores a per-thread cache in a threading.local() object and normalizes the CRS pair into a stable string key so that CRS objects, EPSG codes, and WKT strings that denote the same system collapse to one entry. A bounded eviction keeps the per-thread cache from growing without limit when many distinct CRS pairs are requested.

from __future__ import annotations

import threading
from collections import OrderedDict

from pyproj import CRS, Transformer

# Each thread gets its own attribute namespace here; nothing is shared.
_local = threading.local()

_MAX_ENTRIES = 32   # bound per-thread cache growth across many CRS pairs


def _normalize(crs: str | int | CRS) -> str:
    """Collapse EPSG codes, CRS objects, and WKT to one stable key so that
    equivalent CRS spellings share a single cache entry (avoids key collisions
    where two spellings of the same CRS would build two Transformers)."""
    return CRS.from_user_input(crs).to_wkt()


def get_transformer(source_crs: str | int | CRS,
                    target_crs: str | int | CRS) -> Transformer:
    """Return a thread-local Transformer for the CRS pair, building and caching
    it on first use in this thread. The returned instance is only ever touched
    by the calling thread, so no PROJ context is shared across threads."""
    cache: OrderedDict[tuple[str, str], Transformer] | None = getattr(
        _local, "cache", None)
    if cache is None:
        cache = OrderedDict()
        _local.cache = cache

    key = (_normalize(source_crs), _normalize(target_crs))
    hit = cache.get(key)
    if hit is not None:
        cache.move_to_end(key)          # LRU: mark as most-recently used
        return hit

    tf = Transformer.from_crs(source_crs, target_crs, always_xy=True)
    cache[key] = tf
    if len(cache) > _MAX_ENTRIES:       # bounded: evict least-recently used
        cache.popitem(last=False)
    return tf

For the rarer case where a single transformer must be shared — say a fixed pair reused by every thread and memory is tight — wrap its use in a lock so calls never interleave:

import numpy as np

_shared_lock = threading.Lock()
_shared_tf = Transformer.from_crs("EPSG:4326", "EPSG:32633", always_xy=True)


def locked_transform(lon: np.ndarray, lat: np.ndarray) -> np.ndarray:
    """Serialize access to one shared Transformer. Correct but non-parallel:
    the lock forces threads to take turns, trading throughput for one instance."""
    with _shared_lock:
        x, y = _shared_tf.transform(lon, lat)
    return np.column_stack([x, y]).astype(np.float64, copy=False)

Parameter Reference

Name Type Units Valid range Notes
source_crs str | int | CRS — any PROJ-parseable CRS normalized to WKT for a collision-free key
target_crs str | int | CRS — any PROJ-parseable CRS e.g. "EPSG:32633"
_MAX_ENTRIES int count ≥ 1 per-thread LRU bound; prevents unbounded growth
return of get_transformer Transformer — — thread-local; safe to reuse within the owning thread only
lon, lat np.ndarray degrees (m,) float64 inputs for the shared-lock variant

Minimal Worked Example

Run several threads that each request a transformer for the same CRS pair and confirm the cache serves one instance per thread while transforming correctly.

import numpy as np
from concurrent.futures import ThreadPoolExecutor

pts = np.array([[13.4050, 52.5200]], dtype=np.float64)   # Berlin (lon, lat)


def task() -> tuple[int, float]:
    tf = get_transformer("EPSG:4326", "EPSG:32633")       # thread-local, cached
    x, y = tf.transform(pts[:, 0], pts[:, 1])
    return id(tf), float(x[0])


with ThreadPoolExecutor(max_workers=3) as pool:
    results = list(pool.map(lambda _: task(), range(6)))

eastings = {round(e, 2) for _, e in results}
print(sorted(eastings))
# [392788.16]   every thread transforms Berlin to the same easting in metres

Validation Check

Assert the transformed easting is correct and that repeated calls within one thread reuse the identical cached object rather than rebuilding it.

Cost of building a Transformer against the cost of a transform call Bar chart in milliseconds: building a Transformer from an EPSG pair takes about 18 milliseconds, building one from a WKT string about 24, a single point transform about 0.02, and a ten thousand point array transform about 1.4. Building is roughly a thousand times the cost of using, which is why the cache exists — and why it must be per thread rather than absent. 0.01 0.1 1 10 100 ms 18 build (EPSG) 24 build (WKT) 0.02 transform 1 pt 1.4 transform 10k

Figure — the cost of building a Transformer, against the cost of using one.

import numpy as np

tf1 = get_transformer("EPSG:4326", "EPSG:32633")
tf2 = get_transformer("EPSG:4326", "EPSG:32633")   # same thread, same pair
assert tf1 is tf2, "cache should return the identical instance within a thread"

x, y = tf1.transform(np.array([13.4050]), np.array([52.5200]))
assert abs(float(x[0]) - 392788.16) < 0.05, "Berlin easting drifted"
print("ok — cached instance reused and transform within tolerance")

Common Mistakes

Sharing one Transformer across threads without a lock
Two threads calling .transform on the same instance interleave writes to the PROJ context and return coordinates belonging to neither call — silent corruption, no exception. Give each thread its own instance through threading.local(), or serialize every call behind a lock if a single instance is unavoidable.
Unbounded cache growth
Caching a transformer for every distinct CRS pair a long-running service encounters slowly leaks memory as PROJ contexts pile up. Bound the cache with an LRU policy — an OrderedDict with a maximum size that evicts the least-recently-used entry — so the working set stays fixed.
Key collisions on equivalent CRS spellings
Keying the cache on the raw argument treats 4326, "EPSG:4326", and a WKT string as three different systems, building three transformers for one CRS. Normalize each CRS with CRS.from_user_input(...).to_wkt() before forming the key so equivalent spellings share one entry.

Frequently Asked Questions

Is a Transformer thread-safe if I only read from it?

No. There is no such thing as a read-only call here: transforming mutates state inside the underlying PROJ context, so concurrent calls on one instance can interleave. The failure is not a clean exception either — it can be wrong numbers, which is far worse than a crash. Give every thread its own instance.

Does a per-thread cache leak memory in a long-running server?

It can, if threads are created and destroyed continuously and the cache is keyed by thread. In a pool with a fixed set of worker threads the cache is bounded by the number of threads times the number of CRS pairs, which is small. In a thread-per-request server, bound the cache explicitly and let entries expire.

Should the cache key be the CRS object or its string?

Neither directly — normalise to the authority code where there is one, and fall back to a canonical serialisation otherwise. Two spellings of the same CRS must hit the same entry, and a CRS object may not hash usefully. Remember to include the axis-order flag and the area of interest, because both change which operation gets built.