ADR-007: GPU Backend Selection for ETL Pipeline
Status: Accepted (June 2026)
Driver: Mesh-to-connectomics contact detection is compute-bound
Tags: gpu, etl, performance, cuda
Context
The mesh-to-connectomics ETL pipeline has three compute-heavy stages:
- Voxel contact detection — O(sx·sy·sz) face-adjacency scan, 26-neighbor check per voxel
- Chamfer distance transform — 3-pass O(N) per segment, repeated per neuron (10K+ segments for large volumes)
- Watershed splitting — priority-flood on curvatures for oversized contacts
Skeleton extraction (TEASAR) also runs a chamfer DT per segment. For MICrONS-scale volumes with 50K+ neurons, the CPU path takes hours. GPU acceleration targets this bottleneck.
Decision: Use NVCC-compiled fatbins loaded via cust
We evaluated three GPU approaches:
| Approach | Mechanism | Pros | Cons |
|---|---|---|---|
| cuda-oxide | Rust proc-macro → PTX | Pure Rust | Nightly Rust, git dep, abandoned upstream |
| cust (chosen) | NVCC .cu → fatbin, cust loads at runtime | Stable Rust, crates.io, standard CUDA C | Requires NVCC at build time |
| wgpu | WGSL compute shaders | Cross-platform (AMD, Intel, Apple) | Not implemented; no existing kernels |
Why not cuda-oxide: The NVlabs cuda-oxide project requires nightly Rust, pins a specific rustc version, and is no longer actively maintained. Our Nix build was brittle.
Why not wgpu: No CUDA C → WGSL translation path exists for our kernels. Implementing chamfer DT + contact detection from scratch in WGSL is high effort with unclear performance parity.
Why cust: Standard CUDA C kernels compiled by NVCC, loaded as fatbins via the CUDA driver API. Stable Rust, published on crates.io. Same algorithm as CPU (bit-for-bit equivalent with same weights).
Key architectural decisions
Decision 1: Per-file multi-arch fatbin modules
Each .cu kernel file compiles to a separate fatbin via NVCC 12 (SASS for sm_52..sm_90 plus compute_90 PTX), so every cluster GPU runs precompiled code on drivers >= R525. Concatenating PTX files would produce duplicate .version/.target headers that the CUDA driver rejects, so modules stay separate.
The build.rs generates a modules.rs with per-module accessors:
#![allow(unused)]
fn main() {
// build.rs output
pub fn chamfer_dt() -> FatbinModule { ... }
pub fn detect_contacts() -> FatbinModule { ... }
pub fn all_modules() -> Vec<FatbinModule> { ... }
}
cust.rs loads each module separately via Module::from_fatbin() and searches all modules when looking up kernel functions.
Decision 2: Init runs on a watchdog thread
CustBackend::new builds the context, stream and modules on a helper thread and waits SYNDB_CUDA_INIT_TIMEOUT_SECS (default 120). On timeout the process panics, so a hung cuInit fails the pod fast. CustBackend is already shared across rayon threads: enter() re-binds the (Send + Sync) context with cuCtxSetCurrent on every call, so the thread that created it does not matter. The earlier unproven blanket exclusion of driver-580 nodes is gone; the watchdog replaces it.
Decision 3: Separate chamfer weights for skeleton vs. scoring
Three different chamfer weight schemes existed:
| Scheme | face_w | edge_w | corner_w | Used by |
|---|---|---|---|---|
| skeleton.rs | 1.0 × min_res | 1.4 × min_res | 1.7 × min_res | Skeleton extraction (TEASAR) |
| geometry.rs | 0.3 × min_diag | 0.4 × min_diag | 0.5 × min_diag | Pipeline scoring (normals, curvature) |
| cust.rs (original) | 3.0 × min_res | 4.0 × min_res | 5.0 × min_res | GPU chamfer DT |
The GPU compute_distance_field_full() method (for skeleton extraction) uses skeleton weights (1.0:1.4:1.7) for compatibility. The existing compute_distance_field() (for pipeline scoring) continues using 3:4:5 weights. Changing skeleton weights would alter radius_nm and branch thresholds, producing different skeleton outputs — this is acceptable because the chamfer DT is an approximation of Euclidean DT and both schemes produce equivalent topology.
Decision 4: GPU contact detection must aggregate per-pair
The initial GPU kernel emitted raw (id_a, id_b) pairs per face and deduplicated with a HashSet, discarding geometry:
| Field | CPU (correct) | GPU (original — broken) | GPU (fixed) |
|---|---|---|---|
contact_area_nm2 | Per-direction: res[1]·res[2] for x-faces | res[0]·res[1]·res[2] (a volume, not area) | Per-direction, aggregated |
centroid_nm | Area-weighted average of face midpoints | [0.0, 0.0, 0.0] | Proper centroid |
face_positions_nm | Every face midpoint (for watershed) | Empty | Full list |
face_count | Actual count | Hardcoded 1 | Correct |
Fix: Kernel now emits (id_a, id_b, voxel_idx, direction) per face. Host-side HashMap<(u64,u64), ContactAccum> aggregates per-pair with correct geometry, reusing the same ContactAccum struct as the CPU path.
Decision 5: Serial skeleton extraction when GPU is used
extract_all_skeletons previously used rayon::par_iter() to compute per-segment distance transforms in parallel on CPU. With GPU, each segment’s DT is ~500µs for a 512³ volume, making serial dispatch fast enough (5s for 10K segments) while avoiding CUDA context thread-affinity issues.
Decision 6: Hybrid mode heuristic
SynapseAware datasets (mesh-only BVH) don’t use the GPU. Hybrid mode adds voxel-contact detection (GPU) on top of mesh BVH (CPU) and merges results. We enable Hybrid mode when the heuristic rule triggers:
voxel_volume_nm3 > 10_000
This identifies coarse-resolution volumes where meshes may miss appositions that voxel contact detection catches. By this rule:
| Dataset | Voxel volume | Mode |
|---|---|---|
| Songbird BG (8×8×8) | 512 nm³ | SynapseAware (no GPU) |
| MICrONS Minnie35 (8×8×40) | 2,560 nm³ | SynapseAware (no GPU) |
| BANC seg v888 (16×16×45) | 11,520 nm³ | Hybrid (GPU) |
Consequences
Positive
- GPU contact detection is 1000× faster than CPU (31µs vs 31ms for 8³ volume)
- Skeleton extraction DT is accelerated per-segment
- No thread-safety issues with CUDA contexts
- Binary result matches CPU algorithm (same kernel code, same weights)
Negative
- Requires NVIDIA GPU with driver ≥ 595 (195 nodes available on Nautilus)
- The ETL image carries the CUDA kernels and links
libcuda.so.1(see the update below) - GPU skeleton DT has a race condition (adjacent threads read/write same cells) — results may differ from CPU by <1%
Risks mitigated
| Risk | Mitigation |
|---|---|
cuInit hangs | Init watchdog (SYNDB_CUDA_INIT_TIMEOUT_SECS) panics the pod |
| CUDA context thread safety | No thread spawning; serial dispatch |
| GPU-capable image not deployed | Single oci-syndb-etl image serves CPU and GPU jobs |
| PTX loading failure | Separate per-file modules (no concatenation) |
| Broken contact geometry | Host-side HashMap aggregation, same as CPU |
Update (November 2026): one ETL image
The separate syndb-etl-gpu image is superseded. oci-syndb-etl is built with
the GPU kernels and serves CPU and GPU nodes. cust links libcuda.so.1 with
DT_NEEDED (it is not dlopened), so the build appends a copied stub
libcuda.so.1 to the binary’s RUNPATH as a fallback for CPU nodes. On GPU
pods LD_LIBRARY_PATH (searched before RUNPATH) selects the real driver. CPU
versus GPU execution is still chosen only by gpu.enabled / GPU_BACKEND=cust.
See GPU ETL.
Related files
| File | Purpose |
|---|---|
crates/tools/etl/gpu-kernels/src/*.cu | CUDA C kernel sources |
crates/tools/etl/gpu-kernels/build.rs | NVCC compilation, generates modules.rs |
crates/tools/etl/src/gpu/cust.rs | CustBackend — CUDA backend implementation |
crates/tools/etl/src/gpu/mod.rs | GpuBackend trait, select_backend() dispatch |
infrastructure/helm/syndb-etl/templates/_helpers.tpl | Node affinity, GPU image selection |
infrastructure/helm/syndb-etl/values.yaml | Per-dataset GPU config (gpu: blocks) |
nix/packages.nix | cudaRoot, cudaStubLibcuda, syndb-cli-etl-image derivation |
crates/tools/cli/src/k8s/nautilus/deploy.rs | ETL image build/push |