Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

ADR-007: GPU Backend Selection for ETL Pipeline

Status: Accepted (June 2026)
Driver: Mesh-to-connectomics contact detection is compute-bound
Tags: gpu, etl, performance, cuda

Context

The mesh-to-connectomics ETL pipeline has three compute-heavy stages:

  1. Voxel contact detection — O(sx·sy·sz) face-adjacency scan, 26-neighbor check per voxel
  2. Chamfer distance transform — 3-pass O(N) per segment, repeated per neuron (10K+ segments for large volumes)
  3. Watershed splitting — priority-flood on curvatures for oversized contacts

Skeleton extraction (TEASAR) also runs a chamfer DT per segment. For MICrONS-scale volumes with 50K+ neurons, the CPU path takes hours. GPU acceleration targets this bottleneck.

Decision: Use NVCC-compiled fatbins loaded via cust

We evaluated three GPU approaches:

ApproachMechanismProsCons
cuda-oxideRust proc-macro → PTXPure RustNightly Rust, git dep, abandoned upstream
cust (chosen)NVCC .cu → fatbin, cust loads at runtimeStable Rust, crates.io, standard CUDA CRequires NVCC at build time
wgpuWGSL compute shadersCross-platform (AMD, Intel, Apple)Not implemented; no existing kernels

Why not cuda-oxide: The NVlabs cuda-oxide project requires nightly Rust, pins a specific rustc version, and is no longer actively maintained. Our Nix build was brittle.

Why not wgpu: No CUDA C → WGSL translation path exists for our kernels. Implementing chamfer DT + contact detection from scratch in WGSL is high effort with unclear performance parity.

Why cust: Standard CUDA C kernels compiled by NVCC, loaded as fatbins via the CUDA driver API. Stable Rust, published on crates.io. Same algorithm as CPU (bit-for-bit equivalent with same weights).

Key architectural decisions

Decision 1: Per-file multi-arch fatbin modules

Each .cu kernel file compiles to a separate fatbin via NVCC 12 (SASS for sm_52..sm_90 plus compute_90 PTX), so every cluster GPU runs precompiled code on drivers >= R525. Concatenating PTX files would produce duplicate .version/.target headers that the CUDA driver rejects, so modules stay separate.

The build.rs generates a modules.rs with per-module accessors:

#![allow(unused)]
fn main() {
// build.rs output
pub fn chamfer_dt() -> FatbinModule { ... }
pub fn detect_contacts() -> FatbinModule { ... }
pub fn all_modules() -> Vec<FatbinModule> { ... }
}

cust.rs loads each module separately via Module::from_fatbin() and searches all modules when looking up kernel functions.

Decision 2: Init runs on a watchdog thread

CustBackend::new builds the context, stream and modules on a helper thread and waits SYNDB_CUDA_INIT_TIMEOUT_SECS (default 120). On timeout the process panics, so a hung cuInit fails the pod fast. CustBackend is already shared across rayon threads: enter() re-binds the (Send + Sync) context with cuCtxSetCurrent on every call, so the thread that created it does not matter. The earlier unproven blanket exclusion of driver-580 nodes is gone; the watchdog replaces it.

Decision 3: Separate chamfer weights for skeleton vs. scoring

Three different chamfer weight schemes existed:

Schemeface_wedge_wcorner_wUsed by
skeleton.rs1.0 × min_res1.4 × min_res1.7 × min_resSkeleton extraction (TEASAR)
geometry.rs0.3 × min_diag0.4 × min_diag0.5 × min_diagPipeline scoring (normals, curvature)
cust.rs (original)3.0 × min_res4.0 × min_res5.0 × min_resGPU chamfer DT

The GPU compute_distance_field_full() method (for skeleton extraction) uses skeleton weights (1.0:1.4:1.7) for compatibility. The existing compute_distance_field() (for pipeline scoring) continues using 3:4:5 weights. Changing skeleton weights would alter radius_nm and branch thresholds, producing different skeleton outputs — this is acceptable because the chamfer DT is an approximation of Euclidean DT and both schemes produce equivalent topology.

Decision 4: GPU contact detection must aggregate per-pair

The initial GPU kernel emitted raw (id_a, id_b) pairs per face and deduplicated with a HashSet, discarding geometry:

FieldCPU (correct)GPU (original — broken)GPU (fixed)
contact_area_nm2Per-direction: res[1]·res[2] for x-facesres[0]·res[1]·res[2] (a volume, not area)Per-direction, aggregated
centroid_nmArea-weighted average of face midpoints[0.0, 0.0, 0.0]Proper centroid
face_positions_nmEvery face midpoint (for watershed)EmptyFull list
face_countActual countHardcoded 1Correct

Fix: Kernel now emits (id_a, id_b, voxel_idx, direction) per face. Host-side HashMap<(u64,u64), ContactAccum> aggregates per-pair with correct geometry, reusing the same ContactAccum struct as the CPU path.

Decision 5: Serial skeleton extraction when GPU is used

extract_all_skeletons previously used rayon::par_iter() to compute per-segment distance transforms in parallel on CPU. With GPU, each segment’s DT is ~500µs for a 512³ volume, making serial dispatch fast enough (5s for 10K segments) while avoiding CUDA context thread-affinity issues.

Decision 6: Hybrid mode heuristic

SynapseAware datasets (mesh-only BVH) don’t use the GPU. Hybrid mode adds voxel-contact detection (GPU) on top of mesh BVH (CPU) and merges results. We enable Hybrid mode when the heuristic rule triggers:

voxel_volume_nm3 > 10_000

This identifies coarse-resolution volumes where meshes may miss appositions that voxel contact detection catches. By this rule:

DatasetVoxel volumeMode
Songbird BG (8×8×8)512 nm³SynapseAware (no GPU)
MICrONS Minnie35 (8×8×40)2,560 nm³SynapseAware (no GPU)
BANC seg v888 (16×16×45)11,520 nm³Hybrid (GPU)

Consequences

Positive

  • GPU contact detection is 1000× faster than CPU (31µs vs 31ms for 8³ volume)
  • Skeleton extraction DT is accelerated per-segment
  • No thread-safety issues with CUDA contexts
  • Binary result matches CPU algorithm (same kernel code, same weights)

Negative

  • Requires NVIDIA GPU with driver ≥ 595 (195 nodes available on Nautilus)
  • The ETL image carries the CUDA kernels and links libcuda.so.1 (see the update below)
  • GPU skeleton DT has a race condition (adjacent threads read/write same cells) — results may differ from CPU by <1%

Risks mitigated

RiskMitigation
cuInit hangsInit watchdog (SYNDB_CUDA_INIT_TIMEOUT_SECS) panics the pod
CUDA context thread safetyNo thread spawning; serial dispatch
GPU-capable image not deployedSingle oci-syndb-etl image serves CPU and GPU jobs
PTX loading failureSeparate per-file modules (no concatenation)
Broken contact geometryHost-side HashMap aggregation, same as CPU

Update (November 2026): one ETL image

The separate syndb-etl-gpu image is superseded. oci-syndb-etl is built with the GPU kernels and serves CPU and GPU nodes. cust links libcuda.so.1 with DT_NEEDED (it is not dlopened), so the build appends a copied stub libcuda.so.1 to the binary’s RUNPATH as a fallback for CPU nodes. On GPU pods LD_LIBRARY_PATH (searched before RUNPATH) selects the real driver. CPU versus GPU execution is still chosen only by gpu.enabled / GPU_BACKEND=cust. See GPU ETL.

FilePurpose
crates/tools/etl/gpu-kernels/src/*.cuCUDA C kernel sources
crates/tools/etl/gpu-kernels/build.rsNVCC compilation, generates modules.rs
crates/tools/etl/src/gpu/cust.rsCustBackend — CUDA backend implementation
crates/tools/etl/src/gpu/mod.rsGpuBackend trait, select_backend() dispatch
infrastructure/helm/syndb-etl/templates/_helpers.tplNode affinity, GPU image selection
infrastructure/helm/syndb-etl/values.yamlPer-dataset GPU config (gpu: blocks)
nix/packages.nixcudaRoot, cudaStubLibcuda, syndb-cli-etl-image derivation
crates/tools/cli/src/k8s/nautilus/deploy.rsETL image build/push