GPU-Accelerated ETL
The mesh-to-connectomics pipeline supports GPU acceleration via CUDA C kernels
compiled by NVCC into multi-arch fatbins and loaded at runtime through the cust crate. A pod
that requests a GPU never silently runs on CPU: the build fails without
kernels, and the process panics if GPU_BACKEND=cust cannot initialize CUDA.
Backend selection
The GPU_BACKEND environment variable controls which backend is used:
| Value | Backend | Status |
|---|---|---|
| unset | CPU | Always available, no GPU required |
cust | CUDA (via cust + NVCC fatbin) | Requires gpu-cust feature + NVIDIA driver; panics otherwise |
| any other | CPU | Falls through to default |
select_backend() in crates/tools/etl/src/gpu/mod.rs initializes the
backend once per process (OnceLock):
GPU_BACKEND env var
├── "cust" → CustBackend::new(0) → panic on error (no silent CPU)
└── other → CpuBackend
CustBackend serializes calls on one stream behind a mutex and binds its
context to the calling thread, so rayon workers can share it: skeleton
extraction runs segments in parallel, distance transforms queue on the GPU
while TEASAR path extraction proceeds on the CPU. Per-call kernel failures
(e.g. an oversized line) are logged and that call falls back to CPU.
Architecture
NVCC (CUDA 12.9, build time)
└── .cu → .fatbin (per-kernel file: SASS sm_52..sm_90 + compute_90 PTX)
↓ include_bytes! (build.rs-generated modules.rs)
cust (crates.io, runtime)
├── quick_init() → cuInit(0) + cuDevicePrimaryCtxRetain (init watchdog)
├── Stream::new() → cuStreamCreate
└── Module::from_fatbin() → cuModuleLoadData (per .fatbin file)
Key traits:
| File | Purpose |
|---|---|
gpu/mod.rs | GpuBackend trait + select_backend() dispatch |
gpu/cust.rs | CustBackend — loads fatbins, launches kernels |
gpu/cpu.rs | CpuBackend — CPU reference implementations |
gpu-kernels/build.rs | NVCC compilation, generates modules.rs |
gpu-kernels/src/lib.rs | FatbinModule struct, all_modules() |
Fatbins are kept separate (one per .cu file); the loader searches every
module for a kernel by name.
Architectures. Each fatbin carries SASS for sm_52, 60, 61, 70, 75, 80,
86, 89 and 90 (Maxwell through Hopper), so the cluster’s GPUs run precompiled
code with no JIT, plus compute_90 PTX that newer architectures (Blackwell)
JIT-compile. CUDA 12 minor-version compatibility makes the cubins load on any
driver >= R525. The kernels only use fp64 __dadd_rn/__dmul_rn,
__longlong_as_double and u32 atomicAdd, so compute capability 5.2+ is
enough.
Init watchdog. CustBackend::new runs the driver init and module load on
a helper thread and panics if it has not finished after
SYNDB_CUDA_INIT_TIMEOUT_SECS (default 120), so a hung cuInit fails the pod
within two minutes instead of idling until its deadline. The context is
re-bound to the calling thread on every call, so building it elsewhere is safe.
Prerequisites
Build time
| Requirement | Nix package | Env var |
|---|---|---|
nvcc | cudaPackages.cuda_nvcc (12.9) | CUDA_TOOLKIT_PATH |
| CUDA runtime headers | cudaPackages.cuda_cudart | CUDA_CUDART_INCLUDE |
CUDA CRT headers (crt/host_config.h) | cudaPackages.cuda_nvcc (include/crt) | CUDA_CRT_INCLUDE |
libcuda.so stub | cudaPackages.cuda_cudart | CUDA_LIBRARY_PATH |
All of these are wired in nix/packages.nix via cudaRoot — a runCommand
derivation that symlinks everything into one tree:
cuda-root/
include/
cuda.h → cuda_cudart/include/cuda.h
cuda_runtime.h → cuda_cudart/include/cuda_runtime.h
crt/ → cuda_nvcc/include/crt/
lib64/
libcuda.so → cuda_cudart/lib/stubs/libcuda.so
Runtime (GPU pod)
| Requirement | Source |
|---|---|
| NVIDIA driver | Host kernel module (≥ R525; the fatbin is built with CUDA 12) |
libcuda.so.1 | Mounted by NVIDIA container runtime at /usr/lib/x86_64-linux-gnu/ |
/dev/nvidia* | Exposed via nvidia.com/gpu device plugin |
LD_LIBRARY_PATH | Set by Helm template to include /usr/lib/x86_64-linux-gnu/ |
HOME | Set to /tmp because readOnlyRootFilesystem: true |
Building
# Local (requires CUDA toolkit on host):
CUDA_LIBRARY_PATH=/tmp/cuda-root \
CUDA_TOOLKIT_PATH=/nix/store/...-cuda12.9-cuda_nvcc-12.9.86/bin \
CUDA_HOME=/tmp/cuda-root \
CUDA_CUDART_INCLUDE=/nix/store/...-cuda12.9-cuda_cudart-.../include \
CUDA_CRT_INCLUDE=/nix/store/...-cuda12.9-cuda_nvcc-.../include \
cargo build --release -p cli --features etl-gpu
# Nix (production):
nix build .#syndb-cli-etl-image
The etl-gpu feature flag enables gpu-cust on the etl crate:
etl-gpu → etl-full + etl/gpu-cust
gpu-cust → gpu + dep:cust + dep:meshcon-gpu-kernels
Helm configuration
GPU jobs are configured in infrastructure/helm/syndb-etl/values.yaml:
prepareJobs:
- pipeline: mesh-connectomics
command:
- syndb
- etl
- mesh-connectomics
- prepare
gpu:
enabled: true
count: 1
type: nvidia.com/gpu
shmSize: 8Gi
resources:
requests:
cpu: 4
memory: 24Gi
limits:
cpu: 4800m
memory: 28Gi
When .gpu.enabled is true, the Helm template automatically:
- Keeps the same
syndb-etlimage as CPU jobs; there is no separate GPU image - Sets
GPU_BACKEND=custenv var - Sets
LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:/usr/lib64:/usr/lib/nvidia - Sets
HOME=/tmp - Requests
nvidia.com/gpu(or custom.gpu.type) - Mounts
/dev/shmas a memory-backed emptyDir - Adds node affinity:
nvidia.com/cuda.driver-version.major Gt 524(driver ≥ R525) - Allocates the GPU to the
prepareinit container (mesh tile jobs: the single tile container, sized byprepare.gpu.count/.type)
Driver floor
The kernels ship as CUDA 12 fatbins, so any driver ≥ R525 loads them without
JIT and Helm only requires nvidia.com/cuda.driver-version.major Gt 524.
Older notes excluded driver 580 for a cuInit hang on some GPUs; that was
never reproduced. If a node does hang, the init watchdog panics after
SYNDB_CUDA_INIT_TIMEOUT_SECS (default 120) and the pod fails fast.
Tiled voxel-contact runs
meshConnectome.tiles splits one segmentation volume into
dimensions.x × y × z tile Jobs plus a merge Job:
meshConnectome:
tiles:
enabled: true
runId: k1 # new id per run: Job names + S3 exchange prefix
source: kasthuri-ac4 # registered mesh-connectomics source
segmentationUrl: https://bossdb-open-data.s3.amazonaws.com/Kasthuri/ac4/neuron/
overlap: 32 # contacts must fit within the overlap
prepare:
gpu:
enabled: true # GPU tiles also compute TEASAR skeletons
-
Tiles own a disjoint core box. Each tile writes:
- morphometry accumulators for its core voxels (
neuron_parts.parquet); - the synapses centred in its core;
- its manifest, carrying
compute_backend/gpu_seconds.
Output goes to
s3://<bucket>/mesh-connectomics-tiles/<runId>/tiles/<x-y-z>/with a.readysentinel. - morphometry accumulators for its core voxels (
-
Merge runs the CPU image. It fails as soon as every Job of some tile has failed, and otherwise waits for every sentinel. It sums the accumulators into exact whole-neuron morphometry and writes a manifest that is
completeonly when all tiles arrived. Finally it publishes tos3://<bucket>/mesh-connectomics-<source>/with.ready, which the source’s import job consumes. -
GPU pools: a GPU tile renders one Job per entry of
meshConnectome.prepare.gpu.pools(syndb-etl-mesh-tile-<runId>-<x-y-z>-<pool>) and the first pool the scheduler can place runs it. Default pools:gpu(nvidia.com/gpu),a6000,a40,rtx8000, plusa100,h100andh200withpriorityClassName: opportunistic(those resources have no quota at default priority, so they run preemptible). At start a tile exits 0 if<tile>/.readyalready exists, otherwise it writes<tile>/.claimand runs. The claim is informational, not a lock: two pools placed within seconds both run the tile and upload identical outputs. A tile fails the merge only when all of its pool Jobs failed. Pending losers count against the namespace pod quota until theiractiveDeadlineSeconds, so keep tiles × pools within it or trimpools. An emptypoolslist falls back to the singlegpu.type. -
Run evidence: the merged
gpu_secondsandcompute_backendrecord that the run used the GPU.
GPU node types on NRP Nautilus
The cluster labels nodes with nvidia.com/gpu.product. Selection
as of June 2026:
| GPU | Resource name | Nodes | Driver | Notes |
|---|---|---|---|---|
| RTX 3090 | nvidia.com/gpu | 46 | 595 | |
| A10 | nvidia.com/gpu | 35 | 595 | |
| A100 | nvidia.com/a100 | 22 | 595 | Quota-gated |
| L40 | nvidia.com/gpu | 17 | 595 | |
| RTX 2080 Ti | nvidia.com/gpu | 16 | 595 | Tested + verified |
| RTX A6000 | nvidia.com/rtxa6000 | 10 | 595 | |
| GTX 1080 Ti | nvidia.com/gpu | 13 | 580 | Runs the fatbin |
| V100 | nvidia.com/gpu | 13 | 580 | Runs the fatbin |
For special GPUs (A100, A6000, H100, H200) use their specific resource
name instead of nvidia.com/gpu. Quota-gated GPUs can be accessed with
priorityClassName: opportunistic.
To pin a specific GPU model:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: nvidia.com/gpu.product
operator: In
values:
- NVIDIA-GeForce-RTX-2080-Ti
GPU benchmark suite
syndb etl mesh-connectomics gpu-bench times the GPU-backed algorithms against
their CPU references (cpu-1, cpu-n, gpu) and checks every output bit for bit.
The gpuBench chart key (disabled by default, rendered only with
jobsEnabled) runs it on every pool of gpuBench.pools plus one CPU Job:
syndb-etl-gpu-bench-<runId>-<pool>, each with 2 CPU / 12Gi and one GPU
(CPU Job: 16 CPU / 64Gi, --threads 4 --allow-cpu), requests equal to limits.
GPU pods run with --gpu-only: they time only gpu-1 and gpu-n (the
single-thread reference still runs once per case for the identity check) so the
GPU never idles on a CPU baseline; gpu-bench-report takes cpu-1 and cpu-n
from the cpu pool of the same run. The bench caps crop sizes at a quarter of
the memory limit, so 12Gi reaches 512^3 (EDT); raise gpuBench.resources.gpu.memory
(about 26Gi for 640^3) for larger sizes.
gpuBench.pools uses the meshConnectome.prepare.gpu.pools format plus an
optional products list that pins nvidia.com/gpu.product (p1080ti, v100 and
l40s share the generic nvidia.com/gpu resource). a100, h100 and h200 run as
opportunistic with a 6 h deadline, the others 4 h; a pool that never
schedules is reported as “not scheduled” and blocks nothing. The Jobs carry
app=syndb-etl-bench, so the ETL reset of nautilus apply never deletes
them. Each Job syncs /work/out to s3://syndb-etl/gpu-bench/<runId>/<pool>/
every 5 minutes and on exit, so a preempted pod still leaves partial results.
Run it from a clean export of HEAD with an image that contains gpu-bench.
-l limits the apply to the benchmark Jobs, because jobsEnabled=true also
renders every other enabled ETL Job:
helm template syndb-etl infrastructure/helm/syndb-etl \
-f infrastructure/helm/syndb-etl/gpu-bench-runs/full.yaml \
--set jobsEnabled=true --set gpuBench.runId=<runid> \
--set gpuBench.gitSha=<source revision of the image> \
--set image.digest=sha256:<digest> \
| kubectl apply -n syndb -l app=syndb-etl-bench -f -
gpu-bench-runs/full.yaml also sets the public Kasthuri tile and the BANC
pilot prefix; list the pilot’s tiles (t_X_Y_Z) in gpuBench.bancTile.tiles
once its plan.json exists. Collect and analyse the results:
s5cmd sync 's3://syndb-etl/gpu-bench/<runid>/*' DIR/
syndb etl mesh-connectomics gpu-bench-report --in DIR/* --out OUT
The report writes results.parquet, recommendation.toml and one
gpu-profile-<dataset>.yaml Helm overlay per dataset. Review it, commit it
and apply it to the next run with -f; nothing is applied automatically:
helm template syndb-etl infrastructure/helm/syndb-etl -f OUT/gpu-profile-<dataset>.yaml ...
The profile sets meshConnectome.prepare.gpu.enabled and the pool order that
the tile and contact Jobs read; a pool’s optional products list pins
nvidia.com/gpu.product there as it does for the benchmark Jobs. The
SYNDB_GPU_FIELD_MIN_VOXELS threshold is written to meshConnectome.tiles.env
and meshConnectome.contact.env, name to value maps like prepareJobs[].env.
Verifying GPU operation
Check the pod logs for backend selection:
2026-06-26T16:17:14Z INFO etl::gpu: GPU_BACKEND=cust: attempting CUDA backend
2026-06-26T16:17:17Z INFO etl::gpu::cust: cuST backend: CUDA driver initialized in 2.6s
2026-06-26T16:17:17Z INFO etl::gpu::cust: cuST backend: loaded 2 fatbin module(s)
2026-06-26T16:17:17Z INFO etl::gpu: CUDA backend initialized successfully
Kernel dispatch timings appear at DEBUG level:
cuST contacts kernel done in 31.1µs
cuST contacts found 64 raw pairs in 473µs
cuST contacts total=510µs unique=1
On failure the pod exits instead of running on CPU:
panicked at ... GPU_BACKEND=cust but CUDA backend init failed: ...
The GPU/CPU parity tests (gpu_cpu_dt_equivalence,
gpu_cpu_contact_equivalence) skip on CPU and run for real only with
GPU_BACKEND=cust inside the GPU image.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
Pod stuck in Pending | No GPU node matching affinity | Check GPU type, quota, and nvidia.com/gpu.product labels |
Build panics SYNDB_REQUIRE_GPU_KERNELS=1 but ... | nvcc or .cu sources missing from the Nix source | mkCliEtlSrc must include gpu-kernels/**/*.cu; cuda_nvcc must be in nativeBuildInputs |
Module::from_fatbin: ... InvalidSource | Device compute capability below 5.2, or driver older than R525 | Schedule on a newer GPU/driver |
CUDA init did not finish within 120s panic | Hung cuInit / driver call on that node | Retry on another node; raise SYNDB_CUDA_INIT_TIMEOUT_SECS only for slow-starting devices |
| Process exits 139 after pipeline | Benign cleanup crash in Context drop | Results are already written — fix destructor ordering if critical |
nvcc fatal: Unsupported gpu architecture | Toolkit too new (CUDA 13 dropped sm_52..sm_70) | Keep the build on CUDA 12.x |
False leads
Earlier documentation claimed that NVIDIA_VISIBLE_DEVICES=void was the root
cause of GPU hangs, implying the Nautilus cluster lacked a device plugin.
This was incorrect. The NRP Nautilus cluster does have the NVIDIA device
plugin installed. NVIDIA_VISIBLE_DEVICES is correctly set to the allocated
GPU UUID (e.g. GPU-e143f1a6-...). The suspected cuInit hang on driver 580
was never reproduced.
OCI image
One image, oci-syndb-etl (docker.io/caniko/syndb-etl, pinned by
image.digest), runs on CPU-only and GPU nodes. The binary is built with
--features etl-gpu and links libcuda.so.1 as a normal DT_NEEDED
dependency (cust does not dlopen it), so it needs some libcuda.so.1 to
start:
- GPU pods: the NVIDIA runtime mounts the real driver and the chart’s
LD_LIBRARY_PATHpoints at it.ld.sosearchesLD_LIBRARY_PATHbeforeDT_RUNPATH, so the driver always wins. - CPU pods: no driver is mounted, so the loader falls back to a stub
libcuda.so.1(copied from the CUDA toolkit stubs) that the build appends to the binary’sRUNPATH. The stub is never called: CUDA is only initialized whenGPU_BACKEND=cust, which the chart sets only whengpu.enabled.
The build fails if the binary ends up with DT_RPATH instead of RUNPATH,
since RPATH would outrank LD_LIBRARY_PATH and shadow the driver.
A GPU_BACKEND=cust pod that cannot see the real driver loads the stub and
fails fast at cuInit instead of silently running on CPU.
Files
| File | Purpose |
|---|---|
crates/tools/etl/gpu-kernels/build.rs | NVCC compilation, generates modules.rs |
crates/tools/etl/gpu-kernels/src/lib.rs | FatbinModule, all_modules() |
crates/tools/etl/gpu-kernels/src/edt.cu | Exact separable Euclidean DT kernels (matches skeleton::distance_transform) |
crates/tools/etl/gpu-kernels/src/detect_contacts.cu | Voxel contact detection kernel |
crates/tools/etl/src/gpu/mod.rs | GpuBackend trait + dispatch |
crates/tools/etl/src/gpu/cust.rs | CustBackend implementation |
crates/tools/etl/src/gpu/cpu.rs | CPU fallback |
nix/packages.nix | cudaRoot, cudaStubLibcuda, syndb-cli-etl-image derivation |
infrastructure/helm/syndb-etl/templates/_helpers.tpl | syndb-etl.nodeAffinity |
infrastructure/helm/syndb-etl/templates/etl-prepare-job.yaml | GPU pod spec |
infrastructure/helm/syndb-etl/values.yaml | .gpu block for each job |
Building and publishing in the cluster
syndb ops k8s image-build --tag TAG [--images etl,bench] [--rev REV] builds
Nix OCI images on Nautilus instead of on the operator host. The checkout
is private, so the cluster never clones or evaluates it: the CLI evaluates
git+file://...?rev=<committed SHA> (uncommitted edits are never included),
exports the derivation closure (a few tens of MiB) to s3://syndb-etl/image-build/,
and applies the imageBuild Job of the syndb-etl chart. The pod imports the
closure, realises the derivations (substituting from cache.nixos.org and the
syndb Attic cache, sandbox = false because the pod is unprivileged), and
pushes with skopeo using the dockerhub-push Secret. The CLI prints
repository:TAG@sha256:... per image; pin those digests in
nautilus/values.yaml yourself. It refuses to start while the
syndb-helm-mutation lease is held.
The Nix store is one warm rook-ceph-block PVC, syndb-image-build-store
(NRP rejects generic ephemeral volumes and evicts large emptyDirs), so only the
first build compiles the ~1,800 derivations; later builds reuse the store and
compile or substitute just what the revision changed. The seed container merges
the nixImage’s own store into the PVC on every run with nix copy --all, so
bumping imageBuild.nixImage adds its paths instead of leaving the build
container without a shell. A store above 75% of the 300Gi volume is collected
before the build, which makes that one build substitute from scratch. The claim
is ReadWriteOnce and never shared: the CLI refuses to start while another
app=syndb-image-build Job mounts it. --fresh-store restores the throwaway
<job>-store claim and --store-claim PVC mounts an existing claim instead.
imageBuild.maxJobs (8) and imageBuild.cores (4) set the derivation
parallelism: most cargo-package-* compiles are single-threaded, so parallel
derivations, not threads per derivation, bound the build.
NRP Nautilus policy
Every Job of the chart follows the NRP policies:
- requests and limits are set for cpu, memory and ephemeral-storage, with limits
within 20% of requests (ephemeral-storage is request = limit, the emptyDir size
plus 2Gi);
scripts/check_etl_helm_resources.pyenforces it on the rendered ETL, benchmark and image-build Jobs, and that a100/h100/h200 pods areopportunistic; - CPU use within 20-200% and memory within 20-150% of the request (at most 4
pods outside it; pods of at most 1 CPU / 2GB are exempt): size from
kubectl topof a finished run; the contact tiles run in memory size classes (meshConnectome.contact.tiles.classes, one Job per pool and class, each tile handled by the one class whose owned+halo triangle range holds it), sized from the measured peak oftile-contacts, 135 B per owned+halo triangle plus 240 B per contact node (a 34M-triangle tile needs 10.4 GiB, so a flat 8Gi request OOMs and a flat 24Gi one leaves the median tile at 5%); - GPU utilisation above 40%: contact tile pods download the next batch while
tile-contactsruns, a merge is applied only after its tiles (no waiting pod holds a request), benchmark GPU pods skip the CPU baselines; - at most 50 GPUs without a plan in Nautilus Support: the contact tile pools
share
meshConnectome.contact.tiles.maxConcurrentGpus(48), split over the classes by weight.