Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

GPU-Accelerated ETL

The mesh-to-connectomics pipeline supports GPU acceleration via CUDA C kernels compiled by NVCC into multi-arch fatbins and loaded at runtime through the cust crate. A pod that requests a GPU never silently runs on CPU: the build fails without kernels, and the process panics if GPU_BACKEND=cust cannot initialize CUDA.

Backend selection

The GPU_BACKEND environment variable controls which backend is used:

ValueBackendStatus
unsetCPUAlways available, no GPU required
custCUDA (via cust + NVCC fatbin)Requires gpu-cust feature + NVIDIA driver; panics otherwise
any otherCPUFalls through to default

select_backend() in crates/tools/etl/src/gpu/mod.rs initializes the backend once per process (OnceLock):

GPU_BACKEND env var
  ├── "cust"  →  CustBackend::new(0)  →  panic on error (no silent CPU)
  └── other   →  CpuBackend

CustBackend serializes calls on one stream behind a mutex and binds its context to the calling thread, so rayon workers can share it: skeleton extraction runs segments in parallel, distance transforms queue on the GPU while TEASAR path extraction proceeds on the CPU. Per-call kernel failures (e.g. an oversized line) are logged and that call falls back to CPU.

Architecture

NVCC  (CUDA 12.9, build time)
  └── .cu → .fatbin (per-kernel file: SASS sm_52..sm_90 + compute_90 PTX)
        ↓  include_bytes! (build.rs-generated modules.rs)
cust  (crates.io, runtime)
  ├── quick_init() → cuInit(0) + cuDevicePrimaryCtxRetain   (init watchdog)
  ├── Stream::new() → cuStreamCreate
  └── Module::from_fatbin() → cuModuleLoadData (per .fatbin file)

Key traits:

FilePurpose
gpu/mod.rsGpuBackend trait + select_backend() dispatch
gpu/cust.rsCustBackend — loads fatbins, launches kernels
gpu/cpu.rsCpuBackend — CPU reference implementations
gpu-kernels/build.rsNVCC compilation, generates modules.rs
gpu-kernels/src/lib.rsFatbinModule struct, all_modules()

Fatbins are kept separate (one per .cu file); the loader searches every module for a kernel by name.

Architectures. Each fatbin carries SASS for sm_52, 60, 61, 70, 75, 80, 86, 89 and 90 (Maxwell through Hopper), so the cluster’s GPUs run precompiled code with no JIT, plus compute_90 PTX that newer architectures (Blackwell) JIT-compile. CUDA 12 minor-version compatibility makes the cubins load on any driver >= R525. The kernels only use fp64 __dadd_rn/__dmul_rn, __longlong_as_double and u32 atomicAdd, so compute capability 5.2+ is enough.

Init watchdog. CustBackend::new runs the driver init and module load on a helper thread and panics if it has not finished after SYNDB_CUDA_INIT_TIMEOUT_SECS (default 120), so a hung cuInit fails the pod within two minutes instead of idling until its deadline. The context is re-bound to the calling thread on every call, so building it elsewhere is safe.

Prerequisites

Build time

RequirementNix packageEnv var
nvcccudaPackages.cuda_nvcc (12.9)CUDA_TOOLKIT_PATH
CUDA runtime headerscudaPackages.cuda_cudartCUDA_CUDART_INCLUDE
CUDA CRT headers (crt/host_config.h)cudaPackages.cuda_nvcc (include/crt)CUDA_CRT_INCLUDE
libcuda.so stubcudaPackages.cuda_cudartCUDA_LIBRARY_PATH

All of these are wired in nix/packages.nix via cudaRoot — a runCommand derivation that symlinks everything into one tree:

cuda-root/
  include/
    cuda.h              → cuda_cudart/include/cuda.h
    cuda_runtime.h      → cuda_cudart/include/cuda_runtime.h
    crt/                → cuda_nvcc/include/crt/
  lib64/
    libcuda.so          → cuda_cudart/lib/stubs/libcuda.so

Runtime (GPU pod)

RequirementSource
NVIDIA driverHost kernel module (≥ R525; the fatbin is built with CUDA 12)
libcuda.so.1Mounted by NVIDIA container runtime at /usr/lib/x86_64-linux-gnu/
/dev/nvidia*Exposed via nvidia.com/gpu device plugin
LD_LIBRARY_PATHSet by Helm template to include /usr/lib/x86_64-linux-gnu/
HOMESet to /tmp because readOnlyRootFilesystem: true

Building

# Local (requires CUDA toolkit on host):
CUDA_LIBRARY_PATH=/tmp/cuda-root \
CUDA_TOOLKIT_PATH=/nix/store/...-cuda12.9-cuda_nvcc-12.9.86/bin \
CUDA_HOME=/tmp/cuda-root \
CUDA_CUDART_INCLUDE=/nix/store/...-cuda12.9-cuda_cudart-.../include \
CUDA_CRT_INCLUDE=/nix/store/...-cuda12.9-cuda_nvcc-.../include \
cargo build --release -p cli --features etl-gpu

# Nix (production):
nix build .#syndb-cli-etl-image

The etl-gpu feature flag enables gpu-cust on the etl crate:

etl-gpu  →  etl-full + etl/gpu-cust
gpu-cust →  gpu + dep:cust + dep:meshcon-gpu-kernels

Helm configuration

GPU jobs are configured in infrastructure/helm/syndb-etl/values.yaml:

prepareJobs:
  - pipeline: mesh-connectomics
    command:
      - syndb
      - etl
      - mesh-connectomics
      - prepare
    gpu:
      enabled: true
      count: 1
      type: nvidia.com/gpu
      shmSize: 8Gi
    resources:
      requests:
        cpu: 4
        memory: 24Gi
      limits:
        cpu: 4800m
        memory: 28Gi

When .gpu.enabled is true, the Helm template automatically:

  • Keeps the same syndb-etl image as CPU jobs; there is no separate GPU image
  • Sets GPU_BACKEND=cust env var
  • Sets LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:/usr/lib64:/usr/lib/nvidia
  • Sets HOME=/tmp
  • Requests nvidia.com/gpu (or custom .gpu.type)
  • Mounts /dev/shm as a memory-backed emptyDir
  • Adds node affinity: nvidia.com/cuda.driver-version.major Gt 524 (driver ≥ R525)
  • Allocates the GPU to the prepare init container (mesh tile jobs: the single tile container, sized by prepare.gpu.count/.type)

Driver floor

The kernels ship as CUDA 12 fatbins, so any driver ≥ R525 loads them without JIT and Helm only requires nvidia.com/cuda.driver-version.major Gt 524. Older notes excluded driver 580 for a cuInit hang on some GPUs; that was never reproduced. If a node does hang, the init watchdog panics after SYNDB_CUDA_INIT_TIMEOUT_SECS (default 120) and the pod fails fast.

Tiled voxel-contact runs

meshConnectome.tiles splits one segmentation volume into dimensions.x × y × z tile Jobs plus a merge Job:

meshConnectome:
  tiles:
    enabled: true
    runId: k1               # new id per run: Job names + S3 exchange prefix
    source: kasthuri-ac4    # registered mesh-connectomics source
    segmentationUrl: https://bossdb-open-data.s3.amazonaws.com/Kasthuri/ac4/neuron/
    overlap: 32             # contacts must fit within the overlap
  prepare:
    gpu:
      enabled: true         # GPU tiles also compute TEASAR skeletons
  • Tiles own a disjoint core box. Each tile writes:

    • morphometry accumulators for its core voxels (neuron_parts.parquet);
    • the synapses centred in its core;
    • its manifest, carrying compute_backend / gpu_seconds.

    Output goes to s3://<bucket>/mesh-connectomics-tiles/<runId>/tiles/<x-y-z>/ with a .ready sentinel.

  • Merge runs the CPU image. It fails as soon as every Job of some tile has failed, and otherwise waits for every sentinel. It sums the accumulators into exact whole-neuron morphometry and writes a manifest that is complete only when all tiles arrived. Finally it publishes to s3://<bucket>/mesh-connectomics-<source>/ with .ready, which the source’s import job consumes.

  • GPU pools: a GPU tile renders one Job per entry of meshConnectome.prepare.gpu.pools (syndb-etl-mesh-tile-<runId>-<x-y-z>-<pool>) and the first pool the scheduler can place runs it. Default pools: gpu (nvidia.com/gpu), a6000, a40, rtx8000, plus a100, h100 and h200 with priorityClassName: opportunistic (those resources have no quota at default priority, so they run preemptible). At start a tile exits 0 if <tile>/.ready already exists, otherwise it writes <tile>/.claim and runs. The claim is informational, not a lock: two pools placed within seconds both run the tile and upload identical outputs. A tile fails the merge only when all of its pool Jobs failed. Pending losers count against the namespace pod quota until their activeDeadlineSeconds, so keep tiles × pools within it or trim pools. An empty pools list falls back to the single gpu.type.

  • Run evidence: the merged gpu_seconds and compute_backend record that the run used the GPU.

GPU node types on NRP Nautilus

The cluster labels nodes with nvidia.com/gpu.product. Selection as of June 2026:

GPUResource nameNodesDriverNotes
RTX 3090nvidia.com/gpu46595
A10nvidia.com/gpu35595
A100nvidia.com/a10022595Quota-gated
L40nvidia.com/gpu17595
RTX 2080 Tinvidia.com/gpu16595Tested + verified
RTX A6000nvidia.com/rtxa600010595
GTX 1080 Tinvidia.com/gpu13580Runs the fatbin
V100nvidia.com/gpu13580Runs the fatbin

For special GPUs (A100, A6000, H100, H200) use their specific resource name instead of nvidia.com/gpu. Quota-gated GPUs can be accessed with priorityClassName: opportunistic.

To pin a specific GPU model:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: nvidia.com/gpu.product
          operator: In
          values:
          - NVIDIA-GeForce-RTX-2080-Ti

GPU benchmark suite

syndb etl mesh-connectomics gpu-bench times the GPU-backed algorithms against their CPU references (cpu-1, cpu-n, gpu) and checks every output bit for bit. The gpuBench chart key (disabled by default, rendered only with jobsEnabled) runs it on every pool of gpuBench.pools plus one CPU Job: syndb-etl-gpu-bench-<runId>-<pool>, each with 2 CPU / 12Gi and one GPU (CPU Job: 16 CPU / 64Gi, --threads 4 --allow-cpu), requests equal to limits. GPU pods run with --gpu-only: they time only gpu-1 and gpu-n (the single-thread reference still runs once per case for the identity check) so the GPU never idles on a CPU baseline; gpu-bench-report takes cpu-1 and cpu-n from the cpu pool of the same run. The bench caps crop sizes at a quarter of the memory limit, so 12Gi reaches 512^3 (EDT); raise gpuBench.resources.gpu.memory (about 26Gi for 640^3) for larger sizes. gpuBench.pools uses the meshConnectome.prepare.gpu.pools format plus an optional products list that pins nvidia.com/gpu.product (p1080ti, v100 and l40s share the generic nvidia.com/gpu resource). a100, h100 and h200 run as opportunistic with a 6 h deadline, the others 4 h; a pool that never schedules is reported as “not scheduled” and blocks nothing. The Jobs carry app=syndb-etl-bench, so the ETL reset of nautilus apply never deletes them. Each Job syncs /work/out to s3://syndb-etl/gpu-bench/<runId>/<pool>/ every 5 minutes and on exit, so a preempted pod still leaves partial results.

Run it from a clean export of HEAD with an image that contains gpu-bench. -l limits the apply to the benchmark Jobs, because jobsEnabled=true also renders every other enabled ETL Job:

helm template syndb-etl infrastructure/helm/syndb-etl \
  -f infrastructure/helm/syndb-etl/gpu-bench-runs/full.yaml \
  --set jobsEnabled=true --set gpuBench.runId=<runid> \
  --set gpuBench.gitSha=<source revision of the image> \
  --set image.digest=sha256:<digest> \
  | kubectl apply -n syndb -l app=syndb-etl-bench -f -

gpu-bench-runs/full.yaml also sets the public Kasthuri tile and the BANC pilot prefix; list the pilot’s tiles (t_X_Y_Z) in gpuBench.bancTile.tiles once its plan.json exists. Collect and analyse the results:

s5cmd sync 's3://syndb-etl/gpu-bench/<runid>/*' DIR/
syndb etl mesh-connectomics gpu-bench-report --in DIR/* --out OUT

The report writes results.parquet, recommendation.toml and one gpu-profile-<dataset>.yaml Helm overlay per dataset. Review it, commit it and apply it to the next run with -f; nothing is applied automatically:

helm template syndb-etl infrastructure/helm/syndb-etl -f OUT/gpu-profile-<dataset>.yaml ...

The profile sets meshConnectome.prepare.gpu.enabled and the pool order that the tile and contact Jobs read; a pool’s optional products list pins nvidia.com/gpu.product there as it does for the benchmark Jobs. The SYNDB_GPU_FIELD_MIN_VOXELS threshold is written to meshConnectome.tiles.env and meshConnectome.contact.env, name to value maps like prepareJobs[].env.

Verifying GPU operation

Check the pod logs for backend selection:

2026-06-26T16:17:14Z INFO etl::gpu: GPU_BACKEND=cust: attempting CUDA backend
2026-06-26T16:17:17Z INFO etl::gpu::cust: cuST backend: CUDA driver initialized in 2.6s
2026-06-26T16:17:17Z INFO etl::gpu::cust: cuST backend: loaded 2 fatbin module(s)
2026-06-26T16:17:17Z INFO etl::gpu: CUDA backend initialized successfully

Kernel dispatch timings appear at DEBUG level:

cuST contacts kernel done in 31.1µs
cuST contacts found 64 raw pairs in 473µs
cuST contacts total=510µs unique=1

On failure the pod exits instead of running on CPU:

panicked at ... GPU_BACKEND=cust but CUDA backend init failed: ...

The GPU/CPU parity tests (gpu_cpu_dt_equivalence, gpu_cpu_contact_equivalence) skip on CPU and run for real only with GPU_BACKEND=cust inside the GPU image.

Troubleshooting

SymptomLikely causeFix
Pod stuck in PendingNo GPU node matching affinityCheck GPU type, quota, and nvidia.com/gpu.product labels
Build panics SYNDB_REQUIRE_GPU_KERNELS=1 but ...nvcc or .cu sources missing from the Nix sourcemkCliEtlSrc must include gpu-kernels/**/*.cu; cuda_nvcc must be in nativeBuildInputs
Module::from_fatbin: ... InvalidSourceDevice compute capability below 5.2, or driver older than R525Schedule on a newer GPU/driver
CUDA init did not finish within 120s panicHung cuInit / driver call on that nodeRetry on another node; raise SYNDB_CUDA_INIT_TIMEOUT_SECS only for slow-starting devices
Process exits 139 after pipelineBenign cleanup crash in Context dropResults are already written — fix destructor ordering if critical
nvcc fatal: Unsupported gpu architectureToolkit too new (CUDA 13 dropped sm_52..sm_70)Keep the build on CUDA 12.x

False leads

Earlier documentation claimed that NVIDIA_VISIBLE_DEVICES=void was the root cause of GPU hangs, implying the Nautilus cluster lacked a device plugin. This was incorrect. The NRP Nautilus cluster does have the NVIDIA device plugin installed. NVIDIA_VISIBLE_DEVICES is correctly set to the allocated GPU UUID (e.g. GPU-e143f1a6-...). The suspected cuInit hang on driver 580 was never reproduced.

OCI image

One image, oci-syndb-etl (docker.io/caniko/syndb-etl, pinned by image.digest), runs on CPU-only and GPU nodes. The binary is built with --features etl-gpu and links libcuda.so.1 as a normal DT_NEEDED dependency (cust does not dlopen it), so it needs some libcuda.so.1 to start:

  • GPU pods: the NVIDIA runtime mounts the real driver and the chart’s LD_LIBRARY_PATH points at it. ld.so searches LD_LIBRARY_PATH before DT_RUNPATH, so the driver always wins.
  • CPU pods: no driver is mounted, so the loader falls back to a stub libcuda.so.1 (copied from the CUDA toolkit stubs) that the build appends to the binary’s RUNPATH. The stub is never called: CUDA is only initialized when GPU_BACKEND=cust, which the chart sets only when gpu.enabled.

The build fails if the binary ends up with DT_RPATH instead of RUNPATH, since RPATH would outrank LD_LIBRARY_PATH and shadow the driver. A GPU_BACKEND=cust pod that cannot see the real driver loads the stub and fails fast at cuInit instead of silently running on CPU.

Files

FilePurpose
crates/tools/etl/gpu-kernels/build.rsNVCC compilation, generates modules.rs
crates/tools/etl/gpu-kernels/src/lib.rsFatbinModule, all_modules()
crates/tools/etl/gpu-kernels/src/edt.cuExact separable Euclidean DT kernels (matches skeleton::distance_transform)
crates/tools/etl/gpu-kernels/src/detect_contacts.cuVoxel contact detection kernel
crates/tools/etl/src/gpu/mod.rsGpuBackend trait + dispatch
crates/tools/etl/src/gpu/cust.rsCustBackend implementation
crates/tools/etl/src/gpu/cpu.rsCPU fallback
nix/packages.nixcudaRoot, cudaStubLibcuda, syndb-cli-etl-image derivation
infrastructure/helm/syndb-etl/templates/_helpers.tplsyndb-etl.nodeAffinity
infrastructure/helm/syndb-etl/templates/etl-prepare-job.yamlGPU pod spec
infrastructure/helm/syndb-etl/values.yaml.gpu block for each job

Building and publishing in the cluster

syndb ops k8s image-build --tag TAG [--images etl,bench] [--rev REV] builds Nix OCI images on Nautilus instead of on the operator host. The checkout is private, so the cluster never clones or evaluates it: the CLI evaluates git+file://...?rev=<committed SHA> (uncommitted edits are never included), exports the derivation closure (a few tens of MiB) to s3://syndb-etl/image-build/, and applies the imageBuild Job of the syndb-etl chart. The pod imports the closure, realises the derivations (substituting from cache.nixos.org and the syndb Attic cache, sandbox = false because the pod is unprivileged), and pushes with skopeo using the dockerhub-push Secret. The CLI prints repository:TAG@sha256:... per image; pin those digests in nautilus/values.yaml yourself. It refuses to start while the syndb-helm-mutation lease is held.

The Nix store is one warm rook-ceph-block PVC, syndb-image-build-store (NRP rejects generic ephemeral volumes and evicts large emptyDirs), so only the first build compiles the ~1,800 derivations; later builds reuse the store and compile or substitute just what the revision changed. The seed container merges the nixImage’s own store into the PVC on every run with nix copy --all, so bumping imageBuild.nixImage adds its paths instead of leaving the build container without a shell. A store above 75% of the 300Gi volume is collected before the build, which makes that one build substitute from scratch. The claim is ReadWriteOnce and never shared: the CLI refuses to start while another app=syndb-image-build Job mounts it. --fresh-store restores the throwaway <job>-store claim and --store-claim PVC mounts an existing claim instead. imageBuild.maxJobs (8) and imageBuild.cores (4) set the derivation parallelism: most cargo-package-* compiles are single-threaded, so parallel derivations, not threads per derivation, bound the build.

NRP Nautilus policy

Every Job of the chart follows the NRP policies:

  • requests and limits are set for cpu, memory and ephemeral-storage, with limits within 20% of requests (ephemeral-storage is request = limit, the emptyDir size plus 2Gi); scripts/check_etl_helm_resources.py enforces it on the rendered ETL, benchmark and image-build Jobs, and that a100/h100/h200 pods are opportunistic;
  • CPU use within 20-200% and memory within 20-150% of the request (at most 4 pods outside it; pods of at most 1 CPU / 2GB are exempt): size from kubectl top of a finished run; the contact tiles run in memory size classes (meshConnectome.contact.tiles.classes, one Job per pool and class, each tile handled by the one class whose owned+halo triangle range holds it), sized from the measured peak of tile-contacts, 135 B per owned+halo triangle plus 240 B per contact node (a 34M-triangle tile needs 10.4 GiB, so a flat 8Gi request OOMs and a flat 24Gi one leaves the median tile at 5%);
  • GPU utilisation above 40%: contact tile pods download the next batch while tile-contacts runs, a merge is applied only after its tiles (no waiting pod holds a request), benchmark GPU pods skip the CPU baselines;
  • at most 50 GPUs without a plan in Nautilus Support: the contact tile pools share meshConnectome.contact.tiles.maxConcurrentGpus (48), split over the classes by weight.