Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Mesh Connectomics Pipeline — Phase Execution Plan

This README is the coordination surface. The former phase files were execution scaffolding and have been retired; durable implementation notes live in the design spec, ADRs, and current source.

Current state summary (updated)

Landed (previously planned, now verified in code):

  • ✅ Geometry is wired: score_and_assign calls contact_geometry → geometry::contact_normals + voxel_curvature, and builds real LocalMorphology (pipeline.rs:410, 446). The old P1 “zeros/stubs” gap is closed.
  • ✅ Process classification wired: classify_process_with_skeleton + determine_direction drive pre/post assignment (pipeline.rs:455-491).
  • ✅ Watershed splitting: watershed_split called in pipeline.rs:434 for contacts > 2× typical PSD area.
  • ✅ Configuration-model null model: configuration_model_stats in graph.rs:182 replaces the biased Erdős–Rényi.
  • ✅ Flat-array Dijkstra: FlatDijkstra in skeleton.rs with bit-packed visited and BinaryHeap (partial — extract_path still uses HashMap parents/visited).
  • ✅ CUDA cust backend + PTX kernels: chamfer_dt.cu, detect_contacts.cu compiled to PTX via nvcc; gpu::select_backend() routes to CustBackend when GPU_BACKEND=cust.
  • ✅ Multi-resolution contact detection: detect_voxel_contacts_multires with downsampling + refinement.

Newly implemented (waves A–C + finalize):

  • ✅ Cell-type GMM classification (Stage 7): classify_cell_types in pipeline.rs — runtime EM training or JSON model loading; writes cell_type and cell_type_confidence to neurons.parquet. Closes the previous dead-code gap in celltype.rs.
  • ✅ Hybrid merge preserves BVH signal: merge_contact_candidates now prefers mesh-derived geometry over voxel default placeholders.
  • ✅ Confidence tiers in synapse output: confidence_tier column (Confirmed/Putative/Rejected) in synapses.parquet.
  • ✅ CUDA watershed kernel: watershed.cu — relaxation-based basin labeling kernel.
  • ✅ In-tree tests: Added test coverage for graph.rs (configuration_model_stats, small-world index), assign.rs (confidence tiers), voxel_contact.rs, synapse_aware.rs.
  • ✅ CI validation gate: just validate-mesh-pipeline — conditional on $SYNDB_MICRONS_CACHE.
  • ✅ Parallel skeleton: extract_all_skeletons uses rayon::par_iter() for CPU path; GPU path remains serial (channel-based dispatch).
  • ✅ Flat-array build_skeleton_tree: all HashMaps/HashSets replaced with Vec<> + bit-packed visited.
  • ✅ GB-tree scorer wired: PipelineConfig.scorer + SynapseScorer integrated; score_and_assign uses configurable scorer (logistic or GB). Python training script at scripts/train_gb_tree.py.
  • ✅ Criterion benchmarks: 4 benchmark suites (contact detection, scoring, skeleton, graph construction) in crates/tools/etl/benches/.
  • ✅ Configuration-model parallelism: configuration_model_stats iterations run in parallel via rayon; per-iteration clustering uses adjacency list (O(V+E) instead of O(V·E)).
  • ✅ Pure-Rust Draco decoder: draco-core crate replaces subprocess draco_decoder CLI as primary path; CLI fallback retained. Hermetic container builds.
  • ✅ CI pipeline gate: just validate-mesh-pipeline generates a synthetic 32³ volume and runs the pipeline end-to-end with assertions. No external data needed.
  • ✅ Active learning loop: scripts/active_learn.py — exports top-N uncertain predictions with Neuroglancer state URLs, supports retraining with human labels.

Remaining (research / out-of-scope):

  • ⏳ GNN synapse detection — research-only item; not production-ready.

Phase table

PhaseLabelStatusTouches
P1Cell-type GMM wiring✅ Donepipeline.rs, celltype.rs, morphology.rs
P2Hybrid merge fix✅ Donepipeline.rs
P3Confidence tiers in output✅ Doneassign.rs, pipeline.rs
P4In-tree tests + fixtures✅ Donegraph.rs, voxel_contact.rs, synapse_aware.rs
P5CI validation gate✅ Donejustfile
P6CUDA watershed kernel✅ Donegpu-kernels/src/watershed.cu
P9Parallel skeleton (rayon)✅ Landed; GPU path remains serial by designskeleton.rs
P10Flat-array extract_path◐ Partial; parent/visited maps remain in extract_pathskeleton.rs
P11Criterion benchmarks✅ Landedbenches/
P12GB-tree scoring wiring✅ Landedscoring.rs, Python
P13Active learning pipeline✅ Landed; quality gate remains openscripts/

Whole-set acceptance criteria

  • Pipeline produces cell-type labels and confidence tiers in Parquet output
  • Hybrid mode preserves mesh geometry when available
  • Watershed CUDA kernel compiles alongside existing kernels
  • Skeleton extraction completes a full 8.5K-neuron run in ≤5 min (down from ~2h) — needs parallel
  • MICrONS validation reports precision, recall, F1 per confidence tier
  • GB tree scoring outperforms logistic regression on held-out MICrONS data (higher F1)
  • GPU backend produces bit-identical results to CPU on a small test volume
  • cargo test -p etl passes after every phase
  • cargo clippy -p etl --all-features -- -D warnings passes after every phase

Global constraints

  • All phases work within crates/tools/etl/src/neuroglancer/ unless otherwise noted
  • crates/tools/etl/Cargo.toml may need dependency additions — coordinate with the user to avoid conflicts
  • GPU builds use the stable cust host API plus NVCC-compiled PTX. The gpu-cust feature remains optional; CPU builds require no CUDA toolchain.
  • Python phases are standalone scripts, not production ETL code
  • No phase should introduce non-optional heavy dependencies without a feature flag

Reference

  • Original design proposal
  • Pipeline entry point: crates/tools/etl/src/mesh_connectomics/mod.rs
  • Pipeline core: crates/tools/etl/src/neuroglancer/pipeline.rs
  • GPU abstraction: crates/tools/etl/src/gpu/
  • Codeberg: https://codeberg.org/caniko/syndb