Mesh Connectomics Pipeline — Phase Execution Plan
This README is the coordination surface. The former phase files were execution scaffolding and have been retired; durable implementation notes live in the design spec, ADRs, and current source.
Current state summary (updated)
Landed (previously planned, now verified in code):
- ✅ Geometry is wired:
score_and_assigncallscontact_geometry→geometry::contact_normals+voxel_curvature, and builds realLocalMorphology(pipeline.rs:410, 446). The old P1 “zeros/stubs” gap is closed. - ✅ Process classification wired:
classify_process_with_skeleton+determine_directiondrive pre/post assignment (pipeline.rs:455-491). - ✅ Watershed splitting:
watershed_splitcalled inpipeline.rs:434for contacts > 2× typical PSD area. - ✅ Configuration-model null model:
configuration_model_statsingraph.rs:182replaces the biased Erdős–Rényi. - ✅ Flat-array Dijkstra:
FlatDijkstrainskeleton.rswith bit-packed visited andBinaryHeap(partial —extract_pathstill usesHashMapparents/visited). - ✅ CUDA
custbackend + PTX kernels:chamfer_dt.cu,detect_contacts.cucompiled to PTX via nvcc;gpu::select_backend()routes toCustBackendwhenGPU_BACKEND=cust. - ✅ Multi-resolution contact detection:
detect_voxel_contacts_multireswith downsampling + refinement.
Newly implemented (waves A–C + finalize):
- ✅ Cell-type GMM classification (Stage 7):
classify_cell_typesinpipeline.rs— runtime EM training or JSON model loading; writescell_typeandcell_type_confidencetoneurons.parquet. Closes the previous dead-code gap incelltype.rs. - ✅ Hybrid merge preserves BVH signal:
merge_contact_candidatesnow prefers mesh-derived geometry over voxel default placeholders. - ✅ Confidence tiers in synapse output:
confidence_tiercolumn (Confirmed/Putative/Rejected) insynapses.parquet. - ✅ CUDA watershed kernel:
watershed.cu— relaxation-based basin labeling kernel. - ✅ In-tree tests: Added test coverage for
graph.rs(configuration_model_stats, small-world index),assign.rs(confidence tiers),voxel_contact.rs,synapse_aware.rs. - ✅ CI validation gate:
just validate-mesh-pipeline— conditional on$SYNDB_MICRONS_CACHE. - ✅ Parallel skeleton:
extract_all_skeletonsusesrayon::par_iter()for CPU path; GPU path remains serial (channel-based dispatch). - ✅ Flat-array
build_skeleton_tree: all HashMaps/HashSets replaced withVec<>+ bit-packed visited. - ✅ GB-tree scorer wired:
PipelineConfig.scorer+SynapseScorerintegrated;score_and_assignuses configurable scorer (logistic or GB). Python training script atscripts/train_gb_tree.py. - ✅ Criterion benchmarks: 4 benchmark suites (contact detection, scoring, skeleton, graph construction) in
crates/tools/etl/benches/. - ✅ Configuration-model parallelism:
configuration_model_statsiterations run in parallel via rayon; per-iteration clustering uses adjacency list (O(V+E) instead of O(V·E)). - ✅ Pure-Rust Draco decoder:
draco-corecrate replaces subprocessdraco_decoderCLI as primary path; CLI fallback retained. Hermetic container builds. - ✅ CI pipeline gate:
just validate-mesh-pipelinegenerates a synthetic 32³ volume and runs the pipeline end-to-end with assertions. No external data needed. - ✅ Active learning loop:
scripts/active_learn.py— exports top-N uncertain predictions with Neuroglancer state URLs, supports retraining with human labels.
Remaining (research / out-of-scope):
- ⏳ GNN synapse detection — research-only item; not production-ready.
Phase table
| Phase | Label | Status | Touches |
|---|---|---|---|
| P1 | Cell-type GMM wiring | ✅ Done | pipeline.rs, celltype.rs, morphology.rs |
| P2 | Hybrid merge fix | ✅ Done | pipeline.rs |
| P3 | Confidence tiers in output | ✅ Done | assign.rs, pipeline.rs |
| P4 | In-tree tests + fixtures | ✅ Done | graph.rs, voxel_contact.rs, synapse_aware.rs |
| P5 | CI validation gate | ✅ Done | justfile |
| P6 | CUDA watershed kernel | ✅ Done | gpu-kernels/src/watershed.cu |
| P9 | Parallel skeleton (rayon) | ✅ Landed; GPU path remains serial by design | skeleton.rs |
| P10 | Flat-array extract_path | ◐ Partial; parent/visited maps remain in extract_path | skeleton.rs |
| P11 | Criterion benchmarks | ✅ Landed | benches/ |
| P12 | GB-tree scoring wiring | ✅ Landed | scoring.rs, Python |
| P13 | Active learning pipeline | ✅ Landed; quality gate remains open | scripts/ |
Whole-set acceptance criteria
- Pipeline produces cell-type labels and confidence tiers in Parquet output
- Hybrid mode preserves mesh geometry when available
- Watershed CUDA kernel compiles alongside existing kernels
- Skeleton extraction completes a full 8.5K-neuron run in ≤5 min (down from ~2h) — needs parallel
- MICrONS validation reports precision, recall, F1 per confidence tier
- GB tree scoring outperforms logistic regression on held-out MICrONS data (higher F1)
- GPU backend produces bit-identical results to CPU on a small test volume
-
cargo test -p etlpasses after every phase -
cargo clippy -p etl --all-features -- -D warningspasses after every phase
Global constraints
- All phases work within
crates/tools/etl/src/neuroglancer/unless otherwise noted crates/tools/etl/Cargo.tomlmay need dependency additions — coordinate with the user to avoid conflicts- GPU builds use the stable
custhost API plus NVCC-compiled PTX. Thegpu-custfeature remains optional; CPU builds require no CUDA toolchain. - Python phases are standalone scripts, not production ETL code
- No phase should introduce non-optional heavy dependencies without a feature flag
Reference
- Original design proposal
- Pipeline entry point:
crates/tools/etl/src/mesh_connectomics/mod.rs - Pipeline core:
crates/tools/etl/src/neuroglancer/pipeline.rs - GPU abstraction:
crates/tools/etl/src/gpu/ - Codeberg: https://codeberg.org/caniko/syndb