Plan: heterogeneous-benchmark-etl
Plan-set orchestrator recommendation: GPT-5.4 (codex) — effort
highRouted:
carter route -c complex -r planner→gpt-5.4/high(bar 83, capability 89.8)Five phases, four of them implementing new production ETL pipelines in Rust, converging on a fully assembled heterogeneous real-source benchmark bundle that unblocks the manuscript evidence completion plan. Coordinating requires understanding both the ETL crate architecture and the benchmark suite integration — a weaker model would mis-sequence the pipeline registrations or create incompatible data schemas.
Scope
Implement seven new production ETL pipelines (download → prepare → import) that
feed the heterogeneous real-source benchmark bundle for the SynDB manuscript.
Each pipeline creates a registered SynDB dataset with deterministic UUID,
imports its table into ClickHouse, and produces schema-compliant Parquet that
the existing bench prepare-heterogeneous-live-data command can consume.
The table coverage across the seven pipelines:
| ClickHouse table | Pipeline | Source |
|---|---|---|
mitochondria | dryad-mitochondria | Sager et al. 2025 Dryad archive |
neurons | microns-neurons | MICrONS nucleus_detection_v0 CSV |
synapses | microns-synapses | MICrONS synapses_pni_2 CSV (19 GB) |
dendritic_spines | microns-spines | MICrONS multi_input_spine_predictions CSV (positions→0, deferred join in Phase 05) |
axons | microns-skeletons | MICrONS v661 SWC skeletons (compartment=2) |
dendrites | microns-skeletons | MICrONS v661 SWC skeletons (compartment=3/4) |
vesicles | zhang2025-vesicles | BossDB zhang2025 precomputed chunks |
pre_synaptic_terminals | post-process | Derived from synapses (GROUP BY pre_neuron_id) |
Current state (entry conditions)
- Validated against the codebase at commit
HEADof this repository. - ETL crate exists at
crates/tools/etl/with 36 existing dataset modules. - Helm subchart at
infrastructure/helm/syndb-etl/with download/prepare/import templates. - Heterogeneous benchmark bundle code is implemented and tested in
crates/tools/bench/. - Dryad source data is staged at
external_datasets/doi_10_5061_dryad_sn02v6xg1__v20251211/. - MICrONS source CSVs are accessible at
https://storage.cloud.google.com/mat_dbs/. - BossDB zhang2025 precomputed chunks at
s3://bossdb-open-data/zhang2025/. - MICrONS SWC skeletons at
s3://bossdb-open-data/iarpa_microns/minnie/minnie65/skeletons/v661/skeletons/.
Phase table
| Phase | File | Model / Effort | Provider | Depends on | Blocking? | Parallel with |
|---|---|---|---|---|---|---|
| 01 | shipped (dryad-mitochondria) | GPT-5.3-Codex / medium | codex | — | no | 02, 03 |
| 02 | shipped (microns-neurons, microns-synapses, microns-spines) | DeepSeek v4 Flash / high | opencode | — | no | 01, 03 |
| 03 | shipped (microns-skeletons) | GPT-5.3-Codex / medium | codex | — | no | 01, 02 |
| 04 | shipped (bossdb-zhang2025) | GPT-5.4 / high | codex | 03 (SWC pattern) | no | — |
| 05 | 05-post-process-bundle-retire.md | DeepSeek v4 Flash / high | opencode | 01, 02, 03, 04 | final | — |
Phases 01-04 shipped; their pipelines are registered in
crates/core/shared/src/etl_registry/pipelines.rs and their execution notes
remain in Git history. Phase 05 remains open; its pre-synaptic-terminals
pipeline is already registered.
Phases 01-03 are independent (disjoint repos/modules) and can run concurrently.
Phase 04 benefits from having the SwcSkeletons runtime-family pattern from 03
as reference but does not strictly depend on it at the data level.
Phase 05 requires all prior pipelines to be green and all 8 tables imported.
Parallelism layer
Wave 0 — fan out (no cross-conflicts)
Run concurrently:
- 01 — dryad-mitochondria module,
ManualCaveExportruntime family, local test. - 02 — microns-neurons, microns-synapses, microns-spines modules,
SmallStatic/BulkPublicPreparedfamilies. - 03 — microns-skeletons module,
SwcSkeletonsruntime family, SWC parser.
All three touch different files under crates/tools/etl/src/ and different
ETL_RUNTIME_REGISTRY entries. No shared-file conflicts.
Wave 1 — vesicle extraction
- 04 — zhang2025-vesicles module,
BossDbVesiclesruntime family,compressed_segmentationdecoder. Can reference the SWC pattern from 03.
Wave 2 — final integration
- 05 — post-process pipeline, Helm registration for all 7 pipelines, heterogeneous bundle assembly, manuscript caption edits, plan retirement.
Serialization points
| File | Phase conflict | Order |
|---|---|---|
crates/tools/etl/src/registry/runtime.rs | All phases add entries | 01/02/03 → (git merge) → 04 → 05 |
crates/tools/etl/src/lib.rs | All phases register modules | Same order |
infrastructure/helm/nautilus/values.yaml | 05 only | — |
article/sections/supplementary_figures.tex | 05 only | — |
After each conflict file lands, run the build gate before the next phase rebases onto it.
Known deferred item
- Spine positions (dendritic_spines): Positions are hardcoded to 0 and
parent_links_real: falsebecause the GCS spine predictions CSV has no spatial columns. A post-K8s-run followup in Phase 05 joinstarget_id(spine) →id(synapse) to fillcentroid_{x,y,z}from the synapse Parquet output. Until then, the heterogeneous manifest marks this component withparent_links_real: false.
Whole-set acceptance criteria
- All 8 benchmark tables (
neurons,synapses,dendritic_spines,axons,dendrites,pre_synaptic_terminals,vesicles,mitochondria) have real rows imported from production-quality public sources. - Each pipeline is registered in
ETL_RUNTIME_REGISTRYwith a deterministic UUID and its Helm job declaration invalues.yaml. -
syndb bench prepare-heterogeneous-live-data --source-manifest <heterogeneous-yaml> --scale 100k,1m,10mproduceslive-source-manifest.jsonwithbundle_mode: "heterogeneous_real". -
syndb bench validate-nautilus-results --article-grade --require-real-apipasses against the heterogeneous bundle. -
article/sections/supplementary_figures.texlines 113, 131, 150, 166, 181 no longer claim “controlled synthetic” for CF04/CF11/CF14/CF16; caption references point at the heterogeneous real-source provenance. - The planning tree at
docs/src/planning/manuscript-evidence-completion/is retired fromSUMMARY.mdperretire-docs-planningclean-shipped rules. -
cargo test -p syndb-bench live_sourcepasses with the new heterogeneous manifest fixtures. -
cargo check -p cli --features dev,etl,graph-precompute,benchpasses.
Global constraints (apply to every phase)
- Build gate after every registry edit:
cargo check -p cli --features dev,etl(orcargo check -p cli --features dev,etl,graph-precompute,benchfor bench-touching changes). 0 errors, 0 warnings. - Never commit Dryad zip or other large binaries to the git repo. Stage only code and small test fixtures.
- Dataset UUIDs must be deterministic via
dataset_id!macro +DatasetIdtrait — no random UUIDs at runtime. - Signed commits using the user’s global git config as-is.
- No hand-edited biology numerals in manuscript files.
Reference
- Existing ETL patterns:
crates/tools/etl/src/hemibrain/(download+prepare+import pattern),crates/tools/etl/src/pipeline.rs(streaming import),crates/tools/etl/src/transform.rs(CsvBatchReader). - ClickHouse table schemas:
crates/core/syndb-clickhouse/src/schema/columns.rs(all 8 benchmark tables). - Heterogeneous benchmark bundle:
crates/tools/bench/src/live_source.rs(manifest structs, validation code). - Helm ETL templates:
infrastructure/helm/syndb-etl/templates/etl-job.yaml,etl-download-job.yaml,etl-prepare-job.yaml. - Dryad data:
external_datasets/doi_10_5061_dryad_sn02v6xg1__v20251211/. - Precomputed format spec:
https://github.com/google/neuroglancer/tree/master/src/neuroglancer/datasource/precomputed.