Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Plan: heterogeneous-benchmark-etl

Plan-set orchestrator recommendation: GPT-5.4 (codex) — effort high

Routed: carter route -c complex -r planner → gpt-5.4 / high (bar 83, capability 89.8)

Five phases, four of them implementing new production ETL pipelines in Rust, converging on a fully assembled heterogeneous real-source benchmark bundle that unblocks the manuscript evidence completion plan. Coordinating requires understanding both the ETL crate architecture and the benchmark suite integration — a weaker model would mis-sequence the pipeline registrations or create incompatible data schemas.

Scope

Implement seven new production ETL pipelines (download → prepare → import) that feed the heterogeneous real-source benchmark bundle for the SynDB manuscript. Each pipeline creates a registered SynDB dataset with deterministic UUID, imports its table into ClickHouse, and produces schema-compliant Parquet that the existing bench prepare-heterogeneous-live-data command can consume.

The table coverage across the seven pipelines:

ClickHouse tablePipelineSource
mitochondriadryad-mitochondriaSager et al. 2025 Dryad archive
neuronsmicrons-neuronsMICrONS nucleus_detection_v0 CSV
synapsesmicrons-synapsesMICrONS synapses_pni_2 CSV (19 GB)
dendritic_spinesmicrons-spinesMICrONS multi_input_spine_predictions CSV (positions→0, deferred join in Phase 05)
axonsmicrons-skeletonsMICrONS v661 SWC skeletons (compartment=2)
dendritesmicrons-skeletonsMICrONS v661 SWC skeletons (compartment=3/4)
vesicleszhang2025-vesiclesBossDB zhang2025 precomputed chunks
pre_synaptic_terminalspost-processDerived from synapses (GROUP BY pre_neuron_id)

Current state (entry conditions)

  • Validated against the codebase at commit HEAD of this repository.
  • ETL crate exists at crates/tools/etl/ with 36 existing dataset modules.
  • Helm subchart at infrastructure/helm/syndb-etl/ with download/prepare/import templates.
  • Heterogeneous benchmark bundle code is implemented and tested in crates/tools/bench/.
  • Dryad source data is staged at external_datasets/doi_10_5061_dryad_sn02v6xg1__v20251211/.
  • MICrONS source CSVs are accessible at https://storage.cloud.google.com/mat_dbs/.
  • BossDB zhang2025 precomputed chunks at s3://bossdb-open-data/zhang2025/.
  • MICrONS SWC skeletons at s3://bossdb-open-data/iarpa_microns/minnie/minnie65/skeletons/v661/skeletons/.

Phase table

PhaseFileModel / EffortProviderDepends onBlocking?Parallel with
01shipped (dryad-mitochondria)GPT-5.3-Codex / mediumcodex—no02, 03
02shipped (microns-neurons, microns-synapses, microns-spines)DeepSeek v4 Flash / highopencode—no01, 03
03shipped (microns-skeletons)GPT-5.3-Codex / mediumcodex—no01, 02
04shipped (bossdb-zhang2025)GPT-5.4 / highcodex03 (SWC pattern)no—
0505-post-process-bundle-retire.mdDeepSeek v4 Flash / highopencode01, 02, 03, 04final—

Phases 01-04 shipped; their pipelines are registered in crates/core/shared/src/etl_registry/pipelines.rs and their execution notes remain in Git history. Phase 05 remains open; its pre-synaptic-terminals pipeline is already registered.

Phases 01-03 are independent (disjoint repos/modules) and can run concurrently. Phase 04 benefits from having the SwcSkeletons runtime-family pattern from 03 as reference but does not strictly depend on it at the data level. Phase 05 requires all prior pipelines to be green and all 8 tables imported.

Parallelism layer

Wave 0 — fan out (no cross-conflicts)

Run concurrently:

  • 01 — dryad-mitochondria module, ManualCaveExport runtime family, local test.
  • 02 — microns-neurons, microns-synapses, microns-spines modules, SmallStatic/BulkPublicPrepared families.
  • 03 — microns-skeletons module, SwcSkeletons runtime family, SWC parser.

All three touch different files under crates/tools/etl/src/ and different ETL_RUNTIME_REGISTRY entries. No shared-file conflicts.

Wave 1 — vesicle extraction

  • 04 — zhang2025-vesicles module, BossDbVesicles runtime family, compressed_segmentation decoder. Can reference the SWC pattern from 03.

Wave 2 — final integration

  • 05 — post-process pipeline, Helm registration for all 7 pipelines, heterogeneous bundle assembly, manuscript caption edits, plan retirement.

Serialization points

FilePhase conflictOrder
crates/tools/etl/src/registry/runtime.rsAll phases add entries01/02/03 → (git merge) → 04 → 05
crates/tools/etl/src/lib.rsAll phases register modulesSame order
infrastructure/helm/nautilus/values.yaml05 only—
article/sections/supplementary_figures.tex05 only—

After each conflict file lands, run the build gate before the next phase rebases onto it.

Known deferred item

  • Spine positions (dendritic_spines): Positions are hardcoded to 0 and parent_links_real: false because the GCS spine predictions CSV has no spatial columns. A post-K8s-run followup in Phase 05 joins target_id (spine) → id (synapse) to fill centroid_{x,y,z} from the synapse Parquet output. Until then, the heterogeneous manifest marks this component with parent_links_real: false.

Whole-set acceptance criteria

  • All 8 benchmark tables (neurons, synapses, dendritic_spines, axons, dendrites, pre_synaptic_terminals, vesicles, mitochondria) have real rows imported from production-quality public sources.
  • Each pipeline is registered in ETL_RUNTIME_REGISTRY with a deterministic UUID and its Helm job declaration in values.yaml.
  • syndb bench prepare-heterogeneous-live-data --source-manifest <heterogeneous-yaml> --scale 100k,1m,10m produces live-source-manifest.json with bundle_mode: "heterogeneous_real".
  • syndb bench validate-nautilus-results --article-grade --require-real-api passes against the heterogeneous bundle.
  • article/sections/supplementary_figures.tex lines 113, 131, 150, 166, 181 no longer claim “controlled synthetic” for CF04/CF11/CF14/CF16; caption references point at the heterogeneous real-source provenance.
  • The planning tree at docs/src/planning/manuscript-evidence-completion/ is retired from SUMMARY.md per retire-docs-planning clean-shipped rules.
  • cargo test -p syndb-bench live_source passes with the new heterogeneous manifest fixtures.
  • cargo check -p cli --features dev,etl,graph-precompute,bench passes.

Global constraints (apply to every phase)

  • Build gate after every registry edit: cargo check -p cli --features dev,etl (or cargo check -p cli --features dev,etl,graph-precompute,bench for bench-touching changes). 0 errors, 0 warnings.
  • Never commit Dryad zip or other large binaries to the git repo. Stage only code and small test fixtures.
  • Dataset UUIDs must be deterministic via dataset_id! macro + DatasetId trait — no random UUIDs at runtime.
  • Signed commits using the user’s global git config as-is.
  • No hand-edited biology numerals in manuscript files.

Reference

  • Existing ETL patterns: crates/tools/etl/src/hemibrain/ (download+prepare+import pattern), crates/tools/etl/src/pipeline.rs (streaming import), crates/tools/etl/src/transform.rs (CsvBatchReader).
  • ClickHouse table schemas: crates/core/syndb-clickhouse/src/schema/columns.rs (all 8 benchmark tables).
  • Heterogeneous benchmark bundle: crates/tools/bench/src/live_source.rs (manifest structs, validation code).
  • Helm ETL templates: infrastructure/helm/syndb-etl/templates/etl-job.yaml, etl-download-job.yaml, etl-prepare-job.yaml.
  • Dryad data: external_datasets/doi_10_5061_dryad_sn02v6xg1__v20251211/.
  • Precomputed format spec: https://github.com/google/neuroglancer/tree/master/src/neuroglancer/datasource/precomputed.