Phase 05 — Post-process pipeline, bundle assembly, manuscript update, plan retirement
Recommended model: DeepSeek v4 Flash (opencode) — effort
highRouted:
carter route -c moderate -r subagent --needs coding,writing→deepseek/deepseek-v4-flash/high(bar 72, capability 75.3)Integrates the outputs of all prior phases: creates the post-process (pre_synaptic_terminals) pipeline, writes the final Helm values for all 7 pipelines, assembles the heterogeneous benchmark bundle, updates manuscript captions, and retires the plan-set. Mostly wiring and prose; moderate complexity.
Working tree
Repository root. Requires all prior phases (01-04) to have
completed and their pipelines to be registered in runtime.rs and lib.rs.
Goal
- Create the
post-processpipeline that derivespre_synaptic_terminalsfrom themicrons-synapsesimport output. - Declare all 7 pipeline Helm jobs in
values.yaml. - Build and validate the heterogeneous real-source benchmark bundle.
- Update manuscript captions to reflect real-source provenance.
- Retire the manuscript-evidence-completion plan-set from SUMMARY.md.
Why this matters now
This is the final integration step that converts all individual pipeline outputs into a single coherent benchmark bundle, updates the manuscript evidence claims, and marks the planning work as complete.
Out of scope
- Any new ETL module creation (all 7 pipelines are registered in prior phases).
- Re-implementing existing benchmark or validation logic.
- Running the actual benchmark suite on the cluster (done by the user).
Plan
1. Create post-process pipeline
Create crates/tools/etl/src/post_process/:
id.rs—PostProcessmarker structmod.rs—EtlTableimpl forpre_synaptic_terminals- No download step. The prepare step reads from the already-imported
microns-synapsesdataset via ClickHouse or its prepared Parquet on S3. - Transform:
SELECT pre_neuron_id, COUNT(*) as forms_synapse_count, AVG(centroid_x) as cx, AVG(centroid_y) as cy, AVG(centroid_z) as cz FROM synapses GROUP BY pre_neuron_id - Write
pre_synaptic_terminals.parquetmatchingTABLE_COLUMNS - Register in
runtime.rsandlib.rs
2. Write final Helm values
Update infrastructure/helm/nautilus/values.yaml under syndb-etl::
syndb-etl:
downloadJobs:
- pipeline: dryad-mitochondria
emptyDirSizeLimit: 16Gi
- pipeline: microns-neurons
emptyDirSizeLimit: 1Gi
- pipeline: microns-synapses
emptyDirSizeLimit: 25Gi
- pipeline: microns-spines
emptyDirSizeLimit: 1Gi
- pipeline: microns-skeletons
emptyDirSizeLimit: 4Gi
- pipeline: zhang2025-vesicles
emptyDirSizeLimit: 8Gi
prepareJobs:
- pipeline: dryad-mitochondria
emptyDirSizeLimit: 16Gi
- pipeline: microns-synapses
emptyDirSizeLimit: 25Gi
prepareResources: {requests: {memory: 8Gi}}
- pipeline: microns-skeletons
emptyDirSizeLimit: 4Gi
- pipeline: zhang2025-vesicles
emptyDirSizeLimit: 8Gi
jobs:
- pipeline: dryad-mitochondria
table: mitochondria
- pipeline: microns-neurons
table: neurons
- pipeline: microns-synapses
table: synapses
- pipeline: microns-spines
table: dendritic_spines
- pipeline: microns-skeletons
tables: [axons, dendrites]
- pipeline: zhang2025-vesicles
table: vesicles
- pipeline: post-process
table: pre_synaptic_terminals
2b. Spine reference reconciliation
Implemented source path on 8 October 2026:
syndb etl microns-spines prepare-positions reads local v1412 reference and
detailed synapse CSVs, groups shared postsynaptic compartments, resolves
post_pt_root_id ownership, and publishes hash-bound Parquet plus source.json.
The source contract and commands
describe explicit units, memory bounds and import validation.
The historical proposal treated target_id as a neuron key and assigned
presynaptic ownership. Both are incorrect: it references a synapse, and
group_id identifies a shared postsynaptic spine prediction. Positions are
now explicitly labeled means of member synaptic-junction locations, not
measured spine centroids. Missing morphology remains unknown.
Stage and validate the complete joined output before updating production jobs
or rebuilding the heterogeneous benchmark bundle. Set parent_links_real: true
only after proving that the resulting postsynaptic neuron UUIDs belong to the
bundle’s neuron universe. Full-source and deployment acceptance are still open.
3. Assemble heterogeneous benchmark bundle
- Write the
heterogeneous-source-manifest.yamlthat references all 8 tables and their source components from the 7 imported datasets. - Run
prepare-heterogeneous-live-data:cargo run -p cli --features dev,etl,graph-precompute,bench -- \ bench prepare-heterogeneous-live-data \ --source-manifest benchmark/heterogeneous-source-manifest.yaml \ --output-dir benchmark/live-data \ --scale 100k,1m,10m - Run validation:
test -s benchmark/live-data/live-source-manifest.json grep '"bundle_mode": "heterogeneous_real"' benchmark/live-data/live-source-manifest.json
4. Update manuscript captions
Edit article/sections/supplementary_figures.tex:
- Line 113 (
analytics_engine/CF04): Replace “retained controlled synthetic benchmark artifact” with “heterogeneous real-source benchmark bundle (provenance in Methods §Interoperability round-trip)” - Line 131 (
engineering/CF11): Same pattern - Line 150 (
benchmark_runtime_supplement/CF14): Same - Line 166 (
query_scaling_supplement/CF16): Same - Line 181 (
benchmark_systems_supplement/CF15): Keep as explicit simulated (federation latency requires multi-cluster RTT) - Line 109: Remove
(synthetic controls)from section header
5. Retire manuscript-evidence-completion plan-set
- Update
documentation/src/SUMMARY.md: remove lines 74-90 (the manuscript-evidence-completion section) - Optionally archive the plan files (leave on disk for git history, just remove from published navigation)
- Verify docs build:
mdbook build documentation/
6. Final validation
# Full crate check
cargo check -p cli --features dev,etl,graph-precompute,bench
# Existing heterogeneous manifest tests
cargo test -p syndb-bench live_source
Acceptance criteria
-
post-processpipeline compiles and producespre_synaptic_terminals.parquetmatchingPRE_SYNAPTIC_TERMINALS_COLUMNS. - All 7 pipelines declared in
values.yamlwith per-pipeline resource specs. -
live-source-manifest.jsonhasdata_source: "real"andbundle_mode: "heterogeneous_real". -
article/sections/supplementary_figures.texhas zero occurrences of “controlled synthetic” for CF04/CF11/CF14/CF16; CF15 remains explicitly simulated. -
documentation/src/SUMMARY.mdno longer lists the manuscript-evidence-completion plan. -
mdbook build documentation/succeeds. -
cargo test -p syndb-bench live_sourcepasses.
Files likely touched
crates/tools/etl/src/post_process/mod.rs,id.rs— newcrates/tools/etl/src/lib.rs— register post_processcrates/tools/etl/src/registry/runtime.rs— register post_processinfrastructure/helm/nautilus/values.yaml— all 7 Helm job declarationsbenchmark/heterogeneous-source-manifest.yaml— new manifestarticle/sections/supplementary_figures.tex— caption editsdocumentation/src/SUMMARY.md— remove plan section
Pitfalls
- Symptom:
post-processneeds the synapse Parquet output path. Cause: The pipeline reads from another pipeline’s prepared data. Recovery: Reference it by S3 prefixs3://syndb-etl/microns-synapses-prepared/. - Symptom: Helm values grow very long. Cause: 7 new pipelines declared inline. Recovery: Keep the pattern compact; each pipeline is ~4 lines.
- Symptom: Caption edit inadvertently changes figure numbering. Cause:
supplementary_figures.texuses\suppfigure{label}which maps throughfigure_inputs.tex. Recovery: Only edit caption prose, not\suppfigure{}arguments or order.
Reference
- Existing Helm values:
infrastructure/helm/nautilus/values.yamllines 830-1202 - Supplementary figure labels → CF mapping:
article/figures/figure_inputs.tex - Caption text to edit:
article/sections/supplementary_figures.texlines 109-196 - Heterogeneous benchmark code:
crates/tools/bench/src/live_source.rs - Plan retirement skill:
retire-docs-planning(atdocs/src/planning/retire-docs-planning/SKILL.md)