Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Phase 05 — Post-process pipeline, bundle assembly, manuscript update, plan retirement

Recommended model: DeepSeek v4 Flash (opencode) — effort high

Routed: carter route -c moderate -r subagent --needs coding,writing → deepseek/deepseek-v4-flash / high (bar 72, capability 75.3)

Integrates the outputs of all prior phases: creates the post-process (pre_synaptic_terminals) pipeline, writes the final Helm values for all 7 pipelines, assembles the heterogeneous benchmark bundle, updates manuscript captions, and retires the plan-set. Mostly wiring and prose; moderate complexity.

Working tree

Repository root. Requires all prior phases (01-04) to have completed and their pipelines to be registered in runtime.rs and lib.rs.

Goal

  1. Create the post-process pipeline that derives pre_synaptic_terminals from the microns-synapses import output.
  2. Declare all 7 pipeline Helm jobs in values.yaml.
  3. Build and validate the heterogeneous real-source benchmark bundle.
  4. Update manuscript captions to reflect real-source provenance.
  5. Retire the manuscript-evidence-completion plan-set from SUMMARY.md.

Why this matters now

This is the final integration step that converts all individual pipeline outputs into a single coherent benchmark bundle, updates the manuscript evidence claims, and marks the planning work as complete.

Out of scope

  • Any new ETL module creation (all 7 pipelines are registered in prior phases).
  • Re-implementing existing benchmark or validation logic.
  • Running the actual benchmark suite on the cluster (done by the user).

Plan

1. Create post-process pipeline

Create crates/tools/etl/src/post_process/:

  • id.rs — PostProcess marker struct
  • mod.rs — EtlTable impl for pre_synaptic_terminals
  • No download step. The prepare step reads from the already-imported microns-synapses dataset via ClickHouse or its prepared Parquet on S3.
  • Transform: SELECT pre_neuron_id, COUNT(*) as forms_synapse_count, AVG(centroid_x) as cx, AVG(centroid_y) as cy, AVG(centroid_z) as cz FROM synapses GROUP BY pre_neuron_id
  • Write pre_synaptic_terminals.parquet matching TABLE_COLUMNS
  • Register in runtime.rs and lib.rs

2. Write final Helm values

Update infrastructure/helm/nautilus/values.yaml under syndb-etl::

syndb-etl:
  downloadJobs:
    - pipeline: dryad-mitochondria
      emptyDirSizeLimit: 16Gi
    - pipeline: microns-neurons
      emptyDirSizeLimit: 1Gi
    - pipeline: microns-synapses
      emptyDirSizeLimit: 25Gi
    - pipeline: microns-spines
      emptyDirSizeLimit: 1Gi
    - pipeline: microns-skeletons
      emptyDirSizeLimit: 4Gi
    - pipeline: zhang2025-vesicles
      emptyDirSizeLimit: 8Gi
  prepareJobs:
    - pipeline: dryad-mitochondria
      emptyDirSizeLimit: 16Gi
    - pipeline: microns-synapses
      emptyDirSizeLimit: 25Gi
      prepareResources: {requests: {memory: 8Gi}}
    - pipeline: microns-skeletons
      emptyDirSizeLimit: 4Gi
    - pipeline: zhang2025-vesicles
      emptyDirSizeLimit: 8Gi
  jobs:
    - pipeline: dryad-mitochondria
      table: mitochondria
    - pipeline: microns-neurons
      table: neurons
    - pipeline: microns-synapses
      table: synapses
    - pipeline: microns-spines
      table: dendritic_spines
    - pipeline: microns-skeletons
      tables: [axons, dendrites]
    - pipeline: zhang2025-vesicles
      table: vesicles
    - pipeline: post-process
      table: pre_synaptic_terminals

2b. Spine reference reconciliation

Implemented source path on 8 October 2026: syndb etl microns-spines prepare-positions reads local v1412 reference and detailed synapse CSVs, groups shared postsynaptic compartments, resolves post_pt_root_id ownership, and publishes hash-bound Parquet plus source.json. The source contract and commands describe explicit units, memory bounds and import validation.

The historical proposal treated target_id as a neuron key and assigned presynaptic ownership. Both are incorrect: it references a synapse, and group_id identifies a shared postsynaptic spine prediction. Positions are now explicitly labeled means of member synaptic-junction locations, not measured spine centroids. Missing morphology remains unknown.

Stage and validate the complete joined output before updating production jobs or rebuilding the heterogeneous benchmark bundle. Set parent_links_real: true only after proving that the resulting postsynaptic neuron UUIDs belong to the bundle’s neuron universe. Full-source and deployment acceptance are still open.

3. Assemble heterogeneous benchmark bundle

  • Write the heterogeneous-source-manifest.yaml that references all 8 tables and their source components from the 7 imported datasets.
  • Run prepare-heterogeneous-live-data:
    cargo run -p cli --features dev,etl,graph-precompute,bench -- \
      bench prepare-heterogeneous-live-data \
      --source-manifest benchmark/heterogeneous-source-manifest.yaml \
      --output-dir benchmark/live-data \
      --scale 100k,1m,10m
    
  • Run validation:
    test -s benchmark/live-data/live-source-manifest.json
    grep '"bundle_mode": "heterogeneous_real"' benchmark/live-data/live-source-manifest.json
    

4. Update manuscript captions

Edit article/sections/supplementary_figures.tex:

  • Line 113 (analytics_engine/CF04): Replace “retained controlled synthetic benchmark artifact” with “heterogeneous real-source benchmark bundle (provenance in Methods §Interoperability round-trip)”
  • Line 131 (engineering/CF11): Same pattern
  • Line 150 (benchmark_runtime_supplement/CF14): Same
  • Line 166 (query_scaling_supplement/CF16): Same
  • Line 181 (benchmark_systems_supplement/CF15): Keep as explicit simulated (federation latency requires multi-cluster RTT)
  • Line 109: Remove (synthetic controls) from section header

5. Retire manuscript-evidence-completion plan-set

  • Update documentation/src/SUMMARY.md: remove lines 74-90 (the manuscript-evidence-completion section)
  • Optionally archive the plan files (leave on disk for git history, just remove from published navigation)
  • Verify docs build:
    mdbook build documentation/
    

6. Final validation

# Full crate check
cargo check -p cli --features dev,etl,graph-precompute,bench
# Existing heterogeneous manifest tests
cargo test -p syndb-bench live_source

Acceptance criteria

  • post-process pipeline compiles and produces pre_synaptic_terminals.parquet matching PRE_SYNAPTIC_TERMINALS_COLUMNS.
  • All 7 pipelines declared in values.yaml with per-pipeline resource specs.
  • live-source-manifest.json has data_source: "real" and bundle_mode: "heterogeneous_real".
  • article/sections/supplementary_figures.tex has zero occurrences of “controlled synthetic” for CF04/CF11/CF14/CF16; CF15 remains explicitly simulated.
  • documentation/src/SUMMARY.md no longer lists the manuscript-evidence-completion plan.
  • mdbook build documentation/ succeeds.
  • cargo test -p syndb-bench live_source passes.

Files likely touched

  • crates/tools/etl/src/post_process/mod.rs, id.rs — new
  • crates/tools/etl/src/lib.rs — register post_process
  • crates/tools/etl/src/registry/runtime.rs — register post_process
  • infrastructure/helm/nautilus/values.yaml — all 7 Helm job declarations
  • benchmark/heterogeneous-source-manifest.yaml — new manifest
  • article/sections/supplementary_figures.tex — caption edits
  • documentation/src/SUMMARY.md — remove plan section

Pitfalls

  • Symptom: post-process needs the synapse Parquet output path. Cause: The pipeline reads from another pipeline’s prepared data. Recovery: Reference it by S3 prefix s3://syndb-etl/microns-synapses-prepared/.
  • Symptom: Helm values grow very long. Cause: 7 new pipelines declared inline. Recovery: Keep the pattern compact; each pipeline is ~4 lines.
  • Symptom: Caption edit inadvertently changes figure numbering. Cause: supplementary_figures.tex uses \suppfigure{label} which maps through figure_inputs.tex. Recovery: Only edit caption prose, not \suppfigure{} arguments or order.

Reference

  • Existing Helm values: infrastructure/helm/nautilus/values.yaml lines 830-1202
  • Supplementary figure labels → CF mapping: article/figures/figure_inputs.tex
  • Caption text to edit: article/sections/supplementary_figures.tex lines 109-196
  • Heterogeneous benchmark code: crates/tools/bench/src/live_source.rs
  • Plan retirement skill: retire-docs-planning (at docs/src/planning/retire-docs-planning/SKILL.md)