Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Phase 05 · Sub-layer 02 — Ingest an independent morphometry dataset

Recommended model: DeepSeek v4 Pro (opencode) — effort high

Routed: carter route -c complex -r subagent -n coding → deepseek-v4-pro / high (bar 83, capability 83.5)

A full ETL stream — source → schema map → ingest → validate — for a new morphometry dataset; complex coding with a correctness bar on the measurement mapping.

Working tree

Repository root. Cluster write access + ETL run. Does not edit bio/*.tex/discussion.tex.

Goal

At least one independent, sizeable morphometry dataset (with neuron cable-length / volume measurements) is ingested into SynDB, so the cross-species scaling exponent α no longer rests almost entirely on NeuroMorpho.

Why this matters now

Limitations: morphometric α “rest[s] almost entirely on NeuroMorpho.Org (n=284,373) … Allen Cell Types (n=50) and FlyWire (n=16), neither … large enough to independently constrain the fitted exponent α.” A second large independent source turns α from “NeuroMorpho-dominated” into a cross-dataset estimate.

Out of scope

  • The other coverage streams.
  • Re-fitting α by hand — the Phase 05 merge re-runs the analyses harness.
  • Building a new ETL framework — extend the existing morphometry ETL.

Plan

  1. Choose a public morphometry source with real cable-length/volume per neuron and a usable licence (candidates: a large NeuroMorpho-independent reconstruction archive, or an additional Allen modality). Confirm it measures features comparable to the existing neurons/morphometry schema.
  2. Add an ETL importer following the existing pattern in crates/tools/etl/ (Rust — no Python ETL), mapping its measurements to the SynDB schema; register it in the shared dataset registry with its imaging modality and DOI/licence.
  3. Run the ingest into the cluster; validate row counts and that load_morphometric_scaling_data() now sees the new dataset.
  4. Add the dataset’s citation to references.bib.
  5. Hand the merge: the dataset is live so the analyses re-run includes it.

Acceptance criteria

  • A new morphometry dataset with real per-neuron measurements is ingested and queryable; load_morphometric_scaling_data() returns it.
  • Its ETL importer is Rust (in crates/tools/etl/), registered with modality + DOI/licence in the dataset registry.
  • Row count is large enough to materially contribute to α (state n in provenance).
  • Citation added to references.bib.

Files likely touched

  • crates/tools/etl/ — new importer; crates/core/shared/src/etl_registry/ — registration.
  • article/references.bib — dataset citation.
  • article/reproducibility/ — ingest provenance.

Pitfalls

  • Symptom: units mismatch (µm vs nm) skews α. Cause: source-vs-schema unit difference. Recovery: normalise at ingest; validate against a known neuron’s cable length.
  • Symptom: species labels don’t dedupe. Cause: upstream naming variance. Recovery: the analysis applies a canonical-name mapping at analysis time — ensure the new species map in.
  • Anti-pattern: Python ETL. All ETL is Rust by project rule.

Reference

  • Memory: “No Python in ETL”.
  • packages/plot/src/syndb_plot/manuscript/data.py (load_morphometric_scaling_data).
  • Gap inventory item 9.