Phase 05 · Sub-layer 02 — Ingest an independent morphometry dataset
Recommended model: DeepSeek v4 Pro (opencode) — effort
highRouted:
carter route -c complex -r subagent -n coding→deepseek-v4-pro/high(bar 83, capability 83.5)A full ETL stream — source → schema map → ingest → validate — for a new morphometry dataset; complex coding with a correctness bar on the measurement mapping.
Working tree
Repository root. Cluster write access + ETL run. Does
not edit bio/*.tex/discussion.tex.
Goal
At least one independent, sizeable morphometry dataset (with neuron cable-length / volume measurements) is ingested into SynDB, so the cross-species scaling exponent α no longer rests almost entirely on NeuroMorpho.
Why this matters now
Limitations: morphometric α “rest[s] almost entirely on NeuroMorpho.Org (n=284,373) … Allen Cell Types (n=50) and FlyWire (n=16), neither … large enough to independently constrain the fitted exponent α.” A second large independent source turns α from “NeuroMorpho-dominated” into a cross-dataset estimate.
Out of scope
- The other coverage streams.
- Re-fitting α by hand — the Phase 05 merge re-runs the analyses harness.
- Building a new ETL framework — extend the existing morphometry ETL.
Plan
- Choose a public morphometry source with real cable-length/volume per
neuron and a usable licence (candidates: a large NeuroMorpho-independent
reconstruction archive, or an additional Allen modality). Confirm it
measures features comparable to the existing
neurons/morphometry schema. - Add an ETL importer following the existing pattern in
crates/tools/etl/(Rust — no Python ETL), mapping its measurements to the SynDB schema; register it in the shared dataset registry with its imaging modality and DOI/licence. - Run the ingest into the cluster; validate row counts and that
load_morphometric_scaling_data()now sees the new dataset. - Add the dataset’s citation to
references.bib. - Hand the merge: the dataset is live so the analyses re-run includes it.
Acceptance criteria
- A new morphometry dataset with real per-neuron measurements is ingested and queryable;
load_morphometric_scaling_data()returns it. - Its ETL importer is Rust (in
crates/tools/etl/), registered with modality + DOI/licence in the dataset registry. - Row count is large enough to materially contribute to α (state n in provenance).
- Citation added to
references.bib.
Files likely touched
crates/tools/etl/— new importer;crates/core/shared/src/etl_registry/— registration.article/references.bib— dataset citation.article/reproducibility/— ingest provenance.
Pitfalls
- Symptom: units mismatch (µm vs nm) skews α. Cause: source-vs-schema unit difference. Recovery: normalise at ingest; validate against a known neuron’s cable length.
- Symptom: species labels don’t dedupe. Cause: upstream naming variance. Recovery: the analysis applies a canonical-name mapping at analysis time — ensure the new species map in.
- Anti-pattern: Python ETL. All ETL is Rust by project rule.
Reference
- Memory: “No Python in ETL”.
packages/plot/src/syndb_plot/manuscript/data.py(load_morphometric_scaling_data).- Gap inventory item 9.