Phase 03 · Sub-layer 03 — Regenerate CF14/CF16 (scale-tier benchmarks) at real ingested scales
Recommended model: DeepSeek v4 Pro (opencode) — effort
highRouted:
carter route -c complex -r subagent -n coding,scientific→deepseek-v4-pro/high(bar 83, capability 83.4)Re-running the query/ingestion scale benchmarks against real ingested data and honestly bounding the achievable scale range; complex because the headline risk is over-claiming a scale that isn’t really ingested.
Working tree
Repository root. Cluster access required. Does not edit
supplementary_figures.tex.
Goal
CF14 (runtime scale tiers) and CF16 (query scaling) report per-query p95 latency and ingestion throughput measured against real ingested dataset scales on the cluster, with the achievable scale range stated honestly — not the synthetic 100K/1M/10M tiers unless those scales are genuinely present.
Why this matters now
CF14/CF16 captions: “All panels use the retained controlled synthetic benchmark bundle generated at 100K, 1M, and 10M rows.” The live query path is already benchmarked (CF05) at production scale; extending that to a scale series on real data converts these — bounded by whatever scales are actually ingested.
Out of scope
- CF04, CF11 (siblings); CF15 federation (merge).
- Fabricating scale tiers not present in the cluster — if only one or two real scales exist, report those and say so (no silent extrapolation).
Plan
- Enumerate real per-table row counts across ingested datasets (ClickHouse
system.parts) to find the achievable scale points (e.g. small connectome vs FlyWire/MICrONS) — these become the real “tiers”. - Run the manuscript query benchmark (CF05 controller pattern) at each real scale point for the manuscript query families; capture p95/p99 and ingestion throughput where measurable.
- Write real artifacts + provenance.
- Regenerate CF14 and CF16 from the real artifacts. If fewer than three
real scale points exist, reduce the figure to the real points and
log/caption the limitation (the merge will word it). - Tell the merge the real scale range so the caption states it precisely.
Acceptance criteria
- CF14/CF16 panels derive from real cluster runs at real ingested scales; provenance file names the scale points (dataset → row count) and run metadata.
- No panel claims a scale tier that is not actually ingested; any reduction in tiers is stated in the caption (no silent truncation).
- p95/p99 latencies are directionally consistent with CF05 at the overlapping scale.
- Figures regenerated; build clean.
Files likely touched
packages/plot/src/syndb_plot/manuscript/composite/fig10_developmental.py? No — the runtime/query-scaling generators (fig5_scalability.pysiblings / the supplement generators feeding CF14/CF16).benchmark/results/queries.parquet,ingestion.parquet→ real regenerations.article/reproducibility/— provenance;article/figures/composite/CF14-*,CF16-*.
Pitfalls
- Symptom: only one real scale point exists. Cause: the cluster has production data at one scale, not a synthetic sweep. Recovery: this is the honest outcome — present the real point(s) and keep the synthetic sweep only if explicitly captioned as a control; do not pretend a sweep.
- Symptom: ingestion throughput unmeasurable. Cause: no fresh ingestion to time. Recovery: measure on a controlled re-ingest of one dataset, or drop the ingestion panel and say so.
- Symptom: caption/figure scale mismatch. Cause: merge wrote a tier the data doesn’t have. Recovery: the merge must take the real scale range from this sub-layer’s note verbatim.
Reference
- CF05 live benchmark:
documentation/manuscript/benchmarks/live-cluster-2026-06-06/. - Gap inventory item 3.