Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Phase 03 · Sub-layer 03 — Regenerate CF14/CF16 (scale-tier benchmarks) at real ingested scales

Recommended model: DeepSeek v4 Pro (opencode) — effort high

Routed: carter route -c complex -r subagent -n coding,scientific → deepseek-v4-pro / high (bar 83, capability 83.4)

Re-running the query/ingestion scale benchmarks against real ingested data and honestly bounding the achievable scale range; complex because the headline risk is over-claiming a scale that isn’t really ingested.

Working tree

Repository root. Cluster access required. Does not edit supplementary_figures.tex.

Goal

CF14 (runtime scale tiers) and CF16 (query scaling) report per-query p95 latency and ingestion throughput measured against real ingested dataset scales on the cluster, with the achievable scale range stated honestly — not the synthetic 100K/1M/10M tiers unless those scales are genuinely present.

Why this matters now

CF14/CF16 captions: “All panels use the retained controlled synthetic benchmark bundle generated at 100K, 1M, and 10M rows.” The live query path is already benchmarked (CF05) at production scale; extending that to a scale series on real data converts these — bounded by whatever scales are actually ingested.

Out of scope

  • CF04, CF11 (siblings); CF15 federation (merge).
  • Fabricating scale tiers not present in the cluster — if only one or two real scales exist, report those and say so (no silent extrapolation).

Plan

  1. Enumerate real per-table row counts across ingested datasets (ClickHouse system.parts) to find the achievable scale points (e.g. small connectome vs FlyWire/MICrONS) — these become the real “tiers”.
  2. Run the manuscript query benchmark (CF05 controller pattern) at each real scale point for the manuscript query families; capture p95/p99 and ingestion throughput where measurable.
  3. Write real artifacts + provenance.
  4. Regenerate CF14 and CF16 from the real artifacts. If fewer than three real scale points exist, reduce the figure to the real points and log/caption the limitation (the merge will word it).
  5. Tell the merge the real scale range so the caption states it precisely.

Acceptance criteria

  • CF14/CF16 panels derive from real cluster runs at real ingested scales; provenance file names the scale points (dataset → row count) and run metadata.
  • No panel claims a scale tier that is not actually ingested; any reduction in tiers is stated in the caption (no silent truncation).
  • p95/p99 latencies are directionally consistent with CF05 at the overlapping scale.
  • Figures regenerated; build clean.

Files likely touched

  • packages/plot/src/syndb_plot/manuscript/composite/fig10_developmental.py? No — the runtime/query-scaling generators (fig5_scalability.py siblings / the supplement generators feeding CF14/CF16).
  • benchmark/results/queries.parquet, ingestion.parquet → real regenerations.
  • article/reproducibility/ — provenance; article/figures/composite/CF14-*, CF16-*.

Pitfalls

  • Symptom: only one real scale point exists. Cause: the cluster has production data at one scale, not a synthetic sweep. Recovery: this is the honest outcome — present the real point(s) and keep the synthetic sweep only if explicitly captioned as a control; do not pretend a sweep.
  • Symptom: ingestion throughput unmeasurable. Cause: no fresh ingestion to time. Recovery: measure on a controlled re-ingest of one dataset, or drop the ingestion panel and say so.
  • Symptom: caption/figure scale mismatch. Cause: merge wrote a tier the data doesn’t have. Recovery: the merge must take the real scale range from this sub-layer’s note verbatim.

Reference

  • CF05 live benchmark: documentation/manuscript/benchmarks/live-cluster-2026-06-06/.
  • Gap inventory item 3.