Phase 03 — Replace synthetic benchmark figures with live-cluster data
Goal
The benchmark/engineering figures that currently use synthetic data are either (a) regenerated from real Nautilus-cluster measurements with provenance, and their captions stop saying “synthetic”; or (b) where a real measurement is impossible with one production cluster (federation latency, CF15), honestly resolved — kept as an explicitly simulated control or dropped — so that no synthetic panel masquerades as production.
Why this matters now
Five supplement figures are synthetic: CF04 (MV engine), CF11 (engineering: UUID/UInt64 storage, graph-op scaling, memory), CF14 (runtime scale tiers), CF16 (query scaling), CF15 (federation latency, cross-platform). They are honestly labelled, but converting the convertible ones to live data is the single biggest credibility upgrade available, using the same harness that already produced the real CF05.
Sub-layer table
| # | Slug | Model / Effort | Touches | File |
|---|---|---|---|---|
| 01 | cf04-mv-engine | DeepSeek v4 Pro (opencode) / high | fig4_analytics.py, MV benchmark artifacts, cache | sub-01-cf04-mv-engine.md |
| 02 | cf11-engineering | DeepSeek v4 Pro (opencode) / high | fig11_engineering.py, storage/graph-op artifacts | sub-02-cf11-engineering.md |
| 03 | cf14-16-scale-tiers | DeepSeek v4 Pro (opencode) / high | fig* scale-tier generators, query/ingestion artifacts | sub-03-cf14-16-scale-tiers.md |
Each sub-layer regenerates its own figure from real data (disjoint
Python generators + benchmark artifacts). They do not edit
supplementary_figures.tex — caption edits are batched in the merge to
avoid a shared-file conflict (anti-pattern: sub-layers editing the same
file).
Merge plan
- Collect each sub-layer’s outcome: real-data composite regenerated +
provenance recorded under
article/reproducibility/, or an explicit “not convertible — keep labelled / drop” determination. - In one commit edit
article/sections/supplementary_figures.tex:- For each converted figure, drop “synthetic”/“controlled synthetic benchmark” from the caption and point provenance at the new artifact.
- CF15 (federation latency): not convertible on a single production
cluster. Either keep it with the caption hardened to “simulated
multi-region latency (configured RTT)” — already partly worded — or
drop the latency panels and retain only the cross-platform matrix.
Whichever: ensure the main-text reference (
results.tex, owned by Phase 01) and this caption agree. Coordinate landing order with Phase 04 (both touchsupplementary_figures.tex: 03 → 04).
- Build gate (
lualatex ×2); confirm S-numbering of Figs S1–S11 unchanged.
Acceptance criteria (phase-level)
- CF04, CF11, and CF14/CF16 panels are generated from live-cluster artifacts; each has a provenance file under
article/reproducibility/naming the cluster run (date, dataset, Helm revision, benchmark image). - Their captions no longer contain “synthetic” / “controlled synthetic benchmark”.
- CF15 is either captioned as explicitly simulated (RTT-configured) or its latency panels are dropped; no sentence implies it is production.
- Supplementary figure numbering S1–S11 preserved;
lualatex ×2exits 0, 0 undefined refs.
Global constraints
Inherits the plan README’s build gate, production-data norm, and two-class figure rules. Do not alter the main figures (CF01,02,05,03,06) or their numbering.
Reference
- Gap inventory items 1–4 (synthetic→real, federation latency note).
article/NARRATIVE_AUDIT.md(which figures are synthetic)..claude/skills/manuscript/(figure build pipeline: panels → tikz → composites).- Memory: “Graph precompute in ETL”; benchmark provenance pattern used by CF05.