Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Phase 03 — Replace synthetic benchmark figures with live-cluster data

Goal

The benchmark/engineering figures that currently use synthetic data are either (a) regenerated from real Nautilus-cluster measurements with provenance, and their captions stop saying “synthetic”; or (b) where a real measurement is impossible with one production cluster (federation latency, CF15), honestly resolved — kept as an explicitly simulated control or dropped — so that no synthetic panel masquerades as production.

Why this matters now

Five supplement figures are synthetic: CF04 (MV engine), CF11 (engineering: UUID/UInt64 storage, graph-op scaling, memory), CF14 (runtime scale tiers), CF16 (query scaling), CF15 (federation latency, cross-platform). They are honestly labelled, but converting the convertible ones to live data is the single biggest credibility upgrade available, using the same harness that already produced the real CF05.

Sub-layer table

#SlugModel / EffortTouchesFile
01cf04-mv-engineDeepSeek v4 Pro (opencode) / highfig4_analytics.py, MV benchmark artifacts, cachesub-01-cf04-mv-engine.md
02cf11-engineeringDeepSeek v4 Pro (opencode) / highfig11_engineering.py, storage/graph-op artifactssub-02-cf11-engineering.md
03cf14-16-scale-tiersDeepSeek v4 Pro (opencode) / highfig* scale-tier generators, query/ingestion artifactssub-03-cf14-16-scale-tiers.md

Each sub-layer regenerates its own figure from real data (disjoint Python generators + benchmark artifacts). They do not edit supplementary_figures.tex — caption edits are batched in the merge to avoid a shared-file conflict (anti-pattern: sub-layers editing the same file).

Merge plan

  1. Collect each sub-layer’s outcome: real-data composite regenerated + provenance recorded under article/reproducibility/, or an explicit “not convertible — keep labelled / drop” determination.
  2. In one commit edit article/sections/supplementary_figures.tex:
    • For each converted figure, drop “synthetic”/“controlled synthetic benchmark” from the caption and point provenance at the new artifact.
    • CF15 (federation latency): not convertible on a single production cluster. Either keep it with the caption hardened to “simulated multi-region latency (configured RTT)” — already partly worded — or drop the latency panels and retain only the cross-platform matrix. Whichever: ensure the main-text reference (results.tex, owned by Phase 01) and this caption agree. Coordinate landing order with Phase 04 (both touch supplementary_figures.tex: 03 → 04).
  3. Build gate (lualatex ×2); confirm S-numbering of Figs S1–S11 unchanged.

Acceptance criteria (phase-level)

  • CF04, CF11, and CF14/CF16 panels are generated from live-cluster artifacts; each has a provenance file under article/reproducibility/ naming the cluster run (date, dataset, Helm revision, benchmark image).
  • Their captions no longer contain “synthetic” / “controlled synthetic benchmark”.
  • CF15 is either captioned as explicitly simulated (RTT-configured) or its latency panels are dropped; no sentence implies it is production.
  • Supplementary figure numbering S1–S11 preserved; lualatex ×2 exits 0, 0 undefined refs.

Global constraints

Inherits the plan README’s build gate, production-data norm, and two-class figure rules. Do not alter the main figures (CF01,02,05,03,06) or their numbering.

Reference

  • Gap inventory items 1–4 (synthetic→real, federation latency note).
  • article/NARRATIVE_AUDIT.md (which figures are synthetic).
  • .claude/skills/manuscript/ (figure build pipeline: panels → tikz → composites).
  • Memory: “Graph precompute in ETL”; benchmark provenance pattern used by CF05.