Evidence Recapture
Production evidence does not expire with age. A figure or live capture stays
valid while the data, serving configuration and source versions it binds are
unchanged, and only a changed binding requires recapture. Two commands answer
two separate questions: syndb article evidence audit --production checks that
the committed evidence is self-consistent (the figure-input manifests, the
live_* artifacts, the cited benchmark captures under
documentation/manuscript/benchmarks/{api-endpoints,federation,ingestion,live-cluster}
and the signed BR-02R adoptions); syndb article evidence revalidate compares
that evidence with freshly refreshed inputs and reports unchanged or
changed -> recapture per figure and capture. When new datasets land or serving
changes, recapture with just ms-recapture (steps 1-6 and the audit); step 7
revalidates the result against the same refreshed inputs.
Heavy production reads (the corpus snapshot and the cache populate) run as
in-cluster Jobs through syndb ops k8s job run. Their outputs (about 250 MB of
Parquet plus manifests) are fetched into the checkout, and figure inputs,
tables, generated values and the audit run locally.
Gates (check before starting)
- No dataset import is finalizing. A held Flight finalization Lease means one is:
kubectl -n syndb get lease syndb-flight-finalizer -o jsonpath='{.spec.holderIdentity}'must print nothing. - No ClickHouse replica rollout or CHI change is in progress, and none is planned until the populate finishes.
- No
syndb-api-deploymentrollout until the populate finishes. Restarting the API pod mid-finalization fails an import with “staged upload finalization lost its coordination lease”, and a populate reads through the same Flight service. - No graph precompute Job runs on a dataset the populate reads. A precompute
run deletes, then rewrites, one dataset’s
precomputed_*rows, so an overlapping populate caches empty products. Agree a write freeze with the precompute owner from “populate starting” to “populate done”. - The populate pod stays at 2Gi (it waits on Flight, so a larger request counts
against the NRP low-utilization budget). A failed populate still uploads its
cache and
populate-cache-coverage.json, whosefailureslist names every failed fetch; read them from the Job’s S3 output prefix. syndb-etl.imageininfrastructure/helm/nautilus/values.yamlpins a digest that you verified in the registry (skopeo inspect).job runuses it.
syndb ops k8s nautilus apply and deploy enforce this as a rollout gate. When
helm diff shows a change to syndb-api-deployment (Flight runs inside the API
pod) or to a ClickHouseInstallation, the command refuses if the
syndb-flight-finalizer Lease has a holder or any app=syndb-etl import Job
(no syndb.xyz/stage label) is active, and names the holder with its last renew
time and the Jobs with their start times. Announce to the import owners and wait
for the import to finish. --allow-active-imports overrides the gate, and it is
the only way to roll while an import runs. This is the 2026-11-03 failure: an
apply --no-reset image rollout killed a BANC finalize at 10.15M of 32.6M rows.
Steps
For each live-cluster or federation-real capture, run
syndb article graph capture-deployment --output <capture-dir>/deployment-before.json
immediately before the capture and repeat immediately afterward with
--output <capture-dir>/deployment-after.json. Keep both files beside
provenance.json; merged federation runs need a bracket covering all runs.
The bracket proves claimed ordering and endpoint equality only, not authentic
acquisition time, and cannot detect a change and revert in between.
-
Corpus snapshot.
syndb ops k8s job run --name corpus-snapshot -- data snapshot-corpus-inventoryappends one snapshot per dataset tocorpus_inventory(a few seconds). -
CF12 deployment capture.
syndb article graph capture-deploymentreads thesyndb-etlrelease withhelm getandkubectl getand rewritesarticle/reproducibility/biological-results-br-2026-08/cf12-live-deployment.json. Run it straight after the snapshot so both describe the same deployment. -
Populate. An in-cluster
article cache populate --figure allJob (2Gi, the NRP-exempt shape: it waits on Flight and averages ~0.01 GiB, so a larger request counts as low utilization) fetches every figure product and the CF08 coverage audit, re-freezes the Songbird review, and returns the cache directory andsongbird-source-review.jsoninto the checkout (--fetch-to .). A skipped product means a production data product is missing or empty: fix the product, never the cache. Expected exceptions are FlyWire clique topology (skipped_size_limit, captioned) and MICrONS (gated by design). -
Figure inputs.
syndb article cache build-figure-inputs --figure all, thensyndb article cache validate --figure all. The validator must pass. -
Tables and values. From the repository root, run
python -m syndb_plot.manuscript.run_biological_analyses --tables-dir article/tables --macros-output article/generated_values.tex. Check thatarticle/tables/tab_spine_statistics.texexists. The table is skipped silently whenspine_micronshas no positivemesh_volume, and the manuscript build then fails. -
Figures.
syndb article layout solve, thensyndb article figure all, both insidenix develop .#docs(it providesfigurefit; direnv’s.#devdoes not).figure allfirst validates every figure input and refuses invalid ones. The layouts (article/figures/layouts/*_layout.tex) are git-ignored andfigure alldoes not re-solve them, so a stale local layout can drop panels that a figure gained since the last solve. -
Revalidate, then audit. Pass the refreshed inputs:
syndb article evidence revalidate \ --inventory packages/plot/src/syndb_plot/manuscript/cache/populate-cache-coverage.json \ --corpus packages/plot/src/syndb_plot/manuscript/cache/corpus_snapshot_totals.parquet \ --deployment article/reproducibility/biological-results-br-2026-08/cf12-live-deployment.json \ --federation <capture-federation output file>--inventoryis the populate’s coverage envelope (repeatable),--corpusthe CF01 snapshot totals,--deploymentthe CF12 capture and--federationthe recorded hashes of the published federation tables. Recapture every figure reported aschanged -> recapture; each reason names the binding that moved. Anything uncomparable counts as changed: a missing input, an unreadable corpus snapshot, or an out-of-band serving reference without matching capture-time and current hashes. Thensyndb article evidence audit --productionmust pass. -
Zenodo upload (draft 20692792, never published here). Rebuild the manuscript PDF, run
just ms-preflight, then:export ZENODO_TOKEN="$(cat /run/user/1000/agenix/can_zenodo_token)" # canix home secret can_zenodo_token just ms-bundle # USTAR tar of the working tree + offline validation just ms-zenodo-upload # refreshes metadata, uploads, checks Zenodo's MD5ms-bundlefails unless the production audit passes on the bundle, so a bundle that fails the audit cannot be uploaded. Then stop: the user reviews https://zenodo.org/deposit/20692792 and confirms separately. Only after that confirmation, runjust ms-zenodo-publish 20692792. Publishing is irreversible andms-recapturenever does it. The draft metadata lives incrates/services/api/src/archive/zenodo/manuscript_metadata.json.
Commit the figure-input manifests, populate-cache-coverage.json,
songbird-source-review.json, cf12-live-deployment.json, the tables,
generated_values.tex and the figures. The cache Parquet files are git-ignored.
What is outside this recipe
Benchmark captures (live cluster, federation, ingestion, API endpoints), the
F-UJI assessment and the GPU benchmark (gpu-etl.md) have their own capture
commands. Their stamps record when each was taken, not an expiry; recapture one
when the deployment or data it measured moved. syndb article evidence revalidate also covers the GPU-bench and F-UJI captures and names them when
their committed bytes or pins changed.