Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Evidence Recapture

Production evidence does not expire with age. A figure or live capture stays valid while the data, serving configuration and source versions it binds are unchanged, and only a changed binding requires recapture. Two commands answer two separate questions: syndb article evidence audit --production checks that the committed evidence is self-consistent (the figure-input manifests, the live_* artifacts, the cited benchmark captures under documentation/manuscript/benchmarks/{api-endpoints,federation,ingestion,live-cluster} and the signed BR-02R adoptions); syndb article evidence revalidate compares that evidence with freshly refreshed inputs and reports unchanged or changed -> recapture per figure and capture. When new datasets land or serving changes, recapture with just ms-recapture (steps 1-6 and the audit); step 7 revalidates the result against the same refreshed inputs.

Heavy production reads (the corpus snapshot and the cache populate) run as in-cluster Jobs through syndb ops k8s job run. Their outputs (about 250 MB of Parquet plus manifests) are fetched into the checkout, and figure inputs, tables, generated values and the audit run locally.

Gates (check before starting)

  • No dataset import is finalizing. A held Flight finalization Lease means one is: kubectl -n syndb get lease syndb-flight-finalizer -o jsonpath='{.spec.holderIdentity}' must print nothing.
  • No ClickHouse replica rollout or CHI change is in progress, and none is planned until the populate finishes.
  • No syndb-api-deployment rollout until the populate finishes. Restarting the API pod mid-finalization fails an import with “staged upload finalization lost its coordination lease”, and a populate reads through the same Flight service.
  • No graph precompute Job runs on a dataset the populate reads. A precompute run deletes, then rewrites, one dataset’s precomputed_* rows, so an overlapping populate caches empty products. Agree a write freeze with the precompute owner from “populate starting” to “populate done”.
  • The populate pod stays at 2Gi (it waits on Flight, so a larger request counts against the NRP low-utilization budget). A failed populate still uploads its cache and populate-cache-coverage.json, whose failures list names every failed fetch; read them from the Job’s S3 output prefix.
  • syndb-etl.image in infrastructure/helm/nautilus/values.yaml pins a digest that you verified in the registry (skopeo inspect). job run uses it.

syndb ops k8s nautilus apply and deploy enforce this as a rollout gate. When helm diff shows a change to syndb-api-deployment (Flight runs inside the API pod) or to a ClickHouseInstallation, the command refuses if the syndb-flight-finalizer Lease has a holder or any app=syndb-etl import Job (no syndb.xyz/stage label) is active, and names the holder with its last renew time and the Jobs with their start times. Announce to the import owners and wait for the import to finish. --allow-active-imports overrides the gate, and it is the only way to roll while an import runs. This is the 2026-11-03 failure: an apply --no-reset image rollout killed a BANC finalize at 10.15M of 32.6M rows.

Steps

For each live-cluster or federation-real capture, run syndb article graph capture-deployment --output <capture-dir>/deployment-before.json immediately before the capture and repeat immediately afterward with --output <capture-dir>/deployment-after.json. Keep both files beside provenance.json; merged federation runs need a bracket covering all runs. The bracket proves claimed ordering and endpoint equality only, not authentic acquisition time, and cannot detect a change and revert in between.

  1. Corpus snapshot. syndb ops k8s job run --name corpus-snapshot -- data snapshot-corpus-inventory appends one snapshot per dataset to corpus_inventory (a few seconds).

  2. CF12 deployment capture. syndb article graph capture-deployment reads the syndb-etl release with helm get and kubectl get and rewrites article/reproducibility/biological-results-br-2026-08/cf12-live-deployment.json. Run it straight after the snapshot so both describe the same deployment.

  3. Populate. An in-cluster article cache populate --figure all Job (2Gi, the NRP-exempt shape: it waits on Flight and averages ~0.01 GiB, so a larger request counts as low utilization) fetches every figure product and the CF08 coverage audit, re-freezes the Songbird review, and returns the cache directory and songbird-source-review.json into the checkout (--fetch-to .). A skipped product means a production data product is missing or empty: fix the product, never the cache. Expected exceptions are FlyWire clique topology (skipped_size_limit, captioned) and MICrONS (gated by design).

  4. Figure inputs. syndb article cache build-figure-inputs --figure all, then syndb article cache validate --figure all. The validator must pass.

  5. Tables and values. From the repository root, run python -m syndb_plot.manuscript.run_biological_analyses --tables-dir article/tables --macros-output article/generated_values.tex. Check that article/tables/tab_spine_statistics.tex exists. The table is skipped silently when spine_microns has no positive mesh_volume, and the manuscript build then fails.

  6. Figures. syndb article layout solve, then syndb article figure all, both inside nix develop .#docs (it provides figurefit; direnv’s .#dev does not). figure all first validates every figure input and refuses invalid ones. The layouts (article/figures/layouts/*_layout.tex) are git-ignored and figure all does not re-solve them, so a stale local layout can drop panels that a figure gained since the last solve.

  7. Revalidate, then audit. Pass the refreshed inputs:

    syndb article evidence revalidate \
      --inventory packages/plot/src/syndb_plot/manuscript/cache/populate-cache-coverage.json \
      --corpus packages/plot/src/syndb_plot/manuscript/cache/corpus_snapshot_totals.parquet \
      --deployment article/reproducibility/biological-results-br-2026-08/cf12-live-deployment.json \
      --federation <capture-federation output file>
    

    --inventory is the populate’s coverage envelope (repeatable), --corpus the CF01 snapshot totals, --deployment the CF12 capture and --federation the recorded hashes of the published federation tables. Recapture every figure reported as changed -> recapture; each reason names the binding that moved. Anything uncomparable counts as changed: a missing input, an unreadable corpus snapshot, or an out-of-band serving reference without matching capture-time and current hashes. Then syndb article evidence audit --production must pass.

  8. Zenodo upload (draft 20692792, never published here). Rebuild the manuscript PDF, run just ms-preflight, then:

    export ZENODO_TOKEN="$(cat /run/user/1000/agenix/can_zenodo_token)"  # canix home secret can_zenodo_token
    just ms-bundle                 # USTAR tar of the working tree + offline validation
    just ms-zenodo-upload          # refreshes metadata, uploads, checks Zenodo's MD5
    

    ms-bundle fails unless the production audit passes on the bundle, so a bundle that fails the audit cannot be uploaded. Then stop: the user reviews https://zenodo.org/deposit/20692792 and confirms separately. Only after that confirmation, run just ms-zenodo-publish 20692792. Publishing is irreversible and ms-recapture never does it. The draft metadata lives in crates/services/api/src/archive/zenodo/manuscript_metadata.json.

Commit the figure-input manifests, populate-cache-coverage.json, songbird-source-review.json, cf12-live-deployment.json, the tables, generated_values.tex and the figures. The cache Parquet files are git-ignored.

What is outside this recipe

Benchmark captures (live cluster, federation, ingestion, API endpoints), the F-UJI assessment and the GPU benchmark (gpu-etl.md) have their own capture commands. Their stamps record when each was taken, not an expiry; recapture one when the deployment or data it measured moved. syndb article evidence revalidate also covers the GPU-bench and F-UJI captures and names them when their committed bytes or pins changed.