Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Plan amendment: put the unmounted PVCs to work (replaces slice 6 / O6 as the default)

Design pass, 2026-11-15. Read-only: nothing was run against the cluster. Paths are relative to projects/worktrees/SynDB/nrp-policy (branch fix/nrp-policy). “plan” means plan.md beside this file; “brief” means brief-pvc-utilisation.md.

1. Result

Three claims have a real use. Fourteen do not, and I recommend releasing them (operator questions, §9).

VerdictClaimsSize
In use alreadysyndb-image-build-store300 Gi
Put to use (this plan)BANC mesh cache 250 Gi, MICrONS import scratch 80 Gi, syndb-ctrlbench-source 40 Gi (one run, then release)370 Gi
No credible use; release after inspection8 retired ClickHouse volumes, 3 × 40 Gi MICrONS scratch, 3 Keeper clones950 Gi

Why so few. Three facts from the repo limit what an old claim can become:

  1. A claim can only be used under its own name. PVC names are immutable, and re-binding a volume to a new claim needs write access to PersistentVolumes, which both contexts lack (brief, Unknown). So only workloads that name a claim explicitly (Jobs, CronJobs) can mount one. StatefulSet and CHI claim templates cannot adopt it.
  2. Most of these volumes are capacity, not data. ETL scratch is wiped at the start of every run: the fetch init container runs rm -rf "$DEST_DIR" "$STAGE_DIR" (infrastructure/helm/syndb-etl/templates/_helpers.tpl:368). An empty block volume is recreated in seconds, so keeping one idle buys nothing.
  3. The volumes in use have room. Production ClickHouse is 14% full on 300 Gi (brief, Context), so nothing needs a cold tier, a cache disk or a spool volume.

The rule this plan applies (and writes into the runbook, U1):

A claim stays only if a named workload mounts it on a stated date or cadence, and either the bytes on it are reused across runs or the run needs more than 50 Gi of scratch (NRP evicts above 50 Gi of emptyDir, facts → Local scratch). Every kept claim carries a dated keep marker. When the date passes without a mount, the audit lists it again and it is released.

2. One proposed use per claim

ClaimProposed useWorkload and cadenceExpected fill / fitPlacementData on it today
etl-scratch-prepare-mesh-connectomics-banc-seg-v888 (250 Gi)Decoded-mesh cache for the BANC mesh prepareJob syndb-etl-prepare-mesh-connectomics-banc-seg-v888 (syndb-etl/values.yaml:874-911) mounts it by its rendered name, no chart change. Runs once per BANC mesh re-prepare; next in window v2Sized for the full inventory (values.yaml:884-890); real fill from C-U1linstor-csus, RWO, CSUS nodes only. Must not be shared with another mesh source: the cache key is the bare segment id (crates/tools/etl/src/mesh_connectomics/download.rs:297-303, :339-341)Expected: /data/.mesh-connectomics-cache/<segment>.bin plus the last output directory. The cache survives the fetch wipe and makes a rerun resume (download.rs:335-338). Confirm with C-U1
etl-scratch-microns-synapses-import (80 Gi)Scratch above 50 Gi for the MICrONS synapse reimport: the import Job and its dry run share itsyndb-etl-microns-synapses-import mounts it by name (nautilus/values.yaml:1632-1643); the dry-run variant names it through scratch.claimName (U2). Once per reimport; next in window v2Import needs more than 50 Gi (values.yaml:1639); 80 Gi was sized for itlinstor-csus, RWO, CSUS nodes. Dry run and import must not overlap (U2 guard)Stale copy of s3://syndb-etl/microns-synapses-prepared/; the next run deletes it. Nothing unique expected (C-U1)
etl-scratch-microns-synapses-{dryrun,v1412,v1412dry} (3 × 40 Gi)None.Their Jobs were one-off suffixes for the 2026-11-04 v1412 import. With U2 a dry run shares the 80 Gi claim. Every other job that fits in 40 Gi also fits NRP’s emptyDir allowance—linstor-csusStale copies of the same S3 prefix (C-U1). Recommend release (Q1)
syndb-ctrlbench-source (40 Gi)Source exports for the planned “ctrlbench 10m” runThe ctrlbench prepare Job mounts it at /source (article/reproducibility/controlled-benchmark-suite.md:89-92, :137-143). One run in window v2. Reuse avoids a second read of production ClickHouse, if no in-scope table was re-released since the export0.86 GiB of Parquet plus release-receipt.json (:89, :146-149); fits 40 Gi 40 times overrook-ceph-block, RWOThe 2026-11-09 exports and receipt. The reproducibility doc already ends with “before deleting the source PVC” (:154-156). Release after the run unless ctrlbench is recurring (Q4)
keeper-data-rbdbak-syndb-cluster-keeper-{0,1,2} (3 × 10 Gi)None.They exist only for rollback-member.sh. The runbook kept them 7 days from 2026-11-04 (documentation/runbooks/keeper-storage-linstor-migration.md:98-101, :137-138)under 10 Gi eachrook-ceph-blockKeeper state as of 2026-11-04, now 11 days behind the quorum. Recommend release once the operator accepts Keeper on linstor as final (Q3)
clickhouse-{pacific,us-central,us-east,us-mountain}-data-chi-syndb-cluster-syndb-<region>-{0,1}-0 (8 × 100 Gi)None as claims. The only possible value is data on them that production lacksSee §3unknown; C-U1rook-ceph-block, RWO. Named and (probably) labelled as CHI syndb-cluster shard dataReplica data of the removed shards, kept on purpose by pvcReclaimPolicy: Retain (syndb-clickhouse/values.yaml:182-185, templates/chi.yaml:223-225). Unknown whether production holds all of it. Inspect, recover what is unique, then release (Q1, Q5)
syndb-image-build-store (300 Gi)Warm Nix store (already in use)syndb ops k8s image-build, every image build—rook-ceph-block, ReadWriteOncePodKeep; marker without a date (OU2)

3. The eight retired ClickHouse volumes: angles weighed

AngleVerdictReason
Filesystem cache or S3-backed cold tier for productionnoProduction is 14% full; clickhouse.tieredStorage.enabled: false (nautilus/values.yaml:444-448). It would be a CHI change, a boot test and a replica roll for capacity nobody needs. The CHI cannot mount these names anyway (§1.1; allowStorageMigration guard, syndb-clickhouse/values.yaml:174-181)
Staging for Flight imports or job-result spooling on the APInoThe API scales horizontally (templates/api-hpa.yaml); an RWO volume attaches to one node. Results already stream to S3 (trunk f2b64a86, 8e5632e1)
Re-add regional shards or replicasnoEach replica is another pod under 20% CPU. The allowance is 4 and 2 are used (plan §1.3)
Second warm Nix store (--store-claim)noimage-build --fresh-store already lets a second build run in parallel (crates/tools/cli/src/k8s/image_build.rs:108-109). Wiping retired replica data to host a Nix store under a CHI shard name is a delete plus a trap: the Altinity operator binds claims by name if that shard ever returns
Isolated restore-drill target (one volume)the only candidate; not nowReal benefit: today the weekly drill restores into the production replica (templates/clickhouse-restore-drill-cronjob.yaml:57, ops/backup/restore_drill.rs:83-89), and us-west-1 already sits at 9.0 Gi of a 9830 Mi limit (brief.md §6). Cost: a standalone ClickHouse with Keeper access for replicated tables, a drill that no longer reads the source database on the same server, new RBAC to start and stop it, and a boot test. The volume would still be empty six days in seven. Needs its own design pass after the pilot; uses one volume of eight
Archive the contents to S3 “in case”noNRP storage is not an archive (facts → Retention). Recover only rows production lacks

What decides the eight is their content (C-U1, then §6 table). Structure helps: each shard has two replicas, so at most four volumes hold distinct data.

4. Slice order

slice 0 (land fix/nrp-policy)
  └─ U1  keep marker; etl cleanup respects it; runbook      ── must land before any reimport cleanup
U2  scratch.claimName + collision guard                     ── independent; needed for OU4 only
plan slice 5 ── reuses U1's predicate (amendment in §8)
plan slice 6 ── stays the release tool for claims the operator releases

Qualification is the plan’s (§2 there): treefmt, the named tests in a capped unit, helm unittest infrastructure/helm/syndb-etl for U2.


U1 — dated keep marker; etl cleanup stops deleting kept claims

Goal. A claim with a planned next use survives syndb ops k8s etl cleanup, and a kept claim cannot stay kept silently.

Problem. Slice 0’s cleanup deletes every etl-scratch-* claim that nothing mounts, after one prompt (crates/tools/cli/src/k8s/etl_jobs/jobs.rs, cleanup_orphan_scratch_claims and orphan_scratch_claims). The first cleanup after any import would delete the 250 Gi BANC cache.

Design. One annotation, syndb.io/keep-unmounted (the key plan slices 5–6 already use). Its value is <YYYY-MM-DD> <reason> (keep through that day) or free text (no expiry; only for the image store).

ItemChange
etl_jobs/jobs.rspub(crate) const KEEP_UNMOUNTED_ANNOTATION: &str = "syndb.io/keep-unmounted";
samepub(crate) fn keeps_unmounted(value: Option<&str>, today: &str) -> bool: None → false; value starting with ten characters shaped dddd-dd-dd → date >= today (ISO dates compare as strings); anything else → true
orphan_scratch_claimsnew parameter today: &str; skip a claim when keeps_unmounted(metadata.annotations[KEY], today)
cleanup_orphan_scratch_claimspass today’s UTC date as YYYY-MM-DD (use the date crate the cli crate already depends on); print kept: <name> (<annotation value>) for each skipped claim before the prompt
documentation/runbooks/pvc-unmounted.mdreplace the TODO stub: the rule from §1, the marker format, a register table (claim, workload, keep-until) filled from §2, the inspection Job from §6, and the release command (plan slice 6)
documentation/runbooks/nrp-admission.mdin “Other NRP rules”, storage bullet: add “A claim kept for a planned run carries syndb.io/keep-unmounted; see pvc-unmounted.md”

No chart change: Helm leaves annotations it never rendered alone, so the operator’s annotation (OU2) survives upgrades.

Tests (etl_jobs/tests.rs):

  • keep_marker_with_a_future_date_keeps_and_a_past_date_expires
  • keep_marker_without_a_date_never_expires
  • orphan_scratch_claims_skip_claims_with_a_live_keep_marker
  • existing orphan_scratch_claims_skip_claims_mounted_by_jobs_or_pods passes with the new parameter

Done when. The four tests pass; python3 scripts/check_runbook_anchors.py passes; rg -n TODO documentation/runbooks/pvc-unmounted.md is empty.

RejectedWhy
Separate keep-until annotationtwo keys to keep in step; one value with a leading date is enough
Chart value scratch.keepReason rendering the annotationHelm does not strip unmanaged annotations; the claims to keep already exist
Permanent keep for scratch claimsthat is the idle hoarding the rule forbids; the date forces a new decision

U2 — scratch.claimName: an ETL Job mounts an existing claim instead of minting one

Goal. A dry run or a re-run uses the claim its pipeline already has. No new claim per job suffix (the reason four MICrONS claims exist).

Chart (infrastructure/helm/syndb-etl).

ItemChange
templates/_helpers.tpl, syndb-etl.scratchVolume (:174-198)in pvc mode the claim is get $jobScratch "claimName" | default (printf "etl-scratch-%s" $claimSuffix); fail when claimName is set and the mode is not pvc
templates/etl-scratch-pvc.yamlboth loops (:8-38, :39-67): render no PVC for a job whose scratch.claimName is set. Collect the effective claim name of every selected job in one dict and fail on a repeat: jobs <a> and <b> both mount scratch claim <c>; roll them out one at a time
values.schema.json, scratchConfig (:99-120)add claimName: string, minLength: 1
values.yaml comment block (:1242-1247)document claimName and the one-at-a-time rule

No committed job gets a claimName in this slice. A dry-run entry is a temporary values edit (shape: tests/etl_job_stage_label_test.yaml:15-19) and names the claim there.

CLI guard (the brief’s constraint: reuse must not collide with a running Job). This matters: a dry run and its import share /data/<pipeline>, and the fetch wipes that directory on start.

ItemChange
crates/tools/cli/src/k8s/etl_jobs/rollout.rspub(super) fn selected_rollout_claims(selected: &[String]) -> Result<BTreeSet<String>>, beside selected_rollout_jobs (:52): the scratch.claimName of every selected entry, walking the same entries as add_job_names_from_section (:326)
etl_jobs/jobs.rspub(crate) fn busy_claims(wanted: &BTreeSet<String>, jobs: &serde_json::Value, pods: &serde_json::Value) -> Vec<String>: claims in wanted that claimed_names (slice 0) finds on Jobs without a Complete or Failed condition, or on Pods in phase Pending or Running
selected-rollout apply (k8s/nautilus/mod.rs, the call that adds syndb-etl.jobsEnabled=true, :360-370)after the selected Jobs are deleted and before Helm runs: bail!("scratch claim {c} is mounted by an unfinished Job or Pod; wait for it (syndb ops k8s job status) or delete it") when busy_claims is not empty

// ponytail: the guard reads committed values, so a claim named only through --set on the command line is not checked. Upgrade path: run the guard on the rendered manifest.

Tests.

helm-unittest, new suite tests/etl_scratch_claim_test.yaml (templates etl-job.yaml, etl-prepare-job.yaml, etl-scratch-pvc.yaml):

  • it: mounts the named claim and renders no PVC for it
  • it: still renders a per-job PVC when no claim is named
  • it: prepare jobs honour scratch.claimName
  • it: rejects a named claim on ephemeral scratch
  • it: rejects two selected jobs that mount the same claim

Rust:

  • selected_rollout_claims_collects_named_scratch_claims (rollout.rs tests)
  • busy_claims_reports_claims_on_unfinished_jobs_and_live_pods (etl_jobs/tests.rs)
  • busy_claims_ignores_finished_jobs_and_succeeded_pods

Done when. All tests pass; helm template of the chart with default values is byte-identical before and after (no job names a claim yet); helm unittest infrastructure/helm/syndb-etl is green.

RejectedWhy
Skip the S3 fetch when the claim already holds the prepared setsaves minutes per reimport and puts a stale-cache risk on the import path
One claim per pipeline by defaultchanges every existing job’s claim name; the override covers the cases that exist
Point microns import and microns-synapses import at one claim in committed valuesa full render with both selected would then fail by design; no need yet
Convert scratch claims to ReadWriteOncePodaccess modes of a bound claim cannot change without PV access; linstor-csus support is unverified
Top-level dataPvcNamedeprecated, applies to every job at once (templates/etl-job.yaml:6-7)

5. Conditional work (no slice until a capture or an answer asks for it)

TriggerWork
C-U1 shows retired-shard data that production lacks, and Q5 says recover itOwn design pass: a one-shot Job that reads the part directories and writes the missing tables as Parquet under s3://syndb-etl/, then a normal Flight import. Writes to production need the coordinator and an import-free window
Operator wants the isolated restore drill (§3)Own design pass after the pilot (2026-11-24). Uses one retired volume; the other seven are still released

6. Capture steps

#UnknownWhoHowValidate
C-U1What is on each claim, how full, last writeoperator, or coordinator with operator approval (it creates Jobs)Inspection Jobs below (OU1)each log lists every claim’s top level with size and newest mtime; du exit code 0
C-U2PV reclaim policy (PVs are Forbidden on both contexts)operator (nautilus-oidc)kubectl --context nautilus-oidc get storageclass rook-ceph-block linstor-csus -o custom-columns=NAME:.metadata.name,RECLAIM:.reclaimPolicy,BINDING:.volumeBindingMode,EXPAND:.allowVolumeExpansion. A dynamically provisioned PV inherits the class’s policyDelete → releasing a claim destroys the data; Retain → ask NRP to remove the volume
C-U3Claim metadata: owner labels, Helm ownership, the Keeper clones’ delete-after datecoordinator (SA can read PVCs)kubectl --context nautilus -n syndb get pvc -o json and read metadata.labels, .annotations, .ownerReferences, .finalizers, spec.volumeName for the §2 claimsthe two kept scratch claims carry meta.helm.sh/release-name; if not, the rerun must name them through scratch.claimName (U2) so Helm does not try to create them
C-U4Node affinity of each classcoordinatorafter OU1: kubectl -n syndb get pod -l job-name=<job> -o jsonpath='{.items[0].spec.nodeName}'a node name per Job; CSUS for the linstor Job. The scheduler follows the PV’s node affinity, so the pod shows where the class attaches
C-U5Last mount of each claimcoordinatorplan C5, unchangedtimestamp or empty
C-U6Does NRP reject generic ephemeral volumes? The repo disagrees with itself: syndb-etl/templates/image-build-pvc.yaml:4 says yes, nautilus/templates/bench-job.yaml:192-205 renders onecoordinatorkubectl --context nautilus -n syndb apply --dry-run=server -f a Pod with one ephemeral.volumeClaimTemplate volumeadmitted → large scratch can be a per-pod claim that is never idle (follow-up); rejected → bench.scratch.mode: pvc cannot work and needs its own fix

Inspection Job (C-U1). One Job per group, because a pod mounts all its volumes on one node. One du exec, no shell, no sleep; it exits when the walk ends.

Job nameClaimsrunAsUser / runAsGroup
syndb-pvc-inspect-chthe 8 retired ClickHouse claims101 (ClickHouse uid, nautilus/values.yaml:41-43)
syndb-pvc-inspect-misc3 Keeper clones, syndb-ctrlbench-source101 (values.yaml:131-134); rerun the ctrlbench claim as 1000 if du reports permission errors
syndb-pvc-inspect-scratch5 etl-scratch-* claims1000 (syndb-etl/values.yaml:10-12)
apiVersion: batch/v1
kind: Job
metadata:
  name: syndb-pvc-inspect-ch
  namespace: syndb
  labels: {app: syndb-pvc-inspect}
spec:
  backoffLimit: 0
  activeDeadlineSeconds: 3600
  ttlSecondsAfterFinished: 86400
  template:
    metadata:
      labels: {app: syndb-pvc-inspect}
    spec:
      restartPolicy: Never
      automountServiceAccountToken: false
      # No fsGroup: the kubelet must not change ownership on these volumes.
      securityContext: {runAsUser: 101, runAsGroup: 101, runAsNonRoot: true}
      imagePullSecrets: [{name: dockerhub-credentials}]
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - {key: feature.node.kubernetes.io/pci-10de.present, operator: NotIn, values: ["true"]}
      containers:
        - name: du
          image: docker.io/caniko/syndb-etl:0.10.107@sha256:e4e1d11df254bb3241fcc5913895da25a9b2f6f591852ac4dc5ddbd65eee547f
          command: ["du", "-x", "-d", "3", "--time", "-B", "1M", "--exclude=lost+found", "/claims"]
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: {drop: [ALL]}
          resources:
            requests: {cpu: 500m, memory: 512Mi, ephemeral-storage: 1Gi}
            limits: {cpu: 600m, memory: 614Mi, ephemeral-storage: 1Gi}
          volumeMounts:
            - {name: c0, mountPath: /claims/clickhouse-pacific-data-chi-syndb-cluster-syndb-pacific-0-0, readOnly: true}
            # one mount per claim
      volumes:
        - name: c0
          persistentVolumeClaim: {claimName: clickhouse-pacific-data-chi-syndb-cluster-syndb-pacific-0-0, readOnly: true}
        # one volume per claim

NRP fit: requests and limits on every resource, limits at 1.20x, at or below 1 CPU and 2 Gi so not measured, a TTL, a deadline, a command that ends by itself. readOnly: true on the claim source makes the node mount read-only. If one volume cannot mount read-only (unclean journal), the pod stays in ContainerCreating; kubectl describe pod names the volume; delete the Job and rerun without it. du being in the ETL image is assumed from the fetch script’s use of coreutils (_helpers.tpl:345-383); a missing binary fails the Job at once.

Reading the result for the eight ClickHouse volumes.

FindingMeaningNext
under about 1 GiB, or only system tablesempty replicarelease (Q1)
a -1-0 volume whose listing matches its -0-0 siblingredundant copy of the same shardrelease the -1-0 volume
newest mtime before the single-shard cutover, and every dataset on it was re-imported into us-westsupersededrelease (Q1)
anything elsepossibly unique (provenance rows are the non-reconstructible class, ops/backup/verify.rs:66)Q5, then §5

7. Operator steps

All Helm applies follow the plan’s §5 preamble (clean export of HEAD, --no-reset, Flight lease and import checks).

#StepExact commandPreconditionRollbackNeeds
OU1Run the three inspection Jobskubectl --context nautilus -n syndb apply --dry-run=server -f <job>.yaml, then without --dry-run; kubectl -n syndb logs job/<name>; save the logs under the plan’s evidence directoryno Pod or unfinished Job mounts any listed claim: syndb ops k8s report --only nrp-unmounted-pvcs once slice 5 exists, else kubectl -n syndb get pods -o json | jq -r '.items[].spec.volumes[]?.persistentVolumeClaim.claimName // empty' | sort -ukubectl -n syndb delete job -l app=syndb-pvc-inspect (read-only mounts; nothing else to undo)—
OU2Mark the claims that staykubectl -n syndb annotate pvc etl-scratch-prepare-mesh-connectomics-banc-seg-v888 syndb.io/keep-unmounted="<date> BANC mesh prepare cache"; same for etl-scratch-microns-synapses-import ("<date> MICrONS synapse reimport") and syndb-ctrlbench-source ("<date> ctrlbench 10m prepare"); syndb-image-build-store gets "warm Nix store for ops k8s image-build" (replaces plan O7). <date> = the window v2 end date plus 7 days (Q2)Q2 answered. Do it before anyone runs etl cleanupkubectl -n syndb annotate pvc <name> syndb.io/keep-unmounted-U1 for the cleanup to honour it
OU3BANC mesh re-prepare on the kept cachesyndb ops k8s nautilus diff first: it must not show the 250 Gi claim being created or replaced. Then the same syndb ops k8s etl redeploy <selector> --apply used for the original BANC prepare, without --reset-failed-jobsOU2 done; C-U1 shows the cache directory; C-U3 shows Helm ownership on the claimdelete the Job; the cache is only appended to—
OU4MICrONS synapse reimport: dry run, then import, on the 80 Gi claimadd the dry-run entry with scratch: {mode: pvc, claimName: etl-scratch-microns-synapses-import}, commit, syndb ops k8s etl redeploy <selector> --apply; when the dry run has finished, roll out the importU2 merged and the CLI rebuilt; dry run finished before the import starts (the guard enforces it)delete the Job; scratch holds nothing uniqueU2
OU5ctrlbench 10m on the existing source claimthe prepare and run procedure in controlled-benchmark-suite.md:137-143, :174-186no in-scope table was re-released since 2026-11-09 (else re-export, which is a production read: coordinator first)n/a—
OU6Release what has no useplan slice 6: syndb ops k8s pvc-orphans, then --delete <name> … --yes per group; expired scratch claims go with syndb ops k8s etl cleanupper group: Q1/Q3/Q4 answered yes; C-U1 read; C-U2 recorded; for ClickHouse volumes the §6 table says release, or §5 recovery is done; for syndb-ctrlbench-source, OU5 finished and the receipt, version file and manifest are in the evidence directorynone where the class reclaims with Deleteplan slice 6

8. Changes to plan.md

Plan itemChange
Slice 3, SynDBNRPUnmountedPVC (30 d)unchanged. It is the backstop for a kept claim whose run never happens
Slice 5, nrp-unmounted-pvcs“skip PVCs annotated syndb.io/keep-unmounted” becomes “skip PVCs for which keeps_unmounted (U1) is true”; print the annotation value as evidence. An expired marker is listed like any other unmounted claim
Slice 6, pvc-orphansno longer on hold as a tool. --delete refuses a claim with a live keep marker (same predicate). It runs only for groups the operator releases (OU6)
O6replaced by OU6. The blanket “Scratch claims: etl cleanup” line is safe again once U1 lands
O7replaced by OU2
C6replaced by C-U2 (PVs are Forbidden on nautilus-oidc too)
Row F of §1closed by U1, U2, OU1–OU6

9. Operator-only questions

#QuestionDefault if unanswered
Q1I find no real use for 14 claims (950 Gi): the 8 retired ClickHouse volumes, 3 × 40 Gi MICrONS scratch, 3 Keeper clones. Release them once C-U1 shows nothing unique?keep all, run OU1, ask again with the listing
Q2Which window v2 reimports are scheduled, and when does the window end? The 250 Gi and 80 Gi claims are only worth keeping if BANC mesh prepare and the MICrONS synapse import run againmarker date 2026-12-15
Q3Is Keeper on linstor final (no rollback to RBD)? The three clones serve nothing elsekeep
Q4Is ctrlbench a one-off for the manuscript or a recurring run? One-off → release the source claim after the 10m runone-off
Q5If a retired shard holds rows production lacks: recover them into production, or accept the loss?recover
Q6Do you want the isolated restore-drill design (one retired volume, after the pilot)?no

10. Sources

FactSource
Storage only for data in active use; 6-month purge; emptyDir eviction above 50 Ginrp-policy-facts.md → Storage (URLs there)
ETL fetch wipes the pipeline directory on startinfrastructure/helm/syndb-etl/templates/_helpers.tpl:368
Scratch claim names; helm.sh/resource-policy: keepsyndb-etl/templates/etl-scratch-pvc.yaml:19, :50, :28, :58; _helpers.tpl:183-194
BANC decoded-mesh cache location and reusecrates/tools/etl/src/mesh_connectomics/download.rs:297-306, :335-341; --output-dir /data/mesh-connectomics, syndb-etl/values.yaml:910-911
Slice 0 cleanup deletes every unmounted scratch claimworking-tree diff of crates/tools/cli/src/k8s/etl_jobs/jobs.rs on fix/nrp-policy
Retired shard claims are retained on purposesyndb-clickhouse/values.yaml:182-185; templates/chi.yaml:223-225
Live topology is one shard, two replicasnautilus/values.yaml:27-29
Restore drill runs inside the production replicanautilus/templates/clickhouse-restore-drill-cronjob.yaml:57; crates/tools/cli/src/ops/backup/restore_drill.rs:83-89
Keeper clones: purpose and 7-day windowdocumentation/runbooks/keeper-storage-linstor-migration.md:98-101, :124-138, :207-208
ctrlbench source volume contents and lifetimearticle/reproducibility/controlled-benchmark-suite.md:89-92, :146-156
--store-claim, --fresh-store, ReadWriteOncePod storecrates/tools/cli/src/k8s/image_build.rs:104-109, :123-131; syndb-etl/templates/image-build-pvc.yaml:4-10
Dry-run job entries are values entries with a jobNameSuffixsyndb-etl/tests/etl_job_stage_label_test.yaml:9-19
persistentVolumeClaim.readOnlyhttps://kubernetes.io/docs/reference/kubernetes-api/config-and-storage-resources/volume/#persistentvolumeclaim
A dynamically provisioned PV inherits the class’s reclaim policyhttps://kubernetes.io/docs/concepts/storage/storage-classes/#reclaim-policy
The scheduler honours PV node affinityhttps://kubernetes.io/docs/concepts/storage/persistent-volumes/#node-affinity
fsGroup makes the kubelet change volume ownershiphttps://kubernetes.io/docs/tasks/configure-pod-container/security-context/#configure-volume-permission-and-ownership-change-policy-for-pods
Helm adopts an existing object that carries its ownership metadata; resource-policy: keephttps://helm.sh/docs/howto/charts_tips_and_tricks/#tell-helm-not-to-uninstall-a-resource (adoption: validated by OU3’s nautilus diff, not by this page)
du --time, -d, -x, --excludeGNU coreutils manual, du invocation (validated by OU1’s server dry run and first log)

The Kubernetes, Helm and coreutils references were written from memory, not fetched in this pass. OU1’s server-side dry run and C-U3 are the checks that do not depend on them.

11. Noticed, not planned

ItemWhereWhy left out
The largest idle allocation is mounted: us-west replicas use 41 G of 300 Gi each, geo shards 2–5% of 40 Gibrief, Contextnot an NRP breach as written; RBD volumes cannot shrink, so it means rebuilding replicas
dataVolumeSize: 100Gi is smaller than the live 300 Gi volumesnautilus/values.yaml:47-52known drift, already documented there
bench.scratch.mode: pvc may be unusable on NRPC-U6a bench fix, not a storage-retention item
The image store’s own fill is unknownimage-build-job.yaml:84-98 collects garbage at 75%self-managing; inspect only if builds start substituting from scratch

Operator amendments (2026-11-15)

QuestionAnswerEffect
Q1 retired CH volumes (8×100Gi)“What about further ETL usage? We can expand our horizon to more datasets; this is a lot of space we would be losing access to.”Not released. After C-U1 inspection, and §5 recovery of any unique rows, they become the ETL scratch pool for new-dataset imports. Each new-dataset job names one through scratch.claimName (U2), so the pool needs U2 for all jobs, not just the MICrONS dry run. Each claim carries a dated keep marker tied to a planned dataset import, never an open-ended one. Coordinator note to the operator: NRP provisions claims on demand and the namespace has no storage ResourceQuota, so releasing would not lose access. An idle claim stays non-compliant until a run mounts it. Trap to guard against: these claims keep CHI shard names. If a retired shard is ever re-added, the Altinity operator binds them by name, so U2 must refuse a claimName starting with clickhouse- unless the claim has been wiped (or add an explicit --allow-retired-ch-claim check)
Q1/Q3 small claims (3×40Gi MICrONS scratch, 3×10Gi Keeper clones)Keep allKeep. They get dated markers: MICrONS scratch as more pool members (U2); Keeper clones as rollback copies, marked until the operator declares linstor final
Q2 window v2Both BANC mesh re-prepare and MICrONS synapse reimport run; marker 2026-12-15OU2 date = 2026-12-15 for the 250Gi and 80Gi claims
Q4 ctrlbenchRecurringKeep syndb-ctrlbench-source with a dated marker per planned run; no release after the 10m run
Q5 / Q6not askedDefaults stand: recover unique rows; no isolated restore drill
Consequence for slice 0—Slice 0’s etl cleanup deletes every unmounted etl-scratch-* claim after a prompt. With the operator keeping everything, U1 (keep marker) must land in the same PR as slice 0, or before it, and OU2 markers must be applied before anyone runs etl cleanup