Plan amendment: put the unmounted PVCs to work (replaces slice 6 / O6 as the default)
Design pass, 2026-11-15. Read-only: nothing was run against the cluster. Paths are relative to
projects/worktrees/SynDB/nrp-policy (branch fix/nrp-policy). “plan” means plan.md beside this
file; “brief” means brief-pvc-utilisation.md.
1. Result
Three claims have a real use. Fourteen do not, and I recommend releasing them (operator questions, §9).
| Verdict | Claims | Size |
|---|---|---|
| In use already | syndb-image-build-store | 300 Gi |
| Put to use (this plan) | BANC mesh cache 250 Gi, MICrONS import scratch 80 Gi, syndb-ctrlbench-source 40 Gi (one run, then release) | 370 Gi |
| No credible use; release after inspection | 8 retired ClickHouse volumes, 3 × 40 Gi MICrONS scratch, 3 Keeper clones | 950 Gi |
Why so few. Three facts from the repo limit what an old claim can become:
- A claim can only be used under its own name. PVC names are immutable, and re-binding a volume to a new claim needs write access to PersistentVolumes, which both contexts lack (brief, Unknown). So only workloads that name a claim explicitly (Jobs, CronJobs) can mount one. StatefulSet and CHI claim templates cannot adopt it.
- Most of these volumes are capacity, not data. ETL scratch is wiped at the start of every run:
the fetch init container runs
rm -rf "$DEST_DIR" "$STAGE_DIR"(infrastructure/helm/syndb-etl/templates/_helpers.tpl:368). An empty block volume is recreated in seconds, so keeping one idle buys nothing. - The volumes in use have room. Production ClickHouse is 14% full on 300 Gi (brief, Context), so nothing needs a cold tier, a cache disk or a spool volume.
The rule this plan applies (and writes into the runbook, U1):
A claim stays only if a named workload mounts it on a stated date or cadence, and either the bytes on it are reused across runs or the run needs more than 50 Gi of scratch (NRP evicts above 50 Gi of emptyDir, facts → Local scratch). Every kept claim carries a dated keep marker. When the date passes without a mount, the audit lists it again and it is released.
2. One proposed use per claim
| Claim | Proposed use | Workload and cadence | Expected fill / fit | Placement | Data on it today |
|---|---|---|---|---|---|
etl-scratch-prepare-mesh-connectomics-banc-seg-v888 (250 Gi) | Decoded-mesh cache for the BANC mesh prepare | Job syndb-etl-prepare-mesh-connectomics-banc-seg-v888 (syndb-etl/values.yaml:874-911) mounts it by its rendered name, no chart change. Runs once per BANC mesh re-prepare; next in window v2 | Sized for the full inventory (values.yaml:884-890); real fill from C-U1 | linstor-csus, RWO, CSUS nodes only. Must not be shared with another mesh source: the cache key is the bare segment id (crates/tools/etl/src/mesh_connectomics/download.rs:297-303, :339-341) | Expected: /data/.mesh-connectomics-cache/<segment>.bin plus the last output directory. The cache survives the fetch wipe and makes a rerun resume (download.rs:335-338). Confirm with C-U1 |
etl-scratch-microns-synapses-import (80 Gi) | Scratch above 50 Gi for the MICrONS synapse reimport: the import Job and its dry run share it | syndb-etl-microns-synapses-import mounts it by name (nautilus/values.yaml:1632-1643); the dry-run variant names it through scratch.claimName (U2). Once per reimport; next in window v2 | Import needs more than 50 Gi (values.yaml:1639); 80 Gi was sized for it | linstor-csus, RWO, CSUS nodes. Dry run and import must not overlap (U2 guard) | Stale copy of s3://syndb-etl/microns-synapses-prepared/; the next run deletes it. Nothing unique expected (C-U1) |
etl-scratch-microns-synapses-{dryrun,v1412,v1412dry} (3 × 40 Gi) | None. | Their Jobs were one-off suffixes for the 2026-11-04 v1412 import. With U2 a dry run shares the 80 Gi claim. Every other job that fits in 40 Gi also fits NRP’s emptyDir allowance | — | linstor-csus | Stale copies of the same S3 prefix (C-U1). Recommend release (Q1) |
syndb-ctrlbench-source (40 Gi) | Source exports for the planned “ctrlbench 10m” run | The ctrlbench prepare Job mounts it at /source (article/reproducibility/controlled-benchmark-suite.md:89-92, :137-143). One run in window v2. Reuse avoids a second read of production ClickHouse, if no in-scope table was re-released since the export | 0.86 GiB of Parquet plus release-receipt.json (:89, :146-149); fits 40 Gi 40 times over | rook-ceph-block, RWO | The 2026-11-09 exports and receipt. The reproducibility doc already ends with “before deleting the source PVC” (:154-156). Release after the run unless ctrlbench is recurring (Q4) |
keeper-data-rbdbak-syndb-cluster-keeper-{0,1,2} (3 × 10 Gi) | None. | They exist only for rollback-member.sh. The runbook kept them 7 days from 2026-11-04 (documentation/runbooks/keeper-storage-linstor-migration.md:98-101, :137-138) | under 10 Gi each | rook-ceph-block | Keeper state as of 2026-11-04, now 11 days behind the quorum. Recommend release once the operator accepts Keeper on linstor as final (Q3) |
clickhouse-{pacific,us-central,us-east,us-mountain}-data-chi-syndb-cluster-syndb-<region>-{0,1}-0 (8 × 100 Gi) | None as claims. The only possible value is data on them that production lacks | See §3 | unknown; C-U1 | rook-ceph-block, RWO. Named and (probably) labelled as CHI syndb-cluster shard data | Replica data of the removed shards, kept on purpose by pvcReclaimPolicy: Retain (syndb-clickhouse/values.yaml:182-185, templates/chi.yaml:223-225). Unknown whether production holds all of it. Inspect, recover what is unique, then release (Q1, Q5) |
syndb-image-build-store (300 Gi) | Warm Nix store (already in use) | syndb ops k8s image-build, every image build | — | rook-ceph-block, ReadWriteOncePod | Keep; marker without a date (OU2) |
3. The eight retired ClickHouse volumes: angles weighed
| Angle | Verdict | Reason |
|---|---|---|
| Filesystem cache or S3-backed cold tier for production | no | Production is 14% full; clickhouse.tieredStorage.enabled: false (nautilus/values.yaml:444-448). It would be a CHI change, a boot test and a replica roll for capacity nobody needs. The CHI cannot mount these names anyway (§1.1; allowStorageMigration guard, syndb-clickhouse/values.yaml:174-181) |
| Staging for Flight imports or job-result spooling on the API | no | The API scales horizontally (templates/api-hpa.yaml); an RWO volume attaches to one node. Results already stream to S3 (trunk f2b64a86, 8e5632e1) |
| Re-add regional shards or replicas | no | Each replica is another pod under 20% CPU. The allowance is 4 and 2 are used (plan §1.3) |
Second warm Nix store (--store-claim) | no | image-build --fresh-store already lets a second build run in parallel (crates/tools/cli/src/k8s/image_build.rs:108-109). Wiping retired replica data to host a Nix store under a CHI shard name is a delete plus a trap: the Altinity operator binds claims by name if that shard ever returns |
| Isolated restore-drill target (one volume) | the only candidate; not now | Real benefit: today the weekly drill restores into the production replica (templates/clickhouse-restore-drill-cronjob.yaml:57, ops/backup/restore_drill.rs:83-89), and us-west-1 already sits at 9.0 Gi of a 9830 Mi limit (brief.md §6). Cost: a standalone ClickHouse with Keeper access for replicated tables, a drill that no longer reads the source database on the same server, new RBAC to start and stop it, and a boot test. The volume would still be empty six days in seven. Needs its own design pass after the pilot; uses one volume of eight |
| Archive the contents to S3 “in case” | no | NRP storage is not an archive (facts → Retention). Recover only rows production lacks |
What decides the eight is their content (C-U1, then §6 table). Structure helps: each shard has two replicas, so at most four volumes hold distinct data.
4. Slice order
slice 0 (land fix/nrp-policy)
└─ U1 keep marker; etl cleanup respects it; runbook ── must land before any reimport cleanup
U2 scratch.claimName + collision guard ── independent; needed for OU4 only
plan slice 5 ── reuses U1's predicate (amendment in §8)
plan slice 6 ── stays the release tool for claims the operator releases
Qualification is the plan’s (§2 there): treefmt, the named tests in a capped unit, helm unittest infrastructure/helm/syndb-etl for U2.
U1 — dated keep marker; etl cleanup stops deleting kept claims
Goal. A claim with a planned next use survives syndb ops k8s etl cleanup, and a kept claim
cannot stay kept silently.
Problem. Slice 0’s cleanup deletes every etl-scratch-* claim that nothing mounts, after one
prompt (crates/tools/cli/src/k8s/etl_jobs/jobs.rs, cleanup_orphan_scratch_claims and
orphan_scratch_claims). The first cleanup after any import would delete the 250 Gi BANC cache.
Design. One annotation, syndb.io/keep-unmounted (the key plan slices 5–6 already use). Its
value is <YYYY-MM-DD> <reason> (keep through that day) or free text (no expiry; only for the image
store).
| Item | Change |
|---|---|
etl_jobs/jobs.rs | pub(crate) const KEEP_UNMOUNTED_ANNOTATION: &str = "syndb.io/keep-unmounted"; |
| same | pub(crate) fn keeps_unmounted(value: Option<&str>, today: &str) -> bool: None → false; value starting with ten characters shaped dddd-dd-dd → date >= today (ISO dates compare as strings); anything else → true |
orphan_scratch_claims | new parameter today: &str; skip a claim when keeps_unmounted(metadata.annotations[KEY], today) |
cleanup_orphan_scratch_claims | pass today’s UTC date as YYYY-MM-DD (use the date crate the cli crate already depends on); print kept: <name> (<annotation value>) for each skipped claim before the prompt |
documentation/runbooks/pvc-unmounted.md | replace the TODO stub: the rule from §1, the marker format, a register table (claim, workload, keep-until) filled from §2, the inspection Job from §6, and the release command (plan slice 6) |
documentation/runbooks/nrp-admission.md | in “Other NRP rules”, storage bullet: add “A claim kept for a planned run carries syndb.io/keep-unmounted; see pvc-unmounted.md” |
No chart change: Helm leaves annotations it never rendered alone, so the operator’s annotation (OU2) survives upgrades.
Tests (etl_jobs/tests.rs):
keep_marker_with_a_future_date_keeps_and_a_past_date_expireskeep_marker_without_a_date_never_expiresorphan_scratch_claims_skip_claims_with_a_live_keep_marker- existing
orphan_scratch_claims_skip_claims_mounted_by_jobs_or_podspasses with the new parameter
Done when. The four tests pass; python3 scripts/check_runbook_anchors.py passes;
rg -n TODO documentation/runbooks/pvc-unmounted.md is empty.
| Rejected | Why |
|---|---|
Separate keep-until annotation | two keys to keep in step; one value with a leading date is enough |
Chart value scratch.keepReason rendering the annotation | Helm does not strip unmanaged annotations; the claims to keep already exist |
| Permanent keep for scratch claims | that is the idle hoarding the rule forbids; the date forces a new decision |
U2 — scratch.claimName: an ETL Job mounts an existing claim instead of minting one
Goal. A dry run or a re-run uses the claim its pipeline already has. No new claim per job suffix (the reason four MICrONS claims exist).
Chart (infrastructure/helm/syndb-etl).
| Item | Change |
|---|---|
templates/_helpers.tpl, syndb-etl.scratchVolume (:174-198) | in pvc mode the claim is get $jobScratch "claimName" | default (printf "etl-scratch-%s" $claimSuffix); fail when claimName is set and the mode is not pvc |
templates/etl-scratch-pvc.yaml | both loops (:8-38, :39-67): render no PVC for a job whose scratch.claimName is set. Collect the effective claim name of every selected job in one dict and fail on a repeat: jobs <a> and <b> both mount scratch claim <c>; roll them out one at a time |
values.schema.json, scratchConfig (:99-120) | add claimName: string, minLength: 1 |
values.yaml comment block (:1242-1247) | document claimName and the one-at-a-time rule |
No committed job gets a claimName in this slice. A dry-run entry is a temporary values edit
(shape: tests/etl_job_stage_label_test.yaml:15-19) and names the claim there.
CLI guard (the brief’s constraint: reuse must not collide with a running Job). This matters:
a dry run and its import share /data/<pipeline>, and the fetch wipes that directory on start.
| Item | Change |
|---|---|
crates/tools/cli/src/k8s/etl_jobs/rollout.rs | pub(super) fn selected_rollout_claims(selected: &[String]) -> Result<BTreeSet<String>>, beside selected_rollout_jobs (:52): the scratch.claimName of every selected entry, walking the same entries as add_job_names_from_section (:326) |
etl_jobs/jobs.rs | pub(crate) fn busy_claims(wanted: &BTreeSet<String>, jobs: &serde_json::Value, pods: &serde_json::Value) -> Vec<String>: claims in wanted that claimed_names (slice 0) finds on Jobs without a Complete or Failed condition, or on Pods in phase Pending or Running |
selected-rollout apply (k8s/nautilus/mod.rs, the call that adds syndb-etl.jobsEnabled=true, :360-370) | after the selected Jobs are deleted and before Helm runs: bail!("scratch claim {c} is mounted by an unfinished Job or Pod; wait for it (syndb ops k8s job status) or delete it") when busy_claims is not empty |
// ponytail: the guard reads committed values, so a claim named only through --set on the command
line is not checked. Upgrade path: run the guard on the rendered manifest.
Tests.
helm-unittest, new suite tests/etl_scratch_claim_test.yaml (templates etl-job.yaml,
etl-prepare-job.yaml, etl-scratch-pvc.yaml):
it: mounts the named claim and renders no PVC for itit: still renders a per-job PVC when no claim is namedit: prepare jobs honour scratch.claimNameit: rejects a named claim on ephemeral scratchit: rejects two selected jobs that mount the same claim
Rust:
selected_rollout_claims_collects_named_scratch_claims(rollout.rstests)busy_claims_reports_claims_on_unfinished_jobs_and_live_pods(etl_jobs/tests.rs)busy_claims_ignores_finished_jobs_and_succeeded_pods
Done when. All tests pass; helm template of the chart with default values is byte-identical
before and after (no job names a claim yet); helm unittest infrastructure/helm/syndb-etl is green.
| Rejected | Why |
|---|---|
| Skip the S3 fetch when the claim already holds the prepared set | saves minutes per reimport and puts a stale-cache risk on the import path |
| One claim per pipeline by default | changes every existing job’s claim name; the override covers the cases that exist |
Point microns import and microns-synapses import at one claim in committed values | a full render with both selected would then fail by design; no need yet |
| Convert scratch claims to ReadWriteOncePod | access modes of a bound claim cannot change without PV access; linstor-csus support is unverified |
Top-level dataPvcName | deprecated, applies to every job at once (templates/etl-job.yaml:6-7) |
5. Conditional work (no slice until a capture or an answer asks for it)
| Trigger | Work |
|---|---|
| C-U1 shows retired-shard data that production lacks, and Q5 says recover it | Own design pass: a one-shot Job that reads the part directories and writes the missing tables as Parquet under s3://syndb-etl/, then a normal Flight import. Writes to production need the coordinator and an import-free window |
| Operator wants the isolated restore drill (§3) | Own design pass after the pilot (2026-11-24). Uses one retired volume; the other seven are still released |
6. Capture steps
| # | Unknown | Who | How | Validate |
|---|---|---|---|---|
| C-U1 | What is on each claim, how full, last write | operator, or coordinator with operator approval (it creates Jobs) | Inspection Jobs below (OU1) | each log lists every claim’s top level with size and newest mtime; du exit code 0 |
| C-U2 | PV reclaim policy (PVs are Forbidden on both contexts) | operator (nautilus-oidc) | kubectl --context nautilus-oidc get storageclass rook-ceph-block linstor-csus -o custom-columns=NAME:.metadata.name,RECLAIM:.reclaimPolicy,BINDING:.volumeBindingMode,EXPAND:.allowVolumeExpansion. A dynamically provisioned PV inherits the class’s policy | Delete → releasing a claim destroys the data; Retain → ask NRP to remove the volume |
| C-U3 | Claim metadata: owner labels, Helm ownership, the Keeper clones’ delete-after date | coordinator (SA can read PVCs) | kubectl --context nautilus -n syndb get pvc -o json and read metadata.labels, .annotations, .ownerReferences, .finalizers, spec.volumeName for the §2 claims | the two kept scratch claims carry meta.helm.sh/release-name; if not, the rerun must name them through scratch.claimName (U2) so Helm does not try to create them |
| C-U4 | Node affinity of each class | coordinator | after OU1: kubectl -n syndb get pod -l job-name=<job> -o jsonpath='{.items[0].spec.nodeName}' | a node name per Job; CSUS for the linstor Job. The scheduler follows the PV’s node affinity, so the pod shows where the class attaches |
| C-U5 | Last mount of each claim | coordinator | plan C5, unchanged | timestamp or empty |
| C-U6 | Does NRP reject generic ephemeral volumes? The repo disagrees with itself: syndb-etl/templates/image-build-pvc.yaml:4 says yes, nautilus/templates/bench-job.yaml:192-205 renders one | coordinator | kubectl --context nautilus -n syndb apply --dry-run=server -f a Pod with one ephemeral.volumeClaimTemplate volume | admitted → large scratch can be a per-pod claim that is never idle (follow-up); rejected → bench.scratch.mode: pvc cannot work and needs its own fix |
Inspection Job (C-U1). One Job per group, because a pod mounts all its volumes on one node. One
du exec, no shell, no sleep; it exits when the walk ends.
| Job name | Claims | runAsUser / runAsGroup |
|---|---|---|
syndb-pvc-inspect-ch | the 8 retired ClickHouse claims | 101 (ClickHouse uid, nautilus/values.yaml:41-43) |
syndb-pvc-inspect-misc | 3 Keeper clones, syndb-ctrlbench-source | 101 (values.yaml:131-134); rerun the ctrlbench claim as 1000 if du reports permission errors |
syndb-pvc-inspect-scratch | 5 etl-scratch-* claims | 1000 (syndb-etl/values.yaml:10-12) |
apiVersion: batch/v1
kind: Job
metadata:
name: syndb-pvc-inspect-ch
namespace: syndb
labels: {app: syndb-pvc-inspect}
spec:
backoffLimit: 0
activeDeadlineSeconds: 3600
ttlSecondsAfterFinished: 86400
template:
metadata:
labels: {app: syndb-pvc-inspect}
spec:
restartPolicy: Never
automountServiceAccountToken: false
# No fsGroup: the kubelet must not change ownership on these volumes.
securityContext: {runAsUser: 101, runAsGroup: 101, runAsNonRoot: true}
imagePullSecrets: [{name: dockerhub-credentials}]
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- {key: feature.node.kubernetes.io/pci-10de.present, operator: NotIn, values: ["true"]}
containers:
- name: du
image: docker.io/caniko/syndb-etl:0.10.107@sha256:e4e1d11df254bb3241fcc5913895da25a9b2f6f591852ac4dc5ddbd65eee547f
command: ["du", "-x", "-d", "3", "--time", "-B", "1M", "--exclude=lost+found", "/claims"]
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: {drop: [ALL]}
resources:
requests: {cpu: 500m, memory: 512Mi, ephemeral-storage: 1Gi}
limits: {cpu: 600m, memory: 614Mi, ephemeral-storage: 1Gi}
volumeMounts:
- {name: c0, mountPath: /claims/clickhouse-pacific-data-chi-syndb-cluster-syndb-pacific-0-0, readOnly: true}
# one mount per claim
volumes:
- name: c0
persistentVolumeClaim: {claimName: clickhouse-pacific-data-chi-syndb-cluster-syndb-pacific-0-0, readOnly: true}
# one volume per claim
NRP fit: requests and limits on every resource, limits at 1.20x, at or below 1 CPU and 2 Gi so not
measured, a TTL, a deadline, a command that ends by itself. readOnly: true on the claim source makes
the node mount read-only. If one volume cannot mount read-only (unclean journal), the pod stays in
ContainerCreating; kubectl describe pod names the volume; delete the Job and rerun without it.
du being in the ETL image is assumed from the fetch script’s use of coreutils
(_helpers.tpl:345-383); a missing binary fails the Job at once.
Reading the result for the eight ClickHouse volumes.
| Finding | Meaning | Next |
|---|---|---|
under about 1 GiB, or only system tables | empty replica | release (Q1) |
a -1-0 volume whose listing matches its -0-0 sibling | redundant copy of the same shard | release the -1-0 volume |
| newest mtime before the single-shard cutover, and every dataset on it was re-imported into us-west | superseded | release (Q1) |
| anything else | possibly unique (provenance rows are the non-reconstructible class, ops/backup/verify.rs:66) | Q5, then §5 |
7. Operator steps
All Helm applies follow the plan’s §5 preamble (clean export of HEAD, --no-reset, Flight lease and
import checks).
| # | Step | Exact command | Precondition | Rollback | Needs |
|---|---|---|---|---|---|
| OU1 | Run the three inspection Jobs | kubectl --context nautilus -n syndb apply --dry-run=server -f <job>.yaml, then without --dry-run; kubectl -n syndb logs job/<name>; save the logs under the plan’s evidence directory | no Pod or unfinished Job mounts any listed claim: syndb ops k8s report --only nrp-unmounted-pvcs once slice 5 exists, else kubectl -n syndb get pods -o json | jq -r '.items[].spec.volumes[]?.persistentVolumeClaim.claimName // empty' | sort -u | kubectl -n syndb delete job -l app=syndb-pvc-inspect (read-only mounts; nothing else to undo) | — |
| OU2 | Mark the claims that stay | kubectl -n syndb annotate pvc etl-scratch-prepare-mesh-connectomics-banc-seg-v888 syndb.io/keep-unmounted="<date> BANC mesh prepare cache"; same for etl-scratch-microns-synapses-import ("<date> MICrONS synapse reimport") and syndb-ctrlbench-source ("<date> ctrlbench 10m prepare"); syndb-image-build-store gets "warm Nix store for ops k8s image-build" (replaces plan O7). <date> = the window v2 end date plus 7 days (Q2) | Q2 answered. Do it before anyone runs etl cleanup | kubectl -n syndb annotate pvc <name> syndb.io/keep-unmounted- | U1 for the cleanup to honour it |
| OU3 | BANC mesh re-prepare on the kept cache | syndb ops k8s nautilus diff first: it must not show the 250 Gi claim being created or replaced. Then the same syndb ops k8s etl redeploy <selector> --apply used for the original BANC prepare, without --reset-failed-jobs | OU2 done; C-U1 shows the cache directory; C-U3 shows Helm ownership on the claim | delete the Job; the cache is only appended to | — |
| OU4 | MICrONS synapse reimport: dry run, then import, on the 80 Gi claim | add the dry-run entry with scratch: {mode: pvc, claimName: etl-scratch-microns-synapses-import}, commit, syndb ops k8s etl redeploy <selector> --apply; when the dry run has finished, roll out the import | U2 merged and the CLI rebuilt; dry run finished before the import starts (the guard enforces it) | delete the Job; scratch holds nothing unique | U2 |
| OU5 | ctrlbench 10m on the existing source claim | the prepare and run procedure in controlled-benchmark-suite.md:137-143, :174-186 | no in-scope table was re-released since 2026-11-09 (else re-export, which is a production read: coordinator first) | n/a | — |
| OU6 | Release what has no use | plan slice 6: syndb ops k8s pvc-orphans, then --delete <name> … --yes per group; expired scratch claims go with syndb ops k8s etl cleanup | per group: Q1/Q3/Q4 answered yes; C-U1 read; C-U2 recorded; for ClickHouse volumes the §6 table says release, or §5 recovery is done; for syndb-ctrlbench-source, OU5 finished and the receipt, version file and manifest are in the evidence directory | none where the class reclaims with Delete | plan slice 6 |
8. Changes to plan.md
| Plan item | Change |
|---|---|
Slice 3, SynDBNRPUnmountedPVC (30 d) | unchanged. It is the backstop for a kept claim whose run never happens |
Slice 5, nrp-unmounted-pvcs | “skip PVCs annotated syndb.io/keep-unmounted” becomes “skip PVCs for which keeps_unmounted (U1) is true”; print the annotation value as evidence. An expired marker is listed like any other unmounted claim |
Slice 6, pvc-orphans | no longer on hold as a tool. --delete refuses a claim with a live keep marker (same predicate). It runs only for groups the operator releases (OU6) |
| O6 | replaced by OU6. The blanket “Scratch claims: etl cleanup” line is safe again once U1 lands |
| O7 | replaced by OU2 |
| C6 | replaced by C-U2 (PVs are Forbidden on nautilus-oidc too) |
| Row F of §1 | closed by U1, U2, OU1–OU6 |
9. Operator-only questions
| # | Question | Default if unanswered |
|---|---|---|
| Q1 | I find no real use for 14 claims (950 Gi): the 8 retired ClickHouse volumes, 3 × 40 Gi MICrONS scratch, 3 Keeper clones. Release them once C-U1 shows nothing unique? | keep all, run OU1, ask again with the listing |
| Q2 | Which window v2 reimports are scheduled, and when does the window end? The 250 Gi and 80 Gi claims are only worth keeping if BANC mesh prepare and the MICrONS synapse import run again | marker date 2026-12-15 |
| Q3 | Is Keeper on linstor final (no rollback to RBD)? The three clones serve nothing else | keep |
| Q4 | Is ctrlbench a one-off for the manuscript or a recurring run? One-off → release the source claim after the 10m run | one-off |
| Q5 | If a retired shard holds rows production lacks: recover them into production, or accept the loss? | recover |
| Q6 | Do you want the isolated restore-drill design (one retired volume, after the pilot)? | no |
10. Sources
| Fact | Source |
|---|---|
| Storage only for data in active use; 6-month purge; emptyDir eviction above 50 Gi | nrp-policy-facts.md → Storage (URLs there) |
| ETL fetch wipes the pipeline directory on start | infrastructure/helm/syndb-etl/templates/_helpers.tpl:368 |
Scratch claim names; helm.sh/resource-policy: keep | syndb-etl/templates/etl-scratch-pvc.yaml:19, :50, :28, :58; _helpers.tpl:183-194 |
| BANC decoded-mesh cache location and reuse | crates/tools/etl/src/mesh_connectomics/download.rs:297-306, :335-341; --output-dir /data/mesh-connectomics, syndb-etl/values.yaml:910-911 |
| Slice 0 cleanup deletes every unmounted scratch claim | working-tree diff of crates/tools/cli/src/k8s/etl_jobs/jobs.rs on fix/nrp-policy |
| Retired shard claims are retained on purpose | syndb-clickhouse/values.yaml:182-185; templates/chi.yaml:223-225 |
| Live topology is one shard, two replicas | nautilus/values.yaml:27-29 |
| Restore drill runs inside the production replica | nautilus/templates/clickhouse-restore-drill-cronjob.yaml:57; crates/tools/cli/src/ops/backup/restore_drill.rs:83-89 |
| Keeper clones: purpose and 7-day window | documentation/runbooks/keeper-storage-linstor-migration.md:98-101, :124-138, :207-208 |
| ctrlbench source volume contents and lifetime | article/reproducibility/controlled-benchmark-suite.md:89-92, :146-156 |
--store-claim, --fresh-store, ReadWriteOncePod store | crates/tools/cli/src/k8s/image_build.rs:104-109, :123-131; syndb-etl/templates/image-build-pvc.yaml:4-10 |
Dry-run job entries are values entries with a jobNameSuffix | syndb-etl/tests/etl_job_stage_label_test.yaml:9-19 |
persistentVolumeClaim.readOnly | https://kubernetes.io/docs/reference/kubernetes-api/config-and-storage-resources/volume/#persistentvolumeclaim |
| A dynamically provisioned PV inherits the class’s reclaim policy | https://kubernetes.io/docs/concepts/storage/storage-classes/#reclaim-policy |
| The scheduler honours PV node affinity | https://kubernetes.io/docs/concepts/storage/persistent-volumes/#node-affinity |
| fsGroup makes the kubelet change volume ownership | https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#configure-volume-permission-and-ownership-change-policy-for-pods |
Helm adopts an existing object that carries its ownership metadata; resource-policy: keep | https://helm.sh/docs/howto/charts_tips_and_tricks/#tell-helm-not-to-uninstall-a-resource (adoption: validated by OU3’s nautilus diff, not by this page) |
du --time, -d, -x, --exclude | GNU coreutils manual, du invocation (validated by OU1’s server dry run and first log) |
The Kubernetes, Helm and coreutils references were written from memory, not fetched in this pass. OU1’s server-side dry run and C-U3 are the checks that do not depend on them.
11. Noticed, not planned
| Item | Where | Why left out |
|---|---|---|
| The largest idle allocation is mounted: us-west replicas use 41 G of 300 Gi each, geo shards 2–5% of 40 Gi | brief, Context | not an NRP breach as written; RBD volumes cannot shrink, so it means rebuilding replicas |
dataVolumeSize: 100Gi is smaller than the live 300 Gi volumes | nautilus/values.yaml:47-52 | known drift, already documented there |
bench.scratch.mode: pvc may be unusable on NRP | C-U6 | a bench fix, not a storage-retention item |
| The image store’s own fill is unknown | image-build-job.yaml:84-98 collects garbage at 75% | self-managing; inspect only if builds start substituting from scratch |
Operator amendments (2026-11-15)
| Question | Answer | Effect |
|---|---|---|
| Q1 retired CH volumes (8×100Gi) | “What about further ETL usage? We can expand our horizon to more datasets; this is a lot of space we would be losing access to.” | Not released. After C-U1 inspection, and §5 recovery of any unique rows, they become the ETL scratch pool for new-dataset imports. Each new-dataset job names one through scratch.claimName (U2), so the pool needs U2 for all jobs, not just the MICrONS dry run. Each claim carries a dated keep marker tied to a planned dataset import, never an open-ended one. Coordinator note to the operator: NRP provisions claims on demand and the namespace has no storage ResourceQuota, so releasing would not lose access. An idle claim stays non-compliant until a run mounts it. Trap to guard against: these claims keep CHI shard names. If a retired shard is ever re-added, the Altinity operator binds them by name, so U2 must refuse a claimName starting with clickhouse- unless the claim has been wiped (or add an explicit --allow-retired-ch-claim check) |
| Q1/Q3 small claims (3×40Gi MICrONS scratch, 3×10Gi Keeper clones) | Keep all | Keep. They get dated markers: MICrONS scratch as more pool members (U2); Keeper clones as rollback copies, marked until the operator declares linstor final |
| Q2 window v2 | Both BANC mesh re-prepare and MICrONS synapse reimport run; marker 2026-12-15 | OU2 date = 2026-12-15 for the 250Gi and 80Gi claims |
| Q4 ctrlbench | Recurring | Keep syndb-ctrlbench-source with a dated marker per planned run; no release after the 10m run |
| Q5 / Q6 | not asked | Defaults stand: recover unique rows; no isolated restore drill |
| Consequence for slice 0 | — | Slice 0’s etl cleanup deletes every unmounted etl-scratch-* claim after a prompt. With the operator keeping everything, U1 (keep marker) must land in the same PR as slice 0, or before it, and OU2 markers must be applied before anyone runs etl cleanup |