Plan: finish SynDB’s NRP (Nautilus) policy compliance
Status (2026-11-18): the slices landed as PRs #30 to #36 (trunk e1522805). Deployment status is tracked in Outstanding Work.
Design pass, 2026-11-15. Read-only: nothing was run against the cluster, S3 or ClickHouse.
Paths are relative to the checkout projects/worktrees/SynDB/nrp-policy (branch
fix/nrp-policy, base 77ec7cfa). “facts” means nrp-policy-facts.md beside this file.
1. What is open, and what closes it
| # | Open item (brief §6) | NRP rule | Closed by |
|---|---|---|---|
| A | Incremental backups fail every run; each attempt leaves a prefix in S3 | storage holds only data in active use (facts, Storage → Retention) | slice 1 (gate), slice 2 (prune), O3–O5 |
| B | Nothing prunes clickhouse/v3/* on a schedule; failed prefixes are kept forever | same | slice 2, O5 |
| C | syndb_nrp_* gauges are placeholders; the alerts on them cannot fire | none (our own monitoring) | slice 3 |
| D | No guard stops a chart edit from breaking the 1.20x rule again | limits within 20% of requests (facts, Requests and limits) | slice 4 |
| E | No on-demand audit of the live namespace | all of the above | slice 5 |
| F | ~1.36 TiB of unmounted PVCs | storage retention | slice 6, O6, O7 |
| G | CH us-west-0/1 use 4–6% of a 2 CPU request | utilisation 20–200% CPU (facts, Usage violations) | operator decision Q1; alert in slice 3 keeps the count under 4 |
| H | Deployment purge exception, PII, 2 GB vs 2 Gi exemption | facts, Deployment purge / Restricted data | O8 (NRP support, operator only) |
Three things found while reading that change the brief’s picture:
- The health collector is not deployed.
health.enabled: false(infrastructure/helm/nautilus/values.yaml:1130), sosyndb_nrp_request_limit_ratiois never scraped.SynDBNRPRatioViolationis not merely wrong for limit=request workloads; it has no series at all. External alert delivery is off too (alertNotify.enabled: false,values.yaml:856). - A chain-aware prune already exists (
crates/tools/cli/src/ops/backup/prune.rs), but it keeps every prefix without a manifest (prune.rs:59-65) and only runs by hand. The chart’s own prune CronJob (infrastructure/helm/syndb-clickhouse/templates/backup-prune-cronjob.yaml) belongs to the disabledsyndb-clickhouse.backuppath (values.yaml:262) and does not matchv2_*ids. - Two CH pods under 20% is inside the allowance. The rule blocks a user at more than 4 violating pods (facts, Usage violations). The live risk is a fifth pod (geo shards, an oversized import), not the two replicas themselves.
2. Slice order
slice 0 (land fix/nrp-policy)
└─ slice 1 backup parity gate ── ships alone
└─ slice 2 prune + prune CronJob ── O3 builds one image for 1+2
slice 3 PromQL alerts, delete placeholders ── needs C2
slice 4 rule functions + render guard
└─ slice 5 live audit (report --only nrp)
└─ slice 6 pvc-orphans CLI ── O6 deletes with it
Slices 3 and 4 do not depend on 1–2 and can run in parallel with them.
Qualification for every slice (no hosted CI): treefmt; the named tests in a capped unit, e.g.
systemd-run --user --pipe -p MemoryMax=16G -p MemorySwapMax=0 cargo test -p cli <filter> </dev/null
from the main direnv; helm unittest infrastructure/helm/nautilus for touched suites.
Slice 1 — backup parity gate tolerates live provenance writes (ships alone)
Goal. Backups succeed while the API and bench suites write query_executed provenance rows.
Problem. The gate requires end == start + 1 (ops/backup/clickhouse.rs:937-953, counts taken at
:134-137 and :258-261). That asserts a quiescent table. created_at is DEFAULT now64(3) and
writers insert data columns only (crates/core/syndb-clickhouse/src/provenance.rs:123, :162-166),
so any row written during the backup breaks the equality.
Design. Count only rows at or before the anchor, and fail only when that count shrinks.
| Item | Change |
|---|---|
| New pure fn | bounded_provenance_count_query(database: &str, anchor_id: &uuid::Uuid) -> String returning SELECT count() FROM {db}.provenance_events WHERE created_at <= (SELECT max(created_at) FROM {db}.provenance_events WHERE id = '{anchor}') (uses existing sql_ident, sql_string) |
| Call sites | run it once after the anchor is written or found (clickhouse.rs:143-164) → start_count; once where end_count is read today (:258-261) |
| Gate | validate_backup_provenance_parity(start_count: u64, end_count: u64) -> Result<()>: error when start_count == 0 (anchor not visible) or end_count < start_count. Drop the anchor_preexisted parameter |
| Payload | the existing unbounded count() at :134-137 stays, only as the anchor payload’s per_shard_count |
| Message | keep the prefix backup parity gate failed: provenance rows {start} -> {end}; alerts and runbooks grep it |
end > start is accepted on purpose: an insert stamped just before the anchor can become visible just
after the first count. Rejecting it would recreate today’s failure.
Files. crates/tools/cli/src/ops/backup/clickhouse.rs, crates/tools/cli/src/ops/backup/tests.rs.
Tests (ops/backup/tests.rs; replace the three backup_parity_* tests at :4-20):
backup_parity_accepts_rows_written_during_backupbackup_parity_rejects_rows_lost_before_the_anchorbackup_parity_rejects_an_invisible_anchorbounded_provenance_count_stops_at_the_anchor_row
Done when.
- The four tests pass;
cargo test -p cli backup_is green. - The query text from
bounded_provenance_count_queryruns on a scratch ClickHouse (binary per thelocal-clickhouse-without-podmanmemory) against a table with the DDL fromprovenance.rs:116-127: insert 3 rows, the anchor, then 2 rows; the query returns 4 both times. Paste the output in the PR. This validates the scalar-subquery form, which is otherwise cited only at page level (§6). - No Helm change;
git diff --statshows the two files.
| Rejected | Why |
|---|---|
Change != to < on the unbounded count | new rows would mask an equal number of lost rows |
| Exclude provenance from parity | provenance is the non-reconstructible class the gate exists for (ops/backup/verify.rs:66) |
| Quiesce the API during backups | a 30-minute cadence would freeze writes most of the day |
| Read the cutoff into Rust and bind it | string round trip through server time zone; the subquery needs no conversion |
Slice 2 — prune failed backup prefixes; schedule the prune; scope the image pin
Goal. Failed and expired backup prefixes leave S3 without a human; the fix image reaches only the ClickHouse backup CronJobs.
Rust.
| Item | Change |
|---|---|
PrunePolicy (prune.rs:22) | add incomplete_retention_hours: Option<i64> |
plan_prune (prune.rs:59-65) | a candidate with no manifest and a parseable id time is deleted when Some(h) and now - time > h hours; reason incomplete older than {h}h. Unparseable ids stay kept |
PruneClickhouseArgs (ops/backup.rs:74) | #[arg(long, env = "BACKUP_INCOMPLETE_RETENTION_HOURS")] incomplete_retention_hours: Option<i64>; reject values below 2 (a run may live 3600 s plus a 1200 s lease wait: values.yaml:940, ops/backup.rs:45) |
| Default | None keeps today’s manual behaviour |
A prefix younger than the retention is never touched, so an in-progress backup is safe without taking the backup lease.
Helm (infrastructure/helm/nautilus).
| Item | Change |
|---|---|
templates/clickhouse-backup-cronjob.yaml | inside the per-target loop, one more CronJob syndb-clickhouse-backup-{{ $name }}-prune: command [syndb, ops, backup, prune-clickhouse, --apply]; env S3_ENDPOINT, S3_BUCKET, S3_PREFIX, BACKUP_TARGET, the two AWS secretKeyRefs, BACKUP_DAILY_RETENTION_DAYS, BACKUP_WEEKLY_RETENTION_WEEKS, BACKUP_MINIMUM_RETAINED (from the so far unused backup.clickhouse.* at values.yaml:970-972), BACKUP_INCOMPLETE_RETENTION_HOURS; automountServiceAccountToken: false; restartPolicy: Never; backoffLimit: 1; activeDeadlineSeconds: 3600; concurrencyPolicy: Forbid; pod label app: syndb-clickhouse-backup (already in the app In […] list of cron-networkpolicy.yaml:10-18, so S3 egress applies) plus syndb.io/backup-kind: prune |
| same template | {{- $image := $.Values.backup.clickhouse.image | default $.Values.backup.image }} for full, incremental and prune. Precedent: backup.restoreDrill.image (values.yaml:905-912). backup.image is shared by five producers (values.yaml:887-889) |
values.yaml under backup.clickhouse | image: {}; prune: {suspend: true, schedule: "40 3 * * *", incompleteRetentionHours: 48, resources: {requests: {cpu: 200m, memory: 1Gi}, limits: {cpu: 240m, memory: 1228Mi}}} |
values.schema.json | add the new keys if the schema is closed (tests/schema_validation_test.yaml) |
The prune CronJob ships suspended. O5 turns it on after a reviewed dry run.
// ponytail: prune_clickhouse holds every object path in memory (prune.rs:166-190). 1 Gi covers
roughly 2–3 M objects; C1 reports the real count. Upgrade path: list per backup id.
Tests.
Rust (ops/backup/tests.rs):
prune_deletes_incomplete_backups_past_retentionprune_keeps_recent_incomplete_backupsprune_rejects_incomplete_retention_below_two_hours- existing
prune_keeps_chains_floor_and_unknownspasses unchanged withincomplete_retention_hours: None
helm-unittest, new suite tests/backup_prune_cronjob_test.yaml:
it: renders one prune CronJob per enabled backup targetit: prune CronJob ships suspendedit: prune CronJob passes the incomplete retention and appliesit: prune CronJob keeps limits within 1.20x of requests
tests/backup_cronjob_layout_test.yaml:
it: ClickHouse backup CronJobs use backup.clickhouse.image when setit: ClickHouse backup CronJobs fall back to backup.image
Done when. All named tests pass; helm template with default values renders the prune CronJobs
with suspend: true and an unchanged image for every other CronJob (diff the render before/after).
| Rejected | Why |
|---|---|
S3 lifecycle expiry (backup-lifecycle-configmap.yaml) | raw age expiry breaks incremental chains (values.yaml:925-926) |
| Delete the prefix inside the failing backup Job | loses the forensics the message promises; a crash would still leak |
| Revive the subchart’s bash prune CronJob | inline shell, wrong id format, not chain-aware |
Bump shared backup.image | moves PG verify, Helm drift, Meilisearch and alert-notify at once |
Slice 3 — real NRP alerts from cluster metrics; delete the placeholders
Goal. The syndb-nrp alert group evaluates live objects. No collector code.
Design. NRP’s Prometheus already feeds our rules kube-state-metrics and kubelet series
(templates/etl-prometheusrule.yaml:14, templates/storage-prometheusrule.yaml:14-42). Use the same
source. Blocked on C2; if a series is missing, drop that rule and say so in the PR (slice 5 still
covers it on demand).
Replace templates/syndb-observability-prometheusrule.yaml:94-110 (NS = {{ .Release.Namespace }}):
| Alert | Severity / for | Expression |
|---|---|---|
SynDBNRPRatioViolation | warning / 10m | kube_pod_container_resource_limits{namespace="NS",resource=~"cpu|memory|ephemeral_storage"} / on (pod, container, resource) kube_pod_container_resource_requests{namespace="NS",resource=~"cpu|memory|ephemeral_storage"} > 1.2001 |
SynDBNRPMissingRequestOrLimit | warning / 10m | kube_pod_container_info{namespace="NS"} unless on (pod, container) (count by (pod, container) ({__name__=~"kube_pod_container_resource_(requests|limits)",namespace="NS",resource=~"cpu|memory"}) == 4) |
SynDBNRPUnderUtilisedPods | warning / 30m at >= 3; critical / 10m at > 4 | count(U) where U is below |
SynDBNRPUnmountedPVC | warning / 6h | kube_persistentvolumeclaim_info{namespace="NS"} unless on (persistentvolumeclaim) max_over_time(kube_pod_spec_volumes_persistentvolumeclaims_info{namespace="NS"}[30d]) |
SynDBNRPQuotaExhaustionForecast | warning / 30m | kube_resourcequota{namespace="NS",type="used"} / ignoring (type) (kube_resourcequota{namespace="NS",type="hard"} > 0) > 0.8 (name and anchor kept; if C3 finds no ResourceQuota, delete the alert, its runbook and INDEX row instead) |
U = (
sum by (pod) (rate(container_cpu_usage_seconds_total{namespace="NS",container!=""}[10m]))
/ sum by (pod) (kube_pod_container_resource_requests{namespace="NS",resource="cpu"}) < 0.20
or
sum by (pod) (container_memory_rss{namespace="NS",container!=""})
/ sum by (pod) (kube_pod_container_resource_requests{namespace="NS",resource="memory"}) < 0.20
)
and on (pod) (
sum by (pod) (kube_pod_container_resource_requests{namespace="NS",resource="cpu"}) > 1
or sum by (pod) (kube_pod_container_resource_requests{namespace="NS",resource="memory"}) > 2 * 1024 * 1024 * 1024
)
and on (pod) (kube_pod_status_phase{namespace="NS",phase="Running"} == 1)
Why these numbers: 1.2001 passes 2457Mi/2Gi (1.19971) and catches 2458Mi/2Gi (1.20020); the 10m
rate matches NRP’s 10-minute check and RSS is the memory NRP reads (documentation/runbooks/nrp-admission.md:9-11,
facts → Monitoring); exempt pods are at or below 1 CPU and 2 Gi (nrp-admission.md:45-47); the 30d
PVC window keeps the warm image-build store quiet without an allowlist. Init containers have separate
series and are covered by slices 4–5, not here.
Files.
| File | Change |
|---|---|
templates/syndb-observability-prometheusrule.yaml | rules above, each with runbook_url |
crates/tools/cli/src/ops/health.rs | delete nrp_metrics (:633-657) and its entry (:217) |
documentation/runbooks/nrp-ratio-violation.md, nrp-quota-exhaustion-forecast.md | replace the TODO stubs: what fired, the query to rerun, the fix (floor(1.20 x request); nrp-admission.md) |
documentation/runbooks/INDEX.md | rows/anchors nrp-missing-request-or-limit, nrp-under-utilised-pods, nrp-unmounted-pvc (the three can be sections of nrp-admission.md) |
tests/prometheus_rule_shape_test.yaml | tests below |
Tests (helm-unittest it:):
NRP alerts read cluster metrics, not the removed placeholder gaugeNRP ratio alert covers cpu, memory and ephemeral storage above 1.20xNRP under-utilisation alerts exempt pods at or below 1 CPU and 2GiNRP unmounted PVC alert looks back 30 daysevery NRP alert carries severity and runbook_url
Done when. Suites pass; rg syndb_nrp_ crates infrastructure documentation is empty;
python3 scripts/check_runbook_anchors.py passes (existing check, not edited); and each inner
expression, run once against Thanos (C2 method), returns what the live audit predicts: ratio → the
three keeper pods until slice 0 is applied; U → exactly chi-syndb-cluster-…-us-west-0/1;
unmounted PVC → the brief §6 list minus syndb-image-build-store. Paste the results in the PR.
| Rejected | Why |
|---|---|
Compute real gauges in ops health serve | collector is disabled; its SA lacks PVC and metrics.k8s.io access (templates/syndb-health-collector.yaml:26-35); Prometheus already has the data |
Keep abs(ratio - 1.2) > 0.01 | fires on every compliant limit=request pod |
predict_linear on pod quota | bench suites make pod counts bursty; a ratio is the honest signal |
| PVC allowlist in values | the 30d window removes the need |
Slice 4 — NRP rule functions and a render-time guard
Goal. A chart edit that breaks a rule fails a local test before any apply.
Files. New crates/tools/cli/src/k8s/report/nrp.rs (pure, no kubectl; inside report so
cpu_millis from report/cluster.rs:592 is in scope), report/mod.rs (pub(crate) mod nrp;),
crates/tools/cli/src/parse_tests/helm_templates.rs, crates/tools/bench/src/suite.rs.
Types and functions.
#![allow(unused)]
fn main() {
pub(crate) const MAX_LIMIT_RATIO: f64 = 1.20;
pub(crate) struct Violation {
pub workload: String, // "Kind/name"
pub container: String,
pub rule: &'static str, // "missing-request-limit" | "limit-ratio" | "sleeping-job"
pub detail: String,
}
/// Every object under `doc` that owns a `containers` array is a pod spec
/// (Deployment, StatefulSet, Job, CronJob, Pod, CHI podTemplates).
pub(crate) fn pod_specs(doc: &serde_json::Value) -> Vec<&serde_json::Value>;
/// cpu and memory need request and limit; ephemeral-storage is ratio-checked when set.
/// Covers `initContainers` and `containers`.
pub(crate) fn resource_violations(workload: &str, pod_spec: &serde_json::Value) -> Vec<Violation>;
/// `sleep infinity|inf`, `tail -f /dev/null`, or `sleep N` with N >= 600 s in command + args.
pub(crate) fn sleeping_command(pod_spec: &serde_json::Value) -> Option<String>;
}
Quantities: CPU through cpu_millis; memory and ephemeral storage through
queryfabric_cli_toolbelt::k8s::parse_quantity (vendor/queryfabric/crates/queryfabric-cli-toolbelt/src/k8s.rs:160).
Do not use parse_quantity for CPU: it casts 500m to 0.
// ponytail: the sleep rule is a token heuristic; a script that computes its sleep passes. The live
audit prints the command for review.
Tests.
report/nrp.rs:
ratio_within_twenty_percent_passes_and_one_mebibyte_over_failsmissing_request_or_limit_is_a_violationinit_containers_are_checkedephemeral_storage_ratio_is_checked_only_when_setpod_specs_are_found_in_jobs_cronjobs_statefulsets_and_chi_pod_templatessleeping_job_commands_are_flagged_and_short_backoff_sleeps_are_not
parse_tests/helm_templates.rs (reuses helm_template :10 and rendered_yaml_docs :263):
nautilus_rendered_workloads_meet_nrp_resource_rules— render the umbrella chart with default values, convert each doc to JSON, assertresource_violationsis empty for every pod spec andsleeping_commandisNoneforJobandCronJobdocs. The panic message lists every violation.
crates/tools/bench/src/suite.rs:
suite_resources_limits_equal_requests(pins slice 0’s change at:808)
Done when. Tests pass. If the render test finds real breaches, fix the values in this slice with
floor(1.20 x request) and list each in the PR; do not weaken the test.
Known gap: the Zalando postgresql resource has no containers key, so the render test skips it.
Slice 5 checks the pods it produces.
| Rejected | Why |
|---|---|
Extend scripts/check_etl_helm_resources.py | no new Python; it only sees ETL-labelled Jobs. It stays as is: crates/tools/ci/src/dispatch/host.rs:45 calls it for ETL-specific rules |
| One helm-unittest assert per workload | 38 suites already do this piecemeal; a new workload would not be covered |
Flag a missing ttlSecondsAfterFinished | not an NRP rule; NRP applies a one-week default (facts → Completed Jobs) |
Slice 5 — live audit: syndb ops k8s report --only nrp
Goal. One read-only command answers “does the namespace comply now?”. It feeds the existing
digest (syndb ops health digest renders report --json, ops/health.rs:54-55), which is the
notification path that is actually on.
Design. Add checks to the existing report (report/mod.rs:309-330) instead of a new nrp audit
command: --only, --json, exit codes and the unknown state on Forbidden already exist
(report/mod.rs:365-393).
| Check id | kubectl (all -n <ns> … -o json) | Logic | Result |
|---|---|---|---|
nrp-resources | get pods | Pending/Running pods → resource_violations | fail, actionable |
nrp-sleeping-jobs | get jobs,cronjobs | unfinished Jobs and all CronJobs → sleeping_command | fail, actionable |
nrp-bulk-job-limits | get jobs | group by label app; a group over 100 needs limit == request on cpu and memory | fail, actionable |
nrp-unmounted-pvcs | get pvc; get pods,jobs,cronjobs,deployments,statefulsets | claim names from pod_specs(...).volumes[].persistentVolumeClaim.claimName, plus <template>-<sts>-<ordinal> for ordinals below spec.replicas; skip PVCs annotated syndb.io/keep-unmounted | warn; evidence name size storageClass created; remediation names pvc-orphans |
nrp-utilisation | get --raw /apis/metrics.k8s.io/v1beta1/namespaces/<ns>/pods; get pods | measured pod: summed requests above 1 CPU or 2 Gi; violating: CPU under 20% or over 200%, memory under 20% or over 150% of the request | ok at 0; warn at 1–4 (n of 4 allowance used); fail above 4; unknown on Forbidden with remediation kubectl config use-context nautilus-oidc |
Each check is a pure …_from(json…) -> CheckResult plus a thin kubectl wrapper using
query_kubectl_json. Metrics usage arrives as nanocores ("77123456n"); add a small
usage_cpu_millis that handles n and u and otherwise defers to cpu_millis.
// ponytail: nrp-utilisation reads one instantaneous sample, while NRP checks every 10 minutes.
The slice 3 alert uses the 10-minute rate; the check says so in its evidence line.
Files. crates/tools/cli/src/k8s/report/nrp.rs, report/mod.rs (register five checks, extend the
module doc), documentation/runbooks/nrp-admission.md (one line under “Other NRP rules”: run the
command before and after an apply).
Tests (report/nrp.rs, JSON fixtures):
live_pods_with_ratio_breaches_fail_the_resource_checkfinished_pods_are_ignored_by_the_resource_checka_sleeping_job_fails_and_a_cronjob_that_runs_kubectl_passesbulk_jobs_over_one_hundred_need_limit_equal_to_requestunmounted_pvcs_exclude_mounted_templated_and_annotated_claimsutilisation_counts_only_pods_above_one_cpu_or_two_gibibytesutilisation_parses_nanocore_usageutilisation_forbidden_is_unknown_with_an_oidc_hint
Done when. Tests pass, and one read-only run from the nautilus-oidc context
(syndb ops k8s report --only nrp --json) reproduces brief §6: nrp-utilisation warns with the two
CH replicas, nrp-unmounted-pvcs lists the §6 claims, the other three are ok. The reviewer or
coordinator runs it; it mutates nothing.
| Rejected | Why |
|---|---|
New syndb ops k8s nrp audit command | a second report framework for five checks |
Preflight checks (k8s/preflight.rs) | preflight is the storage-cutover hard-stop list and exits non-zero on any red |
| A Deployment-age check | a static fact; NRP mails three warnings before a purge (facts → Deployment purge); verify the exception once (O8) |
| Query Thanos from the CLI for utilisation | adds an HTTP client and an unverified endpoint to a kubectl-only tool; slice 3 already does it in-cluster |
Slice 6 — syndb ops k8s pvc-orphans: guarded deletion of unmounted claims
Goal. Deleting retired-topology claims goes through one guarded command, not ad hoc kubectl.
CLI. syndb ops k8s pvc-orphans [--delete NAME]... [--yes]
| Invocation | Behaviour |
|---|---|
| no flags | read-only table of unmounted claims: name, size, storage class, created, owner (app, clickhouse.altinity.com/chi, Helm release annotation), spec.volumeName, keep annotation |
--delete NAME (repeatable) | every NAME must be in the orphan set computed in this run and must not carry syndb.io/keep-unmounted; otherwise error naming the referencing workload. Prints kubectl -n <ns> delete pvc … --wait=false and stops |
--delete NAME --yes | runs that command |
There is no --all. The orphan set is slice 5’s function, so the audit and the delete agree.
Files. New crates/tools/cli/src/k8s/pvc_orphans.rs; crates/tools/cli/src/k8s/args.rs
(K8sCommand::PvcOrphans, dispatch at :51-54); parse_tests/command_parsing.rs; and the manifest that
creates syndb-image-build-store (WARM_STORE_CLAIM, k8s/image_build.rs:32; find the creating
template with rg syndb-image-build-store infrastructure crates) gets the annotation
syndb.io/keep-unmounted: "warm Nix store for ops k8s image-build".
Types. fn plan_delete(orphans: &BTreeSet<String>, keep: &BTreeSet<String>, requested: &[String]) -> Result<Vec<String>>.
Tests.
delete_refuses_a_claim_that_a_workload_still_referencesdelete_refuses_a_keep_annotated_claimdelete_without_yes_only_prints_the_commandk8s_pvc_orphans_parses_repeated_delete(command parsing)warm_store_claim_is_marked_keep_unmounted(in the test module or helm suite of whichever file creates the claim)
Done when. Tests pass; a read-only run lists the same claims as nrp-unmounted-pvcs.
| Rejected | Why |
|---|---|
Widen slice 0’s etl cleanup to all claims | it deletes every orphan it finds after one prompt; right for scratch, wrong for 800 Gi of ClickHouse data |
| Print last-mount time | Kubernetes does not record it; C5 reads it from Thanos once |
Patch PVs to Retain first | keeps the storage allocated, which is the violation |
3. ClickHouse CPU under-utilisation (operator decision, no coder slice by default)
Measured: request 2 CPU, use 77m and 124m (brief §6); 1-minute p99 0.45–0.62 cores
(values.yaml:84-85). To reach 20% at idle the request must be at most ~0.4–0.6 CPU, which puts the
1.20x limit (0.48–0.72) at the measured p99.
| Option | Complies? | Cost | Verdict |
|---|---|---|---|
| Keep 2 CPU; hold the namespace under 4 violating pods; ask NRP for an exception | within the 4-pod allowance (2 used) | none; slices 3 and 5 enforce the budget | recommended |
| Request 1 CPU, limit 1200m | no (8–12% at idle) | CHI roll of both replicas; halves the waste | only as a good-faith signal |
| Request 500m, limit 600m | yes at idle | throttles at p99; pilot users arrive 2026-11-24 | no |
| Off-peak downsizing | yes off-peak | a CHI roll per resize; elastic sizing is disabled on purpose (values.yaml:98-101) | no |
opportunistic priority | unverified; documented only as a GPU-quota bypass with preemption (facts → Special GPUs) | preemptible database | no |
| Merge geo bench CHI into fewer pods | reduces other violators | bench redesign | not now; scale geo shards up only for captures (nrp-admission.md:365) |
If the operator picks 1 CPU, that is conditional slice 7: values.yaml:88-89 and
elasticSizing.steadyState (:102-106) → requests {cpu: "1", memory: 8Gi}, limits {cpu: "1200m", memory: 9830Mi}; update tests/burst_overrides_test.yaml it: renders the single-shard ClickHouse sizing with 1.20x limits; then operator step: syndb ops k8s nautilus apply --no-reset --allow-clickhouse-change from a clean export, after the boot check that flag requires
(k8s/nautilus/mod.rs:74-77), with the Flight lease empty. Rollback: revert the commit and apply again.
4. Capture steps
| # | Unknown | Who | How | Validate |
|---|---|---|---|---|
| C1 | Real S3 bytes and object count of prefixes without a manifest (brief’s “30.9 GB per run” is system.backups.total_size from backup_stats, clickhouse.rs:615-632, which may include base-backup files) | operator, or coordinator with the backup secrets | existing dry run, per target, memory-capped: syndb ops backup prune-clickhouse --s3-endpoint https://s3-west.nrp-nautilus.io --s3-bucket syndb-backups --s3-prefix clickhouse/v3/west --target west with AWS_* from Secret syndb-clickhouse-backup-west-s3; repeat for central (values.yaml:977-999) | KEEP … no completed manifest lines ≈ failed pods since 2026-11-14T21:05Z; the summary line gives bytes |
| C2 | Which series NRP’s Thanos holds for syndb, and whether our rules are evaluated | coordinator (read-only HTTP) | curl -fsSG https://thanos.nrp-nautilus.io/api/v1/query --data-urlencode 'query=count by (__name__) ({__name__=~"kube_pod_container_resource_(requests|limits)|kube_pod_container_info|kube_pod_status_phase|kube_persistentvolumeclaim_info|kube_pod_spec_volumes_persistentvolumeclaims_info|kube_resourcequota|container_cpu_usage_seconds_total|container_memory_rss",namespace="syndb"})' (URL: values.yaml:860; same call shape as templates/alert-notify-cronjob.yaml:51); then query=ALERTS{namespace="syndb"} | nine metric names returned; ALERTS shows the backup-stale alert firing (backups have failed for a day), which proves rule evaluation. If atlas cannot reach Thanos, the operator runs the curl from a pod |
| C3 | ResourceQuota objects (what “reached-quota 19/200” is) | coordinator | kubectl --context nautilus -n syndb get resourcequota -o json | names and status.hard/used match the 19/200 figure |
| C4 | Whether keeper pods (2 Gi = 2.147 GB) are measured if NRP’s exemption is 2 GB | operator (login) | open https://nrp.ai/userinfo, Violations | keeper pods absent → Gi reading holds. Present → 5 violators; tell the coordinator at once |
| C5 | Last time each orphan claim was mounted | coordinator | Thanos: max_over_time(timestamp(kube_pod_spec_volumes_persistentvolumeclaims_info{namespace="syndb",persistentvolumeclaim="<name>"})[180d:6h]) | a timestamp, or empty (not mounted within retention) |
| C6 | Reclaim policy of each orphan’s PV | operator (nautilus-oidc; the SA cannot read PVs) | kubectl --context nautilus-oidc get pv <spec.volumeName> -o jsonpath='{.spec.persistentVolumeReclaimPolicy}' | Delete → O6 destroys the data; Retain → the PV must be deleted too |
5. Operator steps
All nautilus apply runs: from a clean export of HEAD, with --no-reset, after
kubectl -n syndb get lease syndb-flight-finalizer shows no holder and no app=syndb-etl import Job
is active.
| # | Step | Exact command | Precondition | Rollback | Needs |
|---|---|---|---|---|---|
| O1 | Measure leftover backup prefixes | C1 | none (read-only) | n/a | — |
| O2 | Optional: stop the failing incrementals until O4 | kubectl -n syndb patch cronjob syndb-clickhouse-backup-west-incremental -p '{"spec":{"suspend":true}}'; same for -central-incremental | C1 shows material growth. Note: failed prefixes hold a finished BACKUP without a manifest and are the only recent restore points | same patch with false; any nautilus apply also resets it (clickhouse-backup-cronjob.yaml:55) | — |
| O3 | Build and pin the fix image | syndb ops k8s image-build --tag <version>-nrp1 --images etl --rev <merged sha>; syndb ops k8s image-digest docker.io/caniko/syndb-etl:<version>-nrp1; coordinator commits backup.clickhouse.image: {repository, tag, digest}; syndb ops k8s nautilus apply --no-reset | slices 1–2 merged; no release build holds the warm store | remove backup.clickhouse.image, apply | slices 1, 2 |
| O4 | Prove a backup completes | syndb ops backup trigger --cronjob syndb-clickhouse-backup-west-incremental --yes; kubectl -n syndb logs job/<printed name> -f | O3 applied | O3 rollback | O3 |
pass: the log ends with a manifest whose status is BACKUP_CREATED; repeat for central | |||||
| O5 | Prune, then schedule it | dry run as C1 plus --incomplete-retention-hours 48; review; rerun with --apply; then set backup.clickhouse.prune.suspend: false, commit, apply | O4 passed for both targets; the dry run marks at least one complete full and its incrementals KEEP per target | none: S3 deletes are final | slice 2, O4 |
| O6 | Delete orphan claims | syndb ops k8s pvc-orphans; then per group syndb ops k8s pvc-orphans --delete <name> … --yes. Scratch claims: syndb ops k8s etl cleanup (slice 0) | per group, below; C5 and C6 recorded; operator answered Q2 | none where the PV policy is Delete | slice 6 |
| O7 | Mark the warm store as intentionally idle (until the slice 6 manifest change is applied) | kubectl -n syndb annotate pvc syndb-image-build-store syndb.io/keep-unmounted="warm Nix store for ops k8s image-build" | none | kubectl annotate … syndb.io/keep-unmounted- | — |
| O8 | Ask NRP support | Matrix or https://nrp.ai/contact/: (a) is syndb on the Deployment purge exception list; (b) do login emails count as PII under the AUP; (c) exception for two stateful DB replicas under 20% CPU; (d) exemption threshold 2 GB or 2 Gi | none | n/a | — |
O6 preconditions per group:
| Group | Size | Precondition before delete |
|---|---|---|
clickhouse-*-data-chi-syndb-cluster-syndb-{pacific,us-central,us-east,us-mountain}-{0,1}-0 | 8×100 Gi | Q2 confirms the data was re-imported into us-west; live CHI declares no such shard (kubectl -n syndb get chi syndb-cluster -o jsonpath='{.spec.configuration.clusters[*].layout.shards[*].name}'); no chi-syndb-cluster-syndb-<region>-* StatefulSet exists; O4 passed. The 2026-10-23 table-drop incident came from the operator removing shards with reclaim Delete, not from deleting a detached claim, but treat it as irreversible |
etl-scratch-* microns ×4, banc-seg-v888 | 450 Gi | no Job mounts them (the command checks); Q2 says whether the BANC reimport reuses the 250 Gi claim |
keeper-data-rbdbak-syndb-cluster-keeper-{0,1,2} | 30 Gi | Keeper has run on linstor since 2026-11-04 without a restore; ruok/mntr healthy on all three |
syndb-ctrlbench-source | 40 Gi | Q2: not needed for the planned ctrlbench run |
syndb-image-build-store | 300 Gi | keep (O7) |
6. Sources for external facts
| Fact | Source |
|---|---|
| Limits within 20% of requests; limit = request above ~100 pods/Jobs | https://nrp.ai/documentation/userdocs/start/policies/#resource-allocation |
| At most 4 violating pods; CPU 20–200%, RAM 20–150%, GPU >40%; small-pod exemption | https://nrp.ai/documentation/userdocs/start/policies/#resource-usage-violations |
| 10-minute check, completed Jobs excluded, exempt zone “CPU ≤ 1 / memory ≤ 2 Gi” | documentation/runbooks/nrp-admission.md:9-16, :45-47 (NRP admin reply 2026-05-21) |
| Sleeping Jobs prohibited | https://nrp.ai/documentation/userdocs/start/policies/#batch-jobs |
| Deployments purged after 2 weeks, three notices, exceptions | https://nrp.ai/documentation/userdocs/start/policies/#workloads-purging-2-weeks-max-runtime |
| Storage only for active data; 6-month purge | https://nrp.ai/documentation/userdocs/start/policies/#data-purging |
| Default Job TTL of one week | https://nrp.ai/documentation/userdocs/tutorial/jobs/ |
| PII prohibited | https://nrp.ai/NRP-AUP.pdf |
opportunistic semantics | https://nrp.ai/documentation/userdocs/running/priority-classes/ |
kube_pod_container_resource_{requests,limits}, kube_pod_container_info, kube_pod_status_phase, kube_pod_spec_volumes_persistentvolumeclaims_info | https://github.com/kubernetes/kube-state-metrics/blob/main/docs/metrics/workload/pod-metrics.md (presence on NRP: C2) |
kube_persistentvolumeclaim_info | https://github.com/kubernetes/kube-state-metrics/blob/main/docs/metrics/storage/persistentvolumeclaim-metrics.md |
kube_resourcequota{type} | https://github.com/kubernetes/kube-state-metrics/blob/main/docs/metrics/policy/resourcequota-metrics.md |
container_cpu_usage_seconds_total, container_memory_rss | https://github.com/google/cadvisor/blob/master/docs/storage/prometheus.md |
Metrics API path and containers[].usage shape | https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/#metrics-api |
CronJob spec.suspend, spec.jobTemplate.spec.template.spec | https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/cron-job-v1/ |
PV spec.persistentVolumeReclaimPolicy | https://kubernetes.io/docs/concepts/storage/persistent-volumes/#reclaiming |
kubectl patch takes TYPE NAME, no label selector | kubectl patch --help on atlas, 2026-11-15 |
created_at DateTime64(3) DEFAULT now64(3); writers omit it | crates/core/syndb-clickhouse/src/provenance.rs:123, :162-166 |
system.backups num_files, total_size | https://clickhouse.com/docs/operations/system-tables/backups (page level; real bytes: C1) |
Scalar subquery in WHERE | https://clickhouse.com/docs/sql-reference/statements/select (page level; proven by slice 1’s scratch run) |
URLs for the kube-state-metrics, cAdvisor and ClickHouse pages were written from memory, not fetched in this pass. C2 and slice 1’s scratch run are the checks that do not depend on them.
7. Noticed, not planned
| Item | Where | Why left out |
|---|---|---|
GD-F and backup trigger default to syndb-clickhouse-backup, which no longer exists | ops/gameday.rs:33, ops/gameday/cluster.rs:43,382,431, ops/backup.rs:121 | functional bug, not an NRP breach: GD-F fails at load_cronjob before it creates anything. Fix needs a loop over the four names (kubectl patch has no selector). Worth its own small PR |
| External alert delivery is off | values.yaml:852-856 | needs an ntfy destination Secret; the digest path covers slice 5 meanwhile |
Keeper rolls when slice 0’s 2457Mi is applied | values.yaml:171 | belongs to slice 0’s apply: same lease and import checks as any CH rollout |
Failed attempts each insert a backup_anchor row | clickhouse.rs:143-164 | harmless noise; stops with slice 1 |
Operator amendments (2026-11-15)
| Question | Answer | Effect on the plan |
|---|---|---|
| Q1 ClickHouse CPU | Keep 2 CPU | §3 recommended row stands; no slice 7. Slices 3 and 5 hold the namespace under 4 violators; O8(c) asks NRP for an exception |
| Q2 Unmounted PVCs | Do not delete. “We should utilize them instead of getting rid of them. Find ways to utilize them for the benefit of SynDB.” | Slice 6 (guarded deletion) and O6 are on hold. A utilisation design replaces them (planner amendment, plan-pvc-utilisation.md). Compliance comes from putting the claims to active use, not from deleting them |
| Q3 Suspend failing incrementals | Keep running | O2 is not run |
| Q4 Failed-prefix prune age | 48 hours | Slice 2 default incompleteRetentionHours: 48 stands |
| Routes | planner: claude-fable-5-2 high (operator-named); coder: opencode-go/deepseek-v4.1-flash max for slice 1, then openai/gpt-6.1-terra medium from slice 2 on (operator choice after the OpenCode Go weekly limit, 2026-11-15); review: openai/gpt-6.1-sol xhigh | — |