Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Plan: finish SynDB’s NRP (Nautilus) policy compliance

Status (2026-11-18): the slices landed as PRs #30 to #36 (trunk e1522805). Deployment status is tracked in Outstanding Work.

Design pass, 2026-11-15. Read-only: nothing was run against the cluster, S3 or ClickHouse. Paths are relative to the checkout projects/worktrees/SynDB/nrp-policy (branch fix/nrp-policy, base 77ec7cfa). “facts” means nrp-policy-facts.md beside this file.

1. What is open, and what closes it

#Open item (brief §6)NRP ruleClosed by
AIncremental backups fail every run; each attempt leaves a prefix in S3storage holds only data in active use (facts, Storage → Retention)slice 1 (gate), slice 2 (prune), O3–O5
BNothing prunes clickhouse/v3/* on a schedule; failed prefixes are kept foreversameslice 2, O5
Csyndb_nrp_* gauges are placeholders; the alerts on them cannot firenone (our own monitoring)slice 3
DNo guard stops a chart edit from breaking the 1.20x rule againlimits within 20% of requests (facts, Requests and limits)slice 4
ENo on-demand audit of the live namespaceall of the aboveslice 5
F~1.36 TiB of unmounted PVCsstorage retentionslice 6, O6, O7
GCH us-west-0/1 use 4–6% of a 2 CPU requestutilisation 20–200% CPU (facts, Usage violations)operator decision Q1; alert in slice 3 keeps the count under 4
HDeployment purge exception, PII, 2 GB vs 2 Gi exemptionfacts, Deployment purge / Restricted dataO8 (NRP support, operator only)

Three things found while reading that change the brief’s picture:

  1. The health collector is not deployed. health.enabled: false (infrastructure/helm/nautilus/values.yaml:1130), so syndb_nrp_request_limit_ratio is never scraped. SynDBNRPRatioViolation is not merely wrong for limit=request workloads; it has no series at all. External alert delivery is off too (alertNotify.enabled: false, values.yaml:856).
  2. A chain-aware prune already exists (crates/tools/cli/src/ops/backup/prune.rs), but it keeps every prefix without a manifest (prune.rs:59-65) and only runs by hand. The chart’s own prune CronJob (infrastructure/helm/syndb-clickhouse/templates/backup-prune-cronjob.yaml) belongs to the disabled syndb-clickhouse.backup path (values.yaml:262) and does not match v2_* ids.
  3. Two CH pods under 20% is inside the allowance. The rule blocks a user at more than 4 violating pods (facts, Usage violations). The live risk is a fifth pod (geo shards, an oversized import), not the two replicas themselves.

2. Slice order

slice 0 (land fix/nrp-policy)
  └─ slice 1  backup parity gate            ── ships alone
       └─ slice 2  prune + prune CronJob    ── O3 builds one image for 1+2
slice 3  PromQL alerts, delete placeholders ── needs C2
slice 4  rule functions + render guard
  └─ slice 5  live audit (report --only nrp)
       └─ slice 6  pvc-orphans CLI          ── O6 deletes with it

Slices 3 and 4 do not depend on 1–2 and can run in parallel with them.

Qualification for every slice (no hosted CI): treefmt; the named tests in a capped unit, e.g. systemd-run --user --pipe -p MemoryMax=16G -p MemorySwapMax=0 cargo test -p cli <filter> </dev/null from the main direnv; helm unittest infrastructure/helm/nautilus for touched suites.


Slice 1 — backup parity gate tolerates live provenance writes (ships alone)

Goal. Backups succeed while the API and bench suites write query_executed provenance rows.

Problem. The gate requires end == start + 1 (ops/backup/clickhouse.rs:937-953, counts taken at :134-137 and :258-261). That asserts a quiescent table. created_at is DEFAULT now64(3) and writers insert data columns only (crates/core/syndb-clickhouse/src/provenance.rs:123, :162-166), so any row written during the backup breaks the equality.

Design. Count only rows at or before the anchor, and fail only when that count shrinks.

ItemChange
New pure fnbounded_provenance_count_query(database: &str, anchor_id: &uuid::Uuid) -> String returning SELECT count() FROM {db}.provenance_events WHERE created_at <= (SELECT max(created_at) FROM {db}.provenance_events WHERE id = '{anchor}') (uses existing sql_ident, sql_string)
Call sitesrun it once after the anchor is written or found (clickhouse.rs:143-164) → start_count; once where end_count is read today (:258-261)
Gatevalidate_backup_provenance_parity(start_count: u64, end_count: u64) -> Result<()>: error when start_count == 0 (anchor not visible) or end_count < start_count. Drop the anchor_preexisted parameter
Payloadthe existing unbounded count() at :134-137 stays, only as the anchor payload’s per_shard_count
Messagekeep the prefix backup parity gate failed: provenance rows {start} -> {end}; alerts and runbooks grep it

end > start is accepted on purpose: an insert stamped just before the anchor can become visible just after the first count. Rejecting it would recreate today’s failure.

Files. crates/tools/cli/src/ops/backup/clickhouse.rs, crates/tools/cli/src/ops/backup/tests.rs.

Tests (ops/backup/tests.rs; replace the three backup_parity_* tests at :4-20):

  • backup_parity_accepts_rows_written_during_backup
  • backup_parity_rejects_rows_lost_before_the_anchor
  • backup_parity_rejects_an_invisible_anchor
  • bounded_provenance_count_stops_at_the_anchor_row

Done when.

  • The four tests pass; cargo test -p cli backup_ is green.
  • The query text from bounded_provenance_count_query runs on a scratch ClickHouse (binary per the local-clickhouse-without-podman memory) against a table with the DDL from provenance.rs:116-127: insert 3 rows, the anchor, then 2 rows; the query returns 4 both times. Paste the output in the PR. This validates the scalar-subquery form, which is otherwise cited only at page level (§6).
  • No Helm change; git diff --stat shows the two files.
RejectedWhy
Change != to < on the unbounded countnew rows would mask an equal number of lost rows
Exclude provenance from parityprovenance is the non-reconstructible class the gate exists for (ops/backup/verify.rs:66)
Quiesce the API during backupsa 30-minute cadence would freeze writes most of the day
Read the cutoff into Rust and bind itstring round trip through server time zone; the subquery needs no conversion

Slice 2 — prune failed backup prefixes; schedule the prune; scope the image pin

Goal. Failed and expired backup prefixes leave S3 without a human; the fix image reaches only the ClickHouse backup CronJobs.

Rust.

ItemChange
PrunePolicy (prune.rs:22)add incomplete_retention_hours: Option<i64>
plan_prune (prune.rs:59-65)a candidate with no manifest and a parseable id time is deleted when Some(h) and now - time > h hours; reason incomplete older than {h}h. Unparseable ids stay kept
PruneClickhouseArgs (ops/backup.rs:74)#[arg(long, env = "BACKUP_INCOMPLETE_RETENTION_HOURS")] incomplete_retention_hours: Option<i64>; reject values below 2 (a run may live 3600 s plus a 1200 s lease wait: values.yaml:940, ops/backup.rs:45)
DefaultNone keeps today’s manual behaviour

A prefix younger than the retention is never touched, so an in-progress backup is safe without taking the backup lease.

Helm (infrastructure/helm/nautilus).

ItemChange
templates/clickhouse-backup-cronjob.yamlinside the per-target loop, one more CronJob syndb-clickhouse-backup-{{ $name }}-prune: command [syndb, ops, backup, prune-clickhouse, --apply]; env S3_ENDPOINT, S3_BUCKET, S3_PREFIX, BACKUP_TARGET, the two AWS secretKeyRefs, BACKUP_DAILY_RETENTION_DAYS, BACKUP_WEEKLY_RETENTION_WEEKS, BACKUP_MINIMUM_RETAINED (from the so far unused backup.clickhouse.* at values.yaml:970-972), BACKUP_INCOMPLETE_RETENTION_HOURS; automountServiceAccountToken: false; restartPolicy: Never; backoffLimit: 1; activeDeadlineSeconds: 3600; concurrencyPolicy: Forbid; pod label app: syndb-clickhouse-backup (already in the app In […] list of cron-networkpolicy.yaml:10-18, so S3 egress applies) plus syndb.io/backup-kind: prune
same template{{- $image := $.Values.backup.clickhouse.image | default $.Values.backup.image }} for full, incremental and prune. Precedent: backup.restoreDrill.image (values.yaml:905-912). backup.image is shared by five producers (values.yaml:887-889)
values.yaml under backup.clickhouseimage: {}; prune: {suspend: true, schedule: "40 3 * * *", incompleteRetentionHours: 48, resources: {requests: {cpu: 200m, memory: 1Gi}, limits: {cpu: 240m, memory: 1228Mi}}}
values.schema.jsonadd the new keys if the schema is closed (tests/schema_validation_test.yaml)

The prune CronJob ships suspended. O5 turns it on after a reviewed dry run.

// ponytail: prune_clickhouse holds every object path in memory (prune.rs:166-190). 1 Gi covers roughly 2–3 M objects; C1 reports the real count. Upgrade path: list per backup id.

Tests.

Rust (ops/backup/tests.rs):

  • prune_deletes_incomplete_backups_past_retention
  • prune_keeps_recent_incomplete_backups
  • prune_rejects_incomplete_retention_below_two_hours
  • existing prune_keeps_chains_floor_and_unknowns passes unchanged with incomplete_retention_hours: None

helm-unittest, new suite tests/backup_prune_cronjob_test.yaml:

  • it: renders one prune CronJob per enabled backup target
  • it: prune CronJob ships suspended
  • it: prune CronJob passes the incomplete retention and applies
  • it: prune CronJob keeps limits within 1.20x of requests

tests/backup_cronjob_layout_test.yaml:

  • it: ClickHouse backup CronJobs use backup.clickhouse.image when set
  • it: ClickHouse backup CronJobs fall back to backup.image

Done when. All named tests pass; helm template with default values renders the prune CronJobs with suspend: true and an unchanged image for every other CronJob (diff the render before/after).

RejectedWhy
S3 lifecycle expiry (backup-lifecycle-configmap.yaml)raw age expiry breaks incremental chains (values.yaml:925-926)
Delete the prefix inside the failing backup Jobloses the forensics the message promises; a crash would still leak
Revive the subchart’s bash prune CronJobinline shell, wrong id format, not chain-aware
Bump shared backup.imagemoves PG verify, Helm drift, Meilisearch and alert-notify at once

Slice 3 — real NRP alerts from cluster metrics; delete the placeholders

Goal. The syndb-nrp alert group evaluates live objects. No collector code.

Design. NRP’s Prometheus already feeds our rules kube-state-metrics and kubelet series (templates/etl-prometheusrule.yaml:14, templates/storage-prometheusrule.yaml:14-42). Use the same source. Blocked on C2; if a series is missing, drop that rule and say so in the PR (slice 5 still covers it on demand).

Replace templates/syndb-observability-prometheusrule.yaml:94-110 (NS = {{ .Release.Namespace }}):

AlertSeverity / forExpression
SynDBNRPRatioViolationwarning / 10mkube_pod_container_resource_limits{namespace="NS",resource=~"cpu|memory|ephemeral_storage"} / on (pod, container, resource) kube_pod_container_resource_requests{namespace="NS",resource=~"cpu|memory|ephemeral_storage"} > 1.2001
SynDBNRPMissingRequestOrLimitwarning / 10mkube_pod_container_info{namespace="NS"} unless on (pod, container) (count by (pod, container) ({__name__=~"kube_pod_container_resource_(requests|limits)",namespace="NS",resource=~"cpu|memory"}) == 4)
SynDBNRPUnderUtilisedPodswarning / 30m at >= 3; critical / 10m at > 4count(U) where U is below
SynDBNRPUnmountedPVCwarning / 6hkube_persistentvolumeclaim_info{namespace="NS"} unless on (persistentvolumeclaim) max_over_time(kube_pod_spec_volumes_persistentvolumeclaims_info{namespace="NS"}[30d])
SynDBNRPQuotaExhaustionForecastwarning / 30mkube_resourcequota{namespace="NS",type="used"} / ignoring (type) (kube_resourcequota{namespace="NS",type="hard"} > 0) > 0.8 (name and anchor kept; if C3 finds no ResourceQuota, delete the alert, its runbook and INDEX row instead)
U = (
      sum by (pod) (rate(container_cpu_usage_seconds_total{namespace="NS",container!=""}[10m]))
        / sum by (pod) (kube_pod_container_resource_requests{namespace="NS",resource="cpu"}) < 0.20
      or
      sum by (pod) (container_memory_rss{namespace="NS",container!=""})
        / sum by (pod) (kube_pod_container_resource_requests{namespace="NS",resource="memory"}) < 0.20
    )
    and on (pod) (
      sum by (pod) (kube_pod_container_resource_requests{namespace="NS",resource="cpu"}) > 1
      or sum by (pod) (kube_pod_container_resource_requests{namespace="NS",resource="memory"}) > 2 * 1024 * 1024 * 1024
    )
    and on (pod) (kube_pod_status_phase{namespace="NS",phase="Running"} == 1)

Why these numbers: 1.2001 passes 2457Mi/2Gi (1.19971) and catches 2458Mi/2Gi (1.20020); the 10m rate matches NRP’s 10-minute check and RSS is the memory NRP reads (documentation/runbooks/nrp-admission.md:9-11, facts → Monitoring); exempt pods are at or below 1 CPU and 2 Gi (nrp-admission.md:45-47); the 30d PVC window keeps the warm image-build store quiet without an allowlist. Init containers have separate series and are covered by slices 4–5, not here.

Files.

FileChange
templates/syndb-observability-prometheusrule.yamlrules above, each with runbook_url
crates/tools/cli/src/ops/health.rsdelete nrp_metrics (:633-657) and its entry (:217)
documentation/runbooks/nrp-ratio-violation.md, nrp-quota-exhaustion-forecast.mdreplace the TODO stubs: what fired, the query to rerun, the fix (floor(1.20 x request); nrp-admission.md)
documentation/runbooks/INDEX.mdrows/anchors nrp-missing-request-or-limit, nrp-under-utilised-pods, nrp-unmounted-pvc (the three can be sections of nrp-admission.md)
tests/prometheus_rule_shape_test.yamltests below

Tests (helm-unittest it:):

  • NRP alerts read cluster metrics, not the removed placeholder gauge
  • NRP ratio alert covers cpu, memory and ephemeral storage above 1.20x
  • NRP under-utilisation alerts exempt pods at or below 1 CPU and 2Gi
  • NRP unmounted PVC alert looks back 30 days
  • every NRP alert carries severity and runbook_url

Done when. Suites pass; rg syndb_nrp_ crates infrastructure documentation is empty; python3 scripts/check_runbook_anchors.py passes (existing check, not edited); and each inner expression, run once against Thanos (C2 method), returns what the live audit predicts: ratio → the three keeper pods until slice 0 is applied; U → exactly chi-syndb-cluster-…-us-west-0/1; unmounted PVC → the brief §6 list minus syndb-image-build-store. Paste the results in the PR.

RejectedWhy
Compute real gauges in ops health servecollector is disabled; its SA lacks PVC and metrics.k8s.io access (templates/syndb-health-collector.yaml:26-35); Prometheus already has the data
Keep abs(ratio - 1.2) > 0.01fires on every compliant limit=request pod
predict_linear on pod quotabench suites make pod counts bursty; a ratio is the honest signal
PVC allowlist in valuesthe 30d window removes the need

Slice 4 — NRP rule functions and a render-time guard

Goal. A chart edit that breaks a rule fails a local test before any apply.

Files. New crates/tools/cli/src/k8s/report/nrp.rs (pure, no kubectl; inside report so cpu_millis from report/cluster.rs:592 is in scope), report/mod.rs (pub(crate) mod nrp;), crates/tools/cli/src/parse_tests/helm_templates.rs, crates/tools/bench/src/suite.rs.

Types and functions.

#![allow(unused)]
fn main() {
pub(crate) const MAX_LIMIT_RATIO: f64 = 1.20;

pub(crate) struct Violation {
    pub workload: String,        // "Kind/name"
    pub container: String,
    pub rule: &'static str,      // "missing-request-limit" | "limit-ratio" | "sleeping-job"
    pub detail: String,
}

/// Every object under `doc` that owns a `containers` array is a pod spec
/// (Deployment, StatefulSet, Job, CronJob, Pod, CHI podTemplates).
pub(crate) fn pod_specs(doc: &serde_json::Value) -> Vec<&serde_json::Value>;

/// cpu and memory need request and limit; ephemeral-storage is ratio-checked when set.
/// Covers `initContainers` and `containers`.
pub(crate) fn resource_violations(workload: &str, pod_spec: &serde_json::Value) -> Vec<Violation>;

/// `sleep infinity|inf`, `tail -f /dev/null`, or `sleep N` with N >= 600 s in command + args.
pub(crate) fn sleeping_command(pod_spec: &serde_json::Value) -> Option<String>;
}

Quantities: CPU through cpu_millis; memory and ephemeral storage through queryfabric_cli_toolbelt::k8s::parse_quantity (vendor/queryfabric/crates/queryfabric-cli-toolbelt/src/k8s.rs:160). Do not use parse_quantity for CPU: it casts 500m to 0.

// ponytail: the sleep rule is a token heuristic; a script that computes its sleep passes. The live audit prints the command for review.

Tests.

report/nrp.rs:

  • ratio_within_twenty_percent_passes_and_one_mebibyte_over_fails
  • missing_request_or_limit_is_a_violation
  • init_containers_are_checked
  • ephemeral_storage_ratio_is_checked_only_when_set
  • pod_specs_are_found_in_jobs_cronjobs_statefulsets_and_chi_pod_templates
  • sleeping_job_commands_are_flagged_and_short_backoff_sleeps_are_not

parse_tests/helm_templates.rs (reuses helm_template :10 and rendered_yaml_docs :263):

  • nautilus_rendered_workloads_meet_nrp_resource_rules — render the umbrella chart with default values, convert each doc to JSON, assert resource_violations is empty for every pod spec and sleeping_command is None for Job and CronJob docs. The panic message lists every violation.

crates/tools/bench/src/suite.rs:

  • suite_resources_limits_equal_requests (pins slice 0’s change at :808)

Done when. Tests pass. If the render test finds real breaches, fix the values in this slice with floor(1.20 x request) and list each in the PR; do not weaken the test.

Known gap: the Zalando postgresql resource has no containers key, so the render test skips it. Slice 5 checks the pods it produces.

RejectedWhy
Extend scripts/check_etl_helm_resources.pyno new Python; it only sees ETL-labelled Jobs. It stays as is: crates/tools/ci/src/dispatch/host.rs:45 calls it for ETL-specific rules
One helm-unittest assert per workload38 suites already do this piecemeal; a new workload would not be covered
Flag a missing ttlSecondsAfterFinishednot an NRP rule; NRP applies a one-week default (facts → Completed Jobs)

Slice 5 — live audit: syndb ops k8s report --only nrp

Goal. One read-only command answers “does the namespace comply now?”. It feeds the existing digest (syndb ops health digest renders report --json, ops/health.rs:54-55), which is the notification path that is actually on.

Design. Add checks to the existing report (report/mod.rs:309-330) instead of a new nrp audit command: --only, --json, exit codes and the unknown state on Forbidden already exist (report/mod.rs:365-393).

Check idkubectl (all -n <ns> … -o json)LogicResult
nrp-resourcesget podsPending/Running pods → resource_violationsfail, actionable
nrp-sleeping-jobsget jobs,cronjobsunfinished Jobs and all CronJobs → sleeping_commandfail, actionable
nrp-bulk-job-limitsget jobsgroup by label app; a group over 100 needs limit == request on cpu and memoryfail, actionable
nrp-unmounted-pvcsget pvc; get pods,jobs,cronjobs,deployments,statefulsetsclaim names from pod_specs(...).volumes[].persistentVolumeClaim.claimName, plus <template>-<sts>-<ordinal> for ordinals below spec.replicas; skip PVCs annotated syndb.io/keep-unmountedwarn; evidence name size storageClass created; remediation names pvc-orphans
nrp-utilisationget --raw /apis/metrics.k8s.io/v1beta1/namespaces/<ns>/pods; get podsmeasured pod: summed requests above 1 CPU or 2 Gi; violating: CPU under 20% or over 200%, memory under 20% or over 150% of the requestok at 0; warn at 1–4 (n of 4 allowance used); fail above 4; unknown on Forbidden with remediation kubectl config use-context nautilus-oidc

Each check is a pure …_from(json…) -> CheckResult plus a thin kubectl wrapper using query_kubectl_json. Metrics usage arrives as nanocores ("77123456n"); add a small usage_cpu_millis that handles n and u and otherwise defers to cpu_millis.

// ponytail: nrp-utilisation reads one instantaneous sample, while NRP checks every 10 minutes. The slice 3 alert uses the 10-minute rate; the check says so in its evidence line.

Files. crates/tools/cli/src/k8s/report/nrp.rs, report/mod.rs (register five checks, extend the module doc), documentation/runbooks/nrp-admission.md (one line under “Other NRP rules”: run the command before and after an apply).

Tests (report/nrp.rs, JSON fixtures):

  • live_pods_with_ratio_breaches_fail_the_resource_check
  • finished_pods_are_ignored_by_the_resource_check
  • a_sleeping_job_fails_and_a_cronjob_that_runs_kubectl_passes
  • bulk_jobs_over_one_hundred_need_limit_equal_to_request
  • unmounted_pvcs_exclude_mounted_templated_and_annotated_claims
  • utilisation_counts_only_pods_above_one_cpu_or_two_gibibytes
  • utilisation_parses_nanocore_usage
  • utilisation_forbidden_is_unknown_with_an_oidc_hint

Done when. Tests pass, and one read-only run from the nautilus-oidc context (syndb ops k8s report --only nrp --json) reproduces brief §6: nrp-utilisation warns with the two CH replicas, nrp-unmounted-pvcs lists the §6 claims, the other three are ok. The reviewer or coordinator runs it; it mutates nothing.

RejectedWhy
New syndb ops k8s nrp audit commanda second report framework for five checks
Preflight checks (k8s/preflight.rs)preflight is the storage-cutover hard-stop list and exits non-zero on any red
A Deployment-age checka static fact; NRP mails three warnings before a purge (facts → Deployment purge); verify the exception once (O8)
Query Thanos from the CLI for utilisationadds an HTTP client and an unverified endpoint to a kubectl-only tool; slice 3 already does it in-cluster

Slice 6 — syndb ops k8s pvc-orphans: guarded deletion of unmounted claims

Goal. Deleting retired-topology claims goes through one guarded command, not ad hoc kubectl.

CLI. syndb ops k8s pvc-orphans [--delete NAME]... [--yes]

InvocationBehaviour
no flagsread-only table of unmounted claims: name, size, storage class, created, owner (app, clickhouse.altinity.com/chi, Helm release annotation), spec.volumeName, keep annotation
--delete NAME (repeatable)every NAME must be in the orphan set computed in this run and must not carry syndb.io/keep-unmounted; otherwise error naming the referencing workload. Prints kubectl -n <ns> delete pvc … --wait=false and stops
--delete NAME --yesruns that command

There is no --all. The orphan set is slice 5’s function, so the audit and the delete agree.

Files. New crates/tools/cli/src/k8s/pvc_orphans.rs; crates/tools/cli/src/k8s/args.rs (K8sCommand::PvcOrphans, dispatch at :51-54); parse_tests/command_parsing.rs; and the manifest that creates syndb-image-build-store (WARM_STORE_CLAIM, k8s/image_build.rs:32; find the creating template with rg syndb-image-build-store infrastructure crates) gets the annotation syndb.io/keep-unmounted: "warm Nix store for ops k8s image-build".

Types. fn plan_delete(orphans: &BTreeSet<String>, keep: &BTreeSet<String>, requested: &[String]) -> Result<Vec<String>>.

Tests.

  • delete_refuses_a_claim_that_a_workload_still_references
  • delete_refuses_a_keep_annotated_claim
  • delete_without_yes_only_prints_the_command
  • k8s_pvc_orphans_parses_repeated_delete (command parsing)
  • warm_store_claim_is_marked_keep_unmounted (in the test module or helm suite of whichever file creates the claim)

Done when. Tests pass; a read-only run lists the same claims as nrp-unmounted-pvcs.

RejectedWhy
Widen slice 0’s etl cleanup to all claimsit deletes every orphan it finds after one prompt; right for scratch, wrong for 800 Gi of ClickHouse data
Print last-mount timeKubernetes does not record it; C5 reads it from Thanos once
Patch PVs to Retain firstkeeps the storage allocated, which is the violation

3. ClickHouse CPU under-utilisation (operator decision, no coder slice by default)

Measured: request 2 CPU, use 77m and 124m (brief §6); 1-minute p99 0.45–0.62 cores (values.yaml:84-85). To reach 20% at idle the request must be at most ~0.4–0.6 CPU, which puts the 1.20x limit (0.48–0.72) at the measured p99.

OptionComplies?CostVerdict
Keep 2 CPU; hold the namespace under 4 violating pods; ask NRP for an exceptionwithin the 4-pod allowance (2 used)none; slices 3 and 5 enforce the budgetrecommended
Request 1 CPU, limit 1200mno (8–12% at idle)CHI roll of both replicas; halves the wasteonly as a good-faith signal
Request 500m, limit 600myes at idlethrottles at p99; pilot users arrive 2026-11-24no
Off-peak downsizingyes off-peaka CHI roll per resize; elastic sizing is disabled on purpose (values.yaml:98-101)no
opportunistic priorityunverified; documented only as a GPU-quota bypass with preemption (facts → Special GPUs)preemptible databaseno
Merge geo bench CHI into fewer podsreduces other violatorsbench redesignnot now; scale geo shards up only for captures (nrp-admission.md:365)

If the operator picks 1 CPU, that is conditional slice 7: values.yaml:88-89 and elasticSizing.steadyState (:102-106) → requests {cpu: "1", memory: 8Gi}, limits {cpu: "1200m", memory: 9830Mi}; update tests/burst_overrides_test.yaml it: renders the single-shard ClickHouse sizing with 1.20x limits; then operator step: syndb ops k8s nautilus apply --no-reset --allow-clickhouse-change from a clean export, after the boot check that flag requires (k8s/nautilus/mod.rs:74-77), with the Flight lease empty. Rollback: revert the commit and apply again.


4. Capture steps

#UnknownWhoHowValidate
C1Real S3 bytes and object count of prefixes without a manifest (brief’s “30.9 GB per run” is system.backups.total_size from backup_stats, clickhouse.rs:615-632, which may include base-backup files)operator, or coordinator with the backup secretsexisting dry run, per target, memory-capped: syndb ops backup prune-clickhouse --s3-endpoint https://s3-west.nrp-nautilus.io --s3-bucket syndb-backups --s3-prefix clickhouse/v3/west --target west with AWS_* from Secret syndb-clickhouse-backup-west-s3; repeat for central (values.yaml:977-999)KEEP … no completed manifest lines ≈ failed pods since 2026-11-14T21:05Z; the summary line gives bytes
C2Which series NRP’s Thanos holds for syndb, and whether our rules are evaluatedcoordinator (read-only HTTP)curl -fsSG https://thanos.nrp-nautilus.io/api/v1/query --data-urlencode 'query=count by (__name__) ({__name__=~"kube_pod_container_resource_(requests|limits)|kube_pod_container_info|kube_pod_status_phase|kube_persistentvolumeclaim_info|kube_pod_spec_volumes_persistentvolumeclaims_info|kube_resourcequota|container_cpu_usage_seconds_total|container_memory_rss",namespace="syndb"})' (URL: values.yaml:860; same call shape as templates/alert-notify-cronjob.yaml:51); then query=ALERTS{namespace="syndb"}nine metric names returned; ALERTS shows the backup-stale alert firing (backups have failed for a day), which proves rule evaluation. If atlas cannot reach Thanos, the operator runs the curl from a pod
C3ResourceQuota objects (what “reached-quota 19/200” is)coordinatorkubectl --context nautilus -n syndb get resourcequota -o jsonnames and status.hard/used match the 19/200 figure
C4Whether keeper pods (2 Gi = 2.147 GB) are measured if NRP’s exemption is 2 GBoperator (login)open https://nrp.ai/userinfo, Violationskeeper pods absent → Gi reading holds. Present → 5 violators; tell the coordinator at once
C5Last time each orphan claim was mountedcoordinatorThanos: max_over_time(timestamp(kube_pod_spec_volumes_persistentvolumeclaims_info{namespace="syndb",persistentvolumeclaim="<name>"})[180d:6h])a timestamp, or empty (not mounted within retention)
C6Reclaim policy of each orphan’s PVoperator (nautilus-oidc; the SA cannot read PVs)kubectl --context nautilus-oidc get pv <spec.volumeName> -o jsonpath='{.spec.persistentVolumeReclaimPolicy}'Delete → O6 destroys the data; Retain → the PV must be deleted too

5. Operator steps

All nautilus apply runs: from a clean export of HEAD, with --no-reset, after kubectl -n syndb get lease syndb-flight-finalizer shows no holder and no app=syndb-etl import Job is active.

#StepExact commandPreconditionRollbackNeeds
O1Measure leftover backup prefixesC1none (read-only)n/a—
O2Optional: stop the failing incrementals until O4kubectl -n syndb patch cronjob syndb-clickhouse-backup-west-incremental -p '{"spec":{"suspend":true}}'; same for -central-incrementalC1 shows material growth. Note: failed prefixes hold a finished BACKUP without a manifest and are the only recent restore pointssame patch with false; any nautilus apply also resets it (clickhouse-backup-cronjob.yaml:55)—
O3Build and pin the fix imagesyndb ops k8s image-build --tag <version>-nrp1 --images etl --rev <merged sha>; syndb ops k8s image-digest docker.io/caniko/syndb-etl:<version>-nrp1; coordinator commits backup.clickhouse.image: {repository, tag, digest}; syndb ops k8s nautilus apply --no-resetslices 1–2 merged; no release build holds the warm storeremove backup.clickhouse.image, applyslices 1, 2
O4Prove a backup completessyndb ops backup trigger --cronjob syndb-clickhouse-backup-west-incremental --yes; kubectl -n syndb logs job/<printed name> -fO3 appliedO3 rollbackO3
pass: the log ends with a manifest whose status is BACKUP_CREATED; repeat for central
O5Prune, then schedule itdry run as C1 plus --incomplete-retention-hours 48; review; rerun with --apply; then set backup.clickhouse.prune.suspend: false, commit, applyO4 passed for both targets; the dry run marks at least one complete full and its incrementals KEEP per targetnone: S3 deletes are finalslice 2, O4
O6Delete orphan claimssyndb ops k8s pvc-orphans; then per group syndb ops k8s pvc-orphans --delete <name> … --yes. Scratch claims: syndb ops k8s etl cleanup (slice 0)per group, below; C5 and C6 recorded; operator answered Q2none where the PV policy is Deleteslice 6
O7Mark the warm store as intentionally idle (until the slice 6 manifest change is applied)kubectl -n syndb annotate pvc syndb-image-build-store syndb.io/keep-unmounted="warm Nix store for ops k8s image-build"nonekubectl annotate … syndb.io/keep-unmounted-—
O8Ask NRP supportMatrix or https://nrp.ai/contact/: (a) is syndb on the Deployment purge exception list; (b) do login emails count as PII under the AUP; (c) exception for two stateful DB replicas under 20% CPU; (d) exemption threshold 2 GB or 2 Ginonen/a—

O6 preconditions per group:

GroupSizePrecondition before delete
clickhouse-*-data-chi-syndb-cluster-syndb-{pacific,us-central,us-east,us-mountain}-{0,1}-08×100 GiQ2 confirms the data was re-imported into us-west; live CHI declares no such shard (kubectl -n syndb get chi syndb-cluster -o jsonpath='{.spec.configuration.clusters[*].layout.shards[*].name}'); no chi-syndb-cluster-syndb-<region>-* StatefulSet exists; O4 passed. The 2026-10-23 table-drop incident came from the operator removing shards with reclaim Delete, not from deleting a detached claim, but treat it as irreversible
etl-scratch-* microns ×4, banc-seg-v888450 Gino Job mounts them (the command checks); Q2 says whether the BANC reimport reuses the 250 Gi claim
keeper-data-rbdbak-syndb-cluster-keeper-{0,1,2}30 GiKeeper has run on linstor since 2026-11-04 without a restore; ruok/mntr healthy on all three
syndb-ctrlbench-source40 GiQ2: not needed for the planned ctrlbench run
syndb-image-build-store300 Gikeep (O7)

6. Sources for external facts

FactSource
Limits within 20% of requests; limit = request above ~100 pods/Jobshttps://nrp.ai/documentation/userdocs/start/policies/#resource-allocation
At most 4 violating pods; CPU 20–200%, RAM 20–150%, GPU >40%; small-pod exemptionhttps://nrp.ai/documentation/userdocs/start/policies/#resource-usage-violations
10-minute check, completed Jobs excluded, exempt zone “CPU ≤ 1 / memory ≤ 2 Gi”documentation/runbooks/nrp-admission.md:9-16, :45-47 (NRP admin reply 2026-05-21)
Sleeping Jobs prohibitedhttps://nrp.ai/documentation/userdocs/start/policies/#batch-jobs
Deployments purged after 2 weeks, three notices, exceptionshttps://nrp.ai/documentation/userdocs/start/policies/#workloads-purging-2-weeks-max-runtime
Storage only for active data; 6-month purgehttps://nrp.ai/documentation/userdocs/start/policies/#data-purging
Default Job TTL of one weekhttps://nrp.ai/documentation/userdocs/tutorial/jobs/
PII prohibitedhttps://nrp.ai/NRP-AUP.pdf
opportunistic semanticshttps://nrp.ai/documentation/userdocs/running/priority-classes/
kube_pod_container_resource_{requests,limits}, kube_pod_container_info, kube_pod_status_phase, kube_pod_spec_volumes_persistentvolumeclaims_infohttps://github.com/kubernetes/kube-state-metrics/blob/main/docs/metrics/workload/pod-metrics.md (presence on NRP: C2)
kube_persistentvolumeclaim_infohttps://github.com/kubernetes/kube-state-metrics/blob/main/docs/metrics/storage/persistentvolumeclaim-metrics.md
kube_resourcequota{type}https://github.com/kubernetes/kube-state-metrics/blob/main/docs/metrics/policy/resourcequota-metrics.md
container_cpu_usage_seconds_total, container_memory_rsshttps://github.com/google/cadvisor/blob/master/docs/storage/prometheus.md
Metrics API path and containers[].usage shapehttps://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/#metrics-api
CronJob spec.suspend, spec.jobTemplate.spec.template.spechttps://kubernetes.io/docs/reference/kubernetes-api/workload-resources/cron-job-v1/
PV spec.persistentVolumeReclaimPolicyhttps://kubernetes.io/docs/concepts/storage/persistent-volumes/#reclaiming
kubectl patch takes TYPE NAME, no label selectorkubectl patch --help on atlas, 2026-11-15
created_at DateTime64(3) DEFAULT now64(3); writers omit itcrates/core/syndb-clickhouse/src/provenance.rs:123, :162-166
system.backups num_files, total_sizehttps://clickhouse.com/docs/operations/system-tables/backups (page level; real bytes: C1)
Scalar subquery in WHEREhttps://clickhouse.com/docs/sql-reference/statements/select (page level; proven by slice 1’s scratch run)

URLs for the kube-state-metrics, cAdvisor and ClickHouse pages were written from memory, not fetched in this pass. C2 and slice 1’s scratch run are the checks that do not depend on them.


7. Noticed, not planned

ItemWhereWhy left out
GD-F and backup trigger default to syndb-clickhouse-backup, which no longer existsops/gameday.rs:33, ops/gameday/cluster.rs:43,382,431, ops/backup.rs:121functional bug, not an NRP breach: GD-F fails at load_cronjob before it creates anything. Fix needs a loop over the four names (kubectl patch has no selector). Worth its own small PR
External alert delivery is offvalues.yaml:852-856needs an ntfy destination Secret; the digest path covers slice 5 meanwhile
Keeper rolls when slice 0’s 2457Mi is appliedvalues.yaml:171belongs to slice 0’s apply: same lease and import checks as any CH rollout
Failed attempts each insert a backup_anchor rowclickhouse.rs:143-164harmless noise; stops with slice 1

Operator amendments (2026-11-15)

QuestionAnswerEffect on the plan
Q1 ClickHouse CPUKeep 2 CPU§3 recommended row stands; no slice 7. Slices 3 and 5 hold the namespace under 4 violators; O8(c) asks NRP for an exception
Q2 Unmounted PVCsDo not delete. “We should utilize them instead of getting rid of them. Find ways to utilize them for the benefit of SynDB.”Slice 6 (guarded deletion) and O6 are on hold. A utilisation design replaces them (planner amendment, plan-pvc-utilisation.md). Compliance comes from putting the claims to active use, not from deleting them
Q3 Suspend failing incrementalsKeep runningO2 is not run
Q4 Failed-prefix prune age48 hoursSlice 2 default incompleteRetentionHours: 48 stands
Routesplanner: claude-fable-5-2 high (operator-named); coder: opencode-go/deepseek-v4.1-flash max for slice 1, then openai/gpt-6.1-terra medium from slice 2 on (operator choice after the OpenCode Go weekly limit, 2026-11-15); review: openai/gpt-6.1-sol xhigh—