Brief: finish SynDB’s NRP (Nautilus) policy compliance
1. Ask
The operator asked for an audit of SynDB on the NRP Nautilus cluster against
https://nrp.ai/documentation/ and wants every violation fixed for the future.
A first pass is done: four latent or code-level violations are fixed and Sol-approved,
uncommitted, on branch fix/nrp-policy (diff: fix-nrp-policy.diff beside this file).
Several live violations and risks remain open. Plan the remaining work.
2. Deliverable
Write plan.md beside this brief. Also send a reply of at most 10 lines: a summary, then only the
questions that the operator alone can answer.
Who executes: a cheap coder model implements one slice at a time, then a reviewer runs a loop on that slice’s diff. Write PR-sized slices. Each slice needs:
- goal; files; types; Helm values/templates; CLI commands and flags;
- the exact test names to add (Rust
#[test]names, helm-unittestit:names); - done criteria that can be checked.
Slice 1 must ship alone. Slice 0 is landing fix/nrp-policy as-is (already
reviewed). Number your slices from 1, with slice 1 stacked on it.
Live-cluster mutations (PVC deletion, CHI resize, CronJob changes applied via
syndb ops k8s nautilus apply) are never part of a coder slice. List them as
operator steps: exact command, precondition, rollback, and which slice must land first.
Every external fact (NRP rule, kubectl/K8s field path, ClickHouse setting) must cite its source: a URL, or a repo path with a line number. Anything unknown gets a capture step: who produces it, how, and how to validate it. Give one opinionated design per problem. List rejected alternatives in a table, one line each. Use tables over prose.
3. Angles worth weighing (not mandates)
- A real compliance check instead of the placeholder metric: e.g.
syndb ops k8s nrp audit(or a new preflight check) that computes from live objects the per-container request/limit ratio, unmounted PVCs, the limit=request rule for workloads with more than 100 Jobs, Jobs whose command is a sleep, Deployment ages, and the utilisation ratio from the metrics API. Emit them as the realsyndb_nrp_*gauges so the existing alerts stop being fake. Note that thesyndb-operatorSA cannot readmetrics.k8s.io, while thenautilus-oidccontext can. - A render-time guard: a helm-unittest or Rust test over
helm templateoutput that fails on any container with limit/request > 1.20. It could also flag a missing request/limit, a sleeping Job command, or a missingttlSecondsAfterFinished. That catches regressions before apply, not after. - Backups: the parity gate fails because provenance rows grow during the
backup (the API logs
query_executedprovenance continuously; bench suites run queries all day). Options: snapshot-bounded counting (rows withevent_time <= start), excluding the provenance table from parity, or quiescing. Each failed run also leaves ~30 GB in S3 “for forensics” every 30 minutes, which is storage hoarding. A prune/retention step for failed backup prefixes is likely needed. - ClickHouse CPU under-utilisation: request smaller with burst headroom is
impossible under the 1.20x cap. Think about off-peak downsizing, merging the geo
bench CHI into fewer pods, an NRP exception request, or using
opportunistic. These are trade-offs for the operator to pick, not something to decide silently. - Retired-topology PVC cleanup as a reusable CLI (
--dry-rundefault, lists owner/last-mount evidence), not ad-hoc kubectl.
4. Hard constraints
- Repo rules:
/data/nvme0/can/canix/projects/owned/SynDB/AGENTS.md(read it). Key points: ETL logic in Rust, no inline Python in Helm values; neverpkgs.helm; Helm applies without--atomic; boot-test ClickHouse config changes before rollout (Helm tests miss CH startup errors). - Operator memory rules that bind this work:
syndb ops k8s nautilus applyneeds--no-resetwhile other sessions’ Jobs run, and must run from a clean export of HEAD, not the shared working tree.- Before an API or ClickHouse rollout, check the
syndb-flight-finalizerlease and that no Flight imports are active. - Prod ClickHouse reads need coordinator approval and run on us-west-1 with
max_threads4, never during imports. - PVC Delete reclaim drops tables.
- No hosted CI for SynDB merges; qualification is local.
- Agent rules: no scratch Python wrappers. Operator-facing scripts are Rust
subcommands. Format via
treefmtonly. Nushell is the operator shell. - Heavy work: Cargo tests run locally in a memory-capped
systemd-rununit. Heavy data or one-shot checks run on Nautilus only as optional Jobs. - The operator’s ponytail rule applies: smallest diff, no abstractions nobody asked for,
reuse existing helpers (preflight
CheckResult,ShellRunner,queryfabric_cli_toolbelt::k8s::kubectl_json, theMetricsbuilder in health.rs).
5. Repo map (do not re-survey; open only what a slice touches)
Checkout for reading: /data/nvme0/can/canix/projects/worktrees/SynDB/nrp-policy
(branch fix/nrp-policy, base upstream trunk 77ec7cfa). Do not edit it.
| Path | Size | What |
|---|---|---|
| infrastructure/helm/nautilus/values.yaml | large | production values; CH resources at ~L86-93 (2 CPU/8Gi, 1.20x), keeper L165-171 |
| infrastructure/helm/nautilus/templates/syndb-observability-prometheusrule.yaml | ~120 | alerts incl. SynDBNRPRatioViolation L96 (abs(ratio-1.2)>0.01, wrong for limit=request workloads) |
| infrastructure/helm/nautilus/templates/clickhouse-backup-cronjob.yaml | — | backup CronJobs (west/central, full/incremental) |
| infrastructure/helm/nautilus/tests/*.yaml | 38 suites | helm-unittest; e.g. burst_overrides_test.yaml asserts 1.20x |
| infrastructure/helm/syndb-etl/templates/etl-scratch-pvc.yaml | 60 | scratch PVCs, resource-policy: keep |
| crates/tools/cli/src/ops/health.rs | 1629 | prom export; syndb_nrp_request_limit_ratio is a hard-coded placeholder 1.2 (L636-656), quota forecast placeholder 999 |
| crates/tools/cli/src/ops/backup/clickhouse.rs | 1110 | backup; validate_backup_provenance_parity L937-953 |
| crates/tools/cli/src/k8s/preflight.rs, preflight/checks.rs | 467 / 953 | preflight check framework (CheckResult, pvc-fill-ratio etc.) |
| crates/tools/cli/src/k8s/etl_jobs/jobs.rs | — | run_cleanup + new orphan scratch PVC cleanup (slice 0) |
| crates/tools/cli/src/ops/gameday.rs, gameday/cluster.rs | — | GD-F; BACKUP_CRONJOB = "syndb-clickhouse-backup" is stale (live names: syndb-clickhouse-backup-{west,central}-{full,incremental}) |
| crates/tools/bench/src/suite.rs | 1507 | bench suite Jobs (suite_resources) |
| documentation/runbooks/nrp-admission.md | — | NRP rules; new “Other NRP rules” section (slice 0) |
| documentation/runbooks/nrp-ratio-violation.md | 3 | TODO stub, linked from the alert |
| documentation/runbooks/nrp-quota-exhaustion-forecast.md | — | forecast runbook |
6. Facts
- NRP policy extract with URLs:
nrp-policy-facts.md(scout, 2026-11-15). Corrections from operator-side evidence (documentation/runbooks/nrp-admission.md, an NRP admin reply dated 2026-05-21): utilisation is checked every 10 minutes, not over a multi-day window. Pods with at most 1 CPU and at most 2Gi are exempt. Completed Jobs do not count. - Live audit, 2026-11-15, namespace
syndb:- Requests and limits: every container sets both, and all are within 1.20x except keeper
2458Mi/2Gi (fixed in slice 0). All Jobs have a TTL (max 7d). Secrets come in via secretKeyRef.
Ingress is
haproxywith no haproxy.org annotations. No nautilus.io tolerations. The only priority class used isopportunistic(GPU bench). CPU-only pods carry NotIn onpci-10de.present. - Utilisation (
kubectl --context nautilus-oidc top): CH us-west-0/1 request 2 CPU/8Gi and use 77m and 124m CPU (4-6%), with 3.4Gi and 9.0Gi memory. These two pods violate the 20% CPU floor. Everything else is at or below 1 CPU / 2Gi, so exempt. - Unmounted PVCs, ~1.36 TiB in total:
- 8×100Gi
clickhouse-*-data-chi-syndb-cluster-syndb-{pacific,us-central,us-east,us-mountain}-{0,1}-0, created Jun 27-29, from the retired multi-region syndb-cluster topology - etl-scratch: microns-synapses-{dryrun,import,v1412,v1412dry} at 40/80/40/40Gi, plus prepare-mesh-connectomics-banc-seg-v888 at 250Gi (since 2026-07-25)
- keeper-data-rbdbak-syndb-cluster-keeper-{0,1,2}, 10Gi each (2026-11-04)
- syndb-ctrlbench-source, 40Gi (2026-11-09)
- syndb-image-build-store, 300Gi: the warm Nix store, mounted only during image builds; arguably in active use
- 8×100Gi
- Deployments: syndb-api and syndb-ui are 141 days old. NRP purges Deployments after two weeks unless the namespace has an exception, so one probably exists. UNVERIFIED.
- Backup CronJobs: west and central incremental have failed every run since at least
2026-11-14T21:05Z, each with backoffLimit 2, so 3 pods per run. Log:
backup parity gate failed: provenance rows 11938 -> 11942 (allowed drift +1); refusing BACKUP_CREATED, leaving files for forensics. Each run uploads about 30.9 GB (18,804 objects) tos3-west.nrp-nautilus.io/syndb-backups/clickhouse/v3/west/.... The S3 usage of the leftover prefixes is UNVERIFIED and needs a capture step. - Pod quota and GPU:
reached-quotapods 19/200. GPU quotas for a100/h100/h200/gh200 are 0. - Personal data: Postgres stores registered users’ emails. The NRP AUP prohibits PII, and whether login emails count is UNVERIFIED. This is an operator question for NRP support.
- Requests and limits: every container sets both, and all are within 1.20x except keeper
2458Mi/2Gi (fixed in slice 0). All Jobs have a TTL (max 7d). Secrets come in via secretKeyRef.
Ingress is
7. References
None (no UI work).