Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Brief: finish SynDB’s NRP (Nautilus) policy compliance

1. Ask

The operator asked for an audit of SynDB on the NRP Nautilus cluster against https://nrp.ai/documentation/ and wants every violation fixed for the future. A first pass is done: four latent or code-level violations are fixed and Sol-approved, uncommitted, on branch fix/nrp-policy (diff: fix-nrp-policy.diff beside this file). Several live violations and risks remain open. Plan the remaining work.

2. Deliverable

Write plan.md beside this brief. Also send a reply of at most 10 lines: a summary, then only the questions that the operator alone can answer.

Who executes: a cheap coder model implements one slice at a time, then a reviewer runs a loop on that slice’s diff. Write PR-sized slices. Each slice needs:

  • goal; files; types; Helm values/templates; CLI commands and flags;
  • the exact test names to add (Rust #[test] names, helm-unittest it: names);
  • done criteria that can be checked.

Slice 1 must ship alone. Slice 0 is landing fix/nrp-policy as-is (already reviewed). Number your slices from 1, with slice 1 stacked on it.

Live-cluster mutations (PVC deletion, CHI resize, CronJob changes applied via syndb ops k8s nautilus apply) are never part of a coder slice. List them as operator steps: exact command, precondition, rollback, and which slice must land first.

Every external fact (NRP rule, kubectl/K8s field path, ClickHouse setting) must cite its source: a URL, or a repo path with a line number. Anything unknown gets a capture step: who produces it, how, and how to validate it. Give one opinionated design per problem. List rejected alternatives in a table, one line each. Use tables over prose.

3. Angles worth weighing (not mandates)

  • A real compliance check instead of the placeholder metric: e.g. syndb ops k8s nrp audit (or a new preflight check) that computes from live objects the per-container request/limit ratio, unmounted PVCs, the limit=request rule for workloads with more than 100 Jobs, Jobs whose command is a sleep, Deployment ages, and the utilisation ratio from the metrics API. Emit them as the real syndb_nrp_* gauges so the existing alerts stop being fake. Note that the syndb-operator SA cannot read metrics.k8s.io, while the nautilus-oidc context can.
  • A render-time guard: a helm-unittest or Rust test over helm template output that fails on any container with limit/request > 1.20. It could also flag a missing request/limit, a sleeping Job command, or a missing ttlSecondsAfterFinished. That catches regressions before apply, not after.
  • Backups: the parity gate fails because provenance rows grow during the backup (the API logs query_executed provenance continuously; bench suites run queries all day). Options: snapshot-bounded counting (rows with event_time <= start), excluding the provenance table from parity, or quiescing. Each failed run also leaves ~30 GB in S3 “for forensics” every 30 minutes, which is storage hoarding. A prune/retention step for failed backup prefixes is likely needed.
  • ClickHouse CPU under-utilisation: request smaller with burst headroom is impossible under the 1.20x cap. Think about off-peak downsizing, merging the geo bench CHI into fewer pods, an NRP exception request, or using opportunistic. These are trade-offs for the operator to pick, not something to decide silently.
  • Retired-topology PVC cleanup as a reusable CLI (--dry-run default, lists owner/last-mount evidence), not ad-hoc kubectl.

4. Hard constraints

  • Repo rules: /data/nvme0/can/canix/projects/owned/SynDB/AGENTS.md (read it). Key points: ETL logic in Rust, no inline Python in Helm values; never pkgs.helm; Helm applies without --atomic; boot-test ClickHouse config changes before rollout (Helm tests miss CH startup errors).
  • Operator memory rules that bind this work:
    • syndb ops k8s nautilus apply needs --no-reset while other sessions’ Jobs run, and must run from a clean export of HEAD, not the shared working tree.
    • Before an API or ClickHouse rollout, check the syndb-flight-finalizer lease and that no Flight imports are active.
    • Prod ClickHouse reads need coordinator approval and run on us-west-1 with max_threads 4, never during imports.
    • PVC Delete reclaim drops tables.
    • No hosted CI for SynDB merges; qualification is local.
  • Agent rules: no scratch Python wrappers. Operator-facing scripts are Rust subcommands. Format via treefmt only. Nushell is the operator shell.
  • Heavy work: Cargo tests run locally in a memory-capped systemd-run unit. Heavy data or one-shot checks run on Nautilus only as optional Jobs.
  • The operator’s ponytail rule applies: smallest diff, no abstractions nobody asked for, reuse existing helpers (preflight CheckResult, ShellRunner, queryfabric_cli_toolbelt::k8s::kubectl_json, the Metrics builder in health.rs).

5. Repo map (do not re-survey; open only what a slice touches)

Checkout for reading: /data/nvme0/can/canix/projects/worktrees/SynDB/nrp-policy (branch fix/nrp-policy, base upstream trunk 77ec7cfa). Do not edit it.

PathSizeWhat
infrastructure/helm/nautilus/values.yamllargeproduction values; CH resources at ~L86-93 (2 CPU/8Gi, 1.20x), keeper L165-171
infrastructure/helm/nautilus/templates/syndb-observability-prometheusrule.yaml~120alerts incl. SynDBNRPRatioViolation L96 (abs(ratio-1.2)>0.01, wrong for limit=request workloads)
infrastructure/helm/nautilus/templates/clickhouse-backup-cronjob.yaml—backup CronJobs (west/central, full/incremental)
infrastructure/helm/nautilus/tests/*.yaml38 suiteshelm-unittest; e.g. burst_overrides_test.yaml asserts 1.20x
infrastructure/helm/syndb-etl/templates/etl-scratch-pvc.yaml60scratch PVCs, resource-policy: keep
crates/tools/cli/src/ops/health.rs1629prom export; syndb_nrp_request_limit_ratio is a hard-coded placeholder 1.2 (L636-656), quota forecast placeholder 999
crates/tools/cli/src/ops/backup/clickhouse.rs1110backup; validate_backup_provenance_parity L937-953
crates/tools/cli/src/k8s/preflight.rs, preflight/checks.rs467 / 953preflight check framework (CheckResult, pvc-fill-ratio etc.)
crates/tools/cli/src/k8s/etl_jobs/jobs.rs—run_cleanup + new orphan scratch PVC cleanup (slice 0)
crates/tools/cli/src/ops/gameday.rs, gameday/cluster.rs—GD-F; BACKUP_CRONJOB = "syndb-clickhouse-backup" is stale (live names: syndb-clickhouse-backup-{west,central}-{full,incremental})
crates/tools/bench/src/suite.rs1507bench suite Jobs (suite_resources)
documentation/runbooks/nrp-admission.md—NRP rules; new “Other NRP rules” section (slice 0)
documentation/runbooks/nrp-ratio-violation.md3TODO stub, linked from the alert
documentation/runbooks/nrp-quota-exhaustion-forecast.md—forecast runbook

6. Facts

  • NRP policy extract with URLs: nrp-policy-facts.md (scout, 2026-11-15). Corrections from operator-side evidence (documentation/runbooks/nrp-admission.md, an NRP admin reply dated 2026-05-21): utilisation is checked every 10 minutes, not over a multi-day window. Pods with at most 1 CPU and at most 2Gi are exempt. Completed Jobs do not count.
  • Live audit, 2026-11-15, namespace syndb:
    • Requests and limits: every container sets both, and all are within 1.20x except keeper 2458Mi/2Gi (fixed in slice 0). All Jobs have a TTL (max 7d). Secrets come in via secretKeyRef. Ingress is haproxy with no haproxy.org annotations. No nautilus.io tolerations. The only priority class used is opportunistic (GPU bench). CPU-only pods carry NotIn on pci-10de.present.
    • Utilisation (kubectl --context nautilus-oidc top): CH us-west-0/1 request 2 CPU/8Gi and use 77m and 124m CPU (4-6%), with 3.4Gi and 9.0Gi memory. These two pods violate the 20% CPU floor. Everything else is at or below 1 CPU / 2Gi, so exempt.
    • Unmounted PVCs, ~1.36 TiB in total:
      • 8×100Gi clickhouse-*-data-chi-syndb-cluster-syndb-{pacific,us-central,us-east,us-mountain}-{0,1}-0, created Jun 27-29, from the retired multi-region syndb-cluster topology
      • etl-scratch: microns-synapses-{dryrun,import,v1412,v1412dry} at 40/80/40/40Gi, plus prepare-mesh-connectomics-banc-seg-v888 at 250Gi (since 2026-07-25)
      • keeper-data-rbdbak-syndb-cluster-keeper-{0,1,2}, 10Gi each (2026-11-04)
      • syndb-ctrlbench-source, 40Gi (2026-11-09)
      • syndb-image-build-store, 300Gi: the warm Nix store, mounted only during image builds; arguably in active use
    • Deployments: syndb-api and syndb-ui are 141 days old. NRP purges Deployments after two weeks unless the namespace has an exception, so one probably exists. UNVERIFIED.
    • Backup CronJobs: west and central incremental have failed every run since at least 2026-11-14T21:05Z, each with backoffLimit 2, so 3 pods per run. Log: backup parity gate failed: provenance rows 11938 -> 11942 (allowed drift +1); refusing BACKUP_CREATED, leaving files for forensics. Each run uploads about 30.9 GB (18,804 objects) to s3-west.nrp-nautilus.io/syndb-backups/clickhouse/v3/west/.... The S3 usage of the leftover prefixes is UNVERIFIED and needs a capture step.
    • Pod quota and GPU: reached-quota pods 19/200. GPU quotas for a100/h100/h200/gh200 are 0.
    • Personal data: Postgres stores registered users’ emails. The NRP AUP prohibits PII, and whether login emails count is UNVERIFIED. This is an operator question for NRP support.

7. References

None (no UI work).