Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Contributing

Quick start for developers

Build and test

git clone --recurse-submodules https://github.com/memorycircuits/SynDB.git
cd SynDB
cargo build --workspace
cargo test --workspace

The workspace is large (~100 crates). Use cargo check -p <crate> for faster iteration.

Prerequisites

  • Rust 1.94+ for the workspace (edition 2024)
  • A running stack for integration tests: syndb stack up (see README)

Code conventions

ConcernStandard
Formattingcargo fmt
Lintingcargo clippy -- -D warnings
Rust edition2024
Commit styleConventional commits (feat:, fix:, refactor:, docs:, etc.)
Module docsEvery public module has a doc comment explaining its purpose

PR workflow

  1. Open a PR against the rapid branch.
  2. Ensure CI passes: cargo fmt --check, cargo clippy, cargo test.
  3. For Helm chart changes, include a Chart Diff: section (see below).
  4. For new features, add a CHANGELOG.md entry under [Unreleased].

Local qualification

Run the local gates before opening a PR:

just qualify   # stop at the first failing gate
just doctor    # run every gate, then print a PASS/FAIL summary

Both run inside nix develop . and check, in order: the required dev-shell tools, treefmt in a non-destructive check mode (it formats a throwaway copy, never the checkout), gitleaks git --exit-code 1 -c .gitleaks.toml --log-opts=origin/trunk..HEAD, the focused k8s::nautilus CLI tests plus scripts/test_verify_cli_clippy_baseline.py, the frozen CLI clippy baseline, and cargo audit. Cargo jobs are capped at 4, and no gate rewrites the working tree or contacts production.

just install-hooks opts this clone into a fast pre-push hook that runs only gitleaks and a format check on the commits being pushed (it sets core.hooksPath scripts/hooks; nothing is installed globally). The hook never inspects untracked files or other worktrees. Remove it with just uninstall-hooks.

just deps-check reports dependency advisories (cargo audit) and outdated direct dependencies (cargo outdated --workspace --root-deps-only). This repository has no dependency-update bot configured; run it weekly and before every release, and record any deferred upgrade with the reason.

Helm chart changes

Pull requests that touch infrastructure/helm/** must describe the intended rendered-manifest drift in a Chart Diff: section of the PR body. Use one of:

  • Chart Diff: none when the PR only changes tests, CI, or documentation.
  • Chart Diff: expected followed by a short list of the workloads, resources, or values whose rendered output should change.

The GitHub helm-validation workflow renders the Nautilus umbrella chart and validates it with Helm, kubeconform, and promtool. Reviewers should treat unexplained rendered-manifest drift as a blocker.

Cluster operator notes

Preflight gates

Every irreversible op goes through syndb-preflight. New Op enum variants must declare their gates via impl OpRequiresGates; new gates must register a runbook_url that resolves to an entry in documentation/runbooks/INDEX.md. Both are enforced at compile time (exhaustive match + build.rs), not at review time.

Commissioning or cutting over a ClickHouseInstallation follows runbooks/chi-commissioning.md.

NRP quota allocations

Do not consume a new Nautilus storage allocation until the ticket/contact thread, approval decision, and tenant CPU/memory ceiling check are recorded here. Missing ceiling data is a deployment blocker, not a warning.

ClickHouse storage shape is immutable per deployed CHI

StatefulSet volumeClaimTemplates are immutable. Changing the clickhouse.disks name set on a live cluster updates pod mounts but cannot update existing claims, leaving a mount with no backing claim leading to the pod being rejected (FailedCreate) on recreate and the clickhouse-operator aborting the whole CHI reconcile. The syndb-clickhouse chart now refuses such in-place changes. Migrate disks by rolling to new host identities, never an in-place edit. See runbooks/incident-2026-05-29-clickhouse-shard-failedcreate.md.

provenance_events is the only ClickHouse-only, non-ETL data class

Of the four ClickHouse table classes created by ensure_schema_in_database (base data tables, materialized views (mv_*), precomputed graph tables (precomputed_*), and provenance_events), only provenance_events is not reconstructible:

  • Base tables are ETL-ingested; MVs re-derive from base tables; precomputed tables are rebuilt by graph-precompute. All three are recoverable.
  • provenance_events is written only at runtime by the graph actor, query worker, and Flight DoGet. No ETL writer, no batch re-derivation, no upstream source.

Per-replica drained cutover is safe for it. A full-shard loss (both replicas) permanently loses that shard’s operational provenance unless it is in the nightly S3 backup.

Smoke tokens for /v1/syql/exec

syndb auth service-login --email smoke@syndb.internal
curl -H "authorization: Bearer $(syndb auth show-token)" \
  https://api.syndb.xyz/v1/syql/exec ...

P09 cutover lessons (2026-06-01)

  1. Use the explicit syndb_distributed remote-server alias for cluster, clusterAllReplicas, and ON CLUSTER; never use the ambiguous syndb identity shared by the CHI topology and Replicated database. An all-replica probe still returns one result per physical replica, so query _local tables and aggregate deliberately for logical counts.
  2. Same-cluster restore drills must validate _local drill tables.
  3. Runtime provenance writes must target Distributed over ReplicatedMergeTree, not a plain MergeTree behind a load-balanced Service.
  4. Burst-worker Job pods do not inherit syndb-api-config envFrom settings.
  5. Never uninstall a whole Helm release as a proxy for removing one old resource.

Submission-grade benchmark capture

A submission-grade benchmark uses the canonical 22-column row schema with 20+ reps per cell: 4 queries x 2 cache_states x 20+ reps = 160+ rows. Drop or annotate the cold-start outlier (the first cold rep of any isolated cell is a Kubernetes pod cold-start, not steady-state latency).

StatefulSet VCT storageClass is immutable

Override to the live shape in the chart/values before helm upgrade if the rendered chart drifts from the deployed VCT.

nrp-admission correction

The ClickHouse-pod S3 egress break was caused by a leftover single-host [ "195.113.147.178/32" ] CIDR pin in s3EndpointCidrs. Commit e237c627 fixed this by defaulting to []. Not an NRP cluster-side defect.

CronJob payload policy

Inline shell business logic in CronJob templates is forbidden. A CronJob container must invoke a reviewed Rust binary directly. Suspending a chronically failing CronJob is allowed only with the three metadata annotations: suspended-reason, suspended-at, and suspended-by.

GPU acceleration wisdom

Backend selection model

The mesh-to-connectomics pipeline uses a three-way backend selection:

BackendEnv varFeature flagStatus
CPUunset or anynoneAlways works
CUDAGPU_BACKEND=custgpu-custProduction on Nautilus
WGpuGPU_BACKEND=wgpufutureNot implemented

The selection is in crates/tools/etl/src/gpu/mod.rs:select_backend(). CPU is always the unconditional fallback. When adding a new backend:

  1. Implement the GpuBackend trait (crates/tools/etl/src/gpu/mod.rs)
  2. Add the implementation file behind a #[cfg(feature = "...")] gate
  3. Register it in select_backend() with the matching env var value
  4. Add the feature flag to crates/tools/etl/Cargo.toml
  5. Document the env var in the GPU deployment guide

CUDA dependencies are optional

The cust runtime and meshcon-gpu-kernels NVCC/PTX crate are optional dependencies in crates/tools/etl/Cargo.toml. Cargo does not build them unless --features gpu-cust (or the umbrella etl-gpu feature) is used. Standard cargo build, cargo test, and cargo clippy remain CPU-only.

Helm template invariants

The etl-prepare-job.yaml template conditionally injects GPU resources. When .gpu is present on a job config, the template must emit all four GPU-related sections as a single coherent block. The known failure mode is a duplicate env: key when .gpu is true but .secretEnv is empty and the clickhouse tiered-storage env helper is empty — this was fixed in commit 58879af by injecting GPU_BACKEND into whichever env: block is active.

Always verify with:

helm template infrastructure/helm/syndb-etl/ \
  --values infrastructure/helm/syndb-etl/values.yaml \
  --set 'prepareJobs[0].gpu.enabled=true' \
  | grep -A3 "GPU_BACKEND\|nvidia.com/gpu\|dshm"

Expected output: one GPU_BACKEND env var, one nvidia.com/gpu resource block, one dshm volume mount, and no duplicate YAML keys.

EmptyDir and GPU shared memory

GPU kernel launches use /dev/shm for shared memory allocations. The Helm template mounts a dedicated emptyDir with medium: Memory at /dev/shm. Default size is 8Gi — tune via gpu.shmSize in the job config. Insufficient shared memory causes cryptic CUDA errors or OOMKilled pods.

Feature flag hygiene for GPU

  • gpu — umbrella feature that enables the GPU module directory (safe to gate behind, no external deps)
  • gpu-cust — enables the CUDA backend (pulls cust and the NVCC/PTX kernel crate; only resolved at build time when the feature is active)
  • WGpu will get its own gpu-wgpu feature in the future

Keep the runtime value and Helm template aligned: gpu-cust builds the backend and GPU_BACKEND=cust selects it at runtime. Any initialization error falls back to the CPU backend.