Contributing
Quick start for developers
Build and test
git clone --recurse-submodules https://github.com/memorycircuits/SynDB.git
cd SynDB
cargo build --workspace
cargo test --workspace
The workspace is large (~100 crates). Use cargo check -p <crate> for faster
iteration.
Prerequisites
- Rust 1.94+ for the workspace (edition 2024)
- A running stack for integration tests:
syndb stack up(see README)
Code conventions
| Concern | Standard |
|---|---|
| Formatting | cargo fmt |
| Linting | cargo clippy -- -D warnings |
| Rust edition | 2024 |
| Commit style | Conventional commits (feat:, fix:, refactor:, docs:, etc.) |
| Module docs | Every public module has a doc comment explaining its purpose |
PR workflow
- Open a PR against the
rapidbranch. - Ensure CI passes:
cargo fmt --check,cargo clippy,cargo test. - For Helm chart changes, include a
Chart Diff:section (see below). - For new features, add a CHANGELOG.md entry under
[Unreleased].
Local qualification
Run the local gates before opening a PR:
just qualify # stop at the first failing gate
just doctor # run every gate, then print a PASS/FAIL summary
Both run inside nix develop . and check, in order: the required dev-shell
tools, treefmt in a non-destructive check mode (it formats a throwaway copy,
never the checkout), gitleaks git --exit-code 1 -c .gitleaks.toml --log-opts=origin/trunk..HEAD, the focused k8s::nautilus CLI tests plus
scripts/test_verify_cli_clippy_baseline.py, the frozen CLI clippy baseline,
and cargo audit. Cargo jobs are capped at 4, and no gate rewrites the working
tree or contacts production.
just install-hooks opts this clone into a fast pre-push hook that runs only
gitleaks and a format check on the commits being pushed (it sets
core.hooksPath scripts/hooks; nothing is installed globally). The hook never
inspects untracked files or other worktrees. Remove it with
just uninstall-hooks.
just deps-check reports dependency advisories (cargo audit) and outdated
direct dependencies (cargo outdated --workspace --root-deps-only). This
repository has no dependency-update bot configured; run it weekly and before
every release, and record any deferred upgrade with the reason.
Helm chart changes
Pull requests that touch infrastructure/helm/** must describe the intended
rendered-manifest drift in a Chart Diff: section of the PR body. Use one of:
Chart Diff: nonewhen the PR only changes tests, CI, or documentation.Chart Diff: expectedfollowed by a short list of the workloads, resources, or values whose rendered output should change.
The GitHub helm-validation workflow renders the Nautilus umbrella chart and
validates it with Helm, kubeconform, and promtool. Reviewers should treat
unexplained rendered-manifest drift as a blocker.
Cluster operator notes
Preflight gates
Every irreversible op goes through syndb-preflight. New Op enum variants
must declare their gates via impl OpRequiresGates; new gates must register a
runbook_url that resolves to an entry in documentation/runbooks/INDEX.md.
Both are enforced at compile time (exhaustive match + build.rs), not at review
time.
Commissioning or cutting over a ClickHouseInstallation follows runbooks/chi-commissioning.md.
NRP quota allocations
Do not consume a new Nautilus storage allocation until the ticket/contact thread, approval decision, and tenant CPU/memory ceiling check are recorded here. Missing ceiling data is a deployment blocker, not a warning.
ClickHouse storage shape is immutable per deployed CHI
StatefulSet volumeClaimTemplates are immutable. Changing the
clickhouse.disks name set on a live cluster updates pod mounts but cannot
update existing claims, leaving a mount with no backing claim leading to the pod
being rejected (FailedCreate) on recreate and the clickhouse-operator aborting
the whole CHI reconcile. The syndb-clickhouse chart now refuses such in-place
changes. Migrate disks by rolling to new host identities, never an in-place
edit. See
runbooks/incident-2026-05-29-clickhouse-shard-failedcreate.md.
provenance_events is the only ClickHouse-only, non-ETL data class
Of the four ClickHouse table classes created by ensure_schema_in_database
(base data tables, materialized views (mv_*), precomputed graph tables
(precomputed_*), and provenance_events), only provenance_events is not
reconstructible:
- Base tables are ETL-ingested; MVs re-derive from base tables; precomputed
tables are rebuilt by
graph-precompute. All three are recoverable. provenance_eventsis written only at runtime by the graph actor, query worker, and Flight DoGet. No ETL writer, no batch re-derivation, no upstream source.
Per-replica drained cutover is safe for it. A full-shard loss (both replicas) permanently loses that shard’s operational provenance unless it is in the nightly S3 backup.
Smoke tokens for /v1/syql/exec
syndb auth service-login --email smoke@syndb.internal
curl -H "authorization: Bearer $(syndb auth show-token)" \
https://api.syndb.xyz/v1/syql/exec ...
P09 cutover lessons (2026-06-01)
- Use the explicit
syndb_distributedremote-server alias forcluster,clusterAllReplicas, andON CLUSTER; never use the ambiguoussyndbidentity shared by the CHI topology and Replicated database. An all-replica probe still returns one result per physical replica, so query_localtables and aggregate deliberately for logical counts. - Same-cluster restore drills must validate
_localdrill tables. - Runtime provenance writes must target Distributed over ReplicatedMergeTree, not a plain MergeTree behind a load-balanced Service.
- Burst-worker Job pods do not inherit
syndb-api-configenvFromsettings. - Never uninstall a whole Helm release as a proxy for removing one old resource.
Submission-grade benchmark capture
A submission-grade benchmark uses the canonical 22-column row schema with
20+ reps per cell: 4 queries x 2 cache_states x 20+ reps = 160+ rows. Drop or
annotate the cold-start outlier (the first cold rep of any isolated cell is a
Kubernetes pod cold-start, not steady-state latency).
StatefulSet VCT storageClass is immutable
Override to the live shape in the chart/values before helm upgrade if the
rendered chart drifts from the deployed VCT.
nrp-admission correction
The ClickHouse-pod S3 egress break was caused by a leftover single-host
[ "195.113.147.178/32" ] CIDR pin in s3EndpointCidrs. Commit e237c627
fixed this by defaulting to []. Not an NRP cluster-side defect.
CronJob payload policy
Inline shell business logic in CronJob templates is forbidden. A CronJob
container must invoke a reviewed Rust binary directly. Suspending a chronically
failing CronJob is allowed only with the three metadata annotations:
suspended-reason, suspended-at, and suspended-by.
GPU acceleration wisdom
Backend selection model
The mesh-to-connectomics pipeline uses a three-way backend selection:
| Backend | Env var | Feature flag | Status |
|---|---|---|---|
| CPU | unset or any | none | Always works |
| CUDA | GPU_BACKEND=cust | gpu-cust | Production on Nautilus |
| WGpu | GPU_BACKEND=wgpu | future | Not implemented |
The selection is in crates/tools/etl/src/gpu/mod.rs:select_backend(). CPU is
always the unconditional fallback. When adding a new backend:
- Implement the
GpuBackendtrait (crates/tools/etl/src/gpu/mod.rs) - Add the implementation file behind a
#[cfg(feature = "...")]gate - Register it in
select_backend()with the matching env var value - Add the feature flag to
crates/tools/etl/Cargo.toml - Document the env var in the GPU deployment guide
CUDA dependencies are optional
The cust runtime and meshcon-gpu-kernels NVCC/PTX crate are optional
dependencies in crates/tools/etl/Cargo.toml. Cargo does not build them unless
--features gpu-cust (or the umbrella etl-gpu feature) is used. Standard
cargo build, cargo test, and cargo clippy remain CPU-only.
Helm template invariants
The etl-prepare-job.yaml template conditionally injects GPU resources. When
.gpu is present on a job config, the template must emit all four GPU-related
sections as a single coherent block. The known failure mode is a duplicate
env: key when .gpu is true but .secretEnv is empty and the
clickhouse tiered-storage env helper is empty — this was fixed in commit
58879af by injecting GPU_BACKEND into whichever env: block is active.
Always verify with:
helm template infrastructure/helm/syndb-etl/ \
--values infrastructure/helm/syndb-etl/values.yaml \
--set 'prepareJobs[0].gpu.enabled=true' \
| grep -A3 "GPU_BACKEND\|nvidia.com/gpu\|dshm"
Expected output: one GPU_BACKEND env var, one nvidia.com/gpu resource
block, one dshm volume mount, and no duplicate YAML keys.
EmptyDir and GPU shared memory
GPU kernel launches use /dev/shm for shared memory allocations. The Helm
template mounts a dedicated emptyDir with medium: Memory at /dev/shm.
Default size is 8Gi — tune via gpu.shmSize in the job config. Insufficient
shared memory causes cryptic CUDA errors or OOMKilled pods.
Feature flag hygiene for GPU
gpu— umbrella feature that enables the GPU module directory (safe to gate behind, no external deps)gpu-cust— enables the CUDA backend (pullscustand the NVCC/PTX kernel crate; only resolved at build time when the feature is active)- WGpu will get its own
gpu-wgpufeature in the future
Keep the runtime value and Helm template aligned: gpu-cust builds the
backend and GPU_BACKEND=cust selects it at runtime. Any initialization error
falls back to the CPU backend.