Plan: SynDB Home Manager module with profiles, and the CLI crate split
Status (2026-11-18): slices 1a and 1b landed at trunk eda5f027; slices 1c, 2, 3 and 5 at 92f28193..861b8ea9. Slice 4 (the canix pin and the developer profile for can) runs in canix; the three CLI packages are prebuilt in canix-fleet.
Baselines: SynDB origin/trunk 68876577, canix origin/trunk a1ee91124. All path:line citations are at those revisions.
Revision 2 (2026-11-17): sections 2 to 6 are rewritten for the operator amendments in section 8 (separate core crate, no ClickHouse client for consumers, syndb mcp api for every profile).
Scope: CLI, Cargo and Nix. No command, route or wire format changes. Slice 1a edits two crates the API server compiles (shared, syndb-clickhouse) without changing its dependency graph (section 2.5); its default is to merge after the 0.11.1 images are built (freeze Tue 2026-11-17 22:00Z).
1. What I found
Counts are unique crate names from cargo tree -e normal,build --prefix none --offline, on a clean export of 68876577 in /data/scratch/tmp/opencode/syndb/hm-profiles-deptree.RNemeax1 (metadata only, nothing compiled). Revision 1 measured the dirty main checkout; these replace its numbers.
| Finding | Evidence |
|---|---|
Features gate little of the floor’s weight today: cli resolves 509 crates with --no-default-features, 678 with prod, 742 with full | measured |
cli-common alone is 395 crates: it depends unconditionally on polars, syndb-clickhouse[polars], arrow-flight, tonic, parquet | crates/tools/cli-common/Cargo.toml:9-27 |
polars in cli-common serves one function with one caller (ETL import); syndb-clickhouse serves that function, ClickHouseArgs (used only by etl/, migrate.rs, ops/) and the Flight payload types | cli-common/src/flight.rs:31-35,334-368, src/clickhouse.rs:25-37; crates/tools/cli/src/etl/import/flight/client.rs:51 |
The Flight payload types and SyndbTable are plain serde types with no ClickHouse code: 5 small files plus table.rs. Nothing outside table.rs implements anything on SyndbTable | crates/core/syndb-clickhouse/src/flight_descriptor.rs:6-18, flight_descriptor/*.rs, table.rs:1-2,37,222,235 |
shared already has every dependency those types need except serde_repr, and is in every CLI’s closure | crates/core/shared/Cargo.toml:15-27 |
user/, api.rs and query/ have no crate:: imports, and nothing but main.rs refers to them | crates/tools/cli/src/main.rs:122-126,221-223; grep at trunk |
kube, k8s-openapi and queryfabric are always-on dependencies used only by gated modules (k8s/, ops/, migrate.rs) | crates/tools/cli/Cargo.toml:16,49-50; main.rs:30-38 |
syndb-lakehouse (328 crates) and syndb-preflight are always-on; only article/ and op_registry.rs use them | Cargo.toml:73-74 |
syndb data mixes three audiences in one enum behind one feature: download (consumer), new/prepare/validate/upload (provider), search reconcile / reconcile-uploads / snapshot-corpus-inventory (operator; PostgreSQL, Meilisearch, ClickHouse) | dataset/args.rs:11-142; dataset/reconcile.rs, dataset/corpus_inventory.rs |
The operator data subcommands are deployed command paths, so they cannot move | infrastructure/helm/nautilus/templates/meilisearch-reconcile-cronjob.yaml:39; infrastructure/helm/syndb-etl/tests/syndb_job_test.yaml:16 |
Prototype of this plan’s manifests: syndb-cli-core resolves 290 crates, 299 with the MCP dependencies, and the consumer floor about 300 (was 509), with no polars*, clickhouse, syndb-clickhouse, kube, k8s-openapi or queryfabric | measured on the edited export |
| Same prototype: the API server’s resolved crates (607 lines with versions) and features (2,134 lines) are identical before and after | cargo tree -p api --no-default-features --features prod, -e normal,build and -e features, sorted and diffed |
The red baseline may already be fixed or still red: the check cargo check -p cli --no-default-features --all-targets was added on 2026-11-14, the day the failure was seen | flake.nix:207, commit 416ef7da. Slice 1c starts by running it |
The five public-API MCP tools sit inside the operator ops server and shell out to the same syndb binary | crates/tools/cli/src/mcp/ops.rs:139-149,204-217,259-272,410-476 |
In canix, “dev host” for can means a home profile of development or higher: atlas, murph, nomad, starlord. bar and thething are base | canix root/user/hosts.nix:10-58; ranked selection flake/_shared/hm.nix:98-108; home/profiles/users/can/default.nix:9-15 |
A consumer still links arrow and parquet (in shared) and arrow-flight and tonic (dataset download).
2. Design
2.1 Tiers
One binary crate (cli, binary syndb) and one new library crate (syndb-cli-core). The consumer CLI is the binary with no Cargo features; each higher tier is a feature of cli. Every feature set that a Nix output or deployed image uses today keeps its exact command tree.
| Tier | Code | Cargo selection | Nix package | Commands |
|---|---|---|---|---|
| consumer | syndb-cli-core plus the cli shell | -p cli --no-default-features | syndb-cli-consumer (new) | auth, api (includes search), query, data download, mcp api, completions |
| provider | + cli features provider, mcp-review | -p cli --no-default-features --features provider,mcp-review | syndb-cli-provider (new) | + data new/prepare/validate/upload/mesh-upload/swb-upload/delete/cache-tags/gen-test-data, review, mcp review |
| developer | + everything | -p cli --features full | syndb-cli-tools (existing) | everything else, all five MCP modes |
| (server images) | unchanged | prod, etl-gpu,article-tools, default + bench | syndb-cli, syndb-cli-etl-image, syndb-cli-bench-image | unchanged |
2.2 What lives where
| Crate | Change | Result |
|---|---|---|
shared | gains pub mod table (SyndbTable) and pub mod flight_descriptor, moved verbatim with their tests from syndb-clickhouse; Cargo.toml gains serde_repr = "0.1" after :24 | the Flight payload types need no ClickHouse crate |
syndb-clickhouse | lib.rs:19,25 become pub use shared::{flight_descriptor, table};; :37 (pub use table::SyndbTable) stays; Cargo.toml:24 (serde_repr) goes | every existing path (syndb_clickhouse::table::SyndbTable, syndb_clickhouse::flight_descriptor::*) still resolves, so none of the roughly 150 files that import them changes. No feature is added |
cli-common | src/flight.rs:31-35 import from shared; syndb-clickhouse and polars become optional behind one new feature clickhouse = ["dep:syndb-clickhouse", "dep:polars"], which gates pub mod clickhouse (src/lib.rs:6) and upload_dataframe_with_artifact (src/flight.rs:334-368) | 395 crates falls to 274 (measured). syndb-bench, syndb-ci and populate-cache use neither gated item (they call config, http, urls, logging, flight::connect_with_timeout, flight::syql_query), so their manifests do not change |
syndb-cli-core (new library, crates/tools/cli-core) | holds user/, api.rs, query/ (moved with git mv, no edits), new download.rs, and later mcp_api.rs. No Cargo features. Dependencies: shared, cli-common, syndb-api-client, clap, tokio (process), serde, serde_json, tracing, reqwest, miette, uuid, chrono, rand, base64, rpassword, webbrowser; slice 5 adds rmcp, queryfabric-cmd-runner. Dev: wiremock, tempfile, rustls (api.rs:136) | Cargo enforces the floor: a consumer command cannot import a crate that is not in this manifest |
cli (binary syndb) | depends on syndb-cli-core always; owns main.rs, every command enum and all provider and developer code behind features (section 2.4) | -p cli --no-default-features is the consumer binary |
2.3 One clap tree across two crates
Rule: the binary owns every enum, so every command name, its order and its help line stay in cli; the core crate owns leaf clap::Args structs and run functions. No #[command(flatten)]. This is the shape main.rs already has (Auth(user::UserArgs)), and clap’s derive is trait-based, so a variant can hold an Args type from another crate.
| Command | In cli | In syndb-cli-core |
|---|---|---|
auth, api, query | main.rs:15,40,47 (mod api; mod query; mod user;) become use syndb_cli_core::{api, query, user};. The variants (:122-126) and dispatch (:221-223) do not change by a character | user::{UserArgs, run} (user/mod.rs:10-11,123), api::{ApiArgs, run} (api.rs:24,60), query::{QueryArgs, run} (query/mod.rs:12,171) are already pub |
data | DatasetArgs and DatasetCommand stay in dataset/args.rs and compile in every build. Download { … } (:72-83) becomes Download(syndb_cli_core::download::DownloadArgs) with the same doc comment; the arm at run_impl.rs:48-52 calls syndb_cli_core::download::run(args, flight_url, server_url). Provider variants get #[cfg(feature = "provider")], the three operator variants #[cfg(feature = "dataset")] | download::DownloadArgs carries the three fields of args.rs:74-82 with the same #[arg] attributes and doc comments; download::run is the body of flight_io.rs:104-135, importing shared::table::SyndbTable |
mcp | McpCommand stays in mcp.rs:9-29 and compiles in every build from slice 5. Api (unit variant) dispatches to the core; Review is #[cfg(feature = "mcp-review")]; Figures, Check, Ops are #[cfg(feature = "mcp")]. The early dispatch before logging setup (main.rs:213-216,266-267) loses its cfg | mcp_api::serve(server_url) and the shared pieces ops reuses (section 2.4, slice 5) |
completions, global flags | unchanged (main.rs:74-108,189-193,269-272): Cli and Cli::command() stay in the binary | none |
#![allow(unused)]
fn main() {
// crates/tools/cli-core/src/download.rs
#[derive(clap::Args)]
pub struct DownloadArgs {
/// Dataset UUID(s) to download (can specify multiple)
#[arg(short, long, required = true)]
pub dataset_id: Vec<Uuid>,
/// SynDB table index to download
#[arg(short, long)]
pub table: i32,
/// Output directory for downloaded parquet files
#[arg(short, long)]
pub output_dir: PathBuf,
}
pub async fn run(args: DownloadArgs, flight_url: &str, server_url: &str) -> Result<()>;
}
2.4 Cargo features of cli (crates/tools/cli/Cargo.toml:101-150)
| Feature | After | Why |
|---|---|---|
| none (floor) | compiles main.rs, dataset/{mod,args,run_impl}.rs with only Download, and from slice 5 mcp.rs with only Api | the consumer binary |
provider (new) | dep:data, dep:polars, dep:tar, dep:flate2; gates the provider data variants, dataset/{create,flight_io,helpers}.rs, mod cache (main.rs:17-18) and mod review (:41) | the client half of today’s dataset |
dataset | provider, repo-tools, dep:database, dep:migration, dep:sea-orm, dep:sea-orm-migration, dep:syndb-search, dep:secrecy, dep:syndb-clickhouse; gates the three operator data variants, dataset/{reconcile,corpus_inventory,tests}.rs | same name and command set as today, so prod, etl, graph-precompute and --features dataset in docs keep working |
repo-tools (new) | gates article, evidence_cli, test_workflows, process, op_registry (ungated today, main.rs:16,25,36,39,46); owns dep:syndb-lakehouse, dep:syndb-preflight, dep:syndb-clickhouse, shared/manuscript-archive (Cargo.toml:15) and each small crate only these modules use | dataset implies it, so every current feature set keeps these commands |
etl | adds dep:kube, dep:k8s-openapi, dep:queryfabric, cli-common/clickhouse | their only users are k8s/, ops/, migrate.rs, ETL import and ClickHouseArgs |
mcp-review (new) | provider, dep:rmcp | review MCP without the figures, check and ops dependencies |
mcp | mcp-review, dep:syndb-check, dep:syndb-mcp, dep:queryfabric-cmd-runner | the four existing modes; api needs no feature |
prod, etl-full, etl-gpu, bench, ci, article-tools, dev, full, default | unchanged | no migration for images or docs |
Rule for cli’s unconditional dependency list: a crate stays unconditional only if main.rs uses it or it is already in syndb-cli-core’s closure (then it costs the floor nothing). Every other dependency becomes optional on the lowest feature whose code uses it. The executor proves each move by compiling the floor and the provider set; the deny-list test (slice 1c) keeps it true.
2.5 The API server’s build
| Claim | Check |
|---|---|
| No API manifest or source changes | git diff --stat origin/trunk -- crates/services is empty; apiCargoExtraArgs (nix/packages.nix:232) is untouched |
| The API compiles the same crates, versions and features | cargo tree -p api --no-default-features --features prod -e features --prefix none | sort -u, before and after slice 1a, has an empty diff (measured on the prototype). serde_repr only moves from syndb-clickhouse’s edge list to shared’s |
| Same wire format and OpenAPI name | the types and their tests move verbatim (table/tests.rs, flight_descriptor/tests/int_core.rs, int_edge_cases.rs); ToSchema::name stays "SyndbTable" (table.rs:235-239) |
| Not claimed: a byte-identical binary or store path | SyndbTable’s symbols move from syndb_clickhouse to shared, and every Nix dependency layer gets a new hash, because the dummy source holds every manifest and Cargo.lock (nix/rust.nix:61-75). Feature-gating syndb-clickhouse would change both as well |
2.6 Home Manager module (nix/home-manager.nix, exported as homeManagerModules.syndb)
| Option | Type | Default | Effect |
|---|---|---|---|
programs.syndb.enable | bool | false | installs package |
.profile | enum consumer, provider, developer | "consumer" | selects the default package and MCP modes |
.package | package | self.packages.<system>.{syndb-cli-consumer, syndb-cli-provider, syndb-cli-tools} by profile | overridable, for example syndb-tools to add the Python export provider |
.serverUrl | null or str | null | sets SYNDB_SERVER_URL in home.sessionVariables and in each MCP server env; null leaves the CLI default from shared::endpoints (cli-common/src/urls.rs:9) |
.flightUrl | null or str | null | same for SYNDB_FLIGHT_URL; a non-production serverUrl needs it, because the Flight default is production (main.rs:84-91) |
.mcp.reviewServerUrl | null or str | null | sets SYNDB_REVIEW_SERVER_URL on the review entry only; never derived from serverUrl (documentation/src/reference/mcp.md:93-101) |
.mcp.servers | read-only attrs of {command, args, env} | by profile | data only. consumer syndb-api; provider syndb-api, syndb-review; developer syndb-api, syndb-figures, syndb-check, syndb-ops, syndb-review. command is lib.getExe cfg.package, args is ["mcp" "<mode>"] |
The module writes no harness configuration. A harness module consumes mcp.servers; the shape matches canix fleetix.mcp.servers (canix home/user/can/personal_pc/mcp.nix:13-38). The module file contains no origin, so the public-URL lint has nothing to flag.
Completions come from the packages, not the module: installShellCompletion for bash, zsh and fish in postInstall of the three native profile packages.
2.7 Rejected alternatives
| Alternative | Why not |
|---|---|
| Feature tiers in one crate (revision 1) | overruled by the operator (section 8) |
#[command(flatten)] of enums owned by the core crate | moves command names and help text into the library and lists the tree in two places; tuple variants are the shape main.rs already has |
The core crate ships its own syndb binary | two workspace binaries named syndb collide in target/, and the surface tests would split across crates |
| A separate provider crate | provider is dataset/ and review in one binary; a feature plus the deny-list test gives the same boundary with no second move |
A new syndb-flight-types crate for the payload types | adds a node to the API’s graph, a workspace member and two more fileset lines; shared is already in every consumer’s closure and needs only serde_repr |
A client feature on syndb-clickhouse | 17 dependents would need features = ["client"] (or a default feature with default-features = false on the workspace dependency), 11 of its 13 modules would carry a cfg, and feature unification switches it back on in any build that includes another user |
Splitting cli-common in two | one feature gates its two heavy items |
A mcp-api Cargo feature | every profile gets mcp api, so there is nothing to switch |
Physical consumer/, provider/, developer/ directories inside cli | renames several hundred files for no compile effect |
Moving operator data commands under ops | they are deployed Helm command paths |
credentialsFile option | canix already delivers the file (canix home/user/can/repositories/syndb_owner_credentials.nix:7-14); for other users it is one xdg.configFile line, and a path-typed option invites copying a secret into the store |
mcp.enable option | mcp.servers is read-only data with no side effect |
| Module writes Claude Code or OpenCode config | couples the module to harnesses; the attrset is enough |
| Home Manager flake input for the check | a new input and lock nodes that canix would inherit; lib.evalModules with two stub options tests the same logic |
| Nushell completions | clap_complete has no Nushell generator; it needs a new dependency. Add when wanted |
Switching the Windows and macOS cross builds to provider | changes release artifacts; follow-up once provider is proven |
syndb-api in the checkout’s .mcp.json and launcher | the checkout’s syndb-ops server already serves the same five tools (scripts/syndb-mcp.sh:19) |
3. Profile matrix
| consumer | provider | developer | |
|---|---|---|---|
| Package | syndb-cli-consumer | syndb-cli-provider | syndb-cli-tools |
| Cargo | -p cli --no-default-features | -p cli --no-default-features --features provider,mcp-review | -p cli --features full |
| Crates | syndb-cli-core and the cli shell | + data, provider and review code in cli | everything |
| Commands | auth, api, query, data download, mcp api, completions | consumer + provider data subcommands, review, mcp review | provider + ops, article, evidence, test, dev, stack, etl, migrate, federation, graph-precompute, bench, ci, operator data subcommands, mcp figures/check/ops |
| Must not link | clickhouse, syndb-clickhouse, queryfabric-adapter-clickhouse, polars*, queryfabric, kube, k8s-openapi, sea-orm, database, migration, syndb-lakehouse, syndb-preflight, data, etl, syndb-search, syndb-federation, opendal, connectome-graph | kube, k8s-openapi, sea-orm, syndb-lakehouse, etl, syndb-search, syndb-federation, opendal, connectome-graph | none |
| Still links | arrow, parquet, arrow-flight, tonic, rmcp (about 300 crates) | + polars and, through data (crates/tools/data/Cargo.toml:11), the ClickHouse client | everything |
| Config files | none | none | none |
| Environment | SYNDB_SERVER_URL, SYNDB_FLIGHT_URL when set | same | same |
| MCP servers | syndb-api | syndb-api, syndb-review | syndb-api, syndb-figures, syndb-check, syndb-ops, syndb-review |
| Secrets | none; syndb auth login writes ~/.config/syndb/auth.json | same; optional ~/.config/syndb/credentials.json supplied by the user’s secret manager (cli-common/src/config.rs:38-66) | same; in canix the agenix file stays where it is |
| Python | none | none | none (syndb etl export needs syndb-export-provider: dev shell, or package = syndb-tools) |
4. Migration table
| Consumer today | Where | After |
|---|---|---|
| Workspace members and dependency table | Cargo.toml:36,59-60,365 | add crates/tools/cli-core to default-members and members, and syndb-cli-core = { path = "crates/tools/cli-core" } (slice 1b) |
Cargo.lock | repository root | 1a: serde_repr moves between two edge lists. 1b: one new syndb-cli-core package block. No third-party entry changes in any slice |
| Rust source filter for every package and cross build | nix/rust.nix:141-143 (mkRustSrc) | add (craneLib.fileset.commonCargoSources (root + "/crates/tools/cli-core")) after :141 (slice 1b). The moved types need no line: shared and syndb-clickhouse are already listed (:117,121) |
| ETL image source list | nix/rust.nix:217-245 (etlCrates in mkCliEtlSrc) | add "crates/tools/cli-core" after :231 (slice 1b). Required: :267 copies sources only for listed crates and :276 rewrites the workspace members to this list, and cli depends on the new crate in every feature set |
| Dummy and metadata sources | nix/rust.nix:61-75,79-86 | no edit: both take every manifest under the root |
Every *-cargo-artifacts dependency layer | nix/packages.nix:103-124,224-238,489-502 | new hashes after 1a, 1b and 1c (manifests and lock are in the dummy source); each rebuilds once, then run scripts/publish-cargo-artifacts.sh |
syndb-api, syndb-node, API images | nix/packages.nix:232-259 | cargo arguments and resolved graph unchanged (section 2.5); dependency layer rebuilt once |
syndb-clickhouse dependents (17 manifests) | for example crates/services/api/Cargo.toml:41, crates/services/flight/Cargo.toml:13 | unchanged: the re-export keeps every path |
cli-common dependents syndb-bench, syndb-ci, populate-cache | crates/tools/bench/Cargo.toml:56, crates/tools/ci/Cargo.toml:21, crates/tools/populate-cache/Cargo.toml:18 | unchanged; cli’s etl feature turns cli-common/clickhouse on |
syndb-cli (prod) | nix/packages.nix:235-236,261-268 | unchanged |
oci-syndb-cli image, K8s Jobs and CronJobs | nix/packages.nix:608-628; Helm templates | unchanged; prod_surface_is_unchanged and data_download_args_are_unchanged guard the command tree |
syndb-cli-etl-image (etl-gpu,article-tools) | nix/packages.nix:71-74,313-334 | same features and commands; source list grows by one crate (row above) |
syndb-cli-bench-image (default + bench) | nix/packages.nix:237-238,337-343 | unchanged |
syndb-cli-tools, syndb-tools | nix/packages.nix:113-124,272-278,302-310 | same features; gains mcp api and shell completions in postInstall |
syndb-export-provider | nix/packages.nix:284-295 | unchanged; in no profile |
Cross builds syndb-cli-{windows,darwin-*} (prod) | nix/packages.nix:188-204 | unchanged; they read mkRustSrc, so they pick the new crate up |
*-cargo-artifacts exports and publisher | nix/packages.nix:489-502; scripts/publish-cargo-artifacts.sh:18-20 | add syndb-cli-consumer-cargo-artifacts, syndb-cli-provider-cargo-artifacts to both |
cli-features check | flake.nix:154-209 | add cargo test -p syndb-cli-core (1b), then the provider check and the floor tests (1c) |
ci clippy step | crates/tools/ci/src/testing/queryfabric.rs:158-170 | add cargo clippy -p syndb-cli-core --all-targets (1b) and cargo clippy -p cli --no-default-features --all-targets (1c) |
CLI clippy baseline (-p cli --features etl --no-deps) | scripts/cli_clippy_baseline.json:2-16 | no longer sees the moved files; the new syndb-cli-core clippy step covers them. Run scripts/verify_cli_clippy_baseline.py and --update only if it reports a change |
Dev-shell syndb wrapper | nix/dev-shells.nix:101-121 | auth, api, query on the floor; review moves to --no-default-features --features provider; test leaves the floor list (falls to full) |
ops MCP server | crates/tools/cli/src/mcp/ops.rs | same 12 tools; the five api_* tools take their types, argv and dispatch from the core crate (slice 5) |
MCP launcher, .mcp.json, dev-shell syndb-mcp shim | scripts/syndb-mcp.sh:19,35; nix/dev-shells.nix:126-132 | unchanged |
| MCP reference | documentation/src/reference/mcp.md:11-12 | add the syndb mcp api row (slice 5) |
| Hosted workflows | .github/workflows/publish.yaml:129-130, nix-builds.yaml:29 | unchanged (Actions are not used before the repository is public) |
| Install docs | README.md:134,158; documentation/src/get-started/install.md:46-48, researchers.md:119-130, quick-start.md:18; crates/tools/cli/README.md:80-83 | slice 3: point researchers at syndb-cli-consumer / syndb-cli-provider and the module; cargo run … --features full lines stay valid |
--features dataset, dev,etl, etl, graph-precompute in docs, scripts, skills | documentation/src/deployment/kubernetes.md:118; justfile:32-33; .skills/predeploy-sweep/SKILL.md:70; others | unchanged: those features keep their meaning |
| Backlog item CLI-01 (minimal-feature CLI) | documentation/src/planning/outstanding-work.md:185 | closed by slice 1c |
canix rollout permission string (cargo run … --features full) | canix home/hosts/atlas/can.nix:132 | unchanged |
| canix credentials delivery | canix home/user/can/repositories/syndb_owner_credentials.nix | unchanged; canix keeps the link |
5. Slices
Order: 1a, 1b, 1c, 5, 2, 3, 4. Slice 5 comes before the packages so the module never lists an MCP mode the binary lacks. Each slice is one PR, reviewed by sol-loop. Slices 1 to 3 and 5 are SynDB; slice 4 is canix.
Common qualification, from the main checkout’s direnv with a private target dir under a task directory in /data/scratch/tmp/opencode/syndb/:
systemd-run --user -p MemoryMax=24G -p MemorySwapMax=0 --pipe --wait … </dev/null (size MemoryMax from fleetix_host_metrics). Long checks run in the background; a check stopped by its time limit is unverified.
Slice 1a: Flight payload types to shared, light cli-common (ships alone)
| Goal | cli-common builds without the ClickHouse client and polars; the API’s dependency graph is unchanged; no behaviour change anywhere |
Moves (git mv, no content edits) | crates/core/syndb-clickhouse/src/table.rs, table/tests.rs, flight_descriptor.rs, flight_descriptor/ to crates/core/shared/src/. Their internal paths (crate::table::SyndbTable at flight_descriptor/get.rs:4, upload.rs:4, tests.rs:3) stay valid |
| Edits | crates/core/shared/src/lib.rs:7-31 (two pub mod lines), shared/Cargo.toml (serde_repr = "0.1"); syndb-clickhouse/src/lib.rs:19,25 (re-export), syndb-clickhouse/Cargo.toml:24; cli-common/Cargo.toml:11,27,32-34, cli-common/src/lib.rs:6, cli-common/src/flight.rs:31-35,334-368; crates/tools/cli/Cargo.toml:117-129 (etl adds cli-common/clickhouse); Cargo.lock |
| Cargo features | cli-common: clickhouse = ["dep:syndb-clickhouse", "dep:polars"] |
New test: crates/tools/cli-common/tests/light_deps.rs | default_features_exclude_clickhouse_and_polars: runs cargo tree -p cli-common -e normal --prefix none --offline --locked and fails if clickhouse, syndb-clickhouse, queryfabric-adapter-clickhouse or any polars* crate appears. Matches crate names in the output. It lives in cli-common because the cli floor may still be red until 1c |
| Local runs | cargo test -p shared; cargo check --workspace --all-targets (the re-export claim spans 17 crates; run once, in the background); cargo check -p cli-common --all-targets; cargo check for -p syndb-bench, -p syndb-ci, -p populate-cache, each alone; cargo test -p cli-common; cargo clippy -p shared -p syndb-clickhouse -p cli-common --all-targets -- -D warnings; treefmt |
| API check | section 2.5: save the sorted cargo tree -p api … -e features output on trunk and on the branch; the diff must be empty and goes in the PR |
| Nix builds | none: no new crate and no fileset change |
| Done | all runs pass; empty API diff recorded; git diff --stat origin/trunk -- crates/services empty; merged after the 0.11.1 images are built |
Slice 1b: syndb-cli-core, a pure move
| Goal | the consumer commands live in their own crate; every feature set of cli keeps its command tree; no feature changes |
| New | crates/tools/cli-core/Cargo.toml (package syndb-cli-core, dependencies of section 2.2, [lints] workspace = true); src/lib.rs (pub mod api; pub mod download; pub mod query; pub mod user;, the #![deny(…)] of main.rs:13, and only the allows clippy asks for); src/download.rs (section 2.3) |
Moves (git mv, no content edits) | crates/tools/cli/src/user/, api.rs, query/ to crates/tools/cli-core/src/ |
| Edits | crates/tools/cli/src/main.rs:15,40,47; dataset/args.rs:72-83, dataset/run_impl.rs:48-52, delete dataset/flight_io.rs:102-135; crates/tools/cli/Cargo.toml (add syndb-cli-core; drop rpassword, webbrowser at :41,43 if nothing else uses them); Cargo.toml:36,59-60,365; Cargo.lock; nix/rust.nix:141,231 (one line each, section 4); flake.nix:205-207 (add cargo test -p syndb-cli-core -- $skip); crates/tools/ci/src/testing/queryfabric.rs:158-170 (clippy spec for -p syndb-cli-core --all-targets) |
New tests: crates/tools/cli/src/parse_tests/tiers.rs | prod_surface_is_unchanged (prod): the top-level commands and the 13 data subcommands of dataset/args.rs:11-142 are all present. data_download_args_are_unchanged (dataset): data download has exactly --dataset-id/-d (required, repeatable), --table/-t, --output-dir/-o |
| Local runs | cargo test -p syndb-cli-core; cargo test -p cli --no-default-features --features prod; cargo test -p cli --features full; cargo clippy -p syndb-cli-core --all-targets -- -D warnings; cargo clippy -p cli --all-targets --features full -- -D warnings; python scripts/verify_cli_clippy_baseline.py; treefmt |
| Nix builds | the ETL source derivation only (seconds): find syndb-etl-rust-src in nix derivation show -r .#oci-syndb-etl, nix build '<that drv>^out', then test -f result/crates/tools/cli-core/src/lib.rs (the ETL CLI build itself is not a flake output, nix/packages.nix:313,651); nix build .#syndb-cli bounded (--max-jobs 1 --cores N) from a clean git+file reference, which proves mkRustSrc and pays one cold dependency layer |
| Done | all runs pass, including the two surface tests; git diff -M --stat shows the three moves as renames with no content change |
Slice 1c: tiers in cli
| Goal | the floor builds and is light; provider builds; every existing feature set keeps its command tree |
| Step 0 | run cargo check -p cli --no-default-features --all-targets on trunk and record whether it is red, with the error count |
| Files | crates/tools/cli/Cargo.toml (section 2.4); src/main.rs (mod list :15-47, Commands :119-194, dispatch :220-273): floor Auth, Api, Query, Data, Completions; #[cfg(feature = "provider")] Review; #[cfg(feature = "repo-tools")] Article, Evidence, Test; #[cfg(feature = "mcp-review")] Mcp; src/mcp.rs:6-50; src/dataset/{mod,args,run_impl}.rs (imports at mod.rs:5-20 and module list :22-28 follow the gates); cfg attributes in parse_tests/* and tests/*.rs (tests/op_exhaustive.rs needs repo-tools); flake.nix:205-207; crates/tools/ci/src/testing/queryfabric.rs:158-170; nix/dev-shells.nix:101-121 |
| Cargo features | section 2.4 |
New test: crates/tools/cli/tests/tier_deps.rs | consumer_and_provider_exclude_heavy_crates: cargo tree -p cli -e normal --prefix none --offline --locked for the floor and for provider,mcp-review; fails if any crate in the “must not link” rows of section 3 appears |
Tests added to parse_tests/tiers.rs | consumer_surface_is_exact (not provider): top level is exactly auth, api, query, data, completions, and data is exactly download. provider_surface_is_exact (provider, not dataset) |
| Check | cli-features gains cargo check -p cli --no-default-features --features provider,mcp-review --all-targets and cargo test -p cli --no-default-features -- $skip. If nested cargo tree cannot run in the build sandbox, add --skip tier_deps there, as the existing skips do (flake.nix:200-204), and keep those tests local |
| Local runs | cargo check -p cli --no-default-features --all-targets; the same with --features provider,mcp-review; cargo test -p cli --no-default-features; cargo test -p cli --no-default-features --features prod; cargo test -p cli --features full; cargo clippy -p cli --no-default-features --all-targets -- -D warnings; cargo clippy -p cli --all-targets --features full -- -D warnings; cargo check -p cli --no-default-features --features federation; treefmt |
| Nix builds | none |
| Done | all runs pass; the floor crate count from cargo tree is in the PR (expect about 290, was 509); git diff shows no command renamed |
Slice 5: syndb mcp api (after 1c)
| Goal | every build serves the five public-API tools over MCP stdio; the ops server’s tool list does not change |
| New | crates/tools/cli-core/src/mcp_api.rs: ApiServer with the tools api_corpus_status, api_datasets, api_dataset, api_search, api_capabilities; ApiTool and its argv (from mcp/ops.rs:139-149,204-217, with push_flag, push_positionals at :223-234, made pub because the ops tools keep using them); the parameter types EmptyParams, ApiDatasetParams, DatasetView, ApiSearchParams (:17-18,63-107); dispatch(server_url, name, tool) (the body of :252-272: it runs the current executable, so mcp api calls syndb api …); serve(server_url) |
| Edits | crates/tools/cli-core/Cargo.toml (rmcp, queryfabric-cmd-runner), src/lib.rs; crates/tools/cli/src/mcp.rs (Api variant, unconditional; mod ops and mod review gated as in 1c); src/main.rs:181-186,213-216,266-267 (Mcp unconditional); src/mcp/ops.rs (the five tool methods at :410-476 stay and delegate to the core’s dispatch; the moved types and argv arms are imported, not copied); documentation/src/reference/mcp.md:11-12 |
| Cargo features | none new |
| Tests | core: router_lists_only_the_public_api_tools; api_tools_render_syndb_api_argv moves from mcp/ops.rs:600. cli (mcp): ops_api_tools_match_the_api_server, which fails if any of the five differs in name, description or input schema between the two routers; router_lists_only_the_read_only_ops_tools (:501) stays as is. parse_tests/tiers.rs: the consumer top level gains mcp with exactly api; provider mcp is exactly api, review |
| Local runs | cargo test -p syndb-cli-core; cargo test -p cli --no-default-features; cargo test -p cli --features full; both clippy lines of 1c plus -p syndb-cli-core; treefmt |
| Nix builds | none |
| Done | all runs pass; syndb mcp api from a floor build answers tools/list with the five tools; rmcp is not on any deny list |
Slice 2: profile packages and completions (Nix)
| Goal | two new packages, completions in the three profile packages, timings recorded |
| Files | nix/packages.nix (beside :235-238, :261-278, :489-515); scripts/publish-cargo-artifacts.sh:18-20 |
| Nix outputs | packages.x86_64-linux.syndb-cli-consumer (-p cli --no-default-features), .syndb-cli-provider (-p cli --no-default-features --features provider,mcp-review), and their -cargo-artifacts, built with depsLayerFor (:224-231), doCheck = false, meta.mainProgram = "syndb" |
| Completions | nativeBuildInputs = (commonArgs.nativeBuildInputs or []) ++ [pkgs.installShellFiles] and installShellCompletion --cmd syndb for bash, zsh, fish from $out/bin/syndb completions <shell>, on syndb-cli-consumer, syndb-cli-provider, syndb-cli-tools |
| Builds | nix build .#syndb-cli-consumer .#syndb-cli-provider .#syndb-cli-tools, bounded (--max-jobs 1 --cores N), from a clean git+file reference. No images, cross builds or syndb-cli |
| Done | each binary’s --help lists exactly its tier’s commands (consumer: auth, api, query, data, mcp, completions); share/bash-completion, share/zsh/site-functions, share/fish/vendor_completions.d exist in all three; cold and warm build times and compiled unit counts for consumer, provider and tools are in the PR; nix path-info -S closure sizes recorded |
Slice 3: Home Manager module, check, docs (Nix)
| Goal | homeManagerModules.syndb exported and evaluated per profile without a build |
| Files | new nix/home-manager.nix; flake.nix (flake.homeManagerModules.syndb = import ./nix/home-manager.nix {inherit (inputs) self;}; beside :297; the check in the checks set at :141-232); new documentation/src/reference/home-manager.md plus its SUMMARY entry; documentation/src/reference/mcp.md (one paragraph: installed binary, mcp.servers); README.md:134; documentation/src/get-started/install.md |
| Module options | section 2.6. Define mcp.servers outside the mkIf cfg.enable block so reading it never fails |
| Check | checks.x86_64-linux.hm-module: lib.evalModules with the module, _module.args.pkgs, and stubs for home.packages and home.sessionVariables; a runCommand that only touches $out. Assertions: per profile package.pname is syndb-cli-consumer, syndb-cli-provider, syndb-cli-tools; attrNames mcp.servers is ["syndb-api"], ["syndb-api" "syndb-review"], the five names; syndb-review.env has no SYNDB_REVIEW_SERVER_URL by default and has it when mcp.reviewServerUrl is set, also when only serverUrl is set it is still absent; home.sessionVariables == {} by default and carries both URLs when set; home.packages == [] when disabled; a package override wins |
| Runs | nix build .#checks.x86_64-linux.hm-module (evaluation only); nix eval .#homeManagerModules.syndb --apply builtins.isFunction; treefmt; cargo test -p cli --test public_urls |
| Nix builds | none beyond the trivial check derivation |
| Done | the check passes; docs show a three-line usage example per profile; no origin string in any new file |
Slice 4: canix consumption (after the SynDB slices are on trunk)
| Step | Action |
|---|---|
| 1. Prebuild | at the merged SynDB revision REV, from the same reference the pin will use (git+ssh://git@github.com/memorycircuits/SynDB.git?ref=trunk&rev=REV, no submodules=1): canix cache binary build '<ref>#syndb-cli-tools' with bounded jobs and memory (if it only takes checkout installables, nix build that reference under the same bounds), then the same for syndb-cli-consumer and syndb-cli-provider; canix cache binary push <path> --cache canix-fleet for each (canix docs/src/operations/attic-cache.md:41-44,141-143). Confirm each path is in canix-fleet before step 2 |
| 2. Pin | pin_request with syndb=git+ssh://git@github.com/memorycircuits/SynDB.git?ref=trunk&rev=REV, no --follows (so derivations match the prebuilt paths), then pin_result (canix docs/src/operations/workspace-update.md:161-195). The private-input form matches canix flake.nix:330 |
| 3. Module | new canix home/modules/development/syndb.nix: imports inputs.syndb.homeManagerModules.syndb; programs.syndb = {enable = true; profile = "developer"; mcp.reviewServerUrl = (import "${inputs.syndb}/endpoints.nix").api;} (section 8; the origin is read from SynDB endpoints.nix:6, so canix hard-codes none); on personal PCs only (hostData.users.can.personalPc, as in canix home/user/can/default.nix:17,39) fleetix.mcp.servers = lib.getAttrs ["syndb-ops" "syndb-review"] config.programs.syndb.mcp.servers (syndb-ops already serves the API tools). Imported from canix home/profiles/users/can/default.nix:9-15, which covers can on atlas, murph, nomad, starlord and not bar or thething |
| 4. Check | one assertion block in canix flake/checks/home_profiles.nix: can@atlas, can@nomad, can@murph, can@starlord have programs.syndb.package.pname == "syndb-cli-tools"; can@atlas has fleetix.mcp.servers.syndb-ops; the review entry’s SYNDB_REVIEW_SERVER_URL equals the api origin of the pinned endpoints.nix |
| 5. Qualify | canix repo eval --checkout --apply on those four attributes; treefmt; canix repo flake-check --profile full bounded. Commit only the new module, the import line and the check |
| Not done here | no host switch: ask the operator, then switch_request. syndb-figures and syndb-check stay with the checkout’s .mcp.json because they act on a checkout |
| Done | pin passed; the four evaluations pass; no new cache miss reported by the pin validation |
6. Risks
| Risk | Mitigation |
|---|---|
Slice 1a edits shared and syndb-clickhouse, which the API compiles, on the day of the 0.11.1 freeze | merge after the 0.11.1 images are built; the types move verbatim with their tests; the API’s feature graph diff is empty (section 2.5); cargo check --workspace --all-targets covers every importer |
| Every Nix dependency layer rebuilds after 1a, 1b and 1c, including the API’s | expected and one-off per slice; publish with scripts/publish-cargo-artifacts.sh after each merge so other machines substitute; slice 4 prebuilds what canix needs |
Another session has uncommitted edits in crates/tools/cli/src/user/mod.rs, which 1b moves, and in crates/core/shared/src/ and Cargo.lock, which 1a touches | the moves are git mv with no content change, so Git follows the rename when that session rebases; start 1b after that edit is committed if its owner prefers; work in a maintained worktree and commit with explicit paths |
| The ETL image source list misses the new crate and only the image build shows it | 1b builds the syndb-etl-rust-src derivation and checks the file; the full ETL image compile is not run in these slices, so the first image build after 1b is the real proof |
A deployed image silently loses a command or data download changes its flags | prod_surface_is_unchanged, data_download_args_are_unchanged, and the unchanged definitions of prod, etl-gpu, bench |
A formerly unconditional module is used by a feature set that does not imply repo-tools (--no-default-features --features federation, bench, ci, mcp, article-tools alone) | no Nix output or doc uses those alone; 1c’s local runs check federation alone, and federation gains repo-tools if it fails |
Feature unification puts the ClickHouse client back into a consumer binary when cli is built together with another workspace package | the package and the deny-list test both build -p cli alone; cargo build --workspace output is not a consumer binary |
| The five API tool descriptions exist in two MCP routers and drift | ops_api_tools_match_the_api_server |
Nested cargo tree fails in the Nix build sandbox | skip tier_deps there and keep it in the local runs, as the check already does for other tests |
| The SynDB flake adds many lock nodes and evaluation cost to canix | the pin train’s validation reports it; no follows is a stated constraint |
This session’s syndb-check and syndb-figures MCP servers fail with “syndb-mcp not found in PATH” | separate from this work: some harness config still names the dev-shell shim. The developer profile puts syndb on PATH, so that config can call syndb mcp <mode>; not changed by this plan |
7. Operator questions
Revision 1 asked these; section 8 answers all five. No new blocking question. One default to confirm: slice 1a merges after the 0.11.1 images are built.
- Should the consumer profile carry an MCP server (slice 5,
syndb mcp apiwith the five public-API tools)? Without it the consumer profile has no MCP at all. - Is a consumer build that still links
arrow-flight,tonicand theclickhouseclient crate acceptable? Removing them means either droppingdata downloadfrom the consumer or feature-gating the core cratesyndb-clickhouse. - Provider excludes
syndb etl export(it needs the Python export provider and the wholeetlfeature). Confirm, or say that providers need it. - In canix: developer profile for
canonly (atlas, murph, nomad, starlord), or fordejanatoo? And which review server URL, if any, should canix set? - Accept feature-tiered modules in the one
clicrate, or do you want the separatesyndb-cli-corecrate?
8. Operator amendments (2026-11-17)
| Question | Answer | Effect on the plan |
|---|---|---|
| Q5 crate layout | Separate core crate (overrides §2.5 row 1) | Redesign slice 1: split a syndb-cli-core (or similarly named) crate for the consumer floor; the cli binary composes it with provider/developer tiers. Solve the data/mcp span problem explicitly |
| Q2 consumer deps | Trim the core crate too | Feature-gate syndb-clickhouse (or split its Flight ticket types out) so the consumer does not link the clickhouse client crate; the API server’s build must be unchanged |
| Q1 consumer MCP | Yes, add syndb mcp api | Slice 5 becomes required; every profile gets syndb-api |
| Q3 provider ETL export | Exclude | Unchanged: etl export stays developer-only |
| Q4 canix users | can only | Unchanged: can on atlas, murph, nomad, starlord; dejana gets nothing |
| Review server URL in canix | (earlier decision 2026-11-17: production with the owner login) | canix sets programs.syndb.mcp.reviewServerUrl to the production API origin for can |
| Merge timing for slice 1a | Default accepted by the orchestrator: merge after the 0.11.1 images are built (freeze Tue 2026-11-17 22:00Z), since it touches crates the API compiles | Develop now; land after the freeze |