Phase 02 · Sub-layer 01 — Populate and serve syndb.datasets from the database
Working tree
Repository root. Requires production cluster access
(Arrow Flight / API auth at ~/.config/syndb/auth.json or
$SYNDB_AUTH_TOKEN) and whatever credentials the deployment uses to write
metadata. Cluster ops touch the Nautilus deployment; the manuscript edit is
a one-line Limitations removal.
Prerequisites (updated 2026-06-13)
- The seed path now auto-resolves the Postgres connection from the
cluster — no manual
POSTGRES_*export.syndb ops seed-datasetsreads the Zalando credentials secret (usyndb.syndb-postgres.credentials.postgresql.acid.zalan.do) viakubectland opens akubectl port-forwardtosvc/syndb-postgresautomatically (seecrates/tools/cli/src/ops/metadata_db.rs). - Therefore the only requirement is a kube context with read access to
the secret and the postgres service in the target namespace (default
syndb). Flags:--namespace <ns>,--from-cluster(force cluster resolution even ifPOSTGRES_*is set). SettingPOSTGRES_HOSTstill takes precedence (back-compat). - The Limitations-bullet removal in
discussion.texis already committed; the remaining work is the cluster-side populate + API verification.
Goal
The syndb.datasets table (Postgres metadata) holds one row per ingested
dataset with DOI, licence, owner, and canonical name, populated from the
existing deployment manifests through the platform’s own ingestion/admin
path — and the public metadata API returns those fields from the database.
Why this matters now
discussion.tex Limitations: “The syndb.datasets table is empty
cluster-wide … dataset-level metadata (DOI, licence, owner, canonical name)
is currently sourced from out-of-band manifests held in the deployment
repository rather than served from the database itself.” The FAIR/provenance
claims in the Discussion imply DB-served metadata; this makes them true.
Out of scope
- Changing the
datasetsschema or the public API contract. - Inventing metadata — only load what the deployment manifests already record.
- Editing any figure or the FAIR table (its scores are unaffected).
Plan
- Ensure a working kube context for the target namespace (default
syndb). The connection resolves automatically — noPOSTGRES_*export needed. - Dry-run to preview:
syndb ops seed-datasets --dry-run. It reports datasets scanned and publications/links that would be added. (The command already exists and now auto-resolves the DB connection from the cluster —crates/tools/cli/src/ops/{seed_datasets,metadata_db}.rs.) - Apply:
syndb ops seed-datasets(add--namespace <ns>if notsyndb); confirm the summary reports rows inserted. - Query through the production metadata API (not the manifest) and confirm DOI/licence/owner/canonical_name come back for a sample of datasets.
- Remove the now-false Limitations bullet in
article/sections/discussion.tex(the\item The \texttt{syndb.datasets} table is empty cluster-wide …paragraph). Coordinate with Phase 05 ondiscussion.tex(land before 05). - Build gate (
lualatex ×2).
Acceptance criteria
-
SELECT count(*) FROM datasets(or API equivalent) ≥ number of ingested datasets, each with non-null DOI, licence, owner, canonical_name. - The production metadata API returns those fields from the DB for ≥3 sampled datasets.
- The
syndb.datasets-empty\itemis gone fromdiscussion.tex; build clean. - Any new seeding logic is a
syndbsubcommand, not a shell CronJob (shell-less image).
Files likely touched
crates/— datasets entity /syndb opsseeding subcommand (if added).- deployment manifests /
infrastructure/(read; maybe a values file). article/sections/discussion.tex— remove one Limitations\item.
Pitfalls
- Symptom: CronJob/StartError with no logs. Cause: the published image is a shell-less static binary; bash job bodies fail. Recovery: move logic into
syndb ops. - Symptom: RoleBinding grant silently no-ops. Cause: ServiceAccount subject missing namespace. Recovery: set the SA namespace in the RoleBinding.
- Symptom: removing the Limitations bullet breaks a
\cref. Cause: another sentence referenced it. Recovery: grep for back-references first.
Reference
- Memory: “syndb image has no shell”; “RoleBinding SA namespace”.
documentation/CONTRIBUTING.md§“Cluster operator notes”.- Gap inventory item 13.