Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Phase 02 · Sub-layer 01 — Populate and serve syndb.datasets from the database

Working tree

Repository root. Requires production cluster access (Arrow Flight / API auth at ~/.config/syndb/auth.json or $SYNDB_AUTH_TOKEN) and whatever credentials the deployment uses to write metadata. Cluster ops touch the Nautilus deployment; the manuscript edit is a one-line Limitations removal.

Prerequisites (updated 2026-06-13)

  • The seed path now auto-resolves the Postgres connection from the cluster — no manual POSTGRES_* export. syndb ops seed-datasets reads the Zalando credentials secret (usyndb.syndb-postgres.credentials.postgresql.acid.zalan.do) via kubectl and opens a kubectl port-forward to svc/syndb-postgres automatically (see crates/tools/cli/src/ops/metadata_db.rs).
  • Therefore the only requirement is a kube context with read access to the secret and the postgres service in the target namespace (default syndb). Flags: --namespace <ns>, --from-cluster (force cluster resolution even if POSTGRES_* is set). Setting POSTGRES_HOST still takes precedence (back-compat).
  • The Limitations-bullet removal in discussion.tex is already committed; the remaining work is the cluster-side populate + API verification.

Goal

The syndb.datasets table (Postgres metadata) holds one row per ingested dataset with DOI, licence, owner, and canonical name, populated from the existing deployment manifests through the platform’s own ingestion/admin path — and the public metadata API returns those fields from the database.

Why this matters now

discussion.tex Limitations: “The syndb.datasets table is empty cluster-wide … dataset-level metadata (DOI, licence, owner, canonical name) is currently sourced from out-of-band manifests held in the deployment repository rather than served from the database itself.” The FAIR/provenance claims in the Discussion imply DB-served metadata; this makes them true.

Out of scope

  • Changing the datasets schema or the public API contract.
  • Inventing metadata — only load what the deployment manifests already record.
  • Editing any figure or the FAIR table (its scores are unaffected).

Plan

  1. Ensure a working kube context for the target namespace (default syndb). The connection resolves automatically — no POSTGRES_* export needed.
  2. Dry-run to preview: syndb ops seed-datasets --dry-run. It reports datasets scanned and publications/links that would be added. (The command already exists and now auto-resolves the DB connection from the cluster — crates/tools/cli/src/ops/{seed_datasets,metadata_db}.rs.)
  3. Apply: syndb ops seed-datasets (add --namespace <ns> if not syndb); confirm the summary reports rows inserted.
  4. Query through the production metadata API (not the manifest) and confirm DOI/licence/owner/canonical_name come back for a sample of datasets.
  5. Remove the now-false Limitations bullet in article/sections/discussion.tex (the \item The \texttt{syndb.datasets} table is empty cluster-wide … paragraph). Coordinate with Phase 05 on discussion.tex (land before 05).
  6. Build gate (lualatex ×2).

Acceptance criteria

  • SELECT count(*) FROM datasets (or API equivalent) ≥ number of ingested datasets, each with non-null DOI, licence, owner, canonical_name.
  • The production metadata API returns those fields from the DB for ≥3 sampled datasets.
  • The syndb.datasets-empty \item is gone from discussion.tex; build clean.
  • Any new seeding logic is a syndb subcommand, not a shell CronJob (shell-less image).

Files likely touched

  • crates/ — datasets entity / syndb ops seeding subcommand (if added).
  • deployment manifests / infrastructure/ (read; maybe a values file).
  • article/sections/discussion.tex — remove one Limitations \item.

Pitfalls

  • Symptom: CronJob/StartError with no logs. Cause: the published image is a shell-less static binary; bash job bodies fail. Recovery: move logic into syndb ops.
  • Symptom: RoleBinding grant silently no-ops. Cause: ServiceAccount subject missing namespace. Recovery: set the SA namespace in the RoleBinding.
  • Symptom: removing the Limitations bullet breaks a \cref. Cause: another sentence referenced it. Recovery: grep for back-references first.

Reference

  • Memory: “syndb image has no shell”; “RoleBinding SA namespace”.
  • documentation/CONTRIBUTING.md §“Cluster operator notes”.
  • Gap inventory item 13.