Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

ADR-008: Scientific Evidence Fabric

FieldValue
StatusAccepted
Date2026-08-22
Decision makersSynDB maintainers
ScopeScientific identity, lineage, execution, eligibility, and review evidence

Context

SynDB already has artifact manifests, table versions, graph-precompute provenance, query-result manifests, W3C PROV-inspired events, and manuscript result contracts. These records answer different questions, but no single content-addressed object binds them into a verifiable transformation graph. ClickHouse rows consequently carry copied provenance strings, and the replayability ledger has become an independently maintained authority.

BR-02 exposed the distinction this architecture must preserve. A raw source may contain 629 rows while a preregistered analysis universe contains 302 nodes. That is not failed parity when an explicit, deterministic projection connects the two. Conversely, an unexplained 406-to-812 row change is not a valid transformation merely because the counts have a simple ratio.

Decision

Extend syndb-lakehouse as the canonical Scientific Evidence Fabric. Existing artifact and table-version types remain the storage-domain foundation. prov-activity remains the event/audit layer and QueryFabric remains a generic query-execution layer. Neither becomes an authority for SynDB scientific identity.

The authoritative stages are:

  1. SourceSnapshot
  2. NormalizedDataset
  3. AnalysisUniverse
  4. GraphArtifact
  5. PrecomputeArtifact
  6. AnalysisResult
  7. ReviewAttestation

Each stage is represented by an immutable EvidenceCapsule. A capsule binds payload digests and schema fingerprints to parent capsule digests, frozen dataset and table-version identities, transformation code and configuration, graph semantics, execution parameters, diagnostics, eligibility reason codes, and optional separately signed review attestations.

flowchart LR
    accTitle: Scientific evidence lineage and trust boundaries
    accDescr: Immutable payloads and capsules form the authority; ClickHouse and generated ledgers are rebuildable indexes, while human review and confirmation access require explicit attestations.

    source[SourceSnapshot] --> normalized[NormalizedDataset]
    normalized --> universe[AnalysisUniverse]
    universe --> graph[GraphArtifact]
    graph --> precompute[PrecomputeArtifact]
    precompute --> result[AnalysisResult]
    inventory[ReviewPacket] --> attestation{ReviewAttestation}
    attestation --> result

    store[(Content-addressed EvidenceStore)]
    source --- store
    normalized --- store
    universe --- store
    graph --- store
    precompute --- store
    result --- store

    result --> index[(ClickHouse evidence index)]
    result --> ledger[Generated replay/readiness views]
    confirmation[Held-out confirmation capsule] --> gate{Attestation gate}
    gate -->|Explicit audited access| result

    classDef authority fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#1e3a5f
    classDef derived fill:#dcfce7,stroke:#16a34a,stroke-width:2px,color:#14532d
    classDef gate fill:#fef9c3,stroke:#ca8a04,stroke-width:2px,color:#713f12
    class source,normalized,universe,graph,precompute,result,store authority
    class index,ledger derived
    class attestation,gate gate

Identity rules

Raw-source parity and derived-output equivalence are different concepts. Exact row and byte equality is required only for an identity-preserving copy. Normalization, filtering, projection, deduplication, endpoint closure, directional expansion, aggregation, and statistical analysis are declared transformations. Their outputs have independent immutable identities.

Every transformation records:

  • input and output capsule digests;
  • transformation name, version, code revision, dirty-tree state, and exact configuration digest;
  • input and output counts, key definitions, duplicate/drop/expansion counts, schema changes, semantic invariants, and verification results;
  • payload and schema digests for every output.

Capsule identity is the BLAKE3 digest of deterministic canonical JSON excluding the capsule digest itself, timestamps, event UUIDs, and human attestations. Those excluded records retain their own immutable digests. Assigning a new event ID or timestamp therefore cannot change scientific identity.

Storage and indexing

Payloads and capsules live in the existing artifact-first object-storage path through an EvidenceStore backed by OpenDAL. Local filesystem storage is the offline/test backend; S3 is the operational backend. Objects are immutable and addressed by digest.

ClickHouse is a query index. A small additive evidence-index table stores lookup fields, capsule and payload digests, schema fingerprint, producer/run identity, publication state, and selected performance columns. Eligibility always loads and verifies the capsule and payload from EvidenceStore; copied row metadata is insufficient. Existing precomputed tables continue serving during migration.

Eligibility and projections

Generic eligibility policies consume frozen analysis requirements, required capsule kinds, verified lineage, diagnostics, and review gates. Decisions use stable reason codes; human-readable blockers are rendered from those codes. Replayability and readiness files are deterministic projections of capsule decisions, not independent authorities.

Confirmation capsules are not resolved by default. Resolution requires an explicit attestation digest and writes a separate immutable audit event. A copied confirmation_outcomes_inspected=false field is not an access control.

Compatibility and migration

Migration is additive and rollback-safe:

  1. Create the evidence index without changing serving tables.
  2. Write new capsules beside existing artifacts.
  3. Import legacy manifests as legacy-unverified unless complete lineage is reconstructed.
  4. Backfill the evidence index from verified capsules in isolated batches.
  5. Compare generated projections with checked-in views before enabling reads.
  6. Roll back by disabling evidence-index reads; immutable capsules and existing serving rows remain untouched.

No production migration, destructive ETL, cutover, or v1 retirement is part of this decision.

Consequences

  • BR-01 through BR-12 share one identity, lineage, eligibility, and review model.
  • Analysis adapters retain only scientific estimator and claim policy.
  • Transformations may legitimately change row counts, but unexplained changes fail closed.
  • Historical BR JSON and caches remain valid blocked evidence; plausible copied metadata cannot retroactively make them eligible.
  • Capsule and schema generation become required drift checks in CI.