ADR-008: Scientific Evidence Fabric
| Field | Value |
|---|---|
| Status | Accepted |
| Date | 2026-08-22 |
| Decision makers | SynDB maintainers |
| Scope | Scientific identity, lineage, execution, eligibility, and review evidence |
Context
SynDB already has artifact manifests, table versions, graph-precompute provenance, query-result manifests, W3C PROV-inspired events, and manuscript result contracts. These records answer different questions, but no single content-addressed object binds them into a verifiable transformation graph. ClickHouse rows consequently carry copied provenance strings, and the replayability ledger has become an independently maintained authority.
BR-02 exposed the distinction this architecture must preserve. A raw source may contain 629 rows while a preregistered analysis universe contains 302 nodes. That is not failed parity when an explicit, deterministic projection connects the two. Conversely, an unexplained 406-to-812 row change is not a valid transformation merely because the counts have a simple ratio.
Decision
Extend syndb-lakehouse as the canonical Scientific Evidence Fabric. Existing
artifact and table-version types remain the storage-domain foundation.
prov-activity remains the event/audit layer and QueryFabric remains a generic
query-execution layer. Neither becomes an authority for SynDB scientific
identity.
The authoritative stages are:
SourceSnapshotNormalizedDatasetAnalysisUniverseGraphArtifactPrecomputeArtifactAnalysisResultReviewAttestation
Each stage is represented by an immutable EvidenceCapsule. A capsule binds
payload digests and schema fingerprints to parent capsule digests, frozen
dataset and table-version identities, transformation code and configuration,
graph semantics, execution parameters, diagnostics, eligibility reason codes,
and optional separately signed review attestations.
flowchart LR
accTitle: Scientific evidence lineage and trust boundaries
accDescr: Immutable payloads and capsules form the authority; ClickHouse and generated ledgers are rebuildable indexes, while human review and confirmation access require explicit attestations.
source[SourceSnapshot] --> normalized[NormalizedDataset]
normalized --> universe[AnalysisUniverse]
universe --> graph[GraphArtifact]
graph --> precompute[PrecomputeArtifact]
precompute --> result[AnalysisResult]
inventory[ReviewPacket] --> attestation{ReviewAttestation}
attestation --> result
store[(Content-addressed EvidenceStore)]
source --- store
normalized --- store
universe --- store
graph --- store
precompute --- store
result --- store
result --> index[(ClickHouse evidence index)]
result --> ledger[Generated replay/readiness views]
confirmation[Held-out confirmation capsule] --> gate{Attestation gate}
gate -->|Explicit audited access| result
classDef authority fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#1e3a5f
classDef derived fill:#dcfce7,stroke:#16a34a,stroke-width:2px,color:#14532d
classDef gate fill:#fef9c3,stroke:#ca8a04,stroke-width:2px,color:#713f12
class source,normalized,universe,graph,precompute,result,store authority
class index,ledger derived
class attestation,gate gate
Identity rules
Raw-source parity and derived-output equivalence are different concepts. Exact row and byte equality is required only for an identity-preserving copy. Normalization, filtering, projection, deduplication, endpoint closure, directional expansion, aggregation, and statistical analysis are declared transformations. Their outputs have independent immutable identities.
Every transformation records:
- input and output capsule digests;
- transformation name, version, code revision, dirty-tree state, and exact configuration digest;
- input and output counts, key definitions, duplicate/drop/expansion counts, schema changes, semantic invariants, and verification results;
- payload and schema digests for every output.
Capsule identity is the BLAKE3 digest of deterministic canonical JSON excluding the capsule digest itself, timestamps, event UUIDs, and human attestations. Those excluded records retain their own immutable digests. Assigning a new event ID or timestamp therefore cannot change scientific identity.
Storage and indexing
Payloads and capsules live in the existing artifact-first object-storage path
through an EvidenceStore backed by OpenDAL. Local filesystem storage is the
offline/test backend; S3 is the operational backend. Objects are immutable and
addressed by digest.
ClickHouse is a query index. A small additive evidence-index table stores lookup
fields, capsule and payload digests, schema fingerprint, producer/run identity,
publication state, and selected performance columns. Eligibility always loads
and verifies the capsule and payload from EvidenceStore; copied row metadata
is insufficient. Existing precomputed tables continue serving during migration.
Eligibility and projections
Generic eligibility policies consume frozen analysis requirements, required capsule kinds, verified lineage, diagnostics, and review gates. Decisions use stable reason codes; human-readable blockers are rendered from those codes. Replayability and readiness files are deterministic projections of capsule decisions, not independent authorities.
Confirmation capsules are not resolved by default. Resolution requires an
explicit attestation digest and writes a separate immutable audit event. A
copied confirmation_outcomes_inspected=false field is not an access control.
Compatibility and migration
Migration is additive and rollback-safe:
- Create the evidence index without changing serving tables.
- Write new capsules beside existing artifacts.
- Import legacy manifests as
legacy-unverifiedunless complete lineage is reconstructed. - Backfill the evidence index from verified capsules in isolated batches.
- Compare generated projections with checked-in views before enabling reads.
- Roll back by disabling evidence-index reads; immutable capsules and existing serving rows remain untouched.
No production migration, destructive ETL, cutover, or v1 retirement is part of this decision.
Consequences
- BR-01 through BR-12 share one identity, lineage, eligibility, and review model.
- Analysis adapters retain only scientific estimator and claim policy.
- Transformations may legitimately change row counts, but unexplained changes fail closed.
- Historical BR JSON and caches remain valid blocked evidence; plausible copied metadata cannot retroactively make them eligible.
- Capsule and schema generation become required drift checks in CI.