Federated Query Across Institutions
SynDB’s federation layer connects independently operated nodes into a single query surface. A researcher on the hub can query datasets hosted at any participating institution as if they were local — without moving data or coordinating per-dataset access.
Scenario
The FlyBrain Institute hosts a Drosophila optic-lobe reconstruction. The Cortex Lab hosts a mouse V1 dataset. Both run SynDB nodes.
A researcher on the hub wants to compare synapse densities between the two species. With federation, this is a single query.
Architecture
┌─────────────────────────────────────┐
│ Hub │
│ ┌──────────────────────────────┐ │
│ │ Single API endpoint │ │
│ │ syql.flybrain.org/v1/syql │ │
│ └──────┬──────────────┬────────┘ │
│ │ │ │
┌─────────▼──────┐ ┌────▼────────────┐
│ FlyBrain Node │ │ Cortex Lab │
│ (drosophila) │ │ (mouse V1) │
│ ClickHouse │ │ ClickHouse │
│ S3 │ │ S3 │
└────────────────┘ └─────────────────┘
Prerequisites
- A hub node with federation enabled
- Two or more remote clusters registered with the hub
- The remote clusters must be reachable from the hub over Arrow Flight (TCP)
Walkthrough
1. Register a cluster
On the hub, register each participating site:
syndb federation register \
--name flybrain \
--api-url https://flybrain.example.com \
--flight-addr flybrain.example.com:50051
2. Verify cluster health
syndb federation list
syndb federation health flybrain
Healthy clusters return dataset catalog entries, version info, and latency.
3. Query across clusters
The SCOPE federation directive in SyQL tells the planner to scatter the
query to all available clusters:
syndb query run "
SCOPE federation
FROM synapses
WHERE species = 'drosophila' OR species = 'mouse'
GROUP BY species
"
The planner:
- Sends the scatter statement to every cluster
- Each cluster executes locally and streams partial results back
- The gather stage merges partials (summing counts, re-aggregating measures)
4. Query a specific cluster
If you only need data from one remote site, use SCOPE remote:
syndb query run "
SCOPE remote
FROM synapses LIMIT 100
"
The planner routes to the first healthy cluster.
5. Monitor federation
syndb federation health # per-cluster status
syndb federation routes # active routing table
Data ownership
Each cluster retains full ownership of its data. The hub never stores remote data — it only caches catalog metadata and routing information. Queries execute on the remote cluster’s infrastructure and the hub streams results through.
Troubleshooting
| Symptom | Likely cause | Check |
|---|---|---|
| Query returns only local results | Remote cluster unreachable | syndb federation health <name> |
| Timeout on large queries | Remote cluster under-provisioned | Increase timeout in K8sDriverConfig |
| “Cluster not found” | Registration expired | Re-register the cluster |