Upload and Analyze a Dataset
This scenario walks through the lifecycle of a dataset in SynDB: find it, query it, get the rows out, then contribute your own.
Important
Repository access
The
syndbCLI is not published yet. Uploading from the command line and downloading whole tables need read access to the private repository, granted on request — see Get help. The web app at app.syndb.xyz needs no installation.
1. Find datasets
In the web app, type a dataset label, DOI, species or note into the search box. Filters narrow the result by table, species, brain region and microscopy.
From the command line, the public catalogue needs no login:
syndb api datasets
syndb api search "Caenorhabditis elegans"
syndb api datasets prints the ID-to-label map; syndb api search runs
full-text search. Each dataset has a stable UUID, and its page lists the tables
it contains with their row counts.
2. Query with SyQL
Open the Query page in the web app, paste a query, press Explain query to validate it and Execute query to run it. SyQL extends SQL with connectomics-aware syntax:
FROM synapses WHERE neurotransmitter = 'GABA' LIMIT 100
Dataset-scoped queries are more precise:
SELECT pre_neuron_id, sum(synapse_count) AS synapses
FROM synapses
WHERE dataset_id = '676f5945-b47e-5ed9-9279-401342059449'
GROUP BY pre_neuron_id
ORDER BY synapses DESC
LIMIT 100
A full walk-through, including how to read the numbers, is the worked example.
From the CLI, the same statement goes to the job queue:
syndb query exec "FROM synapses WHERE dataset_id = '676f5945-b47e-5ed9-9279-401342059449' LIMIT 100"
It prints a job ID. The job page in the web app shows the job status and, when it finishes, the result download.
3. Get the rows out
The job page has a Download result button once the job completes. A query result is always an Arrow IPC stream named after the job. Open it with:
import polars as pl
rows = pl.read_ipc_stream("result.arrow")
To take a whole table rather than a query result, download it over Arrow Flight:
syndb data download \
--dataset-id 676f5945-b47e-5ed9-9279-401342059449 \
--table 6 \
--output-dir ./hemibrain
Table numbers select the table; 6 is synapses, 1 is neurons. The full
list is in the CLI reference.
4. Upload a new dataset
Create the dataset record first — in the web app’s Contribute page, or with
syndb data new — so you have a dataset UUID. Then validate and prepare the
files, and upload:
syndb data validate --input-dir ./my-dataset
syndb data prepare --input-dir ./my-dataset --output-dir ./my-dataset-prepared
syndb data upload --input-dir ./my-dataset-prepared --dataset-id <dataset-uuid>
One file per table, named for the compartment. Validation checks the columns and types against the current schema, so fix anything it reports before uploading. See Upload and Data structuring.
5. Save and share a query
syndb query save-syql --label "hemibrain top synapses" "FROM synapses WHERE dataset_id = '...' LIMIT 100"
syndb query list
syndb query run --id <query-uuid>
syndb query status --id <query-uuid>
Saved queries and jobs belong to the account that created them: another user cannot open your query ID or job ID, and a rerun reads whatever the tables hold on the day it runs. To share or cite a result, give the query text, the dataset IDs, the date you ran it and the exported result file; see Saved queries and Jobs.