Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Upload and Analyze a Dataset

This scenario walks through the lifecycle of a dataset in SynDB: find it, query it, get the rows out, then contribute your own.

Important

Repository access

The syndb CLI is not published yet. Uploading from the command line and downloading whole tables need read access to the private repository, granted on request — see Get help. The web app at app.syndb.xyz needs no installation.

1. Find datasets

In the web app, type a dataset label, DOI, species or note into the search box. Filters narrow the result by table, species, brain region and microscopy.

From the command line, the public catalogue needs no login:

syndb api datasets
syndb api search "Caenorhabditis elegans"

syndb api datasets prints the ID-to-label map; syndb api search runs full-text search. Each dataset has a stable UUID, and its page lists the tables it contains with their row counts.

2. Query with SyQL

Open the Query page in the web app, paste a query, press Explain query to validate it and Execute query to run it. SyQL extends SQL with connectomics-aware syntax:

FROM synapses WHERE neurotransmitter = 'GABA' LIMIT 100

Dataset-scoped queries are more precise:

SELECT pre_neuron_id, sum(synapse_count) AS synapses
FROM synapses
WHERE dataset_id = '676f5945-b47e-5ed9-9279-401342059449'
GROUP BY pre_neuron_id
ORDER BY synapses DESC
LIMIT 100

A full walk-through, including how to read the numbers, is the worked example.

From the CLI, the same statement goes to the job queue:

syndb query exec "FROM synapses WHERE dataset_id = '676f5945-b47e-5ed9-9279-401342059449' LIMIT 100"

It prints a job ID. The job page in the web app shows the job status and, when it finishes, the result download.

3. Get the rows out

The job page has a Download result button once the job completes. A query result is always an Arrow IPC stream named after the job. Open it with:

import polars as pl
rows = pl.read_ipc_stream("result.arrow")

To take a whole table rather than a query result, download it over Arrow Flight:

syndb data download \
  --dataset-id 676f5945-b47e-5ed9-9279-401342059449 \
  --table 6 \
  --output-dir ./hemibrain

Table numbers select the table; 6 is synapses, 1 is neurons. The full list is in the CLI reference.

4. Upload a new dataset

Create the dataset record first — in the web app’s Contribute page, or with syndb data new — so you have a dataset UUID. Then validate and prepare the files, and upload:

syndb data validate --input-dir ./my-dataset
syndb data prepare  --input-dir ./my-dataset --output-dir ./my-dataset-prepared
syndb data upload   --input-dir ./my-dataset-prepared --dataset-id <dataset-uuid>

One file per table, named for the compartment. Validation checks the columns and types against the current schema, so fix anything it reports before uploading. See Upload and Data structuring.

5. Save and share a query

syndb query save-syql --label "hemibrain top synapses" "FROM synapses WHERE dataset_id = '...' LIMIT 100"
syndb query list
syndb query run    --id <query-uuid>
syndb query status --id <query-uuid>

Saved queries and jobs belong to the account that created them: another user cannot open your query ID or job ID, and a rerun reads whatever the tables hold on the day it runs. To share or cite a result, give the query text, the dataset IDs, the date you ran it and the exported result file; see Saved queries and Jobs.

Next steps