Community Edition / Docs / Connectors

Remote storage, in and out

Study definitions rarely start life on the machine running the pipeline, and SDTM outputs rarely end there. The connector bridge covers both directions with a single optional service built on rclone, which speaks Google Drive, OneDrive, WebDAV (Nextcloud, SharePoint), S3 and S3-compatibles, SFTP, and dozens more backends.

Enable

docker compose --profile connectors up -d

Two flows

Inbox. pull copies files from a remote folder into lake/data/inbox/<remote>/, then ingests anything that looks like a USDM study definition (a JSON file with a top-level study object) straight into the broker. Drop a study in a shared folder; it becomes entities.

Publish. publish copies the lake's outputs (SDTM datasets, Dataset-JSON design domains) to a folder on the remote, where biostatisticians actually work.

Configure with the wizard

The setup wizard (http://localhost:8080/setup.html) creates keyed remotes directly: WebDAV, S3 and S3-compatibles, SFTP, FTP. Test the connection, pull, publish, all from the page.

OAuth backends: Google Drive and OneDrive

OAuth needs one browser step that a headless container cannot do. Run the guided flow once:

docker compose --profile connectors exec bridge rclone config

Or authorize on your workstation and paste the token:

# on a machine with a browser
rclone authorize "drive"
# then, in the wizard's terminal flow or rclone config,
# paste the token JSON when asked

Once created, OAuth remotes appear in the wizard's remote list and work with pull and publish like any other backend.

The API, if you skip the wizard

# create a remote (keyed backends)
curl -X POST localhost:8107/remotes -H 'Content-Type: application/json' -d '{
  "name": "sitehub", "type": "webdav",
  "params": {"url": "https://dav.example.com", "vendor": "other",
             "user": "demo", "pass": "..."}}'

# test, pull, publish
curl -X POST localhost:8107/check   -d '{"remote": "sitehub"}' -H 'Content-Type: application/json'
curl -X POST localhost:8107/pull    -d '{"remote": "sitehub", "path": "studies"}' -H 'Content-Type: application/json'
curl -X POST localhost:8107/publish -d '{"remote": "sitehub", "path": "pne-lake"}' -H 'Content-Type: application/json'

Notes on credentials

• rclone stores credentials obscured in its config file, which lives in a named Docker volume (bridge-config) and never leaves the bridge container.
• The bridge has no inbound exposure beyond your localhost port mapping; it only dials out to the remotes you configure.
• S3 here means any S3-compatible endpoint: AWS, MinIO, Ceph, Cloudflare R2. Point endpoint wherever yours lives. Using AWS S3 through the bridge is your deployment's choice; the core pipeline still requires nothing from it.