Community Edition / Docs / Architecture

One broker, a handful of services, a directory

The whole stack is a compose file. Here is what runs, which port it listens on, and why none of it needs a cloud account.

flowchart TB
  web["web :8080<br/>walkthrough UI, context host, /api proxy"]
  scorpio["Scorpio :9090<br/>NGSI-LD broker, in-memory, PostGIS"]
  usdm["usdm :8101<br/>USDM to entities"]
  adaptive["adaptive :8106<br/>NL statement to entity"]
  subs["subscriptions :8103<br/>notification listener"]
  transform["transform :8104<br/>sdtm.oak (R)"]
  projection["projection :8102<br/>trial-design domains"]
  lake["lake :8105<br/>DuckDB over files"]
  bridge["bridge :8107<br/>rclone, optional"]
  data[("lake/data<br/>filesystem")]

  web -- "serves /context/pne-context.jsonld" --> scorpio
  usdm -- "NGSI-LD entities" --> scorpio
  adaptive -- "entities" --> scorpio
  scorpio -- "notification POST" --> subs
  subs -- "POST /run" --> transform
  scorpio -. "reads" .-> projection
  scorpio -. "reads" .-> transform
  projection -- "Dataset-JSON" --> data
  transform -- "SDTM CSV, Dataset-JSON" --> data
  lake <-- "SQL" --> data
  bridge <-- "pull, publish" --> data

┌─────────────────────────────────────────────────┐ │ web :8080 │ │ walkthrough UI · JSON-LD context host · │ │ same-origin /api proxy to every service │ └───────┬─────────────────────────────────────────┘ │ serves /context/pne-context.jsonld ▼ ┌──────────┐ NGSI-LD ┌──────────────────┐ notification ┌───────────────┐ │ usdm │──entities─▶│ Scorpio :9090 │─────POST────────▶│ subscriptions │ │ :8101 │ │ in-memory │ │ :8103 │ └──────────┘ │ all-in-one │ └──────┬────────┘ │ + PostGIS │ │ POST /run ┌──────────┐ reads └──────────────────┘ reads ▼ │projection│◀───────────────────┘ └─────────────────────▶┌───────────────┐ │ :8102 │ │ transform │ └────┬─────┘ ┌──────────────────┐ │ :8104 │ │ Dataset-JSON │ adaptive :8106 │ │ sdtm.oak (R) │ ▼ │ NL -> entity │──entities──▶ └──────┬────────┘ ┌─────────────────┐ │ API or stub │ (broker) │ SDTM CSV + │ lake/data │ └──────────────────┘ │ Dataset-JSON │ (filesystem) │◀─────────────────────────────────────────────┘ │ ▲ │ ┌──────────────────┐ │ │ DuckDB │ │ bridge :8107 │ optional: rclone in/out │ lake :8105 │◀───▶│ connectors │ (GDrive, OneDrive, WebDAV, └─────────────────┘ └──────────────────┘ S3, SFTP, ...)

Every arrow is an HTTP request on localhost. No queue, no bucket, no serverless function.

Service	Port	Role
web	8080	Walkthrough UI, JSON-LD context host, same-origin proxy to every service.
scorpio	9090	NEC Scorpio NGSI-LD broker, upstream image, in-memory profile, PostGIS behind it.
usdm	8101	Turns a USDM-style study definition into broker entities.
projection	8102	Reads broker state, writes Dataset-JSON trial-design domains to the lake.
subscriptions	8103	Receives broker notifications and triggers a transform run.
transform	8104	sdtm.oak in R. Regenerates SDTM from current broker state.
lake	8105	Three HTTP endpoints over a directory; DuckDB for SQL.
adaptive	8106	Natural-language statement to entity proposal. Model call or offline stub.
bridge	8107	Optional rclone connectors, in and out. Off by default.

The broker never needed the cloud

NEC Scorpio ships its messaging layer as first-class build variants, visible in its release tags: -java (in-memory), -kafka, -sqs, -amqp, -mqtt. The in-memory build runs every broker service in one JVM with in-process channels. The AWS coupling in Garnet-style deployments came from choosing the -sqs variant to span Fargate containers; it was packaging, not a broker requirement. This stack runs the upstream image, unmodified.

Three replacements

Subscriptions instead of SQS/SNS. One NGSI-LD subscription covers the execution entity types. The broker POSTs notifications straight to a listener; bursts coalesce into a single idempotent transform run that regenerates SDTM from current broker state.

A directory instead of S3, DuckDB instead of Athena. Everything the pipeline produces is a file you can open: Dataset-JSON design domains, raw EDC-style extracts, SDTM datasets. DuckDB gives SQL over those files with zero infrastructure.

Compose instead of Fargate. Single node is the point. Five of the seven services are standard-library Python and build with no network access at all.

Swapping the lake

The lake sits behind a deliberately small contract: writers drop files, readers use three HTTP endpoints. That makes it the natural place to diverge for production: keep the files and add Postgres as a query replica, sync to S3 and query with Athena, or run Trino, ClickHouse, or MinIO. Recipes with the exact steps are in docs/lake-alternatives.md.

The vocabulary is a plugin

Entities are typed with the cr-domain of The Ontology Project: an open, provenance-native reference model for the clinical-research lifecycle, the first anchored T.O.P. domain. Nothing in the engine is clinical-specific: the broker is standard NGSI-LD, the context is a served JSON-LD file, and the projections are readers of broker state. Anchoring another domain (healthcare delivery, commercialization, claims, manufacturing, supply chain) means a new vocabulary and new projections over the same architecture. The NGSI-LD context is vendored and served locally, so the whole stack runs with no internet access.

Adaptive layer in this stack

The adaptive service turns a natural-language clinical statement into an NGSI-LD entity proposal. POST /construct proposes entities; POST /commit executes only when the PNE_ADAPTIVE_EXECUTE gate is enabled, and every commit attempt keeps a local decision record. With an API key it calls the model once; without one it uses a deterministic offline stub. Broker-native temporal queries cover provenance and time for entities this stack writes.