Jupyter Notebooks

Waratah runs a Jupyter service next to the query engine: a jupyter-server sidecar, started with the cluster and reachable through the same port as everything else, at /jupyter on the console origin — http://127.0.0.1:8080/jupyter/. Kernel channels over WebSockets are proxied too, so notebooks run and stay connected like any local Jupyter.

Enabling it

The service is off by default. Turn it on with the flag:

waratah start --data-dir ./waratah-data --with-notebooks

or permanently in notebooks.toml (see Configuration):

enabled = true

It requires the HTTP server — it is part of the web console’s origin, not a separate listener.

What runs where

Path Role

{data_dir}/notebooks

Your notebooks — the Jupyter contents root (--notebooks-dir).

{data_dir}/notebooks-venv

The sidecar’s Python environment, created automatically on first start.

{data_dir}/notebooks-runtime

Generated sidecar config and Jupyter runtime files.

With the memory backend (no --data-dir) the same three directories are created under the working directory instead.

The auto-created environment is pinned and self-contained — Python 3.11 or newer, with Jupyter Server 2.21.1, IPyKernel 6.31.0, rdflib 7.6.0, pySHACL 0.40.1, pandas 3.0.6, and matplotlib 3.11.2. The point of that stack: your notebooks get a working RDF toolkit plus the usual data science one, with no pip install step. To use a Python you control instead, point --notebooks-python (or the python key in notebooks.toml) at an existing interpreter and auto-provisioning is skipped.

Authentication

Jupyter authenticates with a token, minted fresh at startup whenever you have not set one; the /jupyter proxy holds it. To pin your own:

waratah start --with-notebooks --notebooks-token my-secret

Querying Waratah from a notebook

Every kernel starts with two environment variables pointing back at the engine that launched it:

  • WARATAH_URL — this compute node’s HTTP address.

  • WARATAH_REPO — the repository to query (main).

The pinned rdflib talks SPARQL to that endpoint directly:

import os
from rdflib.plugins.stores.sparqlstore import SPARQLStore

endpoint = os.environ["WARATAH_URL"] + "/rdf4j/repositories/main"
store = SPARQLStore(query_endpoint=endpoint)

for row in store.query("SELECT ?s ?p ?o WHERE { ?s ?p ?o } LIMIT 5"):
    print(row)

Housekeeping

  • Kernels idle for more than an hour are culled (cull_idle_secs, default 3600).

  • The sidecar is a child of the cluster: waratah stop ends it along with everything else.