Remote Execution
GitHub ↗
Remote Execution
GitHub ↗
Rexec is a lightweight, broker-mediated framework that lets scientists run code from Jupyter Notebooks or Python scripts on remote HPC/cloud compute resources — without leaving their local environment.
Rexec serves different audiences. Jump to the guide that fits your role.
You're a data scientist or researcher who wants to transparently offload Python workloads to remote clusters through the NDP Endpoint — without managing infrastructure.
ndp_ep[rexec] via pipYou operate a Kubernetes cluster and want to offer remote execution capability to your NDP Endpoint's users.
You want to understand the internals, contribute, or extend Rexec with new capabilities.
SciDx Remote Execution (Rexec) is a broker-mediated remote code execution system built on ZeroMQ. It allows end users to submit Python workloads from a local Jupyter Notebook or script, route them through a central broker, and have them executed on dedicated per-user server pods running in a remote Kubernetes cluster.
Rexec integrates tightly with the National Data Platform (NDP) Endpoint, which provides dataset discovery, authentication, and the spawn API address. Together, they form a complete remote computation substrate for science workflows.
Full diagram on Architecture page
Deployed together with a single Helm chart, these services enable the full remote execution pipeline. View Helm Chart →
A stateless ZeroMQ proxy that routes messages between many clients and
many server pods. Exposes a client-facing NodePort (30001) and
an internal ClusterIP for server pods.
A FastAPI service that provisions and tears down per-user rexec-server
pods in Kubernetes. Accepts Bearer token auth and supports group-based access
control.
The NDP Endpoint API providing dataset search, authentication, and the Rexec spawn API address to clients. Supports CKAN, MongoDB, Kafka, S3, Pelican, and JupyterLab integrations.
If you have access to an NDP Endpoint with rexec support, you're one pip install away.
# Install the NDP Endpoint library with rexec support
pip install ndp_ep[rexec]from ndp_ep import APIClient, remote_func
# 1) Connect to an NDP Endpoint
client = APIClient(
base_url="https://ndp-endpoint.example.org/api",
token="<your-bearer-token>"
)
# 2) Provision the remote environment with your requirements
client.setup_rexec_environment( requirements="input_requirements.txt" )
# 3) Define and submit a remote function
@remote_func
def collatz_steps(n: int) -> int:
steps = 0
while n != 1:
n = n // 2 if n % 2 == 0 else 3 * n + 1
steps += 1
return steps
result = collatz_steps(27)
print(f"Collatz(27) halts in {result} steps")