Remote Execution
GitHub
Remote Execution
GitHub
Rexec is a broker-mediated framework for executing Python code from Jupyter Notebooks or scripts on remote HPC and cloud compute resources, without requiring users to manage the remote environment.
SciDx Remote Execution (Rexec) is a broker-mediated remote code execution system built on ZeroMQ. It allows end users to submit Python workloads from a local Jupyter Notebook or script, route them through a central broker, and have them executed on dedicated per-user server pods running in a remote Kubernetes cluster.
Rexec integrates with the National Data Platform (NDP) Endpoint, which provides dataset discovery, authentication, and the address of the Server Deploy API. They provide a remote computation platform for scientific workflows.
ndp_ep[rexec] Python library
The architecture diagram could not be loaded. See the Architecture page.
The documentation is organized for three audiences. Select the guide that corresponds to your role.
Researchers and data scientists who execute Python workloads on remote clusters through an NDP Endpoint, without managing the underlying infrastructure.
ndp_ep[rexec]Operators of Kubernetes clusters who provide remote execution capability to the users of an NDP Endpoint.
Contributors who study the system internals, modify existing components, or extend Rexec with new capabilities.
The following three services are deployed together with a single Helm chart and together implement the remote execution pipeline. See the Helm Chart reference for configuration details.
A stateless ZeroMQ proxy that routes messages between many clients and
many server pods. Exposes a client-facing NodePort (30001) and
an internal ClusterIP for server pods.
A FastAPI service that provisions and tears down per-user rexec-server
pods in Kubernetes. Accepts Bearer token auth and supports group-based access
control.
The NDP Endpoint API providing dataset search, authentication, and the Rexec spawn API address to clients. Supports CKAN, MongoDB, Kafka, S3, Pelican, and JupyterLab integrations.
The following example assumes access to an NDP Endpoint with remote execution enabled. Install the client library, connect to the endpoint, and submit a function for remote execution.
# Install the NDP Endpoint library with rexec support
pip install ndp_ep[rexec]from ndp_ep import APIClient, remote_func
# 1) Connect to an NDP Endpoint
client = APIClient(
base_url="https://ndp-endpoint.example.org/api",
token="<your-bearer-token>"
)
# 2) Provision the remote environment with your requirements
client.setup_rexec_environment( requirements="input_requirements.txt" )
# 3) Define and submit a remote function
@remote_func
def collatz_steps(n: int) -> int:
steps = 0
while n != 1:
n = n // 2 if n % 2 == 0 else 3 * n + 1
steps += 1
return steps
result = collatz_steps(27)
print(f"collatz_steps(27) = {result}")