How Rexec Works

This page describes the Rexec client library, scidx-rexec 0.0.9, used through ndp-ep 0.8.1. Items marked Upcoming are not available in this release.

Overview

Rexec turns a decorated Python function into a remote call. You get one dedicated server on the cluster. Each time you call the function, its code and arguments are serialized on your machine, routed through the broker to your server, and run there in a new process.

Output travels back the same way: printed output while the function runs, then the result.

Summary of Execution Semantics

The following nine properties characterize how Rexec executes user code. Each is labelled either Guaranteed, a property of the design that users may rely on, or Limitation, a property of the current release that may change in a future one. Each property is examined in detail in the section indicated.

1. Guaranteed Each user is assigned a single server, shared across all sessions. All notebooks and scripts authenticated under the same user identity dispatch their calls to the same server instance. See Your Environment.
2. Limitation The Python version of the server must match that of the client. Functions are transmitted as Python bytecode, the format of which varies between interpreter versions. The version should therefore be pinned in the requirements specification (e.g., python==3.11). See Your Environment.
3. Guaranteed A call transmits only the function and its arguments. Module-level imports, global variables, and helper functions defined in the calling notebook are not transmitted. They must be defined within the function body or supplied as arguments. See Remote Calls.
4. Limitation Only positional arguments are supported. A call of the form f(16) succeeds, whereas f(x=16) raises TypeError. See Remote Calls.
5. Guaranteed Standard output is streamed during execution. The return value is delivered upon completion. Exceptions raised on the server are reported to the client as RuntimeError. See Live output streaming.
6. Guaranteed Calls are executed sequentially, in order of arrival. A subsequent call does not begin until the preceding call has completed. See State Between Calls.
7. Guaranteed Each call is executed in a new process. In-memory state does not persist between calls. Files written to the server's local disk persist until the server is restarted, replaced, or terminated. See State Between Calls.
8. Limitation Calls are neither retried nor subject to a timeout. If the server or the broker fails during execution, the call blocks until the user interrupts it. See When Things Fail.
9. Limitation A server remains active until it is explicitly terminated. No idle-shutdown mechanism is provided yet. Users should call terminate_rexec_environment() upon completion of their work. See Lifecycle.

Remote Calls

Decorating a function with @remote_func replaces it with a remote call. Calling it serializes the function and its arguments with dill, sends them to the broker with your token, and waits for your server to answer.

Contents of a remote call

On the server, anything the function neither defines nor receives fails with NameError, or ModuleNotFoundError for a local module.

import math
SCALE = 3

@remote_func
def bad(x):
    return math.sqrt(x) * SCALE

@remote_func
def good(x, scale=SCALE):
    import math
    return math.sqrt(x) * scale

good(16)
good(x=16)

1. A module-level import. It is not transmitted to the server.
2. A global variable. It is not transmitted to the server.
3. Raises NameError on the server, since neither math nor SCALE is defined there.
4. The value of SCALE is captured as a default argument when the function is defined.
5. An import within the function body, which therefore executes on the server.
6. Returns 12.0.
7. Raises TypeError: keyword arguments are not supported.

Live output streaming

Output that a remote function writes to standard output or standard error is forwarded to the client as it is produced, rather than collected and delivered when the call completes. Progress messages, logs, and diagnostics therefore appear in the notebook while a long-running function is still executing, and all of a call's output arrives before its return value.

@remote_func
def train(epochs):
    import time
    for epoch in range(epochs):
        time.sleep(5)
        print(f"epoch {epoch}: done")
    return epochs

train(3)

1. Each line appears in the notebook as soon as it is printed, at intervals of about five seconds.
2. Returns 3 after the last line has been printed.

• Each write is forwarded immediately.
• Standard output is printed in the notebook. Standard error is written to the client's standard error.
• Only output written through Python's sys.stdout and sys.stderr is streamed. Output written directly to the process's file descriptors, for example by a subprocess or a compiled extension, is not forwarded and appears only in the server's logs.

What comes back

Where packages must be installed

Where a package must be installed depends on how it is used. Code inside the function runs only on the server. Arguments and return values, however, are serialized on one side and reconstructed on the other, and reconstruction requires importing the package that defines their type.

@remote_func
def sklearn_version():
    import sklearn
    return sklearn.__version__

@remote_func
def load(path):
    import pandas as pd
    return pd.read_csv(path)

1. Executes on the server only; sklearn is not required on the client.
2. Returns a str, so no additional package is required on the client.
3. Returns a DataFrame, so pandas is required on both the server and the client.

Installed versions should also be compatible, since an object pickled by one version of a package is not guaranteed to load in another. Return values must, in addition, be picklable with the standard pickle module: a return value that cannot be pickled is dropped, and the call returns None without raising an error.

Interrupting a call

Stopping the cell (Ctrl-C, or the stop button in Jupyter) cancels the call. The server terminates the process running it (SIGTERM, then SIGKILL after half a second) and KeyboardInterrupt is raised locally.

Your Environment

setup_rexec_environment() asks the Deploy API for a server built from your requirements. The server is identified by your user ID (the sub claim in your token), so you have exactly one, and every session signed in as you uses it.

What's installed

• The official python:<version> container image, for the version you pin.
• scidx-rexec and its dependencies (dill, pyzmq, dxspaces, packaging, cryptography).
• Your requirements, installed with pip install -r.

Requirements file rules

• Pin Python exactly with one python==X.Y line, matching your local Python's major and minor version.
• Use standard pip requirement lines. Direct URL references (pkg @ https://…) are rejected.
• Full-line # comments are fine. Comments at the end of a line are not, and make setup_rexec_environment() raise InvalidRequirement.
• Line order doesn't matter: the same set of lines is the same environment.

# requirements.txt
python==3.11
numpy==2.1.3
pandas>=2.2

What setup_rexec_environment() does

resp = ndp_ep_client.setup_rexec_environment(requirements="requirements.txt")
print(resp)

1. Prints either {"Status": "...", ...} or {"detail": "..."}.

Starting takes a while

setup_rexec_environment() returns as soon as Kubernetes accepts the new server, before the pod has pulled its image and installed your packages. Until the server connects to the broker, calls print Server not ready/available for user <id>. and return None. To wait for it:

import time

@remote_func
def ping():
    return "ready"

while ping() != "ready":
    time.sleep(10)

1. Prints Server not ready… on each attempt until the pod is running.

State Between Calls

Order

Your server runs one call at a time, in the order they arrive. A call from another cell, notebook or script waits until the current one finishes.

Lifecycle

Rexec doesn't report these states yet (there is no status API). They describe what is happening at each stage.

Server states

There is no idle shutdown and no time limit. A server keeps running, and keeps its resource reservation, until you terminate it. Terminating also kills any call it is running.

ndp_ep_client.terminate_rexec_environment()

Call states

When Things Fail

Isolation & Limits

Guaranteed today

• Your server runs in its own Kubernetes namespace, rexec-server-<user-id>, and each call runs in its own process.
• Every message to the broker, and every request to create or delete a server, is checked against the identity provider.
• Your requirements reach the pod as a file for pip install -r, never through a shell command.
• Each server pod has fixed resource limits:

These are the Deploy API's defaults; your sysadmin can change them. You can't request a different size.

Not guaranteed yet

• Network isolation. Rexec doesn't create a network policy for server pods, so what they can reach, inside the cluster and on the internet, depends entirely on the cluster's own policies. By default, Kubernetes allows all of it.
• Unprivileged execution. The server runs as root inside its container.
• Encrypted transport. Traffic between your machine and the broker is plain TCP. Rexec doesn't encrypt your token, code or results on that link.

Not Yet Supported