Sysadmin Guide - Remote Execution

Prerequisites

Kubernetes Cluster

Access to a K8s cluster with kubectl permissions to create namespaces, deployments, services, ClusterRoles, and ServiceAccounts.

Helm ≥ 3.10

Required for the recommended Helm deployment path. Install via brew install helm or the official docs.

Ingress Controller

An Ingress controller (e.g., nginx) must be installed in your cluster to expose the Deploy API and NDP EP API over HTTP(S).

Authentication API

A reachable identity API endpoint that accepts Bearer tokens and returns user info (username, groups). See Auth API section below.

Authentication API

All three components authenticate users against a shared AUTH_API_URL. This URL must accept a GET request with an Authorization: Bearer <token> header and return a JSON body with user information including group membership.

Test IDP available For initial testing, use the NDP test identity provider: https://idp-test.nationaldataplatform.org/temp/information — but use a production IDP for any real deployment.

⚠️

Use the same AUTH_API_URL across all components. Mixing different IDP URLs between the broker, deploy API, and ep-api will cause authentication failures.

What You Deploy

As an NDP Endpoint Provider you are responsible for the three server-side components highlighted below. The client library and the Auth API are external to your infrastructure — you do not manage them.

Rexec Architecture — Sysadmin Components Highlighted

Rexec Broker

ZeroMQ ROUTER–DEALER proxy. Always running. Exposes NodePort 30001 to clients and ClusterIP 5560 to server pods.

Server Deploy API (spawn api)

FastAPI service that spawns and tears down rexec server pods in Kubernetes on demand. Requires kubeconfig access.

NDP EP API

FastAPI gateway that end users call. Authenticates requests, delegates spawn to Deploy API, and returns the broker URL to the client.

Helm Deployment Recommended

The rexec-stack Helm chart is the easiest way to deploy all three Rexec server-side components. It manages three subcharts as one atomic release.

rexec-broker + rexec-server-deployment-api + ndp-ep-api

Add the Helm Repository

helm repo add rexec https://sci-ndp.github.io/rexec
helm repo update

# Verify the chart is available
helm search repo rexec

Configure values.yaml

Download the annotated values template, fill in your environment-specific values, then pass it to helm install with -f my-values.yaml.

Download my-values.yaml template

Edit the key sections:

# my-values.yaml
global:
  # ① Identity provider — must be reachable from all pods
  authApiUrl: https://idp-test.nationaldataplatform.org/temp/information

  # ② (Optional) Group-based access control
  enableGroupBasedAccess: true
  groupNames: /ndp_ep/ep-<your-endpoint-id>

rexec-broker:
  enabled: true
  service:
    external:
      clientNodePort: 30001   # clients connect here
      controlNodePort: 30002   # management port

rexec-server-deployment-api:
  enabled: true
  ingress:
    enabled: true
    className: nginx
    hosts:
      - host: rexec.example.org
        paths:
          - path: /rexec
            pathType: Prefix
  env:
    rootPath: /rexec

ndp-ep-api:
  enabled: true
  ingress:
    enabled: true
    className: nginx
    host: rexec.example.org
    path: /api
  rootPath:
    enabled: true
    value: /api
  env:
    ORGANIZATION: "My University"
    EP_NAME: "My NDP Endpoint"
    REXEC_CONNECTION: True
    REXEC_DEPLOYMENT_API_URL: https://rexec.example.org/rexec

Full values reference See the Helm Chart Reference page for the complete annotated values template covering all NDP EP API integrations (CKAN, MongoDB, Kafka, S3, Pelican, JupyterLab).

Install the Chart

Update subchart dependencies

helm repo update

(Optional) Dry-run — render manifests locally

helm template rexec rexec/rexec-stack \
  -f my-values.yaml \
  --debug

Install or upgrade

helm upgrade --install rexec rexec/rexec-stack \
  -f my-values.yaml \
  -n rexec --create-namespace

Verify Deployment

kubectl get pods -n rexec

NAME                                    READY   STATUS    RESTARTS   AGE
rexec-broker-76c5bbf667-lvxtk           1/1     Running   0          -
rexec-deployment-api-5d554d854c-rdr78   1/1     Running   0          -
rexec-ndp-ep-api-7f9b4f757-j6g9h        1/1     Running   0          -

kubectl get deploy -n rexec

NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
rexec-broker           1/1     1            1           -
rexec-deployment-api   1/1     1            1           -
rexec-ndp-ep-api       1/1     1            1           -

kubectl get svc -n rexec

NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                        AGE
ndp-ep-api-service         ClusterIP   10.152.183.190   <none>        8003/TCP                       -
rexec-broker-external-ip   NodePort    10.152.183.245   <none>        5559:30001/TCP,5561:30002/TCP  -
rexec-broker-internal-ip   ClusterIP   10.152.183.118   <none>        5560/TCP                       -
rexec-deployment-api       ClusterIP   10.152.183.36    <none>        8000/TCP                       -

# Check Deploy API swagger UI
curl http://<ingress-host>/rexec/docs

# Check NDP EP API health
curl https://<ingress-host>/api/docs

Uninstall

helm uninstall rexec -n rexec

Group-Based Access Control

Both the Deploy API and NDP EP API support group-based access control to restrict who can create/modify rexec server instances.

How it works

User authenticates with Bearer token
API validates token against AUTH_API_URL and retrieves user's groups
If enableGroupBasedAccess=true, checks group membership
Access granted only if user's groups overlap with configured groupNames

Group matching is case-insensitive. GET endpoints remain public.

Health Checks

# Broker pods
kubectl get pods -n rexec-broker
kubectl logs -n rexec-broker deployment/rexec-broker

# Deploy API pods
kubectl get pods -n rexec
kubectl logs -n rexec deployment/rexec-server-deployment-api

# NDP EP API
kubectl get pods -n rexec
kubectl logs -n rexec deployment/ndp-ep-api

# Ingress
kubectl get ingress -n rexec

Troubleshooting

Broker NodePort not reachable from clients

Confirm the node's firewall allows inbound TCP on port 30001
Run kubectl get svc -n rexec-broker — ensure NodePort is 30001
Test connectivity: nc -zv <node-ip> 30001

Deploy API returns 403

Check that AUTH_API_URL is reachable from inside the pod
If ACL is enabled, confirm the user belongs to a group in GROUP_NAMES
Inspect pod logs: kubectl logs -n rexec deployment/rexec-server-deployment-api

Pods not starting (ImagePullBackOff)

Verify image registry access from the cluster nodes
Check kubectl describe pod <pod-name> -n rexec for pull errors