NVIDIA DGX
At-Home AI Stack — split-trust shared compute

vLLM is the only clustered service. The model runs with tensor-parallel size 2: spark-01 hosts the Ray master and the vLLM head process; spark-02 hosts a Ray worker. NCCL collectives flow over RoCE/RDMA on the DAC link — not TCP sockets. The earlier hand-built vllm-spark:26.04 approach is obsolete: it ran NCCL over TCP/IP (no /dev/infiniband passthrough, no NCCL_IB_HCA), so steady-state throughput sat at 2–3 tok/s instead of the 45 tok/s the hardware is capable of. The community-maintained eugr/spark-vllm-docker stack ships pre-built SM121a (Blackwell) wheels and wires up the RoCE/infiniband passthrough correctly. Use it.

Production model

Step 1a — Verify RoCE interfaces on both nodes

The DAC link presents two RoCE devices per port-twin. Only the active port (port 0 on each card) is used; the second port stays Down on a standard DAC pair.

#### Run on BOTH spark-01 and spark-02

ibdev2netdev
ls /dev/infiniband/

Step 1b — Clone and build the image (spark-01)

The build pulls the pre-built SM121a (Blackwell) wheels — no compilation required — and the helper script auto-copies the resulting image to spark-02 over the DAC. The --tf5 flag is required for the container variant used by the production recipe. Total build time ~30 minutes (mostly image pull + cross-node copy).

git clone https://github.com/eugr/spark-vllm-docker.git
cd spark-vllm-docker
./build-and-copy.sh --tf5 --copy-to YOUR_NODE2_DAC_IP

Step 1c — Autodiscovery (spark-01)

The discovery step detects the local and peer DAC IPs, identifies the RoCE twins for NCCL_IB_HCA, and writes the values to .env. Every subsequent run-recipe.sh invocation reads from this file — get it right once and the cluster launch becomes a one-liner.

cd ~/spark-vllm-docker
./run-recipe.sh --discover

Accept the prompts. The resulting .env file should contain:

CLUSTER_NODES=YOUR_NODE1_DAC_IP,YOUR_NODE2_DAC_IP
COPY_HOSTS=YOUR_NODE2_DAC_IP
LOCAL_IP=YOUR_NODE1_DAC_IP
ETH_IF=enp1s0f0np0
IB_IF=rocep1s0f0,roceP2p1s0f0
CONTAINER_HF_TOKEN=<your_hf_token>

Append the Hugging Face token after discovery — the discovery prompt won't ask for it:

echo "CONTAINER_HF_TOKEN=YOUR_HF_TOKEN" >> ~/spark-vllm-docker/.env

Step 1d — Download the model and rsync to spark-02 over the DAC

Both nodes need the model weights resident locally. Pull on spark-01 first, then rsync to spark-02 over the DAC — keeps the ~60 GB transfer off the mgmt LAN.

#### spark-01

# Run as YOUR_USERNAME — not root
HF_TOKEN=YOUR_HF_TOKEN hf download Qwen/Qwen3.5-122B-A10B-FP8 \
  --local-dir ~/.cache/huggingface/hub/models--Qwen--Qwen3.5-122B-A10B-FP8/

# Bind explicitly to the local DAC IP so the rsync runs over enp1s0f0np0
rsync -avP \
  -e "ssh -b YOUR_NODE1_DAC_IP" \
  ~/.cache/huggingface/hub/models--Qwen--Qwen3.5-122B-A10B-FP8/ \
  YOUR_NODE2_DAC_IP:~/.cache/huggingface/hub/models--Qwen--Qwen3.5-122B-A10B-FP8/

Step 1e — Launch the cluster

The launch script reads .env, starts the head container on spark-01, SSHes into spark-02 to start the worker container, and forms the Ray cluster. Both containers are started from the same image, so Ray versions are guaranteed to match.

The production recipe is at ~/spark-vllm-docker/recipes/qwen3.5-122b-fp8.yaml. Key parameters baked into the recipe:

model: Qwen/Qwen3.5-122B-A10B-FP8
container: vllm-node-tf5
max_model_len: 262144
max_num_batched_tokens: 8192
mods: mods/fix-qwen3.5-chat-template
env:
  HF_HUB_OFFLINE: 1
  TRANSFORMERS_OFFLINE: 1

These vars prevent vLLM from attempting to reach huggingface.co on every container start. The model weights are already present in the local HF cache — DNS failures after a power outage or network interruption will not block startup.

Additional flags passed to vllm serve by the recipe:

--load-format fastsafetensors --enable-prefix-caching --enable-auto-tool-choice \
--tool-call-parser qwen3_coder --reasoning-parser qwen3 --chat-template unsloth.jinja \
-tp 2 --distributed-executor-backend ray \
--max-num-batched-tokens 8192 \
--default-chat-template-kwargs '{"enable_thinking": false}'

Launch with gpu_memory_utilization passed as a CLI override (the recipe default is overridden here):

cd ~/spark-vllm-docker
./run-recipe.sh qwen3.5-122b-fp8 -d -- \
  --served-model-name qwen3.5-122b \
  --gpu-memory-utilization 0.80

The script starts the Ray head on spark-01, then SSHes into spark-02 to start the worker. It waits until both GPUs register with the Ray cluster before starting vllm serve. Do not interrupt between head start and cluster formation — if you need to restart, stop both containers first (docker stop vllm_node on each node), then re-run the launch command on spark-01.

What run-recipe.sh does automatically:

• Sets NCCL_IB_HCA=rocep1s0f0,roceP2p1s0f0 so NCCL pins to both RoCE twins.
• Passes /dev/infiniband devices into both containers (RDMA verbs + CM).
• Forms the Ray cluster (head on spark-01, worker on spark-02) and waits until both GPUs are registered before starting vllm serve.
• Applies the mods/fix-qwen3.5-chat-template mod and uses fastsafetensors for fast loader I/O.

Watch the launch:

docker logs -f vllm_node

Step 1f — Verification

#### spark-01 — confirm NCCL is using RDMA, not TCP sockets

docker logs vllm_node | grep -E "NET/IB|NET/Socket"

#### spark-01 — Ray cluster status

docker exec vllm_node ray status

#### spark-01 — model endpoint

curl http://localhost:8000/v1/models
curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3.5-122b","messages":[{"role":"user","content":"hi"}],"max_tokens":16}'

#### Both nodes — GPU residency (GB10 quirk)

GB10 uses unified memory. The standard --query-gpu=memory.used,memory.total fields return [N/A] on this hardware — expected. Use plain nvidia-smi and read the Processes section:

nvidia-smi   # run on each node

You should see a vllm / RayWorkerWrapper process on each node with roughly ~61 GB resident at the FP8 weight footprint — leaving headroom for KV cache at gpu_memory_utilization=0.80.

Step 1g — Systemd auto-start

Run only on spark-01 — the cluster launcher SSHes into spark-02 to bring up the worker. The 30-second ExecStartPre sleep gives spark-02 time to finish booting and have Docker running before the head SSHes in.

sudo tee /etc/systemd/system/vllm-cluster.service << 'EOF'
[Unit]
Description=vLLM Cluster - Qwen3.5-122B FP8
After=network-online.target docker.service
Wants=network-online.target
Requires=docker.service

[Service]
Type=oneshot
RemainAfterExit=yes
User=YOUR_USERNAME
WorkingDirectory=/home/YOUR_USERNAME/spark-vllm-docker
ExecStartPre=/bin/sleep 30
ExecStart=/home/YOUR_USERNAME/spark-vllm-docker/run-recipe.sh qwen3.5-122b-fp8 -d -- \
  --served-model-name qwen3.5-122b \
  --gpu-memory-utilization 0.80
TimeoutStartSec=300
TimeoutStopSec=60

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable vllm-cluster.service

To manage the cluster after initial setup:

# Start
sudo systemctl start vllm-cluster.service

# Stop
sudo systemctl stop vllm-cluster.service

# Restart (spark-02 worker restarts automatically via launch-cluster.sh)
sudo systemctl restart vllm-cluster.service

# Logs
sudo journalctl -u vllm-cluster.service -f
docker logs -f vllm_node

Performance results

The performance ceiling is GB10 LPDDR5X memory bandwidth (273 GB/s). With 10B active parameters at INT4 (≈5 GB of weight reads per decoded token), the theoretical maximum is ~55 tok/s per node. 45 tok/s is ~82% of the memory-bandwidth ceiling — there is not much performance left on the table.

Bootstrap fallback — the 35B FP8 model

The 35B FP8 model is still useful for fast iteration on cluster wiring (Ray, NCCL, RoCE) before committing to the longer 122B load. The eugr stack ships a recipe for it; if you've populated the cache, fall back with:

cd ~/spark-vllm-docker
./run-recipe.sh qwen3.6-35b-fp8 -d -- \
  --served-model-name qwen3.6-35b \
  --default-chat-template-kwargs '{"enable_thinking": false}'

Update --served-model-name to match if you fall back — it's the only place the name is set now (there's no LiteLLM config to also update). Open WebUI's model selector on both sides will pick up the new name automatically since it queries vLLM's /v1/models directly.

Your daily-driver chat interface, owned by you, on your node. It points directly at vLLM at http://localhost:8000/v1 — there is no proxy in between. The client gets their own Open WebUI on spark-02 in Step 03 — neither side can see the other's chat history, knowledge bases, RAG documents, or API keys.

#### spark-01 — directory and run

mkdir -p ~/spark-ai-stack
cd ~/spark-ai-stack

docker run -d \
  --name open-webui \
  --restart unless-stopped \
  -p 8080:8080 \
  -v open-webui:/app/backend/data \
  -e OPENAI_API_BASE_URL="http://host.docker.internal:8000/v1" \
  -e OPENAI_API_KEY="not-needed" \
  -e WEBUI_AUTH=True \
  -e ENABLE_OLLAMA_API=False \
  --add-host=host.docker.internal:host-gateway \
  ghcr.io/open-webui/open-webui:main

Visit http://localhost:8080 from spark-01 (or via your tailnet — see Step 04), create your admin account, and confirm in Settings → Connections → OpenAI API:

The client's daily-driver chat interface, owned by the client, on the client's node. It points directly at the shared vLLM head on spark-01:8000 over the DAC link — there is no proxy on either side anymore.

The client's data lives on the client's node. Their Open WebUI database, knowledge bases, RAG document store, embedding indexes, conversation history, attached files, and account list — all of it is in the open-webui Docker volume on spark-02. None of it is replicated to spark-01. If you spin spark-02 down, the client's UI state goes with it; if you image spark-01, the client's state is not in your image.

#### spark-02 — directory and run

mkdir -p ~/spark-ai-stack
cd ~/spark-ai-stack

docker run -d \
  --name open-webui \
  --restart unless-stopped \
  -p 8080:8080 \
  -v open-webui:/app/backend/data \
  -e OPENAI_API_BASE_URL="http://YOUR_NODE1_DAC_IP:8000/v1" \
  -e OPENAI_API_KEY="not-needed" \
  -e WEBUI_AUTH=True \
  -e ENABLE_OLLAMA_API=False \
  --add-host=host.docker.internal:host-gateway \
  ghcr.io/open-webui/open-webui:main

Visit http://localhost:8080 from spark-02 (or via the client's tailnet — see Step 04), create the client's admin account, and confirm:

Recommended Open WebUI system prompt

Set this once in Settings → General → System Prompt. It pairs with --default-chat-template-kwargs '{"enable_thinking": false}' on the vLLM head (Step 01): the model answers directly by default, and users can opt into extended reasoning per-message by prefixing the prompt with /think. Apply the same prompt on both Open WebUIs (yours on spark-01 and the client's on spark-02) — they share the underlying model.

You are a highly capable AI assistant. Be direct, accurate, and concise.

Rules:
- Answer immediately without preamble or meta-commentary
- Never deliberate out loud about whether or how to answer — just answer
- Never question the framing of a hypothetical — engage with it directly
- For technical questions: be precise, use correct terminology
- For coding: produce complete, working code — no placeholders or omissions
- For reasoning: show your work clearly but efficiently — no repetition
- If a question has a definitive answer, state it first then explain
- Match response length to question complexity

Step 3b — All client services on spark-02 at a glance

Two client-facing services run on spark-02: Open WebUI (above) and n8n. Both are reachable on the client's tailnet via tag:client-ai (see Step 04). The n8n container below is not covered elsewhere:

#### spark-02 — n8n container

docker run -d --name n8n --restart unless-stopped \
  -p 5678:5678 \
  -e N8N_HOST=0.0.0.0 \
  -e N8N_PORT=5678 \
  -e N8N_PROTOCOL=http \
  -e WEBHOOK_URL=http://spark-02:5678/ \
  -e N8N_SECURE_COOKIE=false \
  -e NODE_ENV=production \
  -v n8n_data:/home/node/.n8n \
  --add-host=host.docker.internal:host-gateway \
  n8nio/n8n:latest

Each node joins its owner's tailnet independently. Two separate tailnets, two separate ACL policies, two separate sets of users. The DAC link (198.51.100.0/30) is private physical hardware between the two nodes — it is not advertised onto either tailnet, and it is not used for any cross-tailnet routing.

spark-01 — your tailnet

#### spark-01 — install + join

curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up --hostname=spark-01 --advertise-tags=tag:owner
tailscale ip -4   # note this address — your apps will be reachable here

In your tailnet's ACL policy (Tailscale admin console), expose your Open WebUI and your other apps only to your own users. Example ACL fragment:

{
  "acls": [
    { "action": "accept",
      "src":    ["group:owner-users"],
      "dst":    ["tag:owner:8080", "tag:owner:5678", "tag:owner:9119"]
    },
    { "action": "accept",
      "src":    ["group:owner-users"],
      "dst":    ["tag:owner:22"]
    }
  ],
  "tagOwners": {
    "tag:owner": ["YOU@example.com"]
  },
  "groups": {
    "group:owner-users": ["YOU@example.com"]
  }
}

Do NOT add tag:owner:8000 to any allow rule. Port 8000 (vLLM) is unauthenticated and must remain reachable only from localhost (your Open WebUI and Hermes in Steps 02/05) and the DAC IP 198.51.100.1 (the client's Open WebUI in Step 03) — the DAC-scoped ufw rule below enforces the latter; Tailscale never sees port 8000 at all on your side.

spark-02 — client's tailnet

#### spark-02 — install + join (client's auth key)

curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up \
  --authkey=<client-auth-key> \
  --hostname=spark-02 \
  --advertise-tags=tag:client-ai

tailscale ip -4   # client's apps will be reachable here, only on their tailnet

The client's tailnet ACLs are theirs to author. Mirror the structure above with their own users and tags. The client should expose :8080 (their Open WebUI) and :5678 (their n8n) — and nothing else.

#### Client tailnet ACL — replace the default allow-all

The default Tailscale policy allows everything between everyone. Replace it with this grants-based policy. It leaves all non-server devices unrestricted (so the client's existing fleet is untouched), keeps tag:client-server open (their existing Dell server stays as-is), and restricts tag:client-ai (this node) to only its two service ports — 8080 (Open WebUI), 5678 (n8n).

{
  "grants": [
    {
      "src": ["*"],
      "dst": ["autogroup:member"],
      "ip": ["*"]
    },
    {
      "src": ["*"],
      "dst": ["tag:client-server"],
      "ip": ["*"]
    },
    {
      "src": ["*"],
      "dst": ["tag:client-ai"],
      "ip": ["tcp:8080", "tcp:5678"]
    }
  ],
  "tagOwners": {
    "tag:client-ai":         ["autogroup:admin"],
    "tag:client-server":["autogroup:admin"]
  }
}

Lock down host firewalls

Tailscale ACLs are policy; the host firewall is enforcement. Apply ufw rules on both nodes so that even if Tailscale is misconfigured, port 8000 cannot leak to the wrong network. With no proxy layer left, these firewall rules — together with the Tailscale ACLs above — are the entire access-control story for this cluster.

#### spark-01 — host firewall

# Allow from your tailnet (interface tailscale0) and DAC peer only
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow in on tailscale0 to any port 8080 proto tcp           # your Open WebUI
sudo ufw allow in on tailscale0 to any port 5678 proto tcp           # your n8n
sudo ufw allow in on tailscale0 to any port 9119 proto tcp           # your Hermes dashboard
sudo ufw allow in on enp1s0f0np0 from YOUR_NODE2_DAC_IP to any port 8000 proto tcp   # client Open WebUI → vLLM only, over DAC
sudo ufw allow in on enp1s0f0np0 from YOUR_NODE2_DAC_IP to any port 6379 proto tcp   # Ray GCS over DAC
sudo ufw allow ssh                                                   # mgmt LAN ssh
sudo ufw enable

#### spark-02 — host firewall

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow in on tailscale0 to any port 8080 proto tcp           # client Open WebUI
sudo ufw allow in on tailscale0 to any port 5678 proto tcp           # client n8n
sudo ufw allow in on enp1s0f0np0 from YOUR_NODE1_DAC_IP                  # NCCL/Ray over DAC
sudo ufw allow ssh
sudo ufw enable

Hermes is your autonomous agent layer (skills, memory, cron, gateways). It runs on your node and talks directly to vLLM on localhost:8000 — no proxy in between. The client does not get a Hermes — they have their own Open WebUI and n8n on spark-02; if they want an agent they install their own.

#### spark-01 — install

sudo apt install -y ripgrep ffmpeg

curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc && source ~/.bashrc
hermes --version

#### spark-01 — setup wizard

hermes setup

The wizard writes ~/.hermes/config.yaml. base_url is local — Hermes and the vLLM head both live on spark-01.

#### Hermes file locations

Telegram gateway

Create a bot via @BotFather (/newbot, copy the token) and get your user ID from @userinfobot. Then:

hermes setup gateway   # select Telegram, paste token, paste user ID, choose System service

sudo /home/YOUR_USERNAME/.local/bin/hermes gateway install --system
sudo systemctl start hermes-gateway
sudo systemctl status hermes-gateway --no-pager

Built-in dashboard

Hermes ships a built-in web dashboard for managing config, API keys, and sessions. Run it as a systemd service so it starts automatically alongside the gateway.

#### spark-01 — create the dashboard service

sudo tee /etc/systemd/system/hermes-dashboard.service << 'EOF'
[Unit]
Description=Hermes Agent Dashboard
After=network.target hermes-gateway.service
Wants=hermes-gateway.service

[Service]
Type=simple
User=YOUR_USERNAME
ExecStart=/home/YOUR_USERNAME/.local/bin/hermes dashboard --port 9119 --host 0.0.0.0 --insecure --no-open
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable hermes-dashboard
sudo systemctl start hermes-dashboard
sudo systemctl status hermes-dashboard --no-pager

The --insecure flag is required to bind to 0.0.0.0 instead of localhost. Protect port 9119 with Tailscale ACLs (Step 04) — it should only be reachable from your tailnet, not the open internet.

curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash -
sudo apt install -y nodejs
cd ~/.hermes/hermes-agent/web && npm install && npm run build
sudo systemctl restart hermes-dashboard
sudo systemctl status hermes-dashboard --no-pager

Dashboard — basic auth (required for remote access)

Binding the dashboard to 0.0.0.0 exposes it to the local network. Add basic auth so it requires a username and password even inside the tailnet. Set these in ~/.hermes/.env on spark-01 (not in config.yaml):

HERMES_DASHBOARD_BASIC_AUTH_USERNAME=YOUR_DASHBOARD_USERNAME
HERMES_DASHBOARD_BASIC_AUTH_PASSWORD=YOUR_DASHBOARD_PASSWORD

Restart the dashboard after setting these. Verify auth is active:

sudo systemctl restart hermes-dashboard
curl -s http://127.0.0.1:9119/api/status | grep auth

Dashboard — auto-restart on gateway update

After hermes update (or any hermes-gateway restart), the dashboard process goes stale and must be restarted. Wire this up automatically via a systemd override and a sudoers rule — the gateway service runs as your user, which can't call systemctl on system services without sudo.

#### spark-01 — sudoers rule

sudo tee /etc/sudoers.d/hermes-dashboard-restart <<'EOF'
YOUR_USERNAME ALL=(root) NOPASSWD: /usr/bin/systemctl --no-block try-restart hermes-dashboard.service
EOF

#### spark-01 — gateway service override

sudo mkdir -p /etc/systemd/system/hermes-gateway.service.d
sudo tee /etc/systemd/system/hermes-gateway.service.d/override.conf <<'EOF'
[Service]
ExecStartPost=-/usr/bin/sudo -n /usr/bin/systemctl --no-block try-restart hermes-dashboard.service
EOF

sudo systemctl daemon-reload && sudo systemctl restart hermes-gateway

Hermes desktop app (v0.15.2+)

The Hermes desktop app (released June 2026) connects to your dashboard on spark-01 — it does not run its own backend. Without the remote URL set before first launch, the app spins up a local backend, SIGTERMs it, and enters a reset loop.

HERMES_DESKTOP_REMOTE_URL=http://YOUR_NODE1_TAILSCALE_IP:9119
HERMES_DESKTOP_REMOTE_TOKEN=YOUR_DASHBOARD_SESSION_TOKEN

HERMES_DESKTOP_REMOTE_TOKEN must match HERMES_DASHBOARD_SESSION_TOKEN in ~/.hermes/.env on spark-01. The dashboard basic auth credentials (username/password) are separate from this token.

Hermes memory limits

Default memory limits can cause the agent to truncate context or behave inconsistently on long sessions. These values are confirmed stable on this stack:

hermes config set memory_char_limit 6000
hermes config set user_char_limit 3000
sudo systemctl restart hermes-gateway

Keeping Hermes up to date

# spark-01: Update Hermes to the latest release
hermes update
# Pulls latest from main, applies dependency changes, restarts the gateway service.
# Confirm it came back up:
sudo systemctl status hermes-gateway --no-pager

Your single-instance n8n on spark-01. The client runs their own n8n on spark-02 independently — different workflows, different credentials, different Postgres-vs-SQLite state. Neither side can read the other's flows.

#### spark-01 — docker-compose for n8n

cat > ~/spark-ai-stack/n8n.yml << 'EOF'
services:
  n8n:
    image: n8nio/n8n:latest
    container_name: n8n
    restart: unless-stopped
    ports:
      - "5678:5678"
    environment:
      - N8N_HOST=0.0.0.0
      - N8N_PORT=5678
      - N8N_PROTOCOL=http
      - WEBHOOK_URL=http://YOUR_TAILNET_HOSTNAME:5678/
      - N8N_SECURE_COOKIE=false
      - NODE_ENV=production
      - GENERIC_TIMEZONE=America/Los_Angeles
    volumes:
      - n8n_data:/home/node/.n8n
    extra_hosts:
      - "host.docker.internal:host-gateway"

volumes:
  n8n_data:
EOF

docker compose -f ~/spark-ai-stack/n8n.yml up -d

Wire your n8n directly to vLLM

In n8n, add an OpenAI credential pointing directly at vLLM (local on spark-01, no proxy):

Both sides have to pass independently, and each side has to fail in the right places (you should not be able to reach the client's stack, and vice versa). Every item is labeled with the node it should be run from.

Shared compute pool

• spark-01: docker exec vllm_node ray status shows 2 nodes and 2.0/2.0 GPU, with both DAC IPs (YOUR_NODE1_DAC_IP and YOUR_NODE2_DAC_IP) listed.
• spark-01: docker logs vllm_node | grep "NCCL INFO NET/IB" shows both rocep1s0f0 and roceP2p1s0f0 — NCCL is on RoCE/RDMA, not TCP sockets.
• spark-01: nvidia-smi Processes section shows a vllm / RayWorkerWrapper at ~61 GB (122B FP8). The --query-gpu=memory.used / memory.total fields return [N/A] on GB10 — expected, use the Processes section instead.
• spark-02: nvidia-smi Processes section shows a vllm / RayWorkerWrapper at ~61 GB.
• spark-01: curl http://localhost:8000/v1/models returns qwen3.5-122b (vLLM head). A streamed completion runs at ~45 tok/s steady state.

Your side (spark-01)

• spark-01: curl http://localhost:8000/v1/models returns qwen3.5-122b — no auth header needed, this is the vLLM head your Open WebUI and Hermes call directly.
• spark-01: sudo systemctl status hermes-gateway — active (running).
• spark-01: docker ps shows vllm_node, open-webui, and n8n all Up. sudo systemctl status hermes-dashboard is active (running).
• spark-01: Open http://localhost:8080 (or your tailnet hostname), send a chat message — completion streams back directly from vLLM.
• spark-01: Send a Telegram message — Hermes responds. End-to-end your-side path: Telegram → Hermes → vLLM TP=2, directly, no proxy hop.
• spark-01: hermes --version returns a version string. If MCP servers are configured: hermes mcp test <server-name> verifies MCP connectivity.
• spark-01: sudo systemctl status hermes-gateway hermes-dashboard — both active (running).
• spark-01: ~/.hermes/SOUL.md exists (created by installer). ~/.hermes/skills/ is populated with bundled skills seeded by the installer.

Client side (spark-02)

• spark-02: curl http://YOUR_NODE1_DAC_IP:8000/v1/models (over the DAC, no auth header) returns qwen3.5-122b — this is the exact path the client's Open WebUI uses.
• spark-02: curl http://YOUR_NODE1_DAC_IP:8000/v1/chat/completions -H 'Content-Type: application/json' -d '...' returns a completion.
• spark-02: docker ps shows vllm_node (the worker), open-webui, and n8n all Up. (No Hermes container.)
• spark-02: Open http://localhost:8080 (or the client's tailnet hostname), send a chat message — completion streams back over the DAC from spark-01's vLLM head. On spark-01, docker logs --tail 20 vllm_node shows the request arrive.

Cross-stack isolation (the negative tests)

These are now the only tests standing between "working as designed" and "the client has unrestricted access to your node" — there is no proxy-layer key to fall back on if any of these fail. Run every one of them.

• From client's tailnet: curl http://YOUR_NODE1_TAILSCALE_IP:8000/v1/models should fail (timeout or "no route to host"). The unauthenticated vLLM endpoint must not be reachable from the client's tailnet — only from the DAC IP.
• From client's tailnet: curl http://YOUR_NODE1_TAILSCALE_IP:8080 should fail. Your Open WebUI is not reachable from the client's tailnet.
• From your tailnet: curl http://YOUR_NODE2_TAILSCALE_IP:8080 should fail. The client's Open WebUI is not reachable from your tailnet.
• From any host other than spark-02's DAC IP: curl http://YOUR_NODE1_MGMT_IP:8000/v1/models (mgmt LAN, not DAC) should fail. Only the DAC interface, from the client's DAC IP specifically, may reach port 8000.
• spark-01: docker logs --tail 0 -f vllm_node while the client sends a chat message → you should see the request land, since there is no proxy log to check separately anymore. This is the only per-request visibility left on either side — plan accordingly if you need an audit trail.

DAC traffic during inference

• spark-02: nload enp1s0f0np0 spikes into Gb/s during decoding from either side — both your and the client's inference requests traverse the DAC (yours for NCCL collectives, theirs for both the API call to 198.51.100.1:8000 and NCCL).
• Both sides simultaneously: have you and the client send a long prompt at the same time. Both completions should stream concurrently — vLLM's continuous batching handles the overlap.

Config validation

• spark-01 (your Hermes config): YAML validation passes — a parse error will silently break Hermes by falling back to .env:
  bashcopy

  python3 -c "import yaml; yaml.safe_load(open('/home/YOUR_USERNAME/.hermes/config.yaml')); print('YAML valid')"

  Expected output: YAML valid

spark-01 (your node)

spark-02 (client node)

spark-01 — your node

spark-02 — client node (separate ownership)

Backup targets

Two-node-specific failures and their resolutions. Most of these were discovered during live deployment.

docker logs vllm_node | grep -E "NET/IB|NET/Socket"
# NET/IB  = RDMA (good)
# NET/Socket = TCP fallback (bad)

cd ~/spark-vllm-docker
./build-and-copy.sh --tf5 --copy-to YOUR_NODE2_DAC_IP

docker exec vllm_node ray status
# Expected: 2.0/2.0 GPU

docker exec vllm_node ray status
# if Ray isn't running:
cd ~/spark-vllm-docker && ./run-recipe.sh qwen3.5-122b-fp8 -d -- --served-model-name qwen3.5-122b --gpu-memory-utilization 0.80

nc -zv YOUR_NODE1_DAC_IP 6379
# should print: Connection to YOUR_NODE1_DAC_IP 6379 port [tcp/*] succeeded

docker logs vllm_node | grep "NCCL INFO NET/IB"
# Expected: both rocep1s0f0 and roceP2p1s0f0 listed

nvidia-smi
# Read the Processes section — memory.used / memory.total fields return [N/A] on GB10

echo "YOUR_NODE1_MGMT_IP  spark-01" | sudo tee -a /etc/hosts
echo "YOUR_NODE2_MGMT_IP  spark-02" | sudo tee -a /etc/hosts

sudo usermod -aG docker YOUR_USERNAME
newgrp docker

sudo systemctl restart vllm-cluster.service
sudo journalctl -u vllm-cluster.service -f

ping -c 3 YOUR_NODE1_DAC_IP          # DAC link reachable?
nc -zv YOUR_NODE1_DAC_IP 8000        # port open?

docker ps | grep vllm_node           # container running?
sudo ufw status | grep 8000          # firewall rule present?

sudo ufw status verbose | grep 8000
# Only this rule should appear for 8000:
# ALLOW IN  enp1s0f0np0  from YOUR_NODE2_DAC_IP  to any port 8000

cd ~/spark-vllm-docker
./launch-cluster.sh stop
docker rm -f vllm_node 2>/dev/null
ssh YOUR_NODE2_DAC_IP "docker rm -f vllm_node" 2>/dev/null
sudo systemctl restart vllm-cluster.service

docker ps -a | grep vllm_node
docker logs -f vllm_node   # watch for the Uvicorn "running on 0.0.0.0:8000" line
curl http://localhost:8000/v1/models

# confirms vLLM is reachable at all from this node's host
curl http://<url-host>:8000/v1/models

# confirms the Open WebUI container itself can reach it
docker exec open-webui curl -s http://<same-url>/v1/models

Both Open WebUI and n8n have HA modes available, but for a two-node home/lab setup the operational complexity is not worth it. This stack runs them as single instances on spark-02. If you ever want to pursue HA, here are the pointers.

Open WebUI HA

• Switch the open-webui container's storage from a Docker volume to a Postgres backend (env: DATABASE_URL=postgresql://...) and a shared filesystem for uploads and RAG documents.
• Run multiple replicas behind a TCP load balancer. Sticky sessions are recommended for SSE chat streams.
• Postgres can sit on either node; if you put it on spark-01 you'll re-introduce the very latency-disturbance pattern this architecture is designed to avoid. Prefer a third small box or a dedicated HA pair.

n8n HA

• n8n's queue mode requires Postgres for state and Redis for the BullMQ queue. The main container becomes the main instance; one or more worker instances pull jobs off the queue.
• Set EXECUTIONS_MODE=queue, QUEUE_BULL_REDIS_HOST=…, DB_TYPE=postgresdb, and the relevant Postgres env vars on every container. Replicas need the same N8N_ENCRYPTION_KEY.
• For a two-node setup the simplest variant is one main on spark-02 and one worker on a third small box, with Postgres + Redis colocated on the third box.
• Webhook traffic should hit only the main container; long-running executions land on workers transparently.

Obsidian vault sync runs on a dedicated Ubuntu 24.04 LXC (hostname: YOUR_LXC_HOSTNAME, IP: YOUR_LXC_IP) on the Proxmox homelab. Syncthing syncs the vault from Mac/Android to /vault on the LXC. @modelcontextprotocol/server-filesystem exposes the vault as a filesystem MCP server, wrapped by supergateway with streamableHttp transport on port 3000. There is no more central LiteLLM MCP proxy, so this endpoint now has two independent consumers on spark-01: Hermes connects to it directly as a native MCP client, and Your Open WebUI reaches it through mcpo (an MCP-to-OpenAPI bridge), since Open WebUI's Tools UI speaks OpenAPI, not raw MCP.

Obsidian (Mac/Android) <-> Syncthing <-> /vault on LXC <-> server-filesystem <-> supergateway :3000
                                                                                    ├─> Hermes (direct MCP client, spark-01)
                                                                                    └─> mcpo :8000 <-> Your Open WebUI (spark-01)

LXC setup

Run as root on a fresh Ubuntu 24.04 LXC.

#!/bin/bash
set -e

apt update
apt install -y curl gpg apt-transport-https

# Node.js 22
curl -fsSL https://deb.nodesource.com/setup_22.x | bash -
apt install -y nodejs

# Syncthing
curl -fsSL https://syncthing.net/release-key.gpg | \
  gpg --dearmor -o /usr/share/keyrings/syncthing-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/syncthing-archive-keyring.gpg] \
  https://apt.syncthing.net/ syncthing stable" \
  > /etc/apt/sources.list.d/syncthing.list
apt update && apt install -y syncthing

# Vault directory
mkdir -p /vault

# Syncthing — After=network-online.target prevents a race where the service starts
# before the IP is assigned, causing the first vault sync to fail on boot
systemctl enable syncthing@root
systemctl start syncthing@root
sleep 8

# Expose Syncthing GUI on all interfaces
CONFIG_PATH=$(find /root -name "config.xml" 2>/dev/null | grep syncthing | head -1)
sed -i 's|<address>127.0.0.1:8384</address>|<address>0.0.0.0:8384</address>|' "$CONFIG_PATH"
systemctl restart syncthing@root

# @modelcontextprotocol/server-filesystem + supergateway
npm install -g @modelcontextprotocol/server-filesystem supergateway

# systemd service — streamableHttp transport, stateless (no --stateful), protocol 2024-11-05
cat > /etc/systemd/system/obsidian-mcp.service << 'EOF'
[Unit]
Description=Obsidian MCP Server (server-filesystem)
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=root
ExecStart=supergateway \
  --stdio "npx -y @modelcontextprotocol/server-filesystem /vault" \
  --port 3000 \
  --outputTransport streamableHttp
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable obsidian-mcp
systemctl start obsidian-mcp

After the LXC is running

• Configure Syncthing on the LXC (port 8384) to accept a share from your Mac/Android
• Set the shared folder path to /vault
• Connect Mac Obsidian Syncthing client to the LXC Syncthing device

Hermes connection (direct MCP client)

Hermes speaks MCP natively — no proxy needed. Add the server directly to ~/.hermes/config.yaml. The protocol_version must be explicit — server-filesystem ignores 2025-11-25 and the handshake fails silently without it:

mcp_servers:
  - name: obsidian
    url: http://YOUR_LXC_IP:3000/mcp
    transport: streamableHttp
    protocol_version: "2024-11-05"

mcpo — install once on spark-01, for Open WebUI

Open WebUI's Tools UI speaks OpenAPI, not raw MCP, so it needs a bridge. mcpo wraps one or more MCP servers (stdio, SSE, or streamable-http) and re-exposes each as an OpenAPI tool endpoint. Install it once — the Brave Search, Playwright, and Home Assistant sections below all add entries to the same config.json rather than standing up a second instance.

pipx install mcpo   # or: uv tool install mcpo
mkdir -p ~/.mcpo
cat > ~/.mcpo/config.json << 'EOF'
{
  "mcpServers": {
    "obsidian": {
      "type": "streamable-http",
      "url": "http://YOUR_LXC_IP:3000/mcp"
    }
  }
}
EOF

[Unit]
Description=mcpo — MCP-to-OpenAPI bridge for Open WebUI
After=network-online.target

[Service]
Type=simple
User=YOUR_USERNAME
ExecStart=/home/YOUR_USERNAME/.local/bin/mcpo --port 8000 --api-key "YOUR_MCPO_API_KEY" --config /home/YOUR_USERNAME/.mcpo/config.json --hot-reload
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now mcpo

In Your Open WebUI: Settings → Tools → Add Connection, URL http://host.docker.internal:<mcpo-port>/obsidian, API key YOUR_MCPO_API_KEY. Confirm the tool list populates.

Adds live web search as a callable tool — the AI model can run Brave Search queries during a conversation in response to tool calls from Open WebUI and Hermes. Each client spawns the stdio server itself now: mcpo spawns it for Open WebUI, Hermes spawns it directly for its own conversations.

Step 1 — Get a Brave Search API key

Go to api.search.brave.com, create a free account, and generate an API key under the Data for AI plan (free tier supports up to 2,000 queries/month).

Step 2 — Confirm Node.js is installed system-wide

The MCP server is launched via npx. If you completed the Playwright MCP setup, Node.js is already installed system-wide and this step is done. Otherwise:

curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash -
sudo apt install -y nodejs

Step 3 — Add to mcpo's config.json (for Open WebUI)

Add a brave-search entry to ~/.mcpo/config.json on spark-01 (set up in the Obsidian section above), replacing YOUR_BRAVE_API_KEY with your actual key, then restart mcpo — --hot-reload should pick it up automatically, but restart if it doesn't:

{
  "mcpServers": {
    "obsidian": { "...": "unchanged" },
    "brave-search": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-brave-search"],
      "env": { "BRAVE_API_KEY": "YOUR_BRAVE_API_KEY" }
    }
  }
}

sudo systemctl restart mcpo

In Your Open WebUI: Settings → Tools → Add Connection, URL http://host.docker.internal:<mcpo-port>/brave-search, same API key as the Obsidian connection. Confirm the tool list populates.

Step 4 — Add to Hermes (optional, for Hermes' own tool use)

Add the same stdio server directly to ~/.hermes/config.yaml — Hermes spawns it itself, no mcpo involved:

mcp_servers:
  - name: brave-search
    command: npx
    args: ["-y", "@modelcontextprotocol/server-brave-search"]
    env:
      BRAVE_API_KEY: YOUR_BRAVE_API_KEY

Validation

In Open WebUI, send the following prompt:

Expected: the model calls the brave_web_search tool (shown as "Explored" in Open WebUI) and returns a summary drawn from live search results.

Adds browser automation tool use to the stack — the AI model can navigate pages, take screenshots, and scrape content via tool calls in Open WebUI and Hermes. mcpo spawns a headless Chromium process on demand via stdio for Open WebUI; Hermes spawns its own separately. No persistent port is required by the browser process itself.

Step 1 — Install Node.js system-wide

mcpo runs as a systemd service and does not source .bashrc. Node.js must be installed system-wide so npx is available in mcpo's PATH.

curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash -
sudo apt install -y nodejs

Step 2 — Install Playwright MCP Chromium browser

Chrome has no ARM64 build. Use Chromium, installed via the @playwright/mcp package's own browser installer — not via npx playwright install:

npx @playwright/mcp install-browser chromium

Step 3 — Add to mcpo's config.json (for Open WebUI)

Add a playwright entry to ~/.mcpo/config.json on spark-01, then restart mcpo:

{
  "mcpServers": {
    "obsidian": { "...": "unchanged" },
    "brave-search": { "...": "unchanged" },
    "playwright": {
      "command": "npx",
      "args": ["-y", "@playwright/mcp@latest", "--browser", "chromium", "--headless"]
    }
  }
}

sudo systemctl restart mcpo

In Your Open WebUI: Settings → Tools → Add Connection, URL http://host.docker.internal:<mcpo-port>/playwright, same API key as the other mcpo connections.

Step 4 — Add to Hermes (optional, for Hermes' own tool use)

mcp_servers:
  - name: playwright
    command: npx
    args: ["-y", "@playwright/mcp@latest", "--browser", "chromium", "--headless"]

Validation

In Open WebUI, send the following prompt:

Expected: the model calls the navigate and screenshot tools (shown as "Explored" in Open WebUI) and returns a summary of the page.

Home Assistant exposes its own MCP endpoint at /api/mcp using Streamable HTTP transport. This lets Hermes call HA as a tool directly — controlling devices, querying states, reading automations — and lets Your Open WebUI do the same through mcpo.

Step 1 — Enable the integration in Home Assistant

Settings → Devices & Services → Add Integration → search "Model Context Protocol Server"

Step 2 — Generate a long-lived access token

In Home Assistant: Profile → Long-Lived Access Tokens → Create Token. Copy the full token — it is only shown once.

Step 3 — Add to Hermes (direct MCP client)

mcp_servers:
  - name: home-assistant
    url: http://YOUR_HA_IP:8123/api/mcp
    transport: streamableHttp
    headers:
      Authorization: "Bearer YOUR_HA_LONG_LIVED_TOKEN"

Step 4 — Add to mcpo's config.json (for Open WebUI)

{
  "mcpServers": {
    "obsidian": { "...": "unchanged" },
    "brave-search": { "...": "unchanged" },
    "playwright": { "...": "unchanged" },
    "home-assistant": {
      "type": "streamable-http",
      "url": "http://YOUR_HA_IP:8123/api/mcp",
      "headers": { "Authorization": "Bearer YOUR_HA_LONG_LIVED_TOKEN" }
    }
  }
}

sudo systemctl restart mcpo

In Your Open WebUI: Settings → Tools → Add Connection, URL http://host.docker.internal:<mcpo-port>/home-assistant, same API key as the other mcpo connections.