NVIDIA DGX
At-Home AI Stack — split-trust shared compute

vLLM is the only clustered service. The model runs with tensor-parallel size 2: spark-01 hosts the Ray master and the vLLM head process; spark-02 hosts a Ray worker. NCCL traffic for tensor-parallel collectives flows over the DAC link (enp1s0f0np0, MTU 9216).

Production model and bootstrap fallback

Step 1a — Build the custom vLLM image on both nodes

The NGC image nvcr.io/nvidia/vllm:26.04-py3 ships without Ray. vLLM 0.19.0+nv26.04 hard-requires Ray for any multi-node inference — torch.distributed.run is not a substitute, because vLLM validates Ray at engine init regardless of launch method. You must build a custom image that adds Ray, on both nodes, before any cluster launch attempt. This is step zero, not optional.

#### Run on both spark-01 AND spark-02

mkdir -p ~/sparky-ai-stack
cat > ~/sparky-ai-stack/Dockerfile.vllm-spark << 'EOF'
FROM nvcr.io/nvidia/vllm:26.04-py3
RUN pip install ray --quiet
EOF

cd ~/sparky-ai-stack
docker build -f Dockerfile.vllm-spark -t vllm-spark:26.04 .

Step 1b — Sync the Hugging Face cache to spark-02 over DAC

Both nodes need the model weights resident locally. Pull on spark-01 first (or use an existing ~/.cache/huggingface), then rsync to spark-02 across the DAC link.

#### spark-01

# Pre-fetch the production 122B GPTQ-Int4 weights on spark-01.
# Run as your normal user (e.g. cameron) — NOT root. See callout below.
hf download Qwen/Qwen3.5-122B-A10B-GPTQ-Int4 \
  --local-dir ~/.cache/huggingface/hub/models--Qwen--Qwen3.5-122B-A10B-GPTQ-Int4

# Rsync the cache to spark-02 over the DAC link
rsync -avh --progress \
  -e "ssh -o StrictHostKeyChecking=accept-new" \
  ~/.cache/huggingface/ \
  YOUR_NODE2_DAC_IP:/home/YOUR_USERNAME/.cache/huggingface/

Step 1c — Startup scripts on each node

Place a startup script on each node so the cluster can be brought up reproducibly.

#### spark-01 — ~/sparky-ai-stack/scripts/vllm-head.sh

mkdir -p ~/sparky-ai-stack/scripts
cat > ~/sparky-ai-stack/scripts/vllm-head.sh << 'EOF'
#!/usr/bin/env bash
set -euo pipefail

# DAC IP of THIS node (spark-01)
HOST_IP="YOUR_NODE1_DAC_IP"
DAC_IFACE="enp1s0f0np0"

docker rm -f vllm-qwen-122b 2>/dev/null || true

docker run -d \
  --name vllm-qwen-122b \
  --network host \
  --ipc host \
  --gpus all \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  -e VLLM_HOST_IP="${HOST_IP}" \
  -e NCCL_SOCKET_IFNAME="${DAC_IFACE}" \
  -e GLOO_SOCKET_IFNAME="${DAC_IFACE}" \
  -e RAY_ADDRESS="${HOST_IP}:6379" \
  -v "${HOME}/.cache/huggingface:/root/.cache/huggingface" \
  -v "${HOME}/sparky-ai-stack/vllm-compile-cache:/root/.cache/vllm/torch_compile_cache" \
  --restart unless-stopped \
  vllm-spark:26.04 \
  bash -lc '
    set -e
    ray start --head \
      --node-ip-address="'"${HOST_IP}"'" \
      --port=6379 \
      --dashboard-host=0.0.0.0 \
      --num-gpus=1 \
      --block &
    # Was 20s previously — bumped to 60s so a simultaneous power-loss reboot of
    # both nodes still gives the worker container time to come up and register
    # before vLLM begins engine init. The until-loop below is the real guard,
    # but the longer initial sleep avoids racing the loop on a cold boot.
    sleep 60
    until ray status >/dev/null 2>&1; do sleep 1; done
    echo "[head] ray up, waiting for worker to join..."
    until [ "$(ray status 2>/dev/null | grep -c '"'"'1.0/1.0 GPU'"'"')" -gt 1 ] || \
          [ "$(ray status 2>/dev/null | grep -c '"'"'GPU'"'"')" -ge 2 ]; do sleep 2; done
    echo "[head] worker joined, starting vllm serve"

    exec vllm serve Qwen/Qwen3.5-122B-A10B-GPTQ-Int4 \
      --served-model-name qwen3.5-122b \
      --dtype auto \
      --gpu-memory-utilization 0.80 \
      --max-model-len 65536 \
      --max-num-batched-tokens 4096 \
      --enable-auto-tool-choice \
      --tool-call-parser hermes \
      --enable-chunked-prefill \
      --enable-prefix-caching \
      --max-num-seqs 32 \
      --host 0.0.0.0 \
      --port 8000 \
      --reasoning-parser qwen3 \
      --default-chat-template-kwargs '"'"'{"enable_thinking": false}'"'"' \
      --tensor-parallel-size 2 \
      --pipeline-parallel-size 1 \
      --distributed-executor-backend ray
  '
EOF
chmod +x ~/sparky-ai-stack/scripts/vllm-head.sh

#### spark-02 — ~/sparky-ai-stack/scripts/vllm-worker.sh

mkdir -p ~/sparky-ai-stack/scripts
cat > ~/sparky-ai-stack/scripts/vllm-worker.sh << 'EOF'
#!/usr/bin/env bash
set -euo pipefail

# DAC IPs
WORKER_IP="YOUR_NODE2_DAC_IP"   # this node (spark-02)
HEAD_IP="YOUR_NODE1_DAC_IP"     # ray head on spark-01
DAC_IFACE="enp1s0f0np0"

docker rm -f vllm-ray-worker 2>/dev/null || true

docker run -d \
  --name vllm-ray-worker \
  --network host \
  --ipc host \
  --gpus all \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  -e VLLM_HOST_IP="${WORKER_IP}" \
  -e NCCL_SOCKET_IFNAME="${DAC_IFACE}" \
  -e GLOO_SOCKET_IFNAME="${DAC_IFACE}" \
  -v "${HOME}/.cache/huggingface:/root/.cache/huggingface" \
  -v "${HOME}/sparky-ai-stack/vllm-compile-cache:/root/.cache/vllm/torch_compile_cache" \
  --restart unless-stopped \
  vllm-spark:26.04 \
  bash -lc '
    # Retry forever until the head is reachable; this is fine and expected
    until ray start \
            --address="'"${HEAD_IP}"':6379" \
            --node-ip-address="'"${WORKER_IP}"'" \
            --num-gpus=1 \
            --block; do
      echo "[worker] head not yet reachable, retrying in 3s..."
      sleep 3
    done
  '
EOF
chmod +x ~/sparky-ai-stack/scripts/vllm-worker.sh

Step 1c-bis — Persist the vLLM compile cache (both nodes)

On first launch, GPTQ-Marlin and torch.compile run JIT compilation that takes 10–20 minutes silently per node — the log will appear to stall after the engine reports weights loaded. The compiled artifacts land at /root/.cache/vllm/torch_compile_cache inside the container. Because the container is ephemeral (we recreate it on every vllm-head.sh / vllm-worker.sh run), that cache is lost every time and the cluster re-pays the full compile cost on every restart.

Fix: mount a host directory into the container so the cache survives container recreation. The two docker run commands above already include the mount:

-v "${HOME}/sparky-ai-stack/vllm-compile-cache:/root/.cache/vllm/torch_compile_cache"

Create the host directory on each node before the first launch, otherwise Docker will create it root-owned and subsequent unprivileged access will fail:

mkdir -p ~/sparky-ai-stack/vllm-compile-cache   # run on BOTH spark-01 and spark-02

Step 1d — Launch order: worker first, then head

Start the worker container first, then the head. This ordering is intentional:

• The worker's ray start --address …:6379 will retry forever until the head's GCS comes up — this is the expected path and is harmless.
• If the head starts first and vLLM begins engine init before the worker has joined Ray, the placement group fires with only one GPU visible and the run hangs in an indefinite allocation failure with no clean error.
• The head script above blocks vllm serve behind a Ray-status check that waits for two GPUs to be registered, which makes the launch idempotent.

#### spark-02 — start the worker

~/sparky-ai-stack/scripts/vllm-worker.sh
docker logs -f vllm-ray-worker   # leave open in another shell

Wait until the worker logs print Ray runtime started and stop spamming "head not yet reachable" retries (the head isn't up yet, but the container will keep trying).

#### spark-01 — start the head

~/sparky-ai-stack/scripts/vllm-head.sh
docker logs -f vllm-qwen-122b

The head will: (1) start Ray as the master, (2) wait for the worker to register a second GPU into the cluster, then (3) start vllm serve. Allow ~5–8 minutes for the GPTQ-Int4 weights to load on both nodes before the engine is ready. On a cold cache the GPTQ-Marlin JIT compile silently adds another 10–20 minutes after weight loading completes — the log will appear to stall after ray_env.py:111 while the kernels compile inside the RayWorkerWrapper. This is expected; see Cluster issues. Compiled artifacts are cached so subsequent startups skip this step.

Step 1e — Verification

#### spark-01 — Ray cluster status

docker exec vllm-qwen-122b ray status

#### spark-01 — model endpoint

curl http://localhost:8000/v1/models
curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3.5-122b","messages":[{"role":"user","content":"hi"}],"max_tokens":16}'

#### Both nodes — GPU residency (GB10 quirk)

GB10 (Grace Blackwell) uses unified memory. The standard --query-gpu=memory.used,memory.total fields return [N/A] on this hardware — that is expected, not a bug. Use plain nvidia-smi and read the Processes section instead:

nvidia-smi   # run on each node

You should see a RayWorkerWrapper (or vllm) process on each node with roughly ~96 GB resident at --gpu-memory-utilization 0.80 with the 122B GPTQ-Int4 model loaded (≈34 GB of weights per node + KV cache).

#### spark-02 — NCCL traffic on the DAC during inference

nload enp1s0f0np0   # watch this while the curl chat-completion above runs

You should see Gb/s-class spikes on the DAC during decoding. If you see traffic on a different interface, NCCL_SOCKET_IFNAME didn't propagate — see the cluster troubleshooting section.

Bootstrap fallback — the 35B FP8 model

The Qwen/Qwen3.6-35B-A3B-FP8 model is the original bootstrap model and remains useful for fast iteration on cluster wiring (Ray, NCCL, DAC) before committing to the longer 122B load. If the cache is already populated with the 35B FP8 weights, swap the vllm serve line in ~/sparky-ai-stack/scripts/vllm-head.sh and the container name (vllm-qwen-122b → vllm-qwen-35b):

vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
  --served-model-name qwen3.6-35b \
  --dtype auto \
  --gpu-memory-utilization 0.80 \
  --max-model-len 131072 \
  --max-num-batched-tokens 4096 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --max-num-seqs 32 \
  --host 0.0.0.0 \
  --port 8000 \
  --reasoning-parser qwen3 \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --tensor-parallel-size 2 \
  --pipeline-parallel-size 1 \
  --distributed-executor-backend ray

Update --served-model-name in the LiteLLM config in Step 02 (and the client's LiteLLM in Step 03) if you fall back. Expect ~97 GB resident per node at FP8 instead of ~96 GB at GPTQ-Int4.

This is your LiteLLM proxy — your master key, your SQLite log corpus, your routing rules. It serves only your application stack on spark-01 (your Open WebUI, your Hermes, your n8n). The client gets their own separate LiteLLM in Step 03.

Your LiteLLM lives on the same node as the vLLM head and points at localhost:8000. The clustered vLLM presents one logical OpenAI-compatible endpoint — LiteLLM doesn't need to know there are two physical nodes behind it.

#### spark-01 — directories and install

mkdir -p ~/sparky-ai-stack/logs
cd ~/sparky-ai-stack

pip3 install litellm[proxy] --break-system-packages
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc && source ~/.bashrc

#### spark-01 — config (single backend pointing at the local clustered vLLM)

cat > ~/sparky-ai-stack/litellm-config.yaml << 'EOF'
model_list:
  - model_name: qwen3.5-122b
    litellm_params:
      model: openai/qwen3.5-122b
      api_base: http://localhost:8000/v1
      api_key: "not-needed"

litellm_settings:
  verbose: true
  database:
    type: sqlite
    path: /home/YOUR_USERNAME/sparky-ai-stack/logs/litellm.db
  log_config:
    level: INFO
    format: json
    filepath: /home/YOUR_USERNAME/sparky-ai-stack/logs/litellm.log

router_settings:
  num_retries: 0
  timeout: 600
EOF

#### spark-01 — systemd service

sudo tee /etc/systemd/system/litellm.service << 'EOF'
[Unit]
Description=LiteLLM Proxy
After=network.target docker.service
Wants=docker.service

[Service]
Type=simple
User=YOUR_USERNAME
WorkingDirectory=/home/YOUR_USERNAME/sparky-ai-stack
ExecStart=/home/YOUR_USERNAME/.local/bin/litellm --config litellm-config.yaml --port 8001 --host 0.0.0.0
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable litellm
sudo systemctl start litellm

#### spark-01 — clean stale override.conf (only if upgrading from a previous setup)

If LiteLLM was previously run with STORE_MODEL_IN_DB=True and a DATABASE_URL in a systemd drop-in, those env vars persist even after you remove them from litellm-config.yaml. LiteLLM will fail to start with httpx.ConnectError against a Postgres that may no longer exist. Reset the drop-in:

sudo mkdir -p /etc/systemd/system/litellm.service.d
sudo bash -c 'cat > /etc/systemd/system/litellm.service.d/override.conf << EOF
[Service]
Environment=PYTHONPATH=/home/YOUR_USERNAME/.local/lib/python3.12/site-packages
EOF'

sudo systemctl daemon-reload
sudo systemctl restart litellm
sudo systemctl status litellm --no-pager

Step 2d — PostgreSQL for the Admin UI and virtual keys (required)

The LiteLLM Admin UI and virtual-key generation both require PostgreSQL. SQLite is not supported — LiteLLM's Prisma schema is hardcoded for PostgreSQL, and the UI route returns table public.LiteLLM_UserTable does not exist if you try to point it at SQLite. The SQLite block in the config above is fine for request logging; it is not a substitute for the metadata DB.

#### spark-01 — bring up the litellm-db postgres container

On spark-01 the postgres container lives in ~/sparky-ai-stack/docker-compose.yml alongside n8n and hermes-webui:

services:
  litellm-db:
    image: postgres:16
    container_name: litellm-db
    restart: unless-stopped
    environment:
      - POSTGRES_USER=litellm
      - POSTGRES_PASSWORD=litellm
      - POSTGRES_DB=litellm
    volumes:
      - litellm_db:/var/lib/postgresql/data
    ports:
      - "5432:5432"

volumes:
  litellm_db:

cd ~/sparky-ai-stack
docker compose up -d litellm-db
docker compose ps litellm-db

#### spark-01 — apply the Prisma schema

Install the Prisma CLI if missing, then push the LiteLLM Prisma schema into the new database. This must run once on spark-01 before LiteLLM starts, otherwise the UI will return table public.LiteLLM_UserTable does not exist.

pip install prisma --break-system-packages

DATABASE_URL="postgresql://litellm:litellm@localhost:5432/litellm" \
  prisma db push \
  --schema /home/YOUR_USERNAME/.local/lib/python3.12/site-packages/litellm/proxy/schema.prisma

#### spark-01 — wire the database into litellm-config.yaml

Add the database_url to general_settings (not at the top level — see troubleshooting) and enable model-in-DB storage so the UI can edit the model list:

general_settings:
  master_key: YOUR_MASTER_KEY
  database_url: "postgresql://litellm:litellm@localhost:5432/litellm"

litellm_settings:
  store_model_in_db: true

Restart and verify:

sudo systemctl restart litellm
sudo systemctl status litellm --no-pager
curl -s http://localhost:8001/health/readiness | head

The client gets their own LiteLLM proxy on spark-02, with their own master key, their own log corpus, and their own routing rules. It points at the shared vLLM endpoint over the DAC link. This is not a copy of spark-01's LiteLLM — it has no shared config, no shared key, no shared logs. The client controls their own master key and never shares it with you.

#### spark-02 — install

mkdir -p ~/sparky-ai-stack/logs
cd ~/sparky-ai-stack

pip3 install litellm[proxy] --break-system-packages
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc && source ~/.bashrc

#### spark-02 — generate the client master key

Run on the client's terminal — store this key only on spark-02:

echo "sk-client-$(openssl rand -hex 16)"

#### spark-02 — config (points at vLLM over the DAC)

cat > ~/sparky-ai-stack/litellm-config.yaml << 'EOF'
model_list:
  - model_name: qwen3.5-122b
    litellm_params:
      model: openai/qwen3.5-122b
      api_base: http://YOUR_NODE1_DAC_IP:8000/v1   # shared vLLM, over DAC
      api_key: "not-needed"

litellm_settings:
  verbose: true
  database:
    type: sqlite
    path: /home/YOUR_USERNAME/sparky-ai-stack/logs/litellm.db
  log_config:
    level: INFO
    format: json
    filepath: /home/YOUR_USERNAME/sparky-ai-stack/logs/litellm.log

general_settings:
  master_key: YOUR_CLIENT_MASTER_KEY    # set to the sk-client-... value above

router_settings:
  num_retries: 0
  timeout: 600
EOF

#### spark-02 — systemd service (independent of spark-01)

sudo tee /etc/systemd/system/litellm.service << 'EOF'
[Unit]
Description=LiteLLM Proxy (client)
After=network.target docker.service
Wants=docker.service

[Service]
Type=simple
User=YOUR_USERNAME
WorkingDirectory=/home/YOUR_USERNAME/sparky-ai-stack
ExecStart=/home/YOUR_USERNAME/.local/bin/litellm --config litellm-config.yaml --port 8001 --host 0.0.0.0
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable litellm
sudo systemctl start litellm
sudo systemctl status litellm --no-pager

Verification — confirm the request hits spark-01:8000

#### spark-02 — local LiteLLM responds with the client key

curl http://localhost:8001/v1/models \
  -H "Authorization: Bearer YOUR_CLIENT_MASTER_KEY"

#### spark-02 — inference round-trip through the shared backend

curl http://localhost:8001/v1/chat/completions \
  -H "Authorization: Bearer YOUR_CLIENT_MASTER_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3.5-122b","messages":[{"role":"user","content":"hi from client"}],"max_tokens":16}'

Step 3d — PostgreSQL for the client Admin UI and virtual keys (required)

Same constraint as Step 02: the LiteLLM Admin UI and virtual-key generation require PostgreSQL — SQLite is not supported because LiteLLM's Prisma schema is hardcoded for PostgreSQL. spark-02 doesn't run a docker-compose stack, so we use a standalone postgres container.

#### spark-02 — standalone postgres container

docker run -d --name litellm-db --restart unless-stopped \
  -e POSTGRES_USER=litellm \
  -e POSTGRES_PASSWORD=litellm \
  -e POSTGRES_DB=litellm \
  -p 5432:5432 \
  -v litellm_db:/var/lib/postgresql/data \
  postgres:16

#### spark-02 — apply the Prisma schema

pip install prisma --break-system-packages

DATABASE_URL="postgresql://litellm:litellm@localhost:5432/litellm" \
  prisma db push \
  --schema /home/YOUR_USERNAME/.local/lib/python3.12/site-packages/litellm/proxy/schema.prisma

#### spark-02 — wire the database into the client litellm-config.yaml

database_url goes under general_settings, alongside the existing master_key. Add store_model_in_db: true under litellm_settings:

general_settings:
  master_key: YOUR_CLIENT_MASTER_KEY
  database_url: "postgresql://litellm:litellm@localhost:5432/litellm"

litellm_settings:
  store_model_in_db: true

sudo systemctl restart litellm
sudo systemctl status litellm --no-pager

Your daily-driver chat interface, owned by you, on your node. It points at your LiteLLM at http://localhost:8001/v1. The client gets their own Open WebUI on spark-02 in Step 05 — neither side can see the other's chat history, knowledge bases, RAG documents, or API keys.

#### spark-01 — directory and run

mkdir -p ~/sparky-ai-stack
cd ~/sparky-ai-stack

docker run -d \
  --name open-webui \
  --restart unless-stopped \
  -p 8080:8080 \
  -v open-webui:/app/backend/data \
  -e OPENAI_API_BASE_URL="http://host.docker.internal:8001/v1" \
  -e OPENAI_API_KEY="YOUR_MASTER_KEY" \
  -e WEBUI_AUTH=True \
  -e ENABLE_OLLAMA_API=False \
  --add-host=host.docker.internal:host-gateway \
  ghcr.io/open-webui/open-webui:main

Visit http://localhost:8080 from spark-01 (or via your tailnet — see Step 06), create your admin account, and confirm in Settings → Connections → OpenAI API:

The client's daily-driver chat interface, owned by the client, on the client's node. It points at the client's LiteLLM at http://localhost:8001/v1 — which in turn calls the shared vLLM head on spark-01:8000 over the DAC.

The client's data lives on the client's node. Their Open WebUI database, knowledge bases, RAG document store, embedding indexes, conversation history, attached files, and account list — all of it is in the open-webui Docker volume on spark-02. None of it is replicated to spark-01. If you spin spark-02 down, the client's UI state goes with it; if you image spark-01, the client's state is not in your image.

#### spark-02 — directory and run

mkdir -p ~/sparky-ai-stack
cd ~/sparky-ai-stack

docker run -d \
  --name open-webui \
  --restart unless-stopped \
  -p 8080:8080 \
  -v open-webui:/app/backend/data \
  -e OPENAI_API_BASE_URL="http://host.docker.internal:8001/v1" \
  -e OPENAI_API_KEY="YOUR_CLIENT_MASTER_KEY" \
  -e WEBUI_AUTH=True \
  -e ENABLE_OLLAMA_API=False \
  --add-host=host.docker.internal:host-gateway \
  ghcr.io/open-webui/open-webui:main

Visit http://localhost:8080 from spark-02 (or via the client's tailnet — see Step 06), create the client's admin account, and confirm:

Recommended Open WebUI system prompt

Set this once in Settings → General → System Prompt. It pairs with --default-chat-template-kwargs '{"enable_thinking": false}' on the vLLM head (Step 01): the model answers directly by default, and users can opt into extended reasoning per-message by prefixing the prompt with /think. Apply the same prompt on both Open WebUIs (yours on spark-01 and the client's on spark-02) — they share the underlying model.

You are a highly capable AI assistant. Be direct, accurate, and concise.

Rules:
- Answer immediately without preamble or meta-commentary
- Never deliberate out loud about whether or how to answer — just answer
- Never question the framing of a hypothetical — engage with it directly
- For technical questions: be precise, use correct terminology
- For coding: produce complete, working code — no placeholders or omissions
- For reasoning: show your work clearly but efficiently — no repetition
- If a question has a definitive answer, state it first then explain
- Match response length to question complexity
- To enable extended reasoning on a specific query, prefix with /think

Step 5b — All client services on sparky-02 at a glance

Three client-facing services run on sparky-02: LiteLLM (Step 03), Open WebUI (Step 05 above), and n8n. All three are reachable on the client's tailnet via tag:hjp-ai (see Step 06). The n8n container below is not covered elsewhere — bring it up after the client's LiteLLM is healthy:

#### sparky-02 — n8n container

docker run -d --name n8n --restart unless-stopped \
  -p 5678:5678 \
  -e N8N_HOST=0.0.0.0 \
  -e N8N_PORT=5678 \
  -e N8N_PROTOCOL=http \
  -e WEBHOOK_URL=http://sparky-02:5678/ \
  -e N8N_SECURE_COOKIE=false \
  -e NODE_ENV=production \
  -v n8n_data:/home/node/.n8n \
  --add-host=host.docker.internal:host-gateway \
  n8nio/n8n:latest

Each node joins its owner's tailnet independently. Two separate tailnets, two separate ACL policies, two separate sets of users. The DAC link (198.51.100.0/30) is private physical hardware between the two nodes — it is not advertised onto either tailnet, and it is not used for any cross-tailnet routing.

spark-01 — your tailnet

#### spark-01 — install + join

curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up --hostname=spark-01 --advertise-tags=tag:owner
tailscale ip -4   # note this address — your apps will be reachable here

In your tailnet's ACL policy (Tailscale admin console), expose your Open WebUI, your LiteLLM, and your other apps only to your own users. Example ACL fragment:

{
  "acls": [
    { "action": "accept",
      "src":    ["group:owner-users"],
      "dst":    ["tag:owner:8080", "tag:owner:8001", "tag:owner:5678", "tag:owner:8787"]
    },
    { "action": "accept",
      "src":    ["group:owner-users"],
      "dst":    ["tag:owner:22"]
    }
  ],
  "tagOwners": {
    "tag:owner": ["YOU@example.com"]
  },
  "groups": {
    "group:owner-users": ["YOU@example.com"]
  }
}

Do NOT add tag:owner:8000 to any allow rule. Port 8000 (vLLM) is unauthenticated and must remain reachable only from localhost (your LiteLLM in Step 02) and the DAC IP 198.51.100.1 (the client's LiteLLM in Step 03).

spark-02 — client's tailnet

#### spark-02 — install + join (client's auth key)

curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up \
  --authkey=<client-auth-key> \
  --hostname=sparky-02 \
  --advertise-tags=tag:hjp-ai

tailscale ip -4   # client's apps will be reachable here, only on their tailnet

The client's tailnet ACLs are theirs to author. Mirror the structure above with their own users and tags. The client should expose :8080 (their Open WebUI), :8001 (their LiteLLM, only if they want programmatic access from elsewhere), and :5678 (their n8n).

#### Client tailnet ACL — replace the default allow-all

The default Tailscale policy allows everything between everyone. Replace it with this grants-based policy. It leaves all non-server devices unrestricted (so the client's existing fleet is untouched), keeps tag:hjp-dell-server open (their existing Dell server stays as-is), and restricts tag:hjp-ai (this node) to only the three service ports — 8001 (LiteLLM), 8080 (Open WebUI), 5678 (n8n).

{
  "grants": [
    {
      "src": ["*"],
      "dst": ["autogroup:member"],
      "ip": ["*"]
    },
    {
      "src": ["*"],
      "dst": ["tag:hjp-dell-server"],
      "ip": ["*"]
    },
    {
      "src": ["*"],
      "dst": ["tag:hjp-ai"],
      "ip": ["tcp:8001", "tcp:8080", "tcp:5678"]
    }
  ],
  "tagOwners": {
    "tag:hjp-ai":         ["autogroup:admin"],
    "tag:hjp-dell-server":["autogroup:admin"]
  }
}

Lock down host firewalls

Tailscale ACLs are policy; the host firewall is enforcement. Apply ufw rules on both nodes so that even if Tailscale is misconfigured, ports 8000 and 8001 cannot leak to the wrong network.

#### spark-01 — host firewall

# Allow from your tailnet (interface tailscale0) and DAC peer only
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow in on tailscale0 to any port 8001 proto tcp           # your LiteLLM
sudo ufw allow in on tailscale0 to any port 8080 proto tcp           # your Open WebUI
sudo ufw allow in on tailscale0 to any port 5678 proto tcp           # your n8n
sudo ufw allow in on tailscale0 to any port 8787 proto tcp           # your Hermes WebUI
sudo ufw allow in on enp1s0f0np0 from YOUR_NODE2_DAC_IP to any port 8000 proto tcp   # client LiteLLM → vLLM only
sudo ufw allow in on enp1s0f0np0 from YOUR_NODE2_DAC_IP to any port 6379 proto tcp   # Ray GCS over DAC
sudo ufw allow ssh                                                   # mgmt LAN ssh
sudo ufw enable

#### spark-02 — host firewall

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow in on tailscale0 to any port 8001 proto tcp           # client LiteLLM
sudo ufw allow in on tailscale0 to any port 8080 proto tcp           # client Open WebUI
sudo ufw allow in on tailscale0 to any port 5678 proto tcp           # client n8n
sudo ufw allow in on enp1s0f0np0 from YOUR_NODE1_DAC_IP                  # NCCL/Ray over DAC
sudo ufw allow ssh
sudo ufw enable

Hermes is your autonomous agent layer (skills, memory, cron, gateways). It runs on your node and talks to your LiteLLM. The client does not get a Hermes — they have their own Open WebUI and n8n on spark-02; if they want an agent they install their own.

#### spark-01 — install

curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc && source ~/.bashrc
hermes --version

#### spark-01 — setup wizard

hermes setup

The wizard writes ~/.hermes/config.yaml. base_url is local — Hermes and your LiteLLM both live on spark-01.

Telegram gateway

Create a bot via @BotFather (/newbot, copy the token) and get your user ID from @userinfobot. Then:

hermes setup gateway   # select Telegram, paste token, paste user ID, choose System service

sudo /home/YOUR_USERNAME/.local/bin/hermes gateway install --system
sudo systemctl start hermes-gateway
sudo systemctl status hermes-gateway --no-pager

Hermes WebUI (Docker, spark-01)

#### spark-01 — docker-compose for hermes-webui

cat > ~/sparky-ai-stack/hermes-webui.yml << 'EOF'
services:
  hermes-webui:
    image: ghcr.io/nesquena/hermes-webui:latest
    container_name: hermes-webui
    restart: unless-stopped
    extra_hosts:
      - "host.docker.internal:host-gateway"
    environment:
      - WANTED_UID=1000
      - WANTED_GID=1000
      - HERMES_WEBUI_STATE_DIR=/home/hermeswebui/.hermes/webui-mvp
    volumes:
      - /home/YOUR_USERNAME/.hermes:/home/hermeswebui/.hermes
      - /home/YOUR_USERNAME/workspace:/workspace
    ports:
      - "8787:8787"
EOF

mkdir -p ~/workspace
docker compose -f ~/sparky-ai-stack/hermes-webui.yml up -d

Your single-instance n8n on spark-01. The client runs their own n8n on spark-02 independently — different workflows, different credentials, different Postgres-vs-SQLite state. Neither side can read the other's flows.

#### spark-01 — docker-compose for n8n

cat > ~/sparky-ai-stack/n8n.yml << 'EOF'
services:
  n8n:
    image: n8nio/n8n:latest
    container_name: n8n
    restart: unless-stopped
    ports:
      - "5678:5678"
    environment:
      - N8N_HOST=0.0.0.0
      - N8N_PORT=5678
      - N8N_PROTOCOL=http
      - WEBHOOK_URL=http://YOUR_TAILNET_HOSTNAME:5678/
      - N8N_SECURE_COOKIE=false
      - NODE_ENV=production
      - GENERIC_TIMEZONE=America/Los_Angeles
    volumes:
      - n8n_data:/home/node/.n8n
    extra_hosts:
      - "host.docker.internal:host-gateway"

volumes:
  n8n_data:
EOF

docker compose -f ~/sparky-ai-stack/n8n.yml up -d

Wire your n8n to your LiteLLM

In n8n, add an OpenAI credential pointing at your LiteLLM (local on spark-01):

Both sides have to pass independently, and each side has to fail in the right places (you should not be able to reach the client's stack, and vice versa). Every item is labeled with the node it should be run from.

Shared compute pool

• spark-01: docker exec vllm-qwen-122b ray status shows 2 nodes and 2.0/2.0 GPU, with both DAC IPs (YOUR_NODE1_DAC_IP and YOUR_NODE2_DAC_IP) listed.
• spark-01: nvidia-smi Processes section shows a RayWorkerWrapper at ~96 GB (122B GPTQ-Int4 at --gpu-memory-utilization 0.80). The --query-gpu=memory.used / memory.total fields return [N/A] on GB10 — expected, use the Processes section instead.
• spark-02: nvidia-smi Processes section shows a RayWorkerWrapper at ~96 GB.
• spark-01: curl http://localhost:8000/v1/models returns qwen3.5-122b (vLLM head).

Your side (spark-01)

• spark-01: curl http://localhost:8001/v1/models -H "Authorization: Bearer YOUR_MASTER_KEY" returns qwen3.5-122b through your LiteLLM.
• spark-01: sudo systemctl status litellm hermes-gateway — both active (running).
• spark-01: docker ps shows vllm-qwen-122b, open-webui, hermes-webui, and n8n all Up.
• spark-01: Open http://localhost:8080 (or your tailnet hostname), send a chat message — completion streams back. Your LiteLLM log records the request.
• spark-01: Send a Telegram message — Hermes responds. End-to-end your-side path: Telegram → Hermes → your LiteLLM → vLLM TP=2.

Client side (spark-02)

• spark-02: curl http://localhost:8001/v1/models -H "Authorization: Bearer YOUR_CLIENT_MASTER_KEY" returns qwen3.5-122b through the client's LiteLLM.
• spark-02: curl http://localhost:8001/v1/chat/completions -H "Authorization: Bearer YOUR_CLIENT_MASTER_KEY" ... returns a completion.
• spark-02: sudo systemctl status litellm — active (running). Logs are written to ~/sparky-ai-stack/logs/litellm.log on spark-02, separate from your logs.
• spark-02: docker ps shows vllm-ray-worker, open-webui, and n8n all Up. (No Hermes container.)
• spark-02: Open http://localhost:8080 (or the client's tailnet hostname), send a chat message — completion streams back. Client's LiteLLM log records the request; your LiteLLM log on spark-01 does not.

Cross-stack isolation (the negative tests)

• From client's tailnet: curl http://YOUR_NODE1_TAILSCALE_IP:8001/v1/models should fail (timeout or "no route to host"). Your LiteLLM is not exposed to the client's tailnet.
• From client's tailnet: curl http://YOUR_NODE1_TAILSCALE_IP:8000/v1/models should fail. The unauthenticated vLLM endpoint is not reachable from the client's tailnet.
• From client's tailnet: curl http://YOUR_NODE1_TAILSCALE_IP:8080 should fail. Your Open WebUI is not reachable from the client's tailnet.
• From your tailnet: curl http://YOUR_NODE2_TAILSCALE_IP:8080 should fail. The client's Open WebUI is not reachable from your tailnet.
• spark-01: tail -f ~/sparky-ai-stack/logs/litellm.log while the client sends a chat message → your log is silent. Client's traffic does not enter your stack.

DAC traffic during inference

• spark-02: nload enp1s0f0np0 spikes into Gb/s during decoding from either side — both your and the client's inference requests traverse the DAC (yours for NCCL collectives, theirs for both the API call to 198.51.100.1:8000 and NCCL).
• Both sides simultaneously: have you and the client send a long prompt at the same time. Both completions should stream concurrently — vLLM's continuous batching handles the overlap.

Config validation

• spark-01 (your Hermes config): YAML validation passes — a parse error will silently break Hermes by falling back to .env:
  bashcopy

  python3 -c "import yaml; yaml.safe_load(open('/home/YOUR_USERNAME/.hermes/config.yaml')); print('YAML valid')"

  Expected output: YAML valid

spark-01 (your node)

spark-02 (client node)

spark-01 — your node

spark-02 — client node (separate ownership)

Backup targets

Two-node-specific failures and their resolutions. Most of these were discovered during live deployment.

Both Open WebUI and n8n have HA modes available, but for a two-node home/lab setup the operational complexity is not worth it. This stack runs them as single instances on spark-02. If you ever want to pursue HA, here are the pointers.

Open WebUI HA

• Switch the open-webui container's storage from a Docker volume to a Postgres backend (env: DATABASE_URL=postgresql://...) and a shared filesystem for uploads and RAG documents.
• Run multiple replicas behind a TCP load balancer. Sticky sessions are recommended for SSE chat streams.
• Postgres can sit on either node; if you put it on spark-01 you'll re-introduce the very latency-disturbance pattern this architecture is designed to avoid. Prefer a third small box or a dedicated HA pair.

n8n HA

• n8n's queue mode requires Postgres for state and Redis for the BullMQ queue. The main container becomes the main instance; one or more worker instances pull jobs off the queue.
• Set EXECUTIONS_MODE=queue, QUEUE_BULL_REDIS_HOST=…, DB_TYPE=postgresdb, and the relevant Postgres env vars on every container. Replicas need the same N8N_ENCRYPTION_KEY.
• For a two-node setup the simplest variant is one main on spark-02 and one worker on a third small box, with Postgres + Redis colocated on the third box.
• Webhook traffic should hit only the main container; long-running executions land on workers transparently.

Desktop and mobile clients run Obsidian with Syncthing for bidirectional vault sync. The vault syncs to a dedicated Proxmox LXC over the local network. When traveling, Tailscale bridges the connection — clients queue changes offline and sync when Tailscale is enabled on both ends.

Desktop Client (Obsidian + Syncthing) ─┐
Mobile Client  (Obsidian + Syncthing) ─┼─→ Syncthing ─→ LXC /vault
                                       ┘         ↓
                                          obsidian-mcp (stdio)
                                                 ↓
                                          supergateway (streamableHttp, port 3000)
                                                 ↓
                                          LiteLLM ([YOUR-AI-SERVER-HOSTNAME])
                                                 ↓
                                    Chat clients / API consumers

LXC setup

A single script provisions the full stack from a fresh Ubuntu 24.04 LXC.

#!/bin/bash
set -e
# ── Configuration ─────────────────────────────────────────────
SYNCTHING_USER="root"   # Change if running as non-root user
# ──────────────────────────────────────────────────────────────

# 1. Install dependencies
apt update
apt install -y curl gpg apt-transport-https

# 2. Install Node.js 22 + npm
curl -fsSL https://deb.nodesource.com/setup_22.x | bash -
apt install -y nodejs

# 3. Install Syncthing
curl -fsSL https://syncthing.net/release-key.gpg | gpg --dearmor -o /usr/share/keyrings/syncthing-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/syncthing-archive-keyring.gpg] https://apt.syncthing.net/ syncthing stable" > /etc/apt/sources.list.d/syncthing.list
apt update
apt install -y syncthing

# 4. Create vault directory with Obsidian config
mkdir -p /vault/.obsidian
echo '{}' > /vault/.obsidian/app.json

# 5. Enable and start Syncthing
systemctl enable syncthing@${SYNCTHING_USER}
systemctl start syncthing@${SYNCTHING_USER}

# 6. Wait for Syncthing config to generate
sleep 8

# 7. Expose Syncthing GUI on all interfaces
CONFIG_PATH=$(find /root -name "config.xml" 2>/dev/null | grep syncthing | head -1)
sed -i 's|<address>127.0.0.1:8384</address>|<address>0.0.0.0:8384</address>|' "$CONFIG_PATH"
systemctl restart syncthing@${SYNCTHING_USER}

# 8. Install obsidian-mcp and supergateway
npm install -g obsidian-mcp supergateway

# 9. Create obsidian-mcp systemd service
cat > /etc/systemd/system/obsidian-mcp.service << 'EOF'
[Unit]
Description=Obsidian MCP Server
After=network.target

[Service]
Type=simple
User=root
ExecStart=supergateway --stdio "obsidian-mcp /vault" --port 3000 --outputTransport streamableHttp --stateful --protocolVersion 2025-11-25
Restart=on-failure
RestartSec=10
MemoryMax=512M

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable obsidian-mcp
systemctl start obsidian-mcp

echo "Syncthing UI: http://$(hostname -I | awk '{print $1}'):8384"
echo "MCP endpoint: http://$(hostname -I | awk '{print $1}'):3000/mcp"

Manual steps after script

1. Open Syncthing UI at http://[YOUR-LXC-IP]:8384
2. Remove the default folder, add /vault, set a GUI password
3. Pair each client device — add LXC device ID in client Syncthing UI, accept the incoming request on the LXC side, share the vault folder
4. In LiteLLM UI → MCP Servers → Add: URL http://[YOUR-LXC-IP]:3000/mcp, transport http, auth None
5. In Hermes ~/.hermes/config.yaml, add under mcp_servers: obsidian: url: http://[YOUR-LXC-IP]:3000/mcp and transport: streamableHttp
6. Mandatory verification — run this before proceeding. If it fails, the MCP server is not reachable and the next steps will not work:
   bashcopy

   hermes mcp test obsidian

   Expected output: ✓ Connected and ✓ Tools discovered: 11. If this fails, check that the obsidian-mcp systemd service is running on the LXC (systemctl status obsidian-mcp), that port 3000 is reachable from the DGX, and that the URL in config.yaml is correct.
7. Update ~/.hermes/skills/note-taking/obsidian/SKILL.md with the correct MCP tool names (create the directory if it doesn't exist). The built-in Hermes obsidian skill will use filesystem commands unless this file explicitly names the MCP tools:
   bashcopy

   mkdir -p ~/.hermes/skills/note-taking/obsidian
   cat > ~/.hermes/skills/note-taking/obsidian/SKILL.md << 'EOF'
   # Obsidian Vault Skill
   
   Always use MCP tools for all vault operations. Never use filesystem commands.
   
   ## Required: discover vault name first
   Always call `list-available-vaults` before any other tool to discover the vault name.
   Do not assume a vault name — always look it up.
   
   ## Available MCP tools
   - list-available-vaults   — discover vault name (call first, every time)
   - search-vault            — full-text search across all notes
   - read-note               — read a specific note by path
   - create-note             — create a new note
   - edit-note               — edit an existing note
   - move-note               — move or rename a note
   - delete-note             — delete a note
   - create-directory        — create a folder in the vault
   - add-tags                — add tags to a note
   - remove-tags             — remove tags from a note
   - rename-tag              — rename a tag across all notes
   EOF

8. Add mcp to the telegram platform toolset in ~/.hermes/config.yaml and restart hermes-gateway

Services reference

LiteLLM MCP configuration

Adds live web search as a callable tool — the AI model can run Brave Search queries during a conversation in response to tool calls from Open WebUI and other clients. LiteLLM spawns the server on demand via stdio using an API key you supply.

Step 1 — Get a Brave Search API key

Go to api.search.brave.com, create a free account, and generate an API key under the Data for AI plan (free tier supports up to 2,000 queries/month).

Step 2 — Confirm Node.js is installed system-wide

The MCP server is launched via npx. If you completed the Playwright MCP setup, Node.js is already installed system-wide and this step is done. Otherwise:

curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash -
sudo apt install -y nodejs

Step 3 — Add Brave Search MCP server in LiteLLM UI

Navigate to http://YOUR_NODE1_MGMT_IP:8001/ui → MCP Servers → Add New MCP Server. (LiteLLM lives on spark-01.)

Set Stdio Configuration (JSON) — replace YOUR_BRAVE_API_KEY with your actual key:

{
  "command": "npx",
  "args": [
    "-y",
    "@modelcontextprotocol/server-brave-search"
  ],
  "env": {
    "BRAVE_API_KEY": "YOUR_BRAVE_API_KEY"
  }
}

Save and confirm Health Status shows Healthy.

Validation

In Open WebUI, send the following prompt:

Expected: the model calls the brave_web_search tool (shown as "Explored" in Open WebUI) and returns a summary drawn from live search results.

Adds browser automation tool use to the stack — the AI model can navigate pages, take screenshots, and scrape content via tool calls in Open WebUI and other clients. LiteLLM spawns a headless Chromium process on demand via stdio; no persistent port is required.

Step 1 — Install Node.js system-wide

LiteLLM runs as a systemd service and does not source .bashrc. Node.js must be installed system-wide so npx is available in LiteLLM's PATH.

curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash -
sudo apt install -y nodejs

Step 2 — Install Playwright MCP Chromium browser

Chrome has no ARM64 build. Use Chromium, installed via the @playwright/mcp package's own browser installer — not via npx playwright install:

npx @playwright/mcp install-browser chromium

Step 3 — Update litellm-config.yaml

Add model_info blocks to all model entries. Without these, LiteLLM does not advertise function calling support and tool calls will not execute.

model_list:
  - model_name: qwen3.5-122b
    litellm_params:
      model: openai/qwen3.5-122b
      api_base: http://localhost:8000/v1
      api_key: "not-needed"
      max_tokens: 8192
    model_info:
      supports_function_calling: true
      supports_tool_choice: true

Restart LiteLLM after saving:

sudo systemctl restart litellm

Step 4 — Add Playwright MCP server in LiteLLM UI

Navigate to http://YOUR_NODE1_MGMT_IP:8001/ui → MCP Servers → Add New MCP Server. (LiteLLM lives on spark-01.)

Set Stdio Configuration (JSON):

{
  "command": "npx",
  "args": [
    "-y",
    "@playwright/mcp@latest",
    "--browser",
    "chromium",
    "--headless"
  ]
}

Save and confirm Health Status shows Healthy.

Validation

In Open WebUI, send the following prompt:

Expected: the model calls the navigate and screenshot tools (shown as "Explored" in Open WebUI) and returns a summary of the page.

The LiteLLM proxy ships with a built-in web UI at /ui. It requires a master key and a PostgreSQL database — SQLite is not supported for the UI auth layer. The following documents every error encountered during setup, in order.

Step 1 — Set a master key

All commands below run on spark-01 (where LiteLLM lives). Add to ~/sparky-ai-stack/litellm-config.yaml:

general_settings:
  master_key: sk-yourkey
  database_url: "postgresql://litellm:litellm@localhost:5432/litellm"

Generate a secure key:

echo "sk-$(openssl rand -hex 16)"

Step 2 — Add PostgreSQL to a compose file on spark-01

Run Postgres on the same node as LiteLLM. Putting it on spark-02 would re-introduce the very latency-disturbance pattern this architecture is designed to avoid. Add to a new ~/sparky-ai-stack/litellm-db.yml on spark-01:

  litellm-db:
    image: postgres:16
    container_name: litellm-db
    restart: unless-stopped
    environment:
      - POSTGRES_USER=litellm
      - POSTGRES_PASSWORD=litellm
      - POSTGRES_DB=litellm
    ports:
      - "5432:5432"
    volumes:
      - litellm_db:/var/lib/postgresql/data

volumes:
  litellm_db:

docker compose up -d litellm-db

Step 3 — Install Prisma

LiteLLM uses Prisma as its database ORM. It is not included in the base pip install litellm package.

pip install prisma --break-system-packages

--break-system-packages bypasses a Python 3.12 restriction that prevents pip from installing into the system Python environment. It is safe on a dedicated AI server where system tools do not depend on conflicting packages.

Step 4 — Generate Prisma binaries

After installing the package, the binaries must be generated from LiteLLM's bundled schema:

cd ~/.local/lib/python3.12/site-packages/litellm/proxy
prisma generate --schema schema.prisma

Step 5 — Apply the database schema

The Postgres database exists but has no tables yet. Push the schema. DATABASE_URL must be passed inline — Prisma reads it directly from the environment, not from litellm-config.yaml.

DATABASE_URL="postgresql://litellm:litellm@localhost:5432/litellm" \
prisma db push --schema schema.prisma

Step 6 — Restart LiteLLM

sudo systemctl daemon-reload
sudo systemctl restart litellm
sudo systemctl status litellm

Errors encountered in order

Accessing the UI

Navigate to http://YOUR_NODE1_MGMT_IP:8001/ui. Username: admin. Password: your master_key value.