NVIDIA DGX
At-Home AI Stack — split-trust shared compute

vLLM is the only clustered service. The model runs with tensor-parallel size 2: spark-01 hosts the Ray master and the vLLM head process; spark-02 hosts a Ray worker. NCCL collectives flow over RoCE/RDMA on the DAC link — not TCP sockets. The earlier hand-built vllm-spark:26.04 approach is obsolete: it ran NCCL over TCP/IP (no /dev/infiniband passthrough, no NCCL_IB_HCA), so steady-state throughput sat at 2–3 tok/s instead of the 45 tok/s the hardware is capable of. The community-maintained eugr/spark-vllm-docker stack ships pre-built SM121a (Blackwell) wheels and wires up the RoCE/infiniband passthrough correctly. Use it.

Production model

Step 1a — Verify RoCE interfaces on both nodes

The DAC link presents two RoCE devices per port-twin. Only the active port (port 0 on each card) is used; the second port stays Down on a standard DAC pair.

#### Run on BOTH spark-01 and spark-02

ibdev2netdev
ls /dev/infiniband/

Step 1b — Clone and build the image (spark-01)

The build pulls the pre-built SM121a (Blackwell) wheels — no compilation required — and the helper script auto-copies the resulting image to spark-02 over the DAC. The --tf5 flag is required for the container variant used by the production recipe. Total build time ~30 minutes (mostly image pull + cross-node copy).

git clone https://github.com/eugr/spark-vllm-docker.git
cd spark-vllm-docker
./build-and-copy.sh --tf5 --copy-to YOUR_NODE2_DAC_IP

Step 1c — Autodiscovery (spark-01)

The discovery step detects the local and peer DAC IPs, identifies the RoCE twins for NCCL_IB_HCA, and writes the values to .env. Every subsequent run-recipe.sh invocation reads from this file — get it right once and the cluster launch becomes a one-liner.

cd ~/spark-vllm-docker
./run-recipe.sh --discover

Accept the prompts. The resulting .env file should contain:

CLUSTER_NODES=YOUR_NODE1_DAC_IP,YOUR_NODE2_DAC_IP
COPY_HOSTS=YOUR_NODE2_DAC_IP
LOCAL_IP=YOUR_NODE1_DAC_IP
ETH_IF=enp1s0f0np0
IB_IF=rocep1s0f0,roceP2p1s0f0
CONTAINER_HF_TOKEN=<your_hf_token>

Append the Hugging Face token after discovery — the discovery prompt won't ask for it:

echo "CONTAINER_HF_TOKEN=YOUR_HF_TOKEN" >> ~/spark-vllm-docker/.env

Step 1d — Download the model and rsync to spark-02 over the DAC

Both nodes need the model weights resident locally. Pull on spark-01 first, then rsync to spark-02 over the DAC — keeps the ~60 GB transfer off the mgmt LAN.

#### spark-01

# Run as YOUR_USERNAME — not root
HF_TOKEN=YOUR_HF_TOKEN hf download Qwen/Qwen3.5-122B-A10B-FP8 \
  --local-dir ~/.cache/huggingface/hub/models--Qwen--Qwen3.5-122B-A10B-FP8/

# Bind explicitly to the local DAC IP so the rsync runs over enp1s0f0np0
rsync -avP \
  -e "ssh -b YOUR_NODE1_DAC_IP" \
  ~/.cache/huggingface/hub/models--Qwen--Qwen3.5-122B-A10B-FP8/ \
  YOUR_NODE2_DAC_IP:~/.cache/huggingface/hub/models--Qwen--Qwen3.5-122B-A10B-FP8/

Step 1e — Launch the cluster

The launch script reads .env, starts the head container on spark-01, SSHes into spark-02 to start the worker container, and forms the Ray cluster. Both containers are started from the same image, so Ray versions are guaranteed to match.

The production recipe is at ~/spark-vllm-docker/recipes/qwen3.5-122b-fp8.yaml. Key parameters baked into the recipe:

model: Qwen/Qwen3.5-122B-A10B-FP8
container: vllm-node-tf5
max_model_len: 262144
max_num_batched_tokens: 8192
mods: mods/fix-qwen3.5-chat-template
env:
  HF_HUB_OFFLINE: 1
  TRANSFORMERS_OFFLINE: 1

These vars prevent vLLM from attempting to reach huggingface.co on every container start. The model weights are already present in the local HF cache — DNS failures after a power outage or network interruption will not block startup.

Additional flags passed to vllm serve by the recipe:

--load-format fastsafetensors --enable-prefix-caching --enable-auto-tool-choice \
--tool-call-parser qwen3_coder --reasoning-parser qwen3 --chat-template unsloth.jinja \
-tp 2 --distributed-executor-backend ray \
--max-num-batched-tokens 8192 \
--default-chat-template-kwargs '{"enable_thinking": false}'

Launch with gpu_memory_utilization passed as a CLI override (the recipe default is overridden here):

cd ~/spark-vllm-docker
./run-recipe.sh qwen3.5-122b-fp8 -d -- \
  --served-model-name qwen3.5-122b \
  --gpu-memory-utilization 0.80

The script starts the Ray head on spark-01, then SSHes into spark-02 to start the worker. It waits until both GPUs register with the Ray cluster before starting vllm serve. Do not interrupt between head start and cluster formation — if you need to restart, stop both containers first (docker stop vllm_node on each node), then re-run the launch command on spark-01.

What run-recipe.sh does automatically:

• Sets NCCL_IB_HCA=rocep1s0f0,roceP2p1s0f0 so NCCL pins to both RoCE twins.
• Passes /dev/infiniband devices into both containers (RDMA verbs + CM).
• Forms the Ray cluster (head on spark-01, worker on spark-02) and waits until both GPUs are registered before starting vllm serve.
• Applies the mods/fix-qwen3.5-chat-template mod and uses fastsafetensors for fast loader I/O.

Watch the launch:

docker logs -f vllm_node

Step 1f — Verification

#### spark-01 — confirm NCCL is using RDMA, not TCP sockets

docker logs vllm_node | grep -E "NET/IB|NET/Socket"

#### spark-01 — Ray cluster status

docker exec vllm_node ray status

#### spark-01 — model endpoint

curl http://localhost:8000/v1/models
curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3.5-122b","messages":[{"role":"user","content":"hi"}],"max_tokens":16}'

#### Both nodes — GPU residency (GB10 quirk)

GB10 uses unified memory. The standard --query-gpu=memory.used,memory.total fields return [N/A] on this hardware — expected. Use plain nvidia-smi and read the Processes section:

nvidia-smi   # run on each node

You should see a vllm / RayWorkerWrapper process on each node with roughly ~61 GB resident at the FP8 weight footprint — leaving headroom for KV cache at gpu_memory_utilization=0.80.

Step 1g — Systemd auto-start

Run only on spark-01 — the cluster launcher SSHes into spark-02 to bring up the worker. The 30-second ExecStartPre sleep gives spark-02 time to finish booting and have Docker running before the head SSHes in.

sudo tee /etc/systemd/system/vllm-cluster.service << 'EOF'
[Unit]
Description=vLLM Cluster - Qwen3.5-122B FP8
After=network-online.target docker.service
Wants=network-online.target
Requires=docker.service

[Service]
Type=oneshot
RemainAfterExit=yes
User=YOUR_USERNAME
WorkingDirectory=/home/YOUR_USERNAME/spark-vllm-docker
ExecStartPre=/bin/sleep 30
ExecStart=/home/YOUR_USERNAME/spark-vllm-docker/run-recipe.sh qwen3.5-122b-fp8 -d -- \
  --served-model-name qwen3.5-122b \
  --gpu-memory-utilization 0.80
TimeoutStartSec=300
TimeoutStopSec=60

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable vllm-cluster.service

To manage the cluster after initial setup:

# Start
sudo systemctl start vllm-cluster.service

# Stop
sudo systemctl stop vllm-cluster.service

# Restart (spark-02 worker restarts automatically via launch-cluster.sh)
sudo systemctl restart vllm-cluster.service

# Logs
sudo journalctl -u vllm-cluster.service -f
docker logs -f vllm_node

Performance results

The performance ceiling is GB10 LPDDR5X memory bandwidth (273 GB/s). With 10B active parameters at INT4 (≈5 GB of weight reads per decoded token), the theoretical maximum is ~55 tok/s per node. 45 tok/s is ~82% of the memory-bandwidth ceiling — there is not much performance left on the table.

Bootstrap fallback — the 35B FP8 model

The 35B FP8 model is still useful for fast iteration on cluster wiring (Ray, NCCL, RoCE) before committing to the longer 122B load. The eugr stack ships a recipe for it; if you've populated the cache, fall back with:

cd ~/spark-vllm-docker
./run-recipe.sh qwen3.6-35b-fp8 -d -- \
  --served-model-name qwen3.6-35b \
  --default-chat-template-kwargs '{"enable_thinking": false}'

Update --served-model-name in the LiteLLM config in Step 02 (and the client's LiteLLM in Step 03) to match if you fall back.

This is your LiteLLM proxy — your master key, your SQLite log corpus, your routing rules. It serves only your application stack on spark-01 (your Open WebUI, your Hermes, your n8n). The client gets their own separate LiteLLM in Step 03.

Your LiteLLM lives on the same node as the vLLM head and points at localhost:8000. The clustered vLLM presents one logical OpenAI-compatible endpoint — LiteLLM doesn't need to know there are two physical nodes behind it.

#### spark-01 — directories and install

mkdir -p ~/spark-ai-stack/logs
cd ~/spark-ai-stack

pip3 install litellm[proxy] --break-system-packages
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc && source ~/.bashrc

#### spark-01 — config (single backend pointing at the local clustered vLLM)

cat > ~/spark-ai-stack/litellm-config.yaml << 'EOF'
model_list:
  # Default — thinking off. Used by all clients unless they explicitly select the reasoning model.
  - model_name: Qwen3.5-122B-Non-Reasoning
    litellm_params:
      model: openai/qwen3.5-122b
      api_base: http://localhost:8000/v1
      api_key: "not-needed"
      max_tokens: 8192
      extra_body:
        chat_template_kwargs:
          enable_thinking: false
    model_info:
      supports_function_calling: true
      supports_tool_choice: true
      max_context_window: 262144
      max_input_tokens: 229376
      max_output_tokens: 8192

  # Opt-in reasoning — user selects this model explicitly for complex tasks.
  - model_name: Qwen3.5-122B-Reasoning
    litellm_params:
      model: openai/qwen3.5-122b
      api_base: http://localhost:8000/v1
      api_key: "not-needed"
      max_tokens: 32768
      extra_body:
        chat_template_kwargs:
          enable_thinking: true
    model_info:
      supports_function_calling: true
      supports_tool_choice: true
      max_context_window: 262144
      max_input_tokens: 229376
      max_output_tokens: 32768

litellm_settings:
  verbose: true
  store_model_in_db: true
  default_system_message: "You are a highly capable AI assistant. Be direct,
    accurate, and concise. Answer immediately without preamble. Never deliberate
    out loud about whether or how to answer. For coding: produce complete,
    working code. State definitive answers first then explain."

router_settings:
  num_retries: 0
  timeout: 600

general_settings:
  master_key: YOUR_MASTER_KEY
  database_url: "postgresql://litellm:litellm@localhost:5432/litellm"
  mcp_settings:
    allow_all_keys: true
EOF

#### spark-01 — systemd service

sudo tee /etc/systemd/system/litellm.service << 'EOF'
[Unit]
Description=LiteLLM Proxy
After=network.target docker.service
Wants=docker.service

[Service]
Type=simple
User=YOUR_USERNAME
WorkingDirectory=/home/YOUR_USERNAME/spark-ai-stack
ExecStart=/home/YOUR_USERNAME/.local/bin/litellm --config litellm-config.yaml --port 8001 --host 0.0.0.0
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable litellm
sudo systemctl start litellm

Step 2d — PostgreSQL for the Admin UI and virtual keys (required)

The LiteLLM Admin UI and virtual key system require PostgreSQL — this is why the step exists. LiteLLM's Prisma schema is hardcoded for PostgreSQL, and the Admin UI, virtual key generation, and model management all write to the PostgreSQL metadata database.

#### spark-01 — bring up the litellm-db postgres container

On spark-01 the postgres container lives in ~/spark-ai-stack/docker-compose.yml alongside n8n:

services:
  litellm-db:
    image: postgres:16
    container_name: litellm-db
    restart: unless-stopped
    environment:
      - POSTGRES_USER=litellm
      - POSTGRES_PASSWORD=litellm
      - POSTGRES_DB=litellm
    volumes:
      - litellm_db:/var/lib/postgresql/data
    ports:
      - "5432:5432"

volumes:
  litellm_db:

cd ~/spark-ai-stack
docker compose up -d litellm-db
docker compose ps litellm-db

#### spark-01 — apply the Prisma schema

Install the Prisma CLI if missing, then push the LiteLLM Prisma schema into the new database. This must run once on spark-01 before LiteLLM starts, otherwise the UI will return table public.LiteLLM_UserTable does not exist.

pip install prisma --break-system-packages

DATABASE_URL="postgresql://litellm:litellm@localhost:5432/litellm" \
  prisma db push \
  --schema /home/YOUR_USERNAME/.local/lib/python3.12/site-packages/litellm/proxy/schema.prisma

#### spark-01 — wire the database into litellm-config.yaml

Add the database_url to general_settings (not at the top level — see troubleshooting) and enable model-in-DB storage so the UI can edit the model list:

general_settings:
  master_key: YOUR_MASTER_KEY
  database_url: "postgresql://litellm:litellm@localhost:5432/litellm"

litellm_settings:
  store_model_in_db: true

Restart and verify:

sudo systemctl restart litellm
sudo systemctl status litellm --no-pager
curl -s http://localhost:8001/health/readiness | head

The client gets their own LiteLLM proxy on spark-02, with their own master key, their own log corpus, and their own routing rules. It points at the shared vLLM endpoint over the DAC link. This is not a copy of spark-01's LiteLLM — it has no shared config, no shared key, no shared logs. The client controls their own master key and never shares it with you.

#### spark-02 — install

mkdir -p ~/spark-ai-stack/logs
cd ~/spark-ai-stack

pip3 install litellm[proxy] --break-system-packages
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc && source ~/.bashrc

#### spark-02 — generate the client master key

Run on the client's terminal — store this key only on spark-02:

echo "sk-client-$(openssl rand -hex 16)"

#### spark-02 — config (points at vLLM over the DAC)

cat > ~/spark-ai-stack/litellm-config.yaml << 'EOF'
model_list:
  - model_name: Qwen3.5-122B-Non-Reasoning
    litellm_params:
      model: openai/qwen3.5-122b
      api_base: http://YOUR_NODE1_DAC_IP:8000/v1
      api_key: "not-needed"
      max_tokens: 8192
      extra_body:
        chat_template_kwargs:
          enable_thinking: false
    model_info:
      supports_function_calling: true
      supports_tool_choice: true
      max_context_window: 262144
      max_input_tokens: 229376
      max_output_tokens: 8192

  - model_name: Qwen3.5-122B-Reasoning
    litellm_params:
      model: openai/qwen3.5-122b
      api_base: http://YOUR_NODE1_DAC_IP:8000/v1
      api_key: "not-needed"
      max_tokens: 32768
      extra_body:
        chat_template_kwargs:
          enable_thinking: true
    model_info:
      supports_function_calling: true
      supports_tool_choice: true
      max_context_window: 262144
      max_input_tokens: 229376
      max_output_tokens: 32768

litellm_settings:
  verbose: true
  database:
    type: sqlite
    path: /home/YOUR_USERNAME/spark-ai-stack/logs/litellm.db
  log_config:
    level: INFO
    format: json
    filepath: /home/YOUR_USERNAME/spark-ai-stack/logs/litellm.log

general_settings:
  master_key: YOUR_CLIENT_MASTER_KEY    # set to the sk-client-... value above

router_settings:
  num_retries: 0
  timeout: 600
EOF

#### spark-02 — systemd service (independent of spark-01)

sudo tee /etc/systemd/system/litellm.service << 'EOF'
[Unit]
Description=LiteLLM Proxy (client)
After=network.target docker.service
Wants=docker.service

[Service]
Type=simple
User=YOUR_USERNAME
WorkingDirectory=/home/YOUR_USERNAME/spark-ai-stack
ExecStart=/home/YOUR_USERNAME/.local/bin/litellm --config litellm-config.yaml --port 8001 --host 0.0.0.0
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable litellm
sudo systemctl start litellm
sudo systemctl status litellm --no-pager

Verification — confirm the request hits spark-01:8000

#### spark-02 — local LiteLLM responds with the client key

curl http://localhost:8001/v1/models \
  -H "Authorization: Bearer YOUR_CLIENT_MASTER_KEY"

#### spark-02 — inference round-trip through the shared backend

curl http://localhost:8001/v1/chat/completions \
  -H "Authorization: Bearer YOUR_CLIENT_MASTER_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3.5-122b","messages":[{"role":"user","content":"hi from client"}],"max_tokens":16}'

Step 3d — PostgreSQL for the client Admin UI and virtual keys (required)

The LiteLLM Admin UI and virtual key system require PostgreSQL — same requirement as Step 02. Set up PostgreSQL on spark-02 so the client's Admin UI, virtual key generation, and model management work correctly. spark-02 doesn't run a docker-compose stack, so we use a standalone postgres container.

#### spark-02 — standalone postgres container

docker run -d --name litellm-db --restart unless-stopped \
  -e POSTGRES_USER=litellm \
  -e POSTGRES_PASSWORD=litellm \
  -e POSTGRES_DB=litellm \
  -p 5432:5432 \
  -v litellm_db:/var/lib/postgresql/data \
  postgres:16

#### spark-02 — apply the Prisma schema

pip install prisma --break-system-packages

DATABASE_URL="postgresql://litellm:litellm@localhost:5432/litellm" \
  prisma db push \
  --schema /home/YOUR_USERNAME/.local/lib/python3.12/site-packages/litellm/proxy/schema.prisma

#### spark-02 — wire the database into the client litellm-config.yaml

database_url goes under general_settings, alongside the existing master_key. Add store_model_in_db: true under litellm_settings:

general_settings:
  master_key: YOUR_CLIENT_MASTER_KEY
  database_url: "postgresql://litellm:litellm@localhost:5432/litellm"

litellm_settings:
  store_model_in_db: true

sudo systemctl restart litellm
sudo systemctl status litellm --no-pager

Your daily-driver chat interface, owned by you, on your node. It points at your LiteLLM at http://localhost:8001/v1. The client gets their own Open WebUI on spark-02 in Step 05 — neither side can see the other's chat history, knowledge bases, RAG documents, or API keys.

#### spark-01 — directory and run

mkdir -p ~/spark-ai-stack
cd ~/spark-ai-stack

docker run -d \
  --name open-webui \
  --restart unless-stopped \
  -p 8080:8080 \
  -v open-webui:/app/backend/data \
  -e OPENAI_API_BASE_URL="http://host.docker.internal:8001/v1" \
  -e OPENAI_API_KEY="YOUR_MASTER_KEY" \
  -e WEBUI_AUTH=True \
  -e ENABLE_OLLAMA_API=False \
  --add-host=host.docker.internal:host-gateway \
  ghcr.io/open-webui/open-webui:main

Visit http://localhost:8080 from spark-01 (or via your tailnet — see Step 06), create your admin account, and confirm in Settings → Connections → OpenAI API:

The client's daily-driver chat interface, owned by the client, on the client's node. It points at the client's LiteLLM at http://localhost:8001/v1 — which in turn calls the shared vLLM head on spark-01:8000 over the DAC.

The client's data lives on the client's node. Their Open WebUI database, knowledge bases, RAG document store, embedding indexes, conversation history, attached files, and account list — all of it is in the open-webui Docker volume on spark-02. None of it is replicated to spark-01. If you spin spark-02 down, the client's UI state goes with it; if you image spark-01, the client's state is not in your image.

#### spark-02 — directory and run

mkdir -p ~/spark-ai-stack
cd ~/spark-ai-stack

docker run -d \
  --name open-webui \
  --restart unless-stopped \
  -p 8080:8080 \
  -v open-webui:/app/backend/data \
  -e OPENAI_API_BASE_URL="http://host.docker.internal:8001/v1" \
  -e OPENAI_API_KEY="YOUR_CLIENT_MASTER_KEY" \
  -e WEBUI_AUTH=True \
  -e ENABLE_OLLAMA_API=False \
  --add-host=host.docker.internal:host-gateway \
  ghcr.io/open-webui/open-webui:main

Visit http://localhost:8080 from spark-02 (or via the client's tailnet — see Step 06), create the client's admin account, and confirm:

Recommended Open WebUI system prompt

Set this once in Settings → General → System Prompt. It pairs with --default-chat-template-kwargs '{"enable_thinking": false}' on the vLLM head (Step 01): the model answers directly by default, and users can opt into extended reasoning per-message by prefixing the prompt with /think. Apply the same prompt on both Open WebUIs (yours on spark-01 and the client's on spark-02) — they share the underlying model.

You are a highly capable AI assistant. Be direct, accurate, and concise.

Rules:
- Answer immediately without preamble or meta-commentary
- Never deliberate out loud about whether or how to answer — just answer
- Never question the framing of a hypothetical — engage with it directly
- For technical questions: be precise, use correct terminology
- For coding: produce complete, working code — no placeholders or omissions
- For reasoning: show your work clearly but efficiently — no repetition
- If a question has a definitive answer, state it first then explain
- Match response length to question complexity

Step 5b — All client services on spark-02 at a glance

Three client-facing services run on spark-02: LiteLLM (Step 03), Open WebUI (Step 05 above), and n8n. All three are reachable on the client's tailnet via tag:client-ai (see Step 06). The n8n container below is not covered elsewhere — bring it up after the client's LiteLLM is healthy:

#### spark-02 — n8n container

docker run -d --name n8n --restart unless-stopped \
  -p 5678:5678 \
  -e N8N_HOST=0.0.0.0 \
  -e N8N_PORT=5678 \
  -e N8N_PROTOCOL=http \
  -e WEBHOOK_URL=http://spark-02:5678/ \
  -e N8N_SECURE_COOKIE=false \
  -e NODE_ENV=production \
  -v n8n_data:/home/node/.n8n \
  --add-host=host.docker.internal:host-gateway \
  n8nio/n8n:latest

Each node joins its owner's tailnet independently. Two separate tailnets, two separate ACL policies, two separate sets of users. The DAC link (198.51.100.0/30) is private physical hardware between the two nodes — it is not advertised onto either tailnet, and it is not used for any cross-tailnet routing.

spark-01 — your tailnet

#### spark-01 — install + join

curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up --hostname=spark-01 --advertise-tags=tag:owner
tailscale ip -4   # note this address — your apps will be reachable here

In your tailnet's ACL policy (Tailscale admin console), expose your Open WebUI, your LiteLLM, and your other apps only to your own users. Example ACL fragment:

{
  "acls": [
    { "action": "accept",
      "src":    ["group:owner-users"],
      "dst":    ["tag:owner:8080", "tag:owner:8001", "tag:owner:5678", "tag:owner:9119"]
    },
    { "action": "accept",
      "src":    ["group:owner-users"],
      "dst":    ["tag:owner:22"]
    }
  ],
  "tagOwners": {
    "tag:owner": ["YOU@example.com"]
  },
  "groups": {
    "group:owner-users": ["YOU@example.com"]
  }
}

Do NOT add tag:owner:8000 to any allow rule. Port 8000 (vLLM) is unauthenticated and must remain reachable only from localhost (your LiteLLM in Step 02) and the DAC IP 198.51.100.1 (the client's LiteLLM in Step 03).

spark-02 — client's tailnet

#### spark-02 — install + join (client's auth key)

curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up \
  --authkey=<client-auth-key> \
  --hostname=spark-02 \
  --advertise-tags=tag:client-ai

tailscale ip -4   # client's apps will be reachable here, only on their tailnet

The client's tailnet ACLs are theirs to author. Mirror the structure above with their own users and tags. The client should expose :8080 (their Open WebUI), :8001 (their LiteLLM, only if they want programmatic access from elsewhere), and :5678 (their n8n).

#### Client tailnet ACL — replace the default allow-all

The default Tailscale policy allows everything between everyone. Replace it with this grants-based policy. It leaves all non-server devices unrestricted (so the client's existing fleet is untouched), keeps tag:client-server open (their existing Dell server stays as-is), and restricts tag:client-ai (this node) to only the three service ports — 8001 (LiteLLM), 8080 (Open WebUI), 5678 (n8n).

{
  "grants": [
    {
      "src": ["*"],
      "dst": ["autogroup:member"],
      "ip": ["*"]
    },
    {
      "src": ["*"],
      "dst": ["tag:client-server"],
      "ip": ["*"]
    },
    {
      "src": ["*"],
      "dst": ["tag:client-ai"],
      "ip": ["tcp:8001", "tcp:8080", "tcp:5678"]
    }
  ],
  "tagOwners": {
    "tag:client-ai":         ["autogroup:admin"],
    "tag:client-server":["autogroup:admin"]
  }
}

Lock down host firewalls

Tailscale ACLs are policy; the host firewall is enforcement. Apply ufw rules on both nodes so that even if Tailscale is misconfigured, ports 8000 and 8001 cannot leak to the wrong network.

#### spark-01 — host firewall

# Allow from your tailnet (interface tailscale0) and DAC peer only
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow in on tailscale0 to any port 8001 proto tcp           # your LiteLLM
sudo ufw allow in on tailscale0 to any port 8080 proto tcp           # your Open WebUI
sudo ufw allow in on tailscale0 to any port 5678 proto tcp           # your n8n
sudo ufw allow in on tailscale0 to any port 9119 proto tcp           # your Hermes dashboard
sudo ufw allow in on enp1s0f0np0 from YOUR_NODE2_DAC_IP to any port 8000 proto tcp   # client LiteLLM → vLLM only
sudo ufw allow in on enp1s0f0np0 from YOUR_NODE2_DAC_IP to any port 6379 proto tcp   # Ray GCS over DAC
sudo ufw allow ssh                                                   # mgmt LAN ssh
sudo ufw enable

#### spark-02 — host firewall

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow in on tailscale0 to any port 8001 proto tcp           # client LiteLLM
sudo ufw allow in on tailscale0 to any port 8080 proto tcp           # client Open WebUI
sudo ufw allow in on tailscale0 to any port 5678 proto tcp           # client n8n
sudo ufw allow in on enp1s0f0np0 from YOUR_NODE1_DAC_IP                  # NCCL/Ray over DAC
sudo ufw allow ssh
sudo ufw enable

Hermes is your autonomous agent layer (skills, memory, cron, gateways). It runs on your node and talks to your LiteLLM. The client does not get a Hermes — they have their own Open WebUI and n8n on spark-02; if they want an agent they install their own.

#### spark-01 — install

sudo apt install -y ripgrep ffmpeg

curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc && source ~/.bashrc
hermes --version

#### spark-01 — setup wizard

hermes setup

The wizard writes ~/.hermes/config.yaml. base_url is local — Hermes and your LiteLLM both live on spark-01.

#### Hermes file locations

Telegram gateway

Create a bot via @BotFather (/newbot, copy the token) and get your user ID from @userinfobot. Then:

hermes setup gateway   # select Telegram, paste token, paste user ID, choose System service

sudo /home/YOUR_USERNAME/.local/bin/hermes gateway install --system
sudo systemctl start hermes-gateway
sudo systemctl status hermes-gateway --no-pager

Built-in dashboard

Hermes ships a built-in web dashboard for managing config, API keys, and sessions. Run it as a systemd service so it starts automatically alongside the gateway.

#### spark-01 — create the dashboard service

sudo tee /etc/systemd/system/hermes-dashboard.service << 'EOF'
[Unit]
Description=Hermes Agent Dashboard
After=network.target hermes-gateway.service
Wants=hermes-gateway.service

[Service]
Type=simple
User=YOUR_USERNAME
ExecStart=/home/YOUR_USERNAME/.local/bin/hermes dashboard --port 9119 --host 0.0.0.0 --insecure --no-open
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable hermes-dashboard
sudo systemctl start hermes-dashboard
sudo systemctl status hermes-dashboard --no-pager

The --insecure flag is required to bind to 0.0.0.0 instead of localhost. Protect port 9119 with Tailscale ACLs (Step 06) — it should only be reachable from your tailnet, not the open internet.

curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash -
sudo apt install -y nodejs
cd ~/.hermes/hermes-agent/web && npm install && npm run build
sudo systemctl restart hermes-dashboard
sudo systemctl status hermes-dashboard --no-pager

Dashboard — basic auth (required for remote access)

Binding the dashboard to 0.0.0.0 exposes it to the local network. Add basic auth so it requires a username and password even inside the tailnet. Set these in ~/.hermes/.env on spark-01 (not in config.yaml):

HERMES_DASHBOARD_BASIC_AUTH_USERNAME=YOUR_DASHBOARD_USERNAME
HERMES_DASHBOARD_BASIC_AUTH_PASSWORD=YOUR_DASHBOARD_PASSWORD

Restart the dashboard after setting these. Verify auth is active:

sudo systemctl restart hermes-dashboard
curl -s http://127.0.0.1:9119/api/status | grep auth

Dashboard — auto-restart on gateway update

After hermes update (or any hermes-gateway restart), the dashboard process goes stale and must be restarted. Wire this up automatically via a systemd override and a sudoers rule — the gateway service runs as your user, which can't call systemctl on system services without sudo.

#### spark-01 — sudoers rule

sudo tee /etc/sudoers.d/hermes-dashboard-restart <<'EOF'
YOUR_USERNAME ALL=(root) NOPASSWD: /usr/bin/systemctl --no-block try-restart hermes-dashboard.service
EOF

#### spark-01 — gateway service override

sudo mkdir -p /etc/systemd/system/hermes-gateway.service.d
sudo tee /etc/systemd/system/hermes-gateway.service.d/override.conf <<'EOF'
[Service]
ExecStartPost=-/usr/bin/sudo -n /usr/bin/systemctl --no-block try-restart hermes-dashboard.service
EOF

sudo systemctl daemon-reload && sudo systemctl restart hermes-gateway

Hermes desktop app (v0.15.2+)

The Hermes desktop app (released June 2026) connects to your dashboard on spark-01 — it does not run its own backend. Without the remote URL set before first launch, the app spins up a local backend, SIGTERMs it, and enters a reset loop.

HERMES_DESKTOP_REMOTE_URL=http://YOUR_NODE1_TAILSCALE_IP:9119
HERMES_DESKTOP_REMOTE_TOKEN=YOUR_DASHBOARD_SESSION_TOKEN

HERMES_DESKTOP_REMOTE_TOKEN must match HERMES_DASHBOARD_SESSION_TOKEN in ~/.hermes/.env on spark-01. The dashboard basic auth credentials (username/password) are separate from this token.

Hermes memory limits

Default memory limits can cause the agent to truncate context or behave inconsistently on long sessions. These values are confirmed stable on this stack:

hermes config set memory_char_limit 6000
hermes config set user_char_limit 3000
sudo systemctl restart hermes-gateway

Keeping Hermes up to date

# spark-01: Update Hermes to the latest release
hermes update
# Pulls latest from main, applies dependency changes, restarts the gateway service.
# Confirm it came back up:
sudo systemctl status hermes-gateway --no-pager

Your single-instance n8n on spark-01. The client runs their own n8n on spark-02 independently — different workflows, different credentials, different Postgres-vs-SQLite state. Neither side can read the other's flows.

#### spark-01 — docker-compose for n8n

cat > ~/spark-ai-stack/n8n.yml << 'EOF'
services:
  n8n:
    image: n8nio/n8n:latest
    container_name: n8n
    restart: unless-stopped
    ports:
      - "5678:5678"
    environment:
      - N8N_HOST=0.0.0.0
      - N8N_PORT=5678
      - N8N_PROTOCOL=http
      - WEBHOOK_URL=http://YOUR_TAILNET_HOSTNAME:5678/
      - N8N_SECURE_COOKIE=false
      - NODE_ENV=production
      - GENERIC_TIMEZONE=America/Los_Angeles
    volumes:
      - n8n_data:/home/node/.n8n
    extra_hosts:
      - "host.docker.internal:host-gateway"

volumes:
  n8n_data:
EOF

docker compose -f ~/spark-ai-stack/n8n.yml up -d

Wire your n8n to your LiteLLM

In n8n, add an OpenAI credential pointing at your LiteLLM (local on spark-01):

Both sides have to pass independently, and each side has to fail in the right places (you should not be able to reach the client's stack, and vice versa). Every item is labeled with the node it should be run from.

Shared compute pool

• spark-01: docker exec vllm_node ray status shows 2 nodes and 2.0/2.0 GPU, with both DAC IPs (YOUR_NODE1_DAC_IP and YOUR_NODE2_DAC_IP) listed.
• spark-01: docker logs vllm_node | grep "NCCL INFO NET/IB" shows both rocep1s0f0 and roceP2p1s0f0 — NCCL is on RoCE/RDMA, not TCP sockets.
• spark-01: nvidia-smi Processes section shows a vllm / RayWorkerWrapper at ~61 GB (122B FP8). The --query-gpu=memory.used / memory.total fields return [N/A] on GB10 — expected, use the Processes section instead.
• spark-02: nvidia-smi Processes section shows a vllm / RayWorkerWrapper at ~61 GB.
• spark-01: curl http://localhost:8000/v1/models returns qwen3.5-122b (vLLM head). A streamed completion runs at ~45 tok/s steady state.

Your side (spark-01)

• spark-01: curl http://localhost:8001/v1/models -H "Authorization: Bearer YOUR_MASTER_KEY" returns qwen3.5-122b through your LiteLLM.
• spark-01: sudo systemctl status litellm hermes-gateway — both active (running).
• spark-01: docker ps shows vllm_node, open-webui, and n8n all Up. sudo systemctl status hermes-dashboard is active (running).
• spark-01: Open http://localhost:8080 (or your tailnet hostname), send a chat message — completion streams back. Your LiteLLM log records the request.
• spark-01: Send a Telegram message — Hermes responds. End-to-end your-side path: Telegram → Hermes → your LiteLLM → vLLM TP=2. Confirm in LiteLLM logs that the request arrived on the Hermes virtual key, not the master key.
• spark-01: hermes --version returns a version string. If MCP servers are configured: hermes mcp test <server-name> verifies MCP connectivity.
• spark-01: sudo systemctl status hermes-gateway hermes-dashboard — both active (running).
• spark-01: ~/.hermes/SOUL.md exists (created by installer). ~/.hermes/skills/ is populated with bundled skills seeded by the installer.

Client side (spark-02)

• spark-02: curl http://localhost:8001/v1/models -H "Authorization: Bearer YOUR_CLIENT_MASTER_KEY" returns qwen3.5-122b through the client's LiteLLM.
• spark-02: curl http://localhost:8001/v1/chat/completions -H "Authorization: Bearer YOUR_CLIENT_MASTER_KEY" ... returns a completion.
• spark-02: sudo systemctl status litellm — active (running). Logs are written to ~/spark-ai-stack/logs/litellm.log on spark-02, separate from your logs.
• spark-02: docker ps shows vllm_node (the worker), open-webui, and n8n all Up. (No Hermes container.)
• spark-02: Open http://localhost:8080 (or the client's tailnet hostname), send a chat message — completion streams back. Client's LiteLLM log records the request; your LiteLLM log on spark-01 does not.

Cross-stack isolation (the negative tests)

• From client's tailnet: curl http://YOUR_NODE1_TAILSCALE_IP:8001/v1/models should fail (timeout or "no route to host"). Your LiteLLM is not exposed to the client's tailnet.
• From client's tailnet: curl http://YOUR_NODE1_TAILSCALE_IP:8000/v1/models should fail. The unauthenticated vLLM endpoint is not reachable from the client's tailnet.
• From client's tailnet: curl http://YOUR_NODE1_TAILSCALE_IP:8080 should fail. Your Open WebUI is not reachable from the client's tailnet.
• From your tailnet: curl http://YOUR_NODE2_TAILSCALE_IP:8080 should fail. The client's Open WebUI is not reachable from your tailnet.
• spark-01: tail -f ~/spark-ai-stack/logs/litellm.log while the client sends a chat message → your log is silent. Client's traffic does not enter your stack.

DAC traffic during inference

• spark-02: nload enp1s0f0np0 spikes into Gb/s during decoding from either side — both your and the client's inference requests traverse the DAC (yours for NCCL collectives, theirs for both the API call to 198.51.100.1:8000 and NCCL).
• Both sides simultaneously: have you and the client send a long prompt at the same time. Both completions should stream concurrently — vLLM's continuous batching handles the overlap.

Config validation

• spark-01 (your Hermes config): YAML validation passes — a parse error will silently break Hermes by falling back to .env:
  bashcopy

  python3 -c "import yaml; yaml.safe_load(open('/home/YOUR_USERNAME/.hermes/config.yaml')); print('YAML valid')"

  Expected output: YAML valid

spark-01 (your node)

spark-02 (client node)

spark-01 — your node

spark-02 — client node (separate ownership)

Backup targets

Two-node-specific failures and their resolutions. Most of these were discovered during live deployment.

docker logs vllm_node | grep -E "NET/IB|NET/Socket"
# NET/IB  = RDMA (good)
# NET/Socket = TCP fallback (bad)

cd ~/spark-vllm-docker
./build-and-copy.sh --tf5 --copy-to YOUR_NODE2_DAC_IP

docker exec vllm_node ray status
# Expected: 2.0/2.0 GPU

docker exec vllm_node ray status
# if Ray isn't running:
cd ~/spark-vllm-docker && ./run-recipe.sh qwen3.5-122b-fp8 -d -- --served-model-name qwen3.5-122b --gpu-memory-utilization 0.80

nc -zv YOUR_NODE1_DAC_IP 6379
# should print: Connection to YOUR_NODE1_DAC_IP 6379 port [tcp/*] succeeded

docker logs vllm_node | grep "NCCL INFO NET/IB"
# Expected: both rocep1s0f0 and roceP2p1s0f0 listed

nvidia-smi
# Read the Processes section — memory.used / memory.total fields return [N/A] on GB10

echo "YOUR_NODE1_MGMT_IP  spark-01" | sudo tee -a /etc/hosts
echo "YOUR_NODE2_MGMT_IP  spark-02" | sudo tee -a /etc/hosts

sudo usermod -aG docker YOUR_USERNAME
newgrp docker

sudo systemctl restart vllm-cluster.service
sudo journalctl -u vllm-cluster.service -f

ping -c 3 YOUR_NODE1_DAC_IP          # DAC link reachable?
nc -zv YOUR_NODE1_DAC_IP 8000        # port open?

docker ps | grep vllm_node           # container running?
sudo ufw status | grep 8000          # firewall rule present?

sudo ufw status verbose | grep 8000
# Only this rule should appear for 8000:
# ALLOW IN  enp1s0f0np0  from YOUR_NODE2_DAC_IP  to any port 8000

sudo ufw status verbose | grep 8001
# Should only show: ALLOW IN  tailscale0  to any port 8001

DATABASE_URL="postgresql://litellm:litellm@localhost:5432/litellm" \
  prisma db push \
  --schema /home/YOUR_USERNAME/.local/lib/python3.12/site-packages/litellm/proxy/schema.prisma

sudo systemctl restart litellm

general_settings:
  master_key: YOUR_MASTER_KEY
  database_url: "postgresql://litellm:litellm@localhost:5432/litellm"

sudo systemctl restart litellm
journalctl -u litellm -n 50 --no-pager
# Look for: Successfully connected to postgres DB

sudo systemctl restart litellm

Both Open WebUI and n8n have HA modes available, but for a two-node home/lab setup the operational complexity is not worth it. This stack runs them as single instances on spark-02. If you ever want to pursue HA, here are the pointers.

Open WebUI HA

• Switch the open-webui container's storage from a Docker volume to a Postgres backend (env: DATABASE_URL=postgresql://...) and a shared filesystem for uploads and RAG documents.
• Run multiple replicas behind a TCP load balancer. Sticky sessions are recommended for SSE chat streams.
• Postgres can sit on either node; if you put it on spark-01 you'll re-introduce the very latency-disturbance pattern this architecture is designed to avoid. Prefer a third small box or a dedicated HA pair.

n8n HA

• n8n's queue mode requires Postgres for state and Redis for the BullMQ queue. The main container becomes the main instance; one or more worker instances pull jobs off the queue.
• Set EXECUTIONS_MODE=queue, QUEUE_BULL_REDIS_HOST=…, DB_TYPE=postgresdb, and the relevant Postgres env vars on every container. Replicas need the same N8N_ENCRYPTION_KEY.
• For a two-node setup the simplest variant is one main on spark-02 and one worker on a third small box, with Postgres + Redis colocated on the third box.
• Webhook traffic should hit only the main container; long-running executions land on workers transparently.

Obsidian vault sync runs on a dedicated Ubuntu 24.04 LXC (hostname: YOUR_LXC_HOSTNAME, IP: YOUR_LXC_IP) on the Proxmox homelab. Syncthing syncs the vault from Mac/Android to /vault on the LXC. @modelcontextprotocol/server-filesystem exposes the vault as a filesystem MCP server, wrapped by supergateway with streamableHttp transport on port 3000. LiteLLM on spark-01 connects to it at http://YOUR_LXC_IP:3000/mcp.

Obsidian (Mac/Android) <-> Syncthing <-> /vault on LXC <-> server-filesystem <-> supergateway :3000 <-> LiteLLM MCP client

LXC setup

Run as root on a fresh Ubuntu 24.04 LXC.

#!/bin/bash
set -e

apt update
apt install -y curl gpg apt-transport-https

# Node.js 22
curl -fsSL https://deb.nodesource.com/setup_22.x | bash -
apt install -y nodejs

# Syncthing
curl -fsSL https://syncthing.net/release-key.gpg | \
  gpg --dearmor -o /usr/share/keyrings/syncthing-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/syncthing-archive-keyring.gpg] \
  https://apt.syncthing.net/ syncthing stable" \
  > /etc/apt/sources.list.d/syncthing.list
apt update && apt install -y syncthing

# Vault directory
mkdir -p /vault

# Syncthing — After=network-online.target prevents a race where the service starts
# before the IP is assigned, causing the first vault sync to fail on boot
systemctl enable syncthing@root
systemctl start syncthing@root
sleep 8

# Expose Syncthing GUI on all interfaces
CONFIG_PATH=$(find /root -name "config.xml" 2>/dev/null | grep syncthing | head -1)
sed -i 's|<address>127.0.0.1:8384</address>|<address>0.0.0.0:8384</address>|' "$CONFIG_PATH"
systemctl restart syncthing@root

# @modelcontextprotocol/server-filesystem + supergateway
npm install -g @modelcontextprotocol/server-filesystem supergateway

# systemd service — streamableHttp transport, stateless (no --stateful), protocol 2024-11-05
cat > /etc/systemd/system/obsidian-mcp.service << 'EOF'
[Unit]
Description=Obsidian MCP Server (server-filesystem)
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=root
ExecStart=supergateway \
  --stdio "npx -y @modelcontextprotocol/server-filesystem /vault" \
  --port 3000 \
  --outputTransport streamableHttp
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable obsidian-mcp
systemctl start obsidian-mcp

After the LXC is running

• Configure Syncthing on the LXC (port 8384) to accept a share from your Mac/Android
• Set the shared folder path to /vault
• Connect Mac Obsidian Syncthing client to the LXC Syncthing device

LiteLLM connection

Via litellm-config.yaml at the top level (not nested under litellm_settings). The protocol_version must be explicit — server-filesystem ignores 2025-11-25 and the handshake fails silently without it:

mcp_servers:
  - name: obsidian
    url: http://YOUR_LXC_IP:3000/mcp
    transport: streamableHttp
    protocol_version: "2024-11-05"

Adds live web search as a callable tool — the AI model can run Brave Search queries during a conversation in response to tool calls from Open WebUI and other clients. LiteLLM spawns the server on demand via stdio using an API key you supply.

Step 1 — Get a Brave Search API key

Go to api.search.brave.com, create a free account, and generate an API key under the Data for AI plan (free tier supports up to 2,000 queries/month).

Step 2 — Confirm Node.js is installed system-wide

The MCP server is launched via npx. If you completed the Playwright MCP setup, Node.js is already installed system-wide and this step is done. Otherwise:

curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash -
sudo apt install -y nodejs

Step 3 — Add Brave Search MCP server in LiteLLM UI

Navigate to http://YOUR_NODE1_MGMT_IP:8001/ui → MCP Servers → Add New MCP Server. (LiteLLM lives on spark-01.)

Set Stdio Configuration (JSON) — replace YOUR_BRAVE_API_KEY with your actual key:

{
  "command": "npx",
  "args": [
    "-y",
    "@modelcontextprotocol/server-brave-search"
  ],
  "env": {
    "BRAVE_API_KEY": "YOUR_BRAVE_API_KEY"
  }
}

Save and confirm Health Status shows Healthy.

Validation

In Open WebUI, send the following prompt:

Expected: the model calls the brave_web_search tool (shown as "Explored" in Open WebUI) and returns a summary drawn from live search results.

Adds browser automation tool use to the stack — the AI model can navigate pages, take screenshots, and scrape content via tool calls in Open WebUI and other clients. LiteLLM spawns a headless Chromium process on demand via stdio; no persistent port is required.

Step 1 — Install Node.js system-wide

LiteLLM runs as a systemd service and does not source .bashrc. Node.js must be installed system-wide so npx is available in LiteLLM's PATH.

curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash -
sudo apt install -y nodejs

Step 2 — Install Playwright MCP Chromium browser

Chrome has no ARM64 build. Use Chromium, installed via the @playwright/mcp package's own browser installer — not via npx playwright install:

npx @playwright/mcp install-browser chromium

Step 3 — Update litellm-config.yaml

Add model_info blocks to all model entries. Without these, LiteLLM does not advertise function calling support and tool calls will not execute.

model_list:
  - model_name: qwen3.5-122b
    litellm_params:
      model: openai/qwen3.5-122b
      api_base: http://localhost:8000/v1
      api_key: "not-needed"
      max_tokens: 8192
    model_info:
      supports_function_calling: true
      supports_tool_choice: true

Restart LiteLLM after saving:

sudo systemctl restart litellm

Step 4 — Add Playwright MCP server in LiteLLM UI

Navigate to http://YOUR_NODE1_MGMT_IP:8001/ui → MCP Servers → Add New MCP Server. (LiteLLM lives on spark-01.)

Set Stdio Configuration (JSON):

{
  "command": "npx",
  "args": [
    "-y",
    "@playwright/mcp@latest",
    "--browser",
    "chromium",
    "--headless"
  ]
}

Save and confirm Health Status shows Healthy.

Validation

In Open WebUI, send the following prompt:

Expected: the model calls the navigate and screenshot tools (shown as "Explored" in Open WebUI) and returns a summary of the page.

Home Assistant exposes its own MCP endpoint at /api/mcp using Streamable HTTP transport. This lets Hermes (via LiteLLM) call HA as a tool — controlling devices, querying states, reading automations.

Step 1 — Enable the integration in Home Assistant

Settings → Devices & Services → Add Integration → search "Model Context Protocol Server"

Step 2 — Generate a long-lived access token

In Home Assistant: Profile → Long-Lived Access Tokens → Create Token. Copy the full token — it is only shown once.

Step 3 — Add to LiteLLM MCP config

mcp_servers:
  - name: home-assistant
    url: http://YOUR_HA_IP:8123/api/mcp
    transport: streamableHttp
    headers:
      Authorization: "Bearer YOUR_HA_LONG_LIVED_TOKEN"

The LiteLLM proxy ships with a built-in web UI at /ui. It requires a master key and a PostgreSQL database — SQLite is not supported for the UI auth layer. The following documents every error encountered during setup, in order.

Step 1 — Set a master key

All commands below run on spark-01 (where LiteLLM lives). Add to ~/spark-ai-stack/litellm-config.yaml:

general_settings:
  master_key: sk-yourkey
  database_url: "postgresql://litellm:litellm@localhost:5432/litellm"

Generate a secure key:

echo "sk-$(openssl rand -hex 16)"

Step 2 — Add PostgreSQL to a compose file on spark-01

Run Postgres on the same node as LiteLLM. Putting it on spark-02 would re-introduce the very latency-disturbance pattern this architecture is designed to avoid. Add to a new ~/spark-ai-stack/litellm-db.yml on spark-01:

  litellm-db:
    image: postgres:16
    container_name: litellm-db
    restart: unless-stopped
    environment:
      - POSTGRES_USER=litellm
      - POSTGRES_PASSWORD=litellm
      - POSTGRES_DB=litellm
    ports:
      - "5432:5432"
    volumes:
      - litellm_db:/var/lib/postgresql/data

volumes:
  litellm_db:

docker compose up -d litellm-db

Step 3 — Install Prisma

LiteLLM uses Prisma as its database ORM. It is not included in the base pip install litellm package.

pip install prisma --break-system-packages

--break-system-packages bypasses a Python 3.12 restriction that prevents pip from installing into the system Python environment. It is safe on a dedicated AI server where system tools do not depend on conflicting packages.

Step 4 — Generate Prisma binaries

After installing the package, the binaries must be generated from LiteLLM's bundled schema:

cd ~/.local/lib/python3.12/site-packages/litellm/proxy
prisma generate --schema schema.prisma

Step 5 — Apply the database schema

The Postgres database exists but has no tables yet. Push the schema. DATABASE_URL must be passed inline — Prisma reads it directly from the environment, not from litellm-config.yaml.

DATABASE_URL="postgresql://litellm:litellm@localhost:5432/litellm" \
prisma db push --schema schema.prisma

Step 6 — Restart LiteLLM

sudo systemctl daemon-reload
sudo systemctl restart litellm
sudo systemctl status litellm

Errors encountered in order

Accessing the UI

Navigate to http://YOUR_NODE1_MGMT_IP:8001/ui. Username: admin. Password: your master_key value.