Multi-Node Install

Prerequisites

Installed from the ISO (Option B)? The bridge is configured for you — the ISO sets up the host's network interfaces, br-wan included. Skip to Instructions.

Instructions

Step 1. Install Spinifex on Each Server

Choose one method and apply it to every server in the cluster.

Option A — existing OS. On a server already running Ubuntu 26.04 or Debian 13:

curl -fsSL https://install.mulgadc.com | bash

Option B — bare metal, from the ISO. Boot each server from the Spinifex ISO and follow Bootable USB Install. This installs the operating system, partitions the disks, and configures the hostname and network interfaces alongside Spinifex. The ISO installer does not form a cluster — that is what the remaining steps do.

Complete this step on all three servers before continuing. Step 4 requires every node to be installed, reachable, and available at the same time.

Step 2. Set Node IP Variables

On each server, export the management IPs of all three nodes plus the region and AZ. The same values go on every server:

export SPINIFEX_NODE1=192.168.1.10
export SPINIFEX_NODE2=192.168.1.11
export SPINIFEX_NODE3=192.168.1.12
export AWS_REGION=us-east-1
export AWS_AZ=us-east-1a

Adding a fourth server or more? Export SPINIFEX_NODE4 and so on alongside these — the first three stay as they are, because they remain the OVN database nodes.

sudo systemctl stop spinifex.target
sudo /usr/local/lib/spinifex/spinifex-firewall-apply disable

Step 3. Set Up OVN Networking

Servers 1 to 3 run a clustered OVN database — the VPC networking control plane — so it survives losing any one of them. This is the only database limited to three members; storage and NATS run on every server. Server 1 creates the cluster and must be set up first; servers 2 and 3 then join it.

--recreate-db appears in each command below because ovn-central starts a standalone OVN database when the package installs, and a clustered one can only be created from scratch. It replaces that standalone database on both the binary and ISO paths.

If your WAN interface is already a bridge, setup-ovn.sh auto-detects it. Otherwise add --wan-bridge=br-wan --wan-iface=eth1 for a dedicated WAN NIC.

Server 1 — create the cluster:

sudo /usr/local/share/spinifex/setup-ovn.sh \
  --management \
  --db-cluster-local-addr=$SPINIFEX_NODE1 \
  --recreate-db \
  --encap-ip=$SPINIFEX_NODE1

Server 2 (after server 1 is ready):

sudo /usr/local/share/spinifex/setup-ovn.sh \
  --management \
  --db-cluster-local-addr=$SPINIFEX_NODE2 \
  --db-cluster-remote-addr=$SPINIFEX_NODE1 \
  --recreate-db \
  --encap-ip=$SPINIFEX_NODE2

Server 3 (after server 1 is ready):

sudo /usr/local/share/spinifex/setup-ovn.sh \
  --management \
  --db-cluster-local-addr=$SPINIFEX_NODE3 \
  --db-cluster-remote-addr=$SPINIFEX_NODE1 \
  --recreate-db \
  --encap-ip=$SPINIFEX_NODE3

Servers 4 and beyond — repeat for each one, substituting its own address. They point at the three OVN database servers rather than joining the database cluster, so they survive any one of those failing:

sudo /usr/local/share/spinifex/setup-ovn.sh \
  --ovn-remote=tcp:$SPINIFEX_NODE1:6642,tcp:$SPINIFEX_NODE2:6642,tcp:$SPINIFEX_NODE3:6642 \
  --encap-ip=$SPINIFEX_NODE4

Verify the OVN database cluster formed, then that every chassis registered with it:

sudo ovn-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound
sudo ovn-sbctl show

cluster/status should list three servers with one leader, and ovn-sbctl show a chassis for every node in the cluster. If cluster/status reports a standalone database, the OVN cluster did not form — re-check that --db-cluster-local-addr was passed on servers 1, 2 and 3.

Step 4. Form the Cluster

Run init and join concurrently — init blocks until all nodes have joined.

--force is in every command below so the sequence is identical whichever way you installed. It does the work on ISO-installed servers, which arrive as their own single-node cluster: joining replaces that server's CA and master key with server 1's and removes its JetStream store, and --force is the confirmation. On spx admin init it is idempotent — existing keys, credentials and CA are preserved, and only the config files and server certificate are refreshed. On a freshly installed server there is nothing to lose either way.

Every server also discards the JetStream store its single-node cluster wrote. Server 1 removes its own once every server has joined, and each joining server removes its own when it joins. Keeping them is never safe: NATS would adopt each server's copy as a replica of the cluster's stream, and those copies never converge. This is automatic whenever --nodes is 2 or more. --discard-jetstream=false makes init refuse to form over a non-empty store instead, for a server whose store you want to inspect first. Stop spinifex.target on every server first and confirm pgrep -af 'spx service' prints nothing: init and join both refuse while NATS is still running.

Server 1 — initialize:

sudo spx admin init --force \
  --node node1 --nodes 3 \
  --bind $SPINIFEX_NODE1 --cluster-bind $SPINIFEX_NODE1 \
  --port 4432 --region $AWS_REGION --az $AWS_AZ

--nodes 3 is the number of servers init waits for. Set it to your total node count if you are building a larger cluster.

IPsec encrypts the Geneve overlay between servers and is on by default. Joining servers take the setting from server 1, so it is chosen once, on init. On servers that share a trusted private link, --ipsec=false leaves the overlay unencrypted in exchange for considerably higher throughput between instances.

The init output displays the join command including the token:

📡 Formation server started on 10.0.0.1:4432
   Waiting for 2 more node(s) to join...
   Token expires in 30m0s

   Other nodes should run:
   sudo spx admin join --host 10.0.0.1:4432 --token spx_join_a8Bf3x9Kz2mN --node <name> --bind <ip>

Take the token from that output, but run the commands below rather than the line it prints — they add --force and --cluster-bind.

Server 2 — join (while init is still running):

sudo spx admin join --force \
  --node node2 --bind $SPINIFEX_NODE2 --cluster-bind $SPINIFEX_NODE2 \
  --host $SPINIFEX_NODE1:4432 --token <token-from-init-output> \
  --region $AWS_REGION --az $AWS_AZ

Server 3 — join (while init is still running):

sudo spx admin join --force \
  --node node3 --bind $SPINIFEX_NODE3 --cluster-bind $SPINIFEX_NODE3 \
  --host $SPINIFEX_NODE1:4432 --token <token-from-init-output> \
  --region $AWS_REGION --az $AWS_AZ

Each additional server runs the same join command with its own --node name and --bind address.

Note: the join token expires 30 minutes after init by default. For larger deployments with slower provisioning, use --token-ttl 2h.

Step 5. Start Services

On all servers:

sudo systemctl start spinifex.target

Step 6. Verify the Cluster

Run these from any node. Together they confirm that every server joined, that services are healthy on each, and that capacity is being pooled across the cluster.

1. Every node is present and Ready.

spx get nodes

spinifex@node1:~$ spx get nodes
NAME  | STATUS | ROLES         | IP       | REGION    | AZ         | UPTIME | VMs | SERVICES
node1 | Ready  | nats:follower | 10.2.0.2 | us-east-1 | us-east-1a | 21h27m | 0   | nats,predastore,viperblock,daemon,awsgw,vpcd,ui
node2 | Ready  | nats:follower | 10.2.0.3 | us-east-1 | us-east-1a | 21h27m | 1   | nats,predastore,viperblock,daemon,awsgw,vpcd,ui
node3 | Ready  | nats:leader   | 10.2.0.4 | us-east-1 | us-east-1a | 21h27m | 0   | nats,predastore,viperblock,daemon,awsgw,vpcd,ui

What to check:

• Every server you installed is listed. A missing node never joined — see Nodes not joining.
• STATUS is Ready on all of them. NotReady means the node is in the cluster configuration but is not answering, so start with spinifex.target on that host.
• Exactly one nats:leader. The rest are followers.
• SERVICES lists the same set on every node. A short list means something failed to start there; check systemctl status for the missing unit.

2. Capacity is pooled across the cluster.

spx top nodes

spinifex@node1:~$ spx top nodes
NAME  | CPU (used/total) | MEM (used/total) | GPU (used/total) | VMs
node1 | 0/64             | 0Mi/220.2Gi      | -                | 0
node2 | 2/64             | 2.8Gi/251.7Gi    | -                | 1
node3 | 0/64             | 0Mi/251.7Gi      | -                | 0


INSTANCE TYPE | AVAILABLE | VCPU | MEMORY
c6a.12xlarge  | 3         | 48   | 96.0Gi
c6a.16xlarge  | 0         | 64   | 128.0Gi
c6a.24xlarge  | 0         | 96   | 192.0Gi
c6a.2xlarge   | 21        | 8    | 16.0Gi
c6a.4xlarge   | 9         | 16   | 32.0Gi
c6a.8xlarge   | 3         | 32   | 64.0Gi
c6a.large     | 92        | 2    | 4.0Gi
c6a.xlarge    | 45        | 4    | 8.0Gi
c6i.12xlarge  | 3         | 48   | 96.0Gi
c6i.16xlarge  | 0         | 64   | 128.0Gi
c6i.24xlarge  | 0         | 96   | 192.0Gi
c6i.2xlarge   | 21        | 8    | 16.0Gi
c6i.4xlarge   | 9         | 16   | 32.0Gi
c6i.8xlarge   | 3         | 32   | 64.0Gi
c6i.large     | 92        | 2    | 4.0Gi
c6i.xlarge    | 45        | 4    | 8.0Gi
m6a.12xlarge  | 3         | 48   | 192.0Gi

The top table is per-node CPU, memory and GPU usage. The bottom table is what the cluster can actually launch right now: AVAILABLE is the number of instances of that type that would currently fit across all nodes. A type showing 0 does not fit on any single node — instances are not split across servers, so the largest type you can launch is bounded by your biggest node, not by the cluster total.

If capacity looks like a single server rather than the sum of your nodes, the others have not joined.

3. The AWS API answers.

export AWS_PROFILE=spinifex
aws ec2 describe-instance-types

A list of instance types means the gateway, IAM and the cluster behind them are all working.

Congratulations! Your multi-server Spinifex cluster is installed.

Step 7. Set Up Your Cluster

The cluster is running, but it holds nothing yet — no machine images, no networks, no instances.

Continue to Setting Up Your Cluster to import an AMI, create an SSH key pair, create a VPC with a public subnet, and launch your first instance. It ends by arming the host firewall, which is also where you re-arm it if you turned it off to form the cluster.