# Spinifex Documentation — Full Text

> Every published Spinifex document, complete, in one file. Newest change across the site: 2026-09-15. The index-only version is at https://docs.mulgadc.com/llms.txt

---

# Single-Node Install

URL: https://docs.mulgadc.com/docs/install
Category: Installation
Updated: 2026-08-19
Tags: install, single node, quickstart

Install Spinifex on one Ubuntu or Debian server with the binary installer and get an AWS-compatible EC2, S3, and VPC stack running on your own hardware.

## Overview

Spinifex is an open-source infrastructure platform that brings core AWS services to bare-metal, edge, and on-prem environments.

This guide installs Spinifex on a single server using the binary installer. For multi-server clusters, see [Multi-Node Install](https://docs.mulgadc.com/docs/install-multi-node). To build from source, see [Source Install](https://docs.mulgadc.com/docs/install-source).

**Supported Operating Systems:**

- Ubuntu 26.04
- Debian 13

**What Gets Installed:**

- Spinifex daemon and CLI
- QEMU/KVM for virtual machine management
- OVN/Open vSwitch for VPC networking
- Predastore (S3-compatible object storage)
- Viperblock (EBS-compatible block storage)
- AWS CLI v2

## Prerequisites

> [!IMPORTANT]
> **Prerequisite — WAN bridge required.**
>
> Before running the installer, the host's WAN interface **must** already be enslaved to a Linux bridge named `br-wan`. The host IP, default route, and DHCP must all live on the bridge — not on the bare NIC.
>
> The bootstrap installer does **not** create this bridge for you yet. Running it on a host whose default route is still on a bare NIC will leave the install in a non-working state. Auto-provisioning of `br-wan` will land in a future release.
>
> **Verify before continuing:**
>
> - `ip -br link show br-wan` — bridge exists and is `UP`
> - `ip route` — default route's `dev` is `br-wan`
>
> **Setup references:** [VPC Networking → Bridge Setup](https://docs.mulgadc.com/docs/vpc-networking#bridge-setup-physical-network-wiring) for the topology.

## Instructions

## Step 1. Install Spinifex

```bash
curl -fsSL https://install.mulgadc.com | bash
```

The installer downloads the Spinifex binary and bootstraps all dependencies (QEMU, OVN/OVS, AWS CLI).

## Step 2. Setup OVN Networking

If your WAN interface is already a bridge (e.g. `br-wan`), setup-ovn.sh auto-detects it:

```bash
sudo /usr/local/share/spinifex/setup-ovn.sh --management
```

If your WAN is a physical NIC:

```bash
sudo /usr/local/share/spinifex/setup-ovn.sh --management --wan-bridge=br-wan --wan-iface=eth1
```

**Separating VPC tunnel traffic from WAN:** If you want Geneve tunnel traffic (inter-VM east-west for VPC traffic) to use a dedicated interface instead of the WAN IP, add `--encap-ip=<IP>` to specify the tunnel endpoint address:

```bash
sudo /usr/local/share/spinifex/setup-ovn.sh --management --encap-ip=10.0.0.1
```

This is recommended for production and required in multi-node deployments. See [Multi-Node Install](https://docs.mulgadc.com/docs/install-multi-node) for details.

## Step 3. Initialize

```bash
sudo spx admin init --node node1 --nodes 1
```

This auto-detects your network topology, generates configuration and TLS certificates, installs the CA into the system trust store, and configures AWS CLI credentials (saved in `~/.aws/credentials`)

## Step 4. Start Services

```bash
sudo systemctl start spinifex.target
```

## Step 5. Verify

```bash
export AWS_PROFILE=spinifex
aws ec2 describe-instance-types
```

If this returns a list of available instance types, your installation is working.

**Congratulations! Spinifex is installed.**

Continue to [Setting Up Your Cluster](https://docs.mulgadc.com/docs/setting-up-your-cluster) to import an AMI, create a VPC, and launch your first instance.

## Troubleshooting

### spx Command Not Found

```bash
export PATH=$PATH:/usr/local/bin
```

### CA Certificate Not Trusted

`sudo spx admin init` installs the CA automatically. If you need to re-install it manually:

```bash
sudo cp /etc/spinifex/ca.pem /usr/local/share/ca-certificates/spinifex-ca.crt
sudo update-ca-certificates
```

### OVN Services Not Starting

```bash
sudo systemctl is-active ovn-controller
journalctl -u ovn-controller --no-pager -n 20
```

---

# Multi-Node Install

URL: https://docs.mulgadc.com/docs/install-multi-node
Category: Installation
Updated: 2026-09-14
Tags: install, multi node, cluster, iso, bare-metal

Deploy Spinifex across three or more servers to form an availability zone with clustered OVN, replicated object storage, and automatic cluster formation.

## Overview

> [!IMPORTANT]
> **This guide builds a three-node cluster** — the minimum we recommend for any multi-server
> deployment. Every step below is written for three servers and uses `$SPINIFEX_NODE1`,
> `$SPINIFEX_NODE2` and `$SPINIFEX_NODE3`.
>
> **Running more than three?** Install the first three exactly as described, then repeat
> Steps 1, 2, 3 and the join in Step 4 for each additional server. The only difference is in
> Step 3: the OVN database cluster stays at three members, so servers four and beyond point at
> those three instead of joining them.

A Spinifex cluster distributes services across multiple servers for high availability, data durability, and fault tolerance. Cluster formation is automatic — the init node waits for its peers to join, then distributes credentials, CA certificates, and configuration.

**Installing on bare metal?** Step 1 can be done either with the binary installer or by booting each server from the Spinifex ISO — see [Bootable USB Install](https://docs.mulgadc.com/docs/install-usb). The ISO installs the operating system, disks and network configuration as well as Spinifex itself, which makes it the better option for servers with no existing OS. Either way, install every server first, then return here and continue from Step 2.

### Cluster sizing

**Three servers is the minimum we recommend.** Three is the point at which every distributed layer can lose a node and keep running:

| Layer | What three servers gives you |
|---|---|
| **VPC networking** — OVN | Control-plane databases run clustered, surviving the loss of any one node. |
| **Object storage** — Predastore (S3) | Objects are erasure coded `RS(2,1)`, surviving the loss of any one node's shards. |
| **Block storage** — Viperblock (EBS) | Volumes are stored in Predastore, so they inherit the same durability. |

On one or two servers none of that holds. OVN runs standalone on the first node, and the storage metadata quorum has no majority to lose. If that node goes down, running instances keep full networking — but nothing can *change*: no new VPCs, no launches, no security group edits. See [OVN control plane on multi-node clusters](https://docs.mulgadc.com/docs/vpc-networking#ovn-control-plane-on-multi-node-clusters).

Servers beyond the third run the full set of services — storage, gateway and networking agents — and add their capacity to the pool. What they do not do is join the OVN database cluster, which stays at three members, so write latency there stays flat as the cluster grows.

### Hardware

Per server, for the three-node minimum:

| | Minimum | Recommended |
|---|---|---|
| **Nodes** | 3 | 3 or more |
| **RAM** | 32 GB | 128 GB |
| **CPU** | 16 cores | 32+ cores |
| **OS / Spinifex disk** | SSD | NVMe |
| **NICs** | 2 — WAN 1 GbE, LAN/VPC 10 GbE+ | 2 — WAN 10 GbE, LAN/VPC 25 GbE+ |

Two NICs matters more than the raw numbers. One carries WAN traffic; the other carries LAN and VPC traffic between nodes — Geneve tunnels, object shards and OVN replication all cross it, so it is the interface that wants the bandwidth. A single-NIC server will work, but inter-node storage and tunnel traffic then competes with everything going in and out of the cluster.

### Network requirements

Open between all hosts:

| Protocol and port | Used by |
|---|---|
| UDP 6081 | Geneve tunnels |
| TCP 4222, 4248 | NATS |
| TCP 6641, 6642 | OVN northbound and southbound |
| TCP 8443 | Predastore S3 gate |
| UDP 6660, 7660 | Object shards, metadata consensus |

Open between the three OVN database servers (servers 1 to 3) only:

| Protocol and port | Used by |
|---|---|
| TCP 6643, 6644 | OVN database clustering |

Predastore uses the same three ports on every server, so the surface does not widen as the cluster grows. See [How multi-node storage works](#how-multi-node-storage-works).

Those are the ports your **network** has to permit between the servers — switches, upstream firewalls, and anything else in the path.

Spinifex's own **host** firewall is separate, and already carries this policy. It ships armed on the ISO path and off on the binary installer path. Two things follow from that, both of which the ISO box in Step 2 acts on:

- **The formation port needs nothing from you.** `spx admin init` opens 4432 to any source for the length of the formation window and closes it again afterwards, because a node dialling in to join is not a peer yet. The handshake behind it is TLS 1.3 with a bearer token.
- **The rest of the cluster plane is peer-scoped**, and that includes the OVN database ports Step 3 uses. Nodes that do not yet know each other cannot reach them, which is why an ISO-installed node has its firewall taken down before Step 3 and re-armed after Step 6.

## Prerequisites

**Installed from the ISO (Option B)?** The bridge is configured for you — the ISO sets up the host's network interfaces, `br-wan` included. Skip to [Instructions](#instructions).

> [!IMPORTANT]
> **Binary installer only — a WAN bridge is required on every node.**
>
> Before running the installer on any server, that server's WAN interface **must** already be enslaved to a Linux bridge named `br-wan`. The host IP, default route, and DHCP must all live on the bridge — not on the bare NIC.
>
> The binary installer does **not** create this bridge for you yet. Running it on a host whose default route is still on a bare NIC will leave the install in a non-working state. Auto-provisioning of `br-wan` will land in a future release.
>
> **Verify on every node before continuing:**
>
> - `ip -br link show br-wan` — bridge exists and is `UP`
> - `ip route` — default route's `dev` is `br-wan`
>
> See [VPC Networking → Bridge Setup](https://docs.mulgadc.com/docs/vpc-networking#bridge-setup-physical-network-wiring) for the topology.

## Instructions

## Step 1. Install Spinifex on Each Server

Choose one method and apply it to **every** server in the cluster.

**Option A — existing OS.** On a server already running Ubuntu 26.04 or Debian 13:

```bash
curl -fsSL https://install.mulgadc.com | bash
```

**Option B — bare metal, from the ISO.** Boot each server from the Spinifex ISO and follow [Bootable USB Install](https://docs.mulgadc.com/docs/install-usb). This installs the operating system, partitions the disks, and configures the hostname and network interfaces alongside Spinifex. The ISO installer does not form a cluster — that is what the remaining steps do.

Complete this step on all three servers before continuing. Step 4 requires every node to be installed, reachable, and available at the same time.

## Step 2. Set Node IP Variables

On **each server**, export the management IPs of all three nodes plus the region and AZ. The same values go on every server:

```bash
export SPINIFEX_NODE1=192.168.1.10
export SPINIFEX_NODE2=192.168.1.11
export SPINIFEX_NODE3=192.168.1.12
export AWS_REGION=us-east-1
export AWS_AZ=us-east-1a
```

Adding a fourth server or more? Export `SPINIFEX_NODE4` and so on alongside these — the first three stay as they are, because they remain the OVN database nodes.

> [!NOTE]
> **ISO installs only — skip this box if you used the binary installer (Option A).**
>
> The ISO brings each server up as a **running standalone single-node cluster** with its firewall armed, so forming a cluster is a conversion rather than a fresh setup. One thing is needed, on **every** server, before Step 3:
>
> ```bash
> sudo systemctl stop spinifex.target
> sudo /usr/local/lib/spinifex/spinifex-firewall-apply disable
> ```
>
> Each node's firewall currently trusts only itself, because that is the whole cluster as far as it knows, and the cluster plane is peer-scoped — so the OVN database connections Step 3 makes between servers are dropped. Stopping `spinifex.target` is not enough: the firewall lives in the kernel and outlives the services.
>
> Turn it back on after Step 6 — see [Firewall and cluster membership](#firewall-and-cluster-membership).

## Step 3. Set Up OVN Networking

Servers 1 to 3 run a clustered **OVN database** — the VPC networking control plane — so it survives losing any one of them. This is the only database limited to three members; storage and NATS run on every server. **Server 1 creates the cluster and must be set up first**; servers 2 and 3 then join it.

`--recreate-db` appears in each command below because `ovn-central` starts a standalone OVN database when the package installs, and a clustered one can only be created from scratch. It replaces that standalone database on both the binary and ISO paths.

If your WAN interface is already a bridge, `setup-ovn.sh` auto-detects it. Otherwise add `--wan-bridge=br-wan --wan-iface=eth1` for a dedicated WAN NIC.

**Server 1 — create the cluster:**

```bash
sudo /usr/local/share/spinifex/setup-ovn.sh \
  --management \
  --db-cluster-local-addr=$SPINIFEX_NODE1 \
  --recreate-db \
  --encap-ip=$SPINIFEX_NODE1
```

**Server 2** (after server 1 is ready):

```bash
sudo /usr/local/share/spinifex/setup-ovn.sh \
  --management \
  --db-cluster-local-addr=$SPINIFEX_NODE2 \
  --db-cluster-remote-addr=$SPINIFEX_NODE1 \
  --recreate-db \
  --encap-ip=$SPINIFEX_NODE2
```

**Server 3** (after server 1 is ready):

```bash
sudo /usr/local/share/spinifex/setup-ovn.sh \
  --management \
  --db-cluster-local-addr=$SPINIFEX_NODE3 \
  --db-cluster-remote-addr=$SPINIFEX_NODE1 \
  --recreate-db \
  --encap-ip=$SPINIFEX_NODE3
```

**Servers 4 and beyond** — repeat for each one, substituting its own address. They point at the three OVN database servers rather than joining the database cluster, so they survive any one of those failing:

```bash
sudo /usr/local/share/spinifex/setup-ovn.sh \
  --ovn-remote=tcp:$SPINIFEX_NODE1:6642,tcp:$SPINIFEX_NODE2:6642,tcp:$SPINIFEX_NODE3:6642 \
  --encap-ip=$SPINIFEX_NODE4
```

Verify the OVN database cluster formed, then that every chassis registered with it:

```bash
sudo ovn-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound
sudo ovn-sbctl show
```

`cluster/status` should list three servers with one leader, and `ovn-sbctl show` a chassis for every node in the cluster. If `cluster/status` reports a standalone database, the OVN cluster did not form — re-check that `--db-cluster-local-addr` was passed on servers 1, 2 and 3.

## Step 4. Form the Cluster

Run init and join **concurrently** — init blocks until all nodes have joined.

`--force` is in every command below so the sequence is identical whichever way you installed. It does the work on ISO-installed servers, which arrive as their own single-node cluster: joining replaces that server's CA and master key with server 1's and removes its JetStream store, and `--force` is the confirmation. On `spx admin init` it is idempotent — existing keys, credentials and CA are preserved, and only the config files and server certificate are refreshed. On a freshly installed server there is nothing to lose either way.

Every server also discards the JetStream store its single-node cluster wrote. Server 1 removes its own once every server has joined, and each joining server removes its own when it joins. Keeping them is never safe: NATS would adopt each server's copy as a replica of the cluster's stream, and those copies never converge. This is automatic whenever `--nodes` is 2 or more. `--discard-jetstream=false` makes init refuse to form over a non-empty store instead, for a server whose store you want to inspect first. Stop `spinifex.target` on every server first and confirm `pgrep -af 'spx service'` prints nothing: init and join both refuse while NATS is still running.

> [!WARNING]
> Do not point these commands at a server that has already been in service. Joining discards its master key, orphaning every volume and fragment sealed under it. That is what `--force` overrides, and it is unrecoverable.

**Server 1 — initialize:**

```bash
sudo spx admin init --force \
  --node node1 --nodes 3 \
  --bind $SPINIFEX_NODE1 --cluster-bind $SPINIFEX_NODE1 \
  --port 4432 --region $AWS_REGION --az $AWS_AZ
```

`--nodes 3` is the number of servers init waits for. Set it to your total node count if you are building a larger cluster.

IPsec encrypts the Geneve overlay between servers and is on by default. Joining servers take the setting from server 1, so it is chosen once, on init. On servers that share a trusted private link, `--ipsec=false` leaves the overlay unencrypted in exchange for considerably higher throughput between instances.

The init output displays the join command including the token:

```
📡 Formation server started on 10.0.0.1:4432
   Waiting for 2 more node(s) to join...
   Token expires in 30m0s

   Other nodes should run:
   sudo spx admin join --host 10.0.0.1:4432 --token spx_join_a8Bf3x9Kz2mN --node <name> --bind <ip>
```

Take the **token** from that output, but run the commands below rather than the line it prints — they add `--force` and `--cluster-bind`.

**Server 2 — join** (while init is still running):

```bash
sudo spx admin join --force \
  --node node2 --bind $SPINIFEX_NODE2 --cluster-bind $SPINIFEX_NODE2 \
  --host $SPINIFEX_NODE1:4432 --token <token-from-init-output> \
  --region $AWS_REGION --az $AWS_AZ
```

**Server 3 — join** (while init is still running):

```bash
sudo spx admin join --force \
  --node node3 --bind $SPINIFEX_NODE3 --cluster-bind $SPINIFEX_NODE3 \
  --host $SPINIFEX_NODE1:4432 --token <token-from-init-output> \
  --region $AWS_REGION --az $AWS_AZ
```

Each additional server runs the same join command with its own `--node` name and `--bind` address.

**Note:** the join token expires 30 minutes after init by default. For larger deployments with slower provisioning, use `--token-ttl 2h`.

## Step 5. Start Services

On **all servers**:

```bash
sudo systemctl start spinifex.target
```

## Step 6. Verify the Cluster

Run these from any node. Together they confirm that every server joined, that services are healthy on each, and that capacity is being pooled across the cluster.

**1. Every node is present and Ready.**

```bash
spx get nodes
```

```
spinifex@node1:~$ spx get nodes
NAME  | STATUS | ROLES         | IP       | REGION    | AZ         | UPTIME | VMs | SERVICES
node1 | Ready  | nats:follower | 10.2.0.2 | us-east-1 | us-east-1a | 21h27m | 0   | nats,predastore,viperblock,daemon,awsgw,vpcd,ui
node2 | Ready  | nats:follower | 10.2.0.3 | us-east-1 | us-east-1a | 21h27m | 1   | nats,predastore,viperblock,daemon,awsgw,vpcd,ui
node3 | Ready  | nats:leader   | 10.2.0.4 | us-east-1 | us-east-1a | 21h27m | 0   | nats,predastore,viperblock,daemon,awsgw,vpcd,ui
```

What to check:

- **Every server you installed is listed.** A missing node never joined — see [Nodes not joining](#nodes-not-joining).
- **`STATUS` is `Ready`** on all of them. `NotReady` means the node is in the cluster configuration but is not answering, so start with `spinifex.target` on that host.
- **Exactly one `nats:leader`.** The rest are followers.
- **`SERVICES` lists the same set on every node.** A short list means something failed to start there; check `systemctl status` for the missing unit.

**2. Capacity is pooled across the cluster.**

```bash
spx top nodes
```

```
spinifex@node1:~$ spx top nodes
NAME  | CPU (used/total) | MEM (used/total) | GPU (used/total) | VMs
node1 | 0/64             | 0Mi/220.2Gi      | -                | 0
node2 | 2/64             | 2.8Gi/251.7Gi    | -                | 1
node3 | 0/64             | 0Mi/251.7Gi      | -                | 0


INSTANCE TYPE | AVAILABLE | VCPU | MEMORY
c6a.12xlarge  | 3         | 48   | 96.0Gi
c6a.16xlarge  | 0         | 64   | 128.0Gi
c6a.24xlarge  | 0         | 96   | 192.0Gi
c6a.2xlarge   | 21        | 8    | 16.0Gi
c6a.4xlarge   | 9         | 16   | 32.0Gi
c6a.8xlarge   | 3         | 32   | 64.0Gi
c6a.large     | 92        | 2    | 4.0Gi
c6a.xlarge    | 45        | 4    | 8.0Gi
c6i.12xlarge  | 3         | 48   | 96.0Gi
c6i.16xlarge  | 0         | 64   | 128.0Gi
c6i.24xlarge  | 0         | 96   | 192.0Gi
c6i.2xlarge   | 21        | 8    | 16.0Gi
c6i.4xlarge   | 9         | 16   | 32.0Gi
c6i.8xlarge   | 3         | 32   | 64.0Gi
c6i.large     | 92        | 2    | 4.0Gi
c6i.xlarge    | 45        | 4    | 8.0Gi
m6a.12xlarge  | 3         | 48   | 192.0Gi
```

The top table is per-node CPU, memory and GPU usage. The bottom table is what the cluster can actually launch right now: `AVAILABLE` is the number of instances of that type that would currently fit across all nodes. A type showing `0` does not fit on any single node — instances are not split across servers, so the largest type you can launch is bounded by your biggest node, not by the cluster total.

If capacity looks like a single server rather than the sum of your nodes, the others have not joined.

**3. The AWS API answers.**

```bash
export AWS_PROFILE=spinifex
aws ec2 describe-instance-types
```

A list of instance types means the gateway, IAM and the cluster behind them are all working.

**4. Every server holds the same cluster state.**

Services report Ready even over a cluster whose JetStream replicas disagree, so check the replicas themselves. On **every server**, take a digest of its local store:

```bash
sudo spx admin kv digest --json --seqs > kv-digest-$(hostname).json
```

Copy the files to one server and compare them:

```bash
spx admin kv compare kv-digest-*.json
```

```
compared 3 digests: node1@10:02:11 node2@10:02:14 node3@10:02:18
ok        KV_spinifex-iam-users  [node1,node2,node3]
      identical, seqs 1-14
...
42 streams: 42 consistent, 0 diverged
```

The last line must report `0 diverged`, and the command exits non-zero otherwise. The digests are copied from live stores a few seconds apart, so a write landing between two copies can briefly show as a difference: take fresh digests and compare again before acting on one. A difference that persists means the cluster adopted a server's old store as a replica. Do not put that cluster in service — reset the servers and form it again.

A replica that is merely behind is not reported, and neither is a message that one server has already replaced with a newer write to the same key. `spx admin kv compare --help` lists exactly what counts as divergence.

**Congratulations! Your Spinifex cluster is installed.**

Continue to [Setting Up Your Cluster](https://docs.mulgadc.com/docs/setting-up-your-cluster) to import an AMI, create a VPC, and launch your first instance.

## Firewall and Cluster Membership

Spinifex ships an optional host firewall. It divides the node's ports into two groups:

| Group | Ports | Who can reach them |
|---|---|---|
| **Public** | SSH, 443, 3000 (console), 8443 (S3), 9999 (AWS gateway), 53 (DNS) | anyone |
| **Internal** | OVN, NATS, formation, Geneve and the rest of the cluster plane | **cluster members only** |

The internal group is the point. Before this existed, OVN and NATS were reachable from the public internet on a WAN-facing node.

"Cluster members" is not a list you maintain. Each node works it out from the cluster it belongs to and rewrites its own rules whenever membership changes — you never edit the peer list by hand.

### Is it on?

| How the node was installed | Firewall |
|---|---|
| From the ISO | **on** |
| Binary installer (`curl \| bash`) or `setup.sh` | **off** |
| `setup.sh --firewall=on` | **on** |

The binary installer defaults to off deliberately: it runs on servers that already have an operating system and services on them, and switching on a default-deny policy uninvited could cut off something Spinifex knows nothing about. **For production, turn it on** — either at install time:

```bash
curl -fsSL https://install.mulgadc.com | bash -s -- --firewall=on
```

or afterwards, by setting it in `/etc/spinifex/spinifex.toml` and restarting the daemon:

```toml
[network]
firewall_enabled = true
```

Before you do, check what else the machine is serving. Anything listening on a port outside the public group above stops accepting new connections.

### Turning it off and on around cluster changes

A node only recognises the members of the cluster it currently belongs to, so during formation — when the nodes do not yet know each other — internal traffic between them is blocked. Turn the firewall off while you form the cluster, and on again once it is up:

```bash
# Off — before forming or expanding a cluster. Run on every node.
sudo /usr/local/lib/spinifex/spinifex-firewall-apply disable

# On — once the cluster is formed and verified. Run on every node.
sudo systemctl restart spinifex-daemon
```

Restarting the daemon is what re-arms it: the node rebuilds its peer list from the cluster it is now part of, reloads the rules, and re-enables the boot-time unit so the policy survives a reboot. It also happens on its own within five minutes if you would rather wait.

### Checking it

```bash
sudo nft list table inet spinifex_filter
```

The peer list is an nft variable, expanded when the rules load, so it does not appear under a name of its own. Look instead at the `ip saddr { ... }` addresses on the cluster-plane rules — the ones accepting 4222, 6641, 6642 and the rest.

Every node's addresses should be there. On a multi-NIC node that means its WAN, LAN and VPC addresses, so expect several entries per node. A missing node means its cluster traffic is being dropped.

Dropped packets are logged, rate-limited, so this tells you whether a connection problem is the firewall or something else:

```bash
sudo journalctl -k | grep 'spinifex-fw drop'
```

## How Multi-Node Storage Works

Background reading — you do not configure any of this by hand. `spx admin init` and `spx admin join` build the topology from the servers that actually form the cluster in Step 4, and each machine gets the same file with its own host ID recorded in `spinifex.toml`.

Predastore is configured for the whole cluster in `/etc/spinifex/predastore/predastore.toml`. Each server is one `[[host]]` — a single Predastore process owning that machine's data directory and TLS identity — carrying three nodes under `[[host.node]]`:

| Role | Port | Purpose |
|---|---|---|
| `gate` | TCP 8443 | Serves the S3 API. Every server runs one, so any of them answers an S3 request. |
| `blob` | UDP 6660 | Holds erasure-coded object shards. One per machine. |
| `meta` | UDP 7660 | Member of the Raft quorum over global state — buckets and the object index. |

Ports have to be unique within a host but not across the cluster, so every machine uses the same three. Blob and meta traffic between hosts runs over QUIC, authenticated by the cluster CA; nodes on the same machine talk over an in-process pipe and open no socket, which is why a single-server install listens on 8443 alone.

Reed-Solomon parameters are chosen from the cluster size, since each machine contributes exactly one blob node: two servers get `RS(1,1)`, three or more get `RS(2,1)`. `RS(2,1)` survives the loss of any one server's shards — another reason three nodes is the recommended minimum.

## Troubleshooting

### Nodes Not Joining

The init command must still be running when join executes. If init exited, re-run with `--force`.

```bash
curl -sk https://$SPINIFEX_NODE1:4432/health
```

A hang rather than a quick failure means packets are being dropped; a refused connection means nothing is listening.

If it hangs, check node 1's init output before blaming the firewall — `spx admin init` opens the formation port itself while it waits, so this is usually not the cause. It prints `⚠️ Could not open port 4432 in the host firewall` when that fails, which is the case where it is. Confirm on node 1:

```bash
sudo journalctl -k | grep 'spinifex-fw drop'
```

Turn the firewall off on **every** node and retry the join, then re-arm once the cluster is up — see [Firewall and cluster membership](#firewall-and-cluster-membership). The joining node retries for 20 minutes by default, so it is often still waiting while you fix this.

### Join Refuses: "this node is already initialized"

The node has its own cluster configuration — normal for anything installed from the ISO, which initializes a single-node cluster at first boot. Joining replaces that node's CA and master key with the primary's, so it must be confirmed with `--force`.

Safe on a freshly installed node. On one that has been in service it orphans every volume and fragment sealed under the old key, so check before forcing.

### OVN Database Cluster Not Forming

```bash
sudo ovn-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound
```

If this reports a standalone database rather than three servers, the OVN database was created before the cluster flags were supplied. A clustered one can only be created from scratch — re-run Step 3 with `--recreate-db`.

### OVN Chassis Not Registering

```bash
sudo ovn-sbctl show
sudo ss -tlnp | grep 6642
```

### CA Certificate Not Trusted

On a node or any host running `spx`/`aws` against the cluster:

```bash
sudo cp /etc/spinifex/ca.pem /usr/local/share/ca-certificates/spinifex-ca.crt
sudo update-ca-certificates
```

Inside a guest VM there is no `/etc/spinifex`; fetch the CA from IMDS instead:

```bash
sudo curl -fsS http://169.254.169.254/spinifex/ca.pem \
  -o /usr/local/share/ca-certificates/spinifex-ca.crt
sudo update-ca-certificates
```

### Cross-Host VMs Cannot Communicate

```bash
sudo ovs-vsctl show | grep -i geneve
sudo ss -ulnp | grep 6081
```

---

# Source Install

URL: https://docs.mulgadc.com/docs/install-source
Category: Installation
Updated: 2026-08-19
Tags: install, source, development

Build Spinifex from source on Ubuntu or Debian for development, custom builds, or contributing changes, then install and run the resulting binaries locally.

## Overview

This guide builds Spinifex from source. For production deployments, the [binary installer](https://docs.mulgadc.com/docs/install) is recommended.

**Supported Operating Systems:**

- Ubuntu 26.04
- Debian 13

## Instructions

## Step 1. Install Dependencies

```bash
mkdir -p ~/Development/mulga && cd ~/Development/mulga
git clone https://github.com/mulgadc/spinifex.git
sudo make -C spinifex quickinstall
export PATH=$PATH:/usr/local/go/bin
```

## Step 2. Clone Sibling Repositories

```bash
cd spinifex
./scripts/clone-deps.sh
```

This clones Predastore (S3) and Viperblock (EBS) alongside Spinifex.

## Step 3. Development Initialisation

```bash
./scripts/dev-install.sh
```

This bootstraps a single-node development environment: builds binaries, configures OVN, initialises the cluster, installs the CA certificate, and starts all services.

## Step 4. Verify Installation

```bash
export AWS_PROFILE=spinifex
aws ec2 describe-instance-types
```

If this returns a list of available instance types, your installation is working.

**Congratulations! Spinifex is installed from source.**

Continue to [Setting Up Your Cluster](https://docs.mulgadc.com/docs/setting-up-your-cluster) to import an AMI, create a VPC, and launch your first instance.

## Troubleshooting

### Go Not Found in PATH

```bash
export PATH=$PATH:/usr/local/go/bin
```

### CA Certificate Not Trusted

On a node or any host running `spx`/`aws` against the cluster:

```bash
sudo cp /etc/spinifex/ca.pem /usr/local/share/ca-certificates/spinifex-ca.crt
sudo update-ca-certificates
```

Inside a guest VM there is no `/etc/spinifex`; fetch the CA from IMDS instead:

```bash
sudo curl -fsS http://169.254.169.254/spinifex/ca.pem \
  -o /usr/local/share/ca-certificates/spinifex-ca.crt
sudo update-ca-certificates
```

---

# Air-Gapped Install

URL: https://docs.mulgadc.com/docs/install-airgapped
Category: Installation
Updated: 2026-08-21
Tags: install, air-gapped, offline

Deploy Spinifex in environments without internet connectivity. Covers using a pre-built release tarball, mirrored APT packages, and locally-staged cloud images.

## Overview

In air-gapped or disconnected environments, Spinifex can be deployed without internet access. This guide covers preparing offline packages on a connected machine, creating USB deployment media, and installing on the target server with package verification.

## Instructions

## Step 1. Download the Release (on a connected machine)

Each Spinifex release publishes a self-contained tarball, the matching `setup.sh`, and a SHA-256 checksum to GitHub Releases. Resolve the latest tag and download the assets for your architecture:

```bash
ARCH=amd64   # or arm64
TAG=$(curl -fsSL https://api.github.com/repos/mulgadc/spinifex/releases/latest \
  | grep '"tag_name"' | cut -d'"' -f4)

BASE="https://github.com/mulgadc/spinifex/releases/download/${TAG}"
curl -fsSLO "${BASE}/spinifex-${TAG}-linux-${ARCH}.tar.gz"
curl -fsSLO "${BASE}/spinifex-${TAG}-linux-${ARCH}.tar.gz.sha256"
curl -fsSLO "${BASE}/setup.sh"

sha256sum -c "spinifex-${TAG}-linux-${ARCH}.tar.gz.sha256"
```

## Step 2. Stage Dependencies (on the connected machine)

Pre-download the APT packages installed by the production setup. `apt install --download-only` writes `.deb` files to `/var/cache/apt/archives/` without installing them, so the connected machine isn't modified:

```bash
sudo apt update
sudo apt install --download-only -y \
  nbdkit nbdkit-plugin-dev pkg-config \
  qemu-system-x86 qemu-utils \
  ovmf qemu-efi-aarch64 \
  libvirt-daemon-system libvirt-clients libvirt-dev \
  ovn-central ovn-host openvswitch-switch \
  dhcpcd-base make gcc jq curl iproute2 netcat-openbsd \
  wget unzip xz-utils file
```

Download AWS CLI v2:

```bash
curl -fsSL "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" \
  -o awscliv2.zip
```

Mirror the cloud image you intend to run as guest VMs:

```bash
mkdir -p images
curl -fsSL "https://cloud.debian.org/images/cloud/trixie/latest/debian-13-genericcloud-amd64.qcow2" \
  -o images/debian-13-amd64.qcow2
```

## Step 3. Assemble Transfer Media

```bash
mkdir -p /media/spinifex-deploy/{tarball,apt-packages,aws,images}
cp spinifex-${TAG}-linux-${ARCH}.tar.gz /media/spinifex-deploy/tarball/
cp setup.sh /media/spinifex-deploy/
cp /var/cache/apt/archives/*.deb /media/spinifex-deploy/apt-packages/
cp awscliv2.zip /media/spinifex-deploy/aws/
cp images/*.qcow2 /media/spinifex-deploy/images/
```

## Step 4. Install on the Air-Gapped Target

Mount the media, install APT dependencies, install AWS CLI, then run `setup.sh` with the local tarball:

```bash
sudo mount /dev/sdb1 /mnt/usb

sudo dpkg -i /mnt/usb/apt-packages/*.deb
sudo apt-get install -f --no-download   # resolve any leftover deps from local cache

cd /tmp && unzip /mnt/usb/aws/awscliv2.zip && sudo ./aws/install

INSTALL_SPINIFEX_TARBALL=/mnt/usb/tarball/spinifex-*-linux-*.tar.gz \
INSTALL_SPINIFEX_SKIP_APT=1 \
INSTALL_SPINIFEX_SKIP_AWS=1 \
bash /mnt/usb/setup.sh
```

Run `setup.sh` without `sudo` — the script handles privilege escalation internally and ends by `exec`-ing into a `newgrp spinifex` subshell so your current shell picks up `spinifex` group membership. Without that membership, AWS CLI cannot traverse `/etc/spinifex/` (mode `0750`) to read `ca.pem` and Step 9 will fail with a TLS error. Type `exit` to leave the subshell when finished.

## Step 5. Setup OVN Networking

`spx admin init` requires OVN/OVS to be configured before the daemon can manage tenant networks. `setup-ovn.sh` ships in the tarball and runs purely against local commands (no network access required):

```bash
sudo /usr/local/share/spinifex/setup-ovn.sh --management
```

If your WAN interface is a physical NIC rather than a bridge, pass `--wan-bridge=br-wan --wan-iface=<nic>`.

## Step 6. Initialize Without Telemetry

```bash
sudo spx admin init --node node1 --nodes 1 --no-telemetry
```

This generates the cluster configuration, TLS certificates, installs the CA into the system trust store, and writes AWS CLI credentials. See [Single-Node Install](https://docs.mulgadc.com/docs/install) for what `spx admin init` does.

## Step 7. Start Services

`setup.sh` enables `spinifex.target` but does not start it on a fresh install. Start it now so predastore is online before you import images:

```bash
sudo systemctl start spinifex.target
```

## Step 8. Import Cloud Images

Register the pre-staged image with Spinifex:

```bash
sudo spx admin images import --file /mnt/usb/images/debian-13-amd64.qcow2 \
  --distro debian --version 13 --arch x86_64 --boot-mode uefi
```

## Step 9. Verify

```bash
export AWS_PROFILE=spinifex
aws ec2 describe-instance-types
aws ec2 describe-images
```

If both calls return data, your air-gapped install is working. Continue to [Setting Up Your Cluster](https://docs.mulgadc.com/docs/setting-up-your-cluster) to launch your first instance.

## Troubleshooting

### Missing APT Dependencies

`dpkg -i` does not resolve transitive dependencies. After installing, run:

```bash
sudo apt-get install -f --no-download
```

The `--no-download` flag forces apt to use only what's in the local cache. If a dependency is genuinely missing, add it to the `apt --download-only` step on the connected machine and re-stage.

### Setup.sh Tries to Download Anyway

Confirm both skip flags are exported and that `INSTALL_SPINIFEX_TARBALL` points at a readable file:

```bash
sudo INSTALL_SPINIFEX_TARBALL=/mnt/usb/tarball/spinifex-...tar.gz \
     INSTALL_SPINIFEX_SKIP_APT=1 \
     INSTALL_SPINIFEX_SKIP_AWS=1 \
     bash -x /mnt/usb/setup.sh
```

### Init Telemetry Attempted

`spx admin init` posts a one-shot record to `https://install.mulgadc.com/install`. Pass `--no-telemetry` on every `init` and `join` invocation, or export `SPX_NO_TELEMETRY=1` in the operator's shell profile.

### Image Import Fails

`spx admin images import --file` requires the distro/version/arch/boot-mode flags so the image registers in the catalogue. If the import succeeds but `aws ec2 describe-images` returns empty, check `journalctl -u spinifex-daemon -f` for predastore upload errors.

---

# Bootable USB Install

URL: https://docs.mulgadc.com/docs/install-usb
Category: Installation
Updated: 2026-08-21
Tags: install, usb, iso, bare-metal

Install Spinifex on bare-metal x86 hardware by flashing the Spinifex ISO to a USB drive, booting the target server from it, and wiping the disk you select.

## Overview

Spinifex is designed for bare-metal hardware, edge nodes and data-centre use. Follow this guide to install Spinifex from a bootable USB using the Spinifex ISO.

**Note:** this tutorial is for x86 architecture.

**Warning:** this procedure COMPLETELY WIPES the target disk. For systems with multiple disks, ensure the correct one is targeted.

### Booting Media

For this tutorial, a USB drive with at least 8GB of memory is required.

**Note:** flashing the Spinifex ISO onto the USB completely wipes the USB, so ensure no important data is stored on the USB used.

### Balena Etcher Installation

For this tutorial download Balena Etcher to simplify the ISO flash process.

- [Balena Etcher](https://etcher.balena.io)

### Download Spinifex ISO

Download the Spinifex ISO (x86)

- [Spinifex ISO](https://iso.mulgadc.com/spinifex.iso)

### Flash Media

Once installed, open Balena Etcher. Select "Flash From Image," then select the downloaded `spinifex.iso` file.

Next, click "Select target" and choose the USB drive to be used as the boot media. Then click "Flash!"

Balena Etcher will now flash the USB drive with the `spinifex.iso` file.

<img width="1824" height="1184" src="/docs-assets/install-install-usb-balena-complete.png" alt="Balena complete">

You can now safely eject the USB drive if it was not ejected automatically.

### Boot From USB Drive

Insert your newly flashed USB drive into the target device and turn it on.

<img width="800" height="532" src="/docs-assets/install-install-usb-asus-box-new.png" alt="ASUS Box">

As it boots, quickly press the correct key for your device to bring up the BIOS/UEFI menu (commonly F2, F10, F12, ESC or DEL) and change the boot order such that the flashed USB drive has first priority, then continue to boot.

If done successfully, the Spinifex ISO GRUB menu will appear.

<img width="1824" height="1504" src="/docs-assets/install-install-usb-grub-new.png" alt="GRUB menu">

From this menu you can select which method of install is used (console recommended). Headless mode can be configured by mounting the USB on a host device after flashing and editing the `grub.cfg` file with the desired values.

### Setting Up the Spinifex Node

In console mode, follow the installation prompts to set the required networking values for the Spinifex node.

The installer will default to the most sensible disk to install on depending on system configuration — ensure this default is correct before installation, as the process will wipe the disk.

> [!WARNING]
> **Selected drives are erased unconditionally.**
>
> Every drive you select is taken over whatever it currently holds — an existing partition table, a filesystem with data on it, a ZFS pool member, or a previous Spinifex install. The installer unmounts anything mounted from those drives, disables swap on them, clears ZFS labels and filesystem signatures from every partition, and erases the partition table. There is no prompt beyond the confirmation screen and there is no rollback once it begins.
>
> Drives you do not select are never touched. The confirmation screen lists each selected drive with its current contents, and that list is the last point at which the install can be stopped.
>
> If a drive cannot be taken over — because md, LVM or device-mapper still claims it, and the ISO ships no tooling to dismantle those — the install aborts and names the drive and its holder rather than continuing against the old layout.

For network configuration, it is recommended to use automatic IP (DHCP), but this can also be configured manually.

Network interfaces will be automatically detected. In the event that none are detected, the user can manually input the name of the network interface. In the event that multiple network interfaces are detected, the installer will prompt for WAN selection first, followed by LAN.

A hostname (eg `node1`) and admin password must be set.

The installer does not ask about clustering, because it does not need to. It installs and configures a complete, working single node — operating system, disks, hostname, network interfaces, plane addressing, OVN networking, credentials and services — and starts it. A single-node deployment is finished when the installer is. A multi-node cluster is built by joining these servers together afterwards; see [Building a Cluster](#building-a-cluster) below.

Once configuration is complete, a summary of the configuration will be shown.

<img width="2784" height="1904" src="/docs-assets/install-install-usb-installer-complete.png" alt="Installer complete">

The installer will then complete the installation of Spinifex onto the target device. Once complete, remove the USB drive from the device before automatic reboot.

Once the USB drive is removed, press enter or wait for the auto-reboot.

### Log In

The device will reboot and briefly finalise the install — setting the hostname and bringing up the configured network interfaces — then prompt for login. Use the following credentials:

- Login: `spinifex`
- Password: Set by user during installation

Both before and after login, a banner will be printed specifying important information, such as the node's addresses and how to reach the web dashboard.

<img width="3228" height="1616" src="/docs-assets/install-install-usb-banner1.png" alt="banner">

### Start Using It

**There is nothing left to configure.** The node came up as a running single-node cluster: services are started, credentials are issued and networking is up. This is the point of installing from the ISO.

**Web dashboard** — browse to `https://<node-address>:3000`. It is served with the cluster's own CA, so expect a certificate warning on first visit.

**AWS CLI** — credentials were written during install, under the profile `spinifex`:

```bash
cat ~/.aws/credentials
AWS_PROFILE=spinifex aws ec2 describe-instance-types
```

That profile is the operator account, with administrator access. Copy it to your workstation to drive the node remotely — you will also need the cluster CA, available unauthenticated from `https://<node-address>:3000/api/ca.pem`.

From here, [Setting Up Your Cluster](https://docs.mulgadc.com/docs/setting-up-your-cluster) walks through importing an AMI, creating a key pair and a VPC, and launching your first instance.

### Building a Cluster

Skip this if one server is all you need.

To build a multi-node cluster, install from the ISO on **every** server first, following this guide on each one. Each comes up as its own working single node, and they are then joined together — the second and third servers discard the CA and master key they were installed with and adopt the first server's.

> [!IMPORTANT]
> Install all of the servers before joining any of them. The first server's `spx admin init` waits for the others to join, and the join has to happen while it is waiting.

Then follow [Multi-Node Install](https://docs.mulgadc.com/docs/install-multi-node), and read [Joining ISO-installed servers into a cluster](#joining-iso-installed-servers-into-a-cluster) below first — the firewall needs turning off while the cluster forms, and back on afterwards.

### Joining ISO-Installed Servers Into a Cluster

Every ISO install comes up as a **standalone cluster of one**, with a firewall that allows cluster traffic only from servers it recognises as cluster members. Right after installation, the only member each server knows about is itself.

That is exactly what you want for a single server, and it is what gets in the way of building a cluster. Servers cannot recognise each other until the cluster is formed, and they cannot form the cluster until they can talk to each other. So the order is:

1. **Turn the firewall off** on every server.
2. **Form the cluster.**
3. **Turn the firewall back on.** Each server now knows its peers and scopes itself to them automatically.

**Step 1 — before you begin, on every server:**

```bash
sudo /usr/local/lib/spinifex/spinifex-firewall-apply disable
```

Public ports — SSH, 443, 3000 (console), 8443 (S3), 9999 (AWS gateway) and 53 (DNS) — were already open and stay open. What this removes is the restriction on the internal cluster ports: OVN, NATS and the rest. Do this on **every** server, not just the first, because the servers talk to each other in both directions.

> [!WARNING]
> This leaves the internal cluster ports open to anything that can reach the server. On a machine facing the public internet, form the cluster promptly and complete step 3 as soon as it is up.

**Step 2 — form the cluster:** follow [Multi-Node Install](https://docs.mulgadc.com/docs/install-multi-node) from Step 2 onwards.

**Step 3 — once the cluster is up and verified, on every server:**

```bash
sudo systemctl restart spinifex-daemon
```

The node rewrites its peer list from the cluster it is now part of and re-arms itself, including at boot. Confirm every server can see the others:

```bash
sudo nft list table inet spinifex_filter
```

Check the `ip saddr { ... }` addresses on the cluster-plane rules — the peer list is an nft variable expanded at load time, so it has no name of its own in the output. The addresses of **all** your servers should appear. If one is missing, that server's cluster traffic will be blocked — see [Firewall and cluster membership](https://docs.mulgadc.com/docs/install-multi-node#firewall-and-cluster-membership).

### Setup Complete

**Congratulations! Spinifex is installed.**

Once configured and started, continue to [Setting Up Your Cluster](https://docs.mulgadc.com/docs/setting-up-your-cluster) to import an AMI, create a VPC, and launch your first instance.

## Troubleshooting

### Can't Access BIOS/UEFI to Change Boot Order

It can be difficult to get into the BIOS/UEFI — there is only a short window to press the correct key, and this key changes depending on the manufacturer. Search online for your device's BIOS/UEFI key, and press it rapidly as the device boots.

### GRUB Menu Not Appearing

Once in the BIOS/UEFI menu, ensure the correct boot order is set. First boot priority should be set to the USB drive flashed with the ISO — if the USB doesn't appear in the BIOS/UEFI menu, ensure it has been flashed with the ISO correctly. Take note of the name and storage capacity of the USB when flashing, as this should match what appears in the BIOS/UEFI.

### Networking Issues

For a Spinifex node to be properly provisioned, the target device must have at least one NIC.

Spinifex uses DHCP to assign instances within a node their public IP addresses. Instead of a static range, public IPs come from the upstream router's DHCP server. When a VM launches, Spinifex requests a DHCP lease from the router on behalf of the VM. When the VM terminates, the lease is released.

The VM itself never talks to the router's DHCP — it only sees its private VPC IP (from OVN's internal DHCP). The host-side DHCP conversation is invisible to the guest.

**Use when:** You don't control a static IP block but the router's DHCP server has enough leases. Homelabs where you don't want to carve out a range. Environments where IPs are managed centrally by the network team's DHCP.

For further troubleshooting suggestions, refer to the [VPC Networking](https://docs.mulgadc.com/docs/vpc-networking) guide.

---

# Setting Up Your Cluster

URL: https://docs.mulgadc.com/docs/setting-up-your-cluster
Category: Administration
Updated: 2026-09-10
Tags: setup, vpc, ec2, quickstart

Import an AMI, create an SSH key pair and a VPC with a public subnet, then launch your first EC2 instance and connect to it on a fresh Spinifex cluster.

## Overview

This guide walks through the first steps on a freshly installed cluster: importing an AMI, creating an SSH key and a VPC with a public subnet, launching an instance, and connecting to it.

## Prerequisites

- Spinifex installed and running — see [Single-Node Install](https://docs.mulgadc.com/docs/install), [Multi-Node Install](https://docs.mulgadc.com/docs/install-multi-node), or [Source Install](https://docs.mulgadc.com/docs/install-source)
- AWS CLI configured with the `spinifex` profile:

```bash
export AWS_PROFILE=spinifex
```

## 1. Import an AMI

### Option A: Import a bundled image

List the bundled images and import one matching your architecture:

```bash
spx admin images list
```

```
NAME                    | DISTRO | VERSION | ARCH   | BOOT
debian-13-arm64         | debian | 13      | arm64  | uefi
debian-13-x86_64        | debian | 13      | x86_64 | uefi
ubuntu-26.04-arm64      | ubuntu | 26.04   | arm64  | uefi
ubuntu-26.04-x86_64     | ubuntu | 26.04   | x86_64 | uefi
```

```bash
spx admin images import --name ubuntu-26.04-x86_64
```

### Option B: Import a local image file

```bash
spx admin images import --file ~/images/ubuntu-26.04.img --distro ubuntu --version 26.04 --arch x86_64 --boot-mode uefi
```

Verify the import and note the AMI ID:

```bash
AMI_ID=$(aws ec2 describe-images --query 'Images[0].ImageId' --output text)
```

## 2. Create an SSH Key

### Option A: Import an existing key

```bash
aws ec2 import-key-pair \
  --key-name "spinifex-key" \
  --public-key-material fileb://~/.ssh/id_rsa.pub
```

### Option B: Create a new key pair

```bash
aws ec2 create-key-pair --key-name spinifex-key \
  | jq -r '.KeyMaterial | rtrimstr("\n")' > ~/.ssh/spinifex-key

chmod 600 ~/.ssh/spinifex-key
ssh-keygen -y -f ~/.ssh/spinifex-key > ~/.ssh/spinifex-key.pub
```

Verify:

```bash
aws ec2 describe-key-pairs
```

## 3. Create a VPC and Public Subnet

### Create a VPC

```bash
VPC_ID=$(aws ec2 create-vpc --cidr-block 10.200.0.0/16 \
  --query 'Vpc.VpcId' --output text)
```

### Create an Internet Gateway

An Internet Gateway enables instances in public subnets to reach the internet and be reachable from the LAN/WAN.

```bash
IGW_ID=$(aws ec2 create-internet-gateway \
  --query 'InternetGateway.InternetGatewayId' --output text)

aws ec2 attach-internet-gateway \
  --internet-gateway-id $IGW_ID \
  --vpc-id $VPC_ID
```

### Create a Subnet

Create the subnet your instances will launch into. The routing and public-IP steps below are what make it a *public* subnet.

```bash
SUBNET_ID=$(aws ec2 create-subnet \
  --vpc-id $VPC_ID \
  --cidr-block 10.200.1.0/24 \
  --query 'Subnet.SubnetId' --output text)
```

### Create a Route Table

A subnet is only **public** if its route table has a default route to the Internet Gateway. Spinifex does **not** add this route automatically — a new VPC's route table only routes traffic within the VPC (matching AWS). Without a `0.0.0.0/0` route to the IGW, instances in the subnet cannot reach the internet (and inbound connections cannot complete) even with a public IP assigned.

```bash
RT_ID=$(aws ec2 create-route-table \
  --vpc-id $VPC_ID \
  --query 'RouteTable.RouteTableId' --output text)

# Default route to the internet via the IGW
aws ec2 create-route \
  --route-table-id $RT_ID \
  --destination-cidr-block 0.0.0.0/0 \
  --gateway-id $IGW_ID

# Associate the route table with the subnet
aws ec2 associate-route-table \
  --route-table-id $RT_ID \
  --subnet-id $SUBNET_ID
```

### Enable Auto-Assign Public IP

Give every instance launched into the subnet a routable public IP, making it directly reachable from your network.

```bash
aws ec2 modify-subnet-attribute \
  --subnet-id $SUBNET_ID \
  --map-public-ip-on-launch
```

### Allow SSH and ICMP

Every VPC gets a default security group that **blocks inbound traffic from outside the group** (matching AWS). Instances launched without an explicit security group use this default, so you must authorize ingress before you can SSH or ping the instance.

```bash
SG_ID=$(aws ec2 describe-security-groups \
  --filters Name=vpc-id,Values=$VPC_ID \
  --query 'SecurityGroups[0].GroupId' --output text)

# Allow SSH from anywhere
aws ec2 authorize-security-group-ingress \
  --group-id $SG_ID \
  --protocol tcp --port 22 --cidr 0.0.0.0/0

# Allow ICMP (ping) from anywhere
aws ec2 authorize-security-group-ingress \
  --group-id $SG_ID \
  --protocol icmp --port -1 --cidr 0.0.0.0/0
```

**Note:** `0.0.0.0/0` opens these ports to every source. For anything beyond a quick evaluation, scope the `--cidr` to a trusted range. Rule changes apply immediately — no instance restart needed.

### Verify

```bash
aws ec2 describe-vpcs --vpc-ids $VPC_ID
aws ec2 describe-subnets --subnet-ids $SUBNET_ID
aws ec2 describe-route-tables --route-table-ids $RT_ID
aws ec2 describe-security-groups --group-ids $SG_ID
```

## 4. Launch an Instance

Launch an instance in the public subnet on a `t3.micro` (2 vCPU / 1 GiB).

> **Note:** On an **arm64** host, use `t4g.micro` instead of `t3.micro`.

```bash
INSTANCE_ID=$(aws ec2 run-instances \
  --image-id $AMI_ID \
  --instance-type t3.micro \
  --key-name spinifex-key \
  --subnet-id $SUBNET_ID \
  --count 1 \
  --query 'Instances[0].InstanceId' --output text)
```

Wait for the instance to reach `running` state:

```bash
aws ec2 describe-instances --instance-ids $INSTANCE_ID \
  --query 'Reservations[0].Instances[0].[State.Name, PrivateIpAddress, PublicIpAddress]' \
  --output text
```

Expected output:

```
running    10.200.1.4    192.168.1.155
```

The instance has both a private IP (VPC overlay) and a public IP (from your external pool, routable on your network).

## 5. Connect via SSH

SSH directly to the instance's public IP:

```bash
PUBLIC_IP=$(aws ec2 describe-instances --instance-ids $INSTANCE_ID \
  --query 'Reservations[0].Instances[0].PublicIpAddress' --output text)

ssh -i ~/.ssh/spinifex-key ubuntu@$PUBLIC_IP
```

> **Note:** cloud-init takes 30-60 seconds to configure the instance. If SSH is refused, wait and retry.

Once connected, verify the instance has internet access:

```bash
curl -s http://ifconfig.me
```

This should return the instance's public IP or the gateway's SNAT address.

## 6. Managing Instances

### Stop

```bash
aws ec2 stop-instances --instance-ids $INSTANCE_ID
```

### Start

```bash
aws ec2 start-instances --instance-ids $INSTANCE_ID
```

### Terminate

```bash
aws ec2 terminate-instances --instance-ids $INSTANCE_ID
```

### Console Output

View the instance's serial console log (useful for debugging boot issues):

```bash
aws ec2 get-console-output --instance-id $INSTANCE_ID \
  --query 'Output' --output text
```

### Multi-Node: Check Instance Placement

On a multi-node cluster, instances are distributed across nodes:

```bash
spx get vms
```

## 7. Launching the Web UI

Spinifex ships with a built-in web console — an alternative to the AWS CLI, SDKs, and Terraform, analogous to the AWS Management Console. Every action in the UI is a standard AWS SigV4 API call, so the same IAM policies and audit behaviour apply.

<p align="center">
  <img height="621" src="/docs-assets/admin-setting-up-your-cluster-spinifex-ui.jpg" alt="Spinifex Web UI" width="900">
</p>

### Open the Console

The UI is served by each node on port `3000` over TLS. Replace `YOUR_NODE_IP` with the address of the node you installed Spinifex on (or `localhost` if you're on the node itself):

```
https://YOUR_NODE_IP:3000
```

### Trust the Self-Signed Certificate (required)

On first load, your browser will show a TLS warning — Spinifex generates a self-signed certificate at install time. This is expected.

1. Accept the warning to reach the login page (exact wording varies by browser — e.g. Chrome: *Advanced → Proceed to ...*, Firefox: *Advanced → Accept the Risk and Continue*).
2. On the login page, click **Download Certificate** and save `spinifex-ca.pem` to your machine.
3. Install the certificate as a **trusted root** on your workstation, following the steps for your platform below.
4. Restart your browser and reload `https://YOUR_NODE_IP:3000`. The padlock should now show a valid certificate.

Step 3, by platform:

- **macOS:** open `spinifex-ca.pem` in Keychain Access → *System* keychain → set *Trust* to **Always Trust**.
- **Linux:** `sudo cp spinifex-ca.pem /usr/local/share/ca-certificates/spinifex-ca.crt && sudo update-ca-certificates`
- **Windows:** double-click the file → *Install Certificate* → *Local Machine* → *Trusted Root Certification Authorities*.
- **Browser-only (Firefox):** *Settings → Privacy & Security → Certificates → View Certificates → Authorities → Import* and tick *Trust this CA to identify websites*.

> **Why this is required:** the UI logs in by reading your AWS credentials through a trusted TLS channel. Browsers refuse to send credentials over an untrusted connection, so the certificate must be installed as trusted — temporary "Proceed anyway" exceptions won't work for login.

### Log In with AWS Credentials

The console authenticates against the AWS credentials in `~/.aws/credentials` on the node where Spinifex is installed:

```ini
[spinifex]
aws_access_key_id     = AKIA...
aws_secret_access_key = ...
```

At the Spinifex login screen, paste the **Access Key ID** and **Secret Access Key** from the `[spinifex]` profile (or whichever profile maps to the IAM user or role you want to use). Additional users and policies can be managed through the UI or via `aws iam` commands — see [IAM Users and Policies](https://docs.mulgadc.com/docs/iam-users-and-policies).

Once logged in, you have browser-based access to every Spinifex feature: launch and manage instances, attach EBS volumes, browse S3 buckets, configure VPCs and security groups, and manage IAM users and keys — all backed by the same AWS-compatible control plane the CLI uses.

## Additional Options

### Private Subnets (No Public IP)

Create a subnet without `--map-public-ip-on-launch`:

```bash
PRIVATE_SUBNET=$(aws ec2 create-subnet \
  --vpc-id $VPC_ID \
  --cidr-block 10.200.2.0/24 \
  --query 'Subnet.SubnetId' --output text)
```

Instances in a private subnet get a private IP only and are not reachable from your network. By default they also have **no internet access**: with no `0.0.0.0/0` route in the subnet's route table, Spinifex gates egress with a drop policy — matching AWS, where a private subnet has no route off the VPC. They can still reach other instances in the same VPC.

To give a private subnet outbound-only internet access, deploy a NAT Gateway in a public subnet and point the private subnet's default route at it (the AWS pattern):

```bash
# Allocate an Elastic IP for the NAT Gateway
NAT_EIP=$(aws ec2 allocate-address --query AllocationId --output text)

# Create the NAT Gateway in the PUBLIC subnet (the one with the IGW route)
NATGW_ID=$(aws ec2 create-nat-gateway \
  --subnet-id $SUBNET_ID \
  --allocation-id $NAT_EIP \
  --query 'NatGateway.NatGatewayId' --output text)

# Give the private subnet its own route table with a default route to the NAT Gateway
PRIVATE_RT=$(aws ec2 create-route-table \
  --vpc-id $VPC_ID \
  --query 'RouteTable.RouteTableId' --output text)

aws ec2 create-route \
  --route-table-id $PRIVATE_RT \
  --destination-cidr-block 0.0.0.0/0 \
  --nat-gateway-id $NATGW_ID

aws ec2 associate-route-table \
  --route-table-id $PRIVATE_RT \
  --subnet-id $PRIVATE_SUBNET
```

Instances in the private subnet now reach the internet outbound through the NAT Gateway's public IP, but remain unreachable from the WAN.

### Multiple Accounts

Create isolated accounts with their own resources:

```bash
spx admin account create --name myteam
export AWS_PROFILE=spinifex-myteam
```

## Troubleshooting

### Instance Stuck in Pending

```bash
journalctl -u spinifex-daemon -f
aws ec2 describe-images
```

### SSH Connection Refused or Times Out

cloud-init needs 30-60 seconds after boot. Check instance state:

```bash
aws ec2 describe-instances --instance-ids $INSTANCE_ID
```

A connection that **times out** (rather than being refused) usually means the security group is blocking port 22. Confirm the default security group allows SSH ingress:

```bash
aws ec2 describe-security-groups --group-ids $SG_ID \
  --query 'SecurityGroups[0].IpPermissions'
```

If there are no rules for TCP 22, authorize it (see [Allow SSH and ICMP](#allow-ssh-and-icmp)):

```bash
aws ec2 authorize-security-group-ingress \
  --group-id $SG_ID --protocol tcp --port 22 --cidr 0.0.0.0/0
```

### No Public IP Assigned

Verify the subnet has `MapPublicIpOnLaunch` enabled:

```bash
aws ec2 describe-subnets --subnet-ids $SUBNET_ID
```

If `MapPublicIpOnLaunch` is false:

```bash
aws ec2 modify-subnet-attribute --subnet-id $SUBNET_ID --map-public-ip-on-launch
```

Also verify an Internet Gateway is attached to the VPC:

```bash
aws ec2 describe-internet-gateways
```

### Instance Has No Internet Access

First confirm the subnet's route table has a default route to the IGW. Spinifex gates a subnet's egress with a drop policy when this route is missing, so an instance with a public IP still cannot reach the internet:

```bash
aws ec2 describe-route-tables \
  --filters Name=association.subnet-id,Values=$SUBNET_ID \
  --query 'RouteTables[0].Routes'
# Expect a 0.0.0.0/0 route with a GatewayId of igw-...
```

If the route is missing, add it (see [Create a Route Table](#create-a-route-table)):

```bash
aws ec2 create-route --route-table-id $RT_ID \
  --destination-cidr-block 0.0.0.0/0 --gateway-id $IGW_ID
```

If the route is present, check the VPC router's NAT rules (from the host):

```bash
sudo ovn-nbctl lr-nat-list $(sudo ovn-nbctl lr-list | awk '{print $2}' | head -1)
```

Verify the default route exists:

```bash
sudo ovn-nbctl lr-route-list $(sudo ovn-nbctl lr-list | awk '{print $2}' | head -1)
```

---

# Spinifex Admin CLI

URL: https://docs.mulgadc.com/docs/spinifex-admin-cli
Category: Administration
Updated: 2026-08-21
Tags: cli, admin, reference

Complete reference for spx, the Spinifex admin CLI: initialise a cluster, manage accounts and nodes, drive the VM lifecycle, and start or stop services.

## Overview

The `spx` binary is the central administration tool for managing your Spinifex infrastructure. It provides commands for cluster initialization, account management, node operations, VM lifecycle, and service control. All services in the Spinifex platform are managed through this single binary.

**Binary location:** `/usr/local/bin/spx`

## Instructions

## Account Management

Create a new isolated account. This provisions a sequential 12-digit account ID, an `admin` user with an `AdministratorAccess` policy attached, and an access key pair. The credentials are written to `~/.aws/credentials` and `~/.aws/config` under a `spinifex-<name>` profile automatically.

```bash
spx admin account create --name myteam
```

```
Account created successfully!
  Account ID:        000000000002
  Account Name:      myteam
  Admin User:        admin
  Access Key ID:     AKIA1A2B3C4D5E6F7890ABCD
  Secret Access Key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
  AWS Profile:       spinifex-myteam

Use with:
  AWS_PROFILE=spinifex-myteam aws ec2 describe-instances
```

> **The secret access key is only shown once.** It is saved to `~/.aws/credentials` on the node where you ran the command; copy it from there if you need it elsewhere.

Set the profile to start using the account:

```bash
export AWS_PROFILE=spinifex-myteam
```

To create additional users and scoped permissions within the account, see [IAM Users and Policies](https://docs.mulgadc.com/docs/iam-users-and-policies).

List all accounts:

```bash
spx admin account list
```

```
ACCOUNT ID     NAME                 STATUS     CREATED
----------     ----                 ------     -------
000000000000   system               ACTIVE     2026-07-03 02:17
000000000001   spinifex             ACTIVE     2026-07-03 02:17
000000000002   myteam               ACTIVE     2026-07-03 03:32
```

Deleting an account, and creating or removing accounts from your own signup or provisioning system over the private admin API, are covered in [Account Management](https://github.com/mulgadc/spinifex/blob/main/docs/admin/account-management/README.md).

## Node Management

List nodes in the cluster:

```bash
spx get nodes
```

```
NAME    STATUS    IP              REGION           AZ               UPTIME   VMs
node1   Ready     127.0.0.1       ap-southeast-2   ap-southeast-2a  2m       0
node2   Ready     127.0.0.2       ap-southeast-2   ap-southeast-2a  2m       0
node3   Ready     127.0.0.3       ap-southeast-2   ap-southeast-2a  2m       0
```

## Monitor Resources

```bash
spx top nodes
```

Prints per-node CPU/memory usage and cluster-wide instance type availability.

## Image Management

```bash
spx admin images list
spx admin images import --name debian-13-arm64
```

Catalog imports verify the image against the catalog-declared SHA-256/SHA-512 digest before extraction. Use `--file` to import operator-supplied media (verification skipped — operator is responsible for integrity), or `--force` to re-download after a checksum mismatch.

### EKS node image

To run EKS, import the prebuilt node image from the catalog:

```bash
spx admin images import --name spinifex-eks-node
```

This pulls the Alpine + K3s node AMI from `iso.mulgadc.com`, verifies its checksum, and registers it tagged `spinifex:managed-by=eks`. `eks create-cluster` and `eks create-nodegroup` resolve the boot AMI by that tag, so no further configuration is needed.

## Cluster Shutdown

Coordinated, phased shutdown of the entire cluster — running VMs are stopped before storage and control-plane services:

```bash
spx admin cluster shutdown
```

## Troubleshooting

### Permission Denied Running Spinifex

The binary may not be executable. Fix permissions:

```bash
chmod +x /usr/local/bin/spx
```

If you get permission errors during operations, ensure you're running with appropriate privileges. Some OVN and networking commands require `sudo`.

### Services Fail to Start

Check the daemon logs for specific errors via `systemctl`/`journalctl`:

```bash
systemctl status 'spinifex-*'
journalctl -u spinifex-daemon -f
journalctl -u 'spinifex-*' -f
```

Common causes include port conflicts, missing OVN configuration, or untrusted CA certificates.

---

# Updating Spinifex

URL: https://docs.mulgadc.com/docs/update
Category: Administration
Updated: 2026-09-10
Tags: update, upgrade, migrate

Upgrade an existing Spinifex install with the same installer used to deploy it, or take the manual path to review configuration migrations before applying them.

## Overview

Updating Spinifex is the same command used to install it. The installer detects an existing installation, downloads the latest binary and runs any pending configuration migrations before restarting services.

For operators who want to review migrations before they are applied, a manual upgrade path is also supported.

> [!WARNING]
> **Swapping the `spx` binary alone is not an upgrade.** Systemd unit files (`KillMode`, `TimeoutStopSec`, drain ordering, and similar) are written once at install time and never re-asserted just because a new binary is in place — a node "upgraded" by replacing `/usr/local/bin/spx` directly keeps running whatever units it was first installed with, which can silently disagree with the new binary's behaviour. Re-running the installer always reinstalls units unconditionally, so it is unaffected. `spx admin upgrade` now reconciles units too, so it closes this gap for operators who update the binary by hand. See [Checking for Unit Drift](#checking-for-unit-drift).

## Instructions

## Step 1. Re-run the Installer

```bash
curl -fsSL https://install.mulgadc.com | bash
```

That's it. The installer will:

1. Download and install the latest Spinifex binary.
2. Reinstall systemd units so new services are picked up.
3. Run any pending configuration migrations automatically (equivalent to `spx admin upgrade --yes`).
4. Restart `spinifex.target` if the services were already running.

## Step 2. Verify

```bash
export AWS_PROFILE=spinifex
aws ec2 describe-instance-types
```

If this returns a list of instance types, your upgrade is complete.

## Manual Upgrade

If you prefer to review pending migrations before they are applied, Spinifex supports running `spx admin init` to allow you to verify config file migrations.

## Step 1. Install the New Binary Without Running Migrations

```bash
curl -fsSL https://install.mulgadc.com | INSTALL_SPINIFEX_SKIP_MIGRATE=1 bash
```

The installer will download the new binary and reinstall systemd units, but will **not** apply any configuration migrations.

## Step 2. Review Pending Changes

```bash
sudo spx admin upgrade
```

The command prints the current version of each config file and systemd unit, the migrations and unit replacements that would be applied, and a `from → to` description for each. It then prompts for confirmation before making any changes. Answer `n` to abort without touching config or units. Use `--dry-run` instead of the prompt to only report and never apply.

## Step 3. Apply Changes

When you are ready, answer `y` at the prompt, or re-run with `--yes` to apply non-interactively:

```bash
sudo spx admin upgrade --yes
```

This requires root: config files are typically owned by their service user, but writing `/etc/systemd/system` needs root. Run the whole command with `sudo`, not just parts of it.

## Step 4. Restart Services

Migrations modify config files on disk but do not restart running services, and unit reconciliation deliberately never restarts anything either — it writes the unit and runs `systemctl daemon-reload`, so the fix applies to the *next* stop of that service without disturbing a running guest. Apply a config change with:

```bash
sudo systemctl restart spinifex.target
```

A restart preserves any running guests — they are not rebooted, and storage returns within seconds. See [Host and Guest Lifecycle](https://github.com/mulgadc/spinifex/blob/main/docs/admin/host-lifecycle/README.md) for the full contract.

## Checking for Unit Drift

`spx admin upgrade --dry-run` reports whether a node's installed systemd units match the ones shipped in the running `spx` binary, without prompting or changing anything — the fastest way to answer "are this node's units current?" in a support conversation:

```bash
spx admin upgrade --dry-run
```

Each unit is reported as one of:

- **up to date** — installed content matches the embedded copy.
- **stale, will replace** — the installed marker version is older than the embedded one (or has no marker at all, which is version 0 — every node installed before units were versioned).
- **missing → will install** — no unit installed under that name.
- **operator-modified, not touched** — the installed marker version matches, but the content differs. `spx admin upgrade` never overwrites this case; review it with `systemctl cat <unit>` and reconcile by hand with `systemctl edit`.

Run `sudo spx admin upgrade --units-only --yes` to reconcile units without touching config, or `--skip-units` to do the reverse. A replaced unit is backed up alongside the original as `<unit>.pre-reconcile-<from>to<to>.<unix-timestamp>` before being overwritten.

This covers the 16 core units installed by `install_systemd()` (`spinifex-*.service`, `spinifex.target`, `spinifex.slice` and friends). Firstboot, banner, bridge and getty units written by the installer, and the auxiliary units Ansible manages (`wattle-wan-veth-persist`, `wattle-mgmt-bridge-persist`, `obs-agent`), are separate lifecycles with no overlap in unit names and are out of scope for this reconciler.

## Troubleshooting

### No Pending Config Migrations

```
No pending config migrations.
```

Your config is already at the latest version. Nothing to do.

### No Spinifex Installation Found

```
No Spinifex installation found at /etc/spinifex
Run 'spx admin init' first.
```

`spx admin upgrade` requires an initialized installation. If this is a fresh host, follow the [Single-Node Install](https://docs.mulgadc.com/docs/install) guide instead.

### Migration Failure

If a migration fails, the installer and `spx admin upgrade` exit non-zero and leave the config in its prior state where possible. Review the error output, then re-run `sudo spx admin upgrade` once the underlying issue is resolved.

### Services Did Not Pick Up New Config

Migrations edit config files on disk but the running daemons continue to use the config they loaded at start-up. Restart with:

```bash
sudo systemctl restart spinifex.target
```

A restart preserves any running guests — they are not rebooted, and storage returns within seconds. See [Host and Guest Lifecycle](https://github.com/mulgadc/spinifex/blob/main/docs/admin/host-lifecycle/README.md) for the full contract.

### A Node Answers Some Requests as if It Were Still on the Old Build

Replacing `/usr/local/bin/spx` while `spinifex.target` is running does **not** move the running services onto the new binary. They keep executing the replaced file's now-unlinked inode until each unit restarts, so a node can serve the old build indefinitely. Only a service that happens to restart for its own reasons picks the new one up, which leaves a node running a mixture.

This is easy to miss, because the request handlers are NATS queue-group workers spread across nodes: one skewed node in three answers roughly one request in three with the old behaviour, which reads as an intermittent fault rather than a broken node.

Check for it with:

```bash
sudo spx admin preflight
```

Any unit reported `Stale` with kind `service` is running a replaced binary. The check covers every `.service` unit this build ships, and exits non-zero when it finds one, so it also works as a gate in a script. To see the raw state instead:

```bash
for u in spinifex-daemon spinifex-awsgw spinifex-viperblock spinifex-vpcd spinifex-ui spinifex-predastore; do
  pid=$(systemctl show -p MainPID --value "$u")
  [ "$pid" != "0" ] && printf '%-22s %s\n' "$u" "$(sudo readlink /proc/$pid/exe)"
done
```

Any line ending `(deleted)` is running a replaced binary. Restart the target to clear it:

```bash
sudo systemctl restart spinifex.target
```

Re-running the installer avoids this entirely — it restarts services after installing. Prefer it over copying a binary onto a live node.

### Instances Fail to Launch After an Upgrade

```
AMI has no snapshot ID, cannot perform zero-copy clone
```

Or `describe-images --image-ids` reports `InvalidAMIID.NotFound` for an AMI that still appears in the unfiltered `describe-images` list. Two different causes produce this, so check them in order:

1. **A node running a replaced binary**, per the previous entry. Suspect this first if the failure is intermittent — the same command succeeding on some attempts and failing on others is characteristic.
2. **AMI metadata predating the EBS-provider decoupling**, per the warning at the top of this page. This is consistent rather than intermittent, and is resolved by re-importing the AMI.

### Root Privileges Required to Write Systemd Units

```
root privileges required to write systemd units (writing to /etc/systemd/system): ...
Re-run as root to apply the unit changes reported above: sudo spx admin upgrade --units-only --yes
```

`spx admin upgrade` computes and prints unit drift without needing root, but writing `/etc/systemd/system` does. Nothing is written when this happens — re-run the full command with `sudo`.

### Operator-Modified Unit Reported, Not Replaced

A unit whose installed marker version matches the embedded one but whose content differs is never overwritten — this is a deliberate safety property so a hand-tuned unit does not get silently reverted. Compare it against the shipped copy and decide whether to keep, discard or merge the local change:

```bash
systemctl cat <unit>
```

If you want the shipped version, remove the local override and re-run `spx admin upgrade`; systemd falls back to the packaged unit and it reconciles as up to date.

---

# VPC Networking

URL: https://docs.mulgadc.com/docs/vpc-networking
Category: Compute and Networking
Updated: 2026-09-10
Tags: vpc, networking, ovn, public-subnet, security-groups

How Spinifex implements AWS-compatible VPC networking on bare metal with OVN: public and private subnets, security groups, route tables, and Elastic IPs.

## Overview

Spinifex provides AWS-compatible VPC networking on bare-metal. Every EC2 instance runs inside an isolated virtual network backed by OVN (Open Virtual Network). Instances can operate in two modes: **private** (overlay-only, no WAN access) or **public** (routable from the WAN with a unique public IP).

## Instructions

## How It Works

Spinifex maps AWS VPC concepts directly to OVN constructs:

| AWS Concept      | OVN Construct            | What It Does                                       |
| ---------------- | ------------------------ | -------------------------------------------------- |
| VPC              | Logical Router           | Isolates tenant networks, routes between subnets   |
| Subnet           | Logical Switch + DHCP    | L2 broadcast domain with automatic IP assignment   |
| ENI              | Logical Switch Port      | Per-instance network interface with MAC/IP binding |
| Internet Gateway | External Switch + NAT    | Connects VPC router to physical WAN                |
| Security Group   | Port Group + ACLs        | Stateful firewall rules enforced in OVS datapath   |
| Elastic IP       | `dnat_and_snat` NAT rule | Static 1:1 NAT between public and private IP       |

## Network Path

<p align="center">
  <img height="560" src="/docs-assets/compute-vpc-networking-vpc-network-path.svg" alt="VPC logical topology — WAN, br-wan, VPC logical router, subnets, ENIs" width="900">
</p>

Cross-host traffic uses **Geneve tunnels** (UDP 6081) over the management/overlay NIC. Each host runs `ovn-controller` which programs OpenFlow rules on `br-int` (the integration bridge where all VM TAP devices connect).

## Private vs Public Subnets

A subnet's behavior depends on three things: whether the VPC has an Internet Gateway, whether the subnet's route table has a default route to that IGW, and whether the subnet has `MapPublicIpOnLaunch` enabled.

## Private Subnet (Default)

Instances get a private IP only. They can communicate with other instances in the same VPC (even across subnets and hosts via the overlay). They cannot reach the internet or be reached from the WAN.

<p align="center">
  <img height="110" src="/docs-assets/compute-vpc-networking-vpc-private-subnet-flow.svg" alt="Private subnet — instance hits router, no default route, packet dropped" width="900">
</p>

Private subnet instances reach the internet only if their route table has a default route to the IGW (shared SNAT, outbound only — they share the gateway IP) or to a NAT gateway. With no default route, egress is dropped. Either way they cannot be reached from the WAN because they have no public IP.

## Public Subnet

Instances get both a private IP and a public IP. The public IP is a 1:1 NAT managed by OVN — the instance OS only sees its private IP.

<p align="center">
  <img height="170" src="/docs-assets/compute-vpc-networking-vpc-public-subnet-flow.svg" alt="Public subnet — outbound SNAT and inbound DNAT between private and public IPs" width="900">
</p>

**Requirements for a public subnet:**

1. VPC has an Internet Gateway attached
2. A route table associated with the subnet has a `0.0.0.0/0` route to the IGW
3. Subnet has `MapPublicIpOnLaunch = true`
4. External IP pool configured in `spinifex.toml`

Spinifex follows AWS route-table semantics: a subnet is only "public" if its effective route table carries a default route to the IGW. A new VPC's main route table has the local route only — Spinifex does **not** add the IGW route for you. Without it, the subnet's egress is gated with a drop policy, so instances cannot reach the internet (and inbound connections cannot complete because return traffic is dropped) even with a public IP and an attached IGW. Add the route explicitly — either to the main route table, or to a custom route table associated with the subnet (see [Quick Start](#3-create-vpc-with-public-subnet)).

## Comparison

|                          | Private Subnet                    | Public Subnet                  |
| ------------------------ | --------------------------------- | ------------------------------ |
| Private IP               | Yes                               | Yes                            |
| Public IP                | No                                | Auto-assigned from pool        |
| Outbound internet        | Only with a default route to IGW/NAT GW | Yes (own public IP via SNAT)   |
| Inbound from WAN         | No                                | Yes (via 1:1 NAT to public IP) |
| Instance sees public IP? | N/A                               | No — only sees private IP      |
| Elastic IP support       | Only if explicitly associated     | Yes                            |

## External Connectivity Modes

The `[network]` section in `spinifex.toml` controls how VMs reach the outside world. There are three modes, and pool mode has two IP sources (static or DHCP).

## `pool` — Full Public Networking (Recommended)

Each VM in a public subnet gets its own public IP with bidirectional 1:1 NAT. Supports the full AWS feature set: public subnets, auto-assign public IPs, Elastic IPs, and security groups.

Pool mode supports two ways to obtain public IPs:

### Static Range (default)

The admin defines a range of routable IPs that Spinifex manages exclusively.

**Use when:** You have a block of IPs you control — datacenter ISP allocation, homelab range carved out of your router's DHCP scope, enterprise DMZ range.

**Requirement:** The IP range must NOT be served by any other DHCP server. In a homelab, shrink your router's DHCP scope to exclude the Spinifex range.

```toml
[network]
external_mode = "pool"

[[network.external_pools]]
name        = "wan"
range_start = "192.168.1.150"
range_end   = "192.168.1.250"
gateway     = "192.168.1.1"       # Router / next-hop IP
prefix_len  = 24
dns_servers = ["192.168.1.1", "8.8.8.8"]
```

### DHCP Source

Instead of a static range, public IPs come from the upstream router's DHCP server. When a VM launches, Spinifex requests a DHCP lease from the router on behalf of the VM. When the VM terminates, the lease is released.

The VM itself never talks to the router's DHCP — it only sees its private VPC IP (from OVN's internal DHCP). The host-side DHCP conversation is invisible to the guest.

**Use when:** You don't control a static IP block but the router's DHCP server has enough leases. Homelabs where you don't want to carve out a range. Environments where IPs are managed centrally by the network team's DHCP.

**Requirement:** `dhclient` or `dhcpcd-base` installed on the host.

```toml
[network]
external_mode = "pool"

[[network.external_pools]]
name        = "wan"
source      = "dhcp"              # "static" (default) or "dhcp"
gateway     = "192.168.1.1"       # Router / next-hop IP
prefix_len  = 24
dns_servers = ["192.168.1.1", "8.8.8.8"]
# No range_start/range_end — IPs come from router DHCP
```

### How Pool Mode Works (Both Sources)

Regardless of whether IPs come from a static range or DHCP, the OVN behavior is identical:

<p align="center">
  <img height="360" src="/docs-assets/compute-vpc-networking-vpc-dhcp-conversations.svg" alt="Two independent DHCP conversations — host-to-router and VM-to-OVN" width="900">
</p>

### Choosing Static vs DHCP

|                       | Static Range                                 | DHCP Source                                      |
| --------------------- | -------------------------------------------- | ------------------------------------------------ |
| **Public IPs from**   | Admin-defined `range_start`..`range_end`     | Router's DHCP server                             |
| **IP predictability** | You know the exact range                     | Router assigns whatever is available             |
| **Setup effort**      | Must reserve range, shrink router DHCP scope | Just set `source = "dhcp"`                       |
| **Dependency**        | None                                         | Requires `dhclient` on host, working router DHCP |
| **Best for**          | Datacenters, ISP blocks, production          | Homelabs, dev environments, shared networks      |
| **Capacity**          | Exact: `range_end - range_start` IPs         | Limited by router's DHCP pool size               |

Both support the same AWS features: public subnets, Elastic IPs, security groups, DescribeInstances showing public IPs.

## `nat` — Shared SNAT (Simple)

All VMs share a single external IP for outbound SNAT. By default there are no public IPs, no Elastic IPs, and no inbound from WAN — all subnets behave as private subnets with internet access. On routed-NAT nodes, adding a public pool restores full public IP parity (see below).

> **Limitation (routed-NAT v1):** System instances (ECS/EKS/load-balancer
> agents) source egress from ExternalIPAM pool IPs, which do not exist in
> `external_mode=nat`. Those features require `external_mode=pool`. A v2 will
> either allocate transit IPs for system instances or reject the feature at the
> API level in nat mode.

The `gateway_ip` is the IP that OVN uses for SNAT. You can set it statically or use `setup-ovn.sh --dhcp` to obtain one from the router. This is the router's DHCP — not Spinifex's internal OVN DHCP for VMs.

**Use when:** VMs only need outbound access (apt update, pulling images). Edge deployments behind ISP NAT. Single WAN IP available.

```toml
[network]
external_mode = "nat"

[[network.external_pools]]
name       = "wan"
gateway    = "192.168.1.1"
gateway_ip = "192.168.1.100"     # Single IP for all VM outbound SNAT
prefix_len = 24
```

### Host access to instances (jumpbox pattern)

The spinifex host automatically reaches every instance's **private IP**: IGW attach installs a host route into OVN (`<vpc-cidr> via <gateway-transit-ip> dev spx-nat-host`) and exempts the transit net from SNAT, so replies to host-initiated connections come back un-NATted. No per-instance setup.

Security groups still apply and the default SG is closed to the host, same as AWS — open SSH/ICMP from the transit net first:

```bash
aws ec2 authorize-security-group-ingress --group-id $SG \
  --protocol tcp --port 22 --cidr 100.127.0.0/24
```

Then use the host as a jumpbox for remote access:

```bash
ssh -J admin@<spinifex-host> ubuntu@<instance-private-ip>
```

Extra networks that must reach instances without SNAT (e.g. a management LAN) can be added via `[network] nat_exempt_cidrs = ["192.168.50.0/24"]`.

### Public IPs in NAT mode (public pool)

A routed-NAT node (`setup-ovn.sh --nat-uplink` + `spx admin init --external-mode=nat`) can carry a public pool alongside the internal `nat-transit` pool. With one configured, nat mode behaves like pool mode for public IPs: `MapPublicIpOnLaunch` on the default subnet, auto-assigned public IPs, and Elastic IPs all work. Spinifex delivers each public IP at the host — a `/32` route steers it into OVN and a proxy-ARP neighbor entry answers for it on the uplink (L3 only, same MAC, so it works on WiFi and other non-bridgeable uplinks).

Static range carved out of the router's DHCP scope:

```bash
spx admin init --external-mode=nat \
  --external-pool 192.168.1.150-192.168.1.250 \
  --external-gateway 192.168.1.1
```

Or lease public IPs from the upstream router's DHCP:

```bash
spx admin init --external-mode=nat --external-source=dhcp \
  --external-bind-bridge wlan0
```

On WiFi/WWAN uplinks the leases are requested with the interface's own MAC (`dhcp_mac = "interface"`, written automatically) and distinguished by DHCP client-id. Some routers key leases by MAC and ignore the client-id — Spinifex detects this (the router hands the same IP to two client-ids) and fails the allocation with advice to switch to a static range.

Resulting config:

```toml
[network]
external_mode = "nat"
bridge_mode   = "nat"

[[network.external_pools]]
name       = "nat-transit"        # internal transit net (auto-generated)
gateway    = "100.127.0.1"
prefix_len = 24

[[network.external_pools]]
name        = "wan"               # public pool
range_start = "192.168.1.150"
range_end   = "192.168.1.250"
gateway     = "192.168.1.1"
prefix_len  = 24
```

**Caveat — reaching an EIP from the spinifex host itself.** Host-sourced traffic enters OVN from the transit net, which is exempt from NAT (that is what makes the jumpbox pattern work) — so a host connection to an EIP that carries the transit source IP would skip DNAT. Spinifex stamps the EIP route with the uplink's LAN IP as source to avoid this, but if no uplink address can be determined, connect to the instance's **private IP** from the host instead. Other machines on the LAN are unaffected.

## Disabled (Empty/Omitted)

VPC networking is overlay-only. No external connectivity. Instances can only communicate within their VPC.

## Mode Comparison

| Capability                        | `pool` (static) | `pool` (dhcp) | `nat`             | Disabled |
| --------------------------------- | --------------- | ------------- | ----------------- | -------- |
| Outbound internet                 | Yes             | Yes           | Yes               | No       |
| Host reaches instance private IPs | No              | No            | Yes (routed)      | No       |
| Inbound from WAN                  | Yes (1:1 NAT)   | Yes (1:1 NAT) | With public pool  | No       |
| Public subnets                    | Yes             | Yes           | With public pool  | No       |
| Auto-assign public IPs            | Yes             | Yes           | With public pool  | No       |
| Elastic IPs                       | Yes             | Yes           | With public pool  | No       |
| DescribeInstances shows public IP | Yes             | Yes           | With public pool  | No       |
| Admin must reserve IP range       | Yes             | No            | Only static pool  | No       |
| Needs router DHCP                 | No              | Yes           | Optional          | No       |

If you start with `nat` and later need public subnets: on a bridgeable uplink switch to `pool` and define a range (or use `source = "dhcp"`); on a routed-NAT node just add a public pool alongside `nat-transit` — no data migration needed.

## Bridge Setup — Physical Network Wiring

The WAN NIC **must** be enslaved to a Linux bridge. This is a hard requirement — `setup-ovn.sh` will not attach a physical NIC directly to OVS, and macvlan is no longer supported. The Linux bridge owns the host IP, default route, and any DHCP lease, so SSH and management traffic stay up while OVS/OVN are configured underneath.

The full datapath chain looks like this:

```
physical NIC (e.g. `wan`)
   └─ enslaved to ─▶ br-wan         (Linux bridge — host IP, default route, DHCP)
                       │
                       └─ veth pair ─▶ br-ext   (OVS bridge — OVN external uplink)
                                          │
                                          └─ localnet ─▶ br-int  (OVS integration bridge)
                                                            │
                                                            └─▶ TAP devices  (VM NICs)
```

`setup-ovn.sh` auto-detects the Linux bridge that owns the default route (typically `br-wan`, provisioned by cloud-init / netplan / systemd-networkd). You can override the detection with `--wan-bridge=<name>`. Once detected, the script creates the OVS bridge `br-ext` and links it to the WAN bridge with a veth pair. The Linux bridge keeps its IP and routes — no interruption. Bridge-mapping is set to `external:br-ext`.

If the default route is on a bare physical NIC (no bridge), `setup-ovn.sh` stops and prints guidance on how to convert the NIC to a bridge before re-running.

### Example: Required `br-wan` State

The host must have something resembling this before `setup-ovn.sh` is run:

```
7: br-wan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 26:df:3c:de:d0:c2 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.31/23 brd 192.168.1.255 scope global br-wan
       valid_lft forever preferred_lft forever
    inet6 fe80::24df:3cff:fede:d0c2/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
```

The physical NIC (e.g. `wan`, `eth0`, `eno1`) is enslaved to `br-wan` and has no IP of its own — all L3 state lives on the bridge.

Example netplan that produces this:

```yaml
network:
  version: 2
  ethernets:
    wan:
      dhcp4: false
  bridges:
    br-wan:
      interfaces: [wan]
      dhcp4: true
```

## Three Bridges, Three Jobs

Every Spinifex node has three bridges in the datapath:

| Bridge   | Type         | Created By                  | Purpose                                       | Ports                        |
| -------- | ------------ | --------------------------- | --------------------------------------------- | ---------------------------- |
| `br-wan` | Linux bridge | Host (cloud-init / netplan) | Host WAN uplink — owns host IP and default route | Physical WAN NIC, veth peer  |
| `br-ext` | OVS bridge   | `setup-ovn.sh`              | OVN external uplink (`localnet`)              | veth peer to `br-wan`        |
| `br-int` | OVS bridge   | `setup-ovn.sh`              | VM overlay traffic (Geneve tunnels)           | VM TAP devices, tunnel ports |

`br-wan` is provisioned by your distro's network configuration (cloud-init, netplan, systemd-networkd, ifupdown). The name is configurable; `br-wan` is the convention. `br-int` and `br-ext` are always created by `setup-ovn.sh`.

The link between `br-ext` and the VM datapath is logical, not physical: OVN's `localnet` port type maps the logical external switch to `br-ext` via `ovn-bridge-mappings`. Frames egressing a VM travel TAP → `br-int` → OVN pipeline → `br-ext` → veth → `br-wan` → physical NIC → wire.

<p align="center">
  <img height="110" src="/docs-assets/compute-vpc-networking-vpc-datapath-localnet.svg" alt="Data path — VM TAP through br-int, OVN pipeline, br-ext, veth pair, br-wan, physical NIC" width="900">
</p>

## Running setup-ovn.sh

```bash
# Auto-detect the WAN bridge (recommended)
sudo setup-ovn.sh

# Explicitly specify the WAN bridge name
sudo setup-ovn.sh --wan-bridge=br-wan
```

In environments where the WAN IP comes from a router's DHCP server (homelab, small office), add `--dhcp` to obtain a gateway IP from the router automatically:

```bash
sudo setup-ovn.sh --dhcp
```

This requests an IP from the **router's DHCP** (e.g., 192.168.1.1 serving addresses on the LAN). This is not Spinifex's internal OVN DHCP that assigns private IPs to VMs — it's your network's existing DHCP server.

| Flags                          | Result                                                                |
| ------------------------------ | --------------------------------------------------------------------- |
| (no flags)                     | Auto-detect WAN bridge from default route, create `br-int` + `br-ext` |
| `--wan-bridge=<name>`          | Use the specified Linux bridge as the WAN uplink                      |
| `--dhcp`                       | Obtain the OVN gateway IP from the router's DHCP                      |

If no Linux bridge owns the default route, `setup-ovn.sh` exits with guidance rather than silently breaking host connectivity.

## OVN Control Plane on Multi-Node Clusters

How OVN is deployed depends on the size of the cluster, and this is the main reason three servers is the recommended minimum for a multi-server deployment.

| Cluster size | OVN databases | Tolerates |
|---|---|---|
| 1–2 servers | standalone, on the first node | nothing — that node is a single point of failure for the control plane |
| 3 or more | clustered (RAFT) across the first three nodes | loss of any one database node |

Servers beyond the third run the full set of Spinifex services, but do not join the OVN database cluster — they connect to it as clients. The quorum stays at three however large the cluster gets, which is what keeps control-plane write latency stable.

### What a control-plane outage actually costs

Less than it sounds. `ovn-controller` has already programmed the forwarding rules into each host, so **running instances keep full networking** — east-west, north-south, NAT and security groups all continue to work with the databases down.

What stops is *change*. Creating a VPC, launching an instance, and updating a security group all need the control plane, because each has to write new logical topology before anything can program it.

So a standalone OVN deployment is a reasonable choice for a lab or a single-server install. It is a poor one for production, where the inability to launch an instance during an outage is usually as bad as the instances being down.

### Clustering the databases

The three database nodes are set up with `--db-cluster-local-addr`, and nodes 2 and 3 additionally point at node 1 with `--db-cluster-remote-addr`. Compute nodes take `--ovn-remote` listing all three, so they survive any one of them failing. See [Multi-Node Install](https://docs.mulgadc.com/docs/install-multi-node) for the exact commands in sequence.

A clustered database can only be created from scratch — the `ovn-central` package starts a standalone one on install, so the cluster setup passes `--recreate-db` to replace it. Confirm the result with:

```bash
sudo ovn-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound
```

Three servers should be listed with exactly one leader. A report of a standalone database means the cluster flags did not take.

## Per-Node Configuration

Different nodes in a cluster can have different WAN bridges and NICs:

```toml
[nodes.node1.vpcd]
external_interface = "br-wan"

[nodes.node2.vpcd]
external_interface = "br-public"

[nodes.node3.vpcd]
external_interface = "br-wan"      # br-wan enslaving bond0
```

`external_interface` is the **Linux bridge** that owns the WAN uplink on this node — not the physical NIC. The physical NIC lives underneath the bridge. Each node runs `setup-ovn.sh` with its own WAN bridge name (or relies on auto-detection). OVN only requires `ovn-bridge-mappings` to point at `br-ext`.

## Bridge Verification

```bash
# OVS bridges created by setup-ovn.sh
sudo ovs-vsctl br-exists br-int && echo "br-int OK" || echo "br-int MISSING"
sudo ovs-vsctl br-exists br-ext && echo "br-ext OK" || echo "br-ext MISSING"

# Linux WAN bridge owns the host IP and default route
ip -br addr show br-wan
ip route show default
# Default route's dev should be br-wan (or your WAN bridge name)

# br-ext should have one veth port linking it to br-wan
sudo ovs-vsctl list-ports br-ext
# Expect: a veth name (e.g. "veth-wan-ovs")

# Confirm the matching peer is enslaved to the Linux WAN bridge
sudo bridge link show | grep br-wan

# Bridge mappings must point at br-ext
sudo ovs-vsctl get Open_vSwitch . external-ids:ovn-bridge-mappings
# Output: "external:br-ext"

# Physical NIC is enslaved to br-wan (master should be br-wan)
ip -d link show wan
```

## Network Flow Diagram

<p align="center">
  <img height="440" src="/docs-assets/compute-vpc-networking-vpc-host-flow.svg" alt="Bare-metal host — br-int overlay, br-ext OVS uplink, br-wan Linux bridge, physical NIC, OVN NAT pipeline" width="900">
</p>

## Configuration Reference

All network configuration lives in `spinifex.toml`. Settings are split into three levels: cluster-wide mode, IP pool definitions, and per-node NIC settings.

## Configuration Levels

<p align="center">
  <img height="330" src="/docs-assets/compute-vpc-networking-vpc-config-layers.svg" alt="spinifex.toml configuration layers — cluster mode, IP pools, per-node NIC" width="900">
</p>

## Cluster-Wide: external_mode

```toml
[network]
external_mode = "pool"    # "pool", "nat", or "" (disabled)
```

| Value          | Behavior                                                          |
| -------------- | ----------------------------------------------------------------- |
| `"pool"`       | Full public networking — public subnets, auto-assign, Elastic IPs |
| `"nat"`        | Outbound-only SNAT — all VMs share one external IP                |
| `""` / omitted | Overlay-only — no external connectivity                           |

## IP Pools: network.external_pools

Each pool defines where external IPs come from. You can have one pool (homelab) or many (multi-region datacenter).

```toml
[[network.external_pools]]
name        = "wan"                  # Pool identifier (unique within cluster)
source      = "static"              # "static" (default) or "dhcp"
range_start = "192.168.1.150"        # First allocatable IP (static source only)
range_end   = "192.168.1.250"        # Last allocatable IP (static source only)
gateway     = "192.168.1.1"          # WAN default gateway (next hop for 0.0.0.0/0)
gateway_ip  = ""                     # OVN router SNAT address (defaults to range_start)
prefix_len  = 24                     # Subnet mask length
region      = ""                     # Scope to region (optional)
az          = ""                     # Scope to AZ (optional)
dns_servers = ["8.8.8.8"]           # DNS for VMs (optional)
```

### Field Details

| Field         | Required    | Description                                                                                                                                                            |
| ------------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name`        | Yes         | Unique pool name. Referenced by `AllocateAddress` to target a specific pool.                                                                                           |
| `source`      | No          | IP source: `"static"` (default) uses `range_start`/`range_end`. `"dhcp"` obtains IPs from the router's DHCP server on each VM launch.                                  |
| `range_start` | Static only | First IP in the range. First IP is reserved for OVN gateway SNAT (unless `gateway_ip` overrides).                                                                      |
| `range_end`   | Static only | Last IP in the range.                                                                                                                                                  |
| `gateway`     | Yes         | Physical router/switch — the WAN default gateway. OVN sets `0.0.0.0/0 → gateway`.                                                                                      |
| `gateway_ip`  | NAT mode    | Static IP for OVN router SNAT. In pool mode, defaults to `range_start` (static) or first DHCP lease (dhcp). In NAT mode, this is the single external IP all VMs share. |
| `prefix_len`  | Yes         | Subnet mask for the external network (e.g., 24 = /24).                                                                                                                 |
| `region`      | No          | Scopes pool to a region. Instances in this region prefer this pool.                                                                                                    |
| `az`          | No          | Scopes pool to an AZ. More specific than region.                                                                                                                       |
| `dns_servers` | No          | DNS servers propagated to VMs via OVN DHCP.                                                                                                                            |
| `gw_lrp_range_start` / `gw_lrp_range_end` | No | Reserve gateway-LRP IPs for per-VPC OVN routers. When unset, the allocator auto-derives the top 16 host IPs of the pool subnet (~15 concurrent VPCs). Widen to raise the concurrent-VPC ceiling — the `nat-transit` pool defaults to `100.127.0.16`-`100.127.0.254` (239 VPCs). Must not overlap `range_start`/`range_end`. |

### Why range_start/range_end Instead of CIDR?

Customer IP ranges rarely align to CIDR boundaries. A datacenter might have `203.0.113.10-203.0.113.200` from their ISP. Start/end avoids forcing admins to calculate CIDR blocks.

### Gateway vs Gateway_IP

These are different things:

- **`gateway`** = Your network's default gateway (e.g., 192.168.1.1). This is where OVN sends packets destined for the internet. It's your router.
- **`gateway_ip`** = The IP that OVN uses for outbound SNAT. In pool mode, defaults to the first IP in the range. In NAT mode, set this explicitly. Must be on the same subnet as the gateway.

## Per-Node: nodes.NAME.vpcd

```toml
[nodes.spx1.vpcd]
# Comma-separated list of the OVN NB/SB quorum endpoints (3 DB nodes). vpcd and
# ovn-controller fail over across them and follow the RAFT leader.
ovn_nb_addr        = "tcp:10.1.3.181:6641,tcp:10.1.3.182:6641,tcp:10.1.3.183:6641"
ovn_sb_addr        = "tcp:10.1.3.181:6642,tcp:10.1.3.182:6642,tcp:10.1.3.183:6642"
external_interface = "br-wan"                 # WAN Linux bridge name
```

| Field                | Description                                                                                                                                                                          |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `ovn_nb_addr` / `ovn_sb_addr` | OVN Northbound / Southbound DB endpoint(s). The NB and SB databases run clustered via OVSDB RAFT across the first three nodes (client ports 6641/6642, RAFT ports 6643/6644). Each node's config lists all three quorum endpoints so the loss of one DB node does not stall the control plane. A single `tcp:IP:6641` string remains valid for single-node dev. |
| `external_interface` | Linux bridge that owns the WAN uplink on this node (e.g. `br-wan`, `br-public`). The physical NIC is enslaved to this bridge — not configured here. Different nodes may differ. |

## Pool Selection Logic

When an instance needs a public IP:

1. **AZ-scoped pool first**: Pool with matching `region` + `az`
2. **Region-scoped fallback**: Pool with matching `region`, no `az` (overflow)
3. **Unscoped fallback**: Pool with no `region`/`az` (global, homelab configs)
4. **Exhausted**: All pools full → `InsufficientAddressCapacity` error

`AllocateAddress` accepts optional pool name to target a specific block (maps to AWS `PublicIpv4Pool`).

## IPAM Storage

Pool allocation state is stored durably in the cluster (NATS KV bucket `spinifex-external-ipam`, one entry per pool) and survives restarts. Each allocation records the ENI and instance holding the address. Pools are initialized from `spinifex.toml` on vpcd startup (idempotent).

## Deployment Examples

## Homelab / Dev (Single Pool)

```
Network: 192.168.1.0/24
Router: 192.168.1.1 (DHCP .2–.149)
Spinifex: 192.168.1.150–.250 (100 IPs)
```

```toml
[network]
external_mode = "pool"

[[network.external_pools]]
name        = "wan"
range_start = "192.168.1.150"
range_end   = "192.168.1.250"
gateway     = "192.168.1.1"
prefix_len  = 24

[nodes.homelab.vpcd]
external_interface = "br-wan"
```

**Setup:** Configure `br-wan` to enslave your physical WAN NIC (netplan, cloud-init, or systemd-networkd). Change your router's DHCP range to end at .149. Run `sudo setup-ovn.sh` — it auto-detects the WAN bridge from the default route, or specify it with `--wan-bridge=br-wan`.

## Homelab / Dev (DHCP Pool — No Range Reservation)

```
Network: 192.168.1.0/24
Router: 192.168.1.1 (DHCP serves full range, no carve-out needed)
Spinifex: gets IPs from router DHCP on demand
```

```toml
[network]
external_mode = "pool"

[[network.external_pools]]
name        = "wan"
source      = "dhcp"
gateway     = "192.168.1.1"
prefix_len  = 24
dns_servers = ["192.168.1.1", "8.8.8.8"]

[nodes.homelab.vpcd]
external_interface = "br-wan"
```

**Setup:** No router changes needed. Spinifex requests IPs from the router's DHCP server when VMs launch and releases them on terminate. Requires `dhclient` on the host (`apt install isc-dhcp-client`).

## Host-Local Subnet (No Upstream Router)

```
Network: 192.168.10.0/24 — host-local, reachable from the host only
Host WAN: 198.51.100.10/24 on br-wan (existing address — unchanged)
Gateway: 192.168.10.1 — second address added to br-wan
```

Add the VM pool gateway as a second address on `br-wan` alongside the existing WAN IP. The host acts as the gateway for the pool — no upstream router or DHCP server needed for this range.

```yaml
# /etc/netplan/…
bridges:
  br-wan:
    addresses:
      - 192.168.10.1/24      # VM pool gateway — host-local
      - 198.51.100.10/24     # existing WAN IP — unchanged
    routes:
      - to: default
        via: 198.51.100.1
```

```toml
[network]
external_mode = "pool"

[[network.external_pools]]
name        = "wan"
source      = "static"         # required — no upstream DHCP for this range
range_start = "192.168.10.2"
range_end   = "192.168.10.100"
gateway     = "192.168.10.1"   # second address on br-wan
prefix_len  = 24
dns_servers = ["8.8.8.8"]
```

**Setup:** Apply with `sudo netplan apply`. VMs are reachable from the host at `192.168.10.x`. For internet access through the host's WAN interface:

```bash
sysctl -w net.ipv4.ip_forward=1
iptables -t nat -A POSTROUTING -s 192.168.10.0/24 -o br-wan -j MASQUERADE
```

Persist via `/etc/sysctl.d/99-ip-forward.conf` and `netfilter-persistent save`.

## Datacenter / Colo (ISP Block)

```
ISP-assigned: 203.0.113.0/28 (14 usable IPs)
ISP gateway: 203.0.113.1
Servers: 3x with separate mgmt NIC (eth0) and public NIC enslaved to br-wan
```

```toml
[network]
external_mode = "pool"

[[network.external_pools]]
name        = "public"
range_start = "203.0.113.2"
range_end   = "203.0.113.14"
gateway     = "203.0.113.1"
prefix_len  = 28

[nodes.dc1.vpcd]
external_interface = "br-wan"      # br-wan enslaves eth1

[nodes.dc2.vpcd]
external_interface = "br-wan"      # br-wan enslaves eth1

[nodes.dc3.vpcd]
external_interface = "br-public"   # br-public enslaves eno1
```

## Enterprise On-Prem (VLAN)

The Linux WAN bridge enslaves a VLAN sub-interface (e.g. `eth1.200`) instead of a raw NIC. From OVN's perspective nothing changes — `external_interface` still points at the bridge.

```toml
[network]
external_mode = "pool"

[[network.external_pools]]
name        = "dmz"
range_start = "172.16.0.100"
range_end   = "172.16.0.200"
gateway     = "172.16.0.1"
prefix_len  = 24

[nodes.srv1.vpcd]
external_interface = "br-dmz"      # br-dmz enslaves eth1.200

[nodes.srv2.vpcd]
external_interface = "br-dmz"      # br-dmz enslaves bond0.200
```

## Edge / Branch (Outbound Only)

```toml
[network]
external_mode = "nat"

[[network.external_pools]]
name       = "wan"
gateway    = "10.0.0.1"
gateway_ip = "10.0.0.50"
prefix_len = 24

[nodes.edge1.vpcd]
external_interface = "br-wan"
```

## Multi-Region (Multiple Pools)

```toml
[network]
external_mode = "pool"

# US East — AZ-scoped
[[network.external_pools]]
name        = "us-east-1a"
range_start = "203.0.113.2"
range_end   = "203.0.113.254"
gateway     = "203.0.113.1"
prefix_len  = 24
region      = "us-east-1"
az          = "us-east-1a"

# US East — overflow (any AZ in region)
[[network.external_pools]]
name        = "us-east-overflow"
range_start = "192.0.2.2"
range_end   = "192.0.3.254"
gateway     = "192.0.2.1"
prefix_len  = 23
region      = "us-east-1"

# EU West
[[network.external_pools]]
name        = "eu-west"
range_start = "213.189.1.2"
range_end   = "213.189.2.254"
gateway     = "213.189.1.1"
prefix_len  = 23
region      = "eu-west-1"
```

Spinifex allocates from the correct pool based on where the instance launches. An instance in `us-east-1a` gets an IP from `us-east-1a` first; if exhausted, falls back to `us-east-overflow`.

## Security Groups

Security groups are stateful firewalls enforced at the OVS datapath level on each hypervisor. Traffic is filtered before it reaches the wire — equivalent to AWS Nitro card enforcement. The VM never sees dropped packets.

## How Security Groups Work

Each security group maps to an OVN **Port Group**. When an instance launches, its ENI port is added to the port group(s) for its security groups. ACL rules on the port group control traffic:

- **Default deny**: All inbound traffic dropped at priority 900
- **Allow rules**: Specific ports/protocols allowed at priority 1000 (overrides deny)
- **Stateful**: All allow rules use `allow-related` — return traffic is automatically permitted

## Default Security Group

Every VPC gets a default security group that:

- Allows all inbound from instances in the same security group
- Allows all outbound
- Denies all other inbound

## AWS Rule → OVN ACL Translation

| AWS Security Group Rule         | OVN ACL Match                                                      |
| ------------------------------- | ------------------------------------------------------------------ |
| Ingress TCP/22 from 0.0.0.0/0   | `outport == @sg && ip4 && tcp.dst == 22`                           |
| Ingress TCP/443 from 10.0.0.0/8 | `outport == @sg && ip4 && tcp.dst == 443 && ip4.src == 10.0.0.0/8` |
| Ingress ALL from sg-other       | `outport == @sg && ip4 && ip4.src == $sg_other_ip4`                |
| Ingress ICMP from anywhere      | `outport == @sg && ip4 && icmp4`                                   |
| Egress ALL to 0.0.0.0/0         | `inport == @sg && ip4`                                             |
| Default deny inbound            | `outport == @sg && ip4` (priority 900, action=drop)                |

## Example: Allow SSH + HTTP

```bash
# Create security group
SG=$(aws ec2 create-security-group --group-name web \
  --description "Web servers" --vpc-id $VPC \
  --query GroupId --output text)

# Allow SSH from anywhere
aws ec2 authorize-security-group-ingress --group-id $SG \
  --protocol tcp --port 22 --cidr 0.0.0.0/0

# Allow HTTP from anywhere
aws ec2 authorize-security-group-ingress --group-id $SG \
  --protocol tcp --port 80 --cidr 0.0.0.0/0

# Launch instance with this SG
aws ec2 run-instances --image-id $AMI --instance-type t3.small \
  --subnet-id $SUBNET --security-group-ids $SG --key-name mykey
```

Rule changes take effect immediately — no instance restart needed.

## Platform Default Egress Restrictions (Outbound SMTP)

Like AWS, Spinifex blocks outbound SMTP from instances **by default** so a
compromised guest cannot turn a fresh account into a spam relay. Connections to
the mail ports are dropped at the OVS datapath before they reach the wire:

| Port | Protocol | Purpose |
| --- | --- | --- |
| 25 | TCP | SMTP relay |
| 465 | TCP | SMTP over implicit TLS |
| 587 | TCP | SMTP submission |

This is a **platform default, not a security-group rule**. It is enforced as an
OVN egress ACL on every guest's port group at a priority **above** tenant SG
allows, so a tenant **cannot** open these ports by adding a security-group rule —
matching AWS, where lifting the block is an operator action, not a tenant one.
Only **public** destinations are blocked; mail to private ranges (`10/8`,
`172.16/12`, `192.168/16`, `100.64/10`, link-local) is exempt, so an in-VPC or
on-prem relay still works. Dropped attempts are logged for abuse triage.

It sits alongside the security-group ACLs and the host firewall (the `nft`
policy that scopes the node's own ports) as a third datapath control — this one
applies to guest egress specifically.

**Operator controls** (cluster-wide, in `spinifex.toml`):

```toml
[network]
# Omit for the default [25, 465, 587]; set [] to disable entirely.
blocked_ports_wan = [25, 465, 587]

# Workaround to let a specific tenant/VPC send mail, until per-account
# exceptions exist: list the VPC IDs to exempt from the block.
egress_block_exempt_vpcs = ["vpc-0abc123..."]
```

Exempting a VPC removes the block for **all** guests in that VPC, so scope it
narrowly. Add every VPC ID you want exempt to the one list —
`egress_block_exempt_vpcs = ["vpc-a", "vpc-b", ...]`; the match is by VPC ID, so
one entry covers every security group in that VPC.

### Multi-node clusters — keep the value identical on every node, and restart vpcd

`[network]` is a cluster-wide *setting*, but it physically lives in each node's
own `spinifex.toml`, and there is no live distribution of it. Two properties of
the current implementation make the operator responsible for consistency:

- **One node writes the ACLs.** SG reconcile runs on a single CAS-elected vpcd leader, which programs the shared OVN northbound DB. Whichever node holds the lease is the one whose `blocked_ports_wan` / `egress_block_exempt_vpcs` is in force. Leadership moves on restart or crash, so if the node configs disagree, the effective policy **changes when the leader changes** and the drift pass flaps the ACLs between the two states. Edit the value **identically on every node**.
- **It is read at vpcd startup, not hot-reloaded.** The policy is built once when vpcd starts; editing the TOML does nothing until vpcd restarts. Deploy the config change to all nodes and restart vpcd cluster-wide (take the target down and confirm no `spx` process survives — a selective single-service restart is not reliable on the shared binary).

This is the current implementation and is deliberately minimal. A future revision
will move the exemption into shared cluster state (so a single edit propagates
and cannot drift between nodes) and make it a per-account control rather than a
per-VPC operator edit; until then, the two rules above are load-bearing.

## The Instance-to-Host Plane (Metadata and VPC DNS)

Security groups govern traffic **between instances** and **to the outside**. There
is a third path they do not touch, and it is the one most easily left exposed:
traffic from an instance to the **host it runs on**.

Every instance reaches two link-local addresses served by the hypervisor:

| Address | Service | Port |
| --- | --- | --- |
| `169.254.169.254` | Instance metadata (IMDS) — cloud-init, instance-role credentials | TCP 80 |
| `169.254.169.253` | VPC DNS resolver | UDP/TCP 53 |

These are not guest addresses and not OVN routed. Each instance's tap has a
capture rule on the hypervisor that intercepts any packet addressed to `.254` or
`.253` and delivers it to a per-ENI internal port in the host network namespace
(named `ime-*`). That interception is **by destination address only — it does not
match the port.** A packet to `169.254.169.254:22` is delivered to the host just
as readily as one to `169.254.169.254:80`.

That matters because host services — SSH, the AWS gateway on 9999, the console on
3000, the DNS server — bind the wildcard address, so they also answer on these
link-local addresses. Without a control on the `ime-*` path, an instance can
reach every one of them, with none of the source scoping that protects the same
services on the network. A resolver query sent straight to the host's DNS port
this way also bypasses the per-instance DNS rate limit.

### This plane is invisible to a network firewall

The critical point for anyone securing a deployment: **this traffic never crosses
the physical NIC.** It is intra-host, on the OVS bridge between the instance's tap
and the `ime-*` port. A cloud security group, an upstream firewall, `ufw`, or any
rule written against the WAN interface is not in this path and cannot filter it.
The only place to enforce it is on the `ime-*` interface in the host itself.

### How it is protected

Spinifex's host firewall carries a rule that scopes the `ime-*` path to exactly
the two legitimate endpoints and ports — IMDS on `.254:80`, DNS on `.253:53` —
and drops everything else. Whether you have that rule depends entirely on whether
the host firewall is on:

| Install path | Host firewall | This plane |
| --- | --- | --- |
| **From the ISO** | on | protected |
| **Binary installer** (`curl \| bash`) or `setup.sh` | **off** | **exposed** |
| **Binary installer with `--firewall=on`** | on | protected |

The ISO ships the firewall armed, so ISO deployments are protected out of the box.
**The binary installer ships it off** — deliberately, because it runs on servers
that may already have services the installer knows nothing about, and a
default-deny policy could cut them off. The consequence is that a binary install
left at its default has this plane wide open, and so does every host service on
the WAN besides.

If you install any way other than the ISO, turn the host firewall on. At install
time:

```bash
curl -fsSL https://install.mulgadc.com | bash -s -- --firewall=on
# or, from a source checkout:
sudo /usr/local/share/spinifex/setup.sh --firewall=on
```

or afterwards, in `/etc/spinifex/spinifex.toml`, followed by a daemon restart:

```toml
[network]
firewall_enabled = true
```

Before enabling it, check what else the machine is serving — anything listening
outside the public port group stops accepting new connections. See
[Firewall and Cluster Membership](https://docs.mulgadc.com/docs/install-multi-node#firewall-and-cluster-membership)
for the full port policy and the cluster-formation steps.

### Verifying it

On the hypervisor, the `ime-*` accept should be scoped, not blanket:

```bash
sudo nft list chain inet spinifex_filter input | grep ime
# Expect three rules naming 169.254.169.254 tcp dport 80 and 169.254.169.253
# udp/tcp dport 53 — not a bare `iifname "ime-*" accept`.
```

From an instance, the two service ports work and nothing else does:

```bash
curl -s -o /dev/null -w '%{http_code}\n' http://169.254.169.254/latest/meta-data/   # 401 (IMDSv2)
dig +short @169.254.169.253 example.com A                                            # resolves
nc -vz 169.254.169.254 22                                                            # must fail
```

A missing `spinifex_filter` table means the host firewall is off and this plane
is unprotected regardless of the rule above.

## Elastic IPs

Elastic IPs are static public IPs that persist across instance stop/start cycles. Unlike auto-assigned public IPs (which change on stop/start), an Elastic IP stays with your instance.

```bash
# Allocate
EIP=$(aws ec2 allocate-address --query AllocationId --output text)

# Associate with instance
aws ec2 associate-address --allocation-id $EIP --instance-id $INSTANCE

# Stop/start instance — same Elastic IP

# Disassociate
aws ec2 disassociate-address --association-id $ASSOC_ID

# Release back to pool
aws ec2 release-address --allocation-id $EIP
```

When you associate an Elastic IP with an instance that already has an auto-assigned public IP, the auto-assigned IP is released and replaced.

## OVN Reference

For operators debugging or verifying the OVN topology.

## IGW Attach Creates

```bash
# External logical switch with localnet port
ovn-nbctl ls-add ext-{vpcId}
ovn-nbctl lsp-add ext-{vpcId} ext-localnet-{vpcId}
ovn-nbctl lsp-set-type ext-localnet-{vpcId} localnet
ovn-nbctl lsp-set-addresses ext-localnet-{vpcId} unknown
ovn-nbctl lsp-set-options ext-localnet-{vpcId} network_name=external

# Gateway router port with real external IP
ovn-nbctl lrp-add vpc-{vpcId} gw-{vpcId} {mac} 192.168.1.150/24

# Connect external switch to router
ovn-nbctl lsp-add ext-{vpcId} ext-rtr-{vpcId}
ovn-nbctl lsp-set-type ext-rtr-{vpcId} router
ovn-nbctl lsp-set-options ext-rtr-{vpcId} router-port=gw-{vpcId}

# Gateway chassis HA
ovn-nbctl lrp-set-gateway-chassis gw-{vpcId} chassis-1 20
ovn-nbctl lrp-set-gateway-chassis gw-{vpcId} chassis-2 15

# SNAT for all VPC traffic
ovn-nbctl lr-nat-add vpc-{vpcId} snat 192.168.1.150 10.0.0.0/16

# Default route to WAN
ovn-nbctl lr-route-add vpc-{vpcId} 0.0.0.0/0 192.168.1.1
```

## Per-Instance Public IP

```bash
# 1:1 NAT — distributed (DNAT processed on the VM's own chassis)
ovn-nbctl lr-nat-add vpc-{vpcId} dnat_and_snat {public_ip} {private_ip} \
  port-{eniId} {vm_mac}
```

With the WAN NIC on a Linux bridge wired to OVS via veth, OVS sees every frame on the wire regardless of MAC, so OVN can use distributed NAT. The DNAT is processed on the chassis hosting the VM rather than hairpinning through a single gateway chassis.

## Security Group

```bash
# Create port group
ovn-nbctl pg-add sg-{groupId}

# Add VM ports
ovn-nbctl pg-set-ports sg-{groupId} port-{eniId1} port-{eniId2}

# Allow SSH inbound (stateful)
ovn-nbctl acl-add sg-{groupId} to-lport 1000 \
  'outport == @sg_{groupId} && ip4 && tcp.dst == 22' allow-related

# Allow all egress
ovn-nbctl acl-add sg-{groupId} from-lport 1000 \
  'inport == @sg_{groupId} && ip4' allow-related

# Default deny inbound
ovn-nbctl acl-add sg-{groupId} to-lport 900 \
  'outport == @sg_{groupId} && ip4' drop
```

## Useful Debug Commands

```bash
# List all logical routers (VPCs)
sudo ovn-nbctl lr-list

# List all logical switches (subnets + external)
sudo ovn-nbctl ls-list

# Show NAT rules for a VPC
sudo ovn-nbctl lr-nat-list vpc-{vpcId}

# Show routes for a VPC
sudo ovn-nbctl lr-route-list vpc-{vpcId}

# Show chassis (nodes) in the cluster
sudo ovn-sbctl show

# Show port bindings (which VM is on which host)
sudo ovn-sbctl find Port_Binding type="" | grep -E "logical_port|chassis"

# Check ACLs on a security group
sudo ovn-nbctl acl-list sg-{groupId}

# Check port group membership
sudo ovn-nbctl pg-get-ports sg-{groupId}
```

## Quick Start

## 1. Set Up OVN Bridges

Make sure the WAN NIC is enslaved to a Linux bridge (e.g. `br-wan`) and that bridge owns the default route. Then run:

```bash
sudo setup-ovn.sh                       # auto-detect WAN bridge
# or
sudo setup-ovn.sh --wan-bridge=br-wan   # explicit
```

## 2. Configure External IP Pool

```bash
spx admin init
# Follow prompts — auto-detects NICs, suggests IP pool range
# Or edit spinifex.toml manually
```

## 3. Create VPC with Public Subnet

```bash
VPC=$(aws ec2 create-vpc --cidr-block 10.200.0.0/16 \
  --query Vpc.VpcId --output text)

IGW=$(aws ec2 create-internet-gateway \
  --query InternetGateway.InternetGatewayId --output text)
aws ec2 attach-internet-gateway \
  --internet-gateway-id $IGW --vpc-id $VPC

SUBNET=$(aws ec2 create-subnet --vpc-id $VPC \
  --cidr-block 10.200.1.0/24 \
  --query Subnet.SubnetId --output text)

# Route table — give the subnet a default route to the IGW. Spinifex does NOT
# add this automatically; without it the subnet's egress is dropped.
RT=$(aws ec2 create-route-table --vpc-id $VPC \
  --query RouteTable.RouteTableId --output text)

aws ec2 create-route --route-table-id $RT \
  --destination-cidr-block 0.0.0.0/0 --gateway-id $IGW

aws ec2 associate-route-table --route-table-id $RT --subnet-id $SUBNET

# Auto-assign a public IP to instances launched into the subnet
aws ec2 modify-subnet-attribute \
  --subnet-id $SUBNET --map-public-ip-on-launch

# Allow SSH + ICMP on the VPC's default security group (blocks inbound by default)
SG=$(aws ec2 describe-security-groups \
  --filters Name=vpc-id,Values=$VPC \
  --query 'SecurityGroups[0].GroupId' --output text)

aws ec2 authorize-security-group-ingress --group-id $SG \
  --protocol tcp --port 22 --cidr 0.0.0.0/0

aws ec2 authorize-security-group-ingress --group-id $SG \
  --protocol icmp --port -1 --cidr 0.0.0.0/0
```

## 4. Launch Instance

```bash
INSTANCE=$(aws ec2 run-instances \
  --image-id $AMI --instance-type t3.small \
  --subnet-id $SUBNET --key-name mykey \
  --query Instances[0].InstanceId --output text)

aws ec2 describe-instances --instance-ids $INSTANCE \
  --query 'Reservations[0].Instances[0].[PrivateIpAddress,PublicIpAddress]'
```

## Troubleshooting

### Debugging Toolkit

These commands are used throughout the troubleshooting sections below. Learn them — they cover 90% of VPC networking issues.

### OVN Northbound (Logical Topology)

```bash
# Full topology overview (routers, switches, ports)
sudo ovn-nbctl show

# List all VPC routers
sudo ovn-nbctl lr-list

# List all switches (subnets + external)
sudo ovn-nbctl ls-list

# NAT rules for a VPC
sudo ovn-nbctl lr-nat-list vpc-{vpcId}

# Routes for a VPC
sudo ovn-nbctl lr-route-list vpc-{vpcId}

# Port details (check "up" field for DHCP status)
sudo ovn-nbctl list Logical_Switch_Port port-eni-{eniId}

# Gateway chassis assignment
sudo ovn-nbctl list Logical_Router_Port gw-vpc-{vpcId}

# Localnet port options (network_name should be "external")
sudo ovn-nbctl get Logical_Switch_Port ext-port-vpc-{vpcId} options
```

### OVN Southbound (runtime state)

```bash
# Chassis list + port bindings (which VM is on which host)
sudo ovn-sbctl show

# Detailed chassis info (check name matches expectations)
sudo ovn-sbctl list Chassis

# MAC binding table (shows ARP resolution for external traffic)
sudo ovn-sbctl list MAC_Binding

# Trace a packet through the OVN pipeline (invaluable for debugging)
sudo ovn-trace ext-vpc-{vpcId} \
  'inport=="ext-port-vpc-{vpcId}" && eth.dst==ff:ff:ff:ff:ff:ff && \
   arp.op==1 && arp.spa==192.168.1.13 && arp.tpa==192.168.1.201'
```

### OVN DB RAFT cluster (NB/SB replication)

The NB and SB databases run clustered across the first three nodes via native OVSDB RAFT. A 3-node quorum tolerates the loss of one DB node; check status when the control plane stalls.

```bash
# NB cluster status: expect 3 servers and exactly one "Role: leader"
sudo ovn-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound

# SB cluster status
sudo ovn-appctl -t /var/run/ovn/ovnsb_db.ctl cluster/status OVN_Southbound

# Drive a CLI at the whole quorum (fails over past a dead node)
sudo ovn-nbctl --db=tcp:10.1.3.181:6641,tcp:10.1.3.182:6641,tcp:10.1.3.183:6641 show
```

A node showing only itself under `Servers:` never joined the cluster — confirm it was bootstrapped with `--db-cluster-remote-addr` pointing at the creator and that RAFT ports 6643/6644 are reachable between DB nodes.

### OVS (datapath / physical wiring)

```bash
# Full bridge + port topology
sudo ovs-vsctl show

# Kernel datapath ports and stats
sudo ovs-dpctl show

# Kernel datapath flow cache (actual forwarding rules)
sudo ovs-dpctl dump-flows

# OpenFlow rules installed by ovn-controller
sudo ovs-ofctl dump-flows br-int | grep {pattern}

# Conntrack entries (shows active NAT sessions)
sudo ovs-appctl dpctl/dump-conntrack | grep {ip}

# FDB (MAC address table) for a bridge
sudo ovs-appctl fdb/show br-wan

# OVS external_ids (system-id, bridge-mappings, encap-ip)
sudo ovs-vsctl get Open_vSwitch . external_ids
```

### Network Interfaces

```bash
# Physical NIC should be enslaved to br-wan ("master br-wan")
ip -d link show {nic}

# Linux WAN bridge — owns the host IP and default route
ip -br addr show br-wan
ip route show default

# OVS bridges and the veth linking br-ext to br-wan
sudo ovs-vsctl list-ports br-ext
bridge link show | grep br-wan

# Interface traffic stats (RX/TX counts, drops)
ip -s link show {nic}
ip -s link show br-wan
```

### Packet Capture

```bash
# Capture on the OVS uplink bridge (frames between OVN and br-wan)
sudo tcpdump -i br-ext -n -e arp

# Capture on the Linux WAN bridge (frames between br-ext veth and the NIC)
sudo tcpdump -i br-wan -n -e "host {public_ip}"

# Capture on the physical NIC (sees everything on the wire)
sudo tcpdump -i {nic} -n "host {public_ip}"
```

### Service Logs

```bash
# vpcd log (reconcile, NAT, topology operations)
journalctl -u spinifex-vpcd -f

# ovn-controller log (port binding, commit failures)
sudo cat /var/log/ovn/ovn-controller.log | tail -50

# Daemon log (instance launch, network setup)
journalctl -u spinifex-daemon -f
```

### VPC Creation Fails

Check OVN services and vpcd daemon:

```bash
sudo systemctl is-active ovn-controller
journalctl -u spinifex-vpcd -f
```

### Instances Cannot Reach Each Other

Geneve tunnels may not be established:

```bash
sudo ovs-vsctl show | grep -i geneve
sudo ss -ulnp | grep 6081
```

From inside a VM:

```bash
ip addr show
ip route show
```

### Instance Has No Public IP

1. Check subnet has `MapPublicIpOnLaunch`:

   ```bash
   aws ec2 describe-subnets --subnet-ids $SUBNET \
     --query 'Subnets[0].MapPublicIpOnLaunch'
   ```

2. Check IGW is attached:

   ```bash
   aws ec2 describe-internet-gateways \
     --filters Name=attachment.vpc-id,Values=$VPC
   ```

3. Check external IP pool:

   ```bash
   nats kv get spinifex-external-ipam wan
   ```

4. Check OVN NAT rules:
   ```bash
   sudo ovn-nbctl lr-nat-list vpc-$VPC
   ```

### Instance Has Public IP But No Internet

The instance shows a public IP in `describe-instances` but cannot reach the internet, and inbound connections hang. The usual cause is a missing default route: the subnet's effective route table has no `0.0.0.0/0` route to the IGW, so Spinifex gates the subnet with a drop policy.

```bash
# Find the route table that applies to the subnet and check its routes
aws ec2 describe-route-tables \
  --filters Name=association.subnet-id,Values=$SUBNET \
  --query 'RouteTables[0].Routes'
# Expect a route with DestinationCidrBlock 0.0.0.0/0 and a GatewayId of igw-...
```

If the subnet has no explicit association it falls back to the VPC's main route table — check that one too, then add the route to whichever table applies:

```bash
aws ec2 create-route --route-table-id $RT \
  --destination-cidr-block 0.0.0.0/0 --gateway-id $IGW
```

On the host, the drop policy installed when a subnet lacks an IGW route appears as a `Logical_Router_Policy` on the VPC router:

```bash
sudo ovn-nbctl lr-policy-list vpc-$VPC
# A drop policy matching the subnet CIDR means the subnet is gated.
```

### Public IP Not Reachable from WAN

Work through these checks in order — each eliminates a class of issues.

### 1. Verify OVS wiring

```bash
# br-int and br-ext must exist
sudo ovs-vsctl show | grep -E "Bridge (br-int|br-ext)"

# br-ext should have a veth port linking it to br-wan
sudo ovs-vsctl list-ports br-ext

# The Linux WAN bridge should have the physical NIC and the veth peer
bridge link show | grep br-wan

# Bridge mappings must point at br-ext
sudo ovs-vsctl get Open_vSwitch . external-ids:ovn-bridge-mappings
# Expected: "external:br-ext"
```

### 2. Verify chassis and gateway scheduling

```bash
# What OVS thinks the chassis name is
sudo ovs-vsctl get Open_vSwitch . external-ids:system-id

# What OVN SB registered (must match the system-id above)
sudo ovn-sbctl show
# Look for: Chassis {name}

# What vpcd scheduled as gateway chassis (must match SB chassis name)
sudo ovn-nbctl list Logical_Router_Port gw-vpc-{vpcId} | grep gateway_chassis
```

If these don't match, see "Chassis name mismatch" below.

### 3. Verify NAT rule

```bash
sudo ovn-nbctl lr-nat-list vpc-$VPC | grep dnat_and_snat
# Must show the public IP → private IP mapping
# For distributed NAT, external_mac and logical_port should be set
# (the VM's MAC and ENI port name)
```

### 4. Verify ARP resolution

From another host on the same LAN, check if OVN responds to ARP:

```bash
# On the remote host:
ping -c 1 {public_ip}
ip neigh show {public_ip}
# Should show the VM's ENI MAC (distributed NAT) or the OVN router MAC
```

If ARP fails, confirm the WAN bridge is forwarding:

```bash
# Physical NIC must be enslaved to br-wan
ip -d link show {nic} | grep "master br-wan"

# br-wan must be UP and have an IP
ip -br addr show br-wan

# br-ext must have a veth port whose peer is enslaved to br-wan
sudo ovs-vsctl list-ports br-ext
bridge link show | grep br-wan
```

### 5. Verify packet flow with tcpdump

Capture at each layer to find where packets stop:

```bash
# Layer 1: Does the ARP/ICMP arrive on the physical NIC?
sudo tcpdump -i {nic} -n "host {public_ip}"

# Layer 2: Does it cross the Linux WAN bridge?
sudo tcpdump -i br-wan -n "host {public_ip}"

# Layer 3: Does it reach the OVS uplink bridge?
sudo tcpdump -i br-ext -n -e "host {public_ip}"
```

If traffic arrives on the physical NIC but not br-wan, the NIC is not enslaved to the bridge. If it reaches br-wan but not br-ext, the veth pair between them is missing or down — re-run `setup-ovn.sh`.

### 6. Use ovn-trace for pipeline debugging

Simulate a packet through the entire OVN pipeline:

```bash
sudo ovn-trace --ct=new ext-vpc-{vpcId} \
  'inport=="ext-port-vpc-{vpcId}" && eth.dst==ff:ff:ff:ff:ff:ff && \
   arp.op==1 && arp.sha=={remote_mac} && arp.spa=={remote_ip} && \
   arp.tpa=={public_ip}'
```

The output shows every table the packet passes through and what action is taken. Look for `drop` actions or unexpected paths.

### OVN SB Commit Failure Loop

**Symptom:** ovn-controller log shows:

```
OVNSB commit failed, force recompute next time.
```

Repeated millions of times. Port binding never happens (`up: false`).

**Cause:** Stale entries in the OVN Southbound DB (old chassis records, port bindings, datapath bindings) conflict with ovn-controller's expected state, typically after an ungraceful shutdown.

**Fix:** Delete both OVN DB files and restart:

```bash
sudo systemctl stop ovn-central ovn-controller
sudo rm -f /var/lib/ovn/ovnnb_db.db /var/lib/ovn/ovnsb_db.db
sudo systemctl start ovn-central ovn-controller
# vpcd reconcile will recreate the NB topology on next startup
```


### WAN NIC Not Enslaved to a Bridge

**Symptom:** `setup-ovn.sh` exits with an error like "default route is on a physical NIC, not a bridge" and refuses to continue.

**Cause:** Spinifex requires the WAN NIC to be enslaved to a Linux bridge (typically `br-wan`). Macvlan is no longer supported, and attaching the NIC directly to OVS would break SSH and any other host services using the NIC.

**Fix:** Move the host IP and default route onto a Linux bridge. Example netplan:

```yaml
network:
  version: 2
  ethernets:
    wan:
      dhcp4: false
  bridges:
    br-wan:
      interfaces: [wan]
      dhcp4: true
```

Apply (`sudo netplan apply`), confirm the host IP is now on `br-wan` (`ip -br addr show br-wan`), and re-run `setup-ovn.sh`.

### Stale ARP on Remote Hosts

**Symptom:** Ping from a LAN host to a VM public IP fails after a reset, but worked before. The remote host has a stale ARP entry with the old MAC.

**Fix:** Flush the ARP entry on the remote host:

```bash
# On the remote host:
sudo ip neigh flush dev {nic} {public_ip}
ping {public_ip}   # should work now
```

OVN sends periodic gratuitous ARPs from the chassis hosting the VM that will eventually update remote ARP caches, but flushing is faster for testing.

### Security Group Rules Not Taking Effect

```bash
# Check port is in correct port group
sudo ovn-nbctl pg-get-ports sg-$SG_ID

# Check ACLs
sudo ovn-nbctl acl-list sg-$SG_ID
```

---

# Launching Instances

URL: https://docs.mulgadc.com/docs/launching-instances
Category: Compute and Networking
Updated: 2026-08-19
Tags: ec2, instances, vm

Launch, manage, and connect to EC2-compatible virtual machines on Spinifex, with cloud-init, SSH key injection, VPC networking, and AWS lifecycle operations.

## Overview

Spinifex provides EC2-compatible VM management built on QEMU/KVM. Instances support cloud-init, SSH key injection, VPC networking, and standard AWS lifecycle operations.

## Prerequisites

- A running Spinifex cluster (see [Setting Up Your Cluster](https://docs.mulgadc.com/docs/setting-up-your-cluster))
- AWS CLI configured with the `spinifex` profile:

```bash
export AWS_PROFILE=spinifex
```

## Launch

```bash
INSTANCE_ID=$(aws ec2 run-instances \
  --image-id $SPINIFEX_AMI \
  --instance-type t3.small \
  --key-name spinifex-key \
  --query 'Instances[0].InstanceId' --output text)
```

Other supported launch flags: `--subnet-id`, `--security-group-ids`, `--tag-specifications`, `--user-data`, `--iam-instance-profile`, `--block-device-mappings`, `--placement`, `--count`.

## Manage

```bash
aws ec2 describe-instances --instance-ids $INSTANCE_ID
aws ec2 describe-instance-status --instance-ids $INSTANCE_ID --include-all-instances
aws ec2 stop-instances --instance-ids $INSTANCE_ID
aws ec2 start-instances --instance-ids $INSTANCE_ID
aws ec2 terminate-instances --instance-ids $INSTANCE_ID
aws ec2 reboot-instances --instance-ids $INSTANCE_ID
```

## Modify Instance Attributes

Change instance type, user data, or termination protection. Instance type and user data require the instance to be **stopped** first.

### Change Instance Type

```bash
aws ec2 stop-instances --instance-ids $INSTANCE_ID

aws ec2 modify-instance-attribute \
  --instance-id $INSTANCE_ID \
  --instance-type t3.medium

aws ec2 start-instances --instance-ids $INSTANCE_ID
```

### Termination Protection

```bash
aws ec2 modify-instance-attribute \
  --instance-id $INSTANCE_ID \
  --disable-api-termination
```

## Instance Metadata Options

IMDSv2 is always enforced — `--http-tokens optional` and `--http-endpoint disabled` are rejected with `UnsupportedOperation`, matching AWS account-level IMDSv2 enforcement. The hop limit is adjustable (raise it for containerised workloads that reach IMDS through an extra network hop):

```bash
aws ec2 modify-instance-metadata-options \
  --instance-id $INSTANCE_ID \
  --http-put-response-hop-limit 2
```

## Spot Instances

Spot requests are supported as a compatibility layer over the on-demand path: requests launch real VMs on your own compute immediately and report `active`/`fulfilled`. There is no spot market — no bidding, interruption, or reclamation.

```bash
aws ec2 request-spot-instances \
  --instance-count 1 \
  --launch-specification '{"ImageId":"'$SPINIFEX_AMI'","InstanceType":"t3.small","KeyName":"spinifex-key"}'

aws ec2 describe-spot-instance-requests
aws ec2 cancel-spot-instance-requests --spot-instance-request-ids $SIR_ID
```

## Console Output

Retrieve the serial console log for a running instance. Output is base64-encoded.

```bash
aws ec2 get-console-output --instance-id $INSTANCE_ID
```

Decode the output:

```bash
aws ec2 get-console-output --instance-id $INSTANCE_ID \
  --query 'Output' --output text | base64 -d
```

## Instance Types

List instance types available on the current host. The catalog is generated from the host CPU (Intel, AMD, or ARM) and includes burstable (t-family), general purpose (m-family), compute optimised (c-family), and memory optimised (r-family) types.

```bash
aws ec2 describe-instance-types
```

Filter to a specific type:

```bash
aws ec2 describe-instance-types \
  --query "InstanceTypes[?InstanceType=='t3.small']"
```

Show capacity (how many of each type can still be launched):

```bash
aws ec2 describe-instance-types \
  --filters Name=capacity,Values=true
```

## SSH

To SSH into instances via their public IPs, see [Setting Up Your Cluster](https://docs.mulgadc.com/docs/setting-up-your-cluster).

## Troubleshooting

### Instance Fails to Boot

Check QEMU logs for the instance and verify the AMI architecture matches your host:

```bash
journalctl -u spinifex-daemon -f
aws ec2 describe-images --image-ids $SPINIFEX_AMI
```

If the AMI is for a different architecture (e.g. arm64 on an x86_64 host), import the correct image:

```bash
spx admin images list
spx admin images import --name debian-13-x86_64
```

### Cannot SSH Into Instance

cloud-init needs time to configure the instance after boot. Wait 30-60 seconds and retry.

If the connection **times out** rather than being refused, the security group is likely blocking port 22. The default security group denies all inbound traffic (matching AWS), so SSH must be explicitly allowed:

```bash
# Allow SSH from anywhere on the instance's security group
aws ec2 authorize-security-group-ingress \
  --group-id $SG_ID --protocol tcp --port 22 --cidr 0.0.0.0/0
```

See [VPC Networking — Security Groups](https://docs.mulgadc.com/docs/vpc-networking#security-groups) for scoping rules to a trusted CIDR.

Verify the SSH key was specified correctly when launching:

```bash
aws ec2 describe-instances --instance-ids $INSTANCE_ID
```

Check the `KeyName` field matches the key you're using to connect.

---

# Placement Groups

URL: https://docs.mulgadc.com/docs/placement-groups
Category: Compute and Networking
Updated: 2026-08-19
Tags: ec2, placement, spread, cluster

Create and manage spread and cluster placement groups to control how Spinifex places EC2 instances across physical hosts for fault isolation or low latency.

## Overview

Placement groups control how Spinifex distributes instances across physical hosts. Two strategies are supported:

- **Spread** — One instance per physical host, maximising fault isolation
- **Cluster** — All instances on the same host, minimising latency

The `partition` strategy is not supported.

**Supported operations:**

- `create-placement-group` — Create a new group
- `describe-placement-groups` — Query groups with optional filters
- `delete-placement-group` — Remove an empty group

## Prerequisites

- A running Spinifex cluster (see [Setting Up Your Cluster](https://docs.mulgadc.com/docs/setting-up-your-cluster))
- AWS CLI configured with the `spinifex` profile:

```bash
export AWS_PROFILE=spinifex
```

## Create

```bash
aws ec2 create-placement-group \
  --group-name my-spread-group \
  --strategy spread
```

```bash
aws ec2 create-placement-group \
  --group-name my-cluster-group \
  --strategy cluster
```

## Launch Into a Placement Group

```bash
aws ec2 run-instances \
  --image-id $SPINIFEX_AMI \
  --instance-type t3.small \
  --key-name spinifex-key \
  --placement GroupName=my-spread-group \
  --count 3
```

## Describe

```bash
aws ec2 describe-placement-groups
aws ec2 describe-placement-groups --group-names my-spread-group
aws ec2 describe-placement-groups --group-ids pg-abc123
aws ec2 describe-placement-groups \
  --filters Name=strategy,Values=spread
```

## Delete

The group must have no running instances before it can be deleted:

```bash
aws ec2 terminate-instances --instance-ids $INSTANCE_ID
aws ec2 delete-placement-group --group-name my-spread-group
```

## Troubleshooting

### InsufficientInstanceCapacity Error

Not enough distinct physical hosts for a spread launch. Reduce `--count` or terminate existing instances to free host slots:

```bash
spx admin nodes list
```

### Cannot Delete Placement Group

The group still has running instances. Terminate them first:

```bash
aws ec2 describe-instances \
  --filters Name=placement-group-name,Values=my-spread-group
aws ec2 terminate-instances --instance-ids $INSTANCE_ID
```

---

# GPU Passthrough

URL: https://docs.mulgadc.com/docs/gpu-passthrough
Category: Compute and Networking
Updated: 2026-08-19
Tags: gpu, passthrough, vfio, ec2

Configure VFIO GPU passthrough on a Spinifex node to bind NVIDIA or AMD GPUs to guest VMs and expose GPU-enabled EC2 instance types to your workloads.

## Overview

Spinifex supports VFIO-based GPU passthrough, binding NVIDIA and AMD GPUs directly to guest VMs via the `vfio-pci` kernel driver. Once configured, the node exposes GPU-enabled EC2 instance types and the GPU is allocated exclusively to individual instances.

At startup, Spinifex always probes for GPU hardware and surfaces the result in the node banner and in `spx top nodes` — regardless of whether passthrough is enabled. This lets operators verify hardware before activating the feature.

The passthrough state is reflected in the login banner:

<img width="1778" height="302" src="/docs-assets/compute-gpu-passthrough-gpu-banner1.png" alt="GPU not yet enabled">

> **Note:** GPU passthrough is supported on x86_64 hosts only.

## Prerequisites

- IOMMU must be supported and enabled in the host BIOS/UEFI (`Intel VT-d` or `AMD-Vi`).
- An NVIDIA or AMD GPU must be physically installed.
- The Spinifex node must be running on bare metal (not inside a VM).

## Instructions

### Host Setup

Run the one-time host configuration as root. This command is idempotent — safe to re-run after a reboot.

```bash
sudo spx admin gpu setup
```

What it does:

1. Detects installed GPUs and their PCI IDs.
2. Enables IOMMU in GRUB (`intel_iommu=on iommu=pt` or `amd_iommu=on iommu=pt`).
3. Writes the vfio udev rule (`/etc/udev/rules.d/99-spinifex-vfio.rules`).
4. Blacklists `nouveau` (NVIDIA) and `amdgpu` (AMD) so the host kernel does not claim the device.
5. Configures `vfio-pci` early binding in `/etc/modprobe.d/vfio-pci.conf`.
6. Adds vfio modules to initramfs.

If any change requires a reboot, the command prints instructions and exits:

```
Setup complete — reboot required.
  sudo reboot
  Then run: sudo spx admin gpu enable
```

### Enable Passthrough

After setup and reboot, enable GPU passthrough:

```bash
sudo spx admin gpu enable
```

The command writes the configuration, notifies the daemon and waits up to 30 seconds for the daemon to confirm the new state. On success it prints the current GPU status.

<img width="1230" height="644" src="/docs-assets/compute-gpu-passthrough-gpu-enabled.png" alt="GPU enabled">

### Status and Monitoring

**Per-node status:**

```bash
spx admin gpu status
# or for a specific node in the cluster:
spx admin gpu status --node <node-name>
```

Output includes hardware detected, IOMMU state, vfio-pci binding, passthrough enabled/disabled, GPU pool allocation (`allocated/total`), and the GPU-capable EC2 instance types available on that node.

**Cluster view:**

`spx top nodes` includes a `GPU (used/total)` column:

| Value | Meaning |
|-------|---------|
| `1/2` | Passthrough active — 1 of 2 GPUs allocated |
| `0/1*` | Node has a GPU but passthrough is not enabled |
| `-` | No GPU hardware detected |

```bash
spx top nodes
```

### GPU Instance Types

GPU instance types are derived from detected hardware. To list available GPU instance types on the current node:

```bash
aws ec2 describe-instance-types \
  --filters Name=instance-type,Values=g*
```

### Launching GPU Instances

Spinifex ships two pre-built GPU guest images. Import the one that matches your host hardware before launching:

```bash
spx admin images import --name ubuntu-26.04-nvidia-gpu-x86_64   # NVIDIA hosts
spx admin images import --name ubuntu-26.04-amd-gpu-x86_64      # AMD hosts
```

These images are built on Ubuntu 26.04 and serve as a base for GPU workloads. Both include Docker CE, Python 3, and common utilities (`git`, `curl`, `htop`, `tmux`, `ffmpeg`).

**NVIDIA** (`ubuntu-26.04-nvidia-gpu-x86_64`): NVIDIA server driver with DKMS pre-built against the image kernel, `nvidia-smi`, nvidia-container-toolkit wired into Docker (`--gpus all` works on first boot). CUDA toolkit and cuDNN are not included — use an NGC container or install inside the instance.

**AMD** (`ubuntu-26.04-amd-gpu-x86_64`): `linux-firmware` (amdgpu firmware blobs), ROCm 7.2 CLI tools (`rocm-smi`, `rocminfo`, `amd-smi`). Full ROCm compute libraries (`rocblas`, etc.) are not included — install inside the instance as needed.

Launch a GPU instance:

```bash
aws ec2 run-instances \
  --image-id $GPU_AMI \
  --instance-type $INSTANCE_TYPE \
  --key-name spinifex-key
```

To verify the GPU is visible from inside the instance, SSH in and run:

```bash
# NVIDIA
nvidia-smi

# AMD
rocm-smi
```

### Disable Passthrough

Passthrough cannot be disabled while GPU instances are running. Terminate all GPU instances first, then:

```bash
sudo spx admin gpu disable
```

The command signals the daemon and waits for confirmation. The GPU is released back to the host kernel on the next reboot (the vfio-pci binding persists until `setup` is re-run without the blacklists).

## Troubleshooting

### IOMMU not active after reboot

Verify IOMMU is enabled in BIOS/UEFI (`Intel VT-d` or `AMD-Vi`). Then check GRUB picked up the parameters:

```bash
cat /proc/cmdline | grep iommu
ls /sys/kernel/iommu_groups/
```

If `/sys/kernel/iommu_groups/` is empty, IOMMU is not active. Re-run `sudo spx admin gpu setup` after enabling it in firmware.

### GPU bound to native driver instead of vfio-pci

```bash
lspci -k | grep -A3 -i "vga\|3d\|display"
```

If the driver is `amdgpu` or `nvidia` rather than `vfio-pci`, the blacklist or early binding did not take effect. Re-run setup and reboot:

```bash
sudo spx admin gpu setup
sudo reboot
```

### `spx admin gpu enable` fails with "prerequisites not met"

Run `setup` first to configure the host, then retry `enable`:

```bash
sudo spx admin gpu setup
sudo spx admin gpu enable
```

### Daemon does not confirm within 30 seconds

Check the daemon log for errors:

```bash
journalctl -u spinifex-daemon -n 50
```

---

# IAM Users, Policies, and Access Keys

URL: https://docs.mulgadc.com/docs/iam-users-and-policies
Category: Identity and Access
Updated: 2026-09-14
Tags: iam, users, policies, access keys, security

Create AWS IAM users, issue and rotate access keys, and write and attach JSON policies that control exactly what each user can do in a Spinifex account.

## Overview

Spinifex implements AWS-compatible IAM covering user management, access key lifecycle, policy CRUD, and policy attachment. Inline user policies (`put-user-policy`), user and policy tagging, [groups](https://docs.mulgadc.com/docs/iam-groups), and [roles and instance profiles](https://docs.mulgadc.com/docs/iam-roles-and-instance-profiles) are also supported. All IAM resources are scoped to the account that creates them — users in one account cannot see or modify resources in another.

When you create an account with `spx admin account create`, Spinifex bootstraps a root user with an `AdministratorAccess` policy and writes the credentials to `~/.aws/credentials`. From there, you use the standard AWS CLI to manage additional users and permissions.

**How authentication works:** Every AWS CLI request is signed with SigV4 using an access key pair. The gateway verifies the signature, resolves the caller's account, and evaluates attached policies before routing the request. The root user (account `000000000000`) bypasses policy evaluation entirely.

## Prerequisites

- A running Spinifex cluster (see [Setting Up Your Cluster](https://docs.mulgadc.com/docs/setting-up-your-cluster))
- AWS CLI configured with the `spinifex` profile:

```bash
export AWS_PROFILE=spinifex
```

## Instructions

## Users

### Create a User

```bash
aws iam create-user --user-name alice
```

The response includes the user's ARN, unique ID, and creation date:

```json
{
  "User": {
    "UserName": "alice",
    "UserId": "AIDA1A2B3C4D5E6F7890",
    "Arn": "arn:aws:iam::000000000001:user/alice",
    "Path": "/",
    "CreateDate": "2026-03-24T10:00:00Z"
  }
}
```

Use `--path` to organise users into hierarchical groups:

```bash
aws iam create-user --user-name bob --path /developers/
```

### Get a User

```bash
aws iam get-user --user-name alice
```

### List Users

```bash
aws iam list-users
```

Filter by path prefix:

```bash
aws iam list-users --path-prefix /developers/
```

### Delete a User

Before deleting a user, remove all access keys and detach all policies:

```bash
# Remove access keys
aws iam list-access-keys --user-name alice
aws iam delete-access-key --user-name alice --access-key-id AKIA...

# Detach policies
aws iam list-attached-user-policies --user-name alice
aws iam detach-user-policy --user-name alice \
  --policy-arn arn:aws:iam::000000000001:policy/MyPolicy

# Now delete
aws iam delete-user --user-name alice
```

## Access Keys

Access keys are how users authenticate with the AWS CLI. Each user can have up to **2 access keys** at a time, allowing key rotation without downtime. For credentials that expire on their own instead of a long-lived key pair, see [STS and Temporary Credentials](https://docs.mulgadc.com/docs/sts). Workloads running inside an EC2 instance should not carry a user's access key at all — they pick up role credentials automatically through [IMDS](https://docs.mulgadc.com/docs/imds).

### Create an Access Key

```bash
aws iam create-access-key --user-name alice
```

```json
{
  "AccessKey": {
    "UserName": "alice",
    "AccessKeyId": "AKIA1A2B3C4D5E6F7890ABCD",
    "Status": "Active",
    "SecretAccessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    "CreateDate": "2026-03-24T10:05:00Z"
  }
}
```

> **The secret access key is only shown once.** Save it immediately. If lost, delete the key and create a new one.

Configure a profile for the new key. A Spinifex profile also needs the gateway endpoint and CA bundle (copy the values from your existing `spinifex-*` profile in `~/.aws/config`):

```bash
aws configure set aws_access_key_id AKIA1A2B3C4D5E6F7890ABCD --profile spinifex-alice
aws configure set aws_secret_access_key wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY --profile spinifex-alice
aws configure set region ap-southeast-2 --profile spinifex-alice
aws configure set endpoint_url https://localhost:9999 --profile spinifex-alice
aws configure set ca_bundle /var/lib/spinifex/config/ca.pem --profile spinifex-alice
```

Then set `AWS_PROFILE=spinifex-alice` to use it.

### List Access Keys

```bash
aws iam list-access-keys --user-name alice
```

### Rotate Access Keys

Create a second key, update your configuration, then delete the old one:

```bash
# 1. Create new key (while old key still works)
aws iam create-access-key --user-name alice

# 2. Update the profile with the new key
aws configure set aws_access_key_id AKIA_NEW_KEY_ID --profile spinifex-alice
aws configure set aws_secret_access_key NEW_SECRET --profile spinifex-alice

# 3. Verify the new key works
AWS_PROFILE=spinifex-alice aws sts get-caller-identity

# 4. Delete the old key
aws iam delete-access-key --user-name alice --access-key-id AKIA_OLD_KEY_ID
```

### Deactivate an Access Key

Temporarily disable a key without deleting it:

```bash
aws iam update-access-key --user-name alice \
  --access-key-id AKIA1A2B3C4D5E6F7890ABCD \
  --status Inactive
```

Reactivate it later:

```bash
aws iam update-access-key --user-name alice \
  --access-key-id AKIA1A2B3C4D5E6F7890ABCD \
  --status Active
```

## Policies

Policies are JSON documents that define what actions a user can perform on which resources.

### Policy Document Format

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["ec2:RunInstances", "ec2:DescribeInstances"],
      "Resource": "*"
    }
  ]
}
```

Each statement has:

- **Effect** — `Allow` or `Deny`
- **Action** — Service actions (e.g. `ec2:RunInstances`, `s3:GetObject`). Supports wildcards: `ec2:*`, `s3:Get*`, or `*` for all actions.
- **Resource** — Target resources. Use `*` for all resources.

**Evaluation order:** An explicit `Deny` always wins. If no statement matches, access is denied by default.

### Create a Policy

Save the policy document to a file, then create the policy:

```bash
cat > /tmp/ec2-readonly.json << 'EOF'
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeImages",
                "ec2:DescribeKeyPairs",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSubnets",
                "ec2:DescribeVpcs"
            ],
            "Resource": "*"
        }
    ]
}
EOF

aws iam create-policy \
  --policy-name EC2ReadOnly \
  --policy-document file:///tmp/ec2-readonly.json
```

```json
{
  "Policy": {
    "PolicyName": "EC2ReadOnly",
    "PolicyId": "ANPA1A2B3C4D5E6F7890",
    "Arn": "arn:aws:iam::000000000001:policy/EC2ReadOnly",
    "Path": "/",
    "DefaultVersionId": "v1",
    "CreateDate": "2026-03-24T10:10:00Z"
  }
}
```

### Common Policy Examples

**Full administrator access:**

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "*",
      "Resource": "*"
    }
  ]
}
```

**S3 read/write (for hybrid sync, backups):**

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:ListBucket",
        "s3:DeleteObject"
      ],
      "Resource": "*"
    }
  ]
}
```

**EC2 operator (launch and manage instances, no VPC changes):**

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:RunInstances",
        "ec2:StartInstances",
        "ec2:StopInstances",
        "ec2:TerminateInstances",
        "ec2:DescribeInstances",
        "ec2:DescribeImages"
      ],
      "Resource": "*"
    }
  ]
}
```

**Deny terminate (attach alongside a broader Allow to prevent accidental termination):**

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Deny",
      "Action": "ec2:TerminateInstances",
      "Resource": "*"
    }
  ]
}
```

### Get a Policy

```bash
aws iam get-policy \
  --policy-arn arn:aws:iam::000000000001:policy/EC2ReadOnly
```

### Get the Policy Document

Use `get-policy-version` with version `v1` to retrieve the actual JSON document:

```bash
aws iam get-policy-version \
  --policy-arn arn:aws:iam::000000000001:policy/EC2ReadOnly \
  --version-id v1
```

### List Policies

```bash
aws iam list-policies
```

### Delete a Policy

A policy must be detached from all users before it can be deleted:

```bash
aws iam delete-policy \
  --policy-arn arn:aws:iam::000000000001:policy/EC2ReadOnly
```

## Attaching Policies

Policies have no effect until attached to a user.

### Attach a Policy to a User

```bash
aws iam attach-user-policy --user-name alice \
  --policy-arn arn:aws:iam::000000000001:policy/EC2ReadOnly
```

### List a User's Policies

```bash
aws iam list-attached-user-policies --user-name alice
```

### Detach a Policy

```bash
aws iam detach-user-policy --user-name alice \
  --policy-arn arn:aws:iam::000000000001:policy/EC2ReadOnly
```

## Putting It All Together

A complete workflow for onboarding a developer with scoped EC2 and S3 access:

```bash
# 1. Create the user
aws iam create-user --user-name dev-carol --path /developers/

# 2. Create an access key
aws iam create-access-key --user-name dev-carol
# Save the AccessKeyId and SecretAccessKey from the output

# 3. Create a scoped policy
cat > /tmp/dev-policy.json << 'EOF'
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "EC2Access",
            "Effect": "Allow",
            "Action": ["ec2:*"],
            "Resource": "*"
        },
        {
            "Sid": "S3ReadOnly",
            "Effect": "Allow",
            "Action": ["s3:GetObject", "s3:ListBucket"],
            "Resource": "*"
        },
        {
            "Sid": "NoTerminate",
            "Effect": "Deny",
            "Action": "ec2:TerminateInstances",
            "Resource": "*"
        }
    ]
}
EOF

aws iam create-policy \
  --policy-name DeveloperAccess \
  --policy-document file:///tmp/dev-policy.json

# 4. Attach the policy
aws iam attach-user-policy --user-name dev-carol \
  --policy-arn arn:aws:iam::000000000001:policy/DeveloperAccess

# 5. Configure the developer's AWS CLI profile with the key from step 2.
#    endpoint_url and ca_bundle: copy from an existing spinifex-* profile in ~/.aws/config
aws configure set aws_access_key_id AKIA_CAROL_KEY_ID --profile spinifex-carol
aws configure set aws_secret_access_key CAROL_SECRET --profile spinifex-carol
aws configure set region ap-southeast-2 --profile spinifex-carol
aws configure set endpoint_url https://localhost:9999 --profile spinifex-carol
aws configure set ca_bundle /var/lib/spinifex/config/ca.pem --profile spinifex-carol

# 6. Verify
AWS_PROFILE=spinifex-carol aws ec2 describe-instances
AWS_PROFILE=spinifex-carol aws ec2 terminate-instances --instance-ids i-123
# ^ This will be denied by the NoTerminate statement
```

## Troubleshooting

### AccessDenied on IAM Commands

The calling user must have IAM permissions. Attach a policy with `iam:*` actions, or use the account's admin profile:

```bash
export AWS_PROFILE=spinifex
aws iam list-users
```

### InvalidClientTokenId

The access key is either inactive or does not exist. Check the key status:

```bash
aws iam list-access-keys --user-name alice
```

If the key shows `Inactive`, reactivate it:

```bash
aws iam update-access-key --user-name alice \
  --access-key-id AKIA... --status Active
```

If the key was deleted, create a new one.

### DeleteConflict When Deleting a User

The user still has access keys, attached policies, or [group memberships](https://docs.mulgadc.com/docs/iam-groups). Remove them first:

```bash
# Check for access keys
aws iam list-access-keys --user-name alice

# Check for attached policies
aws iam list-attached-user-policies --user-name alice

# Check for group memberships
aws iam list-groups-for-user --user-name alice
```

### DeleteConflict When Deleting a Policy

The policy is still attached to one or more users. Detach it from all users before deleting:

```bash
# Find who has it attached, then detach
aws iam detach-user-policy --user-name alice \
  --policy-arn arn:aws:iam::000000000001:policy/MyPolicy

aws iam delete-policy \
  --policy-arn arn:aws:iam::000000000001:policy/MyPolicy
```

### LimitExceeded When Creating Access Keys

Each user can have at most 2 access keys. Delete an existing key before creating a new one:

```bash
aws iam list-access-keys --user-name alice
aws iam delete-access-key --user-name alice --access-key-id AKIA_OLD
aws iam create-access-key --user-name alice
```

### MalformedPolicyDocument

The policy JSON is invalid. Check that:

- `Version` is exactly `"2012-10-17"`
- At least one `Statement` exists
- Each statement has `Effect` (`Allow`/`Deny`), `Action`, and `Resource`
- The document is under 6144 bytes
- The JSON is well-formed (no trailing commas, correct quoting)

---

# IAM Groups for Shared User Permissions

URL: https://docs.mulgadc.com/docs/iam-groups
Category: Identity and Access
Updated: 2026-09-14
Tags: iam, groups, users, policies, security

Organise AWS IAM users into groups, manage membership, and attach managed or inline policies once for the whole team instead of granting them user by user.

## Overview

A group is a collection of [IAM users](https://docs.mulgadc.com/docs/iam-users-and-policies). Policies attached to a group apply to every member, so instead of attaching the same policy to each developer individually, you attach it once to a `developers` group and manage membership.

Members inherit both the group's **attached managed policies** and its **inline policies**, combined with any policies on the user itself. Policy evaluation is the same as everywhere else: an explicit `Deny` in any applicable policy wins, and anything not allowed is denied.

Groups cannot be nested, and a group is not a principal — it cannot sign requests or own access keys. The identity you assume to get credentials is an [IAM role](https://docs.mulgadc.com/docs/iam-roles-and-instance-profiles), not a group. Like all IAM resources, groups are scoped to the account that creates them.

## Prerequisites

- A running Spinifex cluster (see [Setting Up Your Cluster](https://docs.mulgadc.com/docs/setting-up-your-cluster))
- AWS CLI configured with the `spinifex` profile:

```bash
export AWS_PROFILE=spinifex
```

## Instructions

## Groups

### Create a Group

```bash
aws iam create-group --group-name developers
```

```json
{
  "Group": {
    "Path": "/",
    "GroupName": "developers",
    "GroupId": "AGPA1A2B3C4D5E6F7890",
    "Arn": "arn:aws:iam::000000000001:group/developers",
    "CreateDate": "2026-07-03T10:00:00Z"
  }
}
```

Use `--path` to organise groups hierarchically:

```bash
aws iam create-group --group-name ops --path /teams/
```

### Get and List Groups

`get-group` returns the group and its members:

```bash
aws iam get-group --group-name developers
aws iam list-groups
aws iam list-groups --path-prefix /teams/
```

### Delete a Group

A group must have no members and no policies before it can be deleted — see [Cleaning up a group](#deleteconflict-when-deleting-a-group).

```bash
aws iam delete-group --group-name developers
```

## Membership

Add and remove users:

```bash
aws iam add-user-to-group --group-name developers --user-name carol
aws iam remove-user-from-group --group-name developers --user-name carol
```

List a group's members (the `Users` array of `get-group`):

```bash
aws iam get-group --group-name developers --query 'Users[].UserName'
```

List the groups a user belongs to:

```bash
aws iam list-groups-for-user --user-name carol --query 'Groups[].GroupName'
```

> **Group membership blocks user deletion.** `delete-user` returns `DeleteConflict` while the user is still in any group — remove them from all groups first.

## Group Policies

### Attach a Managed Policy

```bash
aws iam attach-group-policy --group-name developers \
  --policy-arn arn:aws:iam::000000000001:policy/EC2ReadOnly

aws iam list-attached-group-policies --group-name developers

aws iam detach-group-policy --group-name developers \
  --policy-arn arn:aws:iam::000000000001:policy/EC2ReadOnly
```

See [IAM Users and Policies](https://docs.mulgadc.com/docs/iam-users-and-policies) for creating managed policies and the policy document format.

### Inline Group Policies

Inline policies live inside the group and are deleted with it:

```bash
aws iam put-group-policy --group-name developers \
  --policy-name keypair-mgmt \
  --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["ec2:CreateKeyPair","ec2:DeleteKeyPair"],"Resource":"*"}]}'

aws iam list-group-policies --group-name developers
aws iam get-group-policy --group-name developers --policy-name keypair-mgmt
aws iam delete-group-policy --group-name developers --policy-name keypair-mgmt
```

Members are authorised by the union of the group's attached and inline policies. A user in `developers` with the policies above can call `ec2:DescribeInstances` (via the attached `EC2ReadOnly`) and `ec2:CreateKeyPair` (via the inline policy), while anything not granted — say `ec2:RunInstances` — is denied.

## Putting It All Together

Onboard a team with shared permissions:

```bash
# 1. Create the group and grant it permissions
aws iam create-group --group-name developers
aws iam attach-group-policy --group-name developers \
  --policy-arn arn:aws:iam::000000000001:policy/EC2ReadOnly

# 2. Create users and add them to the group
aws iam create-user --user-name carol
aws iam add-user-to-group --group-name developers --user-name carol

# 3. Create access keys and configure profiles as usual
aws iam create-access-key --user-name carol

# 4. Verify: carol can describe instances via group policy, nothing more
AWS_PROFILE=spinifex-carol aws ec2 describe-instances
AWS_PROFILE=spinifex-carol aws ec2 run-instances --image-id ami-123 --instance-type t3.micro
# ^ AccessDenied — not granted by the group
```

To change the whole team's permissions later, edit the group's policies once — no per-user changes needed.

## Troubleshooting

### DeleteConflict When Deleting a Group

The group still has members, attached policies, or inline policies. Empty it first:

```bash
# Members
aws iam get-group --group-name developers --query 'Users[].UserName'
aws iam remove-user-from-group --group-name developers --user-name carol

# Attached policies
aws iam list-attached-group-policies --group-name developers
aws iam detach-group-policy --group-name developers --policy-arn arn:aws:iam::000000000001:policy/EC2ReadOnly

# Inline policies
aws iam list-group-policies --group-name developers
aws iam delete-group-policy --group-name developers --policy-name keypair-mgmt

aws iam delete-group --group-name developers
```

### DeleteConflict When Deleting a User

Group membership counts as a subordinate entity, alongside access keys and attached policies. Find and leave the user's groups:

```bash
aws iam list-groups-for-user --user-name carol
aws iam remove-user-from-group --group-name developers --user-name carol
```

### NoSuchEntity

The group or user referenced does not exist — both `get-group` on a missing group and `add-user-to-group` with a missing user return this. Check spelling with `list-groups` / `list-users`.

### Member Still Denied After Attaching a Group Policy

Check the policy landed on the right group and the user is actually a member:

```bash
aws iam list-groups-for-user --user-name carol
aws iam list-attached-group-policies --group-name developers
aws iam list-group-policies --group-name developers
```

Remember an explicit `Deny` in any policy that applies to the user — their own or any of their groups' — overrides the `Allow`.

---

# IAM Roles and Instance Profiles for EC2

URL: https://docs.mulgadc.com/docs/iam-roles-and-instance-profiles
Category: Identity and Access
Updated: 2026-09-14
Tags: iam, roles, instance profiles, trust policies, credentials, security

Create AWS IAM roles with trust policies, wrap them in instance profiles, and launch EC2 instances that receive credentials with no static access keys at all.

## Overview

A role is an IAM identity with permissions but no long-lived credentials. Unlike an [IAM group](https://docs.mulgadc.com/docs/iam-groups), which only bundles users together, a role is a principal in its own right: instead of an access key pair it has a **trust policy** declaring who may assume it, and whoever assumes the role receives short-lived, auto-rotating STS credentials.

An **instance profile** is the container that binds a role to an EC2 instance. An instance launched with a profile gets the role's credentials delivered through [IMDS](https://docs.mulgadc.com/docs/imds) — no static keys baked into the image, no `~/.aws/credentials` inside the guest.

The full arc is: create a role → attach permissions → wrap it in an instance profile → launch an instance with the profile → the guest picks up credentials automatically. Each instance profile holds exactly **one role** (matching AWS).

Like all IAM resources, roles and instance profiles are scoped to the account that creates them.

## Prerequisites

- A running Spinifex cluster (see [Setting Up Your Cluster](https://docs.mulgadc.com/docs/setting-up-your-cluster))
- AWS CLI configured with the `spinifex` profile:

```bash
export AWS_PROFILE=spinifex
```

## Instructions

## Roles

### Create a Role

Every role needs a trust policy. For a role that EC2 instances will use, trust the `ec2.amazonaws.com` service principal:

```bash
cat > /tmp/ec2-trust.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "ec2.amazonaws.com" },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

aws iam create-role --role-name app-server \
  --assume-role-policy-document file:///tmp/ec2-trust.json \
  --description "Role for app servers"
```

```json
{
  "Role": {
    "Path": "/",
    "RoleName": "app-server",
    "RoleId": "AROA1A2B3C4D5E6F7890",
    "Arn": "arn:aws:iam::000000000001:role/app-server",
    "CreateDate": "2026-07-03T10:00:00Z",
    "AssumeRolePolicyDocument": {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": { "Service": "ec2.amazonaws.com" },
          "Action": "sts:AssumeRole"
        }
      ]
    },
    "Description": "Role for app servers",
    "MaxSessionDuration": 3600
  }
}
```

To let a user (rather than EC2) assume the role, trust an `AWS` principal instead:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::000000000001:user/admin" },
      "Action": "sts:AssumeRole"
    }
  ]
}
```

**Trust policy validation:** Spinifex rejects `NotPrincipal`, `NotAction`, empty `Principal` blocks, and empty-string `Action` elements at write time with `MalformedPolicyDocument`. `Condition` blocks are rejected except the `StringEquals` form used for web identity federation.

### Get, List, and Update Roles

```bash
aws iam get-role --role-name app-server
aws iam list-roles
aws iam list-roles --path-prefix /services/
```

Update the description or the maximum STS session duration (3600 seconds by default):

```bash
aws iam update-role --role-name app-server \
  --description "Updated" --max-session-duration 7200
```

Replace the trust policy on an existing role:

```bash
aws iam update-assume-role-policy --role-name app-server \
  --policy-document file:///tmp/ec2-trust.json
```

## Granting Permissions to a Role

A freshly created role can do nothing. Grant permissions the same two ways as users: attach managed policies, or embed inline policies.

### Attach a Managed Policy

```bash
aws iam attach-role-policy --role-name app-server \
  --policy-arn arn:aws:iam::000000000001:policy/S3ReadOnly

aws iam list-attached-role-policies --role-name app-server
aws iam detach-role-policy --role-name app-server \
  --policy-arn arn:aws:iam::000000000001:policy/S3ReadOnly
```

See [IAM Users and Policies](https://docs.mulgadc.com/docs/iam-users-and-policies) for creating managed policies and the policy document format.

### Inline Role Policies

Inline policies live inside the role and are deleted with it:

```bash
aws iam put-role-policy --role-name app-server \
  --policy-name ec2-describe \
  --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["ec2:DescribeInstances"],"Resource":"*"}]}'

aws iam list-role-policies --role-name app-server
aws iam get-role-policy --role-name app-server --policy-name ec2-describe
aws iam delete-role-policy --role-name app-server --policy-name ec2-describe
```

## Instance Profiles

Create a profile and add the role to it:

```bash
aws iam create-instance-profile --instance-profile-name app-server-profile
aws iam add-role-to-instance-profile \
  --instance-profile-name app-server-profile --role-name app-server
```

Inspect it — the `Roles` array shows the bound role:

```bash
aws iam get-instance-profile --instance-profile-name app-server-profile
aws iam list-instance-profiles
aws iam list-instance-profiles-for-role --role-name app-server
```

A profile holds at most one role; adding a second returns `LimitExceeded`. To swap roles, remove the current one first:

```bash
aws iam remove-role-from-instance-profile \
  --instance-profile-name app-server-profile --role-name app-server
```

## Launching an Instance with a Role

Pass the profile by name (or ARN) at launch:

```bash
aws ec2 run-instances \
  --image-id ami-0dd52c90440ff4150 \
  --instance-type t3.micro \
  --subnet-id subnet-a0e5fc381376d82a1 \
  --iam-instance-profile Name=app-server-profile
```

The instance description shows the association:

```bash
aws ec2 describe-instances --instance-ids i-d2e09ff7de71b6341 \
  --query 'Reservations[0].Instances[0].IamInstanceProfile'
```

```json
{
  "Arn": "arn:aws:iam::000000000001:instance-profile/app-server-profile",
  "Id": "AIPA1A2B3C4D5E6F7890"
}
```

Inside the guest, the AWS CLI and SDKs pick up the role's credentials from IMDS with no configuration — see [IMDS](https://docs.mulgadc.com/docs/imds) for fetching them manually, rotation timing, and limits.

## Managing Profile Associations at Runtime

Profiles can be attached to, removed from, or swapped on a **running** instance — no relaunch needed.

List associations:

```bash
aws ec2 describe-iam-instance-profile-associations \
  --filters Name=instance-id,Values=i-d2e09ff7de71b6341
```

```json
{
  "IamInstanceProfileAssociations": [
    {
      "AssociationId": "iip-assoc-9d7a617c605e337cd",
      "InstanceId": "i-d2e09ff7de71b6341",
      "IamInstanceProfile": {
        "Arn": "arn:aws:iam::000000000001:instance-profile/app-server-profile"
      },
      "State": "associated"
    }
  ]
}
```

Attach a profile to an instance launched without one:

```bash
aws ec2 associate-iam-instance-profile \
  --instance-id i-d2e09ff7de71b6341 \
  --iam-instance-profile Name=app-server-profile
```

Swap to a different profile (returns a new association ID):

```bash
aws ec2 replace-iam-instance-profile-association \
  --association-id iip-assoc-9d7a617c605e337cd \
  --iam-instance-profile Name=other-profile
```

Remove the profile — the guest's `iam/` metadata subtree starts returning 404:

```bash
aws ec2 disassociate-iam-instance-profile \
  --association-id iip-assoc-9d7a617c605e337cd
```

## Assuming a Role

Users and services can assume a role directly with STS, provided **both** gates pass: the role's trust policy admits their principal, and their own identity policy grants `sts:AssumeRole` on the role's ARN. See [STS](https://docs.mulgadc.com/docs/sts) for the identity-side grant.

```bash
aws sts assume-role \
  --role-arn arn:aws:iam::000000000001:role/deploy \
  --role-session-name release-42
```

```json
{
  "Credentials": {
    "AccessKeyId": "ASIA1A2B3C4D5E6F7890",
    "SecretAccessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    "SessionToken": "IQoJb3JpZ2luX2VjE...",
    "Expiration": "2026-07-03T11:00:00Z"
  },
  "AssumedRoleUser": {
    "AssumedRoleId": "AROA1A2B3C4D5E6F7890:release-42",
    "Arn": "arn:aws:sts::000000000001:assumed-role/deploy/release-42"
  }
}
```

The returned `ASIA`-prefixed credentials are temporary, valid up to the role's `MaxSessionDuration`. See [STS and Temporary Credentials](https://docs.mulgadc.com/docs/sts) for session durations, using the credentials, and the other STS flows.

## Cleaning Up

Deletion order matters — a role must be empty and unreferenced before it can be deleted:

```bash
# 1. Remove the role from any instance profiles
aws iam remove-role-from-instance-profile \
  --instance-profile-name app-server-profile --role-name app-server
aws iam delete-instance-profile --instance-profile-name app-server-profile

# 2. Delete inline policies and detach managed policies
aws iam delete-role-policy --role-name app-server --policy-name ec2-describe
aws iam detach-role-policy --role-name app-server \
  --policy-arn arn:aws:iam::000000000001:policy/S3ReadOnly

# 3. Delete the role
aws iam delete-role --role-name app-server
```

## Troubleshooting

### MalformedPolicyDocument When Creating a Role

The trust policy is invalid. In addition to the JSON checks that apply to all policies, trust policies must not use `NotPrincipal`, `NotAction`, an empty `Principal` block, or an empty-string `Action`. `Condition` blocks are rejected except `StringEquals` conditions used with web identity federation.

### DeleteConflict When Deleting a Role

The role still has attached managed policies, inline policies, or is bound to an instance profile. Remove all three first:

```bash
aws iam list-attached-role-policies --role-name app-server
aws iam list-role-policies --role-name app-server
aws iam list-instance-profiles-for-role --role-name app-server
```

### DeleteConflict When Deleting an Instance Profile

The profile still contains a role:

```bash
aws iam remove-role-from-instance-profile \
  --instance-profile-name app-server-profile --role-name app-server
aws iam delete-instance-profile --instance-profile-name app-server-profile
```

### LimitExceeded When Adding a Role to a Profile

The profile already holds a role — the limit is one per profile. Remove the existing role first, or create a second profile.

### InvalidIamInstanceProfile.NotFound at Launch

The profile name or ARN passed to `run-instances` does not exist in your account. Check the spelling:

```bash
aws iam list-instance-profiles
```

### Instance Has No Credentials in IMDS

If the `iam/` metadata subtree returns 404, the instance has no associated profile. Attach one without relaunching:

```bash
aws ec2 associate-iam-instance-profile \
  --instance-id i-d2e09ff7de71b6341 \
  --iam-instance-profile Name=app-server-profile
```

If the profile is associated but contains no role, the `security-credentials/` listing is empty; add the role and credentials appear on the next request.

---

# STS Temporary Credentials and AssumeRole

URL: https://docs.mulgadc.com/docs/sts
Category: Identity and Access
Updated: 2026-09-14
Tags: sts, iam, temporary credentials, assume role, oidc, security

Use AWS STS to issue temporary credentials: assume an IAM role, request a session token, or federate Kubernetes workloads with OIDC web identity on Spinifex.

## Overview

STS (Security Token Service) issues **temporary credentials**: an `ASIA`-prefixed access key, a secret key, and a session token, all of which expire together at a fixed time. Nothing to rotate, nothing to revoke — the credentials simply stop working.

Spinifex supports the three main ways to obtain them:

- **`assume-role`** — exchange your IAM user credentials for a [role's](https://docs.mulgadc.com/docs/iam-roles-and-instance-profiles) permissions, gated by both the role's trust policy and your own `sts:AssumeRole` grant.
- **`get-session-token`** — get a time-boxed copy of your own [IAM user's](https://docs.mulgadc.com/docs/iam-users-and-policies) permissions.
- **`assume-role-with-web-identity`** — exchange an OIDC ID token (a Kubernetes ServiceAccount token) for role credentials, with no IAM credentials at all.

EC2 instances get role credentials a fourth way, automatically through [IMDS](https://docs.mulgadc.com/docs/imds) — no STS call needed in the guest.

## Prerequisites

- A running Spinifex cluster (see [Setting Up Your Cluster](https://docs.mulgadc.com/docs/setting-up-your-cluster))
- AWS CLI configured with the `spinifex` profile:

```bash
export AWS_PROFILE=spinifex
```

## Instructions

## Checking Your Identity

`get-caller-identity` returns whoever signed the request — it needs no permissions and never fails for authorisation reasons, which makes it the standard "which credentials am I using?" check:

```bash
aws sts get-caller-identity
```

```json
{
    "UserId": "AIDA1A2B3C4D5E6F7890",
    "Account": "000000000001",
    "Arn": "arn:aws:iam::000000000001:user/admin"
}
```

Run with temporary role credentials, the ARN switches to the assumed-role form — see [Using Temporary Credentials](#using-temporary-credentials).

## Assuming a Role

`assume-role` needs two independent grants, and a missing one denies the call:

1. The target role's **trust policy** must admit your principal.
2. Your own **identity policy** must grant `sts:AssumeRole` on the role's ARN.

The second is why `"Principal": {"AWS": "arn:aws:iam::000000000001:root"}` in a trust policy is safe: it delegates the decision to identity policies in that account rather than opening the role to everyone in it.

Create a role that trusts an IAM user (see [IAM Roles and Instance Profiles](https://docs.mulgadc.com/docs/iam-roles-and-instance-profiles) for the full role lifecycle):

```bash
cat > /tmp/deploy-trust.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::000000000001:user/admin" },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

aws iam create-role --role-name deploy \
  --assume-role-policy-document file:///tmp/deploy-trust.json

aws iam put-role-policy --role-name deploy --policy-name ec2-read \
  --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["ec2:Describe*"],"Resource":"*"}]}'
```

That policy is what the *session* may do. The caller still needs its own grant to make the call at all:

```bash
aws iam put-user-policy --user-name admin --policy-name assume-deploy \
  --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"sts:AssumeRole","Resource":"arn:aws:iam::000000000001:role/deploy"}]}'
```

Scope `Resource` to the role ARN exactly as `create-role` returned it, path included — a wildcard such as `arn:aws:iam::000000000001:role/*` grants assumption of every role the account's trust policies admit you to.

Then assume it:

```bash
aws sts assume-role \
  --role-arn arn:aws:iam::000000000001:role/deploy \
  --role-session-name release-42
```

```json
{
    "Credentials": {
        "AccessKeyId": "ASIA1A2B3C4D5E6F7890",
        "SecretAccessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
        "SessionToken": "IQoJb3JpZ2luX2VjE...",
        "Expiration": "2026-07-03T11:00:00Z"
    },
    "AssumedRoleUser": {
        "AssumedRoleId": "AROA1A2B3C4D5E6F7890:release-42",
        "Arn": "arn:aws:sts::000000000001:assumed-role/deploy/release-42"
    },
    "PackedPolicySize": 0
}
```

The session name tags every session so audit logs can tell *who* used the role; it appears in the assumed-role ARN and the `UserId`.

If the trust policy does not allow your principal, `assume-role` fails with `AccessDenied` — no matter what IAM permissions you hold. It fails the same way when your own identity policy carries no `sts:AssumeRole` grant on the role, no matter what the trust policy says.

### Session Duration

`--duration-seconds` must be between **900** (15 minutes) and the smaller of the role's `MaxSessionDuration` and **43200** (12 hours); the default is **3600** (1 hour). A value outside that window is rejected with `ValidationError`, so to run longer sessions, raise the role's ceiling first:

```bash
# 7200 > MaxSessionDuration (3600) → ValidationError
aws sts assume-role --role-arn arn:aws:iam::000000000001:role/deploy \
  --role-session-name long-build --duration-seconds 7200

aws iam update-role --role-name deploy --max-session-duration 43200

aws sts assume-role --role-arn arn:aws:iam::000000000001:role/deploy \
  --role-session-name long-build --duration-seconds 7200
```

### Flags That Differ from AWS

- `--external-id` and `--source-identity` are accepted and logged but **not enforced** — trust policies cannot carry the `Condition` blocks that would check them (see [Trust Policy Validation](#trust-policy-validation)). Do not rely on an external ID as a security boundary.
- `--serial-number` / `--token-code` (MFA) are not supported and return `InvalidParameterValue`.
- `--policy` / `--policy-arns` (session policies) are not supported and return `PackedPolicyTooLarge`.

## Using Temporary Credentials

Temporary credentials are used exactly like access keys, plus a third value — the session token. Export all three as environment variables (which take precedence over any profile):

```bash
export AWS_ACCESS_KEY_ID=ASIA1A2B3C4D5E6F7890
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export AWS_SESSION_TOKEN=IQoJb3JpZ2luX2VjE...
export AWS_ENDPOINT_URL=https://localhost:9999
export AWS_CA_BUNDLE=/var/lib/spinifex/config/ca.pem
export AWS_DEFAULT_REGION=ap-southeast-2
```

Or store them in a named profile — the same shape as a user profile with `aws_session_token` added:

```bash
aws configure set aws_access_key_id ASIA1A2B3C4D5E6F7890 --profile spinifex-deploy
aws configure set aws_secret_access_key wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY --profile spinifex-deploy
aws configure set aws_session_token IQoJb3JpZ2luX2VjE... --profile spinifex-deploy
aws configure set region ap-southeast-2 --profile spinifex-deploy
aws configure set endpoint_url https://localhost:9999 --profile spinifex-deploy
aws configure set ca_bundle /var/lib/spinifex/config/ca.pem --profile spinifex-deploy
```

Requests are then authorised as the assumed role, not your user:

```bash
aws sts get-caller-identity
# "Arn": "arn:aws:sts::000000000001:assumed-role/deploy/release-42"

aws ec2 describe-instances     # allowed by the role's ec2-read policy
aws iam list-users             # AccessDenied — the role grants ec2:Describe* only
```

When the expiration time passes, every call fails until you fetch fresh credentials — there is no refresh; run `assume-role` again.

## Session Tokens for Your Own User

`get-session-token` issues temporary credentials with your **own user's** permissions — no role, no trust policy. Useful for handing a build script credentials that expire on their own instead of your long-lived access key:

```bash
aws sts get-session-token --duration-seconds 3600
```

```json
{
    "Credentials": {
        "AccessKeyId": "ASIA1A2B3C4D5E6F7890",
        "SecretAccessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
        "SessionToken": "IQoJb3JpZ2luX2VjE...",
        "Expiration": "2026-07-03T11:00:00Z"
    }
}
```

Duration runs from **900** to **129600** seconds (36 hours), defaulting to **43200** (12 hours). Unlike `assume-role`, out-of-range values are **clamped, not rejected** — asking for 200000 seconds quietly returns a 36-hour session. Use the credentials exactly as in [Using Temporary Credentials](#using-temporary-credentials); `get-caller-identity` keeps reporting your user ARN, since no role is involved.

MFA (`--serial-number` / `--token-code`) is not supported and returns `InvalidParameterValue`.

## Web Identity Federation (IRSA)

`assume-role-with-web-identity` exchanges an OIDC ID token for role credentials — the mechanism behind **IRSA** (IAM Roles for Service Accounts), where a Kubernetes pod's ServiceAccount token becomes its cloud identity. The call is anonymous: the JWT is the identity, so no IAM credentials sign the request.

In Spinifex the token issuer is an [EKS](https://docs.mulgadc.com/docs/eks) cluster. Each cluster publishes a signing-key set (JWKS) under an issuer URL of the form `https://{host}/oidc/eks/{region}/{account-id}/{cluster-name}`, and STS verifies tokens against it directly — no external identity provider is contacted.

### Register the OIDC Provider

The role account must register the issuer as an IAM OIDC provider before any token from it is accepted:

```bash
aws iam create-open-id-connect-provider \
  --url https://spinifex.example.com/oidc/eks/ap-southeast-2/000000000001/prod \
  --client-id-list sts.amazonaws.com \
  --thumbprint-list 9e99a48a9960b14926bb7f3b02e22da2b0ab7280
```

```json
{
    "OpenIDConnectProviderArn": "arn:aws:iam::000000000001:oidc-provider/spinifex.example.com/oidc/eks/ap-southeast-2/000000000001/prod",
    "Tags": []
}
```

Manage providers with the usual verbs — `get-open-id-connect-provider` and `list-open-id-connect-provider-tags` take the ARN, and tags work like every other IAM resource:

```bash
aws iam list-open-id-connect-providers
aws iam get-open-id-connect-provider \
  --open-id-connect-provider-arn arn:aws:iam::000000000001:oidc-provider/spinifex.example.com/oidc/eks/ap-southeast-2/000000000001/prod
aws iam delete-open-id-connect-provider \
  --open-id-connect-provider-arn arn:aws:iam::000000000001:oidc-provider/spinifex.example.com/oidc/eks/ap-southeast-2/000000000001/prod
```

### The IRSA Trust Policy

The role's trust policy names the provider as a `Federated` principal and pins the token's `sub` (the ServiceAccount) and `aud` claims with `StringEquals` — the one place a `Condition` block is allowed:

```bash
cat > /tmp/irsa-trust.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::000000000001:oidc-provider/spinifex.example.com/oidc/eks/ap-southeast-2/000000000001/prod"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "spinifex.example.com/oidc/eks/ap-southeast-2/000000000001/prod:sub": "system:serviceaccount:default:app",
          "spinifex.example.com/oidc/eks/ap-southeast-2/000000000001/prod:aud": "sts.amazonaws.com"
        }
      }
    }
  ]
}
EOF

aws iam create-role --role-name irsa-app \
  --assume-role-policy-document file:///tmp/irsa-trust.json
```

The condition keys are the issuer URL (scheme stripped) followed by `:sub` or `:aud`. Only these two keys and only `StringEquals` are accepted — anything wider (`StringLike`, wildcards, other keys) is rejected at `create-role` with `MalformedPolicyDocument` rather than silently over-granting.

### Exchange the Token

```bash
aws sts assume-role-with-web-identity \
  --role-arn arn:aws:iam::000000000001:role/irsa-app \
  --role-session-name pod-app-7f9c4 \
  --web-identity-token "$(cat /var/run/secrets/eks.amazonaws.com/serviceaccount/token)"
```

The response has the same `Credentials` block as `assume-role`, plus the verified `SubjectFromWebIdentityToken`, `Provider`, and `Audience`. Duration runs 900–43200 seconds (default 3600), independent of the role's `MaxSessionDuration`.

For a token to be accepted it must be an ES256-signed JWT whose `iss` matches a registered OIDC provider in the role's account, whose `aud` contains `sts.amazonaws.com`, whose signature verifies against the cluster's published JWKS, and which has not expired — any failure returns `InvalidIdentityToken`. In-cluster, none of this is manual: pods with an annotated ServiceAccount get the token mounted and the AWS SDK performs the exchange automatically.

## Trust Policy Validation

Spinifex validates trust policies (`AssumeRolePolicyDocument`) more strictly than AWS, rejecting at write time with `MalformedPolicyDocument`:

- `Condition` blocks anywhere **except** the `StringEquals` IRSA form on `sts:AssumeRoleWithWebIdentity` described above. This is why `--external-id` cannot be enforced.
- `NotPrincipal` and `NotAction` elements.
- Empty `Principal` blocks and empty-string `Action` elements.

Everything the validator accepts is evaluated at assume time; explicit `Deny` statements win, and a policy that matches no statement denies.

## Troubleshooting

### AccessDenied When Assuming a Role

Either gate can produce this. A denial from your own identity policy names the principal, action and resource:

```
User: arn:aws:iam::000000000001:user/admin is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::000000000001:role/deploy
```

A bare `AccessDenied` means the role's trust policy does not allow your principal, or the role does not exist — a missing role is deliberately indistinguishable from a denied one, matching AWS. Check who you are, that the role exists, what it trusts, and what you are granted:

```bash
aws sts get-caller-identity
aws iam list-roles --query 'Roles[].Arn'
aws iam get-role --role-name deploy \
  --query 'Role.AssumeRolePolicyDocument'
```

For web identity, `AccessDenied` also covers a trust-policy `Condition` that doesn't match the token's `sub`/`aud` claims.

### ValidationError on Duration

The requested `--duration-seconds` is outside 900–min(`MaxSessionDuration`, 43200). Check and raise the role's ceiling:

```bash
aws iam get-role --role-name deploy --query 'Role.MaxSessionDuration'
aws iam update-role --role-name deploy --max-session-duration 43200
```

Note `get-session-token` never returns this — it clamps instead.

### InvalidParameterValue

You passed an MFA flag (`--serial-number` / `--token-code`) — MFA is not supported. `--tags` and `--transitive-tag-keys` on `assume-role` return the same error.

### PackedPolicyTooLarge

You passed `--policy` or `--policy-arns`. Session policies are not supported — scope the role's own permissions instead.

### InvalidIdentityToken

The web identity token failed verification: malformed JWT, wrong signature algorithm (only ES256 is accepted), expired, `aud` missing `sts.amazonaws.com`, issuer not registered as an OIDC provider in the role's account, or the cluster's JWKS is unavailable. Confirm the provider is registered:

```bash
aws iam list-open-id-connect-providers
```

### Credentials Suddenly Rejected

Temporary credentials expired. Check the `Expiration` from the original response and mint a fresh set — sessions cannot be renewed or extended.

---

# IMDS: Instance Metadata Service (IMDSv2)

URL: https://docs.mulgadc.com/docs/imds
Category: Identity and Access
Updated: 2026-09-14
Tags: imds, metadata, instance identity, iam roles, credentials, security

Query AWS instance metadata, read user data, and fetch short-lived IAM role credentials from inside a guest VM using IMDSv2 session tokens on Spinifex.

## Overview

Every running guest VM can reach the Instance Metadata Service at `http://169.254.169.254`, exactly as on EC2. There is no in-VM agent to install and no in-guest route configuration — DHCP and fully static guests reach it identically.

IMDS answers questions a workload asks about itself: instance ID, instance type, private and public IPs, hostname, availability zone, security groups, the launch SSH key, user data, and — when the instance has an IAM instance profile — short-lived, auto-rotating role credentials.

**Spinifex is IMDSv2-only.** Every read requires a session token obtained with a `PUT` request. A tokenless (IMDSv1-style) `GET` returns `401 Unauthorized` with an empty body.

Requests are attributed to an instance by the network interface they arrive on and tokens are bound to that interface, so one instance can never read another's metadata or replay its tokens.

## Prerequisites

- A running Spinifex cluster (see [Setting Up Your Cluster](https://docs.mulgadc.com/docs/setting-up-your-cluster))
- A running instance to query from
- For IAM role credentials: an instance launched with `--iam-instance-profile`

## Instructions

## Using IMDS from a Guest

All commands in this section run **inside the guest VM**.

### Get a Session Token

Request a token with a TTL between 1 and 21600 seconds (6 hours):

```bash
TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
```

The token endpoint only accepts `PUT` — a `GET` returns `405 Method Not Allowed`. A missing or out-of-range TTL header returns `400 Bad Request`.

### Read Metadata

Send the token back in the `X-aws-ec2-metadata-token` header on every read:

```bash
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/instance-id
```

```
i-0a1b2c3d4e5f67890
```

Directory paths (ending in `/`) return a newline-separated listing of children:

```bash
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/
```

### One-Liner Pattern

For scripts, mint a short-lived token inline:

```bash
imds() {
  local token=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" \
    -H "X-aws-ec2-metadata-token-ttl-seconds: 60")
  curl -s -H "X-aws-ec2-metadata-token: $token" "http://169.254.169.254/latest/$1"
}

imds meta-data/instance-id
imds meta-data/local-ipv4
imds meta-data/placement/availability-zone
```

## Metadata Paths

`GET /` lists supported API versions (`2021-07-15`, `latest`). Any dated version segment (e.g. `/2016-09-02/...`, as probed by cloud-init) aliases to `/latest`.

Commonly used paths under `/latest/meta-data/`:

| Path | Returns |
| ---- | ------- |
| `instance-id` | Instance ID |
| `instance-type` | Instance type (e.g. `t3.micro`) |
| `ami-id` | Image the instance was launched from |
| `ami-launch-index` | Launch index within the reservation (`0..n-1`) |
| `reservation-id` | Reservation ID |
| `instance-life-cycle` | `spot` or `on-demand` |
| `local-ipv4` | Primary private IP |
| `public-ipv4` | Elastic/public IP; 404 when none |
| `public-hostname` | Mirrors `public-ipv4`; 404 when no public IP |
| `mac` | Primary interface MAC address |
| `hostname`, `local-hostname` | `ip-<dashed-ip>.<region>.compute.internal` |
| `security-groups` | Security group names, one per line |
| `placement/availability-zone` | Availability zone |
| `placement/region` | Region (AZ with trailing letter stripped) |
| `services/domain`, `services/partition` | `amazonaws.com` / `aws` |
| `public-keys/0/openssh-key` | Launch key pair's SSH public key; 404 if no key pair or the key was deleted |
| `iam/info` | Instance profile ARN and ID; 404 if no profile |
| `iam/security-credentials/<role>` | Temporary role credentials (see below) |
| `network/interfaces/macs/<mac>/...` | Primary interface subtree: `interface-id`, `owner-id`, `subnet-id`, `vpc-id`, `local-ipv4s`, `security-group-ids`, `subnet-ipv4-cidr-block`, `vpc-ipv4-cidr-block`, and more |

The `network/interfaces/macs/` subtree covers the **primary interface only**; querying another MAC returns 404 (multi-ENI metadata is deferred).

Paths that intentionally return **404**: `tags/instance/*`, `block-device-mapping/*`, `placement/{group-name,partition-number,availability-zone-id,host-id}`, `instance-action`, and `spot/{instance-action,termination-time}` (404 is the correct "no interruption scheduled" answer for spot pollers).

## User Data

User data supplied at launch (`run-instances --user-data`) is served at:

```bash
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/user-data
```

Returns the decoded user data, or 404 if the instance was launched without any. cloud-init consumes this automatically at first boot.

## IAM Role Credentials

Instances launched with an IAM instance profile get short-lived, auto-rotating credentials through IMDS — no static keys baked into the image. Creating roles and instance profiles is covered in [IAM Roles and Instance Profiles](https://docs.mulgadc.com/docs/iam-roles-and-instance-profiles). These are the same temporary credentials [STS](https://docs.mulgadc.com/docs/sts) issues, delivered to the guest automatically instead of through an explicit `assume-role` call.

```bash
# Discover the role name
ROLE=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/iam/security-credentials/)

# Fetch credentials
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  "http://169.254.169.254/latest/meta-data/iam/security-credentials/$ROLE"
```

```json
{
  "Code": "Success",
  "LastUpdated": "2026-07-03T10:00:00Z",
  "Type": "AWS-HMAC",
  "AccessKeyId": "ASIA1A2B3C4D5E6F7890",
  "SecretAccessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
  "Token": "IQoJb3JpZ2luX2VjE...",
  "Expiration": "2026-07-03T11:00:00Z",
  "AccountId": "000000000001"
}
```

Credentials are ASIA-prefixed temporary STS credentials valid for **1 hour**, re-minted automatically **5 minutes before expiry**. The AWS SDKs and CLI pick them up with no configuration:

```bash
# Inside the guest, with no ~/.aws/credentials:
aws sts get-caller-identity
```

If the instance has no profile, the whole `iam/` subtree returns 404 (and is omitted from the `meta-data/` listing).

## Instance Identity Document

An unsigned identity document (schema `2017-09-30`) is available at:

```bash
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/dynamic/instance-identity/document
```

```json
{
  "accountId": "000000000001",
  "architecture": "x86_64",
  "availabilityZone": "ap-southeast-2a",
  "imageId": "ami-0123456789abcdef0",
  "instanceId": "i-0a1b2c3d4e5f67890",
  "instanceType": "t3.micro",
  "pendingTime": "2026-07-03T09:58:12Z",
  "privateIp": "10.0.1.15",
  "region": "ap-southeast-2",
  "version": "2017-09-30"
}
```

The signed forms (`signature`, `pkcs7`, `rsa2048`) currently return 404 — they require a per-cluster signing key and land with EKS IRSA support.

## Metadata Options

Metadata options are managed from the **control plane** with the standard EC2 commands. The mutable knobs are the PUT response hop limit and `HttpTokens`; the remaining fields are fixed.

### Inspect

```bash
aws ec2 describe-instances --instance-ids i-0a1b2c3d4e5f67890 \
  --query 'Reservations[0].Instances[0].MetadataOptions'
```

```json
{
  "State": "applied",
  "HttpTokens": "required",
  "HttpPutResponseHopLimit": 1,
  "HttpEndpoint": "enabled",
  "HttpProtocolIpv6": "disabled",
  "InstanceMetadataTags": "disabled"
}
```

### Set at Launch

```bash
aws ec2 run-instances \
  --image-id ami-0123456789abcdef0 \
  --instance-type t3.micro \
  --metadata-options "HttpPutResponseHopLimit=2"
```

### Modify a Running or Stopped Instance

```bash
aws ec2 modify-instance-metadata-options \
  --instance-id i-0a1b2c3d4e5f67890 \
  --http-put-response-hop-limit 2
```

Raise the hop limit above the default of 1 when containers on the instance need IMDS access through an extra routing hop (e.g. containers on a bridge network fetching role credentials). Valid range is 1–64.

### Enabling IMDSv1

Instances default to `HttpTokens=required`, so every read needs a token. Set `optional` to also serve untokened reads:

```bash
aws ec2 run-instances \
  --image-id ami-0123456789abcdef0 \
  --instance-type t3.micro \
  --metadata-options "HttpTokens=optional"
```

This exists for guest agents that cannot perform the IMDSv2 handshake. The main one is **cloudbase-init**, the standard Windows bootstrap agent, whose `EC2Service` has no token support in any released version — a Windows instance left at `required` gets 401 on every read and so never receives its hostname, injected administrator password or user-data.

Prefer `required` everywhere else. IMDSv1 has no defence against a confused-deputy SSRF in a guest workload reaching the metadata endpoint on an attacker's behalf, which is the whole reason IMDSv2 binds tokens to the requesting interface.

Rejected settings:

- `--http-endpoint disabled` → `UnsupportedOperation` (the endpoint cannot be turned off)
- `--http-protocol-ipv6 enabled` → `UnsupportedOperation`
- `--instance-metadata-tags enabled` → `UnsupportedOperation`
- Hop limit outside 1–64 → `InvalidParameterValue`

## How It Works

Useful background for host-side troubleshooting:

- The IMDS HTTP server runs inside the `spinifex-vpcd` service on each host, listening on `169.254.169.254:80` with one listener per instance's primary interface — attribution is structural, with no source-IP trust.
- A freshly launched guest's metadata service comes up within a few seconds of launch; cloud-init's built-in retries absorb this window.
- Session tokens and cached role credentials are not persisted. A vpcd restart drops them; SDKs and cloud-init transparently reissue. Datapath state on the host survives service restarts.

## Limits and Defaults

| Setting | Value |
| ------- | ----- |
| Token TTL | 1–21600 seconds (request-scoped, via header) |
| Token binding | Issuing network interface; in-memory, not persisted |
| Role credential lifetime | 3600 seconds, refreshed 5 minutes before expiry |
| PUT response hop limit | Default 1, valid 1–64 |
| `HttpTokens` | `required` (default) or `optional` |
| `HttpEndpoint` | Always `enabled` (immutable) |
| Tokenless GET | `401 Unauthorized`, empty body, unless `HttpTokens=optional` |
| `X-Forwarded-For` present | `403 Forbidden` |
| Wrong method | `405 Method Not Allowed` |
| Unknown/unsupported path | `404 Not Found` |

## Troubleshooting

### 401 Unauthorized on Every Read

The request is missing a valid token. Common causes:

- No `X-aws-ec2-metadata-token` header, on an instance at the default `HttpTokens=required` — obtain a token first, or set `HttpTokens=optional` if the guest agent cannot do the handshake.
- A change to `HttpTokens` can take up to 30 seconds to take effect: the per-instance setting is cached on the untokened path so an unauthenticated caller cannot drive repeated `DescribeInstances` fan-outs.
- The token expired — reissue with a fresh `PUT /latest/api/token`.
- The token was issued to a different instance/interface — tokens are interface-bound and rejected elsewhere, identically to unknown tokens.
- The token endpoint itself never 401s; if the `PUT` fails, check for a `400` (bad TTL header) instead.

### 403 Forbidden

The request carried an `X-Forwarded-For` header. IMDS rejects proxied requests outright; call it directly from the workload, not through a forward proxy.

### Software in a Container Cannot Reach IMDS

The default hop limit of 1 means the PUT response dies at the first routed hop, e.g. a Docker bridge network. Raise it from the control plane:

```bash
aws ec2 modify-instance-metadata-options \
  --instance-id i-0a1b2c3d4e5f67890 \
  --http-put-response-hop-limit 2
```

### UnsupportedOperation When Setting Metadata Options

You attempted to relax IMDSv2 enforcement (`--http-tokens optional`) or disable the endpoint (`--http-endpoint disabled`). Neither is supported — the platform posture is fixed. Only the hop limit can be changed.

### IAM Credentials Return 404, an Empty List, or "Code": "Failed"

A 404 or empty list means the instance has no IAM instance profile, or the profile has no role. Verify from the control plane:

```bash
aws ec2 describe-instances --instance-ids i-0a1b2c3d4e5f67890 \
  --query 'Reservations[0].Instances[0].IamInstanceProfile'
```

A credential body with `"Code": "Failed"` means the backend could not mint credentials (for example, the role was deleted after launch). Check the role still exists and the vpcd logs on the host for `IMDS: AssumeRoleForInstance failed`. See the IAM Roles and Instance Profiles guide for setting up profiles.

### Connection Refused / Timeout to 169.254.169.254

Metadata is served per-interface by `spinifex-vpcd` on the host. In order:

1. Immediately after launch, wait a few seconds — the listener converges within one 15-second reconcile tick, and cloud-init retries through it.
2. On the host, confirm vpcd is running and serving:

```bash
systemctl status spinifex-vpcd
journalctl -u spinifex-vpcd | grep 'IMDS:'
```

Look for `IMDS: tap responder serving` (listener bound) and `IMDS: issued IMDSv2 token` (proof guest packets are traversing the datapath). A bound responder with no token issuance points at the host datapath rather than the HTTP service:

```bash
ovs-vsctl list-ports br-imds          # per-instance endpoint + patch ports
ovs-vsctl list-ports br-int | grep imi-
```

### cloud-init Did Not Apply SSH Key or User Data

cloud-init sources both from IMDS at first boot. Check `public-keys/0/openssh-key` and `user-data` respond from inside the guest (see [Using IMDS from a Guest](#using-imds-from-a-guest)), and review `/var/log/cloud-init.log` in the guest. A 404 on `public-keys/` means the instance was launched without `--key-name` or the key pair was since deleted.

---

# EKS (Managed Kubernetes)

URL: https://docs.mulgadc.com/docs/eks
Category: Containers
Updated: 2026-09-14
Tags: eks, kubernetes, containers, vpc, iam

Provision an AWS-compatible EKS control plane and managed node group on Spinifex, wire up the VPC, IAM, and security groups, then deploy your first workload.

## Overview

An EKS cluster on Spinifex has two parts: a **control plane** (the managed Kubernetes API server, provisioned on a Spinifex-managed VM and fronted by a network load balancer) and one or more **managed node groups** of worker VMs that run your pods. You provide the VPC, subnets, and IAM roles; Spinifex creates and reconciles the cluster, its load balancer, and its security groups.

Authentication uses **EKS access entries** (the API authentication mode) — IAM principals are mapped to Kubernetes RBAC groups, and `kubectl` authenticates with `aws eks get-token` exactly as it does on AWS.

<p align="center">
  <img height="470" src="/docs-assets/containers-eks-eks-topology.svg" alt="EKS on Spinifex — a managed control plane (public API NLB plus apiserver VM) fronts a managed node group of eks-node worker VMs running pods across two subnets, pulling images from ECR" width="900">
</p>

**What you'll create:**

| Resource | Purpose |
|---|---|
| VPC + subnets | The network the cluster and workers run in |
| Internet Gateway / NAT Gateway | Egress so workers can pull container images |
| Cluster IAM role | Lets the control plane manage cluster resources |
| Node IAM role | Lets workers join the cluster and pull images |
| EKS cluster | The managed Kubernetes API endpoint |
| Managed node group | The worker VMs that run your pods |
| Security-group rule | Opens your app's port to reach the workers |

**Spinifex specifics**

- **Auth mode is API-only.** `authentication_mode` must be `API`; the legacy `CONFIG_MAP` and `API_AND_CONFIG_MAP` modes are rejected. Grant access with access entries, not an `aws-auth` ConfigMap.
- **Security groups are auto-managed.** Spinifex creates the cluster and node-group security groups deterministically (e.g. `eks-cluster-<name>-nodegroup-sg`); `vpc_config.security_group_ids` is ignored. To expose a workload you add an ingress rule to the auto-managed node-group SG.
- **The worker image is fixed.** Node groups always boot Spinifex's `eks-node` image. `ami_type` is recorded but does not select the image, and the image must be registered on the cluster before you create a node group.
- **Default Kubernetes version is `1.32`.**

## Prerequisites

> [!IMPORTANT]
> **Prerequisite — `eks-node` image required.**
>
> Node groups always boot Spinifex's **`eks-node`** image, and it **must** be registered before you create a cluster — the console blocks cluster creation until it is present. Import it during `spx admin init` (or via the image catalogue) ahead of time.
>
> **Verify before continuing:**
>
> ```bash
> aws ec2 describe-images \
>   --filters 'Name=tag:spinifex:managed-by,Values=eks' \
>   --query 'Images[].[ImageId,Name]' --output text
> ```
>
> No rows means the image is not imported — register it before continuing.

Before creating a cluster you need the supporting infrastructure in place. The Terraform workbooks build all of this for you; if you are using the CLI or console, create it first.

- **Spinifex running**, with the AWS CLI configured for the `spinifex` profile (see [Installing Spinifex](https://docs.mulgadc.com/docs/install)) and `kubectl` installed.
- **The `eks-node` image** registered on the cluster. The console blocks cluster creation until it is present.
- **A VPC with at least one subnet.** The control plane launches in your first subnet; workers run in the subnets you pass to the node group.
- **Egress for the workers.** Workers must reach the internet to pull container images.
- **Public workers** need an **Internet Gateway** with a default route (`0.0.0.0/0 → IGW`) and public IPs.
- **Private workers** need a **NAT Gateway** in a public subnet with a default route (`0.0.0.0/0 → NAT`) from the private subnets.
- **A cluster role** trusted by `eks.amazonaws.com` with `AmazonEKSClusterPolicy` attached.
- **A node role** trusted by `ec2.amazonaws.com` with `AmazonEKSWorkerNodePolicy`, `AmazonEKS_CNI_Policy`, and `AmazonEC2ContainerRegistryReadOnly` attached (the last lets workers pull images from [ECR](https://docs.mulgadc.com/docs/ecr)).

## Instructions

The same cluster can be created three ways. Pick your tool — each path reaches the same working cluster.

:::tabs
@tab AWS CLI

### 1. Create the cluster

Point the cluster at your VPC subnets and the cluster role, with API authentication and a public endpoint:

```bash
export AWS_PROFILE=spinifex

aws eks create-cluster \
  --name demo \
  --role-arn arn:aws:iam::000000000000:role/eks-cluster-role \
  --resources-vpc-config subnetIds=subnet-aaaa,subnet-bbbb,endpointPublicAccess=true \
  --access-config authenticationMode=API \
  --kubernetes-version 1.32
```

Wait for it to become active:

```bash
aws eks wait cluster-active --name demo
aws eks describe-cluster --name demo --query 'cluster.status'
```

### 2. Add a node group

```bash
aws eks create-nodegroup \
  --cluster-name demo \
  --nodegroup-name default \
  --node-role-arn arn:aws:iam::000000000000:role/eks-node-role \
  --subnets subnet-aaaa subnet-bbbb \
  --scaling-config minSize=1,maxSize=2,desiredSize=1 \
  --instance-types t3.medium

aws eks wait nodegroup-active --cluster-name demo --nodegroup-name default
```

### 3. Connect kubectl

```bash
aws eks update-kubeconfig --name demo
kubectl get nodes
```

### 4. Expose a workload

Workers join an auto-managed security group that only allows intra-cluster traffic. To reach a NodePort service, add one ingress rule to that SG (look it up by its deterministic name):

```bash
SG=$(aws ec2 describe-security-groups \
  --filters Name=group-name,Values=eks-cluster-demo-nodegroup-sg \
  --query 'SecurityGroups[0].GroupId' --output text)

aws ec2 authorize-security-group-ingress \
  --group-id "$SG" --protocol tcp --port 30080 --cidr 0.0.0.0/0
```

Then deploy and publish your app:

```bash
kubectl create deployment hello --image=nginxdemos/hello --replicas=2
kubectl expose deployment hello --type=NodePort --port=80 \
  --overrides='{"spec":{"ports":[{"port":80,"nodePort":30080}]}}'
```

@tab Spinifex UI

The Spinifex console drives the same workflow through a guided form. From the left navigation open **EKS → Clusters**.

<img width="2880" height="1800" src="/docs-assets/containers-eks-eks-list-clusters.png" alt="The EKS Clusters list in the Spinifex console, with a Create Cluster button">

### 1. Create the cluster

1. Click **Create Cluster**.
2. Enter a **name** and choose the **Kubernetes version** (default `1.32`).
3. Choose the **cluster IAM role**, or create one inline from the dialog.
4. Select the **VPC and subnets** the cluster will run in.
5. Set **endpoint access** (public, private, or both) and submit.

<img width="2880" height="1800" src="/docs-assets/containers-eks-eks-create-cluster.png" alt="The Create Cluster form in the Spinifex console, showing name, Kubernetes version, IAM role, VPC, subnets, and endpoint access fields">

The form notes that the authentication mode is API and that the legacy aws-auth ConfigMap is not supported. The cluster appears in the **Clusters** list as `CREATING` and flips to `ACTIVE` once the control plane is healthy.

### 2. Add a node group

1. Open the cluster and switch to the **Node groups** tab.
2. Click **Add node group**, choose the **node IAM role**, **subnets**, **instance type**, and **scaling** (desired / min / max).
3. Submit and wait for the node group to reach `ACTIVE`.

### 3. Manage access and addons

- The **Access** tab lists IAM principals and their cluster-access policies — add an access entry to let another user or role reach the cluster.
- The **Addons** tab installs and shows the health of managed addons.
- The **Networking** tab shows the cluster's VPC, subnets, endpoint access, and the auto-managed security groups.

To run `kubectl` against the cluster, follow the AWS CLI tab's `update-kubeconfig` step.

@tab Terraform

The Terraform workbooks are the fastest way to a working cluster — they build the VPC, IAM roles, cluster, node group, and a demo workload in one `apply`. Start with **eks-quickstart**:

```bash
git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
cd spinifex-tf
git sparse-checkout set docs/terraform-workbooks
cd docs/terraform-workbooks/eks-quickstart
```

The cluster config points the AWS provider at Spinifex's `eks`, `ec2`, `iam`, and `sts` endpoints, then creates the cluster and node group:

```hcl
resource "aws_eks_cluster" "this" {
  name     = "eks-quickstart"
  role_arn = aws_iam_role.cluster.arn
  version  = "1.32"

  access_config {
    authentication_mode = "API"
  }

  vpc_config {
    subnet_ids              = [aws_subnet.a.id, aws_subnet.b.id]
    endpoint_public_access  = true
  }
}

resource "aws_eks_node_group" "default" {
  cluster_name    = aws_eks_cluster.this.name
  node_group_name = "default"
  node_role_arn   = aws_iam_role.node.arn
  subnet_ids      = [aws_subnet.a.id, aws_subnet.b.id]

  scaling_config {
    desired_size = 1
    min_size     = 1
    max_size     = 2
  }

  instance_types = ["t3.medium"]
}
```

Apply it, then connect:

```bash
export AWS_PROFILE=spinifex
tofu init
tofu apply
aws eks update-kubeconfig --name eks-quickstart
kubectl get nodes
```

The workbooks form a ladder. **eks-quickstart** is a NodePort demo; **eks-https-ingress** adds private subnets, a NAT gateway, and HTTPS via the AWS Load Balancer Controller + ACM; **eks-gitops-argocd** delivers the app from a git repo with the Argo CD addon and persists state on an EBS-CSI (Viperblock-backed) volume. See the full templates linked in Resources.

:::

## Troubleshooting

**Cluster stuck in `CREATING`.** The control plane is still booting. Check `aws eks describe-cluster --name <name> --query 'cluster.health'` for reported issues. The control plane needs egress to bootstrap, so confirm its subnet has an Internet Gateway route.

**`kubectl` cannot connect.** Re-run `aws eks update-kubeconfig --name <name>` to refresh the endpoint and CA, and confirm the cluster is `ACTIVE`. `aws sts get-caller-identity` verifies your credentials reach the Spinifex endpoint.

**Workers stay `NotReady`.** Almost always an egress or IAM problem:
- Confirm the workers can pull images — a default route to an Internet Gateway (public) or NAT Gateway (private).
- Confirm the node role has `AmazonEKSWorkerNodePolicy`, `AmazonEKS_CNI_Policy`, and `AmazonEC2ContainerRegistryReadOnly`.

**Workload unreachable from outside the cluster.** The node-group security group only allows intra-cluster traffic by default. Add an ingress rule for your NodePort to `eks-cluster-<name>-nodegroup-sg`, as shown in the CLI tab.

**`create-cluster` rejected with an authentication-mode error.** `authentication_mode` must be `API`. Remove any `CONFIG_MAP` / `API_AND_CONFIG_MAP` setting and grant access with access entries instead.

**Addon install fails.** Only addons bundled into the registered `eks-node` image install successfully. If you pin an addon version in Terraform, leave `addon_version` unset so Spinifex selects the catalog default.

## Reference Architectures

### EKS AI Platform on Bare Metal: llama.cpp + YOLO on RTX Pro 6000

A complete end-to-end example of Spinifex EKS in production: a GPU-accelerated AI inference platform running on a Supermicro X14 2U with two NVIDIA RTX Pro 6000 Blackwell GPUs. The guide provisions an EKS cluster with two GPU worker nodes via VFIO PCIe passthrough, deploys an OpenAI-compatible LLM API (llama.cpp / Llama 3.2 3B) and a real-time YOLO object-detection stream on separate GPU nodes, and wires them together behind an ALB using IAM, ECR, ACM, and EBS — all with standard AWS tooling and a single `AWS_PROFILE` swap.

→ [EKS AI Platform Reference Architecture](https://docs.mulgadc.com/hardware/supermicro/rtx-pro-6000)

---

# ECS (Elastic Container Service)

URL: https://docs.mulgadc.com/docs/ecs
Category: Containers
Updated: 2026-08-19
Tags: ecs, containers, tasks, alb, iam

Create an ECS cluster on Spinifex, register a task definition, boot container instances, run tasks, and front a service with an Application Load Balancer.

## Overview

ECS on Spinifex follows the AWS **EC2 launch type**: you supply the compute. A cluster is a logical grouping; the capacity behind it is **container instances** — ordinary EC2 instances booted from Spinifex's `spinifex-ecs-node` image, each running the Spinifex ECS agent. The agent registers the instance with the cluster, reports its CPU/memory, and runs the containers the scheduler places on it.

There is **no ECS-specific launch API** — this is AWS-faithful. You add capacity by launching EC2 instances from the ECS node image (with the `ecsInstanceRole` instance profile), exactly as on AWS. The Spinifex console wraps this in a one-click **Provision capacity** action, but underneath it is just `RunInstances`.

A **task definition** describes one or more containers (image, CPU/memory, ports, environment, and an optional task IAM role). A **task** is a running instantiation of a task definition; the scheduler bin-packs tasks onto instances with free capacity. A **service** keeps a desired number of tasks running, replaces failed ones, and — when configured — registers each task's IP with an Application Load Balancer target group.

<p align="center">
  <img height="470" src="/docs-assets/containers-ecs-ecs-topology.svg" alt="ECS on Spinifex — an Application Load Balancer fronts awsvpc tasks running on container instances booted from the spinifex-ecs-node AMI; the ECS agent registers with the Spinifex gateway over TLS and SigV4 and serves task-role credentials to containers" width="900">
</p>

**What you'll create:**

| Resource | Purpose |
|---|---|
| VPC + subnets | The network the cluster and tasks run in |
| Internet Gateway | Egress so instances can pull container images |
| `ecsInstanceRole` | Instance profile letting the agent reach the control plane |
| ECS cluster | The logical grouping tasks and instances join |
| Task definition | The container spec (image, CPU/memory, ports, task role) |
| Container instance(s) | EC2 VMs from the ECS node image that run tasks |
| Service + ALB target group | Keeps N tasks running and load-balances them |

**Spinifex specifics**

- **EC2 launch type only.** There is no Fargate-equivalent serverless capacity; you run and pay for the container instances. `RequiresCompatibilities` of `FARGATE` is not honoured.
- **`awsvpc` network mode.** Each task gets its own ENI and private IP in your subnet. Target groups must use `target_type = "ip"`; the service's `network_configuration` selects the subnets and security groups.
- **The container-instance image is fixed.** Instances must boot Spinifex's `spinifex-ecs-node` image (resolve it by the `spinifex:managed-by=ecs` tag) and carry the `ecsInstanceRole` instance profile.
- **Task IAM roles via the credential endpoint.** A task with a `taskRoleArn` gets `AWS_CONTAINER_CREDENTIALS_RELATIVE_URI` injected; the agent serves short-lived credentials for that role at `169.254.170.2`, AWS-faithful — your container's SDK picks them up with no static keys.

### Limitations

ECS v1 is deliberately minimal. Keep these in mind:

- **No service discovery.** `serviceRegistries` / Cloud Map integration is not implemented — reach a service through its load balancer, not a DNS name.
- **Only the `json-file` log driver is collected.** Container stdout/stderr is captured host-side on the container instance (see [Logging](#logging)); it is not shipped to CloudWatch Logs. Any other driver (e.g. `awslogs`) is accepted for parity but its logs are discarded — `RegisterTaskDefinition` logs a warning naming the container so the drop is not silent.
- **No capacity providers or managed scaling.** Capacity is the static total of your registered instances; there is no scale-out/in or ASG binding — you manage the instance count.
- **`secrets[]` are rejected.** A task definition that declares container `secrets[]` fails `RegisterTaskDefinition` with `InvalidParameterException` rather than running without the secrets it expects. Tags set via `TagResource` are not persisted, and health-check settings beyond the target group default are not forwarded.

### Logging

Spinifex honours the host-side **`json-file`** log driver (the containerd default). A container's stdout/stderr is written on its container instance and retrievable there — no CloudWatch Logs path exists.

To read a task's logs, find the container instance running it (`aws ecs describe-tasks` → `containerInstanceArn` → the EC2 instance), then on that host inspect the containerd task output. Containers are named `{taskId}-{containerName}` and carry `mulga.ecs.*` labels:

```bash
# On the container instance:
ctr -n default containers ls                 # find {taskId}-{containerName}
ctr -n default tasks ls
journalctl -u containerd | grep <taskId>     # container stdout/stderr via the host journal
```

A task definition may still declare `awslogs` (or any other driver) for AWS compatibility — Spinifex accepts it, warns at registration that the driver is not implemented, and falls back to the host-side `json-file` behaviour.

### Deployments

An `UpdateService` to a new task-definition revision performs a **health-gated rolling update** honouring `deploymentConfiguration`: `minimumHealthyPercent` keeps that fraction of the desired count running while `maximumPercent` bounds how many extra tasks launch during the roll. Enabling the **deployment circuit breaker** fails a rollout whose tasks repeatedly fail to start, and — with `rollback` set — automatically reverts to the last-good task definition.

### Execution role

If a task definition sets `executionRoleArn`, the agent assumes that role to authorise ECR image pulls (instead of the container-instance role). When it is unset, pulls fall back to the instance role.

## Prerequisites

> [!IMPORTANT]
> **Prerequisite — `spinifex-ecs-node` image required.**
>
> Container instances **must** boot Spinifex's **`spinifex-ecs-node`** image — it carries the ECS agent. Import it before provisioning capacity (during `spx admin init` or via the image catalogue); the console's provision-capacity action and the Terraform workbook both resolve it by tag.
>
> **Verify before continuing:**
>
> ```bash
> aws ec2 describe-images \
>   --filters 'Name=tag:spinifex:managed-by,Values=ecs' \
>   --query 'Images[].[ImageId,Name]' --output text
> ```
>
> No rows means the image is not imported — register it before continuing.

The Terraform workbook builds all of this for you. For the CLI or console paths, have it in place first.

- **Spinifex running**, with the AWS CLI configured for the `spinifex` profile (see [Installing Spinifex](https://docs.mulgadc.com/docs/install)).
- **The `spinifex-ecs-node` image** imported. Confirm it resolves:

  ```bash
  aws ec2 describe-images \
    --filters 'Name=tag:spinifex:managed-by,Values=ecs' \
    --query 'Images[].[ImageId,Name]' --output text
  ```

- **A VPC with at least one subnet** and an **Internet Gateway** with a default route (`0.0.0.0/0 → IGW`), so instances can pull container images.
- **The `ecsInstanceRole` instance profile.** It is account-global: a role trusted by `ec2.amazonaws.com` with an `ecs:*` policy, exposed through an instance profile of the same name. The console's provision-capacity action creates it on first use; the Terraform workbook creates it unless you opt out.
- **An optional task IAM role** trusted by `ecs-tasks.amazonaws.com`, if your containers call AWS APIs.

## Instructions

The same workload can be created three ways. Pick your tool — each path reaches the same running service.

:::tabs
@tab AWS CLI

### 1. Create the cluster

```bash
export AWS_PROFILE=spinifex

aws ecs create-cluster --cluster-name demo
```

### 2. Register a task definition

`awsvpc` network mode, EC2 launch type, one nginx container on port 80:

```bash
aws ecs register-task-definition \
  --family web \
  --network-mode awsvpc \
  --requires-compatibilities EC2 \
  --cpu 256 --memory 512 \
  --container-definitions '[{
    "name":"web",
    "image":"docker.io/library/nginx:1.27-alpine",
    "portMappings":[{"containerPort":80,"protocol":"tcp"}],
    "essential":true
  }]'
```

### 3. Add capacity

Launch one or more EC2 instances from the ECS node image with the `ecsInstanceRole` instance profile and a cloud-init that points the agent at the cluster. The [ECS Quickstart workbook](https://github.com/mulgadc/spinifex/tree/main/docs/terraform-workbooks/ecs-quickstart) produces this user-data for you; the console **Provision capacity** action is the one-click equivalent. Once an instance boots and its agent registers, it appears here:

```bash
aws ecs list-container-instances --cluster demo
```

### 4. Run a task or create a service

Run a one-off task:

```bash
aws ecs run-task \
  --cluster demo --task-definition web --count 1 \
  --network-configuration 'awsvpcConfiguration={subnets=[subnet-aaaa]}'
```

Or keep it running behind an ALB target group (`target_type = ip`):

```bash
aws ecs create-service \
  --cluster demo --service-name web --task-definition web \
  --desired-count 2 \
  --network-configuration 'awsvpcConfiguration={subnets=[subnet-aaaa]}' \
  --load-balancers 'targetGroupArn=<tg-arn>,containerName=web,containerPort=80'

aws ecs describe-services --cluster demo --services web \
  --query 'services[0].[runningCount,desiredCount]'
```

@tab Spinifex UI

The Spinifex console drives the same workflow. From the left navigation open **ECS → Clusters**.

### 1. Create the cluster

Click **Create Cluster**, give it a name, and submit. It appears in the **Clusters** list.

### 2. Provision capacity

Open the cluster and use **Provision capacity** on the **Infrastructure** (container instances) tab. The console launches instances from the `spinifex-ecs-node` image, attaches the `ecsInstanceRole` instance profile (creating it if needed), and injects the gateway URL and CA into the instance's cloud-init — so the agent registers without any manual key handling. The instances appear under **Infrastructure** as they register.

### 3. Register a task definition and run it

- The **Task definitions** view lists families and revisions; register a new revision with the container image, CPU/memory, and ports.
- From a task definition, **Run task** for a one-off, or create a **Service** to keep a desired count running.
- A cluster's **Services** and **Tasks** tabs show what is running; open a task or service for its detail page (containers, networking, and public/private endpoints).

### 4. Attach a load balancer

When creating a service, select an ALB target group (`target_type = ip`) so each task's ENI is registered and traffic is balanced across tasks.

@tab Terraform

The [ECS Quickstart workbook](https://github.com/mulgadc/spinifex/tree/main/docs/terraform-workbooks/ecs-quickstart) builds the whole stack in one `apply`: VPC, IAM, cluster, task definition, container instances, and an ALB-fronted service. It is the Terraform-native equivalent of the console's provision-capacity action.

```bash
git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
cd spinifex-tf
git sparse-checkout set docs/terraform-workbooks
cd docs/terraform-workbooks/ecs-quickstart
```

The core resources point the AWS provider at Spinifex's `ecs`, `ec2`, `iam`, `sts`, and `elasticloadbalancingv2` endpoints, then create the cluster, an `awsvpc` task definition with a task role, and a service wired to a target group:

```hcl
resource "aws_ecs_cluster" "main" {
  name = "ecs-quickstart"
}

resource "aws_ecs_task_definition" "web" {
  family                   = "ecs-quickstart-web"
  network_mode             = "awsvpc"
  requires_compatibilities = ["EC2"]
  cpu                      = "256"
  memory                   = "512"
  task_role_arn            = aws_iam_role.task.arn

  container_definitions = jsonencode([{
    name         = "web"
    image        = "docker.io/library/nginx:1.27-alpine"
    essential    = true
    portMappings = [{ containerPort = 80, protocol = "tcp" }]
  }])
}

resource "aws_ecs_service" "web" {
  name            = "ecs-quickstart-web"
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.web.arn
  desired_count   = 1

  network_configuration {
    subnets         = [aws_subnet.a.id, aws_subnet.b.id]
    security_groups = [aws_security_group.node.id]
  }

  load_balancer {
    target_group_arn = aws_lb_target_group.web.arn
    container_name   = "web"
    container_port   = 80
  }
}
```

Because a container instance reaches the control plane over the gateway (TLS + SigV4) rather than a managed AWS endpoint, the workbook makes the two things the console injects explicit: a **LAN-reachable** `gateway_url` (the host's bridge IP, not `127.0.0.1`) and the gateway CA, both baked into the instance's cloud-init. Apply it:

```bash
export AWS_PROFILE=spinifex
tofu init
tofu apply -var 'gateway_url=https://<host-lan-ip>:9999'
```

`ecsInstanceRole` is account-global — if it already exists (from the console), pass `-var 'create_instance_role=false'`. See the workbook README for the full variable list and teardown.

:::

## Troubleshooting

**No container instances appear after launching them.** The agent registers over the gateway, not a managed endpoint, so check its cloud-init injected a correct **LAN-reachable** gateway URL (not `127.0.0.1` — a guest VM cannot reach the host loopback) and the gateway CA. `cloud-init write_files` runs once per instance, so fixing the user-data needs an instance **replacement**, not an in-place modify. Confirm the instance carries the `ecsInstanceRole` instance profile.

**Tasks stay `PENDING`.** No instance has free capacity for the task's CPU/memory reservation. Add capacity, or lower the task definition's reservations. `aws ecs list-container-instances` should show at least one `ACTIVE` instance.

**Service `runningCount` below `desiredCount`.** Either there is not enough capacity (see above), or tasks are failing to start — check the task's containers can pull their image (instances need an egress route) and that the image reference is valid.

**Load balancer target unhealthy / app unreachable.** The target group must be `target_type = ip` (tasks register their ENI IP, not an instance ID). The ALB DNS name ends in `.elb.spinifex.local` and does not resolve from outside; fetch its public IP with `aws elbv2 describe-load-balancers` and curl that. Note health-check settings beyond the target group default are not forwarded.

**Container cannot assume its task role.** Confirm the task definition sets `taskRoleArn` and the role is trusted by `ecs-tasks.amazonaws.com`. The agent injects `AWS_CONTAINER_CREDENTIALS_RELATIVE_URI`; a container that overrides this variable, or an SDK too old to honour it, will not pick up the credentials.

**Expected a feature that is missing.** ECS v1 omits service discovery, CloudWatch Logs (`awslogs`) shipping, and capacity providers / managed scaling. See the Limitations in the Overview.

---

# ECR (Container Registry)

URL: https://docs.mulgadc.com/docs/ecr
Category: Containers
Updated: 2026-08-19
Tags: ecr, containers, docker, registry, oci

Store and serve container images from Spinifex's AWS-compatible ECR: create a repository, authenticate Docker, push and pull images, and let EKS workers pull.

## Overview

ECR on Spinifex serves a private container registry on the Spinifex gateway endpoint — the same `host:9999` you already use for the AWS API. Authentication is account-scoped by token, so you push and pull at the gateway host and never need extra DNS:

```
<spinifex-gateway-host>:9999/<repository>
```

`aws ecr describe-repositories` and `aws ecr get-login-password` return the exact host to use — always prefer that value. Where real DNS is configured, the AWS-parity per-account host `<account-id>.dkr.ecr.<region>.<your-spinifex-domain>:9999` also works.

Each repository holds image manifests and their layers. Images are stored in object storage ([Predastore](https://github.com/mulgadc/predastore)) in a per-account bucket, with blobs content-addressed and de-duplicated across repositories. Authentication is token-based: `aws ecr get-login-password` mints a short-lived bearer token that `docker login` uses against the registry's `/v2/` endpoint.

**What works today**

- Repository lifecycle — `CreateRepository`, `DeleteRepository`, `DescribeRepositories`.
- OCI push and pull — blobs, manifests, and tags, including cross-repository blob mounts.
- Token auth — `GetAuthorizationToken` plus bearer/basic auth on `/v2/`.
- Repository policies — set, get, and delete a repository policy document.

**Current limitations**

- **Image scanning is not supported** — scan APIs return an explicit "operation not supported" error.
- **No automatic garbage collection yet** — deleting images does not yet reclaim underlying blobs.
- Lifecycle-policy enforcement and tag-immutability enforcement are not active.

## Prerequisites

- **Spinifex running**, with the AWS CLI configured for the `spinifex` profile (see [Installing Spinifex](https://docs.mulgadc.com/docs/install)).
- **Docker** installed locally to build, push, and pull images.
- **Docker trusting the Spinifex CA.** The registry serves TLS signed by the Spinifex local CA. Install it into the host trust store once (Docker uses the host root pool for registry TLS, so this covers every account endpoint without per-host config):

```bash
sudo cp /etc/spinifex/ca.pem /usr/local/share/ca-certificates/spinifex-local-ca.crt
sudo update-ca-certificates
sudo systemctl restart docker
```

- **The registry endpoint** — the Spinifex gateway `host:9999`. `aws ecr get-login-password` and `aws ecr describe-repositories` return the exact `host:port`; copy it from there rather than constructing it by hand.
- **For EKS pulls:** worker nodes need the `AmazonEC2ContainerRegistryReadOnly` policy on their node IAM role (the EKS prerequisites already include this) and network egress to the registry endpoint.

## Instructions

Create a repository and push an image using your preferred tool.

:::tabs
@tab AWS CLI

### 1. Create a repository

```bash
export AWS_PROFILE=spinifex

aws ecr create-repository --repository-name my-app
aws ecr describe-repositories --repository-names my-app \
  --query 'repositories[0].repositoryUri'
```

The `repositoryUri` is the value you tag and push to — the gateway `host:9999` with the repository path, port included (for example `192.0.2.10:9999/my-app`). Use it verbatim; don't hand-build a hostname.

### 2. Authenticate Docker

Take the registry host straight from the API so the `:9999` port is correct:

```bash
REGISTRY=$(aws ecr describe-repositories --repository-names my-app \
  --query 'repositories[0].repositoryUri' --output text | cut -d/ -f1)

aws ecr get-login-password --region <region> \
  | docker login --username AWS --password-stdin "$REGISTRY"
```

### 3. Build, tag, and push

```bash
docker build -t my-app:latest .
docker tag my-app:latest "$REGISTRY/my-app:latest"
docker push "$REGISTRY/my-app:latest"
```

### 4. Verify and pull

```bash
aws ecr describe-images --repository-name my-app
docker pull "$REGISTRY/my-app:latest"
```

@tab Spinifex UI

From the left navigation open **ECR → Repositories**.

<img width="2880" height="1800" src="/docs-assets/containers-ecr-ecr-list-repositories.png" alt="The ECR Repositories list in the Spinifex console, showing a repository with its registry URI, tag mutability, and creation date">

### 1. Create a repository

1. Click **Create Repository**.
2. Enter the **repository name** and, optionally, set **tag mutability** (mutable or immutable).
3. Submit — the repository appears in the list with its full registry URI.

### 2. Copy the push commands

Open the repository to see its detail page. The **Push commands** panel lists the exact `aws ecr get-login-password | docker login`, `docker build`, `docker tag`, and `docker push` commands, pre-filled with your registry host and repository URI — use the **Copy** button and run them locally.

<img width="2880" height="1800" src="/docs-assets/containers-ecr-ecr-push-commands.png" alt="A repository detail page in the Spinifex console, with the Push commands panel showing the docker login, build, tag, and push commands">

### 3. Manage the repository

- The **Permissions** tab edits the repository's access policy.
- The **Lifecycle** tab manages retention rules, and **Tag immutability** can be toggled from the detail page.
- The **Scan** tab reflects that image scanning is not supported on Spinifex.
- Delete a repository from the **Delete** action in the repositories list.

@tab Terraform

Repositories can be managed as code with the standard `aws_ecr_repository` resource, with the AWS provider pointed at Spinifex's `ecr` endpoint:

```hcl
provider "aws" {
  region                      = "ap-southeast-2"
  skip_credentials_validation = true
  skip_requesting_account_id  = true

  endpoints {
    ecr = "https://<your-spinifex-gateway>"
  }
}

resource "aws_ecr_repository" "my_app" {
  name = "my-app"
}

output "repository_url" {
  value = aws_ecr_repository.my_app.repository_url
}
```

```bash
export AWS_PROFILE=spinifex
tofu init
tofu apply
```

Then authenticate Docker and push using the `repository_url` output, following steps 2–4 in the AWS CLI tab.

:::

## Troubleshooting

**`docker login` fails with an authorization error.** The token is account-scoped and short-lived. Re-run `aws ecr get-login-password` to mint a fresh token, and confirm the registry host in `docker login` matches your account's endpoint (`aws ecr describe-repositories` returns it).

**`docker push` cannot reach the registry.** Use the exact host from `aws ecr describe-repositories` — the Spinifex gateway `host:9999`. Docker dials it directly, so the `:9999` port must be present (without it docker tries `:443` and fails). If you are using the AWS-parity `<account-id>.dkr.ecr…` hostname instead, that path needs DNS resolving it to the gateway — prefer the gateway host the API returns.

**EKS workers cannot pull an image.** Confirm the node IAM role has `AmazonEC2ContainerRegistryReadOnly` and that the workers have egress to the registry endpoint (an Internet Gateway or NAT Gateway route). See the [EKS prerequisites](https://docs.mulgadc.com/docs/eks).

**Image-scanning commands return an error.** Image scanning is not supported on Spinifex; the scan APIs intentionally reject these calls. Remove scan-on-push configuration from your tooling.

**A pushed image still appears after deletion frees no space.** Garbage collection of unreferenced blobs is not yet automatic — deleting an image removes its manifest and tags but does not immediately reclaim the underlying layers.

---

# Moving an AWS Workload to Mulga

URL: https://docs.mulgadc.com/docs/moving-aws-workload
Category: Migration
Updated: 2026-08-19
Tags: migration, aws, terraform

Move existing AWS workloads onto Spinifex using compatible APIs, SDKs, and Terraform across EC2, VPC, EBS, S3, IAM, STS, ELBv2, ACM, ECR, ECS, and EKS.

## Overview

Spinifex provides drop-in compatibility with AWS APIs, making it possible to migrate existing workloads with minimal changes.

**Supported Services:** EC2, VPC, EBS, S3, IAM, STS, ELBv2 (ALB/NLB), ACM, ECR, ECS, EKS

**Compatible Tools:** AWS CLI, AWS SDKs, Terraform, kubectl (via `aws eks get-token`), any S3-compatible client

## Prerequisites

- A running Spinifex cluster (see [Setting Up Your Cluster](https://docs.mulgadc.com/docs/setting-up-your-cluster))
- AWS CLI configured with the `spinifex` profile:

```bash
export AWS_PROFILE=spinifex
```

## Configure AWS CLI

```bash
aws ec2 describe-instances
aws s3 ls
```

## Terraform

```hcl
provider "aws" {
  region     = "ap-southeast-2"
  access_key = "your-spinifex-access-key"
  secret_key = "your-spinifex-secret-key"

  endpoints {
    ec2                = "https://localhost:9999"
    iam                = "https://localhost:9999"
    sts                = "https://localhost:9999"
    elbv2              = "https://localhost:9999"
    acm                = "https://localhost:9999"
    ecr                = "https://localhost:9999"
    ecs                = "https://localhost:9999"
    eks                = "https://localhost:9999"
    s3                 = "https://localhost:8443"
  }

  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true
  skip_region_validation      = true
}
```

## Hybrid S3 Sync

```bash
aws s3 sync s3://local-bucket/ s3://cloud-bucket/ --source-region spinifex --region us-east-1
```

## Troubleshooting

## Terraform Provider Errors

Ensure all four skip flags are set in your provider configuration:

```hcl
skip_credentials_validation = true
skip_metadata_api_check     = true
skip_requesting_account_id  = true
skip_region_validation      = true
```

Without these, Terraform will try to validate credentials and metadata against real AWS endpoints.

## S3 Signature Errors

Spinifex uses AWS Signature V4. Ensure your AWS CLI is version 2.0 or higher:

```bash
aws --version
```

If using an older version, upgrade:

```bash
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install --update
```

---

# Hybrid Sync

URL: https://docs.mulgadc.com/docs/hybrid-sync
Category: Migration
Updated: 2026-08-19
Tags: hybrid, sync, s3

Synchronise data bidirectionally between Spinifex and AWS when connectivity allows, so local infrastructure stays usable at intermittently connected sites.

## Overview

Spinifex's hybrid mode enables bidirectional data synchronization between local infrastructure and AWS cloud services. Ideal for intermittent connectivity environments.

## Prerequisites

- A running Spinifex cluster (see [Setting Up Your Cluster](https://docs.mulgadc.com/docs/setting-up-your-cluster))
- AWS CLI configured with the `spinifex` profile:

```bash
export AWS_PROFILE=spinifex
```

## Push Local to Cloud

```bash
aws s3 sync s3://local-bucket/ s3://cloud-bucket/ \
  --source-region spinifex --region us-east-1
```

## Pull Cloud to Local

```bash
aws s3 sync s3://cloud-bucket/ s3://local-bucket/ \
  --source-region us-east-1 --region spinifex
```

## EBS Volume Backup

```bash
rsync -avz /data/ user@cloud-server:/backup/spinifex-data/
```

## Troubleshooting

## Sync Fails Mid Transfer

S3 sync is idempotent — re-run the same command to resume where it left off:

```bash
aws s3 sync s3://local-bucket/ s3://cloud-bucket/ \
  --source-region spinifex --region us-east-1
```

Only changed or missing files will be transferred on subsequent runs.

---

# Bastion with Private Subnet

URL: https://docs.mulgadc.com/docs/bastion-private-subnet
Category: Terraform Workbooks
Updated: 2026-08-19
Tags: terraform, bastion, vpc, security, private subnet, workbook

Deploy a VPC with public and private subnets on Spinifex, then use a bastion host as the only route to an isolated EC2 instance that has no internet access.

## Overview

Deploy a VPC with public and private subnets where the private subnet has no internet access. A bastion host in the public subnet is the only way to reach instances in the private subnet. This pattern is used for sensitive workloads that must remain isolated from the internet — the private instance cannot make or receive any connections outside the VPC.

**Architecture:**

<p align="center">
  <img height="180" src="/docs-assets/terraform-workbooks-bastion-private-subnet-tf-bastion.svg" alt="Bastion architecture — WAN reaches bastion in public subnet, SSH hop to app server in private subnet" width="900">
</p>

**What you'll learn:**

- Creating public and private subnets in a VPC
- Isolating compute instances from the internet using route tables and security groups
- Using a bastion host as the sole access point to private resources
- SSH hopping through a bastion to reach private instances

**Use cases:**

- Sensitive data processing that must not have internet egress
- Internal services that should only be reachable within the VPC
- Compliance workloads requiring network isolation

**Prerequisites:**

- Spinifex installed and running (see [Installing Spinifex](https://docs.mulgadc.com/docs/install))
- An Ubuntu 26.04 AMI imported (see [Setting Up Your Cluster](https://docs.mulgadc.com/docs/setting-up-your-cluster))
- OpenTofu or Terraform installed

## Instructions

### Step 1. Get the Template

Clone the Terraform examples from the Spinifex repository:

```bash
git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
cd spinifex-tf
git sparse-checkout set docs/terraform
cd docs/terraform/bastion-private-subnet
```

Or create a `main.tf` file and paste the full configuration below.

```hcl
# Example 2: Bastion Host with Private Subnet
#
# Deploys a VPC with both public and private subnets. A bastion host in the
# public subnet provides SSH access to an isolated instance in the private
# subnet. The private instance has no internet connectivity — ideal for
# sensitive workloads that must remain air-gapped from the internet.
#
# Architecture:
#
#   WAN ──SSH──▶ Bastion (public subnet)
#                    │
#                    ▼ SSH (private IP)
#                 App Server (private subnet, no internet)
#
# Usage:
#   export AWS_PROFILE=spinifex
#   tofu init && tofu apply
#
# After apply:
#   # SSH to the bastion
#   ssh -i bastion-demo.pem ubuntu@<bastion_ip>
#
#   # From the bastion, SSH to the private instance
#   # (the key is pre-installed at ~/.ssh/bastion-demo.pem via cloud-init)
#   ssh -i ~/.ssh/bastion-demo.pem ubuntu@<private_ip>

terraform {
  required_version = ">= 1.6.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 5.0"
    }
    tls = {
      source  = "hashicorp/tls"
      version = ">= 4.0"
    }
    local = {
      source  = "hashicorp/local"
      version = ">= 2.0"
    }
  }
}

# ---------------------------------------------------------------------------
# Variables
# ---------------------------------------------------------------------------

variable "region" {
  type    = string
  default = "ap-southeast-2"
}

variable "instance_type" {
  type    = string
  default = "t3.small"
}

variable "spinifex_endpoint" {
  type        = string
  default     = "https://127.0.0.1:9999"
  description = "Spinifex AWS gateway endpoint"
}

# ---------------------------------------------------------------------------
# Provider
# ---------------------------------------------------------------------------

provider "aws" {
  region = var.region

  endpoints {
    ec2 = var.spinifex_endpoint
    iam = var.spinifex_endpoint
    sts = var.spinifex_endpoint
  }

  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true
  skip_region_validation      = true
}

# ---------------------------------------------------------------------------
# Data sources
# ---------------------------------------------------------------------------

data "aws_availability_zones" "available" {
  state = "available"
}

data "aws_ami" "ubuntu" {
  most_recent = true
  owners      = ["000000000000"]

  filter {
    name   = "name"
    values = ["*ubuntu-26.04*", "*ubuntu-24.04*"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }

  filter {
    name   = "root-device-type"
    values = ["ebs"]
  }
}

# ---------------------------------------------------------------------------
# SSH Key Pair (shared by bastion and private instances)
# ---------------------------------------------------------------------------

resource "tls_private_key" "bastion" {
  algorithm = "ED25519"
}

resource "aws_key_pair" "bastion" {
  key_name   = "bastion-demo"
  public_key = tls_private_key.bastion.public_key_openssh
}

resource "local_file" "bastion_pem" {
  filename        = "${path.module}/bastion-demo.pem"
  content         = tls_private_key.bastion.private_key_openssh
  file_permission = "0600"
}

# ---------------------------------------------------------------------------
# VPC
# ---------------------------------------------------------------------------

resource "aws_vpc" "main" {
  cidr_block           = "10.20.0.0/16"
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    Name = "bastion-demo-vpc"
  }
}

# ---------------------------------------------------------------------------
# Internet Gateway — only the public subnet routes through this
# ---------------------------------------------------------------------------

resource "aws_internet_gateway" "igw" {
  vpc_id = aws_vpc.main.id

  tags = {
    Name = "bastion-demo-igw"
  }
}

# ---------------------------------------------------------------------------
# Public Subnet — bastion host lives here
# ---------------------------------------------------------------------------

resource "aws_subnet" "public" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = "10.20.1.0/24"
  availability_zone       = data.aws_availability_zones.available.names[0]
  map_public_ip_on_launch = true

  tags = {
    Name = "bastion-demo-public"
  }
}

resource "aws_route_table" "public" {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.igw.id
  }

  tags = {
    Name = "bastion-demo-public-rt"
  }
}

resource "aws_route_table_association" "public" {
  subnet_id      = aws_subnet.public.id
  route_table_id = aws_route_table.public.id
}

# ---------------------------------------------------------------------------
# Private Subnet — isolated instances live here (no public IPs, no internet)
# ---------------------------------------------------------------------------

resource "aws_subnet" "private" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = "10.20.2.0/24"
  availability_zone       = data.aws_availability_zones.available.names[0]
  map_public_ip_on_launch = false

  tags = {
    Name = "bastion-demo-private"
  }
}

# Private route table — no default route, no internet access
resource "aws_route_table" "private" {
  vpc_id = aws_vpc.main.id

  tags = {
    Name = "bastion-demo-private-rt"
  }
}

resource "aws_route_table_association" "private" {
  subnet_id      = aws_subnet.private.id
  route_table_id = aws_route_table.private.id
}

# ---------------------------------------------------------------------------
# Security Groups
# ---------------------------------------------------------------------------

# Bastion: SSH from anywhere
resource "aws_security_group" "bastion" {
  name        = "bastion-demo-bastion-sg"
  description = "Bastion: SSH from WAN"
  vpc_id      = aws_vpc.main.id

  ingress {
    description = "SSH"
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    description = "All outbound"
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "bastion-demo-bastion-sg"
  }
}

# Private instances: SSH only from the bastion security group
resource "aws_security_group" "private" {
  name        = "bastion-demo-private-sg"
  description = "Private: SSH from bastion only"
  vpc_id      = aws_vpc.main.id

  ingress {
    description     = "SSH from bastion"
    from_port       = 22
    to_port         = 22
    protocol        = "tcp"
    security_groups = [aws_security_group.bastion.id]
  }

  egress {
    description = "VPC internal only"
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["10.20.0.0/16"]
  }

  tags = {
    Name = "bastion-demo-private-sg"
  }
}

# ---------------------------------------------------------------------------
# Bastion Host (public subnet)
# ---------------------------------------------------------------------------

resource "aws_instance" "bastion" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = var.instance_type

  subnet_id              = aws_subnet.public.id
  vpc_security_group_ids = [aws_security_group.bastion.id]
  key_name               = aws_key_pair.bastion.key_name

  associate_public_ip_address = true

  # Copy the SSH private key onto the bastion so you can hop to private instances
  user_data_base64 = base64encode(<<-USERDATA
    #!/bin/bash
    set -euo pipefail
    mkdir -p /home/ubuntu/.ssh
    cat > /home/ubuntu/.ssh/bastion-demo.pem <<'KEY'
    ${tls_private_key.bastion.private_key_openssh}
    KEY
    chmod 600 /home/ubuntu/.ssh/bastion-demo.pem
    chown -R ubuntu:ubuntu /home/ubuntu/.ssh
  USERDATA
  )

  tags = {
    Name = "bastion-demo-bastion"
  }
}

# ---------------------------------------------------------------------------
# Private Instance (private subnet — no public IP, no internet)
# ---------------------------------------------------------------------------

resource "aws_instance" "private" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = var.instance_type

  subnet_id              = aws_subnet.private.id
  vpc_security_group_ids = [aws_security_group.private.id]
  key_name               = aws_key_pair.bastion.key_name

  tags = {
    Name = "bastion-demo-private-app"
  }
}

# ---------------------------------------------------------------------------
# Outputs
# ---------------------------------------------------------------------------

output "note" {
  value = "EC2 instances can take 30+ seconds to boot after apply. If SSH is unreachable, wait and retry."
}

output "bastion_public_ip" {
  value = aws_instance.bastion.public_ip
}

output "private_instance_ip" {
  value = aws_instance.private.private_ip
}

output "ssh_to_bastion" {
  description = "SSH to the bastion host"
  value       = "ssh -i bastion-demo.pem ubuntu@${aws_instance.bastion.public_ip}"
}

output "ssh_to_private_from_bastion" {
  description = "From the bastion, SSH to the private instance (key is pre-installed via cloud-init)"
  value       = "ssh -i ~/.ssh/bastion-demo.pem ubuntu@${aws_instance.private.private_ip}"
}
```

### Step 2. Deploy

The workbook defaults to `t3.small` (2 vCPU, 2 GiB) for both bastion and private instance. On clusters without that type registered, override with `TF_VAR_instance_type` — query what's available with `aws ec2 describe-instance-types`.

```bash
export AWS_PROFILE=spinifex
tofu init
tofu apply
```

### Step 3. Connect

> **Note:** EC2 instances can take 30+ seconds to boot after apply. If SSH is unreachable, wait and retry.

SSH into the bastion:

```bash
ssh -i bastion-demo.pem ubuntu@<bastion_public_ip>
```

From the bastion, SSH to the private instance. The key is pre-installed at `~/.ssh/bastion-demo.pem` via cloud-init:

```bash
ssh -i ~/.ssh/bastion-demo.pem ubuntu@<private_ip>
```

### Step 4. Verify Isolation

From the private instance, confirm there is no internet connectivity:

```bash
# This should time out — the private instance has no route to the internet
curl --connect-timeout 5 https://deb.debian.org || echo "No internet access (expected)"
```

The private instance can only communicate within the VPC (`10.20.0.0/16`). Its security group restricts egress to VPC-internal traffic only.

### Clean Up

```bash
tofu destroy
```

## Troubleshooting

### Cannot SSH to Private Instance

The private instance has no public IP — it is only reachable from the bastion. SSH to the bastion first, then use the pre-installed key to hop:

```bash
ssh -i bastion-demo.pem ubuntu@<bastion_ip>
ssh -i ~/.ssh/bastion-demo.pem ubuntu@<private_ip>
```

If the key is missing on the bastion, check that cloud-init completed successfully:

```bash
sudo journalctl -u cloud-init --no-pager
ls -la ~/.ssh/bastion-demo.pem
```

### Private Instance Can Reach the Internet

If the private instance unexpectedly has internet access, check that its route table has no default route:

```bash
aws ec2 describe-route-tables --profile spinifex
```

The private route table should have no `0.0.0.0/0` route. Also verify the private security group egress is restricted to the VPC CIDR (`10.20.0.0/16`).

### AMI Not Found

Ensure you have imported an Ubuntu 26.04 image:

```bash
aws ec2 describe-images --owners 000000000000 --profile spinifex
```

---

# S3-Backed Web App

URL: https://docs.mulgadc.com/docs/s3-webapp
Category: Terraform Workbooks
Updated: 2026-08-19
Tags: terraform, s3, predastore, flask, webapp, workbook, imds, sts, instance-profile

Deploy a Flask file-sharing app on EC2 backed by S3 (Predastore) with Terraform, using an IAM instance profile and short-lived STS credentials from IMDS.

## Overview

Deploy an EC2 instance running a Flask file-sharing web application backed by S3 (Predastore). Users can upload files through a web form and browse uploaded content — demonstrating Terraform managing both compute and object storage together.

This workbook uses the idiomatic AWS credential model: the instance is launched with an **IAM instance profile** and the app pulls short-lived STS credentials from **IMDS** (`169.254.169.254`) through boto3's default credential chain. **No long-lived S3 keys are baked into the instance.** The Terraform run still uses the operator's admin credentials (to create the bucket, role, profile, and instance), but the running instance authenticates to S3 with credentials it fetches at runtime and which expire in ~1 hour.

**Architecture:**

<p align="center">
  <img height="180" src="/docs-assets/terraform-workbooks-s3-webapp-tf-s3-webapp.svg" alt="S3 webapp — browser to Flask EC2 instance; the instance fetches short-lived STS credentials from IMDS (169.254.169.254) via its instance profile, then calls Predastore over the S3 API" width="900">
</p>

**What you'll learn:**

- Configuring the AWS provider with both Spinifex and Predastore endpoints
- Creating S3 buckets on Predastore via Terraform
- Defining an IAM role, least-privilege managed policy, and instance profile, and passing the role to an instance (`iam:PassRole`)
- How the instance obtains short-lived credentials from IMDS via boto3's default credential chain — no static keys on the instance
- Deploying a Python webapp with cloud-init that signs S3 requests with the STS session token

**Prerequisites:**

- Spinifex installed and running (see [Installing Spinifex](https://docs.mulgadc.com/docs/install))
- Predastore running (S3 API on port 8443)
- An Ubuntu 26.04 AMI imported (see [Setting Up Your Cluster](https://docs.mulgadc.com/docs/setting-up-your-cluster))
- OpenTofu or Terraform installed
- The operator identity running the Terraform apply (`AWS_PROFILE=spinifex`) must be allowed to manage IAM and pass the role: `iam:CreateRole`, `iam:CreatePolicy`, `iam:AttachRolePolicy`, `iam:CreateInstanceProfile`, `iam:AddRoleToInstanceProfile`, and `iam:PassRole` on the new role. The bootstrap admin profile satisfies this.
- The EC2 instance must be able to reach Predastore — use the host's br-wan IP, not localhost

## Instructions

### Step 1. Get the Template

Clone the Terraform examples from the Spinifex repository:

```bash
git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
cd spinifex-tf
git sparse-checkout set docs/terraform
cd docs/terraform/s3-webapp
```

Or create the files manually and paste the full configuration below.

### Step 2. Create terraform.tfvars

Before deploying, create a `terraform.tfvars` with your Predastore credentials. The `predastore_host` must be reachable from inside the VPC — use the host's br-wan or LAN IP, not localhost.

```hcl
# Copy this to terraform.tfvars and fill in your values.
#
# The predastore_host must be reachable from INSIDE the VPC guest — not
# localhost. Use the host's br-wan or LAN IP, e.g. "192.168.1.10:8443".
#
# s3_access_key / s3_secret_key are the operator credentials for the Terraform
# run only — they create the bucket on Predastore and the IAM role/profile.
# They are NOT passed to the instance; the instance pulls short-lived STS
# credentials from IMDS via its instance profile.

predastore_host = "192.168.1.10:8443"
s3_access_key   = "AKIAIOSFODNN7EXAMPLE"
s3_secret_key   = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

# Optional overrides:
# spinifex_endpoint    = "https://localhost:9999"
# predastore_endpoint  = "https://localhost:8443"
# bucket_name          = "webapp-uploads"
# region               = "ap-southeast-2"
```

### Step 3. Create main.tf

```hcl
# Example 3: S3-Backed Web Application
#
# Deploys an EC2 instance running a simple file-sharing webapp backed by S3
# (Predastore). Users can upload files through a web form and browse uploaded
# content — demonstrating Terraform managing both compute (Spinifex) and
# object storage (Predastore) resources together.
#
# Architecture:
#
#   Browser ──HTTP──▶ EC2 Instance (Flask webapp, port 80)
#                         │  ▲ IMDS: short-lived STS creds (ASIA + token)
#                         │  └──── 169.254.169.254
#                         ▼ S3 API (boto3, SigV4 with session token)
#                     Predastore (port 8443)
#
# Credentials:
#   The Terraform run uses the operator's admin keys (s3_access_key /
#   s3_secret_key) to create the bucket, IAM role, instance profile, and
#   instance. The INSTANCE receives no long-lived secret — boto3's default
#   credential chain pulls short-lived STS credentials from IMDS
#   (169.254.169.254), resolved via the attached instance profile -> role.
#
# Prerequisites:
#   - Spinifex services running (gateway on port 9999)
#   - Predastore running (S3 API on port 8443)
#   - The operator identity (AWS_PROFILE=spinifex) must be allowed
#     iam:CreateRole / iam:CreatePolicy / iam:AttachRolePolicy /
#     iam:CreateInstanceProfile / iam:AddRoleToInstanceProfile and
#     iam:PassRole on the new role.
#   - The EC2 instance must be able to reach the Predastore endpoint.
#     Set `predastore_host` to the IP reachable from inside the VPC
#     (e.g. the host's br-wan IP, NOT localhost).
#
# Usage:
#   cd spinifex/scripts/iac/aws/examples/03-s3-webapp
#   export AWS_PROFILE=spinifex
#   tofu init && tofu apply
#
# After apply:
#   curl http://<public_ip>          # File browser UI
#   ssh -i s3-webapp-demo.pem ubuntu@<public_ip>

terraform {
  required_version = ">= 1.6.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 5.0"
    }
    tls = {
      source  = "hashicorp/tls"
      version = ">= 4.0"
    }
    local = {
      source  = "hashicorp/local"
      version = ">= 2.0"
    }
  }
}

# ---------------------------------------------------------------------------
# Variables
# ---------------------------------------------------------------------------

variable "region" {
  type    = string
  default = "ap-southeast-2"
}

variable "instance_type" {
  type    = string
  default = "t3.small"
}

variable "spinifex_endpoint" {
  type        = string
  default     = "https://127.0.0.1:9999"
  description = "Spinifex AWS gateway endpoint (EC2/IAM)"
}

variable "predastore_endpoint" {
  type        = string
  default     = "https://127.0.0.1:8443"
  description = "Predastore S3 endpoint (for Terraform to create buckets)"
}

variable "predastore_host" {
  type        = string
  description = "Predastore host:port reachable from inside the VPC (e.g. 192.168.1.10:8443)"
}

variable "s3_access_key" {
  type        = string
  description = "Operator S3 access key for the Terraform run (creates the bucket on Predastore); the instance uses IMDS, not this key"
}

variable "s3_secret_key" {
  type        = string
  sensitive   = true
  description = "Operator S3 secret key for the Terraform run; the instance uses IMDS, not this key"
}

variable "bucket_name" {
  type    = string
  default = "webapp-uploads"
}

# ---------------------------------------------------------------------------
# Provider — EC2 via Spinifex gateway, S3 via Predastore
# ---------------------------------------------------------------------------

provider "aws" {
  region     = var.region
  access_key = var.s3_access_key
  secret_key = var.s3_secret_key

  endpoints {
    ec2 = var.spinifex_endpoint
    s3  = var.predastore_endpoint
    iam = var.spinifex_endpoint
    sts = var.spinifex_endpoint
  }

  s3_use_path_style = true

  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true
  skip_region_validation      = true
}

# ---------------------------------------------------------------------------
# Data sources
# ---------------------------------------------------------------------------

data "aws_availability_zones" "available" {
  state = "available"
}

data "aws_ami" "ubuntu" {
  most_recent = true
  owners      = ["000000000000"]

  filter {
    name   = "name"
    values = ["*ubuntu-26.04*", "*ubuntu-24.04*"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }

  filter {
    name   = "root-device-type"
    values = ["ebs"]
  }
}

# ---------------------------------------------------------------------------
# S3 Bucket (created on Predastore)
# ---------------------------------------------------------------------------

resource "aws_s3_bucket" "uploads" {
  bucket = var.bucket_name
}

# ---------------------------------------------------------------------------
# IAM Instance Role — least-privilege S3 access fetched at runtime via IMDS
# ---------------------------------------------------------------------------

resource "aws_iam_role" "webapp" {
  name = "s3-webapp-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "ec2.amazonaws.com" }
      Action    = "sts:AssumeRole"
    }]
  })
}

resource "aws_iam_policy" "webapp_s3" {
  name = "s3-webapp-policy"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid      = "ListBucket"
        Effect   = "Allow"
        Action   = ["s3:ListBucket"]
        Resource = ["arn:aws:s3:::${var.bucket_name}"]
      },
      {
        Sid      = "ObjectRW"
        Effect   = "Allow"
        Action   = ["s3:GetObject", "s3:PutObject"]
        Resource = ["arn:aws:s3:::${var.bucket_name}/*"]
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "webapp" {
  role       = aws_iam_role.webapp.name
  policy_arn = aws_iam_policy.webapp_s3.arn
}

resource "aws_iam_instance_profile" "webapp" {
  name = "s3-webapp-profile"
  role = aws_iam_role.webapp.name
}

# ---------------------------------------------------------------------------
# SSH Key Pair
# ---------------------------------------------------------------------------

resource "tls_private_key" "webapp" {
  algorithm = "ED25519"
}

resource "aws_key_pair" "webapp" {
  key_name   = "s3-webapp-demo"
  public_key = tls_private_key.webapp.public_key_openssh
}

resource "local_file" "webapp_pem" {
  filename        = "${path.module}/s3-webapp-demo.pem"
  content         = tls_private_key.webapp.private_key_openssh
  file_permission = "0600"
}

# ---------------------------------------------------------------------------
# VPC + Public Subnet
# ---------------------------------------------------------------------------

resource "aws_vpc" "main" {
  cidr_block           = "10.30.0.0/16"
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    Name = "s3-webapp-demo-vpc"
  }
}

resource "aws_internet_gateway" "igw" {
  vpc_id = aws_vpc.main.id

  tags = {
    Name = "s3-webapp-demo-igw"
  }
}

resource "aws_subnet" "public" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = "10.30.1.0/24"
  availability_zone       = data.aws_availability_zones.available.names[0]
  map_public_ip_on_launch = true

  tags = {
    Name = "s3-webapp-demo-public"
  }
}

resource "aws_route_table" "public" {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.igw.id
  }

  tags = {
    Name = "s3-webapp-demo-public-rt"
  }
}

resource "aws_route_table_association" "public" {
  subnet_id      = aws_subnet.public.id
  route_table_id = aws_route_table.public.id
}

# ---------------------------------------------------------------------------
# Security Group — SSH + HTTP inbound, all outbound
# ---------------------------------------------------------------------------

resource "aws_security_group" "webapp" {
  name        = "s3-webapp-demo-sg"
  description = "Allow SSH and HTTP inbound"
  vpc_id      = aws_vpc.main.id

  ingress {
    description = "SSH"
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    description = "HTTP"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    description = "All outbound"
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "s3-webapp-demo-sg"
  }
}

# ---------------------------------------------------------------------------
# EC2 Instance — Flask webapp that talks to Predastore S3
# ---------------------------------------------------------------------------

resource "aws_instance" "webapp" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = var.instance_type

  subnet_id              = aws_subnet.public.id
  vpc_security_group_ids = [aws_security_group.webapp.id]
  key_name               = aws_key_pair.webapp.key_name
  iam_instance_profile   = aws_iam_instance_profile.webapp.name

  associate_public_ip_address = true

  # The role's permissions must exist before the instance boots and makes its
  # first IMDS-credentialed S3 call
  depends_on = [aws_iam_role_policy_attachment.webapp]

  user_data_base64 = base64encode(<<-USERDATA
    #!/bin/bash
    set -euo pipefail

    # Install dependencies
    apt-get update -y
    apt-get install -y python3-pip python3-venv

    # Create app directory and virtualenv
    mkdir -p /opt/webapp
    python3 -m venv /opt/webapp/venv
    /opt/webapp/venv/bin/pip install flask boto3

    # Write S3 credentials config
    cat > /opt/webapp/.env <<'ENVFILE'
    S3_ENDPOINT=https://${var.predastore_host}
    S3_BUCKET=${var.bucket_name}
    S3_REGION=${var.region}
    ENVFILE

    # Write the Flask application
    cat > /opt/webapp/app.py <<'PYEOF'
    import os, io, urllib3
    from flask import Flask, request, redirect, url_for, Response

    # Suppress TLS warnings for self-signed certs
    urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

    # Load env
    env = {}
    with open("/opt/webapp/.env") as f:
        for line in f:
            line = line.strip()
            if "=" in line and not line.startswith("#"):
                k, v = line.split("=", 1)
                env[k] = v

    import boto3
    from botocore.config import Config

    s3 = boto3.client(
        "s3",
        endpoint_url=env["S3_ENDPOINT"],
        region_name=env["S3_REGION"],
        verify=False,
        config=Config(s3={"addressing_style": "path"}),
    )

    BUCKET = env["S3_BUCKET"]
    app = Flask(__name__)

    @app.route("/")
    def index():
        # List objects in the bucket
        try:
            resp = s3.list_objects_v2(Bucket=BUCKET)
            objects = resp.get("Contents", [])
        except Exception as e:
            objects = []

        rows = ""
        for obj in objects:
            key = obj["Key"]
            size = obj["Size"]
            rows += f'<tr><td><a href="/files/{key}">{key}</a></td><td>{size} bytes</td></tr>\n'

        return f"""<!DOCTYPE html>
    <html>
    <head><title>Spinifex S3 File Browser</title></head>
    <body style="font-family: sans-serif; max-width: 700px; margin: 40px auto;">
      <h1>Spinifex S3 File Browser</h1>
      <p>Bucket: <code>{BUCKET}</code></p>

      <h2>Upload a File</h2>
      <form method="POST" action="/upload" enctype="multipart/form-data">
        <input type="file" name="file" required>
        <button type="submit">Upload</button>
      </form>

      <h2>Files</h2>
      <table border="1" cellpadding="6" cellspacing="0" style="border-collapse: collapse;">
        <tr><th>Key</th><th>Size</th></tr>
        {rows if rows else '<tr><td colspan="2">No files yet</td></tr>'}
      </table>

      <hr>
      <p><small>Powered by Spinifex + Predastore</small></p>
    </body>
    </html>"""

    @app.route("/upload", methods=["POST"])
    def upload():
        f = request.files.get("file")
        if not f or not f.filename:
            return redirect("/")
        s3.put_object(Bucket=BUCKET, Key=f.filename, Body=f.read())
        return redirect("/")

    @app.route("/files/<path:key>")
    def download(key):
        try:
            obj = s3.get_object(Bucket=BUCKET, Key=key)
            return Response(
                obj["Body"].read(),
                headers={"Content-Disposition": f'inline; filename="{key}"'},
            )
        except Exception:
            return "Not found", 404

    if __name__ == "__main__":
        app.run(host="0.0.0.0", port=80)
    PYEOF

    # Create a systemd service so the webapp starts on boot
    cat > /etc/systemd/system/s3-webapp.service <<'SVCEOF'
    [Unit]
    Description=S3 File Browser Webapp
    After=network.target

    [Service]
    Type=simple
    ExecStart=/opt/webapp/venv/bin/python /opt/webapp/app.py
    WorkingDirectory=/opt/webapp
    Restart=always
    RestartSec=3

    [Install]
    WantedBy=multi-user.target
    SVCEOF

    systemctl daemon-reload
    systemctl enable s3-webapp
    systemctl start s3-webapp
  USERDATA
  )

  tags = {
    Name = "s3-webapp-demo"
  }
}

# ---------------------------------------------------------------------------
# Outputs
# ---------------------------------------------------------------------------

output "note" {
  value = "EC2 instances can take 30+ seconds to boot after apply. If SSH or HTTP is unreachable, wait and retry."
}

output "instance_id" {
  value = aws_instance.webapp.id
}

output "public_ip" {
  value = aws_instance.webapp.public_ip
}

output "bucket_name" {
  value = aws_s3_bucket.uploads.id
}

output "ssh_command" {
  value = "ssh -i s3-webapp-demo.pem ubuntu@${aws_instance.webapp.public_ip}"
}

output "web_url" {
  value = "http://${aws_instance.webapp.public_ip}"
}
```

### Step 4. Deploy

The workbook defaults to `t3.small` (2 vCPU, 2 GiB). On clusters without that type registered, override with `TF_VAR_instance_type` — query what's available with `aws ec2 describe-instance-types`.

```bash
export AWS_PROFILE=spinifex
tofu init
tofu apply
```

### Step 5. Test the Application

> **Note:** EC2 instances can take 30+ seconds to boot after apply. If SSH or HTTP is unreachable, wait and retry.

Open the `web_url` output in your browser. You should see the file browser UI. Upload a file and verify it appears in the list.

```bash
# Verify via CLI
curl http://<public_ip>

# Check the S3 bucket directly
aws s3 ls s3://webapp-uploads/ --profile spinifex --endpoint-url https://localhost:8443
```

### Clean Up

```bash
tofu destroy
```

## Troubleshooting

### Predastore Connection Refused from Instance

The EC2 instance cannot reach `localhost` on the host. Set `predastore_host` to the host's br-wan or LAN IP address:

```hcl
predastore_host = "192.168.1.10:8443"
```

### S3 Bucket Creation Fails

Verify Predastore is running and accessible:

```bash
curl -k https://localhost:8443/
aws s3 ls --profile spinifex --endpoint-url https://localhost:8443
```

### Flask App Not Starting

SSH into the instance and check the service:

```bash
ssh -i s3-webapp-demo.pem ubuntu@<public_ip>
sudo systemctl status s3-webapp
sudo journalctl -u s3-webapp --no-pager -n 50
```

### Upload Fails with `403 AccessDenied`

The instance reached S3 but was not authorized. The `.env` file deliberately holds **no** keys — credentials come from IMDS:

```bash
ssh -i s3-webapp-demo.pem ubuntu@<public_ip>
cat /opt/webapp/.env   # S3_ENDPOINT / S3_BUCKET / S3_REGION only — no keys
```

Check that:

- the managed policy is attached to the role (`s3:ListBucket` on the bucket, `s3:GetObject`/`s3:PutObject` on its objects), and the policy's bucket name matches `bucket_name`;
- the assumed-role session's account matches the bucket owner's account — Predastore enforces bucket ownership. Both are created by the same operator identity, so they align by construction; a mismatch surfaces as `403`.

### `NoCredentialsError` / No Credentials

boto3 could not obtain credentials, which means the IMDS credential chain did not resolve. From the instance:

```bash
ssh -i s3-webapp-demo.pem ubuntu@<public_ip>

# IMDSv2: fetch a token, then the role name behind the instance profile
TOKEN=$(curl -sX PUT http://169.254.169.254/latest/api/token \
  -H 'X-aws-ec2-metadata-token-ttl-seconds: 60')
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/iam/security-credentials/

# Confirm the resolved identity — expect assumed-role/s3-webapp-role/<instance-id>
aws sts get-caller-identity --endpoint-url https://<predastore_host>
```

If the role name is empty, the instance profile was not attached — confirm `iam_instance_profile` on the instance and that `iam:PassRole` is allowed for the operator.

### AMI Not Found

Ensure you have imported an Ubuntu 26.04 image:

```bash
aws ec2 describe-images --owners 000000000000 --profile spinifex
```

---

# Nginx Web Server (Load Balanced)

URL: https://docs.mulgadc.com/docs/nginx-alb
Category: Terraform Workbooks
Updated: 2026-08-19
Tags: terraform, nginx, ec2, elbv2, alb, vpc, workbook

Deploy a VPC with two private EC2 instances running Nginx behind an internet-facing Application Load Balancer on Spinifex, using Terraform or OpenTofu.

## Overview

Deploy two Nginx web servers behind an internet-facing Application Load Balancer on Spinifex using Terraform/OpenTofu. This workbook provisions a VPC with public and private subnets, an internet gateway and NAT Gateway, route tables, security group, SSH key pair, an application load balancer (ALB) and two EC2 instances with cloud-init user-data that installs and starts Nginx. Only the ALB is reachable from outside the VPC — the Nginx instances live in the **private subnets** and reach the internet only for cloud-init bootstrapping via the NAT Gateway.

<p align="center">
  <img height="480" src="/docs-assets/terraform-workbooks-nginx-alb-tf-nginx-alb.svg" alt="Nginx + ALB VPC — IGW, two public subnets with ALB ENIs and NAT GW, two private subnets hosting nginx" width="900">
</p>

**What you'll learn:**

- Configuring the AWS Terraform provider to target Spinifex
- Creating a VPC with public + private subnets, an IGW and a NAT Gateway
- Provisioning a multi-AZ internet-facing ALB fronting private workers
- Provisioning an EC2 instance with cloud-init user-data
- Generating SSH key pairs with the TLS provider

**What gets created**

| Resource | Name | Purpose |
|---|---|---|
| VPC | `nginx-alb-vpc` | Isolated network (10.20.0.0/16) |
| Public Subnets | `nginx-alb-public-a`, `nginx-alb-public-b` | Two AZs hosting the ALB and NAT Gateway |
| Private Subnets | `nginx-alb-private-a`, `nginx-alb-private-b` | Two AZs hosting the Nginx workers |
| Internet Gateway | `nginx-alb-igw` | Routes internet traffic for the public subnets |
| Elastic IP | `nginx-alb-nat-eip` | Public address for the NAT Gateway |
| NAT Gateway | `nginx-alb-nat` | Outbound internet for the private subnets (cloud-init apt bootstrap) |
| Security Group | `nginx-alb-sg` | Allows SSH (22) and HTTP (80) inbound |
| EC2 Instances | `nginx-alb-1`, `nginx-alb-2` | Ubuntu 26.04 with Nginx via cloud-init (private subnets) |
| ALB | `nginx-alb` | Internet-facing Application Load Balancer on port 80 |
| Target Group | `nginx-alb-tg` | HTTP health-checked group for both instances |
| Listener | HTTP :80 | Forwards traffic to the target group |

**Prerequisites:**

- Spinifex installed and running (see [Installing Spinifex](https://docs.mulgadc.com/docs/install))
- An Ubuntu 26.04 AMI imported (see [Setting Up Your Cluster](https://docs.mulgadc.com/docs/setting-up-your-cluster))
- OpenTofu or Terraform installed

## Instructions

### Step 1. Get the Template

Clone the Terraform examples from the Spinifex repository:

```bash
git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
cd spinifex-tf
git sparse-checkout set docs/terraform
cd docs/terraform/nginx-alb
```

Or create a `main.tf` file and paste the full configuration below.

```hcl
# Example: Nginx Web Servers with ALB on Spinifex
#
# Deploys a VPC with two public subnets hosting Nginx EC2 instances and an
# internet-facing Application Load Balancer. Workers sit in public subnets
# with auto-assigned public IPs purely so cloud-init can apt-install nginx.
# The ALB targets them by primary private IP (target_type=instance), so
# load-balanced traffic stays on the private VPC network.
#
# In a production deployment, workers would be in private subnets with
# nginx baked into a custom AMI (or installed via a private repository
# mirror), and a NAT Gateway would be unnecessary for this workload.
#
# Demonstrates: VPC, public subnets, internet gateway, route tables,
# security group, key pair, cloud-init user-data, EC2 instances with
# auto-assigned public IPs, ALB, target group, and listener.
#
# Usage:
#   cd spinifex/docs/terraform/nginx-alb
#   export AWS_PROFILE=spinifex
#   tofu init && tofu apply
#
# After apply, fetch the ALB's public IP (the *.elb.spinifex.local DNS
# name does not resolve from your host):
#
#   aws elbv2 describe-load-balancers --names nginx-alb \
#     --query 'LoadBalancers[0].AvailabilityZones[].LoadBalancerAddresses[].IpAddress' \
#     --output text
#
# Then:
#   curl http://<alb_public_ip>    # Load-balanced Nginx (alternates between instances)

terraform {
  required_version = ">= 1.6.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.65.0, < 5.66.0"
    }
    tls = {
      source  = "hashicorp/tls"
      version = ">= 4.0"
    }
    local = {
      source  = "hashicorp/local"
      version = ">= 2.0"
    }
  }
}

# ---------------------------------------------------------------------------
# Variables
# ---------------------------------------------------------------------------

variable "region" {
  type    = string
  default = "ap-southeast-2"
}

variable "instance_type" {
  type = string
  default = "t3.small"
}

variable "spinifex_endpoint" {
  type        = string
  default     = "https://127.0.0.1:9999"
  description = "Spinifex AWS gateway endpoint"
}

# ---------------------------------------------------------------------------
# Provider — point the AWS provider at Spinifex
# ---------------------------------------------------------------------------

provider "aws" {
  region = var.region

  endpoints {
    ec2                    = var.spinifex_endpoint
    iam                    = var.spinifex_endpoint
    sts                    = var.spinifex_endpoint
    elasticloadbalancingv2 = var.spinifex_endpoint
  }

  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true
  skip_region_validation      = true
}

# ---------------------------------------------------------------------------
# Data sources
# ---------------------------------------------------------------------------

data "aws_availability_zones" "available" {
  state = "available"
}

data "aws_ami" "ubuntu" {
  most_recent = true
  owners      = ["000000000000"] # Spinifex system images

  filter {
    name   = "name"
    values = ["*ubuntu-26.04*", "*ubuntu-24.04*"]
  }
}

# ---------------------------------------------------------------------------
# SSH Key Pair
# ---------------------------------------------------------------------------

resource "tls_private_key" "nginx" {
  algorithm = "ED25519"
}

resource "aws_key_pair" "nginx" {
  key_name   = "nginx-alb-demo"
  public_key = tls_private_key.nginx.public_key_openssh
}

resource "local_file" "nginx_pem" {
  filename        = "${path.module}/nginx-alb-demo.pem"
  content         = tls_private_key.nginx.private_key_openssh
  file_permission = "0600"
}

# ---------------------------------------------------------------------------
# VPC
# ---------------------------------------------------------------------------

resource "aws_vpc" "main" {
  cidr_block           = "10.20.0.0/16"
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    Name = "nginx-alb-vpc"
  }
}

# ---------------------------------------------------------------------------
# Internet Gateway
# ---------------------------------------------------------------------------

resource "aws_internet_gateway" "igw" {
  vpc_id = aws_vpc.main.id

  tags = {
    Name = "nginx-alb-igw"
  }
}

# ---------------------------------------------------------------------------
# Public Subnets (two AZs for the ALB and NAT Gateway)
# ---------------------------------------------------------------------------

resource "aws_subnet" "public_a" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = "10.20.1.0/24"
  availability_zone       = data.aws_availability_zones.available.names[0]
  map_public_ip_on_launch = true

  tags = {
    Name = "nginx-alb-public-a"
  }
}

resource "aws_subnet" "public_b" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = "10.20.2.0/24"
  availability_zone       = data.aws_availability_zones.available.names[0]
  map_public_ip_on_launch = true

  tags = {
    Name = "nginx-alb-public-b"
  }
}

# ---------------------------------------------------------------------------
# Route Table — public subnets egress via IGW
# ---------------------------------------------------------------------------

resource "aws_route_table" "public" {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.igw.id
  }

  tags = {
    Name = "nginx-alb-public-rt"
  }
}

resource "aws_route_table_association" "public_a" {
  subnet_id      = aws_subnet.public_a.id
  route_table_id = aws_route_table.public.id
}

resource "aws_route_table_association" "public_b" {
  subnet_id      = aws_subnet.public_b.id
  route_table_id = aws_route_table.public.id
}

# ---------------------------------------------------------------------------
# Security Group — SSH + HTTP inbound, all outbound
# ---------------------------------------------------------------------------

resource "aws_security_group" "web" {
  name        = "nginx-alb-sg"
  description = "Allow SSH and HTTP inbound"
  vpc_id      = aws_vpc.main.id

  ingress {
    description = "SSH"
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    description = "HTTP"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    description = "All outbound"
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "nginx-alb-sg"
  }
}

# ---------------------------------------------------------------------------
# EC2 Instances — two Nginx servers with distinct landing pages
# ---------------------------------------------------------------------------

resource "aws_instance" "nginx_1" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = var.instance_type

  subnet_id              = aws_subnet.public_a.id
  vpc_security_group_ids = [aws_security_group.web.id]
  key_name               = aws_key_pair.nginx.key_name

  user_data_base64 = base64encode(<<-USERDATA
    #!/bin/bash
    set -euo pipefail

    apt-get update -y
    apt-get install -y nginx

    INSTANCE_ID=$(cat /var/lib/cloud/data/instance-id 2>/dev/null || hostname)
    cat > /var/www/html/index.html <<HTML
    <!DOCTYPE html>
    <html>
    <head><title>Spinifex ALB Demo</title></head>
    <body style="font-family: sans-serif; max-width: 600px; margin: 80px auto;">
      <h1>Hello from Spinifex!</h1>
      <p><strong>Instance:</strong> $INSTANCE_ID (Server 1)</p>
      <p>This Nginx server is behind an Application Load Balancer.</p>
      <hr>
      <p><small>Provisioned via cloud-init user-data.</small></p>
    </body>
    </html>
    HTML

    systemctl enable nginx
    systemctl restart nginx
  USERDATA
  )

  tags = {
    Name = "nginx-alb-1"
  }
}

resource "aws_instance" "nginx_2" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = var.instance_type

  subnet_id              = aws_subnet.public_b.id
  vpc_security_group_ids = [aws_security_group.web.id]
  key_name               = aws_key_pair.nginx.key_name

  user_data_base64 = base64encode(<<-USERDATA
    #!/bin/bash
    set -euo pipefail

    apt-get update -y
    apt-get install -y nginx

    INSTANCE_ID=$(cat /var/lib/cloud/data/instance-id 2>/dev/null || hostname)
    cat > /var/www/html/index.html <<HTML
    <!DOCTYPE html>
    <html>
    <head><title>Spinifex ALB Demo</title></head>
    <body style="font-family: sans-serif; max-width: 600px; margin: 80px auto;">
      <h1>Hello from Spinifex!</h1>
      <p><strong>Instance:</strong> $INSTANCE_ID (Server 2)</p>
      <p>This Nginx server is behind an Application Load Balancer.</p>
      <hr>
      <p><small>Provisioned via cloud-init user-data.</small></p>
    </body>
    </html>
    HTML

    systemctl enable nginx
    systemctl restart nginx
  USERDATA
  )

  tags = {
    Name = "nginx-alb-2"
  }
}

# ---------------------------------------------------------------------------
# Application Load Balancer
# ---------------------------------------------------------------------------

resource "aws_lb" "web" {
  name               = "nginx-alb"
  internal           = false
  load_balancer_type = "application"
  security_groups    = [aws_security_group.web.id]
  subnets            = [aws_subnet.public_a.id, aws_subnet.public_b.id]

  tags = {
    Name = "nginx-alb"
  }
}

# ---------------------------------------------------------------------------
# Target Group — HTTP health-checked on port 80
# ---------------------------------------------------------------------------

resource "aws_lb_target_group" "nginx" {
  name     = "nginx-alb-tg"
  port     = 80
  protocol = "HTTP"
  vpc_id   = aws_vpc.main.id

  health_check {
    path                = "/"
    protocol            = "HTTP"
    healthy_threshold   = 2
    unhealthy_threshold = 3
    timeout             = 5
    interval            = 10
  }

  tags = {
    Name = "nginx-alb-tg"
  }
}

# ---------------------------------------------------------------------------
# Register both instances as targets
# ---------------------------------------------------------------------------

resource "aws_lb_target_group_attachment" "nginx_1" {
  target_group_arn = aws_lb_target_group.nginx.arn
  target_id        = aws_instance.nginx_1.id
  port             = 80
}

resource "aws_lb_target_group_attachment" "nginx_2" {
  target_group_arn = aws_lb_target_group.nginx.arn
  target_id        = aws_instance.nginx_2.id
  port             = 80
}

# ---------------------------------------------------------------------------
# Listener — forward HTTP :80 to the target group
# ---------------------------------------------------------------------------

resource "aws_lb_listener" "http" {
  load_balancer_arn = aws_lb.web.arn
  port              = 80
  protocol          = "HTTP"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.nginx.arn
  }
}

# ---------------------------------------------------------------------------
# Outputs
# ---------------------------------------------------------------------------

output "note" {
  value = <<-EOT
    EC2 instances can take 30+ seconds to boot after apply — if HTTP is
    unreachable, wait and retry.

    The Nginx instances have private IPs only. The ALB DNS name ends in
    .elb.spinifex.local and will not resolve from your host, so fetch the
    ALB's public IP with:

      aws elbv2 describe-load-balancers --names nginx-alb \
        --query 'LoadBalancers[0].AvailabilityZones[].LoadBalancerAddresses[].IpAddress' \
        --output text

    Then: curl http://<that-ip>
  EOT
}

output "alb_name" {
  value = aws_lb.web.name
}

output "alb_arn" {
  value = aws_lb.web.arn
}

output "alb_dns_name" {
  value = aws_lb.web.dns_name
}

output "instance_1_id" {
  value = aws_instance.nginx_1.id
}

output "instance_1_private_ip" {
  value = aws_instance.nginx_1.private_ip
}

output "instance_2_id" {
  value = aws_instance.nginx_2.id
}

output "instance_2_private_ip" {
  value = aws_instance.nginx_2.private_ip
}
```

### Step 2. Install Debian AMI

Install the Debian 13 AMI which is used in the example to host the `nginx` webservers as an EC2 instance.

```bash
spx admin images import --name debian-13-x86_64
```

> **Note:** The load balancer itself runs as a direct-boot QEMU microvm using the kernel + initramfs bundled with the Spinifex distribution (`/usr/share/spinifex/microvm/`), so no separate LB AMI import is required.

### Step 3. Deploy

```bash
export AWS_PROFILE=spinifex
tofu init
```


### Step 4. Specify instance and apply

Next, depending on your architecture and CPU/memory requirements you must specify an instance type to launch.

Either specify an instance type directly (e.g Intel)

```bash
# AMD instance
export TF_VAR_instance_type="t3a.small"

# Or, Intel
export TF_VAR_instance_type="t3.small"
```

Or alternatively, using the AWS CLI tool query your instance for available types (e.g Intel, AMD, ARM) that support 2 vCPUs and 1 GB RAM.

```bash
export TF_VAR_instance_type=$(aws ec2 describe-instance-types \
  --query "sort_by(InstanceTypes[?VCpuInfo.DefaultVCpus==\`2\` && MemoryInfo.SizeInMiB>=\`1024\`], &MemoryInfo.SizeInMiB)[0].InstanceType" \
  --output text)
```

Next, apply and launch the template:

```bash
tofu apply
```

### Step 5. Verify

> **Note:** EC2 instances can take 30+ seconds to boot after apply, and the NAT Gateway must be `available` before cloud-init on the workers can reach the apt repository. If the ALB returns 5xx or HTTP is unreachable, wait and retry — the target group health checks need a moment to mark both instances healthy once Nginx has installed.

The ALB is internet-facing, but the DNS name Spinifex returns (`*.elb.spinifex.local`) will not resolve from your host. Fetch the ALB's public IP with the AWS CLI:

```bash
ALB_IP=$(aws elbv2 describe-load-balancers --names nginx-alb \
  --query 'LoadBalancers[0].AvailabilityZones[].LoadBalancerAddresses[].IpAddress' \
  --output text)
```

Then hit the ALB — successive requests should alternate between Server 1 and Server 2:

```bash
curl http://$ALB_IP
curl http://$ALB_IP
```

Open `http://$ALB_IP` in your browser and refresh to see the page alternate content served from each instance.

The Nginx instances themselves only have private IPs (see the `instance_1_private_ip` / `instance_2_private_ip` outputs) and are only reachable from inside the VPC — go through the ALB.

Check target health via AWS CLI:

```bash
TG_ARN=$(aws elbv2 describe-target-groups \
  --query 'TargetGroups[0].TargetGroupArn' \
  --output text)

aws elbv2 describe-target-health --target-group-arn $TG_ARN
```

### Cleanup

```bash
tofu destroy
```

## Troubleshooting

### AMI Not Found

Ensure you have imported an Ubuntu 26.04 image. Check available AMIs:

```bash
aws ec2 describe-images --owners 000000000000 --profile spinifex
```

If missing import:

```bash
spx admin images import --name ubuntu-26.04-x86_64
```

### Provider Connection Refused

Verify Spinifex services are running:

```bash
sudo systemctl status spinifex.target
curl -k https://localhost:9999/
```

### ALB Returns 5xx / Targets Unhealthy

Give the instances a moment to finish cloud-init (Nginx has to install before it can answer health checks). Check target health:

```bash
TG_ARN=$(aws elbv2 describe-target-groups --names nginx-alb-tg \
  --query 'TargetGroups[0].TargetGroupArn' --output text)
aws elbv2 describe-target-health --target-group-arn "$TG_ARN"
```

If targets stay unhealthy, verify the instances are running:

```bash
aws ec2 describe-instances --profile spinifex
```

If cloud-init on the workers never finished, confirm the NAT Gateway is `available` (the private subnets rely on it for outbound apt access):

```bash
aws ec2 describe-nat-gateways --query 'NatGateways[].[NatGatewayId,State]'
```

### Nginx Not Responding

The Nginx instances have no public IP, so you can't SSH in directly from your host. If you need to inspect cloud-init logs, launch a small jump host in the same VPC or run commands via the Spinifex console, then:

```bash
ssh -i nginx-alb-demo.pem ubuntu@<instance_private_ip>
sudo journalctl -u cloud-init --no-pager
sudo systemctl status nginx
```

---

# Nginx Web Server

URL: https://docs.mulgadc.com/docs/nginx-webserver
Category: Terraform Workbooks
Updated: 2026-08-19
Tags: terraform, nginx, ec2, vpc, workbook

Provision a VPC, public subnet, internet gateway, route table, security group, and an EC2 instance that installs and starts Nginx from cloud-init user-data.

## Overview

Deploy a complete Nginx web server on Spinifex using Terraform/OpenTofu. This workbook provisions a VPC, public subnet, internet gateway, route table, security group, SSH key pair, and an EC2 instance with cloud-init user-data that installs and starts Nginx.

**What you'll learn:**

- Configuring the AWS Terraform provider to target Spinifex
- Creating a VPC with public internet access
- Provisioning an EC2 instance with cloud-init user-data
- Generating SSH key pairs with the TLS provider

**Prerequisites:**

- Spinifex installed and running (see [Installing Spinifex](https://docs.mulgadc.com/docs/install))
- An Ubuntu 26.04 AMI imported (see [Setting Up Your Cluster](https://docs.mulgadc.com/docs/setting-up-your-cluster))
- OpenTofu or Terraform installed

## Instructions

### Step 1. Get the Template

Clone the Terraform examples from the Spinifex repository:

```bash
git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
cd spinifex-tf
git sparse-checkout set docs/terraform
cd docs/terraform/nginx-webserver
```

Or create a `main.tf` file and paste the full configuration below.

```hcl
# Example 1: Nginx Web Server on Spinifex
#
# Deploys a VPC with a public subnet and an EC2 instance running Nginx.
# Demonstrates: VPC, subnet, internet gateway, route table, security group,
# key pair, cloud-init user-data, and EC2 instance provisioning.
#
# Usage:
#   cd spinifex/scripts/iac/aws/examples/01-nginx-webserver
#   export AWS_PROFILE=spinifex
#   tofu init && tofu apply
#
# After apply:
#   curl http://<public_ip>        # Nginx welcome page
#   ssh -i nginx-demo.pem ubuntu@<public_ip>

terraform {
  required_version = ">= 1.6.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 5.0"
    }
    tls = {
      source  = "hashicorp/tls"
      version = ">= 4.0"
    }
    local = {
      source  = "hashicorp/local"
      version = ">= 2.0"
    }
  }
}

# ---------------------------------------------------------------------------
# Variables
# ---------------------------------------------------------------------------

variable "region" {
  type    = string
  default = "ap-southeast-2"
}

variable "instance_type" {
  type    = string
  default = "t3.small"
}

variable "spinifex_endpoint" {
  type        = string
  default     = "https://127.0.0.1:9999"
  description = "Spinifex AWS gateway endpoint"
}

# ---------------------------------------------------------------------------
# Provider — point the AWS provider at Spinifex
# ---------------------------------------------------------------------------

provider "aws" {
  region = var.region

  endpoints {
    ec2 = var.spinifex_endpoint
    iam = var.spinifex_endpoint
    sts = var.spinifex_endpoint
  }

  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true
  skip_region_validation      = true
}

# ---------------------------------------------------------------------------
# Data sources
# ---------------------------------------------------------------------------

data "aws_availability_zones" "available" {
  state = "available"
}

data "aws_ami" "ubuntu" {
  most_recent = true
  owners      = ["000000000000"] # Spinifex system images

  filter {
    name   = "name"
    values = ["*ubuntu-26.04*", "*ubuntu-24.04*"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }

  filter {
    name   = "root-device-type"
    values = ["ebs"]
  }
}

# ---------------------------------------------------------------------------
# SSH Key Pair
# ---------------------------------------------------------------------------

resource "tls_private_key" "nginx" {
  algorithm = "ED25519"
}

resource "aws_key_pair" "nginx" {
  key_name   = "nginx-demo"
  public_key = tls_private_key.nginx.public_key_openssh
}

resource "local_file" "nginx_pem" {
  filename        = "${path.module}/nginx-demo.pem"
  content         = tls_private_key.nginx.private_key_openssh
  file_permission = "0600"
}

# ---------------------------------------------------------------------------
# VPC
# ---------------------------------------------------------------------------

resource "aws_vpc" "main" {
  cidr_block           = "10.10.0.0/16"
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    Name = "nginx-demo-vpc"
  }
}

# ---------------------------------------------------------------------------
# Internet Gateway — gives the public subnet a route to the internet
# ---------------------------------------------------------------------------

resource "aws_internet_gateway" "igw" {
  vpc_id = aws_vpc.main.id

  tags = {
    Name = "nginx-demo-igw"
  }
}

# ---------------------------------------------------------------------------
# Public Subnet
# ---------------------------------------------------------------------------

resource "aws_subnet" "public" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = "10.10.1.0/24"
  availability_zone       = data.aws_availability_zones.available.names[0]
  map_public_ip_on_launch = true

  tags = {
    Name = "nginx-demo-public"
  }
}

# ---------------------------------------------------------------------------
# Route Table — send 0.0.0.0/0 through the internet gateway
# ---------------------------------------------------------------------------

resource "aws_route_table" "public" {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.igw.id
  }

  tags = {
    Name = "nginx-demo-public-rt"
  }
}

resource "aws_route_table_association" "public" {
  subnet_id      = aws_subnet.public.id
  route_table_id = aws_route_table.public.id
}

# ---------------------------------------------------------------------------
# Security Group — SSH + HTTP inbound, all outbound
# ---------------------------------------------------------------------------

resource "aws_security_group" "web" {
  name        = "nginx-demo-sg"
  description = "Allow SSH and HTTP inbound"
  vpc_id      = aws_vpc.main.id

  ingress {
    description = "SSH"
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    description = "HTTP"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    description = "All outbound"
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "nginx-demo-sg"
  }
}

# ---------------------------------------------------------------------------
# EC2 Instance — Ubuntu 26.04 with Nginx installed via cloud-init
# ---------------------------------------------------------------------------

resource "aws_instance" "nginx" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = var.instance_type

  subnet_id              = aws_subnet.public.id
  vpc_security_group_ids = [aws_security_group.web.id]
  key_name               = aws_key_pair.nginx.key_name

  associate_public_ip_address = true

  user_data_base64 = base64encode(<<-USERDATA
    #!/bin/bash
    set -euo pipefail

    # Install Nginx
    apt-get update -y
    apt-get install -y nginx

    # Write a custom landing page
    cat > /var/www/html/index.html <<'HTML'
    <!DOCTYPE html>
    <html>
    <head><title>Spinifex Demo</title></head>
    <body style="font-family: sans-serif; max-width: 600px; margin: 80px auto;">
      <h1>Hello from Spinifex!</h1>
      <p>This Nginx server was deployed with Terraform on Spinifex infrastructure.</p>
      <hr>
      <p><small>Instance provisioned via cloud-init user-data.</small></p>
    </body>
    </html>
    HTML

    # Ensure Nginx is running
    systemctl enable nginx
    systemctl restart nginx
  USERDATA
  )

  tags = {
    Name = "nginx-demo"
  }
}

# ---------------------------------------------------------------------------
# Outputs
# ---------------------------------------------------------------------------

output "note" {
  value = "EC2 instances can take 30+ seconds to boot after apply. If SSH or HTTP is unreachable, wait and retry."
}

output "instance_id" {
  value = aws_instance.nginx.id
}

output "public_ip" {
  value = aws_instance.nginx.public_ip
}

output "ssh_command" {
  value = "ssh -i nginx-demo.pem ubuntu@${aws_instance.nginx.public_ip}"
}

output "web_url" {
  value = "http://${aws_instance.nginx.public_ip}"
}
```

### Step 2. Deploy

The workbook defaults to `t3.small` (2 vCPU, 2 GiB). On clusters without that type registered, override with `TF_VAR_instance_type` — query what's available with `aws ec2 describe-instance-types`.

```bash
export AWS_PROFILE=spinifex
tofu init
tofu apply
```

### Step 3. Verify

> **Note:** EC2 instances can take 30+ seconds to boot after apply. If SSH or HTTP is unreachable, wait and retry.

After apply completes, use the outputs to test:

```bash
curl http://<public_ip>
ssh -i nginx-demo.pem ubuntu@<public_ip>
```

Open the `web_url` output in your browser to see the Nginx welcome page.

### Clean Up

```bash
tofu destroy
```

## Troubleshooting

### AMI Not Found

Ensure you have imported an Ubuntu 26.04 image. Check available AMIs:

```bash
aws ec2 describe-images --owners 000000000000 --profile spinifex
```

### Provider Connection Refused

Verify Spinifex services are running:

```bash
sudo systemctl status spinifex.target
curl -k https://localhost:9999/
```

### SSH Connection Timeout

Check that the security group allows inbound SSH (port 22) and that the instance has a public IP assigned. Verify the instance is running:

```bash
aws ec2 describe-instances --profile spinifex
```

### Nginx Not Responding

SSH into the instance and check cloud-init logs:

```bash
ssh -i nginx-demo.pem ubuntu@<public_ip>
sudo journalctl -u cloud-init --no-pager
sudo systemctl status nginx
```

---

# EKS Quickstart

URL: https://docs.mulgadc.com/docs/eks-quickstart
Category: Terraform Workbooks
Updated: 2026-08-21
Tags: terraform, eks, kubernetes, iam, vpc, workbook

Stand up a minimal managed Kubernetes cluster with Terraform: a VPC, IAM roles, an EKS cluster, a worker node group, an ECR repository, and a demo web app.

## Overview

Provision a managed Kubernetes cluster on Spinifex with Terraform/OpenTofu — and finish with something you can actually see. As well as the cluster, this workbook creates an **ECR repository**, then deploys the Spinifex-themed demo app onto the cluster and publishes it on the worker's public IP. When `apply` finishes, open the **`demo_url`** output: the page reports the **pod, node, cluster, and region** that answered. Refresh it and the answering pod changes, so even a non-technical viewer can watch Kubernetes scheduling and load-balancing in real time.

Under the hood it keeps things minimal: a VPC with two public subnets, an IAM role for the control plane, an IAM role for the workers with the three managed policies AWS-managed node groups expect, a public-endpoint `aws_eks_cluster`, and a node group sized by **`node_desired_size`** (1 for a single-node demo, or 3 for an HA-shaped cluster). You build the demo image and push it to the ECR repository, then Terraform uses the **Kubernetes provider** (authenticating exactly like `kubectl` does, via `aws eks get-token`) to deploy the Deployment + NodePort Service.

**What you'll learn:**

- Configuring the AWS provider to target Spinifex's `eks`, `ec2`, `iam` and `sts` endpoints
- Creating the EKS cluster and worker IAM roles with faithful managed-policy attachments
- Provisioning an `aws_eks_cluster` and a managed `aws_eks_node_group`
- Pointing the Kubernetes provider at the new cluster and deploying a workload from the same `apply`
- Opening one rule on the auto-managed worker security group to expose a NodePort

**What gets created**

| Resource | Name | Purpose |
|---|---|---|
| VPC | `eks-quickstart-vpc` | Isolated network (10.30.0.0/16) |
| Subnets | `eks-quickstart-subnet-a/-b` | Public subnets for the cluster and worker |
| Internet Gateway | `eks-quickstart-igw` | Egress so the worker can pull the demo image |
| IAM Roles | `eks-quickstart-cluster-role`, `eks-quickstart-node-role` | Control-plane and worker roles |
| ECR Repository | `spinifex-demo` | Holds the demo image the workers pull |
| EKS Cluster | `eks-quickstart` | Public API endpoint, API auth mode, Kubernetes 1.32 |
| Node Group | `default` | `node_desired_size` `t3.medium` worker(s) — 1 or 3 |
| SG Ingress Rule | `eks-quickstart-demo-nodeport` | Opens the demo NodePort on the auto-managed worker SG |
| K8s Deployment | `spinifex-demo` | The themed demo image from ECR, 2 replicas |
| K8s Service | `spinifex-demo` | NodePort publishing the demo on the worker |

**Spinifex specifics**

- The cluster's security groups are **auto-managed** — `vpc_config.security_group_ids` is ignored. To reach a NodePort, this workbook looks the worker SG up by its deterministic name (`eks-cluster-<name>-nodegroup-sg`) and adds a single ingress rule.
- The worker AMI is always Spinifex's `eks-node` image; `ami_type` is recorded but does **not** select the image.
- `authentication_mode` must be `"API"` (the `API_AND_CONFIG_MAP` mode is rejected).
- The workers need outbound internet (here via the IGW) to pull the demo container image.

**Prerequisites:**

- Spinifex installed and running (see [Installing Spinifex](https://docs.mulgadc.com/docs/install))
- The Spinifex `eks-node` image available on the cluster
- OpenTofu or Terraform, plus `kubectl` and the AWS CLI
- Docker, to build and push the demo image to ECR

## Instructions

### Step 1. Get the Template

```bash
git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
cd spinifex-tf
git sparse-checkout set docs/terraform-workbooks
cd docs/terraform-workbooks/eks-quickstart
```

Or create a `main.tf` file and paste the full configuration below.

```hcl
# Example: EKS Quickstart on Spinifex
#
# A minimal managed-Kubernetes cluster that ends in something you can see: a
# VPC, the two IAM roles EKS needs, a one- or three-worker cluster, an ECR
# repository, and the Spinifex-themed demo app deployed onto it by Terraform and
# exposed on the worker's public IP.
#
# Once `apply` finishes, open the demo_url output in a browser. The page reports
# the pod, node, cluster, and region that answered — refresh and it bounces
# between the app's replicas, showing the cluster is really scheduling and
# load-balancing pods.
#
# Demonstrates: VPC + subnets, EKS cluster + node IAM roles, an aws_eks_cluster
# and managed aws_eks_node_group, an ECR repository the workers pull from, the
# Kubernetes provider authenticating to the cluster, and a Deployment + NodePort
# Service reachable from your browser.
#
# Usage:
#   cd spinifex/docs/terraform-workbooks/eks-quickstart
#   export AWS_PROFILE=spinifex
#   tofu init && tofu apply
#   # build + push the demo image to the ECR repo this creates (see README),
#   # then: cd workloads && tofu init && tofu apply
#   # finally open the demo_url output

terraform {
  required_version = ">= 1.6.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 5.40, < 6.0"
    }
  }
}

# ---------------------------------------------------------------------------
# Variables
# ---------------------------------------------------------------------------

variable "region" {
  type    = string
  default = "ap-southeast-2"
}

variable "cluster_name" {
  type    = string
  default = "eks-quickstart"
}

variable "k8s_version" {
  type        = string
  default     = "1.32"
  description = "Kubernetes minor version for the control plane and workers"
}

variable "instance_type" {
  type    = string
  default = "t3.medium"
}

variable "node_desired_size" {
  type        = number
  default     = 1
  description = "Worker count. Use 1 for a single-node demo, or 3 for an HA-shaped cluster."

  validation {
    condition     = var.node_desired_size == 1 || var.node_desired_size == 3
    error_message = "node_desired_size must be 1 or 3."
  }
}

variable "node_port" {
  type        = number
  default     = 30080
  description = "NodePort the demo Service is published on"
}

variable "browse_cidr" {
  type        = string
  default     = "0.0.0.0/0"
  description = "CIDR allowed to reach the demo NodePort; tighten to your own IP in production"
}

variable "spinifex_endpoint" {
  type        = string
  default     = "https://127.0.0.1:9999"
  description = "Spinifex AWS gateway endpoint"
}

# ---------------------------------------------------------------------------
# Providers
# ---------------------------------------------------------------------------

provider "aws" {
  region = var.region

  endpoints {
    ec2 = var.spinifex_endpoint
    iam = var.spinifex_endpoint
    sts = var.spinifex_endpoint
    eks = var.spinifex_endpoint
    ecr = var.spinifex_endpoint
  }

  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true
  skip_region_validation      = true
}

# ---------------------------------------------------------------------------
# Data sources
# ---------------------------------------------------------------------------

data "aws_availability_zones" "available" {
  state = "available"
}

# ---------------------------------------------------------------------------
# VPC + two public subnets (workers get public IPs to pull the demo image)
# ---------------------------------------------------------------------------

resource "aws_vpc" "main" {
  cidr_block           = "10.30.0.0/16"
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    Name = "${var.cluster_name}-vpc"
  }
}

resource "aws_internet_gateway" "igw" {
  vpc_id = aws_vpc.main.id

  tags = {
    Name = "${var.cluster_name}-igw"
  }
}

resource "aws_subnet" "a" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = "10.30.1.0/24"
  availability_zone       = data.aws_availability_zones.available.names[0]
  map_public_ip_on_launch = true

  tags = {
    Name = "${var.cluster_name}-subnet-a"
  }
}

resource "aws_subnet" "b" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = "10.30.2.0/24"
  availability_zone       = data.aws_availability_zones.available.names[0]
  map_public_ip_on_launch = true

  tags = {
    Name = "${var.cluster_name}-subnet-b"
  }
}

resource "aws_route_table" "public" {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.igw.id
  }

  tags = {
    Name = "${var.cluster_name}-public-rt"
  }
}

resource "aws_route_table_association" "a" {
  subnet_id      = aws_subnet.a.id
  route_table_id = aws_route_table.public.id
}

resource "aws_route_table_association" "b" {
  subnet_id      = aws_subnet.b.id
  route_table_id = aws_route_table.public.id
}

# ---------------------------------------------------------------------------
# IAM — EKS cluster role
# ---------------------------------------------------------------------------

resource "aws_iam_role" "cluster" {
  name = "${var.cluster_name}-cluster-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Action    = "sts:AssumeRole"
      Principal = { Service = "eks.amazonaws.com" }
    }]
  })
}

resource "aws_iam_role_policy_attachment" "cluster" {
  role       = aws_iam_role.cluster.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
}

# ---------------------------------------------------------------------------
# IAM — worker node role
# ---------------------------------------------------------------------------

resource "aws_iam_role" "node" {
  name = "${var.cluster_name}-node-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Action    = "sts:AssumeRole"
      Principal = { Service = "ec2.amazonaws.com" }
    }]
  })
}

resource "aws_iam_role_policy_attachment" "node_worker" {
  role       = aws_iam_role.node.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
}

resource "aws_iam_role_policy_attachment" "node_cni" {
  role       = aws_iam_role.node.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
}

resource "aws_iam_role_policy_attachment" "node_ecr" {
  role       = aws_iam_role.node.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
}

# ---------------------------------------------------------------------------
# ECR — repository the workers pull the demo image from
#
# Build and push the demo-app image here before applying the workloads module
# (see ../demo-app/README.md). The node role already carries
# AmazonEC2ContainerRegistryReadOnly, so workers can pull from it.
# ---------------------------------------------------------------------------

resource "aws_ecr_repository" "demo" {
  name = "spinifex-demo"

  # tofu destroy must remove the repo even though it still holds the pushed demo
  # image; without this, DeleteRepository (force=false) fails RepositoryNotEmpty.
  force_delete = true
}

# ---------------------------------------------------------------------------
# EKS cluster — public API endpoint, API (access-entry) auth mode
# ---------------------------------------------------------------------------

resource "aws_eks_cluster" "this" {
  name     = var.cluster_name
  role_arn = aws_iam_role.cluster.arn
  version  = var.k8s_version

  vpc_config {
    subnet_ids              = [aws_subnet.a.id, aws_subnet.b.id]
    endpoint_public_access  = true
    endpoint_private_access = false
  }

  access_config {
    authentication_mode                         = "API"
    bootstrap_cluster_creator_admin_permissions = true
  }

  depends_on = [aws_iam_role_policy_attachment.cluster]

  tags = {
    Name = var.cluster_name
  }
}

# ---------------------------------------------------------------------------
# Managed node group — one worker
# ---------------------------------------------------------------------------

resource "aws_eks_node_group" "default" {
  cluster_name    = aws_eks_cluster.this.name
  node_group_name = "default"
  node_role_arn   = aws_iam_role.node.arn
  subnet_ids      = [aws_subnet.a.id, aws_subnet.b.id]

  scaling_config {
    desired_size = var.node_desired_size
    min_size     = var.node_desired_size
    max_size     = var.node_desired_size * 2
  }

  instance_types = [var.instance_type]
  ami_type       = "AL2_x86_64"

  depends_on = [
    aws_iam_role_policy_attachment.node_worker,
    aws_iam_role_policy_attachment.node_cni,
    aws_iam_role_policy_attachment.node_ecr,
  ]

  tags = {
    Name = "${var.cluster_name}-default"
  }
}

# ---------------------------------------------------------------------------
# Open the NodePort on the auto-managed nodegroup SG
#
# Spinifex creates the worker SG itself (vpc_config.security_group_ids is
# ignored) and admits only intra-cluster traffic. To reach the demo NodePort
# from a browser, look the SG up by its deterministic name and add one rule.
# ---------------------------------------------------------------------------

data "aws_security_group" "nodegroup" {
  filter {
    name   = "group-name"
    values = ["eks-cluster-${var.cluster_name}-nodegroup-sg"]
  }

  filter {
    name   = "vpc-id"
    values = [aws_vpc.main.id]
  }

  depends_on = [aws_eks_node_group.default]
}

resource "aws_vpc_security_group_ingress_rule" "nodeport" {
  security_group_id = data.aws_security_group.nodegroup.id
  cidr_ipv4         = var.browse_cidr
  from_port         = var.node_port
  to_port           = var.node_port
  ip_protocol       = "tcp"

  tags = {
    Name = "${var.cluster_name}-demo-nodeport"
  }
}

# ---------------------------------------------------------------------------
# Discover the worker's public IP for the demo URL
# ---------------------------------------------------------------------------

data "aws_instances" "workers" {
  instance_tags = {
    "spinifex:eks-cluster" = aws_eks_cluster.this.name
  }

  depends_on = [aws_eks_node_group.default]
}

# ---------------------------------------------------------------------------
# Outputs
# ---------------------------------------------------------------------------

output "cluster_name" {
  value = aws_eks_cluster.this.name
}

output "region" {
  value = var.region
}

output "node_port" {
  value = var.node_port
}

output "node_desired_size" {
  value = var.node_desired_size
}

output "ecr_repository_url" {
  value       = aws_ecr_repository.demo.repository_url
  description = "Push the demo-app image here, then apply the workloads module"
}

output "demo_url" {
  value       = "http://${data.aws_instances.workers.public_ips[0]}:${var.node_port}"
  description = "Open in a browser; refresh to see different pods answer"
}

output "update_kubeconfig" {
  value = "aws eks update-kubeconfig --name ${aws_eks_cluster.this.name} --region ${var.region}"
}
```

### Step 2. Deploy

The cluster and the demo app are **two root modules with separate state**. Apply the cluster first:

```bash
export AWS_PROFILE=spinifex
tofu init
tofu apply
```

This creates the cluster (which bootstraps a control-plane VM and brings up k3s — a few minutes in `CREATING`), launches the worker(s), creates the `spinifex-demo` ECR repository, and opens the NodePort.

Set the worker count with `node_desired_size` (`1` for a single node, `3` for an HA-shaped cluster):

```bash
tofu apply -var node_desired_size=3
```

Once the cluster is `ACTIVE`, **build and push the demo image** to the ECR repository this created (full commands in [`docs/terraform-workbooks/demo-app`](https://github.com/mulgadc/spinifex/blob/main/docs/terraform-workbooks/demo-app/README.md)):

```bash
cd ../demo-app
REGISTRY=$(cd ../eks-quickstart && tofu output -raw ecr_repository_url)
REGISTRY_HOST=${REGISTRY%%/*}
aws ecr get-login-password | docker login --username AWS --password-stdin "$REGISTRY_HOST"
docker build -t "${REGISTRY}:latest" .
docker push "${REGISTRY}:latest"
cd ../eks-quickstart
```

Then deploy the demo app from the nested `workloads/` module — it defaults `demo_image` to the parent's ECR repository at `:latest`:

```bash
cd workloads
tofu init
tofu apply
```

> **Why two modules?** The Kubernetes provider in `workloads/` reads the cluster endpoint from a live `data "aws_eks_cluster"` source, so it's only ever configured while the cluster exists. Keeping it out of the cluster module means `destroy` never tries to refresh a workload against a cluster that's already gone — the failure mode where the provider falls back to `http://localhost:80` and reports `connection refused`. Always destroy `workloads/` before the cluster.

> **Same profile for the Kubernetes provider.** The Kubernetes provider authenticates by shelling out to `aws eks get-token`, which has to reach the Spinifex STS endpoint. Keep `AWS_PROFILE=spinifex` exported for both applies.

```hcl
# Demo workload for eks-quickstart — separate root module / state.
#
# Kept apart from the cluster root module on purpose: the kubernetes provider
# below reads the cluster endpoint from a *live* data source, so it is only ever
# configured while the cluster exists. Destroy this module before the parent and
# the provider never falls back to localhost.
#
# Usage:
#   cd spinifex/docs/terraform-workbooks/eks-quickstart
#   tofu init && tofu apply                    # parent: cluster + infra
#   cd workloads && tofu init && tofu apply    # this module: demo app
#   # teardown is the reverse — destroy here first, then the parent.

terraform {
  required_version = ">= 1.6.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 5.40, < 6.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = ">= 2.20"
    }
  }
}

variable "spinifex_endpoint" {
  type        = string
  default     = "https://127.0.0.1:9999"
  description = "Spinifex AWS gateway endpoint"
}

variable "replicas" {
  type        = number
  default     = 2
  description = "Demo app replicas; refresh the page to see requests land on different pods"
}

variable "demo_image" {
  type        = string
  default     = ""
  description = "Demo image ref. Defaults to the parent's ECR repository URL at :latest."
}

provider "aws" {
  region = data.terraform_remote_state.infra.outputs.region

  endpoints {
    ec2 = var.spinifex_endpoint
    iam = var.spinifex_endpoint
    sts = var.spinifex_endpoint
    eks = var.spinifex_endpoint
  }

  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true
  skip_region_validation      = true
}

# Cluster identity comes from the parent module's state; the live endpoint/CA
# come from a data source so the provider is only configured while the cluster
# is up.
data "terraform_remote_state" "infra" {
  backend = "local"

  config = {
    path = "../terraform.tfstate"
  }
}

locals {
  cluster_name = data.terraform_remote_state.infra.outputs.cluster_name
  region       = data.terraform_remote_state.infra.outputs.region
  node_port    = data.terraform_remote_state.infra.outputs.node_port
  demo_image   = var.demo_image != "" ? var.demo_image : "${data.terraform_remote_state.infra.outputs.ecr_repository_url}:latest"
}

data "aws_eks_cluster" "this" {
  name = local.cluster_name
}

# Authenticates with the same `aws eks get-token` exec flow the generated
# kubeconfig uses, so the Kubernetes provider can deploy the demo app.
provider "kubernetes" {
  host                   = data.aws_eks_cluster.this.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)

  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "aws"
    args        = ["eks", "get-token", "--cluster-name", local.cluster_name, "--region", local.region]
  }
}

# The Spinifex-themed demo app reports the pod, node, cluster, and region that
# served the request. With multiple replicas, refreshing the demo_url alternates
# between them. The pod and node names come from the downward API.
resource "kubernetes_deployment_v1" "demo" {
  metadata {
    name      = "spinifex-demo"
    namespace = "default"
    labels    = { app = "spinifex-demo" }
  }

  spec {
    replicas = var.replicas

    selector {
      match_labels = { app = "spinifex-demo" }
    }

    template {
      metadata {
        labels = { app = "spinifex-demo" }
      }

      spec {
        container {
          name  = "spinifex-demo"
          image = local.demo_image

          port {
            container_port = 8080
          }

          env {
            name = "POD_NAME"
            value_from {
              field_ref {
                field_path = "metadata.name"
              }
            }
          }

          env {
            name = "NODE_NAME"
            value_from {
              field_ref {
                field_path = "spec.nodeName"
              }
            }
          }

          env {
            name = "POD_NAMESPACE"
            value_from {
              field_ref {
                field_path = "metadata.namespace"
              }
            }
          }

          env {
            name  = "CLUSTER_NAME"
            value = local.cluster_name
          }

          env {
            name  = "AWS_REGION"
            value = local.region
          }

          readiness_probe {
            http_get {
              path = "/healthz"
              port = 8080
            }
            initial_delay_seconds = 3
            period_seconds        = 10
          }
        }
      }
    }
  }
}

resource "kubernetes_service_v1" "demo" {
  metadata {
    name      = "spinifex-demo"
    namespace = "default"
  }

  spec {
    selector = { app = "spinifex-demo" }
    type     = "NodePort"

    port {
      port        = 80
      target_port = 8080
      node_port   = local.node_port
    }
  }

  depends_on = [kubernetes_deployment_v1.demo]
}
```

### Step 3. Open the Demo

Run from the cluster module directory (`cd ..` if you're still in `workloads/`):

```bash
tofu output demo_url
```

Open that URL in a browser. You'll see the Spinifex-themed page reporting the **pod, node, cluster, and region** that handled the request. Refresh a few times — with two replicas, the pod name alternates, demonstrating that the Service is load-balancing across the cluster.

### Step 4. Inspect with kubectl (optional)

```bash
aws eks update-kubeconfig --name eks-quickstart --region ap-southeast-2
kubectl get nodes
kubectl get pods -o wide
```

`kubectl get pods -o wide` shows the demo pods and which node each landed on.

### Cleanup

Destroy in reverse — the demo app first (while the cluster is still up), then the cluster:

```bash
cd workloads
tofu destroy
cd ..
tofu destroy
```

## Troubleshooting

### Demo URL Doesn't Load

The page is served from a NodePort on the worker's public IP. Work through the chain:

```bash
# Is the worker running with a public IP?
aws ec2 describe-instances --filters "Name=tag:spinifex:eks-cluster,Values=eks-quickstart" \
  --query 'Reservations[].Instances[].[InstanceId,State.Name,PublicIpAddress]' --output text

# Did the demo pods roll out?
kubectl get pods -o wide

# Is the NodePort rule on the worker SG?
aws ec2 describe-security-groups --filters "Name=group-name,Values=eks-cluster-eks-quickstart-nodegroup-sg" \
  --query 'SecurityGroups[0].IpPermissions'
```

If the pods are `Pending`, the worker may not be `Ready` yet — give it a moment. If they're `ImagePullBackOff`, the worker can't pull from ECR: confirm you built and pushed the image (`tofu output ecr_repository_url`), that the IGW route and the worker's public IP are in place, and that the node role carries `AmazonEC2ContainerRegistryReadOnly`.

### kubernetes provider: connection refused / Unauthorized

The provider runs `aws eks get-token` against the Spinifex STS endpoint. Confirm the AWS CLI is still pointed at Spinifex and the cluster is `ACTIVE`:

```bash
aws sts get-caller-identity
aws eks describe-cluster --name eks-quickstart --query 'cluster.status'
```

If the cluster was still `CREATING` when the provider first tried to connect, just re-run `tofu apply`.

### Cluster Stuck in CREATING

Control-plane bootstrap takes a few minutes. Confirm the underlying VM came up:

```bash
aws eks describe-cluster --name eks-quickstart --query 'cluster.status'
aws ec2 describe-instances --profile spinifex
```

### Provider Connection Refused

```bash
sudo systemctl status spinifex.target
curl -k https://localhost:9999/
```

---

# EKS HTTPS Ingress (LBC + ACM)

URL: https://docs.mulgadc.com/docs/eks-https-ingress
Category: Terraform Workbooks
Updated: 2026-08-21
Tags: terraform, eks, kubernetes, ingress, alb, acm, https, workbook

Serve a demo app over HTTPS on EKS using the AWS Load Balancer Controller addon and an ACM certificate, with an internet-facing ALB built from an Ingress.

## Overview

This is the second rung of the EKS ladder. It takes the cluster + demo app from [EKS Quickstart](https://docs.mulgadc.com/docs/eks-quickstart) and publishes the app the way you would in production: a Kubernetes `Ingress` reconciled by the **AWS Load Balancer Controller (LBC)** into an internet-facing **Application Load Balancer** that terminates TLS with an **ACM** certificate.

To make room for an internet-facing ALB with private workers, the network grows from the quickstart's two public subnets to a public/private split: the ALB and a **NAT gateway** sit in the public subnets, the workers sit in the private subnets and reach ECR through the NAT. The cluster is tagged `spinifex.io/managed-ingress = "false"`, which disables K3s' built-in traefik/servicelb so the LBC owns ingress.

The LBC is installed as a managed **addon** (`aws_eks_addon`). Spinifex wires the controller's AWS credentials and the cluster's ELB-eligible subnets at the node level and injects those subnets into the `alb` `IngressClassParams`, so the Ingress needs **no IRSA role and no subnet tags** — just `ingressClassName: alb` and a handful of annotations.

**What you'll learn:**

- Installing the `aws-load-balancer-controller` addon and letting it own ingress
- Disabling the built-in K3s ingress with the `spinifex.io/managed-ingress` tag
- Importing a self-signed certificate into ACM and attaching it to an ALB listener
- Driving an ALB entirely from a Kubernetes `Ingress` (`ingressClassName: alb`)
- A public/private VPC with a NAT gateway for private workers

**What gets created**

| Resource | Name | Purpose |
|---|---|---|
| VPC | `eks-https-vpc` | Isolated network (10.31.0.0/16) |
| Subnets | `eks-https-public-a/-b`, `eks-https-private-a/-b` | Public (ALB + NAT) and private (workers) |
| Internet + NAT Gateway | `eks-https-igw`, `eks-https-nat` | Public egress; private-worker egress to ECR |
| IAM Roles | `eks-https-cluster-role`, `eks-https-node-role` | Control-plane and worker roles |
| ECR Repository | `spinifex-demo` | Holds the demo image the workers pull |
| EKS Cluster | `eks-https` | Public + private endpoints; `managed-ingress=false` |
| Node Group | `workers` | `node_desired_size` `t3.large` worker(s) — 1 or 3 |
| Addon | `aws-load-balancer-controller` | Provisions the ALB from the Ingress |
| ACM Certificate | `eks-https-ingress` | Self-signed, imported; attached to the HTTPS listener |
| SG Ingress Rule | `eks-https-alb-nodeport` | Admits the VPC CIDR to the workers' NodePort |
| K8s Deployment + Service | `spinifex-demo` | Themed demo image, 2 replicas, NodePort |
| K8s Ingress | `spinifex-demo` | `ingressClassName: alb`, HTTPS via the ACM cert |

**Spinifex specifics**

- **The LBC needs no IRSA or subnet tagging here.** Credentials and ELB-eligible subnets are injected by Spinifex at the node level and into the `alb` `IngressClassParams`. The Ingress just sets `ingressClassName: alb` plus the `alb.ingress.kubernetes.io/*` annotations.
- **`spinifex.io/managed-ingress = "false"`** disables the K3s built-in traefik/servicelb so the LBC owns ingress, matching AWS parity. Omit the tag (or set it `true`) and the built-in ingress stays on.
- The ALB targets the workers' **NodePort** (`target-type: instance`). The one SG rule admits the VPC CIDR on that port so the in-VPC ALB can reach the workers; the worker SG is otherwise auto-managed.
- Spinifex ACM supports **ImportCertificate only** — the cert is generated by the `tls` provider and imported, not requested.
- **Leave `addon_version` unset** (see the [quickstart/addons notes](https://docs.mulgadc.com/docs/eks-quickstart)) — the AWS provider and the Spinifex catalog disagree on the version string format, so let the server choose.

**Prerequisites:**

- Spinifex installed and running, with the `eks-node` image (carrying the LBC bundle) available
- OpenTofu or Terraform, plus `kubectl`, the AWS CLI, and Docker
- The demo image built and pushed to ECR (see [`docs/terraform-workbooks/demo-app`](https://github.com/mulgadc/spinifex/blob/main/docs/terraform-workbooks/demo-app/README.md))

## Instructions

### Step 1. Get the Template

```bash
git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
cd spinifex-tf
git sparse-checkout set docs/terraform-workbooks
cd docs/terraform-workbooks/eks-https-ingress
```

Or create a `main.tf` file and paste the full configuration below.

```hcl
# Example: EKS HTTPS Ingress via the AWS Load Balancer Controller + ACM
#
# Builds on eks-quickstart. Instead of poking a NodePort open on the worker, this
# workbook serves the Spinifex-themed demo app over HTTPS through an Application
# Load Balancer that the AWS Load Balancer Controller (LBC) provisions from a
# Kubernetes Ingress — the same pattern you'd use on AWS EKS.
#
# What it adds over the quickstart:
#   * Public subnets for the ALB + a NAT gateway, private subnets for the workers.
#   * The aws-load-balancer-controller addon, installed through the EKS API.
#   * The cluster tag spinifex.io/managed-ingress = "false", which disables K3s'
#     built-in traefik/servicelb so the LBC owns ingress (AWS parity).
#   * A self-signed certificate imported into ACM and attached to the ALB's HTTPS
#     listener via an Ingress annotation.
#
# The workloads/ module then creates a Deployment, a NodePort Service, and an
# Ingress (ingressClassName: alb). The LBC reconciles that Ingress into an
# internet-facing ALB that terminates TLS with the ACM cert and forwards to the
# workers' NodePort. The cluster's ELB-eligible subnets are injected by Spinifex
# into the alb IngressClassParams, so the Ingress needs no subnet annotations.
#
# Usage:
#   cd spinifex/docs/terraform-workbooks/eks-https-ingress
#   export AWS_PROFILE=spinifex
#   tofu init && tofu apply
#   # build + push the demo image to the ECR repo this creates (see README),
#   # then: cd workloads && tofu init && tofu apply
#   # finally: kubectl get ingress spinifex-demo -o wide  → open https://<address>

terraform {
  required_version = ">= 1.6.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 5.40, < 6.0"
    }
    tls = {
      source  = "hashicorp/tls"
      version = ">= 4.0"
    }
  }
}

# ---------------------------------------------------------------------------
# Variables
# ---------------------------------------------------------------------------

variable "region" {
  type    = string
  default = "ap-southeast-2"
}

variable "cluster_name" {
  type    = string
  default = "eks-https"
}

variable "k8s_version" {
  type    = string
  default = "1.32"
}

variable "node_instance_type" {
  type    = string
  default = "t3.medium"
}

variable "node_desired_size" {
  type        = number
  default     = 1
  description = "Worker count. Use 1 for a single-node demo, or 3 for an HA-shaped cluster."

  validation {
    condition     = var.node_desired_size == 1 || var.node_desired_size == 3
    error_message = "node_desired_size must be 1 or 3."
  }
}

variable "node_port" {
  type        = number
  default     = 30080
  description = "NodePort the demo Service is published on and the ALB forwards to"
}

variable "cert_common_name" {
  type    = string
  default = "eks-https.spinifex.local"
}

variable "api_public_access_cidr" {
  type        = string
  default     = "0.0.0.0/0"
  description = "CIDR allowed to reach the public Kubernetes API endpoint; tighten in production"
}

variable "spinifex_endpoint" {
  type    = string
  default = "https://127.0.0.1:9999"
}

# ---------------------------------------------------------------------------
# Providers
# ---------------------------------------------------------------------------

provider "aws" {
  region = var.region

  endpoints {
    ec2 = var.spinifex_endpoint
    iam = var.spinifex_endpoint
    sts = var.spinifex_endpoint
    eks = var.spinifex_endpoint
    ecr = var.spinifex_endpoint
    acm = var.spinifex_endpoint
  }

  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true
  skip_region_validation      = true
}

data "aws_availability_zones" "available" {
  state = "available"
}

# ---------------------------------------------------------------------------
# VPC — public subnets for the ALB + NAT, private subnets for the workers
# ---------------------------------------------------------------------------

resource "aws_vpc" "main" {
  cidr_block           = "10.31.0.0/16"
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    Name = "${var.cluster_name}-vpc"
  }
}

resource "aws_internet_gateway" "igw" {
  vpc_id = aws_vpc.main.id

  tags = {
    Name = "${var.cluster_name}-igw"
  }
}

resource "aws_subnet" "public_a" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = "10.31.1.0/24"
  availability_zone       = data.aws_availability_zones.available.names[0]
  map_public_ip_on_launch = true

  tags = {
    Name = "${var.cluster_name}-public-a"
  }
}

resource "aws_subnet" "public_b" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = "10.31.2.0/24"
  availability_zone       = data.aws_availability_zones.available.names[0]
  map_public_ip_on_launch = true

  tags = {
    Name = "${var.cluster_name}-public-b"
  }
}

resource "aws_subnet" "private_a" {
  vpc_id            = aws_vpc.main.id
  cidr_block        = "10.31.11.0/24"
  availability_zone = data.aws_availability_zones.available.names[0]

  tags = {
    Name = "${var.cluster_name}-private-a"
  }
}

resource "aws_subnet" "private_b" {
  vpc_id            = aws_vpc.main.id
  cidr_block        = "10.31.12.0/24"
  availability_zone = data.aws_availability_zones.available.names[0]

  tags = {
    Name = "${var.cluster_name}-private-b"
  }
}

resource "aws_route_table" "public" {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.igw.id
  }

  tags = {
    Name = "${var.cluster_name}-public-rt"
  }
}

resource "aws_route_table_association" "public_a" {
  subnet_id      = aws_subnet.public_a.id
  route_table_id = aws_route_table.public.id
}

resource "aws_route_table_association" "public_b" {
  subnet_id      = aws_subnet.public_b.id
  route_table_id = aws_route_table.public.id
}

# NAT gateway gives the private workers outbound internet to pull the demo image
# from ECR. The workers still join the cluster over the in-VPC private endpoint.
resource "aws_eip" "nat" {
  domain = "vpc"

  tags = {
    Name = "${var.cluster_name}-nat-eip"
  }
}

resource "aws_nat_gateway" "nat" {
  allocation_id = aws_eip.nat.id
  subnet_id     = aws_subnet.public_a.id

  depends_on = [aws_internet_gateway.igw]

  tags = {
    Name = "${var.cluster_name}-nat"
  }
}

resource "aws_route_table" "private" {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block     = "0.0.0.0/0"
    nat_gateway_id = aws_nat_gateway.nat.id
  }

  tags = {
    Name = "${var.cluster_name}-private-rt"
  }
}

resource "aws_route_table_association" "private_a" {
  subnet_id      = aws_subnet.private_a.id
  route_table_id = aws_route_table.private.id
}

resource "aws_route_table_association" "private_b" {
  subnet_id      = aws_subnet.private_b.id
  route_table_id = aws_route_table.private.id
}

# ---------------------------------------------------------------------------
# IAM — cluster role
# ---------------------------------------------------------------------------

resource "aws_iam_role" "cluster" {
  name = "${var.cluster_name}-cluster-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Action    = "sts:AssumeRole"
      Principal = { Service = "eks.amazonaws.com" }
    }]
  })
}

resource "aws_iam_role_policy_attachment" "cluster" {
  role       = aws_iam_role.cluster.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
}

# ---------------------------------------------------------------------------
# IAM — node role
# ---------------------------------------------------------------------------

resource "aws_iam_role" "node" {
  name = "${var.cluster_name}-node-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Action    = "sts:AssumeRole"
      Principal = { Service = "ec2.amazonaws.com" }
    }]
  })
}

resource "aws_iam_role_policy_attachment" "node_worker" {
  role       = aws_iam_role.node.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
}

resource "aws_iam_role_policy_attachment" "node_cni" {
  role       = aws_iam_role.node.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
}

resource "aws_iam_role_policy_attachment" "node_ecr" {
  role       = aws_iam_role.node.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
}

# The AWS Load Balancer Controller runs with the node's instance-profile
# credentials (Spinifex wires creds at the node level, not IRSA — see the addon
# block below), so the permissions it needs to manage ALBs must live on the node
# role. This mirrors the upstream AWSLoadBalancerControllerIAMPolicy, trimmed to
# the actions an ALB Ingress exercises (Shield/WAF/Cognito are disabled on the
# controller). Resources are "*" since Spinifex evaluates grants by action.
# A customer-managed policy + attachment is used (Spinifex implements
# CreatePolicy/AttachRolePolicy, not inline PutRolePolicy).
resource "aws_iam_policy" "node_lbc" {
  name = "${var.cluster_name}-node-lbc"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "ec2:DescribeAccountAttributes",
          "ec2:DescribeAddresses",
          "ec2:DescribeAvailabilityZones",
          "ec2:DescribeInternetGateways",
          "ec2:DescribeVpcs",
          "ec2:DescribeSubnets",
          "ec2:DescribeSecurityGroups",
          "ec2:DescribeInstances",
          "ec2:DescribeNetworkInterfaces",
          "ec2:DescribeTags",
          "ec2:GetCoipPoolUsage",
          "ec2:DescribeCoipPools",
          "ec2:CreateSecurityGroup",
          "ec2:CreateTags",
          "ec2:DeleteTags",
          "ec2:AuthorizeSecurityGroupIngress",
          "ec2:RevokeSecurityGroupIngress",
          "ec2:DeleteSecurityGroup",
          "elasticloadbalancing:*",
          "acm:ListCertificates",
          "acm:DescribeCertificate",
          "acm:GetCertificate",
          "iam:CreateServiceLinkedRole",
          "iam:ListServerCertificates",
          "iam:GetServerCertificate",
          "tag:GetResources",
          "tag:TagResources",
          "wafv2:GetWebACLForResource",
          "shield:GetSubscriptionState"
        ]
        Resource = "*"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "node_lbc" {
  role       = aws_iam_role.node.name
  policy_arn = aws_iam_policy.node_lbc.arn
}

# ---------------------------------------------------------------------------
# ECR — repository the workers pull the demo image from
# ---------------------------------------------------------------------------

resource "aws_ecr_repository" "demo" {
  name = "spinifex-demo"

  # tofu destroy must remove the repo even though it still holds the pushed demo
  # image; without this, DeleteRepository (force=false) fails RepositoryNotEmpty.
  force_delete = true
}

# ---------------------------------------------------------------------------
# EKS cluster — public + private endpoints, LBC-owned ingress
#
# The spinifex.io/managed-ingress = "false" tag disables K3s' built-in
# traefik/servicelb so the AWS Load Balancer Controller owns ingress, matching
# how ingress works on AWS EKS.
# ---------------------------------------------------------------------------

resource "aws_eks_cluster" "this" {
  name     = var.cluster_name
  role_arn = aws_iam_role.cluster.arn
  version  = var.k8s_version

  vpc_config {
    subnet_ids              = [aws_subnet.private_a.id, aws_subnet.private_b.id]
    endpoint_public_access  = true
    endpoint_private_access = true
    public_access_cidrs     = [var.api_public_access_cidr]
  }

  access_config {
    authentication_mode                         = "API"
    bootstrap_cluster_creator_admin_permissions = true
  }

  depends_on = [aws_iam_role_policy_attachment.cluster]

  tags = {
    Name                          = var.cluster_name
    "spinifex.io/managed-ingress" = "false"
  }
}

# ---------------------------------------------------------------------------
# Managed node group — workers in the private subnets
# ---------------------------------------------------------------------------

resource "aws_eks_node_group" "workers" {
  cluster_name    = aws_eks_cluster.this.name
  node_group_name = "workers"
  node_role_arn   = aws_iam_role.node.arn
  subnet_ids      = [aws_subnet.private_a.id, aws_subnet.private_b.id]

  scaling_config {
    desired_size = var.node_desired_size
    min_size     = var.node_desired_size
    max_size     = var.node_desired_size * 2
  }

  instance_types = [var.node_instance_type]
  ami_type       = "AL2_x86_64"

  depends_on = [
    aws_iam_role_policy_attachment.node_worker,
    aws_iam_role_policy_attachment.node_cni,
    aws_iam_role_policy_attachment.node_ecr,
  ]

  tags = {
    Name = "${var.cluster_name}-workers"
  }
}

# ---------------------------------------------------------------------------
# Addon — AWS Load Balancer Controller
#
# Spinifex wires the controller's AWS credentials and ELB-eligible subnets at the
# node level, so no IRSA role or subnet tagging is needed here. addon_version is
# omitted: the AWS provider demands a v-prefixed version that the catalog rejects,
# so let Spinifex default to its catalog version.
# ---------------------------------------------------------------------------

resource "aws_eks_addon" "lbc" {
  cluster_name                = aws_eks_cluster.this.name
  addon_name                  = "aws-load-balancer-controller"
  resolve_conflicts_on_create = "OVERWRITE"

  depends_on = [aws_eks_node_group.workers]

  tags = {
    Name = "${var.cluster_name}-lbc"
  }
}

# ---------------------------------------------------------------------------
# TLS — self-signed certificate imported into ACM
#
# Spinifex ACM supports ImportCertificate (not RequestCertificate), so the cert
# is generated locally by the tls provider and imported. The workloads Ingress
# attaches it to the ALB's HTTPS listener by ARN.
# ---------------------------------------------------------------------------

resource "tls_private_key" "ingress" {
  algorithm = "RSA"
  rsa_bits  = 2048
}

resource "tls_self_signed_cert" "ingress" {
  private_key_pem = tls_private_key.ingress.private_key_pem

  subject {
    common_name  = var.cert_common_name
    organization = "Spinifex EKS Demo"
  }

  dns_names             = [var.cert_common_name]
  validity_period_hours = 8760
  early_renewal_hours   = 720

  allowed_uses = [
    "key_encipherment",
    "digital_signature",
    "server_auth",
  ]
}

resource "aws_acm_certificate" "ingress" {
  private_key      = tls_private_key.ingress.private_key_pem
  certificate_body = tls_self_signed_cert.ingress.cert_pem

  tags = {
    Name = "${var.cluster_name}-ingress"
  }
}

# ---------------------------------------------------------------------------
# Let the ALB reach the workers' NodePort
#
# Spinifex auto-manages the nodegroup SG and admits only intra-cluster traffic.
# The LBC-provisioned ALB lives in this VPC, so admit the VPC CIDR on the
# NodePort. Look the SG up by its deterministic name and add one rule.
# ---------------------------------------------------------------------------

data "aws_security_group" "nodegroup" {
  filter {
    name   = "group-name"
    values = ["eks-cluster-${var.cluster_name}-nodegroup-sg"]
  }

  filter {
    name   = "vpc-id"
    values = [aws_vpc.main.id]
  }

  depends_on = [aws_eks_node_group.workers]
}

resource "aws_vpc_security_group_ingress_rule" "nodeport_from_vpc" {
  security_group_id = data.aws_security_group.nodegroup.id
  cidr_ipv4         = aws_vpc.main.cidr_block
  from_port         = var.node_port
  to_port           = var.node_port
  ip_protocol       = "tcp"

  tags = {
    Name = "${var.cluster_name}-alb-nodeport"
  }
}

# ---------------------------------------------------------------------------
# Outputs
# ---------------------------------------------------------------------------

output "cluster_name" {
  value = aws_eks_cluster.this.name
}

output "region" {
  value = var.region
}

output "node_port" {
  value = var.node_port
}

output "node_desired_size" {
  value = var.node_desired_size
}

output "certificate_arn" {
  value = aws_acm_certificate.ingress.arn
}

output "ecr_repository_url" {
  value       = aws_ecr_repository.demo.repository_url
  description = "Push the demo-app image here, then apply the workloads module"
}

output "ingress_address_hint" {
  value = "After applying workloads: kubectl get ingress spinifex-demo -o jsonpath='{.status.loadBalancer.ingress[0].hostname}{\"\\n\"}' — then open https://<that-address> (self-signed cert: curl -k)."
}

output "update_kubeconfig" {
  value = "aws eks update-kubeconfig --name ${aws_eks_cluster.this.name} --region ${var.region}"
}
```

### Step 2. Deploy the Cluster

The cluster and the demo app are **two root modules with separate state**. Apply the cluster first (set `node_desired_size=3` for an HA-shaped cluster):

```bash
export AWS_PROFILE=spinifex
tofu init
tofu apply
```

This brings up the cluster, the workers, the NAT gateway, the LBC addon, the ACM cert, and the `spinifex-demo` ECR repository. Give it a few minutes to reach `ACTIVE` and for the LBC addon to report healthy.

### Step 3. Build and Push the Demo Image

```bash
cd ../demo-app
REGISTRY=$(cd ../eks-https-ingress && tofu output -raw ecr_repository_url)
REGISTRY_HOST=${REGISTRY%%/*}
aws ecr get-login-password | docker login --username AWS --password-stdin "$REGISTRY_HOST"
docker build -t "${REGISTRY}:latest" .
docker push "${REGISTRY}:latest"
cd ../eks-https-ingress
```

### Step 4. Deploy the App and Ingress

```bash
cd workloads
tofu init
tofu apply
```

> **Why two modules?** The Kubernetes provider in `workloads/` reads the cluster endpoint from a live `data "aws_eks_cluster"` source, so it's only configured while the cluster exists. Always destroy `workloads/` before the cluster, or the provider falls back to `http://localhost:80` and reports `connection refused`. Keep `AWS_PROFILE=spinifex` exported for both applies.

```hcl
# Demo workload for eks-https-ingress — separate root module / state.
#
# Kept apart from the cluster root module on purpose: the kubernetes provider
# below reads the cluster endpoint from a *live* data source, so it is only ever
# configured while the cluster exists. Destroy this module before the parent and
# the provider never falls back to localhost.
#
# Creates the Spinifex-themed demo Deployment, a NodePort Service, and an Ingress
# (ingressClassName: alb). The AWS Load Balancer Controller reconciles the
# Ingress into an internet-facing ALB that terminates TLS with the imported ACM
# certificate and forwards to the workers' NodePort.
#
# Usage:
#   cd spinifex/docs/terraform-workbooks/eks-https-ingress
#   tofu init && tofu apply                    # parent: cluster + LBC + ACM
#   cd workloads && tofu init && tofu apply    # this module: demo app + Ingress
#   # teardown is the reverse — destroy here first, then the parent.

terraform {
  required_version = ">= 1.6.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 5.40, < 6.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = ">= 2.20"
    }
  }
}

variable "spinifex_endpoint" {
  type    = string
  default = "https://127.0.0.1:9999"
}

variable "replicas" {
  type    = number
  default = 2
}

variable "demo_image" {
  type        = string
  default     = ""
  description = "Demo image ref. Defaults to the parent's ECR repository URL at :latest."
}

variable "inbound_cidr" {
  type        = string
  default     = "0.0.0.0/0"
  description = "CIDR allowed to reach the ALB HTTPS listener"
}

provider "aws" {
  region = data.terraform_remote_state.infra.outputs.region

  endpoints {
    ec2 = var.spinifex_endpoint
    iam = var.spinifex_endpoint
    sts = var.spinifex_endpoint
    eks = var.spinifex_endpoint
  }

  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true
  skip_region_validation      = true
}

# Cluster identity comes from the parent module's state; the live endpoint/CA
# come from a data source so the provider is only configured while the cluster
# is up.
data "terraform_remote_state" "infra" {
  backend = "local"

  config = {
    path = "../terraform.tfstate"
  }
}

locals {
  cluster_name = data.terraform_remote_state.infra.outputs.cluster_name
  region       = data.terraform_remote_state.infra.outputs.region
  node_port    = data.terraform_remote_state.infra.outputs.node_port
  cert_arn     = data.terraform_remote_state.infra.outputs.certificate_arn
  demo_image   = var.demo_image != "" ? var.demo_image : "${data.terraform_remote_state.infra.outputs.ecr_repository_url}:latest"
}

data "aws_eks_cluster" "this" {
  name = local.cluster_name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.this.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)

  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "aws"
    args        = ["eks", "get-token", "--cluster-name", local.cluster_name, "--region", local.region]
  }
}

# The Spinifex-themed demo app reports the pod, node, cluster, and region that
# served the request. The pod and node names come from the downward API.
resource "kubernetes_deployment_v1" "demo" {
  metadata {
    name      = "spinifex-demo"
    namespace = "default"
    labels    = { app = "spinifex-demo" }
  }

  spec {
    replicas = var.replicas

    selector {
      match_labels = { app = "spinifex-demo" }
    }

    template {
      metadata {
        labels = { app = "spinifex-demo" }
      }

      spec {
        container {
          name  = "spinifex-demo"
          image = local.demo_image

          port {
            container_port = 8080
          }

          env {
            name = "POD_NAME"
            value_from {
              field_ref {
                field_path = "metadata.name"
              }
            }
          }

          env {
            name = "NODE_NAME"
            value_from {
              field_ref {
                field_path = "spec.nodeName"
              }
            }
          }

          env {
            name = "POD_NAMESPACE"
            value_from {
              field_ref {
                field_path = "metadata.namespace"
              }
            }
          }

          env {
            name  = "CLUSTER_NAME"
            value = local.cluster_name
          }

          env {
            name  = "AWS_REGION"
            value = local.region
          }

          env {
            name  = "APP_TITLE"
            value = "Spinifex EKS — HTTPS Ingress"
          }

          readiness_probe {
            http_get {
              path = "/healthz"
              port = 8080
            }
            initial_delay_seconds = 3
            period_seconds        = 10
          }
        }
      }
    }
  }
}

# NodePort Service: the ALB target group registers the workers on this port.
resource "kubernetes_service_v1" "demo" {
  metadata {
    name      = "spinifex-demo"
    namespace = "default"
  }

  spec {
    selector = { app = "spinifex-demo" }
    type     = "NodePort"

    port {
      port        = 80
      target_port = 8080
      node_port   = local.node_port
    }
  }

  depends_on = [kubernetes_deployment_v1.demo]
}

# Ingress reconciled by the AWS Load Balancer Controller into an internet-facing
# ALB. target-type instance registers the workers' NodePort; the ACM cert ARN
# attaches to the HTTPS:443 listener; HTTP is redirected to HTTPS. ELB subnets
# are injected by Spinifex into the alb IngressClassParams, so none are set here.
resource "kubernetes_ingress_v1" "demo" {
  metadata {
    name      = "spinifex-demo"
    namespace = "default"

    annotations = {
      "alb.ingress.kubernetes.io/scheme"           = "internet-facing"
      "alb.ingress.kubernetes.io/target-type"      = "instance"
      "alb.ingress.kubernetes.io/listen-ports"     = "[{\"HTTP\":80},{\"HTTPS\":443}]"
      "alb.ingress.kubernetes.io/certificate-arn"  = local.cert_arn
      "alb.ingress.kubernetes.io/ssl-redirect"     = "443"
      "alb.ingress.kubernetes.io/healthcheck-path" = "/healthz"
      "alb.ingress.kubernetes.io/inbound-cidrs"    = var.inbound_cidr
    }
  }

  spec {
    ingress_class_name = "alb"

    rule {
      http {
        path {
          path      = "/"
          path_type = "Prefix"

          backend {
            service {
              name = kubernetes_service_v1.demo.metadata[0].name
              port {
                number = 80
              }
            }
          }
        }
      }
    }
  }
}

output "ingress_name" {
  value = kubernetes_ingress_v1.demo.metadata[0].name
}
```

### Step 5. Open the Demo over HTTPS

The LBC takes a minute or two to provision the ALB after the Ingress is created. The Ingress publishes the ALB's **DNS name**, which has no resolver yet (northstar will add one) — resolve it to the ALB's public IP with `describe-load-balancers` so you can reach it without DNS or `/etc/hosts`:

```bash
# Point kubectl at the cluster first (writes ~/.kube/config) — without this every
# kubectl call fails to connect:
aws eks update-kubeconfig --name eks-https --region ap-southeast-2

kubectl get ingress spinifex-demo -o wide
DNSNAME=$(kubectl get ingress spinifex-demo -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
IP=$(aws elbv2 describe-load-balancers \
  --query "LoadBalancers[?DNSName=='${DNSNAME}'].AvailabilityZones[].LoadBalancerAddresses[].IpAddress | [0]" \
  --output text)
curl -k "https://$IP"          # self-signed cert: -k skips verification
```

Open `https://$IP` in a browser (accept the self-signed certificate). The Spinifex-themed page reports the pod, node, cluster, and region that answered — refresh to watch requests land on different replicas, now over HTTPS through the ALB.

### Cleanup

Destroy in reverse — the app/Ingress first (so the LBC tears the ALB down while the cluster is up), then the cluster:

```bash
cd workloads
tofu destroy
cd ..
tofu destroy
```

## Troubleshooting

### The Ingress Has No Address

The LBC populates `status.loadBalancer` once it has provisioned the ALB. If it stays empty:

```bash
kubectl describe ingress spinifex-demo            # events show LBC reconcile errors
kubectl -n kube-system logs deploy/aws-load-balancer-controller --tail=50
aws eks describe-addon --cluster-name eks-https --addon-name aws-load-balancer-controller --query 'addon.status'
```

Confirm the addon is `ACTIVE` and the cluster carries `spinifex.io/managed-ingress = "false"` (otherwise the built-in ingress competes).

### HTTPS Loads but Returns 502 / 504

The ALB reached the workers but the backend didn't answer. Check the demo pods and the NodePort SG rule:

```bash
kubectl get pods -o wide
kubectl get svc spinifex-demo
aws ec2 describe-security-groups --filters "Name=group-name,Values=eks-cluster-eks-https-nodegroup-sg" \
  --query 'SecurityGroups[0].IpPermissions'
```

The `eks-https-alb-nodeport` rule must admit the VPC CIDR on the NodePort, and the target group health check (`/healthz`) must pass.

### Addon Fails With "no baked bundle"

Spinifex only installs addons whose manifests are baked into the `eks-node` AMI. If `describe-addon` reports `CREATE_FAILED` with `no baked bundle`, the running worker image predates the LBC bundle — rebuild and republish the `eks-node` image, then recreate the addon.

### Pods Stuck in ImagePullBackOff

The private workers pull from ECR through the NAT gateway. Confirm the image was pushed (`tofu output ecr_repository_url`), the NAT route is in place, and the node role carries `AmazonEC2ContainerRegistryReadOnly`.

### Provider Connection Refused

```bash
sudo systemctl status spinifex.target
curl -k https://localhost:9999/
```

---

# GitOps on EKS (Argo CD + EBS-CSI)

URL: https://docs.mulgadc.com/docs/eks-gitops-argocd
Category: Terraform Workbooks
Updated: 2026-08-21
Tags: terraform, eks, kubernetes, argocd, gitops, ebs-csi, storage, workbook

Deliver an app to EKS with GitOps: the Argo CD addon syncs it from git, an EBS-CSI PersistentVolume holds its state, and HTTPS is served via LBC and ACM.

## Overview

This workbook keeps everything from [EKS HTTPS Ingress](https://docs.mulgadc.com/docs/eks-https-ingress) — the public/private VPC, the NAT gateway, the LBC addon, the ACM cert, the HTTPS Ingress — and adds the parts you reach for once an app is real: **GitOps delivery** and **persistent storage**.

Three changes over the previous rung:

1. **Argo CD delivers the app.** Instead of Terraform applying the Deployment, the **argocd** addon syncs the app from a git repository ([`mulgadc/eks-demo-app`](https://github.com/mulgadc/eks-demo-app)). Terraform only registers the repo credential and creates the Argo CD `Application`; Argo CD reconciles the manifests and self-heals drift.
2. **State lives on an EBS volume.** The **aws-ebs-csi-driver** addon ships a default gp3 StorageClass. The app's `PersistentVolumeClaim` (in the git repo) dynamically provisions a **Viperblock-backed EBS volume**; the demo's hit counter persists to it and survives pod restarts.
3. **Access-entry RBAC.** A second IAM principal is granted read-only cluster access via an access entry bound to `AmazonEKSViewPolicy`.

Terraform manages the cluster, the addons, the ACM cert, and the HTTPS Ingress; **Argo CD** manages the app's Deployment, Service, and PVC from git. The Ingress points at the Service Argo CD creates, so the app is reachable over HTTPS the moment Argo CD finishes its first sync.

**What you'll learn:**

- Installing the `argocd` and `aws-ebs-csi-driver` addons through the EKS API
- Registering a private git repo with Argo CD and driving an `Application` from Terraform
- Dynamically provisioning a Viperblock-backed EBS volume with a `PersistentVolumeClaim`
- Splitting ownership: Terraform owns infra + Ingress, Argo CD owns the workload
- Granting scoped read-only access with an EKS access entry

**What gets created**

| Resource | Name | Purpose |
|---|---|---|
| VPC + subnets | `eks-gitops-*` | Public/private network (10.32.0.0/16) with a NAT gateway |
| IAM Roles | `eks-gitops-cluster-role`, `-node-role`, `-viewer` | Control-plane, worker, and read-only viewer roles |
| ECR Repository | `spinifex-demo` | Holds the demo image the workers pull |
| EKS Cluster | `eks-gitops` | Public + private endpoints; `managed-ingress=false` |
| Node Group | `workers` | `node_desired_size` `t3.large` worker(s) — 1 or 3 |
| Addons | `aws-load-balancer-controller`, `argocd`, `aws-ebs-csi-driver` | Ingress, GitOps delivery, persistent storage |
| ACM Certificate | `eks-gitops-ingress` | Self-signed, imported; on the HTTPS listener |
| Access Entry + Assoc. | `eks-gitops-viewer` → `AmazonEKSViewPolicy` | Read-only, cluster scope |
| Argo CD repo Secret | `eks-demo-app-repo` | Credential for the private git repo |
| Argo CD Application | `spinifex-demo` | Syncs the app from git |
| K8s Ingress | `spinifex-demo` | `ingressClassName: alb`, HTTPS via the ACM cert |
| Git-managed (by Argo CD) | Deployment, Service, **PVC** | The app + its Viperblock-backed volume |

**Spinifex specifics**

- The `argocd` and `aws-ebs-csi-driver` bundles must be baked into the `eks-node` AMI — `describe-addon` returns `no baked bundle` otherwise.
- The EBS-CSI default StorageClass uses `provisioner: ebs.csi.aws.com`; a PVC against it provisions a **Viperblock-backed EBS volume**.
- If the PVC stays `Pending`, the app still serves (the counter falls back to in-memory) — see [Troubleshooting](#troubleshooting).
- Leave addon versions unset (the AWS provider and the catalog disagree on the version-string format).
- The Argo CD `Application` is a `kubernetes_manifest`, so the `argoproj.io` CRDs must exist before `apply` — apply the parent and let the addon reach `ACTIVE` first.

**Prerequisites:**

- Spinifex running with an `eks-node` image carrying the LBC, Argo CD, and EBS-CSI bundles
- OpenTofu or Terraform, plus `kubectl`, the AWS CLI, and Docker
- The demo image built and pushed to ECR (see [`docs/terraform-workbooks/demo-app`](https://github.com/mulgadc/spinifex/blob/main/docs/terraform-workbooks/demo-app/README.md))
- A git repo with the app manifests ([`mulgadc/eks-demo-app`](https://github.com/mulgadc/eks-demo-app)); a read-only PAT if it is private

## Instructions

### Step 1. Get the Template

```bash
git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
cd spinifex-tf
git sparse-checkout set docs/terraform-workbooks
cd docs/terraform-workbooks/eks-gitops-argocd
```

```hcl
# Example: GitOps on EKS with Argo CD + a persistent EBS-CSI volume
#
# The top rung of the EKS ladder. It extends eks-https-ingress (LBC + ACM HTTPS)
# and changes how the app is delivered: instead of Terraform applying the
# workload, the Argo CD addon syncs a more elaborate Spinifex-themed app from a
# git repository, and the app stores state on a Viperblock-backed EBS volume
# provisioned dynamically through the EBS-CSI driver.
#
# What it adds over eks-https-ingress:
#   * The argocd addon — GitOps continuous delivery, installed via the EKS API.
#   * The aws-ebs-csi-driver addon — dynamic EBS (Viperblock) PersistentVolumes.
#   * An access entry granting a second IAM principal read-only cluster access.
#
# The workloads/ module registers the (private) git repo with Argo CD, creates an
# Argo CD Application that syncs the app from it, and keeps the HTTPS Ingress
# (LBC + ACM) pointing at the git-managed Service. The app's manifests — including
# the PersistentVolumeClaim — live in the git repo (see ../../../../eks-demo-app).
#
# Usage:
#   cd spinifex/docs/terraform-workbooks/eks-gitops-argocd
#   export AWS_PROFILE=spinifex
#   tofu init && tofu apply
#   # build + push the demo image to the ECR repo this creates (see README),
#   # then: cd workloads && tofu init && tofu apply -var git_repo_url=... -var git_token=...
#   # finally: kubectl get ingress spinifex-demo -o wide  → open https://<address>

terraform {
  required_version = ">= 1.6.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 5.40, < 6.0"
    }
    tls = {
      source  = "hashicorp/tls"
      version = ">= 4.0"
    }
  }
}

# ---------------------------------------------------------------------------
# Variables
# ---------------------------------------------------------------------------

variable "region" {
  type    = string
  default = "ap-southeast-2"
}

variable "cluster_name" {
  type    = string
  default = "eks-gitops"
}

variable "k8s_version" {
  type    = string
  default = "1.32"
}

variable "node_instance_type" {
  type    = string
  default = "t3.medium"
}

variable "node_desired_size" {
  type        = number
  default     = 1
  description = "Worker count. Use 1 for a single-node demo, or 3 for an HA-shaped cluster."

  validation {
    condition     = var.node_desired_size == 1 || var.node_desired_size == 3
    error_message = "node_desired_size must be 1 or 3."
  }
}

variable "node_port" {
  type        = number
  default     = 30080
  description = "NodePort the demo Service is published on and the ALB forwards to"
}

variable "argocd_node_port" {
  type        = number
  default     = 30081
  description = "NodePort the Argo CD UI Service is published on and its ALB forwards to"
}

variable "cert_common_name" {
  type    = string
  default = "eks-gitops.spinifex.local"
}

variable "api_public_access_cidr" {
  type        = string
  default     = "0.0.0.0/0"
  description = "CIDR allowed to reach the public Kubernetes API endpoint; tighten in production"
}

variable "spinifex_endpoint" {
  type    = string
  default = "https://127.0.0.1:9999"
}

# ---------------------------------------------------------------------------
# Providers
# ---------------------------------------------------------------------------

provider "aws" {
  region = var.region

  endpoints {
    ec2 = var.spinifex_endpoint
    iam = var.spinifex_endpoint
    sts = var.spinifex_endpoint
    eks = var.spinifex_endpoint
    ecr = var.spinifex_endpoint
    acm = var.spinifex_endpoint
  }

  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true
  skip_region_validation      = true
}

data "aws_availability_zones" "available" {
  state = "available"
}

# ---------------------------------------------------------------------------
# VPC — public subnets for the ALB + NAT, private subnets for the workers
# ---------------------------------------------------------------------------

resource "aws_vpc" "main" {
  cidr_block           = "10.32.0.0/16"
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    Name = "${var.cluster_name}-vpc"
  }
}

resource "aws_internet_gateway" "igw" {
  vpc_id = aws_vpc.main.id

  tags = {
    Name = "${var.cluster_name}-igw"
  }
}

resource "aws_subnet" "public_a" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = "10.32.1.0/24"
  availability_zone       = data.aws_availability_zones.available.names[0]
  map_public_ip_on_launch = true

  tags = {
    Name = "${var.cluster_name}-public-a"
  }
}

resource "aws_subnet" "public_b" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = "10.32.2.0/24"
  availability_zone       = data.aws_availability_zones.available.names[0]
  map_public_ip_on_launch = true

  tags = {
    Name = "${var.cluster_name}-public-b"
  }
}

resource "aws_subnet" "private_a" {
  vpc_id            = aws_vpc.main.id
  cidr_block        = "10.32.11.0/24"
  availability_zone = data.aws_availability_zones.available.names[0]

  tags = {
    Name = "${var.cluster_name}-private-a"
  }
}

resource "aws_subnet" "private_b" {
  vpc_id            = aws_vpc.main.id
  cidr_block        = "10.32.12.0/24"
  availability_zone = data.aws_availability_zones.available.names[0]

  tags = {
    Name = "${var.cluster_name}-private-b"
  }
}

resource "aws_route_table" "public" {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.igw.id
  }

  tags = {
    Name = "${var.cluster_name}-public-rt"
  }
}

resource "aws_route_table_association" "public_a" {
  subnet_id      = aws_subnet.public_a.id
  route_table_id = aws_route_table.public.id
}

resource "aws_route_table_association" "public_b" {
  subnet_id      = aws_subnet.public_b.id
  route_table_id = aws_route_table.public.id
}

# NAT gateway gives the private workers outbound internet to pull the demo image
# from ECR. The workers still join the cluster over the in-VPC private endpoint.
resource "aws_eip" "nat" {
  domain = "vpc"

  tags = {
    Name = "${var.cluster_name}-nat-eip"
  }
}

resource "aws_nat_gateway" "nat" {
  allocation_id = aws_eip.nat.id
  subnet_id     = aws_subnet.public_a.id

  depends_on = [aws_internet_gateway.igw]

  tags = {
    Name = "${var.cluster_name}-nat"
  }
}

resource "aws_route_table" "private" {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block     = "0.0.0.0/0"
    nat_gateway_id = aws_nat_gateway.nat.id
  }

  tags = {
    Name = "${var.cluster_name}-private-rt"
  }
}

resource "aws_route_table_association" "private_a" {
  subnet_id      = aws_subnet.private_a.id
  route_table_id = aws_route_table.private.id
}

resource "aws_route_table_association" "private_b" {
  subnet_id      = aws_subnet.private_b.id
  route_table_id = aws_route_table.private.id
}

# ---------------------------------------------------------------------------
# IAM — cluster role
# ---------------------------------------------------------------------------

resource "aws_iam_role" "cluster" {
  name = "${var.cluster_name}-cluster-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Action    = "sts:AssumeRole"
      Principal = { Service = "eks.amazonaws.com" }
    }]
  })
}

resource "aws_iam_role_policy_attachment" "cluster" {
  role       = aws_iam_role.cluster.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
}

# ---------------------------------------------------------------------------
# IAM — node role
# ---------------------------------------------------------------------------

resource "aws_iam_role" "node" {
  name = "${var.cluster_name}-node-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Action    = "sts:AssumeRole"
      Principal = { Service = "ec2.amazonaws.com" }
    }]
  })
}

resource "aws_iam_role_policy_attachment" "node_worker" {
  role       = aws_iam_role.node.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
}

resource "aws_iam_role_policy_attachment" "node_cni" {
  role       = aws_iam_role.node.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
}

resource "aws_iam_role_policy_attachment" "node_ecr" {
  role       = aws_iam_role.node.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
}

# The AWS Load Balancer Controller runs with the node's instance-profile
# credentials (Spinifex wires creds at the node level, not IRSA — see the addon
# block below), so the permissions it needs to manage ALBs must live on the node
# role. This mirrors the upstream AWSLoadBalancerControllerIAMPolicy, trimmed to
# the actions an ALB Ingress exercises (Shield/WAF/Cognito are disabled on the
# controller). Resources are "*" since Spinifex evaluates grants by action.
# A customer-managed policy + attachment is used (Spinifex implements
# CreatePolicy/AttachRolePolicy, not inline PutRolePolicy).
resource "aws_iam_policy" "node_lbc" {
  name = "${var.cluster_name}-node-lbc"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "ec2:DescribeAccountAttributes",
          "ec2:DescribeAddresses",
          "ec2:DescribeAvailabilityZones",
          "ec2:DescribeInternetGateways",
          "ec2:DescribeVpcs",
          "ec2:DescribeSubnets",
          "ec2:DescribeSecurityGroups",
          "ec2:DescribeInstances",
          "ec2:DescribeNetworkInterfaces",
          "ec2:DescribeTags",
          "ec2:GetCoipPoolUsage",
          "ec2:DescribeCoipPools",
          "ec2:CreateSecurityGroup",
          "ec2:CreateTags",
          "ec2:DeleteTags",
          "ec2:AuthorizeSecurityGroupIngress",
          "ec2:RevokeSecurityGroupIngress",
          "ec2:DeleteSecurityGroup",
          "elasticloadbalancing:*",
          "acm:ListCertificates",
          "acm:DescribeCertificate",
          "acm:GetCertificate",
          "iam:CreateServiceLinkedRole",
          "iam:ListServerCertificates",
          "iam:GetServerCertificate",
          "tag:GetResources",
          "tag:TagResources",
          "wafv2:GetWebACLForResource",
          "shield:GetSubscriptionState"
        ]
        Resource = "*"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "node_lbc" {
  role       = aws_iam_role.node.name
  policy_arn = aws_iam_policy.node_lbc.arn
}

# The EBS CSI driver also runs with the node's instance-profile credentials, so
# the volume-lifecycle permissions it needs must live on the node role too. This
# mirrors the upstream AmazonEBSCSIDriverPolicy. Resources are "*" since Spinifex
# evaluates grants by action; the CreateVolume/CreateSnapshot/CreateTags grants
# upstream scope with request/resource tag conditions, omitted here because
# Spinifex does not yet evaluate IAM tag conditions.
resource "aws_iam_policy" "node_ebs_csi" {
  name = "${var.cluster_name}-node-ebs-csi"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "ec2:CreateSnapshot",
          "ec2:AttachVolume",
          "ec2:DetachVolume",
          "ec2:ModifyVolume",
          "ec2:DescribeAvailabilityZones",
          "ec2:DescribeInstances",
          "ec2:DescribeSnapshots",
          "ec2:DescribeTags",
          "ec2:DescribeVolumes",
          "ec2:DescribeVolumesModifications",
          "ec2:CreateTags",
          "ec2:DeleteTags",
          "ec2:CreateVolume",
          "ec2:DeleteVolume",
          "ec2:DeleteSnapshot"
        ]
        Resource = "*"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "node_ebs_csi" {
  role       = aws_iam_role.node.name
  policy_arn = aws_iam_policy.node_ebs_csi.arn
}

# ---------------------------------------------------------------------------
# IAM — a second principal to grant read-only cluster access to
# ---------------------------------------------------------------------------

resource "aws_iam_role" "viewer" {
  name = "${var.cluster_name}-viewer"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Action    = "sts:AssumeRole"
      Principal = { Service = "ec2.amazonaws.com" }
    }]
  })
}

# ---------------------------------------------------------------------------
# ECR — repository the workers pull the demo image from
# ---------------------------------------------------------------------------

resource "aws_ecr_repository" "demo" {
  name = "spinifex-demo"

  # tofu destroy must remove the repo even though it still holds the pushed demo
  # image; without this, DeleteRepository (force=false) fails RepositoryNotEmpty.
  force_delete = true
}

# ---------------------------------------------------------------------------
# EKS cluster — public + private endpoints, LBC-owned ingress
#
# The spinifex.io/managed-ingress = "false" tag disables K3s' built-in
# traefik/servicelb so the AWS Load Balancer Controller owns ingress, matching
# how ingress works on AWS EKS.
# ---------------------------------------------------------------------------

resource "aws_eks_cluster" "this" {
  name     = var.cluster_name
  role_arn = aws_iam_role.cluster.arn
  version  = var.k8s_version

  vpc_config {
    subnet_ids              = [aws_subnet.private_a.id, aws_subnet.private_b.id]
    endpoint_public_access  = true
    endpoint_private_access = true
    public_access_cidrs     = [var.api_public_access_cidr]
  }

  access_config {
    authentication_mode                         = "API"
    bootstrap_cluster_creator_admin_permissions = true
  }

  depends_on = [aws_iam_role_policy_attachment.cluster]

  # Spinifex's DescribeCluster doesn't echo these back the way the AWS provider
  # expects: access_config reads as absent (and its ForceNew
  # bootstrap_cluster_creator_admin_permissions would replace the live cluster on
  # re-add), and vpc_config.security_group_ids reads back as drift. Ignore both so
  # later applies (e.g. adding the Argo CD NodePort rule) neither replace the
  # cluster nor detach its security groups.
  lifecycle {
    ignore_changes = [access_config, vpc_config[0].security_group_ids]
  }

  tags = {
    Name                          = var.cluster_name
    "spinifex.io/managed-ingress" = "false"
  }
}

# ---------------------------------------------------------------------------
# Managed node group — workers in the private subnets
# ---------------------------------------------------------------------------

resource "aws_eks_node_group" "workers" {
  cluster_name    = aws_eks_cluster.this.name
  node_group_name = "workers"
  node_role_arn   = aws_iam_role.node.arn
  subnet_ids      = [aws_subnet.private_a.id, aws_subnet.private_b.id]

  scaling_config {
    desired_size = var.node_desired_size
    min_size     = var.node_desired_size
    max_size     = var.node_desired_size * 2
  }

  instance_types = [var.node_instance_type]
  ami_type       = "AL2_x86_64"

  depends_on = [
    aws_iam_role_policy_attachment.node_worker,
    aws_iam_role_policy_attachment.node_cni,
    aws_iam_role_policy_attachment.node_ecr,
    aws_iam_role_policy_attachment.node_lbc,
    aws_iam_role_policy_attachment.node_ebs_csi,
  ]

  tags = {
    Name = "${var.cluster_name}-workers"
  }
}

# ---------------------------------------------------------------------------
# Addon — AWS Load Balancer Controller
#
# Spinifex wires the controller's AWS credentials and ELB-eligible subnets at the
# node level, so no IRSA role or subnet tagging is needed here. addon_version is
# omitted: the AWS provider demands a v-prefixed version that the catalog rejects,
# so let Spinifex default to its catalog version.
# ---------------------------------------------------------------------------

resource "aws_eks_addon" "lbc" {
  cluster_name                = aws_eks_cluster.this.name
  addon_name                  = "aws-load-balancer-controller"
  resolve_conflicts_on_create = "OVERWRITE"

  depends_on = [aws_eks_node_group.workers]

  tags = {
    Name = "${var.cluster_name}-lbc"
  }
}

# ---------------------------------------------------------------------------
# Addon — Argo CD (GitOps delivery)
#
# Installed through the EKS API; Spinifex stages the bundle host-side and the
# worker renders it into the K3s auto-deploy dir. The workloads module registers
# the git repo and creates the Argo CD Application that syncs the demo app.
# addon_version omitted (see the LBC note).
# ---------------------------------------------------------------------------

resource "aws_eks_addon" "argocd" {
  cluster_name                = aws_eks_cluster.this.name
  addon_name                  = "argocd"
  resolve_conflicts_on_create = "OVERWRITE"

  depends_on = [aws_eks_node_group.workers]

  tags = {
    Name = "${var.cluster_name}-argocd"
  }
}

# ---------------------------------------------------------------------------
# Addon — EBS CSI driver (Viperblock-backed PersistentVolumes)
#
# Ships a default gp3 StorageClass (provisioner ebs.csi.aws.com). A PVC in the
# demo app's git manifests dynamically provisions a Viperblock-backed EBS volume
# so the app's state survives pod restarts. addon_version omitted (see LBC note).
# ---------------------------------------------------------------------------

resource "aws_eks_addon" "ebs_csi" {
  cluster_name                = aws_eks_cluster.this.name
  addon_name                  = "aws-ebs-csi-driver"
  resolve_conflicts_on_create = "OVERWRITE"

  depends_on = [aws_eks_node_group.workers]

  tags = {
    Name = "${var.cluster_name}-ebs-csi"
  }
}

# ---------------------------------------------------------------------------
# Access entry — grant the viewer principal read-only access (API auth mode)
# ---------------------------------------------------------------------------

resource "aws_eks_access_entry" "viewer" {
  cluster_name  = aws_eks_cluster.this.name
  principal_arn = aws_iam_role.viewer.arn
  type          = "STANDARD"
}

resource "aws_eks_access_policy_association" "viewer" {
  cluster_name  = aws_eks_cluster.this.name
  principal_arn = aws_iam_role.viewer.arn
  policy_arn    = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy"

  access_scope {
    type = "cluster"
  }

  depends_on = [aws_eks_access_entry.viewer]
}

# ---------------------------------------------------------------------------
# TLS — self-signed certificate imported into ACM
#
# Spinifex ACM supports ImportCertificate (not RequestCertificate), so the cert
# is generated locally by the tls provider and imported. The workloads Ingress
# attaches it to the ALB's HTTPS listener by ARN.
# ---------------------------------------------------------------------------

resource "tls_private_key" "ingress" {
  algorithm = "RSA"
  rsa_bits  = 2048
}

resource "tls_self_signed_cert" "ingress" {
  private_key_pem = tls_private_key.ingress.private_key_pem

  subject {
    common_name  = var.cert_common_name
    organization = "Spinifex EKS Demo"
  }

  # Wildcard SAN so one ALB (shared IngressGroup) can host-route both the demo
  # app (app.<cn>) and the Argo CD UI (argocd.<cn>) off this single cert.
  dns_names             = [var.cert_common_name, "*.${var.cert_common_name}"]
  validity_period_hours = 8760
  early_renewal_hours   = 720

  allowed_uses = [
    "key_encipherment",
    "digital_signature",
    "server_auth",
  ]
}

resource "aws_acm_certificate" "ingress" {
  private_key      = tls_private_key.ingress.private_key_pem
  certificate_body = tls_self_signed_cert.ingress.cert_pem

  tags = {
    Name = "${var.cluster_name}-ingress"
  }
}

# ---------------------------------------------------------------------------
# Let the ALB reach the workers' NodePort
#
# Spinifex auto-manages the nodegroup SG and admits only intra-cluster traffic.
# The LBC-provisioned ALB lives in this VPC, so admit the VPC CIDR on the
# NodePort. Look the SG up by its deterministic name and add one rule.
# ---------------------------------------------------------------------------

data "aws_security_group" "nodegroup" {
  filter {
    name   = "group-name"
    values = ["eks-cluster-${var.cluster_name}-nodegroup-sg"]
  }

  filter {
    name   = "vpc-id"
    values = [aws_vpc.main.id]
  }

  depends_on = [aws_eks_node_group.workers]
}

resource "aws_vpc_security_group_ingress_rule" "nodeport_from_vpc" {
  security_group_id = data.aws_security_group.nodegroup.id
  cidr_ipv4         = aws_vpc.main.cidr_block
  from_port         = var.node_port
  to_port           = var.node_port
  ip_protocol       = "tcp"

  tags = {
    Name = "${var.cluster_name}-alb-nodeport"
  }
}

# Same, for the Argo CD UI NodePort its ALB forwards to.
resource "aws_vpc_security_group_ingress_rule" "argocd_nodeport_from_vpc" {
  security_group_id = data.aws_security_group.nodegroup.id
  cidr_ipv4         = aws_vpc.main.cidr_block
  from_port         = var.argocd_node_port
  to_port           = var.argocd_node_port
  ip_protocol       = "tcp"

  tags = {
    Name = "${var.cluster_name}-argocd-nodeport"
  }
}

# ---------------------------------------------------------------------------
# Outputs
# ---------------------------------------------------------------------------

output "cluster_name" {
  value = aws_eks_cluster.this.name
}

output "region" {
  value = var.region
}

output "node_port" {
  value = var.node_port
}

output "argocd_node_port" {
  value = var.argocd_node_port
}

output "node_desired_size" {
  value = var.node_desired_size
}

output "certificate_arn" {
  value = aws_acm_certificate.ingress.arn
}

output "cert_common_name" {
  value = var.cert_common_name
}

output "viewer_principal_arn" {
  value = aws_iam_role.viewer.arn
}

output "ecr_repository_url" {
  value       = aws_ecr_repository.demo.repository_url
  description = "Push the demo-app image here, then apply the workloads module"
}

output "ingress_address_hint" {
  value = "The demo app and Argo CD UI share one ALB (LBC IngressGroup), host-routed: app.${var.cert_common_name} and argocd.${var.cert_common_name}. Get the address: kubectl get ingress spinifex-demo -o jsonpath='{.status.loadBalancer.ingress[0].hostname}{\"\\n\"}'."
}

output "argocd_ingress_hint" {
  value = "Same ALB as the app. Open https://argocd.${var.cert_common_name} (admin password: kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d). Resolve the host to the ALB address, or curl -k --resolve."
}

output "update_kubeconfig" {
  value = "aws eks update-kubeconfig --name ${aws_eks_cluster.this.name} --region ${var.region}"
}
```

### Step 2. Deploy the Cluster + Addons

```bash
export AWS_PROFILE=spinifex
tofu init
tofu apply        # add -var node_desired_size=3 for an HA-shaped cluster
```

Wait for the cluster to reach `ACTIVE` and for all three addons (`aws-load-balancer-controller`, `argocd`, `aws-ebs-csi-driver`) to report healthy:

```bash
aws eks list-addons --cluster-name eks-gitops
aws eks describe-addon --cluster-name eks-gitops --addon-name argocd --query 'addon.status'
```

### Step 3. Build and Push the Demo Image

```bash
cd ../demo-app
REGISTRY=$(cd ../eks-gitops-argocd && tofu output -raw ecr_repository_url)
REGISTRY_HOST=${REGISTRY%%/*}
aws ecr get-login-password | docker login --username AWS --password-stdin "$REGISTRY_HOST"
docker build -t "${REGISTRY}:latest" .
docker push "${REGISTRY}:latest"
cd ../eks-gitops-argocd
```

Point the app's manifests at this image: set the image ref in `eks-demo-app/manifests` (a kustomize `images:` override) to your `${REGISTRY}:latest`, and push that to the git repo.

### Step 4. Hand Delivery to Argo CD

```bash
cd workloads
tofu init
tofu apply \
  -var git_repo_url=https://github.com/mulgadc/eks-demo-app.git \
  -var git_token=<read-only-PAT>     # omit for a public repo
```

This registers the repo credential, creates the Argo CD `Application`, the demo app's HTTPS Ingress, and an HTTPS Ingress for the Argo CD UI itself. Argo CD then syncs the Deployment, Service, and PVC from git.

```hcl
# GitOps workload for eks-gitops-argocd — separate root module / state.
#
# Instead of applying the app directly, this module hands delivery to Argo CD:
# it registers the (private) git repo as an Argo CD repository credential and
# creates an Argo CD Application that syncs the demo app's manifests from it. The
# app's Deployment, Service, and PersistentVolumeClaim live in the git repo (see
# ../../../../eks-demo-app); the HTTPS Ingress (LBC + ACM) stays here and points
# at the Service that Argo CD creates.
#
# The Argo CD Application is a kubernetes_manifest, so the argoproj.io CRDs must
# already exist — apply the parent module (which installs the argocd addon) and
# let it reach ACTIVE before applying this module.
#
# Usage:
#   cd spinifex/docs/terraform-workbooks/eks-gitops-argocd
#   tofu init && tofu apply                       # parent: cluster + addons + ACM
#   cd workloads && tofu init
#   tofu apply -var git_repo_url=https://github.com/mulgadc/eks-demo-app.git \
#              -var git_token=<a-read-only-PAT>
#   # teardown is the reverse — destroy here first, then the parent.

terraform {
  required_version = ">= 1.6.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 5.40, < 6.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = ">= 2.20"
    }
  }
}

variable "spinifex_endpoint" {
  type    = string
  default = "https://127.0.0.1:9999"
}

variable "git_repo_url" {
  type        = string
  default     = "https://github.com/mulgadc/eks-demo-app.git"
  description = "Git repo Argo CD syncs the demo app from"
}

variable "git_revision" {
  type        = string
  default     = "main"
  description = "Git branch, tag, or commit Argo CD tracks"
}

variable "git_path" {
  type        = string
  default     = "manifests"
  description = "Path within the repo holding the app manifests"
}

variable "git_username" {
  type        = string
  default     = "git"
  description = "Username for the git credential (any non-empty value for a PAT)"
}

variable "git_token" {
  type        = string
  default     = ""
  sensitive   = true
  description = "Read-only personal access token for the private repo. Leave empty for a public repo."
}

provider "aws" {
  region = data.terraform_remote_state.infra.outputs.region

  endpoints {
    ec2 = var.spinifex_endpoint
    iam = var.spinifex_endpoint
    sts = var.spinifex_endpoint
    eks = var.spinifex_endpoint
  }

  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true
  skip_region_validation      = true
}

data "terraform_remote_state" "infra" {
  backend = "local"

  config = {
    path = "../terraform.tfstate"
  }
}

locals {
  cluster_name     = data.terraform_remote_state.infra.outputs.cluster_name
  region           = data.terraform_remote_state.infra.outputs.region
  cert_arn         = data.terraform_remote_state.infra.outputs.certificate_arn
  argocd_node_port = data.terraform_remote_state.infra.outputs.argocd_node_port
  cert_cn          = data.terraform_remote_state.infra.outputs.cert_common_name
  private_repo     = var.git_token != ""

  # The demo app and the Argo CD UI share one ALB via an LBC IngressGroup; LBC
  # gives each backend its own target group and host-routes between them.
  alb_group   = local.cluster_name
  app_host    = "app.${local.cert_cn}"
  argocd_host = "argocd.${local.cert_cn}"
}

data "aws_eks_cluster" "this" {
  name = local.cluster_name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.this.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)

  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "aws"
    args        = ["eks", "get-token", "--cluster-name", local.cluster_name, "--region", local.region]
  }
}

# Repository credential for the private repo. Argo CD picks up Secrets in its
# namespace labelled argocd.argoproj.io/secret-type=repository. Created only when
# a token is supplied (a public repo needs no credential).
resource "kubernetes_secret_v1" "repo" {
  count = local.private_repo ? 1 : 0

  metadata {
    name      = "eks-demo-app-repo"
    namespace = "argocd"
    labels = {
      "argocd.argoproj.io/secret-type" = "repository"
    }
  }

  data = {
    type     = "git"
    url      = var.git_repo_url
    username = var.git_username
    password = var.git_token
  }
}

# Argo CD Application: syncs the demo app from the git repo into the default
# namespace, self-healing and pruning so the cluster tracks the repo.
resource "kubernetes_manifest" "demo_app" {
  manifest = {
    apiVersion = "argoproj.io/v1alpha1"
    kind       = "Application"

    # The resources finalizer makes deleting this Application cascade-prune the
    # synced resources (Deployment, Service, PVC). Without it, tofu destroy drops
    # the Application CR but orphans the live PVC, whose Delete-reclaim PV then
    # leaks an EBS volume once the cluster (and ebs-csi) is torn down.
    metadata = {
      name       = "spinifex-demo"
      namespace  = "argocd"
      finalizers = ["resources-finalizer.argocd.argoproj.io"]
    }

    spec = {
      project = "default"

      source = {
        repoURL        = var.git_repo_url
        targetRevision = var.git_revision
        path           = var.git_path
      }

      destination = {
        server    = "https://kubernetes.default.svc"
        namespace = "default"
      }

      syncPolicy = {
        automated = {
          prune    = true
          selfHeal = true
        }
        syncOptions = ["CreateNamespace=true"]
      }
    }
  }

  depends_on = [kubernetes_secret_v1.repo]
}

# HTTPS Ingress (LBC + ACM), carried over from eks-https-ingress. It points at
# the spinifex-demo Service that Argo CD creates from the git manifests.
resource "kubernetes_ingress_v1" "demo" {
  metadata {
    name      = "spinifex-demo"
    namespace = "default"

    annotations = {
      "alb.ingress.kubernetes.io/scheme"           = "internet-facing"
      "alb.ingress.kubernetes.io/target-type"      = "instance"
      "alb.ingress.kubernetes.io/group.name"       = local.alb_group
      "alb.ingress.kubernetes.io/listen-ports"     = "[{\"HTTP\":80},{\"HTTPS\":443}]"
      "alb.ingress.kubernetes.io/certificate-arn"  = local.cert_arn
      "alb.ingress.kubernetes.io/ssl-redirect"     = "443"
      "alb.ingress.kubernetes.io/healthcheck-path" = "/healthz"
    }
  }

  spec {
    ingress_class_name = "alb"

    # No host condition: the app is the ALB's catch-all on :443, so it answers on
    # the raw ALB IP as well as app.<cn> once DNS is wired. Argo CD's host rule is
    # more specific and still wins for argocd.<cn>.
    rule {
      http {
        path {
          path      = "/"
          path_type = "Prefix"

          backend {
            service {
              name = "spinifex-demo"
              port {
                number = 80
              }
            }
          }
        }
      }
    }
  }

  depends_on = [kubernetes_manifest.demo_app]
}

# ---------------------------------------------------------------------------
# Argo CD UI exposure — managing the cluster through the GitOps console is the
# point of this demo, so the UI gets the same HTTPS-Ingress treatment as the app
# rather than a port-forward. The argocd addon ships argocd-server as a ClusterIP
# only; expose it via a NodePort the ALB targets. argocd-server serves TLS on
# 8080 (no --insecure), so the ALB speaks HTTPS to the backend.
#
# Same group.name as the demo Ingress, so LBC folds both onto ONE ALB. On :443
# the app is the catch-all and argocd.<cn> host-routes to the UI. Argo CD also
# gets a hostless :8443 listener (argocd_ip below) so the UI is reachable on the
# raw ALB IP before DNS exists — https://<alb-ip>:8443.
# ---------------------------------------------------------------------------

resource "kubernetes_service_v1" "argocd_server_nodeport" {
  metadata {
    name      = "argocd-server-nodeport"
    namespace = "argocd"
  }

  spec {
    type     = "NodePort"
    selector = { "app.kubernetes.io/name" = "argocd-server" }

    port {
      port        = 443
      target_port = 8080
      node_port   = local.argocd_node_port
      protocol    = "TCP"
    }
  }
}

resource "kubernetes_ingress_v1" "argocd" {
  metadata {
    name      = "argocd-server"
    namespace = "argocd"

    annotations = {
      "alb.ingress.kubernetes.io/scheme"               = "internet-facing"
      "alb.ingress.kubernetes.io/target-type"          = "instance"
      "alb.ingress.kubernetes.io/group.name"           = local.alb_group
      "alb.ingress.kubernetes.io/listen-ports"         = "[{\"HTTPS\":443}]"
      "alb.ingress.kubernetes.io/certificate-arn"      = local.cert_arn
      "alb.ingress.kubernetes.io/backend-protocol"     = "HTTPS"
      "alb.ingress.kubernetes.io/healthcheck-protocol" = "HTTPS"
      "alb.ingress.kubernetes.io/healthcheck-path"     = "/healthz"
    }
  }

  spec {
    ingress_class_name = "alb"

    rule {
      host = local.argocd_host

      http {
        path {
          path      = "/"
          path_type = "Prefix"

          backend {
            service {
              name = kubernetes_service_v1.argocd_server_nodeport.metadata[0].name
              port {
                number = 443
              }
            }
          }
        }
      }
    }
  }
}

# Hostless Argo CD listener on :8443 of the same ALB, so the UI is reachable on
# the raw ALB IP (https://<alb-ip>:8443) without DNS or a host header. argocd.<cn>
# on :443 still works via the host-routed ingress above.
resource "kubernetes_ingress_v1" "argocd_ip" {
  metadata {
    name      = "argocd-server-ip"
    namespace = "argocd"

    annotations = {
      "alb.ingress.kubernetes.io/scheme"               = "internet-facing"
      "alb.ingress.kubernetes.io/target-type"          = "instance"
      "alb.ingress.kubernetes.io/group.name"           = local.alb_group
      "alb.ingress.kubernetes.io/listen-ports"         = "[{\"HTTPS\":8443}]"
      "alb.ingress.kubernetes.io/certificate-arn"      = local.cert_arn
      "alb.ingress.kubernetes.io/backend-protocol"     = "HTTPS"
      "alb.ingress.kubernetes.io/healthcheck-protocol" = "HTTPS"
      "alb.ingress.kubernetes.io/healthcheck-path"     = "/healthz"
    }
  }

  spec {
    ingress_class_name = "alb"

    rule {
      http {
        path {
          path      = "/"
          path_type = "Prefix"

          backend {
            service {
              name = kubernetes_service_v1.argocd_server_nodeport.metadata[0].name
              port {
                number = 443
              }
            }
          }
        }
      }
    }
  }
}

output "application_name" {
  value = kubernetes_manifest.demo_app.manifest.metadata.name
}

output "alb_address_hint" {
  value = "One shared ALB. Get its IP: kubectl get ingress spinifex-demo -o jsonpath='{.status.loadBalancer.ingress[0].ip}{\"\\n\"}' (or [0].hostname). Reach by raw IP — app: https://<alb-ip>/  Argo CD: https://<alb-ip>:8443/ . By DNS — app: ${local.app_host}  Argo CD: ${local.argocd_host}."
}

output "argocd_admin_password_cmd" {
  value = "kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d ; echo"
}

output "argocd_url_hint" {
  value = "Open https://${local.argocd_host} (admin password: kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d). Resolve the host to the ALB address (northstar/CNAME), or curl -k --resolve ${local.argocd_host}:443:<alb-ip>."
}
```

### Step 5. Watch the Sync and Open the App

The demo app and the Argo CD UI share **one ALB** (an LBC IngressGroup). On `:443` the app is the catch-all and `argocd.eks-gitops.spinifex.local` host-routes to the UI; Argo CD also gets a hostless `:8443` listener so it is reachable on the raw ALB IP before DNS exists. Grab the shared ALB address once:

```bash
# Point kubectl at the cluster first (writes ~/.kube/config) — without this every
# kubectl call fails to connect:
aws eks update-kubeconfig --name eks-gitops --region ap-southeast-2

kubectl -n argocd get applications
kubectl get pods -o wide
kubectl get pvc                    # the EBS-CSI volume should be Bound

DNSNAME=$(kubectl get ingress spinifex-demo -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
ALB_IP=$(aws elbv2 describe-load-balancers \
  --query "LoadBalancers[?DNSName=='${DNSNAME}'].AvailabilityZones[].LoadBalancerAddresses[].IpAddress | [0]" \
  --output text)
echo "$ALB_IP"
```

**Reach it by raw IP (no DNS, no host header):**

| Service | URL |
|---|---|
| Demo app | `https://<ALB_IP>/` |
| Argo CD UI | `https://<ALB_IP>:8443/` |

```bash
curl -k https://"$ALB_IP"/            # app — returns the Spinifex page
curl -k https://"$ALB_IP":8443/       # Argo CD UI
```

The bare ALB IP serves the **app** on `:443`. Hitting `https://<ALB_IP>/` and expecting Argo CD gives the app instead — use `:8443` for the UI. Once northstar (or Route 53) resolves the hostnames to the ALB, `app.eks-gitops.spinifex.local` and `argocd.eks-gitops.spinifex.local` both work on `:443`. Self-signed cert — accept the browser warning.

The page shows the **"persisted to EBS volume"** badge and a hit counter that keeps climbing — delete the pod (`kubectl delete pod -l app=spinifex-demo`) and the count survives, proving the volume is durable.

### Step 6. Open the Argo CD UI

Managing deployments through the Argo CD console is the point of this workbook, so the UI is exposed on the **same ALB** as the app, not behind a port-forward. The `workloads` module adds a `NodePort` Service in front of `argocd-server` (the addon ships it `ClusterIP` only) and two Ingresses in the shared group — a host-routed one on `:443` and a hostless one on `:8443` for raw-IP access. `argocd-server` serves TLS on its own port, so both use `backend-protocol: HTTPS`.

**Get the admin credentials.** Argo CD generates a one-time `admin` password into a Secret on install; the username is always `admin`:

```bash
kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath='{.data.password}' | base64 -d ; echo
```

(`tofu output argocd_admin_password_cmd` prints this same command.)

Open the UI and log in as `admin`:

- By raw IP: `https://<ALB_IP>:8443/`
- By DNS once wired: `https://argocd.eks-gitops.spinifex.local`

The `spinifex-demo` Application shows the sync status, the resource tree, and live diffs against git — change a manifest in the repo and watch Argo CD reconcile it.

If you'd rather not expose the UI at all, port-forward instead:

```bash
kubectl -n argocd port-forward svc/argocd-server 8080:443
# then open https://localhost:8080
```

### Cleanup

Destroy in reverse — the Argo CD Application + Ingress first (so Argo CD prunes the workload and the LBC tears the ALB down), then the cluster:

```bash
cd workloads
tofu destroy
cd ..
tofu destroy
```

## Troubleshooting

### Application Won't Sync

```bash
kubectl -n argocd get applications
kubectl -n argocd describe application spinifex-demo     # conditions show repo/auth errors
kubectl -n argocd logs deploy/argocd-repo-server --tail=50
```

For a private repo, confirm the `eks-demo-app-repo` Secret exists in the `argocd` namespace with the `argocd.argoproj.io/secret-type=repository` label and a valid token.

### `kubernetes_manifest` Fails: no matches for kind "Application"

The `argoproj.io` CRDs aren't installed yet. Apply the parent module and wait for the `argocd` addon to reach `ACTIVE` before applying `workloads/`.

### PVC Stuck in Pending

```bash
kubectl get pvc
kubectl describe pvc <name>
kubectl get storageclass
```

Confirm the `aws-ebs-csi-driver` addon is `ACTIVE` and the default StorageClass exists. The app still serves with an in-memory counter if the volume never binds.

### The Ingress Has No Address

The LBC populates `status.loadBalancer` once it provisions the ALB. Check the addon and the controller logs:

```bash
kubectl describe ingress spinifex-demo
kubectl -n kube-system logs deploy/aws-load-balancer-controller --tail=50
```

Confirm the cluster carries `spinifex.io/managed-ingress = "false"` so the LBC owns ingress.

If the controller logs show `FailedBuildModel ... DescribeAvailabilityZones ... 403 ... AccessDenied`, the node role is missing the LBC permissions. The controller runs with the node instance-profile credentials, so the `${var.cluster_name}-node-lbc` policy (`aws_iam_policy.node_lbc`) must be attached to the node role — re-run `tofu apply` on the parent module if it was provisioned before that policy existed.

### Provider Connection Refused

```bash
sudo systemctl status spinifex.target
curl -k https://localhost:9999/
```

---

# ECS Quickstart

URL: https://docs.mulgadc.com/docs/ecs-quickstart
Category: Terraform Workbooks
Updated: 2026-08-19
Tags: terraform, ecs, containers, alb, iam, vpc, workbook

Stand up a full AWS-compatible ECS stack with Terraform: a VPC, IAM roles, a cluster, a task definition, container instances, and a load-balanced service.

## Overview

This workbook is the Terraform-native equivalent of the console's **Provision capacity** action. It provisions a full ECS stack in one `apply`: a VPC with two public subnets, the `ecsInstanceRole` instance profile, a task IAM role, an ECS cluster, an `awsvpc` task definition running nginx, one or more container instances launched from the `spinifex-ecs-node` AMI, and an Application Load Balancer with a target group the service registers into.

Because a Spinifex container instance reaches the control plane over the **gateway** (TLS + SigV4) rather than a managed AWS endpoint, the workbook makes the two things the console injects for you explicit: a **LAN-reachable** gateway URL and the gateway CA, both baked into the instance's cloud-init user-data. The agent draws its credentials from IMDS via the `ecsInstanceRole` instance profile, so no static keys are written.

## Prerequisites

- **Spinifex running**, with the AWS CLI configured for the `spinifex` profile (see [Installing Spinifex](https://docs.mulgadc.com/docs/install)) and OpenTofu (or Terraform) installed.
- **The `spinifex-ecs-node` AMI imported** — resolved here by the `tag:spinifex:managed-by=ecs` filter:

  ```bash
  aws ec2 describe-images \
    --filters 'Name=tag:spinifex:managed-by,Values=ecs' \
    --query 'Images[].[ImageId,Name]' --output text
  ```

- **A LAN-reachable gateway URL** — the host's WAN/bridge IP, not `127.0.0.1` (a guest VM cannot reach the host loopback). For example `https://192.168.1.33:9999`.
- **The gateway CA PEM** readable at `gateway_ca_cert_path` (default `/etc/spinifex/ca.pem`).

## Instructions

### 1. Fetch the workbook

```bash
git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
cd spinifex-tf
git sparse-checkout set docs/terraform-workbooks
cd docs/terraform-workbooks/ecs-quickstart
```

### 2. Apply

```bash
export AWS_PROFILE=spinifex

tofu init
tofu apply -var 'gateway_url=https://<host-lan-ip>:9999'
```

`ecsInstanceRole` is account-global. If you have already used the console's provision-capacity action it exists already, so skip re-creating it:

```bash
tofu apply \
  -var 'gateway_url=https://<host-lan-ip>:9999' \
  -var 'create_instance_role=false'
```

### 3. Verify

Container instances take ~30-60s to boot and register:

```bash
aws ecs list-container-instances --cluster ecs-quickstart
aws ecs describe-services --cluster ecs-quickstart --services ecs-quickstart-web \
  --query 'services[0].[runningCount,desiredCount]'
```

The ALB DNS name ends in `.elb.spinifex.local` and does not resolve from your host. Fetch its public IP and curl it:

```bash
aws elbv2 describe-load-balancers --names ecs-quickstart-alb \
  --query 'LoadBalancers[0].AvailabilityZones[].LoadBalancerAddresses[].IpAddress' \
  --output text
curl http://<alb_public_ip>
```

### 4. Variables

| Variable | Default | Purpose |
|---|---|---|
| `region` | `ap-southeast-2` | AWS region. |
| `cluster_name` | `ecs-quickstart` | Cluster + resource name prefix. |
| `instance_type` | `t3.small` | Container instance type. |
| `container_count` | `1` | Container instances + service desired count. |
| `task_image` | `docker.io/library/nginx:1.27-alpine` | Image the task runs. |
| `spinifex_endpoint` | `https://127.0.0.1:9999` | Gateway as seen from the host running Terraform. |
| `gateway_url` | _(required)_ | Gateway as seen from a guest VM (LAN-reachable). |
| `gateway_ca_cert_path` | `/etc/spinifex/ca.pem` | Host-readable gateway CA PEM. |
| `create_instance_role` | `true` | Create `ecsInstanceRole`; set `false` if it already exists. |

### 5. Teardown

```bash
tofu destroy -var 'gateway_url=https://<host-lan-ip>:9999'
```

`DeleteCluster` cascades through the service, its tasks, and the container instance registrations, so the destroy round-trips cleanly.

## Troubleshooting

**Container instances never register.** The agent registers over the gateway, not a managed endpoint. Confirm `gateway_url` is the host's **LAN-reachable** IP (not `127.0.0.1`) and that `gateway_ca_cert_path` points at the real gateway CA. Because `cloud-init write_files` runs once per instance, a corrected user-data needs an instance **replacement** — `tofu apply -replace='aws_instance.node[0]'` — not an in-place modify.

**`apply` fails resolving the AMI.** The `spinifex-ecs-node` image is not imported, or its `spinifex:managed-by=ecs` tag is missing. Re-run the `describe-images` check in Prerequisites.

**`MissingParameter` on the instance.** The gateway's `RunInstances` requires a `KeyName`; the workbook generates a key pair for you, so this only appears if you have stripped that out.

**Service `runningCount` stays below `desiredCount`.** No instance has free capacity, or tasks cannot pull the image. Confirm at least one `ACTIVE` container instance and that the subnets have an Internet Gateway route.

**`create_instance_role` conflict.** If `ecsInstanceRole` already exists (from the console), pass `-var 'create_instance_role=false'` so Terraform does not try to recreate it.

---

# RDS Quickstart (PostgreSQL)

URL: https://docs.mulgadc.com/docs/rds-quickstart
Category: Terraform Workbooks
Updated: 2026-09-14
Tags: terraform, rds, postgres, database, vpc, workbook

Stand up a managed PostgreSQL database on Spinifex with Terraform: a VPC, DB subnet group, parameter group, aws_db_instance, and a client VM that runs psql.

## Overview

A Spinifex DB instance runs in a VM you never see. What lands in your VPC is a single **endpoint ENI** in one of the DB subnet group's subnets, so the endpoint is **always private** — there is no `publicly_accessible` mode, and a request for one is rejected. That is why this workbook builds a client VM: it is the only place from which the database can be reached.

The layout is the one that shape implies — the tier that talks to the database in its own subnet, and the database in a subnet with no route off the VPC:

```
     IGW
      │
   client subnet 10.60.1.0/24 ──── client VM
                                       │
                                    psql:5432
                                       │
   db subnet     10.60.2.0/24 ──── endpoint ENI ──▶ DB VM (platform-owned)
```

The endpoint is reachable from **any subnet of the VPC**, so the client does not have to share the subnet the endpoint ENI landed in — which also means a subnet group spanning several subnets works wherever the endpoint is placed within it.

The client subnet is public in that it has an internet-gateway route, which the client needs to `apt-get` a psql. The DB subnet has no route table association, so it stays on the main route table `create-vpc` writes: intra-VPC routing and nothing else. The endpoint ENI gets no public address and `publicly_accessible = true` is rejected either way.

The database security group admits `5432` from the client security group and nothing else, which compiles to an ACL on the endpoint ENI itself — the port is deny-by-default for everything else in the VPC.

That holds for the instance, not just for the endpoint ENI. The DB VM has two further NICs the platform uses to manage it, and no customer security group governs either; the engine binds the endpoint ENI's address alone, so it is not listening on them. PostgreSQL's client authentication rules are scoped to this VPC's own range as well, so the endpoint really is the whole of the reachable surface.

## Prerequisites

- **Spinifex running**, with the AWS CLI configured for the `spinifex` profile (see [Installing Spinifex](https://docs.mulgadc.com/docs/install)) and OpenTofu (or Terraform) installed.
- **The `spinifex-rds-postgres` image registered.** DB instances boot from this system image. Import it once per cluster, then verify that all tags used by the RDS engine resolver are present:

  ```bash
  spx admin images import --name spinifex-rds-postgres

  AWS_PROFILE=spinifex aws ec2 describe-images \
    --filters \
      'Name=tag:spinifex:managed-by,Values=rds' \
      'Name=tag:engine,Values=postgres' \
      'Name=tag:engine-version,Values=18' \
    --query 'Images[].[ImageId,Name]' --output text
  ```

- **An Ubuntu image** for the client VM, resolved here by a `*ubuntu-26.04*` / `*ubuntu-24.04*` name filter.
- **Roughly 2 GiB of free guest memory** — one `db.t3.micro` DB VM plus one `t3.small` client.

## Instructions

### 1. Fetch the workbook

```bash
git clone --depth 1 --filter=blob:none --sparse https://github.com/mulgadc/spinifex.git spinifex-tf
cd spinifex-tf
git sparse-checkout set docs/terraform-workbooks
cd docs/terraform-workbooks/rds-quickstart
```

### 2. Apply

```bash
export AWS_PROFILE=spinifex

tofu init
tofu apply -var 'db_password=<choose-one>'
```

Creating the instance boots a VM, runs `initdb` and waits for the in-guest agent's first healthy heartbeat, so the DB is several minutes of the apply on its own. Terraform waits for `available` before it launches the client, because the client's cloud-init needs the endpoint address.

### 3. Connect

```bash
tofu output ssh_to_client
ssh -i rds-quickstart-client.pem ubuntu@<client_public_ip>
```

`PGHOST`, `PGPORT`, `PGUSER` and `PGDATABASE` are exported from `/etc/profile.d`, and the password is in `~/.pgpass` at `0600` — so `psql` on its own connects:

```bash
psql -c 'SELECT version();'
psql -c 'CREATE TABLE hello (id int primary key, note text);'
psql -c "INSERT INTO hello VALUES (1, 'it works');"
psql -c 'SELECT note FROM hello;'
```

Give the client a minute after apply: `postgresql-client` is installed by cloud-init on first boot.

**TLS is required**, and the commands above already satisfy it: libpq negotiates TLS whenever the server offers it, so nothing here needs setting. Only a client that explicitly disables it — `psql "sslmode=disable"` — is refused. To validate the certificate as well, fetch the cluster CA in the guest with `curl -fsS http://169.254.169.254/spinifex/ca.pem -o ~/ca.pem` and add `sslmode=verify-full sslrootcert=~/ca.pem`; both the endpoint name and its IP are in the certificate's SAN set, so either address verifies.

### 4. Variables

| Variable | Default | Purpose |
|---|---|---|
| `region` | `ap-southeast-2` | AWS region. |
| `name` | `rds-quickstart` | Resource name prefix and the DB instance identifier. |
| `spinifex_endpoint` | `https://127.0.0.1:9999` | Gateway as seen from the host running Terraform. |
| `instance_type` | `t3.small` | EC2 type for the **client** VM. |
| `db_instance_class` | `db.t3.micro` | DB class. One of the curated `db.*` subset. |
| `engine_version` | `18` | PostgreSQL major. `18` is the only version served, and a minor such as `18.4` is rejected. |
| `db_name` | `appdb` | Database created at bootstrap. |
| `db_username` | `appuser` | Master user. `postgres`, `rdsadmin`, `rds_superuser` and `pg_*` are reserved. |
| `db_password` | `QuickstartS3cret1` | Master password — override it. No `/`, `"`, `@` or spaces. |
| `allocated_storage` | `20` | Data-volume GiB. Grow-only, and a grow is stop/start. |

### 5. Teardown

```bash
tofu destroy -var 'db_password=<the-one-you-used>'
```

The instance is created with `skip_final_snapshot = true` on purpose. A final snapshot **pins the data volume alive** until that snapshot is deleted, so a workbook that took one would leave storage behind on every destroy. For anything you care about, take one:

```bash
aws rds delete-db-instance --db-instance-identifier rds-quickstart \
  --final-db-snapshot-identifier rds-quickstart-final
```

## Things this workbook does deliberately

- **`storage_encrypted = true` is set explicitly.** Storage is always encrypted, so the instance reports `true` whether or not you asked. The provider's attribute is optional but *not* computed, so leaving it out means the read-back carries a value your configuration does not — and every subsequent `plan` shows a change on an instance nothing has touched.
- **`instance_class` and `engine_version` are literal.** `describe-db-engine-versions` and `describe-orderable-db-instance-options` are not implemented, so the `aws_rds_engine_version` and `aws_rds_orderable_db_instance` data sources are unavailable. `aws_db_instance` needs neither.
- **A DB subnet group over one subnet.** AWS requires a DB subnet group to span two AZs. Spinifex is single-AZ — every subnet reports the same zone — so the group accepts any count, and a second private subnet here would buy nothing but a second CIDR.
- **The password is written to `~/.pgpass`, not to the environment.** A `PGPASSWORD` in `/etc/profile.d` is readable by every user on the client and leaks into `ps` output.

## Troubleshooting

**`apply` fails resolving the DB AMI.** The `spinifex-rds-postgres` image is not registered with the required engine tags. Run `spx admin images import --name spinifex-rds-postgres`, then repeat the tagged `describe-images` check in Prerequisites before applying again.

**The apply sits on `aws_db_instance.main: Still creating...`.** Several minutes is normal — a VM boot plus `initdb` plus the first healthy heartbeat. Much longer is a bootstrap that did not finish:

```bash
aws rds describe-db-instances --db-instance-identifier rds-quickstart \
  --query 'DBInstances[0].[DBInstanceStatus,StatusInfos[0].Message]' --output text
aws rds describe-events --source-type db-instance --source-identifier rds-quickstart
```

**`psql` hangs on the client.** Three causes, in the order worth checking:

- cloud-init has not finished installing `postgresql-client` — check `cloud-init status --long`.
- the security group is not letting you through. Nothing outside the client security group can open `5432` — that is the point of the rule — so a psql from your workstation will always time out.
- the client is outside the VPC. The endpoint exists only inside it: any subnet will do, but there is no path to it from anywhere else.

**`InsufficientInstanceCapacity` on the DB instance.** The node admits a launch against live free memory. Free some, or drop to a smaller `db_instance_class`.

**`tofu destroy` leaves a DB instance behind.** `deletion_protection` blocks a delete outright. This workbook sets it to `false`; if you turned it on, clear it before destroying:

```bash
aws rds modify-db-instance --db-instance-identifier rds-quickstart \
  --no-deletion-protection --apply-immediately
```

---

# Flaw Remediation Policy

URL: https://docs.mulgadc.com/docs/flaw-remediation-policy
Category: Security and Compliance
Updated: 2026-08-21
Tags: security, compliance, cmmc, vulnerabilities, cvss, patching

CVSS-tiered SLAs for identifying, reporting, and correcting software flaws in Spinifex and its direct dependencies, for maintainers and CMMC Level 1 operators.

## Overview

**Audience:** Spinifex maintainers who publish releases, and operators deploying Spinifex into CMMC Level 1 environments.

**Scope:** Flaws in Spinifex's own code and its direct dependencies (Go modules, UI npm packages, GitHub Actions). Operating system, kernel, OVN/OVS, and hypervisor CVEs are the operator's responsibility — see [§5](#5-operator-responsibilities).

## CMMC Practices Covered

| Practice | Title | Objective |
|----------|-------|-----------|
| SI.L1-3.14.1 | Flaw Remediation | [a] Time to identify flaws is specified. [b] Flaws identified in time. [c] Time to report flaws is specified. [d] Flaws reported in time. [e] Time to correct flaws is specified. [f] Flaws corrected in time. |

## Approach

Spinifex ships a short dependency chain and identification is already automated: Dependabot raises security PRs the moment GitHub links a CVE to a shipped dependency, `govulncheck` runs on every PR, and GitHub Security Advisories watch all three `mulgadc/*` repositories.

Sections map to the three scored objectives: [§1](#1-identification) names the identification sources (objectives [a]/[b]); [§3](#3-remediation-slas) and [§4](#4-reporting-workflow) set the report and correct timeframes (objectives [c]–[f]). [§5](#5-operator-responsibilities) delineates layers the operator owns, and [§6](#6-evidence-and-record-keeping) covers the records needed to demonstrate compliance at assessment.

## 1. Identification

| Source | Frequency | Coverage |
|--------|-----------|----------|
| Dependabot security updates | Immediate — raised as soon as GitHub links an advisory to a dependency | Go modules, GitHub Actions, UI npm packages |
| Dependabot version updates | Weekly | Routine version bumps (non-security) |
| `govulncheck` (CI `lint_and_security` job) | Every PR and push to `main` | Go stdlib + module CVEs reachable in the binary |
| GitHub Security Advisories | Continuous | `mulgadc/spinifex`, `mulgadc/predastore`, `mulgadc/viperblock` |
| Third-party reports (sensitive) | Ad hoc | [GitHub private vulnerability reporting](https://github.com/mulgadc/spinifex/security/advisories/new) |
| Third-party reports (non-sensitive) | Ad hoc | Public GitHub issue, labelled `security` |
| Internal discovery | Ad hoc | Review, audit, red team, incident response |

Dependabot security PRs are merged immediately once CI passes — for dependency CVEs this collapses identify, report, and correct into a single action. The separate weekly cadence applies only to non-security version bumps.

Reporters: use private vulnerability reporting for anything plausibly Critical or High; a public issue is fine for hardening suggestions and non-exploitable bugs. A flaw is **identified** once it appears in any channel above with enough detail to score CVSS.

## 2. Severity

CVSS v3.1 base score from the upstream advisory; otherwise scored by maintainers and the vector recorded on the tracking issue.

| Severity | CVSS v3.1 | Examples |
|----------|-----------|----------|
| Critical | 9.0 – 10.0 | Unauthenticated RCE reachable from tenant networks; `awsgw` auth bypass; master-key disclosure |
| High | 7.0 – 8.9 | Authenticated RCE; intra-service privilege escalation; cluster-internal auth bypass; memory disclosure exposing credentials. |
| Medium | 4.0 – 6.9 | Authenticated DoS; non-secret information disclosure; limited-blast-radius logic errors |
| Low | 0.1 – 3.9 | Heavily mitigated issues — local-only, non-default config, negligible impact |

**Severity modifiers.** Bump one tier **up** if any of the following apply: the flaw is under active exploitation, reaches the master encryption key or cluster CA private key, or allows cross-tenant data access. Bump one tier **down** if the flaw is only reachable by a cluster-internal process already holding credentials equivalent to the attack's outcome.

## 3. Remediation SLAs

Clocks start at the moment of identification ([§1](#1-identification)). All three objectives — identify [a], report [c], correct [e] — are time-bound.

| Severity | Identify | Report | Correct |
|----------|----------|--------|---------|
| Critical | 24 hours | 48 hours, with patch | 48 hours |
| High     | 48 hours | 7 days, with patch | 7 days |
| Medium   | 7 days   | Next release changelog | 30 days |
| Low      | 30 days  | Next release changelog | 90 days |

**Identify** - triaged into a tracking issue, CVSS scored, affected components and versions determined, severity label applied.

**Report** - GitHub Security Advisory is published (for Critical/High) or the fix is described in the release notes (for Medium/Low). Operators are the audience; the report must let them determine whether their deployment is affected.

**Correct** - tagged Spinifex release containing the fix is available for operators to pull. For dependency CVEs, this is a release that bumps the vulnerable module to a fixed version. For first-party code, it is a release containing the patch.

If an upstream embargo applies, the correct-SLA pauses until disclosure; the identify-SLA does not. Record the embargo on the tracking issue.

## 4. Reporting Workflow

1. **Intake** — External findings arrive via GitHub (private for sensitive, public issue otherwise). Maintainers mirror each finding into the internal tracker with CVSS score, affected components, and advisory link. Private disclosures stay private until the advisory is drafted.
2. **Triage** — Validate severity within the identify-SLA, confirm reachability in Spinifex, and either close as not-applicable (with justification) or proceed.
3. **Fix** — Patch lands on `main` via the normal PR + E2E workflow. Commit message references the CVE / GHSA ID.
4. **Release** — Tagged release cut; release notes state affected versions, CVSS, and upgrade guidance. Critical/High get a published GHSA.
5. **Notify** — GitHub release/advisory notifications reach subscribed operators automatically. Out-of-band email notification is triggered only for Critical.

Not-applicable findings (unreachable, already patched, scanner false positive) are still tracked to closure so every identification has an audit trail.

## 5. Operator Responsibilities

Operators own remediation for layers Spinifex does not ship:

| Layer | Responsibility |
|-------|----------------|
| Spinifex releases | Apply released patches within the correct-SLA above. |
| Host OS + kernel | Track distribution advisories (Debian DSA, Ubuntu USN) at matching cadence. |
| OVN / OVS | Patched via the operator's package channel alongside the OS. |
| QEMU / KVM / libvirt | As above — hypervisor CVEs affect tenant isolation. |
| Host AV / EDR agent | See [Malware Protection §2](https://docs.mulgadc.com/docs/malware-protection#2-update-requirements-sil1-3144). |

Operators should subscribe to the `mulgadc/spinifex` release feed (Watch → Custom → Releases + Security Advisories) so Critical/High advisories arrive via GitHub's notification channel rather than polling.

## 6. Evidence and Record Keeping

For CMMC assessment, retain the following for at least 12 months:

- Tracking record for every identified flaw (GitHub issue, GHSA, or internal tracker ID) with identify / triage / close timestamps.
- Dependabot history and `govulncheck` CI logs — export before GitHub Actions retention expires if shorter.
- Release notes and GHSAs for each corrected flaw.
- Operator patch-apply records (change tickets, config-management runs) showing releases deployed within SLA.

These demonstrate objectives [b], [d], [f] — "within the specified time" — in practice.

## 7. Operator Checklist

- System security plan references this policy as the identify / report / correct timeframes for Spinifex.
- Change-management records patch-apply timestamps for every Spinifex release.
- Operator subscribes to `mulgadc/spinifex`, `mulgadc/predastore`, `mulgadc/viperblock` releases and advisories.
- OS / kernel / hypervisor patching cadence documented and at least as strict as [§3](#3-remediation-slas).
- Annual review confirms at least one patch cycle completed within SLA in the prior 12 months, with evidence per [§6](#6-evidence-and-record-keeping).

---

# Malware Protection

URL: https://docs.mulgadc.com/docs/malware-protection
Category: Security and Compliance
Updated: 2026-08-19
Tags: security, compliance, cmmc, malware, antivirus, file integrity

Operator guide to host-based malware protection and file integrity monitoring on the Linux hosts running Spinifex services, aligned to CMMC Level 1 needs.

## Overview

**Audience:** Operators deploying Spinifex into environments subject to CMMC Level 1 (or organisations that otherwise require host-based malware protection).

**Scope:** The Spinifex nodes themselves — the Linux hosts running `spinifex-daemon`, `spinifex-awsgw`, `spinifex-nats`, `spinifex-predastore`, `spinifex-viperblock`, `spinifex-vpcd`, `spinifex-ui` and the OVN control plane. Guest VMs launched on the platform are the responsibility of the workload owner and are out of scope.

## CMMC Practices Covered

This guide captures the operator-side controls required to meet three CMMC Level 1 practices that Spinifex itself cannot enforce — they depend on host-based tooling chosen and operated by the deployment team.

| Practice | Title | Objective |
|----------|-------|-----------|
| SI.L1-3.14.2 | Malicious Code Protection | [a] Designated locations for malicious code protection are identified. [b] Protection from malicious code at designated locations is provided. |
| SI.L1-3.14.4 | Update Malicious Code Protection | [a] Malicious code protection mechanisms are updated when new releases are available. |
| SI.L1-3.14.5 | System & File Scanning | [a] Periodic scans of the information system are defined. [b] Periodic scans of the information system are performed. [c] Real-time scans of files from external sources are defined and performed as files are downloaded, opened, or executed. |

## Approach

Spinifex does not bundle or mandate a specific anti-malware product. Operators will typically have an existing endpoint security standard (CrowdStrike Falcon, Microsoft Defender for Linux, Wazuh, ClamAV, Sophos, ESET, etc.). This guide:

1. Names the Spinifex components and filesystem paths that must be included in scan and integrity-monitoring coverage (the "designated locations").
2. Specifies update and scan cadence required to meet SI.L1-3.14.4 and SI.L1-3.14.5.
3. Provides reference configurations for open-source stacks (ClamAV for AV, AIDE for file integrity, Wazuh for centralised monitoring) as a known-good baseline. Operators running a commercial EDR should map these locations and cadences into their existing product.

## 1. Designated Locations (SI.L1-3.14.2 [a])

All Spinifex nodes must include the following in their malware protection scope. Paths assume the production install layout (`/etc/spinifex` exists); development layouts under `$HOME/spinifex` are not supported for compliance deployments.

### 1.1 Executable Binaries

| Path | Contents |
|------|----------|
| `/usr/local/bin/spx` | Spinifex CLI and service entry point. All `spinifex-*.service` units invoke `spx service <name> start`, so every service binary is reached through this one path. |
| `/usr/bin/ovn-*`, `/usr/bin/ovs-*` | OVN / OVS control-plane and datapath binaries (installed via `apt` from the `ovn-central`, `ovn-host`, `openvswitch-switch` packages). |
| `/usr/lib/<arch>-linux-gnu/nbdkit/plugins/nbdkit-viperblock-plugin.so` | Viperblock NBD plugin loaded by `nbdkit` to expose EBS volumes as block devices. The exact path comes from `nbdkit --dump-config` (`plugindir=`); on Debian 13 amd64 it is `/usr/lib/x86_64-linux-gnu/nbdkit/plugins/`. |
| `/usr/local/share/ca-certificates/spinifex-ca.crt` | Cluster CA cert installed into the system trust store |

### 1.2 Configuration and Secrets

| Path | Contents |
|------|----------|
| `/etc/spinifex/` | Cluster config directory: `spinifex.toml`, `awsgw.toml`, `nats.conf`, and `predastore/predastore.toml`. |
| `/etc/spinifex/ca.pem`, `/etc/spinifex/ca.key` | Cluster root CA certificate and private key (leader only). |
| `/etc/spinifex/server.pem`, `/etc/spinifex/server.key` | Per-node TLS certificate and key. |
| `/etc/spinifex/master.key` | Master encryption key that wraps per-volume DEKs and IAM secrets. **Highest-value artifact on the node.** |
| `/etc/spinifex/systemd.env` | Service environment file (shared by all `spinifex-*.service` units). |

### 1.3 Service Units and Privilege Configuration

| Path | Contents |
|------|----------|
| `/etc/systemd/system/spinifex-*.service` | Unit files for daemon, awsgw, nats, predastore, viperblock, vpcd, ui. |
| `/etc/sudoers`, `/etc/sudoers.d/` | Any drop-ins granting privileges to service accounts (OVN/OVS management, NBD mounts, etc.). |
| `/etc/ssh/sshd_config`, `/etc/ssh/sshd_config.d/` | Host SSH daemon configuration. |

### 1.4 Runtime and Data Locations

| Path | Purpose | Notes |
|------|---------|-------|
| `/var/lib/spinifex/spinifex/` | Daemon state and local metadata. | Include in scan scope. |
| `/var/lib/spinifex/awsgw/` | AWS gateway state. | Include in scan scope. |
| `/var/lib/spinifex/vpcd/` | VPC daemon state. | Include in scan scope. |
| `/var/lib/spinifex/predastore/` | Local Predastore S3 backing data (chunks, metadata). | See exclusion note below. |
| `/var/lib/spinifex/viperblock/` | Local Viperblock WAL and chunk data for EBS volumes. | See exclusion note below. |
| `/var/lib/spinifex/nats/` | NATS JetStream data. | See exclusion note below. |
| `/run/spinifex/`, `/run/spinifex/nbd/` | Runtime sockets and PID files. | Integrity monitoring not meaningful (transient); exclude from AIDE. |
| `/var/log/spinifex/` | Service logs. | Scan for malicious content indicators; do not block on writes. |

**Performance exclusions.** Viperblock WAL, Predastore chunks, and NATS JetStream files are large, high-churn, block-level storage that contains customer VM data. Real-time on-access AV scanning of these paths is both a significant performance cost and a confused-deputy problem (the daemon, not the operator, is the writer). Cover these paths with **periodic scheduled scans** (SI.L1-3.14.5 [b]) but exclude them from real-time on-access hooks. Record the exclusion and its justification in the system security plan.

### 1.5 External File Ingest — Real-time Scan Required (SI.L1-3.14.5 [c])

The following are "files from external sources" within the meaning of 3.14.5 [c] and **must** be scanned at ingest, before Spinifex consumes them:

| Ingest point | What lands here | How it reaches the node |
|--------------|-----------------|-------------------------|
| VM image catalogue downloads | Debian / Ubuntu / Alpine / Rocky cloud images extracted during `spx admin images import`. | Spinifex downloads over HTTPS and verifies the catalogue SHA checksum. Operators layer AV on top: scan the downloaded image file before it is registered. |
| `ImportImage` / user-uploaded AMIs | Operator- or tenant-supplied VMDK/VHD/RAW images. | Uploaded to Predastore and registered via the EC2 API. |
| Cloud-init user-data | Base64 payloads attached to instances. | Passed via `RunInstances`; written to the CIDATA seed ISO before boot. |
| ISO / media attached to instances | Arbitrary ISO images attached as CD-ROM to VMs. | Uploaded to Predastore. |

Spinifex does not currently invoke an AV scanner on these ingest paths in-process. Until that integration exists, operators must either (a) use an on-access AV hook (ClamAV `clamonacc`, EDR file-write monitor) that sees Predastore writes, or (b) run a pre-ingest workflow that scans artifacts before they are uploaded to Spinifex.

## 2. Update Requirements (SI.L1-3.14.4)

Malicious code protection mechanisms must be updated when new releases are available. For signature-based AV this means signature feeds; for EDR products it means agent version pins. Apply to every node.

| Component | Cadence | Mechanism |
|-----------|---------|-----------|
| AV signatures (ClamAV `freshclam`, commercial EDR cloud feeds) | At least every 4 hours; real-time if the vendor supports it. | Systemd timer for `freshclam` (see [§4.1](#41-clamav-signature-based-av)); EDR agents typically self-update. |
| AV engine / agent version | Within 7 days of vendor release for non-critical; within 48 hours for vendor-marked critical. | OS package manager or vendor deployment pipeline; align with the flaw-remediation SLAs. |
| File-integrity baseline (AIDE) | Re-baseline on every Spinifex upgrade or config change. | Re-run `aide --init` as part of the upgrade runbook; commit the resulting hash to the change record. |

**Evidence:** the system security plan must record (a) which product is deployed, (b) how updates are delivered, and (c) a sample of update-success log entries from the last 30 days.

## 3. Scan Schedule (SI.L1-3.14.5)

### 3.1 Periodic Scans (objectives [a], [b])

| Target | Frequency | Type |
|--------|-----------|------|
| Designated locations [§1.1–§1.3](#11-executable-binaries) (binaries, config, secrets, unit files) | Daily | Full AV scan + AIDE integrity check. |
| Runtime data under `/var/lib/spinifex/*` ([§1.4](#14-runtime-and-data-locations)) | Weekly | Full AV scan. |
| Whole root filesystem | Monthly | Full AV scan. |

### 3.2 Real-time Scans (objective [c])

| Trigger | Action |
|---------|--------|
| Write to the external-ingest paths in [§1.5](#1-designated-locations-sil1-3142-a) | AV scan before the file is considered available. A positive detection must fail the ingest and emit a structured log line to `/var/log/spinifex/` and the operator's SIEM. |
| Execution of a binary in `/usr/local/bin/`, `/usr/bin/ovn-*`, `/usr/bin/ovs-*`, or load of `nbdkit-viperblock-plugin.so` from the nbdkit plugin dir | On-access AV check (ClamAV `clamonacc` or EDR equivalent). |

### 3.3 Logging

All scan runs — scheduled and real-time — must produce a log entry that includes scanner name, version, signature version, target, outcome, and duration. Forward to the same log collection used for Spinifex service logs.

## 4. Reference Configurations

The configurations below are reference baselines. They are known to meet the CMMC L1 objectives on a standard Debian 13 Spinifex node; they are not the only valid implementation. Operators running a commercial EDR should treat [§1](#1-designated-locations-sil1-3142-a) and [§3](#3-scan-schedule-sil1-3145) as the contract and configure their chosen product to match.

### 4.1 ClamAV (signature-based AV)

Install:

```bash
apt-get install -y clamav clamav-daemon clamav-freshclam clamonacc
```

Signature updates — drop-in `/etc/systemd/system/clamav-freshclam.timer.d/spinifex.conf`:

```ini
[Timer]
OnCalendar=*-*-* 00,04,08,12,16,20:00:00
Persistent=true
```

Enable and start:

```bash
systemctl enable --now clamav-freshclam.timer clamav-daemon.service clamav-clamonacc.service
```

On-access scanning — `/etc/clamav/clamd.conf` additions:

```
OnAccessIncludePath /usr/local/bin
OnAccessIncludePath /usr/lib/x86_64-linux-gnu/nbdkit/plugins
OnAccessIncludePath /etc/spinifex
OnAccessIncludePath /etc/systemd/system
OnAccessPreventionAction clamd-block
OnAccessExcludePath /var/lib/spinifex/viperblock
OnAccessExcludePath /var/lib/spinifex/predastore
OnAccessExcludePath /var/lib/spinifex/nats
```

Scheduled scans — `/etc/systemd/system/spinifex-clamav-scan.service`:

```ini
[Unit]
Description=Spinifex scheduled AV scan
After=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/bin/clamdscan --fdpass --multiscan \
    /usr/local/bin /usr/lib/x86_64-linux-gnu/nbdkit/plugins \
    /etc/spinifex /etc/systemd/system \
    /var/lib/spinifex/awsgw /var/lib/spinifex/vpcd \
    /var/lib/spinifex/spinifex /var/log/spinifex
StandardOutput=append:/var/log/spinifex/clamav-scheduled.log
StandardError=append:/var/log/spinifex/clamav-scheduled.log
```

Paired timer `spinifex-clamav-scan.timer`:

```ini
[Unit]
Description=Daily Spinifex AV scan

[Timer]
OnCalendar=*-*-* 03:15:00
Persistent=true

[Install]
WantedBy=timers.target
```

For the weekly `/var/lib/spinifex/*` data scan and the monthly root scan, add two additional timer/service pairs with `OnCalendar=weekly` / `OnCalendar=monthly` and the appropriate scan targets.

### 4.2 AIDE (file integrity monitoring)

Install:

```bash
apt-get install -y aide aide-common
```

Policy — `/etc/aide/aide.conf.d/90_spinifex`:

```
# Binaries — any change is suspicious
/usr/local/bin/spx                                f+p+u+g+s+m+c+sha256
# Viperblock NBD plugin — adjust <arch> for the node (see `nbdkit --dump-config`)
/usr/lib/x86_64-linux-gnu/nbdkit/plugins/nbdkit-viperblock-plugin.so  f+p+u+g+s+m+c+sha256

# Config and secrets
/etc/spinifex                                     f+p+u+g+s+m+c+sha256
/etc/systemd/system/spinifex-.*\.service$         f+p+u+g+s+m+c+sha256

# CA trust store
/usr/local/share/ca-certificates/spinifex-ca.crt  f+p+u+g+s+m+c+sha256

# Exclude high-churn runtime data (covered by scheduled AV instead)
!/var/lib/spinifex/viperblock
!/var/lib/spinifex/predastore
!/var/lib/spinifex/nats
!/var/log/spinifex
!/run/spinifex
```

Baseline:

```bash
aideinit
mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db
```

Daily check — a systemd timer invoking `aide --check`. Non-zero exit must alert. Re-baseline after every Spinifex upgrade, package upgrade, or config change and record the new hash.

### 4.3 Wazuh (centralised AV + FIM + alerting)

For multi-node clusters, a Wazuh agent on each Spinifex node feeding a central Wazuh manager is the simplest way to discharge 3.14.2 / 3.14.4 / 3.14.5 with consolidated evidence.

Key agent configuration (`/var/ossec/etc/ossec.conf` fragments):

```xml
<syscheck>
  <frequency>43200</frequency>   <!-- 12h FIM baseline -->
  <directories check_all="yes" realtime="yes">/etc/spinifex</directories>
  <directories check_all="yes" realtime="yes">/usr/local/bin</directories>
  <directories check_all="yes" realtime="yes">/etc/systemd/system</directories>
  <ignore>/var/lib/spinifex/viperblock</ignore>
  <ignore>/var/lib/spinifex/predastore</ignore>
  <ignore>/var/lib/spinifex/nats</ignore>
  <ignore>/run/spinifex</ignore>
</syscheck>

<rootcheck>
  <frequency>3600</frequency>
</rootcheck>
```

Pair with the Wazuh ClamAV integration (or a commercial AV module) for signature-based detection, and use the Wazuh manager's rule engine to alert on signature updates failing (SI.L1-3.14.4 evidence) and on scheduled-scan failures (SI.L1-3.14.5 evidence).

## 5. Operator Checklist

Use this list to confirm a node meets the three CMMC practices before the node is admitted to a production cluster:

- Host AV product installed, running, and reporting to a central console or log sink.
- Signature update timer/job running; last successful update within the cadence in [§2](#2-update-requirements-sil1-3144).
- AV and file-integrity policy covers every path in [§1.1–§1.4](#11-executable-binaries), with documented exclusions for [§1.4](#14-runtime-and-data-locations) high-churn paths.
- External-ingest paths in [§1.5](#1-designated-locations-sil1-3142-a) have real-time scanning in place (either at the Predastore write layer or in the operator's upload workflow).
- Daily, weekly, and monthly scheduled scans defined per [§3.1](#31-periodic-scans-objectives-a-b) and visible in recent run history.
- Scan and update logs forwarded to the same SIEM / log collector used for Spinifex service logs.
- AIDE (or equivalent FIM) baseline exists; re-baseline step is part of the Spinifex upgrade runbook.
- System security plan documents the deployed product, update mechanism, scan cadence, and exclusions.

---

# Media Sanitization and Disposal

URL: https://docs.mulgadc.com/docs/media-sanitization
Category: Security and Compliance
Updated: 2026-08-21
Tags: security, compliance, cmmc, media, sanitization, disposal, decommissioning

Operator guide to sanitizing and disposing of storage media used by Spinifex nodes, covering system disks, Viperblock and Predastore volumes, and key tokens.

## Overview

**Audience:** Operators decommissioning, reassigning, returning under warranty, or disposing of hardware that has been used to run Spinifex nodes in environments subject to CMMC Level 1, or any site that requires documented media sanitization.

**Scope:** Every storage medium that has, or may have, held Federal Contract Information (FCI) as part of a Spinifex deployment — node system disks, Viperblock WAL/chunk disks, Predastore object stores, NATS JetStream disks, removable master-key tokens, backup media, and any loose drives that previously occupied a Spinifex role. Guest VM internal media (disks as seen from inside a tenant VM) is the workload owner's responsibility and out of scope.

## CMMC Practices Covered

| Practice | Title | Objective |
|----------|-------|-----------|
| MP.L1-3.8.3 | Media Disposal | [a] System media containing FCI is sanitized or destroyed before disposal. [b] System media containing FCI is sanitized before it is released for reuse. |

## Approach

NIST SP 800-88 Rev 1 defines three sanitization categories — **Clear**, **Purge**, and **Destroy** — chosen by the confidentiality of the data and whether the medium will leave the operator's control. Spinifex discharges MP.L1-3.8.3 along two tracks:

1. **Volume-level sanitization** for EBS volumes and VM disks returning to the free pool. Handled in-platform by **cryptographic erase** — the per-volume data encryption key (DEK) is deleted, rendering the ciphertext in Predastore S3 irrecoverable. This is the NIST SP 800-88 **Purge**-level method for encrypted media.
2. **Whole-drive sanitization** for physical media leaving a Spinifex node — drive replacement, node retirement, warranty return, resale, or destruction. The operator runs the sanitization; Spinifex does not and cannot reach the firmware commands required. This guide prescribes the method per media type and the records to keep.

The key principle: every piece of media that ever held plaintext FCI (or keys that wrapped FCI ciphertext) must be sanitized to at least **Purge** before release for reuse, and to **Purge** or **Destroy** before leaving operator control. When in doubt, Destroy.

## 1. Media in Scope

Any medium in any of these roles on a Spinifex node has held, or may have held, FCI:

| Medium | Typical hardware | Contents |
|--------|------------------|----------|
| Node system disk | NVMe/SATA SSD | `/etc/spinifex/master.key`, cluster CA key, per-node TLS keys, service configs, logs, journal, swap. |
| Viperblock WAL device | NVMe/SATA SSD | Plaintext of in-flight block writes before chunking. High-churn, short residency, but FCI lands here. |
| Viperblock chunk cache / local backing | NVMe/SATA SSD or HDD | Encrypted volume chunks. |
| Predastore object store | NVMe/SATA SSD or HDD | All S3 object data — AMIs, snapshots, user-uploaded artifacts, IAM state files, tenant data. |
| NATS JetStream disk | NVMe/SATA SSD | IAM NATS token, cluster metadata, pending job state, NATS KV entries (including wrapped DEKs). |
| Removable master-key media | USB flash token, HSM, smart card | Master encryption key (USB-mount path). Destroying this sanitizes every DEK-encrypted volume cluster-wide. |
| Backup media | LTO tape, removable HDD/SSD, off-site cloud backup | Any of the above, point-in-time. |
| BMC / iDRAC / iLO storage | Embedded flash | Console recordings, SEL logs, BMC credentials, cached operator certificates. Sanitize via the BMC "Reset to defaults" / "Erase user data" command before disposal. |
| Switch / router config storage | Embedded flash | Cluster VLAN, management IPs, ACLs. Operator network kit, out of Spinifex scope but noted for completeness. |
| Optical / write-once media | DVD/BD-R | If used to transport keys or images. Always Destroy. |

Any drive whose history cannot be traced — pulled from a spares bin, recovered from a failed host, found unlabelled — must be treated as though it held FCI.

## 2. Sanitization Method Selection

The method depends on (a) media type and (b) whether the media remains inside the protected boundary after sanitization.

| Scenario | Minimum method | Reference |
|----------|----------------|-----------|
| Encrypted volume returning to free pool, DEK deletable | **Cryptographic Erase** (Purge) via DEK deletion | [§3](#3-volume-level-sanitization-before-reuse-mpl1-383-b) |
| SSD / NVMe leaving the node | **Cryptographic Erase** + **Purge** (SANITIZE block-erase or ATA Secure Erase) | [§4.1](#41-nvme-ssd) |
| Magnetic HDD leaving the node, operational | **Purge** (single-pass overwrite with verify, or ATA Secure Erase on drives that support it) | [§4.3](#43-hdd) |
| Magnetic HDD leaving the node, faulty (not writable) | **Destroy** (degauss followed by shred/incinerate) | [§4.3](#43-hdd) |
| Any media leaving operator control (resale, warranty, disposal) | **Destroy** if residual confidentiality concern remains after Purge; otherwise Purge + third-party attestation | [§4.4](#44-destruction) |
| Tape | **Destroy** (degauss + shred) — overwrite is not reliable on LTO with compression | [§6](#6-removable-and-backup-media) |
| Optical write-once | **Destroy** (shred/incinerate) | [§6](#6-removable-and-backup-media) |
| BMC / switch flash | Vendor "factory reset + erase user data"; Destroy chip if procedure is not available | [§1](#1-media-in-scope) |

**Cryptographic erase is only valid when** the key was generated and handled under modern cryptographic hygiene (AES-256 or equivalent, key never exfiltrated, key store itself sanitized) and the encrypted data was not also written anywhere in plaintext. For Spinifex volumes these conditions hold.

**Self-encrypting drives (SEDs)** support a single-command cryptographic erase via the TCG Opal "RevertSP" / `sedutil-cli --revertNoErase` or the ATA `SECURITY ERASE UNIT ENHANCED` that triggers the on-drive MEK rotation. When using an SED as a node system or data disk, record the SED model and method in the decommissioning procedure.

## 3. Volume-Level Sanitization — Before Reuse (MP.L1-3.8.3 [b])

Applies to EBS volumes, AMIs, and snapshots whose underlying storage will be reused for a different tenant or workload.

### 3.1 Cryptographic Erase via DEK Deletion

Deletion is sanitization:

- `DeleteVolume` removes the wrapped DEK from NATS KV. The chunks in Predastore S3 remain but are unrecoverable ciphertext. No further operator action is required.
- `TerminateInstances` with the default `DeleteOnTermination=true` on root volumes performs the same cryptographic erase for the terminated instance's root.
- `DeleteSnapshot` removes the snapshot's DEK wrapper; the underlying chunks become irrecoverable ciphertext.

This is the sanitization-before-reuse path for every in-platform object. Operators do not need to take separate action — the API call is the sanitization.

### 3.2 What Happens to the Physical Storage

Cryptographic erase leaves ciphertext chunks in Predastore. These chunks are eventually overwritten as new volumes reuse the space. For CMMC purposes, the ciphertext without the DEK satisfies Purge. The underlying drive still requires whole-drive sanitization when it eventually leaves the cluster — see [§4](#4-whole-drive-and-node-decommissioning-before-disposal-mpl1-383-a).

## 4. Whole-Drive and Node Decommissioning — Before Disposal (MP.L1-3.8.3 [a])

Applies when a physical drive — or a whole node — leaves the Spinifex cluster: drive swap, node retirement, warranty return, resale, recycling, destruction.

### 4.1 NVMe SSD

Preferred: the NVMe **SANITIZE** command with the **Block Erase** action. Supported on most enterprise NVMe drives.

```bash
# Confirm support
nvme id-ctrl /dev/nvme0 | grep -i sanicap

# Purge — block erase
nvme sanitize /dev/nvme0 --sanact=2

# Poll until complete
nvme sanitize-log /dev/nvme0
```

Fallback for drives without SANITIZE support: NVMe **Format** with Secure Erase set to 1 (`--ses=1`), or vendor tool. Cryptographic Erase (`--sanact=4`) is acceptable for SEDs whose encryption posture is documented; otherwise Block Erase is the safer default.

Boot media: NVMe targets must be the non-boot drive when running from the OS. For the node system disk, boot a sanitization-purpose live USB (e.g. [PartedMagic](https://partedmagic.com/), the vendor's diagnostic ISO) or pull the drive and sanitize in a dedicated sanitization workstation.

### 4.2 SATA/SAS SSD

Preferred: **ATA Secure Erase (Enhanced)** via `hdparm`:

```bash
# Confirm support and not frozen
hdparm -I /dev/sdX | grep -A1 "Security"

# If frozen, power-cycle (suspend/resume or hot-swap) without unplugging the OS disk

# Set password and issue enhanced erase
hdparm --user-master u --security-set-pass p /dev/sdX
hdparm --user-master u --security-erase-enhanced p /dev/sdX
```

Fallback: vendor Secure Erase utility, or `blkdiscard --secure /dev/sdX` on drives that report `TRIM deterministic + RZAT`. A single-pass overwrite with `shred -v -n 1 /dev/sdX` is **not** sufficient for SSDs because of wear-levelling reserve blocks — use Secure Erase or Destroy.

### 4.3 HDD

Preferred: **ATA Secure Erase** (same `hdparm` sequence as [§4.2](#42-satasas-ssd)). Most modern HDDs support it.

Fallback: single-pass overwrite with verify:

```bash
shred -v -n 1 -z /dev/sdX
# or
dd if=/dev/zero of=/dev/sdX bs=1M status=progress && \
  cmp /dev/zero /dev/sdX          # expect "EOF on /dev/zero" only
```

For HDDs that fail or refuse sanitization, or that held high-sensitivity data, **degauss then Destroy** — degaussing alone renders a modern HDD non-functional, so there is no reuse path after degauss. Degaussing requires an NSA/CSS-listed degausser appropriate to the drive's coercivity.

### 4.4 Destruction

When a drive cannot be sanitized (failed, SED with no working password, sanitize aborts) or when leaving operator control with any residual confidentiality concern, Destroy:

- **SSD/NVMe:** shred to ≤2 mm particle size (NSA/CSS EPL-listed shredder) or incinerate. Crushing alone is not sufficient for modern flash — chips can survive.
- **HDD:** degauss (on drives with magnetic platters) followed by shredding, drilling multiple holes through the platters and the head assembly, or incineration.
- **Tape, optical:** shredding.
- **Chain of custody:** transport destroyed media inside tamper-evident containers. The destruction record ([§7](#7-evidence-and-record-keeping)) must name the destruction method, destruction vendor (if third-party), and operator witness.

Third-party destruction vendors must provide a certificate of destruction listing every drive serial. Reconcile against the asset register before the destruction record is closed.

### 4.5 Node Decommissioning Runbook

When retiring a whole node:

1. **Drain** — migrate or terminate all instances hosted on the node; confirm Predastore and Viperblock roles are reassigned if the node held one. Volume DEKs of terminated instances are already cryptographically erased per [§3.1](#31-cryptographic-erase-via-dek-deletion).
2. **Deregister** — raise a change ticket; the operator removes the node from the cluster (`spx admin node remove`) so NATS routes, OVN chassis, and predastore distributed membership drop it.
3. **Stop services and unmount** — `systemctl stop 'spinifex-*'`, `nbdkit` sessions, OVN agent. Confirm `/run/spinifex/nbd/` is empty.
4. **Destroy on-disk keys** — see [§5](#5-key-destruction). This is the single most important step: sanitization of volume data reduces to sanitization of keys once data is encrypted.
5. **Sanitize each drive** by media type per [§4.1](#41-nvme-ssd) – [§4.3](#43-hdd). Every drive bay is sanitized, including any unpopulated cache or WAL devices.
6. **Verify** — capture the output of the sanitize/erase command (exit code + sanitize-log for NVMe, `hdparm -I` "not enabled, not locked" for ATA, overwrite verify for HDD overwrite) into the decommissioning record.
7. **Label** drives "Sanitized — <method> — <date> — <operator>" before they leave the rack or enter a transit container. Unlabelled drives are treated as unsanitized if reintroduced.
8. **Record** per [§7](#7-evidence-and-record-keeping).

## 5. Key Destruction

Cryptographic erase relies on the key being unrecoverable. A drive that previously held `/etc/spinifex/master.key` must have that key destroyed — otherwise an attacker with both the drive image and the key recovers the cleartext even after "erase".

| Key | Location | Destruction |
|-----|----------|-------------|
| `/etc/spinifex/master.key` | Node system disk (every node) | Overwrite the file (`shred -u /etc/spinifex/master.key`) before the drive is sanitized, in addition to drive sanitization. Once the master key is gone cluster-wide, every DEK it wrapped is effectively destroyed. |
| Cluster CA private key (`/etc/spinifex/ca.key`) | Leader node system disk | Same treatment. Loss of the CA key does not sanitize data, but prevents impersonation of the cluster identity post-disposal. |
| Per-node TLS private key (`/etc/spinifex/server.key`) | Each node system disk | `shred -u` before drive sanitization. |
| Wrapped DEKs in NATS KV | `/var/lib/spinifex/nats/` on NATS-carrying nodes | Deleted via the `DeleteVolume` API; the KV record is compacted. Sanitization of the NATS JetStream disk per [§4](#4-whole-drive-and-node-decommissioning-before-disposal-mpl1-383-a) ensures no recoverable copies. |
| Removable master-key media (USB) | USB flash token | Physically destroy the token when the cluster is decommissioned; alternatively retain inside a secured location as evidence of cluster-wide cryptographic erase. Do not reuse for another cluster without full sanitization (see [§6](#6-removable-and-backup-media)). |
| Backup-copy keys | Any off-site or escrow copy | Destroy simultaneously with the primary. A surviving backup defeats the cryptographic erase. |

If even one copy of the master key survives and the encrypted drives are recoverable, cryptographic erase has **not** been achieved. Track every key copy in the device register ([Physical Security Guide §5](https://docs.mulgadc.com/docs/physical-security-guide#5-manage-physical-access-devices-pel1-3105)).

## 6. Removable and Backup Media

| Media | Sanitization |
|-------|--------------|
| USB flash (master-key token, key-transport USB, image-import USB) | Destroy. The price of a replacement does not justify the risk of partial wear-levelled sanitization. Record the serial in the device register and the destruction in the decommissioning log. |
| External HDD/SSD (backup) | Treat as [§4.2](#42-satasas-ssd) / [§4.3](#43-hdd). |
| LTO tape | Degauss with a tape-rated degausser, then shred. Overwrite is not reliable under LTO hardware compression and is not accepted by 800-88 for tape Purge. |
| Optical (CD/DVD/BD-R) | Shred or incinerate. |
| Off-site cloud backup | Issue the cloud provider's delete-and-purge API; retain the provider's deletion-confirmation receipt. For cryptographically wrapped backups, destroy the wrapping key cluster-side and retain that deletion as evidence. |
| Printed material (console photos, recovery codes, admin handover sheets) | Cross-cut shred. |

Any removable medium that has entered the protected boundary and been written to must be tracked in the device register (see [Physical Security Guide §5](https://docs.mulgadc.com/docs/physical-security-guide#5-manage-physical-access-devices-pel1-3105)) and accounted for at decommissioning.

## 7. Evidence and Record Keeping

For CMMC assessment, retain the following for at least three years (or longer where contract policy requires):

- **Asset register** listing every drive and removable medium by serial number, media type, role in the cluster (system / WAL / chunk / predastore / NATS / backup), and in-service / decommissioned status.
- **Decommissioning record** per drive or node with: date, operator, sanitization method, command output or tool report, verification result, final disposition (reuse / return / destroy), destination (e.g. "warranty RMA #12345", "shredder vendor X, CoD #6789").
- **Certificates of destruction** from third-party vendors, reconciled against the asset register.
- **Key-destruction record** confirming `master.key`, CA key, and per-node keys were shredded before drive sanitization.
- **Exceptions log** — faulty drives that could not be sanitized and were destroyed instead; any sanitize failures and their resolution.
- **Annual attestation** from the operator confirming the procedures in this guide operated for the prior 12 months, signed by the named security owner.

Cross-reference each decommissioning record to the change ticket that drove the retirement, so drive disposal can be audited against cluster state change.

## 8. Operator Checklist

Use this list to confirm a Spinifex deployment meets MP.L1-3.8.3:

- Asset register enumerates every drive and removable medium by serial, media type, and role.
- Every volume-producing path uses encryption so that `DeleteVolume` is sanitization.
- Node-decommissioning runbook exists and matches [§4.5](#45-node-decommissioning-runbook), including the drain → deregister → key-destroy → drive-sanitize sequence.
- Sanitization method is pre-selected per media type per [§2](#2-sanitization-method-selection); tool availability (hdparm, nvme-cli, vendor ISOs, degausser, shredder or destruction vendor) is verified before a decommissioning begins.
- `master.key`, CA key, and per-node TLS keys are shredded before the drive holding them is sanitized.
- Every backup copy of any key is destroyed at the same time as the primary; surviving backup copies are tracked and accounted for.
- Removable media (USB, tape, optical) is destroyed rather than erased unless the medium is an SED with documented cryptographic erase.
- Decommissioning records capture method, tool output, verification, final disposition, and operator; certificates of destruction are reconciled against the asset register.
- Drives awaiting sanitization or awaiting pickup by a destruction vendor are held inside the physical protection boundary ([Physical Security Guide §1](https://docs.mulgadc.com/docs/physical-security-guide#1-protected-assets)) with chain of custody logged.
- System security plan references this guide and names the sanitization tooling, destruction vendor (if any), retention period for records, and the security owner attesting annually.

---

# External Connection Inventory

URL: https://docs.mulgadc.com/docs/network-connections
Category: Security and Compliance
Updated: 2026-09-04
Tags: security, compliance, cmmc, network, connections, boundary

Operator inventory of every inbound listener and outbound connection on Spinifex nodes, with ports, protocols, and purpose, for documented CMMC Level 1 sites.

## Overview

**Audience:** Operators deploying Spinifex into environments subject to CMMC Level 1, or any site that requires a documented inventory of system connections.

**Scope:** Network connections originated by or terminated at the Spinifex nodes — the Linux hosts running `spinifex-daemon`, `spinifex-awsgw`, `spinifex-nats`, `spinifex-predastore`, `spinifex-viperblock`, `spinifex-vpcd`, `spinifex-ui`, and the OVN control plane. Guest VM traffic is the workload owner's responsibility and out of scope.

**Boundary definition.** For the purposes of this document:

- **External** means outside the Spinifex cluster's trusted network perimeter — the public internet, the operator's corporate network, tenant users of the AWS API, and guest VMs.
- **Internal** means between Spinifex nodes inside the cluster subnet(s) defined in `spinifex.toml`.

AC.L1-3.1.20 applies specifically to **external** connections. Internal cluster connections are documented here as well so operators can build an accurate firewall policy.

## CMMC Practices Covered

This guide addresses AC.L1-3.1.20. The related boundary-protection practice SC.L1-3.13.1 is covered by OVN ACL and security-group enforcement in `vpcd` and is documented separately.

| Practice | Title | Objective |
|----------|-------|-----------|
| AC.L1-3.1.20 | External Connections | [a] Connections to external systems are identified. [b] The use of external systems is identified. [c] Connections to external systems are verified. [d] The use of external systems is verified. [e] Connections to external systems are controlled/limited. [f] The use of external systems is controlled/limited. |

## Approach

Spinifex has a small, enumerable set of network surfaces:

1. **Inbound listeners** — the TCP/UDP ports each node binds. These are the attack surface exposed to whoever can reach the node.
2. **Outbound connections** — the destinations the node's services reach out to. Today this is a short list: peer Spinifex nodes, OS image mirrors, and install telemetry.
3. **Cross-node connections** — inter-node control- and data-plane traffic inside the cluster subnet.

The inventory in [§1](#1-inbound-listeners)–[§2](#2-outbound-connections) satisfies objectives [a]/[b]. The **Auth / Verification** columns throughout satisfy [c]/[d]. [§4](#4-limiting-controls) and [§5](#5-configuration-surface) satisfy [e]/[f]. The default install meets [c]–[f] for every listed connection; the operator's remaining work is to record the inventory in the system security plan, apply host/network firewall rules per [§4](#4-limiting-controls), and audit [§5](#5-configuration-surface) on a recurring schedule.

## 1. Inbound Listeners

"Scope" classifies intended reach. It maps onto the node's network planes — see
*Planes and scope* below, because a node with fewer NICs collapses them:

- **External** — reachable by tenant/operator networks, on the `wan` plane. Authenticated and TLS-protected.
- **Cluster** — reachable only from peer Spinifex nodes, on the `lan` plane. Operator must restrict via host or network firewall.
- **Encap** — reachable only from peer chassis, on the `vpc` plane. Carries the tenant overlay and the IPsec that protects it.
- **Guest** — bound to a per-instance interface and reachable only by that instance's VM.
- **Localhost** — bound to `127.0.0.1`; not reachable off-node.

The listener invariant tests (`spinifex/network/invariants` and the multinode e2e suite) read
this table and fail any Cluster- or Encap-scope port found bound to the wildcard address,
unless that row's Purpose or Auth text contains the exact phrase **"binds the wildcard by
design"**. That phrase is load-bearing, not incidental wording — a row that merely mentions
"wildcard" or "0.0.0.0", negated or not, grants no exception. Adding a new wildcard-bound
Cluster/Encap listener means adding that literal phrase to its row, not just describing the
behavior in other words.

| Port | Service | Protocol | Scope | Purpose | Auth / Verification |
|------|---------|----------|-------|---------|--------------------|
| 9999 | spinifex-awsgw | HTTPS | External | AWS-compatible API (EC2, S3, ELBv2, IAM) — customer endpoint | AWS SigV4 + TLS (cluster CA) |
| 3000 | spinifex-ui | HTTPS | External | Operator web dashboard | Session cookie + TLS |
| 22 | OpenSSH | SSH | External | Operator administration | Key-based auth (operator-managed) |
| 53 | northstar | DNS (UDP + TCP) | External | Authoritative DNS for the cluster's zones, resolved directly by instances and by operator networks. Binds the node's advertise (wan) address specifically, not the wildcard, so it does not collide with the `systemd-resolved` stub. | None — public authoritative DNS |
| 8443 | spinifex-predastore (gate) | HTTPS | External | S3-compatible object storage (AMIs, snapshots, user objects). S3 is a public plane: the gate binds `0.0.0.0` by design. | AWS SigV4 + TLS |
| 4432 | Formation server | HTTPS | Cluster (bootstrap only) | Cluster join coordination; active only while a join token is valid. Binds the node's `--bind` (lan) address. See *Formation port lifecycle* below. | Short-lived bearer token + TLS¹ |
| 4222 | spinifex-nats (client) | NATS + TLS | Cluster | Internal service bus for EC2/EBS/VPC/S3 handlers | Token + mutual TLS (cluster CA) |
| 4248 | spinifex-nats (cluster) | NATS + TLS | Cluster | Inter-node NATS federation | Token + mutual TLS (cluster CA) |
| 5300 | northstar | DNS (UDP + TCP) | Cluster | Forward target for every node's per-instance DNS shim, dialled cross-node. Binds the wildcard by design. | None — restrict by firewall |
| 6660 | predastore (blob node) | QUIC / UDP | Cluster | Erasure-coded object shard transport between hosts. Multi-node clusters only — see *Predastore ports* below. | TLS 1.3, server certificate verified against the cluster CA |
| 7660 | predastore (meta node) | QUIC / UDP | Cluster | Raft consensus over global state — buckets and the object index — between hosts. Multi-node clusters only. | TLS 1.3, server certificate verified against the cluster CA |
| 8660 | predastore (admin) | HTTP | Cluster | `/healthz` and `/readyz`. Readiness names the blob peers a gate cannot reach, so it must not face the WAN. Binds the host's cluster `bind_addr`, never the wildcard. | None — unauthenticated by design; restrict by firewall |
| 6641 | OVN Northbound DB (client) | OVSDB/TCP | Cluster | Logical network topology consumed by vpcd. Binds `127.0.0.1` plus the node's lan-plane address (`--lan-addr`), never the wildcard address. On a node with no separate lan plane that address is the public one, and a host firewall is the only remaining control. | Cluster network only; TLS planned |
| 6642 | OVN Southbound DB (client) | OVSDB/TCP | Cluster | Chassis / port / MAC binding state. Binds `127.0.0.1` plus the node's lan-plane address (`--lan-addr`), never the wildcard address. On a node with no separate lan plane that address is the public one, and a host firewall is the only remaining control. | Cluster network only; TLS planned |
| 6643 | OVN Northbound DB (RAFT) | OVSDB/TCP | Cluster | NB database RAFT replication between the 3 quorum nodes | Cluster network only; TLS planned |
| 6644 | OVN Southbound DB (RAFT) | OVSDB/TCP | Cluster | SB database RAFT replication between the 3 quorum nodes | Cluster network only; TLS planned |
| 6081 | OVN (Geneve) | UDP | Encap | Tenant traffic overlay between chassis. A kernel UDP-tunnel socket, so packets are delivered locally and traverse the host's netfilter input hook before OVS sees them — a host firewall must accept them explicitly. The socket is opened by the kernel tunnel driver and takes no bind address, so it binds the wildcard by design and reach must be restricted by firewall. | None — see 500/4500 |
| 500, 4500 | strongSwan `charon` | IKEv2 / UDP | Encap | IKE and NAT-T for the IPsec protecting Geneve, managed entirely by `ovs-monitor-ipsec`. Binds the wildcard by design (the upstream strongSwan default, accepted rather than overridden), so reach must be restricted by firewall. | Certificate-based, against the cluster CA |
| — | ESP | IP proto 50 | Encap | The IPsec payload itself, once IKE has negotiated an SA | Cluster CA |
| 8222 | spinifex-nats (monitoring) | HTTP | Localhost | `varz`/`subsz` metrics consumed by the daemon | Loopback only |
| 323 | chronyd | NTP | Localhost | Time sync client control socket | Loopback only |
| 169.254.169.254:80 | spinifex-vpcd (IMDS) | HTTP | Guest | Instance metadata service. One socket per instance, bound to that instance's `ime-*` interface. Terminates on the host, so it traverses the netfilter input hook. | Instance identity by interface; IMDSv2 tokens |
| 169.254.169.253:53 | spinifex-vpcd (VPC DNS) | DNS (UDP + TCP) | Guest | Per-VPC DNS resolver, forwarding to northstar `:5300` on peer nodes. Same per-instance binding as above. | Instance identity by interface |
| socket / dynamic TCP | nbdkit (Viperblock) | NBD | Host-local / cluster | Block device transport for guest EBS volumes | Unix socket by default; TCP only in remote/DPU mode |

**Planes and scope.** A node resolves three planes — `wan`, `lan` and `vpc` — from its
interfaces, and collapses `vpc` ← `lan` ← `wan` when a plane has no interface of its own. On
a single-NIC node every scope in this table lands on the public address, so **Cluster** and
**Encap** describe intent, not a guarantee. Verify with `ss -tulnp` against the node's actual
addresses rather than assuming the classification holds.

¹ **Formation port lifecycle.** 4432 opens during `spx admin init` / `spx admin join` while a bootstrap token is outstanding and closes once the cluster is formed (token TTL default 30 min, `--token-ttl`). The server presents an ephemeral self-signed cert that pre-dates trust bootstrap, so the joining node does not verify the certificate chain for this single dial. Authenticity rests on the operator supplying the leader address out-of-band plus possession of the bearer token. Document in the security plan so reviewers do not flag 4432 as a persistent open port.

**Predastore ports.** A Predastore cluster is described in `/etc/spinifex/predastore/predastore.toml` as a set of `[[host]]` blocks — one per machine, each running a single process — with the nodes pinned to it declared under `[[host.node]]`. There are three roles: a `gate` serving the S3 API, a `blob` node holding erasure-coded object shards, and a `meta` node in the Raft quorum over global state. Ports must be unique within a host but are not unique across the cluster, so **every machine uses the same three ports** — 8443, 6660 and 7660. It is three fixed ports per machine, not a range, and adding machines does not widen it.

Nodes on the same host talk over an in-process pipe and bind no socket at all, so a single-node install opens only 8443; 6660 and 7660 appear only once a second host exists. The gate is dialled by S3 clients but never by peer nodes, so it binds no QUIC socket of its own — it takes an ephemeral UDP port to dial out with.

**Development-only listeners.** When `dev_networking=true`, QEMU opens arbitrary host TCP ports for SSH port-forwarding into guest VMs. Production installs (the `/etc/spinifex` layout) do not enable this; it must not appear on compliance nodes.

## 2. Outbound Connections

Spinifex nodes initiate a small, fixed set of outbound connections.

**To external destinations:**

| Destination | Purpose | Protocol | Verification |
|-------------|---------|----------|--------------|
| `https://cloud.debian.org/images/cloud/trixie/latest/` | Debian 13 cloud image download | HTTPS | TLS + checksum verification |
| `https://cloud-images.ubuntu.com/releases/resolute/release/` | Ubuntu 26.04 LTS cloud image download | HTTPS | TLS + checksum verification |
| `https://dl.rockylinux.org/pub/rocky/10/images/` | Rocky Linux 10 cloud image download | HTTPS | TLS + checksum verification |
| `https://dl-cdn.alpinelinux.org/alpine/` | Alpine Linux cloud image download | HTTPS | TLS + checksum verification |
| `https://d2yp8ipz5jfqcw.cloudfront.net` | Alpine image for managed HAProxy load-balancer | HTTPS | TLS + checksum verification |
| `https://install.mulgadc.com/install` | One-shot install telemetry POST on `spx admin init` / `join`. | HTTPS | TLS |

**To peer nodes (cluster-internal):** NATS federation (4248), Predastore S3 (8443), OVN NB/SB (6641/6642), northstar DNS forwarding (5300), and the Geneve/IPsec overlay (6081, 500, 4500, ESP) — see [§3](#3-cross-node-internal-connections) for encryption and verification of each. The daemon also polls local NATS monitoring at `127.0.0.1:8222/varz` (loopback HTTP). It opens no connection to Predastore for status: the storage topology it reports comes from reading `predastore.toml`, and no Predastore node serves a status endpoint.

**Update checks and metadata.** Spinifex does not check for updates and does not consume a cloud metadata service (`169.254.169.254` is served *by* the cluster to guest VMs). Node software updates come from the operator's OS package channel. The install-telemetry endpoint above is the only vendor-operated destination contacted by a node; closed-egress deployments should disable it and record the opt-out in the security plan.

**Air-gapped deployments.** The image URLs above are the only destinations needed for the standard image catalogue. Mirror them locally and use `spx admin images import --file` with pre-staged files. Telemetry must also be disabled. See [Air-Gapped Install](https://docs.mulgadc.com/docs/install-airgapped).

## 3. Cross-Node (Internal) Connections

Control-plane and data-plane traffic between Spinifex nodes, for completeness and firewall planning:

| Connection | Port(s) | Encryption / Auth | Notes |
|-----------|---------|-------------------|-------|
| NATS cluster routes | 4248 | Mutual TLS + cluster token | Full mesh between NATS servers |
| Predastore S3 (gate) | TCP 8443 | TLS + AWS SigV4 | Cross-node object reads/writes |
| Predastore blob | UDP 6660 | QUIC with TLS 1.3; server certificate verified against the cluster CA | Erasure-coded object shards. Same port on every machine. |
| Predastore meta | UDP 7660 | QUIC with TLS 1.3; server certificate verified against the cluster CA | Raft consensus over buckets and the object index. Same port on every machine. |
| OVN NB/SB (client) | 6641 / 6642 | Cluster network only (TLS planned) | Network control plane; vpcd and ovn-controller dial the quorum |
| OVN NB/SB (RAFT) | 6643 / 6644 | Cluster network only (TLS planned) | NB/SB database replication across the 3 quorum nodes |
| OVN tunnels (Geneve) | UDP 6081 | Encapsulated by IPsec when `network.ipsec_enabled` is true, which is the default on multi-node clusters | Tenant traffic overlay between chassis, on the `vpc` plane |
| OVN IPsec (IKE / NAT-T / ESP) | UDP 500, UDP 4500, IP proto 50 | Certificate-based against the cluster CA, negotiated by `ovs-monitor-ipsec` | Protects the Geneve tunnels above. OVN-native only — no layer manages strongSwan directly. |
| Instance DNS forwarding | UDP/TCP 5300 | None | Each node's per-instance DNS shim forwards guest queries to peer nodes' northstar `:5300` |

Nodes **must** sit on a network segment that is not routed to tenant/guest VMs or to the internet. The Predastore blob and meta transports and the OVN DBs are cluster-internal and must not be reachable from anywhere else.

## 4. Limiting Controls

Default external surface is five listeners — **9999** (AWS API), **3000** (UI), **22** (SSH), **8443** (S3) and **53** (DNS) — plus **4432** transiently during bootstrap. Every other listener is cluster- or encap-scoped and the operator must enforce this with a host firewall or an upstream network ACL.

The nodes ship no firewall policy today. Until one is installed, this is the operator's responsibility and the reference below is the starting point.

> **Read the notes under the ruleset before applying it.** A default-deny input policy that
> omits any of the loopback, conntrack, `ime-*` or Geneve rules will break instance
> networking, guest metadata or your own SSH session. Apply it to one node and verify before
> applying it cluster-wide.

```
table inet spinifex_filter {
  chain input {
    type filter hook input priority filter; policy drop;

    ct state established,related accept
    ct state invalid drop
    iif lo accept

    # Guest metadata and per-VPC DNS terminate on the host, on per-instance
    # interfaces. Omitting this breaks cloud-init, instance role credentials
    # and all guest DNS.
    iifname "ime-*" accept

    # Path MTU discovery is not optional under a Geneve overlay: dropping
    # destination-unreachable produces silent blackholes on large flows.
    icmp type { echo-request, destination-unreachable, time-exceeded, parameter-problem } accept
    icmpv6 type { echo-request, destination-unreachable, packet-too-big, time-exceeded,
                  parameter-problem, nd-neighbor-solicit, nd-neighbor-advert,
                  nd-router-advert } accept

    # External, from anywhere
    tcp dport { 22, 3000, 8443, 9999 } accept
    tcp dport 53 accept
    udp dport 53 accept

    # Cluster, from peer nodes only. Replace with your nodes' lan-plane
    # addresses — a CIDR does not generalise to nodes on different subnets.
    ip saddr { 10.0.1.1, 10.0.1.2, 10.0.1.3 } tcp dport { 4222, 4248, 4432, 5300, 6641, 6642, 6643, 6644, 8660 } accept
    ip saddr { 10.0.1.1, 10.0.1.2, 10.0.1.3 } udp dport { 5300, 6660, 7660 } accept

    # Encap, from peer chassis only. Replace with your nodes' vpc-plane
    # addresses. Geneve arrives as host-local UDP and must be accepted here.
    ip saddr { 10.0.2.1, 10.0.2.2, 10.0.2.3 } udp dport { 6081, 500, 4500 } accept
    ip saddr { 10.0.2.1, 10.0.2.2, 10.0.2.3 } meta l4proto esp accept
  }
}
```

Notes that make the difference between this working and locking you out:

- **Use a dedicated table and never flush the others.** `vpcd` writes MASQUERADE and per-EIP FORWARD rules into the `ip filter` and `ip nat` tables, and only reinstalls them when the service starts. `iptables -F`, `nft flush ruleset`, `ufw enable` and `firewalld` all destroy them silently, and instance networking stays broken until the next restart.
- **Do not add a `forward` hook.** nftables evaluates every table registered on a hook and any `drop` is final, so a default-deny forward chain here would override `vpcd`'s accepts in the other table and break every routed-NAT instance and every EIP. Filtering forwarded guest traffic is the security group's job, not the host firewall's.
- **`output` is deliberately untouched.** The metadata service's reply path egresses through per-instance policy routing; filtering `output` breaks it in ways that are hard to attribute.
- **On a single-NIC node the cluster and encap sets are the node's public addresses**, because the planes collapse. The rules are still correct, but they are no longer a boundary — an upstream ACL is the only real control there.

Port 4432 must be closed outside the bootstrap window; `spx admin join` opens it transiently. Outbound egress can be limited to the image-catalogue hostnames in [§2](#2-outbound-connections) plus the operator's OS package repositories; on air-gapped nodes, block all outbound HTTPS and use `spx admin images import --file`.

## 5. Configuration Surface

Every listener and outbound destination is controlled by one of these files. Changes require a service restart.

| File | Keys | Controls |
|------|------|----------|
| `/etc/spinifex/spinifex.toml` | `nodes.<node>.{awsgw,nats,predastore,daemon}.host`, `nodes.<node>.vpcd.ovn_{nb,sb}_addr`, `nodes.<node>.daemon.dev_networking`, `network.ipsec_enabled` | Per-service bind addresses/ports; dev-mode QEMU port forwarding. `network.ipsec_enabled` (default `true`) decides whether OVN-native IPsec protects the Geneve tunnels, and therefore whether `charon` listens on 500/4500 — single-node clusters never enable it. |
| `/etc/spinifex/nats.conf` | `listen`, `cluster.listen`, `cluster.routes`, `http`, `tls`, `cluster.authorization` | NATS client/cluster/monitoring listeners, peer routes, TLS, cluster token. |
| `/etc/spinifex/predastore/predastore.toml` | `[[host]].bind_addr`, `[[host]].addr`, `[[host]].tls_cert`, `[[host]].tls_key`, `[[host.node]].role`, `[[host.node]].port` | Predastore host and node layout: `bind_addr` is the address the host's sockets bind, `addr` is the address peer hosts dial it on, and both carry no port — the nodes pinned to the host supply their own. The service's `--host`/`--port` override the bind address and the gate's S3 port. |
| `/etc/spinifex/northstar/northstar.toml` | `listen`, `forwarders` | DNS listen addresses (`:53` on the advertise address, `:5300` wildcard) and upstream resolvers. |
| OVN packages (`ovn-central`, `ovn-host`) | `ovn-nb-db`, `ovn-sb-db` (via `ovs-vsctl set open_vswitch …`); `setup-ovn.sh --lan-addr` for the NB/SB client bind; `--encap-ip` for the Geneve endpoint | OVN DB bind addresses and the encap plane. |
| Spinifex UI service | Built-in defaults: `host = "0.0.0.0"`, `port = 3000`. No `spinifex.toml` block today. | UI listener. |
| `spx admin init` / `spx admin join` | `--port`, `--token-ttl`, `--no-telemetry` (or `SPX_NO_TELEMETRY=1`) | Formation port, token TTL, telemetry opt-out. |
| Image catalogue (built-in) | Fixed URLs listed in [§2](#2-outbound-connections); not operator-configurable | Outbound HTTPS destinations for image downloads. |

## 6. Operator Checklist

- Inventory recorded in the system security plan — inbound ([§1](#1-inbound-listeners)), outbound ([§2](#2-outbound-connections)), cross-node ([§3](#3-cross-node-internal-connections)) — matches what is observed on the node (`ss -tlnp`, `ss -unlp`).
- Host firewall enforces the scope split in [§4](#4-limiting-controls): external surface limited to 9999, 3000, 22, 8443 and 53 (and 4432 only during bootstrap).
- Node planes verified: `ss -tulnp` shows cluster-scope listeners on the `lan` address and encap-scope listeners reachable only from peer chassis. On a single-NIC node, record that the planes are collapsed and that an upstream ACL is the only boundary.
- Cluster subnet is isolated from tenant guest VM networks and from the public internet.
- Formation port 4432 is closed on nodes not actively running a bootstrap token.
- Outbound HTTPS restricted to the [§2](#2-outbound-connections) image-catalogue hosts, or replaced with air-gapped import.
- Install telemetry (`install.mulgadc.com`) is either permitted and recorded in the security plan, or disabled via `SPX_NO_TELEMETRY=1` / `--no-telemetry`.
- OVN NB/SB client and RAFT ports (6641–6644) exposure limited to the cluster subnet pending the L2 TLS work.
- SSH (22) configured to operator-managed keys only; password auth disabled in `sshd_config`.
- Periodic review (at least annually, and after any topology change) confirms this inventory still matches the deployed configuration.

---

# Physical Security Operator Guide

URL: https://docs.mulgadc.com/docs/physical-security-guide
Category: Security and Compliance
Updated: 2026-08-21
Tags: security, compliance, cmmc, physical, facilities, access control

Operator guide to physical access controls, visitor handling, access logging, and access-device management at sites hosting Spinifex nodes and network gear.

## Overview

**Audience:** Operators deploying Spinifex into environments subject to CMMC Level 1, or any site that requires documented physical protection of compute and storage infrastructure.

**Scope:** The physical environment housing Spinifex nodes — the Linux hosts running `spinifex-daemon`, `spinifex-awsgw`, `spinifex-nats`, `spinifex-predastore`, `spinifex-viperblock`, `spinifex-vpcd`, `spinifex-ui`, and the OVN control plane — together with the network equipment, cabling, console/KVM access paths, and backup media that support them. Tenant workloads and operator endpoints (laptops, jump hosts) are out of scope.

## CMMC Practices Covered

PE.L1 objectives are discharged by the operator's facility, not by Spinifex. This guide names the assets, cadences, and evidence required.

| Practice | Title | Objective |
|----------|-------|-----------|
| PE.L1-3.10.1 | Limit Physical Access | [a] Authorized individuals allowed physical access are identified. [b] Physical access to organizational systems, equipment, and operating environments is limited to authorized individuals. |
| PE.L1-3.10.3 | Escort Visitors | [a] Visitors are escorted. [b] Visitor activity is monitored. |
| PE.L1-3.10.4 | Physical Access Logs | [a] Audit logs of physical access are maintained. |
| PE.L1-3.10.5 | Manage Physical Access Devices | [a] Physical access devices are identified. [b] Physical access devices are controlled. [c] Physical access devices are managed. |

## Approach

Spinifex does not mandate a specific access-control product. Operators typically deploy into a facility that already has badge readers, CCTV, and a visitor-management system (Kisi, HID, Lenel, Genetec, Envoy, Traction Guest, paper logs, etc.). Manual procedures (locked cabinet, paper sign-in, tracked key list) are acceptable for small sites provided the evidence trail exists; multi-rack sites should feed an electronic access-control system into the same SIEM used for Spinifex service logs.

## 1. Protected Assets

The physical protection boundary must enclose every item in this table. An asset is "protected" when access to it requires passing through a controlled barrier (locked room, cage, or cabinet).

| Asset | Why it must be inside the boundary |
|-------|-----------------------------------|
| Spinifex node chassis | Host console, BMC, and disks hold `/etc/spinifex/master.key`, cluster CA key, per-node TLS keys, and all tenant volume data. |
| Network switches and routers serving the cluster subnet | Physical access permits traffic capture, port mirroring, and control-plane tampering. |
| Structured cabling (top-of-rack to host, host to storage) | Passive taps are trivial on unprotected cabling. |
| Console / KVM / serial aggregators | Bypass host authentication; reach GRUB, single-user mode, BMC. |
| Backup media (tapes, removable disks, off-site copies) | Carry the same data as primary storage; covered by MP.L1-3.8.3 for disposal. |
| Facility power and HVAC cutoffs serving the rack | Unauthorised de-energise is a denial-of-service and a risk to in-flight writes. |

Nodes deployed in an unstaffed remote or edge location must be installed in a locked enclosure with tamper-evident seals and, where feasible, a sensor (door contact, accelerometer) feeding the central monitoring system. Record the enclosure location, seal serial, and sensor channel in the asset register.

## 2. Limit Physical Access (PE.L1-3.10.1)

### 2.1 Authorized Individuals ([a])

Maintain a written access list naming every individual authorized to enter each protected space. For each entry record:

- Full name and employing organization.
- Role justifying access (e.g. "Spinifex operator", "facilities", "vendor: OEM field service").
- Scope — which protected spaces, and whether escorted or unescorted.
- Start date, scheduled review date, and end date when access is removed.
- Approver (named individual, not a role mailbox).

Review the list at least quarterly and immediately on personnel change (role change, departure, contractor end-of-engagement). Revocations must be effective in the access-control system promptly per operator policy.

### 2.2 Enforcement ([b])

Access to protected spaces must require authentication at the barrier — a badge, PIN, key, biometric, or combination. Unaccompanied access by individuals not on the list in [§2.1](#21-authorized-individuals-a) must not be possible. Specifically:

- Doors and cages: electronic access control (badge/PIN) with door-forced and door-held alarms wired to the monitoring system. Mechanical-only locks are acceptable for lab/edge deployments provided key distribution is tracked under [§5](#5-manage-physical-access-devices-pel1-3105).
- Racks and cabinets holding Spinifex nodes: locked at all times when unattended. Key or combination distribution tracked as an access device under [§5](#5-manage-physical-access-devices-pel1-3105).
- Remote/edge enclosures: locked with tamper-evident seal; seal integrity checked on every site visit and recorded in the visit log.

Shared credentials (one badge used by several people, a rack key left in the cage) are not acceptable.

## 3. Escort and Monitor Visitors (PE.L1-3.10.3)

A visitor is any individual entering a protected space who is not on the authorized-access list in [§2.1](#21-authorized-individuals-a). This includes vendor field engineers, auditors, janitorial staff, and employees of the operator who do not hold Spinifex access.

### 3.1 Escort ([a])

- Every visitor must be signed in by a named authorized escort before entering the protected space.
- The escort must remain physically present with the visitor for the duration of the visit. Passing a visitor between escorts is permitted; leaving a visitor unaccompanied is not.
- Work inside a rack or at a node console requires one escort per visitor.

### 3.2 Monitoring ([b])

Visitor activity must be monitored by at least one of:

- Continuous physical presence of the escort in line-of-sight of the visitor. This is the minimum.
- CCTV coverage of the protected space with recordings retained for at least 30 days. Cameras must cover rack fronts and rears, door entries, and any console/KVM positions.
- For vendor maintenance involving system access (BMC, console, disk swap): a second operator witness on-site, or a recorded screen share if the work is performed from the console. Record the witness or session recording ID in the visitor log.

CCTV is strongly recommended for any facility hosting more than a single rack. It also serves as corroborating evidence for [§4](#4-physical-access-logs-pel1-3104) logs and is useful for incident response.

## 4. Physical Access Logs (PE.L1-3.10.4)

### 4.1 What to Log

Every entry into a protected space must produce a log entry capturing:

| Field | Source |
|-------|--------|
| Identity (name, badge ID, or visitor record ID) | Access control system or sign-in sheet. |
| Timestamp in and timestamp out | Access control system; manual for paper logs. |
| Protected space entered | Door/cage ID. |
| Purpose | Free-text; required for visitors, recommended for authorized staff. |
| Escort (for visitors) | Named individual from the authorized list. |
| Associated change or ticket ID | When the visit is tied to a Spinifex change (upgrade, disk swap, node rebuild). |

### 4.2 Retention and Review

- **Retention:** at least 12 months. CCTV recordings associated with the same visit, where used to discharge [§3.2](#32-monitoring-b), should be retained for the same period.
- **Review cadence:** monthly spot-check of 10% of entries, quarterly full review against the authorized-access list. Discrepancies (unknown badge ID, visitor with no matching sign-in, escort named who was off-site) must be investigated and the finding recorded.
- **Alerting:** out-of-hours entries, repeated failed reads at a single door, and door-forced / door-held events must fire an alert to the on-call operator. Treat these as incidents until triaged.

### 4.3 Integration with Spinifex Logs

Forward physical-access events to the same SIEM or log collector used for Spinifex service logs (see [Malware Protection §3](https://docs.mulgadc.com/docs/malware-protection#3-scan-schedule-sil1-3145)) so a `master.key` access attempt can be correlated with a physical entry into the rack.

## 5. Manage Physical Access Devices (PE.L1-3.10.5)

Physical access devices are badges, key cards, PINs, mechanical keys, rack keys, safe combinations, and tamper seals. USB tokens or HSMs holding the cluster CA key or master encryption key also belong in the register.

### 5.1 Identification ([a])

Maintain a device register with one entry per issued device. Minimum fields:

| Field | Notes |
|-------|-------|
| Device ID | Badge serial, key stamp, seal serial, HSM serial. |
| Device type | Badge / mechanical key / PIN / combination / seal / hardware token. |
| Scope | Which barriers it opens, or which asset it protects. |
| Holder | Named individual. Shared holders are not acceptable for badges or PINs. |
| Issued date, issued by | Audit trail for [c]. |
| Returned or revoked date, reason | Populated on personnel change. |

### 5.2 Control ([b])

- Issue only to individuals named in the [§2.1](#21-authorized-individuals-a) access list. Scope of the device must not exceed that individual's authorized scope.
- Badges and PINs must be unique per holder. Mechanical keys that must be shared (e.g. a single rack key) are permitted only when distribution is tracked by check-out/check-in against the register.
- Lost or compromised devices trigger immediate revocation: badges deactivated in the access-control system, PINs changed, affected mechanical locks re-keyed within 30 days, tamper seals re-applied on the next site visit. Record the event and remediation in the register.
- Terminated or reassigned personnel surrender all devices on last day. Mark returned in the register.

### 5.3 Management ([c])

- **Inventory review:** at least quarterly, reconcile the device register against (a) the access-control system's badge database, (b) the physical key count, and (c) the authorized-access list in [§2.1](#21-authorized-individuals-a). Discrepancies must be resolved before the review is marked complete.
- **Rotation / re-keying:** mechanical locks re-keyed on loss of any key, and at least every five years. Default combinations (safe, cabinet, BMC default passwords at install) must be changed before the device enters service.
- **Spares and master keys:** held in a secured location (locked cabinet or safe inside a protected space), accessible only to named individuals, with any access itself logged under [§4](#4-physical-access-logs-pel1-3104).

## 6. Evidence and Record Keeping

For CMMC assessment, retain the following for at least 12 months (longer where facility or contract policy requires):

- **Authorized access list** ([§2.1](#21-authorized-individuals-a)) with review dates and approver names.
- **Access-control system configuration** showing which badges are permitted at which readers, exported or screenshotted at each quarterly review.
- **Physical access logs** ([§4](#4-physical-access-logs-pel1-3104)) and, where applicable, CCTV retention policy and sample recordings.
- **Visitor logs** with escort names for every visit.
- **Device register** ([§5.1](#51-identification-a)) with issue, return, and revocation history.
- **Incident records** for door-forced/door-held events, lost devices, failed seal checks, and any access-review discrepancies, with remediation outcomes.
- **Annual review attestation** from the facility or security owner confirming the controls above operated for the prior 12 months.

## 7. Operator Checklist

Use this list to confirm a site meets the four CMMC practices before admitting Spinifex nodes to a production cluster:

- Authorized-access list exists, names every protected space, is reviewed at least quarterly, and has an approver for every entry.
- Every protected space enforces authentication at the barrier; shared credentials are not in use for badged/PIN'd spaces.
- Remote/edge enclosures are locked, sealed, and listed in the asset register with seal serials recorded.
- Visitor procedure is documented: sign-in, escort (one-per-visitor at equipment), sign-out.
- CCTV or equivalent monitoring covers the rack fronts, rears, and console positions, with 30-day minimum retention.
- Physical access logs are produced for every entry, retained 12 months, and reviewed on the cadence in [§4.2](#42-retention-and-review).
- Door-forced, door-held, and out-of-hours events alert the on-call operator.
- Physical access events forwarded to the SIEM used for Spinifex service logs (see [Malware Protection §3](https://docs.mulgadc.com/docs/malware-protection#3-scan-schedule-sil1-3145)).
- Device register reconciles quarterly against the access-control system and physical key count; discrepancies are closed before sign-off.
- Lost-device and termination procedures trigger immediate revocation per [§5.2](#52-control-b).
- System security plan references this guide and records the facility, access-control product, CCTV retention, and log-forwarding destination.

---

# AWS API Coverage

URL: https://docs.mulgadc.com/coverage
Category: Coverage
Updated: 2026-09-15
Tags: aws, compatibility, coverage, api, operations

The AWS API operations Spinifex serves, generated from its gateway dispatch tables and the pinned AWS SDK service models on each build, service by service.

## Overview

The platform serves **381 operations** across the AWS APIs below. Every page names the operations its service implements from the pinned model, generated from the dispatch tables on each build rather than written by hand.

| Service | Operations |
|---|---:|
| [ACM](https://docs.mulgadc.com/coverage/acm) | 9 |
| [EC2](https://docs.mulgadc.com/coverage/ec2) | 125 |
| [ECR](https://docs.mulgadc.com/coverage/ecr) | 22 |
| [ECS](https://docs.mulgadc.com/coverage/ecs) | 31 |
| [EKS](https://docs.mulgadc.com/coverage/eks) | 34 |
| [ELBv2](https://docs.mulgadc.com/coverage/elbv2) | 34 |
| [IAM](https://docs.mulgadc.com/coverage/iam) | 76 |
| [RDS](https://docs.mulgadc.com/coverage/rds) | 26 |
| [S3](https://docs.mulgadc.com/coverage/s3) | 19 |
| [STS](https://docs.mulgadc.com/coverage/sts) | 5 |
| **Total** | **381** |

---

# EC2 API Coverage

URL: https://docs.mulgadc.com/coverage/ec2
Category: Coverage
Updated: 2026-09-15
Tags: aws, compatibility, coverage, ec2, compute, vpc

The Amazon EC2 API operations Spinifex implements, covering instances, EBS volumes, VPC networking, tags, security groups and the rest of the compute surface.

## Overview

Spinifex implements **125 operations** in the EC2 `2016-11-15` API model.

### Spot Instances Are a Mock

Spot Instance Requests are a mock over the on-demand `RunInstances` path. A request synchronously launches real VMs on the operator's own compute and is then reported `active` and `fulfilled`. There is no spot market: no bidding, no price rejection, no interruption and no reclamation, and instances are never taken back.

### Operations

| Operation |
|---|
| `AllocateAddress` |
| `AssociateAddress` |
| `AssociateIamInstanceProfile` |
| `AssociateRouteTable` |
| `AttachInternetGateway` |
| `AttachNetworkInterface` |
| `AttachVolume` |
| `AuthorizeSecurityGroupEgress` |
| `AuthorizeSecurityGroupIngress` |
| `CancelCapacityReservation` |
| `CancelSpotInstanceRequests` |
| `CopyImage` |
| `CopySnapshot` |
| `CreateCapacityReservation` |
| `CreateEgressOnlyInternetGateway` |
| `CreateImage` |
| `CreateInternetGateway` |
| `CreateKeyPair` |
| `CreateLaunchTemplate` |
| `CreateLaunchTemplateVersion` |
| `CreateNatGateway` |
| `CreateNetworkInterface` |
| `CreatePlacementGroup` |
| `CreateRoute` |
| `CreateRouteTable` |
| `CreateSecurityGroup` |
| `CreateSnapshot` |
| `CreateSubnet` |
| `CreateTags` |
| `CreateVolume` |
| `CreateVpc` |
| `DeleteEgressOnlyInternetGateway` |
| `DeleteInternetGateway` |
| `DeleteKeyPair` |
| `DeleteLaunchTemplate` |
| `DeleteLaunchTemplateVersions` |
| `DeleteNatGateway` |
| `DeleteNetworkInterface` |
| `DeletePlacementGroup` |
| `DeleteRoute` |
| `DeleteRouteTable` |
| `DeleteSecurityGroup` |
| `DeleteSnapshot` |
| `DeleteSubnet` |
| `DeleteTags` |
| `DeleteVolume` |
| `DeleteVpc` |
| `DeregisterImage` |
| `DescribeAccountAttributes` |
| `DescribeAddresses` |
| `DescribeAddressesAttribute` |
| `DescribeAvailabilityZones` |
| `DescribeCapacityReservations` |
| `DescribeEgressOnlyInternetGateways` |
| `DescribeIamInstanceProfileAssociations` |
| `DescribeImageAttribute` |
| `DescribeImages` |
| `DescribeInstanceAttribute` |
| `DescribeInstanceCreditSpecifications` |
| `DescribeInstanceStatus` |
| `DescribeInstanceTypeOfferings` |
| `DescribeInstanceTypes` |
| `DescribeInstances` |
| `DescribeInternetGateways` |
| `DescribeKeyPairs` |
| `DescribeLaunchTemplateVersions` |
| `DescribeLaunchTemplates` |
| `DescribeNatGateways` |
| `DescribeNetworkInterfaces` |
| `DescribePlacementGroups` |
| `DescribeRegions` |
| `DescribeRouteTables` |
| `DescribeSecurityGroupRules` |
| `DescribeSecurityGroups` |
| `DescribeSnapshots` |
| `DescribeSpotInstanceRequests` |
| `DescribeSubnets` |
| `DescribeTags` |
| `DescribeVolumeStatus` |
| `DescribeVolumes` |
| `DescribeVolumesModifications` |
| `DescribeVpcAttribute` |
| `DescribeVpcs` |
| `DetachInternetGateway` |
| `DetachNetworkInterface` |
| `DetachVolume` |
| `DisableEbsEncryptionByDefault` |
| `DisableSerialConsoleAccess` |
| `DisassociateAddress` |
| `DisassociateIamInstanceProfile` |
| `DisassociateRouteTable` |
| `EnableEbsEncryptionByDefault` |
| `EnableSerialConsoleAccess` |
| `GetConsoleOutput` |
| `GetEbsEncryptionByDefault` |
| `GetPasswordData` |
| `GetSecurityGroupsForVpc` |
| `GetSerialConsoleAccessStatus` |
| `ImportKeyPair` |
| `ModifyImageAttribute` |
| `ModifyInstanceAttribute` |
| `ModifyInstanceMetadataOptions` |
| `ModifyLaunchTemplate` |
| `ModifyNetworkInterfaceAttribute` |
| `ModifySubnetAttribute` |
| `ModifyVolume` |
| `ModifyVpcAttribute` |
| `MonitorInstances` |
| `RebootInstances` |
| `RegisterImage` |
| `ReleaseAddress` |
| `ReplaceIamInstanceProfileAssociation` |
| `ReplaceRoute` |
| `ReplaceRouteTableAssociation` |
| `RequestSpotInstances` |
| `ResetImageAttribute` |
| `RevokeSecurityGroupEgress` |
| `RevokeSecurityGroupIngress` |
| `RunInstances` |
| `StartInstances` |
| `StopInstances` |
| `TerminateInstances` |
| `UnmonitorInstances` |
| `UpdateSecurityGroupRuleDescriptionsEgress` |
| `UpdateSecurityGroupRuleDescriptionsIngress` |

---

# IAM API Coverage

URL: https://docs.mulgadc.com/coverage/iam
Category: Coverage
Updated: 2026-09-15
Tags: aws, compatibility, coverage, iam, identity, policies

The AWS IAM API operations Spinifex implements, covering users, roles, policies, groups, instance profiles and the access keys that authenticate them.

## Overview

Spinifex implements **76 operations** in the IAM `2010-05-08` API model.

### Scope

All IAM operations are account-scoped. The root user of an account bypasses policy evaluation entirely, as it does on AWS.

### Operations

| Operation |
|---|
| `AddRoleToInstanceProfile` |
| `AddUserToGroup` |
| `AttachGroupPolicy` |
| `AttachRolePolicy` |
| `AttachUserPolicy` |
| `CreateAccessKey` |
| `CreateGroup` |
| `CreateInstanceProfile` |
| `CreateOpenIDConnectProvider` |
| `CreatePolicy` |
| `CreateRole` |
| `CreateUser` |
| `DeleteAccessKey` |
| `DeleteGroup` |
| `DeleteGroupPolicy` |
| `DeleteInstanceProfile` |
| `DeleteOpenIDConnectProvider` |
| `DeletePolicy` |
| `DeleteRole` |
| `DeleteRolePolicy` |
| `DeleteUser` |
| `DeleteUserPolicy` |
| `DetachGroupPolicy` |
| `DetachRolePolicy` |
| `DetachUserPolicy` |
| `GetAccountSummary` |
| `GetGroup` |
| `GetGroupPolicy` |
| `GetInstanceProfile` |
| `GetOpenIDConnectProvider` |
| `GetPolicy` |
| `GetPolicyVersion` |
| `GetRole` |
| `GetRolePolicy` |
| `GetUser` |
| `GetUserPolicy` |
| `ListAccessKeys` |
| `ListAttachedGroupPolicies` |
| `ListAttachedRolePolicies` |
| `ListAttachedUserPolicies` |
| `ListEntitiesForPolicy` |
| `ListGroupPolicies` |
| `ListGroups` |
| `ListGroupsForUser` |
| `ListInstanceProfileTags` |
| `ListInstanceProfiles` |
| `ListInstanceProfilesForRole` |
| `ListOpenIDConnectProviderTags` |
| `ListOpenIDConnectProviders` |
| `ListPolicies` |
| `ListPolicyTags` |
| `ListPolicyVersions` |
| `ListRolePolicies` |
| `ListRoleTags` |
| `ListRoles` |
| `ListUserPolicies` |
| `ListUserTags` |
| `ListUsers` |
| `PutGroupPolicy` |
| `PutRolePolicy` |
| `PutUserPolicy` |
| `RemoveRoleFromInstanceProfile` |
| `RemoveUserFromGroup` |
| `TagInstanceProfile` |
| `TagOpenIDConnectProvider` |
| `TagPolicy` |
| `TagRole` |
| `TagUser` |
| `UntagInstanceProfile` |
| `UntagOpenIDConnectProvider` |
| `UntagPolicy` |
| `UntagRole` |
| `UntagUser` |
| `UpdateAccessKey` |
| `UpdateAssumeRolePolicy` |
| `UpdateRole` |

---

# S3 API Coverage

URL: https://docs.mulgadc.com/coverage/s3
Category: Coverage
Updated: 2026-09-15
Tags: aws, compatibility, coverage, s3, storage, objects

The Amazon S3 API operations Predastore serves on the platform's S3 endpoint, covering buckets, objects, multipart uploads and the policies that guard them.

## Overview

Predastore implements **19 operations** in the S3 `2006-03-01` API model.

### Predastore serves this endpoint

S3 is the one surface the AWS gateway does not answer itself. Object storage runs on [Predastore](https://github.com/mulgadc/predastore), which serves the S3 REST API directly over its own endpoint.

### Routed is not the same as conforming

This page says an operation is routed to a handler. It does not say the handler's behaviour matches S3 in every case.

That behaviour is measured separately, against the `ceph/s3-tests` suite Ceph RGW, MinIO and Garage are all validated with, and the results are published in Predastore's [S3 compatibility report](https://github.com/mulgadc/predastore/blob/dev/docs/S3-COMPATIBILITY.md).

### Operations

| Operation |
|---|
| `AbortMultipartUpload` |
| `CompleteMultipartUpload` |
| `CopyObject` |
| `CreateBucket` |
| `CreateMultipartUpload` |
| `DeleteBucket` |
| `DeleteObject` |
| `DeleteObjects` |
| `GetObject` |
| `HeadBucket` |
| `HeadObject` |
| `ListBuckets` |
| `ListMultipartUploads` |
| `ListObjects` |
| `ListObjectsV2` |
| `ListParts` |
| `PutObject` |
| `UploadPart` |
| `UploadPartCopy` |

---

# EKS API Coverage

URL: https://docs.mulgadc.com/coverage/eks
Category: Coverage
Updated: 2026-09-15
Tags: aws, compatibility, coverage, eks, kubernetes, containers

The Amazon EKS API operations Spinifex implements, covering clusters, nodegroups, add-ons, access entries and the identity providers they authenticate.

## Overview

Spinifex implements **34 operations** in the EKS `2017-11-01` API model.

### Operations

| Operation |
|---|
| `AssociateAccessPolicy` |
| `AssociateIdentityProviderConfig` |
| `CreateAccessEntry` |
| `CreateAddon` |
| `CreateCluster` |
| `CreateNodegroup` |
| `DeleteAccessEntry` |
| `DeleteAddon` |
| `DeleteCluster` |
| `DeleteNodegroup` |
| `DescribeAccessEntry` |
| `DescribeAddon` |
| `DescribeAddonVersions` |
| `DescribeCluster` |
| `DescribeIdentityProviderConfig` |
| `DescribeNodegroup` |
| `DisassociateAccessPolicy` |
| `DisassociateIdentityProviderConfig` |
| `ListAccessEntries` |
| `ListAccessPolicies` |
| `ListAddons` |
| `ListAssociatedAccessPolicies` |
| `ListClusters` |
| `ListIdentityProviderConfigs` |
| `ListNodegroups` |
| `ListTagsForResource` |
| `TagResource` |
| `UntagResource` |
| `UpdateAccessEntry` |
| `UpdateAddon` |
| `UpdateClusterConfig` |
| `UpdateClusterVersion` |
| `UpdateNodegroupConfig` |
| `UpdateNodegroupVersion` |

---

# ECS API Coverage

URL: https://docs.mulgadc.com/coverage/ecs
Category: Coverage
Updated: 2026-09-15
Tags: aws, compatibility, coverage, ecs, containers, orchestration

The Amazon ECS API operations Spinifex implements, covering clusters, services, tasks, container instances and the task definitions they are launched from.

## Overview

Spinifex implements **31 operations** in the ECS `2014-11-13` API model.

### No Fargate

Clusters, services and tasks run on EC2 container instances. There is no Fargate launch type, and no operation below provides one.

### Operations

| Operation |
|---|
| `CreateCapacityProvider` |
| `CreateCluster` |
| `CreateService` |
| `DeleteCapacityProvider` |
| `DeleteCluster` |
| `DeleteService` |
| `DeregisterContainerInstance` |
| `DeregisterTaskDefinition` |
| `DescribeCapacityProviders` |
| `DescribeClusters` |
| `DescribeContainerInstances` |
| `DescribeServices` |
| `DescribeTaskDefinition` |
| `DescribeTasks` |
| `ListClusters` |
| `ListContainerInstances` |
| `ListServices` |
| `ListTagsForResource` |
| `ListTaskDefinitions` |
| `ListTasks` |
| `PutClusterCapacityProviders` |
| `RegisterContainerInstance` |
| `RegisterTaskDefinition` |
| `RunTask` |
| `StartTask` |
| `StopTask` |
| `SubmitTaskStateChange` |
| `TagResource` |
| `UntagResource` |
| `UpdateContainerInstancesState` |
| `UpdateService` |

---

# ECR API Coverage

URL: https://docs.mulgadc.com/coverage/ecr
Category: Coverage
Updated: 2026-09-15
Tags: aws, compatibility, coverage, ecr, containers, registry

The Amazon ECR API operations Spinifex implements, alongside the OCI distribution endpoint that carries the image layers for every repository it serves.

## Overview

Spinifex implements **22 operations** in the ECR `2015-09-21` API model.

### Two endpoints, one registry

Repository metadata is served over the AWS API on the gateway endpoint. Image data moves over the OCI Distribution `/v2/` endpoint on that same host, authenticated by the bearer token `GetAuthorizationToken` mints for `docker login`.

The split explains the stubs below. The layer-transfer operations — `BatchCheckLayerAvailability`, `InitiateLayerUpload`, `UploadLayerPart`, `CompleteLayerUpload` and `GetDownloadUrlForLayer` — are registered stubs because the `/v2/` endpoint carries that traffic instead. A client using `docker` or any OCI-compatible tool never calls them.

Registry replication is a stub for the same kind of reason: a deployment is a single registry, with no cross-region peer to replicate to.

### Operations

| Operation |
|---|
| `BatchDeleteImage` |
| `BatchGetImage` |
| `CreateRepository` |
| `DeleteLifecyclePolicy` |
| `DeleteRepository` |
| `DeleteRepositoryPolicy` |
| `DescribeImages` |
| `DescribeRepositories` |
| `GetAuthorizationToken` |
| `GetLifecyclePolicy` |
| `GetLifecyclePolicyPreview` |
| `GetRepositoryPolicy` |
| `ListImages` |
| `ListTagsForResource` |
| `PutImage` |
| `PutImageScanningConfiguration` |
| `PutImageTagMutability` |
| `PutLifecyclePolicy` |
| `SetRepositoryPolicy` |
| `StartLifecyclePolicyPreview` |
| `TagResource` |
| `UntagResource` |

---

# ELBv2 API Coverage

URL: https://docs.mulgadc.com/coverage/elbv2
Category: Coverage
Updated: 2026-09-15
Tags: aws, compatibility, coverage, elbv2, load balancing, networking

The Elastic Load Balancing v2 API operations Spinifex implements, for both the Application and Network Load Balancers it serves, listeners and target groups.

## Overview

Spinifex implements **34 operations** in the ELBv2 `2015-12-01` API model.

### Two data planes

The data plane is a system-managed load balancer VM, launched automatically during `CreateLoadBalancer`. Application Load Balancers run HAProxy for the L7 surface — rules, fixed responses and redirects over HTTP and HTTPS. Network Load Balancers run nginx `stream` for the L4 surface — TCP, UDP, TLS and TCP_UDP — because HAProxy cannot load-balance UDP. The agent selects the engine from the configuration the control plane delivers, so the choice follows the load balancer type and is not separately configurable.

### Operations

| Operation |
|---|
| `AddListenerCertificates` |
| `AddTags` |
| `CreateListener` |
| `CreateLoadBalancer` |
| `CreateRule` |
| `CreateTargetGroup` |
| `DeleteListener` |
| `DeleteLoadBalancer` |
| `DeleteRule` |
| `DeleteTargetGroup` |
| `DeregisterTargets` |
| `DescribeAccountLimits` |
| `DescribeListenerCertificates` |
| `DescribeListeners` |
| `DescribeLoadBalancerAttributes` |
| `DescribeLoadBalancers` |
| `DescribeRules` |
| `DescribeSSLPolicies` |
| `DescribeTags` |
| `DescribeTargetGroupAttributes` |
| `DescribeTargetGroups` |
| `DescribeTargetHealth` |
| `ModifyListener` |
| `ModifyLoadBalancerAttributes` |
| `ModifyRule` |
| `ModifyTargetGroup` |
| `ModifyTargetGroupAttributes` |
| `RegisterTargets` |
| `RemoveListenerCertificates` |
| `RemoveTags` |
| `SetIpAddressType` |
| `SetRulePriorities` |
| `SetSecurityGroups` |
| `SetSubnets` |

---

# RDS API Coverage

URL: https://docs.mulgadc.com/coverage/rds
Category: Coverage
Updated: 2026-09-15
Tags: aws, compatibility, coverage, rds, databases, postgresql, mariadb

The Amazon RDS API operations Spinifex implements, for the managed PostgreSQL and MariaDB engines it offers, covering instances, snapshots and parameters.

## Overview

Spinifex implements **26 operations** in the RDS `2014-10-31` API model.

### Engines

Spinifex offers PostgreSQL and MariaDB. Each DB instance is one dedicated system-owned VM running the engine directly, launched from a platform AMI and hidden from the customer's EC2 API. `Engine` is fixed at create: there is no in-place engine change, no cross-engine snapshot restore and no migration between the two.

`mysql` is not an accepted engine and is not an alias for `mariadb`. MariaDB is offered under its own AWS engine name, exactly as AWS RDS offers it, so a client — including Terraform's `aws_db_instance` — must set `engine = "mariadb"`.

Engine versions are pinned per engine. An `EngineVersion` naming anything but the pin is rejected, including a narrower minor version, because the AMI makes no promise about which minor it carries.

### The endpoint is private

The engine is reached over a customer-account ENI injected into a subnet of the DB subnet group, so the endpoint is reachable from inside the VPC only. The DB VM has other NICs that no customer security group governs, and the engine binds none of them — the port is not open there at all, rather than open and gated.

### TLS is required by default

Both engines enforce encrypted connections by default: `rds.force_ssl` on PostgreSQL and `require_secure_transport` on MariaDB. For MariaDB this is a deliberate divergence from AWS, which leaves it off. Both are boolean, modifiable and dynamic, so setting either to `0` in a parameter group restores plaintext without a reboot.

### Rejected parameters

A parameter whose omission would create a false safety, security or availability guarantee is rejected with `InvalidParameterValue` rather than silently dropped.

| Parameter | Why it is rejected |
|-----------|--------------------|
| `MultiAZ=true` | Single-AZ platform; a standby would not exist |
| `PubliclyAccessible=true` | The endpoint is a private VPC address |
| `StorageEncrypted=false` | Unencrypted storage is not offered |
| `EnableIAMDatabaseAuthentication` | IAM database authentication is not implemented |
| `Iops`, `StorageThroughput`, `StorageType` ≠ `gp3` | Provisioned performance classes are not implemented |
| `KmsKeyId`, `TdeCredentialArn` | Storage is encrypted with the cluster key, not a customer-managed one |
| `AvailabilityZone` | The platform exposes a single zone |
| `AvailabilityZoneGroup` (orderable options) | It selects a zone or local-zone group, and naming a zone is already refused |
| `DBSecurityGroups` | EC2-Classic security groups — use `VpcSecurityGroupIds` |
| `DBClusterIdentifier`, `DBClusterSnapshotIdentifier` | Clustered engines are not offered |
| `EnableCloudwatchLogsExports` | Log export is not implemented |
| `EngineVersion` other than the engine's pin, `Engine` on modify | No in-place engine or version change |
| `Engine=mysql` (and Aurora engines) | Oracle MySQL is not offered; `mariadb` is a distinct engine, not an alias for it |
| `NewDBInstanceIdentifier` | The identifier is the DNS label and the KV key |
| `DBPortNumber`, `DBSubnetGroupName` on modify | Both would move the endpoint |
| `MaxAllocatedStorage` | Storage autoscaling is not implemented |
| `ManageMasterUserPassword`, `RotateMasterUserPassword` | Secrets Manager integration is not offered |
| `CACertificateIdentifier` | The serving certificate is minted from the cluster CA |
| `Domain`, `DomainFqdn` | Active Directory domain join is not offered |
| `OptionGroupName` | Option groups are not offered |
| `CustomIamInstanceProfile` | The DB VM's instance profile is platform-owned |
| `EnableCustomerOwnedIp` | An Outposts feature |
| `ForceFailover` (reboot) | No standby to fail over to |
| `DBSnapshotIdentifier` (stop) | Snapshot-on-stop is not implemented |

### Operations

| Operation |
|---|
| `AddTagsToResource` |
| `CreateDBInstance` |
| `CreateDBParameterGroup` |
| `CreateDBSnapshot` |
| `CreateDBSubnetGroup` |
| `DeleteDBInstance` |
| `DeleteDBParameterGroup` |
| `DeleteDBSnapshot` |
| `DeleteDBSubnetGroup` |
| `DescribeDBEngineVersions` |
| `DescribeDBInstanceAutomatedBackups` |
| `DescribeDBInstances` |
| `DescribeDBParameterGroups` |
| `DescribeDBParameters` |
| `DescribeDBSnapshots` |
| `DescribeDBSubnetGroups` |
| `DescribeEvents` |
| `DescribeOrderableDBInstanceOptions` |
| `ListTagsForResource` |
| `ModifyDBInstance` |
| `ModifyDBParameterGroup` |
| `RebootDBInstance` |
| `RemoveTagsFromResource` |
| `RestoreDBInstanceFromDBSnapshot` |
| `StartDBInstance` |
| `StopDBInstance` |

---

# ACM API Coverage

URL: https://docs.mulgadc.com/coverage/acm
Category: Coverage
Updated: 2026-09-15
Tags: aws, compatibility, coverage, acm, certificates, tls

The AWS Certificate Manager API operations Spinifex implements, generated from the gateway dispatch tables and the pinned AWS service model on each build.

## Overview

Spinifex implements **9 operations** in the ACM `2015-12-08` API model.

### Import and issuance

Spinifex both stores externally-issued certificates and issues its own for load balancer listener references. Certificates are account-scoped, and a delete is refused while any listener still references the ARN — there is no force flag, matching AWS.

`RequestCertificate` mints an ARN immediately and returns `PENDING_VALIDATION`; it never issues inline except against a tenant private CA, which has no domain to validate. The validation mode is derived from deployment state rather than configured: the DNS provider API where a credential exists, a manual TXT record where the platform hosts the zone, and a private CA otherwise — the only option for a deployment with no publicly delegated domain.

Terraform's canonical certificate, DNS record and validation flow works unmodified in every mode. Where Spinifex owns the record write it emits no `ResourceRecord`, so iterating the validation options yields zero records and the validation resource still blocks correctly by polling until the certificate is issued.

### Operations

| Operation |
|---|
| `AddTagsToCertificate` |
| `DeleteCertificate` |
| `DescribeCertificate` |
| `GetCertificate` |
| `ImportCertificate` |
| `ListCertificates` |
| `ListTagsForCertificate` |
| `RemoveTagsFromCertificate` |
| `RequestCertificate` |

---

# STS API Coverage

URL: https://docs.mulgadc.com/coverage/sts
Category: Coverage
Updated: 2026-09-15
Tags: aws, compatibility, coverage, sts, identity, credentials

The AWS STS API operations Spinifex implements, covering role assumption, session tokens and web identity federation (IRSA), with those it does not offer.

## Overview

Spinifex implements **5 operations** in the STS `2011-06-15` API model.

### Trust policies

Trust policies are validated at write time rather than silently narrowed at assume time. `NotPrincipal`, `NotAction`, empty-string `Action` elements and empty `Principal` blocks are all rejected as malformed.

`Condition` blocks are rejected except on `AssumeRoleWithWebIdentity` with `StringEquals`, which is the shape IRSA needs and which Spinifex evaluates at assume time against the token's issuer, subject and audience. Anything wider is refused rather than accepted and ignored, because an accepted-but-unevaluated condition is a silent over-grant.

### Parameters that are refused, not ignored

Several inputs the model describes are deliberately rejected rather than accepted as no-ops: inline session policies and policy ARNs, session tags, and MFA serial numbers and token codes. Each of them would otherwise appear to restrict or strengthen a session that in fact carries the role's full permissions.

### Operations

| Operation |
|---|
| `AssumeRole` |
| `AssumeRoleWithWebIdentity` |
| `GetAccessKeyInfo` |
| `GetCallerIdentity` |
| `GetSessionToken` |

---

# Multi-Tenant AI on Supermicro H14 with AMD MI350X

URL: https://docs.mulgadc.com/hardware/supermicro/smci-h14
Category: Hardware / Supermicro
Updated: 2026-09-14
Tags: supermicro, amd, mi350x, gpu-passthrough, multi-tenant, vllm

Provision isolated GPU VMs for simultaneous AI workloads on a Supermicro H14 bare-metal node with AMD MI350X GPUs, using Spinifex's EC2-compatible API.

## Overview

Spinifex is an open-source infrastructure platform that brings core AWS services including EC2, EBS and S3 to bare-metal, edge, and on-prem environments. It exposes an EC2-compatible API, so any tooling that works against AWS (the `aws` CLI, Terraform, SDKs) works against a Spinifex node unchanged, with a single profile swap.

This document serves as a tutorial for setting up multi-tenant AI workloads with Spinifex. The reference architecture for this tutorial is the Supermicro H14 platform - specifications listed below.

The H14 chassis comes equipped with **8× AMD Instinct MI350X** (288 GB HBM3e each, 2.3 TB total). We used Spinifex to provision three isolated VMs, each assigned 2× MI350X via direct PCIe passthrough, and ran three computationally intensive workloads simultaneously: a YOLO11x vision model, a Qwen3-VL 235B FP8 vision language model, and a Llama 3.3 70B large language model.

<p><video src="https://iso.mulgadc.com/clipped-demo.mp4" controls width="100%" style="border-radius:6px"></video></p>

### Platform

| Component | Specification |
|---|---|
| **Bare-metal host** | Supermicro H14 |
| **Host OS** | Ubuntu 24.04 LTS or Debian 13 (minimum) |
| **Orchestration** | Spinifex — EC2-compatible bare-metal API |
| **Guest OS** | Ubuntu 26.04 LTS |
| **GPUs** | 8× AMD Instinct MI350X (288 GB HBM3e each, 2.3 TB total) |
| **GPU passthrough** | PCIe passthrough via vfio-pci |
| **Instance type** | `g7e.12xlarge` — 2× MI350X per VM |
| **Container runtime** | Docker (with ROCm device access) |
| **Inference runtime** | vLLM (`rocm/vllm` image) |
| **VM network** | Internal VPC, 192.168.10.0/24 |
| **Block storage** | Viperblock — EBS-compatible, local NVMe-backed |

Spinifex runs on the bare-metal host and exposes an EC2-compatible API to any standard `aws ec2` tooling. Guest VMs receive their GPU allocation at the hardware level — each VM's OS sees the MI350Xs as native PCIe devices with no virtualisation layer in the data path. Workloads across VMs share no GPU memory, no address space, and no network segment except through the VPC.

GPU passthrough requires a host kernel and OS that supports vfio-pci. Ubuntu 24.04 LTS (kernel 6.8+) and Debian 13 (kernel 6.12+) are the tested minimum baselines for the host. Guest VMs run Ubuntu 26.04 LTS.

## Prerequisites

- Supermicro H14 with 8× AMD Instinct MI350X installed
- Host OS: **Ubuntu 26.04 LTS** (kernel 6.8+) or **Debian 13** (kernel 6.12+) — minimum for vfio-pci support
- Spinifex installed and all services running (`systemctl status spinifex.target`)
- AMD GPU AMI registered (`ubuntu-26.04-amd-gpu-x86_64`) — Ubuntu 26.04 LTS with ROCm-compatible kernel
- AWS CLI configured with `AWS_PROFILE=spinifex` pointing at the Spinifex endpoint
- SSH key pair imported into Spinifex, VPC and security group created (see [Launching Instances](https://docs.mulgadc.com/docs/launching-instances))

GPU passthrough must be configured before launching GPU instances. This is a one-time step per host.

```bash
# Detect GPUs and bind to vfio-pci (requires reboot)
sudo spx admin gpu setup

# After rebooting — confirm passthrough is active and signal the daemon
sudo spx admin gpu enable
```

`spx admin gpu setup` blacklists the AMD driver on the host and binds each GPU to `vfio-pci`. After reboot, `spx admin gpu enable` verifies the binding and makes the GPU pool available to `RunInstances`.

<img width="1230" height="644" src="/docs-assets/hardware-supermicro-smci-h14-gpu-enabled.png" alt="Spinifex console confirming PCIe passthrough is active for each GPU">

To check available GPU instance types:

```bash
export AWS_PROFILE=spinifex
aws ec2 describe-instance-types \
    --query 'InstanceTypes[?GpuInfo].[InstanceType,GpuInfo.Gpus[0].Count,GpuInfo.Gpus[0].Name]' \
    --output table
```

## Instructions

### 1. Provision the VMs

Each VM was provisioned using standard AWS EC2 CLI commands — the only change from a normal AWS workflow is `AWS_PROFILE=spinifex`, which redirects the CLI to Spinifex's local EC2-compatible endpoint instead of AWS:

| VM | Instance type | GPUs | Role | Model |
|---|---|---|---|---|
| `vm-yolo` | g7e.12xlarge | 2× MI350X | Real-time object detection | YOLO11x |
| `vm-vlm`  | g7e.12xlarge | 2× MI350X | Multi-modal scene analysis | Qwen3-VL 235B FP8 |
| `vm-chat` | g7e.12xlarge | 2× MI350X | Conversational LLM | Llama 3.3 70B |

Six of the eight available MI350Xs are claimed here; the remaining two stay idle on the host, available for a fourth tenant without touching the existing three.

```bash
export AWS_PROFILE=spinifex

# vm-yolo — YOLO11x object detection, 200 GB disk
YOLO_ID=$(aws ec2 run-instances \
    --image-id ami-ubuntu-amd-gpu \
    --instance-type g7e.12xlarge \
    --key-name spinifex-key \
    --subnet-id <subnet-id> \
    --security-group-ids <sg-id> \
    --block-device-mappings 'DeviceName=/dev/sda1,Ebs={VolumeSize=200,DeleteOnTermination=true}' \
    --count 1 \
    --query 'Instances[0].InstanceId' --output text)
echo "vm-yolo launched: $YOLO_ID"

# vm-vlm — Qwen3-VL 235B FP8, 600 GB disk for weights
VLM_ID=$(aws ec2 run-instances \
    --image-id ami-ubuntu-amd-gpu \
    --instance-type g7e.12xlarge \
    --key-name spinifex-key \
    --subnet-id <subnet-id> \
    --security-group-ids <sg-id> \
    --block-device-mappings 'DeviceName=/dev/sda1,Ebs={VolumeSize=600,DeleteOnTermination=true}' \
    --count 1 \
    --query 'Instances[0].InstanceId' --output text)
echo "vm-vlm launched: $VLM_ID"

# vm-chat — Llama 3.3 70B, 300 GB disk
CHAT_ID=$(aws ec2 run-instances \
    --image-id ami-ubuntu-amd-gpu \
    --instance-type g7e.12xlarge \
    --key-name spinifex-key \
    --subnet-id <subnet-id> \
    --security-group-ids <sg-id> \
    --block-device-mappings 'DeviceName=/dev/sda1,Ebs={VolumeSize=300,DeleteOnTermination=true}' \
    --count 1 \
    --query 'Instances[0].InstanceId' --output text)
echo "vm-chat launched: $CHAT_ID"

# Wait for all three to reach running state
for ID in "$YOLO_ID" "$VLM_ID" "$CHAT_ID"; do
    aws ec2 wait instance-running --instance-ids "$ID"
    echo "$ID is running"
done

# Retrieve IPs (Spinifex assigns addresses from the internal VPC)
YOLO_IP=$(aws ec2 describe-instances --instance-ids "$YOLO_ID" \
    --query 'Reservations[0].Instances[0].PublicIpAddress' --output text)
VLM_IP=$(aws ec2 describe-instances --instance-ids "$VLM_ID" \
    --query 'Reservations[0].Instances[0].PublicIpAddress' --output text)
CHAT_IP=$(aws ec2 describe-instances --instance-ids "$CHAT_ID" \
    --query 'Reservations[0].Instances[0].PublicIpAddress' --output text)

echo "vm-yolo: $YOLO_IP"
echo "vm-vlm:  $VLM_IP"
echo "vm-chat: $CHAT_IP"
```

### 2. Verify GPU access

Once SSH is available, confirm both GPUs are visible inside each VM:

```bash
ssh -i ~/.ssh/spinifex-key ubuntu@$CHAT_IP 'lspci | grep -i amd'
```

Or install and run `amd-smi`:

```bash
ssh -i ~/.ssh/spinifex-key ubuntu@$CHAT_IP \
    'curl -fsSL https://repo.radeon.com/rocm/rocm.gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/rocm.gpg >/dev/null && \
     echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/6.3 noble main" | sudo tee /etc/apt/sources.list.d/rocm.list && \
     sudo apt-get update -qq && sudo apt-get install -y -q amd-smi-lib && \
     amd-smi list'
```

<img width="1520" height="580" src="/docs-assets/hardware-supermicro-smci-h14-2gpus.png" alt="amd-smi inside vm-chat confirming two MI350Xs are directly attached, each with a unique UUID">

Two MI350X entries with distinct UUIDs confirm direct PCIe passthrough is working.

### 3. The orchestration layer

Spinifex provides an EC2-compatible API on bare metal. The relevant capabilities for this demo:

- **Multi-GPU claim per VM.** Each `g7e.12xlarge` requires two PCIe addresses claimed as a unit. Spinifex queries the available GPU pool on the host and assigns them to the guest VM at creation time, rolling back if the full allocation cannot be satisfied.
- **PCIe passthrough via vfio-pci.** All 8 MI350Xs are bound to `vfio-pci` on the bare-metal host. Each VM gets its GPUs at the hardware level.
- **Internal VPC (192.168.10.0/24).** VMs communicate directly without traversing the host's public network.

The result: three completely independent EC2 instances, each running their own separate workloads using only the GPU resources they were allocated.

### 4. Test methodology

Two capture runs were made against the same workload script: **30 s baseline** (YOLO and VLM models running, no chat load) → **25 scripted Llama chat prompts** at 15 s intervals → **30 s settle tail**.

The only difference between the two runs was tensor parallelism:

- **Demo 1 (TP=1):** each VM uses one of its two allocated GPUs. The second sits idle.
- **Demo 2 (TP=2):** each VM shards its model symmetrically across both GPUs using AMD Infinity Fabric (XGMI).

| Metric | TP=1 baseline (Demo 1) | TP=2 (Demo 2) |
|---|---|---|
| Wall clock | 613 s | 600 s |
| YOLO avg fps | 16.91 (min 15.3 / max 18.0) | 17.09 (min 15.6 / max 18.1) |
| Qwen peak tok/s | 61.7 | 53.0 |
| Llama peak tok/s | 36.4 | **53.1** |
| Both GPUs active (chat VM) | no — GPU 1 idle | **yes — symmetric** |
| VRAM per GPU (Qwen) | 259 GB on one, 0.3 GB on other | 241 GB on **both** |
| VRAM per GPU (Llama) | 261 GB on one, 0.3 GB on other | 248 GB on **both** |

### 5. Results

### Inter-tenant isolation

The charts below show the three independent workloads spiking activity on their associated GPUs, measured in GFX utilisation and power draw. YOLO average FPS stays at ~17 despite utilisation in the YOLO VM never exceeding 20%, indicating a bottleneck in networking or encoding rather than inference. Each chart highlights workload independence — pulses of GPU activity share no correlation across VMs.

<img width="1400" height="630" src="/docs-assets/hardware-supermicro-smci-h14-smci-demo1-gpu-utilisation.png" alt="GPU utilisation across all three VMs — Demo 1 TP=1">

GPU utilisation per VM across the full Demo 1 capture.

<img width="1400" height="630" src="/docs-assets/hardware-supermicro-smci-h14-smci-demo2-gpu-power.png" alt="GPU power draw per workload — Demo 2 TP=2">

Power draw per VM during Demo 2.

### Single vs double GPU

When TP=2 is enabled, each VM shards its model symmetrically across both GPUs. Inside the chat VM, both GPUs pulse together on every request, each holding exactly **248 GB of VRAM** (weights split + KV cache shards). With TP=1, GPU 1 sat at 0% for the entire run.

<img width="1400" height="630" src="/docs-assets/hardware-supermicro-smci-h14-smci-demo2-per-gpu-chat.png" alt="Llama 3.3 70B per-GPU activity — TP=2">

Per-GPU utilisation inside `vm-chat` with TP=2.

<img width="1400" height="630" src="/docs-assets/hardware-supermicro-smci-h14-smci-demo1-per-gpu-chat.png" alt="Llama 3.3 70B per-GPU activity — TP=1">

Same VM with TP=1.

### vLLM generation throughput

With TP=1, Qwen peaks at ~60 tok/s and Llama caps around 35 tok/s — both compute-bound on a single MI350X. Enabling TP=2 lifts Llama's peak to ~50 tok/s (+47%) as generation is distributed symmetrically across both GPUs. Qwen's throughput is slightly lower under TP=2 due to collective overhead on the shared-memory transport.

<img width="1400" height="630" src="/docs-assets/hardware-supermicro-smci-h14-smci-demo1-vllm-throughput.png" alt="vLLM generation throughput — TP=1">

Generation throughput during Demo 1 (TP=1).

<img width="1400" height="630" src="/docs-assets/hardware-supermicro-smci-h14-smci-demo2-vllm-throughput.png" alt="vLLM generation throughput — TP=2">

Generation throughput during Demo 2 (TP=2).

### 6. Teardown

```bash
# Stop workloads on each VM
for IP in "$YOLO_IP" "$VLM_IP" "$CHAT_IP"; do
    ssh -i ~/.ssh/spinifex-key ubuntu@$IP 'docker stop $(docker ps -q)' || true
done

# Terminate all three instances — releases 6× MI350X back to the Spinifex pool
aws ec2 terminate-instances --instance-ids "$YOLO_ID" "$VLM_ID" "$CHAT_ID"
```

The six MI350Xs are immediately returned to the pool once the instances terminate, ready for a new allocation without touching the host.

### 7. Conclusion

This tutorial demonstrates how Spinifex can turn a single bare-metal chassis into a multi-tenant AI serving platform. Spinifex utilises the flexibility of PCIe passthrough to allocate GPU resources in whichever configuration is required by the workload/s.

Importantly, it does so with standard `aws ec2` CLI calls — `run-instances`, `describe-instances`, `terminate-instances` — against Spinifex's EC2-compatible endpoint. Teams already operating AWS infrastructure can point their existing tooling at a Spinifex node with a single profile change, against GPUs that sit in their own rack.

---

# Mixed AI Workloads on a Single H200 Chassis: Guest-Managed MIG

URL: https://docs.mulgadc.com/hardware/supermicro/smci-mig
Category: Hardware / Supermicro
Updated: 2026-09-14
Tags: nvidia, h200, mig, vllm, yolo, predastore, bare-metal

Install Spinifex from source, configure host-local networking, attach Predastore storage, and run four concurrent AI workloads across guest-managed MIG slices.

## Overview

Spinifex is an open-source infrastructure platform that brings core AWS services including EC2, EBS and S3 to bare-metal, edge, and on-prem environments. It exposes an AWS compatible API, so any tooling that works against AWS (the `aws` CLI, Terraform, SDKs) works against a Spinifex node unchanged, with a single profile swap.

This guide documents a full bare-metal AI deployment on a single [Supermicro X13](https://www.supermicro.com/en/products/x13) chassis using NVIDIA H200 SXM GPUs. These GPUs include the [NVIDIA Multi-Instance GPU](https://www.nvidia.com/en-au/technologies/multi-instance-gpu/) (MIG) capability, which allows a single GPU to be "sliced" into up to seven independent GPU partitions that are hardware isolated. Each partition is capable of running its own workloads with reserved resources from the "host" GPU.

<p align="center"><img width="600" height="486" src="/docs-assets/hardware-supermicro-smci-mig-X13.png" alt="Supermicro X13 8U GPU System"></p>

In this setup, each EC2 instance receives an entire H200 via PCIe passthrough and manages its own MIG partitions. This gives each tenant full control over how they slice their GPU — including the ability to run heterogeneous workloads at different partition sizes on the same physical card.

A key consideration for this example is that we do not have access to the router upstream of the X13 host, or knowledge of potentially available IP addresses for the VMs we provision on it. The first step in the Instructions section outlines how we attach our own local IP address range to the bridge on the host, effectively creating a pseudo-airgapped environment - although in this specific case outbound connectivity is still required for remote access to the host, downloading required packages etc, with this configuration it is *not* required for host/EC2 instance communication.

### EC2 workload layout

| EC2 Workload | IP | MIG config | Workload | Model |
|---|---|---|---|---|
| `vm-llama3b` | 192.168.10.X  | 7 × 1g.18gb | Chat inference × 7 | Llama-3.2-3B-Instruct |
| `vm-qwen32b` | 192.168.10.X  | 2 × 3g.71gb | Chat inference × 2 | Qwen2.5-32B-Instruct |
| `vm-llama70b` | 192.168.10.X | 1 × 7g.141gb | Chat inference × 1 | Llama-3.1-70B-FP8 |
| `vm-yolo`    | 192.168.10.X | 2 × 3g.71gb | Object detection × 2 | YOLO11x + YOLO11s |

Twelve concurrent inference endpoints in total: 7 fast 3B slots, 2 mid-tier 32B slots, 1 full-GPU 70B slot, and 2 real-time vision streams running side-by-side to compare detection models.


### Platform

| Component | Specification |
|---|---|
| **Chassis** | Supermicro X13 8U GPU System |
| **CPUs** | 2× Intel Xeon Platinum 8568Y+ (48 cores each, 96 cores / 192 threads total) |
| **RAM** | 2 TB DDR5-4800 (32× 64 GB SK Hynix DIMMs) |
| **Storage** | 4× 7.68 TB KIOXIA CD6 NVMe SSDs (30.72 TB raw) |
| **GPUs** | 8× NVIDIA H200 SXM5 (141 GiB HBM3e per GPU, ~1.13 TB total) — 4 used in this demo |
| **Host OS** | Ubuntu 26.04 LTS |
| **Orchestration** | Spinifex — EC2-compatible bare-metal API |
| **Guest OS** | Ubuntu 26.04 LTS |
| **GPU partitioning** | NVIDIA MIG — managed inside each guest VM |
| **Block storage** | Predastore — S3-compatible, NVMe-backed |
| **Container runtime** | Docker |
| **Inference runtime** | vLLM (`vllm-openai` image from local registry) |
| **Vision runtime** | Ultralytics YOLO11 |

## Prerequisites

### 1. Verify GPU visibility on host
Before making any changes, verify that the host can see its GPUs:

<img width="1690" height="1770" src="/docs-assets/hardware-supermicro-smci-mig-nvidia-smi.png" alt="NVIDIA SMI">

## Instructions

### 1. Configure host-local VPC networking

Spinifex utilises bridged networking via OVN. In this example, the remote host exposes only one public IP address, attached to a single physical NIC. Before provisioning Spinifex, we must first ensure that `br-wan` exists and enslave the physical NIC to it.

Since we have no access to an upstream router or knowledge of available IP address pools, we must also attach our own `192.168.10.0/24` address range to `br-wan` which will be used by the guest VMs to communicate with the host and each other.

For example:

```yaml
# /etc/netplan/…
bridges:
  br-wan:
    addresses:
      - 192.168.10.1/24      # VM pool gateway — host-local
      - 198.51.100.10/24     # existing WAN IP — unchanged
    routes:
      - to: default
        via: 198.51.100.1
```

Then apply with `sudo netplan apply`. VMs, once provisioned, are reachable from the host at `192.168.10.x`. For internet access through the host's WAN interface:

```bash
sysctl -w net.ipv4.ip_forward=1
iptables -t nat -A POSTROUTING -s 192.168.10.0/24 -o br-wan -j MASQUERADE
```
The exact process for this is described in the [VPC Networking](https://docs.mulgadc.com/docs/vpc-networking#host-local-subnet-no-upstream-router) guide.



### 2. Install Spinifex

Follow the [Single Node Install](https://docs.mulgadc.com/docs/install) guide. This process will install Spinifex and start Spinifex services, however `spinifex.toml` needs to be edited to finalise the changes made to the networking in the previous section, as described in the following section.

### 3. Configure spinifex.toml and restart services

Edit `/etc/spinifex/spinifex.toml` to point the external pool and VPCD at the bridge created in step 1:

```toml
[network]
external_mode = "pool"

[[network.external_pools]]
name        = "wan"
source      = "static"         # required — no upstream DHCP for this range
range_start = "192.168.10.2"
range_end   = "192.168.10.100"
gateway     = "192.168.10.1"   # second address on br-wan
prefix_len  = 24
dns_servers = ["8.8.8.8"]
```

Then restart all services:

```bash
sudo systemctl start spinifex.target
sudo systemctl status spinifex.target
```

### 4. Attach Predastore storage

Spinifex spreads Predastore's object shards across multiple blob nodes using Reed–Solomon encoding for data redundancy. This chassis has four NVMe drives — one for the OS, three dedicated to data — so we back each blob node with its own physical drive, giving us fault-tolerant storage on a single machine — conceptually similar to RAID 5, where data and parity are spread across drives so a single drive failure is recoverable.

A single-node install is one Predastore host running seven nodes, each with a directory named for its node ID under `/var/lib/spinifex/predastore/cluster`. The three blob nodes are `node-2`, `node-3` and `node-4`, and they are the ones worth their own drive: the gate keeps no data, and the three meta nodes hold only the Raft log for buckets and the object index.

Confirm drive assignments with `lsblk` before proceeding, as device names vary between systems.

```bash
lsblk   # identify the OS drive and the three data drives

# Stop services so Predastore isn't writing while we relocate its data directories
sudo systemctl stop spinifex.target

# --- Repeat the block below for each data drive (node-2/nvme-1, node-3/nvme-2, node-4/nvme-3) ---

# Mount the physical drive at a stable path
sudo mkdir -p /mnt/nvme-1
sudo mount /dev/nvme1n1 /mnt/nvme-1

# Move the blob node's data off the OS drive onto the physical NVMe
sudo mv /var/lib/spinifex/predastore/cluster/node-2 /mnt/nvme-1/node-2

# Symlink the original path back so Predastore finds its data unchanged
sudo ln -s /mnt/nvme-1/node-2 /var/lib/spinifex/predastore/cluster/node-2

# Persist the mount across reboots
echo "/dev/nvme1n1  /mnt/nvme-1  auto  defaults  0  2" | sudo tee -a /etc/fstab

# --- End of per-drive block ---

sudo systemctl start spinifex.target
```

Verify the nodes are healthy before proceeding:

```bash
export AWS_PROFILE=spinifex

aws s3 ls --endpoint-url https://localhost:8443
# Should return without error (empty bucket list is fine)
```

### 5. Enable GPU Passthrough
Spinifex allows GPUs to be utilised by guest VMS via PCIe-passthrough. This can be enabled via `sudo spx admin gpu setup`. Then, after a reboot, run `sudo spx admin gpu enable`. The Spinifex banner should update to reflect GPU passthrough state after every step:

<p align="center"><img width="1196" height="170" src="/docs-assets/hardware-supermicro-smci-mig-spinifex-banner1.png" alt="GPU Enabled"></p>


### 6. Import the GPU AMI

The demo uses the standard Spinifex NVIDIA GPU AMI (`ubuntu-26.04-nvidia-gpu-x86_64`), which includes:

- Ubuntu 26.04 LTS guest image
- NVIDIA server driver (DKMS pre-built against the pinned kernel)
- Docker CE + nvidia-container-toolkit (`--gpus` support enabled at boot)
- Python 3 + venv, common utilities (tmux, curl, ffmpeg, etc.)

```bash
spx admin images import --name ubuntu-26.04-nvidia-gpu-x86_64
```

Confirm and set the AMI ID:
```bash
AMI_ID=$(aws ec2 describe-images \
    --filters "Name=name,Values=ubuntu-26.04-nvidia-gpu-x86_64" \
    --query 'Images[0].ImageId' --output text)
```

### 7. Create VPC resources

```bash
# Create VPC and subnet
VPC_ID=$(aws ec2 create-vpc --cidr-block 192.168.10.0/24 \
    --query 'Vpc.VpcId' --output text)
SUBNET_ID=$(aws ec2 create-subnet --vpc-id $VPC_ID \
    --cidr-block 192.168.10.0/24 \
    --query 'Subnet.SubnetId' --output text)

# Security group — allow SSH + all vLLM/YOLO ports
SG_ID=$(aws ec2 create-security-group \
    --group-name demo-sg --description "H200 MIG demo" \
    --vpc-id $VPC_ID --query 'GroupId' --output text)

aws ec2 authorize-security-group-ingress --group-id $SG_ID \
    --protocol tcp --port 22 --cidr 0.0.0.0/0
aws ec2 authorize-security-group-ingress --group-id $SG_ID \
    --protocol tcp --port 8000-8011 --cidr 0.0.0.0/0
```

```bash
# Create SSH key
aws ec2 create-key-pair --key-name spinifex-key \
  | jq -r '.KeyMaterial | rtrimstr("\n")' > ~/.ssh/spinifex-key

chmod 600 ~/.ssh/spinifex-key
ssh-keygen -y -f ~/.ssh/spinifex-key > ~/.ssh/spinifex-key.pub
```

Verify:

```bash
aws ec2 describe-key-pairs
```

### 8. Launch the four VMs

Each VM gets a whole H200 via PCIe passthrough. The `p5e.4xlarge` instance type maps one H200 per VM:

```bash
aws ec2 run-instances \
  --image-id $AMI_ID \
  --instance-type p5e.4xlarge \
  --key-name spinifex-key \
  --subnet-id $SUBNET_ID \
  --security-group-ids $SG_ID \
  --count 4
```

Wait until all four reach running state, then take note of the public IPs assigned to each instance:

```bash
aws ec2 describe-instances \
    --query 'Reservations[*].Instances[*].[InstanceId,PublicIpAddress,State.Name]' \
    --output table
```

### 9. Enable MIG inside each VM

MIG is enabled inside each guest VM, not on the host. SSH into each VM and enable it:

```bash
# Repeat on each VM IP
ssh -i ~/.ssh/spinifex-key ubuntu@<vm-ip>

sudo nvidia-smi -mig 1
nvidia-smi | grep "MIG M."
# Should show: MIG M.  Enabled
```

### 10. Create MIG partitions inside each VM

With MIG enabled, we are now able to partition each VM's assigned H200 GPU into several separate GPU instances. This process assigns each GPU slice its own UUID, so each VM goes from seeing one whole GPU to seeing a number of "MIG devices":

<p align="center"><img width="1390" height="484" src="/docs-assets/hardware-supermicro-smci-mig-mig-activated.png" alt="NVIDIA SMI"></p>

This allows us to run several separate workloads, each assigned to its own GPU instance, on the same physical GPU, and thus utilise more of the overall GPU's resources.

Run `nvidia-smi mig -lgip` to see the available partition types. MIG profiles are named Xg.Ygb, where:

* X is the number of GPU slices (GPU Instances, or GIs) allocated to the partition. On an H200 there are 7 allocatable GPU slices, so the largest profile is 7g.
* Y is the amount of HBM memory allocated to that partition.

Larger profiles also receive proportionally more SMs (Streaming Multiprocessors - analagous to CPU cores), cache, copy engines, encoders/decoders, and other GPU resources.

Thus we partition the GPUs assigned to our VMs as follows:

**vm-llama3b — 7 × 1g.18gb (Llama 3B):**
```bash
sudo nvidia-smi mig -cgi 1g.18gb,1g.18gb,1g.18gb,1g.18gb,1g.18gb,1g.18gb,1g.18gb -C
nvidia-smi -L  # Verify 7 MIG devices
```

**vm-qwen32b — 2 × 3g.71gb (Qwen 32B):**
```bash
sudo nvidia-smi mig -cgi 3g.71gb,3g.71gb -C
nvidia-smi -L  # Verify 2 MIG devices
```

**vm-llama70b — 1 × 7g.141gb (Llama 70B, full GPU):**
```bash
sudo nvidia-smi mig -cgi 7g.141gb -C
nvidia-smi -L  # Verify 1 MIG device
```

**vm-yolo — 2 × 3g.71gb (YOLO):**
```bash
sudo nvidia-smi mig -cgi 3g.71gb,3g.71gb -C
nvidia-smi -L  # Verify 2 MIG devices
```

Each MIG device gets a UUID of the form `MIG-xxxxxxxx-...`. These UUIDs are used to pin individual containers to their slice via `--gpus "device=<UUID>"` (for Docker) or `CUDA_VISIBLE_DEVICES=<UUID>` (for direct Python processes).

### 11. Configure Docker to use the local image registry

The vLLM image is served from the host's local Docker registry (`192.168.10.1:5000`).

Setting up the local registry on the host:

```bash
# Configure Docker to trust the local registry address
sudo tee /etc/docker/daemon.json > /dev/null <<'EOF'
{
  "insecure-registries": ["192.168.10.1:5000"]
}
EOF
sudo systemctl restart docker

# Start the registry container, bound to the bridge interface only
docker run -d \
    -p 192.168.10.1:5000:5000 \
    --name registry \
    --restart=always \
    registry:2

# Pull the vLLM image and push it into the local registry
docker pull vllm/vllm-openai:latest
docker tag vllm/vllm-openai:latest 192.168.10.1:5000/vllm-openai:latest
docker push 192.168.10.1:5000/vllm-openai:latest

# Verify the image is available
curl http://192.168.10.1:5000/v2/_catalog
# {"repositories":["vllm-openai"]}
```

Each VM then needs to trust it as an insecure registry:

```bash
# On each VM — vm-yolo uses direct Python, not Docker for YOLO
sudo tee /etc/docker/daemon.json > /dev/null <<'EOF'
{
  "insecure-registries": ["192.168.10.1:5000"]
}
EOF
sudo systemctl restart docker
```

### 12. Deploy the LLM workloads

Each LLM is downloaded using the Huggingface CLI onto the respective VM. First install the CLI:

```bash
pip install -U "huggingface_hub[cli]"
```

Then download the model and determine its snapshot directory. This directory contains the model’s config.json, tokenizer files, and weight files, and will be used as MODEL_DIR when launching vLLM.

```bash
#On vm-llama3b
hf download meta-llama/Llama-3.2-3B-Instruct

MODEL_DIR=$(dirname "$(find ~/.cache/huggingface/hub -path '*Llama-3.2-3B-Instruct*' -name config.json | head -1)")
echo "$MODEL_DIR"

#On vm-qwen32b
hf download Qwen/Qwen2.5-32B-Instruct

MODEL_DIR=$(dirname "$(find ~/.cache/huggingface/hub -path '*Qwen2.5-32B-Instruct*' -name config.json | head -1)")
echo "$MODEL_DIR"

#On vm-llama70b
hf download RedHatAI/Meta-Llama-3.1-70B-Instruct-FP8

MODEL_DIR=$(dirname "$(find ~/.cache/huggingface/hub -path '*Meta-Llama-3.1-70B-Instruct-FP8*' -name config.json | head -1)")
echo "$MODEL_DIR"
```

Then we run the docker containers as follows:

**vm-llama3b — 7 × Llama-3.2-3B-Instruct (one container per MIG slice):**

Enumerate the MIG UUIDs and start one vLLM container per slice:


```bash
mapfile -t UUIDS < <(nvidia-smi -L | grep -oP 'MIG-[0-9a-f-]+')
for i in "${!UUIDS[@]}"; do
    docker run -d --rm \
        --name "vllm-$i" \
        --gpus "device=${UUIDS[$i]}" \
        --ipc host \
        -p $((8000 + i)):8000 \
        -v "${MODEL_DIR}:/models:ro" \
        192.168.10.1:5000/vllm-openai:latest \
        vllm serve /models \
            --served-model-name llama-3b \
            --dtype bfloat16 \
            --max-model-len 4096 \
            --gpu-memory-utilization 0.90 \
            --port 8000
done
```

**vm-qwen32b — 2 × Qwen2.5-32B-Instruct:**

```bash
mapfile -t UUIDS < <(nvidia-smi -L | grep -oP 'MIG-[0-9a-f-]+')
for i in "${!UUIDS[@]}"; do
    docker run -d --rm \
        --name "qwen32b-$i" \
        --gpus "device=${UUIDS[$i]}" \
        --ipc host \
        -p $((8000 + i)):8000 \
        -v "${MODEL_DIR}:/models:ro" \
        192.168.10.1:5000/vllm-openai:latest \
        vllm serve /models \
            --served-model-name qwen2.5-32b \
            --dtype bfloat16 \
            --max-model-len 4096 \
            --gpu-memory-utilization 0.90 \
            --port 8000
done
```

**vm-llama70b — Llama-3.1-70B-Instruct-FP8 (full GPU slice):**

```bash
UUID=$(nvidia-smi -L | grep -oP 'MIG-[0-9a-f-]+' | head -1)
docker run -d --rm \
    --name vllm \
    --gpus "device=${UUID}" \
    --ipc host \
    -p 8000:8000 \
    -v "${MODEL_DIR}:/models:ro" \
    192.168.10.1:5000/vllm-openai:latest \
    vllm serve /models \
        --served-model-name meta-llama-3.1-70b \
        --dtype auto \
        --max-model-len 8192 \
        --gpu-memory-utilization 0.90 \
        --port 8000
```

### 13. Deploy YOLO object detection (vm-yolo)

For this demo we built a simple YOLO inference server that runs detection on a looping video file and serves the annotated output as an MJPEG stream. The same script runs twice on vm-yolo — once for YOLO11x (larger, more accurate) and once for YOLO11s (smaller, faster) — each pinned to its own MIG slice and listening on a different port.

The MIG pinning works through `CUDA_VISIBLE_DEVICES`: setting it to a MIG UUID before launch scopes the process to that slice, and CUDA remaps it to device index 0 inside the process. From the script's perspective it always sees one GPU at index 0 — the MIG boundary is invisible to it. `YOLO_MODEL` and `PORT` are what actually differentiate the two instances:

```bash
python3 -m venv ~/yolo-venv
~/yolo-venv/bin/pip install ultralytics fastapi "uvicorn[standard]" opencv-python-headless httpx

# Get MIG UUIDs
nvidia-smi -L
# GPU 0: NVIDIA H200 (UUID: GPU-...)
#   MIG 3g.71gb  Device 0: (UUID: MIG-<uuid0>)
#   MIG 3g.71gb  Device 1: (UUID: MIG-<uuid1>)

# YOLO11x on slice 0 — larger model, port 8010
CUDA_VISIBLE_DEVICES=MIG-<uuid0> \
  VIDEO_PATH=<path-to-video> \
  YOLO_MODEL=yolo11x.pt \
  YOLO_DEVICE=0 \
  PORT=8010 \
  ~/yolo-venv/bin/python ~/yolo_stream.py >> ~/yolo-x.log 2>&1 &

# YOLO11s on slice 1 — smaller/faster model, port 8011
CUDA_VISIBLE_DEVICES=MIG-<uuid1> \
  VIDEO_PATH=<path-to-video> \
  YOLO_MODEL=yolo11s.pt \
  YOLO_DEVICE=0 \
  PORT=8011 \
  ~/yolo-venv/bin/python ~/yolo_stream.py >> ~/yolo-s.log 2>&1 &
```

Each instance exposes a `/video` endpoint serving a `multipart/x-mixed-replace` MJPEG stream, consumable directly by browsers and most HTTP clients. YOLO11x (~75 MB) and YOLO11s (~9 MB) weights download automatically on first run from the Ultralytics model hub.

### 14. Dashboard

We also built a simple host-side dashboard — another FastAPI application that proxies all the VM streams to the browser so only one port on the host needs to be exposed. Each VM endpoint is wired in by address at startup, mapping directly to the IPs assigned in step 8:

```
http://vm-yolo-IP:8010  →  vm-yolo  YOLO11x MJPEG stream
http://vm-yolo-IP:8011  →  vm-yolo  YOLO11s MJPEG stream
http://vm-llama3b-IP:8000   →  vm-llama3b  vLLM endpoint 0
http://vm-llama3b-IP:8001   →  vm-llama3b  vLLM endpoint 1
... (one entry per endpoint across all three LLM VMs)
```

The dashboard has two proxy patterns: MJPEG passthrough for the YOLO feeds (forwarding the raw boundary stream from vm-yolo to the browser), and SSE passthrough for the LLM token streams (subscribing to each vLLM endpoint and re-emitting tokens as server-sent events). Both use a reconnect loop so the browser connection stays open if a VM is temporarily unreachable.



<p><video src="https://iso.mulgadc.com/h200-demo.mp4" controls width="100%" style="border-radius:6px"></video></p>

The dashboard shows:
- GPU allocation bars for all four EC2 instancess (proportional to MIG slice size)
- Live streaming LLM responses per endpoint, colour-coded by tier
- Side-by-side YOLO11x vs YOLO11s video feeds with FPS and detection counts

The dashboard also displays the overall GPU utilisation, derived as a percentage of maximum power usage. Importantly, it demonstrates how Spinifex combined with NVIDIA's MIG capability enables the deployment of multiple heterogeneous workloads on owned hardware.

### 15. Teardown

```bash
# On vm-yolo:
# Stop YOLO processes
pkill -f yolo_stream.py

# On each LLM instance:
# Stop docker containers

docker stop $(docker ps -q)

# Disable MIG inside each VM before terminating.
sudo nvidia-smi mig -dci
sudo nvidia-smi mig -dgi
sudo nvidia-smi -mig 0

# On the host:
# Terminate all four instances — releases 4× H200 back to the Spinifex pool
aws ec2 terminate-instances --instance-ids \
    $(aws ec2 describe-instances \
        --filters "Name=instance-state-name,Values=running" \
        --query 'Reservations[*].Instances[*].InstanceId' \
        --output text)
```

The four H200s are immediately returned to the host GPU pool once the instances terminate, ready for reallocation.

### 16. Conclusion

This document highlights how Spinifex can turn a single bare-metal chassis into a multi-tenant AI serving platform. Spinifex utilises the flexibility of PCIe passthrough combined with NVIDIA's MIG capability to allocate GPU resources in whichever configuration is required by the workload/s. This flexibility and fine-grain control ensures maximum GPU utilisation.

Importantly, it does so with standard `aws ec2` CLI calls — `run-instances`, `describe-instances`, `terminate-instances` — against Spinifex's EC2-compatible endpoint. Teams already operating AWS infrastructure can point their existing tooling at a Spinifex node with a single profile change, against GPUs that sit in their own rack.

---

# Spinifex EKS AI Platform on Dual RTX Pro 6000 Baremetal

URL: https://docs.mulgadc.com/hardware/supermicro/rtx-pro-6000
Category: Hardware / Supermicro
Updated: 2026-09-14
Tags: nvidia, rtx-pro-6000, eks, kubernetes, llama.cpp, yolo, ecr, bare-metal, terraform

Deploy a GPU-accelerated AI inference platform — an OpenAI-compatible LLM API and a real-time CV stream — on Kubernetes on bare metal with standard AWS tooling.

## Overview

Spinifex is an open-source infrastructure platform that brings core AWS services such as EC2, S3, EBS and EKS, to bare-metal, edge, and on-prem deployments. It exposes a fully AWS-compatible API, so any tooling that works against AWS — the `aws` CLI, OpenTofu, `kubectl` — works against a Spinifex node unchanged, with a single profile swap.

This guide walks through the use of Spinifex to deploy a self-contained AI inference platform on Supermicro's bare metal X14 platform using only standard AWS tooling. We use Terraform to create resources (wrapped in simple `make` commands) in the exact same way you would create AWS resources.


Specifically, we create an EKS cluster with two worker nodes, each consisting of a g7e.2xlarge EC2 instance with an attached GPU via VFIO passthrough, an ALB to route traffic to each node, ECR for storing and managing our workload images, and all of the associated security and certificate management requirements (IAM, ACM) you would expect from real AWS.

<p align="center"><img width="2078" height="1350" src="/docs-assets/hardware-supermicro-rtx-pro-6000-platform.png" alt="AI Platform request flow: HTTPS ingress → ALB routing → EKS ai-platform GPU workers"></p>

### Platform

| Component | Specification |
|---|---|
| **Chassis** | Supermicro X14 2U CloudDC with 2× Intel Xeon 6730P |
| **Memory** | 8 × 64 GB DDR5 6400 MHz ECC RDIMM (512 GB total) |
| **GPUs** | 2× NVIDIA RTX Pro 6000 Blackwell Server Edition (96 GiB GDDR7 each, 192 GiB total) |
| **Storage** | 4× NVMe SSD: 2× 1.5 TB, 2× 880 GB — one 1.5 TB drive carries the OS; the remaining three back Predastore |
| **Spinifex instance family** | `g7e` — one RTX Pro 6000 per instance via VFIO PCIe passthrough |
| **Kubernetes** | k3s, managed via Spinifex EKS API |
| **API endpoint** | `https://<host>:9999` (AWS-compatible) |

### Workloads
Three workloads run in the `inference` namespace of an EKS cluster backed by two GPU worker nodes:

| Pod | Role | GPU | Model | Image source |
|---|---|---|---|---|
| `llm-server` | OpenAI-compatible chat API (llama.cpp) | 1× RTX Pro 6000 | Llama 3.2 3B Instruct Q4_K_M GGUF | ECR `llm-server:latest` |
| `yolo-stream` | MJPEG object-detection stream (CUDA) | 1× RTX Pro 6000 | YOLO11x | ECR `yolo-stream:latest` |
| `ai-dashboard` | Web UI: chat + live detection feed | CPU only | — | ECR `ai-dashboard:latest` |

Both GPU workloads bake their model weights into the Docker image at build time — `llm-server`'s GGUF weights (~2 GiB) and `yolo-stream`'s YOLO11x checkpoint (~109 MB) are present in the image when it starts.

`yolo-stream` renders a 1280×720 sample video through YOLO11x once on startup (~25 s), caches the annotated frames in memory, then serves `/stream` from that cache — smooth MJPEG playback decoupled from per-frame inference cost after the initial warm-up.

In this case, the two workloads demonstrated could both run comfortably on a single RTX Pro 6000 with ample headroom. However, this reference architecture primarily seeks to show how Spinifex can use EKS to provision infrastructure with resources in mind - With one RTX Pro 6000 per node and both pods requesting `nvidia.com/gpu: "1"`, the scheduler assigns one GPU and one workload per node. Thus if larger models were used (such as Llama 3.3 70B, Q4_K_M, ~40 GiB for the LLM workload), Spinifex's EKS implementation would ensure the worker nodes do not compete for resources.

## Architecture

<p align="center"><img width="2880" height="1512" src="/docs-assets/hardware-supermicro-rtx-pro-6000-ai-platform-request-flow-2.png" alt="AI Platform request flow: HTTPS ingress → ALB routing → EKS ai-platform GPU workers"></p>

### AWS services exercised

| Service | Role |
|---|---|
| **IAM** | Cluster role, node role (with ECR, LBC, EBS-CSI permissions inline), viewer access entry |
| **ECR** | Private registry for all three images; source of truth for builds, though nodes receive images via sideload rather than live OCI pull for this demo |
| **EC2** | 2× GPU microVM (`g7e.2xlarge`), one RTX Pro 6000 each via VFIO PCIe passthrough |
| **EKS** | Cluster, GPU nodegroup (`desired_size = 2`), LBC and EBS-CSI managed addons, access entries |
| **ELBv2** | ALB provisioned by the LBC addon; single shared IngressGroup across all three services |
| **ACM** | Self-signed cert (`ai-platform.spinifex.local`) imported and attached to the ALB HTTPS listener |
| **EBS (Viperblock)** | 200 GB root volume per GPU worker node, provisioned by the nodegroup (`disk_size = 200`)|

All permissions for the LBC and EBS-CSI addons are attached directly to the node role. Both addons support IRSA, but fall back to the node's instance profile when no `service_account_role_arn` is supplied — sufficient for a single-cluster deployment.

## Prerequisites

### On the Spinifex host

**1. Install Spinifex**

Follow the [Single Node Install](https://docs.mulgadc.com/docs/install) guide. This installs Spinifex and starts all services.

**2. Configure spinifex.toml and restart services**

Spinifex uses OVN for bridged networking. EC2 instances receive IP addresses from a pool configured in `spinifex.toml`. For a standard install, reserve a range of addresses from your local network — either a static block or let Spinifex request addresses from an upstream DHCP server:

```toml
[network]
external_mode = "pool"

[[network.external_pools]]
name        = "wan"
source      = "static"          # or "dhcp" to use an upstream DHCP server
range_start = "<pool-start>"
range_end   = "<pool-end>"
gateway     = "<upstream-gateway>"
prefix_len  = <prefix>
dns_servers = ["8.8.8.8"]
```

Then restart all services:

```bash
sudo systemctl restart spinifex.target
sudo systemctl status spinifex.target
```

See the [VPC Networking](https://docs.mulgadc.com/docs/vpc-networking) guide for full configuration options.

**3. Bind the GPUs to VFIO**

```bash
sudo spx admin gpu setup
# Reboot, then:
sudo spx admin gpu enable
```

Confirm both GPUs are bound:

```bash
lspci -d 10de: -nn
# Expect: NVIDIA Corporation Device [10de:2bb5] appearing twice
```

**4. Attach Predastore storage**

The X14 has four NVMe drives: two 1.5 TB SSDs (one carries the OS) and two ~880 GB SSDs. The OS occupies its own dedicated NVMe; the remaining three drives are pre-formatted and already mounted at `/mnt/nvme-1`, `/mnt/nvme-2`, and `/mnt/nvme-3`. Predastore spreads object shards across these three drives — one blob node per physical drive, with Reed–Solomon redundancy so a single drive failure is recoverable.

A single-node install is one Predastore host running seven nodes, each with a directory named for its node ID under `/var/lib/spinifex/predastore/cluster`. The three blob nodes are `node-2`, `node-3` and `node-4`, so `nvme-$i` takes `node-$((i + 1))`. The gate keeps no data at all, and the three meta nodes hold only the Raft log for buckets and the object index, which is small enough to leave on the OS drive.

Relocate the blob node directories onto the mounted drives:

```bash
sudo systemctl stop spinifex.target

for i in 1 2 3; do
  node="node-$((i + 1))"
  sudo mv /var/lib/spinifex/predastore/cluster/$node /mnt/nvme-$i/$node
  sudo ln -s /mnt/nvme-$i/$node /var/lib/spinifex/predastore/cluster/$node
done

sudo systemctl start spinifex.target
```

Verify the nodes are healthy before proceeding:

```bash
export AWS_PROFILE=spinifex
aws s3 ls
# Should return without error (empty bucket list is fine)
```

> **Future direction:** Predastore will support ZFS for cross-disk redundancy on a single node, eliminating the need for Step 4 and reserving Predastore's Reed–Solomon for the multi-node level.

**5. Verify the GPU instance type**

```bash
sudo spx admin gpu status
```

This will print confirmation that GPU passthrough has been configured correctly along with the available GPU instance types. The RTX Pro 6000 Blackwell Server Edition (PCI device `10de:2bb5`) maps to the `g7e` family. The workbook defaults to `g7e.2xlarge` (one GPU per node); override with `GPU_TYPE=g7e.4xlarge` (or the size your host reports) if needed. Do not use `g7e.12xlarge` — that is the 2× GPU size.

### Local tooling

- [OpenTofu](https://opentofu.org/) >= 1.6
- [kubectl](https://kubernetes.io/docs/tasks/tools/)
- [Docker](https://docs.docker.com/get-docker/)


### Clone the workbook

```bash
git clone https://github.com/mulgadc/eks-ai-platform
cd eks-ai-platform
```

## Instructions

### 1. Import the EKS GPU node AMI

GPU worker nodes require the `ecr-credential-provider` binary so kubelet can call `GetAuthorizationToken` against ECR. This binary is included in the dedicated EKS GPU node AMI in the Spinifex image catalogue. List available images and import it:

```bash
spx admin images list
# Look for the EKS GPU node image

spx admin images import --name spinifex-eks-node-gpu
```

Confirm the AMI is registered:

```bash
aws ec2 describe-images --query 'Images[*].[Name,ImageId]' --output table
```

### 2. Provision the cluster

```bash
make infra ENDPOINT=https://<host>:9999
```

This provisions the VPC (`10.33.0.0/16`, two public and two private subnets with a NAT gateway), IAM roles, ECR repositories, EKS cluster, GPU nodegroup (2× `g7e.2xlarge`, 200 GB disk each), LBC and EBS-CSI managed addons, a self-signed ACM certificate, and the NodePort security group rules the ALB needs to reach the worker nodes.

Update your kubeconfig once the cluster reports `ACTIVE`:

```bash
$(tofu -chdir=workbook output -raw update_kubeconfig)
kubectl get nodes
# Expect: 2 Ready nodes in the gpu-workers nodegroup
```

The Makefile wraps `tofu -chdir=workbook apply -var spinifex_endpoint=... -var gpu_instance_type=...` — running Tofu directly is equivalent and lets you pass any additional variables. The workbook provisions `aws_vpc`, `aws_subnet` (two public, two private), `aws_eks_cluster`, `aws_eks_node_group` (two `g7e.2xlarge` nodes, each with a 200 GB Viperblock root volume via `disk_size = 200`), `aws_eks_addon` for LBC and EBS-CSI, three `aws_ecr_repository` resources, three IAM roles, and a self-signed `aws_acm_certificate` — all via Spinifex's AWS-compatible endpoint at `:9999`.

The full workbook is at [`workbook/main.tf`](https://github.com/mulgadc/eks-ai-platform/blob/main/workbook/main.tf). The AWS provider points all standard API calls at Spinifex's endpoint — the same Terraform resources that work on AWS work here unchanged:

```hcl
provider "aws" {
  endpoints {
    ec2 = var.spinifex_endpoint
    iam = var.spinifex_endpoint
    sts = var.spinifex_endpoint
    eks = var.spinifex_endpoint
    ecr = var.spinifex_endpoint
    acm = var.spinifex_endpoint
  }
}

resource "aws_eks_cluster" "this" {
  name     = var.cluster_name
  role_arn = aws_iam_role.cluster.arn
  version  = var.k8s_version

  access_config {
    authentication_mode = "API"
  }
}

resource "aws_eks_node_group" "gpu_workers" {
  cluster_name   = aws_eks_cluster.this.name
  instance_types = [var.gpu_instance_type]  # g7e.2xlarge — one RTX Pro 6000 per node
  disk_size      = 200

  scaling_config {
    desired_size = 2
    min_size     = 2
    max_size     = 2
  }
}
```



### 3. Build and push container images

```bash
make images
```

This authenticates to ECR, then builds and pushes all three images:

- **`llm-server`** — based on `ghcr.io/ggml-org/llama.cpp:server-cuda`; downloads Llama 3.2 3B Instruct Q4_K_M GGUF (~2 GiB) from Hugging Face at build time and bakes it into the image. Exposes an OpenAI-compatible `/v1/chat/completions` API.
- **`yolo-stream`** — based on `pytorch/pytorch:2.7.1-cuda12.8-cudnn9-runtime`; installs Ultralytics and downloads YOLO11x weights at build time. PyTorch 2.7.1+cu128 is required: the RTX Pro 6000 Blackwell is compute capability sm_120, and earlier PyTorch releases ship no sm_120 kernels.
- **`ai-dashboard`** — lightweight Flask proxy (`FROM python:3.11-slim`) that aggregates the LLM API and YOLO stream into a single page.

The ECR registry URI always includes `:9999` — for example, `<account>.dkr.ecr.ap-southeast-2.<suffix>:9999`. Use the `ecr_registry` Tofu output directly in `docker login` and image references; do not construct the hostname manually.

Image URIs come from `tofu -chdir=workbook output -raw ecr_registry`. ECR authentication uses the same API as AWS: `aws ecr get-login-password` calls `GetAuthorizationToken` against the Spinifex ECR endpoint and returns a short-lived JWT that Docker accepts as a registry password. The `make images` target is equivalent to running those `docker build` and `docker push` commands directly against `$REGISTRY` from that Tofu output.

The three ECR repositories are provisioned by the infra workbook:

```hcl
locals {
  ecr_repos = toset(["llm-server", "yolo-stream", "ai-dashboard"])
}

resource "aws_ecr_repository" "app" {
  for_each     = local.ecr_repos
  name         = each.key
  force_delete = true
}
```

### 4. Sideload images onto GPU worker nodes

```bash
make sideload
```

ECR is the source of truth for all three images — authentication, push, and registry management all work identically to AWS. In a standard EKS deployment, nodes would pull images directly from ECR at scheduling time. In this demo, live pulls of these image sizes (~2 GiB for `llm-server`, ~4.5 GiB for `yolo-stream`) through the Spinifex ECR gateway proved unreliable — large transfers stalled or failed mid-stream, a combination of network conditions on this single-host setup and a rough edge in early Spinifex ECR support. This step works around that by staging the images directly into each node's containerd store before the pods are scheduled.

It exports each image from the local Docker daemon, serves the tarballs over HTTP from the Spinifex host, and imports them directly into each worker node's containerd store via short-lived privileged pods. All three Deployments use `imagePullPolicy: IfNotPresent` and depend on the images already being present in containerd.

The script also labels the two GPU nodes deterministically (`workload=llm-server` / `workload=yolo-stream`, sorted by node name), and the Deployments use matching `nodeSelector` values so each pod lands on the node that already has its image. `ai-dashboard` has no GPU requirement and is imported on both nodes since it can schedule onto either.

Expect several minutes for `yolo-stream`'s image (CUDA + PyTorch + YOLO11x weights, ~4.5 GiB).

This step has no Tofu equivalent — it operates directly on the running cluster via `kubectl`. The ECR registry URI is read from the Tofu state; node labelling uses `kubectl label` and image import uses short-lived privileged pods that run `ctr images import` into each node's containerd store.

### 5. Deploy workloads

```bash
make workloads ENDPOINT=https://<host>:9999
```

This deploys the NVIDIA GPU Operator via Helm, then all three application Deployments, ClusterIP and NodePort services, and the shared ALB Ingresses.

The GPU Operator requires two adjustments for k3s:

- `driver.enabled=false` — the NVIDIA driver is pre-built into the GPU AMI at image creation time; the Operator installs only the toolkit and device plugin.
- `CONTAINERD_SOCKET=/run/k3s/containerd/containerd.sock` — k3s bundles its own containerd at a different socket and config path than the standalone containerd default. Without this override the toolkit DaemonSet crash-loops with `no such file or directory`.

Watch the GPU Operator complete, then the inference pods come up:

```bash
kubectl -n gpu-operator get pods -w
kubectl -n inference get pods -w
```

Confirm each GPU node reports `nvidia.com/gpu: 1` in allocatables:

```bash
$(tofu -chdir=workbook output -raw gpu_verify_hint)
```

All three routes share a single ALB via `alb.ingress.kubernetes.io/group.name`. Explicit `group.order` values (`/v1` = 10, `/stream` = 20, `/` = 100) ensure the dashboard's catch-all path evaluates last — without them, the LBC sorts Ingress resources alphabetically, which places the catch-all first and swallows the other routes.

Retrieve the ALB IP (the DNS name `*.elb.spinifex.local` is a label, not a resolvable entry):

```bash
$(tofu -chdir=workbook output -raw alb_ip_hint)
ALB_IP=<output>
```

Validate the LLM endpoint:

```bash
curl -sk https://$ALB_IP/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"llama-3.2-3b-instruct","messages":[{"role":"user","content":"What is Spinifex?"}]}' \
  | python3 -m json.tool
```

Open `https://$ALB_IP/` to reach the dashboard. If accessing from a remote machine:

```bash
ssh -L 8443:$ALB_IP:443 <spinifex-host>
# Open https://ai-platform.spinifex.local:8443/
# Add 127.0.0.1 ai-platform.spinifex.local to /etc/hosts if the browser requires hostname match
```

The workloads module reads cluster coordinates, ECR image URIs, NodePort values, and the ACM cert ARN from the parent module's Tofu state, then creates the NVIDIA GPU Operator `helm_release`, three `kubernetes_deployment_v1` resources, six `kubernetes_service_v1` resources (ClusterIP + NodePort per workload), and three `kubernetes_ingress_v1` resources — all through the Terraform Kubernetes and Helm providers, which authenticate to the cluster via `aws eks get-token`.

The full workloads module is at [`workbook/workloads/main.tf`](https://github.com/mulgadc/eks-ai-platform/blob/main/workbook/workloads/main.tf). Each GPU pod requests one `nvidia.com/gpu` resource — the scheduler enforces the one-per-node split automatically — and all three services share a single ALB provisioned by the Load Balancer Controller via standard Kubernetes ingress annotations:

```hcl
container {
  image = local.images.llm_server  # ECR URI from parent module state
  resources {
    limits   = { "nvidia.com/gpu" = "1", memory = "8Gi" }
    requests = { "nvidia.com/gpu" = "1", memory = "4Gi" }
  }
}

resource "kubernetes_ingress_v1" "llm" {
  metadata {
    annotations = {
      "alb.ingress.kubernetes.io/group.name"      = local.alb_group
      "alb.ingress.kubernetes.io/group.order"     = "10"
      "alb.ingress.kubernetes.io/certificate-arn" = local.cert_arn
      "alb.ingress.kubernetes.io/listen-ports"    = "[{\"HTTPS\":443}]"
    }
  }
  spec {
    rule { http { path { path = "/v1"; path_type = "Prefix" } } }
  }
}
```


### Dashboard

After the successful completion of the `make workloads` step, the dashboard should be available and displaying the outputs of the two worker nodes; YOLO computer vision in the left pane, and a LLM chat in the right pane, as shown in the images below.

<p><video src="https://iso.mulgadc.com/full-dashboard.mp4" controls width="100%" style="border-radius:6px"></video></p>

### 6. Teardown

```bash
make destroy ENDPOINT=https://<host>:9999
```

Workloads are destroyed before infra. Both GPU worker instances terminate, immediately returning their RTX Pro 6000s to the Spinifex pool.

The Makefile runs the two-module destroy sequence — `tofu -chdir=workbook/workloads destroy` first, then `tofu -chdir=workbook destroy` — because the parent module's security group rules are referenced by the ALB created in the workloads layer. Running Tofu directly in that order is equivalent.

## Troubleshooting

### `llm-server` or `yolo-stream` pod stuck in `Pending`

The GPU Operator DaemonSet must complete before `nvidia.com/gpu` appears in node allocatables:

```bash
kubectl -n gpu-operator get daemonset -w
kubectl get node -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.'nvidia\.com/gpu'
```

If a pod shows `Insufficient nvidia.com/gpu` on a node that looks healthy, the `workload=` labels from `make sideload` may be stale — for example, after a nodegroup recreation that assigned new node names. Re-run `make sideload` to relabel the nodes and re-import the images.

### GPU worker instances fail to launch (`bind ... to vfio-pci: invalid argument`)

On hosts where the GPU's IOMMU group contains an upstream PCIe root-port bridge (no ACS isolation), Spinifex attempts to bind the bridge to `vfio-pci`. Bridges are non-endpoint devices that `vfio-pci` refuses to bind, causing the instance to crash immediately after launch:

```
GPU claim failed... bind IOMMU group member 0000:14:02.0: bind 0000:14:02.0 to vfio-pci: invalid argument
```

Check IOMMU group membership with `lspci -nnk` and `/sys/kernel/iommu_groups/*/devices/`. This is fixed in Spinifex by excluding bridge-class PCI devices from the bind lifecycle — ensure your Spinifex build includes that fix. If the failed bind left a bridge without a driver, restore it:

```bash
echo | sudo tee /sys/bus/pci/devices/<bridge-addr>/driver_override
echo <bridge-addr> | sudo tee /sys/bus/pci/drivers/pcieport/bind
```

If the nodegroup got wedged in `CREATING` after hitting this, delete it and let Terraform recreate it:

```bash
aws eks delete-nodegroup --cluster-name ai-platform --nodegroup-name gpu-workers
```

### Nodegroup stuck `CREATING` with healthy nodes, Tofu times out

`tofu apply` hangs for 20 minutes and fails with `workers did not become Ready: timed out`, even though `kubectl get nodes` shows both workers `Ready`. The cause is a missing `eks.amazonaws.com/nodegroup` label on the node — without it, the control plane never tallies the nodegroup as satisfied. Confirm with:

```bash
kubectl get node <worker> -o jsonpath='{.metadata.labels}'
```

This is fixed upstream in Spinifex. If stuck, delete both the nodegroup and cluster and let Terraform recreate them cleanly:

```bash
aws eks delete-nodegroup --cluster-name ai-platform --nodegroup-name gpu-workers
aws eks delete-cluster --name ai-platform
```

### ALB returns 502 immediately after rollout

Both `llm-server` and `yolo-stream` require a few seconds after container start before their readiness probes pass — CUDA and GPU Operator initialisation contribute to the delay. The ALB marks targets unhealthy during this window. Watch the pods become ready:

```bash
kubectl -n inference get pods -w
kubectl -n inference logs -f deploy/llm-server
```


## Conclusion

This guide demonstrates how Spinifex turns a single bare-metal chassis into a production-shaped AI serving platform managed entirely with standard AWS tooling. IAM roles, ECR repositories, an EKS cluster, GPU worker nodes, addons, and an ALB are all provisioned with the same Terraform resources and AWS CLI commands that work on real AWS — with a single `AWS_PROFILE` swap.

The RTX Pro 6000 Blackwell Server Edition's 96 GiB GDDR7 fits substantial GPU workloads in a single PCIe slot, and Spinifex's `g7e` instance family exposes each GPU as a standard EC2 instance. Teams already operating AWS infrastructure can point their existing tooling at a Spinifex node and retain the full EKS workflow — from `aws ecr get-login-password` to `kubectl get ingress` — on hardware they own. ECR acts as the canonical registry throughout: image build, push, and authentication are identical to AWS, with direct kubelet pulls being the intended path as Spinifex's ECR gateway matures.

---

# Arcee Trinity 400B FP8 on AMD MI350X with Spinifex

URL: https://docs.mulgadc.com/hardware/supermicro/arcee-trinity-mi350x
Category: Hardware / Supermicro
Updated: 2026-09-14
Tags: supermicro, amd, mi350x, gpu-passthrough, vllm, llm, arcee

Provision, load, and benchmark arcee-ai/Trinity-Large-Preview-FP8 across two AMD MI350X GPUs on a Supermicro H14 node using Spinifex's EC2-compatible API.

## Overview

Spinifex is an open-source infrastructure platform that brings core AWS services including EC2, EBS and S3 to bare-metal, edge, and on-prem environments. It exposes an EC2-compatible API, so any tooling that works against AWS (the `aws` CLI, Terraform, SDKs) works against a Spinifex node unchanged, with a single profile swap.

### About Trinity

[Trinity](https://www.arcee.ai/trinity) is a family of open-weight language models from [Arcee AI](https://www.arcee.ai), built specifically for production agentic workloads: reliable tool calling, structured JSON output, coherent multi-turn conversations, and long-context reasoning. The family spans four sizes — Nano (6B), Mini (26B), Large Preview (400B), and Large Thinking (400B reasoning-optimised), with consistent capabilities and API surface across all of them. A workflow validated on Nano can be promoted to Large with no prompt changes.

Trinity Large Preview is a **sparse Mixture-of-Experts (MoE)** model. The headline number is 400B total parameters, but only **13B parameters activate per token** — the MoE routing selects a subset of experts for each forward pass, so inference cost is closer to a much smaller dense model than the raw parameter count suggests. The result is a 400B-class model with a **512K token context window** that fits on two high-memory GPUs and generates at competitive throughput.

Arcee has trained Trinity with a heavy focus on agent reliability — function selection accuracy, valid parameter generation, graceful failure recovery, and schema adherence for structured outputs. It is available as open weights on HuggingFace and is natively compatible with vLLM, SGLang, and llama.cpp.

### This guide

This writeup covers running Trinity Large Preview FP8 on a Supermicro H14 bare-metal node via Spinifex — our EC2-compatible orchestration layer. Two AMD Instinct MI350X GPUs (288 GB HBM3e each, 576 GB combined) provide just enough headroom for the FP8 weights plus active KV cache, provisioned and managed with standard `aws ec2` CLI commands.

At that scale the FP8 weights require around 400 GB of GPU memory — more than a single MI350X can hold. This guide covers splitting the model symmetrically across two MI350Xs using vLLM tensor parallelism (TP=2), running on a `g7e.12xlarge` instance provisioned via Spinifex on the H14.

The complete stack:

| Layer | Detail |
|---|---|
| Bare-metal host | Supermicro H14, 8× AMD Instinct MI350X |
| Orchestration | Spinifex (EC2-compatible API) |
| Instance type | `g7e.12xlarge` — 2× MI350X via PCIe passthrough |
| Disk | 800 GB (model weights alone are ~400 GB) |
| Inference runtime | vLLM (`rocm/vllm` Docker image) |
| Tensor parallelism | TP=2 — model sharded symmetrically across GPU 0 and GPU 1 |
| Model | `arcee-ai/Trinity-Large-Preview-FP8` |

### Platform

| Component | Specification |
|---|---|
| **Bare-metal host** | Supermicro H14 |
| **Host OS** | Ubuntu 24.04 LTS or Debian 13 (minimum) |
| **Orchestration** | Spinifex — EC2-compatible bare-metal API |
| **Guest OS** | Ubuntu 26.04 LTS |
| **GPUs** | 2× AMD Instinct MI350X (288 GB HBM3e each, 576 GB combined) |
| **GPU passthrough** | PCIe passthrough via vfio-pci |
| **Instance type** | `g7e.12xlarge` |
| **Container runtime** | Docker (with ROCm device access) |
| **Inference runtime** | vLLM (`rocm/vllm` image), tensor parallelism TP=2 |
| **Block storage** | Viperblock — EBS-compatible, 800 GB |
| **Model** | `arcee-ai/Trinity-Large-Preview-FP8` (400B MoE, ~13B active/token) |

Spinifex runs on the bare-metal host and presents an EC2-compatible API endpoint. Launching a `g7e.12xlarge` atomically claims two MI350Xs from the host's GPU pool and binds them to the guest VM via PCIe passthrough — the guest OS communicates with the hardware directly, with no software virtualisation layer in the data path. When the instance terminates, those two GPUs are immediately returned to the pool.

GPU passthrough requires a host kernel and OS that supports vfio-pci. Ubuntu 24.04 LTS (kernel 6.8+) and Debian 13 (kernel 6.12+) are the tested minimum baselines for the host. Guest VMs run Ubuntu 26.04 LTS, which ships with the ROCm-compatible kernel and userspace expected by the AMD GPU AMI.

## Prerequisites

- Supermicro H14 with at least 2× AMD Instinct MI350X installed
- Host OS: **Ubuntu 24.04 LTS** (kernel 6.8+) or **Debian 13** (kernel 6.12+) — minimum for vfio-pci support
- Spinifex installed and all services running (`systemctl status spinifex.target`)
- GPU passthrough configured — `spx admin gpu setup` (reboot required) then `spx admin gpu enable`
- AMD GPU AMI registered (`ami-ubuntu-amd-gpu`) — Ubuntu 26.04 LTS with ROCm-compatible kernel
- AWS CLI configured with `AWS_PROFILE=spinifex` pointing at the Spinifex endpoint
- SSH key pair imported, VPC and security group created (see [Launching Instances](https://docs.mulgadc.com/docs/launching-instances))
- Docker installed in the guest VM (included in the AMD GPU AMI)
- HuggingFace account with access to `arcee-ai/Trinity-Large-Preview-FP8`

GPU passthrough must be configured before launching GPU instances:

```bash
sudo spx admin gpu setup   # binds GPUs to vfio-pci — requires reboot
# ... reboot ...
sudo spx admin gpu enable  # confirms passthrough and makes GPU pool available
```

## Instructions

### 1. Provision the instance

Spinifex exposes a standard EC2 API on the bare-metal host. The only difference from a real AWS workflow is `AWS_PROFILE=spinifex`.

```bash
export AWS_PROFILE=spinifex

# Check available GPU instance types
aws ec2 describe-instance-types \
    --query 'InstanceTypes[?GpuInfo].[InstanceType,GpuInfo.Gpus[0].Count,GpuInfo.Gpus[0].Name]' \
    --output table

# Launch a g7e.12xlarge with 800 GB disk
# Spinifex atomically claims 2× MI350X via PCIe passthrough for this instance type
INSTANCE_ID=$(aws ec2 run-instances \
    --image-id ami-ubuntu-amd-gpu \
    --instance-type g7e.12xlarge \
    --key-name spinifex-key \
    --subnet-id <subnet-id> \
    --security-group-ids <sg-id> \
    --block-device-mappings 'DeviceName=/dev/sda1,Ebs={VolumeSize=800,DeleteOnTermination=true}' \
    --count 1 \
    --query 'Instances[0].InstanceId' --output text)

echo "Launched: $INSTANCE_ID"

# Wait for running state
aws ec2 wait instance-running --instance-ids "$INSTANCE_ID"

# Get the IP
INSTANCE_IP=$(aws ec2 describe-instances \
    --instance-ids "$INSTANCE_ID" \
    --query 'Reservations[0].Instances[0].PublicIpAddress' --output text)

echo "IP: $INSTANCE_IP"
```

Once SSH is available, confirm both GPUs are visible:

```bash
ssh -i ~/.ssh/spinifex-key ubuntu@$INSTANCE_IP 'lspci | grep -i amd'
```

Or install and run `amd-smi` on the instance:

<img width="1520" height="580" src="/docs-assets/hardware-supermicro-arcee-trinity-mi350x-2gpus.png" alt="amd-smi inside the guest VM confirming two MI350Xs are directly attached, each with a unique UUID">

Two MI350X entries with distinct UUIDs confirm direct PCIe passthrough.

### 2. Pull the vLLM Docker image

```bash
ssh -i ~/.ssh/spinifex-key ubuntu@$INSTANCE_IP

# On the VM:
docker pull rocm/vllm:latest
```

> The `rocm/vllm` image is large (~20 GB). Pull it while the model is downloading in parallel if bandwidth allows.

### 3. Pull the model

Trinity Large Preview FP8 weighs roughly 400 GB, so expect the model download to take some time.

```bash
# On the VM:
pip install 'huggingface-hub[hf_xet]'
hf download 'arcee-ai/Trinity-Large-Preview-FP8' --repo-type model

# Monitor progress — looking for ~400 GB total
du -sh ~/.cache/huggingface/hub/
```

The model lands in `~/.cache/huggingface/hub/`. vLLM picks it up automatically from there on serve.

### 4. Run vLLM

Trinity needs TP=2 to fit across both MI350Xs:

```bash
# On the VM:
docker run --rm -d \
    --name trinity \
    --device /dev/kfd \
    --device /dev/dri \
    --group-add video \
    --ipc host \
    --network host \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    rocm/vllm:latest \
    vllm serve arcee-ai/Trinity-Large-Preview-FP8 \
        --tensor-parallel-size 2 \
        --dtype auto \
        --max-model-len 8192 \
        --gpu-memory-utilization 0.90 \
        --host 0.0.0.0 \
        --port 8003
```

Watch the startup log:

```bash
docker logs -f trinity
```

Loading a 400B FP8 model across two GPUs takes **2–4 minutes**. Wait for:

```
INFO:     Application startup complete.
```

Then verify:

```bash
curl http://localhost:8003/health
curl http://localhost:8003/v1/models | python3 -m json.tool
```

### 5. Test it

A quick sanity check via the OpenAI-compatible API before running benchmarks:

```bash
curl http://localhost:8003/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "arcee-ai/Trinity-Large-Preview-FP8",
        "messages": [{"role": "user", "content": "Explain tensor parallelism in three sentences."}],
        "max_tokens": 256
    }'
```

The first request will be slowest — KV cache is cold. Subsequent requests warm up noticeably.

### 6. Dashboard

We created a quick dashboard for a live view of the model's telemetry — throughput, latency, GPU utilisation, VRAM, and KV cache — streamed via SSE from the vLLM metrics endpoint and the GPU stats sidecar.

<p><video src="https://iso.mulgadc.com/trinity-workload.mp4" controls width="100%" style="border-radius:6px"></video></p>

### 7. Benchmark results

Two captures were run against the live model:

- **Chat session:** A short interactive session — low concurrency, variable cadence, 56 s total. Reflects real conversational use.
- **Scripted benchmark:** 5-minute sustained load driver — rotating batch of 15 long-form technical prompts at 1.5 s cadence. Reflects peak throughput.

| Metric | Chat session | Benchmark (5 min) |
|---|---|---|
| Peak tok/s | 67.1 | 67.0 |
| Avg tok/s (under load) | 18.7 | **52.9** |
| Avg TTFT | 1668 ms | 758 ms |
| Min TTFT | 1502 ms | 538 ms |
| Max TTFT | 1871 ms | 1165 ms |
| GPU 0 util (avg) | 30.2% | 88.8% |
| GPU 1 util (avg) | 30.1% | 88.7% |
| VRAM per GPU | 259.6 GB | 259.6 GB |
| Avg power (both GPUs) | 664 W | 788 W |
| Peak power | 828 W | 839 W |
| GPU temp | 65 °C | 65 °C |

The lower avg utilisation in the chat session reflects idle gaps between conversational turns; peak throughput is identical at ~67 tok/s in both runs, indicating the hardware ceiling rather than a software one.

### Generation throughput

<img width="1540" height="672" src="/docs-assets/hardware-supermicro-arcee-trinity-mi350x-trinity-throughput.png" alt="Trinity generation throughput — scripted benchmark">

Generation and prompt-ingest tok/s over the 5-minute benchmark. The model sustains ~53 tok/s average at continuous load, peaking at 67 tok/s.

### Per-GPU compute utilisation

<img width="1540" height="672" src="/docs-assets/hardware-supermicro-arcee-trinity-mi350x-trinity-per-gpu-util.png" alt="Trinity per-GPU utilisation — scripted benchmark">

GPU 0 and GPU 1 track each other closely throughout the benchmark — TP=2 distributes attention and FFN layers symmetrically across both MI350Xs.

### Per-GPU VRAM

<img width="1540" height="672" src="/docs-assets/hardware-supermicro-arcee-trinity-mi350x-trinity-per-gpu-vram.png" alt="Trinity per-GPU VRAM — scripted benchmark">

Both GPUs hold 259.6 GB — model weights sharded evenly, plus KV cache. 90.2% of HBM3e is occupied at rest, leaving ~26 GB headroom per GPU for active KV cache during generation.

### Per-GPU power

<img width="1540" height="672" src="/docs-assets/hardware-supermicro-arcee-trinity-mi350x-trinity-per-gpu-power.png" alt="Trinity per-GPU power — scripted benchmark">

Power draw during active generation peaks at ~420 W per GPU, for a combined chassis draw well under the MI350X's 750 W TDP per card. Idle between prompts returns to ~300 W.

### Latency

<img width="1540" height="672" src="/docs-assets/hardware-supermicro-arcee-trinity-mi350x-trinity-latency.png" alt="Trinity latency — scripted benchmark">

Cumulative average TTFT and E2E latency over the benchmark. TTFT settles to ~758 ms avg once the KV cache is warm; E2E tracks request length as expected.

### KV cache utilisation

<img width="1540" height="672" src="/docs-assets/hardware-supermicro-arcee-trinity-mi350x-trinity-kv-cache.png" alt="Trinity KV cache utilisation — scripted benchmark">

KV cache occupancy rises as requests pile up during the sustained load phase. The queue depth stays near zero — the model keeps pace with the 1.5 s prompt cadence.

### 8. Teardown

```bash
# Stop the container
ssh -i ~/.ssh/spinifex-key ubuntu@$INSTANCE_IP 'docker stop trinity'

# Terminate the instance — releases the 2× MI350X back to the Spinifex pool
aws ec2 terminate-instances --instance-ids "$INSTANCE_ID"
```

The two MI350Xs are immediately available to Spinifex for a new workload once the instance terminates.

### 9. Conclusion

Trinity Large Preview demonstrates that frontier-class open-weight models are now within reach of a two-GPU bare-metal setup. The MoE architecture is crucial: 400B total parameters with only 13B active per token means the model generates at ~53 tok/s sustained and peaks at 67 tok/s on a pair of MI350Xs — throughput that would be difficult to match with a dense model of comparable quality at that memory footprint.

Combining this with Spinifex as the infrastructure layer allows teams to own the entire workflow. Provisioning a `g7e.12xlarge` instance, pulling a 400 GB model, and serving it through vLLM took a handful of standard `aws ec2` commands and a single `docker run`. The same workflow runs identically on any Spinifex-managed bare-metal node, with the GPUs returned to the pool when instances terminate.

For teams evaluating Trinity for production agentic workloads — tool calling, structured outputs, long-context reasoning — this represents a credible self-hosted deployment path: open weights, Apache 2.0 license, owned hardware and familiar tooling.

---

# Cisco UCS: AWS-compatible cloud at the edge

URL: https://docs.mulgadc.com/hardware/cisco/platform-benchmark
Category: Hardware / Cisco UCS
Updated: 2026-09-14
Tags: cisco, ucs, edge, aws-compatible, xeon-6, intel-amx, predastore, viperblock, raft

Place EC2 instances, EBS volumes, S3 object storage and Kubernetes workloads on a resilient three-node Cisco UCS cluster, using familiar AWS APIs and tooling.

## Overview

This reference architecture turns three Cisco Unified Edge servers into a small,
resilient cloud that can run where data is produced: a factory, retail estate,
branch, lab, sovereign environment, or disconnected site. Spinifex exposes EC2,
EBS, S3 and EKS-compatible APIs on the cluster, so the operational model is
familiar: use the AWS CLI, SDKs, Terraform or OpenTofu, Kubernetes manifests and
CI/CD systems already used for AWS. The change is the endpoint, not the workflow.

**Companion architectures:** [Vision Pipeline on Cisco UCS](https://docs.mulgadc.com/hardware/cisco/vision-pipeline) shows a local AI pipeline that streams inputs from S3; [vLLM Serving on Cisco UCS](https://docs.mulgadc.com/hardware/cisco/llm-serving) shows CPU and GPU model serving on the same cluster.

### Platform

| Component | Specification |
|---|---|
| **Chassis** | 3× Cisco Unified Edge, single-socket each |
| **CPU** | 1× Intel Xeon 6543P-B per node — 32 cores / 64 threads, 800 MHz–3.30 GHz |
| **Cache / NUMA** | L3 128 MB (unified), single NUMA node — no vCPU/memory locality tuning needed |
| **ISA** | AVX-512 (F/DQ/BW/VL/VNNI/BF16/FP16), **Intel AMX** (tile/BF16/INT8), VT-x, VT-d |
| **Memory** | 499 GiB usable per node (≈1.5 TiB aggregate) |
| **GPU** | 1× NVIDIA L4 (23,034 MiB) via VFIO PCIe passthrough on one node; the other two nodes are CPU-only |
| **Storage** | 4× KIOXIA CD8P NVMe (1.92 TB each) per node, raw — Predastore and Viperblock claim these directly |
| **Boot** | Cisco SATA RAID VD (Marvell 88SE9230 controller) |
| **Network** | 2× Intel E825-C 25GbE ports per node — one bond for management/WAN (VLAN 1337), one dedicated to storage/cluster traffic (VLAN 1336, full 25GbE) |
| **Aggregate** | 96 cores / 192 threads, ~1.5 TiB RAM, ~23 TiB raw NVMe, 1× L4 GPU |

Single-NUMA-per-node simplifies placement — no vCPU/memory locality tuning required.
GPU workloads schedule to the L4 node; CPU instances can use all three. Storage and
object data are distributed across the cluster rather than tied to the node that
launched an instance.

### Hardware Chassis

<img width="1024" height="576" src="/docs-assets/hardware-cisco-platform-benchmark-reference-architectures-cisco-ucs-platform-hardware1.png" alt="Cisco UCS x Spinifex">

## Architecture

### AWS services exercised

| Service | Role |
|---|---|
| **EC2** | Guest instances across all three nodes; spread placement groups distribute across physical nodes |
| **EBS (Viperblock)** | Root and data volumes per instance, surviving termination independently when `delete_on_termination = false` |
| **S3 (Predastore)** | Cluster-wide object storage for datasets, model weights, artifacts and backups |
| **VPC (OVN)** | Overlay networking between instances |
| **EKS** | (Optional) Kubernetes control plane with pod rescheduling on node failure |

## Prerequisites

- 3-node Spinifex cluster installed and services healthy on all nodes — follow the [Multi-Node Install](https://docs.mulgadc.com/docs/install-multi-node) guide.
- VPC, subnet, network address pool, SSH key pair, and security group configured — see [VPC Networking](https://docs.mulgadc.com/docs/vpc-networking) and [Launching Instances](https://docs.mulgadc.com/docs/launching-instances).
- GPU passthrough enabled on the L4 node when GPU instances are required — see [GPU Passthrough](https://docs.mulgadc.com/docs/gpu-passthrough):

```bash
sudo spx admin gpu setup   # reboot required after this step
sudo spx admin gpu enable
```

- AWS CLI configured with `AWS_PROFILE=spinifex` pointing at the cluster's EC2-compatible endpoint (`https://<host>:9999`).

## Instructions

### 1. Verify the cluster and its control plane

Before placing workloads, confirm all three nodes are ready and that the storage and
control-plane services are running on each:

```bash
spx get nodes
systemctl is-active spinifex-predastore spinifex-viperblock spinifex-daemon
```

Predastore's Raft group and OVN's northbound/southbound databases are both distributed
across all three nodes — losing one node triggers a clean leader re-election among the
remaining two, and object storage and networking continue serving. Check that all three
nodes are contributing to the Raft group before exposing the cluster to workload traffic.

### 2. Launch instances through the EC2-compatible API

Point an existing AWS profile at the Spinifex endpoint and use the standard EC2
instance lifecycle. Terraform is the recommended path — Spinifex's spread placement
group implementation holds node reservations that are not always released on destroy,
so a fixed group name can cause a subsequent apply to hang. Using a per-deploy unique
name (via `random_id` or equivalent) avoids this:

```hcl
resource "random_id" "suffix" {
  byte_length = 4
}

resource "aws_placement_group" "spread" {
  name     = "edge-spread-${random_id.suffix.hex}"
  strategy = "spread"
}

resource "aws_instance" "worker" {
  count                  = 2
  ami                    = var.ami_id
  instance_type          = "m8i.2xlarge"
  subnet_id              = var.subnet_id
  key_name               = var.key_name
  placement_group        = aws_placement_group.spread.name
  vpc_security_group_ids = [var.security_group_id]
}
```

For a one-off launch via the CLI, use a unique group name each time for the same reason:

```bash
export AWS_PROFILE=spinifex
GROUP="edge-spread-$(openssl rand -hex 4)"

aws ec2 create-placement-group \
  --group-name "$GROUP" \
  --strategy spread

aws ec2 run-instances \
  --image-id "$AMI" --instance-type m8i.2xlarge \
  --count 2 \
  --subnet-id "$SUBNET" --security-group-ids "$SECURITY_GROUP" \
  --key-name "$KEY_NAME" \
  --placement "{\"GroupName\":\"$GROUP\"}"
```

A `g6.2xlarge` instance type routes to the node with GPU passthrough configured;
`m8i` types schedule across all three nodes.

### 3. Attach Viperblock volumes and access Predastore object storage

Create and attach an EBS-compatible data volume to a running instance:

```bash
VOLUME_ID=$(aws ec2 create-volume \
  --availability-zone "$AZ" \
  --size 100 --volume-type gp2 \
  --query VolumeId --output text)

aws ec2 attach-volume \
  --volume-id "$VOLUME_ID" \
  --instance-id "$INSTANCE_ID" \
  --device /dev/sdf
```

Setting `delete_on_termination = false` in Terraform keeps the volume alive across
instance replacement — relaunching a terminated instance re-attaches the same volume
rather than starting from empty storage:

```hcl
resource "aws_ebs_volume" "data" {
  availability_zone = var.az
  size              = 100
  type              = "gp2"
}

resource "aws_volume_attachment" "data" {
  device_name           = "/dev/sdf"
  volume_id             = aws_ebs_volume.data.id
  instance_id           = aws_instance.worker[0].id
  delete_on_termination = false
}
```

Predastore presents an S3-compatible endpoint — standard S3 tooling works without
modification:

```bash
aws s3 mb s3://my-bucket
aws s3 cp ./dataset.tar.gz s3://my-bucket/
aws s3 sync ./results/ s3://my-bucket/results/
```

All three nodes share the same bucket over the cluster's internal storage fabric, so
datasets, model weights and pipeline outputs are accessible to any instance without
mounting a shared filesystem.

### 4. Optional: deploy Kubernetes workloads through the EKS-compatible API

Spinifex also exposes an EKS-compatible control plane — this was not part of the Cisco
exercise documented here, which used EC2, EBS and S3 directly. For workloads where
automatic pod rescheduling matters, Spinifex supports the same `kubectl` workflows as
a standard EKS cluster:

```bash
kubectl get nodes
kubectl apply -f deployment.yaml
kubectl get pods -n default
```

Kubernetes pod rescheduling is the platform's primary answer to node failure for
scheduled workloads. The companion [vision pipeline](https://docs.mulgadc.com/hardware/cisco/vision-pipeline)
and [vLLM serving](https://docs.mulgadc.com/hardware/cisco/llm-serving) architectures use raw EC2 instances;
EKS is the pattern to reach for when automatic rescheduling matters over manual placement.

### 5. Storage benchmark

Predastore and Viperblock — Spinifex's S3 and EBS implementations — represent the
most substantial engineering work in the platform. Implementing reliable erasure-coded
object storage and a distributed block device stack on commodity NVMe, while exposing
the AWS API surface that most workloads depend on, is where most of the hard problems
live. Both subsystems are under active development, and a production-grade environment
like this cluster is where integration-level issues surface before they reach users. Run this check after cluster setup or hardware
changes to confirm both services are performing as expected.

The methodology: one guest per physical node, fio against an ext4 Viperblock data
volume (four jobs, iodepth 32, direct I/O, 512 MiB file, 30-second run), three
sequential repetitions per guest.

| Guest | 16K mixed read/write | 16K random read | 128K mixed read/write | 128K random read |
|---|---:|---:|---:|---:|
| GPU | 75 / 32 MiB/s | 75 MiB/s | 210 / 91 MiB/s | 332 MiB/s |
| CPU1 | 74 / 32 MiB/s | 76 MiB/s | 218 / 94 MiB/s | 333 MiB/s |
| CPU2 | 69 / 30 MiB/s | 107 MiB/s | 226 / 98 MiB/s | 405 MiB/s |

The 128K guest random-read numbers (332–405 MiB/s) sit between the [AWS EBS gp3](https://docs.aws.amazon.com/ebs/latest/userguide/general-purpose.html)
baseline (125 MiB/s) and its provisioned maximum (1,000 MiB/s) — a meaningful result
given that gp3 baseline is what most AWS operators treat as the default floor for
general-purpose block storage. The 16K numbers (75–107 MiB/s) land near that baseline.
The gap to the raw NVMe host performance (the [KIOXIA CD8P](https://americas.kioxia.com/en-us/business/ssd/data-center-ssd/cd8p-r.html) delivers over 4,000 MiB/s
host-side at comparable queue depths) is not architectural — it is the current
single-queue virtio-blk attach path and unthreaded NBD backend, both of which have
a clear source-level fix. These numbers are the baseline to improve against in future
releases, not the ceiling.

128K mixed I/O improves throughput but carries a high p99 latency envelope (roughly
0.47–0.75 s); size queues and write patterns accordingly for latency-sensitive
applications.

Predastore S3 validation used 1 GiB objects with three sequential and three
distributed-concurrent repetitions per host:

| Workload | Write, median | Read, median |
|---|---:|---:|
| One host client | 121.9 MiB/s | 213.2 MiB/s |
| Three-host aggregate | 184.2 MiB/s | 381.7 MiB/s |

The single-host sequential read (213 MiB/s) is a credible result for a single-node
S3-compatible store at low-to-moderate concurrency with 1 GiB objects — throughput at
this scale is typically network- and per-request-latency-bound rather than disk-bound,
and 213 MiB/s uses roughly 17% of the 25GbE storage fabric. The three-node distributed read (381.7 MiB/s, 1.79× single-host) confirms
Predastore is distributing reads across the cluster. The write scaling ratio (184.2 vs
121.9 MiB/s, 1.51×) is narrower than read, so worth tracking as a leading indicator of
backend contention as the cluster grows and write load increases.

All sequential S3 checksum validations passed. Full methodology, raw fio JSON and S3
metrics are in the [benchmark repository](https://github.com/tomnewton-mulga/CISCO-refarch).

## What this architecture unlocks

- **Cloud-to-edge placement:** run the same EC2, volume, S3 and Kubernetes patterns at the site where latency, privacy, bandwidth or sovereignty requires it.
- **Familiar operations:** keep AWS CLI profiles, Terraform/OpenTofu modules, SDKs, CI/CD pipelines and deployment tooling rather than introducing a separate edge-only platform.
- **Local AI with shared data:** place GPU and AMX-capable CPU inference next to the data source while keeping datasets and artifacts available to all instances through S3.
- **A resilient foundation:** distribute control and storage services across three physical Cisco nodes instead of concentrating the site on one server. Kubernetes rescheduling and Raft-based storage both tolerate a single node loss without operator intervention.
- **A path back to cloud:** the same APIs at the edge and on AWS make workload movement, burst strategies and consistent application packaging straightforward.

The raw measurements are evidence that the platform is functioning as designed; the
larger outcome is a portable, resilient operating model for workloads that need cloud
interfaces outside a public-cloud region.

---

# Spinifex Vision Pipeline on Cisco UCS

URL: https://docs.mulgadc.com/hardware/cisco/vision-pipeline
Category: Hardware / Cisco UCS
Updated: 2026-09-14
Tags: cisco, ucs, intel-amx, nvidia-l4, yolo, computer-vision, edge-ai, predastore, terraform

A YOLO11m detection and Qwen2-VL captioning pipeline streaming from a shared Predastore bucket across two EC2 instances, with Intel AMX and NVIDIA L4 compared.

## Overview

Spinifex is an open-source infrastructure platform that brings core AWS services — EC2,
S3, EBS, VPC and EKS — to bare-metal, edge, and on-prem deployments. It exposes a fully
AWS-compatible API, so standard tooling (the `aws` CLI, Terraform) works against a
Spinifex cluster unchanged, with a single profile/endpoint swap.

This guide walks through a computer-vision pipeline built on a 3-node Cisco Unified Edge
cluster running Spinifex: real-time object detection plus a vision-language model
producing plain-English scene descriptions, both streaming from the same shared Predastore
(S3-compatible) bucket as two entirely independent EC2-compatible instances.

**Companion architectures:** [Cisco UCS: AWS-compatible cloud at the edge](https://docs.mulgadc.com/hardware/cisco/platform-benchmark) · [vLLM Serving on Cisco UCS: Intel AMX vs NVIDIA L4](https://docs.mulgadc.com/hardware/cisco/llm-serving)

### Platform

| Component | Specification |
|---|---|
| **Chassis** | 3× Cisco Unified Edge, single-socket each |
| **CPU** | 1× Intel Xeon 6543P-B per node — 32 cores / 64 threads, 800 MHz–3.30 GHz |
| **Cache / NUMA** | L3 128 MB (unified), single NUMA node per host |
| **ISA** | AVX-512 (F/DQ/BW/VL/VNNI/BF16/FP16), **Intel AMX** (tile/BF16/INT8), VT-x, VT-d |
| **Memory** | 499 GiB usable per node (≈1.5 TiB aggregate) |
| **GPU** | 1× NVIDIA L4 (23,034 MiB) via VFIO PCIe passthrough on one node; the other two nodes are CPU-only |
| **Storage** | 4× KIOXIA CD8P NVMe (1.92 TB each) per node, raw — Predastore/Viperblock claim these directly |
| **Boot** | Cisco SATA RAID VD (Marvell 88SE9230 controller) |
| **Network** | 2× Intel E825-C 25GbE ports per node — one bond for management/WAN (VLAN 1337), one for storage/cluster traffic (VLAN 1336, full 25GbE) |
| **Aggregate** | 96 cores / 192 threads, ~1.5 TiB RAM, ~23 TiB raw NVMe, 1× L4 GPU |

### Workloads

| Instance | Role | Engine | Node |
|---|---|---|---|
| `cpu1` (`m8i.2xlarge`) | Real-time YOLO11m object detection | OpenVINO, CPU (AMX/BF16) | either CPU-only node (spread placement group) |
| `gpu` (`g6.2xlarge`) | Qwen2-VL-2B scene captioning | PyTorch/Transformers, NVIDIA L4 | the GPU-equipped node (only one of three has an L4) |
| `cpu2` (`m8i.2xlarge`) | Consolidation/failure-testing capacity | OpenVINO, CPU | either CPU-only node (spread placement group) |

The detection worker reads a frame from Predastore, runs YOLO11m, and posts the
annotated frame; the captioning worker independently reads the same raw frame
from Predastore and produces a one-sentence description with
Qwen2-VL-2B — two instances, two different accelerators, one shared object store.

## Architecture

### AWS services exercised

| Service | Role |
|---|---|
| **EC2** | 3× guest instances (1 GPU, 2 CPU), spread placement group — one instance per physical node |
| **S3 (Predastore)** | Central image bucket; both YOLO and VLM workers read from it independently over HTTPS |
| **EBS (Viperblock)** | Root + dedicated data volume per instance — survives instance termination/replacement independently |
| **VPC (OVN)** | Overlay networking between guests |

## Prerequisites

- 3-node Spinifex cluster installed and services healthy on all nodes — follow the [Multi-Node Install](https://docs.mulgadc.com/docs/install-multi-node) guide.
- VPC, network pool, SSH key pair, and security group configured — see [VPC Networking](https://docs.mulgadc.com/docs/vpc-networking) and [Launching Instances](https://docs.mulgadc.com/docs/launching-instances)
- GPU passthrough configured on the one node with the NVIDIA L4 — see [GPU Passthrough](https://docs.mulgadc.com/docs/gpu-passthrough):

```bash
sudo spx admin gpu setup   # reboot required after this step
sudo spx admin gpu enable
```

- AWS CLI configured with `AWS_PROFILE=spinifex` pointing at the cluster endpoint (`https://<host>:9999`)
- [OpenTofu](https://opentofu.org/) >= 1.6 installed locally

### Clone the workbook

```bash
git clone https://github.com/mulgadc/cisco-ucs-vision-pipeline
cd cisco-ucs-vision-pipeline
```

## Instructions

### 1. Provision the three instances

```bash
cd terraform/
terraform apply
```

One `g6.2xlarge` (GPU) and two `m8i.2xlarge` (CPU) instances in a spread placement
group — Spinifex schedules spread-group members onto distinct physical nodes, so each
instance lands on a different one of the three. The `gpu` instance must land on the one
node with GPU passthrough configured; the spread group guarantees the two CPU instances
land on the remaining two nodes. A dedicated Viperblock data volume per
instance (separate from the root volume, `delete_on_termination = false`) survives
instance termination/replacement independently — relaunching a terminated instance
reattaches the same volume rather than starting from empty storage.

```
gpu_instance  = { id = "i-...", public_ip = "192.168.12.151", type = "g6.2xlarge" }
cpu_instances = [
  { id = "i-...", public_ip = "192.168.12.152", type = "m8i.2xlarge" },
  { id = "i-...", public_ip = "192.168.12.154", type = "m8i.2xlarge" },
]
```

### 2. CPU (Intel AMX) vs GPU precision comparison

Before running the live pipeline, this section characterises each accelerator's
throughput and accuracy on YOLO11m in isolation. Intel AMX (Advanced Matrix Extensions)
is a relatively new instruction set, first introduced with 4th-gen Xeon Scalable
(Sapphire Rapids, 2023) and supported by the Xeon 6543P-B CPUs in these nodes, designed
to accelerate the matrix operations that dominate AI and ML workloads. Running AMX head-to-head against
the NVIDIA L4 directly answers how much a modern, AI-targeted CPU instruction set can
close the gap to GPU without any discrete accelerator.

YOLO11m at 640×640, COCO val2017 (5,000 images), fixed image order, 20-image warm-up
excluded, COCO mAP validated per configuration (not just throughput — an unvalidated
speed number is not a valid quantization comparison). CPUID flags alone don't prove AMX
is executing — the proof is oneDNN's own kernel-selection trace during real inference. On
this hardware, BF16 workloads select `avx10_1_512_amx` (the current naming scheme for
this CPU generation — not the older `brgemm_avx512_amx*` string some documentation
references); FP32 selects `avx512_core` only, with zero AMX selections. The table below
measures what that kernel difference is worth on YOLO11m:

| Precision / engine | Batch | images/s | mAP50-95 |
|---|---:|---:|---:|
| CPU, FP32 (forced, negative control) | 1 | 13.88 | 0.4993 |
| CPU, BF16 (AMX) | 1 | 33.02 | 0.4992 |
| CPU, BF16 (AMX) | 8 | 40.23 | — |
| CPU, INT8 (AMX) | 1 | 28.48 | 0.4946 |
| GPU, TensorRT FP16 | 1 | 62.60 | 0.5067 |
| GPU, TensorRT FP16 | 8 | 118.90 | — |

<img width="1500" height="960" src="/docs-assets/hardware-cisco-vision-pipeline-precision-throughput.png" alt="YOLO11m throughput by precision and engine — CPU FP32/BF16/INT8 versus GPU FP16, batch 1 and 8">

<img width="1200" height="900" src="/docs-assets/hardware-cisco-vision-pipeline-precision-map.png" alt="COCO mAP50-95 by precision, axis zoomed to 0.49-0.502 to show the near-zero accuracy cost of BF16">

**BF16/AMX gives a real 2.4–3.1x throughput gain over true FP32 on this hardware, at
essentially zero accuracy cost** (Δ mAP50-95 = −0.0001). INT8 costs a small but real
−0.9% relative mAP and, counter to the usual expectation, ran *slower* than BF16 — the
default quantization left some layers unquantized, introducing dequant/requant overhead
that BF16's uniform precision avoids entirely.

GPU beats the best CPU path by ~3x at only 22–35% GPU utilisation — headroom-rich, with
host-side pre/post-processing (letterbox, NMS) bundled into the reported figures
alongside GPU inference itself.

### 3. Local volume vs Predastore-streamed

Same engines, now reading frames over S3 and writing detections back, instead of from
the instance's own local Viperblock volume:

| Worker | Local images/s | S3-streamed images/s | Drop |
|---|---:|---:|---:|
| gpu (TensorRT FP16) | 62.60 | 23.54 | −62% |
| cpu1 (OpenVINO BF16) | 33.02 | 13.44 | −59% |
| cpu2 (OpenVINO default) | ~34.79 | 13.44 | −61% |

<img width="1400" height="960" src="/docs-assets/hardware-cisco-vision-pipeline-local-vs-s3.png" alt="Local Viperblock volume versus Predastore-streamed throughput, by worker">

A solo cpu1 S3 run (no concurrent workers) scored 12.53 img/s — essentially identical to
its 3-worker-concurrent number (13.44). **Concurrency from the other two workers cost
cpu1 almost nothing** — the clearest evidence the storage-request latency itself, not
contention between workers, is the limiter. Per-image latency breakdown confirms it:
GPU's S3 round-trip (read+write ≈ 30.6 ms) is ~5x its actual inference cost (5.7 ms) —
each frame is a small object (~163 KB), so this is a per-request-latency-bound access
pattern, not a bandwidth-bound one. A production pipeline would batch/pipeline S3 reads
rather than one GET per frame.

The ~30 ms round-trip at this object size (~163 KB) is dominated by fixed per-request
cost — HTTPS/TLS, sigv4 signing, and the OVN gateway-chassis hop — rather than
bandwidth. S3 latency grows only ~24–40% as workers scale while inference time grows
~11x, confirming the round-trip is a fixed per-request tax, not a contention effect.

### 4. Demo dashboard

```bash
cd demo-dashboard/
./venv/bin/uvicorn server:app --host 0.0.0.0 --port 8090
```

A local FastAPI dashboard renders the pipeline live: the annotated detection feed, the
VLM's scene caption, GPU
utilisation/power, Predastore CPU%, and per-instance Predastore `GetObject` latency.

Of particular note during the video below are the spikes in Predastore CPU %, attributed to its automatic compaction process.

<p><video src="https://iso.mulgadc.com/CISCO-vision.mp4" controls width="100%" style="border-radius:6px"></video></p>

### 5. Teardown

```bash
cd terraform/
terraform destroy
```

All three instances terminate and the NVIDIA L4 is immediately returned to the Spinifex
GPU pool. The dedicated Viperblock data volumes (`delete_on_termination = false`) survive
instance termination but are destroyed explicitly by Terraform here — any results or
model artefacts worth keeping should be copied to Predastore before running this step.

### 6. Conclusion

This pipeline demonstrates a mixed CPU/GPU edge-AI workload — Intel
AMX-accelerated detection and NVIDIA L4-accelerated captioning, running as independent
EC2-compatible instances against a shared S3-compatible object store, on Cisco Unified Edge
hardware managed entirely through standard AWS tooling. AMX delivers a real,
accuracy-neutral 2.4–3x throughput gain on this hardware; the L4 leaves significant
headroom at this workload's current scale; and streaming from a central bucket rather
than a private local volume costs 59–62% throughput, because each frame is a
small, latency- rather than bandwidth-bound request. Teams already operating
AWS infrastructure can point their existing tooling at a Spinifex node with a
single profile swap — EC2 instances, Viperblock EBS volumes, Predastore S3
buckets, and OVN VPC networking all provisioned from the same Terraform
resources that work on AWS.

---

# vLLM Serving on Cisco UCS: Intel AMX vs NVIDIA L4

URL: https://docs.mulgadc.com/hardware/cisco/llm-serving
Category: Hardware / Cisco UCS
Updated: 2026-09-14
Tags: cisco, ucs, intel-amx, nvidia-l4, vllm, llm-serving, qwen

Qwen2.5-7B-Instruct served with vLLM on a Cisco UCS Spinifex cluster, comparing Intel AMX-accelerated CPU serving with NVIDIA L4 GPU serving under concurrency.

## Overview

Spinifex is an open-source infrastructure platform that brings core AWS services — EC2,
S3, EBS, VPC and EKS — to bare-metal, edge, and on-prem deployments, exposing a fully
AWS-compatible API.

This document serves Qwen2.5-7B-Instruct with vLLM on the same 3-node Cisco Unified Edge
cluster, comparing Intel AMX-accelerated CPU serving on an `m8i.2xlarge`-class instance
against NVIDIA L4 GPU serving on a `g6.2xlarge`-class instance — a real,
production-representative LLM-serving stack, with the two accelerators measured independently at matched model, version, and serving configuration.

**Companion architectures:** [Cisco UCS: AWS-compatible cloud at the edge](https://docs.mulgadc.com/hardware/cisco/platform-benchmark) · [Spinifex Vision Pipeline on Cisco UCS](https://docs.mulgadc.com/hardware/cisco/vision-pipeline)

### Platform

| Component | Specification |
|---|---|
| **Chassis** | 3× Cisco Unified Edge, single-socket each |
| **CPU** | 1× Intel Xeon 6543P-B per node — 32 cores / 64 threads, 800 MHz–3.30 GHz |
| **Cache / NUMA** | L3 128 MB (unified), single NUMA node per host |
| **ISA** | AVX-512 (F/DQ/BW/VL/VNNI/BF16/FP16), **Intel AMX** (tile/BF16/INT8), VT-x, VT-d |
| **Memory** | 499 GiB usable per node (≈1.5 TiB aggregate) |
| **GPU** | 1× NVIDIA L4 (23,034 MiB) via VFIO PCIe passthrough on one node; the other two nodes are CPU-only |
| **Storage** | 4× KIOXIA CD8P NVMe (1.92 TB each) per node, raw — Predastore/Viperblock claim these directly |
| **Boot** | Cisco SATA RAID VD (Marvell 88SE9230 controller) |
| **Network** | 2× Intel E825-C 25GbE ports per node — one bond for management/WAN (VLAN 1337), one for storage/cluster traffic (VLAN 1336, full 25GbE) |
| **Aggregate** | 96 cores / 192 threads, ~1.5 TiB RAM, ~23 TiB raw NVMe, 1× L4 GPU |

### Workloads

| Instance | Role | Engine | Precision |
|---|---|---|---|
| `cpu1` (`m8i.2xlarge`) | LLM serving | vLLM (CPU backend), Intel AMX | BF16 |
| `gpu` (`g6.2xlarge`) | LLM serving | vLLM (CUDA backend), NVIDIA L4 | BF16 |

Same model, same vLLM version, same serving stack, deployed independently on each
instance — no request routing or load balancing between them; this measures each
accelerator's serving characteristics in isolation.

## Architecture

### AWS services exercised

| Service | Role |
|---|---|
| **EC2** | One `m8i.2xlarge` instance (CPU serving) and one `g6.2xlarge` instance (GPU serving), on separate physical nodes — the latter on the one node with GPU passthrough configured |

## Prerequisites

- 3-node Spinifex cluster installed and services healthy on all nodes — follow the [Multi-Node Install](https://docs.mulgadc.com/docs/install-multi-node) guide, then verify with `spx get nodes`
- GPU passthrough configured on the one node with the NVIDIA L4 — see [GPU Passthrough](https://docs.mulgadc.com/docs/gpu-passthrough):

```bash
sudo spx admin gpu setup   # reboot required after this step
sudo spx admin gpu enable
```

- One `m8i.2xlarge` (`cpu1`) and one `g6.2xlarge` (`gpu`) instance provisioned and reachable — the `gpu` instance must land on the node with passthrough configured; see [Launching Instances](https://docs.mulgadc.com/docs/launching-instances) for the full provisioning workflow including VPC, key pair, and security group setup
- ~20 GB free disk per instance for model weights
- Python 3.14, `pip install vllm` (GPU instance) or the CPU wheel with `--extra-index-url https://download.pytorch.org/whl/cpu` (CPU instance) — vLLM 0.26.0 / PyTorch 2.11.0 on both, `torch+cu130` on the GPU instance, `torch+cpu` on the CPU instance

## Instructions

### 1. Download the model

```bash
hf download Qwen/Qwen2.5-7B-Instruct \
  --revision a09a35458c702b33eeacc393d103063234e8bc28 \
  --local-dir ./Qwen2.5-7B-Instruct
```

Pinned to a specific commit (Apache-2.0 licensed) for reproducibility, on both
instances identically.

### 2. Launch the server

```bash
# cpu1 — Intel AMX via the CPU backend
VLLM_CPU_KVCACHE_SPACE=4 vllm serve ./Qwen2.5-7B-Instruct \
  --served-model-name Qwen2.5-7B-Instruct \
  --host 0.0.0.0 --port 8000 --dtype bfloat16 --max-model-len 4096

# gpu — NVIDIA L4 via the CUDA backend
VLLM_USE_FLASHINFER_SAMPLER=0 vllm serve ./Qwen2.5-7B-Instruct \
  --served-model-name Qwen2.5-7B-Instruct \
  --host 0.0.0.0 --port 8000 --dtype bfloat16 --max-model-len 4096 \
  --gpu-memory-utilization 0.85
```

Two environment-specific workarounds were needed on this cluster, worth recording for
anyone reproducing this:

- **cpu1**: vLLM's compiled extension (`vllm/_C.abi3.so`) shipped with an executable-stack ELF flag (`GNU_STACK` = RWE) that this guest kernel refuses to `mprotect` (`cannot enable executable stack as shared object requires: Invalid argument`), crashing the server at import time. Fixed with `patchelf --clear-execstack vllm/_C.abi3.so` — a one-time, host-local binary patch, not a vLLM or model issue.
- **gpu**: this instance has no CUDA toolkit (`nvcc`) installed, only the driver/runtime — fine for running pre-built PyTorch/CUDA kernels, but vLLM's default sampler (FlashInfer) JIT-compiles a kernel on first use and fails without `nvcc`. `VLLM_USE_FLASHINFER_SAMPLER=0` falls back to vLLM's native PyTorch sampler, which needs no compilation step.

### 3. Confirm Intel AMX is executing for this workload

CPUID flags alone don't prove AMX is in use. The proof is oneDNN's own kernel-selection
trace during real vLLM chat-completion requests:

```bash
ONEDNN_VERBOSE=1 vllm serve ./Qwen2.5-7B-Instruct ... 2> serve_cpu1.log
# ...send requests...
grep -o 'avx10_1_512_amx[a-z_0-9]*\|avx512_core[a-z_0-9]*' serve_cpu1.log | sort -u
```

| Kernel | Selections during serving |
|---|---:|
| `avx10_1_512_amx` | 452 |
| `avx512_core` (fallback) | 0 |

**Zero fallback** — every matmul in this run went through AMX. A full-BF16 transformer's
matmuls are more uniformly AMX-eligible than a quantized detection model, where
unquantized layers can still fall back to `avx512_core`.

### 4. Benchmark: `vllm bench serve`

vLLM's own benchmark CLI (`vllm bench serve`) is used directly rather than a custom
harness — it already reports the percentiles that matter for serving (TTFT, TPOT, ITL,
E2E latency) against a fixed-length synthetic dataset with controllable concurrency:

```bash
vllm bench serve \
  --backend openai-chat --base-url http://localhost:8000 \
  --endpoint /v1/chat/completions --model Qwen2.5-7B-Instruct \
  --dataset-name random --random-input-len <128|512> --random-output-len 128 \
  --max-concurrency <1|4|8> --num-prompts $((concurrency * 10)) \
  --percentile-metrics ttft,tpot,itl,e2el --ignore-eos --save-result
```

Matrix: 2 input lengths (128, 512 tokens) × 3 concurrency levels (1, 4, 8) × 3 repeats
per cell, 128 output tokens fixed throughout, 36 runs total, **zero failed requests**
across the full matrix. Figures below are the mean of each cell's 3 repeats (median
metric per run). Raw per-run JSON results are available in the
[LLM serving repository](https://github.com/mulgadc/cisco-ucs-llm-serving).

<img width="1350" height="900" src="/docs-assets/hardware-cisco-llm-serving-tpot-vs-concurrency.png" alt="Median time per output token versus concurrency, CPU AMX versus GPU, by input length">

<img width="1350" height="900" src="/docs-assets/hardware-cisco-llm-serving-throughput-vs-concurrency.png" alt="Total token throughput versus concurrency, CPU AMX versus GPU, by input length">

| Engine | Input tok | Concurrency | TTFT p50 (ms) | TPOT p50 (ms) | E2E p50 (ms) | Output tok/s | Total tok/s |
|---|---:|---:|---:|---:|---:|---:|---:|
| CPU (AMX) | 128 | 1 | 409.8 | 184.4 | 23,832 | 5.37 | 11.97 |
| CPU (AMX) | 128 | 4 | 1,432.0 | 192.6 | 25,795 | 19.89 | 44.28 |
| CPU (AMX) | 128 | 8 | 868.0 | 207.1 | 28,510 | 35.94 | 80.02 |
| CPU (AMX) | 512 | 1 | 932.4 | 182.7 | 24,141 | 5.29 | 27.65 |
| CPU (AMX) | 512 | 4 | 5,737.0 | 193.0 | 30,259 | 17.61 | 92.04 |
| CPU (AMX) | 512 | 8 | 6,289.1 | 250.3 | 38,029 | 26.86 | 140.40 |
| GPU (L4) | 128 | 1 | 91.9 | 56.7 | 7,297 | 17.53 | 39.04 |
| GPU (L4) | 128 | 4 | 264.0 | 58.6 | 7,724 | 66.54 | 148.17 |
| GPU (L4) | 128 | 8 | 240.6 | 58.8 | 7,807 | 129.14 | 287.54 |
| GPU (L4) | 512 | 1 | 171.4 | 56.8 | 7,389 | 17.33 | 90.59 |
| GPU (L4) | 512 | 4 | 520.8 | 59.0 | 8,173 | 63.62 | 332.52 |
| GPU (L4) | 512 | 8 | 894.1 | 61.5 | 8,746 | 117.02 | 611.63 |

**Decode speed (TPOT) is ~3.2–3.3x faster on the GPU** at concurrency 1 — a
load-independent ratio that reflects the accelerators themselves. Under concurrency the
GPU holds nearly flat (TPOT +8.3% from C=1 to C=8 at 512-token input) while the CPU
degrades (+37.0% over the same range), widening the total throughput gap from 3.3x solo
to 4.4x at C=8.

**Prefill (TTFT) degrades more sharply than decode on both engines, and worse on CPU**:
at 512-token input, CPU TTFT rises 6.7x (932→6,289 ms) versus the GPU's 5.2x
(171→894 ms). Both effects are expected — concurrent requests share the same finite
compute for both prefill and decode, and the CPU simply has less of it.

### 5. Teardown

```bash
# Stop the vLLM server on each instance
pkill -f 'vllm serve'
```

### 6. Conclusion

vLLM serving Qwen2.5-7B-Instruct on this cluster confirms Intel AMX executing at the
kernel level with zero fallback, and quantifies what that's actually worth against the
NVIDIA L4: a stable ~3.2–3.3x GPU decode-speed advantage at low load, widening to
~4.4x aggregate throughput under concurrent load as CPU prefill and decode both degrade
faster than the GPU's dedicated compute on the same 8-vCPU guest width. The degradation
is inherent finite-compute sharing inside a properly-threaded engine, and sets a concrete
ceiling for how many concurrent LLM-serving requests to expect from a single
`m8i.2xlarge`-class guest.

For a CPU-only path, the absolute numbers hold up better than the GPU-relative gap might
suggest. AMX delivers ~5.4 output tok/s per request at C=1 on a 7B model — practical for
non-interactive or batch workloads — and scales to 26.9 tok/s aggregate at C=8 on a
single instance, with zero fallback from the AMX instruction path confirmed. In edge or
cost-constrained deployments where a GPU isn't available, or where request volume is low
enough that dedicated GPU compute would sit largely idle between bursts, the CPU path is a
deployable option rather than a fallback of last resort. The GPU wins clearly on
latency and throughput at scale; AMX makes the CPU competitive enough that the choice is
meaningful rather than obvious.

Both serving instances were provisioned through Spinifex's EC2-compatible endpoint as standard EC2 instance types — `m8i.2xlarge` for the CPU path, `g6.2xlarge` for the GPU path — with standard `aws ec2` CLI calls. Teams already operating AWS infrastructure can point their existing tooling at a Spinifex node with a single profile swap.

---

# Single-Node Spinifex on OnLogic HX401

URL: https://docs.mulgadc.com/hardware/onlogic/hx401
Category: Hardware / OnLogic
Updated: 2026-09-14
Tags: onlogic, hx401, edge, single-node, viperblock, predastore

EC2, EBS and S3 on a single fanless OnLogic HX401 — the full Spinifex service set on a passive-cooled industrial node for factory and remote edge deployments.

## Overview

The [OnLogic HX401](https://www.onlogic.com/hx401/) is a fanless, passive-cooled industrial edge computer that runs the complete Spinifex service set — EC2, EBS, S3 and VPC networking — in a form factor small enough to panel-mount inside a factory cabinet, attach to a DIN rail in a control room, or deploy unattended at a remote site without active cooling or rack infrastructure. Spinifex exposes EC2, EBS and S3-compatible APIs on the node, so the operational model is unchanged from AWS: use the `aws` CLI, SDKs, Terraform or OpenTofu, and CI/CD pipelines already used for cloud workloads. The change is the endpoint, not the workflow.

Single-node removes the distributed-storage guarantees of a multi-node cluster — Predastore runs RS(1,0) on one disk, so there is no erasure-coded redundancy. For edge deployments where data is generated locally, processed on-node, and shipped upstream on a schedule, or where the box provides compute capacity next to a sensor or control network rather than acting as a primary data store, this is the expected and appropriate configuration.

<p align="center"><img width="782" height="648" src="/docs-assets/hardware-onlogic-hx401-HX401.png" alt="OnLogic HX401 fanless edge node"></p>

### Platform

| Component | Specification |
|---|---|
| **Bare-metal host** | [OnLogic HX401](https://www.onlogic.com/hx401/) — fanless, passive-cooled industrial edge computer |
| **CPU** | Intel i5-1250PE — 12 physical cores / 16 threads |
| **Memory** | 31 GiB |
| **Storage** | Transcend TS256GMTE652T2, 256 GB NVMe (PCIe Gen3 x4) |
| **Networking** | 2× Intel GbE (I210-IT + I219-LM) |
| **Host OS** | Debian 13 |
| **Guest OS** | Ubuntu 26.04 LTS |
| **Instance type** | `c6i.2xlarge` — 8 vCPU, 16 GiB |
| **Form factor** | Fanless, passive-cooled; DIN rail, wall and VESA mountable |

## Architecture

### AWS services exercised

| Service | Role |
|---|---|
| **EC2** | Guest instances on the host NVMe; no spread placement groups on a single node |
| **EBS (Viperblock)** | Local NVMe-backed block volumes per instance, surviving termination independently when `delete_on_termination = false` |
| **S3 (Predastore)** | RS(1,0) single-node object storage — no erasure-coding overhead, no distributed redundancy |
| **VPC (OVN)** | Standalone OVN northbound/southbound databases on the node itself |

## Prerequisites

- OnLogic HX401 (or comparable x86 node) running **Debian 13** or **Ubuntu 26.04 LTS**
- The WAN interface enslaved to a Linux bridge named `br-wan` — the host IP, default route and DHCP must live on the bridge, not the bare NIC
- A reserved range of addresses on your LAN for guest instances — this deployment uses `192.168.157.201-250`
- AWS CLI configured with `AWS_PROFILE=spinifex` pointing at the node's EC2-compatible endpoint (`https://<host>:9999`)

Verify the bridge before installing:

```bash
ip -br link show br-wan
ip route
```

## Instructions

### 1. Install Spinifex and verify the node

Follow the [Single-Node Install](https://docs.mulgadc.com/docs/install) guide, using `--nodes 1` to select the single-node templates — RS(1,0) storage and a single-member NATS cluster.

**Set a static external pool.** `init` auto-detects the external network and selects `source = "dhcp"` when the host holds a DHCP lease. That does not work for guests on every network: the upstream DHCP server answers the host on `br-wan` but does not necessarily offer addresses to guest ENI MAC addresses, causing instance launches to fail with:

```text
PrepareRunInstances: public IP allocation failed — aborting launch
dhcp DORA on br-wan: unable to receive an offer: context deadline exceeded
```

Edit `/etc/spinifex/spinifex.toml` to use a static range reserved for guests on your LAN:

```toml
[[network.external_pools]]
name        = "wan"
source      = "static"
range_start = "192.168.157.201"
range_end   = "192.168.157.250"
gateway     = "192.168.157.1"
prefix_len  = 24
dns_servers = ["1.1.1.1", "8.8.8.8"]
```

See the [VPC Networking](https://docs.mulgadc.com/docs/vpc-networking) guide for full configuration options. Then start the platform and confirm the node is ready:

```bash
sudo systemctl start spinifex.target
sudo spx get nodes
```

```text
NAME  | STATUS | ROLES       | IP              | REGION         | AZ              | VMs | SERVICES
node1 | Ready  | nats:leader | 192.168.157.134 | ap-southeast-2 | ap-southeast-2a | 0   | nats,predastore,viperblock,daemon,awsgw,vpcd,ui
```

All services — Predastore, Viperblock, and the OVN databases — run on the single node. There is no Raft group to converge and no leader election to wait for.

### 2. Launch instances through the EC2-compatible API

Point an existing AWS profile at the Spinifex endpoint and use the standard EC2 instance lifecycle:

```bash
export AWS_PROFILE=spinifex

aws ec2 run-instances \
  --image-id "$AMI" --instance-type c6i.2xlarge \
  --count 1 \
  --subnet-id "$SUBNET" --security-group-ids "$SECURITY_GROUP" \
  --key-name "$KEY_NAME"
```

`c6i.2xlarge` provisions 8 vCPU and 16 GiB from the host's 12 cores and 31 GiB, leaving headroom for the platform services. On a single node there are no spread placement groups — all instances land on the same physical host, which is expected.

### 3. Attach Viperblock volumes and access Predastore object storage

Create and attach a dedicated data volume. The default root volume on the stock Ubuntu AMI is 4 GiB — too small for most workloads. Attach a separate volume and benchmark or write data there:

```bash
VOLUME_ID=$(aws ec2 create-volume \
  --availability-zone ap-southeast-2a \
  --size 30 --volume-type gp2 \
  --query VolumeId --output text)

aws ec2 attach-volume \
  --volume-id "$VOLUME_ID" \
  --instance-id "$INSTANCE_ID" \
  --device /dev/sdf
```

Inside the guest, format and mount:

```bash
sudo mkfs.ext4 -q -L data /dev/vdb
sudo mkdir -p /mnt/data && sudo mount /dev/vdb /mnt/data
```

Setting `delete_on_termination = false` in Terraform keeps the volume alive across instance replacement — relaunching a terminated instance re-attaches the same volume rather than starting from empty storage:

```hcl
resource "aws_ebs_volume" "data" {
  availability_zone = var.az
  size              = 30
  type              = "gp2"
}

resource "aws_volume_attachment" "data" {
  device_name           = "/dev/sdf"
  volume_id             = aws_ebs_volume.data.id
  instance_id           = aws_instance.worker.id
  delete_on_termination = false
}
```

Predastore presents an S3-compatible endpoint — standard S3 tooling works without modification:

```bash
aws s3 mb s3://my-bucket
aws s3 cp ./data.tar.gz s3://my-bucket/
aws s3 sync ./results/ s3://my-bucket/results/
```

### 4. Storage benchmark

Run this after setup or hardware changes to confirm both storage services are performing as expected. The methodology: fio against an ext4-formatted Viperblock data volume (four jobs, iodepth 32, direct I/O, 30-second run), with a bare-metal host baseline run first so the two results diff directly using the same `spx-bench.sh` script.

**Host bare-metal baseline** (platform idle, no guests):

```bash
./spx-bench.sh --tag host-baremetal
```

```text
--- CPU (sysbench, events/sec) ---------------------------------
  threads=1         1456.50 events/s   95th 0.70 ms
  threads=16       12822.85 events/s   95th 2.00 ms

--- Disk (fio, NVMe root) --------------------------------------
  randrw 70/30 4k    read      66,169 IOPS    258 MiB/s
  randrw 70/30 4k    write     28,426 IOPS    111 MiB/s
  randread 4k        read     171,326 IOPS    669 MiB/s
  seqread 1M         read       1,818 MiB/s
  seqwrite 1M        write        308 MiB/s
```

**Guest** (`c6i.2xlarge`, 8 vCPU, attached gp2 volume):

```bash
BENCHDIR=/mnt/bench ./spx-bench.sh --tag guest --require-mount
```

| Test | Host | Guest | Guest/Host |
|---|---:|---:|---:|
| sysbench single-thread | 1,456 events/s | 1,358 events/s | **93.2%** |
| randrw 4k read | 66,169 IOPS | 6,860 IOPS | 10.4% |
| randrw 4k write | 28,426 IOPS | 2,951 IOPS | 10.4% |
| randread 4k | 171,326 IOPS | 9,263 IOPS | 5.4% |
| seqread 1M | 1,818 MiB/s | 484 MiB/s | 26.6% |
| seqwrite 1M | 308 MiB/s | 73 MiB/s | 23.7% |

<p align="center">
<img width="731" height="582" src="/docs-assets/hardware-onlogic-hx401-cpu-comparison.png" alt="sysbench single-thread: host 1,456 events/s vs guest 1,358 events/s (93.2%)">
</p>

Guest CPU lands at **93.2% of bare metal** — roughly 7% single-thread virtualisation overhead, a result that reproduced on separate hardware. Block storage reflects the cost of the current single-queue virtio-blk path and unthreaded NBD backend sharing a single 256 GB NVMe between the host OS, platform services, and guest volumes; these are the same implementation constraints as on the Cisco cluster, expressed more sharply on constrained hardware.

Variance across three runs is notable for two metrics:

| Test | Run 1 | Run 2 | Run 3 | Spread |
|---|---:|---:|---:|---:|
| randrw 4k read | 7,321 | 6,424 | 6,860 | 14% |
| randread 4k | 9,959 | **3,813** | 9,263 | **2.6×** |
| seqread 1M | 518 | 484 | 453 | 14% |
| seqwrite 1M | 83 | 73 | **31** | **2.7×** |
| seqwrite p99 | 7.95 s | 7.28 s | **15.90 s** | 2.2× |

<p align="center">
<img width="1031" height="659" src="/docs-assets/hardware-onlogic-hx401-storage-iops-comparison.png" alt="Random I/O IOPS: host vs guest">
<img width="806" height="658" src="/docs-assets/hardware-onlogic-hx401-storage-seq-comparison.png" alt="Sequential throughput MiB/s: host vs guest">
<img width="1330" height="687" src="/docs-assets/hardware-onlogic-hx401-storage-variance.png" alt="Guest storage variance across 3 runs: randread and seqwrite show bimodal behaviour">
</p>

`randread` and `seqwrite` are bimodal rather than noisy — one run in three falls to roughly a third of its neighbours, likely a Viperblock WAL flush or Predastore compaction cycle intersecting some runs but not others. Treat those two as directionally correct rather than stable point values at this sample size. The `seqwrite` p99 reaching 7–16 seconds is worth noting for write-heavy workloads.

**Predastore S3 validation** (256 MiB objects, three runs, write + read + checksum verify):

```bash
./s3-bench.sh 256
```

| | v1.15.0 | v1.16.0 | Change |
|---|---:|---:|---:|
| Write | 75.5 MiB/s | 146.6 MiB/s | **1.94×** |
| Read | 243.8 MiB/s | 298.1 MiB/s | **1.22×** |

<p align="center"><img width="881" height="658" src="/docs-assets/hardware-onlogic-hx401-s3-version-comparison.png" alt="S3 throughput: v1.15.0 vs v1.16.0 — 1.94× write, 1.22× read"></p>

The v1.16.0 write improvement comes from the single-node storage template switching to RS(1,0): v1.15.0 split every object into two data shards plus a parity shard and wrote all three to the same disk, paying 1.5× write amplification for redundancy a single-node cluster cannot deliver. The read gain is smaller because the read path never had to reconstruct anything. All checksum validations passed on every run.

Raw fio output, S3 results, and the v1.15 / v1.16 annotated comparisons are in the
[benchmark repository](https://github.com/tomnewton-mulga/OnLogic-refarch).

## What this architecture unlocks

- **Cloud tooling at the edge:** run EC2, EBS and S3 workflows on hardware that fits inside a panel, on a DIN rail, or in a small enclosure at a factory, retail site, branch or remote location — without a rack, a UPS, or active cooling.
- **No-moving-parts reliability:** the fanless passive-cooled design tolerates dusty, vibration-prone, or thermally variable environments where server-class hardware is impractical to operate or maintain.
- **Familiar operations:** keep AWS CLI profiles, Terraform or OpenTofu modules, SDKs and deployment pipelines rather than adopting a separate edge-only platform.
- **Local processing with S3 access:** workloads running in EC2-compatible instances read from and write to Predastore over the standard S3 API — sensor data, inference outputs and logs remain on-node until explicitly synced upstream.
- **A path back to cloud:** the same AWS API surface at the edge and on AWS makes workload movement, cloud burst, and consistent application packaging straightforward — the profile swap that points tooling at the HX401 is the same one that points it back at AWS.