Host Firewall

Instructions

What the Policy Allows

The public plane is open to any source, IPv4 only — an accept with no source scope would otherwise also serve on any global IPv6 address the node picks up from an upstream router advertisement:

SSH is public too, but scoped by a variable you own — see Narrowing SSH access.

The cluster plane is peer-scoped. On a single-NIC node the planes collapse onto one address, which is why these rules match peer addresses rather than an interface or a CIDR:

Two of those are peer-scoped for reasons worth knowing. Outside a formation window, 4432 is the daemon cluster manager, whose routes report node topology, service inventory and running instances to anyone who asks. And 8660 is Predastore's unauthenticated /healthz and /readyz, which name the peers a gate cannot reach.

The formation port needs nothing from you. spx admin init opens 4432 to any source for the length of the formation window and closes it again afterwards, because a node dialling in to join is not a peer yet. It is one slot rather than a list, so it cannot be grown into a general-purpose hole. The handshake behind it is TLS 1.3 with a bearer token.

Beyond those groups the policy accepts what the platform cannot work without: established and related connections, loopback, the ICMP types PMTUD needs under a Geneve overlay, broadcast DHCP replies for the uplink and external-pool leases, and guest traffic to the metadata and VPC DNS endpoints on 169.254.169.254 and 169.254.169.253. The guest rules are scoped to those two addresses and their ports rather than accepting everything arriving on a guest port, which would expose every wildcard-bound host service to any tenant.

Cluster Membership

The daemon resolves who the node's peers are from the cluster it belongs to, writes them into peers.nft, and reloads the policy. A node that has not resolved membership yet is left reachable rather than cut off from a cluster it cannot name: the apply helper fails without touching the ruleset when the peer file is missing.

One consequence matters in practice. A node only recognises the members of the cluster it currently belongs to. During formation the nodes do not yet know each other, so internal traffic between them is dropped — which is why forming or expanding a cluster starts by turning the policy off.

Is It On?

The binary installer defaults to off deliberately: it runs on servers that already have an operating system and services on them, and switching on a default-deny policy uninvited could cut off something Spinifex knows nothing about. The ISO is an appliance on hardware we define, so there the policy is the product.

For production, turn it on — either at install time:

curl -fsSL https://install.mulgadc.com | bash -s -- --firewall=on

or afterwards, by setting it in /etc/spinifex/spinifex.toml and restarting the daemon:

[network]
firewall_enabled = true

Before you do, check what else the machine is serving. Anything listening on a port outside the public group above stops accepting new connections.

Narrowing SSH Access

SSH is accepted from every source by default. That is deliberate — arming the policy should not lock an operator out of the node they are arming it from — but it is the first thing to change on a production cluster.

The sources live in /etc/spinifex/firewall/custom.nft, which the installer creates once and never rewrites, so an edit here survives upgrades, re-installs and node resets:

define trusted_ssh_peers = { 10.0.0.0/8, 192.168.1.0/24 }

Check the change before applying it, and check the policy file rather than this one — on its own custom.nft parses as a bare define and proves nothing, because the rule that uses the variable lives elsewhere:

sudo nft -c -f /etc/spinifex/firewall/spinifex.nft
sudo nft -f /etc/spinifex/firewall/spinifex.nft

Never leave the variable empty. An undefined or empty set fails the whole ruleset, not just its own rule.

The ports come from sshd's own configuration, re-read on every setup.sh run, so moving sshd to a non-standard port is picked up by an upgrade rather than needing an edit here. IPv4 only: sshd reached over IPv6 is not matched by this rule.

Turning It Off and On Around Cluster Changes

Turn the firewall off while you form or expand a cluster, and on again once it is up:

# Off — before forming or expanding a cluster. Run on every node.
sudo /usr/local/lib/spinifex/spinifex-firewall-apply disable

# On — once the cluster is formed and verified. Run on every node.
sudo systemctl restart spinifex-daemon

Restarting the daemon is what re-arms it: the node rebuilds its peer list from the cluster it is now part of, reloads the rules, and re-enables the boot-time unit so the policy survives a reboot. It also happens on its own within five minutes if you would rather wait.

disable removes Spinifex's own table and the peer file and leaves every other nftables table alone. It is idempotent, and it is the recovery path — it runs even on a node with no policy installed, so it cannot leave you with a node you can neither arm nor disarm.

The Files

Everything lives under /etc/spinifex/firewall/. Which files are safe to edit follows from who writes them:

The daemon cannot write /etc itself, and a blanket nft grant would be root-equivalent, so changes go through /usr/local/lib/spinifex/spinifex-firewall-apply — a root helper that accepts a fixed set of verbs and re-validates every address handed to it.

Checking It

sudo nft list table inet spinifex_filter

The peer list is an nft variable, expanded when the rules load, so it does not appear under a name of its own. Look instead at the ip saddr { ... } addresses on the cluster-plane rules — the ones accepting 4222, 6641, 6642 and the rest.

Every node's addresses should be there. On a multi-NIC node that means its WAN, LAN and VPC addresses, so expect several entries per node. A missing node means its cluster traffic is being dropped.

Dropped packets are logged, rate-limited to five a minute, so this tells you whether a connection problem is the firewall or something else:

sudo journalctl -k | grep 'spinifex-fw drop'

An empty log with a connection still failing means the policy is not the cause. A node with no table loaded at all is not armed:

systemctl is-enabled spinifex-firewall.service