External Connection Inventory

Overview

Audience: Operators deploying Spinifex into environments subject to CMMC Level 1, or any site that requires a documented inventory of system connections.

Scope: Network connections originated by or terminated at the Spinifex nodes — the Linux hosts running spinifex-daemon, spinifex-awsgw, spinifex-nats, spinifex-predastore, spinifex-viperblock, spinifex-vpcd, spinifex-ui, and the OVN control plane. Guest VM traffic is the workload owner's responsibility and out of scope.

Boundary definition. For the purposes of this document:

• External means outside the Spinifex cluster's trusted network perimeter — the public internet, the operator's corporate network, tenant users of the AWS API, and guest VMs.
• Internal means between Spinifex nodes inside the cluster subnet(s) defined in spinifex.toml.

AC.L1-3.1.20 applies specifically to external connections. Internal cluster connections are documented here as well so operators can build an accurate firewall policy.

CMMC Practices Covered

This guide addresses AC.L1-3.1.20. The related boundary-protection practice SC.L1-3.13.1 is covered by OVN ACL and security-group enforcement in vpcd and is documented separately.

Approach

Spinifex has a small, enumerable set of network surfaces:

1. Inbound listeners — the TCP/UDP ports each node binds. These are the attack surface exposed to whoever can reach the node.
2. Outbound connections — the destinations the node's services reach out to. Today this is a short list: peer Spinifex nodes, OS image mirrors, and install telemetry.
3. Cross-node connections — inter-node control- and data-plane traffic inside the cluster subnet.

The inventory in §1–§2 satisfies objectives [a]/[b]. The Auth / Verification columns throughout satisfy [c]/[d]. §4 and §5 satisfy [e]/[f]. The default install meets [c]–[f] for every listed connection; the operator's remaining work is to record the inventory in the system security plan, apply host/network firewall rules per §4, and audit §5 on a recurring schedule.

1. Inbound Listeners

"Scope" classifies intended reach. It maps onto the node's network planes — see Planes and scope below, because a node with fewer NICs collapses them:

• External — reachable by tenant/operator networks, on the wan plane. Authenticated and TLS-protected.
• Cluster — reachable only from peer Spinifex nodes, on the lan plane. Operator must restrict via host or network firewall.
• Encap — reachable only from peer chassis, on the vpc plane. Carries the tenant overlay and the IPsec that protects it.
• Guest — bound to a per-instance interface and reachable only by that instance's VM.
• Localhost — bound to 127.0.0.1; not reachable off-node.

The listener invariant tests (spinifex/network/invariants and the multinode e2e suite) read this table and fail any Cluster- or Encap-scope port found bound to the wildcard address, unless that row's Purpose or Auth text contains the exact phrase "binds the wildcard by design". That phrase is load-bearing, not incidental wording — a row that merely mentions "wildcard" or "0.0.0.0", negated or not, grants no exception. Adding a new wildcard-bound Cluster/Encap listener means adding that literal phrase to its row, not just describing the behavior in other words.

Planes and scope. A node resolves three planes — wan, lan and vpc — from its interfaces, and collapses vpc ← lan ← wan when a plane has no interface of its own. On a single-NIC node every scope in this table lands on the public address, so Cluster and Encap describe intent, not a guarantee. Verify with ss -tulnp against the node's actual addresses rather than assuming the classification holds.

¹ Formation port lifecycle. 4432 opens during spx admin init / spx admin join while a bootstrap token is outstanding and closes once the cluster is formed (token TTL default 30 min, --token-ttl). The server presents an ephemeral self-signed cert that pre-dates trust bootstrap, so the joining node does not verify the certificate chain for this single dial. Authenticity rests on the operator supplying the leader address out-of-band plus possession of the bearer token. Document in the security plan so reviewers do not flag 4432 as a persistent open port.

Predastore ports. A Predastore cluster is described in /etc/spinifex/predastore/predastore.toml as a set of [[host]] blocks — one per machine, each running a single process — with the nodes pinned to it declared under [[host.node]]. There are three roles: a gate serving the S3 API, a blob node holding erasure-coded object shards, and a meta node in the Raft quorum over global state. Ports must be unique within a host but are not unique across the cluster, so every machine uses the same three ports — 8443, 6660 and 7660. It is three fixed ports per machine, not a range, and adding machines does not widen it.

Nodes on the same host talk over an in-process pipe and bind no socket at all, so a single-node install opens only 8443; 6660 and 7660 appear only once a second host exists. The gate is dialled by S3 clients but never by peer nodes, so it binds no QUIC socket of its own — it takes an ephemeral UDP port to dial out with.

Development-only listeners. When dev_networking=true, QEMU opens arbitrary host TCP ports for SSH port-forwarding into guest VMs. Production installs (the /etc/spinifex layout) do not enable this; it must not appear on compliance nodes.

2. Outbound Connections

Spinifex nodes initiate a small, fixed set of outbound connections.

To external destinations:

To peer nodes (cluster-internal): NATS federation (4248), Predastore S3 (8443), OVN NB/SB (6641/6642), northstar DNS forwarding (5300), and the Geneve/IPsec overlay (6081, 500, 4500, ESP) — see §3 for encryption and verification of each. The daemon also polls local NATS monitoring at 127.0.0.1:8222/varz (loopback HTTP). It opens no connection to Predastore for status: the storage topology it reports comes from reading predastore.toml, and no Predastore node serves a status endpoint.

Update checks and metadata. Spinifex does not check for updates and does not consume a cloud metadata service (169.254.169.254 is served by the cluster to guest VMs). Node software updates come from the operator's OS package channel. The install-telemetry endpoint above is the only vendor-operated destination contacted by a node; closed-egress deployments should disable it and record the opt-out in the security plan.

Air-gapped deployments. The image URLs above are the only destinations needed for the standard image catalogue. Mirror them locally and use spx admin images import --file with pre-staged files. Telemetry must also be disabled. See Air-Gapped Install.

3. Cross-Node (Internal) Connections

Control-plane and data-plane traffic between Spinifex nodes, for completeness and firewall planning:

Nodes must sit on a network segment that is not routed to tenant/guest VMs or to the internet. The Predastore blob and meta transports and the OVN DBs are cluster-internal and must not be reachable from anywhere else.

4. Limiting Controls

Default external surface is five listeners — 9999 (AWS API), 3000 (UI), 22 (SSH), 8443 (S3) and 53 (DNS) — plus 4432 transiently during bootstrap. Every other listener is cluster- or encap-scoped and the operator must enforce this with a host firewall or an upstream network ACL.

The nodes ship no firewall policy today. Until one is installed, this is the operator's responsibility and the reference below is the starting point.

Read the notes under the ruleset before applying it. A default-deny input policy that omits any of the loopback, conntrack, ime-* or Geneve rules will break instance networking, guest metadata or your own SSH session. Apply it to one node and verify before applying it cluster-wide.

table inet spinifex_filter {
  chain input {
    type filter hook input priority filter; policy drop;

    ct state established,related accept
    ct state invalid drop
    iif lo accept

    # Guest metadata and per-VPC DNS terminate on the host, on per-instance
    # interfaces. Omitting this breaks cloud-init, instance role credentials
    # and all guest DNS.
    iifname "ime-*" accept

    # Path MTU discovery is not optional under a Geneve overlay: dropping
    # destination-unreachable produces silent blackholes on large flows.
    icmp type { echo-request, destination-unreachable, time-exceeded, parameter-problem } accept
    icmpv6 type { echo-request, destination-unreachable, packet-too-big, time-exceeded,
                  parameter-problem, nd-neighbor-solicit, nd-neighbor-advert,
                  nd-router-advert } accept

    # External, from anywhere
    tcp dport { 22, 3000, 8443, 9999 } accept
    tcp dport 53 accept
    udp dport 53 accept

    # Cluster, from peer nodes only. Replace with your nodes' lan-plane
    # addresses — a CIDR does not generalise to nodes on different subnets.
    ip saddr { 10.0.1.1, 10.0.1.2, 10.0.1.3 } tcp dport { 4222, 4248, 4432, 5300, 6641, 6642, 6643, 6644, 8660 } accept
    ip saddr { 10.0.1.1, 10.0.1.2, 10.0.1.3 } udp dport { 5300, 6660, 7660 } accept

    # Encap, from peer chassis only. Replace with your nodes' vpc-plane
    # addresses. Geneve arrives as host-local UDP and must be accepted here.
    ip saddr { 10.0.2.1, 10.0.2.2, 10.0.2.3 } udp dport { 6081, 500, 4500 } accept
    ip saddr { 10.0.2.1, 10.0.2.2, 10.0.2.3 } meta l4proto esp accept
  }
}

Notes that make the difference between this working and locking you out:

• Use a dedicated table and never flush the others. vpcd writes MASQUERADE and per-EIP FORWARD rules into the ip filter and ip nat tables, and only reinstalls them when the service starts. iptables -F, nft flush ruleset, ufw enable and firewalld all destroy them silently, and instance networking stays broken until the next restart.
• Do not add a forward hook. nftables evaluates every table registered on a hook and any drop is final, so a default-deny forward chain here would override vpcd's accepts in the other table and break every routed-NAT instance and every EIP. Filtering forwarded guest traffic is the security group's job, not the host firewall's.
• output is deliberately untouched. The metadata service's reply path egresses through per-instance policy routing; filtering output breaks it in ways that are hard to attribute.
• On a single-NIC node the cluster and encap sets are the node's public addresses, because the planes collapse. The rules are still correct, but they are no longer a boundary — an upstream ACL is the only real control there.

Port 4432 must be closed outside the bootstrap window; spx admin join opens it transiently. Outbound egress can be limited to the image-catalogue hostnames in §2 plus the operator's OS package repositories; on air-gapped nodes, block all outbound HTTPS and use spx admin images import --file.

5. Configuration Surface

Every listener and outbound destination is controlled by one of these files. Changes require a service restart.

6. Operator Checklist

• Inventory recorded in the system security plan — inbound (§1), outbound (§2), cross-node (§3) — matches what is observed on the node (ss -tlnp, ss -unlp).
• Host firewall enforces the scope split in §4: external surface limited to 9999, 3000, 22, 8443 and 53 (and 4432 only during bootstrap).
• Node planes verified: ss -tulnp shows cluster-scope listeners on the lan address and encap-scope listeners reachable only from peer chassis. On a single-NIC node, record that the planes are collapsed and that an upstream ACL is the only boundary.
• Cluster subnet is isolated from tenant guest VM networks and from the public internet.
• Formation port 4432 is closed on nodes not actively running a bootstrap token.
• Outbound HTTPS restricted to the §2 image-catalogue hosts, or replaced with air-gapped import.
• Install telemetry (install.mulgadc.com) is either permitted and recorded in the security plan, or disabled via SPX_NO_TELEMETRY=1 / --no-telemetry.
• OVN NB/SB client and RAFT ports (6641–6644) exposure limited to the cluster subnet pending the L2 TLS work.
• SSH (22) configured to operator-managed keys only; password auth disabled in sshd_config.
• Periodic review (at least annually, and after any topology change) confirms this inventory still matches the deployed configuration.