Single-Node Spinifex on OnLogic HX401

Instructions

1. Install Spinifex and verify the node

Follow the Single-Node Install guide, using --nodes 1 to select the single-node templates — RS(1,0) storage and a single-member NATS cluster.

Set a static external pool. init auto-detects the external network and selects source = "dhcp" when the host holds a DHCP lease. That does not work for guests on every network: the upstream DHCP server answers the host on br-wan but does not necessarily offer addresses to guest ENI MAC addresses, causing instance launches to fail with:

PrepareRunInstances: public IP allocation failed — aborting launch
dhcp DORA on br-wan: unable to receive an offer: context deadline exceeded

Edit /etc/spinifex/spinifex.toml to use a static range reserved for guests on your LAN:

[[network.external_pools]]
name        = "wan"
source      = "static"
range_start = "192.168.157.201"
range_end   = "192.168.157.250"
gateway     = "192.168.157.1"
prefix_len  = 24
dns_servers = ["1.1.1.1", "8.8.8.8"]

See the VPC Networking guide for full configuration options. Then start the platform and confirm the node is ready:

sudo systemctl start spinifex.target
sudo spx get nodes

NAME  | STATUS | ROLES       | IP              | REGION         | AZ              | VMs | SERVICES
node1 | Ready  | nats:leader | 192.168.157.134 | ap-southeast-2 | ap-southeast-2a | 0   | nats,predastore,viperblock,daemon,awsgw,vpcd,ui

All services — Predastore, Viperblock, and the OVN databases — run on the single node. There is no Raft group to converge and no leader election to wait for.

2. Launch instances through the EC2-compatible API

Point an existing AWS profile at the Spinifex endpoint and use the standard EC2 instance lifecycle:

export AWS_PROFILE=spinifex

aws ec2 run-instances \
  --image-id "$AMI" --instance-type c6i.2xlarge \
  --count 1 \
  --subnet-id "$SUBNET" --security-group-ids "$SECURITY_GROUP" \
  --key-name "$KEY_NAME"

c6i.2xlarge provisions 8 vCPU and 16 GiB from the host's 12 cores and 31 GiB, leaving headroom for the platform services. On a single node there are no spread placement groups — all instances land on the same physical host, which is expected.

3. Attach Viperblock volumes and access Predastore object storage

Create and attach a dedicated data volume. The default root volume on the stock Ubuntu AMI is 4 GiB — too small for most workloads. Attach a separate volume and benchmark or write data there:

VOLUME_ID=$(aws ec2 create-volume \
  --availability-zone ap-southeast-2a \
  --size 30 --volume-type gp2 \
  --query VolumeId --output text)

aws ec2 attach-volume \
  --volume-id "$VOLUME_ID" \
  --instance-id "$INSTANCE_ID" \
  --device /dev/sdf

Inside the guest, format and mount:

sudo mkfs.ext4 -q -L data /dev/vdb
sudo mkdir -p /mnt/data && sudo mount /dev/vdb /mnt/data

Setting delete_on_termination = false in Terraform keeps the volume alive across instance replacement — relaunching a terminated instance re-attaches the same volume rather than starting from empty storage:

resource "aws_ebs_volume" "data" {
  availability_zone = var.az
  size              = 30
  type              = "gp2"
}

resource "aws_volume_attachment" "data" {
  device_name           = "/dev/sdf"
  volume_id             = aws_ebs_volume.data.id
  instance_id           = aws_instance.worker.id
  delete_on_termination = false
}

Predastore presents an S3-compatible endpoint — standard S3 tooling works without modification:

aws s3 mb s3://my-bucket
aws s3 cp ./data.tar.gz s3://my-bucket/
aws s3 sync ./results/ s3://my-bucket/results/

4. Storage benchmark

Run this after setup or hardware changes to confirm both storage services are performing as expected. The methodology: fio against an ext4-formatted Viperblock data volume (four jobs, iodepth 32, direct I/O, 30-second run), with a bare-metal host baseline run first so the two results diff directly using the same spx-bench.sh script.

Host bare-metal baseline (platform idle, no guests):

./spx-bench.sh --tag host-baremetal

--- CPU (sysbench, events/sec) ---------------------------------
  threads=1         1456.50 events/s   95th 0.70 ms
  threads=16       12822.85 events/s   95th 2.00 ms

--- Disk (fio, NVMe root) --------------------------------------
  randrw 70/30 4k    read      66,169 IOPS    258 MiB/s
  randrw 70/30 4k    write     28,426 IOPS    111 MiB/s
  randread 4k        read     171,326 IOPS    669 MiB/s
  seqread 1M         read       1,818 MiB/s
  seqwrite 1M        write        308 MiB/s

Guest (c6i.2xlarge, 8 vCPU, attached gp2 volume):

BENCHDIR=/mnt/bench ./spx-bench.sh --tag guest --require-mount

Guest CPU lands at 93.2% of bare metal — roughly 7% single-thread virtualisation overhead, a result that reproduced on separate hardware. Block storage reflects the cost of the current single-queue virtio-blk path and unthreaded NBD backend sharing a single 256 GB NVMe between the host OS, platform services, and guest volumes; these are the same implementation constraints as on the Cisco cluster, expressed more sharply on constrained hardware.

Variance across three runs is notable for two metrics:

randread and seqwrite are bimodal rather than noisy — one run in three falls to roughly a third of its neighbours, likely a Viperblock WAL flush or Predastore compaction cycle intersecting some runs but not others. Treat those two as directionally correct rather than stable point values at this sample size. The seqwrite p99 reaching 7–16 seconds is worth noting for write-heavy workloads.

Predastore S3 validation (256 MiB objects, three runs, write + read + checksum verify):

./s3-bench.sh 256

The v1.16.0 write improvement comes from the single-node storage template switching to RS(1,0): v1.15.0 split every object into two data shards plus a parity shard and wrote all three to the same disk, paying 1.5× write amplification for redundancy a single-node cluster cannot deliver. The read gain is smaller because the read path never had to reconstruct anything. All checksum validations passed on every run.

Raw fio output, S3 results, and the v1.15 / v1.16 annotated comparisons are in the benchmark repository.

What this architecture unlocks

• Cloud tooling at the edge: run EC2, EBS and S3 workflows on hardware that fits inside a panel, on a DIN rail, or in a small enclosure at a factory, retail site, branch or remote location — without a rack, a UPS, or active cooling.
• No-moving-parts reliability: the fanless passive-cooled design tolerates dusty, vibration-prone, or thermally variable environments where server-class hardware is impractical to operate or maintain.
• Familiar operations: keep AWS CLI profiles, Terraform or OpenTofu modules, SDKs and deployment pipelines rather than adopting a separate edge-only platform.
• Local processing with S3 access: workloads running in EC2-compatible instances read from and write to Predastore over the standard S3 API — sensor data, inference outputs and logs remain on-node until explicitly synced upstream.
• A path back to cloud: the same AWS API surface at the edge and on AWS makes workload movement, cloud burst, and consistent application packaging straightforward — the profile swap that points tooling at the HX401 is the same one that points it back at AWS.