# Working with Spinifex

These are the official instructions from Mulga for using **Spinifex** from a coding agent.

Spinifex is an open-source infrastructure platform that runs the core AWS services on
bare-metal, edge and on-premise hardware: EC2, VPC, EBS, S3, IAM, STS, IMDS, ECS, EKS, ECR,
ELBv2, ACM and RDS. It speaks the real AWS APIs with SigV4 authentication, so you drive it
with the ordinary `aws` CLI, the AWS SDKs, and Terraform or OpenTofu pointed at a local
endpoint. Most existing AWS workloads and IaC move across with only an endpoint change.

**Do not install Spinifex unless the user explicitly asks you to.** Installation writes to
system paths as root and reconfigures host networking. Start by finding out what is already
present, and work with that.

---

## Step 1 — Detect what is already there

Run these and read the results before doing anything else. Do not treat a failure as fatal;
each one narrows down the situation.

```bash
# Is the CLI installed, and which version?
command -v spx && spx version

# Is a cluster configured on this host?
ls /etc/spinifex/ 2>/dev/null
systemctl list-units 'spinifex*' --all --no-pager 2>/dev/null

# Is there a Spinifex AWS profile?
grep -l 'spinifex' ~/.aws/credentials ~/.aws/config 2>/dev/null
aws configure list-profiles 2>/dev/null | grep -i spinifex
```

Interpret it like this:

- **`spx` present and a `spinifex` profile exists** — a cluster is set up. Go to step 2.
- **`spx` present but no profile** — Spinifex is installed but not initialised, or you are on
  a client machine rather than the node. Go to step 3.
- **Nothing found** — Spinifex is not installed here. Tell the user, point them at
  https://docs.mulgadc.com/docs/install for a single server or https://docs.mulgadc.com/docs/install-multi-node for a cluster,
  and stop. Do not run the installer for them.

## Step 2 — Confirm you can authenticate

Spinifex profiles live in the standard AWS config files. `~/.aws/credentials` holds an
`AKIA`-prefixed key pair:

```ini
[spinifex]
aws_access_key_id     = AKIA...
aws_secret_access_key = ...
```

and `~/.aws/config` carries the three settings that point the AWS tooling at the local
cluster instead of Amazon:

```ini
[profile spinifex]
region      = ap-southeast-2
endpoint_url = https://localhost:9999
ca_bundle    = /var/lib/spinifex/config/ca.pem
```

Read the real values rather than assuming them — the endpoint host and the CA path vary by
install, and `spx admin init` installs the CA into the system trust store as well:

```bash
aws configure get region       --profile spinifex
aws configure get endpoint_url --profile spinifex
aws configure get ca_bundle    --profile spinifex
```

Then prove the connection end to end. Export the profile once and reuse it:

```bash
export AWS_PROFILE=spinifex
aws sts get-caller-identity      # who am I
aws ec2 describe-instance-types  # can I reach the gateway
```

If those succeed you are connected and can use ordinary AWS commands for the rest of the
session. Common failures:

- **`SSLError` / certificate verify failed** — the CA is not trusted. Use the `ca_bundle` from
  the profile, or re-install it from `/etc/spinifex/ca.pem`.
- **Connection refused on :9999** — the AWS gateway is not running. Check with
  `spx service awsgw status` and the cluster with `spx get nodes`.
- **`InvalidClientTokenId`** — the key in `~/.aws/credentials` no longer matches the cluster.

## Step 3 — Know the two endpoints

Spinifex exposes two ports, and this trips people up:

| Port | Serves | Use |
|---|---|---|
| `9999` | AWS gateway — EC2, IAM, STS, ECS, EKS, ECR, ELBv2, ACM, RDS | the profile's `endpoint_url` |
| `8443` | Predastore, the S3-compatible object store | `aws s3 --endpoint-url https://<host>:8443` |

S3 calls need the storage endpoint explicitly, because the profile's `endpoint_url` points at
the gateway:

```bash
aws s3 ls --endpoint-url https://localhost:8443
```

## Step 4 — Use spx for what the AWS API does not cover

`spx` is the admin CLI for platform operations that have no AWS equivalent. Everything else
should go through the `aws` CLI.

```bash
spx get nodes          # cluster members and readiness
spx get vms            # running instances across all nodes (alias: spx get instances)
spx top nodes          # CPU/memory per node, and instance-type capacity cluster-wide
spx version

spx admin images list          # AMI catalogue
spx admin images import        # import an AMI
spx admin account create --name <team>   # new tenant account; writes a spinifex-<team> profile
spx admin account list
spx admin principal list       # IAM principals
spx admin gpu status           # GPU passthrough state
spx service <name> status      # awsgw | spinifex | predastore | viperblock | nats | vpcd | northstar
```

Multi-tenancy note: `spx admin account create` provisions an isolated account and writes a
`spinifex-<name>` profile into the AWS config files. Switch tenants with
`AWS_PROFILE=spinifex-<name>`.

## Step 5 — Terraform and OpenTofu

The docs use **OpenTofu** (`tofu`), and the AWS provider works unmodified — you override the
per-service endpoints and switch off the checks that assume real AWS:

```hcl
provider "aws" {
  region = var.region

  endpoints {
    ec2                    = var.spinifex_endpoint
    iam                    = var.spinifex_endpoint
    sts                    = var.spinifex_endpoint
    ecs                    = var.spinifex_endpoint
    eks                    = var.spinifex_endpoint
    elasticloadbalancingv2 = var.spinifex_endpoint
  }

  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true
  skip_region_validation      = true
}
```

Set `spinifex_endpoint` to the gateway (`https://<host>:9999`). Declare an endpoint for every
service the configuration touches — an omitted one silently resolves to real AWS. Run with
`AWS_PROFILE=spinifex tofu init && tofu plan`. Complete worked examples, including EKS with
HTTPS ingress and GitOps, are in the Terraform Workbooks listed below.

## Step 6 — Check coverage before promising a feature

Spinifex implements a large subset of each service, not the whole API. Before you rely on a
specific call, verify it exists rather than assuming parity: EC2 implements 119 modelled
operations, IAM 75, ELBv2 33, ECS 31, RDS 26, ECR 19, ACM 9, STS 4. If an operation returns
`InvalidAction` or an unimplemented error, it is out of scope — say so plainly and propose a
supported alternative instead of retrying.

## Working rules

- Prefer the `aws` CLI and SDKs over `spx` for anything with an AWS equivalent; that is the
  whole point of the platform.
- Always set `AWS_PROFILE`, or pass `--profile`. The default profile will hit real AWS.
- Remember the `:8443` S3 endpoint; it is the most common mistake.
- Destructive operations — terminating instances, deleting volumes or buckets, `tofu destroy`,
  `spx admin cluster shutdown` — affect real hardware the user owns. Confirm before running.
- Do not install, upgrade or re-initialise Spinifex unless asked.

## Documentation

Fetch these when you need detail. The full text of every document in one file is at
https://docs.mulgadc.com/llms-full.txt, and a linked index is at https://docs.mulgadc.com/llms.txt.

### Installation

- [Single-Node Install](https://docs.mulgadc.com/docs/install): Install Spinifex on one Ubuntu or Debian server with the binary installer and get an AWS-compatible EC2, S3, and VPC stack running on your own hardware.
- [Multi-Node Install](https://docs.mulgadc.com/docs/install-multi-node): Deploy Spinifex across three or more servers to form an availability zone with clustered OVN, replicated object storage, and automatic cluster formation.
- [Source Install](https://docs.mulgadc.com/docs/install-source): Build Spinifex from source on Ubuntu or Debian for development, custom builds, or contributing changes, then install and run the resulting binaries locally.
- [Air-Gapped Install](https://docs.mulgadc.com/docs/install-airgapped): Deploy Spinifex in environments without internet connectivity. Covers using a pre-built release tarball, mirrored APT packages, and locally-staged cloud images.
- [Bootable USB Install](https://docs.mulgadc.com/docs/install-usb): Install Spinifex on bare-metal x86 hardware by flashing the Spinifex ISO to a USB drive, booting the target server from it, and wiping the disk you select.

### Administration

- [Setting Up Your Cluster](https://docs.mulgadc.com/docs/setting-up-your-cluster): Import an AMI, create an SSH key pair and a VPC with a public subnet, then launch your first EC2 instance and connect to it on a fresh Spinifex cluster.
- [Spinifex Admin CLI](https://docs.mulgadc.com/docs/spinifex-admin-cli): Complete reference for spx, the Spinifex admin CLI: initialise a cluster, manage accounts and nodes, drive the VM lifecycle, and start or stop services.
- [Updating Spinifex](https://docs.mulgadc.com/docs/update): Upgrade an existing Spinifex install with the same installer used to deploy it, or take the manual path to review configuration migrations before applying them.

### Compute and Networking

- [VPC Networking](https://docs.mulgadc.com/docs/vpc-networking): How Spinifex implements AWS-compatible VPC networking on bare metal with OVN: public and private subnets, security groups, route tables, and Elastic IPs.
- [Launching Instances](https://docs.mulgadc.com/docs/launching-instances): Launch, manage, and connect to EC2-compatible virtual machines on Spinifex, with cloud-init, SSH key injection, VPC networking, and AWS lifecycle operations.
- [Placement Groups](https://docs.mulgadc.com/docs/placement-groups): Create and manage spread and cluster placement groups to control how Spinifex places EC2 instances across physical hosts for fault isolation or low latency.
- [GPU Passthrough](https://docs.mulgadc.com/docs/gpu-passthrough): Configure VFIO GPU passthrough on a Spinifex node to bind NVIDIA or AMD GPUs to guest VMs and expose GPU-enabled EC2 instance types to your workloads.

### Identity and Access

- [IAM Users, Policies, and Access Keys](https://docs.mulgadc.com/docs/iam-users-and-policies): Create AWS IAM users, issue and rotate access keys, and write and attach JSON policies that control exactly what each user can do in a Spinifex account.
- [IAM Groups for Shared User Permissions](https://docs.mulgadc.com/docs/iam-groups): Organise AWS IAM users into groups, manage membership, and attach managed or inline policies once for the whole team instead of granting them user by user.
- [IAM Roles and Instance Profiles for EC2](https://docs.mulgadc.com/docs/iam-roles-and-instance-profiles): Create AWS IAM roles with trust policies, wrap them in instance profiles, and launch EC2 instances that receive credentials with no static access keys at all.
- [STS Temporary Credentials and AssumeRole](https://docs.mulgadc.com/docs/sts): Use AWS STS to issue temporary credentials: assume an IAM role, request a session token, or federate Kubernetes workloads with OIDC web identity on Spinifex.
- [IMDS: Instance Metadata Service (IMDSv2)](https://docs.mulgadc.com/docs/imds): Query AWS instance metadata, read user data, and fetch short-lived IAM role credentials from inside a guest VM using IMDSv2 session tokens on Spinifex.

### Containers

- [EKS (Managed Kubernetes)](https://docs.mulgadc.com/docs/eks): Provision an AWS-compatible EKS control plane and managed node group on Spinifex, wire up the VPC, IAM, and security groups, then deploy your first workload.
- [ECS (Elastic Container Service)](https://docs.mulgadc.com/docs/ecs): Create an ECS cluster on Spinifex, register a task definition, boot container instances, run tasks, and front a service with an Application Load Balancer.
- [ECR (Container Registry)](https://docs.mulgadc.com/docs/ecr): Store and serve container images from Spinifex's AWS-compatible ECR: create a repository, authenticate Docker, push and pull images, and let EKS workers pull.

### Migration

- [Moving an AWS Workload to Mulga](https://docs.mulgadc.com/docs/moving-aws-workload): Move existing AWS workloads onto Spinifex using compatible APIs, SDKs, and Terraform across EC2, VPC, EBS, S3, IAM, STS, ELBv2, ACM, ECR, ECS, and EKS.
- [Hybrid Sync](https://docs.mulgadc.com/docs/hybrid-sync): Synchronise data bidirectionally between Spinifex and AWS when connectivity allows, so local infrastructure stays usable at intermittently connected sites.

### Terraform Workbooks

- [Bastion with Private Subnet](https://docs.mulgadc.com/docs/bastion-private-subnet): Deploy a VPC with public and private subnets on Spinifex, then use a bastion host as the only route to an isolated EC2 instance that has no internet access.
- [S3-Backed Web App](https://docs.mulgadc.com/docs/s3-webapp): Deploy a Flask file-sharing app on EC2 backed by S3 (Predastore) with Terraform, using an IAM instance profile and short-lived STS credentials from IMDS.
- [Nginx Web Server (Load Balanced)](https://docs.mulgadc.com/docs/nginx-alb): Deploy a VPC with two private EC2 instances running Nginx behind an internet-facing Application Load Balancer on Spinifex, using Terraform or OpenTofu.
- [Nginx Web Server](https://docs.mulgadc.com/docs/nginx-webserver): Provision a VPC, public subnet, internet gateway, route table, security group, and an EC2 instance that installs and starts Nginx from cloud-init user-data.
- [EKS Quickstart](https://docs.mulgadc.com/docs/eks-quickstart): Stand up a minimal managed Kubernetes cluster with Terraform: a VPC, IAM roles, an EKS cluster, a worker node group, an ECR repository, and a demo web app.
- [EKS HTTPS Ingress (LBC + ACM)](https://docs.mulgadc.com/docs/eks-https-ingress): Serve a demo app over HTTPS on EKS using the AWS Load Balancer Controller addon and an ACM certificate, with an internet-facing ALB built from an Ingress.
- [GitOps on EKS (Argo CD + EBS-CSI)](https://docs.mulgadc.com/docs/eks-gitops-argocd): Deliver an app to EKS with GitOps: the Argo CD addon syncs it from git, an EBS-CSI PersistentVolume holds its state, and HTTPS is served via LBC and ACM.
- [ECS Quickstart](https://docs.mulgadc.com/docs/ecs-quickstart): Stand up a full AWS-compatible ECS stack with Terraform: a VPC, IAM roles, a cluster, a task definition, container instances, and a load-balanced service.
- [RDS Quickstart (PostgreSQL)](https://docs.mulgadc.com/docs/rds-quickstart): Stand up a managed PostgreSQL database on Spinifex with Terraform: a VPC, DB subnet group, parameter group, aws_db_instance, and a client VM that runs psql.

### Security and Compliance

- [Flaw Remediation Policy](https://docs.mulgadc.com/docs/flaw-remediation-policy): CVSS-tiered SLAs for identifying, reporting, and correcting software flaws in Spinifex and its direct dependencies, for maintainers and CMMC Level 1 operators.
- [Malware Protection](https://docs.mulgadc.com/docs/malware-protection): Operator guide to host-based malware protection and file integrity monitoring on the Linux hosts running Spinifex services, aligned to CMMC Level 1 needs.
- [Media Sanitization and Disposal](https://docs.mulgadc.com/docs/media-sanitization): Operator guide to sanitizing and disposing of storage media used by Spinifex nodes, covering system disks, Viperblock and Predastore volumes, and key tokens.
- [External Connection Inventory](https://docs.mulgadc.com/docs/network-connections): Operator inventory of every inbound listener and outbound connection on Spinifex nodes, with ports, protocols, and purpose, for documented CMMC Level 1 sites.
- [Physical Security Operator Guide](https://docs.mulgadc.com/docs/physical-security-guide): Operator guide to physical access controls, visitor handling, access logging, and access-device management at sites hosting Spinifex nodes and network gear.

### Coverage

- [AWS API Coverage](https://docs.mulgadc.com/coverage): The AWS API operations Spinifex serves, generated from its gateway dispatch tables and the pinned AWS SDK service models on each build, service by service.
- [EC2 API Coverage](https://docs.mulgadc.com/coverage/ec2): The Amazon EC2 API operations Spinifex implements, covering instances, EBS volumes, VPC networking, tags, security groups and the rest of the compute surface.
- [IAM API Coverage](https://docs.mulgadc.com/coverage/iam): The AWS IAM API operations Spinifex implements, covering users, roles, policies, groups, instance profiles and the access keys that authenticate them.
- [S3 API Coverage](https://docs.mulgadc.com/coverage/s3): The Amazon S3 API operations Predastore serves on the platform's S3 endpoint, covering buckets, objects, multipart uploads and the policies that guard them.
- [EKS API Coverage](https://docs.mulgadc.com/coverage/eks): The Amazon EKS API operations Spinifex implements, covering clusters, nodegroups, add-ons, access entries and the identity providers they authenticate.
- [ECS API Coverage](https://docs.mulgadc.com/coverage/ecs): The Amazon ECS API operations Spinifex implements, covering clusters, services, tasks, container instances and the task definitions they are launched from.
- [ECR API Coverage](https://docs.mulgadc.com/coverage/ecr): The Amazon ECR API operations Spinifex implements, alongside the OCI distribution endpoint that carries the image layers for every repository it serves.
- [ELBv2 API Coverage](https://docs.mulgadc.com/coverage/elbv2): The Elastic Load Balancing v2 API operations Spinifex implements, for both the Application and Network Load Balancers it serves, listeners and target groups.
- [RDS API Coverage](https://docs.mulgadc.com/coverage/rds): The Amazon RDS API operations Spinifex implements, for the managed PostgreSQL and MariaDB engines it offers, covering instances, snapshots and parameters.
- [ACM API Coverage](https://docs.mulgadc.com/coverage/acm): The AWS Certificate Manager API operations Spinifex implements, generated from the gateway dispatch tables and the pinned AWS service model on each build.
- [STS API Coverage](https://docs.mulgadc.com/coverage/sts): The AWS STS API operations Spinifex implements, covering role assumption, session tokens and web identity federation (IRSA), with those it does not offer.

### Hardware — Supermicro

- [Multi-Tenant AI on Supermicro H14 with AMD MI350X](https://docs.mulgadc.com/hardware/supermicro/smci-h14): Provision isolated GPU VMs for simultaneous AI workloads on a Supermicro H14 bare-metal node with AMD MI350X GPUs, using Spinifex's EC2-compatible API.
- [Mixed AI Workloads on a Single H200 Chassis: Guest-Managed MIG](https://docs.mulgadc.com/hardware/supermicro/smci-mig): Install Spinifex from source, configure host-local networking, attach Predastore storage, and run four concurrent AI workloads across guest-managed MIG slices.
- [Spinifex EKS AI Platform on Dual RTX Pro 6000 Baremetal](https://docs.mulgadc.com/hardware/supermicro/rtx-pro-6000): Deploy a GPU-accelerated AI inference platform — an OpenAI-compatible LLM API and a real-time CV stream — on Kubernetes on bare metal with standard AWS tooling.
- [Arcee Trinity 400B FP8 on AMD MI350X with Spinifex](https://docs.mulgadc.com/hardware/supermicro/arcee-trinity-mi350x): Provision, load, and benchmark arcee-ai/Trinity-Large-Preview-FP8 across two AMD MI350X GPUs on a Supermicro H14 node using Spinifex's EC2-compatible API.

### Hardware — Cisco UCS

- [Cisco UCS: AWS-compatible cloud at the edge](https://docs.mulgadc.com/hardware/cisco/platform-benchmark): Place EC2 instances, EBS volumes, S3 object storage and Kubernetes workloads on a resilient three-node Cisco UCS cluster, using familiar AWS APIs and tooling.
- [Spinifex Vision Pipeline on Cisco UCS](https://docs.mulgadc.com/hardware/cisco/vision-pipeline): A YOLO11m detection and Qwen2-VL captioning pipeline streaming from a shared Predastore bucket across two EC2 instances, with Intel AMX and NVIDIA L4 compared.
- [vLLM Serving on Cisco UCS: Intel AMX vs NVIDIA L4](https://docs.mulgadc.com/hardware/cisco/llm-serving): Qwen2.5-7B-Instruct served with vLLM on a Cisco UCS Spinifex cluster, comparing Intel AMX-accelerated CPU serving with NVIDIA L4 GPU serving under concurrency.

### Hardware — OnLogic

- [Single-Node Spinifex on OnLogic HX401](https://docs.mulgadc.com/hardware/onlogic/hx401): EC2, EBS and S3 on a single fanless OnLogic HX401 — the full Spinifex service set on a passive-cooled industrial node for factory and remote edge deployments.

Last updated 2026-09-15. Retrieved from https://docs.mulgadc.com/agent-setup/prompt.md