OCI Architecture

Overview

Why run Spinifex on OCI

Lower cost for the same shape of workload. OCI prices compute and block storage well below the large US clouds for equivalent OCPU and IOPS, and Spinifex turns one large instance into a whole EC2/EBS/S3 surface. A tenant's twenty small instances become twenty QEMU guests on hardware you are billed for once, instead of twenty billable cloud instances.

Egress stops being the line item that decides the architecture. Oracle publishes a large monthly outbound-transfer allowance per tenancy, then charges per GB beyond it at roughly an order of magnitude below the majors. Check the current pricing page rather than a number in a document, but that gap is the reason this integration exists. Traffic _between_ guests never leaves the host, or on a cluster never leaves the VCN: it is Geneve over the private plane, which is not billed as internet egress at all.

The workload stops being AWS-shaped and becomes yours. Your tenants code against Spinifex's API, so one set of Terraform, SDK calls and AMIs runs on OCI today and on bare metal in your own rack whenever you decide to move it. Leaving AWS becomes a deployment decision rather than a rewrite, and the second move costs nothing extra.

You keep the parts of OCI worth keeping. Bare-metal shapes, fault domains, block volumes with a chosen performance tier, and real public addresses out of OCI's own pool. Spinifex registers each guest's address with OCI rather than hiding the cloud underneath it, so the guest is genuinely reachable on the internet instead of sitting behind a shared NAT.

Sizing

Bare metal is the recommendation. BM.Standard.E2.64 or similar gives you the hardware directly, with no nested virtualisation, full NIC performance, and nothing between your guests and the wire. Where bare metal is out of scope, a single VM.Standard.E6.Flex suits testing and development, or three of them production.

How it fits together

Everything in Single-Node Install and Multi-Node Install still applies: formation, storage and OVN behave exactly as they do on hardware you own. The layer underneath is what changes. An OCI VCN is not an Ethernet segment, so the external datapath is wired differently, and public addresses come from OCI's API rather than from a range you write in a config file.

Single node: the whole AWS surface inside one OCI instance

One instance runs the control plane and every tenant guest. Customers reach the AWS APIs on the node's own address, while their instances reach the internet through addresses registered on the node's second VNIC.

Two things in that picture are the whole integration. The guest's private address (10.0.1.4) never leaves the host, because OVN and the host route both hold it. The address OCI delivers to is a secondary private IP registered on VNIC 1, appearing on br-wan, and the customer's public address (150.230.13.131) is a reserved public IP that OCI 1:1-NATs onto it upstream. describe-instances reports the public one while every datapath object holds the private one, which is why the two never appear together on the host.

The third is storage. Every guest disk, every S3 object and the JetStream state all live under /var/lib/spinifex, which is an OCI block volume attached over iSCSI, not the boot disk. Size it for the guests you intend to run. A boot volume that quietly ends up carrying the stack is the most common way an install on OCI fails later rather than immediately.

Three nodes: the same surface, distributed

Each node registers the addresses for its own guests on its own second VNIC, and the allocator runs per node, so nothing has to know another node's OCIDs. When a guest moves, whether through a stop/start that lands elsewhere or a host failure, the node it lands on claims the address's private half from the old VNIC with a single OCI call. The OCID and the public half are untouched, so the customer's address never changes.

The overlay, the object shards and the OVN databases all cross VNIC 0, inside the VCN. Size that plane rather than the external one: guest-to-guest traffic between nodes is Geneve over the VCN, and it is the link every distributed layer shares.

Credentials: an API key, or an instance principal

The allocator needs OCI credentials at runtime. A node with none forms, passes every health check, and then refuses every launch that wants a public address with InsufficientAddressCapacity, naming no cause outside its own journal.

An API key, the documented path

An API key needs nothing from a tenancy admin, which also makes it the only option when the compartment belongs to someone else's tenancy. The cost is key material on disk, with a rotation obligation, on every node:

# On your workstation, not the node.
openssl genrsa -out ~/.oci/oci_api_key.pem 2048
chmod 600 ~/.oci/oci_api_key.pem
openssl rsa -pubout -in ~/.oci/oci_api_key.pem -out ~/.oci/oci_api_key_public.pem
openssl rsa -pubout -outform DER -in ~/.oci/oci_api_key.pem \
  | openssl md5 -c        # this is the fingerprint

Upload the public key to the user under Identity → Users → API Keys, and record the user OCID, the fingerprint and the tenancy OCID.

Each node then needs two files, and a node holding only the key is the silent failure described above.

The profile name must be spinifex to match oci_config_profile in the pool:

[spinifex]
user=ocid1.user.oc1..aaaa...
fingerprint=12:34:56:78:90:ab:cd:ef:12:34:56:78:90:ab:cd:ef
tenancy=ocid1.tenancy.oc1..aaaa...
region=ap-sydney-1
key_file=/etc/spinifex/oci/oci_api_key.pem

spx-oci-config.sh writes both files on every node, and the deploy calls it for you. It runs after formation and before the pool is configured, which is the only window where a node has /etc/spinifex but has not started the allocator, and it reads the credential from your [spinifex] profile, a dedicated key, a set of OCI_SPX_* environment variables, or failing all three the credential Terraform itself used — resolved through the same scripts/oci_env.py as the apply, so the two cannot disagree.

The key reaches the nodes over SSH and goes nowhere else. It is never rendered into user-data or Terraform state, where it would be readable from instance metadata for the life of the instance, and it is carried in the remote shell's stdin rather than its arguments, where ps would expose it to every other tenant on the host. --credential-hook PATH still takes any executable called as hook <ssh-key> <host>..., so a tenancy that keeps credentials in a vault can substitute its own.

An instance principal: no key material, but it needs a tenancy admin

The instance certificate is served at /opc/v2/identity/cert.pem and the Go SDK authenticates as the instance with no key on disk. It must also be _authorised_, by a dynamic group and a policy, and both are tenancy-root resources: oci_identity_dynamic_group is created with compartment_id set to the tenancy OCID. A compartment-scoped user cannot create them, and the attempt fails with 404-NotAuthorizedOrNotFound on CreateDynamicGroup, which reads like a misconfiguration and is not one.

cd scripts/terraform/oci-spx
./setup-identity.sh --dry-run     # always first
./setup-identity.sh               # needs a tenancy-admin OCI profile

Then set instance_principal = "adopt" in your tfvars and pass --instance-principal to the deploy instead of --credential-hook. The dynamic group matches on instance.compartment.id, so it covers every node you ever build in that compartment and nothing needs updating when nodes are replaced.

The IAM policy

Spinifex calls exactly ten operations, all of them in spinifex/providers/cloud/oci/client.go. Grant no more than these:

The matching policy, scoped to the one compartment Spinifex allocates in:

Allow group SpinifexOperators to use    vnics       in compartment <compartment-name>
Allow group SpinifexOperators to manage private-ips in compartment <compartment-name>
Allow group SpinifexOperators to manage public-ips  in compartment <compartment-name>
Allow group SpinifexOperators to use    subnets     in compartment <compartment-name>

instance-principal.tf grants the dynamic group these same four statements, so an instance principal can allocate a public address exactly as an IAM user can. The two credential routes differ in who is authorised, never in what.

To tell the two apart, make one call per permission class and see which is the first to fail. GetVnic tests VNIC_READ, ListPrivateIps tests PRIVATE_IP_READ, and GetSubnet tests SUBNET_READ. If the first two succeed and the third does not, the credential is authentic and in the right compartment, and the policy is missing its subnet statement.

Scope it to a compartment, not the tenancy. manage public-ips at tenancy level lets a compromised node consume the whole regional reserved-public-IP quota.

Do not grant manage vnics or manage instances. Spinifex never creates, attaches or deletes a VNIC, and never touches an instance. It only adds addresses to a VNIC that already exists, which use permits. manage also permits VNIC lifecycle, which is strictly more than the integration can use.

Quotas

Two limits bound how many public addresses a deployment can hand out, and they do not scale the same way:

On a new tenancy, check the compute limits too. A fresh account commonly carries a service limit of zero for bare-metal shapes and a low one for flex VMs, which can leave BM.Standard.E2.64 unavailable until you request it. That refusal arrives at apply time, not at plan time.

A raise is a support request, not a setting: Console → Governance & Administration → Limits, Quotas and Usage → "Request a service limit increase". Cite the exact limit name from oci limits definition list, not a name from documentation.

oci limits definition list --service-name vcn --compartment-id <tenancy-ocid>
oci limits value list       --service-name vcn --compartment-id <tenancy-ocid>
oci limits resource-availability get --service-name vcn \
    --limit-name <name-from-above> --compartment-id <tenancy-ocid>

These calls need a grant the node does not have, and must not be given. Limits and usage are tenancy-level, which cannot be compartment-scoped the way the policy above insists the node's is. Create a second, read-only principal for an operator or a reporting job, and keep it off every node:

Allow group SpinifexReporting to read limits        in tenancy
Allow group SpinifexReporting to read usage-reports in tenancy

The second verb is what Cost Analysis and oci usage-api usage-summary request-summarized-usages read. Adding either to SpinifexOperators would hand every node tenancy-wide read, which is exactly what the node policy refuses.

Firewalls, in all three layers

Three independent things filter traffic to an OCI deployment, and they are easy to mistake for one. Nearly every "my port is closed" report on OCI turns out to be the wrong layer being adjusted.

The OCI security list is deliberately open, and should stay that way

# network.tf, security_list.public
ingress_security_rules { protocol = "all", source = var.vcn_cidr }      # intra-VCN
dynamic "ingress_security_rules" { ... source = each allow-list CIDR }  # protocol = "all"

A guest's public address is a secondary private IP on VNIC 1, in the public subnet, so inbound traffic to a guest crosses this security list. Narrowing node_client_cidr_allow_list to your own office /32 therefore cuts internet ingress to every guest, not just to the node. A tenant who then opens 443 on their own security group finds it does nothing, with no error anywhere naming the cause.

That is also why the rule is protocol = "all" rather than the three service ports. Enumerating ports here would mean a Terraform change for every customer port.

So leave it at 0.0.0.0/0 for any deployment that serves guests, and do the restricting in the host firewall, which is the layer that can tell the node's own listeners apart from a guest's. Narrow the allow list only for a private lab with no public guest services.

The separate intra-VCN rule exists so that narrowing the allow list can never cut Geneve, OVN, NATS or predastore between nodes.

The host firewall is what you harden

Terraform's cloud-init runs open-service-ports.sh, which opens all protocols from the peer CIDR plus node_service_ports (3000 8443 9999) to any source, in the Ubuntu image's iptables. SSH is already open to the world from the image's own INPUT rule, and nothing in that script narrows it.

The Spinifex nft policy is installed but not armed on this path, because the installer arms it only when asked. Arming it is what gives you the proper policy: public plane, peer-scoped cluster plane, and an SSH scope you own.

# On each node. Arms the policy the ISO installer arms by default.
sudo env SETUP_STAGES=firewall /usr/local/share/spinifex/setup.sh --firewall on

Then narrow SSH in the one file setup.sh creates and never overwrites:

# /etc/spinifex/firewall/custom.nft is yours, and upgrades leave it alone.
define trusted_ssh_peers = { 203.0.113.10/32 }

sudo nft -c -f /etc/spinifex/firewall/spinifex.nft    # check before applying
sudo /usr/local/lib/spinifex/spinifex-firewall-apply

What the armed policy allows, and why each port is where it is, is in Host Firewall, along with Narrowing SSH access and the rule that forming or expanding a cluster starts by turning the policy off, because nodes mid-formation are not yet each other's peers.

Two OCI-specific notes on top of that guide:

• Northstar's DNS on 53 is in the armed policy's public plane, but node_service_ports does not include it. Until the nft policy is armed, a node serving a public zone does not answer on 53 from the internet.
• Never flush iptables on an OCI node. The image's InstanceServices chain in OUTPUT is what permits iSCSI to 169.254.2.0/24:3260 and the metadata service, so clearing it takes the data volume and IMDS with it.

What Terraform builds, per node

The resource-by-resource architecture is in scripts/terraform/oci-spx.

The data volume is iSCSI, and that is the detail that bites. OCI presents the volume as an iSCSI target reachable at 169.254.2.2:3260, so attaching it in the API does not put a block device on the host. is_agent_auto_iscsi_login_enabled makes the Oracle Cloud Agent do the login, and it does that _asynchronously_, so the device is not there when cloud-init first runs. The mount script waits up to ten minutes for it, formats it only if blkid reports no filesystem so that a re-run cannot erase a populated volume, and writes an fstab entry by UUID:

UUID=<uuid>  /var/lib/spinifex  ext4  defaults,_netdev,nofail  0 2

_netdev orders the mount after the network, and therefore after iscsid has a session. nofail keeps a missing volume from wedging the boot. Mounting /var/lib/spinifex _itself_ rather than somewhere else and symlinking is deliberate: viperblock, predastore and JetStream all land on the volume with no configuration pointing anywhere unusual.

Every Terraform variable

All of these have defaults that build a working single node, except the first:

Six more exist and are not yours to set: tenancy_ocid, user_ocid, fingerprint, private_key, private_key_path and region. scripts/oci_env.py fills them as TF_VAR_* from the profile in ~/.oci/config on every terraform call the driver makes, so the region you deploy into is the profile's region and the credential never reaches a .tfvars file. Setting them by hand works, and is how a run with no config file at all would have to work, but then two sources disagree about which tenancy this is.

The profile is resolved as --oci-profile, then $OCI_CLI_PROFILE, then the reference tenancy, then DEFAULT. A name passed explicitly and absent from the config is an error that lists the profiles present; only an unasked-for name falls through. CI takes a different route entirely, reading OCI_TENANCY_OCID, OCI_USER_OCID, OCI_FINGERPRINT, OCI_PRIVATE_KEY and OCI_REGION from the environment so a runner writes no key to disk. A partial set there is an error rather than a fallback, because half a credential means a secret failed to reach the job.

The oci CLI

The oci CLI is not on the Ubuntu image, and apt has no python3-oci-cli package. Nothing in the deployment needs it; install it when you want to inspect allocations by hand:

sudo apt-get update && sudo apt-get install -y python3-venv
python3 -m venv ~/.oci-cli-venv
~/.oci-cli-venv/bin/pip install --upgrade pip oci-cli
sudo ln -sf ~/.oci-cli-venv/bin/oci /usr/local/bin/oci
oci --version

What the CLI is and is not for. Spinifex's daemon uses the OCI Go SDK for every allocation, so the CLI is not on that path at all. You need it for bootstrap and for operational inspection. Do not build automation that shells out to it during an allocation: allocations run inside a request budget that a forked Python process will not respect.

---

Deploying without Terraform

Terraform automates every step of a normal install, and that is the point of it. The oci-spx configuration builds the VCN, the subnets, the gateways, the block volumes and both VNICs per node, then renders the cloud-init that installs Spinifex and sets up OVN. The sequence below happens without anyone typing it, in the same order every time, on one node or three.

If you do need to install by hand, whether for an existing OCI tenancy you cannot run Terraform against, an unsupported shape, or a stage you are debugging, the install itself is not OCI-specific. Follow the standard guides:

• Installing Spinifex for a single node.
• Multi-Node Installation for a cluster, which covers spx admin init on the leader and spx admin join on the rest.

Four things about OCI are not in those guides, and each has its own section above:

• The external pool is source = "oci", which needs the API key or instance principal from Credentials and the IAM policy. Omit it and you get the silent failure that section opens with: a healthy node that cannot hand out an address.
• br-wan stays a Linux bridge owned by netplan, linked to OVS br-ext by a veth pair. Pass setup-ovn.sh --wan-bridge=br-wan and it detects this itself. The NIC never becomes an OVS port, so the VNIC keeps its MAC identity. See How it fits together.
• The block volume must be mounted with _netdev, because it arrives over iSCSI and the mount is otherwise attempted before a session exists.
• IMDS needs the 169.254 remap described under Guests cannot reach instance metadata, since the guest's metadata address collides with OCI's own.

---