Arcee Trinity 400B FP8 on AMD MI350X with Spinifex

Instructions

1. Provision the instance

Spinifex exposes a standard EC2 API on the bare-metal host. The only difference from a real AWS workflow is AWS_PROFILE=spinifex.

export AWS_PROFILE=spinifex

# Check available GPU instance types
aws ec2 describe-instance-types \
    --query 'InstanceTypes[?GpuInfo].[InstanceType,GpuInfo.Gpus[0].Count,GpuInfo.Gpus[0].Name]' \
    --output table

# Launch a g7e.12xlarge with 800 GB disk
# Spinifex atomically claims 2× MI350X via PCIe passthrough for this instance type
INSTANCE_ID=$(aws ec2 run-instances \
    --image-id ami-ubuntu-amd-gpu \
    --instance-type g7e.12xlarge \
    --key-name spinifex-key \
    --subnet-id <subnet-id> \
    --security-group-ids <sg-id> \
    --block-device-mappings 'DeviceName=/dev/sda1,Ebs={VolumeSize=800,DeleteOnTermination=true}' \
    --count 1 \
    --query 'Instances[0].InstanceId' --output text)

echo "Launched: $INSTANCE_ID"

# Wait for running state
aws ec2 wait instance-running --instance-ids "$INSTANCE_ID"

# Get the IP
INSTANCE_IP=$(aws ec2 describe-instances \
    --instance-ids "$INSTANCE_ID" \
    --query 'Reservations[0].Instances[0].PublicIpAddress' --output text)

echo "IP: $INSTANCE_IP"

Once SSH is available, confirm both GPUs are visible:

ssh -i ~/.ssh/spinifex-key ubuntu@$INSTANCE_IP 'lspci | grep -i amd'

Or install and run amd-smi on the instance:

Two MI350X entries with distinct UUIDs confirm direct PCIe passthrough.

2. Pull the vLLM Docker image

ssh -i ~/.ssh/spinifex-key ubuntu@$INSTANCE_IP

# On the VM:
docker pull rocm/vllm:latest

The rocm/vllm image is large (~20 GB). Pull it while the model is downloading in parallel if bandwidth allows.

3. Pull the model

Trinity Large Preview FP8 weighs roughly 400 GB, so expect the model download to take some time.

# On the VM:
pip install 'huggingface-hub[hf_xet]'
hf download 'arcee-ai/Trinity-Large-Preview-FP8' --repo-type model

# Monitor progress — looking for ~400 GB total
du -sh ~/.cache/huggingface/hub/

The model lands in ~/.cache/huggingface/hub/. vLLM picks it up automatically from there on serve.

4. Run vLLM

Trinity needs TP=2 to fit across both MI350Xs:

# On the VM:
docker run --rm -d \
    --name trinity \
    --device /dev/kfd \
    --device /dev/dri \
    --group-add video \
    --ipc host \
    --network host \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    rocm/vllm:latest \
    vllm serve arcee-ai/Trinity-Large-Preview-FP8 \
        --tensor-parallel-size 2 \
        --dtype auto \
        --max-model-len 8192 \
        --gpu-memory-utilization 0.90 \
        --host 0.0.0.0 \
        --port 8003

Watch the startup log:

docker logs -f trinity

Loading a 400B FP8 model across two GPUs takes 2–4 minutes. Wait for:

INFO:     Application startup complete.

Then verify:

curl http://localhost:8003/health
curl http://localhost:8003/v1/models | python3 -m json.tool

5. Test it

A quick sanity check via the OpenAI-compatible API before running benchmarks:

curl http://localhost:8003/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "arcee-ai/Trinity-Large-Preview-FP8",
        "messages": [{"role": "user", "content": "Explain tensor parallelism in three sentences."}],
        "max_tokens": 256
    }'

The first request will be slowest — KV cache is cold. Subsequent requests warm up noticeably.

6. Dashboard

We created a quick dashboard for a live view of the model's telemetry — throughput, latency, GPU utilisation, VRAM, and KV cache — streamed via SSE from the vLLM metrics endpoint and the GPU stats sidecar.

7. Benchmark results

Two captures were run against the live model:

• Chat session: A short interactive session — low concurrency, variable cadence, 56 s total. Reflects real conversational use.
• Scripted benchmark: 5-minute sustained load driver — rotating batch of 15 long-form technical prompts at 1.5 s cadence. Reflects peak throughput.

The lower avg utilisation in the chat session reflects idle gaps between conversational turns; peak throughput is identical at ~67 tok/s in both runs, indicating the hardware ceiling rather than a software one.

Generation throughput

Generation and prompt-ingest tok/s over the 5-minute benchmark. The model sustains ~53 tok/s average at continuous load, peaking at 67 tok/s.

Per-GPU compute utilisation

GPU 0 and GPU 1 track each other closely throughout the benchmark — TP=2 distributes attention and FFN layers symmetrically across both MI350Xs.

Per-GPU VRAM

Both GPUs hold 259.6 GB — model weights sharded evenly, plus KV cache. 90.2% of HBM3e is occupied at rest, leaving ~26 GB headroom per GPU for active KV cache during generation.

Per-GPU power

Power draw during active generation peaks at ~420 W per GPU, for a combined chassis draw well under the MI350X's 750 W TDP per card. Idle between prompts returns to ~300 W.

Latency

Cumulative average TTFT and E2E latency over the benchmark. TTFT settles to ~758 ms avg once the KV cache is warm; E2E tracks request length as expected.

KV cache utilisation

KV cache occupancy rises as requests pile up during the sustained load phase. The queue depth stays near zero — the model keeps pace with the 1.5 s prompt cadence.

8. Teardown

# Stop the container
ssh -i ~/.ssh/spinifex-key ubuntu@$INSTANCE_IP 'docker stop trinity'

# Terminate the instance — releases the 2× MI350X back to the Spinifex pool
aws ec2 terminate-instances --instance-ids "$INSTANCE_ID"

The two MI350Xs are immediately available to Spinifex for a new workload once the instance terminates.

9. Conclusion

Trinity Large Preview demonstrates that frontier-class open-weight models are now within reach of a two-GPU bare-metal setup. The MoE architecture is crucial: 400B total parameters with only 13B active per token means the model generates at ~53 tok/s sustained and peaks at 67 tok/s on a pair of MI350Xs — throughput that would be difficult to match with a dense model of comparable quality at that memory footprint.

Combining this with Spinifex as the infrastructure layer allows teams to own the entire workflow. Provisioning a g7e.12xlarge instance, pulling a 400 GB model, and serving it through vLLM took a handful of standard aws ec2 commands and a single docker run. The same workflow runs identically on any Spinifex-managed bare-metal node, with the GPUs returned to the pool when instances terminate.

For teams evaluating Trinity for production agentic workloads — tool calling, structured outputs, long-context reasoning — this represents a credible self-hosted deployment path: open weights, Apache 2.0 license, owned hardware and familiar tooling.