vLLM Serving on Cisco UCS: Intel AMX vs NVIDIA L4

Instructions

1. Download the model

hf download Qwen/Qwen2.5-7B-Instruct \
  --revision a09a35458c702b33eeacc393d103063234e8bc28 \
  --local-dir ./Qwen2.5-7B-Instruct

Pinned to a specific commit (Apache-2.0 licensed) for reproducibility, on both instances identically.

2. Launch the server

# cpu1 — Intel AMX via the CPU backend
VLLM_CPU_KVCACHE_SPACE=4 vllm serve ./Qwen2.5-7B-Instruct \
  --served-model-name Qwen2.5-7B-Instruct \
  --host 0.0.0.0 --port 8000 --dtype bfloat16 --max-model-len 4096

# gpu — NVIDIA L4 via the CUDA backend
VLLM_USE_FLASHINFER_SAMPLER=0 vllm serve ./Qwen2.5-7B-Instruct \
  --served-model-name Qwen2.5-7B-Instruct \
  --host 0.0.0.0 --port 8000 --dtype bfloat16 --max-model-len 4096 \
  --gpu-memory-utilization 0.85

Two environment-specific workarounds were needed on this cluster, worth recording for anyone reproducing this:

• cpu1: vLLM's compiled extension (vllm/_C.abi3.so) shipped with an executable-stack

ELF flag (GNU_STACK = RWE) that this guest kernel refuses to mprotect (cannot enable executable stack as shared object requires: Invalid argument), crashing the server at import time. Fixed with patchelf --clear-execstack vllm/_C.abi3.so — a one-time, host-local binary patch, not a vLLM or model issue.

• gpu: this instance has no CUDA toolkit (nvcc) installed, only the driver/runtime —

fine for running pre-built PyTorch/CUDA kernels, but vLLM's default sampler (FlashInfer) JIT-compiles a kernel on first use and fails without nvcc. VLLM_USE_FLASHINFER_SAMPLER=0 falls back to vLLM's native PyTorch sampler, which needs no compilation step.

3. Confirm Intel AMX is executing for this workload

CPUID flags alone don't prove AMX is in use. The proof is oneDNN's own kernel-selection trace during real vLLM chat-completion requests:

ONEDNN_VERBOSE=1 vllm serve ./Qwen2.5-7B-Instruct ... 2> serve_cpu1.log
# ...send requests...
grep -o 'avx10_1_512_amx[a-z_0-9]*\|avx512_core[a-z_0-9]*' serve_cpu1.log | sort -u

Zero fallback — every matmul in this run went through AMX. A full-BF16 transformer's matmuls are more uniformly AMX-eligible than a quantized detection model, where unquantized layers can still fall back to avx512_core.

4. Benchmark: vllm bench serve

vLLM's own benchmark CLI (vllm bench serve) is used directly rather than a custom harness — it already reports the percentiles that matter for serving (TTFT, TPOT, ITL, E2E latency) against a fixed-length synthetic dataset with controllable concurrency:

vllm bench serve \
  --backend openai-chat --base-url http://localhost:8000 \
  --endpoint /v1/chat/completions --model Qwen2.5-7B-Instruct \
  --dataset-name random --random-input-len <128|512> --random-output-len 128 \
  --max-concurrency <1|4|8> --num-prompts $((concurrency * 10)) \
  --percentile-metrics ttft,tpot,itl,e2el --ignore-eos --save-result

Matrix: 2 input lengths (128, 512 tokens) × 3 concurrency levels (1, 4, 8) × 3 repeats per cell, 128 output tokens fixed throughout, 36 runs total, zero failed requests across the full matrix. Figures below are the mean of each cell's 3 repeats (median metric per run). Raw per-run JSON results are available in the LLM serving repository.

Decode speed (TPOT) is ~3.2–3.3x faster on the GPU at concurrency 1 — a load-independent ratio that reflects the accelerators themselves. Under concurrency the GPU holds nearly flat (TPOT +8.3% from C=1 to C=8 at 512-token input) while the CPU degrades (+37.0% over the same range), widening the total throughput gap from 3.3x solo to 4.4x at C=8.

Prefill (TTFT) degrades more sharply than decode on both engines, and worse on CPU: at 512-token input, CPU TTFT rises 6.7x (932→6,289 ms) versus the GPU's 5.2x (171→894 ms). Both effects are expected — concurrent requests share the same finite compute for both prefill and decode, and the CPU simply has less of it.

5. Teardown

# Stop the vLLM server on each instance
pkill -f 'vllm serve'

6. Conclusion

vLLM serving Qwen2.5-7B-Instruct on this cluster confirms Intel AMX executing at the kernel level with zero fallback, and quantifies what that's actually worth against the NVIDIA L4: a stable ~3.2–3.3x GPU decode-speed advantage at low load, widening to ~4.4x aggregate throughput under concurrent load as CPU prefill and decode both degrade faster than the GPU's dedicated compute on the same 8-vCPU guest width. The degradation is inherent finite-compute sharing inside a properly-threaded engine, and sets a concrete ceiling for how many concurrent LLM-serving requests to expect from a single m8i.2xlarge-class guest.

For a CPU-only path, the absolute numbers hold up better than the GPU-relative gap might suggest. AMX delivers ~5.4 output tok/s per request at C=1 on a 7B model — practical for non-interactive or batch workloads — and scales to 26.9 tok/s aggregate at C=8 on a single instance, with zero fallback from the AMX instruction path confirmed. In edge or cost-constrained deployments where a GPU isn't available, or where request volume is low enough that dedicated GPU compute would sit largely idle between bursts, the CPU path is a deployable option rather than a fallback of last resort. The GPU wins clearly on latency and throughput at scale; AMX makes the CPU competitive enough that the choice is meaningful rather than obvious.

Both serving instances were provisioned through Spinifex's EC2-compatible endpoint as standard EC2 instance types — m8i.2xlarge for the CPU path, g6.2xlarge for the GPU path — with standard aws ec2 CLI calls. Teams already operating AWS infrastructure can point their existing tooling at a Spinifex node with a single profile swap.