ROCm with two different GPUs? How hard can it be?

/ 23/04/2026

I spent a couple of days this week trying to get vllm to work with more than one GPU.. not the easy way, but two different architecture GPU’s.

One gfx1100 (Radeon 7900 XTX) and one gfx908 (Radeon Instinct mi100) – that’s one RDNA3 and one CDNA GPU. Yeah, I’m pushing water uphill here.

So now, having decided vllm is much happier with multiples of the same GPU, or at least the same architecture, I’m looking closely at llama.cpp. Just to make it more challenging, I’m also hoping to get the cutting edge, also known as the bleeding edge, working at the same time.

Inspiration came from this discussion on github about testing Turboquant in more detail on gfx1100, so I figured I’d have a go at building my own llama.cpp which has support for both my GPU’s in it.

This is my log of commands – not so much a story as keeping notes for what I’ve tried which worked (if it doesn’t work I’ll try to make it obvious).

Currently running Fedora 44 with ROCm 7.2.2 – latest EL10 build from Redhat installed from ROCm repo.

He’res info on ROCm 7.2.2: https://github.com/ROCm/ROCm/releases

And further info including links to pytorch, tensorflow etc.: https://rocm.docs.amd.com/projects/install-on-linux/en/latest/

Install ROCm on the host (Fed 44)

sudo dnf install https://repo.radeon.com/amdgpu-install/7.2.1/rhel/10/amdgpu-install-7.2.1.70201-1.el10.noarch.rpm
sudo dnf install rocm

Check it all worked:

$ amd-smi

+------------------------------------------------------------------------------+
| AMD-SMI 26.2.2+671d39a71e amdgpu version: Linuxver ROCm version: 7.2.2 |
| VBIOS version: 535818 |
| Platform: Linux Baremetal |
|-------------------------------------+----------------------------------------|
| BDF GPU-Name | Mem-Uti Temp UEC Power-Usage |
| GPU HIP-ID OAM-ID Partition-Mode | GFX-Uti Fan Mem-Usage |
|=====================================+========================================|
| 0000:03:00.0 AMD Instinct MI100 | 0 % 49 °C 0 41/290 W |
| 0 1 N/A N/A | 0 % N/A 6/32752 MB |
|-------------------------------------+----------------------------------------|
| 0000:83:00.0 Radeon RX 7900 XTX | 0 % 46 °C 0 10/303 W |
| 1 0 N/A N/A | 0 % 0.0 % 1699/24560 MB |
+-------------------------------------+----------------------------------------+
+------------------------------------------------------------------------------+
| Processes: |
| GPU PID Process Name GTT_MEM VRAM_MEM MEM_USAGE CU % |
|==============================================================================|
| No running processes found |
+------------------------------------------------------------------------------+

Happy days, OS can see both cards, so a solid start. Also note the packages installed are 7.2.2.

Fetch Docker Image

Next lets fetch a docker image with ROCm/pytorch in it and build llama.cpp in – using the most recent release from AMD as they now moved to Ubuntu 24.04 for LTS, and also upgraded pytorch (which may make this a fools errand, but lets run with it for now). I’m hoping that using the same ROCm from AMD for both the Docker and the host OS should give me at least a smooth path to llama.cpp having a consistent set of new drivers with the latest fixes in.

docker pull rocm/pytorch:rocm7.2.2_ubuntu24.04_py3.12_pytorch_release_2.10.0

And then lets try to Build your own docker image.

export MODEL_PATH='./models'

docker run -it \
      --name=llama_cpp \
      --privileged --network=host \
      --device=/dev/kfd --device=/dev/dri \
      --group-add video --cap-add=SYS_PTRACE \
      --security-opt seccomp=unconfined \
      --ipc=host -v $MODEL_PATH:/data rocm/ \ pytorch:rocm7.2.2_ubuntu24.04_py3.12_pytorch_release_2.10.0

And we’re in.. like Flynn! You should now have a root console inside the docker container. If for some reason you don’t, you can try connecting to it like this:

docker exec -it llama_cpp bash

Ok, now you’re in, lets carry on..

Pull and build llama.cpp

apt-get update && apt-get install -y nano libcurl4-openssl-dev cmake git
mkdir -p /workspace && cd /workspace

Would like to try turboquant but its early days so lets aim for more stable to start with 🙂

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp

And then choose architectures.. gfx908 is mi100, and gfx1100 is R7900 (RDNA3)

export LLAMACPP_ROCM_ARCH=gfx908,gfx1100

And then compile!

HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
cmake -S . -B build \
-DGGML_HIP=ON \
-DAMDGPU_TARGETS=$LLAMACPP_ROCM_ARCH \
-DCMAKE_BUILD_TYPE=Release \
-DLLAMA_CURL=ON && cmake --build build --config Release -j$(nproc)

And once its done, lets try:

./build/bin/test-backend-ops

And yay! Here’s what it shows:

root@fedora:/workspace/llama.cpp# ./build/bin/test-backend-ops
ggml_cuda_init: found 2 ROCm devices (Total VRAM: 57312 MiB):
Device 0: AMD Radeon RX 7900 XTX, gfx1100 (0x1100), VMM: no, Wave Size: 32, VRAM: 24560 MiB
Device 1: AMD Instinct MI100, gfx908:sramecc+:xnack- (0x908), VMM: no, Wave Size: 64, VRAM: 32752 MiB
Testing 3 devices
Backend 1/3: ROCm0
Device description: AMD Radeon RX 7900 XTX
Device memory: 24560 MB (24460 MB free)
ABS(type=f16,ne_a=[128,2,2,2],v=0): OK
ABS(type=f16,ne_a=[5,7,11,13],v=0): OK
SGN(type=f16,ne_a=[128,2,2,2],v=0): OK
etc. etc.

Ok, it sees both cards, and recognises the total VRAM.. I’ll call that a good place to stop for tonight.. almost! Lets just quickly Add in the HF tools

pip install huggingface_hub hf_transfer

Enabling my HF cache

Inside the Docker again

export HF_HOME=/data

And then lets give it a shot

root@fedora:/workspace/llama.cpp# hf cache ls
ID                                    SIZE LAST_ACCESSED LAST_MODIFIED REFS
----------------------------------- ------ ------------- ------------- ----
model/Qwen/Qwen3.5-9B                19.3G 5 days ago    2 weeks ago   main
model/unsloth/Llama-3.2-3B-Instr...   3.4G 1 week ago    1 week ago    main
model/unsloth/Qwen3.5-27B-GGUF       22.5G 2 weeks ago   2 weeks ago   main
model/unsloth/Qwen3.5-9B-GGUF         9.5G 2 weeks ago   2 weeks ago   main

Oh, also this is useful for managing fetching HF models:

https://github.com/bodaay/HuggingFaceModelDownloader

Woopee! Excitement building!

Running llama-something

First try, llama-cli inside the docker to see if I can get it to load a model.

./build/bin/llama-cli --hf-repo Qwen/Qwen3.5-9B --no-mmap --ctx-size 32768
ggml_cuda_init: found 2 ROCm devices (Total VRAM: 57312 MiB):
Device 0: AMD Radeon RX 7900 XTX, gfx1100 (0x1100), VMM: no, Wave Size: 32, VRAM: 24560 MiB
Device 1: AMD Instinct MI100, gfx908:sramecc+:xnack- (0x908), VMM: no, Wave Size: 64, VRAM: 32752 MiB
HTTPS is not supported. Please rebuild with one of:
-DLLAMA_BUILD_BORINGSSL=ON
-DLLAMA_BUILD_LIBRESSL=ON
-DLLAMA_OPENSSL=ON (default, requires OpenSSL dev files installed)
root@fedora:/workspace/llama.cpp#

Ouch.. surprises. So lets do what it says and enable openssl..

apt-get install libssl-dev

And then re-build (which of course only rebuilds the new parts based on the headers..)

HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
cmake -S . -B build \
-DGGML_HIP=ON \
-DAMDGPU_TARGETS=$LLAMACPP_ROCM_ARCH \
-DCMAKE_BUILD_TYPE=Release \
-DLLAMA_CURL=ON 
-DLLAMA_OPENSSL=ON \
&& cmake --build build --config Release -j$(nproc)

And lets go again

./build/bin/llama-cli –hf-repo Qwen/Qwen3.5-9B –no-mmap –ctx-size 32768

…

Very cool, now working – probably not even remotely optimised, but it works! Here’s the result of a couple of prompts end in generation of a limerick..

In Sydney in April the weather is wild,
You're wearing a coat, but you're sweating for a while.
You ask yourself, why?
As the sun starts to fly,
Then you shiver and say 'The koalas are chill'!
[ Prompt: 489.2 t/s | Generation: 58.9 t/s ]

And while its inferencing..

So using both GPU’s.

Checking the temperatures

Ok, so I know 3.6 is released yesterday, but I already downloaded 3.5 and this is just sorting out environment still, so while I download Qwen-3.6-27B, lets get things moving.

root@fedora:/workspace/llama.cpp# ./build/bin/llama-bench --hf-repo unsloth/Qwen3.5-27B-GGUF:Q6_K

First thing I notice is that temps climb.. so for prolonged running / coding, I’ll probably limit power for the mi100 to 200W as I’ll take a small performance hit, but hopefully avoid throttling. Maybe Cooling the mi100 in a desktop case needs to be reconsidered? I might have to try one of the higher-current fans which fit this 3d print.

This command needs to be run in the host, not inside the docker:

sudo rocm-smi -d 0 --setpoweroverdrive 200
============================ ROCm System Management Interface ============================
================================ Set GPU Power OverDrive =================================
GPU[0]		: Successfully set power to: 200W
==========================================================================================
================================== End of ROCm SMI Log ===================================

Also, Q6 only takes 21GB of VRam.. nice! This means I have enough to run 256k context without quantizing KV. Anyway.. back to the bench..

mi100 vs 7900XTX performance

So lets run the Q6 dense 27B model on each device and see who’s hamstringing who..

./build/bin/llama-bench -p 2048 -n 128 -dev ROCm0 --hf-repo unsloth/Qwen3.6-27B-GGUF:Q6_K
ggml_cuda_init: found 2 ROCm devices (Total VRAM: 57312 MiB):
Device 0: AMD Radeon RX 7900 XTX, gfx1100 (0x1100), VMM: no, Wave Size: 32, VRAM: 24560 MiB
Device 1: AMD Instinct MI100, gfx908:sramecc+:xnack- (0x908), VMM: no, Wave Size: 64, VRAM: 32752 MiB
modelsizeparamsbackendngldevtestt/s
qwen35 27B Q6_K20.97 GiB26.90 BROCm99ROCm0pp2048834.41 ± 2.01
qwen35 27B Q6_K20.97 GiB26.90 BROCm99ROCm0tg12827.82 ± 0.04
./build/bin/llama-bench -p 2048 -n 128 -dev ROCm1 --hf-repo unsloth/Qwen3.6-27B-GGUF:Q6_K
ggml_cuda_init: found 2 ROCm devices (Total VRAM: 57312 MiB):
Device 0: AMD Radeon RX 7900 XTX, gfx1100 (0x1100), VMM: no, Wave Size: 32, VRAM: 24560 MiB
Device 1: AMD Instinct MI100, gfx908:sramecc+:xnack- (0x908), VMM: no, Wave Size: 64, VRAM: 32752 MiB
modelsizeparamsbackendngldevtestt/s
qwen35 27B Q6_K20.97 GiB26.90 BROCm99ROCm1pp2048679.41 ± 3.18
qwen35 27B Q6_K20.97 GiB26.90 BROCm99ROCm1tg12821.65 ± 3.25

Not a surprise, CDNA is older – but the extra VRAM is key here, and the combined performance is improved by the faster card.

Flash Attention

Reading the llama.cpp build notes here there is also this handy little tidbit:

To enhance flash attention performance on RDNA3+ or CDNA architectures, you can utilize the rocWMMA library by enabling the -DGGML_HIP_ROCWMMA_FATTN=ON option. This requires rocWMMA headers to be installed on the build system.

Given my cards are RDNA3 and CDNA, this is a tick for me, so its time to rebuild once again, with another flag 🙂

HIPCXX=”$(hipconfig -l)/clang” HIP_PATH=”$(hipconfig -R)” \
cmake -S . -B build \
-DGGML_HIP=ON \
-DAMDGPU_TARGETS=$LLAMACPP_ROCM_ARCH \
-DCMAKE_BUILD_TYPE=Release \
-DLLAMA_CURL=ON \
-DLLAMA_OPENSSL=ON \
-DGGML_HIP_ROCWMMA_FATTN=ON \
&& cmake –build build –config Release -j$(nproc)

Also, I spotted in the build log – it sppears the flash attention only works for a few types of quant.. BF16, Q8_0, Q4_0, and F16.. 

./build/bin/llama-bench -p 2048 -n 128 -fa 1 --hf-repo unsloth/Qwen3.6-27B-GGUF:Q8_0
ggml_cuda_init: found 2 ROCm devices (Total VRAM: 57312 MiB):
Device 0: AMD Radeon RX 7900 XTX, gfx1100 (0x1100), VMM: no, Wave Size: 32, VRAM: 24560 MiB
Device 1: AMD Instinct MI100, gfx908:sramecc+:xnack- (0x908), VMM: no, Wave Size: 64, VRAM: 32752 MiB
modelsizeparamsbackendnglfatestt/s
qwen35 27B Q8_026.62 GiB26.90 BROCm991pp20481172.00 ± 1.91
qwen35 27B Q8_026.62 GiB26.90 BROCm991tg12823.49 ± 0.02

Woot! Getting better speeds now than I had on the Q6_K quant before for prompt, and token gen is only slightly slower, but Q8! Very cool (actually, hot.. need to get a beefier fan running so I don’t have to wait between benchs to see impact of changes!).

Q8_0 Benchmarks

The Q6 previously was becuase it would fit in each card individually, but thats not why we came.. we want full Q8 action! (or at least, something to stretch the combined VRAM pool!)

Qwen3.6-27B

Er.. look at the one right above?

Qwen3.6-35B MOE

./build/bin/llama-bench -p 2048 -n 128 -fa 1 --hf-repo unsloth/Qwen3.6-35B-A3B-GGUF:Q8_0
ggml_cuda_init: found 2 ROCm devices (Total VRAM: 57312 MiB)
Device 0: AMD Radeon RX 7900 XTX, gfx1100 (0x1100), VMM: no, Wave Size: 32, VRAM: 24560 MiB
Device 1: AMD Instinct MI100, gfx908:sramecc+:xnack- (0x908), VMM: no, Wave Size: 64, VRAM: 32752 MiB
modelsizeparamsbackendnglfatestt/s
qwen35moe 35B.A3B Q8_034.36 GiB34.66 BROCm991pp20481930.35 ± 8.28
qwen35moe 35B.A3B Q8_034.36 GiB34.66 BROCm991tg12862.12 ± 0.20

gemma-4-31B-it

Yeah, the instruct version.. I like being instructed by dense models 🙂

./build/bin/llama-bench -p 2048 -n 128 -mg 0 -fa 1 --hf-repo unsloth/gemma-4-31B-it-GGUF:Q8_0
ggml_cuda_init: found 2 ROCm devices (Total VRAM: 57312 MiB):
Device 0: AMD Radeon RX 7900 XTX, gfx1100 (0x1100), VMM: no, Wave Size: 32, VRAM: 24560 MiB
Device 1: AMD Instinct MI100, gfx908:sramecc+:xnack- (0x908), VMM: no, Wave Size: 64, VRAM: 32752 MiB
modelsizeparamsbackendnglfatestt/s
gemma4 31B Q8_030.38 GiB30.70 BROCm991pp2048857.99 ± 5.21
gemma4 31B Q8_030.38 GiB30.70 BROCm991tg12821.10 ± 0.02

gemma-4-26B-A4B-it

./build/bin/llama-bench -p 2048 -n 128 -fa 1 --hf-repo bartowski/google_gemma-4-26B-A4B-it-GGUF:Q8_0
ggml_cuda_init: found 2 ROCm devices (Total VRAM: 57312 MiB):
Device 0: AMD Radeon RX 7900 XTX, gfx1100 (0x1100), VMM: no, Wave Size: 32, VRAM: 24560 MiB
Device 1: AMD Instinct MI100, gfx908:sramecc+:xnack- (0x908), VMM: no, Wave Size: 64, VRAM: 32752 MiB
modelsizeparamsbackendnglfatestt/s
gemma4 26B.A4B Q8_025.00 GiB25.23 BROCm991pp20482280.04 ± 4.95
gemma4 26B.A4B Q8_025.00 GiB25.23 BROCm991tg12860.94 ± 0.15

Pretty much spot on with where I’d expect compard to A3B for Qwen.. slightly faster PP as fewer params, and slower TG as more active params.

Llama-3.3-70B-Instruct

Yeah, maybe not as great today as it once was, but I’m interested to see where it lands speed wise. Mostly because all the llama-bench results I can find are a year old, e.g. this one Q4_K with dual 7900xtx (Vulkan) where pp512 was 108, and tg128 was 14.57, or this one with dual 3090 and dual 3060 which got 17.4t/s Q4_K_M.
I’m curious to see how these old cards perform with current software!

./build/bin/llama-bench -p 2048,512 -n 128 -fa 1 --hf-repo unsloth/Llama-3.3-70B-Instruct-GGUF:Q4_0
ggml_cuda_init: found 2 ROCm devices (Total VRAM: 57312 MiB):
Device 0: AMD Radeon RX 7900 XTX, gfx1100 (0x1100), VMM: no, Wave Size: 32, VRAM: 24560 MiB
Device 1: AMD Instinct MI100, gfx908:sramecc+:xnack- (0x908), VMM: no, Wave Size: 64, VRAM: 32752 MiB
modelsizeparamsbackendnglfatestt/s
llama 70B Q4_037.35 GiB70.55 BROCm991pp2048443.61 ± 0.35
llama 70B Q4_037.35 GiB70.55 BROCm991pp512313.77 ± 0.13
llama 70B Q4_037.35 GiB70.55 BROCm991tg12817.37 ± 0.10

Pretty respectable! Looks like AMD has closed the software gap a fair ways since this time last year.

Balancing Uneven VRAM across devices

You may have noticed I don’t have cards with equal amounts of VRAM. This gives me a problem – which is I don’t want the 7900XTX to error due to memory allocation issues when there’s still RAM free on the mi100. Luckily, there’s a runtime flag for it, which sets a ratio.

Technically, I could go with 3/4 as I’ve got 24GB and 32GB, which divides nicely in 8GB chunks, but.. the 7900xtx is also my OS video output, so I need to keep 2GB free there as well. For now, I’ve settled on 5/8 ratio, so if the 7900xtx allocated 20GB, the mi100 will have allocated 32GB, which gives me enough safety margin that my OS won’t hang or worse, crash wayland and Gnome. I could maybe stretch this to 11/16 but this split works for now.

Doing this gives me a slight hit to my PP performance, because more layers are on the mi100, but it doesn’t hurt my TG much, and it gives me more context ability. Effectively I have 52GB of useable VRAM!

./build/bin/llama-bench -p 2048 -n 128 -fa 1 -ts 5/8 --hf-repo unsloth/Qwen3.6-27B-GGUF:Q8_0
ggml_cuda_init: found 2 ROCm devices (Total VRAM: 57312 MiB):
Device 0: AMD Radeon RX 7900 XTX, gfx1100 (0x1100), VMM: no, Wave Size: 32, VRAM: 24560 MiB
Device 1: AMD Instinct MI100, gfx908:sramecc+:xnack- (0x908), VMM: no, Wave Size: 64, VRAM: 32752 MiB
modelsizeparamsbackendnglfatstestt/s
qwen35 27B Q8_026.62 GiB26.90 BROCm9915.00/8.00pp20481102.67 ± 0.84
qwen35 27B Q8_026.62 GiB26.90 BROCm9915.00/8.00tg12823.48 ± 0.07

Serving it up

Here’s a command which will serve the models to the host, on port 8000. This means tools like Opencode can connect to it. Note I’m setting it to allow for serving 3 threads at a time, so that subagents can make parallel calls.

./build/bin/llama-server -fa 1 --parallel 3 --kv-unified -ts 5/8 --hf-repo unsloth/Qwen3.6-35B-A3B-GGUF:Q8_0 --ctx-size 262144 --host 0.0.0.0 --port 8000

And yeah, thats 256k of context! Here’s what it looks like loaded:

Every 2.0s: amd-smi monitor fedora: Thu 30 Apr 2026 01:13:31 AEST
in 0.491s (0)
GPU XCP POWER GPU_T MEM_T GFX_CLK GFX% MEM% ENC% DEC% VRAM_USAGE
0 0 48 W 70 °C 69 °C 300 MHz 0 % 0 % N/A 0 % 25.7/ 32.0 GB
1 0 13 W 56 °C 62 °C 22 MHz 2 % 1 % N/A 0 % 21.0/ 24.0 GB

Where next?

Next additions will be:

  1. Updating params for models in the server runtime
  2. Connecting OpenCode
  3. Setting up to build a docker image of the above as a working baseline (and maybe adding more architecture support for other gfx? might help others)
  4. Seeing if I can get the forms with TurboQuant on them working

Hooroo!

Leave a Reply

Your email address will not be published. Required fields are marked *