Model Watch #2: speed runs of three open models on one V100

We ran three open models with vLLM on one V100 GPU on 28 Oct 2026. Every number from these runs is measured by Starfield. This is not a ranking of models.

The three models are Qwen3-8B, granite-4.2-8b and Olmo-3-7B-Instruct. Each one answered the same 150 prompts, 3 times. All 450 requests per model completed, and none failed.

What we ran

Model revisions: Qwen3-8B b968826d9c46dd6066d109eabc6255188de91218, granite-4.2-8b a7042144125ce5ae64f89610e8289647fc000b9a, Olmo-3-7B-Instruct 6e5971d9eba42665f5bd5a0fcf047f299ce1dccc.

Results

Each cell is the mean of 3 repeats. The range is in brackets when the repeats differ. Rows are in alphabetical order. This is not a ranking.

How to read these numbers

• This is not a ranking. It covers three models, one GPU, one engine and one load of 8 requests at once.
• The models stop at different lengths. For the same 150 prompts, granite-4.2-8b wrote 27,622 tokens and Qwen3-8B wrote about 21,586. A model that writes more has more to generate. Compare token counts before you compare speeds.
• The first Olmo-3 repeat was slower to start: 264.9 ms to first token, against 182.8 and 183.2 ms. We did not find the cause. That one repeat pulls up the Olmo-3 mean.
• Greedy decoding is not exact when 8 requests run at once. Qwen3-8B wrote 26 more tokens in repeat 3 than in the other two.
• The vLLM tool counts input tokens from the raw prompt, without the chat template. We do not use its input token total or its total tokens per second.
• Speed says nothing about answer quality. No quality check has run.

What we found about the method

• Olmo-3 does not load with transformers 5.x at vLLM 0.19.1. All three runs use transformers 4.57.6, so they are comparable.
• granite-4.2-8b runs out of memory at its default context of 131,072 tokens. Every model now runs with a context limit of 4096 tokens.
• The longest prompt plus 256 output tokens is about 1,300 tokens, so no prompt is cut.
• On a V100, vLLM uses the Triton attention backend. FlashAttention 2 needs a newer GPU, so its "not supported" log line is expected.
• The run now names the one card it uses, and the run record lists only that card.

The raw results and the findings note are public on GitHub.

What has not been run yet

• vLLM 0.32.0 on an A100. We have not checked the prefix caching setting at that version.
• llama.cpp. It has not run on a GPU yet. Model conversion and flash attention on the V100 are unchecked.
• Quality checks.
• Energy use and cost per million tokens.

The full method is on the methodology page. If you find a mistake in a number, it goes in the corrections log.