# How many tokens will an old 3090 produce? **URL:** \ **Category:** Beginners\ **Created:** [August 25, 2026, 10:25pm UTC](https://discuss.huggingface.co/t/how-many-tokens-will-an-old-3090-produce/179251 "2026-08-25T22:25:39Z")\ **Posts on this page:** 1\ **Showing post:** 2
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [August 26, 2026, 1:00am UTC](https://discuss.huggingface.co/t/how-many-tokens-will-an-old-3090-produce/179251/2 "2026-08-26T01:00:33Z")
Oh. It depends on the backend you’re using, but that **seems to be about the expected speed** : * * * 482 tokens in about 12 seconds works out to roughly **40 tokens/s**. If those 12 seconds are the actual generation/decode time, that number does not look suspiciously high for Qwen3.8-27B on a 3090. As a sanity check rather than an exact apples-to-apples benchmark, the current crowd-sourced [llamabench.ai results for Qwen3.8-27B on an RTX 3090](https://llamabench.ai/browse/rtx3090/qwen3-8-27b) include non-speculative runs at **38.2, 40.0, 41.1, and 45.6 tok/s**. Those runs use llama.cpp and somewhat different quant/cache settings, so I would not treat them as a reproduction of your Ollama run, but your ~40.2 tok/s sits right in that range. The cheapest way to verify what you are seeing is probably: ```bash ollama run --verbose ollama ps ``` For the first command, the useful lines are approximately: ```plaintext prompt eval count: ... prompt eval duration: ... prompt eval rate: ... eval count: ... eval duration: ... eval rate: ... tokens/s ``` Ollama’s [API usage documentation](https://docs.ollama.com/api/usage) makes the distinction explicit: - `prompt_eval_*` is processing the input prompt. - `eval_count` is the number of generated output tokens. - `eval_duration` is the time spent generating those output tokens. - `total_duration` also includes other work such as model loading and prompt processing. So if Ollama itself reports an `eval rate` around 40 tok/s, there is not much ambiguity left: you really are getting about 40 generated tokens/s. `ollama ps` gives the other high-value check. According to the [Ollama FAQ](https://docs.ollama.com/faq), `100% GPU` means the model is loaded entirely on the GPU, while a CPU/GPU percentage indicates that it is split between system RAM and VRAM. A useful decision tree is therefore: ```plaintext Is ~12 s the generation time / Ollama eval_duration? | +-- Yes | | | +-- ~482 / ~12 s = ~40 tok/s | | | +-- ollama ps says 100% GPU | | | | | +-- This looks quite normal for this model/GPU class. | | | +-- CPU/GPU split | | | +-- Then context size/offloading becomes important | before comparing the number with other benchmarks. | +-- No / not sure | +-- Use --verbose (or the API metrics) once and compare eval_rate, rather than dividing visible tokens by wall-clock time. ``` In other words, I would not worry that the 3090 has somehow produced an impossible result. Its age is a bit misleading here: **24 GB of VRAM is still a very useful configuration for local inference** , because a quantized model of this size can fit without having to spill a large part of it into CPU memory. > **Why comparing this to an older '27B model on a 3090' can be misleading** > > Qwen3.8-27B is also not just a conventional 27B full-attention Transformer with a new name. > > The [official Qwen3.8-27B model card](https://huggingface.co/Qwen/Qwen3.8-27B) describes: > > - 27B parameters > - 64 language-model layers > - a layout of > `16 × (3 × (Gated DeltaNet → FFN) → 1 × (Gated Attention → FFN))` > - MTP (Multi-Token Prediction) training > - native 262K context > > So it is a hybrid recurrent/attention architecture. That does **not** mean “DeltaNet explains exactly why you get 40 tok/s”, but it does mean an old benchmark for an unrelated 27B full-attention model is not a particularly strong prediction for this model. > > Qwen’s own model card also warns that inference throughput can vary significantly across serving frameworks. That matters here because: > > ```plaintext > GPU + parameter count + nominal quant > > ``` > > is not enough to define a benchmark. > > At minimum, these can change the result: > > ```plaintext > backend / backend version > exact GGUF > quantization > GPU offload > context length > KV/state-cache settings > Flash Attention > prompt length > prefill vs decode > speculative decoding / MTP > sampling/application overhead > > ``` > > So a result from llama.cpp, Ollama, vLLM, or another backend should usually be treated as a range/check rather than an exact prediction unless those conditions match. > > There is an additional timing detail here: [Ollama added Qwen3.8-27B support in v0.32.12](https://github.com/ollama/ollama/releases), released on August 14, 2026, and v0.32.13 immediately included another Qwen3.8-specific change. This is a very new model/runtime combination, so older “3090 + ~27B” reports can easily be measuring a materially different software path. > **About the GGUF you pulled from Hugging Face** > > There is a small artifact-identification wrinkle that is probably worth keeping separate from the performance question. > > Hugging Face’s [Ollama integration documentation](https://huggingface.co/docs/hub/ollama) supports tags of the form: > > ```plaintext > hf.co//: > > ``` > > and also allows using a complete GGUF filename when an exact artifact matters. > > At the moment, the Unsloth repository recommends: > > ```bash > ollama run hf.co/unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M > > ``` > > The current [`UD-Q4_K_M` file](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/blob/main/Qwen3.8-27B-UD-Q4_K_M.gguf) is **16.5 GB** , and the current [`mmproj-F16.gguf`](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/blob/main/mmproj-F16.gguf) is **928 MB**. > > Those sizes match the numbers in your log remarkably closely: > > ```plaintext > model: ~16.5 GB > projector: ~928 MB > > ``` > > So the current `UD-Q4_K_M` + F16 projector is a very plausible match. > > I would still call that a **strong size match, not proof of the exact historical artifact** , because the contents of a model repository and its quantization aliases can change over time. If exact reproducibility ever matters, using/recording the full GGUF filename or digest is safer than relying only on a short quantization label. > > This distinction does not really change the answer to your speed question; it just makes later benchmark comparisons less ambiguous. > **What about MTP / speculative decoding?** > > I would keep MTP separate from the explanation of your current ~40 tok/s result. > > Qwen3.8-27B was trained with Multi-Token Prediction, but: > > ```plaintext > "the model has MTP capability" > > ``` > > and > > ```plaintext > "this particular inference run is using MTP/speculative decoding" > > ``` > > are not the same statement. > > There is already enough evidence to explain ~40 tok/s **without** speculative decoding: the 3090/Qwen3.8-27B community results linked above include several non-speculative runs in almost exactly that range. > > For comparison, Ollama exposes a separate current model variant, [`qwen3.8:27b-mtp-q4_K_M`](https://ollama.com/library/qwen3.8:27b-mtp-q4_K_M), whose parameters explicitly include: > > ```plaintext > draft_num_predict: 4 > > ``` > > Likewise, [llama.cpp documents speculative decoding](https://github.com/ggml-org/llama.cpp/blob/master/docs/speculative.md) as an explicit inference mechanism: draft tokens are proposed and then verified by the target model. > > That can raise decode throughput substantially when draft acceptance and the rest of the configuration cooperate. The same 3090 benchmark page currently has MTP runs well above the non-speculative ~40 tok/s range. > > But I would not use those faster MTP numbers as the baseline for your run unless you have confirmed that MTP is actually active. > > A simple way to keep the comparison clean is: > > ```plaintext > normal decode <-> normal decode > MTP/speculative <-> MTP/speculative > > ``` > > rather than mixing both sets of numbers. > **Why two people can report very different 'tokens/s' for the same GPU** > > There are really two separate performance phases in ordinary autoregressive inference: > > ### 1. Prompt processing / prefill > > The existing prompt is processed, often many tokens at once. > > Ollama reports this separately as: > > ```plaintext > prompt eval rate > > ``` > > ### 2. Token generation / decode > > The model repeatedly generates the next token. > > Ollama reports this as: > > ```plaintext > eval rate > > ``` > > For an interactive chat, decode rate is usually what someone means when they say “the model generates at 40 tok/s”, but prompt processing has a large effect on time-to-first-token and therefore on how fast the application _feels_. > > So these are quite different comparisons: > > ```plaintext > "I get 40 tok/s decode" > > ``` > > ```plaintext > "The entire request, including a giant prompt, finishes at an average of 40 tok/s" > > ``` > > ```plaintext > "I counted 482 visible words/tokens and divided by stopwatch time" > > ``` > > The first one is the cleanest model-generation benchmark. > > Context length is another reason benchmark numbers move. If a configuration fits entirely in VRAM at one context size but needs CPU offload or substantially more cache/state memory at another, the performance regime can change. > > That is why `ollama ps` is useful: it provides a direct observation instead of requiring you to guess from the nominal GGUF file size. > > For a reproducible comparison, something like this is enough: > > ```plaintext > GPU: RTX 3090 24 GB > Ollama version: ... > model: exact tag / GGUF > context: ... > processor: 100% GPU (or split) > prompt eval rate: ... > eval rate: ... > MTP/speculative: on/off > > ``` > > You do not need all of that just to answer “is 40 tok/s plausible?”, but it makes later comparisons much more useful. > **One more Qwen3.8-specific detail: thinking tokens** > > The [Qwen3.8-27B model card](https://huggingface.co/Qwen/Qwen3.8-27B) says that the model thinks by default before producing its direct response. > > That means “generated tokens” and “the tokens I visibly counted in the final answer” are not necessarily identical concepts, depending on the frontend/API and how thinking is displayed. > > For benchmarking, I would therefore use Ollama’s own `eval_count` and `eval_duration` rather than manually counting the final visible answer. > > Again, this is mostly a measurement/detail issue rather than evidence of a problem with your setup. > **If the performance changes later** > > I would not troubleshoot anything further while it is producing clean output at ~40 tok/s. But if the behavior changes, the symptom gives you a reasonably cheap next branch: > > | Symptom | First thing I would check | > | --- | --- | > | Decode suddenly much slower | `ollama ps`, context length, CPU/GPU split | > | First token is slow but generation is fast | prompt evaluation / prompt size | > | Short chats are fast, long chats become slow | context/cache/state size and GPU residency | > | Enabling MTP gives little/no gain | whether MTP is actually active and draft acceptance | > | Speed is normal but output becomes malformed | Ollama/backend version before assuming hardware failure | > | Two benchmarks disagree badly | backend, exact GGUF, context, cache, MTP, and measurement boundary | > > That keeps several different problems separate instead of treating every performance change as “the GPU got slower”. So, for the result you posted, my default interpretation would simply be: **yes, ~40 tok/s is believable.** If `ollama run ... --verbose` reports roughly that `eval rate` and `ollama ps` says `100% GPU`, I would consider the basic mystery solved. The fact that a 3090 is an older card does not prevent it from still being a very capable local-LLM card when a ~27B quantized model fits comfortably inside its 24 GB VRAM. --- _[View the full topic](https://discuss.huggingface.co/t/how-many-tokens-will-an-old-3090-produce/179251)._