# Why can't I reproduce benchmark scores from papers like Phi, Llama, or Qwen? Am I doing something wrong or is this normal? **URL:** \ **Category:** Models\ **Created:** [May 27, 2025, 4:49pm UTC](https://discuss.huggingface.co/t/why-cant-i-reproduce-benchmark-scores-from-papers-like-phi-llama-or-qwen-am-i-doing-something-wrong-or-is-this-normal/157027 "2025-05-27T16:49:47Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![jh-lim](https://avatars.discourse-cdn.com/v4/letter/j/e0b2c6/32.png) [@jh-lim](https://discuss.huggingface.co/u/jh-lim)\ **Post date:** [May 27, 2025, 4:49pm UTC](https://discuss.huggingface.co/t/why-cant-i-reproduce-benchmark-scores-from-papers-like-phi-llama-or-qwen-am-i-doing-something-wrong-or-is-this-normal/157027/1 "2025-05-27T16:49:47Z")
I’m working on evaluating open-source LLMs (e.g., Phi, Llama, Qwen), and I’ve noticed that **the benchmark scores I get are consistently different** from the ones reported in their tech reports or papers — sometimes by a wide margin. Sometimes the results are **lower** than expected, but surprisingly, sometimes they’re **higher** too. My point is: there are (many) cases where the difference is quite large, and it’s not clear _why_. I’ve tried: - Using lm-eval-harness with the default settings - Matching tokenizers and prompt formats as best as possible - Evaluating on standard benchmarks like MMLU, GSM8K, ARC, etc, in the reports under the same few-shot conditions Despite this, the scores I get are often significantly different from what’s published — and I can’t find any official scripts or clear explanations of the exact benchmarking setup used in those papers. This seems to happen _not just with one model_, but across many open-source models. **Is this a common experience in the community?** - Are papers using special prompt engineering or internal eval setups they don’t release? - Am I missing some key benchmarking tricks? - Is this just part of the game at this point? Would really appreciate if anyone can share: - Experience trying to reproduce scores - Any evaluation tips - Benchmarking setups that _actually_ match reported numbers Thanks in advance! ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [May 28, 2025, 12:55am UTC](https://discuss.huggingface.co/t/why-cant-i-reproduce-benchmark-scores-from-papers-like-phi-llama-or-qwen-am-i-doing-something-wrong-or-is-this-normal/157027/2 "2025-05-28T00:55:36Z")
The backend, or rather, if the library version or options passed to the generation function (such as temperature) are different, the results may vary, so I think it can only be used as a rough guide. Leaderboards are easy to compare because they use the same criteria within the same leaderboard, but I don’t think there are many absolute indicators that can be used. For large companies, I think the output of the endpoints officially provided by the company can be used as a reference. > **[LLM inference server performances comparison llama.cpp / TGI / vLLM ·...](https://github.com/ggml-org/llama.cpp/discussions/6730)** > > Performances and improvment area This thread objective is to gather llama.cpp performance 📈 and improvement ideas💡against other popular LLM inference frameworks, especially on the CUDA backend. Let... > **[Spaces - Hugging Face](https://huggingface.co/spaces?sort=trending&search=leader)** > > Discover amazing ML apps made by the community ---
**Author:** ![Pimpcat-AU](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/pimpcat-au/32/48989_2.png) [@Pimpcat-AU](https://discuss.huggingface.co/u/Pimpcat-AU)\ **Post date:** [June 10, 2025, 7:54pm UTC](https://discuss.huggingface.co/t/why-cant-i-reproduce-benchmark-scores-from-papers-like-phi-llama-or-qwen-am-i-doing-something-wrong-or-is-this-normal/157027/3 "2025-06-10T19:54:31Z")
Yes, this is normal. Benchmark results in LLM papers are often not exactly reproducible due to differences in evaluation code, tokenizer versions, prompt formatting, dataset splits, hardware, random seeds, and sometimes undocumented “internal” settings or prompt engineering. Even “official” scripts can produce different results if any dependency changes. Tips: ``` Use the exact tokenizer and prompt format as the original paper or repo. Check for hidden preprocessing, special instructions, or test set modifications. Use the same version of lm-eval-harness and its dependencies. Run multiple seeds and average results if possible. ``` Even then, you may still see margin differences. This is a common and known issue in the LLM community. Solution provided by Triskel Data Deterministic AI.