# Llama 2 70B on a cpu

**URL:** <https://discuss.huggingface.co/t/llama-2-70b-on-a-cpu/48526>\
**Category:** Beginners\
**Created:** [July 28, 2023, 2:09pm UTC](https://discuss.huggingface.co/t/llama-2-70b-on-a-cpu/48526 "2023-07-28T14:09:00Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![malikarumi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/malikarumi/32/31844_2.png) [@malikarumi](https://discuss.huggingface.co/u/malikarumi)\
**Post date:** [July 28, 2023, 2:09pm UTC](https://discuss.huggingface.co/t/llama-2-70b-on-a-cpu/48526/1 "2023-07-28T14:09:00Z")

</div>

I was just using this model here on HuggingFace. I noticed that it referenced a cpu, which I didn’t expect at all. Here’s the link:

> **[Explore Llamav2 With TGI - a Hugging Face Space by ysharma](https://huggingface.co/spaces/ysharma/Explore_llamav2_with_TGI)**
>
> Discover amazing ML apps made by the community

Beside the title it says:

“Running on cpu. Upgrade.”

Is this just the _endpoint_ running on a CPU?

---

<div class="post-metadata">

**Author:** ![YaTharThShaRma999](https://avatars.discourse-cdn.com/v4/letter/y/41988e/32.png) [@YaTharThShaRma999](https://discuss.huggingface.co/u/YaTharThShaRma999)\
**Post date:** [July 28, 2023, 4:19pm UTC](https://discuss.huggingface.co/t/llama-2-70b-on-a-cpu/48526/2 "2023-07-28T16:19:49Z")

</div>

No its running with inference endpoints which is probably running with several powerful gpus(a100). Running a 70b model on cpu would be extremely slow and take over 100 gb ram.

I don’t know why its running on cpu upgrade however.

---

<div class="post-metadata">

**Author:** ![xinchun](https://avatars.discourse-cdn.com/v4/letter/x/ea666f/32.png) [@xinchun](https://discuss.huggingface.co/u/xinchun)\
**Post date:** [August 23, 2023, 1:37am UTC](https://discuss.huggingface.co/t/llama-2-70b-on-a-cpu/48526/3 "2023-08-23T01:37:37Z")

</div>

I hava test use llama.cpp infer Llama2 7B、13B 70B on different CPU

The fast 70B INT8 speed as **3.77 token /s** ( AMD 9654P 96C/768G memory)

run command:  
**build/bin/main -m /home/apps/models/wizardlm-70b-q8\_0.bin -gqa ` `8` `-eps 1e-` `5` `-t ` `96` `-n ` `1024` `–repeat\_penalty ` `1.0` `–color -c ` `512` `–temp ` `0.6` `-p ` `“Please introduce me something about vipshop holdings ltd.”**

token speed:

`llama_print_timings: load time = ` `1576.96` `ms`  
`llama_print_timings: sample time = ` `10.12` `ms / ` `396` `runs ( ` `0.03` `ms per token, ` `39114.97` `tokens per second)`  
`llama_print_timings: prompt eval time = ` `1500.66` `ms / ` `15` `tokens ( ` `100.04` `ms per token, ` `10.00` `tokens per second)`

`llama_print_timings: eval time = ` `104635.98` `ms / ` `395` `runs ( ` `264.90` `ms per token, ` \*\*`3.77` `tokens per second**)`

It’s make sense:)