# Running 8B Llama on Jetson Orin Nano (using only 2.5GB of GPU memory)

**URL:** <https://discuss.huggingface.co/t/running-8b-llama-on-jetson-orin-nano-using-only-2-5gb-of-gpu-memory/174180>\
**Category:** Show and Tell\
**Created:** [March 11, 2026, 8:41am UTC](https://discuss.huggingface.co/t/running-8b-llama-on-jetson-orin-nano-using-only-2-5gb-of-gpu-memory/174180 "2026-03-11T08:41:21Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![roroep](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/roroep/32/58533_2.png) [@roroep](https://discuss.huggingface.co/u/roroep)\
**Post date:** [March 11, 2026, 8:41am UTC](https://discuss.huggingface.co/t/running-8b-llama-on-jetson-orin-nano-using-only-2-5gb-of-gpu-memory/174180/1 "2026-03-11T08:41:21Z")

</div>

Hi, we would like to share our project on deploying **8B Llama on Jetson Orin Nano** , using only 2.5GB of GPU shared memory (peak), with a comparison against a llama.cpp INT4 baseline.

### **Baseline (llama.cpp INT4)**

In our baseline setup, Llama-3.1-8B INT4 reached:

- 5.2GB GPU shared memory (peak)

- 6.8GB total RAM (peak)

On Jetson Orin Nano, this uses most of the available memory budget and leaves limited headroom for other edge workloads.

### **Our result**

Using our own extreme low-bit (1.58-bit) deployment pipeline, we ran an 8B-class Llama model with:

- 2.5GB GPU shared memory (peak)

- 4.1GB total RAM (peak)

This makes the deployment more practical on Orin Nano when the LLM needs to coexist with other components on the device.

### **Main Techniques**

- 1.58-bit quantization (Mixed-precision QAT)

- Kernel-level optimizations (Custom kernel for embedding access and layer fusion)

### **Demo Video**

 ![화면 캡처 2026-03-11 174209](https://us1.discourse-cdn.com/hellohellohello/original/3X/a/1/a11f39bc4c1da12a5c5d5c960999afa663b9e7f3.jpeg)

- Link: [https://youtu.be/yVZSksaqf08](https://github.com/ggml-org/llama.cpp/discussions/url)

**Notes**

- For our 1.58-bit Llama model, instruction tuning has been limited to date and we expect further improvements with additional tuning.

### **Why this may be useful**

For edge deployments, memory headroom matters because the LLM often needs to run alongside other components such as:

- Other AI models including STT, TTS, and more

- System workloads including perception, logging, control, networking, and more

Reducing the model footprint makes on-device LLM deployment more realistic even on Nano-class edge SoCs.

**And we are sharing more details at GTC 2026!**

If you are blocked by memory footprint or latency while building Llama or other LLMs on Jetson or other SoC platforms, please leave us a message.

Contact: [https://enerzai.com/contact](https://github.com/ggml-org/llama.cpp/discussions/url)