# Utilizing GPU for ggml model with Langchain

**URL:** <https://discuss.huggingface.co/t/utilizing-gpu-for-ggml-model-with-langchain/49412>\
**Category:** Beginners\
**Created:** [August 4, 2023, 2:02am UTC](https://discuss.huggingface.co/t/utilizing-gpu-for-ggml-model-with-langchain/49412 "2023-08-04T02:02:43Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![carpenterjrds](https://avatars.discourse-cdn.com/v4/letter/c/ccd318/32.png) [@carpenterjrds](https://discuss.huggingface.co/u/carpenterjrds)\
**Post date:** [August 4, 2023, 2:02am UTC](https://discuss.huggingface.co/t/utilizing-gpu-for-ggml-model-with-langchain/49412/1 "2023-08-04T02:02:43Z")

</div>

I’m currently using a ggml-format model (13b-chimera.ggmlv3.q4\_1.bin) in an app using Langchain. I’ve found that the program is still only using the CPU, despite running it on a VM with a GPU.

I’ve tried using the line `torch.cuda.set_device(torch.device("cuda:0"))` which returns the error `AttributeError: module 'torch._C' has no attribute '_cuda_setDevice'`. But running the line `torch.cuda.is_available()` returns true, indicating that CUDA can find the GPU.

Can ggml models work with GPUs in the first place? Do I need to use another format like GPTQ? If so, how would I implement it in the program, since ggml models are a single .bin file while GPTQ models seem to be a collection of files?

 ![llm_snip](https://us1.discourse-cdn.com/hellohellohello/original/3X/e/c/ec08fb3bb2321fb1866130a63bddfafe5c422d7e.png)

---

<div class="post-metadata">

**Author:** ![Wi1](https://avatars.discourse-cdn.com/v4/letter/w/34f0e0/32.png) [@Wi1](https://discuss.huggingface.co/u/Wi1)\
**Post date:** [August 19, 2023, 1:17am UTC](https://discuss.huggingface.co/t/utilizing-gpu-for-ggml-model-with-langchain/49412/2 "2023-08-19T01:17:26Z")

</div>

Can check out the GPU instrucitons: [https://python.langchain.com/docs/integrations/llms/llamacpp#gpu](https://python.langchain.com/docs/integrations/llms/llamacpp#gpu)