# ERROR | Expected a cuda device, but got: cpu **URL:** \ **Category:** Inference Endpoints on the Hub\ **Created:** [December 22, 2023, 10:33am UTC](https://discuss.huggingface.co/t/error-expected-a-cuda-device-but-got-cpu/66724 "2023-12-22T10:33:50Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![coreaiteam](https://avatars.discourse-cdn.com/v4/letter/c/7993a0/32.png) [@coreaiteam](https://discuss.huggingface.co/u/coreaiteam)\ **Post date:** [December 22, 2023, 10:33am UTC](https://discuss.huggingface.co/t/error-expected-a-cuda-device-but-got-cpu/66724/1 "2023-12-22T10:33:50Z")
I have trained a model and using the new feature of deploying the adapter model directory from repo on Huggingface but I am getting the following error when generating a response from the endpoint. However when I used the same handler.py in colab it is giving response. ERROR | Expected a cuda device, but got: cpu Along with a warning : UserWarning: Merge lora module to 4-bit linear may get different generations due to rounding errors. ![Screenshot 2023-12-22 at 4.33.54 PM 1](https://us1.discourse-cdn.com/hellohellohello/original/3X/4/c/4c2a2d80920051b4029acee28f679a6b8b62cba5.png) ---
**Author:** ![fthor](https://avatars.discourse-cdn.com/v4/letter/f/43a26b/32.png) [@fthor](https://discuss.huggingface.co/u/fthor)\ **Post date:** [January 1, 2024, 2:48pm UTC](https://discuss.huggingface.co/t/error-expected-a-cuda-device-but-got-cpu/66724/2 "2024-01-01T14:48:20Z")
I have a similar issue when using inference endpoints. I defined a Custom Handler following the doc, but when I try to load the model (llava) using bitsandbytes to quantize, it fails because the GPU is not found. Before trying to setup an endpoint i played around with a space via gradio using the same hardware specs and everything worked fine. To me it seems like the GPU is not present when the endpoint is initialized, but I could be wrong as I’m completely new with using the inference endpoints