# 8 bit precision error

**URL:** <https://discuss.huggingface.co/t/8-bit-precision-error/79424>\
**Category:** Models\
**Created:** [March 30, 2024, 1:31pm UTC](https://discuss.huggingface.co/t/8-bit-precision-error/79424 "2024-03-30T13:31:44Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![anon53969503](https://avatars.discourse-cdn.com/v4/letter/a/ecccb3/32.png) [@anon53969503](https://discuss.huggingface.co/u/anon53969503)\
**Post date:** [March 30, 2024, 1:31pm UTC](https://discuss.huggingface.co/t/8-bit-precision-error/79424/1 "2024-03-30T13:31:44Z")

</div>

Hi

I am finetuning Mistral 7B model for multilabel classification. I am getting the following error -

ValueError: You can’t train a model that has been loaded in 8-bit precision on a different device than the one you’re training on. Make sure you loaded the model on the correct device using for example `device\_map={‘’:torch.cuda.current\_device() or device\_map={‘’:torch.xpu.current\_device()}  
I0000 00:00:1711804411.121734 181893 cpu\_client.cc:373] TfrtCpuClient destroyed.

Here is my code -

# qunatization config

quantization\_config = BitsAndBytesConfig(  
load\_in\_4bit = True, # enable 4-bit quantization  
bnb\_4bit\_quant\_type = ‘nf4’, # information theoretically optimal dtype for normally distributed weights  
bnb\_4bit\_use\_double\_quant = True, # quantize quantized weights //insert xzibit meme  
bnb\_4bit\_compute\_dtype = torch.bfloat16 # optimized fp format for ML  
)

lora\_config = LoraConfig(  
r = 16, # the dimension of the low-rank matrices  
lora\_alpha = 8, # scaling factor for LoRA activations vs pre-trained weight activations  
target\_modules = [‘q\_proj’, ‘k\_proj’, ‘v\_proj’, ‘o\_proj’],  
lora\_dropout = 0.05, # dropout probability of the LoRA layers  
bias = ‘none’, # wether to train bias weights, set to ‘none’ for attention layers  
task\_type = ‘SEQ\_CLS’  
)

from accelerate import PartialState

model = AutoModelForSequenceClassification.from\_pretrained(  
model\_name,  
quantization\_config=quantization\_config,  
num\_labels=len(le.classes\_),  
device\_map={“”: PartialState().process\_index},  
)

model = prepare\_model\_for\_kbit\_training(model)  
model = get\_peft\_model(model, lora\_config)  
model.config.pad\_token\_id = tokenizer.pad\_token\_id

# define training args

training\_args = TrainingArguments(  
output\_dir = ‘multilabel\_classification’,  
learning\_rate = 1e-4,  
per\_device\_train\_batch\_size = 8, # tested with 16gb gpu ram  
per\_device\_eval\_batch\_size = 8,  
num\_train\_epochs = 10,  
weight\_decay = 0.01,  
evaluation\_strategy = ‘epoch’,  
save\_strategy = ‘epoch’,  
load\_best\_model\_at\_end = True  
)

label\_weights = 1 - labels\_encoded/labels\_encoded.sum()

# train

trainer = CustomTrainer(  
model = model,  
args = training\_args,  
train\_dataset = tokenized\_ds[‘train’],  
eval\_dataset = tokenized\_ds[‘val’],  
tokenizer = tokenizer,  
data\_collator = functools.partial(collate\_fn, tokenizer=tokenizer),  
compute\_metrics = compute\_metrics,  
label\_weights = torch.tensor(label\_weights, device=model.device)  
)

trainer.train()

I have tried many things discussed on google but nothing worked. Any help would be appreciated.