# Error while training Mixtral in 8bit **URL:** \ **Category:** Intermediate\ **Created:** [January 16, 2024, 8:13am UTC](https://discuss.huggingface.co/t/error-while-training-mixtral-in-8bit/69313 "2024-01-16T08:13:26Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![devroy](https://avatars.discourse-cdn.com/v4/letter/d/779978/32.png) [@devroy](https://discuss.huggingface.co/u/devroy)\ **Post date:** [January 16, 2024, 8:13am UTC](https://discuss.huggingface.co/t/error-while-training-mixtral-in-8bit/69313/1 "2024-01-16T08:13:26Z")
Hi when I try to train my model using 8bit I get an error as shown below my code is as follows from peft import LoraConfig, get\_peft\_model, prepare\_model\_for\_kbit\_training MODEL\_NAME =“mistralai/Mixtral-8x7B-v0.1” peft\_config = LoraConfig( lora\_alpha=16, lora\_dropout=0.1, r=64, bias=“none”, target\_modules=[ “q\_proj”, “k\_proj”, “v\_proj”, “o\_proj”, ], task\_type=“CAUSAL\_LM” ) model = AutoModelForCausalLM.from\_pretrained( MODEL\_NAME, load\_in\_8bit=True, device\_map=“auto”, use\_flash\_attention\_2=True, torch\_dtype=torch.bfloat16, ) model = prepare\_model\_for\_kbit\_training(model) model = get\_peft\_model(model, peft\_config) LEN = 2500 args = TrainingArguments( output\_dir = MODEL\_NAME+“\_”+str(LEN)+“\_ner”, num\_train\_epochs = 3, per\_device\_train\_batch\_size = 1, warmup\_steps = 0.04, logging\_steps=100, # save\_strategy=“epoch”, save\_steps=1000, learning\_rate=2e-5, optim=“adamw\_torch” lr\_scheduler\_type=‘cosine’, gradient\_accumulation\_steps= 8, bf16=True, # evaluation\_strategy=“epoch”, eval\_steps=1000, # gradient\_checkpointing=True, save\_total\_limit=2, ) trainer = SFTTrainer( model=model, max\_seq\_length=LEN, tokenizer=tokenizer, args=args, peft\_config=peft\_config, train\_dataset=data\_train, eval\_dataset=data\_val, dataset\_text\_field=“text”, packing= False, neftune\_noise\_alpha=5, ) trainer.train() The error I get is as follows. --------------------------------------------------------------------------- RuntimeError Traceback (most recent call last) Cell In[22], line 1 ----\> 1 trainer.train() File [~/anaconda3/envs/moe/lib/python3.11/site-packages/trl/trainer/sft\_trainer.py:315](… /trl/trainer/sft\_trainer.py:315), in SFTTrainer.train(self, \*args, \*\*kwargs) 312 if self.neftune\_noise\_alpha is not None and not self.\_trainer\_supports\_neftune: 313 self.model = self.\_trl\_activate\_neftune(self.model) → 315 output = super().train(\*args, \*\*kwargs) 317 # After training we make sure to retrieve back the original forward pass method 318 # for the embedding layer by removing the forward post hook. 319 if self.neftune\_noise\_alpha is not None and not self.\_trainer\_supports\_neftune: File ~/anaconda3/envs/moe/lib/python3.11/site-packages/transformers/trainer.py:1537, in Trainer.train(self, resume\_from\_checkpoint, trial, ignore\_keys\_for\_eval, \*\*kwargs) 1535 hf\_hub\_utils.enable\_progress\_bars() 1536 else: → 1537 return inner\_training\_loop( 1538 args=args, 1539 resume\_from\_checkpoint=resume\_from\_checkpoint, 1540 trial=trial, 1541 ignore\_keys\_for\_eval=ignore\_keys\_for\_eval, 1542 ) File …/transformers/trainer.py:1854, in Trainer.\_inner\_training\_loop(self, batch\_size, args, resume\_from\_checkpoint, trial, ignore\_keys\_for\_eval) 1851 self.control = self.callback\_handler.on\_step\_begin(args, self.state, self.control) … 486 ) 487 grad\_A = torch.matmul(grad\_output, CB).view(ctx.grad\_shape).to(ctx.dtype\_A) 488 else: RuntimeError: The size of tensor a (32) must match the size of tensor b (8) at non-singleton dimension 0 Thanks in advance