# LoRA finetuning without quantization (8bit)

**URL:** <https://discuss.huggingface.co/t/lora-finetuning-without-quantization-8bit/62977>\
**Category:** 🤗Transformers\
**Created:** [November 21, 2023, 7:35am UTC](https://discuss.huggingface.co/t/lora-finetuning-without-quantization-8bit/62977 "2023-11-21T07:35:38Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![abhishek-wfx](https://avatars.discourse-cdn.com/v4/letter/a/edb3f5/32.png) [@abhishek-wfx](https://discuss.huggingface.co/u/abhishek-wfx)\
**Post date:** [November 21, 2023, 7:35am UTC](https://discuss.huggingface.co/t/lora-finetuning-without-quantization-8bit/62977/1 "2023-11-21T07:35:38Z")

</div>

When I load gpt2 model without using load\_in\_8bit = True, I get below error while training using SFTTrainer.

RuntimeError: element 0 of tensors does not require grad and does not have a grad\_fn

Arguments I am using -

> def get\_args():  
> parser = argparse.ArgumentParser()  
> parser.add\_argument(‘-f’) #added to work on notebook  
> parser.add\_argument(“–model\_path”, type=str, default=“gpt2”) # model-id from HF
> 
> ```
> parser.add_argument("--seq_length", type=int, default=1024) #specific for gpt2
> parser.add_argument("--max_steps", type=int, default=500) #no. of batches to be fine tuned
> parser.add_argument("--batch_size", type=int, default=1) #single batch at a time due to memory issue
> parser.add_argument("--gradient_accumulation_steps", type=int, default=16) #accumulating gradient as batch_size 1 is not stable
> # parser.add_argument("--eos_token_id", type=int, default=49152) #eos already specified in gpt2
> 
> parser.add_argument("--learning_rate", type=float, default=1e-4)
> # parser.add_argument("--lr_scheduler_type", type=str, default="cosine")
> parser.add_argument("--num_warmup_steps", type=int, default=100) #fixed learning rate on first warmup steps
> # parser.add_argument("--weight_decay", type=float, default=0.05)
> 
> parser.add_argument("--no_gradient_checkpointing", action="store_false", default=False) #store gradients to be retrived during backpropagation, used to reduce memory. not req
> parser.add_argument("--local_rank", type=int, default=0) #process id, not useful in single process training
> parser.add_argument("--no_fp16", action="store_false") #fp16 not supported
> parser.add_argument("--bf16", action="store_true", default=False) #using bf16
> 
> parser.add_argument("--seed", type=int, default=0) #seed
> parser.add_argument("--num_workers", type=int, default=None) #using single worker
> 
> parser.add_argument("--output_dir", type=str, default="./gpt2-lora/checkpoints2") #save model to this folder
> parser.add_argument("--log_freq", default=100, type=int) #display output after log steps
> parser.add_argument("--eval_freq", default=100, type=int) #evaluate after eval steps
> parser.add_argument("--save_freq", default=100, type=int) #save to folder after save steps
> 
> return parser.parse_args()
> 
> ```

> lora\_config = LoraConfig(  
> r=8,  
> lora\_alpha=16,  
> lora\_dropout=0.05,  
> bias=“none”,  
> task\_type=“CAUSAL\_LM”,  
> fan\_in\_fan\_out = True # added from docs; [https://github.com/huggingface/peft/blob/v0.6.2/src/peft/tuners/lora/config.py#L24](https://github.com/huggingface/peft/blob/v0.6.2/src/peft/tuners/lora/config.py#L24)  
> )

> training\_args = TrainingArguments(  
> output\_dir=args.output\_dir,  
> dataloader\_drop\_last=True, #drops last batch if dataset is not divisble by batch\_size, not useful for batch\_size = 1  
> evaluation\_strategy=“steps”,  
> max\_steps=args.max\_steps,  
> eval\_steps=args.eval\_freq,  
> save\_steps=args.save\_freq,  
> logging\_steps=args.log\_freq,  
> per\_device\_train\_batch\_size=args.batch\_size,  
> per\_device\_eval\_batch\_size=args.batch\_size,  
> learning\_rate=args.learning\_rate,  
> # lr\_scheduler\_type=args.lr\_scheduler\_type,  
> warmup\_steps=args.num\_warmup\_steps,  
> gradient\_accumulation\_steps=args.gradient\_accumulation\_steps,  
> gradient\_checkpointing=not args.no\_gradient\_checkpointing,  
> fp16=not args.no\_fp16,  
> bf16=args.bf16,  
> # weight\_decay=args.weight\_decay,  
> run\_name=“gpt2-finetuned2”,  
> # report\_to=“wandb”,  
> ddp\_find\_unused\_parameters=False,  
> )

training code -

> model = AutoModelForCausalLM.from\_pretrained(  
> args.model\_path, device\_map={“”: Accelerator().process\_index}  
> )

> trainer = SFTTrainer(  
> model=model, #checked  
> args=training\_args,  
> train\_dataset=train\_data, #checked  
> eval\_dataset=val\_data, #checked  
> peft\_config=lora\_config, #checked  
> packing=True, #checked  
> )

Is quantization mandatory or can i make this work somehow?

---

<div class="post-metadata">

**Author:** ![JulioSanchezD](https://avatars.discourse-cdn.com/v4/letter/j/a88e4f/32.png) [@JulioSanchezD](https://discuss.huggingface.co/u/JulioSanchezD)\
**Post date:** [February 23, 2024, 11:08pm UTC](https://discuss.huggingface.co/t/lora-finetuning-without-quantization-8bit/62977/2 "2024-02-23T23:08:07Z")

</div>

I’m having the exact same issue, have you found a solution?