# QLoRA - model isn't training **URL:** \ **Category:** Beginners\ **Created:** [October 22, 2025, 11:19am UTC](https://discuss.huggingface.co/t/qlora-model-isnt-training/169337 "2025-10-22T11:19:32Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![antbartash](https://avatars.discourse-cdn.com/v4/letter/a/46a35a/32.png) [@antbartash](https://discuss.huggingface.co/u/antbartash)\ **Post date:** [October 22, 2025, 11:19am UTC](https://discuss.huggingface.co/t/qlora-model-isnt-training/169337/1 "2025-10-22T11:19:32Z")
Hi everyone, I’ve been trying to switch from LoRA to QLoRA on an Nvidia T4, but I’m running into an issue where the evaluation loss stays completely flat, while the training loss fluctuates around its initial value. My LoRA setup works fine, but adding `bnb_config`, `model.gradient_checkpointing_enable()`, and `model = prepare_model_for_kbit_training(model)` causes the issue described above. ![1000000396](https://us1.discourse-cdn.com/hellohellohello/original/3X/5/d/5d755be17cacac8fc8637104730fdb9b8cb38d49.jpeg) Since the non-quantized version runs without problems, I don’t think the issue is related to the LoRA config, dataset, or formatting functions. The number of trainable parameters is non-zero for both the LoRA and QLoRA setups. Below is the code I’m using for QLoRA. Any help would be appreciated! ```python ds_train_with_assistant_content = ds_train.map(construct_message_with_assistant_content) ds_valid_with_assistant_content = ds_valid.map(construct_message_with_assistant_content) bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_use_double_quant=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16 ) checkpoint = "Qwen/Qwen3-0.6B" tokenizer = AutoTokenizer.from_pretrained(checkpoint) model = AutoModelForCausalLM.from_pretrained( checkpoint, device_map="auto", quantization_config=bnb_config ) model.config.use_cache = False model.gradient_checkpointing_enable() model = prepare_model_for_kbit_training(model) model.enable_input_require_grads() timestamp = datetime.now().strftime('%Y-%m-%d_%H-%M-%S') RUN_NAME = f'qlora-final-model-all-linear-r64-{timestamp}' wandb.init( project=os.environ["WANDB_PROJECT"], name=RUN_NAME, # id=run_id, # resume previous run if available resume="allow", # allows resuming crashed run ) RESUME_TRAINING = False OUTPUT_DIR = "./qlora-final_model_all_linear_r64-output" PER_DEVICE_BATCH_SIZE = 2 # higher values --> OOM optimizer = 'paged_adamw_8bit' effective_batch_size = 16 learning_rate = 1e-5 weight_decay = 0.0 betas = (0.9, 0.9999) warmup_ratio = 0.2 epochs = 1 gradient_accumulation_steps = int(effective_batch_size / PER_DEVICE_BATCH_SIZE) lora_r = 16*4 lora_alpha = 64*4 lora_dropout = 0.01 training_args = TrainingArguments( output_dir=OUTPUT_DIR, per_device_train_batch_size=PER_DEVICE_BATCH_SIZE, gradient_accumulation_steps=gradient_accumulation_steps, learning_rate=learning_rate, optim=optimizer, num_train_epochs=epochs, weight_decay=weight_decay, lr_scheduler_type="cosine", warmup_ratio=warmup_ratio, save_strategy="steps", save_steps=gradient_accumulation_steps*5, save_total_limit=2, eval_strategy="steps", eval_steps=gradient_accumulation_steps*5, logging_strategy="steps", logging_steps=gradient_accumulation_steps*5, report_to=['wandb'], run_name=RUN_NAME, bf16=True, # fp16=True, # fp16_full_eval=True, metric_for_best_model="eval_loss", greater_is_better=False, max_grad_norm=1, load_best_model_at_end=True, gradient_checkpointing=True, gradient_checkpointing_kwargs={"use_reentrant": False} ) peft_config = LoraConfig( r=lora_r, lora_alpha=lora_alpha, lora_dropout=lora_dropout, bias="none", task_type="CAUSAL_LM", target_modules='all-linear' ) # model.requires_grad_(False) # freeze base weights (precautionary) model_peft = get_peft_model(model, peft_config) # inject a LoRA adapter print_trainable_parameters(model_peft) trainer = SFTTrainer( model=model_peft, train_dataset=ds_train_with_assistant_content, eval_dataset=ds_valid_with_assistant_content, formatting_func=formatting_func, args=training_args, callbacks=[EarlyStoppingCallback(early_stopping_patience=25)] ) # Training setup summary dataset_size = len(ds_train_with_assistant_content) steps_per_epoch = dataset_size // (PER_DEVICE_BATCH_SIZE * gradient_accumulation_steps) total_steps = steps_per_epoch * epochs warmup_steps = int(total_steps * warmup_ratio) print("===== Training Setup Summary =====") print(f"Num epochs: {epochs}") print(f"Effective batch size: {effective_batch_size}") print(f"Per-device batch size: {PER_DEVICE_BATCH_SIZE}") print(f"Gradient accumulation: {gradient_accumulation_steps}") print(f"Dataset size: {dataset_size}") print(f"Steps per epoch: {steps_per_epoch}") print(f"Total training steps: {total_steps}") print(f"Warmup steps: {warmup_steps}") print(f"Logging steps: {training_args.logging_steps}") print("===================================") print(f"Start time: {datetime.now().strftime('%Y-%m-%d_%H-%M-%S')}") # Training last_checkpoint = None if RESUME_TRAINING and os.path.isdir(OUTPUT_DIR): last_checkpoint = get_last_checkpoint(OUTPUT_DIR) if last_checkpoint is not None: print(f"Resuming training from checkpoint: {last_checkpoint}") trainer.train(resume_from_checkpoint=last_checkpoint) else: print("Starting fresh training run") trainer.train() print(f"End time: {datetime.now().strftime('%Y-%m-%d_%H-%M-%S')}") # WandB logging of eval metrics for log in trainer.state.log_history: if 'eval_loss' in log: wandb.log({ "eval_loss": log['eval_loss'], "eval_perplexity": math.exp(log['eval_loss']), "step": log['step'], "learning_rate": learning_rate, "weight_decay": weight_decay, "betas": betas, "warmup_ratio": warmup_ratio, "effective_batch_size": effective_batch_size, "optimizer": optimizer }) wandb.finish() # finish the run ``` ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [October 22, 2025, 12:52pm UTC](https://discuss.huggingface.co/t/qlora-model-isnt-training/169337/2 "2025-10-22T12:52:50Z")
> Nvidia T4 Since T4 doesn’t natively support `torch.bfloat16`, using `torch.float16`/ `fp16=True` instead might resolve the error. No other major issues appear to exist. ---
**Author:** ![antbartash](https://avatars.discourse-cdn.com/v4/letter/a/46a35a/32.png) [@antbartash](https://discuss.huggingface.co/u/antbartash)\ **Post date:** [October 23, 2025, 7:19am UTC](https://discuss.huggingface.co/t/qlora-model-isnt-training/169337/3 "2025-10-23T07:19:01Z")
Thanks for the suggestion It turned out the issue was environment-related — I was able to get the expected results using the exact same code on Colab. In my local environment, clearing the caches for transformers, torch, etc., and upgrading all the libraries resolved the problem. ---
**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\ **Post date:** [October 24, 2025, 6:16pm UTC](https://discuss.huggingface.co/t/qlora-model-isnt-training/169337/4 "2025-10-24T18:16:57Z")
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.