# Finetuned LLM model conversion to GGUF - performance drop **URL:** \ **Category:** Models\ **Created:** [June 20, 2024, 11:41am UTC](https://discuss.huggingface.co/t/finetuned-llm-model-conversion-to-gguf-performance-drop/93179 "2024-06-20T11:41:03Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![tprochenka](https://avatars.discourse-cdn.com/v4/letter/t/e274bd/32.png) [@tprochenka](https://discuss.huggingface.co/u/tprochenka)\ **Post date:** [June 20, 2024, 11:41am UTC](https://discuss.huggingface.co/t/finetuned-llm-model-conversion-to-gguf-performance-drop/93179/1 "2024-06-20T11:41:03Z")
Hi, I’m finetunning LLM on my data using SFTTrainer, bitsandbytes quatization and peft with configs like listed below. When I convert the model to GGUF for CPU inference, the model performance significantly drops. Any idea what could be a problem? ```auto bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.float16, ) peft_config = LoraConfig( lora_alpha=16, lora_dropout=0.1, r=16, bias="none", task_type="CAUSAL_LM" ) ``` I do conversion to gguf in the following way. First, merge trained adapter with base model. Then such merged model is converted to gguf using llama.cpp, ‘convert.py’ script, I do q8\_0 quantization, tested other types without success. I tested as well conversion using unsloath, as well w/o positive result. ```auto python convert.py \ --outfile \ --outtype q8_0 \ --vocab_dir ``` ---
**Author:** ![iamz80](https://avatars.discourse-cdn.com/v4/letter/i/f05b48/32.png) [@iamz80](https://discuss.huggingface.co/u/iamz80)\ **Post date:** [June 21, 2024, 5:49pm UTC](https://discuss.huggingface.co/t/finetuned-llm-model-conversion-to-gguf-performance-drop/93179/3 "2024-06-21T17:49:53Z")
Hello! I have two concerns regarding your setup. 1. Are you sure alpha=16 isn’t too much? 2. When you merge, do you merge with the original non-quantized model or with a quantized version? My concern is that if LoRA is trained with the quantized version but merged with the original model, the model output might not be the same. And when it is quantized to GGUF format it adds on top of that additional error. And the two stacked together might be enough to degrade the output considerably. Thank you! ---
**Author:** ![meowman](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/meowman/32/28718_2.png) [@meowman](https://discuss.huggingface.co/u/meowman)\ **Post date:** [June 26, 2024, 10:01pm UTC](https://discuss.huggingface.co/t/finetuned-llm-model-conversion-to-gguf-performance-drop/93179/4 "2024-06-26T22:01:18Z")
That’s because you quantized the model. Quantization reduces the precision of the model’s parameters from floating-point representations to lower-bit representations. Lower bit numbers mean using fewer digits to represent a number. It’s like when you count from 1 to 10 instead of 1 to 100. So, lower bit numbers are simpler and take up less space! ---
**Author:** ![tprochenka](https://avatars.discourse-cdn.com/v4/letter/t/e274bd/32.png) [@tprochenka](https://discuss.huggingface.co/u/tprochenka)\ **Post date:** [July 31, 2024, 6:11am UTC](https://discuss.huggingface.co/t/finetuned-llm-model-conversion-to-gguf-performance-drop/93179/5 "2024-07-31T06:11:37Z")
HI @iamz80, thank you for your insights and sorry for the late response. ad 1. I tested different configurations of alpha and r - and it is ok. When is a best setup in your opinion? I saw different opinions. ad 2. When I fine tune a model a load a base model with quantization: ```auto model = AutoPeftModelForCausalLM.from_pretrained( model_path, quantization_config=bnb_config, device_map={"": 0}, trust_remote_code=True, use_cache=True, is_trainable=False, ) ``` Then adapter is bf16 or tf32: ```auto args = TrainingArguments( output_dir=OUTPUT_MODEL_DIR, num_train_epochs=ft_parameters.EPOCHS, per_device_train_batch_size=ft_parameters.BATCH_SIZE, # 4 --> 2 becouse gpu per_device_eval_batch_size=1, gradient_accumulation_steps=ft_parameters.ACCUMULATION_STEP, gradient_checkpointing=True, optim="paged_adamw_32bit", logging_steps=10, save_strategy="epoch", learning_rate=ft_parameters.LEARINIG_RATE, bf16=True, tf32=True, max_grad_norm=0.3, warmup_ratio=0.03, lr_scheduler_type="constant", evaluation_strategy="epoch" if ft_parameters.DO_EVAL else "no", report_to="none", disable_tqdm=False, # disable tqdm since with packing values are in correct full_determinism=True, ) ``` Then when I load a model for merging I do the following I load it in bf16 as I thought it could only help - but your point is that it could deteriorate performance? ```auto model = AutoPeftModelForCausalLM.from_pretrained( lora_adapter_dir, device_map=device_map, torch_dtype=torch.bfloat16 ) model = model.merge_and_unload() ``` So it you opinion adding “quantization\_config=bnb\_config,” to from\_pretrained when merging could help? But then I will quantize as well adapter which was in higher precision so far… And probably the best option is to take bigger machine and finetune without any quantization then we avoid double quantization? I tested that quantization at gguf conversion is needed because w/o it the model has endless inference times… ---
**Author:** ![tprochenka](https://avatars.discourse-cdn.com/v4/letter/t/e274bd/32.png) [@tprochenka](https://discuss.huggingface.co/u/tprochenka)\ **Post date:** [July 31, 2024, 6:13am UTC](https://discuss.huggingface.co/t/finetuned-llm-model-conversion-to-gguf-performance-drop/93179/6 "2024-07-31T06:13:18Z")
Hi @meowman, if I do a conversion without quantization model is huge and inference times are endless…