# Fine-Tuning LLMs on Large Proprietary Codebases

**URL:** <https://discuss.huggingface.co/t/fine-tuning-llms-on-large-proprietary-codebases/155828>\
**Category:** Models\
**Created:** [May 19, 2025, 5:17am UTC](https://discuss.huggingface.co/t/fine-tuning-llms-on-large-proprietary-codebases/155828 "2025-05-19T05:17:04Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![anon82315112](https://avatars.discourse-cdn.com/v4/letter/a/f04885/32.png) [@anon82315112](https://discuss.huggingface.co/u/anon82315112)\
**Post date:** [May 19, 2025, 5:17am UTC](https://discuss.huggingface.co/t/fine-tuning-llms-on-large-proprietary-codebases/155828/1 "2025-05-19T05:17:04Z")

</div>

I’m currently fine-tuning a large language model (LLM) on a proprietary codebase. The fine-tuning process itself has completed without technical issues, but the performance of the resulting model is very poor—its responses are largely irrelevant, even when asked questions that are directly taken from the training dataset.

The objective of this fine-tuning effort is to enable the model to assist with tasks related to the private codebase, such as code generation, code explanation, and guidance on using internal APIs. From my observations, the real challenge lies in how the dataset is being prepared.

Currently, my dataset generation process is relatively straightforward: I traverse the codebase and use a local deployment of the `qwne3-235b` model to generate question-answer pairs for individual code files. For security reasons, only locally deployed models are used. Here’s the prompt I’m using:

```plaintext
# Role Tasks
You are a senior game engine code analysis expert who is good at understanding complex game engine code structure, core algorithms and system architecture. Your task is to analyze the company's internal engine code provided and generate high-quality training data for large model fine-tuning.

## Core Tasks
Based on the provided code, generate no less than {question_num} high-quality technical questions, as well as corresponding comprehensive, accurate and professional answers.

## Analysis focus
1. Code structure analysis
2. Core function implementation
3. Algorithms and data structures
4. API usage
5. Common problem solving

## Question type guide
- Code interpretation questions
- Function search questions
- Architecture design questions
- Optimization-related questions

## Output restrictions
Please output in JSON list format following the Alpaca training structure:
[
  {
    "instruction": "xxx",
    "input": "xxx",
    "output": "xxx"
  },
  ...
]

## Constraints (important!)
- Must be grounded in code content; avoid speculation
- Questions should be factually answerable from the code
- Cover both high-level and low-level aspects
- No hypothetical or vague questions
- Each instruction must have unique, factual value
- Reference specific classes/functions/files and include code snippets where needed
- Avoid generating questions about programming language syntax itself
- For code explanations, use implementation-heavy code snippets
- Instructions should sound like real developer questions
- Keep `input` empty if unnecessary

```

I only used the qwne3-235b for data generation, while the fine-tuning was based on the very small qwen2.5 model. It’s still just some experimentation on very small models, and shouldn’t be applied to larger models until there are good results.

The codebase is large and spans multiple programming languages. So far, I’ve only been feeding individual files into the model. I’ve considered including related files as context, but that presents two main challenges:

1. Code parsing becomes extremely complex
2. The model’s context window gets exceeded quickly

I also experimented with generating training data from Git commit diffs and messages, slightly modifying the prompt to suit that format. However, like the file-based method, the resulting datasets don’t appear to be of high quality.

I’ve tried pre-training on the private codebase before fine-tuning, but this didn’t yield noticeable improvements either.

From what I’ve seen in the community, most discussions focus on dataset creation from documents. There’s far less guidance on dataset generation from codebases—especially large, complex, multi-language ones. Most existing practices seem geared toward code pre-training or FIM (Fill In the Middle) tasks.

I’m not sure whether my current methodology is flawed or if I need to significantly revise my approach. Any guidance would be greatly appreciated:

- Is my current approach to dataset construction fundamentally sound?
- Are there improvements I should make to the prompt or generation method?
- Why might the fine-tuned model be performing so poorly despite the seemingly relevant dataset?

I’m continuing to experiment and will update here with any new progress. Thanks in advance for any help.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [May 19, 2025, 8:32am UTC](https://discuss.huggingface.co/t/fine-tuning-llms-on-large-proprietary-codebases/155828/2 "2025-05-19T08:32:44Z")

</div>

Let’s put aside the creation of the dataset for now. It is possible that the weights of Qwen 3 or the implementation of Transformers are not yet complete.

I think it would be quick to test the suitability of the dataset by training a very small, reasonably well-established model on the same dataset to see if it produces results.

> **[Qwen/Qwen3-235B-A22B · The Qwen3-235B-A22B model is not as effective as the...](https://huggingface.co/Qwen/Qwen3-235B-A22B/discussions/35)**
>
> I evaluated the models in the Qwen3 series based on Humaneval and found that the Qwen3-32B model performs better than the larger 235B model, with the non-thinking results even better than the think...

> **[🚀\[Fine-tuning\] Qwen3-MoE Megatron Training Implementation and Best Practices👋...](https://github.com/QwenLM/Qwen3/discussions/1301)**
>
> 中文版notebook: https://modelscope.cn/notebook/share/ipynb/d4d8765f/qwen3.ipynb Hello, everyone! We are thrilled to hear about the open-source release of Qwen3 and Qwen3-MoE. The ms-swift large model ...

> <https://github.com/unslothai/unsloth/issues/2072>
>
> I tried to train Qwen-2.5-3b with GRPO on the LIMO dataset using revised code fr…om the colab example: https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1\_(8B)-GRPO.ipynb
> 
> However, the context length can not be scaled up above 1.5k due to OOM errors, which is not an expected situation considering the blog (https://unsloth.ai/blog/grpo) using 54.3GB of VRAM for Llama 3.1 8B.
> 
> The revised code I use collects VRAM usage messages which shows that the maximum allocation of VRAM by pytorch is far below 48GB.
> The "RuntimeError: CUDA error: out of memory" does not prompt the allocation behavior that causes this error with only a confusing stack trace pointing to a non-CUDA operation, which is shown in the debugging messages below at the end of this description. 
> 
> I am not familiar with the multi-worker mechanism of Pytorch could you please kindly point out what I have done wrong?
> 
> I would appreciate any of your responses. Thanks.
>  
> Code snip pasted below:
> 
> \`\`\`python
> from unsloth import FastLanguageModel
> import torch
> import gc
> from functools import wraps
> import time
> from UnslothGRPOTrainer import (
> UnslothGRPOConfig as GRPOConfig,
> UnslothGRPOTrainer as GRPOTrainer,
> )
> from transformers import TrainerCallback
> import numpy as np
> from functools import wraps
> import re
> from datasets import load\_dataset, Dataset
> 
> \# Load and prep dataset
> SYSTEM\_PROMPT = """
> Respond in the following format:
> \<reasoning\>
> ...
> \</reasoning\>
> \<answer\>
> ...
> \</answer\>
> """
> 
> XML\_COT\_FORMAT = """\\
> \<reasoning\>
> {reasoning}
> \</reasoning\>
> \<answer\>
> {answer}
> \</answer\>
> """
> def extract\_hash\_answer(text: str) -\> str | None:
> if "####" not in text:
> return None
> return text.split("####")\[1\].strip()
> 
> def track\_memory(func):
> @wraps(func)
> def wrapper(\*args, \*\*kwargs):
> print(f"\\n=== Entering {func.\_\_name\_\_} ===")
> print(f"Memory before: {torch.cuda.memory\_allocated() / 1024\*\*3:.2f} GB")
> print(f"Memory reserved before: {torch.cuda.memory\_reserved() / 1024\*\*3:.2f} GB")
>         
> start\_time = time.time()
> result = func(\*args, \*\*kwargs)
> end\_time = time.time()
>         
> print(f"Time taken: {end\_time - start\_time:.2f} seconds")
> print(f"Memory after: {torch.cuda.memory\_allocated() / 1024\*\*3:.2f} GB")
> print(f"Memory reserved after: {torch.cuda.memory\_reserved() / 1024\*\*3:.2f} GB")
> print(f"=== Exiting {func.\_\_name\_\_} ===\\n")
>         
> return result
> return wrapper
> 
> 
> def get\_gsm8k\_questions(split = "train") -\> Dataset:
> # data = load\_dataset('openai/gsm8k', 'main')\[split\] # type: ignore
> data = load\_dataset('GAIR/LIMO')\[split\] # type: ignore
> data = data.map(lambda x: { # type: ignore
> 'prompt': \[
> {'role': 'system', 'content': SYSTEM\_PROMPT},
> {'role': 'user', 'content': x\['question'\]}
> \],
> # 'answer': extract\_hash\_answer(x\['answer'\])
> 'answer': x\['answer'\]
> }) # type: ignore
> return data # type: ignore
> 
> 
> @track\_memory
> def load\_dataset\_with\_tracking():
> return get\_gsm8k\_questions()
> 
> dataset = load\_dataset\_with\_tracking()
> 
> 
> max\_seq\_length = 4096 \* 2 # Can increase for longer reasoning traces
> lora\_rank = 32 # Larger rank = smarter, but slower
> max\_prompt\_length = 512 # Moved here
> 
> model, tokenizer = FastLanguageModel.from\_pretrained(
> model\_name = "Qwen/Qwen2.5-3B-Instruct",
> max\_seq\_length = max\_seq\_length,
> load\_in\_4bit = True, # False for LoRA 16bit
> fast\_inference = True, # Enable vLLM fast inference
> max\_lora\_rank = lora\_rank,
> gpu\_memory\_utilization = 0.10, # Reduce if out of memory
> float8\_kv\_cache=True
> )
> 
> model = FastLanguageModel.get\_peft\_model(
> model,
> r = lora\_rank, # Choose any number \> 0 ! Suggested 8, 16, 32, 64, 128
> target\_modules = \[
> # "q\_proj", "k\_proj", "v\_proj", "o\_proj",
> "gate\_proj", "up\_proj", "down\_proj",
> \], # Remove QKVO if out of memory
> lora\_alpha = lora\_rank,
> # use\_gradient\_checkpointing=False,
> use\_gradient\_checkpointing = "unsloth", # Enable long context finetuning
> random\_state = 3407,
> # float8\_kv\_cache=True
> )
> 
> 
> def extract\_xml\_answer(text: str) -\> str:
> answer = text.split("\<answer\>")\[-1\]
> answer = answer.split("\</answer\>")\[0\]
> return answer.strip()
> 
> 
> 
> \# uncomment middle messages for 1-shot prompting
> 
> 
> \# Modify training configuration
> training\_args = GRPOConfig(
> learning\_rate = 5e-6,
> adam\_beta1 = 0.9,
> adam\_beta2 = 0.99,
> weight\_decay = 0.1,
> warmup\_ratio = 0.1,
> lr\_scheduler\_type = "cosine",
> optim = "paged\_adamw\_8bit",
> logging\_steps = 1,
> per\_device\_eval\_batch\_size = 2,
> per\_device\_train\_batch\_size = 1,
> gradient\_accumulation\_steps = 1, # Increased for better memory management
> num\_generations = 4, # Reduced to help identify memory issues
> max\_prompt\_length = max\_prompt\_length,
> max\_completion\_length = max\_seq\_length - max\_prompt\_length,
> max\_steps = 1000000, # Reduced for testing
> save\_steps = 50,
> max\_grad\_norm = 0.1,
> report\_to = "none",
> output\_dir = "outputs",
> )
> 
> \# Add memory tracking before model initialization
> print("\\n=== Model Initialization Memory Usage ===")
> print(f"Memory before model init: {torch.cuda.memory\_allocated() / 1024\*\*3:.2f} GB")
> print(f"Memory reserved before model init: {torch.cuda.memory\_reserved() / 1024\*\*3:.2f} GB")
> 
> \# Add memory tracking for dataset loading
> 
> 
> \# Reward functions
> def correctness\_reward\_func(prompts, completions, answer, \*\*kwargs) -\> list\[float\]:
> responses = \[completion\[0\]\['content'\] for completion in completions\]
> q = prompts\[0\]\[-1\]\['content'\]
> extracted\_responses = \[extract\_xml\_answer(r) for r in responses\]
> print('-'\*20, f"Question:\\n{q}", f"\\nAnswer:\\n{answer\[0\]}", f"\\nResponse:\\n{responses\[0\]}", f"\\nExtracted:\\n{extracted\_responses\[0\]}")
> return \[2.0 if r == a else 0.0 for r, a in zip(extracted\_responses, answer)\]
> 
> def int\_reward\_func(completions, \*\*kwargs) -\> list\[float\]:
> responses = \[completion\[0\]\['content'\] for completion in completions\]
> extracted\_responses = \[extract\_xml\_answer(r) for r in responses\]
> return \[0.5 if r.isdigit() else 0.0 for r in extracted\_responses\]
> 
> def strict\_format\_reward\_func(completions, \*\*kwargs) -\> list\[float\]:
> """Reward function that checks if the completion has a specific format."""
> pattern = r"^\<reasoning\>\\n.\*?\\n\</reasoning\>\\n\<answer\>\\n.\*?\\n\</answer\>\\n$"
> responses = \[completion\[0\]\["content"\] for completion in completions\]
> matches = \[re.match(pattern, r) for r in responses\]
> return \[0.5 if match else 0.0 for match in matches\]
> 
> def soft\_format\_reward\_func(completions, \*\*kwargs) -\> list\[float\]:
> """Reward function that checks if the completion has a specific format."""
> pattern = r"\<reasoning\>.\*?\</reasoning\>\\s\*\<answer\>.\*?\</answer\>"
> responses = \[completion\[0\]\["content"\] for completion in completions\]
> matches = \[re.match(pattern, r) for r in responses\]
> return \[0.5 if match else 0.0 for match in matches\]
> 
> def count\_xml(text) -\> float:
> count = 0.0
> if text.count("\<reasoning\>\\n") == 1:
> count += 0.125
> if text.count("\\n\</reasoning\>\\n") == 1:
> count += 0.125
> if text.count("\\n\<answer\>\\n") == 1:
> count += 0.125
> count -= len(text.split("\\n\</answer\>\\n")\[-1\])\*0.001
> if text.count("\\n\</answer\>") == 1:
> count += 0.125
> count -= (len(text.split("\\n\</answer\>")\[-1\]) - 1)\*0.001
> return count
> 
> def xmlcount\_reward\_func(completions, \*\*kwargs) -\> list\[float\]:
> contents = \[completion\[0\]\["content"\] for completion in completions\]
> return \[count\_xml(c) for c in contents\]
> 
> \# Add memory tracking to key functions
> @track\_memory
> def compute\_loss(self, \*args, \*\*kwargs):
> return super().compute\_loss(\*args, \*\*kwargs)
> 
> \# Modify the trainer class to include memory tracking
> class MemoryTrackedGRPOTrainer(GRPOTrainer):
> def \_\_init\_\_(self, \*args, \*\*kwargs):
> super().\_\_init\_\_(\*args, \*\*kwargs)
> self.compute\_loss = track\_memory(self.compute\_loss)
> 
> \# Replace the trainer instantiation with the memory tracked version
> trainer = MemoryTrackedGRPOTrainer(
> model = model,
> processing\_class = tokenizer,
> reward\_funcs = \[
> xmlcount\_reward\_func,
> soft\_format\_reward\_func,
> strict\_format\_reward\_func,
> int\_reward\_func,
> correctness\_reward\_func,
> \],
> args = training\_args,
> train\_dataset = dataset,
> )
> 
> \# Add memory tracking to the training loop
> @track\_memory
> def train\_with\_memory\_tracking():
> trainer.train()
> 
> \# Replace trainer.train() with the tracked version
> train\_with\_memory\_tracking()
> 
> text = tokenizer.apply\_chat\_template(\[
> {"role" : "user", "content" : "Which is bigger? 9.11 or 9.9?"},
> \], tokenize = False, add\_generation\_prompt = True)
> \`\`\`
> 
> The debugging messages:
> 
> \`\`\`
> 
> === Model Initialization Memory Usage ===
> Memory before model init: 3.93 GB
> Memory reserved before model init: 4.02 GB
> 
> === Entering train\_with\_memory\_tracking ===
> Memory before: 3.93 GB
> Memory reserved before: 4.02 GB
> 
> 0%| | 0/1000000 \[00:00\<?, ?it/s\]
> 0%| | 1/1000000 \[00:14\<4033:48:31, 14.52s/it\]
>                                                         
> 
> 0%| | 1/1000000 \[00:14\<4033:48:31, 14.52s/it\]\[rank0\]: Traceback (most recent call last):
> \[rank0\]: File "\<frozen runpy\>", line 198, in \_run\_module\_as\_main
> \[rank0\]: File "\<frozen runpy\>", line 88, in \_run\_code
> \[rank0\]: File "/home/liangchen/ts\_agent/train\_phi4.py", line 236, in \<module\>
> \[rank0\]: train\_with\_memory\_tracking()
> \[rank0\]: File "/home/liangchen/ts\_agent/train\_phi4.py", line 48, in wrapper
> \[rank0\]: result = func(\*args, \*\*kwargs)
> ...
> \[rank0\]: File "/home/liangchen/miniconda3/envs/ts\_agent/lib/python3.11/site-packages/unsloth\_zoo/gradient\_checkpointing.py", line 767, in unsloth\_checkpoint
> \[rank0\]: return UnslothCheckpointFunction.apply(function, preserve, \*args)
> \[rank0\]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> \[rank0\]: File "/home/liangchen/miniconda3/envs/ts\_agent/lib/python3.11/site-packages/torch/autograd/function.py", line 575, in apply
> \[rank0\]: return super().apply(\*args, \*\*kwargs) # type: ignore\[misc\]
> \[rank0\]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> \[rank0\]: File "/home/liangchen/miniconda3/envs/ts\_agent/lib/python3.11/site-packages/unsloth\_zoo/gradient\_checkpointing.py", line 441, in forward
> \[rank0\]: if new\_size \> x.numel(): x.resize\_(new\_size)
> \[rank0\]: ^^^^^^^^^^^^^^^^^^^
> \[rank0\]: RuntimeError: CUDA error: out of memory
> \[rank0\]: CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
> \[rank0\]: For debugging consider passing CUDA\_LAUNCH\_BLOCKING=1
> \[rank0\]: Compile with \`TORCH\_USE\_CUDA\_DSA\` to enable device-side assertions.
> 
> \-------------------- Question:
> Triangle $ABC$ has $AB=21$ ,
> ...
> === Entering compute\_loss ===
> Memory before: 4.05 GB
> Memory reserved before: 4.97 GB
> input\_ids.shape:torch.Size(\[4, 1040\])
> \[EOF\]
> \`\`\`

---

<div class="post-metadata">

**Author:** ![anon82315112](https://avatars.discourse-cdn.com/v4/letter/a/f04885/32.png) [@anon82315112](https://discuss.huggingface.co/u/anon82315112)\
**Post date:** [May 19, 2025, 9:07am UTC](https://discuss.huggingface.co/t/fine-tuning-llms-on-large-proprietary-codebases/155828/4 "2025-05-19T09:07:05Z")

</div>

Thank you for your response. I apologize for not explaining this clearly earlier. Currently, I only used the qwne3-235b for data generation, while the fine-tuning was based on the very small qwen2.5 model. It’s still just some experimentation on very small models, and shouldn’t be applied to larger models until there are good results

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [May 19, 2025, 9:16am UTC](https://discuss.huggingface.co/t/fine-tuning-llms-on-large-proprietary-codebases/155828/5 "2025-05-19T09:16:26Z")

</div>

I think that’s a good approach. In that case, it might be an issue with the dataset or the Trainer settings themselves…🤔

It might be a good idea to ask on Hugging Face’s Discord or unsloth’s Discord.  
Even if it’s not documented in articles or documentation, it’s common for people who know the ins and outs to be willing to share their expertise.

If the fine-tuning know-how is publicly available online, we can quickly summarize it using ChatGPT or Gemini, but for things that aren’t publicly available, they’re not very helpful…

* * *

by Hugging Chat: [HuggingChat](https://huggingface.co/chat/)

### **Summary of Key Recommendations**

Your model’s poor performance stems from **fragmented context** and **low-quality QA pairs** in training data. Here’s how to fix it:

* * *

#### **1. Core Problems**

- **Isolated Code Files** : Training on single files prevents systemic understanding (e.g., API interactions, architecture).
- **Shallow Questions** : Generated QA pairs focus on syntax rather than real tasks (debugging, optimization).
- **Self-Supervised Flaws** : Using a model to generate its own training data compounds errors.
- **Empty Input Fields** : Model isn’t trained to use external context (e.g., code diffs, error logs).

* * *

#### **2. Critical Fixes**

- **Context-Aware QA** :

- **Task-Driven Questions** :

- **RAG-Augmented Training** :

- **Data Quality Control** :

* * *

#### **3. Training Adjustments**

- **Curriculum Learning** :

- **Loss Masking** :

- **Evaluation Metrics** :

* * *

#### **4. Implementation Roadmap**

1. **Weeks 1-2** : Map code dependencies and generate cross-file QA pairs.
2. **Weeks 3-4** : Redesign prompts for task-driven QA and validate data.
3. **Weeks 5-6** : Implement RAG-augmented training with retrieval context.
4. **Weeks 7-8** : Add human evaluation and custom metrics.

* * *

**Final Takeaway** :  
Your current approach lacks **systemic understanding** and **real-task alignment**. Shift from isolated file analysis to **context-aware, task-specific training** (e.g., debugging, optimization) using cross-file code dependencies and RAG. This will enable the model to reason about the entire codebase, not just individual files.

---

<div class="post-metadata">

**Author:** ![anon82315112](https://avatars.discourse-cdn.com/v4/letter/a/f04885/32.png) [@anon82315112](https://discuss.huggingface.co/u/anon82315112)\
**Post date:** [May 19, 2025, 9:35am UTC](https://discuss.huggingface.co/t/fine-tuning-llms-on-large-proprietary-codebases/155828/6 "2025-05-19T09:35:42Z")

</div>

I’ll try to ask in Discord. I’ve tried asking ChatGPT Gemini Grok …, also use Deep Research, but haven’t found a better resolution yet. 🥹  
Anyway, thanks for your answer!

---

<div class="post-metadata">

**Author:** ![Pimpcat-AU](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/pimpcat-au/32/48989_2.png) [@Pimpcat-AU](https://discuss.huggingface.co/u/Pimpcat-AU)\
**Post date:** [June 10, 2025, 8:14pm UTC](https://discuss.huggingface.co/t/fine-tuning-llms-on-large-proprietary-codebases/155828/7 "2025-06-10T20:14:55Z")

</div>

Fix:

```
Your current approach (auto-generating Q&A pairs from single files) produces synthetic data with low diversity and context loss.

For better results:

    Include neighboring/related files in context windows (where feasible).

    Add real human-written questions/answers or high-quality curated examples, not just auto-generated ones.

    Mix in Fill-In-the-Middle or code-completion style samples, not only Q&A format.

    Filter/generated data for relevance/quality, and balance with some language-only (docstring, commit) data.

Model performance is poor because synthetic Q&A lacks true variation and may not match real user queries or context complexity.

```

Solution provided by Triskel Data Deterministic AI.

---

<div class="post-metadata">

**Author:** ![Gayanukaa](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/gayanukaa/32/50028_2.png) [@Gayanukaa](https://discuss.huggingface.co/u/Gayanukaa)\
**Post date:** [June 24, 2025, 3:10pm UTC](https://discuss.huggingface.co/t/fine-tuning-llms-on-large-proprietary-codebases/155828/8 "2025-06-24T15:10:48Z")

</div>

Hi, I’m in a similar situation to finetune a model on a private codebase for guidance on using internal APIs

I want to check if it’s possible to change on the code files itself without generating QA pairs.  
With QA pairs is also fine if the outputs are good but more lenient towards without QA pairs.

Were you successful in achieving good results where the methodology could be shared. Greatly appreciated!

---

<div class="post-metadata">

**Author:** ![anon82315112](https://avatars.discourse-cdn.com/v4/letter/a/f04885/32.png) [@anon82315112](https://discuss.huggingface.co/u/anon82315112)\
**Post date:** [June 24, 2025, 3:32pm UTC](https://discuss.huggingface.co/t/fine-tuning-llms-on-large-proprietary-codebases/155828/9 "2025-06-24T15:32:55Z")

</div>

We made some further attempts, but the results were not very good. Therefore, we did not continue. We are currently experimenting with a solution that indexes code based on questions and then provides it as context to LLM.

We are also simultaneously generating question-answer pairs based on internal documents for model fine-tuning, and this is progressing well with good results.

---

<div class="post-metadata">

**Author:** ![Gayanukaa](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/gayanukaa/32/50028_2.png) [@Gayanukaa](https://discuss.huggingface.co/u/Gayanukaa)\
**Post date:** [June 24, 2025, 3:38pm UTC](https://discuss.huggingface.co/t/fine-tuning-llms-on-large-proprietary-codebases/155828/10 "2025-06-24T15:38:05Z")

</div>

Any technique or sample repository you can share on the - indexes code based on questions and then provides it as context to LLM.

I would like to try that as well 🙌

---

<div class="post-metadata">

**Author:** ![anon82315112](https://avatars.discourse-cdn.com/v4/letter/a/f04885/32.png) [@anon82315112](https://discuss.huggingface.co/u/anon82315112)\
**Post date:** [June 24, 2025, 3:54pm UTC](https://discuss.huggingface.co/t/fine-tuning-llms-on-large-proprietary-codebases/155828/11 "2025-06-24T15:54:51Z")

</div>

The project we are referencing is [cline](https://github.com/cline/cline)

---

<div class="post-metadata">

**Author:** ![Issacluffy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/issacluffy/32/55274_2.png) [@Issacluffy](https://discuss.huggingface.co/u/Issacluffy)\
**Post date:** [October 31, 2025, 5:43am UTC](https://discuss.huggingface.co/t/fine-tuning-llms-on-large-proprietary-codebases/155828/12 "2025-10-31T05:43:54Z")

</div>

So I am also working on the same topic,

I tried different ways like for creation of dataset, due to constraints of resources i used a parser for .py,.xml,.js formats which are majority of the files so i prepared a hardcoded dataset like write a py file on so and so and output will also be there . I tried fine tuning on the dataset and training loss and validation loss are also good but when checking the inference results those are not as good as i thought.

Now trying to do it by rag approach

if any found out the way to do or if i am doing wrong please do let me know .

Thanks