# How to fine tune fine tune GitHub Copilot?

**URL:** <https://discuss.huggingface.co/t/how-to-fine-tune-fine-tune-github-copilot/18889>\
**Category:** Research\
**Created:** [June 9, 2022, 4:26am UTC](https://discuss.huggingface.co/t/how-to-fine-tune-fine-tune-github-copilot/18889 "2022-06-09T04:26:50Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![neo-benjamin](https://avatars.discourse-cdn.com/v4/letter/n/f6c823/32.png) [@neo-benjamin](https://discuss.huggingface.co/u/neo-benjamin)\
**Post date:** [June 9, 2022, 4:26am UTC](https://discuss.huggingface.co/t/how-to-fine-tune-fine-tune-github-copilot/18889/1 "2022-06-09T04:26:50Z")

</div>

We can fine tune language models like BERT, GPT-3.

Can I fine tune GitHub Copilot model?

I have already looked into the [https://copilot.github.com/](https://copilot.github.com/) but cant find the details. Would really appreciate if someone had fine tuned Github Copilot.

---

<div class="post-metadata">

**Author:** ![lvwerra](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lvwerra/32/2427_2.png) [@lvwerra](https://discuss.huggingface.co/u/lvwerra)\
**Post date:** [June 9, 2022, 12:34pm UTC](https://discuss.huggingface.co/t/how-to-fine-tune-fine-tune-github-copilot/18889/2 "2022-06-09T12:34:10Z")

</div>

Hi @neo-benjamin

The Codex model that’s powering the Copilot product is not open sourced. However, there are a few models similar to Codex available on the Hugging Face Hub such as Incoder or CodeGen:

> **[facebook/incoder-6B · Hugging Face](https://huggingface.co/facebook/incoder-6B)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

> **[Salesforce/codegen-16B-multi · Hugging Face](https://huggingface.co/Salesforce/codegen-16B-multi)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

---

<div class="post-metadata">

**Author:** ![neo-benjamin](https://avatars.discourse-cdn.com/v4/letter/n/f6c823/32.png) [@neo-benjamin](https://discuss.huggingface.co/u/neo-benjamin)\
**Post date:** [June 9, 2022, 8:19pm UTC](https://discuss.huggingface.co/t/how-to-fine-tune-fine-tune-github-copilot/18889/3 "2022-06-09T20:19:57Z")

</div>

How to fine tune Codegen? Are the steps documented?

---

<div class="post-metadata">

**Author:** ![lvwerra](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lvwerra/32/2427_2.png) [@lvwerra](https://discuss.huggingface.co/u/lvwerra)\
**Post date:** [June 24, 2022, 1:40pm UTC](https://discuss.huggingface.co/t/how-to-fine-tune-fine-tune-github-copilot/18889/4 "2022-06-24T13:40:30Z")

</div>

You can have a look at the language modeling examples. The should work for any auto regressive model such as GPT-2 or CodeGen: [transformers/examples/pytorch/language-modeling at main · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling)