# Saving underlying language model after trained on downstream task

**URL:** <https://discuss.huggingface.co/t/saving-underlying-language-model-after-trained-on-downstream-task/1121>\
**Category:** 🤗Transformers\
**Created:** [September 14, 2020, 11:48am UTC](https://discuss.huggingface.co/t/saving-underlying-language-model-after-trained-on-downstream-task/1121 "2020-09-14T11:48:14Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![eldarab](https://avatars.discourse-cdn.com/v4/letter/e/3ec8ea/32.png) [@eldarab](https://discuss.huggingface.co/u/eldarab)\
**Post date:** [September 14, 2020, 11:48am UTC](https://discuss.huggingface.co/t/saving-underlying-language-model-after-trained-on-downstream-task/1121/1 "2020-09-14T11:48:14Z")

</div>

Hello everyone,

Although I am quite new here (first post!), I feel like I know huggingface transformers pretty well since I read the documentation and tutorials, and I even made a small project by myself recently.

My goal is to create a module that receives a pretrained language model, fine tunes/trains it on some downstream task, NER for example, and then saves only the language model but without the NER classifier at the end (only the underlying finetuned language model) .

Some requirements that make this task non-trivial:

- The module should be general enough (inherit from PretrainedModel) so it can be trained, saved to disk, loaded from disk, and then trained, all using huggingface _Trainer_.
- The module must be flexible enough to do this for all huggingface’s models with as little code changes as possible.
- The module should be as forward compatible as possible, meaning that I prefer using AutoModel as much as I can.

I hope I got myself clear. if you have any questions regarding my post, please don’t hesitate to reply.