# Extracting token embeddings from pretrained language models

**URL:** <https://discuss.huggingface.co/t/extracting-token-embeddings-from-pretrained-language-models/6834>\
**Category:** Beginners\
**Created:** [June 15, 2021, 4:16am UTC](https://discuss.huggingface.co/t/extracting-token-embeddings-from-pretrained-language-models/6834 "2021-06-15T04:16:55Z")\
**Posts on this page:** 1\
**Showing post:** 4

<div class="post-metadata">

**Author:** ![osanseviero](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/osanseviero/32/38738_2.png) [@osanseviero](https://discuss.huggingface.co/u/osanseviero)\
**Post date:** [June 15, 2021, 9:00am UTC](https://discuss.huggingface.co/t/extracting-token-embeddings-from-pretrained-language-models/6834/4 "2021-06-15T09:00:51Z")

</div>

You can specify the tokenizer with the `tokenizer` argument and do what is suggested in the error message.

Here is an example based on the [documentation](https://huggingface.co/transformers/internal/tokenization_utils.html?highlight=resize_token_embeddings#transformers.tokenization_utils_base.SpecialTokensMixin.add_special_tokens).

```auto
tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModel.from_pretrained("gpt2")

tokenizer.add_special_tokens({'pad_token': '[PAD]'})
model.resize_token_embeddings(len(tokenizer))

pipe = pipeline('feature-extraction', model=model, tokenizer=tokenizer)

```

This works for me. Let me know if you have any other questions 🙂

---

_[View the full topic](https://discuss.huggingface.co/t/extracting-token-embeddings-from-pretrained-language-models/6834)._