# Where to pick-up embedding data from BERT model?

**URL:** <https://discuss.huggingface.co/t/where-to-pick-up-embedding-data-from-bert-model/14451>\
**Category:** Models\
**Created:** [February 6, 2022, 9:11am UTC](https://discuss.huggingface.co/t/where-to-pick-up-embedding-data-from-bert-model/14451 "2022-02-06T09:11:39Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ThetaPhiPsi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thetaphipsi/32/19404_2.png) [@ThetaPhiPsi](https://discuss.huggingface.co/u/ThetaPhiPsi)\
**Post date:** [February 6, 2022, 9:11am UTC](https://discuss.huggingface.co/t/where-to-pick-up-embedding-data-from-bert-model/14451/1 "2022-02-06T09:11:39Z")

</div>

Hi all,

I would like to use the embeddings model developed by Emilia Alsentzer, Bio\_ClinicalBERT, but I am not sure where to get the embedding data.

For now I use the model’s `pooler_output`, is that the correct way?

```auto
tokenizer = AutoTokenizer.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")
model = AutoModel.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")

inputs = tokenizer(...)
sentence_vector = model(**inputs).pooler_output

```

---

<div class="post-metadata">

**Author:** ![osanseviero](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/osanseviero/32/38738_2.png) [@osanseviero](https://discuss.huggingface.co/u/osanseviero)\
**Post date:** [February 7, 2022, 8:38pm UTC](https://discuss.huggingface.co/t/where-to-pick-up-embedding-data-from-bert-model/14451/2 "2022-02-07T20:38:11Z")

</div>

The embedding of each token is in hidden\_state.

```python
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")
model = AutoModel.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")

text = "this is a test"
inputs = tokenizer(text, return_tensors="pt")
model(**inputs)["last_hidden_state"].shape
>>> torch.Size([1, 6, 768])

```

`pooler_output` corresponds to the CLS token which is used for classification. So it can be seen as a hidden state for the whole sentence.

> - **pooler\_output** ( `torch.FloatTensor` of shape `(batch_size, hidden_size)` ) — Last layer hidden-state of the first token of the sequence (classification token) after further processing through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns the classification token after processing through a linear layer and a tanh activation function. The linear layer weights are trained from the next sentence prediction (classification) objective during pretraining.

---

<div class="post-metadata">

**Author:** ![ThetaPhiPsi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thetaphipsi/32/19404_2.png) [@ThetaPhiPsi](https://discuss.huggingface.co/u/ThetaPhiPsi)\
**Post date:** [February 8, 2022, 8:50pm UTC](https://discuss.huggingface.co/t/where-to-pick-up-embedding-data-from-bert-model/14451/3 "2022-02-08T20:50:35Z")

</div>

Thank you for your explanation.  
So, if I consider `pooler_output` to be a (kind of) embedding of the whole sentence, that is the correct method.