# Decode embeddings of BERT hidden layers

**URL:** <https://discuss.huggingface.co/t/decode-embeddings-of-bert-hidden-layers/30775>\
**Category:** Beginners\
**Created:** [January 30, 2023, 8:15pm UTC](https://discuss.huggingface.co/t/decode-embeddings-of-bert-hidden-layers/30775 "2023-01-30T20:15:07Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![yuna786](https://avatars.discourse-cdn.com/v4/letter/y/9de053/32.png) [@yuna786](https://discuss.huggingface.co/u/yuna786)\
**Post date:** [January 30, 2023, 8:15pm UTC](https://discuss.huggingface.co/t/decode-embeddings-of-bert-hidden-layers/30775/1 "2023-01-30T20:15:07Z")

</div>

I am concatenating the last 4 hidden layers of BERT to generate my embeddings, with the method from [here](https://mccormickml.com/2019/05/14/BERT-word-embeddings-tutorial/#33-creating-word-and-sentence-vectors-from-hidden-states). The embedding is a 1D numpy array of length 3,072 (4\*768)

However, I can’t seem to figure out how to decode these embeddings back into sentences.

I’ve tried reshaping the embedding to work with `get_output_embedding()`:

```
    bert = transformers.BertForMaskedLM.from_pretrained("bert-base-uncased")
    tok = transformers.BertTokenizer.from_pretrained("bert-base-uncased")
    dec = bert.get_output_embeddings()(torch.from_numpy(embedding.reshape(4,768)).float())
    print("Decoded sentence:", tok.decode(dec.softmax(0).argmax(1)))

```

Although the output of this code indeed returns a string of characters, these characters are not the original sentence.

How can I decode the embeddings (generated from the last 4 hidden layers of BERT)?