# Extracting sentence embeddings from NLP models from each layer seperately

**URL:** <https://discuss.huggingface.co/t/extracting-sentence-embeddings-from-nlp-models-from-each-layer-seperately/9316>\
**Category:** Beginners\
**Created:** [August 18, 2021, 6:31am UTC](https://discuss.huggingface.co/t/extracting-sentence-embeddings-from-nlp-models-from-each-layer-seperately/9316 "2021-08-18T06:31:10Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![kadaj13](https://avatars.discourse-cdn.com/v4/letter/k/76d3ee/32.png) [@kadaj13](https://discuss.huggingface.co/u/kadaj13)\
**Post date:** [August 18, 2021, 6:31am UTC](https://discuss.huggingface.co/t/extracting-sentence-embeddings-from-nlp-models-from-each-layer-seperately/9316/1 "2021-08-18T06:31:10Z")

</div>

I am using the following codes to extract word embeddings from famous NLP models such BERT:

```auto
from transformers import pipeline, AutoTokenizer, AutoModel
import numpy as np
import re

model_name = "bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)

tokenizer.add_special_tokens({'pad_token': '[PAD]'})
model.resize_token_embeddings(len(tokenizer))

model_pipeline = pipeline('feature-extraction', model=model_name, tokenizer=tokenizer)

def find_wordNo_sentence(word, sentence):
    
    print(sentence)
    splitted_sen = sentence.split(" ")
    print(splitted_sen)
    index = splitted_sen.index(word)

    for i,w in enumerate(splitted_sen):
        if(word == w):
            return i

    print("not found") #0 base

def return_bert_embedding(word, sentence):
        
    word = re.sub(r'[^\w]', " ", word)
    word = " ".join(word.split())
    
    sentence = re.sub(r'[^\w]', ' ', sentence)
    sentence = " ".join(sentence.split())
    
    id_word = find_wordNo_sentence(word, sentence)
    
   
        
    try:
        data = model_pipeline(sentence)
        
        n_words = len(sentence.split(" "))
        #print(sentence_emb.shape)
        n_embs = len(data[0])
        print(n_embs, n_words)
        print(len(data[0]))
    
        if (n_words != n_embs):
            "There is extra tokenized word"
            
            
        results = data[0][id_word]  
        return np.array(results)
    
    except:
        return "word not found"

print(return_bert_embedding('your', "what is your name?"))

```

However, I know that these NLP models have different layers. How can I specify the layer number for extracting the features?