# MaskedLMOutput does not have last\_hidden\_state

**URL:** <https://discuss.huggingface.co/t/maskedlmoutput-does-not-have-last-hidden-state/6422>\
**Category:** 🤗Transformers\
**Created:** [May 27, 2021, 2:29pm UTC](https://discuss.huggingface.co/t/maskedlmoutput-does-not-have-last-hidden-state/6422 "2021-05-27T14:29:45Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Fruits](https://avatars.discourse-cdn.com/v4/letter/f/58f4c7/32.png) [@Fruits](https://discuss.huggingface.co/u/Fruits)\
**Post date:** [May 27, 2021, 2:29pm UTC](https://discuss.huggingface.co/t/maskedlmoutput-does-not-have-last-hidden-state/6422/1 "2021-05-27T14:29:45Z")

</div>

I am trying to access the last hidden state here at `pooler1` and `pooler2` variables towards the end of the code.

```
   class BERT_Arch(nn.Module):

    def __init__ (self, bert):
      
      super(BERT_Arch, self). __init__ ()

      self.bert = MODEL_TYPE.from_pretrained(PRETRAINED_MODEL_NAME)
      freeze(self.bert, params['freeze']) 
      for name, param in self.bert.named_parameters():
        print("{} -> {}".format(name, param.requires_grad))

      self.dropout = nn.Dropout(params['dropout'])
      self.relu = nn.ReLU()
      self.fc1 = nn.Linear(params['linear_in'], params['linear_out'])

      def _qa_embedding(self, pooler):
         x = self.fc1(pooler)
         x = normalize_vectors(x)
         return x

      #define the forward pass
      def forward(self, tokensq, maskq, tokens1, mask1, tokens2, mask2):

      #pass the inputs to the model  
      outputs1 = self.bert(torch.cat((tokensq, tokens1), 1), attention_mask=torch.cat((maskq, mask1), 1), output_hidden_states=True)
      outputs2 = self.bert(torch.cat((tokensq, tokens2), 1), attention_mask=torch.cat((maskq, mask2), 1), output_hidden_states=True)

      # last_hidden_state does only exist with the non "auto-masked" models use hidden_states for the rest
      pooler1 = torch.cat((outputs1.last_hidden_state[:,0], outputs1.last_hidden_state[:,params['max_q_len']]), 1)
      pooler2 = torch.cat((outputs2.last_hidden_state[:,0], outputs2.last_hidden_state[:,params['max_q_len']]), 1)

      v1 = self._qa_embedding(pooler1)
      v2 = self._qa_embedding(pooler2)
      
      return v1, v2

```

It did work when I used this BERT multiling uncased which uses [BaseModelOutput](https://github.com/huggingface/transformers/blob/80d712fac6ccae308a2f408ebbc0c4d8c482d509/src/transformers/modeling_outputs.py#L24). `last_hidden_state` is declared as avariable.

Unfortunately, now that I am using BERT mutliling cased, the class [MaskedLMOutput](https://github.com/huggingface/transformers/blob/80d712fac6ccae308a2f408ebbc0c4d8c482d509/src/transformers/modeling_outputs.py#L453) is being used which does not seem to have the `last_hidden_state` variable.

How can I access the `last_hidden_state`?  
Do I really need to copy the `class MaskedLMOutput` including all the dependencies and add `last_hidden_state: torch.FloatTensor = None` ? It seems a bit overkill as I need to import all the dependencies then.

Thanks so much