# Transformers with additional external data

**URL:** <https://discuss.huggingface.co/t/transformers-with-additional-external-data/8041>\
**Category:** Beginners\
**Created:** [July 5, 2021, 12:31pm UTC](https://discuss.huggingface.co/t/transformers-with-additional-external-data/8041 "2021-07-05T12:31:17Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Roman](https://avatars.discourse-cdn.com/v4/letter/r/ed8c4c/32.png) [@Roman](https://discuss.huggingface.co/u/Roman)\
**Post date:** [July 5, 2021, 12:31pm UTC](https://discuss.huggingface.co/t/transformers-with-additional-external-data/8041/1 "2021-07-05T12:31:17Z")

</div>

Hello.  
Working with pretrained BERT model for regression task. I have idea that dataset has **usefull not text feature** , which can help to improve result. Is it possible to modify my model so that the text field is processed by bert taking into account this **useful feature**?

```auto
class MyModel(torch.nn.Module):
    def __init__ (self):
        super(). __init__ ()

        config = AutoConfig.from_pretrained(model_name)
        config.update({"output_hidden_states":True, 
                       "hidden_dropout_prob": 0.0,
                       "layer_norm_eps": 1e-7})                       
        
        self.roberta = AutoModel.from_pretrained(model_name, config=config)  
            
        self.attention = torch.nn.Sequential(            
            torch.nn.Linear(768, 512),            
            torch.nn.Tanh(),                       
            torch.nn.Linear(512, 1),
            torch.nn.Softmax(dim=1)
        )        

        self.regressor = torch.nn.Sequential(                        
            torch.nn.Linear(768, 1)                        
        )
        

    def forward(self, input_ids, attention_mask):
        roberta_output = self.roberta(input_ids=input_ids,
                                      attention_mask=attention_mask)        

        last_layer_hidden_states = roberta_output.hidden_states[-1]
        weights = self.attention(last_layer_hidden_states)
        context_vector = torch.sum(weights * last_layer_hidden_states, dim=1)        
        return self.regressor(context_vector)

```

df - some dataset with ‘text’ feature processed by Bert and usefull feature **‘stat’** (float). I what to use **‘stat’** feature to improve my predictions.

---

<div class="post-metadata">

**Author:** ![muralidandu](https://avatars.discourse-cdn.com/v4/letter/m/73ab20/32.png) [@muralidandu](https://discuss.huggingface.co/u/muralidandu)\
**Post date:** [August 24, 2021, 11:18pm UTC](https://discuss.huggingface.co/t/transformers-with-additional-external-data/8041/2 "2021-08-24T23:18:21Z")

</div>

Hi,  
Did you finalize combining context vector + manual features?