# How to implement custom vision encoder-decoder?

**URL:** <https://discuss.huggingface.co/t/how-to-implement-custom-vision-encoder-decoder/45848>\
**Category:** 🤗Transformers\
**Created:** [July 6, 2023, 7:11am UTC](https://discuss.huggingface.co/t/how-to-implement-custom-vision-encoder-decoder/45848 "2023-07-06T07:11:13Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![bongseok](https://avatars.discourse-cdn.com/v4/letter/b/a6a055/32.png) [@bongseok](https://discuss.huggingface.co/u/bongseok)\
**Post date:** [July 6, 2023, 7:11am UTC](https://discuss.huggingface.co/t/how-to-implement-custom-vision-encoder-decoder/45848/1 "2023-07-06T07:11:13Z")

</div>

I’m trying to make a custom vision encoder-decoder model.  
I want to use pre-trained encoder while want to use decoder from scratch, So I cannot use `VisionEncoderDecoderModel.from_pretrained()`.  
Specifically, I want to use pre-trained `deit` model as a encoder, and custom trained `Electra` as a decoder. (using `AutoModelForCausalLM`)

I write code like below. In train step, there is no problem.  
But I got a problem which says “model have no attribute ‘generate’”. **How can I implement or import `generate` function?**

```auto
class CustomEncoderDecoderModel(nn.Module):
    config_class = VisionEncoderDecoderConfig

    def __init__ (self, encoder_name, decoder_config,
                 config=None):
        super(CustomEncoderDecoderModel, self). __init__ ()
        self.encoder = AutoModel.from_pretrained(encoder_name)
        self.decoder_config = decoder_config
        self.decoder = AutoModelForCausalLM.from_config(self.decoder_config)
        self.config = VisionEncoderDecoderConfig.from_encoder_decoder_configs(self.encoder.config, self.decoder.config)
        
        self.criterion = nn.CrossEntropyLoss()
        self.enc_to_dec_proj = nn.Linear(self.encoder.config.hidden_size, self.decoder.config.hidden_size)

    def forward(self, pixel_values, labels, decoder_input_ids=None,
                decoder_input_embeds=None,
                decoder_attention_mask=None,
                decoder_inputs_embeds=None,
                past_key_values=None):
        encoder_outputs = self.encoder(pixel_values,
                                       output_attentions=True)
        encoder_hidden_states = encoder_outputs[0]
        encoder_attention_mask = None

        if decoder_input_ids is None and decoder_input_embeds is None:
            decoder_input_ids = shift_tokens_right(
                labels, self.decoder.config.pad_token_id, decoder_start_token_id=2
            )

        if self.encoder.config.hidden_size != self.decoder.config.hidden_size:
            
            encoder_hidden_states = self.enc_to_dec_proj(encoder_hidden_states)
        decoder_outputs = self.decoder(
            input_ids = decoder_input_ids,
            attention_mask = decoder_attention_mask,
            encoder_hidden_states=encoder_hidden_states,
            encoder_attention_mask=encoder_attention_mask,
            inputs_embeds=decoder_inputs_embeds,
            output_attentions=True,
            use_cache=True,
            past_key_values=past_key_values,
        )
        logits = decoder_outputs[0]

        loss = self.criterion(logits.reshape(-1, self.decoder.config.vocab_size), labels.reshape(-1))

        return {'loss': loss, 'logits': logits,
                'past_key_values': decoder_outputs.past_key_values,
                'decoder_hidden_states': decoder_outputs.hidden_states,
                'decoder_attentions': decoder_outputs.attentions,
                'cross_attentions': decoder_outputs.cross_attentions,
                'encoder_hidden_state': encoder_outputs.hidden_states,
                'encoder_attentions': encoder_attention_mask,
                'encoder_attentions': encoder_outputs.attentions,
                }

```

---

<div class="post-metadata">

**Author:** ![anon86149130](https://avatars.discourse-cdn.com/v4/letter/a/2bfe46/32.png) [@anon86149130](https://discuss.huggingface.co/u/anon86149130)\
**Post date:** [August 1, 2023, 1:22am UTC](https://discuss.huggingface.co/t/how-to-implement-custom-vision-encoder-decoder/45848/2 "2023-08-01T01:22:57Z")

</div>

Just Inherit from the `VisionEncoderDecoderModel` parent class and then change whatever you need to and leave the rest of it as is. `generate` function is not an attribute of the `nn.Module` so it won’t be automatically available here.  
You can use the code for Donut model as a reference.