# Encoder Decoder Embedding layer shared in BartModel code **URL:** \ **Category:** 🤗Transformers\ **Created:** [September 20, 2023, 2:39am UTC](https://discuss.huggingface.co/t/encoder-decoder-embedding-layer-shared-in-bartmodel-code/55696 "2023-09-20T02:39:57Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![Hakase-Noonna](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/hakase-noonna/32/17504_2.png) [@Hakase-Noonna](https://discuss.huggingface.co/u/Hakase-Noonna)\ **Post date:** [September 20, 2023, 2:39am UTC](https://discuss.huggingface.co/t/encoder-decoder-embedding-layer-shared-in-bartmodel-code/55696/1 "2023-09-20T02:39:58Z")
Hello, I am currently studying Bart model code in transformers library. I looked up the **BartModel class** in **modeling\_bart.py** python code and found the following code. ```auto @add_start_docstrings( "The bare BART Model outputting raw hidden-states without any specific head on top.", BART_START_DOCSTRING, ) class BartModel(BartPretrainedModel): _keys_to_ignore_on_load_missing = ["encoder.embed_tokens.weight", "decoder.embed_tokens.weight"] def __init__ (self, config: BartConfig): super(). __init__ (config) padding_idx, vocab_size = config.pad_token_id, config.vocab_size self.shared = nn.Embedding(vocab_size, config.d_model, padding_idx) self.encoder = BartEncoder(config, self.shared) self.decoder = BartDecoder(config, self.shared) # Initialize weights and apply final processing self.post_init() def get_input_embeddings(self): return self.shared def set_input_embeddings(self, value): self.shared = value self.encoder.embed_tokens = self.shared self.decoder.embed_tokens = self.shared def get_encoder(self): return self.encoder def get_decoder(self): return self.decoder @add_start_docstrings_to_model_forward(BART_INPUTS_DOCSTRING) @add_code_sample_docstrings( checkpoint=_CHECKPOINT_FOR_DOC, output_type=Seq2SeqModelOutput, config_class=_CONFIG_FOR_DOC, expected_output=_EXPECTED_OUTPUT_SHAPE, ) def forward( self, input_ids: torch.LongTensor = None, attention_mask: Optional[torch.Tensor] = None, decoder_input_ids: Optional[torch.LongTensor] = None, decoder_attention_mask: Optional[torch.LongTensor] = None, head_mask: Optional[torch.Tensor] = None, decoder_head_mask: Optional[torch.Tensor] = None, cross_attn_head_mask: Optional[torch.Tensor] = None, encoder_outputs: Optional[List[torch.FloatTensor]] = None, past_key_values: Optional[List[torch.FloatTensor]] = None, inputs_embeds: Optional[torch.FloatTensor] = None, decoder_inputs_embeds: Optional[torch.FloatTensor] = None, use_cache: Optional[bool] = None, output_attentions: Optional[bool] = None, output_hidden_states: Optional[bool] = None, return_dict: Optional[bool] = None, ) -> Union[Tuple, Seq2SeqModelOutput]: # different to other models, Bart automatically creates decoder_input_ids from # input_ids if no decoder_input_ids are provided if decoder_input_ids is None and decoder_inputs_embeds is None: if input_ids is None: raise ValueError( "If no `decoder_input_ids` or `decoder_inputs_embeds` are " "passed, `input_ids` cannot be `None`. Please pass either " "`input_ids` or `decoder_input_ids` or `decoder_inputs_embeds`." ) decoder_input_ids = shift_tokens_right( input_ids, self.config.pad_token_id, self.config.decoder_start_token_id ) output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions output_hidden_states = ( output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states ) use_cache = use_cache if use_cache is not None else self.config.use_cache return_dict = return_dict if return_dict is not None else self.config.use_return_dict if encoder_outputs is None: encoder_outputs = self.encoder( input_ids=input_ids, attention_mask=attention_mask, head_mask=head_mask, inputs_embeds=inputs_embeds, output_attentions=output_attentions, output_hidden_states=output_hidden_states, return_dict=return_dict, ) # If the user passed a tuple for encoder_outputs, we wrap it in a BaseModelOutput when return_dict=True elif return_dict and not isinstance(encoder_outputs, BaseModelOutput): encoder_outputs = BaseModelOutput( last_hidden_state=encoder_outputs[0], hidden_states=encoder_outputs[1] if len(encoder_outputs) > 1 else None, attentions=encoder_outputs[2] if len(encoder_outputs) > 2 else None, ) # decoder outputs consists of (dec_features, past_key_value, dec_hidden, dec_attn) decoder_outputs = self.decoder( input_ids=decoder_input_ids, attention_mask=decoder_attention_mask, encoder_hidden_states=encoder_outputs[0], encoder_attention_mask=attention_mask, head_mask=decoder_head_mask, cross_attn_head_mask=cross_attn_head_mask, past_key_values=past_key_values, inputs_embeds=decoder_inputs_embeds, use_cache=use_cache, output_attentions=output_attentions, output_hidden_states=output_hidden_states, return_dict=return_dict, ) if not return_dict: return decoder_outputs + encoder_outputs return Seq2SeqModelOutput( last_hidden_state=decoder_outputs.last_hidden_state, past_key_values=decoder_outputs.past_key_values, decoder_hidden_states=decoder_outputs.hidden_states, decoder_attentions=decoder_outputs.attentions, cross_attentions=decoder_outputs.cross_attentions, encoder_last_hidden_state=encoder_outputs.last_hidden_state, encoder_hidden_states=encoder_outputs.hidden_states, encoder_attentions=encoder_outputs.attentions, ) ``` I tried to use this model to solve translation task. What I’ve learned is that I have to use **two vocabs(src\_vocab, target\_vocab)**. That means I need two Embedding Layers, dosen’t it? However, I only can see there is one nn.Embedding layer as **self.shared**. This is very confusing to me. How can this model translate one language to another? Any informative answer is much appreciated!! Thx!! 😆 ---
**Author:** ![Hakase-Noonna](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/hakase-noonna/32/17504_2.png) [@Hakase-Noonna](https://discuss.huggingface.co/u/Hakase-Noonna)\ **Post date:** [September 20, 2023, 2:55am UTC](https://discuss.huggingface.co/t/encoder-decoder-embedding-layer-shared-in-bartmodel-code/55696/2 "2023-09-20T02:55:59Z")
Okay, This was dumb question… 🤪 One Embedding layer goes to Encoder and Decoder classes separately. So there will be two embedding layers in the end. Question is Answered.