# mBART embedding matrix prunning

**URL:** <https://discuss.huggingface.co/t/mbart-embedding-matrix-prunning/6125>\
**Category:** Intermediate\
**Created:** [May 11, 2021, 2:31pm UTC](https://discuss.huggingface.co/t/mbart-embedding-matrix-prunning/6125 "2021-05-11T14:31:47Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![anunez](https://avatars.discourse-cdn.com/v4/letter/a/e9a140/32.png) [@anunez](https://discuss.huggingface.co/u/anunez)\
**Post date:** [May 11, 2021, 2:31pm UTC](https://discuss.huggingface.co/t/mbart-embedding-matrix-prunning/6125/1 "2021-05-11T14:31:47Z")

</div>

I hope this posts is okay in the forums.

I am using an mBART model and I want to reduce the size of the vocabulary. To do so:  
(1) I mapped the old vocabulary to the new one.  
(2) I got the original embeddings and created the new embedding matrix following my new vocabulary mapping.  
(3) I put the new embedding matrix into the mBART model.

Specifically, for (3) I did:

> #ft\_tensor contains the input embedding matrix  
> ft\_tensor = torch.load(args.pruned\_model + ‘/input\_embedding\_weights.pt’)  
> new\_wte = Embedding.from\_pretrained(ft\_tensor)  
> new\_wte.padding\_idx = 0  
> model.set\_input\_embeddings(new\_wte)  
> # LM head  
> model.lm\_head.weight = model.get\_input\_embeddings().weight  
> # The procedure is repeated for the output embeddings  
> ft\_tensor = torch.load(args.pruned\_model + ‘/output\_embedding\_weights.pt’)  
> new\_wte = Embedding.from\_pretrained(ft\_tensor)  
> new\_wte.padding\_idx = 0  
> model.set\_output\_embeddings(new\_wte)

I also adapted the Tokenizer to the new vocabulary. When running this (in the forward pass) I get the following error:

> Traceback (most recent call last):  
> File “main.py”, line 71, in   
> main()  
> File “main.py”, line 49, in main  
> outputs = model(\*\*batch)  
> File “/var/python3envs/transformers-4.5.1/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 727, in \_call\_impl  
> result = self.forward(\*input, \*\*kwargs)  
> File “/var/python3envs/transformers-4.5.1/lib/python3.6/site-packages/transformers/models/mbart/modeling\_mbart.py”, line 1305, in forward  
> lm\_logits = self.lm\_head(outputs[0]) + self.final\_logits\_bias  
> File “/var/python3envs/transformers-4.5.1/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 727, in \_call\_impl  
> result = self.forward(\*input, \*\*kwargs)  
> File “/var/python3envs/transformers-4.5.1/lib/python3.6/site-packages/torch/nn/modules/sparse.py”, line 126, in forward  
> self.norm\_type, self.scale\_grad\_by\_freq, self.sparse)  
> File “/var/python3envs/transformers-4.5.1/lib/python3.6/site-packages/torch/nn/functional.py”, line 1852, in embedding  
> return torch.embedding(weight, input, padding\_idx, scale\_grad\_by\_freq, sparse)  
> RuntimeError: Expected tensor for argument #1 ‘indices’ to have scalar type Long; but got torch.FloatTensor instead (while checking arguments for embedding)

The first thing I did was casting the input to Long, but this did not solve the error. In fact, if I do not modify the embeddings of the model (keep the original model), the code works well. If I get the original embeddings (model.get\_input\_embeddings().weight) and copy them again (model.set\_input\_embeddings(…)) the same error arises. I hope someone can shed light on this issue.