# RAG Embeddings: German language **URL:** \ **Category:** Beginners\ **Created:** [November 2, 2023, 3:05pm UTC](https://discuss.huggingface.co/t/rag-embeddings-german-language/60840 "2023-11-02T15:05:49Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![mox](https://avatars.discourse-cdn.com/v4/letter/m/f04885/32.png) [@mox](https://discuss.huggingface.co/u/mox)\ **Post date:** [November 2, 2023, 3:05pm UTC](https://discuss.huggingface.co/t/rag-embeddings-german-language/60840/1 "2023-11-02T15:05:49Z")
Hi, I am new to the RAG approach and looked at many tutorials. For the Embedding Step I saw that many people are using the “sentence-transformers/all-MiniLM-L6-v2” from Huggingface, which works fine for English texts. Now I want to use RAG on German texts, so can I still use the Sentece Transformers Embedding or do I have to use a German one? If so, what are good open-source embeddings for the german language? Thanks in advance! ---
**Author:** ![Tim1785](https://avatars.discourse-cdn.com/v4/letter/t/c4cdca/32.png) [@Tim1785](https://discuss.huggingface.co/u/Tim1785)\ **Post date:** [December 5, 2023, 2:35pm UTC](https://discuss.huggingface.co/t/rag-embeddings-german-language/60840/2 "2023-12-05T14:35:54Z")
Hi @mox I just saw your post and i was wondering If you had come across something specific. Currently I am checking / experiementing with [LeoLM/leo-mistral-hessianai-7b-chat · Hugging Face](https://huggingface.co/LeoLM/leo-mistral-hessianai-7b-chat) model and its applications for QA retrieval using llama index. I am also looking for the same answer as i think the accuracy depends heavily on the embedding model . Let me know your thoughts ---
**Author:** ![mox](https://avatars.discourse-cdn.com/v4/letter/m/f04885/32.png) [@mox](https://discuss.huggingface.co/u/mox)\ **Post date:** [February 2, 2024, 7:19am UTC](https://discuss.huggingface.co/t/rag-embeddings-german-language/60840/3 "2024-02-02T07:19:42Z")
Hi Tim, atm I am using intfloat/multilingual-e5-large which works okay. Also I wanted to check the JinaAI embeddings which look promising (“Jina Ich bin ein Berliner Embeddings”) ---
**Author:** ![MarcGrumpyOlejak](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marcgrumpyolejak/32/47693_2.png) [@MarcGrumpyOlejak](https://discuss.huggingface.co/u/MarcGrumpyOlejak)\ **Post date:** [February 19, 2024, 2:36pm UTC](https://discuss.huggingface.co/t/rag-embeddings-german-language/60840/4 "2024-02-19T14:36:17Z")
Hi @mox – a late answer. I tested several different “smaller” embeddings before I start using possible Mistral-Embeddings and I stumbled upon “danielheinz/e5-base-sts-en-de”. My “Use-Cases” are 100% for administrational documents – so there is a “huge” context (f.ex. check all german laws) which has to be embedded. I’ve tested (very very simple) the embeddings with 6 “short” searches with “synonyms” (~13.000 different lines of text) to find services of the administration. Only with “e5-base-sts-en-de” I got 100% – failed RAG-embeddings-searches were the following: - multilingual-e5-base - paraphrase-multilingual-MiniLM-L12-v2 - paraphrase-multilingual-mpnet-base-v2 - gte-large - gbert-base But this is only a momentary snapshot an is not representative. ---
**Author:** ![hansegermeier](https://avatars.discourse-cdn.com/v4/letter/h/b2d939/32.png) [@hansegermeier](https://discuss.huggingface.co/u/hansegermeier)\ **Post date:** [February 27, 2024, 4:02pm UTC](https://discuss.huggingface.co/t/rag-embeddings-german-language/60840/5 "2024-02-27T16:02:05Z")
Hi Marc, we tried to use your recommended embedding model but we ran into an index out of range error: IndexError: index out of range in self according to this link this problem seems to be fixable by adjusting the vocab size: > > > \# 🐛 Bug > > \## Information > > The model I am using Bert ('bert-large-uncased') an…d I am facing two issues related to this model > > The language I am using the model on English > > The problem arises when using: > > When I am trying to encode a large sentence ( sentence length 500 words ), I am getting this error : > > \`IndexError: index out of range in self\` > > I tried to set max\_words length as 400, still getting same error : > > > Data I am using can be downloaded like this : > > \`\`\` > from sklearn.datasets import fetch\_20newsgroups > import re > > categories = \['alt.atheism', 'soc.religion.christian','comp.graphics', 'sci.med'\] > twenty\_train = fetch\_20newsgroups(subset='train',categories=categories, shuffle=True, random\_state=42) > > print("\\n".join(twenty\_train.data\[0\].split("\\n")\[:3\])) > > > X\_tratado = \[\] > > for email in range(0, len(twenty\_train.data)): > > # Remover caracteres especiais > texto = re.sub(r'\\\\r\\\\n', ' ', str(twenty\_train.data\[email\])) > texto = re.sub(r'\\W', ' ', texto) > > # Remove caracteres simples de uma letra > texto = re.sub(r'\\s+\[a-zA-Z\]\\s+', ' ', texto) > texto = re.sub(r'\\^\[a-zA-Z\]\\s+', ' ', texto) > > # Substitui multiplos espaços por um unico espaço > texto = re.sub(r'\\s+', ' ', texto, flags=re.I) > > # Remove o 'b' que aparece no começo > texto = re.sub(r'^b\\s+', '', texto) > > # Converte para minúsculo > texto = texto.lower() > > X\_tratado.append(texto) > > dr = {} > dr \['text'\] = X\_tratado > dr \['labels'\] = twenty\_train.target > \`\`\` > > Now I am using bert model to encode the sentences : > > > \`\`\` > from transformers import BertModel, BertConfig, BertTokenizer > import torch > > tokenizer = BertTokenizer.from\_pretrained('bert-large-uncased') > model = BertModel.from\_pretrained('bert-large-uncased') > inputs = tokenizer(datar\[7\], return\_tensors="pt") > outputs = model(\*\*inputs) > features = outputs\[0\]\[:,0,:\].detach().numpy().squeeze() > \`\`\` > > Which is giving this error : > > \`\`\` > \--------------------------------------------------------------------------- > IndexError Traceback (most recent call last) > \ in \ > 5 model = BertModel.from\_pretrained('bert-large-uncased') > 6 inputs = tokenizer(datar\[7\], return\_tensors="pt") > \----\> 7 outputs = model(\*\*inputs) > 8 features = outputs\[0\]\[:,0,:\].detach().numpy().squeeze() > > ~/tfproject/tfenv/lib/python3.7/site-packages/torch/nn/modules/module.py in \_\_call\_\_(self, \*input, \*\*kwargs) > 548 result = self.\_slow\_forward(\*input, \*\*kwargs) > 549 else: > \--\> 550 result = self.forward(\*input, \*\*kwargs) > 551 for hook in self.\_forward\_hooks.values(): > 552 hook\_result = hook(self, input, result) > > ~/tfproject/tfenv/lib/python3.7/site-packages/transformers/modeling\_bert.py in forward(self, input\_ids, attention\_mask, token\_type\_ids, position\_ids, head\_mask, inputs\_embeds, encoder\_hidden\_states, encoder\_attention\_mask, output\_attentions, output\_hidden\_states) > 751 > 752 embedding\_output = self.embeddings( > \--\> 753 input\_ids=input\_ids, position\_ids=position\_ids, token\_type\_ids=token\_type\_ids, inputs\_embeds=inputs\_embeds > 754 ) > 755 encoder\_outputs = self.encoder( > > ~/tfproject/tfenv/lib/python3.7/site-packages/torch/nn/modules/module.py in \_\_call\_\_(self, \*input, \*\*kwargs) > 548 result = self.\_slow\_forward(\*input, \*\*kwargs) > 549 else: > \--\> 550 result = self.forward(\*input, \*\*kwargs) > 551 for hook in self.\_forward\_hooks.values(): > 552 hook\_result = hook(self, input, result) > > ~/tfproject/tfenv/lib/python3.7/site-packages/transformers/modeling\_bert.py in forward(self, input\_ids, token\_type\_ids, position\_ids, inputs\_embeds) > 177 if inputs\_embeds is None: > 178 inputs\_embeds = self.word\_embeddings(input\_ids) > \--\> 179 position\_embeddings = self.position\_embeddings(position\_ids) > 180 token\_type\_embeddings = self.token\_type\_embeddings(token\_type\_ids) > 181 > > ~/tfproject/tfenv/lib/python3.7/site-packages/torch/nn/modules/module.py in \_\_call\_\_(self, \*input, \*\*kwargs) > 548 result = self.\_slow\_forward(\*input, \*\*kwargs) > 549 else: > \--\> 550 result = self.forward(\*input, \*\*kwargs) > 551 for hook in self.\_forward\_hooks.values(): > 552 hook\_result = hook(self, input, result) > > ~/tfproject/tfenv/lib/python3.7/site-packages/torch/nn/modules/sparse.py in forward(self, input) > 112 return F.embedding( > 113 input, self.weight, self.padding\_idx, self.max\_norm, > \--\> 114 self.norm\_type, self.scale\_grad\_by\_freq, self.sparse) > 115 > 116 def extra\_repr(self): > > ~/tfproject/tfenv/lib/python3.7/site-packages/torch/nn/functional.py in embedding(input, weight, padding\_idx, max\_norm, norm\_type, scale\_grad\_by\_freq, sparse) > 1722 # remove once script supports set\_grad\_enabled > 1723 \_no\_grad\_embedding\_renorm\_(weight, input, max\_norm, norm\_type) > \-\> 1724 return torch.embedding(weight, input, padding\_idx, scale\_grad\_by\_freq, sparse) > 1725 > 1726 > > IndexError: index out of range in self > \`\`\` > > The second issue I am facing, When I am using this bert model to encode many sentences, It seems Bert is not using GPU : > > !\[Screenshot 2020-07-09 at 12 45 14 AM\](https://user-images.githubusercontent.com/17107749/86960748-9c905980-c17d-11ea-8d1e-bb72141cbf37.png) > > How to accelerate GPU while using bert model? > > > \## Environment info > \