# RAG isnt working as expected

**URL:** <https://discuss.huggingface.co/t/rag-isnt-working-as-expected/84866>\
**Category:** Beginners\
**Created:** [May 2, 2024, 3:29pm UTC](https://discuss.huggingface.co/t/rag-isnt-working-as-expected/84866 "2024-05-02T15:29:53Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![DavidAfonsoValente](https://avatars.discourse-cdn.com/v4/letter/d/59ef9b/32.png) [@DavidAfonsoValente](https://discuss.huggingface.co/u/DavidAfonsoValente)\
**Post date:** [May 2, 2024, 3:29pm UTC](https://discuss.huggingface.co/t/rag-isnt-working-as-expected/84866/1 "2024-05-02T15:29:53Z")

</div>

This is my code:

import faiss  
from sentence\_transformers import SentenceTransformer  
from transformers import AutoTokenizer, AutoModelForCausalLM  
import torch  
import pickle  
import os

# Load pickle files

data =   
for filename in os.listdir(‘.’): # replace with your directory  
if filename.endswith(‘.pickle’):  
with open(os.path.join(‘./’, filename), ‘rb’) as f:  
datapoint = pickle.load(f)  
if datapoint != “nothing”:  
data.append(datapoint)

# Load pre-trained model for encoding text to vectors

model\_data\_encoding = SentenceTransformer(‘all-MiniLM-L6-v2’)

# Encode all data to vectors

data\_vectors = [model\_data\_encoding.encode(f"{datapoint[‘names’]}: {datapoint[‘axioms’]}") for datapoint in data]

# Build a FAISS index

dimension = data\_vectors[0].shape[0]  
index = faiss.IndexFlatL2(dimension)  
for vector in data\_vectors:  
index.add(vector.reshape(1, -1))

# Load pre-trained LLM

tokenizer = AutoTokenizer.from\_pretrained(“gpt2”)  
model = AutoModelForCausalLM.from\_pretrained(“gpt2”)  
tokenizer.pad\_token\_id = tokenizer.eos\_token\_id

def answer\_question(question):  
# Encode question to vector  
question\_vector = model\_data\_encoding.encode(question)

```
# Find the most similar documents in the database
distances, indices = index.search(question_vector.reshape(1, -1), 5)

# Get the corresponding data
relevant_data = [data[index] for index in indices[0]]

# Format the data for the LLM
inputs = tokenizer("Given these retrieved axioms from a mathematical database:\n" +" ".join([f"{datapoint['names'][0]}: {datapoint['axioms']}" for datapoint in relevant_data]) + "\nAnswer this question:\n" + question + "\nAnswer: ", return_tensors="pt")

print("Given these retrieved axioms from a mathematical database:\n" +"\n".join([f"{datapoint['names'][0]}: {datapoint['axioms']}" for datapoint in relevant_data]) + "\nAnswer this question:\n" + question + "\nAnswer: ")
# Generate a response
outputs = model.generate(**inputs, max_new_tokens=64, temperature=0.7)

# Decode the response
response = tokenizer.decode(outputs[0], skip_special_tokens=True)

# Parsing out the first sentence
first_sentence = response.split("Answer:")[1].split(".")[0]

return first_sentence

```

question = “what is the LA-semihypergroup axiom?”

response = answer\_question(question)

print(response)

The response I am getting is:  
The LA-semihypergroup axiom is a generalization of the axiom of the inverse semigroup

Instead of the “obvious” answer:  
(x\*y)_z = (z_y)\*x

Why could this be happening and how can I improve it?

---

<div class="post-metadata">

**Author:** ![nimatov](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nimatov/32/18720_2.png) [@nimatov](https://discuss.huggingface.co/u/nimatov)\
**Post date:** [May 2, 2024, 3:55pm UTC](https://discuss.huggingface.co/t/rag-isnt-working-as-expected/84866/2 "2024-05-02T15:55:01Z")

</div>

To get this , formula based answer you should have this answer in expernal data (pickle , I guess). And embedding model with tokenizer that support mathematics language . I am afraid all-MiniLM-L6-v2 is not like this. Still hard to give precise answer , but these two direction I would check.

---

<div class="post-metadata">

**Author:** ![DavidAfonsoValente](https://avatars.discourse-cdn.com/v4/letter/d/59ef9b/32.png) [@DavidAfonsoValente](https://discuss.huggingface.co/u/DavidAfonsoValente)\
**Post date:** [May 2, 2024, 4:06pm UTC](https://discuss.huggingface.co/t/rag-isnt-working-as-expected/84866/3 "2024-05-02T16:06:26Z")

</div>

the retrieval is working perfectly fine though, the problem is really the generation. If i put he prompt with the retrieved axioms into chatgpt it works fine. Could this just be a problem with the gpt2 model?

---

<div class="post-metadata">

**Author:** ![nimatov](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nimatov/32/18720_2.png) [@nimatov](https://discuss.huggingface.co/u/nimatov)\
**Post date:** [May 2, 2024, 6:10pm UTC](https://discuss.huggingface.co/t/rag-isnt-working-as-expected/84866/4 "2024-05-02T18:10:28Z")

</div>

Very likely. If retrieve part works , the problem with synthesizing the answer by GPT2 . Try to run your Retriever with GPT4 API with llama\_index and see if your get the same you get in chatGPT . You have tones of examples on the llma\_index doc site.