# Mistral 7B RAG Langchaing

**URL:** <https://discuss.huggingface.co/t/mistral-7b-rag-langchaing/73973>\
**Category:** Models\
**Created:** [February 20, 2024, 12:44pm UTC](https://discuss.huggingface.co/t/mistral-7b-rag-langchaing/73973 "2024-02-20T12:44:11Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![victoriglesias5](https://avatars.discourse-cdn.com/v4/letter/v/e19adc/32.png) [@victoriglesias5](https://discuss.huggingface.co/u/victoriglesias5)\
**Post date:** [February 20, 2024, 12:44pm UTC](https://discuss.huggingface.co/t/mistral-7b-rag-langchaing/73973/1 "2024-02-20T12:44:12Z")

</div>

Hi guys!  
I’ve been working with Mistral 7B model in order to chat with my own data.  
I’m workin with a MongoDB dataset about restaurants, but when I ask my model about anything related with this dataset, it returns me a wrong outpur.

I’ve been checking the context and it seems to be there the main problem.

I post the code here. Hope someone can help me. I appreciate it guys, thank you all.

!pip install -q -U torch datasets transformers==4.36.1 tensorflow langchain playwright html2text sentence\_transformers faiss-cpu  
!pip install -q accelerate==0.21.0 peft==0.4.0 bitsandbytes==0.40.2 trl==0.4.7

import os  
import torch  
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, pipeline

from datasets import load\_dataset  
from peft import LoraConfig, PeftModel

from langchain.text\_splitter import CharacterTextSplitter  
from langchain.document\_transformers import Html2TextTransformer  
#from langchain.document\_loaders import AsyncChromiumLoader  
from langchain\_community.document\_loaders.mongodb import MongodbLoader

from langchain.embeddings.huggingface import HuggingFaceEmbeddings  
from langchain.vectorstores import FAISS, Chroma

from langchain.prompts import PromptTemplate  
from langchain.schema.runnable import RunnablePassthrough  
from langchain.llms import HuggingFacePipeline  
from langchain.chains import LLMChain

from huggingface\_hub import login  
login(“xxxxxxxxxxxxxxxxxxxx”)

#################################################################

# Tokenizer

#################################################################

model\_name=‘mistralai/Mistral-7B-Instruct-v0.2’

tokenizer = AutoTokenizer.from\_pretrained(model\_name, trust\_remote\_code=True)  
tokenizer.pad\_token = tokenizer.eos\_token  
tokenizer.padding\_side = “right”

#################################################################

# bitsandbytes parameters

#################################################################

# Activate 4-bit precision base model loading

use\_4bit = True

# Compute dtype for 4-bit base models

bnb\_4bit\_compute\_dtype = “float16”

# Quantization type (fp4 or nf4)

bnb\_4bit\_quant\_type = “nf4”

# Activate nested quantization for 4-bit base models (double quantization)

use\_nested\_quant = False

#################################################################

# Set up quantization config

#################################################################  
compute\_dtype = getattr(torch, bnb\_4bit\_compute\_dtype)

bnb\_config = BitsAndBytesConfig(  
load\_in\_4bit=use\_4bit,  
bnb\_4bit\_quant\_type=bnb\_4bit\_quant\_type,  
bnb\_4bit\_compute\_dtype=compute\_dtype,  
bnb\_4bit\_use\_double\_quant=use\_nested\_quant,  
)

# Check GPU compatibility with bfloat16

# if compute\_dtype == torch.float16 and use\_4bit:

# major, \_ = torch.cuda.get\_device\_capability()

# if major \>= 8:

# print(“=” \* 80)

# print(“Your GPU supports bfloat16: accelerate training with bf16=True”)

# print(“=” \* 80)

#################################################################

# Load pre-trained config

#################################################################  
model = AutoModelForCausalLM.from\_pretrained(  
model\_name,  
quantization\_config=bnb\_config,  
)

def print\_number\_of\_trainable\_model\_parameters(model):  
trainable\_model\_params = 0  
all\_model\_params = 0  
for \_, param in model.named\_parameters():  
all\_model\_params += param.numel()  
if param.requires\_grad:  
trainable\_model\_params += param.numel()  
return f"trainable model parameters: {trainable\_model\_params}\nall model parameters: {all\_model\_params}\npercentage of trainable model parameters: {100 \* trainable\_model\_params / all\_model\_params:.2f}%"

print(print\_number\_of\_trainable\_model\_parameters(model))

text\_generation\_pipeline = pipeline(  
model=model,  
tokenizer=tokenizer,  
task=“text-generation”,  
temperature=0,  
repetition\_penalty=1.1,  
return\_full\_text=True,  
max\_new\_tokens=1000,  
)

mistral\_llm = HuggingFacePipeline(pipeline=text\_generation\_pipeline)

!playwright install  
!playwright install-deps

import nest\_asyncio  
nest\_asyncio.apply()  
#!pip install motor

# Articles to index

loader = MongodbLoader(  
connection\_string=“mongodb+srv://victoriglesias5:\*\*\*\*\*\*@prueba1.hdlxqaf.mongodb.net/”,  
db\_name = “sample\_restaurants”,  
collection\_name=“restaurants”,  
filter\_criteria={“borough”: “Bronx”, “cuisine”: “Bakery”}  
)

docs = loader.load()

# Chunk text

text\_splitter = CharacterTextSplitter(chunk\_size=100,  
chunk\_overlap=0)  
chunked\_documents = text\_splitter.split\_documents(docs)

#!pip install chromadb

# Load chunked documents into the FAISS index

db = FAISS.from\_documents(chunked\_documents,  
HuggingFaceEmbeddings(model\_name=‘sentence-transformers/all-mpnet-base-v2’))

retriever = db.as\_retriever(  
#search\_type=“similarity”,  
#search\_kwargs={‘k’: 20}  
)

query = “¿Which restaurant has an id: ‘30075445’?”  
docs = db.similarity\_search(query)  
print(docs[0].page\_content)

prompt\_template = “”"

### [INST] Instruction: Answer the following question according to your knowledge about the dataset. Here is some context to help you:

{context}

### QUESTION:

{question} [/INST]  
“”"

# Create prompt from prompt template

prompt = PromptTemplate(  
input\_variables=[“context”, “question”],  
template=prompt\_template,  
)

# Create llm chain

llm\_chain = LLMChain(llm=mistral\_llm, prompt=prompt)

rag\_chain = (  
{“context”: retriever, “question”: RunnablePassthrough()}  
| llm\_chain  
)

result = rag\_chain.invoke(“According to the database, ¿What is the ‘name’ of the restaurant with a ‘restaurant\_id’ = ‘30075445’”)

result[‘context’]

resultado = result[‘text’]

print(resultado)

I beleive the error is in the context. Please help ☹

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/c/d/cdb457578aeb48159aead2ba76312812a8c3995a.png)