# Use RAGAS with huggingface LLM

**URL:** <https://discuss.huggingface.co/t/use-ragas-with-huggingface-llm/75769>\
**Category:** Intermediate\
**Created:** [March 4, 2024, 7:23am UTC](https://discuss.huggingface.co/t/use-ragas-with-huggingface-llm/75769 "2024-03-04T07:23:58Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![Christian2901](https://avatars.discourse-cdn.com/v4/letter/c/898d66/32.png) [@Christian2901](https://discuss.huggingface.co/u/Christian2901)\
**Post date:** [March 4, 2024, 7:23am UTC](https://discuss.huggingface.co/t/use-ragas-with-huggingface-llm/75769/1 "2024-03-04T07:23:58Z")

</div>

Hello everybody,

I want to use the [RAGAS lib](https://github.com/explodinggradients/ragas/tree/main) to evaluate my RAG pipeline. The evaluation model should be a huggingface model like Llama-2, Mistral, Gemma and more. How can I implement it with the named library or is there another solution?

The examples by the team [Examples by RAGAS team](https://docs.ragas.io/en/stable/howtos/customisations/bring-your-own-llm-or-embs.html) aren’t helpful for me, because they doesn’t show, how to use specific Huggingface model. Also a specifc langchain pipeline doesn’t work, because it isn’t the required class.

Best regards  
Christian

---

<div class="post-metadata">

**Author:** ![dkoterwa](https://avatars.discourse-cdn.com/v4/letter/d/e9c0ed/32.png) [@dkoterwa](https://discuss.huggingface.co/u/dkoterwa)\
**Post date:** [March 4, 2024, 12:34pm UTC](https://discuss.huggingface.co/t/use-ragas-with-huggingface-llm/75769/2 "2024-03-04T12:34:08Z")

</div>

Hi,  
take a look at [this post](https://docs.ragas.io/en/latest/howtos/applications/compare_llms.html). I think you will find it helpful 🙂

---

<div class="post-metadata">

**Author:** ![CKeibel](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ckeibel/32/55416_2.png) [@CKeibel](https://discuss.huggingface.co/u/CKeibel)\
**Post date:** [March 7, 2024, 7:38am UTC](https://discuss.huggingface.co/t/use-ragas-with-huggingface-llm/75769/3 "2024-03-07T07:38:59Z")

</div>

If I read this correctly, some metrics also need an embedding model. The easiest way would be to put them in langchain wrappers as follows:

```auto
from transformers import pipeline, AutoTokenizer, AutoModelForCausalLM
from langchain import HuggingFacePipeline
from langchain_community.embeddings import HuggingFaceEmbeddings
from ragas.metrics import (
    answer_relevancy,
    faithfulness,
    context_recall,
    context_precision,
)
from ragas import evaluate

# embedding model
embedding_model = HuggingFaceEmbeddings("my-model-id")

# evaluator
model_id = "my-evaluator-id"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

pipe = pipeline(
    model=model,
    tokenizer=tokenizer,
    return_full_text=True, # langchain expects the full text
    task='text-generation',
    temperature=0.1, 
    repetition_penalty=1.1 # without this output begins repeating
)

evaluator = HuggingFacePipeline(pipeline=pipe)

# ragas
result = evaluate(
    dataset=dataset,
    llm=evaluator,
    embeddings=embedding_model,
    metrics=[
        context_precision,
        faithfulness,
        answer_relevancy,
        context_recall,
    ],
)

```

I had recently implemented something similar. You have to have `sentence-transformers` and the latest version of `transformers` installed, otherwise I got problems with `sentence-transformers`.

But with the latest version of `transformers` I had problems importing the `pipeline` (some tensorflow error). But you can also create them yourself.

```auto
class CustomPipeline(Pipeline):
    def _sanitize_parameters(self, **kwargs):
        preprocess_kwargs = {}
        return preprocess_kwargs, {}, {}

    def preprocess(self, text):
        return self.tokenizer(text, return_tensors="pt").to(self.device)

    def _forward(self, inputs):
        outputs = self.model.generate(**inputs)
        return outputs

    def postprocess(self, outputs):
        outputs = self.tokenizer.decode(outputs[0])
        return outputs

# init pipe
pipe = CustomPipelne(
    model=model,
    tokenizer=tokenizer,
    return_full_text=True, # langchain expects the full text
    task='text-generation',
    temperature=0.1, 
    repetition_penalty=1.1
)

# langchain wrapper
evaluator = HuggingFacePipeline(pipeline=pipe)

```

---

<div class="post-metadata">

**Author:** ![kp1264](https://avatars.discourse-cdn.com/v4/letter/k/ee7513/32.png) [@kp1264](https://discuss.huggingface.co/u/kp1264)\
**Post date:** [April 30, 2024, 3:08pm UTC](https://discuss.huggingface.co/t/use-ragas-with-huggingface-llm/75769/4 "2024-04-30T15:08:02Z")

</div>

Hii @CKeibel ,

I want to evaluate my RAG using open-source LLM instead of GPT-4. I attached code snippets for the RAGAS evaluation. I want to make sure that I am passing llm in the evaluate function correctly or not? I am getting error saying that “WARNING:ragas.llms.output\_parser:Failed to parse output. Returning None.” I have checked the several logs and I think the problem is in the passing the llm in the evaluate function.

```auto

from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_community.chat_models.huggingface import ChatHuggingFace
from langchain_community.llms import HuggingFaceHub

embedding_model = HuggingFaceEmbeddings(model_name="BAAI/bge-small-en-v1.5")
hugging_llm = HuggingFaceHub(
repo_id="HuggingFaceH4/zephyr-7b-beta",
task="text-generation",
model_kwargs={
"max_new_tokens": 512,
"top_k": 30,
"temperature": 0.1,
"repetition_penalty": 1.03,
},
)

llm = LangchainLLMWrapper(hugging_llm)
embeddings = LangchainEmbeddingsWrapper(embedding_model)

from ragas import evaluate
from ragas.metrics import context_precision
result = evaluate(
dataset=eval_dataset,
llm=llm,
embeddings=embeddings,
metrics=[
context_precision,
],
)

```

Thanks!

---

<div class="post-metadata">

**Author:** ![scarte](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@scarte](https://discuss.huggingface.co/u/scarte)\
**Post date:** [May 7, 2024, 12:41pm UTC](https://discuss.huggingface.co/t/use-ragas-with-huggingface-llm/75769/5 "2024-05-07T12:41:26Z")

</div>

Hi Keibel!  
Thank you very much for your developed response. Even so, it still gives me some errors.

When you define the following class:

```auto
class CustomPipeline(Pipeline):

```

The `Pipeline` you use is imported from `from transformers.pipelines.base import Pipeline`?

Following your second code option gives me the following error:

```auto
  File "/usr/local/lib/python3.10/dist-packages/ragas/llms/base.py", line 147, in generate_text
    result = self.langchain_llm.generate_prompt(
  File "/usr/local/lib/python3.10/dist-packages/langchain_core/language_models/llms.py", line 633, in generate_prompt
    return self.generate(prompt_strings, stop=stop, callbacks=callbacks, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/langchain_core/language_models/llms.py", line 803, in generate
    output = self._generate_helper(
  File "/usr/local/lib/python3.10/dist-packages/langchain_core/language_models/llms.py", line 670, in _generate_helper
    raise e
  File "/usr/local/lib/python3.10/dist-packages/langchain_core/language_models/llms.py", line 657, in _generate_helper
    self._generate(
  File "/usr/local/lib/python3.10/dist-packages/langchain_community/llms/huggingface_pipeline.py", line 279, in _generate
    text = response["generated_text"]
TypeError: string indices must be integers

```

Do you know what it could be? Maybe `Pipeline` isn’t importing it well.

Thank you very much first of all! 🙂  
Saioa

---

<div class="post-metadata">

**Author:** ![scarte](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@scarte](https://discuss.huggingface.co/u/scarte)\
**Post date:** [May 8, 2024, 10:31am UTC](https://discuss.huggingface.co/t/use-ragas-with-huggingface-llm/75769/6 "2024-05-08T10:31:32Z")

</div>

Hello @kp1264!

I’m looking for different ways to use Ragas with Llama-3 as the evaluator LLM, and I get this same warning that you mentioned:

> “WARNING:ragas.llms.output\_parser:Failed to parse output. Returning None.”

Besides, the metrics are Nan of course.

> {‘faithfulness’: nan}

Did you manage to solve it?

Thank you very much!

---

<div class="post-metadata">

**Author:** ![sheetalkamthe55](https://avatars.discourse-cdn.com/v4/letter/s/ecccb3/32.png) [@sheetalkamthe55](https://discuss.huggingface.co/u/sheetalkamthe55)\
**Post date:** [May 12, 2024, 4:29pm UTC](https://discuss.huggingface.co/t/use-ragas-with-huggingface-llm/75769/7 "2024-05-12T16:29:19Z")

</div>

I am getting the same error of,

> “WARNING:ragas.llms.output\_parser:Failed to parse output. Returning None.”

I tried including the trace using Langsmith to check for requests and responses. For the given input prompt I believe it is an issue of context length because I do get a blank response. I tried different LLMs but the error remains the same.

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/4/6/4604d43578364f54213b90f8547d0d04dc95679e.jpeg)

Please suggest any open-source models to evaluate.

---

<div class="post-metadata">

**Author:** ![codegood](https://avatars.discourse-cdn.com/v4/letter/c/ed655f/32.png) [@codegood](https://discuss.huggingface.co/u/codegood)\
**Post date:** [May 17, 2024, 7:37pm UTC](https://discuss.huggingface.co/t/use-ragas-with-huggingface-llm/75769/8 "2024-05-17T19:37:23Z")

</div>

What eval datasets can I use for this? I’m getting error’s in the Squad dataset.

---

<div class="post-metadata">

**Author:** ![himjoshi](https://avatars.discourse-cdn.com/v4/letter/h/7ba0ec/32.png) [@himjoshi](https://discuss.huggingface.co/u/himjoshi)\
**Post date:** [June 3, 2024, 6:26pm UTC](https://discuss.huggingface.co/t/use-ragas-with-huggingface-llm/75769/9 "2024-06-03T18:26:16Z")

</div>

@scarte @sheetalkamthe55  
Were you able to resolve the error? I’m also facing similar error. I’m using mistral 7B, and utilized the max context length of 8k, but still stuck at this part.

---

<div class="post-metadata">

**Author:** ![scarte](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@scarte](https://discuss.huggingface.co/u/scarte)\
**Post date:** [June 4, 2024, 6:41am UTC](https://discuss.huggingface.co/t/use-ragas-with-huggingface-llm/75769/10 "2024-06-04T06:41:00Z")

</div>

Hi @himjoshi !

In the end I solved it by loading the model with Ollama, as explained in this tutorial:

[![](https://us1.discourse-cdn.com/hellohellohello/original/3X/6/e/6ec15410f7c2904b8f905f32c11b53c51c7ce7bb.jpeg "Evaluating RAG using Llama 3") ](https://www.youtube.com/watch?v=Ts2wDG6OEko&t=287s)

You load the model you want with `ChatOllama`, which in your case will be [mistral](https://ollama.com/library/mistral):

```auto
from langchain_community.chat_models import ChatOllama

llm = ChatOllama(model="mistral")

```

and the embeddings if you want them also with `OllamaEmbeddings`.

Sorry if it’s not very helpful, but it’s how I was able to move forward.  
If you find another solution, please share it here! thank you!

Saioa 🙂

---

<div class="post-metadata">

**Author:** ![scarte](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@scarte](https://discuss.huggingface.co/u/scarte)\
**Post date:** [June 4, 2024, 6:45am UTC](https://discuss.huggingface.co/t/use-ragas-with-huggingface-llm/75769/11 "2024-06-04T06:45:03Z")

</div>

Hi @codegood !

You can use “[explodinggradients/amnesty\_qa](https://huggingface.co/datasets/explodinggradients/amnesty_qa)” dataset, loading it as:

```auto
from datasets import load_dataset

data = load_dataset("explodinggradients/amnesty_qa", "english_v2")

```

---

<div class="post-metadata">

**Author:** ![Saurabh8255](https://avatars.discourse-cdn.com/v4/letter/s/3da27b/32.png) [@Saurabh8255](https://discuss.huggingface.co/u/Saurabh8255)\
**Post date:** [July 10, 2024, 11:42am UTC](https://discuss.huggingface.co/t/use-ragas-with-huggingface-llm/75769/12 "2024-07-10T11:42:39Z")

</div>

Hey I am facing same error did you solve the error please let me know  
I am using code as  
from ragas.llms import LangchainLLMWrapper  
from ragas.embeddings import LangchainEmbeddingsWrapper  
from langchain\_community.embeddings import HuggingFaceEmbeddings  
from langchain\_community.chat\_models.huggingface import ChatHuggingFace  
from langchain.llms import HuggingFaceHub

embedding\_model = HuggingFaceEmbeddings(model\_name=“BAAI/bge-small-en-v1.5”)  
hugging\_llm = HuggingFaceHub(  
repo\_id=“HuggingFaceH4/zephyr-7b-beta”,  
task=“text-generation”,  
model\_kwargs={  
“max\_new\_tokens”: 512,  
“top\_k”: 30,  
“temperature”: 0.1,  
“repetition\_penalty”: 1.03,  
},  
)

import nest\_asyncio  
nest\_asyncio.apply()  
from datasets import Dataset  
import os  
from ragas import evaluate  
from ragas.metrics import faithfulness, answer\_correctness

data\_samples = {  
‘question’: [‘When was the first super bowl?’, ‘Who won the most super bowls?’],  
‘answer’: [‘The first superbowl was held on Jan 15, 1967’, ‘The most super bowls have been won by The New England Patriots’],  
‘contexts’ : [[‘The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,’],  
[‘The Green Bay Packers…Green Bay, Wisconsin.’,‘The Packers compete…Football Conference’]],  
‘ground\_truth’: [‘The first superbowl was held on January 15, 1967’, ‘The New England Patriots have won the Super Bowl a record six times’]  
}

dataset = Dataset.from\_dict(data\_samples)

score = evaluate(dataset,metrics=[faithfulness,answer\_correctness], llm=hugging\_llm, embeddings = embedding\_model )  
score.to\_pandas()

Output:  
WARNING:ragas.llms.output\_parser:Failed to parse output. Returning None.  
WARNING:ragas.llms.output\_parser:Failed to parse output. Returning None.  
WARNING:ragas.llms.output\_parser:Failed to parse output. Returning None.

---

<div class="post-metadata">

**Author:** ![iammano](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/iammano/32/29438_2.png) [@iammano](https://discuss.huggingface.co/u/iammano)\
**Post date:** [July 16, 2024, 8:44am UTC](https://discuss.huggingface.co/t/use-ragas-with-huggingface-llm/75769/13 "2024-07-16T08:44:29Z")

</div>

Hey @CKeibel , By default when importing the RAGAs in my jupyter notebook it’s throwing an error regarding the OAI keys, did you come across this kind of error while implementing it? If possible can you provide me the package version you are using?

 ![Screenshot from 2024-07-16 14-12-54](https://us1.discourse-cdn.com/hellohellohello/original/3X/f/7/f7e38c5a4e2ae3605122cf2a47cdc76104c21ce2.png)

Thanks,  
Mano

---

<div class="post-metadata">

**Author:** ![ctrlaltdelete](https://avatars.discourse-cdn.com/v4/letter/c/ecc23a/32.png) [@ctrlaltdelete](https://discuss.huggingface.co/u/ctrlaltdelete)\
**Post date:** [July 30, 2024, 7:53am UTC](https://discuss.huggingface.co/t/use-ragas-with-huggingface-llm/75769/14 "2024-07-30T07:53:00Z")

</div>

Hi @scarte !  
Could you tell me which model do you load by Ollama? I load llama3.1 by using Ollama, but still have the error 😭

---

<div class="post-metadata">

**Author:** ![sohammhatre](https://avatars.discourse-cdn.com/v4/letter/s/b5a626/32.png) [@sohammhatre](https://discuss.huggingface.co/u/sohammhatre)\
**Post date:** [October 10, 2024, 11:51am UTC](https://discuss.huggingface.co/t/use-ragas-with-huggingface-llm/75769/15 "2024-10-10T11:51:02Z")

</div>

I’m getting the error

```auto
/usr/local/lib/python3.10/dist-packages/ragas/metrics/base.py in init(self, run_config)
    151 f"Metric '{self.name}' has no valid LLM provided (self.llm is None). Please initantiate a the metric with an LLM to run." # noqa
    152 )
--> 153 self.llm.set_run_config(run_config)
    154 
    155 

AttributeError: 'Mistral' object has no attribute 'set_run_config

```

for

```python
from langchain_ollama.chat_models import ChatOllama
from langchain_ollama.embeddings import OllamaEmbeddings
from ragas import evaluate
from ragas.metrics import answer_relevancy
from datasets import Dataset
import json

# The model should be specified using the `model` parameter
req_llm = ChatOllama(model="mistral")

embeddings = OllamaEmbeddings(model="mistral")

# for m in metrics:
# m. __setattr__ ("llm", req_llm)
# m. __setattr__ ("embeddings", embeddings)

data = [
    {
        'question': 'Which is the most popular global sport?',
        'contexts': [
            "The popularity of sports can be measured in various ways, including TV viewership, social media presence, number of participants, and economic impact.",
            "Football is undoubtedly the world's most popular sport with major events like the FIFA World Cup.",
            "Sports personalities like Ronaldo and Messi draw a followership of more than 4 billion people."
        ],
        'response': 'Football is the most popular sport with around 4 billion followers worldwide',
        'answer': 'Football is the most popular sport with around 4 billion followers worldwide'
    }
]

# Convert the list to a Hugging Face Dataset object
dataset = Dataset.from_list(data)

# Step 3: Run the evaluation
results = evaluate(
    dataset=dataset, # Use the Hugging Face Dataset object
    metrics=[answer_relevancy],
    llm=req_llm,
    embeddings=embeddings,
)

# Step 4: Print the results
print(json.dumps(results, indent=3))

```

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [October 10, 2024, 12:40pm UTC](https://discuss.huggingface.co/t/use-ragas-with-huggingface-llm/75769/16 "2024-10-10T12:40:44Z")

</div>

It seems to be a bug.

> <https://github.com/explodinggradients/ragas/issues/595>
>
> Hi, 
> 
> I've been encountering the error shown below when I try to run \`evaluate…()\` inside one of my projects (run on a Docker env). For some reason it works when I try to run it on a Notebook (using the Azure example in the docs). I've made sure that the Azure Model on both the docker image and notebook have the same configuration.
> 
> \`\`\`
> File "/usr/local/lib/python3.11/site-packages/ragas/evaluation.py", line 179, in evaluate
> \[m.init(run\_config) for m in metrics\]
> File "/usr/local/lib/python3.11/site-packages/ragas/evaluation.py", line 179, in \<listcomp\>
> \[m.init(run\_config) for m in metrics\]
> ^^^^^^^^^^^^^^^^^^
> File "/usr/local/lib/python3.11/site-packages/ragas/metrics/base.py", line 116, in init
> self.llm.set\_run\_config(run\_config)
> ^^^^^^^^^^^^^^^^^^^^^^^
> AttributeError: 'AzureChatOpenAI' object has no attribute 'set\_run\_config'
> \`\`\`
> 
> Implementation-wise, the notebook is purely straight-forward but for the one inside the docker, I call \`evaluate()\` inside an async function. I'm not sure if this would have an effect but I invoke \`evaluate()\` the same way as I did in the Notebook:
> 
> \`ragas\_result = evaluate(ragas\_dataset, metrics=DEFAULT\_METRICS, llm=DEFAULT\_LLM\_MODEL, embeddings=DEFAULT\_EMBEDDINGS)\`
> 
> Here's the config for the Azure model I'm using (I got the settings from the notebook and they match the one inside the docker image):
> 
> \`AzureChatOpenAI(client=\<openai.resources.chat.completions.Completions object at 0x157f7b910\>, async\_client=\<openai.resources.chat.completions.AsyncCompletions object at 0x157f7ee10\>, model\_name='gpt-35-turbo', temperature=0.0, openai\_api\_key='...', openai\_proxy='', request\_timeout=60, azure\_endpoint='https://...openai.azure.com/', deployment\_name='...gpt-35-turbo', openai\_api\_version='2023-03-15-preview', openai\_api\_type='azure')
> \`
> 
> Here are also some library versions:
> \`\`\`
> langchain==0.1.6
> langchain-community==0.0.19
> langchain-core==0.1.22
> langchain-openai==0.0.5
> ragas==0.1.0
> python==3.11.5
> \`\`\`
> 
> Any help would be highly appreciated. I can provide more information if needed. Thank you.

---

<div class="post-metadata">

**Author:** ![RyanTree](https://avatars.discourse-cdn.com/v4/letter/r/9de053/32.png) [@RyanTree](https://discuss.huggingface.co/u/RyanTree)\
**Post date:** [October 20, 2024, 2:15am UTC](https://discuss.huggingface.co/t/use-ragas-with-huggingface-llm/75769/17 "2024-10-20T02:15:03Z")

</div>

Hi @scarte ,

dataset=load\_dataset(“explodinggradients/amnesty\_qa”,“english\_v2”,trust\_remote\_code=True)  
dataset\_subset = dataset[“eval”].select(range(2))

llm = ChatOllama(model=“mistral”)  
embedding\_model = OllamaEmbeddings(model=“mistral”)

result = evaluate(

dataset=dataset\_subset,

llm=llm,

embeddings=embedding\_model,

metrics=[

context\_precision

],

run\_config=RunConfig(timeout=180.0, max\_workers=16)

)

I did ollama run mistral and it was launched successfully but with the above mentioned code, it is still having response error(). Would be super appreciated if you could take a look?

---

<div class="post-metadata">

**Author:** ![laoyang103](https://avatars.discourse-cdn.com/v4/letter/l/ccd318/32.png) [@laoyang103](https://discuss.huggingface.co/u/laoyang103)\
**Post date:** [March 17, 2025, 10:25am UTC](https://discuss.huggingface.co/t/use-ragas-with-huggingface-llm/75769/18 "2025-03-17T10:25:04Z")

</div>

try this:

> <https://github.com/Microflow-IO/modsecurity-for-anylog/blob/main/ragas_local_llm.ipynb>