# Error using datasets with pipeline for text generation

**URL:** <https://discuss.huggingface.co/t/error-using-datasets-with-pipeline-for-text-generation/94747>\
**Category:** 🤗Datasets\
**Created:** [June 29, 2024, 10:53pm UTC](https://discuss.huggingface.co/t/error-using-datasets-with-pipeline-for-text-generation/94747 "2024-06-29T22:53:33Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![radicalprotnns](https://avatars.discourse-cdn.com/v4/letter/r/97f17d/32.png) [@radicalprotnns](https://discuss.huggingface.co/u/radicalprotnns)\
**Post date:** [June 29, 2024, 10:53pm UTC](https://discuss.huggingface.co/t/error-using-datasets-with-pipeline-for-text-generation/94747/1 "2024-06-29T22:53:33Z")

</div>

I am performing inference with llama-3-8b for the purposes of text generation. If I only pass 1 prompt at a time, my code works. However, since I have a for loop that loops over 500 prompts and calling the model for each prompt, hugging face gave me the following warning:

> UserWarning: You seem to be using the pipelines sequentially on GPU. In order to maximize efficiency please use a dataset

I have thus tested out the suggestion in the warning but I am getting an error and I can’t figure out what’s the mistake. Here’s a snippet of my code:

```auto
from transformers import (
    AutoTokenizer,
    AutoModelForCausalLM,
    pipeline
)
import torch
from datasets import Dataset
from transformers.pipelines.pt_utils import KeyDataset

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3-8B-Instruct",
    load_in_8bit=True,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")

pipe = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    temperature=1
)

message = [
    {
        "role": "system",
        "content": "You are a friendly chatbot who always responds in the style of a pirate",
    },
    {"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
]
messages = [message, message]
dataset_messages = {"messages": messages}
messages_dataset = Dataset.from_dict(dataset_messages)

sequences = pipe(
    KeyDataset(messages_dataset,"messages"),
    max_new_tokens=200,
    do_sample=True,
    return_full_text=False,
    top_k=1
)

for seq in sequences:
    print(f"Result: {seq['generated_text']}")

```

However, I get the following error:

> text\_generation.py", line 266, in preprocess  
> prefix + prompt\_text,  
> ~~~^~~~~~~~~  
> TypeError: can only concatenate str (not “list”) to str

when I call pipe on the datasets object. My datasets version is 2.18.0, I am aware of [this issue](https://github.com/huggingface/datasets/issues/5406) but I believe this is not due to the update anymore. Can anyone help me figure out if I used datasets correctly?

Thank you so much!

---

<div class="post-metadata">

**Author:** ![JanaSJ](https://avatars.discourse-cdn.com/v4/letter/j/50afbb/32.png) [@JanaSJ](https://discuss.huggingface.co/u/JanaSJ)\
**Post date:** [December 29, 2024, 10:19pm UTC](https://discuss.huggingface.co/t/error-using-datasets-with-pipeline-for-text-generation/94747/2 "2024-12-29T22:19:14Z")

</div>

Hi there, were you able to solve this in some way? I have the same problem (datasets version 2.19.1)

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [December 30, 2024, 2:10am UTC](https://discuss.huggingface.co/t/error-using-datasets-with-pipeline-for-text-generation/94747/3 "2024-12-30T02:10:16Z")

</div>

Perhaps this?

> <https://stackoverflow.com/questions/77159136/efficiently-using-hugging-face-transformers-pipelines-on-gpu-with-large-datasets>

---

<div class="post-metadata">

**Author:** ![JanaSJ](https://avatars.discourse-cdn.com/v4/letter/j/50afbb/32.png) [@JanaSJ](https://discuss.huggingface.co/u/JanaSJ)\
**Post date:** [December 30, 2024, 3:40pm UTC](https://discuss.huggingface.co/t/error-using-datasets-with-pipeline-for-text-generation/94747/4 "2024-12-30T15:40:06Z")

</div>

I have a running code which is similar to the answer proposed to that stack overflow question (i.e. a function that returns a model response using a huggingface pipeline and has one pd row as input and then applying that function on the whole pd df).

However, I would like to test if directly using an iterable of the whole data (e.g. a KeyDataset object) as input to the pipeline, which then enables batching, speeds things up. But that’s where I get the same error as OP

---

<div class="post-metadata">

**Author:** ![Alanturner2](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/alanturner2/32/37542_2.png) [@Alanturner2](https://discuss.huggingface.co/u/Alanturner2)\
**Post date:** [December 30, 2024, 3:42pm UTC](https://discuss.huggingface.co/t/error-using-datasets-with-pipeline-for-text-generation/94747/5 "2024-12-30T15:42:44Z")

</div>

```auto
from transformers import (
    AutoTokenizer,
    AutoModelForCausalLM,
    pipeline
)
import torch
from datasets import Dataset
from transformers.pipelines.pt_utils import KeyDataset

# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3-8B-Instruct",
    load_in_8bit=True,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")

# Create pipeline for text generation
pipe = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    temperature=1
)

# Define the messages
message = [
    {
        "role": "system",
        "content": "You are a friendly chatbot who always responds in the style of a pirate",
    },
    {"role": "user", "content": "How many helicopters can a human eat in one sitting?"}
]

# Create a dataset with strings, not lists of messages
dataset_messages = {"messages": [f"{msg['content']}" for msg in message] * 2} # Repeat for 2 entries
messages_dataset = Dataset.from_dict(dataset_messages)

# Use KeyDataset to pass to the pipeline
sequences = pipe(
    KeyDataset(messages_dataset, "messages"),
    max_new_tokens=200,
    do_sample=True,
    return_full_text=False,
    top_k=1
)

# Print results
for seq in sequences:
    print(f"Result: {seq['generated_text']}")

```

---

<div class="post-metadata">

**Author:** ![KateWinslet](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/katewinslet/32/26764_2.png) [@KateWinslet](https://discuss.huggingface.co/u/KateWinslet)\
**Post date:** [December 30, 2024, 6:59pm UTC](https://discuss.huggingface.co/t/error-using-datasets-with-pipeline-for-text-generation/94747/6 "2024-12-30T18:59:36Z")

</div>

> [@radicalprotnns](#):
>
> I am performing inference with llama-3-8b for the purposes of text generation. If I only pass 1 prompt at a time, my code works. However, since I have a for loop that loops over 500 prompts and calling the model for each prompt, hugging face gave me the following warning:
> 
> > UserWarning: You seem to be using the pipelines sequentially on GPU. In order to maximize efficiency please use a dataset
> 
> I have thus tested out the suggestion in the warning but I am getting an error and I can’t figure out what’s the mistake. Here’s a snippet of my code:
> 
> ```auto
> from transformers import (
> AutoTokenizer,
> AutoModelForCausalLM,
> pipeline
> )
> import torch
> from datasets import Dataset
> from transformers.pipelines.pt_utils import KeyDataset
> 
> model = AutoModelForCausalLM.from_pretrained(
> "meta-llama/Meta-Llama-3-8B-Instruct",
> load_in_8bit=True,
> device_map="auto",
> )
> tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")
> 
> pipe = pipeline(
> "text-generation",
> model=model,
> tokenizer=tokenizer,
> torch_dtype=torch.bfloat16,
> device_map="auto",
> temperature=1
> )
> 
> message = [
> {
> "role": "system",
> "content": "You are a friendly chatbot who always responds in the style of a pirate",
> },
> {"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
> ]
> messages = [message, message]
> dataset_messages = {"messages": messages}
> messages_dataset = Dataset.from_dict(dataset_messages)
> 
> sequences = pipe(
> KeyDataset(messages_dataset,"messages"),
> max_new_tokens=200,
> do_sample=True,
> return_full_text=False,
> top_k=1
> )
> 
> for seq in sequences:
> print(f"Result: {seq['generated_text']}")
> 
> ```
> 
> However, I get the following error:
> 
> > text\_generation.py", line 266, in preprocess  
> > prefix + prompt\_text,
> > 
> > ```auto
> > TypeError: can only concatenate str (not “list”) to str
> > 
> > ```
> 
> when I call pipe on the datasets object. My datasets version is 2.18.0, I am aware of [this issue](https://github.com/huggingface/datasets/issues/5406) but I believe this is not due to the update anymore. Can anyone help me figure out if I used datasets correctly?
> 
> Thank you so much!

The error occurs because the pipeline expects strings, not lists. Flatten the message dictionary to a string for each prompt before passing it to the dataset.