# Batch inference using open source LLMs

**URL:** <https://discuss.huggingface.co/t/batch-inference-using-open-source-llms/42570>\
**Category:** 🤗Transformers\
**Created:** [June 8, 2023, 6:19pm UTC](https://discuss.huggingface.co/t/batch-inference-using-open-source-llms/42570 "2023-06-08T18:19:55Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![galprz](https://avatars.discourse-cdn.com/v4/letter/g/838e76/32.png) [@galprz](https://discuss.huggingface.co/u/galprz)\
**Post date:** [June 8, 2023, 6:19pm UTC](https://discuss.huggingface.co/t/batch-inference-using-open-source-llms/42570/1 "2023-06-08T18:19:55Z")

</div>

Hey, i am trying to perform batch inference using oasst-sft-7-llamba-30b (open assistent model but i don’t think it is really related to the model’s type) and i cannot get it to work with batch\>1 if i set the batch size to more than 1 it just output low quality text (compre to batch=1) here is the code that i use:

```auto
import bitsandbytes as bnb
load_8bit: bool = True
base_model: str = "/storage/oasst-sft-7-llama-30b"
prompt_template: str = "oasst"
model = AutoModelForCausalLM.from_pretrained(
    base_model,
    load_in_8bit=load_8bit,
    torch_dtype=torch.float16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(base_model)
generator = pipeline('text-generation', max_length=512, model=model, tokenizer=tokenizer, device=0,
                     batch_size=4, num_beams=1, top_k=40, top_p=0.1, temperature=0.0)
for outputs in tqdm(generator(KeyDataset(dataset, "text" ))):
    print([out["generated_text"] for out in outputs])

```

---

<div class="post-metadata">

**Author:** ![shubhamagarwal92](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/shubhamagarwal92/32/18417_2.png) [@shubhamagarwal92](https://discuss.huggingface.co/u/shubhamagarwal92)\
**Post date:** [August 30, 2023, 1:39am UTC](https://discuss.huggingface.co/t/batch-inference-using-open-source-llms/42570/2 "2023-08-30T01:39:09Z")

</div>

Hi @galprz !

Did you find a solution for this?