# Reduce inference time with batches

**URL:** <https://discuss.huggingface.co/t/reduce-inference-time-with-batches/9985>\
**Category:** Beginners\
**Created:** [September 14, 2021, 3:00pm UTC](https://discuss.huggingface.co/t/reduce-inference-time-with-batches/9985 "2021-09-14T15:00:00Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![otatopeht](https://avatars.discourse-cdn.com/v4/letter/o/e19b73/32.png) [@otatopeht](https://discuss.huggingface.co/u/otatopeht)\
**Post date:** [September 14, 2021, 3:00pm UTC](https://discuss.huggingface.co/t/reduce-inference-time-with-batches/9985/1 "2021-09-14T15:00:00Z")

</div>

Hi everyone.

I want to further improve the inference time from BERT.  
Here is the code below:

```auto
for sentence in list(data_dict.values()):
    tokens = {'input_ids': [], 'attention_mask': []}
    new_tokens = tokenizer.encode_plus(sentence, max_length=512,
                                       truncation=True, padding='max_length',
                                       return_tensors='pt',
                                       return_attention_mask=True)
    tokens['input_ids'].append(new_tokens['input_ids'][0])
    tokens['attention_mask'].append(new_tokens['attention_mask'][0])

    # reformat list of tensors into single tensor
    tokens['input_ids'] = torch.stack(tokens['input_ids'])
    tokens['attention_mask'] = torch.stack(tokens['attention_mask'])

    outputs = model(**tokens)
    embeddings = outputs[0]

```

How can I provide batches like in training instead of the whole dataset?