# Batch processing for stream dataset

**URL:** <https://discuss.huggingface.co/t/batch-processing-for-stream-dataset/21551>\
**Category:** Intermediate\
**Created:** [August 12, 2022, 8:29am UTC](https://discuss.huggingface.co/t/batch-processing-for-stream-dataset/21551 "2022-08-12T08:29:50Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Zak](https://avatars.discourse-cdn.com/v4/letter/z/8e7dd6/32.png) [@Zak](https://discuss.huggingface.co/u/Zak)\
**Post date:** [August 12, 2022, 8:29am UTC](https://discuss.huggingface.co/t/batch-processing-for-stream-dataset/21551/1 "2022-08-12T08:29:51Z")

</div>

Hello,

I am working on the Oscar dataset and trying to filter some entries based on the results of the One-Shot classification model.

Since the Oscars dataset is huge and I can’t load at once, I’ m using the stream mode of Datasets.

To filter the dataset, I use this function

```auto
def infernce_batch(examples):

   outputs, scores, texts = [], [], []

   for exemple in exemples['text']:
       res = classifier_ort(exemple, classes)
       res_mean = np.mean(np.array(res['scores']))

       outputs.append(True if res_mean > 0.5 else False)
       scores.append(res_mean)
       texts.append(exemple)

   return {"text": texts, "is_class": outputs, "score": scores} 

```

After that I call the `MAP` function to iterate throw the dataset

```auto
updated_dataset = oscar_dataset_streamed.map(infernce_batch, 
                                  batched=True, 
                                  batch_size=1000)

```

And to get back the results

```auto
for exemple in updated_dataset.take(100):

    print(exemple['id'])
    print(exemple['text'])

```

To me, this is not the most efficient way to process this dataset, as it is streaming data row by row and not by batch, and I didn’t see any difference between using the `Map` function with `batched=True` and the `MAP` function with ` batched=False`.

So my question is, is there an efficient way to process the Oscar dataset faster using stream mode and batch processing?

Thank you