# Dataset.map() with batching and multiprocessing

**URL:** <https://discuss.huggingface.co/t/dataset-map-with-batching-and-multiprocessing/75439>\
**Category:** 🤗Datasets\
**Created:** [March 1, 2024, 6:57am UTC](https://discuss.huggingface.co/t/dataset-map-with-batching-and-multiprocessing/75439 "2024-03-01T06:57:42Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![varadhbhatnagar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/varadhbhatnagar/32/31383_2.png) [@varadhbhatnagar](https://discuss.huggingface.co/u/varadhbhatnagar)\
**Post date:** [March 1, 2024, 6:57am UTC](https://discuss.huggingface.co/t/dataset-map-with-batching-and-multiprocessing/75439/1 "2024-03-01T06:57:42Z")

</div>

I am seeing different results when I do

`dataset.map(..., batched=True, num_proc=4)`  
vs  
`dataset.map(..., batched=True, num_proc=16)`

Here is the output:

```auto
Map (num_proc=4): 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 500/500 [00:00<00:00, 1019.49 examples/s]
Map (num_proc=16): 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 500/500 [00:00<00:00, 1684.70 examples/s]
Dataset({
    features: ['input_ids', 'attention_mask'],
    num_rows: 148
}) Dataset({
    features: ['input_ids', 'attention_mask'],
    num_rows: 143
})

```

Is it expected to have different `num_rows` in the output?

---

<div class="post-metadata">

**Author:** ![varadhbhatnagar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/varadhbhatnagar/32/31383_2.png) [@varadhbhatnagar](https://discuss.huggingface.co/u/varadhbhatnagar)\
**Post date:** [March 5, 2024, 6:31am UTC](https://discuss.huggingface.co/t/dataset-map-with-batching-and-multiprocessing/75439/2 "2024-03-05T06:31:41Z")

</div>

@muellerzr Any thoughts on this?