# Mapping large datasets

**URL:** <https://discuss.huggingface.co/t/mapping-large-datasets/14569>\
**Category:** 🤗Datasets\
**Created:** [February 10, 2022, 6:48am UTC](https://discuss.huggingface.co/t/mapping-large-datasets/14569 "2022-02-10T06:48:17Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![danielbubiola](https://avatars.discourse-cdn.com/v4/letter/d/49beb7/32.png) [@danielbubiola](https://discuss.huggingface.co/u/danielbubiola)\
**Post date:** [February 10, 2022, 6:48am UTC](https://discuss.huggingface.co/t/mapping-large-datasets/14569/1 "2022-02-10T06:48:17Z")

</div>

I have a large dataset of over 10000 audio data. whenever I tried to map it, it exceeded available memory. I have used batching but still it failed. any better way to do it?

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [February 10, 2022, 1:17pm UTC](https://discuss.huggingface.co/t/mapping-large-datasets/14569/2 "2022-02-10T13:17:11Z")

</div>

Hi! You can lower memory usage by reducing `batch_size` in `map` (default is 1000).

---

<div class="post-metadata">

**Author:** ![danielbubiola](https://avatars.discourse-cdn.com/v4/letter/d/49beb7/32.png) [@danielbubiola](https://discuss.huggingface.co/u/danielbubiola)\
**Post date:** [February 12, 2022, 11:34am UTC](https://discuss.huggingface.co/t/mapping-large-datasets/14569/3 "2022-02-12T11:34:49Z")

</div>

Thanks Mario. I have used that. But that didn’t work for me. any other Ideas?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [February 15, 2022, 3:51pm UTC](https://discuss.huggingface.co/t/mapping-large-datasets/14569/4 "2022-02-15T15:51:31Z")

</div>

I think the parameter you’re looking for is the `writer_batch_size`: it’s the number of rows per write operation for the cache file writer.

Default is currently 1,000. Higher value makes the processing do fewer data flushing, lower value consume less temporary memory while running `.map()`.

Feel free to set both `batch_size` and `writer_batch_size` to lower values 🙂

cc @mariosasko don’t you think we should have writer\_batch\_size = batch\_size by default ?

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [February 15, 2022, 4:05pm UTC](https://discuss.huggingface.co/t/mapping-large-datasets/14569/5 "2022-02-15T16:05:35Z")

</div>

@lhoestq Yes, I agree. We can set the default value of `writer_batch_size` to `None` in the signature of `map`, and do:

```python
writer_batch_size = batch_size if writer_batch_size is None else writer_batch_size

```

in the body.