# Using a generator for the map function in an iterable dataset **URL:** \ **Category:** Beginners\ **Created:** [December 29, 2023, 5:45am UTC](https://discuss.huggingface.co/t/using-a-generator-for-the-map-function-in-an-iterable-dataset/67354 "2023-12-29T05:45:20Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![mikesol](https://avatars.discourse-cdn.com/v4/letter/m/41988e/32.png) [@mikesol](https://discuss.huggingface.co/u/mikesol)\ **Post date:** [December 29, 2023, 5:45am UTC](https://discuss.huggingface.co/t/using-a-generator-for-the-map-function-in-an-iterable-dataset/67354/1 "2023-12-29T05:45:20Z")
Hi! I have an audio dataset, and I’m using the standard `load_dataset` pattern to get the dataset into memory. ```python dataset = load_dataset("audiofolder", data_dir="../data", streaming=streaming) ``` And then I use `map` with `batched=True` to do preprocessing. The problem is that the preprocessing explodes the size of the dataset due to the batching. Is it possible to pass a generator to `map` instead of a function? That’d fix the problem in that a single sample would be emitted whenever `next` is called instead of making the whole batch in one go. I know that it’s possible to create an `IterableDataset` from a generator, and I can do that as a last resort, but the `load_dataset` is quite nice and saves me from doing a bunch of manual librosa stuff. Thanks in advance for any tips! ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [December 30, 2023, 12:52am UTC](https://discuss.huggingface.co/t/using-a-generator-for-the-map-function-in-an-iterable-dataset/67354/2 "2023-12-30T00:52:46Z")
You can reduce the `batch_size` to emit examples small batches a a time. Or simply not use batching