# Problems while filtering large datasets using \`map\`

**URL:** <https://discuss.huggingface.co/t/problems-while-filtering-large-datasets-using-map/8335>\
**Category:** 🤗Datasets\
**Created:** [July 14, 2021, 10:19am UTC](https://discuss.huggingface.co/t/problems-while-filtering-large-datasets-using-map/8335 "2021-07-14T10:19:05Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![pcuenq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/pcuenq/32/32135_2.png) [@pcuenq](https://discuss.huggingface.co/u/pcuenq)\
**Post date:** [July 14, 2021, 10:19am UTC](https://discuss.huggingface.co/t/problems-while-filtering-large-datasets-using-map/8335/1 "2021-07-14T10:19:05Z")

</div>

As part of [out flax-jax-community-week project](https://discuss.huggingface.co/t/dall-e-mini-version/7324), I encountered a problem when attempting to filter a large dataset to select only the items matching a certain condition. I’m processing the items with `map` using batched mode, but some of the batches contain no matching items. We need to return an empty result in that case, but I’m having trouble doing so. If I return an empty dictionary `{}`, then I encounter schema errors. I think they happen when empty batches are followed by non-empty ones. If I return a dictionary with keys containing empty lists `{'column_name': []}`, then I get an index out of bounds error.

I filed [this issue](https://github.com/huggingface/datasets/issues/2644) describing the problem in slightly more detail. If this is not a bug, or someone else has found a workaround, that’d be great to know.

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [July 20, 2021, 9:24am UTC](https://discuss.huggingface.co/t/problems-while-filtering-large-datasets-using-map/8335/2 "2021-07-20T09:24:51Z")

</div>

Thanks for reporting ! Hopefully we can have a fix soon 🙂

---

<div class="post-metadata">

**Author:** ![pcuenq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/pcuenq/32/32135_2.png) [@pcuenq](https://discuss.huggingface.co/u/pcuenq)\
**Post date:** [July 21, 2021, 10:52pm UTC](https://discuss.huggingface.co/t/problems-while-filtering-large-datasets-using-map/8335/3 "2021-07-21T22:52:59Z")

</div>

I just took [a first go](https://github.com/huggingface/datasets/pull/2698) at it. Let me know if I’m off track!