# Dealing with large objects as arguments in datasets.map

**URL:** <https://discuss.huggingface.co/t/dealing-with-large-objects-as-arguments-in-datasets-map/10946>\
**Category:** 🤗Datasets\
**Created:** [October 20, 2021, 6:16pm UTC](https://discuss.huggingface.co/t/dealing-with-large-objects-as-arguments-in-datasets-map/10946 "2021-10-20T18:16:57Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![wish](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/wish/32/2816_2.png) [@wish](https://discuss.huggingface.co/u/wish)\
**Post date:** [October 20, 2021, 6:16pm UTC](https://discuss.huggingface.co/t/dealing-with-large-objects-as-arguments-in-datasets-map/10946/1 "2021-10-20T18:16:57Z")

</div>

Hi, I’ve recently noticed that fingerprinting takes a lot of time when function arguments are large, e.g.

```auto
import datasets
from datasets import load_dataset
# Random dataset
dataset = load_dataset('glue', 'mrpc', split='test')

def do_nothing(examples, important_dict):
    return examples

# A very big dictionary
important_dict = {}
for i in range(10**8):
    important_dict[i] = 'nothing'

dataset = dataset.map(
    lambda examples: do_nothing(examples, important_dict),
    batched=True,
    load_from_cache_file=False,
    num_proc=10, # The larger the slower
    keep_in_memory=True) 

```

Fingerprinting of the `do_nothing` function takes a lot of time even when only one processor is used, but for many, it is basically N times more (which can be days when the arguments of the `do_nothing` function are really big).

It would be great if it was possible to disable fingerprinting altogether, or somehow _exclude_ certain function arguments from being fingerprinted (like the `imporant_dict`), or if not exclude then mark them as _immutable_ so that the fingerprint can be calculated only once. Or am I doing something completely wrong and there is a better way?

System info:

- Ubuntu
- Python 3.8
- datasets==1.14.0

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [October 21, 2021, 9:06am UTC](https://discuss.huggingface.co/t/dealing-with-large-objects-as-arguments-in-datasets-map/10946/2 "2021-10-21T09:06:26Z")

</div>

Hi ! Computing the fingerprint of the mapped dataset is necessary for the caching mechanism to work.  
So you can disable this with `set_caching_enabled(True)`, but every time you re-run your code it will recompute the `map` call.

The fingerprint is computed by hashing the code and the variables in your `map` function. So it takes time because it hashes your big dictionary. It could be helpful if you could specify the hash of your dictionary in advance maybe. Or if you could specify in advance the fingerprint of the resulting dataset. What do you think ?

---

<div class="post-metadata">

**Author:** ![wish](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/wish/32/2816_2.png) [@wish](https://discuss.huggingface.co/u/wish)\
**Post date:** [October 21, 2021, 3:39pm UTC](https://discuss.huggingface.co/t/dealing-with-large-objects-as-arguments-in-datasets-map/10946/3 "2021-10-21T15:39:06Z")

</div>

Thank you for the explanation. I get what `set_caching_enabled` does, but it does not disable fingerprinting of `.map` arguments.

Regarding the provision of hashes in advance, this is probably the easiest option, I’ve tested a quick workaround where the `update_fingerprint` receives another argument `precalculated_hashes - dictionary[name, hash]` that I’ve provided to the `.map` function. In my case, I’ve provided a hash for the whole function (ie `do_nothing` from the example). Does this seem right?

Irrelevant of this, would be still great if `set_caching_enabled` also killed fingerprinting - unless it is also used for something else.