# Avoiding hashing in \`map\`

**URL:** <https://discuss.huggingface.co/t/avoiding-hashing-in-map/134356>\
**Category:** 🤗Datasets\
**Created:** [January 5, 2025, 6:55pm UTC](https://discuss.huggingface.co/t/avoiding-hashing-in-map/134356 "2025-01-05T18:55:39Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![chriswolfram](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/chriswolfram/32/38646_2.png) [@chriswolfram](https://discuss.huggingface.co/u/chriswolfram)\
**Post date:** [January 5, 2025, 6:55pm UTC](https://discuss.huggingface.co/t/avoiding-hashing-in-map/134356/1 "2025-01-05T18:55:39Z")

</div>

I have a function (let’s call it `f`) that involves running an LLM on strings from the IMDB movie reviews dataset. However, when I run

```auto
imdb.map(f)

```

It grinds of a halt when I am using a large model. I think the problem is that `map` is trying to hash `f` which references a large model in its body, and this is very expensive. I found that setting `new_fingerprint='foo', load_from_cache_file=False` fixes this, but my understanding is that this still creates a cache file that will never be used.

I’ve tried using `datasets.disable_caching()` before running `map`, but it still appears to be trying to hash `f`.

What is the right way to do this? I would have thought that mapping LLMs over datasets is a common use case, and was surprised that this (obscure?) hashing problem doesn’t have an obvious workaround (unless I’m missing something)?

Thanks

---

<div class="post-metadata">

**Author:** ![ValdeJunior](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valdejunior/32/38351_2.png) [@ValdeJunior](https://discuss.huggingface.co/u/ValdeJunior)\
**Post date:** [January 6, 2025, 7:23am UTC](https://discuss.huggingface.co/t/avoiding-hashing-in-map/134356/2 "2025-01-06T07:23:57Z")

</div>

1. **Use a Simple Lambda or Wrapper for the Function** :

```auto
def f_wrapper(example):
    model = load_model() # Load the model in this function
    return model(example['text']) # Process the example with the model

```

1. **Use `batched=True` in `map()`**:

```auto
imdb = imdb.map(f_wrapper, batched=True, load_from_cache_file=False)

```

1. **Avoid Using `datasets.map()` Entirely**:

```auto
results = []
for example in imdb:
    result = f(example)
    results.append(result)

```

1. **Caching at the Function Level** :

```auto
model = None

def f(example):
    global model
    if model is None:
        model = load_model() # Load the model only once
    return model(example['text'])

```

1. **Disable the `hashing` in `map()`**:

```auto
imdb = imdb.map(f, new_fingerprint='foo', load_from_cache_file=False)

```

1. **Manual Parallelization** :

```auto
imdb = imdb.map(f_wrapper, batched=True, num_proc=4, load_from_cache_file=False)

```