# How to cache tokenization for the data

**URL:** <https://discuss.huggingface.co/t/how-to-cache-tokenization-for-the-data/68869>\
**Category:** Beginners\
**Created:** [January 12, 2024, 12:42am UTC](https://discuss.huggingface.co/t/how-to-cache-tokenization-for-the-data/68869 "2024-01-12T00:42:21Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kason123](https://avatars.discourse-cdn.com/v4/letter/k/54ee81/32.png) [@Kason123](https://discuss.huggingface.co/u/Kason123)\
**Post date:** [January 12, 2024, 12:42am UTC](https://discuss.huggingface.co/t/how-to-cache-tokenization-for-the-data/68869/1 "2024-01-12T00:42:21Z")

</div>

Hi I tokenize my data as follows but every time I try to run it, the code does the mapping scratch although there is a cached one in the respective folder. Can anyone help to avoid this redundant process?

```auto
tokenizer=AutoTokenizer.from_pretrained(script_args.model_name, cache_dir="hf_cache_dir", local_files_only=True)
def tokenize_function(example):
    return tokenizer(example["text"], truncation=True)

tokenized_datasets = dataset.map(tokenize_function, batched=True, load_from_cache_file=True)

```

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [January 15, 2024, 7:11pm UTC](https://discuss.huggingface.co/t/how-to-cache-tokenization-for-the-data/68869/2 "2024-01-15T19:11:32Z")

</div>

Hi! This is a known issue: [AutoTokenizer hash value got change after datasets.map · Issue #3638 · huggingface/datasets · GitHub](https://github.com/huggingface/datasets/issues/3638). It requires rewriting large parts of the `tokenizers` lib, so we haven’t fixed it yet. In the meantime, you can bypass it by setting the tokenizer’s state with a dummy call such as `_ = tokenizer("Dummy text", truncation=True)` before the map.

---

<div class="post-metadata">

**Author:** ![Kason123](https://avatars.discourse-cdn.com/v4/letter/k/54ee81/32.png) [@Kason123](https://discuss.huggingface.co/u/Kason123)\
**Post date:** [January 16, 2024, 11:59pm UTC](https://discuss.huggingface.co/t/how-to-cache-tokenization-for-the-data/68869/3 "2024-01-16T23:59:44Z")

</div>

Thanks for your suggestion but this didn’t work out for me.