# Caching a dataset with map() when loaded with from\_dict() **URL:** \ **Category:** 🤗Datasets\ **Created:** [June 26, 2021, 7:44am UTC](https://discuss.huggingface.co/t/caching-a-dataset-with-map-when-loaded-with-from-dict/7411 "2021-06-26T07:44:34Z")\ **Posts on this page:** 4\ **Page:** 1

**Author:** ![jasonyoun](https://avatars.discourse-cdn.com/v4/letter/j/f14d63/32.png) [@jasonyoun](https://discuss.huggingface.co/u/jasonyoun)\ **Post date:** [June 26, 2021, 7:44am UTC](https://discuss.huggingface.co/t/caching-a-dataset-with-map-when-loaded-with-from-dict/7411/1 "2021-06-26T07:44:35Z")

Using **datasets 1.8.0**. **Normal? Situations** If I use [`load_dataset()`](https://huggingface.co/docs/datasets/package_reference/loading_methods.html#datasets.load_dataset) to load data, it generates cache files. If you then apply [`.map()`](https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.map) on that dataset, corresponding cache files are generated as expected. Following is a simple code to reproduce the results. ```auto from datasets import load_dataset, Dataset def add_prefix(example): example['sentence1'] = 'My sentence: ' + example['sentence1'] return example def main(): dataset = load_dataset('glue', 'mrpc', split='train') print(dataset) dataset = dataset.map(add_prefix) print(dataset) if __name__ == " __main__": main() ``` Running the above will generate the following output. As we can see with the tqdm progress bar, the `.map()` is applied and the cache files are saved accordingly. ```auto Downloading: 28.8kB [00:00, 25.6MB/s] Downloading: 28.7kB [00:00, 28.8MB/s] Downloading and preparing dataset glue/mrpc (download: 1.43 MiB, generated: 1.43 MiB, post-processed: Unknown size, total: 2.85 MiB) to /home/jasonyoun/.cache/huggingface/datasets/glue/mrpc/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad... Downloading: 6.22kB [00:00, 7.73MB/s] Downloading: 1.05MB [00:00, 9.16MB/s] Downloading: 441kB [00:00, 4.83MB/s] 0 examples [00:00, ? examples/s]2021-06-26 00:07:58.740056: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0 Dataset glue downloaded and prepared to /home/jasonyoun/.cache/huggingface/datasets/glue/mrpc/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad. Subsequent calls will reuse this data. Dataset({ features: ['idx', 'label', 'sentence1', 'sentence2'], num_rows: 3668 }) 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3668/3668 [00:00<00:00, 20903.25ex/s] Dataset({ features: ['idx', 'label', 'sentence1', 'sentence2'], num_rows: 3668 }) ``` Running the exact code again will use the saved cache for both `load_dataset()` and `.map()`. ```auto Reusing dataset glue (/home/jasonyoun/.cache/huggingface/datasets/glue/mrpc/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad) Dataset({ features: ['idx', 'label', 'sentence1', 'sentence2'], num_rows: 3668 }) Loading cached processed dataset at /home/jasonyoun/.cache/huggingface/datasets/glue/mrpc/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad/cache-53c01e7abeb2b20b.arrow Dataset({ features: ['idx', 'label', 'sentence1', 'sentence2'], num_rows: 3668 }) ``` **Not-so-normal Situations** However, using [`from_dict()`](https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.from_dict) to load dataset from the memory **does not** generate cache files. Because the dataset did not initially generate the cache files, applying `.map()` on this dataset also does not generate cache files. This can be observed in [arrow\_dataset.py](https://github.com/huggingface/datasets/blob/master/src/datasets/arrow_dataset.py) lines 1803-1813. ```auto # Check if we've already cached this computation (indexed by a hash) if self.cache_files: # !!!THIS WILL BE AN EMPTY LIST when loaded with .from_dict()!!! if cache_file_name is None: # we create a unique hash from the function, # current dataset file and the mapping args cache_file_name = self._get_cache_file_path(new_fingerprint) if os.path.exists(cache_file_name) and load_from_cache_file: logger.warning("Loading cached processed dataset at %s", cache_file_name) info = self.info.copy() info.features = features return Dataset.from_file(cache_file_name, info=info, split=self.split) ``` **Why is this a problem?** In the thread below, it was suggested that one use `torch.distributed.barrier()` when using a distributed framework to process a large dataset. > [@Slow processing with map when using deepspeed or fairscale](https://discuss.huggingface.co/t/slow-processing-with-map-when-using-deepspeed-or-fairscale/7229/11): > > I was also able to reproduce the result. Thanks for the prompt support @stas and @sgugger. For my specific use-case, I create the dataset using the `.from_dict()` method. I then process the dataset using `.map()` using the main process (no cache files get saved automatically unless you specify the `cache_file_name` parameter of `.map()`). Once the non-main processes are resumed, they are not able to locate the cache files. **Possible Workarounds** 1. I could save all data (maybe a csv) and load it using `load_dataset()`. This way, cache files are generated and all is well. However, my dataset is large (~100GB), and multiple processing using `.map()` further increases the data size making it very difficult to store and load data efficiently with limited HW resources. That’s why I’m using `from_dict()` to load smaller data on-the-fly. 2. Instead of using in-memory data to create the dataset, I could first save the data as csv or json. I could then use `load_dataset('csv' or 'json', ...)`. I will try this method for now. I think it will work. ---

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [July 20, 2021, 9:14am UTC](https://discuss.huggingface.co/t/caching-a-dataset-with-map-when-loaded-with-from-dict/7411/2 "2021-07-20T09:14:01Z")

Hi ! When `dataset.cache_files` is empty (i.e. when your dataset comes from python objects, not from data from your disk), the `map()` method doesn’t know where to write the resulting dataset. In this case, you have to pass `cache_file_name=` manually to `map()`. For the subsequent calls to `.map()`, you won’t need to specify this anymore, since it will store the cache files in the same directory as the path your provided in the first place. ---

**Author:** ![jxm](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jxm/32/29614_2.png) [@jxm](https://discuss.huggingface.co/u/jxm)\ **Post date:** [November 16, 2022, 1:34pm UTC](https://discuss.huggingface.co/t/caching-a-dataset-with-map-when-loaded-with-from-dict/7411/3 "2022-11-16T13:34:36Z")

For what it’s worth, I got around this issue by using `datasets.Dataset.from_file` in conjunction with `datasets.Dataset.from_dict`. The pattern looks something like: ```python dataset_path = '/some/path' if not os.path.exists(dataset_path): d = datasets.Dataset.from_dict(...) d.save_to_disk(dataset_path) d = datasets.Dataset.from_file( filename=os.path.join(dataset_path, 'dataset.arrow'), info=datasets.DatasetInfo( builder_name="my_dataset", config_name="my_dataset_config" ) ) ``` Note that the first time this code runs, when we create the dataset using `from_dict`, we have to save it and load it again so that `dataset.cache_files` will be non-empty. The reason this works is that `dataset.cache_files` will be automatically set if the dataset is loaded from disk. ---

**Author:** ![eli100](https://avatars.discourse-cdn.com/v4/letter/e/fbc32d/32.png) [@eli100](https://discuss.huggingface.co/u/eli100)\ **Post date:** [March 22, 2023, 6:27pm UTC](https://discuss.huggingface.co/t/caching-a-dataset-with-map-when-loaded-with-from-dict/7411/4 "2023-03-22T18:27:49Z")

Also stumbled on this bug. Any plans to fix it?