# Create a dataset from generator **URL:** \ **Category:** 🤗Datasets\ **Created:** [January 10, 2021, 12:08pm UTC](https://discuss.huggingface.co/t/create-a-dataset-from-generator/3119 "2021-01-10T12:08:20Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![shpotes](https://avatars.discourse-cdn.com/v4/letter/s/87869e/32.png) [@shpotes](https://discuss.huggingface.co/u/shpotes)\ **Post date:** [January 10, 2021, 12:08pm UTC](https://discuss.huggingface.co/t/create-a-dataset-from-generator/3119/1 "2021-01-10T12:08:20Z")
There is any way to create a dataset from a generator (without it being loaded into memory). Something similar to `tf.data.Dataset.from_generator` ---
**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\ **Post date:** [January 10, 2021, 2:26pm UTC](https://discuss.huggingface.co/t/create-a-dataset-from-generator/3119/2 "2021-01-10T14:26:44Z")
The datasets are not completely read into memory so you should not have to worry about memory usage. It’s mostly fast on-disk access thanks to memory mapping. ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [January 19, 2021, 3:00pm UTC](https://discuss.huggingface.co/t/create-a-dataset-from-generator/3119/3 "2021-01-19T15:00:52Z")
We don’t have a `.from_generator` method yet but this is something we may add ! If you want to generate a dataset from text/json/csv files, then you can do it directly using `load_dataset`. More information in the [documentation](https://huggingface.co/docs/datasets/loading_datasets.html#from-local-files) Currently to make a dataset from a custom generator you can make a dataset script that can `yield` the examples. When calling `load_dataset("path/to/my/dataset/script")` it will iterate through the generator to write all the examples in an arrow file without loading them into memory. Then a `Dataset` object will be created containing your data that are memory-mapped from your disk. Memory-mapping allows to load the dataset without loading it into memory. You can find how to write a dataset script [here](https://huggingface.co/docs/datasets/add_dataset.html) ---
**Author:** ![StephennFernandes](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/stephennfernandes/32/17230_2.png) [@StephennFernandes](https://discuss.huggingface.co/u/StephennFernandes)\ **Post date:** [May 23, 2022, 12:36am UTC](https://discuss.huggingface.co/t/create-a-dataset-from-generator/3119/4 "2022-05-23T00:36:15Z")
@shpotes we’re you able to solve this problem, if so would you please share the solution code ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [June 27, 2022, 2:09pm UTC](https://discuss.huggingface.co/t/create-a-dataset-from-generator/3119/5 "2022-06-27T14:09:58Z")
> You can find how to write a dataset script [here](https://huggingface.co/docs/datasets/add_dataset.html) new link is [https://huggingface.co/docs/datasets/dataset\_script](https://huggingface.co/docs/datasets/dataset_script) EDIT: you should use `.from_generator ()` now, which doesn’t require to implement a custom class ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [November 23, 2022, 10:24am UTC](https://discuss.huggingface.co/t/create-a-dataset-from-generator/3119/6 "2022-11-23T10:24:41Z")
For those interested, we added `.from_generator` recently 🙂 docs: - for Dataset: [Main classes](https://huggingface.co/docs/datasets/v2.7.0/en/package_reference/main_classes#datasets.Dataset.from_generator) - For IterableDataset: [Main classes](https://huggingface.co/docs/datasets/v2.7.0/en/package_reference/main_classes#datasets.IterableDataset.from_generator) ---
**Author:** ![deathcrush](https://avatars.discourse-cdn.com/v4/letter/d/58f4c7/32.png) [@deathcrush](https://discuss.huggingface.co/u/deathcrush)\ **Post date:** [January 15, 2024, 11:49am UTC](https://discuss.huggingface.co/t/create-a-dataset-from-generator/3119/7 "2024-01-15T11:49:00Z")
@lhoestq I was wondering where can I find some detailed information about how caching works (in particular about what informs the decision about loading from cache or doing the processing again)? I am generating datasets for inference on-the-fly during `evaluate` and `predict` and I passed the `cache_dir` option so that the generated datasets are generated only under the main process (so I’m using `Trainer'`s `accelerator.main_process_first()` context manager during `evaluate` calls). However, when evaluating at different points I need to generate the dataset again. Am I correct in thinking that I should wipe off the `cache/generator` in between eval calls to ensure the evaluation does not used previously generated datasets? ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [January 30, 2024, 2:43pm UTC](https://discuss.huggingface.co/t/create-a-dataset-from-generator/3119/8 "2024-01-30T14:43:51Z")
You can find the documentation here 🙂 > **[Cache management](https://huggingface.co/docs/datasets/v2.16.1/en/cache#cache-management)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science.