# Create large dataset with PIL image loaded in 'image' column

**URL:** <https://discuss.huggingface.co/t/create-large-dataset-with-pil-image-loaded-in-image-column/76292>\
**Category:** Beginners\
**Created:** [March 6, 2024, 11:31pm UTC](https://discuss.huggingface.co/t/create-large-dataset-with-pil-image-loaded-in-image-column/76292 "2024-03-06T23:31:17Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![csegalin](https://avatars.discourse-cdn.com/v4/letter/c/ecc23a/32.png) [@csegalin](https://discuss.huggingface.co/u/csegalin)\
**Post date:** [March 6, 2024, 11:31pm UTC](https://discuss.huggingface.co/t/create-large-dataset-with-pil-image-loaded-in-image-column/76292/1 "2024-03-06T23:31:17Z")

</div>

I have a dataset of 4M images with metadata that I already formatted as HF dataset from the pandas DF.  
I would like to add a column that has the PIL image as a lot of these dataset have.  
With the size of the dataset I need to process I am not sure what is the best approach to do it as running a lip or map will def go OOM  
Any ideas?

also how to create the json file so that if I save to disk then I can load it using `load_dataset` and not `load_from_disk`?