# Uploading large datasets iteratively

**URL:** <https://discuss.huggingface.co/t/uploading-large-datasets-iteratively/60253>\
**Category:** 🤗Datasets\
**Created:** [October 28, 2023, 2:42pm UTC](https://discuss.huggingface.co/t/uploading-large-datasets-iteratively/60253 "2023-10-28T14:42:04Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![316usman](https://avatars.discourse-cdn.com/v4/letter/3/9d8465/32.png) [@316usman](https://discuss.huggingface.co/u/316usman)\
**Post date:** [October 28, 2023, 2:42pm UTC](https://discuss.huggingface.co/t/uploading-large-datasets-iteratively/60253/1 "2023-10-28T14:42:04Z")

</div>

Hi, I am trying to upload about 300GB of text. First I would like to do some processing on the text and then upload to the dataset for e.g I open a text file and make it clean and then upload it and then do this with the next file. How do i do this iteratively?

To use .push\_to\_hub you need to have added a new item to the dataset that would require alot of ram.

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [October 29, 2023, 3:25pm UTC](https://discuss.huggingface.co/t/uploading-large-datasets-iteratively/60253/2 "2023-10-29T15:25:02Z")

</div>

## How to load, process and upload 300GB of text

You can actually do it on all your text files **at once**.  
Indeed `datasets` can work on datasets bigger than memory, since it memory maps the datasets files from your disk.

1. load the dataset using `load_dataset`, e.g.

```python
ds = load_dataset("text", data_files=list_of_text_files, split="train")

```

1. use `map` to clean your dataset (use `num_proc` to make it faster), e.g.

```python
ds = ds.map(clean_document, num_proc=num_proc)

```

1. upload with `push_to_hub`, e.g.

```python
ds.push_to_hub("316usman/my_text_dataset")

```

* * *

## An alternative to upload iteratively anyway

There’s no “append” mode yet in `push_to_hub`, but you can execute the above steps once for each file as a separate split:

```python
ds.push_to_hub("316usman/my_text_dataset", split=f"part_{i:05d}")

```

and then regroup all the splits together by modifying the YAML at the top of the README.md in the dataset repository on HF:

from

```yaml
configs:
- config_name: default
  data_files:
  - split: part_0000
    path: data/part_0000-*
  - split: part_0001
    path: data/part_0001-*
  ...

```

to

```yaml
configs:
- config_name: default
  data_files:
  - split: train
    path: data/part_*

```

---

<div class="post-metadata">

**Author:** ![316usman](https://avatars.discourse-cdn.com/v4/letter/3/9d8465/32.png) [@316usman](https://discuss.huggingface.co/u/316usman)\
**Post date:** [October 29, 2023, 9:08pm UTC](https://discuss.huggingface.co/t/uploading-large-datasets-iteratively/60253/3 "2023-10-29T21:08:23Z")

</div>

@lhoestq Thank you so much for replying

What I get from your first approach is: Datasets library does not load the files into the RAM rather they are stored on the Disk (HDD/SSD). Please correct me if I am wrong.

Also I came across the **from\_generator()** class Now my approach is that I download a file from a block storage and process it using the data generator function, in this case where would the data be store until i have used **push\_to\_hub()** on the disk perhaps?.

The alternative approach looks promising I will try that out too. In the alternative approach I would be uploading text files? (With each row in my file representing an entry in the dataset).

Again thanks so much for replying. Hope to connect with you.

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [October 30, 2023, 9:43am UTC](https://discuss.huggingface.co/t/uploading-large-datasets-iteratively/60253/4 "2023-10-30T09:43:24Z")

</div>

Both `load_dataset` and `from_generator` write the dataset on disk and then memory maps the data on disk to not fill out the RAM (you can see it as using virtual memory from your disk).

> The alternative approach looks promising I will try that out too. In the alternative approach I would be uploading text files? (With each row in my file representing an entry in the dataset).

It uploads Parquet files. Parquet a compressed columnar format that is particularly suited to store all sorts of datasets and is easy to load in python 🙂

---

<div class="post-metadata">

**Author:** ![316usman](https://avatars.discourse-cdn.com/v4/letter/3/9d8465/32.png) [@316usman](https://discuss.huggingface.co/u/316usman)\
**Post date:** [October 30, 2023, 7:20pm UTC](https://discuss.huggingface.co/t/uploading-large-datasets-iteratively/60253/5 "2023-10-30T19:20:38Z")

</div>

Thank you so much I’ll try and get back to you