# How to handle big data?

**URL:** <https://discuss.huggingface.co/t/how-to-handle-big-data/40143>\
**Category:** 🤗Datasets\
**Created:** [May 17, 2023, 8:35am UTC](https://discuss.huggingface.co/t/how-to-handle-big-data/40143 "2023-05-17T08:35:36Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Lucia001](https://avatars.discourse-cdn.com/v4/letter/l/a3d4f5/32.png) [@Lucia001](https://discuss.huggingface.co/u/Lucia001)\
**Post date:** [May 17, 2023, 8:35am UTC](https://discuss.huggingface.co/t/how-to-handle-big-data/40143/1 "2023-05-17T08:35:36Z")

</div>

I am trying to use datasets.load\_dataset to load multiple big files from disk. I noticed it could merge the content of the files together to be a single dataset. And then I called the datasets.map to tokenize the dataset.  
My question is: If the total content of the files are much bigger than RAM, what will happen? Will the program break down or consume very long time? Can the map be executed in training lazily?

Thank you very much for your reply!

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [May 17, 2023, 6:15pm UTC](https://discuss.huggingface.co/t/how-to-handle-big-data/40143/2 "2023-05-17T18:15:46Z")

</div>

> If the total content of the files are much bigger than RAM, what will happen? Will the program break down or consume very long time?

It shouldn’t crash. `map` processes and writes data to disk in batches to support transforming datasets bigger than RAM.

> Can the map be executed in training lazily?

You can use [`set_transform`](https://huggingface.co/docs/datasets/process#format-transform) instead of `map` for lazy execution.

---

<div class="post-metadata">

**Author:** ![Lucia001](https://avatars.discourse-cdn.com/v4/letter/l/a3d4f5/32.png) [@Lucia001](https://discuss.huggingface.co/u/Lucia001)\
**Post date:** [May 18, 2023, 1:41am UTC](https://discuss.huggingface.co/t/how-to-handle-big-data/40143/3 "2023-05-18T01:41:12Z")

</div>

Thank you for your reply!  
I tried the set\_transform, but the generated dataset cannot work with Trainer.train. It seems the dataloader cannot load data from the dataset bacause of index issue. The exception is like below:  
File “D:\Environment\anaconda\envs\virsyn\lib\site-packages\torch\utils\data\dataloader.py”, line 521, in **next**  
data = self.\_next\_data()  
File “D:\Environment\anaconda\envs\virsyn\lib\site-packages\torch\utils\data\dataloader.py”, line 1203, in \_next\_data  
return self.\_process\_data(data)  
File “D:\Environment\anaconda\envs\virsyn\lib\site-packages\torch\utils\data\dataloader.py”, line 1229, in \_process\_data  
data.reraise()  
File “D:\Environment\anaconda\envs\virsyn\lib\site-packages\torch\_utils.py”, line 434, in reraise  
raise exception  
IndexError: Caught IndexError in DataLoader worker process 0.  
Original Traceback (most recent call last):  
File “D:\Environment\anaconda\envs\virsyn\lib\site-packages\torch\utils\data\_utils\worker.py”, line 287, in \_worker\_loop  
data = fetcher.fetch(index)  
File “D:\Environment\anaconda\envs\virsyn\lib\site-packages\torch\utils\data\_utils\fetch.py”, line 49, in fetch  
data = [self.dataset[idx] for idx in possibly\_batched\_index]  
File “D:\Environment\anaconda\envs\virsyn\lib\site-packages\torch\utils\data\_utils\fetch.py”, line 49, in   
data = [self.dataset[idx] for idx in possibly\_batched\_index]  
File “D:\Environment\anaconda\envs\virsyn\lib\site-packages\datasets\arrow\_dataset.py”, line 2155, in **getitem**  
key,  
File “D:\Environment\anaconda\envs\virsyn\lib\site-packages\datasets\arrow\_dataset.py”, line 2138, in \_getitem  
pa\_subtable = query\_table(self.\_data, key, indices=self.\_indices if self.\_indices is not None else None)  
File “D:\Environment\anaconda\envs\virsyn\lib\site-packages\datasets\formatting\formatting.py”, line 486, in query\_table  
\_check\_valid\_index\_key(key, size)  
File “D:\Environment\anaconda\envs\virsyn\lib\site-packages\datasets\formatting\formatting.py”, line 429, in \_check\_valid\_index\_key  
raise IndexError(f"Invalid key: {key} is out of bounds for size {size}")  
IndexError: Invalid key: 293 is out of bounds for size 0

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [May 18, 2023, 2:30pm UTC](https://discuss.huggingface.co/t/how-to-handle-big-data/40143/4 "2023-05-18T14:30:05Z")

</div>

Debugging an error like this without the code (reproducible example) is hard. Can you share the code that instantiates the dataset and the trainer? Feel free to share a dummy dataset (with the same structure) to keep the data private.

---

<div class="post-metadata">

**Author:** ![Lucia001](https://avatars.discourse-cdn.com/v4/letter/l/a3d4f5/32.png) [@Lucia001](https://discuss.huggingface.co/u/Lucia001)\
**Post date:** [May 19, 2023, 3:07am UTC](https://discuss.huggingface.co/t/how-to-handle-big-data/40143/5 "2023-05-19T03:07:39Z")

</div>

To provide lazily tokenize function, I extract the tokenize function to be a class:

```auto
> class LazyTokenize:
> def __init__ (self, tokenizer, text_column_name, padding, max_seq_length):
> self.tokenizer = tokenizer
> self.text_column_name = text_column_name
> self.padding = padding
> self.max_seq_length = max_seq_length
> 
> def tokenize_function(self, examples):
> # Remove empty lines
> examples[self.text_column_name] = [
> line for line in examples[self.text_column_name] if len(line) > 0 and not line.isspace()
> ]
> return self.tokenizer(
> examples[self.text_column_name],
> padding=self.padding,
> truncation=True,
> max_length=self.max_seq_length,
> return_special_tokens_mask=True,
> )

```

Then load dataset like below:

```auto
> data_files={'train': ['../data/pretrain/train\\file1.csv', '../data/pretrain/train\\file2.csv', '../data/pretrain/train\\file3.csv'], 'validation': ['../data/pretrain/eval\\file1.csv', '../data/pretrain/eval\\file2.csv', '../data/pretrain/eval\\file3.csv']}
> raw_datasets = load_dataset(extension, data_files=data_files, cache_dir=model_args.cache_dir)
> lazy_tokenizer = LazyTokenize(tokenizer, text_column_name, padding, max_seq_length)
> tokenized_datasets = raw_datasets.with_transform(lazy_tokenizer.tokenize_function)

```

Then use the trainer.py of transformer, the calling is like below:

```auto
> train_dataset = tokenized_datasets["train"]
> eval_dataset = tokenized_datasets["validation"]
> trainer = Trainer(
> model=model,
> args=training_args,
> train_dataset=train_dataset if training_args.do_train else None,
> eval_dataset=eval_dataset if training_args.do_eval else None,
> tokenizer=tokenizer,
> data_collator=data_collator,
> callbacks=[early_stopping_callback]
> )
> 
> trainer.train(resume_from_checkpoint=checkpoint)

```

Before using the with\_transform, the datasets.map function worked well with the tokenize function.

Thank you very much for your help!

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [May 22, 2023, 11:38am UTC](https://discuss.huggingface.co/t/how-to-handle-big-data/40143/6 "2023-05-22T11:38:32Z")

</div>

The number of output examples should match the number of input examples in the transform function, so instead of removing empty lines with:

```python
examples[self.text_column_name] = [
    line for line in examples[self.text_column_name] if len(line) > 0 and not line.isspace()
]

```

run a filter before `with_transform` like so:

```python
raw_datasets.filter(lambda ex: len(ex[self.text_column_name]) > 0 and not ex[self.text_column_name].isspace())

```

Or, even better, use `map` instead of `with_transform` if there is enough disk space for the cache file (tokenization is the same across the epochs, so it’s more efficient to do it once than lazily each epoch):

```python
def tokenize_function(self, examples):
    # Remove empty lines
    examples[self.text_column_name] = [
        line for line in examples[self.text_column_name] if len(line) > 0 and not line.isspace()
    ]
    return self.tokenizer(
       examples[self.text_column_name],
       padding=self.padding,
       truncation=True,
       max_length=self.max_seq_length,
       return_special_tokens_mask=True,
       return_tensors="np", # makes converting to Arrow faster (needed for caching)
    )

# map with `batched=True` can return less/more examples than there are in the input
tokenized_datasets = raw_datasets.map(tokenizer.tokenize_function, batched=True, remove_columns=raw_datasets["train"].column_names)

```

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [May 25, 2023, 3:08pm UTC](https://discuss.huggingface.co/t/how-to-handle-big-data/40143/7 "2023-05-25T15:08:21Z")

</div>

> [@Lucia001](#):
>
> IndexError: Invalid key: 293 is out of bounds for size 0

it can be related to [IndexError: Invalid key: 16 is out of bounds for size 0 - #3 by Isma](https://discuss.huggingface.co/t/indexerror-invalid-key-16-is-out-of-bounds-for-size-0/14298/3)

---

<div class="post-metadata">

**Author:** ![Lucia001](https://avatars.discourse-cdn.com/v4/letter/l/a3d4f5/32.png) [@Lucia001](https://discuss.huggingface.co/u/Lucia001)\
**Post date:** [June 15, 2023, 3:07am UTC](https://discuss.huggingface.co/t/how-to-handle-big-data/40143/8 "2023-06-15T03:07:38Z")

</div>

Thank you all for your help!  
I decided not to use transform to implement lazy tokenization.  
But now I am facing another problem: the data is too big, and the tokenization, DataCollator steps before training will cost too much time!  
Does huggingface transformer support to distribute such data processes into many computers to enhance executing speed?