# Is split\_dataset\_by\_node (streaming dataset) compatible with multi processing?

**URL:** <https://discuss.huggingface.co/t/is-split-dataset-by-node-streaming-dataset-compatible-with-multi-processing/108725>\
**Category:** 🤗Datasets\
**Created:** [September 26, 2024, 5:38am UTC](https://discuss.huggingface.co/t/is-split-dataset-by-node-streaming-dataset-compatible-with-multi-processing/108725 "2024-09-26T05:38:46Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![proj-persona](https://avatars.discourse-cdn.com/v4/letter/p/f6c823/32.png) [@proj-persona](https://discuss.huggingface.co/u/proj-persona)\
**Post date:** [September 26, 2024, 5:38am UTC](https://discuss.huggingface.co/t/is-split-dataset-by-node-streaming-dataset-compatible-with-multi-processing/108725/1 "2024-09-26T05:38:46Z")

</div>

Is split\_dataset\_by\_node (streaming dataset) compatible with multi processing?

Streaming dataset (IterableDataset) doesn’t support multi processing because we need to make sure each rank loads different data. However, if we use split\_dataset\_by\_node, then it is possible to process data with multiple workers, right?

It is important when we process text data with on-the-fly tokenization. Naive streaming dataloader only has 1 worker to tokenize text, which is inefficient. Is there any good way to improve the efficiency? At least, each rank should have its own worker to tokenize?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [October 23, 2024, 2:32pm UTC](https://discuss.huggingface.co/t/is-split-dataset-by-node-streaming-dataset-compatible-with-multi-processing/108725/2 "2024-10-23T14:32:59Z")

</div>

> Is split\_dataset\_by\_node (streaming dataset) compatible with multi processing?

yes !

> Streaming dataset (IterableDataset) doesn’t support multi processing because we need to make sure each rank loads different data. However, if we use split\_dataset\_by\_node, then it is possible to process data with multiple workers, right?

correct, actually that’s what split\_dataset\_by\_node was made for in the first place

> It is important when we process text data with on-the-fly tokenization. Naive streaming dataloader only has 1 worker to tokenize text, which is inefficient. Is there any good way to improve the efficiency? At least, each rank should have its own worker to tokenize?

you can use `DataLoader(..., num_workers=num_workers)` to use one or more processes per rank to tokenize the data.