# A streaming dataset's memory footprint continually grows

**URL:** <https://discuss.huggingface.co/t/a-streaming-datasets-memory-footprint-continually-grows/159404>\
**Category:** 🤗Datasets\
**Created:** [June 15, 2025, 6:42pm UTC](https://discuss.huggingface.co/t/a-streaming-datasets-memory-footprint-continually-grows/159404 "2025-06-15T18:42:00Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![joell1](https://avatars.discourse-cdn.com/v4/letter/j/7cd45c/32.png) [@joell1](https://discuss.huggingface.co/u/joell1)\
**Post date:** [June 15, 2025, 6:42pm UTC](https://discuss.huggingface.co/t/a-streaming-datasets-memory-footprint-continually-grows/159404/1 "2025-06-15T18:42:00Z")

</div>

Hey Datasets community. I am having a minor issue with a streaming a dataset. I have a very large dataset, over 500GB, that I have sharded into 5 files. Naturally I choose to use a streaming dataset since my machine does not have enough RAM to hold the entire dataset. I create my HF dataset with the following line

```auto
dataset = load_dataset(
            'parquet',
            data_files=data_files,
            streaming=True
        )

```

and I create the pytorch dataloader with:

```auto
def build_dataloader(self, dataset, batch_size, num_workers):
        return DataLoader(
            dataset,
            batch_size=batch_size,
            num_workers=num_workers,
            collate_fn=collate_fn
        )

```

I choose to have 4 workers. It works and is plenty fast, but during an epoch the memory footprint continually grows. When I look at htop in Linux, my training program spawned about 54 processes, all of which currently take up ~4.8GB of memory. Earlier today that same number for those processes was ~3GB. My conception of a streaming dataloader is that it should load in a batch and after a training step forfeit that memory so that it can load in the next batch. Therefore dataloading should always maintain the memory footprint of 1 batch\*num\_workers. Is there a memory leak here? I have attached a screenshot of htop to help you get an idea of what is going on.

 ![Screenshot (361)](https://us1.discourse-cdn.com/hellohellohello/original/3X/3/6/3662a42fd3570bd2c5ad17238974cfb875ec9b8c.png)

Many thanks in advance  
-Joel

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [June 15, 2025, 10:17pm UTC](https://discuss.huggingface.co/t/a-streaming-datasets-memory-footprint-continually-grows/159404/2 "2025-06-15T22:17:09Z")

</div>

I think it’s similar to this phenomenon.

> <https://github.com/huggingface/datasets/issues/7269>
>
> \### Describe the bug
> 
> I try to use a dataset with streaming=True, the issue I …have is that the RAM usage becomes higher and higher until it is no longer sustainable.
> 
> I understand that huggingface store data in ram during the streaming, and more worker in dataloader there are, more a lot of shard will be stored in ram, but the issue I have is that the ram usage is not constant. So after each new shard loaded, the ram usage will be higher and higher.
> 
> \### Steps to reproduce the bug
> 
> You can run this code and see you ram usage, after each shard of 255 examples, your ram usage will be extended.
> \`\`\`py
> from datasets import load\_dataset
> from torch.utils.data import DataLoader
> 
> dataset = load\_dataset("WaveGenAI/dataset", streaming=True)
> 
> dataloader = DataLoader(dataset\["train"\], num\_workers=3)
> 
> for i, data in enumerate(dataloader):
> print(i, end="\\r")
> \`\`\`
> 
> \### Expected behavior
> 
> The Ram usage should be always the same (just 3 shards loaded in the ram).
> 
> \### Environment info
> 
> \- \`datasets\` version: 3.0.1
> \- Platform: Linux-6.10.5-arch1-1-x86\_64-with-glibc2.40
> \- Python version: 3.12.4
> \- \`huggingface\_hub\` version: 0.26.0
> \- PyArrow version: 17.0.0
> \- Pandas version: 2.2.3
> \- \`fsspec\` version: 2024.6.1

> [@OOM issue with large dataset streaming](https://discuss.huggingface.co/t/oom-issue-with-large-dataset-streaming/144488):
>
> For training, we load a multiple datasets (each with 1000+ 1GB+ each arrow files saved using dataset) with load\_dataset(…streaming=true) and then interleave them together with interleave\_dataset. Even with streaming dataset we are seeing OOM at the interleaving\_datasets step. Note no data has been read from datasets yet. Sharding the initial datasets per gpu before interleaving alleviated the memory pressure. Why should there be such memory issues with streaming dataset?

---

<div class="post-metadata">

**Author:** ![joell1](https://avatars.discourse-cdn.com/v4/letter/j/7cd45c/32.png) [@joell1](https://discuss.huggingface.co/u/joell1)\
**Post date:** [June 16, 2025, 7:19pm UTC](https://discuss.huggingface.co/t/a-streaming-datasets-memory-footprint-continually-grows/159404/3 "2025-06-16T19:19:04Z")

</div>

Thanks John

After reading both posts, its not clear that there is a resolution to the reported behavior. I see that Quentin recommended `dataset.with_format("torch")`. I tried this, but it slows down the dataloading compared to the dataset with the “with\_format” method, almost by a factor of 2.

I have spent some time observing the different behavior of the two methods. Without the “with\_format” it loads data in very quickly, but gets hung-up at certain points, at which the dataloader is suspended few seconds, sometimes tens of seconds, while when using “with\_format” it never gets hung-up, but as I previously said is only half as fast. Both methods take up considerable memory, although “with\_format” a little less.

Furthermore whether using “with\_format” or not, dataloading with streaming appears to need 5-6 GB per worker, and may be growing and shrinking unpredictably throughout training. Is there anyone that can give me a rationalization for this behavior during streaming? If it is streaming, why is the memory footprint still kinda large. A single batch has maybe 20,000 float32s, and thus should have a running memory footprint on the order of MB, not GB, right?

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [June 17, 2025, 2:46am UTC](https://discuss.huggingface.co/t/a-streaming-datasets-memory-footprint-continually-grows/159404/4 "2025-06-17T02:46:24Z")

</div>

Hmm… @lhoestq

---

<div class="post-metadata">

**Author:** ![Pimpcat-AU](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/pimpcat-au/32/48989_2.png) [@Pimpcat-AU](https://discuss.huggingface.co/u/Pimpcat-AU)\
**Post date:** [June 17, 2025, 6:16am UTC](https://discuss.huggingface.co/t/a-streaming-datasets-memory-footprint-continually-grows/159404/5 "2025-06-17T06:16:46Z")

</div>

You’re using streaming=True with num\_workers \> 0.  
This causes each worker to hold its own persistent iterator state, and Python doesn’t automatically garbage collect iterators across workers unless manually torn down.

Over time, especially with large DataLoader iterations, this creates persistent object growth in each process, even though you’re “streaming.”  
Fix / Mitigation Options:

1. Force shared workers to reinitialize frequently.  
Set:

persistent\_workers=False

Or reinstantiate the DataLoader per epoch:

for epoch in range(num\_epochs):  
loader = build\_dataloader(…)

1. Manually trigger GC cleanup inside collate\_fn:

import gc

def collate\_fn(batch):  
gc.collect()  
return batch

1. Use num\_workers=0 for streaming datasets if memory is highly constrained you sacrifice speed but regain determinism.

2. Use HuggingFace’s iter(dataset) directly instead of wrapping it in DataLoader if you want tightest control.

Why it happens:  
StreamingDataset + multiprocessing = replicated internal state across subprocesses.  
Memory isn’t leaked it’s just retained longer than expected because Python workers don’t reset iterator scope automatically.

Fix provided by Triskel Data Deterministic AI.  
Loop logic only works when memory is sealed.

Let me know if you’d like a memory-safe symbolic loader pattern.

---

<div class="post-metadata">

**Author:** ![joell1](https://avatars.discourse-cdn.com/v4/letter/j/7cd45c/32.png) [@joell1](https://discuss.huggingface.co/u/joell1)\
**Post date:** [June 18, 2025, 5:51am UTC](https://discuss.huggingface.co/t/a-streaming-datasets-memory-footprint-continually-grows/159404/6 "2025-06-18T05:51:30Z")

</div>

Hey Pimpcat-AU, thank you so much for all this information. I tried both `persistent_workers=False` and `gc.collect()`, unfortunately neither one worked. The latter fix really slowed things down, so I implemented it in the collate function randomly 1% of the time. Ultimately my program had a failed worker at exactly the same step in training.

Could you possibly tell me why the memory footprint is so high to begin with? 1 batch is on the order of MB and with multiple workers it probably should be tens of MB. Is the program asking for memory that it isn’t actually using?

---

<div class="post-metadata">

**Author:** ![Pimpcat-AU](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/pimpcat-au/32/48989_2.png) [@Pimpcat-AU](https://discuss.huggingface.co/u/Pimpcat-AU)\
**Post date:** [June 18, 2025, 6:27am UTC](https://discuss.huggingface.co/t/a-streaming-datasets-memory-footprint-continually-grows/159404/7 "2025-06-18T06:27:29Z")

</div>

from datasets import load\_dataset

class SymbolicStreamingLoader:  
def **init** (self, dataset, batch\_size):  
self.dataset = dataset  
self.batch\_size = batch\_size  
self.iterator = iter(dataset)

```
def __iter__ (self):
    return self

def __next__ (self):
    batch = []
    try:
        for _ in range(self.batch_size):
            batch.append(next(self.iterator))
    except StopIteration:
        if not batch:
            raise
    return batch

```

Usage:

dataset = load\_dataset(‘parquet’, data\_files=data\_files, streaming=True)  
loader = SymbolicStreamingLoader(dataset, batch\_size=32)

for epoch in range(num\_epochs):  
for batch in loader:  
train\_step(batch)  
loader = SymbolicStreamingLoader(dataset, batch\_size=32) # re-init each epoch

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [June 18, 2025, 6:37am UTC](https://discuss.huggingface.co/t/a-streaming-datasets-memory-footprint-continually-grows/159404/8 "2025-06-18T06:37:22Z")

</div>

It seems like this is a specification that does not occur with streaming of parquet-formatted datasets, but if that is not the case, could it be a bug?

> <https://github.com/huggingface/datasets/issues/7269>
>
> \### Describe the bug
> 
> I try to use a dataset with streaming=True, the issue I …have is that the RAM usage becomes higher and higher until it is no longer sustainable.
> 
> I understand that huggingface store data in ram during the streaming, and more worker in dataloader there are, more a lot of shard will be stored in ram, but the issue I have is that the ram usage is not constant. So after each new shard loaded, the ram usage will be higher and higher.
> 
> \### Steps to reproduce the bug
> 
> You can run this code and see you ram usage, after each shard of 255 examples, your ram usage will be extended.
> \`\`\`py
> from datasets import load\_dataset
> from torch.utils.data import DataLoader
> 
> dataset = load\_dataset("WaveGenAI/dataset", streaming=True)
> 
> dataloader = DataLoader(dataset\["train"\], num\_workers=3)
> 
> for i, data in enumerate(dataloader):
> print(i, end="\\r")
> \`\`\`
> 
> \### Expected behavior
> 
> The Ram usage should be always the same (just 3 shards loaded in the ram).
> 
> \### Environment info
> 
> \- \`datasets\` version: 3.0.1
> \- Platform: Linux-6.10.5-arch1-1-x86\_64-with-glibc2.40
> \- Python version: 3.12.4
> \- \`huggingface\_hub\` version: 0.26.0
> \- PyArrow version: 17.0.0
> \- Pandas version: 2.2.3
> \- \`fsspec\` version: 2024.6.1

> [@Multiprocessing map taking too much memory footprint](https://discuss.huggingface.co/t/multiprocessing-map-taking-too-much-memory-footprint/27238/16):
>
> A dataset that comes from memory (e.g. using .from\_dict()) doesn’t have a cache file yet, so if you want your map() to write on disk instead of filling up your memory you should pass a cache\_file\_name to map(). Note that at one point we might allocate a cache automatically to such datasets in memory to align with the general behavior.

---

<div class="post-metadata">

**Author:** ![joell1](https://avatars.discourse-cdn.com/v4/letter/j/7cd45c/32.png) [@joell1](https://discuss.huggingface.co/u/joell1)\
**Post date:** [June 19, 2025, 7:00am UTC](https://discuss.huggingface.co/t/a-streaming-datasets-memory-footprint-continually-grows/159404/9 "2025-06-19T07:00:02Z")

</div>

Is the take away here that streaming and multiple workers is unfeasible with very large datasets?