# Roadmap/timeline for dataset streaming

**URL:** <https://discuss.huggingface.co/t/roadmap-timeline-for-dataset-streaming/6789>\
**Category:** 🤗Datasets\
**Created:** [June 14, 2021, 10:41am UTC](https://discuss.huggingface.co/t/roadmap-timeline-for-dataset-streaming/6789 "2021-06-14T10:41:03Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](https://discuss.huggingface.co/u/vblagoje)\
**Post date:** [June 14, 2021, 10:41am UTC](https://discuss.huggingface.co/t/roadmap-timeline-for-dataset-streaming/6789/1 "2021-06-14T10:41:03Z")

</div>

Hey @lhoestq ,

Is there a roadmap/timeline for dataset streaming?

Cheers

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\
**Post date:** [June 14, 2021, 12:12pm UTC](https://discuss.huggingface.co/t/roadmap-timeline-for-dataset-streaming/6789/2 "2021-06-14T12:12:21Z")

</div>

hey @vblagoje even better, there’s an open PR: [Dataset Streaming by lhoestq · Pull Request #2375 · huggingface/datasets · GitHub](https://github.com/huggingface/datasets/pull/2375)

---

<div class="post-metadata">

**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](https://discuss.huggingface.co/u/vblagoje)\
**Post date:** [June 14, 2021, 1:18pm UTC](https://discuss.huggingface.co/t/roadmap-timeline-for-dataset-streaming/6789/3 "2021-06-14T13:18:19Z")

</div>

Excellent, thanks @lewtun so we are talking 1.9 as the most likely release or 2.0?

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\
**Post date:** [June 14, 2021, 5:15pm UTC](https://discuss.huggingface.co/t/roadmap-timeline-for-dataset-streaming/6789/4 "2021-06-14T17:15:51Z")

</div>

i’ll let quentin answer that since he’s the master of all `datasets` releases 😉

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [June 14, 2021, 5:43pm UTC](https://discuss.huggingface.co/t/roadmap-timeline-for-dataset-streaming/6789/5 "2021-06-14T17:43:24Z")

</div>

Hi ! Most likely in 1.9 (next week maybe ?)

It adds a new `datasets.IterableDataset` object that you can load by passing `streaming=True` in `load_dataset`. You can iterate over it using a `for` loop for example.

You can use it to load your dataset, define your data processing using `map` and `shuffle`, and to train models.

After 1.9 we’ll also keep developing more features around streaming after the release. For example:

- a nice integration with pytorch/tensorflow/jax
- a conversion method to get back a map-style `datasets.Dataset` object.
- additional methods like `filter`

And also improve the caching/buffering mechanism as well as the streaming from compressed data.

---

<div class="post-metadata">

**Author:** ![Leena](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/leena/32/2110_2.png) [@Leena](https://discuss.huggingface.co/u/Leena)\
**Post date:** [June 30, 2021, 10:09pm UTC](https://discuss.huggingface.co/t/roadmap-timeline-for-dataset-streaming/6789/6 "2021-06-30T22:09:38Z")

</div>

Hi,

I have a general question regarding Streaming mode – Is IterableDataset not to be used with Pytorch DataLoader? I can use Dataset with the DataLoader without any issues (as is also mentioned in the examples), but I cannot do so with the former. I am quite new to the HF Dataset library so my apologies if this is already mentioned somewhere (I am still looking).

I get the following error, which makes sense because this is streaming mode, but I am unclear about how to design so that I can do batching then:  
`
File “/data/leshekha/lib/HFDatasets/lib/python3.6/site-packages/torch/utils/data/sampler.py”, line 67, in iter
return iter(range(len(self.data_source)))
TypeError: object of type ‘IterableDataset’ has no len()
`  
Any help is appreciated. Thank you.

---

<div class="post-metadata">

**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](https://discuss.huggingface.co/u/vblagoje)\
**Post date:** [July 2, 2021, 10:50am UTC](https://discuss.huggingface.co/t/roadmap-timeline-for-dataset-streaming/6789/7 "2021-07-02T10:50:44Z")

</div>

Hi Leena,

If you are using HF Trainer look into overriding `get_train_dataloader` where you can provide your own instance of torch Sampler initialized with a dataset size.

HTH,  
Vladimir

---

<div class="post-metadata">

**Author:** ![Leena](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/leena/32/2110_2.png) [@Leena](https://discuss.huggingface.co/u/Leena)\
**Post date:** [July 2, 2021, 5:58pm UTC](https://discuss.huggingface.co/t/roadmap-timeline-for-dataset-streaming/6789/8 "2021-07-02T17:58:15Z")

</div>

Hi Vladimir,

Thank you for your response. I am not using HF trainer, I have my own trainer, which essentially means I create my own Dataset, DataLoader, training loop etc (I have not used HF trainer till now may be I will explore that). If my understanding is right then, _get\_train\_dataloader_ returns a Torch DataLoader so working with HF Trainer and my own Trainer should be very similar.

Looking at the Pytorch code it seems like when IterableDataset is used then \_InfiniteConstantSampler is the what the sampler is set to. I created a custom IterableDataset in Torch (not HF) and am able to create batches using this. When I create a HF IterableDataset (using streaming=True) then I am not able to iterate on this as it throws error and you can see that the sampler is SequentialSampler.

Here is a reproducible example:

1. Using Torch IterableDataset and Torch DataLoader:

```python
class CustomIterableDataset(IterableDataset):

  def __init__ (self, data):
        self.data = data

  def __iter__ (self):
        return iter(self.data)

data = list(range(12))
dataset = CustomIterableDataset(data)
dataloader = DataLoader(dataset, batch_size=4)
print("dataloader: ", dataloader.sampler)
for batch in dataloader:
    print(batch)

```

Output is:  
dataloader: \<torch.utils.data.dataloader.\_InfiniteConstantSampler object at 0x7f1cc29e2c50\>  
tensor([0, 1, 2, 3])  
tensor([4, 5, 6, 7])  
tensor([8, 9, 10, 11])

1. Using HF IterableDataset and Torch DataLoader:

```python
from datasets import load_dataset

dataset = load_dataset('oscar', "unshuffled_deduplicated_en", split='train', streaming=True)
dataloader = torch.utils.data.DataLoader(dataset, batch_size=32)
print(dataloader.sampler)
<torch.utils.data.sampler.SequentialSampler object at 0x7f245a510208>
for batch in dataloader:
    print(batch)

```

Error is:  
Traceback (most recent call last):  
File “”, line 1, in   
File “/data/leshekha/lib/HFDatasets/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 435, in **next**  
data = self.\_next\_data()  
File “/data/leshekha/lib/HFDatasets/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 474, in \_next\_data  
index = self.\_next\_index() # may raise StopIteration  
File “/data/leshekha/lib/HFDatasets/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 427, in \_next\_index  
return next(self.\_sampler\_iter) # may raise StopIteration  
File “/data/leshekha/lib/HFDatasets/lib/python3.6/site-packages/torch/utils/data/sampler.py”, line 227, in **iter**  
for idx in self.sampler:  
File “/data/leshekha/lib/HFDatasets/lib/python3.6/site-packages/torch/utils/data/sampler.py”, line 67, in **iter**  
return iter(range(len(self.data\_source)))  
TypeError: object of type ‘IterableDataset’ has no len()

Thanks,  
Leena

---

<div class="post-metadata">

**Author:** ![thomwolf](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thomwolf/32/2198_2.png) [@thomwolf](https://discuss.huggingface.co/u/thomwolf)\
**Post date:** [July 5, 2021, 8:12pm UTC](https://discuss.huggingface.co/t/roadmap-timeline-for-dataset-streaming/6789/9 "2021-07-05T20:12:58Z")

</div>

Solution at [Error iteration over IterableDataset using Torch DataLoader · Issue #2583 · huggingface/datasets · GitHub](https://github.com/huggingface/datasets/issues/2583)

---

<div class="post-metadata">

**Author:** ![Leena](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/leena/32/2110_2.png) [@Leena](https://discuss.huggingface.co/u/Leena)\
**Post date:** [July 5, 2021, 11:59pm UTC](https://discuss.huggingface.co/t/roadmap-timeline-for-dataset-streaming/6789/10 "2021-07-05T23:59:38Z")

</div>

Thank you for the solution and the link. I was just about to paste it here. This has been solved.