# Convert dataset to pytorch dataloader

**URL:** <https://discuss.huggingface.co/t/convert-dataset-to-pytorch-dataloader/35753>\
**Category:** 🤗Datasets\
**Created:** [April 6, 2023, 10:45am UTC](https://discuss.huggingface.co/t/convert-dataset-to-pytorch-dataloader/35753 "2023-04-06T10:45:31Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![pfjf2](https://avatars.discourse-cdn.com/v4/letter/p/c2a13f/32.png) [@pfjf2](https://discuss.huggingface.co/u/pfjf2)\
**Post date:** [April 6, 2023, 10:45am UTC](https://discuss.huggingface.co/t/convert-dataset-to-pytorch-dataloader/35753/1 "2023-04-06T10:45:31Z")

</div>

I’m trying to convert a Huggingface dataset into a pytorch dataloader.  
I’m trying to do it in streaming mode to avoid downloading a huge amount of data. I have the following so far:

```auto
dataset = load_dataset("speech_commands", "v0.02", streaming=True)
all_columns = dataset["train"].column_names
columns_to_remove = set(all_columns) - set(['audio', 'label'])
trainset = dataset["train"].remove_columns(columns_to_remove)
trainset = trainset.map(lambda e: {'audio': e['audio']['array'], 'label': e['label']})
trainset = trainset.with_format(type='torch')
dataloader = DataLoader(trainset, batch_size=4)

for batch in dataloader:
   audioTensor = batch['audio']
   targetTensor = batch['label']

```

Is this the best way to do it?  
This seems clunky.

---

<div class="post-metadata">

**Author:** ![pfjf2](https://avatars.discourse-cdn.com/v4/letter/p/c2a13f/32.png) [@pfjf2](https://discuss.huggingface.co/u/pfjf2)\
**Post date:** [April 6, 2023, 1:13pm UTC](https://discuss.huggingface.co/t/convert-dataset-to-pytorch-dataloader/35753/2 "2023-04-06T13:13:26Z")

</div>

Also, for whatever reason, it’s not caching the `.map()` transform, even though the hash for the lambda is the same on every session. What’s going on there?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [April 7, 2023, 6:10pm UTC](https://discuss.huggingface.co/t/convert-dataset-to-pytorch-dataloader/35753/3 "2023-04-07T18:10:08Z")

</div>

Hi ! A bit simpler:

```python
dataset = load_dataset("speech_commands", "v0.02", streaming=True)
trainset = dataset["train"].select_columns(['audio', 'label'])
trainset = trainset.map(lambda e: {'audio': e['audio']['array'], 'label': e['label']})
dataloader = DataLoader(trainset, batch_size=4)

for batch in dataloader:
   audioTensor = batch['audio']
   targetTensor = batch['label']

```

PS: `IterableDataset.with_format` is currently a no-op, but soon will implement the conversion to torch Tensors directly in an optimized way.

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [April 7, 2023, 6:10pm UTC](https://discuss.huggingface.co/t/convert-dataset-to-pytorch-dataloader/35753/4 "2023-04-07T18:10:53Z")

</div>

> [@pfjf2](#):
>
> Also, for whatever reason, it’s not caching the `.map()` transform, even though the hash for the lambda is the same on every session. What’s going on there?

Caching only happens with datastes on disk. On the other hand, nothing is written on disk or cached for datasets in streaming mode