# Odd dataset.map() behavior with PyTorch dataloader

**URL:** <https://discuss.huggingface.co/t/odd-dataset-map-behavior-with-pytorch-dataloader/78692>\
**Category:** 🤗Datasets\
**Created:** [March 25, 2024, 10:06pm UTC](https://discuss.huggingface.co/t/odd-dataset-map-behavior-with-pytorch-dataloader/78692 "2024-03-25T22:06:54Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![irow](https://avatars.discourse-cdn.com/v4/letter/i/a4c791/32.png) [@irow](https://discuss.huggingface.co/u/irow)\
**Post date:** [March 25, 2024, 10:06pm UTC](https://discuss.huggingface.co/t/odd-dataset-map-behavior-with-pytorch-dataloader/78692/1 "2024-03-25T22:06:54Z")

</div>

Hi all, I’m using a streaming dataset, and then passing that into a normal PyTorch DataLoader. I have a simple map function that pads various sequences in the dataset.

```auto
def pad_map(x):
    x['words1'] = pad(x['words1'])
    x['words2'] = pad(x['words2'])
    x['tags1'] = pad(x['tags1'])
    x['tags2'] = pad(x['tags2'])
    return x

```

when I pass the dataset into the PyTorch DataLoader (for example with a batch\_size of 4), I get this unexpected result.

```auto
batch = {'words1': [tensor([44, 3399, 419, 3256]),
  tensor([312, 222, 667, 290]),
  tensor([120, 883, 99, 101]),
...
  tensor([0, 0, 0, 0]),
  tensor([0, 0, 0, 0]),
  tensor([0, 0, 0, 0]),
  tensor([0, 0, 0, 0])],
 'label': tensor([1, 0, 0, 0])}

```

The only way I see to fix this is to do `torch.stack(batch['words1']).T` to get the proper format.  
Is there any way for the map function to produce an output that like the following ?  
`{'words1': tensor of shape (batch_size, seq_len), ..., 'labels': tensor of shape (batch_size, )}`  
Thanks for your help!

---

<div class="post-metadata">

**Author:** ![irow](https://avatars.discourse-cdn.com/v4/letter/i/a4c791/32.png) [@irow](https://discuss.huggingface.co/u/irow)\
**Post date:** [March 25, 2024, 11:09pm UTC](https://discuss.huggingface.co/t/odd-dataset-map-behavior-with-pytorch-dataloader/78692/2 "2024-03-25T23:09:54Z")

</div>

I figured it out, creating a tensor in the map function stopped it from batching it strangely. So like

```auto
def pad_map(x):
    x['words1'] = torch.tensor(pad(x['words1']))
    x['words2'] = torch.tensor(pad(x['words2']))
    x['tags1'] = torch.tensor(pad(x['tags1']))
    x['tags2'] = torch.tensor(pad(x['tags2']))
    return x

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [March 26, 2024, 11:09am UTC](https://discuss.huggingface.co/t/odd-dataset-map-behavior-with-pytorch-dataloader/78692/3 "2024-03-26T11:09:59Z")

</div>

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.