# Padding in datasets

**URL:** <https://discuss.huggingface.co/t/padding-in-datasets/2345>\
**Category:** 🤗Datasets\
**Created:** [November 29, 2020, 8:45am UTC](https://discuss.huggingface.co/t/padding-in-datasets/2345 "2020-11-29T08:45:40Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![maximin](https://avatars.discourse-cdn.com/v4/letter/m/90db22/32.png) [@maximin](https://discuss.huggingface.co/u/maximin)\
**Post date:** [November 29, 2020, 8:45am UTC](https://discuss.huggingface.co/t/padding-in-datasets/2345/1 "2020-11-29T08:45:40Z")

</div>

I usually use padding in batches before I get into the `datasets` library.

I found that `dataset.map` support `batched` and `batch_size`. But it seems that only padding all examples (in `dataset.map`) to fixed length or `max_length` make sense with subsequent `batch_size` in creating `DataLoader`.

Otherwise, if I use map function like `lambda x: tokenizer(x["sentence"], padding=True, truncation=True)` I get errors like `RuntimeError: stack expects each tensor to be equal size, but got [56] at entry 0 and [53] at entry 8` when iterating the dataloader since I could not find a way to iterating the same batches in `datasets.map` in `dataloader`.

Am I right?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [December 28, 2020, 1:20pm UTC](https://discuss.huggingface.co/t/padding-in-datasets/2345/2 "2020-12-28T13:20:24Z")

</div>

That’s because `padding=True` makes the tokenization pad to the longest sequence in the batch. Therefore two batches may have different length

---

<div class="post-metadata">

**Author:** ![maximin](https://avatars.discourse-cdn.com/v4/letter/m/90db22/32.png) [@maximin](https://discuss.huggingface.co/u/maximin)\
**Post date:** [March 21, 2021, 2:38pm UTC](https://discuss.huggingface.co/t/padding-in-datasets/2345/3 "2021-03-21T14:38:21Z")

</div>

Then having all the examples in the dataset padded to the same length could slow down training, right?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [March 22, 2021, 1:31pm UTC](https://discuss.huggingface.co/t/padding-in-datasets/2345/4 "2021-03-22T13:31:33Z")

</div>

Padding all the examples to the same length makes the training slower compared to training with padding to the maximum length per batch.  
You can have a `data_collator` in your pytorch dataloader that does the padding to the maximum length per batch.

---

<div class="post-metadata">

**Author:** ![RylanSchaeffer](https://avatars.discourse-cdn.com/v4/letter/r/6f9a4e/32.png) [@RylanSchaeffer](https://discuss.huggingface.co/u/RylanSchaeffer)\
**Post date:** [October 11, 2021, 8:08pm UTC](https://discuss.huggingface.co/t/padding-in-datasets/2345/5 "2021-10-11T20:08:14Z")

</div>

@lhoestq do you have an example of that?

---

<div class="post-metadata">

**Author:** ![RylanSchaeffer](https://avatars.discourse-cdn.com/v4/letter/r/6f9a4e/32.png) [@RylanSchaeffer](https://discuss.huggingface.co/u/RylanSchaeffer)\
**Post date:** [October 11, 2021, 8:12pm UTC](https://discuss.huggingface.co/t/padding-in-datasets/2345/6 "2021-10-11T20:12:38Z")

</div>

@maximin what was your solution in place of `lambda entry: self.tokenizer(entry[ padding=True,)`?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [October 21, 2021, 8:49am UTC](https://discuss.huggingface.co/t/padding-in-datasets/2345/7 "2021-10-21T08:49:01Z")

</div>

`padding=True` in the data\_collator does the padding to the maximum length of the batch, so that’s the way to go 🙂

But if you want to do the tokenization in `map` instead of in the data collator you can, but you must add an extra padding step in the data\_collator to make sure all the examples in each batch have the same length