# Weird example of batching in Dataset.map document

**URL:** <https://discuss.huggingface.co/t/weird-example-of-batching-in-dataset-map-document/24861>\
**Category:** 🤗Datasets\
**Created:** [October 24, 2022, 1:39am UTC](https://discuss.huggingface.co/t/weird-example-of-batching-in-dataset-map-document/24861 "2022-10-24T01:39:58Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![liujxing](https://avatars.discourse-cdn.com/v4/letter/l/13edae/32.png) [@liujxing](https://discuss.huggingface.co/u/liujxing)\
**Post date:** [October 24, 2022, 1:39am UTC](https://discuss.huggingface.co/t/weird-example-of-batching-in-dataset-map-document/24861/1 "2022-10-24T01:39:58Z")

</div>

In the document of `Dataset.map` ([here](https://huggingface.co/docs/datasets/process)), the example given in “Batch processing” → “Split long examples” says “Batch processing enables interesting applications such as splitting long sentences into shorter chunks and data augmentation” with the following code:

```auto
def chunk_examples(examples):
    chunks = []
    for sentence in examples["sentence1"]:
        chunks += [sentence[i:i + 50] for i in range(0, len(sentence), 50)]
    return {"chunks": chunks}
chunked_dataset = dataset.map(chunk_examples, batched=True, remove_columns=dataset.column_names)

```

This example looks weird to me, because it seems it’s just doing the chunking row by row, so it does not matter whether we are doing it row by row or in a batched way. Did I misunderstand anything here?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [October 24, 2022, 9:32am UTC](https://discuss.huggingface.co/t/weird-example-of-batching-in-dataset-map-document/24861/2 "2022-10-24T09:32:32Z")

</div>

By default, `map` requires an input one 1 example and to output 1 example.

But a batched `map` can take a input batch of size `N` and output a batch of size `M`.  
The code you provided indeed returns a batch with more examples than the input.

---

<div class="post-metadata">

**Author:** ![matthewchung74](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/matthewchung74/32/15244_2.png) [@matthewchung74](https://discuss.huggingface.co/u/matthewchung74)\
**Post date:** [September 1, 2023, 5:19pm UTC](https://discuss.huggingface.co/t/weird-example-of-batching-in-dataset-map-document/24861/3 "2023-09-01T17:19:18Z")

</div>

Hi @lhoestq , with that example above, it is just splitting sentence1, but not saving the labels, ie the label and idx columns in the chunked\_dataset. How does one split a long sentence and save the labels in a format for the Trainer to understand?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [September 4, 2023, 9:29am UTC](https://discuss.huggingface.co/t/weird-example-of-batching-in-dataset-map-document/24861/4 "2023-09-04T09:29:25Z")

</div>

You can duplicate the label for each chunk:

```python
def chunk_examples(examples):
    chunks = []
    labels = []
    for sentence, label in zip(examples["sentence1"], examples["label"]):
        chunks += [sentence[i:i + 50] for i in range(0, len(sentence), 50)]
        labels += [label] * len(range(0, len(sentence), 50))
    return {"chunk": chunks, "label": labels}
chunked_dataset = dataset.map(chunk_examples, batched=True, remove_columns=dataset.column_names)

```

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [September 4, 2023, 9:57am UTC](https://discuss.huggingface.co/t/weird-example-of-batching-in-dataset-map-document/24861/5 "2023-09-04T09:57:06Z")

</div>

And if you’re doing NER and have multiple labels per sentence:

```python
def chunk_examples(examples):
    chunks = []
    labels = []
    for sentence, labels in zip(examples["sentence1"], examples["labels"]):
        chunks += [sentence[i:i + 50] for i in range(0, len(sentence), 50)]
        labels += [labels[i:i + 50] for i in range(0, len(labels), 50)]
    return {"chunk": chunks, "labels": labels}
chunked_dataset = dataset.map(chunk_examples, batched=True, remove_columns=dataset.column_names)

```