# Clarification on Batch mapping

**URL:** <https://discuss.huggingface.co/t/clarification-on-batch-mapping/60366>\
**Category:** 🤗Datasets\
**Created:** [October 30, 2023, 2:44am UTC](https://discuss.huggingface.co/t/clarification-on-batch-mapping/60366 "2023-10-30T02:44:08Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![supercoolaj](https://avatars.discourse-cdn.com/v4/letter/s/ac91a4/32.png) [@supercoolaj](https://discuss.huggingface.co/u/supercoolaj)\
**Post date:** [October 30, 2023, 2:44am UTC](https://discuss.huggingface.co/t/clarification-on-batch-mapping/60366/1 "2023-10-30T02:44:09Z")

</div>

Hello!

Does batch mapping ( i.e. `dataset.map(batched=True)`) preserve individual data samples? How do I access each individual sample after batch mapping?

I have a 50K dataset and after batching, that reduces the dataset to 50 examples (which is expected because `batch_size=1000`). When I look at the first sample, it reached the maximum token length.

But if I set `batched=False`, the first sample does not reach the maximum token length.

Does this mean that batch mapping concatenates the examples? If I want my model to read every example (without losing info because of concatenation), should I set `batched=False`?

---

<div class="post-metadata">

**Author:** ![supercoolaj](https://avatars.discourse-cdn.com/v4/letter/s/ac91a4/32.png) [@supercoolaj](https://discuss.huggingface.co/u/supercoolaj)\
**Post date:** [October 30, 2023, 5:09am UTC](https://discuss.huggingface.co/t/clarification-on-batch-mapping/60366/2 "2023-10-30T05:09:22Z")

</div>

Ok, my understanding was wrong. If I preprocess the dataset before passing `.map(batched=True)`, it will preserve _all samples_. In other words:

```python
def tokenizer_func(examples):
        return tokenizer(examples['final'], 
        truncation=True, padding=True, max_length=128, return_tensors="pt")

tokenized_train = train_dset.map(tokenizer_func,
                                 remove_columns=train_dset.column_names, batched=True)
>>> Dataset({
    features: ['input_ids', 'attention_mask'],
    num_rows: 50000
    })

```

However, if I preprocess the dataset during the `.map()` call, I end up with 50 examples.

```python
def tokenizer_func(examples):
        return tokenizer(generate_prompt(examples['raw']), 
        truncation=True, padding=True, max_length=128, return_tensors="pt")

# examples follow format of resp json files
tokenized_train = train_dset.map(tokenizer_func,
                                 remove_columns=train_dset.column_names, batched=True)
>>> Dataset({
    features: ['input_ids', 'attention_mask'],
    num_rows: 50
    })

```

Can someone explain the difference in behavior? `generate_prompt` is simply a function that returns a string.

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [November 2, 2023, 5:11pm UTC](https://discuss.huggingface.co/t/clarification-on-batch-mapping/60366/3 "2023-11-02T17:11:37Z")

</div>

When `batched=True`, both `examples['final']` and `examples['raw']` will be a list of `batch_size` elements. You may have to use

```python
def tokenizer_func(examples):
        return tokenizer([generate_prompt(raw_text) for raw_text in examples['raw']]),
                                  truncation=True, padding=True, max_length=128, return_tensors="pt")

```