# Error when following Transformers Language modeling tutorial step by step

**URL:** <https://discuss.huggingface.co/t/error-when-following-transformers-language-modeling-tutorial-step-by-step/20936>\
**Category:** Beginners\
**Created:** [July 27, 2022, 8:32pm UTC](https://discuss.huggingface.co/t/error-when-following-transformers-language-modeling-tutorial-step-by-step/20936 "2022-07-27T20:32:32Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kickbub](https://avatars.discourse-cdn.com/v4/letter/k/e480ec/32.png) [@Kickbub](https://discuss.huggingface.co/u/Kickbub)\
**Post date:** [July 27, 2022, 8:32pm UTC](https://discuss.huggingface.co/t/error-when-following-transformers-language-modeling-tutorial-step-by-step/20936/1 "2022-07-27T20:32:32Z")

</div>

I am following this [tutorial](https://huggingface.co/docs/transformers/tasks/language_modeling#load-eli5-dataset) and copy-pasting all the code blocks into Colab. However when it gets to the training segment, this error pops up.

```auto
/usr/local/lib/python3.7/dist-packages/transformers/optimization.py:310: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning
  FutureWarning,
*****Running training*****
  Num examples = 8466
  Num Epochs = 3
  Instantaneous batch size per device = 8
  Total train batch size (w. parallel, distributed & accumulation) = 8
  Gradient Accumulation steps = 1
  Total optimization steps = 3177
 [254/3177 00:47 < 09:10, 5.31 it/s, Epoch 0.24/3]
Epoch	Training Loss	Validation Loss
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/transformers/tokenization_utils_base.py in convert_to_tensors(self, tensor_type, prepend_batch_axis)
    718 if not is_tensor(value):
--> 719 tensor = as_tensor(value)
    720 

ValueError: expected sequence of length 128 at dim 1 (got 127)

During handling of the above exception, another exception occurred:

ValueError Traceback (most recent call last)
10 frames
/usr/local/lib/python3.7/dist-packages/transformers/tokenization_utils_base.py in convert_to_tensors(self, tensor_type, prepend_batch_axis)
    734 )
    735 raise ValueError(
--> 736 "Unable to create tensor, you should probably activate truncation and/or padding with"
    737 " 'padding=True' 'truncation=True' to have batched tensors with the same length. Perhaps your"
    738 f" features (`{key}` in this case) have excessive nesting (inputs type `list` where type `int` is"

ValueError: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length. Perhaps your features (`labels` in this case) have excessive nesting (inputs type `list` where type `int` is expected).

```

I have already initialized the data\_collator/ran all the code blocks in the tutorial but am still receiving an error about truncation?

---

<div class="post-metadata">

**Author:** ![Kickbub](https://avatars.discourse-cdn.com/v4/letter/k/e480ec/32.png) [@Kickbub](https://discuss.huggingface.co/u/Kickbub)\
**Post date:** [July 28, 2022, 8:04pm UTC](https://discuss.huggingface.co/t/error-when-following-transformers-language-modeling-tutorial-step-by-step/20936/2 "2022-07-28T20:04:31Z")

</div>

Ok, apparently distilgpt2 doesn’t support dynamic padding through the data\_collator (or padding in general? not too sure). So I opted to drop the remainders.

```auto
block_size = 128

def group_texts(examples):
    # Concatenate all texts.
    concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}
    total_length = len(concatenated_examples[list(examples.keys())[0]])
    # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can
        # customize this part to your needs.
    total_length = (total_length // block_size) * block_size
    # Split by chunks of max_len.
    result = {
        k: [t[i : i + block_size] for i in range(0, total_length, block_size)]
        for k, t in concatenated_examples.items()
    }
    result["labels"] = result["input_ids"].copy()
    return result

```