# TrainingArguments class - max\_steps formula when using streaming dataset

**URL:** <https://discuss.huggingface.co/t/trainingarguments-class-max-steps-formula-when-using-streaming-dataset/36531>\
**Category:** 🤗Transformers\
**Created:** [April 14, 2023, 3:37am UTC](https://discuss.huggingface.co/t/trainingarguments-class-max-steps-formula-when-using-streaming-dataset/36531 "2023-04-14T03:37:49Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![monta](https://avatars.discourse-cdn.com/v4/letter/m/c77e96/32.png) [@monta](https://discuss.huggingface.co/u/monta)\
**Post date:** [April 14, 2023, 3:37am UTC](https://discuss.huggingface.co/t/trainingarguments-class-max-steps-formula-when-using-streaming-dataset/36531/1 "2023-04-14T03:37:49Z")

</div>

# Objective

Need a definite formula to decide the value to set max\_steps when using streaming dataset.

# Background

There are several questions raised about `max_steps` when using streaming dataset.

- [Explicitly set number of training steps using Trainer](https://discuss.huggingface.co/t/explicitly-set-number-of-training-steps-using-trainer/1127)
- [Streaming dataset into Trainer: does not implement **len** , max\_steps has to be specified](https://discuss.huggingface.co/t/streaming-dataset-into-trainer-does-not-implement-len-max-steps-has-to-be-specified/32893)]([Streaming dataset into Trainer: does not implement \_\_len\_\_, max\_steps has to be specified](https://discuss.huggingface.co/t/streaming-dataset-into-trainer-does-not-implement-len-max-steps-has-to-be-specified/32893))

According to the documents, it is set to **the total number of training steps** which should be number of total mini-batches.

- [max\_steps](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments.max_steps)

> If set to a positive number, **the total number of training steps** to perform. Overrides num\_train\_epochs.

I am afraid it is not clear.

# Question

Suppose there is a small dataset of 2048 rows in the train split of a Huggingface Dataset, and the training arguments are set as below except max\_steps as below.

```auto
training_args = TrainingArguments(
    output_dir="bloom_finetuned",
    max_steps=MAX_STEPS,
    num_train_epochs=3,
    per_device_train_batch_size=1,
    per_device_eval_batch_size=1,
    learning_rate=2e-5,
    weight_decay=0.01, 
    no_cuda=False,
)

```

Then for a system that has single GPU:

```auto
MAX_STEPS =num_train_epochs * num_rows_in_train / per_device_train_batch_size

```

Where:

- num\_rows\_in\_train=2048 is total number of records in the training dataset
- per\_device\_train\_batch\_size=1 is the batch size to be sent to GPU
- num\_train\_epochs=1 is the number of epochs to run

Is this correct?

If there are multiple GPU devices being used in parallel, then:

```auto
MAX_STEPS =num_train_epochs * num_rows_in_train / per_device_train_batch_size / num_gpu_devices

```

Is this correct?

# Confirmation

The Trainer training shows huge number of epochs for the above setting. Is this supposed to be like this?

```auto
*****Running training*****
  Num examples = 6,144
  Num Epochs = 9,223,372,036,854,775,807 <-----
  Instantaneous batch size per device = 1
  Total train batch size (w. parallel, distributed & accumulation) = 1
  Gradient Accumulation steps = 1
  Total optimization steps = 6,144
  Number of trainable parameters = 559,214,592

```

The reason `per_device_train_batch_size=1` is because training BLOOM takes up GPU memory and cannot set \> 1.

---

<div class="post-metadata">

**Author:** ![suranap](https://avatars.discourse-cdn.com/v4/letter/s/b2d939/32.png) [@suranap](https://discuss.huggingface.co/u/suranap)\
**Post date:** [September 14, 2023, 5:35pm UTC](https://discuss.huggingface.co/t/trainingarguments-class-max-steps-formula-when-using-streaming-dataset/36531/2 "2023-09-14T17:35:16Z")

</div>

I can’t answer your questions, but I did see Num Epochs hit this large number. In the code [here](https://github.com/huggingface/transformers/blob/17fdd35481e6b462989c1c600e6cc0987dc88621/src/transformers/trainer.py#L1625), if you set [max\_steps then that overrides num\_train\_epochs](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.Seq2SeqTrainingArguments.max_steps). They assign sys.maxsize so the rest of the code basically ignores num\_train\_epochs.