# Streaming dataset into Trainer: does not implement \_\_len\_\_, max\_steps has to be specified **URL:** \ **Category:** 🤗Datasets\ **Created:** [March 2, 2023, 2:25pm UTC](https://discuss.huggingface.co/t/streaming-dataset-into-trainer-does-not-implement-len-max-steps-has-to-be-specified/32893 "2023-03-02T14:25:38Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![nesty](https://avatars.discourse-cdn.com/v4/letter/n/ecae2f/32.png) [@nesty](https://discuss.huggingface.co/u/nesty)\ **Post date:** [March 2, 2023, 2:25pm UTC](https://discuss.huggingface.co/t/streaming-dataset-into-trainer-does-not-implement-len-max-steps-has-to-be-specified/32893/1 "2023-03-02T14:25:38Z")
Greetings, I am using datasets API to load a csv file in streaming mode, using this code: ```auto train_dataset = load_dataset("csv",data_files='train.csv', streaming=True) ``` Then I am converting it into a pytorch format: ```auto train_dataset = train_dataset.with_format("torch") ``` I am using a map function to tokenize the input and batch them: ```auto def encode(batch): inputs = tokenizer(list(map(join_commit_codes,batch['code'])), truncation=True, padding='max_length',max_length=max_commit_code_length) batch["input_ids"] = inputs.input_ids batch["attention_mask"] = inputs.attention_mask batch["token_type_ids"] = inputs.token_type_ids return batch train_dataset = train_dataset.map(encode, batch_size=batch_size,batched=True, remove_columns=["code"]) ``` And then passing it into a Trainer instance: ```auto trainer = Trainer( model=model, args=training_args, compute_metrics=compute_metrics, train_dataset=train_dataset.with_format("torch")['train'], eval_dataset =valid_dataset.with_format("torch")['train'] ) ``` But here I am getting this error: ```auto ValueError: train_dataset does not implement __len__ , max_steps has to be specified ``` I have tried other solutions like specifying a **len** function: ```auto class FromIterableDataset: def __init__ (self,iterable_dataset,length): self.dataset = iterable_dataset self.length = length def __len__ (self): return self.length ``` Then wrapping the dataset, but then I get an error that indexing is not supported. We have a large dataset and streaming it into the model for training is the best option. I would greatly appreciate your help in fixing this error. Thank you. ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [March 3, 2023, 3:23pm UTC](https://discuss.huggingface.co/t/streaming-dataset-into-trainer-does-not-implement-len-max-steps-has-to-be-specified/32893/2 "2023-03-03T15:23:02Z")
Have you tried passing `max_steps` to the trainer ? ---
**Author:** ![nesty](https://avatars.discourse-cdn.com/v4/letter/n/ecae2f/32.png) [@nesty](https://discuss.huggingface.co/u/nesty)\ **Post date:** [March 3, 2023, 3:49pm UTC](https://discuss.huggingface.co/t/streaming-dataset-into-trainer-does-not-implement-len-max-steps-has-to-be-specified/32893/3 "2023-03-03T15:49:08Z")
Hello, Thanks for the reply. The problem is I do not know the needed max\_steps in advance, is it dependent on the batch size and epochs? ---
**Author:** ![amitport](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/amitport/32/14639_2.png) [@amitport](https://discuss.huggingface.co/u/amitport)\ **Post date:** [March 10, 2023, 9:23am UTC](https://discuss.huggingface.co/t/streaming-dataset-into-trainer-does-not-implement-len-max-steps-has-to-be-specified/32893/5 "2023-03-10T09:23:24Z")
I’m using IterableDataset for reading large datasets (larger than 100G). I do not know how many rows they have and counting this in itself could take quite a while. In my opinion, hugging face should have just a notion of end of dataset in iterabledatasets. I believe this exists in torch/tf and this could be used to mark when an epoch is finished without requiring the number of rows. ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [March 10, 2023, 1:33pm UTC](https://discuss.huggingface.co/t/streaming-dataset-into-trainer-does-not-implement-len-max-steps-has-to-be-specified/32893/6 "2023-03-10T13:33:38Z")
I agree it would be nice if the Trainer could just check when the dataset ends and count that as an epoch. There is probably an issue on github about this though ---
**Author:** ![amitport](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/amitport/32/14639_2.png) [@amitport](https://discuss.huggingface.co/u/amitport)\ **Post date:** [March 21, 2023, 11:25am UTC](https://discuss.huggingface.co/t/streaming-dataset-into-trainer-does-not-implement-len-max-steps-has-to-be-specified/32893/7 "2023-03-21T11:25:34Z")
> [@lhoestq](#): > > e Trainer could just check when the dataset ends and count that as an epoch. There is probably an issue on github about this though There was, but AFAIKT it was resolved with a workaround (specifying max\_steps), which is less than ideal. ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\ **Post date:** [March 21, 2023, 2:09pm UTC](https://discuss.huggingface.co/t/streaming-dataset-into-trainer-does-not-implement-len-max-steps-has-to-be-specified/32893/8 "2023-03-21T14:09:47Z")
The Trainer can’t just wait for the end of an epoch as the number of steps needs to be known in advance for the learning rate scheduler. This is why we have no choice but to ask for `max_steps` to be set in this case.