# Error training with iterabledatasets

**URL:** <https://discuss.huggingface.co/t/error-training-with-iterabledatasets/20699>\
**Category:** Beginners\
**Created:** [July 21, 2022, 9:34pm UTC](https://discuss.huggingface.co/t/error-training-with-iterabledatasets/20699 "2022-07-21T21:34:28Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![jrf](https://avatars.discourse-cdn.com/v4/letter/j/3be4f8/32.png) [@jrf](https://discuss.huggingface.co/u/jrf)\
**Post date:** [July 21, 2022, 9:34pm UTC](https://discuss.huggingface.co/t/error-training-with-iterabledatasets/20699/1 "2022-07-21T21:34:28Z")

</div>

Hello, I’m stuck on an error trying to train Wave2Vec2 with an IterableDatasetDict. I pasted in some key code below, but I can just send the .py file if that would be better.

Here is the error:

Num examples = 16  
Num Epochs = 9223372036854775807  
Instantaneous batch size per device = 16  
Total train batch size (w. parallel, distributed & accumulation) = 16  
Gradient Accumulation steps = 1  
Total optimization steps = 1  
0%| | 0/1 [00:00\<?, ?it/s]Traceback (most recent call last):  
File “C:\Users\jfolstein\Documents\Projects\HOAX\Wav2Vec2v2\venv\lib\site-packages\torch\utils\data\dataloader.py”, line 652, in **next**  
data = self.\_next\_data()  
File “C:\Users\jfolstein\Documents\Projects\HOAX\Wav2Vec2v2\venv\lib\site-packages\torch\utils\data\dataloader.py”, line 692, in \_next\_data  
data = self.\_dataset\_fetcher.fetch(index) # may raise StopIteration  
File “C:\Users\jfolstein\Documents\Projects\HOAX\Wav2Vec2v2\venv\lib\site-packages\torch\utils\data\_utils\fetch.py”, line 32, in fetch  
data.append(next(self.dataset\_iter))  
File “C:\Users\jfolstein\Documents\Projects\HOAX\Wav2Vec2v2\venv\lib\site-packages\datasets\formatting\dataset\_wrappers\torch\_iterable\_dataset.py”, line 28, in **iter**  
yield from IterableDataset. **iter** (self)  
File “C:\Users\jfolstein\Documents\Projects\HOAX\Wav2Vec2v2\venv\lib\site-packages\datasets\iterable\_dataset.py”, line 599, in **iter**  
for key, example in self.\_iter():  
File “C:\Users\jfolstein\Documents\Projects\HOAX\Wav2Vec2v2\venv\lib\site-packages\datasets\iterable\_dataset.py”, line 579, in \_iter  
yield from ex\_iterable  
File “C:\Users\jfolstein\Documents\Projects\HOAX\Wav2Vec2v2\venv\lib\site-packages\datasets\iterable\_dataset.py”, line 280, in **iter**  
for key, example in iterator:  
File “C:\Users\jfolstein\Documents\Projects\HOAX\Wav2Vec2v2\venv\lib\site-packages\datasets\iterable\_dataset.py”, line 457, in **iter**  
for \_ in islice(ex\_iterator, self.n):  
ValueError: Stop argument for islice() must be None or an integer: 0 \<= x \<= sys.maxsize.

When I follow the debugger into iterable\_dataset.py to the line  
for \_ in islice(ex\_iterator, self.n)  
self.n is a float

I haven’t yet been able to figure out what self.n is exactly.

Here is how I am creating the iterable datasets:

trnsets =   
ntrnfiles = 0  
for set in trnsets2use:  
myset = loaddsetshards(set[‘path’])  
trnsets.append(myset[‘train’])  
ntrnfiles += set[‘nfiles’]

print(f"total training files: {str(ntrnfiles)}")  
dset\_training = hfds.interleave\_datasets(trnsets, probabilities=[.1, .9])  
dset\_training = dset\_training.shuffle(seed=1)  
dset\_trn = dset\_training.skip(ntrnfiles/4)  
dset\_val = dset\_training.take(ntrnfiles/4)

dset\_tst = loaddsetshards(testsets2use[0][‘path’])

dset = hfds.IterableDatasetDict({‘train’: dset\_trn, ‘test’: dset\_tst, ‘validation’: dset\_val})

loaddsetshards loads a bunch of .json shards in a diectory:

def loaddsetshards(dsetdir):  
\_, \_, filepaths = get\_filepaths(dsetdir)  
dslist =   
print(f"loading all shards in {dsetdir}")  
dsetout = datasets.load\_dataset(dsetdir, streaming=True)  
return dsetout

here are the training args…

numtrnepochs = 4  
batchperdev = 16  
stepsperep = numtrnepochs/batchperdev  
maxsteps = int(numtrnepochs\*stepsperep)

training\_args = TrainingArguments(  
output\_dir=outputpath,  
logging\_dir=logpath,  
per\_device\_train\_batch\_size=batchperdev,  
evaluation\_strategy=“steps”,  
num\_train\_epochs=numtrnepochs,  
fp16=True,  
gradient\_checkpointing=True,  
save\_steps=300,  
eval\_steps=300,  
logging\_steps=300,  
learning\_rate=1e-3, # 1e-4,  
weight\_decay=0.005,  
warmup\_steps=1000,  
save\_total\_limit=2,  
remove\_unused\_columns=True,  
max\_steps=maxsteps  
)  
print(‘making trainer’)  
trainer = Trainer(  
model=model,  
data\_collator=data\_collator,  
args=training\_args,  
compute\_metrics=compute\_metrics,  
train\_dataset=dset[“train”],  
eval\_dataset=dset[“validation”],  
tokenizer=processor.feature\_extractor,  
)

---

<div class="post-metadata">

**Author:** ![jrf](https://avatars.discourse-cdn.com/v4/letter/j/3be4f8/32.png) [@jrf](https://discuss.huggingface.co/u/jrf)\
**Post date:** [July 22, 2022, 12:49pm UTC](https://discuss.huggingface.co/t/error-training-with-iterabledatasets/20699/2 "2022-07-22T12:49:07Z")

</div>

Ok, I fixed this by changing

dset\_trn = dset\_training.skip( **ntrnfiles/4** )  
dset\_val = dset\_training.take( **ntrnfiles/4** )

to

dset\_trn = dset\_training.skip(**int(ntrnfiles/4)**)  
dset\_val = dset\_training.take(**int(ntrnfiles/4)**)

Dividing by 4 changed a counter into a float somewhere. Might be worth throwing an informative error message in there, but I seem to be the only one dumb enough to make that mistake so far.