# Using External Datasets with HuggingFace Data Loader

**URL:** <https://discuss.huggingface.co/t/using-external-datasets-with-huggingface-data-loader/17021>\
**Category:** Beginners\
**Created:** [April 20, 2022, 8:22pm UTC](https://discuss.huggingface.co/t/using-external-datasets-with-huggingface-data-loader/17021 "2022-04-20T20:22:28Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![PaulYoung](https://avatars.discourse-cdn.com/v4/letter/p/6bbea6/32.png) [@PaulYoung](https://discuss.huggingface.co/u/PaulYoung)\
**Post date:** [April 20, 2022, 8:22pm UTC](https://discuss.huggingface.co/t/using-external-datasets-with-huggingface-data-loader/17021/1 "2022-04-20T20:22:28Z")

</div>

Hi, I am a beginner with HuggingFace and PyTorch and I am having trouble doing a simple task. I took the ViT tutorial [Fine-Tune ViT for Image Classification with 🤗 Transformers](https://huggingface.co/blog/fine-tune-vit) and replaced the second block with this:

> from datasets import load\_dataset  
> ds = load\_dataset(  
> ‘./tiny-imagenet-200’)  
> #data\_files= {“train”: “train”, “test”: “test”, “validate”: “val”})  
> ds

As the name implies, ./tiny-imagenet-200 is the directory containing the tiny-imagenet dataset in its default configuration. Here is the data structure of that dataset:  
./tiny-imagenet-200  
test/  
images/  
[all test JPEG images]  
train/  
[directories for each class number]/  
images/  
[current class JPEG images]  
[class name]\_boxes.txt  
val/  
images/  
[all validation JPEG images]  
val\_annotations.txt  
wnids.txt (appears to be a list of class numbers)  
words.txt (table of class numbers to names)

The error returned is

> Unable to resolve any data file that matches ‘[’\*_test_’, ‘\*_eval_’]’ at /home/omniverse03/Documents/BeansTransformerTutorial/tiny-imagenet-200 with any supported extension [‘csv’, ‘tsv’, ‘json’, ‘jsonl’, ‘parquet’, ‘txt’, ‘blp’, ‘bmp’, ‘dib’, ‘bufr’, ‘cur’, ‘pcx’, ‘dcx’, ‘dds’, ‘ps’, ‘eps’, ‘fit’, ‘fits’, ‘fli’, ‘flc’, ‘ftc’, ‘ftu’, ‘gbr’, ‘gif’, ‘grib’, ‘h5’, ‘hdf’, ‘png’, ‘apng’, ‘jp2’, ‘j2k’, ‘jpc’, ‘jpf’, ‘jpx’, ‘j2c’, ‘icns’, ‘ico’, ‘im’, ‘iim’, ‘tif’, ‘tiff’, ‘jfif’, ‘jpe’, ‘jpg’, ‘jpeg’, ‘mpg’, ‘mpeg’, ‘msp’, ‘pcd’, ‘pxr’, ‘pbm’, ‘pgm’, ‘ppm’, ‘pnm’, ‘psd’, ‘bw’, ‘rgb’, ‘rgba’, ‘sgi’, ‘ras’, ‘tga’, ‘icb’, ‘vda’, ‘vst’, ‘webp’, ‘wmf’, ‘emf’, ‘xbm’, ‘xpm’, ‘zip’]

---

<div class="post-metadata">

**Author:** ![mapama247](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mapama247/32/33424_2.png) [@mapama247](https://discuss.huggingface.co/u/mapama247)\
**Post date:** [April 21, 2022, 9:13am UTC](https://discuss.huggingface.co/t/using-external-datasets-with-huggingface-data-loader/17021/2 "2022-04-21T09:13:38Z")

</div>

Hi!

I assume that you are using the official dataset from Stanford, which can be downloaded by doing:

```auto
wget http://cs231n.stanford.edu/tiny-imagenet-200.zip

```

After unzipping it I simply did:

```auto
from datasets import load_dataset
ds = load_dataset('imagefolder', data_dir='./tiny-imagenet-200')

```

This loads the 120k examples into a single set. To load a specific set (i.e. test) you could do something like:

```auto
ds_test = load_dataset('imagefolder', data_dir='./tiny-imagenet-200/test')

```

As you can see this uses an ImageFolder dataset builder, the alternative would be to build your own custom dataloader (e.g. [food101.py](https://huggingface.co/datasets/food101/blob/main/food101.py)). Here’s some information on [how to write data loading scripts](https://huggingface.co/docs/datasets/v1.9.0/add_dataset.html), I believe all the examples are for text data but you can just check the loading script of any image dataset available on the hub.

PS: Make sure that you have installed the “Image” feature from the datasets library!

```auto
pip install datasets[vision]

```

You can find more info in the [official documentation](https://huggingface.co/docs/datasets/image_process#process-image-data).

---

<div class="post-metadata">

**Author:** ![PaulYoung](https://avatars.discourse-cdn.com/v4/letter/p/6bbea6/32.png) [@PaulYoung](https://discuss.huggingface.co/u/PaulYoung)\
**Post date:** [April 21, 2022, 4:11pm UTC](https://discuss.huggingface.co/t/using-external-datasets-with-huggingface-data-loader/17021/3 "2022-04-21T16:11:55Z")

</div>

Thank you so much. I was really struggling with that.  
Do you know why the tutorial I was using was able to use an image dataset without installing datasets[vision] (it only installed datasets transformers)? Is it because they were built in huggingface datasets?

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [April 21, 2022, 4:26pm UTC](https://discuss.huggingface.co/t/using-external-datasets-with-huggingface-data-loader/17021/4 "2022-04-21T16:26:51Z")

</div>

Hi! Note that

```python
ds = load_dataset('imagefolder', data_dir='./tiny-imagenet-200')

```

puts all the images in a singe train split. Also, this fails to infer the lables since tiny imagenet is not a valid image folder.

This is the corrected code:

```python
import os
from datasets import DownloadConfig, load_dataset, Image, DatasetDict

PATH_TO_TINY_IMAGENET_DIR = "..." 

# avoids issues with filelock + multiprocessing on some platforms
dc = DownloadConfig(num_proc=1)

# prepare train split
train_dset = load_dataset("imagefolder", download_config=dc, data_dir=os.path.join(PATH_TO_TINY_IMAGENET_DIR, "train"), split="train", ignore_verifications=True)

train_dset = train_dset.remove_columns("label")
train_dset = train_dset.cast_column("image", Image(decode=False))
train_dset = train_dset.map(lambda ex: {"label": ex["image"]["path"].replace("\\", "/").split("/")[-3]})
train_dset = train_dset.cast_column("image", Image(decode=True))

train_dset = train_dset.class_encode_column("label")

# prepare validation split
val_dset = load_dataset("imagefolder", download_config=dc, data_dir=os.path.join(PATH_TO_TINY_IMAGENET_DIR, "val", "images"), split="train", ignore_verifications=True)

annotations = {}
with open(os.path.join(PATH_TO_TINY_IMAGENET_DIR, "val", "val_annotations.txt")) as f:
    for line in f:
        if line:
            img_file, label, *_ = line.split()
            annotations[img_file.lower()] = label

val_dset = val_dset.remove_columns("label")
val_dset = val_dset.cast_column("image", Image(decode=False))
val_dset = val_dset.map(lambda ex: {"label": train_dset.features["label"].str2int(annotations[os.path.basename(ex["image"]["path"]).lower()])}, features=train_dset.features)
val_dset = val_dset.cast_column("image", Image(decode=True))

# prepare test split
test_dset = load_dataset("imagefolder", download_config=dc, data_dir=os.path.join(PATH_TO_TINY_IMAGENET_DIR, "test", "images"), split="train", ignore_verifications=True)
test_dset = test_dset.remove_columns("label")
test_dset = test_dset.map(lambda ex: {"label": None}, features=train_dset.features)

# final dataset with all the splits
ds = DatasetDict({"train": train_dset, "validation": val_dset, "test": test_dset})

```

-

---

<div class="post-metadata">

**Author:** ![PaulYoung](https://avatars.discourse-cdn.com/v4/letter/p/6bbea6/32.png) [@PaulYoung](https://discuss.huggingface.co/u/PaulYoung)\
**Post date:** [April 22, 2022, 6:14pm UTC](https://discuss.huggingface.co/t/using-external-datasets-with-huggingface-data-loader/17021/5 "2022-04-22T18:14:59Z")

</div>

Thank you Mario. It’s surprising that such a prominent dataset is so complicated to load. Note that even this solution leaves the class names as n000XXX rather than the correlated text in words.txt but that is not critical to my current task.

---

<div class="post-metadata">

**Author:** ![PaulYoung](https://avatars.discourse-cdn.com/v4/letter/p/6bbea6/32.png) [@PaulYoung](https://discuss.huggingface.co/u/PaulYoung)\
**Post date:** [April 22, 2022, 6:40pm UTC](https://discuss.huggingface.co/t/using-external-datasets-with-huggingface-data-loader/17021/6 "2022-04-22T18:40:16Z")

</div>

Continuing with my quest to run the tutorial with the tinyimagenet dataset, I got to this block

> train\_results = trainer.train()  
> trainer.save\_model()  
> trainer.log\_metrics(“train”, train\_results.metrics)  
> trainer.save\_metrics(“train”, train\_results.metrics)  
> trainer.save\_state()

and it threw “ValueError: operands could not be broadcast together with shapes (224,224) (3,)” on the first line. This appears to be indicating a numpy dimensions error based on my investigations, but it is deep within trainer.train(). Moreover, there is not any relevant 3’s or 224’s in all the earlier code.

> **[Google Colaboratory](https://colab.research.google.com/drive/1Au0xDvmD1ahyNXWOAJQ_2UReRMWT3EkF?usp=sharing)**

All code is identical to the tutorial I mentioned with these exceptions:  
the pip install mapama mentioned  
block 2 replaced by mario’s code  
Some printouts removed  
save, eval, and logging steps multiplied by 10 to reduce output

---

<div class="post-metadata">

**Author:** ![PaulYoung](https://avatars.discourse-cdn.com/v4/letter/p/6bbea6/32.png) [@PaulYoung](https://discuss.huggingface.co/u/PaulYoung)\
**Post date:** [April 25, 2022, 6:33pm UTC](https://discuss.huggingface.co/t/using-external-datasets-with-huggingface-data-loader/17021/7 "2022-04-25T18:33:46Z")

</div>

## Full error message below:

ValueError Traceback (most recent call last)  
/[mypath]/nextClassification.ipynb Cell 29’ in \<cell line: 1\>()  
----\> 1 train\_results = trainer.train()  
2 trainer.save\_model()  
3 trainer.log\_metrics(“train”, train\_results.metrics)

File ~/.local/lib/python3.8/site-packages/transformers/trainer.py:1396, in Trainer.train(self, resume\_from\_checkpoint, trial, ignore\_keys\_for\_eval, \*\*kwargs)  
1393 self.control = self.callback\_handler.on\_epoch\_begin(args, self.state, self.control)  
1395 step = -1  
 → 1396 for step, inputs in enumerate(epoch\_iterator):  
1397  
1398 # Skip past any already trained steps if resuming training  
1399 if steps\_trained\_in\_current\_epoch \> 0:  
1400 steps\_trained\_in\_current\_epoch -= 1

File ~/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py:530, in \_BaseDataLoaderIter. **next** (self)  
528 if self.\_sampler\_iter is None:  
529 self.\_reset()  
 → 530 data = self.\_next\_data()  
531 self.\_num\_yielded += 1  
532 if self.\_dataset\_kind == \_DatasetKind.Iterable and   
533 self.\_IterableDataset\_len\_called is not None and   
534 self.\_num\_yielded \> self.\_IterableDataset\_len\_called:

File ~/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py:570, in \_SingleProcessDataLoaderIter.\_next\_data(self)  
568 def \_next\_data(self):  
569 index = self.\_next\_index() # may raise StopIteration  
 → 570 data = self.\_dataset\_fetcher.fetch(index) # may raise StopIteration  
571 if self.\_pin\_memory:  
572 data = \_utils.pin\_memory.pin\_memory(data)

File ~/.local/lib/python3.8/site-packages/torch/utils/data/\_utils/fetch.py:49, in \_MapDatasetFetcher.fetch(self, possibly\_batched\_index)  
47 def fetch(self, possibly\_batched\_index):  
48 if self.auto\_collation:  
—\> 49 data = [self.dataset[idx] for idx in possibly\_batched\_index]  
50 else:  
51 data = self.dataset[possibly\_batched\_index]

File ~/.local/lib/python3.8/site-packages/torch/utils/data/\_utils/fetch.py:49, in (.0)  
47 def fetch(self, possibly\_batched\_index):  
48 if self.auto\_collation:  
—\> 49 data = [self.dataset[idx] for idx in possibly\_batched\_index]  
50 else:  
51 data = self.dataset[possibly\_batched\_index]

File ~/.local/lib/python3.8/site-packages/datasets/arrow\_dataset.py:1764, in Dataset. **getitem** (self, key)  
1762 def **getitem** (self, key): # noqa: F811  
1763 “”“Can be used to index columns (by string names) or rows (by integer index or iterable of indices or bools).”“”  
 → 1764 return self.\_getitem(  
1765 key,  
1766 )

File ~/.local/lib/python3.8/site-packages/datasets/arrow\_dataset.py:1749, in Dataset.\_getitem(self, key, decoded, \*\*kwargs)  
1747 formatter = get\_formatter(format\_type, features=self.features, decoded=decoded, \*\*format\_kwargs)  
1748 pa\_subtable = query\_table(self.\_data, key, indices=self.\_indices if self.\_indices is not None else None)  
 → 1749 formatted\_output = format\_table(  
1750 pa\_subtable, key, formatter=formatter, format\_columns=format\_columns, output\_all\_columns=output\_all\_columns  
1751 )  
1752 return formatted\_output

File ~/.local/lib/python3.8/site-packages/datasets/formatting/formatting.py:532, in format\_table(table, key, formatter, format\_columns, output\_all\_columns)  
530 python\_formatter = PythonFormatter(features=None)  
531 if format\_columns is None:  
 → 532 return formatter(pa\_table, query\_type=query\_type)  
533 elif query\_type == “column”:  
534 if key in format\_columns:

File ~/.local/lib/python3.8/site-packages/datasets/formatting/formatting.py:281, in Formatter. **call** (self, pa\_table, query\_type)  
279 def **call** (self, pa\_table: pa.Table, query\_type: str) → Union[RowFormat, ColumnFormat, BatchFormat]:  
280 if query\_type == “row”:  
 → 281 return self.format\_row(pa\_table)  
282 elif query\_type == “column”:  
283 return self.format\_column(pa\_table)

File ~/.local/lib/python3.8/site-packages/datasets/formatting/formatting.py:387, in CustomFormatter.format\_row(self, pa\_table)  
386 def format\_row(self, pa\_table: pa.Table) → dict:  
 → 387 formatted\_batch = self.format\_batch(pa\_table)  
388 try:  
389 return \_unnest(formatted\_batch)

File ~/.local/lib/python3.8/site-packages/datasets/formatting/formatting.py:418, in CustomFormatter.format\_batch(self, pa\_table)  
416 if self.decoded:  
417 batch = self.python\_features\_decoder.decode\_batch(batch)  
 → 418 return self.transform(batch)

[mypath]/nextClassification.ipynb Cell 17’ in transform(example\_batch)  
1 def transform(example\_batch):  
2 # Take a list of PIL images and turn them to pixel values  
----\> 3 inputs = feature\_extractor([x for x in example\_batch[‘image’]], return\_tensors=‘pt’)  
5 # Don’t forget to include the labels!  
6 inputs[‘labels’] = example\_batch[‘labels’]

File ~/.local/lib/python3.8/site-packages/transformers/models/vit/feature\_extraction\_vit.py:143, in ViTFeatureExtractor. **call** (self, images, return\_tensors, \*\*kwargs)  
141 images = [self.resize(image=image, size=self.size, resample=self.resample) for image in images]  
142 if self.do\_normalize:  
 → 143 images = [self.normalize(image=image, mean=self.image\_mean, std=self.image\_std) for image in images]  
145 # return as BatchFeature  
146 data = {“pixel\_values”: images}

File ~/.local/lib/python3.8/site-packages/transformers/models/vit/feature\_extraction\_vit.py:143, in (.0)  
141 images = [self.resize(image=image, size=self.size, resample=self.resample) for image in images]  
142 if self.do\_normalize:  
 → 143 images = [self.normalize(image=image, mean=self.image\_mean, std=self.image\_std) for image in images]  
145 # return as BatchFeature  
146 data = {“pixel\_values”: images}

File ~/.local/lib/python3.8/site-packages/transformers/image\_utils.py:186, in ImageFeatureExtractionMixin.normalize(self, image, mean, std)  
184 return (image - mean[:, None, None]) / std[:, None, None]  
185 else:  
 → 186 return (image - mean) / std

ValueError: operands could not be broadcast together with shapes (224,224) (3,)

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [April 26, 2022, 1:20pm UTC](https://discuss.huggingface.co/t/using-external-datasets-with-huggingface-data-loader/17021/8 "2022-04-26T13:20:38Z")

</div>

> It’s surprising that such a prominent dataset is so complicated to load. Note that even this solution leaves the class names as n000XXX rather than the correlated text in words.txt but that is not critical to my current task.

This dataset is tricky to load because it doesn’t follow the standard image folder structure. And you can use `map` similar to the `map` calls from my snippet to replace the class names with the correlated words.

> and it threw “ValueError: operands could not be broadcast together with shapes (224,224) (3,)” on the first line.

It would be easier to debug this error from the actual code, but your notebook is not public, so I’d assume that some of the images are grayscale. Replacing the line:

```python
inputs = feature_extractor([x for x in example_batch[‘image’]], return_tensors=‘pt’)

```

with

```python
inputs = feature_extractor([x.convert("RGB") for x in example_batch[‘image’]], return_tensors=‘pt’)

```

should fix the issue.

---

<div class="post-metadata">

**Author:** ![PaulYoung](https://avatars.discourse-cdn.com/v4/letter/p/6bbea6/32.png) [@PaulYoung](https://discuss.huggingface.co/u/PaulYoung)\
**Post date:** [April 26, 2022, 3:25pm UTC](https://discuss.huggingface.co/t/using-external-datasets-with-huggingface-data-loader/17021/9 "2022-04-26T15:25:01Z")

</div>

You are an absolute legend. May I ask what made you think that was the issue so perhaps I can solve something like this in the future?

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [April 27, 2022, 7:13pm UTC](https://discuss.huggingface.co/t/using-external-datasets-with-huggingface-data-loader/17021/10 "2022-04-27T19:13:01Z")

</div>

ViT expects 3 input channels by default.