# Loading nested dataset for training

**URL:** <https://discuss.huggingface.co/t/loading-nested-dataset-for-training/139133>\
**Category:** 🤗Datasets\
**Created:** [February 5, 2025, 1:00pm UTC](https://discuss.huggingface.co/t/loading-nested-dataset-for-training/139133 "2025-02-05T13:00:03Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![sourabhY](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sourabhy/32/40447_2.png) [@sourabhY](https://discuss.huggingface.co/u/sourabhY)\
**Post date:** [February 5, 2025, 1:00pm UTC](https://discuss.huggingface.co/t/loading-nested-dataset-for-training/139133/1 "2025-02-05T13:00:03Z")

</div>

I am trying to use allenai/pixmo-docs which has structure as

dataset\_info:

- config\_name: charts  
features:
  - name: image  
dtype: image
  - name: image\_id  
dtype: string
  - name: questions  
sequence:
    - name: question  
dtype: string
    - name: answer  
dtype: string  
and I am using this code and getting list indices must be integers/slices error and don’t know what to do. please help!!!

def preprocess\_function(examples):  
processed\_inputs = {  
‘input\_ids’: ,  
‘attention\_mask’: ,  
‘pixel\_values’: ,  
‘labels’:   
}

```
for img, questions, answers in zip(examples['image'], examples['questions']['question'], examples['questions']['answer']):
    for q, a in zip(questions, answers):
        inputs = processor(images=img, text=q, padding="max_length", truncation=True, return_tensors="pt")
        
        processed_inputs['input_ids'].append(inputs['input_ids'][0])
        processed_inputs['attention_mask'].append(inputs['attention_mask'][0])
        processed_inputs['pixel_values'].append(inputs['pixel_values'][0])
        processed_inputs['labels'].append(a)

return processed_inputs

```

processed\_dataset = dataset.map(preprocess\_function, batched=True, remove\_columns=dataset.column\_names)

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [February 5, 2025, 1:54pm UTC](https://discuss.huggingface.co/t/loading-nested-dataset-for-training/139133/2 "2025-02-05T13:54:37Z")

</div>

It would be easy if this were the cause…

> <https://stackoverflow.com/questions/77513504/typeerror-list-indices-must-be-integers-or-slices-not-str-when-import-hf-datas>

---

<div class="post-metadata">

**Author:** ![sourabhY](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sourabhy/32/40447_2.png) [@sourabhY](https://discuss.huggingface.co/u/sourabhY)\
**Post date:** [February 5, 2025, 2:08pm UTC](https://discuss.huggingface.co/t/loading-nested-dataset-for-training/139133/3 "2025-02-05T14:08:02Z")

</div>

Man thank you but when i run same script in colab i get different error.  
Like remove column names train, validation not present found image… Something like this

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [February 5, 2025, 2:42pm UTC](https://discuss.huggingface.co/t/loading-nested-dataset-for-training/139133/4 "2025-02-05T14:42:17Z")

</div>

There are actually errors that only occur in Colab, but the most common one is when the library version is different and the error content changes. Let’s try raising and lowering the library version in Colab. accelerate, transformers, datasets, and huggingface\_hub have a significant impact.

> [@TrainingArgument does not work on colab](https://discuss.huggingface.co/t/trainingargument-does-not-work-on-colab/43372):
>
> I was following chapter 3 of the NLP course on Fine-tuning a model with the Trainer API and when executing the command in colab training\_args = TrainingArguments(“test-trainer”) I get: ImportError Traceback (most recent call last) in \<cell line: 1\>() ----\> 1 training\_args = TrainingArguments(output\_dir=“test-trainer”) 4 frames /usr/local/lib/python3.10/dist-packages/transformers/training\_args.py in \_setup\_devices(self) 1670 if not is\_sagemaker\_mp…

---

<div class="post-metadata">

**Author:** ![sourabhY](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sourabhy/32/40447_2.png) [@sourabhY](https://discuss.huggingface.co/u/sourabhY)\
**Post date:** [February 5, 2025, 3:16pm UTC](https://discuss.huggingface.co/t/loading-nested-dataset-for-training/139133/5 "2025-02-05T15:16:32Z")

</div>

Can you tell how should i approach fine tuning and how to get started  
Like i am stuck on loading dataset only from past 2 days.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [February 5, 2025, 3:30pm UTC](https://discuss.huggingface.co/t/loading-nested-dataset-for-training/139133/6 "2025-02-05T15:30:58Z")

</div>

I have some knowledge of trouble patterns, but I’m still a beginner when it comes to generative AI. I’m not suited to teaching because I don’t know much about the essence of training.

I think the best way to get started is to try out some courses.

Another option is to ask questions on HF Discord, which has more people than the forum. There are a lot of people on there who are knowledgeable about NLP. If you’re used to using Discord, it should be easy, but if you’re not, I’ll explain.

> **[Introduction - Hugging Face NLP Course](https://huggingface.co/learn/nlp-course/chapter1/1)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.