# Exceeded maximum rows when load\_dataset for JSON

**URL:** <https://discuss.huggingface.co/t/exceeded-maximum-rows-when-load-dataset-for-json/35155>\
**Category:** 🤗Datasets\
**Created:** [March 31, 2023, 3:31am UTC](https://discuss.huggingface.co/t/exceeded-maximum-rows-when-load-dataset-for-json/35155 "2023-03-31T03:31:05Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![chjun](https://avatars.discourse-cdn.com/v4/letter/c/b3f665/32.png) [@chjun](https://discuss.huggingface.co/u/chjun)\
**Post date:** [March 31, 2023, 3:31am UTC](https://discuss.huggingface.co/t/exceeded-maximum-rows-when-load-dataset-for-json/35155/1 "2023-03-31T03:31:05Z")

</div>

I have a file in 150M size, but 100k rows inside, when i use load\_dataset() to load the data, i got the following error:

> ## Downloading and preparing dataset json/chjun–signal\_5 to /root/.cache/huggingface/datasets/chjun\_\_\_json/chjun–signal\_5-943deb579ded8031/0.0.0/fe5dd6ea2639a6df622901539cb550cf8797e5a6b2dd7af1cf934bed8e233e6e… Downloading data files: 100% 1/1 [00:00\<00:00, 54.64it/s] Extracting data files: 100% 1/1 [00:00\<00:00, 43.17it/s] ERROR:datasets.packaged\_modules.json.json:Failed to read file ‘/root/.cache/huggingface/datasets/downloads/d14d13fd8ba2262eff9553aeb2de18f0c0d8f661c6d500d0afd795dea9606792’ with error \<class ‘pyarrow.lib.ArrowInvalid’\>: Exceeded maximum rows
> 
> JSONDecodeError Traceback (most recent call last)  
> /usr/local/lib/python3.9/dist-packages/datasets/packaged\_modules/json/json.py in \_generate\_tables(self, files)  
> 132 with open(file, encoding=“utf-8”) as f:  
> → 133 dataset = json.load(f)  
> 134 except json.JSONDecodeError:
> 
> 14 frames  
> JSONDecodeError: Extra data: line 1 column 1478 (char 1477)
> 
> During handling of the above exception, another exception occurred:
> 
> ArrowInvalid Traceback (most recent call last)  
> ArrowInvalid: Exceeded maximum rows
> 
> The above exception was the direct cause of the following exception:
> 
> DatasetGenerationError Traceback (most recent call last)  
> /usr/local/lib/python3.9/dist-packages/datasets/builder.py in \_prepare\_split\_single(self, gen\_kwargs, fpath, file\_format, max\_shard\_size, job\_id)  
> 1891 if isinstance(e, SchemaInferenceError) and e. **context** is not None:  
> 1892 e = e. **context**  
> → 1893 raise DatasetGenerationError(“An error occurred while generating the dataset”) from e  
> 1894  
> 1895 yield job\_id, True, (total\_num\_examples, total\_num\_bytes, writer.\_features, num\_shards, shard\_lengths)
> 
> DatasetGenerationError: An error occurred while generating the dataset

Is this a bug? I’m trying to decrease the size for my training data.

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [March 31, 2023, 2:12pm UTC](https://discuss.huggingface.co/t/exceeded-maximum-rows-when-load-dataset-for-json/35155/2 "2023-03-31T14:12:21Z")

</div>

Hi! It seems like your JSON file is not formatted correctly - it should either contain JSON Lines or a single JSON array of objects

---

<div class="post-metadata">

**Author:** ![chjun](https://avatars.discourse-cdn.com/v4/letter/c/b3f665/32.png) [@chjun](https://discuss.huggingface.co/u/chjun)\
**Post date:** [April 6, 2023, 1:26am UTC](https://discuss.huggingface.co/t/exceeded-maximum-rows-when-load-dataset-for-json/35155/3 "2023-04-06T01:26:44Z")

</div>

But if I reduced the rows of json file to 91000, it worked well. I used FOR loop to feed data to my JSON file, there should be no difference for rows.

---

<div class="post-metadata">

**Author:** ![chjun](https://avatars.discourse-cdn.com/v4/letter/c/b3f665/32.png) [@chjun](https://discuss.huggingface.co/u/chjun)\
**Post date:** [April 6, 2023, 9:52am UTC](https://discuss.huggingface.co/t/exceeded-maximum-rows-when-load-dataset-for-json/35155/4 "2023-04-06T09:52:48Z")

</div>

btw, this is the my code to write out the json file ready loading to HF.

```auto
    list_json = []
    for i in range(100000):
        merged_json = {"data_in": padded_flip_psf[1], 'data_out': padded_flip_psf[1]}
        list_json.append(merged_json)

    print(len(list_json))
    print(list_json[0])
    print(len(list_json[0]["data_in"]))
    print(len(list_json[0]["data_out"]))
    out_file = open('test.json', 'w')
    for item in list_json:
        json.dump(item, out_file)

    out_file.close()

```

---

<div class="post-metadata">

**Author:** ![chjun](https://avatars.discourse-cdn.com/v4/letter/c/b3f665/32.png) [@chjun](https://discuss.huggingface.co/u/chjun)\
**Post date:** [April 6, 2023, 11:08am UTC](https://discuss.huggingface.co/t/exceeded-maximum-rows-when-load-dataset-for-json/35155/5 "2023-04-06T11:08:21Z")

</div>

updates: If I write json file using

> json.dump(list\_json, out\_file)

directly instead of

> for item in list\_json:  
> json.dump(item, out\_file)

, it works when load\_dataset in HF.