# Using datasets to open jsonl

**URL:** <https://discuss.huggingface.co/t/using-datasets-to-open-jsonl/161037>\
**Category:** 🤗Datasets\
**Created:** [June 28, 2025, 6:33pm UTC](https://discuss.huggingface.co/t/using-datasets-to-open-jsonl/161037 "2025-06-28T18:33:58Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![bluebingo](https://avatars.discourse-cdn.com/v4/letter/b/f4b2a3/32.png) [@bluebingo](https://discuss.huggingface.co/u/bluebingo)\
**Post date:** [June 28, 2025, 6:33pm UTC](https://discuss.huggingface.co/t/using-datasets-to-open-jsonl/161037/1 "2025-06-28T18:33:58Z")

</div>

## Problem When Using Datasets to Open JSONL

I am trying to open a JSONL format file using the `datasets` library. Here is my code:

```auto
from datasets import load_dataset

path = "./testdata.jsonl"
dataset = load_dataset('json', data_files=path, split='train')

```

The contents of testdata.jsonl are organized as follows (just for testing):

```auto
{"src":"hello","term":{"a":"aa"}}
{"src":"hi","term":{"b":"bb"}}

```

When I use the code above to load the dataset and attempt to print the second item, like this:

```auto
print(dataset[1])

```

I get the following output:

```auto
{'src': 'hi', 'term': {'a': None, 'b': 'bb'}}

```

Instead of the expected output:

```auto
{'src': 'hi', 'term': {'b': 'bb'}}

```

How can I obtain the second format of the dataset? Is it possible that I simply forgot to include a parameter?

---

<div class="post-metadata">

**Author:** ![Pimpcat-AU](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/pimpcat-au/32/48989_2.png) [@Pimpcat-AU](https://discuss.huggingface.co/u/Pimpcat-AU)\
**Post date:** [June 28, 2025, 10:47pm UTC](https://discuss.huggingface.co/t/using-datasets-to-open-jsonl/161037/2 "2025-06-28T22:47:45Z")

</div>

Ensure the JSONL file is correctly formatted:  
Each line in the file should be a valid JSON object with no extra commas or brackets. For example, the file should look like this:

{“src”:“hello”,“term”:{“a”:“aa”}}  
{“src”:“hi”,“term”:{“b”:“bb”}}

After fixing the JSONL format, use the following code to load the dataset properly:

from datasets import load\_dataset

path = “./testdata.jsonl”  
dataset = load\_dataset(‘json’, data\_files=path, split=‘train’)

print(dataset[1]) # This should now work correctly

After these changes, the second entry should now print the correct data:

{‘src’: ‘hi’, ‘term’: {‘b’: ‘bb’}}

Also, ensure there are no extra spaces or line breaks in the dataset if it’s large. Each line should be a valid JSON object.

**Response generated by Triskel Data Deterministic Ai**

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [June 28, 2025, 10:55pm UTC](https://discuss.huggingface.co/t/using-datasets-to-open-jsonl/161037/3 "2025-06-28T22:55:56Z")

</div>

Another option, albeit a bit rough, is this:

```py
from datasets import load_dataset

def process(example):
    example["term"] = str({k: v for k, v in example["term"].items() if v is not None})
    return example

path = "./testdata.jsonl"
dataset = load_dataset('json', data_files=path, split='train')

print(dataset[1]) # {'src': 'hi', 'term': {'a': None, 'b': 'bb'}}

dataset = dataset.map(process)

print(dataset[1]) # {'src': 'hi', 'term': "{'b': 'bb'}"}

```

---

<div class="post-metadata">

**Author:** ![bluebingo](https://avatars.discourse-cdn.com/v4/letter/b/f4b2a3/32.png) [@bluebingo](https://discuss.huggingface.co/u/bluebingo)\
**Post date:** [June 29, 2025, 6:35pm UTC](https://discuss.huggingface.co/t/using-datasets-to-open-jsonl/161037/4 "2025-06-29T18:35:49Z")

</div>

Thank you for your advice. I appreciate your efforts, but unfortunately, it hasn’t been effective for me.

---

<div class="post-metadata">

**Author:** ![bluebingo](https://avatars.discourse-cdn.com/v4/letter/b/f4b2a3/32.png) [@bluebingo](https://discuss.huggingface.co/u/bluebingo)\
**Post date:** [June 29, 2025, 6:38pm UTC](https://discuss.huggingface.co/t/using-datasets-to-open-jsonl/161037/5 "2025-06-29T18:38:28Z")

</div>

Thank you for your advice; it was really helpful in solving the problem! However, I find it a bit cumbersome to map the datasets each time I want to open a JSONL file with JSON elements. I wonder if there might be a more permanent solution to address this issue.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [June 30, 2025, 1:50am UTC](https://discuss.huggingface.co/t/using-datasets-to-open-jsonl/161037/8 "2025-06-30T01:50:35Z")

</div>

> I find it a bit cumbersome to map the datasets each time I want to open a JSONL file with JSON elements. I wonder if there might be a more permanent solution to address this issue.

That’s true. There may be a more concise method (including potential ones). I’ll mention it to the library developer. @lhoestq

---

<div class="post-metadata">

**Author:** ![bluebingo](https://avatars.discourse-cdn.com/v4/letter/b/f4b2a3/32.png) [@bluebingo](https://discuss.huggingface.co/u/bluebingo)\
**Post date:** [June 30, 2025, 8:03am UTC](https://discuss.huggingface.co/t/using-datasets-to-open-jsonl/161037/9 "2025-06-30T08:03:11Z")

</div>

Thank you! I look forward to any official solutions that the developer might provide.

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [July 1, 2025, 12:27pm UTC](https://discuss.huggingface.co/t/using-datasets-to-open-jsonl/161037/10 "2025-07-01T12:27:46Z")

</div>

Hi ! This behavior is expected since `datasets` uses Arrow which has fixed types. This means each sample should have the same subfields with the same types. Missing subfields are filled with None.

You can restructure your data differently to fit this paradigm: either converting nested data as one string, or use one list for keys and one list for values.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [July 1, 2025, 8:18pm UTC](https://discuss.huggingface.co/t/using-datasets-to-open-jsonl/161037/11 "2025-07-01T20:18:09Z")

</div>

Thank you, lhonestq!

---

<div class="post-metadata">

**Author:** ![bluebingo](https://avatars.discourse-cdn.com/v4/letter/b/f4b2a3/32.png) [@bluebingo](https://discuss.huggingface.co/u/bluebingo)\
**Post date:** [July 2, 2025, 1:16am UTC](https://discuss.huggingface.co/t/using-datasets-to-open-jsonl/161037/12 "2025-07-02T01:16:11Z")

</div>

Thank you, lhonestq!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [July 2, 2025, 1:17pm UTC](https://discuss.huggingface.co/t/using-datasets-to-open-jsonl/161037/13 "2025-07-02T13:17:03Z")

</div>

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.