# Loading HF datasets with variable size array using pyarrow with the appropriate schema

**URL:** <https://discuss.huggingface.co/t/loading-hf-datasets-with-variable-size-array-using-pyarrow-with-the-appropriate-schema/117094>\
**Category:** 🤗Datasets\
**Created:** [November 11, 2024, 7:32pm UTC](https://discuss.huggingface.co/t/loading-hf-datasets-with-variable-size-array-using-pyarrow-with-the-appropriate-schema/117094 "2024-11-11T19:32:40Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Raz0rr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/raz0rr/32/33373_2.png) [@Raz0rr](https://discuss.huggingface.co/u/Raz0rr)\
**Post date:** [November 11, 2024, 7:32pm UTC](https://discuss.huggingface.co/t/loading-hf-datasets-with-variable-size-array-using-pyarrow-with-the-appropriate-schema/117094/1 "2024-11-11T19:32:40Z")

</div>

I want to load my HF dataset using `pyarrow`. What should I give for schema in the below code:

```auto
import pyarrow as pa
import pyarrow.dataset as ds

data_path = "/some/path/to/huggingface/dataset"
schema = pa.schema([])
dataset = ds.dataset(data_path, format="arrow", schema=schema)

```

from the huggingface dataset directory “dataset\_info.json” file, I have the following:

```auto
  "features": {
    "index": {
      "dtype": "int32",
      "_type": "Value"
    },
    "example": {
      "shape": [
        null,
        2
      ],
      "dtype": "float32",
      "_type": "Array2D"
    },
    "label": {
      "shape": [
        null,
        3
      ],
      "dtype": "float32",
      "_type": "Array2D"
    },
    "coords_label": {
      "shape": [
        null,
        3
      ],
      "dtype": "float32",
      "_type": "Array2D"
    },
    "coords_num": {
      "feature": {
        "dtype": "int32",
        "_type": "Value"
      },
      "_type": "Sequence"
    },
    "has_noise": {
      "dtype": "bool",
      "_type": "Value"
    }
  },

```