# Huggingface-cli to load\_dataset

**URL:** <https://discuss.huggingface.co/t/huggingface-cli-to-load-dataset/71158>\
**Category:** 🤗Datasets\
**Created:** [January 30, 2024, 8:42am UTC](https://discuss.huggingface.co/t/huggingface-cli-to-load-dataset/71158 "2024-01-30T08:42:24Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![pseudotensor](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/pseudotensor/32/23448_2.png) [@pseudotensor](https://discuss.huggingface.co/u/pseudotensor)\
**Post date:** [January 30, 2024, 8:42am UTC](https://discuss.huggingface.co/t/huggingface-cli-to-load-dataset/71158/1 "2024-01-30T08:42:24Z")

</div>

I can’t understand how to go from huggingface-cli or git clone to load\_dataset() from that cached location. Despite the 3 methods all here: [Downloading datasets](https://huggingface.co/docs/hub/datasets-downloading) they seem incompatible. The only poor suggestion here was to never use the other 2 methods: [python - How to load a huggingface dataset from local path? - Stack Overflow](https://stackoverflow.com/questions/77020278/how-to-load-a-huggingface-dataset-from-local-path)

So idea is I want to use hf\_transfer since that is about 10x faster for downloads than load\_dataset. So I do:

```bash
pip install hf_transfer
export HF_HUB_ENABLE_HF_TRANSFER=1
huggingface-cli download --repo-type dataset vietgpt/the_pile_openwebtext2

```

I get much faster results than load\_dataset:

```python
num_proc_load_dataset=32
dataset = load_dataset('vietgpt/the_pile_openwebtext2', num_proc=num_proc_load_dataset, trust_remote_code=True)

```

But, then I spent about 2 hours trying to figure out how to load the dang result using load\_dataset by playing with cache\_dir, data\_dir, data\_files, dataset name, etc. I tried linking, copying, all sorts of files everywhere and no luck.

I can see all the files in:

```auto
/home/fsuser/.cache/huggingface/hub/datasets--vietgpt--the_pile_openwebtext2/snapshots/1de27c660aefd991700e5c13865a041591394834/data

```

as same ones in the original repo as if I would do `git clone` to that same hash folder.

But when using load\_dataset, seems to go to totally different location, and on separate computer where I did that, all different file names. Very confusing.

I was able to specify some hacked setup where it stopped complaining about the files, but then asked for 1 example of a feature or something related, which I didn’t have and shouldn’t need.

Generally, suppose one uses huggingface-cli download or git clone, how does one go directly from that do load\_dataset? It must be possible since the original files are exactly like that in the repo that is the origin.

Thanks!  
Jon

---

<div class="post-metadata">

**Author:** ![Hung190100](https://avatars.discourse-cdn.com/v4/letter/h/ed8c4c/32.png) [@Hung190100](https://discuss.huggingface.co/u/Hung190100)\
**Post date:** [January 30, 2024, 8:48am UTC](https://discuss.huggingface.co/t/huggingface-cli-to-load-dataset/71158/2 "2024-01-30T08:48:32Z")

</div>

You need login Hugging Face ( [Quickstart (huggingface.co)](https://huggingface.co/docs/huggingface_hub/quick-start)), then load data set  
dataset = load\_dataset(dataset\_name)

---

<div class="post-metadata">

**Author:** ![pseudotensor](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/pseudotensor/32/23448_2.png) [@pseudotensor](https://discuss.huggingface.co/u/pseudotensor)\
**Post date:** [January 30, 2024, 8:53am UTC](https://discuss.huggingface.co/t/huggingface-cli-to-load-dataset/71158/3 "2024-01-30T08:53:48Z")

</div>

I don’t understand why logging in matters, but I tried it and it still re-downloads the data instead of using the huggingface-cli download result I already downloaded

```auto
Python 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from huggingface_hub import login
>>> login()

    _| _| _| _| _|_|_| _|_|_| _|_|_| _| _| _|_|_| _|_|_|_| _|_| _|_|_| _|_|_|_|
    _| _| _| _| _| _| _| _|_| _| _| _| _| _| _| _|
    _|_|_|_| _| _| _| _|_| _| _|_| _| _| _| _| _| _|_| _|_|_| _|_|_|_| _| _|_|_|
    _| _| _| _| _| _| _| _| _| _| _|_| _| _| _| _| _| _| _|
    _| _| _|_| _|_|_| _|_|_| _|_|_| _| _| _|_|_| _| _| _| _|_|_| _|_|_|_|

    A token is already saved on your machine. Run `huggingface-cli whoami` to get more information or `huggingface-cli logout` if you want to log out.
    Setting a new token will erase the existing one.
    To login, `huggingface_hub` requires a token generated from https://huggingface.co/settings/tokens .
Token: 
Add token as git credential? (Y/n) y
Token is valid (permission: read).
Your token has been saved in your configured git credential helpers (store).
Your token has been saved to /home/fsuser/.cache/huggingface/token
Login successful
>>> num_proc = 32
>>> num_proc_load_dataset = num_proc
>>> from datasets import load_dataset
>>> dataset = load_dataset('vietgpt/the_pile_openwebtext2', num_proc=num_proc_load_dataset, trust_remote_code=True)
Resolving data files: 100%|???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 35/35 [00:02<00:00, 16.72it/s]
Downloading data: 3%|???? | 29.4M/954M [00:01<00:54, 17.0MB/s]
Downloading data: 4%|?????                                                                                                                      

```

---

<div class="post-metadata">

**Author:** ![Hung190100](https://avatars.discourse-cdn.com/v4/letter/h/ed8c4c/32.png) [@Hung190100](https://discuss.huggingface.co/u/Hung190100)\
**Post date:** [January 30, 2024, 9:05am UTC](https://discuss.huggingface.co/t/huggingface-cli-to-load-dataset/71158/4 "2024-01-30T09:05:24Z")

</div>

It seems to be stored in cache, so you can save it to a folder you can see easily

---

<div class="post-metadata">

**Author:** ![GuoDanding](https://avatars.discourse-cdn.com/v4/letter/g/74df32/32.png) [@GuoDanding](https://discuss.huggingface.co/u/GuoDanding)\
**Post date:** [March 6, 2024, 2:27pm UTC](https://discuss.huggingface.co/t/huggingface-cli-to-load-dataset/71158/5 "2024-03-06T14:27:11Z")

</div>

I am also confused with this! 😭 I try to download HuggingFaceH4/ultrachat\_200k. When I use **cli** , the raw dataset _.parquet file_ (organized as [HuggingFaceH4/ultrachat\_200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k/tree/main)) is downloaded and saved in **~/.cache/huggingface/hub**. But when I use **load\_dataset** , it is saved in **~/.cache/huggingface/datasets** and seems to be preprocessed to _.arrow file_. I just follow the step to load my HuggingFaceH4/ultrachat\_200k dataset downloaded by cli:

1. Use the cli download option --local-dir to save the file. (which can be skipped, just load\_datasets from ~/.cache/huggingface/hub/your datasets is ok):

```auto
huggingface-cli download --resume-download --repo-type dataset HuggingFaceH4/ultrachat_200k --local-dir ./modeling/datasets/HuggingFaceH4/ultrachat_200k

```

1. See the readme in the downloaded file to know how to organize dataset:

> configs:
> 
> - config\_name: default
> - data\_files:
> - split: train\_sft  
> path: data/train\_sft-\*
> - split: test\_sft  
> path: data/test\_sft-\*
> - split: train\_gen  
> path: data/train\_gen-\*
> - split: test\_gen  
> path: data/test\_gen-\*
> 
> dataset\_info:
> 
> - features:
> - name: prompt  
> dtype: string
> - name: prompt\_id  
> dtype: string
> - name: messages  
> list:
> - name: content  
> dtype: string
> - name: role  
> dtype: string

1. Process it!

```auto
from datasets import load_dataset
from datasets import load_dataset,Features,Value

context_feat = Features({
    'prompt': Value(dtype='string', id=None),
    'prompt_id': Value(dtype='string', id=None),
    'messages': [{'content': Value(dtype='string', id=None), 'role': Value(dtype='string', id=None)}]
})

url = "./modeling/datasets/HuggingFaceH4/ultrachat_200k/data/"
>
data_files = {'train_sft': [url + "train_sft-00000-of-00003-a3ecf92756993583.parquet", url + "train_sft-00001-of-00003-0a1804bcb6ae68c6.parquet", url + "train_sft-00002-of-00003-ee46ed25cfae92c6.parquet"],
              'test_sft': url + "test_sft-00000-of-00001-f7dfac4afe5b93f4.parquet",
              'train_gen': [url + "train_gen-00000-of-00003-a6c9fb894be3e50b.parquet", url + "train_gen-00001-of-00003-d6a0402e417f35ca.parquet", url + "train_gen-00002-of-00003-c0db75b92a2f48fd.parquet"],
              'test_gen': url + "test_gen-00000-of-00001-3d4cd8309148a71f.parquet"}

```

1. Load the dataset! 🤩

```auto
raw_datasets = load_dataset("parquet", data_files=data_files, features=context_feat)

```

Maybe some official document of load\_dataset and how to process different type of data file will be helpful.  
Hope that help! 🥰

---

<div class="post-metadata">

**Author:** ![Payoto](https://avatars.discourse-cdn.com/v4/letter/p/5f8ce5/32.png) [@Payoto](https://discuss.huggingface.co/u/Payoto)\
**Post date:** [July 23, 2025, 1:15pm UTC](https://discuss.huggingface.co/t/huggingface-cli-to-load-dataset/71158/6 "2025-07-23T13:15:52Z")

</div>

I faced the same issue - while @GuoDanding 's solution works (thanks 🙂 ) in later versions of `load_dataset` (I’m using `datasets==3.6`) you can just call it with the path to the snapshot you have downloaded.

So simply do:

```auto
dataset = load_dataset("/home/fsuser/.cache/huggingface/hub/datasets--vietgpt--the_pile_openwebtext2/snapshots/1de27c660aefd991700e5c13865a041591394834/")  

```

Now for a big dataset this will try to prepare and load the entire dataset into memory so you will want to pass `streaming=True`:

```auto
ds = load_dataset("/path/where/you/downloaded/dataset/<snapshot-id>/", streaming=True)

```

Since you’re streaming from a local folder you don’t have any of the remote access issues you might normally expect from “streaming=True”.

For big datasets `streaming=True` might be a good idea for @GuoDanding answer as well - when I used it for FineWeb-Edu, it spent 20h preparing all the parquet files into arrow files.

Hope this saves someone some time!