# Read CSV multi threading

**URL:** <https://discuss.huggingface.co/t/read-csv-multi-threading/8490>\
**Category:** 🤗Datasets\
**Created:** [July 19, 2021, 11:04am UTC](https://discuss.huggingface.co/t/read-csv-multi-threading/8490 "2021-07-19T11:04:49Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![imthanhlv](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/imthanhlv/32/2326_2.png) [@imthanhlv](https://discuss.huggingface.co/u/imthanhlv)\
**Post date:** [July 19, 2021, 11:04am UTC](https://discuss.huggingface.co/t/read-csv-multi-threading/8490/1 "2021-07-19T11:04:49Z")

</div>

According to [datasets document](https://huggingface.co/docs/datasets/loading_datasets.html#csv-files):

```auto
A few interesting features are provided out-of-the-box by the Apache Arrow backend:
    - **multi-threaded** or single-threaded reading
    - automatic decompression of input files (based on the filename extension, such as my_data.csv.gz)
    - fetching column names from the first row in the CSV file
    - column-wise type inference and conversion to one of null, int64, float64, timestamp[s], string or binary data
    - detecting various spellings of null values such as NaN or #N/A

```

How can I read, for example `train.csv.gz` in multi-threaded mode ?

I use `datasets.load_dataset("csv", data_files="./train.csv.gz")` but htop only show one cpu core is running

---

<div class="post-metadata">

**Author:** ![imthanhlv](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/imthanhlv/32/2326_2.png) [@imthanhlv](https://discuss.huggingface.co/u/imthanhlv)\
**Post date:** [July 19, 2021, 11:06am UTC](https://discuss.huggingface.co/t/read-csv-multi-threading/8490/2 "2021-07-19T11:06:32Z")

</div>

pinging @lhoestq , sorry I can’t find the multi-threaded part in document

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [July 20, 2021, 9:02am UTC](https://discuss.huggingface.co/t/read-csv-multi-threading/8490/3 "2021-07-20T09:02:33Z")

</div>

Hi ! Currently the CSV loader doesn’t leverage multithreading nor multiprocessing.

This is something we are working on, see issue [[load\_dataset] shard and parallelize the process · Issue #2650 · huggingface/datasets · GitHub](https://github.com/huggingface/datasets/issues/2650) which should allow to parallelize the conversion over multiple csv files.

However I’m not very familiar tools that allow to do multithreading on single files though. So if you have any idea/direction that could speed up the conversion of csv files to arrow, feel free to share it here 🙂

---

<div class="post-metadata">

**Author:** ![imthanhlv](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/imthanhlv/32/2326_2.png) [@imthanhlv](https://discuss.huggingface.co/u/imthanhlv)\
**Post date:** [July 20, 2021, 1:28pm UTC](https://discuss.huggingface.co/t/read-csv-multi-threading/8490/4 "2021-07-20T13:28:26Z")

</div>

Thank you for your answer, that issue is also what I’m wondering.

I have another question, my custom dataset [here](https://huggingface.co/datasets/imthanhlv/binhvq_dedup/blob/main/binhvq_dedup.py) I’m using [lm\_dataformat](https://github.com/leogao2/lm_dataformat/) to read `.zstd` files. I notice that after download and read the data the first time (for training tokenizer), it create 3 files:

- `xxx-train.arrow`
- `xxx-validation.arrow` (corresponding to my split name in loading script)
- `dataset_info.json`

I have two question:

- 1, Can I create a new dataset from these arrow and json files so that next time loading I don’t have to decompress again?
- 2, I’m running [run\_clm\_flax.py](https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/run_clm_flax.py) and passing my dataset name to this script, using TPU-VM. I set `$HF_HOME` to `~/hfcache`, which is 270 GB tmpfs that I mount from TPU-VM Ram. But I still faced out of memory. What is your suggestion to overcome this problem? My dataset is ~25G compress, two arrow files are around ~60GB. Can I delete these `xxx-train.arrow` and `xxx-validation.arrow` when creating `tmpxxx` and `cachexxx` files ?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [July 20, 2021, 1:46pm UTC](https://discuss.huggingface.co/t/read-csv-multi-threading/8490/5 "2021-07-20T13:46:02Z")

</div>

Once you have you processed data, you can save them using `save_to_disk` to save them in the directory of your choice. When this is done, you can completely delete your cache and reload your processed dataset with `load_from_disk` for example (see the [documentation](https://huggingface.co/docs/datasets/processing.html#saving-a-processed-dataset-on-disk-and-reload-it))

Also note that if you really need to, you can load any arrow file with `Dataset.from_file("path/to/any/arrow/file")`

---

<div class="post-metadata">

**Author:** ![imthanhlv](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/imthanhlv/32/2326_2.png) [@imthanhlv](https://discuss.huggingface.co/u/imthanhlv)\
**Post date:** [July 21, 2021, 11:53am UTC](https://discuss.huggingface.co/t/read-csv-multi-threading/8490/6 "2021-07-21T11:53:13Z")

</div>

Thank you @lhoestq

The arrow file reading worked beaultifully.  
In my case, I resized the tmpfs to 300GB and finally it fits my dataset ;\_\_;

I will modify the script so that next time it will not load the raw dataset again but load the preprocessed instead using your suggestion `load_from_disk`