# Dataset set\_format **URL:** \ **Category:** 🤗Datasets\ **Created:** [November 10, 2020, 11:43am UTC](https://discuss.huggingface.co/t/dataset-set-format/1961 "2020-11-10T11:43:31Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](https://discuss.huggingface.co/u/vblagoje)\ **Post date:** [November 10, 2020, 11:43am UTC](https://discuss.huggingface.co/t/dataset-set-format/1961/1 "2020-11-10T11:43:31Z")
Hello everyone, Datasets provide this great feature of formatting datasets using set\_format and then choosing the desired format (numpy, torch etc). The encoded dataset I prepared has columns/features of various data types (int32, int8 etc) but HF models require all features to be dtype torch.long/int64. Is there a simple trick to convert all features to torch.long tensors when selecting torch format? I understand that I could have prepared the dataset with int64 type but that significantly increases the dataset file size footprint. Thanks, Vladimir ---
**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](https://discuss.huggingface.co/u/vblagoje)\ **Post date:** [November 10, 2020, 1:00pm UTC](https://discuss.huggingface.co/t/dataset-set-format/1961/2 "2020-11-10T13:00:36Z")
Nevermind, I found a way. RTFM. `format = {'type': 'torch', 'format_kwargs' :{'dtype': torch.long}}` `dataset.set_format(**format)` ---
**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\ **Post date:** [November 10, 2020, 1:31pm UTC](https://discuss.huggingface.co/t/dataset-set-format/1961/3 "2020-11-10T13:31:21Z")
Here is [a related issue](https://github.com/huggingface/datasets/issues/625#issuecomment-696127826) that may be helpful. ---
**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](https://discuss.huggingface.co/u/vblagoje)\ **Post date:** [November 10, 2020, 3:28pm UTC](https://discuss.huggingface.co/t/dataset-set-format/1961/4 "2020-11-10T15:28:16Z")
@BramVanroy it really was. You got me at “After a weekend of debugging”. I can totally relate 🙂 I have pre-processed my model input data and saved it as python lists. When loading the training data, I am setting format to torch. From your discussion with @lhoestq this conversion is as fast because loading converts numpy arrays to torch tensors, right? Do I need to save my data as numpy arrays as well? Doesn’t seem to be needed. What is the best format to save your training data so it can be loaded as fast as possible into torch tensors? ---
**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\ **Post date:** [November 10, 2020, 3:47pm UTC](https://discuss.huggingface.co/t/dataset-set-format/1961/5 "2020-11-10T15:47:37Z")
It doesn’t really matter: your data is converted into a list to be compatible with the Arrow data format. Then, when you call or request items from the dataset, the data will be cast from the underlying Arrow format (list-like) to numpy, and then to the returned format that you request (with `set_format`). My guess is that this last step also includes the precision information (e.g. float 32 vs float64), although ideally the precision should already be correct in the numpy conversion to prevent data loss. @lhoestq Please correct me if I’m wrong. ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [November 10, 2020, 3:53pm UTC](https://discuss.huggingface.co/t/dataset-set-format/1961/6 "2020-11-10T15:53:01Z")
Hi ! You’re right @BramVanroy Under the hood all the data are in Arrow format. Arrow has a good interoperability with numpy that allows to cast arrow objets to numpy fast and with zero-copy. Then from numpy the conversion to pytorch is also very fast. To summarize this is what it looks like when you load a pytorch tensor from a dataset: memory-mapped arrow file -\> load a sample in arrow format -\> convert to numpy -\> convert to pytorch (using the set\_format args) ---
**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\ **Post date:** [November 10, 2020, 3:55pm UTC](https://discuss.huggingface.co/t/dataset-set-format/1961/7 "2020-11-10T15:55:31Z")
Thanks for the clarification @lhoestq! Would it be possible to move the precision to the numpy cast, or is it ensured that the numpy call is always the highest possible precision? I’m asking this because, imagine that you want to save data as double precision, and you set the `set_format` to double precision torch tensors, but then your data is first cast to single precision and only when casting to torch it is in double. You’d lose a lot of data, I think? ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [November 10, 2020, 4:06pm UTC](https://discuss.huggingface.co/t/dataset-set-format/1961/8 "2020-11-10T16:06:42Z")
The numpy cast from arrow will reuse the same precision of the arrow data. It doesn’t reduce the precision 🙂 The precision of the arrow data is defined by the features field of the dataset. For example to have double precision you need ```python features = Features({"col1": Value("double")}) ``` ---
**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\ **Post date:** [November 10, 2020, 4:32pm UTC](https://discuss.huggingface.co/t/dataset-set-format/1961/9 "2020-11-10T16:32:30Z")
That is not entirely correct I think. If you remember, I [had an issue](https://github.com/huggingface/datasets/issues/625) where a torch.float32 would unexpectedly end up as a torch.float64 if you do not manually specify the precision. So the input precision is _not_ automatically used in the output. It went like this: torch.float32 -\> list -\> float64 (numpy) -\> torch.float64. ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [November 10, 2020, 4:52pm UTC](https://discuss.huggingface.co/t/dataset-set-format/1961/10 "2020-11-10T16:52:42Z")
Well the precision that is used is the one of the arrow format, that is read from the file on disk. You’re right that when writing an arrow file, on the other hand, there’s currently an issue. You’re having this issue because when you write floats in a dataset using `map` for example then it will try to infer the precision and take the highest one if it’s not manually specified which one to use (here float64 even though you’re providing float32 tensors using `map`). ---
**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\ **Post date:** [November 10, 2020, 5:52pm UTC](https://discuss.huggingface.co/t/dataset-set-format/1961/11 "2020-11-10T17:52:40Z")
Well, I guess that if it picks the highest possible precision available then no information is lost ever, so that’s okay. Thanks for the reply! ---
**Author:** ![dhruvgrammarly](https://avatars.discourse-cdn.com/v4/letter/d/ccd318/32.png) [@dhruvgrammarly](https://discuss.huggingface.co/u/dhruvgrammarly)\ **Post date:** [November 24, 2024, 4:22am UTC](https://discuss.huggingface.co/t/dataset-set-format/1961/12 "2024-11-24T04:22:13Z")
> [@lhoestq](#): > > Under the hood all the data are in Arrow format. Arrow has a good interoperability with numpy that allows to cast arrow objets to numpy fast and with zero-copy. Then from numpy the conversion to pytorch is also very fast. This seems inconsistent with what the documentation at ```auto to_numpy(*self*, *zero_copy_only=False* ) Return a NumPy copy of this array (experimental). Parameters: **zero_copy_only** [bool](), default `False` Introduced for signature consistence with pyarrow.Array.to_numpy. This must be False here since NumPy arrays’ buffer must be contiguous. ``` This suggests that it’s making a copy of the data and not doing a zero copy to numpy arrays. I’m also running into a problem when loading it as numpy array or python list seems equally slow. Maybe I’m doing something horribly wrong. [Create batch from list of ids in the dataset is very slow - #4 by dhruvgrammarly](https://discuss.huggingface.co/t/create-batch-from-list-of-ids-in-the-dataset-is-very-slow/15997/4)