# Map() with a tokenizer does not return pytorch tensors

**URL:** <https://discuss.huggingface.co/t/map-with-a-tokenizer-does-not-return-pytorch-tensors/51723>\
**Category:** Beginners\
**Created:** [August 22, 2023, 7:07pm UTC](https://discuss.huggingface.co/t/map-with-a-tokenizer-does-not-return-pytorch-tensors/51723 "2023-08-22T19:07:26Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![alonmiz](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/alonmiz/32/18708_2.png) [@alonmiz](https://discuss.huggingface.co/u/alonmiz)\
**Post date:** [August 22, 2023, 7:07pm UTC](https://discuss.huggingface.co/t/map-with-a-tokenizer-does-not-return-pytorch-tensors/51723/1 "2023-08-22T19:07:26Z")

</div>

I’m trying to tokenize a dataset and move all the torch tensors to gpu, but somehow this doesn’t work:

> import datasets
> 
> cola = datasets.load\_dataset(‘linxinyuan/cola’)  
> cola\_tokenized = cola.map(lambda examples: tokenizer(examples[‘text’], return\_tensors=“pt”, padding=True, truncation=True).to(‘cuda’), batched=True, batch\_size=16)
> 
> print(cola\_tokenized[‘train’][0][‘input\_ids’])

The output is a python list and not even a tensor:

> [0, 2522, …, 2]

I tried calling cola.set\_format(“torch”) before map() (still doesn’t explain why the previous run didn’t return tensors) but then I get pytorch tensors that are placed on the cpu (even though I call to(‘cuda’)).

Any idea why this happens?

---

<div class="post-metadata">

**Author:** ![Luan77777](https://avatars.discourse-cdn.com/v4/letter/l/ecc23a/32.png) [@Luan77777](https://discuss.huggingface.co/u/Luan77777)\
**Post date:** [August 22, 2023, 7:20pm UTC](https://discuss.huggingface.co/t/map-with-a-tokenizer-does-not-return-pytorch-tensors/51723/2 "2023-08-22T19:20:38Z")

</div>

I have the same issue… Sometimes I get tensors and then sometimes only a list. I notices that when your data ist larger then your batching size your doent get a tensor and when your data is smaller you get some tensors… However when you try to print the shape or dtype of your data before the .map function there should be tensors only. When i tried that there where tensors right before i returned the element zu the map function. Maybe you couald try that! But still this function remains random and does what it wants…

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [August 23, 2023, 1:09pm UTC](https://discuss.huggingface.co/t/map-with-a-tokenizer-does-not-return-pytorch-tensors/51723/3 "2023-08-23T13:09:12Z")

</div>

You can get Torch Tensors by changing the format Torch with `.set_format("torch")` (`.set_format("torch", device="cuda")` to put them on GPU)

`map` caches results into an Arrow file, and Arrow doesn’t understand Torch tensors, so this would require storing additional metadata for each example to recover the initial type for decoding later. We decided not to keep this metadata for simplicity and instead support changing the default (Arrow → Python) decoding with `set_format`/`set_transform` ([docs](https://huggingface.co/docs/datasets/process#format)).