# Tensorflow Huggingface Datasets Equivalent to PyTorch

**URL:** <https://discuss.huggingface.co/t/tensorflow-huggingface-datasets-equivalent-to-pytorch/19223>\
**Category:** 🤗Datasets\
**Created:** [June 16, 2022, 3:25am UTC](https://discuss.huggingface.co/t/tensorflow-huggingface-datasets-equivalent-to-pytorch/19223 "2022-06-16T03:25:11Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![conceptofmind](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/conceptofmind/32/55565_2.png) [@conceptofmind](https://discuss.huggingface.co/u/conceptofmind)\
**Post date:** [June 16, 2022, 3:25am UTC](https://discuss.huggingface.co/t/tensorflow-huggingface-datasets-equivalent-to-pytorch/19223/1 "2022-06-16T03:25:11Z")

</div>

Hello,

Does anyone know the Tensorflow equivalent to something like this in PyTorch:

```auto
# Convert the format of the tokenized train dataset to Tensors

train_with_pytorch = tokenized_train_dataset.with_format("torch")

# Convert the format of the tokenized validation dataset to Tensors

eval_with_pytorch = tokenized_eval_dataset.with_format("torch")

# Create the Iterable train dataloader. If the length of a tokenized input sequence is less than 2048 drop it.

train_dataloader = DataLoader(train_with_pytorch, shuffle = True, drop_last = True, collate_fn = default_data_collator, batch_size = 8)

# Create the Iterable validation dataloader. If the length of a tokenized input sequence is less than 2048 drop it.

eval_dataloader = DataLoader(eval_with_pytorch, shuffle = True, drop_last = True, collate_fn = default_data_collator, batch_size = 8)

```

I have been able to successfully build and integrate multiple Huggingface datasets and data loaders with PyTorch but I am currently having difficulty reproducing the same results in Tensorflow.

I greatly appreciate any help!

---

<div class="post-metadata">

**Author:** ![stevhliu](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/stevhliu/32/18373_2.png) [@stevhliu](https://discuss.huggingface.co/u/stevhliu)\
**Post date:** [June 16, 2022, 4:10pm UTC](https://discuss.huggingface.co/t/tensorflow-huggingface-datasets-equivalent-to-pytorch/19223/2 "2022-06-16T16:10:01Z")

</div>

Hi!

You can use the [`to_tf_dataset()`](https://huggingface.co/docs/datasets/use_with_tensorflow#using-totfdataset) function which just got a nice rework. If your elements are all the same length, then the built-in collator will handle it (otherwise you’ll need a custom collator). You can just do:

```auto
tf_train = dataset.to_tf_dataset(columns=["input"], 
                                 label_cols=["labels"],
                                 batch_size=8,
                                 shuffle=True)

```

Check out the [docs](https://huggingface.co/docs/datasets/use_with_tensorflow#using-totfdataset) here for more details about 🙂

---

<div class="post-metadata">

**Author:** ![conceptofmind](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/conceptofmind/32/55565_2.png) [@conceptofmind](https://discuss.huggingface.co/u/conceptofmind)\
**Post date:** [June 27, 2022, 3:36pm UTC](https://discuss.huggingface.co/t/tensorflow-huggingface-datasets-equivalent-to-pytorch/19223/3 "2022-06-27T15:36:13Z")

</div>

Hi @stevhliu ,

I appreciate the response.

I should have been more explicit. I had already reviewed the documentation but it was not clear to me whether everything that was available in tf.data.Dataset could be used interchangeably with Huggingface’s to\_tf\_dataset() or not.

For example to drop the last batch in TensorFlow:

```auto
dataset = tf.data.Dataset.range(8)
dataset = dataset.batch(3, drop_remainder=True)

```

Would this be the same thing in Huggingface Streaming Dataloaders?

```auto
tf_train = dataset.to_tf_dataset(columns=["input"], 
                                 label_cols=["labels"],
                                 batch_size=8,
                                 shuffle=True,
                                 drop_remainder = True
                                 )

```

Thank you,

Enrico