# Transform a tf.data.dataset to a datasets.dataset? **URL:** \ **Category:** Beginners\ **Created:** [March 12, 2022, 2:57am UTC](https://discuss.huggingface.co/t/transform-a-tf-data-dataset-to-a-datasets-dataset/15626 "2022-03-12T02:57:37Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![HeNan](https://avatars.discourse-cdn.com/v4/letter/h/f6c823/32.png) [@HeNan](https://discuss.huggingface.co/u/HeNan)\ **Post date:** [March 12, 2022, 2:57am UTC](https://discuss.huggingface.co/t/transform-a-tf-data-dataset-to-a-datasets-dataset/15626/1 "2022-03-12T02:57:37Z")
I want to ask if there exists a method that can help me transform a tf.data.dataset to a datasets.dataset. ---
**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\ **Post date:** [March 31, 2022, 11:45am UTC](https://discuss.huggingface.co/t/transform-a-tf-data-dataset-to-a-datasets-dataset/15626/2 "2022-03-31T11:45:24Z")
Hi! We don’t have a dedicated method for converting `tf.data.Dataset` to `datasets.Dataset`, only the other way around. So you have two options: - If the dataset is small and fits in RAM, you can convert the TF dataset to a Python dictionary and then call `datasets.Dataset.from_dict` on it. Another approach that might be easier is to install Tensorflow Datasets and convert the TF dataset to a Pandas DataFrame, on which you can call `datasets.Dataset.from_pandas`: `datasets.Dataset.from_pandas(tfds.as_dataframe(tf_dataset))` - If the dataset doesn’t fit in RAM, you can create a simple loading script in which you iterate over the dataset and yield its examples in `_generate_examples`. You can find more info here: [Create a dataset loading script](https://huggingface.co/docs/datasets/dataset_script). ---
**Author:** ![eppane](https://avatars.discourse-cdn.com/v4/letter/e/43a26b/32.png) [@eppane](https://discuss.huggingface.co/u/eppane)\ **Post date:** [September 28, 2022, 6:22am UTC](https://discuss.huggingface.co/t/transform-a-tf-data-dataset-to-a-datasets-dataset/15626/3 "2022-09-28T06:22:13Z")
Hello @mariosasko, any updates on this? I have my own dataset with images and masks, and I am trying to make it work with the TFSegformerForSemanticSegmentation. I have created a tf.data.Dataset that generates the images and masks from files (filepaths), but when I try to train the model, I run into this error: ```auto Node: 'tf_segformer_for_semantic_segmentation_2/segformer/transpose' 2 root error(s) found. (0) INVALID_ARGUMENT: transpose expects a vector of size 5. But input(1) is a vector of size 4 [[{{node tf_segformer_for_semantic_segmentation_2/segformer/transpose}}]] [[tf_segformer_for_semantic_segmentation_2/sparse_categorical_crossentropy/cond/then/_0/tf_segformer_for_semantic_segmentation_2/sparse_categorical_crossentropy/cond/cond/then/_44/tf_segformer_for_semantic_segmentation_2/sparse_categorical_crossentropy/cond/cond/remove_squeezable_dimensions/cond/pivot_t/_102/_2229]] (1) INVALID_ARGUMENT: transpose expects a vector of size 5. But input(1) is a vector of size 4 [[{{node tf_segformer_for_semantic_segmentation_2/segformer/transpose}}]] 0 successful operations. 0 derived errors ignored. [Op:__inference_train_function_152606] ``` Where is that fifth dimension coming from? I have something like this: ```auto list_ds = tf.data.Dataset.from_tensor_slices((image_files, mask_files)) ds = list_ds.map(lambda x, y: tf.py_function(parse_function, [x, y], [tf.float32, tf.int32]), num_parallel_calls=1) ``` and in the end of the `parse_function`, I use: ```auto encoded_inputs = feature_extractor(image, mask, return_tensors="tf") return encoded_inputs['pixel_values'], encoded_inputs['labels'] ``` So now I am wondering whether I should convert the dataset into datasets.Dataset for better compatibility. I noticed some tips in [here](https://github.com/huggingface/datasets/issues/4417), would it be better to write the data into an .arrow-file with the ArrowWriter? And then load it with Dataset.from\_file()? Or can I make the tf.data.Dataset work? I can make a dummy forward pass with the model (49 classes) without errors: ```auto outputs = model(batch[0], batch[1]) print(outputs.loss, outputs.logits.shape) > tf.Tensor([nan], shape=(1,), dtype=float32) (1, 49, 128, 128) ``` Although the loss is `nan`. Many thanks! eppane ---
**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\ **Post date:** [September 30, 2022, 11:17am UTC](https://discuss.huggingface.co/t/transform-a-tf-data-dataset-to-a-datasets-dataset/15626/4 "2022-09-30T11:17:51Z")
Perhaps @Rocketknight1 can help