# Space Connection Error - Dataset Map

**URL:** <https://discuss.huggingface.co/t/space-connection-error-dataset-map/133574>\
**Category:** Spaces\
**Created:** [December 30, 2024, 8:37am UTC](https://discuss.huggingface.co/t/space-connection-error-dataset-map/133574 "2024-12-30T08:37:09Z")\
**Posts on this page:** 1\
**Showing post:** 7

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [December 30, 2024, 10:23am UTC](https://discuss.huggingface.co/t/space-connection-error-dataset-map/133574/7 "2024-12-30T10:23:37Z")

</div>

> How can i do to load “tokenized dataset” from the dataset repos?

I’ve found a way to save it. However, I think it would be more reliable to [split the data set into several parts](https://stackoverflow.com/questions/76001128/splitting-dataset-into-train-test-and-validation-using-huggingface-datasets-fun) and run the training in several sessions if possible. It would be difficult if the data cannot be divided due to its nature…

> <https://github.com/huggingface/transformers/issues/14185>
>
> It takes a lot of time to tokenize my dataset, is there a way to save it and loa…d it?
> 
> Let's say I'm using the IMDB toy dataset, How to save the \`inputs\` object?
> \`\`\`
> from datasets import load\_dataset
> raw\_datasets = load\_dataset("imdb")
> from transformers import AutoTokenizer
> tokenizer = AutoTokenizer.from\_pretrained("bert-base-cased")
> inputs = tokenizer(sentences, padding="max\_length", truncation=True)

Another option would be to customize the Trainer’s DataCollator to incorporate a tokenizer. And [use trainer with IterableDataset](https://discuss.huggingface.co/t/using-iterabledataset-with-trainer-iterabledataset-has-no-len/15790).

> <https://stackoverflow.com/questions/76879872/how-to-use-huggingface-hf-trainer-train-with-custom-collate-function>

> [@DataCollator vs. Tokenizers](https://discuss.huggingface.co/t/datacollator-vs-tokenizers/5897/2):
>
> hey @hamel, welcome to the hugs forum! you’re spot on about using data collators to do padding on-the-fly. to understand why this helps, consider the following scenarios: use the tokenizer to pad each example in the dataset to the length of the longest example in the dataset use the tokenizer and DataCollatorWithPadding ([docs](https://huggingface.co/transformers/master/main_classes/data_collator.html?highlight=datacollator#datacollatorwithpadding)) to pad each example in a batch to the length of the longest example in the batch clearly, scenario 2 is more efficient, especially in cases where a few examples happ…

---

_[View the full topic](https://discuss.huggingface.co/t/space-connection-error-dataset-map/133574)._