# Space Connection Error - Dataset Map **URL:** \ **Category:** Spaces\ **Created:** [December 30, 2024, 8:37am UTC](https://discuss.huggingface.co/t/space-connection-error-dataset-map/133574 "2024-12-30T08:37:09Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Cicciokr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cicciokr/32/38336_2.png) [@Cicciokr](https://discuss.huggingface.co/u/Cicciokr)\ **Post date:** [December 30, 2024, 8:37am UTC](https://discuss.huggingface.co/t/space-connection-error-dataset-map/133574/1 "2024-12-30T08:37:09Z")
Hi, i have created a Streamlit space with a CPU UPGRADED and i try to do a Dataset.map, after about 20 minute the space go on “Connection timeout” on container log i see “stopping…” and my procedure of Dataset.map stop and i need to restart… On the screen i see a popup with message: “Sorry, there is an error on our side.” How can i resolve? I have tried also to do a tokenized\_dataset local on my pc but it is 45 GB and i don’t know how to load it on huggingface to use on space and so skip this step of dataset.map Thank you ---
**Author:** ![Alanturner2](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/alanturner2/32/37542_2.png) [@Alanturner2](https://discuss.huggingface.co/u/Alanturner2)\ **Post date:** [December 30, 2024, 9:14am UTC](https://discuss.huggingface.co/t/space-connection-error-dataset-map/133574/2 "2024-12-30T09:14:52Z")
To resolve the connection timeout and error during `Dataset.map` in your Streamlit Space: 1. **Increase Resources** : Ensure your Streamlit Space has sufficient CPU and memory for processing large datasets. 2. **Optimize Dataset Processing** : Process the dataset in smaller batches using a chunked approach to avoid timeouts. 3. **Pre-tokenize Locally** : Tokenize the dataset locally, save it in a format like Parquet/JSON, and upload it to Hugging Face. Load the tokenized dataset directly in your Streamlit Space. 4. **Use `map` Optimizations** : Use `batched=True` and `num_proc` in `Dataset.map` for better performance. 5. **Check Logs** : Review container logs for detailed error messages to identify specific issues. ---
**Author:** ![Cicciokr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cicciokr/32/38336_2.png) [@Cicciokr](https://discuss.huggingface.co/u/Cicciokr)\ **Post date:** [December 30, 2024, 9:30am UTC](https://discuss.huggingface.co/t/space-connection-error-dataset-map/133574/3 "2024-12-30T09:30:50Z")
Hi, i have loaded my dataset and converted it on parquet format _dataset = load\_dataset(“Cicciokr/CC-100-Latin”, revision=“refs/convert/parquet”)_ i have also tokenized locally my dataset and is about 45GB but is not possible to upload it on the Spaces, error “reach limit 1GB”. To use the Dataset it in the Trainer function, i need to do a mapping with tokenizer right or is not necessary and i can use directly the dataset? _tokenizer = RobertaTokenizerFast(_ - vocab\_file=“./vocab.json”,\* - merges\_file=“./merges.txt”,\* _)_ _def preprocess\_function(examples):_ - return tokenizer(examples[‘text’], truncation=True, padding=‘max\_length’, max\_length=512)\* _tokenized\_dataset = dataset.map(preprocess\_function, batched=True, num\_proc=8)_ _trainer = Trainer(_ - train\_dataset=tokenized\_dataset[‘train’]\* _)_ In the log i have no other information only connection timeout, and i use the CPU Upgraded, is not possible to have connection timeout while is in run mode with no other error, i pay the hourly use of this resources. Thank you ---
**Author:** ![Alanturner2](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/alanturner2/32/37542_2.png) [@Alanturner2](https://discuss.huggingface.co/u/Alanturner2)\ **Post date:** [December 30, 2024, 9:34am UTC](https://discuss.huggingface.co/t/space-connection-error-dataset-map/133574/4 "2024-12-30T09:34:47Z")
Yes, to use your dataset with the `Trainer` function, you need to map the tokenizer over the dataset to tokenize the text data. You’ve already set up the tokenizer and preprocessing function correctly. By using the `dataset.map` function, you’re applying the tokenizer to your dataset, which prepares it for use with the Trainer. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [December 30, 2024, 10:16am UTC](https://discuss.huggingface.co/t/space-connection-error-dataset-map/133574/5 "2024-12-30T10:16:30Z")
> i have also tokenized locally my dataset and is about 45GB but is not possible to upload it on the Spaces, error “reach limit 1GB”. Model repos and dataset repos are large and fast, but you cannot upload large files to Spaces. It is possible to download from model repos and dataset repos after starting Spaces, and it is fast, so it is good to put datasets in dataset repos. Edit: However, the free Spaces disk space is still 50GB even after starting up, so if you only want to read a 45GB data set, it should be fine, but it might be difficult to process… it’s 50-50. ---
**Author:** ![Cicciokr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cicciokr/32/38336_2.png) [@Cicciokr](https://discuss.huggingface.co/u/Cicciokr)\ **Post date:** [December 30, 2024, 10:19am UTC](https://discuss.huggingface.co/t/space-connection-error-dataset-map/133574/6 "2024-12-30T10:19:50Z")
How can i do to load “tokenized dataset” from the dataset repos? And also my question is… if the space can’t even run a dataset map, is it possible to run training on a T4 small or will it always go into connection timeout? Now i’m trying to do a Dataset.map with a T4 Small, but is slower then a CPU Upgraded ( T4 Small about 7000 examples/s, CPU Upgraded 20000 examples/s ) Thank you ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [December 30, 2024, 10:23am UTC](https://discuss.huggingface.co/t/space-connection-error-dataset-map/133574/7 "2024-12-30T10:23:37Z")
> How can i do to load “tokenized dataset” from the dataset repos? I’ve found a way to save it. However, I think it would be more reliable to [split the data set into several parts](https://stackoverflow.com/questions/76001128/splitting-dataset-into-train-test-and-validation-using-huggingface-datasets-fun) and run the training in several sessions if possible. It would be difficult if the data cannot be divided due to its nature… > > > It takes a lot of time to tokenize my dataset, is there a way to save it and loa…d it? > > Let's say I'm using the IMDB toy dataset, How to save the \`inputs\` object? > \`\`\` > from datasets import load\_dataset > raw\_datasets = load\_dataset("imdb") > from transformers import AutoTokenizer > tokenizer = AutoTokenizer.from\_pretrained("bert-base-cased") > inputs = tokenizer(sentences, padding="max\_length", truncation=True) Another option would be to customize the Trainer’s DataCollator to incorporate a tokenizer. And [use trainer with IterableDataset](https://discuss.huggingface.co/t/using-iterabledataset-with-trainer-iterabledataset-has-no-len/15790). > > [@DataCollator vs. Tokenizers](https://discuss.huggingface.co/t/datacollator-vs-tokenizers/5897/2): > > hey @hamel, welcome to the hugs forum! you’re spot on about using data collators to do padding on-the-fly. to understand why this helps, consider the following scenarios: use the tokenizer to pad each example in the dataset to the length of the longest example in the dataset use the tokenizer and DataCollatorWithPadding ([docs](https://huggingface.co/transformers/master/main_classes/data_collator.html?highlight=datacollator#datacollatorwithpadding)) to pad each example in a batch to the length of the longest example in the batch clearly, scenario 2 is more efficient, especially in cases where a few examples happ…