# Map with tokenize function stuck in the beginning **URL:** \ **Category:** 🤗Datasets\ **Created:** [December 26, 2024, 2:21am UTC](https://discuss.huggingface.co/t/map-with-tokenize-function-stuck-in-the-beginning/133047 "2024-12-26T02:21:00Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![sleepywalker](https://avatars.discourse-cdn.com/v4/letter/s/6f9a4e/32.png) [@sleepywalker](https://discuss.huggingface.co/u/sleepywalker)\ **Post date:** [December 26, 2024, 2:21am UTC](https://discuss.huggingface.co/t/map-with-tokenize-function-stuck-in-the-beginning/133047/1 "2024-12-26T02:21:00Z")
Hello. I have a function ```auto block_size = 8192 def tokenize_and_chunk(examples): tokenized_inputs = tokenizer(examples['content'], return_tensors='pt', add_special_tokens=False) input_ids = tokenized_inputs['input_ids'][0] total_length = len(input_ids) if total_length >= block_size: input_ids = input_ids[:(total_length // block_size) * block_size] input_chunks = [input_ids[i:i + block_size] for i in range(0, len(input_ids), block_size)] return {'input_ids': input_chunks} ``` and a dataset with 25000 samples where every sample is a string with ~50000 symbols. When I try to tokenize this dataset using `tokenized_dataset = concatenated_data.map(tokenize_and_chunk, batched=True, num_proc=200, remove_columns=concatenated_data.column_names)` it stucks in the beginning. I tried to change `num_proc` to 1 to use tokenizer parallelism but it didn’t help. When I try to tokenize with this function only one sample like `tokenize_and_chunk(concatenated_data[0])` it succesfully tokenized in a moment. How can I fix that problem with `dataset.map` for tokenization? ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [December 27, 2024, 6:41am UTC](https://discuss.huggingface.co/t/map-with-tokenize-function-stuck-in-the-beginning/133047/2 "2024-12-27T06:41:27Z")
Maybe try IterableDataset? > [@Training a Tokenizer on a Streamed Dataset](https://discuss.huggingface.co/t/training-a-tokenizer-on-a-streamed-dataset/8026): > > Hi, I’m trying to train a tokenizer on a dataset that uses streaming. I followed the instructions provided [here](https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling#train-tokenizer), with the addition of streaming=True during the dataset loading step. However, it quickly failed as the IterableDataset class does not have a length property (unlike the normal Dataset class). How can I work around this issue without having to download the dataset file entirely, i.e. purely relying on dataset streaming? Here’s the [Colab link](https://colab.research.google.com/drive/1rOdAzmfdp2S1tGDk4iaiM2d4pai4QGES?usp=sharing) to reproduce the error. Any help would be … ---
**Author:** ![sleepywalker](https://avatars.discourse-cdn.com/v4/letter/s/6f9a4e/32.png) [@sleepywalker](https://discuss.huggingface.co/u/sleepywalker)\ **Post date:** [December 27, 2024, 11:34am UTC](https://discuss.huggingface.co/t/map-with-tokenize-function-stuck-in-the-beginning/133047/3 "2024-12-27T11:34:07Z")
With IterableDataset like ConstnatLengthDataset model will train longer than with predefined tokenized dataset. Moreover, with IterableDataset multigpu training is more complex. It needs something like DistributedSample, DistributedLoader and so on. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [December 27, 2024, 1:59pm UTC](https://discuss.huggingface.co/t/map-with-tokenize-function-stuck-in-the-beginning/133047/5 "2024-12-27T13:59:10Z")
As you say, I don’t want to recommend it either. The slowdown and difficulty of implementation are unavoidable overheads, and this is a last resort. However, your dataset is probably too large even in an environment with a lot of RAM… There may be a way to train it [by slicing the dataset itself in advance simply](https://huggingface.co/docs/datasets/v3.2.0/loading#slice-splits) and feeding it in small amounts without streaming. > [@Keeping IterableDataset node-wise split fixed during DDP](https://discuss.huggingface.co/t/keeping-iterabledataset-node-wise-split-fixed-during-ddp/58713/3): > > Thanks, yes, the dataset has many shards. But could you give the example how to keep the dataset split fixed for each node during the whole DDP run? The documentation had the following example for split\_dataset\_by\_node: from datasets.distributed import split\_dataset\_by\_node ids = ds.to\_iterable\_dataset(num\_shards=512) ids = ids.shuffle(buffer\_size=10\_000) # will shuffle the shards order and use a shuffle buffer when you start iterating ids = split\_dataset\_by\_node(ids, world\_size=8, rank=0) … > **[Stream](https://huggingface.co/docs/datasets/stream#tokenization)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. ---
**Author:** ![sleepywalker](https://avatars.discourse-cdn.com/v4/letter/s/6f9a4e/32.png) [@sleepywalker](https://discuss.huggingface.co/u/sleepywalker)\ **Post date:** [December 27, 2024, 10:47pm UTC](https://discuss.huggingface.co/t/map-with-tokenize-function-stuck-in-the-beginning/133047/6 "2024-12-27T22:47:03Z")
I mean why function with tokenization takes so much time to implement it relaitively to other functions. I understand that tokenization is not a simple transformation but this tokenizer is fast and as I know tokenization has multiprocessing by itself and I can map with only 1 `num_proc` and anyway it will parallelizae tokenization by itself. But I can’t understand problem that mapping with `batch_size=100` for example just stuck in the beginning and not working. Due to that behaviour, other mappings takes like 1-2 minute for implementation while tokenization may take 15-20 minutes with `batched=False` to not to stuck in the beginnging