# Map with tokenize function stuck in the beginning **URL:** \ **Category:** 🤗Datasets\ **Created:** [December 26, 2024, 2:21am UTC](https://discuss.huggingface.co/t/map-with-tokenize-function-stuck-in-the-beginning/133047 "2024-12-26T02:21:00Z")\ **Posts on this page:** 1\ **Showing post:** 5
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [December 27, 2024, 1:59pm UTC](https://discuss.huggingface.co/t/map-with-tokenize-function-stuck-in-the-beginning/133047/5 "2024-12-27T13:59:10Z")
As you say, I don’t want to recommend it either. The slowdown and difficulty of implementation are unavoidable overheads, and this is a last resort. However, your dataset is probably too large even in an environment with a lot of RAM… There may be a way to train it [by slicing the dataset itself in advance simply](https://huggingface.co/docs/datasets/v3.2.0/loading#slice-splits) and feeding it in small amounts without streaming. > [@Keeping IterableDataset node-wise split fixed during DDP](https://discuss.huggingface.co/t/keeping-iterabledataset-node-wise-split-fixed-during-ddp/58713/3): > > Thanks, yes, the dataset has many shards. But could you give the example how to keep the dataset split fixed for each node during the whole DDP run? The documentation had the following example for split\_dataset\_by\_node: from datasets.distributed import split\_dataset\_by\_node ids = ds.to\_iterable\_dataset(num\_shards=512) ids = ids.shuffle(buffer\_size=10\_000) # will shuffle the shards order and use a shuffle buffer when you start iterating ids = split\_dataset\_by\_node(ids, world\_size=8, rank=0) … > **[Stream](https://huggingface.co/docs/datasets/stream#tokenization)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. --- _[View the full topic](https://discuss.huggingface.co/t/map-with-tokenize-function-stuck-in-the-beginning/133047)._