# Understanding set\_transform **URL:** \ **Category:** 🤗Datasets\ **Created:** [February 20, 2021, 12:21am UTC](https://discuss.huggingface.co/t/understanding-set-transform/3740 "2021-02-20T00:21:13Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![jncasey](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jncasey/32/28195_2.png) [@jncasey](https://discuss.huggingface.co/u/jncasey)\ **Post date:** [February 20, 2021, 12:21am UTC](https://discuss.huggingface.co/t/understanding-set-transform/3740/1 "2021-02-20T00:21:13Z")
I’ve been working on a side project that uses phonetic English language models for text generation. Since I’m not aware of any existing phonetic English datasets, I’ve been preprocessing existing English text datasets with my phonemization script to give myself enough training data. Mainly OSCAR for pretraining the model, and then my own small datasets for fine-tuning on specific tasks. My workflow has been: 1. downloading the 2.5TB oscar\_en shuffled text file 2. processing it (in chunks) to its phonetic representation and saving those text files to disk 3. batch tokenizing those files and saving them to a local HuggingFace dataset, because it takes hours (or days) to tokenize the whole thing at the beginning of a training Even with only 3% of the original OSCAR corpus phonemized, my dataset is up to over a 1.2TB on disk. Which I was okay with – I’m running out of local storage, but I was never going to be able to use the whole OSCAR corpus anyway on my rinky-dink home setup. But this month has brought two things to HuggingFace – OSCAR in the datasets library, and on-the-fly transforms. Am I right in understanding that I could load the oscar\_en corpus from the HF Dataset, and then pass to `set_transform` a function that would phonemize and tokenize the samples, and the only hit to my disk would be the arrow cache of the original OSCAR dataset? And that I would be able to quickly resume training from my most recent checkpoint, since it’d just be loading from that cache? I imagine it’ll slow the overall training down and I might not be able to feed my GPUs as quickly as I’d like, but the simplified workflow might be worth the performance hit (especially if I find another bug in my phonemizer script that makes me want to redo everything) Building off of that, if one wanted to do a BART-style pretraining, would it be possible start with a single-column dataset, and pass to `set_transform` a function that returns the tokenized original dataset as the targets, and a randomly-masked version of the original tokens as the inputs, all on the fly? [Forgive me if this last question is dumb or nonsensical, I have a very limited understanding of seq2seq training / how BART works] ---
**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\ **Post date:** [February 20, 2021, 9:09am UTC](https://discuss.huggingface.co/t/understanding-set-transform/3740/2 "2021-02-20T09:09:55Z")
`set_transform` [does not cache](https://twitter.com/BramVanroy/status/1361741729695731715) the resulting data. Depending on the data/storage you have available you may want to opt for [`map`](https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.map). Both have a low memory footprint. ---
**Author:** ![jncasey](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jncasey/32/28195_2.png) [@jncasey](https://discuss.huggingface.co/u/jncasey)\ **Post date:** [February 20, 2021, 4:16pm UTC](https://discuss.huggingface.co/t/understanding-set-transform/3740/3 "2021-02-20T16:16:39Z")
Great, thanks for confirming that it doesn’t cache to disk. That’s exactly what I was hoping. I guess now I’ll have to update to the latest masters and start testing how much on-the-fly tokenization and other data transforms slow my training down. ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [February 21, 2021, 7:56pm UTC](https://discuss.huggingface.co/t/understanding-set-transform/3740/4 "2021-02-21T19:56:31Z")
Indeed if you use set\_transform then the resulting phonemized data are created on-the-fly and not stored/cached. Only the original OSCAR data are stored on your disk as an arrow file. And you’re right your second point about BART-style pretraining: you can pass a function to set\_transform that returns two fields, one that is the original text and one that is randomly masked, even if you have only one column in your dataset. ---
**Author:** ![jncasey](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jncasey/32/28195_2.png) [@jncasey](https://discuss.huggingface.co/u/jncasey)\ **Post date:** [February 21, 2021, 9:20pm UTC](https://discuss.huggingface.co/t/understanding-set-transform/3740/5 "2021-02-21T21:20:12Z")
Thanks, @lhoestq! That’s an incredibly cool and useful feature. Can’t wait to play with it. ---
**Author:** ![jncasey](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jncasey/32/28195_2.png) [@jncasey](https://discuss.huggingface.co/u/jncasey)\ **Post date:** [March 8, 2021, 11:51pm UTC](https://discuss.huggingface.co/t/understanding-set-transform/3740/6 "2021-03-08T23:51:52Z")
Hi @lhoestq! I’m finally getting around to testing some `set_transform` workflows and I have a question. I’ve passed a fairly CPU-heavy preprocessing function to `set_transform`. After about an hour of training, I forced the training to stop and then tried to resume from the last checkpoint. It’s been over 15 minutes of heavy CPU activity since I resumed, and the training progress indicator is still on step zero. [UPDATE: training finally resumed after 46 minutes] Is it possible that my transform function being called on every sample as the trainer advances to the last checkpoint step? If that is what’s happening, I’m not sure if it’s due to the dataset or trainer. Is there any way to avoid it? ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [March 9, 2021, 9:48am UTC](https://discuss.huggingface.co/t/understanding-set-transform/3740/7 "2021-03-09T09:48:57Z")
You’re right, your transform is being called for every sample until the last checkpoint step. This is done in the Trainer code [here](https://github.com/huggingface/transformers/blob/9a06b6b11bdfc42eea08fa91d0c737d1863c99e3/src/transformers/trainer.py#L993-L998) Maybe at that point in the code you can remove the transform and then re-add it ? To remove the formatting transform you can use `dataset.reset_format()`, and then you can set the transform again. ---
**Author:** ![jncasey](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jncasey/32/28195_2.png) [@jncasey](https://discuss.huggingface.co/u/jncasey)\ **Post date:** [March 9, 2021, 4:33pm UTC](https://discuss.huggingface.co/t/understanding-set-transform/3740/8 "2021-03-09T16:33:55Z")
Thanks for that pointer. If I’m reading the code right, I could save the current transform function by storing the dict returned by `dataset.format()`, remove the transform and let the trainer advance to the current checkpoint, and then use `dataset.set_format()` with the stored format values for args to reload the original transform. Would that get me back to the checkpoint faster and still give me the same values from the dataset? ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [March 9, 2021, 5:13pm UTC](https://discuss.huggingface.co/t/understanding-set-transform/3740/9 "2021-03-09T17:13:18Z")
Yes exactly. You can get the format with `dataset.format`, then you can remove the formatting transform with `dataset.reset_format`. At this point you can run the `for` loop that iterates over the dataloader to make it reach the requested checkpoint. Finally after that you can set the transform back with `dataset.set_format`. Hope that helps ! ---
**Author:** ![jncasey](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jncasey/32/28195_2.png) [@jncasey](https://discuss.huggingface.co/u/jncasey)\ **Post date:** [March 9, 2021, 6:15pm UTC](https://discuss.huggingface.co/t/understanding-set-transform/3740/10 "2021-03-09T18:15:43Z")
Do you think this might be a generally useful update to the trainer? Or are there use cases where you’d want to maintain a dataset’s formatting function while iterating up to the last checkpoint? ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [March 9, 2021, 9:48pm UTC](https://discuss.huggingface.co/t/understanding-set-transform/3740/11 "2021-03-09T21:48:35Z")
This may be a nice addition indeed ! Feel free to open an issue on the `transformers` repo on github. You can tag me `@lhoestq` and also `@sgugger ` who’s in charge of the `Trainer` related stuff 🙂