# Cannot preprocess wikipedia dataset

**URL:** <https://discuss.huggingface.co/t/cannot-preprocess-wikipedia-dataset/34098>\
**Category:** 🤗Datasets\
**Created:** [March 18, 2023, 1:16pm UTC](https://discuss.huggingface.co/t/cannot-preprocess-wikipedia-dataset/34098 "2023-03-18T13:16:45Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Waterhorse](https://avatars.discourse-cdn.com/v4/letter/w/f1d935/32.png) [@Waterhorse](https://discuss.huggingface.co/u/Waterhorse)\
**Post date:** [March 18, 2023, 1:16pm UTC](https://discuss.huggingface.co/t/cannot-preprocess-wikipedia-dataset/34098/1 "2023-03-18T13:16:45Z")

</div>

I am using load\_dataset to load the new Wikipedia dataset by:

load\_dataset(“wikipedia”, language=“en”, date=“20230301”, beam\_runner=“DirectRunner”)

I have successfully download the whole data, however, the procedure gets stuck after downloading it and I have to interrupt it:

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/d00c2d997ead3554b383b891f2ed4d0bdf110463.png)

Any solutions?

---

<div class="post-metadata">

**Author:** ![graelo](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/graelo/32/16537_2.png) [@graelo](https://discuss.huggingface.co/u/graelo)\
**Post date:** [June 3, 2023, 8:47pm UTC](https://discuss.huggingface.co/t/cannot-preprocess-wikipedia-dataset/34098/2 "2023-06-03T20:47:54Z")

</div>

Hi, just a hint from my experience preprocessing the `es` dataset:

I successfully preprocessed the `es` dump from `20230320`, and that dataset is roughly 4 GB which is much smaller than the `en` dataset you chose. The preprocessing took about 3 hours and consumed up to 35 GB of ram, but it worked. Currently, there is one annoyance: no indication of progress, except if you monitor the size of the temporary file being created in the folder `~/.cache/huggingface/datasets/wikipedia/en/2.0.0/...`. You can see that file growing about every 2 minutes.

So I’m guessing in your case the processing will take at least 12 hours and consume much more ram than 35 GB.

A good alternative to preprocessing locally would be to use DataflowRunner, but I don’t precisely know how to craft the `beam_options` arguments to do so.

My 2 cts!  
Cheers and take care