# Read data from hdfs

**URL:** <https://discuss.huggingface.co/t/read-data-from-hdfs/5339>\
**Category:** Beginners\
**Created:** [April 7, 2021, 3:10am UTC](https://discuss.huggingface.co/t/read-data-from-hdfs/5339 "2021-04-07T03:10:33Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![gfork](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/gfork/32/1479_2.png) [@gfork](https://discuss.huggingface.co/u/gfork)\
**Post date:** [April 7, 2021, 3:10am UTC](https://discuss.huggingface.co/t/read-data-from-hdfs/5339/1 "2021-04-07T03:10:33Z")

</div>

Hi, I’m new to huggingface. For my current exeperience, the code I’ve read about shows that the data will be downloaded to local before next steps.  
Is that possible to read stream data from remote source like hdfs? Since my training data mab be huge, it will exhaust my disk storage for downloading the data.  
Will be thankful for any hint.

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\
**Post date:** [April 7, 2021, 7:59am UTC](https://discuss.huggingface.co/t/read-data-from-hdfs/5339/2 "2021-04-07T07:59:47Z")

</div>

hey @gfork, i’ve never tried it myself but the `datasets` library let’s you process data with Apache Beam: [Beam Datasets — datasets 1.5.0 documentation](https://huggingface.co/docs/datasets/beam_dataset.html?highlight=hdfs#beam-datasets)

perhaps that is suitable for your use case?

---

<div class="post-metadata">

**Author:** ![gfork](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/gfork/32/1479_2.png) [@gfork](https://discuss.huggingface.co/u/gfork)\
**Post date:** [April 7, 2021, 8:05am UTC](https://discuss.huggingface.co/t/read-data-from-hdfs/5339/3 "2021-04-07T08:05:34Z")

</div>

Thanks @lewtun, I’ll try that, and give you feedback if it works. 😄