# Azure ML Datastore

**URL:** <https://discuss.huggingface.co/t/azure-ml-datastore/31732>\
**Category:** 🤗Datasets\
**Created:** [February 15, 2023, 1:06am UTC](https://discuss.huggingface.co/t/azure-ml-datastore/31732 "2023-02-15T01:06:55Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![sahuabinash](https://avatars.discourse-cdn.com/v4/letter/s/b38774/32.png) [@sahuabinash](https://discuss.huggingface.co/u/sahuabinash)\
**Post date:** [February 15, 2023, 1:06am UTC](https://discuss.huggingface.co/t/azure-ml-datastore/31732/1 "2023-02-15T01:06:55Z")

</div>

Hello. My dataset is in Azure Blob Storage as a parquet. I am able to create a dataset(azureml) using Tablular.from\_parquet\_files. My question is how do I convert this to Huggingface dataset?  
from datasets import load\_dataset  
dataset = load\_dataset(“parquet”, data\_files={‘train’: ‘train.parquet’, ‘test’: ‘test.parquet’})

Any general recommendation on how to create a Pytorch Dataloader for large parquet files in Azure?  
Thanks

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [February 16, 2023, 11:44am UTC](https://discuss.huggingface.co/t/azure-ml-datastore/31732/2 "2023-02-16T11:44:44Z")

</div>

Hi ! If your parquet files are public, you can load them using their HTTP urls in `load_dataset`. You can also stream the dataset using `streaming=True` (useful especially if your dataset is super big),  
then you can pass your dataset directly to a PyTorch DataLoader (see [documentation](https://huggingface.co/docs/datasets/use_with_pytorch#data-loading)).

We don’t support accessing private azure blob storage yet, though there is an issue opened about it: [Support cloud storage in load\_dataset · Issue #5281 · huggingface/datasets · GitHub](https://github.com/huggingface/datasets/issues/5281)