# Download data is slow from github lfs

**URL:** <https://discuss.huggingface.co/t/download-data-is-slow-from-github-lfs/118294>\
**Category:** Beginners\
**Created:** [November 13, 2024, 7:52am UTC](https://discuss.huggingface.co/t/download-data-is-slow-from-github-lfs/118294 "2024-11-13T07:52:58Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![ljw20180420](https://avatars.discourse-cdn.com/v4/letter/l/d6d6ee/32.png) [@ljw20180420](https://discuss.huggingface.co/u/ljw20180420)\
**Post date:** [November 13, 2024, 7:52am UTC](https://discuss.huggingface.co/t/download-data-is-slow-from-github-lfs/118294/1 "2024-11-13T07:52:58Z")

</div>

I download my data from huggingface by DownloadManager. It is very fast.

```python
from datasets.download.download_manager import DownloadManager
dl_manager = DownloadManager()
dl_manager.download("https://huggingface.co/datasets/ljw20180420/CRISPR_data/resolve/main/dataset.json.gz")

```

However, when I download it from github lfs, it is very slow.

```python
downloaded_files = dl_manager.download("https://github.com/ljw20180420/CRISPRdata/raw/refs/heads/main/dataset.json.gz")

```

Does anyone know the reason? Thank you.

---

<div class="post-metadata">

**Author:** ![ljw20180420](https://avatars.discourse-cdn.com/v4/letter/l/d6d6ee/32.png) [@ljw20180420](https://discuss.huggingface.co/u/ljw20180420)\
**Post date:** [November 13, 2024, 9:26am UTC](https://discuss.huggingface.co/t/download-data-is-slow-from-github-lfs/118294/2 "2024-11-13T09:26:24Z")

</div>

I know the reason. I only set proxy by

```python
from huggingface_hub import configure_http_backend
import requests

url="socks5h://127.0.0.1:1080"
# Create a factory function that returns a Session with configured proxies
def backend_factory() -> requests.Session:
    session = requests.Session()
    session.proxies.update({
        "http": url,
        "https": url
    })
    return session
# Set it as the default session factory
configure_http_backend(backend_factory=backend_factory)

```

This works only for huggingface\_hub, but not datasets. I need to set proxy for DownloadManager.

```python
from datasets.download.download_manager import DownloadManager, DownloadConfig
dl_manager = DownloadManager()
df = dl_manager.download("https://github.com/ljw20180420/CRISPRdata/raw/refs/heads/main/dataset.json.gz") # slow
dl_manager_proxy = DownloadManager(download_config=DownloadConfig(proxies={"http": "socks5h://127.0.0.1:1080", "https": "socks5h://127.0.0.1:1080"}))
df_proxy = dl_manager_proxy.download("https://github.com/ljw20180420/CRISPRdata/raw/refs/heads/main/dataset.json.gz") # fast

```