# HF Hub Commit API isn't accepting LFS files

**URL:** <https://discuss.huggingface.co/t/hf-hub-commit-api-isnt-accepting-lfs-files/113997>\
**Category:** 🤗Hub\
**Created:** [October 25, 2024, 6:55pm UTC](https://discuss.huggingface.co/t/hf-hub-commit-api-isnt-accepting-lfs-files/113997 "2024-10-25T18:55:25Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![apyh](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/apyh/32/33440_2.png) [@apyh](https://discuss.huggingface.co/u/apyh)\
**Post date:** [October 25, 2024, 6:55pm UTC](https://discuss.huggingface.co/t/hf-hub-commit-api-isnt-accepting-lfs-files/113997/1 "2024-10-25T18:55:25Z")

</div>

Hello! I’m re-implementing the HF commit API (don’t ask) and I’m having troubles getting LFS files to be accepted as part of a commit.  
I’ve managed to get the files preuploaded to the HF S3 buckets, and sent the completion request to the HF API.

I send

```ts
{
  oid: "7d34cce2c40a7089ea8b1d8ea9c25c573c46fcce7aa60579748118183a03f272",
  parts: [
    { part_number: 1, etag: "\"bbb437b7fe0d1d765cfe7c1af9156484\"" },
    { part_number: 2, etag: "\"dc335cf70c0ca26b4d1bf7059ad9226c\"" }
  ]
}

```

and the API sends me back `{"success":true}` (thank you API!)  
however, when I try to call `/api/models/username/repo/commit/main`, with this payload (note the identical OID that the LFS multipart finalize endpoint has accepted as uploaded)

```ts
[
  {
    "key": "header",
    "value": {
      "summary": "Upload 20m_file.bin with hf_hub",
      "description": "lalalala"
    }
  },
  {
    "key": "lfsFile",
    "value": {
      "path": "20m_file.bin",
      "algo": "sha256",
      "oid": "7d34cce2c40a7089ea8b1d8ea9c25c573c46fcce7aa60579748118183a03f272",
      "size": 20970496
    }
  }
]

```

I get a 400 back from the API with `{"error":"Your push was rejected because an LFS pointer pointed to a file that does not exist. For instance, this can happen if you used git push --no-verify to push your changes. Offending file: - 20m_file.bin"}`.

Any ideas as to what I’m doing wrong?

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [October 26, 2024, 1:30am UTC](https://discuss.huggingface.co/t/hf-hub-commit-api-isnt-accepting-lfs-files/113997/2 "2024-10-26T01:30:09Z")

</div>

Is it the same case as this one?

> [@Your push was rejected because an LFS pointer pointed to a file that does not exist](https://discuss.huggingface.co/t/your-push-was-rejected-because-an-lfs-pointer-pointed-to-a-file-that-does-not-exist/24583/4):
>
> Thanks @pierric!

---

<div class="post-metadata">

**Author:** ![apyh](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/apyh/32/33440_2.png) [@apyh](https://discuss.huggingface.co/u/apyh)\
**Post date:** [October 26, 2024, 5:28am UTC](https://discuss.huggingface.co/t/hf-hub-commit-api-isnt-accepting-lfs-files/113997/3 "2024-10-26T05:28:52Z")

</div>

while this is the same error, the circumstances thru which it arose and can be troubleshooted are vastly different & doesn’t help me fix my issue at all :\<

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [October 26, 2024, 7:15am UTC](https://discuss.huggingface.co/t/hf-hub-commit-api-isnt-accepting-lfs-files/113997/4 "2024-10-26T07:15:04Z")

</div>

@pierric It’s a rare error. Any idea what it is?

---

<div class="post-metadata">

**Author:** ![apyh](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/apyh/32/33440_2.png) [@apyh](https://discuss.huggingface.co/u/apyh)\
**Post date:** [October 28, 2024, 8:23pm UTC](https://discuss.huggingface.co/t/hf-hub-commit-api-isnt-accepting-lfs-files/113997/5 "2024-10-28T20:23:49Z")

</div>

`celina` fixed my issue !  
Quote from her:

> I took a quick look at your code, the issue might come from a mismatch in the size between the actual file uploaded to S3 and what it is sent in the verify request. in `process_stream`, you’re not taking the `sample` size in `total_bytes`, which makes the size off by 1024 bytes. you might also want to include the first 1024 bytes of the file in the SHA256 hash.
> 
> ```rust
> async fn process_stream<R>(mut reader: R, size: u64) -> io::Result<UploadInfo>
> where
> R: AsyncRead + Unpin,
> {
> let mut sample = vec![0u8; SAMPLE_SIZE.min(size as usize)];
> reader.read_exact(&mut sample).await?;
> 
> let mut hasher = Sha256::new();
> hasher.update(&sample); // Hash the sample bytes too
> let mut total_bytes = sample.len() as u64; // Start with `sample` size
> let mut buffer = vec![0u8; CHUNK_SIZE];
> 
> loop {
> let bytes_read = reader.read(&mut buffer).await?;
> if bytes_read == 0 {
> break;
> }
> hasher.update(&buffer[..bytes_read]);
> total_bytes += bytes_read as u64;
> }
> 
> Ok(UploadInfo {
> size: total_bytes,
> sample,
> sha256: hasher.finalize().to_vec(),
> })
> }
> 
> ```
> 
> with these changes, I managed to upload lfs files with your script.  
> I hope this helped! 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [October 29, 2024, 8:24am UTC](https://discuss.huggingface.co/t/hf-hub-commit-api-isnt-accepting-lfs-files/113997/6 "2024-10-29T08:24:26Z")

</div>

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.