# Custom inference endpoint with multiple models

**URL:** <https://discuss.huggingface.co/t/custom-inference-endpoint-with-multiple-models/68858>\
**Category:** Inference Endpoints on the Hub\
**Created:** [January 11, 2024, 9:51pm UTC](https://discuss.huggingface.co/t/custom-inference-endpoint-with-multiple-models/68858 "2024-01-11T21:51:48Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![hbredin](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/hbredin/32/31902_2.png) [@hbredin](https://discuss.huggingface.co/u/hbredin)\
**Post date:** [January 11, 2024, 9:51pm UTC](https://discuss.huggingface.co/t/custom-inference-endpoint-with-multiple-models/68858/1 "2024-01-11T21:51:48Z")

</div>

I would like to deploy an inference endpoint based on a combination of multiple models.  
Here is the repository structure

```auto
/handler.py
/requirements.txt
/model1.pt
/model2.pt

```

What will be the value of `path` here? Adding a few prints in the handler tells me that it is set to `/repository/`.  
However, it also looks like `/repository/` does not contain `model1.pt` and `model2.pt`.  
Is this on purpose? How can I update the code below to access `model1.pt` and `model2.pt`?

```python
class EndpointHandler:
    def __init__ (self, path=""):
        self.model1 = torch.load('model1.pt')
        self.model2 = torch.load('model2.pt')

    def __call__ (self, data):
        pass

```

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [January 12, 2024, 8:40am UTC](https://discuss.huggingface.co/t/custom-inference-endpoint-with-multiple-models/68858/2 "2024-01-12T08:40:13Z")

</div>

the torch load should be `os.path.join(path,“model1.pt”). The path var provided in the init points to where your files are.

---

<div class="post-metadata">

**Author:** ![hbredin](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/hbredin/32/31902_2.png) [@hbredin](https://discuss.huggingface.co/u/hbredin)\
**Post date:** [January 12, 2024, 12:27pm UTC](https://discuss.huggingface.co/t/custom-inference-endpoint-with-multiple-models/68858/3 "2024-01-12T12:27:10Z")

</div>

Thanks.

The problem was that, for some reason, not all files are pulled from the model repository.

For instance, files with `.ckpt` extension (it was the case for me) are not available in `path` directory. I had to change the extension of the model files for them to be downloaded.

This behavior is probably documented somewhere but I could not find it.

The problem is now solved. `os.path.join(path, "model.pt")` does the trick.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [January 13, 2024, 12:27am UTC](https://discuss.huggingface.co/t/custom-inference-endpoint-with-multiple-models/68858/4 "2024-01-13T00:27:35Z")

</div>

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.