# Function calling not working with inference clients on (seemingly) any model

**URL:** <https://discuss.huggingface.co/t/function-calling-not-working-with-inference-clients-on-seemingly-any-model/138581>\
**Category:** Beginners\
**Created:** [February 2, 2025, 3:31pm UTC](https://discuss.huggingface.co/t/function-calling-not-working-with-inference-clients-on-seemingly-any-model/138581 "2025-02-02T15:31:43Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![anomalus](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/anomalus/32/17744_2.png) [@anomalus](https://discuss.huggingface.co/u/anomalus)\
**Post date:** [February 2, 2025, 3:31pm UTC](https://discuss.huggingface.co/t/function-calling-not-working-with-inference-clients-on-seemingly-any-model/138581/1 "2025-02-02T15:31:43Z")

</div>

I’m attempting to use function-calling capabilities with text inference models, but none of the ways I’ve attempted to do with the inference clients work. I use OpenAI’s function calling regularly, but Hugging Face capabilities seem fragile and unpredictable, but I suspect I’m just “doing it wrong.”

I’d appreciate any insights into the following cases I’m seeing:

## JavaScript inference client fails

If I just include the first message (e.g. “What’s the weather in X place?”), the models will properly return the function call required to finish the completion. But when including the subsequent messages, typically required for OpenAI, all models I try fail thusly:

Mistral models error out claiming `tool_call_id`s should be 9-character alphanumeric strings. Even when they are, it still gives the same error.

The Qwen model (and others) error out with “An error fetching the blob.”

And still yet, some Llama models will just return the original tool call message again, resulting in an infinite loop if I don’t catch it.

> <https://gist.github.com/awmartin/d13a6c7369a61acc564343925061ec7c>

## Python inference client fails

Similarly, the same code in Python fails. Again, Mistral models error out with the 9-character claim, even though the `tool_call_id` is indeed a 9-char alphanumeric.

Other models fail by not returning a final completion, but just repeat the tool\_call message.

> <https://gist.github.com/awmartin/c64c84fbbdc3a9f0c2ce6e5ae0dab3dc>

Again, any insights you’d have I’d appreciate!

---

<div class="post-metadata">

**Author:** ![Alanturner2](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/alanturner2/32/37542_2.png) [@Alanturner2](https://discuss.huggingface.co/u/Alanturner2)\
**Post date:** [February 3, 2025, 7:17am UTC](https://discuss.huggingface.co/t/function-calling-not-working-with-inference-clients-on-seemingly-any-model/138581/4 "2025-02-03T07:17:58Z")

</div>

> [@Remote: Repository not found](https://discuss.huggingface.co/t/remote-repository-not-found/136676/2):
>
> Hey @aqpeeb ! I think you should check your authentication with huggingface. Authenticate with Hugging Face: If authentication is required, make sure you’re logged in to Hugging Face via the CLI: bash CopyEdit huggingface-cli login Enter your token when prompted. If the Space belongs to an organization, ensure you have the proper permissions. And then you should check your huggingface api token has the authorization to use the organization or repository. Hope this help!

---

<div class="post-metadata">

**Author:** ![anomalus](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/anomalus/32/17744_2.png) [@anomalus](https://discuss.huggingface.co/u/anomalus)\
**Post date:** [February 3, 2025, 1:44pm UTC](https://discuss.huggingface.co/t/function-calling-not-working-with-inference-clients-on-seemingly-any-model/138581/6 "2025-02-03T13:44:44Z")

</div>

@Alanturner2 I don’t think it’s a token or authentication problem for this function calling. Was that comment intended to help here as well? I have a token with the proper permissions and all that needs to happen to reproduce behavior is setting the HF\_TOKEN variable.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [February 3, 2025, 2:00pm UTC](https://discuss.huggingface.co/t/function-calling-not-working-with-inference-clients-on-seemingly-any-model/138581/7 "2025-02-03T14:00:15Z")

</div>

I tried it twice with Qwen and twice with Zephyr. I wonder what the expected behavior is…

```auto
ChatCompletionOutput(choices=[ChatCompletionOutputComplete(finish_reason='stop', index=0, message=ChatCompletionOutputMessage(role='assistant', content=None, tool_calls=[ChatCompletionOutputToolCall(function=ChatCompletionOutputFunctionDefinition(arguments={'format': 'celsius', 'location': 'San Giustino, Italy'}, name='get_current_weather', description=None), id='0', type='function')]), logprobs=None)], created=1738590766, id='', model='Qwen/Qwen2.5-72B-Instruct', system_fingerprint='3.0.2-sha-b70f29d', usage=ChatCompletionOutputUsage(completion_tokens=30, prompt_tokens=312, total_tokens=342))   
ChatCompletionOutput(choices=[ChatCompletionOutputComplete(finish_reason='stop', index=0, message=ChatCompletionOutputMessage(role='assistant', content=None, tool_calls=[ChatCompletionOutputToolCall(function=ChatCompletionOutputFunctionDefinition(arguments={'format': 'celsius', 'location': 'San Giustino, Italy'}, name='get_current_weather', description=None), id='0', type='function')]), logprobs=None)], created=1738590769, id='', model='Qwen/Qwen2.5-72B-Instruct', system_fingerprint='3.0.2-sha-b70f29d', usage=ChatCompletionOutputUsage(completion_tokens=30, prompt_tokens=339, total_tokens=369)) 

ChatCompletionOutput(choices=[ChatCompletionOutputComplete(finish_reason='stop', index=0, message=ChatCompletionOutputMessage(role='assistant', content=None, tool_calls=[ChatCompletionOutputToolCall(fu
nction=ChatCompletionOutputFunctionDefinition(arguments={'format': 'celsius', 'location': 'San Giustino (Italy)'}, name='get_current_weather', description=None), id='0', type='function')]), logprobs=None)], created=1738590905, id='', model='HuggingFaceH4/zephyr-7b-beta', system_fingerprint='3.0.1-sha-bb9095a', usage=ChatCompletionOutputUsage(completion_tokens=41, prompt_tokens=300, total_tokens=341))
ChatCompletionOutput(choices=[ChatCompletionOutputComplete(finish_reason='stop', index=0, message=ChatCompletionOutputMessage(role='assistant', content=None, tool_calls=[ChatCompletionOutputToolCall(function=ChatCompletionOutputFunctionDefinition(arguments={'format': 'celsius', 'location': 'San Giustino, Italy'}, name='get_current_weather', description=None), id='0', type='function')]), logprobs=None)], created=1738590906, id='', model='HuggingFaceH4/zephyr-7b-beta', system_fingerprint='3.0.1-sha-bb9095a', usage=ChatCompletionOutputUsage(completion_tokens=40, prompt_tokens=101, total_tokens=141))

```

> <https://github.com/huggingface/huggingface_hub/issues/2606>
>
> \### Describe the bug
> 
> Let's start by thanking you for this great resource :blu…e\_heart: 
> 
> The \`InferenceClient\` supports tool calling, as explained \[here\](https://huggingface.co/docs/huggingface\_hub/en/package\_reference/inference\_client#huggingface\_hub.InferenceClient.chat\_completion.example-4).
> 
> In many use cases, \*\*it is useful to pass back the tool call to the Language Model and also the tool result in a message from \`tool\` role\*\*.
> In this way, the LM can for example respond in a human-readable way.
> This is supported in \[HF Transformers\](https://huggingface.co/docs/transformers/main/chat\_templating#a-complete-tool-use-example).
> 
> When using the \`InferenceClient\` (for Serverless Inference API or TGI), I'm struggling to find a way to reproduce this desired behavior.
> (I mostly experimented with Mistral and Llama models supporting tool/function calling, with similar results)
> 
> @Wauplin @hanouticelina Is this supported or planned?
> Is there any workaround you suggest? So far, I've only tried to wrap the tool result in a message from \`user\` and this somehow works...
> 
> Probably related issue (in TGI): https://github.com/huggingface/text-generation-inference/issues/2461
> 
> \### Reproduction
> \`\`\`python
> from huggingface\_hub import InferenceClient
> 
> client = InferenceClient("mistralai/Mistral-Nemo-Instruct-2407")
> 
> messages = \[
> {
> "role": "system",
> "content": "Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous.",
> },
> {
> "role": "user",
> "content": "What's the weather like in San Giustino (Italy) in Celsius?",
> },
> \]
> tools = \[
> {
> "type": "function",
> "function": {
> "name": "get\_current\_weather",
> "description": "Get the current weather",
> "parameters": {
> "type": "object",
> "properties": {
> "location": {
> "type": "string",
> "description": "The city and state, e.g. San Francisco, CA",
> },
> "format": {
> "type": "string",
> "enum": \["celsius", "fahrenheit"\],
> "description": "The temperature unit to use. Infer this from the users location.",
> },
> },
> "required": \["location", "format"\],
> },
> },
> }\]
> 
> client.chat\_completion(messages=messages, tools=tools, max\_tokens=500, temperature=0.3)
> 
> \# this works great and produces a similar output:
> \# ChatCompletionOutput(choices=\[ChatCompletionOutputComplete(finish\_reason='stop', index=0, message=ChatCompletionOutputMessage(role='assistant', content=None, tool\_calls=\[ChatCompletionOutputToolCall(function=ChatCompletionOutputFunctionDefinition(arguments={'format': 'celsius', 'location': 'San Giustino, Italy'}, name='get\_current\_weather', description=None), id='0', type='function')\]), logprobs=None)\], ...)
> 
> \# TRYING TO PASS BACK TOOL CALLS AND TOOL RESULT
> new\_messages = \[el for el in messages\]
> id\_ = "9Ae3bDc2F" # fake ID needed to use Mistral models
> 
> tool\_call = {"name": "get\_current\_weather", "arguments": {"location": "San Giustino, Italy", "format": "celsius"}}
> new\_messages.append({"role": "assistant", "content":"", "tool\_calls": \[{"type": "function", "function": tool\_call, "id": id\_}\]})
> new\_messages.append({"role": "tool", "name": "get\_current\_temperature", "content": "22.0", "tool\_call\_id": id\_})
> 
> client.chat\_completion(messages=new\_messages, tools=tools, max\_tokens=500, temperature=0.3)
> 
> \# HfHubHTTPError: 422 Client Error: Unprocessable Entity for url: https://api-inference.huggingface.co/models/mistralai/Mistral-Nemo-Instruct-2407/v1/chat/completions (Request ID: ...)
> \# Template error: unknown filter: filter string is unknown (in \<string\>:79)
> \`\`\`
> 
> \### System info
> 
> \`\`\`shell
> \- huggingface\_hub version: 0.25.2
> \`\`\`

---

<div class="post-metadata">

**Author:** ![anomalus](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/anomalus/32/17744_2.png) [@anomalus](https://discuss.huggingface.co/u/anomalus)\
**Post date:** [February 3, 2025, 2:23pm UTC](https://discuss.huggingface.co/t/function-calling-not-working-with-inference-clients-on-seemingly-any-model/138581/8 "2025-02-03T14:23:53Z")

</div>

Glad to see someone can reproduce it! With OpenAI at least, the expected behavior is to get an “assistant” message with a text content that presents a human-readable response, like “The temperature is 4ºC.”

Thanks for the GitHub link; I’ll look through it. I opened a [similar bug](https://github.com/huggingface/text-generation-inference/issues/2986) in the inference client repo; HF is a big place so I don’t know if that was the right repo.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [February 3, 2025, 3:02pm UTC](https://discuss.huggingface.co/t/function-calling-not-working-with-inference-clients-on-seemingly-any-model/138581/9 "2025-02-03T15:02:27Z")

</div>

Even within the InferenceClient class, chat\_completion is quite special because it is modeled on the OpenAI API. For now, the return value needs to be plain like this.

```py
output = client.chat_completion(messages=messages, tools=tools, max_tokens=500, temperature=0.3)
print(output.choices[0].message.tool_calls[0].function)
print(output.choices[0].message.content)

output = client.chat_completion(messages=messages, max_tokens=500, temperature=0.3)
print(output.choices[0].message.content)

```

```auto
ChatCompletionOutputFunctionDefinition(arguments={'format': 'celsius', 'location': 'San Giustino, Italy'}, name='get_current_weather', description=None)

None

The current temperature in San Giustino, Italy, is 22.0 degrees Celsius. However, please note that this information might change depending on the time of day and season. For the most accurate and up-to-date weather information, you can check a reliable weather website or app.

```

> **[Inference](https://huggingface.co/docs/huggingface_hub/package_reference/inference_client#huggingface_hub.InferenceClient.chat_completion.tools)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

> **[Function Calling](https://huggingface.co/docs/hugs/en/guides/function-calling)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

It’s probably buggy.

---

<div class="post-metadata">

**Author:** ![anomalus](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/anomalus/32/17744_2.png) [@anomalus](https://discuss.huggingface.co/u/anomalus)\
**Post date:** [February 3, 2025, 3:18pm UTC](https://discuss.huggingface.co/t/function-calling-not-working-with-inference-clients-on-seemingly-any-model/138581/10 "2025-02-03T15:18:17Z")

</div>

Hmm, very interesting. So one omits the “tools=tools” once manually determining the calls are satisfied…

I’m quite familiar with the OpenAI API, and there are some quirks in HF that I’m obviously stumbling through. For example, in OpenAI, the tools array can always be provided, in case subsequent function calls are needed based on the result.

But I think I can work through that quirk for the time being. Thanks @John6666!

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [February 3, 2025, 3:19pm UTC](https://discuss.huggingface.co/t/function-calling-not-working-with-inference-clients-on-seemingly-any-model/138581/11 "2025-02-03T15:19:12Z")

</div>

Sorry, I was wrong. I think the Inference API is buggy.  
The function is defined correctly. When that happens, None is returned.

Edit:  
I reported the symptoms on Discord.

---

<div class="post-metadata">

**Author:** ![eight0805](https://avatars.discourse-cdn.com/v4/letter/e/ce7236/32.png) [@eight0805](https://discuss.huggingface.co/u/eight0805)\
**Post date:** [February 7, 2025, 8:52pm UTC](https://discuss.huggingface.co/t/function-calling-not-working-with-inference-clients-on-seemingly-any-model/138581/12 "2025-02-07T20:52:18Z")

</div>

Is there any update on this issue, I am facing the same issue and trying to find a walkaround.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [February 7, 2025, 9:00pm UTC](https://discuss.huggingface.co/t/function-calling-not-working-with-inference-clients-on-seemingly-any-model/138581/13 "2025-02-07T21:00:30Z")

</div>

So far, there’s been no progress.

---

<div class="post-metadata">

**Author:** ![anomalus](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/anomalus/32/17744_2.png) [@anomalus](https://discuss.huggingface.co/u/anomalus)\
**Post date:** [February 8, 2025, 5:02pm UTC](https://discuss.huggingface.co/t/function-calling-not-working-with-inference-clients-on-seemingly-any-model/138581/14 "2025-02-08T17:02:25Z")

</div>

I’ve had some [acknowledgement on the GitHub issue I filed](https://github.com/huggingface/text-generation-inference/issues/2986#issuecomment-2643403042).

I’ve tried a workaround that entails omitting the “tools” argument on subsequent instantiations of the chat completion object, and it seems to work for now, at least for simple tool situations.

Vision models also fail in weird ways, when sending messages with an image\_url specified. The workaround is the same.

Both of these situations require special hacky logic for HF inference, [like this](https://github.com/spatialpixel/funkify/blob/4229858c009c4b5d4194a2076e46ac57daa9f219/src/service-huggingface.js#L39).