# Batch inference with huggingface\_hub for serverless providers

**URL:** <https://discuss.huggingface.co/t/batch-inference-with-huggingface-hub-for-serverless-providers/170649>\
**Category:** Beginners\
**Created:** [November 19, 2025, 7:11pm UTC](https://discuss.huggingface.co/t/batch-inference-with-huggingface-hub-for-serverless-providers/170649 "2025-11-19T19:11:13Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [November 19, 2025, 11:30pm UTC](https://discuss.huggingface.co/t/batch-inference-with-huggingface-hub-for-serverless-providers/170649/2 "2025-11-19T23:30:09Z")

</div>

Whether batch inference is supported depends on the endpoint. I don’t think many HF endpoints support it…

* * *

Short, direct answer first, then details:

- There is **no direct equivalent** of OpenAI’s Batch API or Vertex AI’s Gemini batch prediction (with special discounted pricing and job objects) on **Hugging Face serverless Inference Providers** via `huggingface_hub`.

- Hugging Face serverless (HF Inference + other Providers) is priced as a **normal online API** : you pay for compute time per request, not a separate “batch mode” with cheaper tokens.([Hugging Face](https://huggingface.co/docs/inference-providers/en/pricing "Pricing and Billing"))

- If you want “batch-like” behavior and cost reductions, you have to use:

Below, I’ll spell out:

1. What OpenAI / Gemini batch actually give you (so we can compare).

2. How Hugging Face serverless Inference Providers work.

3. Concrete ways to reduce cost with Hugging Face:

* * *

## 1. What you’re doing with OpenAI / Gemini batch

With OpenAI and Gemini:

- **OpenAI Batch API** :

- **Vertex AI Gemini Batch prediction** :

So your mental model now is:

> “I have a large offline workload. I can pay less per token by submitting it as a batch job instead of hammering the online API.”

That’s exactly what we need to compare against on Hugging Face.

* * *

## 2. How Hugging Face serverless Inference Providers work

Hugging Face’s **Inference Providers** layer is a unified router over many serverless providers (HF Inference, Together, SambaNova, etc.), exposed via the `InferenceClient` and OpenAI-compatible endpoints.([Hugging Face](https://huggingface.co/docs/inference-providers/en/index "Inference Providers"))

Key points:

- **HF Inference** is the serverless Inference API that used to be called “Inference API (serverless)”. It’s now just one provider (“hf-inference”) behind the router.([Hugging Face](https://huggingface.co/docs/inference-providers/en/providers/hf-inference "HF Inference"))

- **Pricing model for HF Inference** :

- **Other Inference Providers** (Together, SambaNova, etc.) are also billed pay-as-you-go. HF’s own forum answers state that “usage fees for each Inference Provider’s endpoint would apply directly.”([Hugging Face Forums](https://discuss.huggingface.co/t/hf-routed-inference-pricing/167789 "HF routed Inference Pricing? - Models"))

Important for your question:

- The Inference Providers docs and pricing pages **do not define any “Batch API”, “batch jobs”, or special discounted batch pricing**.
- The `InferenceClient` reference provides methods like `chat_completion`, `text_generation`, `embeddings`, etc., but again **no `create_batch_job` / `get_batch_result` API**.([Hugging Face](https://huggingface.co/docs/huggingface_hub/en/package_reference/inference_client "Inference"))

So:

> For serverless Inference Providers, Hugging Face only exposes **online** (synchronous or streaming) APIs, not an asynchronous batch job API with separate discounted pricing.

There is no 1:1 equivalent of “OpenAI Batch API” or “Vertex Batch prediction” at the Provider-router level.

* * *

## 3. Options to reduce cost with Hugging Face

Even though there is no direct batch-discount API, you can still reduce effective cost per token with a combination of **model choice** , **infrastructure choice** , and **usage patterns**.

Think of three levels:

1. **Within serverless Providers** : what you can do while staying purely “serverless”.
2. **Offline jobs on Hugging Face** : HF Jobs for batch inference (closest conceptual match to Vertex batch).
3. **Self-host / third-party infra** : vLLM / TEI / other offline inference, still using HF models.

I’ll walk through each, plus general token-saving techniques.

* * *

### 3.1 Inside serverless Providers: efficiency, not discounts

Within the serverless Providers themselves, you mainly have two levers:

#### 3.1.1 Choose cheaper models and cheaper providers

HF Inference pricing is compute-based: **hardware-time × price**.([Hugging Face](https://huggingface.co/docs/inference-providers/en/pricing "Pricing and Billing"))

Implications:

- Smaller / more efficient models (e.g. 1–8B LLMs, distilled models, specialized classifiers) will:

You can:

- Use the model selector on [huggingface.co](http://huggingface.co) and check which Providers are available and what their prices are.([Hugging Face](https://huggingface.co/docs/hub/en/models-inference "Inference Providers"))

- In the router, you can choose:

This is analogous to switching from GPT-4 Turbo to GPT-4o-mini: you’re changing the **model** as your main cost lever.

#### 3.1.2 Client-side micro-batching and concurrency

This does not change the **price per token** , but it can:

- Increase throughput.
- Reduce wall-clock time.
- Potentially lower effective cost if you’re billed in coarse blocks of compute time by the provider.

Two main approaches:

1. **Batch multiple inputs into a single call** , if the task supports it (e.g. some classification or embedding endpoints accept lists).

2. **Use `AsyncInferenceClient` to run many requests in parallel** with a controlled concurrency limit:

Again: **this improves throughput and utilization** , not pricing tiers. But if your load is modest and you just want to process offline datasets faster, this might already be “good enough” without leaving serverless.

* * *

### 3.2 HF Jobs: closest analogue to Vertex batch prediction

If you want something that genuinely looks like “batch jobs”, you should look at **Hugging Face Jobs** , which is a separate feature from Providers.

From the official Jobs docs and CLI guide:([Hugging Face](https://huggingface.co/docs/huggingface_hub/en/guides/jobs "Run and manage Jobs"))

- HF Jobs lets you run containerized code (e.g. Python scripts) on HF-managed CPUs/GPUs.
- The docs explicitly list **“Batch Inference: Run offline inference on thousands of samples using optimized GPU setups”** as a primary use case.
- Other use cases include training, data processing, and synthetic data generation.

Conceptually, HF Jobs behaves like:

> “Docker run on Hugging Face GPUs” with job scheduling and logs.

How this differs from serverless Providers:

- You are not billed per Provider request; you’re billed for **compute instance time** (e.g. `a10g-small` for N hours).
- You pull models from the Hub (or your private repos) and run inference in your code.
- You control batching (batch size, data loader, etc.), so you can approach the theoretical throughput of the hardware.

This is much closer to Vertex batch prediction than Providers are.

Very rough workflow:

1. Put your batch inference script in a repo:

2. In `run.py`, write your offline inference loop using `transformers`, vLLM, or TEI, reading from a dataset and writing out predictions.

3. Launch a job:

Advantages for “token cost”:

- By running on your own job, you can:

The HF Jobs docs and related blog posts show examples of this, including large-scale LLM batch inference with vLLM.([Hugging Face](https://huggingface.co/docs/huggingface_hub/en/guides/jobs "Run and manage Jobs"))

* * *

### 3.3 Self-hosted vLLM / TEI / local endpoints (still using HF models)

HF’s general inference guide says you can point `InferenceClient` to a **local or remote server** running vLLM, TGI, llama.cpp, etc., as long as it uses an OpenAI-compatible API.([Hugging Face](https://huggingface.co/docs/huggingface_hub/en/guides/inference "Run Inference on servers"))

Example pattern from the docs:

```python
from huggingface_hub import InferenceClient

# Example server: vLLM / TGI / litellm proxy exposing OpenAI-style /v1/chat/completions
# See: https://huggingface.co/docs/huggingface_hub/en/guides/inference # noqa: E501

client = InferenceClient(model="http://localhost:8080")

resp = client.chat_completion(
    model="meta-llama/Llama-3.2-3B-Instruct",
    messages=[{"role": "user", "content": "Short answer: why batch inference?"}],
    max_tokens=128,
)
print(resp.choices[0].message.content)

```

With something like **vLLM** :

- The docs describe an **offline inference mode** where you create a `LLM` object, load an HF model, and pass lists of prompts, with strong throughput as batch size increases.([docs.vllm.ai](https://docs.vllm.ai/en/v0.11.0/serving/offline_inference.html "Offline Inference - vLLM"))
- Benchmarks show vLLM significantly outperforming plain Transformers for offline inference, especially for large batches.

You can deploy this:

- On your own infra (Kubernetes, bare-metal, etc.).
- On other clouds (GCP / AWS / Azure) using managed job systems (Cloud Run Jobs, SageMaker Processing, etc.).
- Or _inside_ HF Jobs, combining the job system with vLLM for maximum throughput.

Cost-wise:

- You pay for compute (GPU/CPU) directly (cloud or HF Jobs).
- If you keep the GPU highly utilized with large batches, your effective cost per token is often much lower than serverless.

This is essentially “build your own Vertex Batch but using open models from Hugging Face and a general-purpose job system.”

* * *

### 3.4 General token-usage reduction strategies (work regardless of provider)

These are independent of whether you use serverless Providers, HF Jobs, or your own infra. They directly reduce tokens → reduce cost.

1. **Shorter prompts and contexts** :

2. **Control `max_new_tokens` / `max_tokens`** :

3. **Specialized small models instead of big general LLMs** :

4. **Embeddings** :

5. **Caching and deduplication** :

Even if HF does not give you a “batch token discount,” these simple steps can easily cut your total token usage by 2–5×, which usually dominates any 50% discount you get from Batch in other ecosystems.

* * *

## 4. Putting it together: mapping your current pattern to HF

Given how you’re already using OpenAI and Gemini batch, here’s a practical mapping to Hugging Face:

1. **If you want a pure serverless, “no infra” approach** :

2. **If you want a true batch-optimized workflow** (closest to Vertex Batch):

3. **If you already have cloud infra** (GCP / AWS / etc.):

In other words:

- **No direct Batch API / discount on HF serverless Providers.**

- **Yes, you can absolutely get “batch-style” cost savings by:**

---

_[View the full topic](https://discuss.huggingface.co/t/batch-inference-with-huggingface-hub-for-serverless-providers/170649)._