# OverflowML: Auto-optimal model loading for any hardware

**URL:** <https://discuss.huggingface.co/t/overflowml-auto-optimal-model-loading-for-any-hardware/174144>\
**Category:** Show and Tell\
**Created:** [March 10, 2026, 8:13pm UTC](https://discuss.huggingface.co/t/overflowml-auto-optimal-model-loading-for-any-hardware/174144 "2026-03-10T20:13:07Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Khaeldur](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/khaeldur/32/58515_2.png) [@Khaeldur](https://discuss.huggingface.co/u/Khaeldur)\
**Post date:** [March 10, 2026, 8:13pm UTC](https://discuss.huggingface.co/t/overflowml-auto-optimal-model-loading-for-any-hardware/174144/1 "2026-03-10T20:13:07Z")

</div>

Sharing a library I built to solve the “model too big for GPU” problem automatically.

**Problem:** Loading large models requires knowing which combination of device\_map, quantization, and offloading to use — and it varies by hardware. FP8 doesn’t work with CPU offload on Windows. INT4 needs bitsandbytes. Sequential offload and attention\_slicing crash together.

**Solution:**

```python
import overflowml

# Detects your hardware, picks strategy, loads with optimal config
model, tokenizer = overflowml.load_model("meta-llama/Llama-3-70B")

```

Under the hood it:

- Detects GPU type, VRAM, RAM, FP8/BF16 support
- Estimates model size from config (no weight download needed)
- Picks the best strategy: direct load, FP8, BitsAndBytes INT4/INT8, model\_cpu\_offload, or sequential\_cpu\_offload
- Sets up device\_map, max\_memory, quantization\_config automatically
- Avoids known incompatibilities

Also works with diffusers pipelines:

```python
overflowml.optimize_pipeline(pipe, model_size_gb=40)

```

CLI tool included:

```auto
$ overflowml benchmark # shows what models your hardware can run
$ overflowml plan 70 # detailed strategy for a 70GB model
$ overflowml detect # show hardware capabilities

```

Cross-platform: NVIDIA (CUDA), Apple Silicon (MPS/MLX unified memory), AMD (ROCm planned).

`pip install overflowml[transformers]`

GitHub: [GitHub - Khaeldur/overflowml: Run AI models larger than your GPU. Auto-detects hardware, picks optimal memory strategy. · GitHub](https://github.com/Khaeldur/overflowml)  
PyPI: [overflowml · PyPI](https://pypi.org/project/overflowml/)