# FitCheck — Estimate LLM Training and Serving VRAM Before You Run **URL:** \ **Category:** Show and Tell\ **Created:** [September 25, 2026, 1:57pm UTC](https://discuss.huggingface.co/t/fitcheck-estimate-llm-training-and-serving-vram-before-you-run/180737 "2026-09-25T13:57:33Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![mlanvvs](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mlanvvs/32/63357_2.png) [@mlanvvs](https://discuss.huggingface.co/u/mlanvvs)\ **Post date:** [September 25, 2026, 1:57pm UTC](https://discuss.huggingface.co/t/fitcheck-estimate-llm-training-and-serving-vram-before-you-run/180737/1 "2026-09-25T13:57:33Z")
![fitcheck-overview](https://us1.discourse-cdn.com/hellohellohello/original/3X/c/1/c16a2cbce30c93512437390a52c9bb6bc630960c.svg) Hi everyone, I built **FitCheck** to answer a common question before starting an LLM job: **Will this configuration fit in my GPU’s memory?** FitCheck estimates peak VRAM for LoRA, QLoRA, and full fine-tuning. It reports the memory breakdown, usable GPU capacity, headroom, fit verdict, and largest estimated micro-batch that fits. It also supports serving estimates based on model weights and KV cache, plus an advisor that explores batch size, sequence length, and LoRA rank. FitCheck reads the model’s Hugging Face `config.json` and parameter-count metadata. It does not download model weights or require PyTorch, CUDA, or a GPU to run the estimate. I tested the estimator against real GPU measurements. Across 57 calibration and repeat runs, the full-process estimate has a **2.4% mean absolute error** and a **13.9% worst absolute error**. A separate 12-run holdout has a **5.4% mean absolute error** and produced **12/12 correct fit-boundary verdicts** for the tested setup. The current measurements are all from one Tesla T4, so this is not a universal accuracy claim. FitCheck is a sizing tool, not a guarantee that every workload will avoid an OOM. - **Try the Space:** [FitCheck - a Hugging Face Space by mlanvvs](https://huggingface.co/spaces/mlanvvs/fitcheck) - **GitHub and technical details:** [GitHub - Anassbzdd/fitcheck: See if your LLM fits in your GPU's VRAM before loading it — no more guessing or OOM errors. · GitHub](https://github.com/Anassbzdd/fitcheck) - **Install:** `pip install fitcheck-llm` I would appreciate feedback, especially: - Is the result easy to understand? - Which models or configurations should I test next? - Does the estimate match your measured GPU usage? Positive or critical feedback is welcome. My goal is to find where the estimator is wrong and improve it with reproducible measurements. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [September 28, 2026, 8:20am UTC](https://discuss.huggingface.co/t/fitcheck-estimate-llm-training-and-serving-vram-before-you-run/180737/2 "2026-09-28T08:20:48Z")
For now, I ran some measurements on an L4: * * * Since the current [measurement archive](https://github.com/Anassbzdd/fitcheck/blob/4da83a58f95864003734f5e9fd92af1756c0d685/data/measurements/README.md) is still T4-only, I focused mostly on your “does the estimate match measured GPU usage?” question and tried a small second-GPU panel on a Colab NVIDIA L4. I pinned FitCheck to [this commit](https://github.com/Anassbzdd/fitcheck/tree/4da83a58f95864003734f5e9fd92af1756c0d685) and tested BF16 with: - Qwen2.5-Coder-1.5B and TinyLlama-1.1B - NF4 QLoRA and ordinary non-quantized LoRA - eager attention and real `flash_attention_2` - batch/sequence variations, including a same-session 2×2 grid Across three sessions I got 17 successful measurement rows covering 12 unique configurations. I used repeats only as repeatability checks, not as extra independent samples. The short version is: | subset | tensor-tier error | full-process error | | --- | --- | --- | | Qwen anchor cases | -0.44% to -0.08% | -2.72% to +2.20% | | TinyLlama eager, 2×2 grid | -0.95% to +2.83% | -7.83% to +9.06% | | TinyLlama real FA2, 2×2 grid | -0.91% to -0.75% | -8.61% to +1.87% | Here, signed error is `(predicted - measured) / measured`, so negative means under-prediction. So, at least in this small L4/BF16 panel, the **tensor accounting transferred quite well** , including the real FlashAttention-2 path. The larger residual showed up mainly in the **full-process tier** instead. That seems consistent with the separation already present in FitCheck: the physical tensor terms and the card/runtime overhead are different problems. At the tested commit, L4 has no fitted overhead profile, so it falls back to the [unmeasured default profile](https://github.com/Anassbzdd/fitcheck/blob/4da83a58f95864003734f5e9fd92af1756c0d685/fitcheck/overhead_db.py). In these measurements, that residual was shape-dependent and went in both directions, rather than looking like one simple constant offset. If I were choosing one next step, I would probably use these as **second-card calibration/holdout candidates before adding another random model**. I would still keep some rows out of the fit rather than fitting and grading on the same L4 points. Also, on the presentation side: the component breakdown was clear enough for me to follow. In practice, the tensor/process split was especially useful because it made the location of the larger miss much easier to see. If useful, I can also provide the raw `measure.py --json` rows, environment capture, and the external process-memory traces. > **Test setup and full measurements** > > ### Environment > > The effective environment was the same across the three probe sessions: > > - GPU: NVIDIA L4 (`sm_89`) > - reported device memory: 23,034 MiB > - NVIDIA driver: 580.82.07 > - Python: 3.13.15 > - PyTorch: 2.11.0+cu128 > - CUDA reported by PyTorch: 12.8 > - Transformers: 5.17.0 > - PEFT: 0.20.0 > - bitsandbytes: 0.50.2 > - Accelerate: 1.14.0 > - `huggingface_hub`: 1.29.0 > - FitCheck source: [`4da83a58f95864003734f5e9fd92af1756c0d685`](https://github.com/Anassbzdd/fitcheck/tree/4da83a58f95864003734f5e9fd92af1756c0d685) > > The real FA2 runs used `flash-attn` 2.8.3 and actually selected `flash_attention_2`, rather than the T4 SDPA memory-efficient stand-in. > > That path is within the upstream FlashAttention-2 CUDA support envelope: the [official implementation](https://github.com/Dao-AILab/flash-attention) lists Ampere/Ada/Hopper and fp16/bf16 as supported. > > ### Qwen anchor > > Model: [Qwen/Qwen2.5-Coder-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct) > > | configuration | tensor error | process error | > | --- | --- | --- | > | NF4, BF16, eager, bs=2, seq=1024 | -0.294% | +0.319% | > | NF4, BF16, eager, bs=2, seq=2048 | -0.442% | -2.723% | > | NF4, BF16, real FA2, bs=2, seq=1024 | -0.177% | +2.197% | > | non-quantized LoRA, BF16, eager, bs=2, seq=1024 | -0.080% | +0.112% | > > The QLoRA rows used rank 32, the standard `[q,k,v,o]` target set, AdamW, FP32 optimizer states, and gradient checkpointing. > > One representative equivalent command is: > > ```bash > python scripts/measure.py Qwen/Qwen2.5-Coder-1.5B-Instruct \ > --gpu l4 \ > --qlora \ > --precision bf16 \ > --lora-r 32 \ > --batch-size 2 \ > --seq-len 1024 \ > --optimizer adamw \ > --json > > ``` > > and the FA2 version adds: > > ```bash > --flash-attn > > ``` > > ### TinyLlama diagnostic grid > > Model: [TinyLlama/TinyLlama-1.1B-Chat-v1.0](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0) > > I used TinyLlama because its smaller vocabulary makes the eager attention/layer hump much easier to expose as sequence length grows. The final probe ran the complete `batch ∈ {1,2}` × `seq ∈ {1024,2048}` grid in one session, under both eager and real FA2. > > Common settings: > > - NF4 QLoRA > - BF16 compute > - LoRA rank 32 > - standard `[q,k,v,o]` targets > - AdamW / FP32 optimizer state > - gradient checkpointing > - no double quantization > > #### Eager > > | batch | seq | tensors predicted | tensors measured | tensor error | process predicted | process measured | process error | > | --- | --- | --- | --- | --- | --- | --- | --- | > | 1 | 1024 | 2003.1 MiB | 2019.4 MiB | -0.81% | 2596.4 MiB | 2636 MiB | -1.50% | > | 2 | 1024 | 2848.6 MiB | 2875.8 MiB | -0.95% | 3484.2 MiB | 3780 MiB | -7.83% | > | 1 | 2048 | 4000.6 MiB | 3908.2 MiB | +2.36% | 4693.8 MiB | 4304 MiB | +9.06% | > | 2 | 2048 | 6843.6 MiB | 6655.0 MiB | +2.83% | 7678.9 MiB | 7888 MiB | -2.65% | > > #### Real FlashAttention-2 > > | batch | seq | tensors predicted | tensors measured | tensor error | process predicted | process measured | process error | > | --- | --- | --- | --- | --- | --- | --- | --- | > | 1 | 1024 | 1833.6 MiB | 1847.6 MiB | -0.76% | 2418.4 MiB | 2374 MiB | +1.87% | > | 2 | 1024 | 2509.6 MiB | 2528.5 MiB | -0.75% | 3128.2 MiB | 3140 MiB | -0.38% | > | 1 | 2048 | 2509.6 MiB | 2529.0 MiB | -0.77% | 3128.2 MiB | 3140 MiB | -0.38% | > | 2 | 2048 | 3861.6 MiB | 3897.0 MiB | -0.91% | 4547.8 MiB | 4976 MiB | -8.61% | > > The largest absolute tensor-tier error I saw anywhere in the successful L4 rows was about 2.8%. > > I would **not** turn that into an “L4 accuracy = X%” result, though: this was a small, deliberately diagnostic panel, not a random independent benchmark. > **Where the remaining error seems to live** > > I kept FitCheck’s own distinction between the physical tensor terms and the full-process total, because the two behaved quite differently. > > For the final TinyLlama grid, the tensor prediction tracks the batch/sequence interaction fairly closely. > > I used the ordinary 2×2 interaction here — effectively, how much the batch-size effect changes when sequence length changes. > > ### Eager interaction > > Tensor tier: > > - predicted interaction: **+1997.5 MiB** > - measured interaction: **+1890.4 MiB** > > Process-overhead residual: > > - predicted interaction: **+99.9 MiB** > - observed diagnostic residual: **+549.6 MiB** > > ### FA2 interaction > > Tensor tier: > > - predicted interaction: **+676.0 MiB** > - measured interaction: **+687.1 MiB** > > Process-overhead residual: > > - predicted interaction: **+33.8 MiB** > - observed diagnostic residual: **+382.9 MiB** > > So the larger shape-dependent discrepancy appears **after** the tensor tier. > > I am deliberately calling that a process/runtime-overhead residual rather than saying “fragmentation is the cause.” The measurements do not uniquely identify the mechanism. > > For localization, I used: > > ```plaintext > diagnostic process-overhead residual > = harness process total > - measured peak tensor allocation > > ``` > > That is useful diagnostically, but it should not be interpreted as two values that necessarily peaked at exactly the same instant. > > This distinction is also consistent with PyTorch’s own CUDA memory model. PyTorch separates memory occupied by tensors (`memory_allocated` / `max_memory_allocated`) from memory managed by its caching allocator (`memory_reserved` / `max_memory_reserved`). See [CUDA memory management](https://docs.pytorch.org/docs/main/notes/cuda.html#cuda-memory-management). > > ### External process-memory cross-check > > I also sampled process GPU memory independently through NVML every 50 ms. > > Across the successful rows, the sampled NVML peak was consistently **8 MiB below** the FitCheck harness’s process total. > > I would not treat a 50 ms sampler as an oracle — it can miss a very short transient — but that stable relationship makes a large measurement-definition artifact less likely in this particular panel. > > PyTorch’s [CUDA memory debugging documentation](https://docs.pytorch.org/docs/main/torch_cuda_memory.html) recommends comparing against raw device memory when allocations outside the PyTorch allocator are relevant, which is why I added this cross-check. > > ### Cross-session repeatability > > Four TinyLlama configurations were independently repeated in a later session and reproduced the same headline process totals: > > - eager, bs=2, seq=1024: **3780 MiB** > - eager, bs=1, seq=2048: **4304 MiB** > - FA2, bs=2, seq=1024: **3140 MiB** > - FA2, bs=1, seq=2048: **3140 MiB** > > The harness-reported CUDA context was also **248 MiB** in every cell of the final eight-run grid. > > That does not prove the behavior is universal or deterministic under another driver/runtime, but it makes the observed shape pattern look less like a one-off allocation-noise event. > > ### How I read this > > At the tested commit, [`OVERHEAD_DB`](https://github.com/Anassbzdd/fitcheck/blob/4da83a58f95864003734f5e9fd92af1756c0d685/fitcheck/overhead_db.py) has measured profiles for T4 × kernel × quantization, while an L4 falls back to the unmeasured default overhead profile. > > So my current reading would be: > > 1. the tensor formulas transferred quite well to this L4/BF16 panel; > 2. real FA2 also behaved well at the tensor tier; > 3. the uncalibrated full-process overhead is where the larger L4 residual remains; > 4. that residual depends on batch/sequence shape, so it is not obviously repairable by only changing one fixed context constant. > > This seems more like useful input for the existing card/kernel-specific calibration layer than evidence against the overall decomposition. > **What I would and would not conclude / possible next steps** > > ### What I think the measurements support > > For **this exact L4/runtime/model/config panel** : > > - the tensor-tier estimates were close; > - NF4 and ordinary LoRA both worked well in the tested Qwen points; > - the real `flash_attention_2` tensor estimates were close; > - eager attention’s larger sequence-dependent memory behavior was broadly captured; > - the main remaining error was concentrated in the full-process tier; > - that process residual could be either positive or negative depending on shape. > > This also lines up fairly well with the project’s existing architecture: the [measurement archive](https://github.com/Anassbzdd/fitcheck/blob/4da83a58f95864003734f5e9fd92af1756c0d685/data/measurements/README.md) already treats the GPU/kernel/quantization overhead calibration separately, and explicitly calls a second card the remaining gap. > > ### What I would _not_ conclude > > I would not use these measurements to claim: > > - a universal FitCheck accuracy number for L4; > - an Ada-wide accuracy number; > - that allocator fragmentation is proven to be the root cause; > - that an L4 overhead profile is already fully characterized; > - that the same result holds under other PyTorch / Transformers / PEFT / driver versions; > - that inference/serving has been validated on L4; > - that INT8, checkpointing-off, full fine-tuning, MoE, multimodal, compiled, or multi-GPU cases behave the same way; > - that L4 fit/no-fit boundary accuracy has been validated. > > I also would not count the repeated configurations as additional independent accuracy samples. > > ### What I would do next > > If the goal is still “find where the estimator is wrong with reproducible measurements,” I probably would **not** jump immediately to another unrelated model. > > A higher-information path seems to be: > > 1. treat some of these L4 rows as candidate second-card calibration data; > 2. keep other L4 rows as a genuine holdout; > 3. see how much of the process residual an L4-specific `(GPU, kernel, quantization)` profile absorbs; > 4. only if the residual remains interesting, collect allocator-level diagnostics. > > For the last branch, PyTorch exposes useful counters through [`torch.cuda.memory_stats()`](https://docs.pytorch.org/docs/stable/generated/torch.cuda.memory.memory_stats.html), including allocated/reserved bytes and segment counts. The CUDA memory snapshot tooling can then give a more detailed allocation history. > > I would treat that as a diagnostic option rather than assuming in advance that fragmentation is the explanation. > > A separate L4 boundary/OOM holdout would also be useful eventually, because sizing tools care more about a small **under-prediction near the fit boundary** than about the same percentage error when there is plenty of headroom. But I think the current rows already answer the immediate cross-GPU question well enough that this does not need to block anything. > > ### Raw reproducibility data > > If these measurements are useful for the project, I can provide the raw `scripts/measure.py --json` outputs rather than only the summarized tables. > > I also kept: > > - exact FitCheck commit > - full Python/package environment > - raw stderr/stdout > - NVML process-memory samples > - model config embedded in each measurement row > - checksums for the result artifacts > > so the rows should be reasonably straightforward to inspect or re-score without relying on the summary above.