# CPU offloading error scenario

**URL:** <https://discuss.huggingface.co/t/cpu-offloading-error-scenario/175522>\
**Category:** 🤗Transformers\
**Created:** [April 24, 2026, 10:06am UTC](https://discuss.huggingface.co/t/cpu-offloading-error-scenario/175522 "2026-04-24T10:06:02Z")\
**Posts on this page:** 1\
**Showing post:** 5

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [April 24, 2026, 10:04pm UTC](https://discuss.huggingface.co/t/cpu-offloading-error-scenario/175522/5 "2026-04-24T22:04:38Z")

</div>

I’ll post a draft of the issue for now (edited):

* * *

The **good actual issues** to raise are these, in this order.

## Issue 1 — Primary: PEFT adapter loading fails only on an already CPU/GPU-dispatched bnb 4-bit Gemma 4 model

**File first at:** `huggingface/transformers`  
**Mention/cross-link:** `huggingface/peft`, `huggingface/accelerate`, `bitsandbytes-foundation/bitsandbytes`

### Suggested title

```plaintext
PeftModel.from_pretrained fails only on CPU/GPU-dispatched 4-bit Gemma4: Params4bit. __new__ got unexpected _is_hf_initialized

```

### Why this is the strongest issue

This is the core, now updated with the latest Colab T4 evidence:

```plaintext
R05 all-GPU 4-bit:
PASS

R03 split CPU/GPU 4-bit:
FAIL_BNB_PARAM_CONSTRUCTOR
TypeError: Params4bit. __new__ () got an unexpected keyword argument '_is_hf_initialized'

```

The important contrast is:

```plaintext
Works:
device_map = {"": 0}

Fails:
device_map = {
    "model.vision_tower": "cpu",
    "model.audio_tower": "cpu",
    "": 0,
}

```

This means the current primary failure is not PEFT + bitsandbytes 4-bit in general. The all-GPU path works. The failure is specific to loading a PEFT adapter on top of an already CPU/GPU-dispatched bitsandbytes 4-bit Gemma 4 base model.

The previous `Gemma4ClippableLinear` blocker is still important, but it is no longer the primary issue because the latest repro bypasses it by targeting the inner `.linear` modules in Gemma 4 multimodal towers.

### Current environment

```plaintext
Runtime: Colab Free / Tesla T4
Python: 3.12.13
torch: 2.10.0+cu128
transformers: 5.6.2
accelerate: 1.13.0
peft: 0.19.1
bitsandbytes: 0.49.2
huggingface_hub: 1.12.0
safetensors: 0.7.0
torchao_importable: false

```

### Model and adapter

```plaintext
Base model:
unsloth/gemma-4-E2B-it

Adapter:
Ayodele01/gemma-4-E2B-Gemini-3.1-Pro-Reasoning-Distill

```

Adapter probe:

```plaintext
adapter_model.safetensors keys: 786

audio: 72
language_like: 490
vision_or_patch: 224

```

This adapter is genuinely multimodal, so simply excluding vision/audio LoRA would change the adapter semantics.

### Why Transformers first

Transformers is still the best first repo because this issue crosses:

- Gemma 4 model integration;
- bitsandbytes quantization integration;
- PEFT adapter integration expectations;
- device-map loading behavior;
- Accelerate redispatch/hook behavior;
- current `_is_hf_initialized` parameter reconstruction behavior.

Accelerate owns `dispatch_model()` and hook attachment. bitsandbytes owns `Params4bit`. PEFT triggers the adapter-loading path. But the user-facing break is in the Transformers + PEFT + bnb integration path.

* * *

## Issue 2 — Supporting / separate if requested: PEFT target-module compatibility for Gemma4ClippableLinear

**File at:** `huggingface/peft`

### Suggested title

```plaintext
Gemma4 multimodal LoRA adapters need inner .linear targeting or Gemma4ClippableLinear support

```

### Why it is supporting, not primary

Before patching the local adapter config, PEFT fails earlier:

```plaintext
ValueError: Target module Gemma4ClippableLinear(...) is not supported.

```

However, the latest repro bypasses that blocker by targeting the inner `.linear` modules for Gemma 4 vision/audio towers. With that patch:

```plaintext
R05 all-GPU 4-bit:
PASS

```

So this is real, but it is not the primary split/offload failure.

* * *

## Issue 3 — Deferred / separate only if you have the exact trace: PEFT device\_map breaks Gemma 4 shared-KV generation

**File first at:** `huggingface/transformers`  
**Mention/cross-link:** `huggingface/peft`, `huggingface/accelerate`

### Suggested title

```plaintext
Gemma4 + PeftModel.from_pretrained(device_map=...) breaks shared_kv_states during generate

```

### Why this is deferred

The latest Colab E2B matrix did **not** reproduce this failure:

```plaintext
R05 all-GPU, no PEFT device_map:
PASS

R03 split CPU/GPU, no PEFT device_map:
FAIL_BNB_PARAM_CONSTRUCTOR before generation

```

Only file the shared-KV issue if you attach a separate trace where adapter loading succeeds and generation fails here:

```plaintext
Gemma4Attention.forward
→ shared_kv_states[self.kv_shared_layer_index]
→ KeyError: 22

```

* * *

## Optional Issue 4 — Docs / UX: PEFT offload-dir / offload-folder handling is confusing

**File at:** `huggingface/peft`

### Suggested title

```plaintext
Clarify offload_dir/offload_folder behavior for PeftModel.from_pretrained on already-dispatched models

```

### Why it is lower priority

This is useful, but it is not the core bug. The current core bug has a concrete split/offload `Params4bit` constructor trace.

* * *

# What I would not file

## Not this

```plaintext
CPU offloading is broken.

```

Too broad. The base model can load in split CPU/GPU form. The failure occurs during PEFT adapter loading on top of the already-dispatched quantized base model.

Better:

```plaintext
PeftModel.from_pretrained fails on an already CPU/GPU-dispatched bitsandbytes 4-bit Gemma 4 model during Accelerate hook setup.

```

## Not this

```plaintext
PEFT expects vision/audio towers to be on GPU.

```

The all-GPU path works, and the split path fails in a bnb/Accelerate parameter reconstruction path. The evidence points to split/offload redispatch, not a generic PEFT requirement that vision/audio must be GPU-resident.

## Not this as the main issue

```plaintext
Tensor.item() cannot be called on meta tensors

```

That may be a related double-quant / `QuantState.as_dict()` variant, but the latest controlled Colab matrix with `double_quant=False` lands on:

```plaintext
Params4bit. __new__ () got an unexpected keyword argument '_is_hf_initialized'

```

Mention the `nested_offset.item()` issue only if you attach that exact separate trace.

* * *

# Recommended filing plan

## Best plan

Open **one primary Transformers issue** :

```plaintext
PeftModel.from_pretrained fails only on CPU/GPU-dispatched 4-bit Gemma4: Params4bit. __new__ got unexpected _is_hf_initialized

```

Include the Gemma4ClippableLinear patch as a “repro setup note,” not as the headline.

Then say:

```plaintext
I can split the Gemma4ClippableLinear target-module compatibility issue into a PEFT issue if maintainers prefer.

```

## If you want the cleanest tracking

Open two separate issues:

1. **Transformers Issue A:** split/offload bnb 4-bit + PEFT + Accelerate `Params4bit. __new__ (_is_hf_initialized)` failure.
2. **PEFT Issue B:** Gemma 4 multimodal LoRA adapters and `Gemma4ClippableLinear` target-module compatibility.

Do not open the shared-KV issue from the latest Colab matrix alone, because it was not reproduced there.

* * *

# Key evidence to include

Include this exact contrast:

```plaintext
Works:
device_map = {"": 0}

Fails:
device_map = {
    "model.vision_tower": "cpu",
    "model.audio_tower": "cpu",
    "": 0,
}

```

Include this exact result pair:

```plaintext
R05_ALL_GPU_4BIT:
PASS
generate output shape: (1, 8)
CUDA after generate: allocated 6.507 GiB, reserved 6.693 GiB, free 7.742 GiB

R03_SPLIT_4BIT:
FAIL_BNB_PARAM_CONSTRUCTOR
TypeError: Params4bit. __new__ () got an unexpected keyword argument '_is_hf_initialized'

```

Include the adapter probe:

```plaintext
Ayodele01/gemma-4-E2B-Gemini-3.1-Pro-Reasoning-Distill

audio: 72
language_like: 490
vision_or_patch: 224

```

* * *

# Bottom line

The actual issue to raise now is:

1. **Primary bug:** `PeftModel.from_pretrained()` fails on a CPU/GPU-dispatched bitsandbytes 4-bit Gemma 4 base model during Accelerate hook setup, while the same patched multimodal adapter works all-GPU.

2. **Supporting bug:** vanilla PEFT currently trips over Gemma 4 multimodal tower `Gemma4ClippableLinear` wrappers unless the adapter targets inner `.linear` modules or PEFT supports the wrapper.

3. **Deferred bug:** passing `device_map` into PEFT may break Gemma 4 shared-KV generation, but this needs its own exact trace and should not be merged into the latest Colab E2B primary issue.

* * *

Below are **ready-to-paste GitHub sections**. The ready-to-copy bodies use an outer 4-backtick fence so the inner 3-backtick code fences remain intact when copied.

* * *

# Issue 1

## Target repo

```plaintext
huggingface/transformers

```

## Suggested title

```plaintext
PeftModel.from_pretrained fails only on CPU/GPU-dispatched 4-bit Gemma4: Params4bit. __new__ got unexpected _is_hf_initialized

```

## Suggested labels

```plaintext
bug, Gemma4, PEFT, Accelerate, bitsandbytes, quantization, device_map, cpu-offload

```

## Body

````markdown
### System Info

- Runtime: Colab Free / Tesla T4
- Python: 3.12.13
- torch: 2.10.0+cu128
- transformers: 5.6.2
- accelerate: 1.13.0
- peft: 0.19.1
- bitsandbytes: 0.49.2
- huggingface_hub: 1.12.0
- safetensors: 0.7.0
- torchao: not importable / removed from environment
- model: `unsloth/gemma-4-E2B-it`
- adapter: `Ayodele01/gemma-4-E2B-Gemini-3.1-Pro-Reasoning-Distill`
- quantization: bitsandbytes 4-bit NF4
- attention implementation: `sdpa`
- trust_remote_code: `False`

### Summary

I can load `unsloth/gemma-4-E2B-it` in bitsandbytes 4-bit, load a patched multimodal LoRA adapter with PEFT, and run a tiny `generate()` smoke test when the whole model is placed on GPU:

```python
device_map = {"": 0}
```

However, the same model/adapter path fails when the base model is loaded with a split CPU/GPU `device_map`:

```python
device_map = {
    "model.vision_tower": "cpu",
    "model.audio_tower": "cpu",
    "": 0,
}
```

The failure happens during `PeftModel.from_pretrained()`, after PEFT starts adapter loading and calls into Accelerate `dispatch_model()` / hook attachment. The failing path reconstructs a bitsandbytes `Params4bit` object and passes `_is_hf_initialized` into its constructor:

```text
TypeError: Params4bit. __new__ () got an unexpected keyword argument '_is_hf_initialized'
```

The contrast is important:

```text
R05 all-GPU 4-bit:
PASS

R03 split CPU/GPU 4-bit:
FAIL_BNB_PARAM_CONSTRUCTOR
```

This suggests the issue is not PEFT + bitsandbytes 4-bit in general. It appears specific to loading a PEFT adapter on top of an already CPU/GPU-dispatched bitsandbytes 4-bit Gemma 4 model, where PEFT triggers an additional Accelerate dispatch/hook path.

### Why the adapter config is patched in this repro

The adapter is a real multimodal LoRA adapter. Its safetensors keys include:

```text
audio: 72
language_like: 490
vision_or_patch: 224
```

Without patching, PEFT first fails earlier with:

```text
ValueError: Target module Gemma4ClippableLinear(...) is not supported.
```

That is because Gemma 4 vision/audio towers expose wrapper modules such as:

```text
model.vision_tower.encoder.layers.0.self_attn.q_proj
→ Gemma4ClippableLinear

model.vision_tower.encoder.layers.0.self_attn.q_proj.linear
→ Linear / Linear4bit
```

The adapter weights use inner `.linear` paths for multimodal towers, for example:

```text
base_model.model.model.audio_tower.layers.0.self_attn.k_proj.linear.lora_A.weight
base_model.model.model.vision_tower.encoder.layers.0.self_attn.q_proj.linear.lora_A.weight
```

So the repro patches only the local copy of `adapter_config.json` to target inner `.linear` modules for Gemma 4 multimodal towers while leaving language tower targets at the usual projection modules.

Patched target expression:

```text
.*(?:model\.language_model\.layers\.\d+\.(?:self_attn|mlp)\.(?:q_proj|k_proj|v_proj|o_proj|gate_proj|up_proj|down_proj)|model\.vision_tower\.encoder\.layers\.\d+\.(?:self_attn|mlp)\.(?:q_proj|k_proj|v_proj|o_proj|gate_proj|up_proj|down_proj)\.linear|model\.audio_tower\.layers\.\d+\.self_attn\.(?:q_proj|k_proj|v_proj)\.linear)$
```

After this patch, the all-GPU case works, which confirms the Gemma4ClippableLinear target-module issue is bypassed for this repro.

### Quantization config

```python
from transformers import BitsAndBytesConfig
import torch

quant_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=False,
    bnb_4bit_compute_dtype=torch.bfloat16,
    llm_int8_enable_fp32_cpu_offload=True,
)
```

### Case A — all-GPU path works

```python
device_map = {"": 0}
max_memory = {0: "14GiB"}

base_model = Gemma4ForConditionalGeneration.from_pretrained(
    "unsloth/gemma-4-E2B-it",
    quantization_config=quant_config,
    device_map=device_map,
    max_memory=max_memory,
    dtype=torch.bfloat16,
    attn_implementation="sdpa",
    trust_remote_code=False,
    low_cpu_mem_usage=True,
)

model = PeftModel.from_pretrained(
    base_model,
    "/content/patched_gemma4_multimodal_inner_linear_adapter_v4",
    is_trainable=False,
)

out = model.generate(**inputs, max_new_tokens=4, do_sample=False, use_cache=True)
```

Observed result:

```text
R05_ALL_GPU_4BIT: PASS
generate output shape: (1, 8)

CUDA after generate:
free_gib: 7.742
total_gib: 14.563
allocated_gib: 6.507
reserved_gib: 6.693
```

### Case B — split CPU/GPU path fails

```python
device_map = {
    "model.vision_tower": "cpu",
    "model.audio_tower": "cpu",
    "": 0,
}
max_memory = {0: "13GiB", "cpu": "10GiB"}

base_model = Gemma4ForConditionalGeneration.from_pretrained(
    "unsloth/gemma-4-E2B-it",
    quantization_config=quant_config,
    device_map=device_map,
    max_memory=max_memory,
    offload_folder="/content/gemma4_offload_v4",
    dtype=torch.bfloat16,
    attn_implementation="sdpa",
    trust_remote_code=False,
    low_cpu_mem_usage=True,
)

model = PeftModel.from_pretrained(
    base_model,
    "/content/patched_gemma4_multimodal_inner_linear_adapter_v4",
    is_trainable=False,
    offload_dir="/content/gemma4_offload_v4",
    offload_buffers=True,
    ephemeral_gpu_offload=True,
    torch_device="cuda:0",
)
```

Observed result:

```text
R03_SPLIT_4BIT: FAIL_BNB_PARAM_CONSTRUCTOR

TypeError: Params4bit. __new__ () got an unexpected keyword argument '_is_hf_initialized'
```

Trace tail:

```text
File ".../peft/peft_model.py", line 582, in from_pretrained
    load_result = model.load_adapter(

File ".../peft/peft_model.py", line 1475, in load_adapter
    dispatch_model(

File ".../accelerate/big_modeling.py", line 432, in dispatch_model
    attach_align_device_hook_on_blocks(

File ".../accelerate/hooks.py", line 540, in attach_align_device_hook
    add_hook_to_module(module, hook, append=True)

File ".../accelerate/hooks.py", line 183, in add_hook_to_module
    module = hook.init_hook(module)

File ".../accelerate/hooks.py", line 330, in init_hook
    set_module_tensor_to_device(module, name, "meta")

File ".../accelerate/utils/modeling.py", line 363, in set_module_tensor_to_device
    new_value = param_cls(new_value, requires_grad=old_value.requires_grad, **kwargs).to(

TypeError: Params4bit. __new__ () got an unexpected keyword argument '_is_hf_initialized'
```

### Expected behavior

One of the following:

1. `PeftModel.from_pretrained()` should support adapter loading on an already CPU/GPU-dispatched bitsandbytes 4-bit model without reconstructing `Params4bit` with unsupported kwargs.
2. PEFT should avoid redispatching an already-dispatched quantized base model, or should do so without passing `_is_hf_initialized` to bitsandbytes constructors.
3. Accelerate `set_module_tensor_to_device()` should avoid forwarding HF-internal parameter attributes into `bitsandbytes.Params4bit. __new__ ()` if that constructor does not accept them.
4. bitsandbytes `Params4bit. __new__ ()` should accept or ignore `_is_hf_initialized` if this attribute is expected in the HF integration path.
5. If this configuration is unsupported, the error should be raised early with a clear message.

### Actual behavior

- All-GPU 4-bit base + patched multimodal PEFT adapter: works.
- Split CPU/GPU 4-bit base + the same patched multimodal PEFT adapter: fails during PEFT adapter loading.
- The failure occurs before generation.
- The failure occurs in the Accelerate dispatch/hook path when reconstructing a bitsandbytes `Params4bit` parameter on the meta-device path.

### Why this seems split/offload-specific

The same environment, same model, same adapter, same adapter patch, same quantization config, and same PEFT call pattern work when the model is all-GPU:

```python
device_map = {"": 0}
```

The failure appears only with:

```python
device_map = {
    "model.vision_tower": "cpu",
    "model.audio_tower": "cpu",
    "": 0,
}
```

Therefore, the issue seems tied to PEFT adapter loading on an already split/offloaded bitsandbytes 4-bit model, not to PEFT + bnb 4-bit generally.

### Related observations

#### 1. Gemma4ClippableLinear target-module blocker

Before applying the local adapter-config patch, both tested Gemma 4 E2B adapters failed earlier with:

```text
ValueError: Target module Gemma4ClippableLinear(...) is not supported.
```

That appears to be the same class of issue as the public PEFT `Gemma4ClippableLinear` support issue. The current repro works around that by targeting the inner `.linear` modules for vision/audio towers.

#### 2. Related `_is_hf_initialized` issue family

There is an existing issue for a similar error family with `Int8Params`:

```text
TypeError: Int8Params. __new__ () got an unexpected keyword argument '_is_hf_initialized'
```

This repro appears to be the `Params4bit` variant of that family, reached through PEFT adapter loading and Accelerate hook setup on a split-device model.

### Questions

1. Is `PeftModel.from_pretrained()` expected to support adapter loading on an already CPU/GPU-dispatched bitsandbytes 4-bit model?
2. Should PEFT skip redispatch when the base model already has an `hf_device_map`?
3. Should Accelerate filter `_is_hf_initialized` before reconstructing bitsandbytes parameter classes?
4. Should bitsandbytes `Params4bit` accept or ignore `_is_hf_initialized`?
5. Is the recommended workaround for Gemma 4 E2B on T4 to keep the whole 4-bit model on GPU rather than offloading vision/audio towers to CPU?

### Relevant links

```text
Accelerate big model dispatch docs:
https://huggingface.co/docs/accelerate/package_reference/big_modeling

PEFT PeftModel docs:
https://huggingface.co/docs/peft/package_reference/peft_model

Transformers PEFT integration docs:
https://huggingface.co/docs/transformers/en/peft

Transformers bitsandbytes docs:
https://huggingface.co/docs/transformers/quantization/bitsandbytes

PEFT Gemma4ClippableLinear issue:
https://github.com/huggingface/peft/issues/3129

Related _is_hf_initialized issue:
https://github.com/huggingface/transformers/issues/43872
```

````

* * *

# Optional Issue 2

## Target repo

```plaintext
huggingface/peft

```

## Suggested title

```plaintext
Gemma4 multimodal LoRA adapters need inner .linear targeting or Gemma4ClippableLinear support

```

## Body

````markdown
There is also a Gemma 4 target-module compatibility issue before this failure. Without the local `target_modules` patch, PEFT tries to inject LoRA into outer `Gemma4ClippableLinear` wrappers and fails with:

```text
ValueError: Target module Gemma4ClippableLinear(...) is not supported.
```

The adapter is genuinely multimodal, so simply excluding vision/audio targets would change the adapter semantics. The repro instead patches the local adapter config to target the inner `.linear` modules in Gemma 4 vision/audio towers.

With that patch, the all-GPU case passes, but the CPU/GPU-dispatched case still fails with:

```text
Params4bit. __new__ () got an unexpected keyword argument '_is_hf_initialized'
```

````

* * *

# My recommendation

Open **Issue 1** first. It is now much cleaner than the original draft because the latest evidence isolates the trigger:

```plaintext
all-GPU 4-bit path works
split/offloaded 4-bit path fails during PEFT adapter loading

```

Do **not** open the shared-KV issue from this latest Colab run alone, because this run did not reach that failure path.

---

_[View the full topic](https://discuss.huggingface.co/t/cpu-offloading-error-scenario/175522)._