# Loading model from local disk

**URL:** <https://discuss.huggingface.co/t/loading-model-from-local-disk/115880>\
**Category:** Beginners\
**Created:** [November 5, 2024, 8:40pm UTC](https://discuss.huggingface.co/t/loading-model-from-local-disk/115880 "2024-11-05T20:40:14Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![miloskec](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/miloskec/32/33628_2.png) [@miloskec](https://discuss.huggingface.co/u/miloskec)\
**Post date:** [November 5, 2024, 8:40pm UTC](https://discuss.huggingface.co/t/loading-model-from-local-disk/115880/1 "2024-11-05T20:40:14Z")

</div>

Im trying to load mlx-community/Mamba-Codestral-7B-v0.1-8bit model from disk.

```auto
from mlx_lm import load, generate
self.model, self.tokenizer = load("../models/Mamba-Codestral-7B-v0.1-8bit_model")

```

_ERROR:root:Model type mamba2 not supported._  
_codestral-mistral-models | Error loading model mlx-community/Mamba-Codestral-7B-v0.1-8bit: Model type mamba2 not supported._  
_codestral-mistral-models | ERROR:root:Error checking if model is saved: Model type mamba2 not supported._  
_codestral-mistral-models | ERROR:root:Error loading model: Model type mamba2 not supported._

OR

```auto
from transformers import Mamba2ForCausalLM, AutoTokenizer
self.tokenizer = AutoTokenizer.from_pretrained(self.save_dir)
self.model = AutoModelForCausalLM.from_pretrained(self.save_dir)
//self.model = Mamba2ForCausalLM.from_pretrained(self.save_dir)

```

_Error loading model mlx-community/Mamba-Codestral-7B-v0.1-8bit: The model’s quantization config from the arguments has no `quant_method` attribute. Make sure that the model has been correctly quantized…_

Model is cloned using “git clone [mlx-community/Mamba-Codestral-7B-v0.1-8bit · Hugging Face](https://huggingface.co/mlx-community/Mamba-Codestral-7B-v0.1-8bit)”

Model content:  
.git  
.gitattributes  
README.md  
config.json  
model-00001-of-00002.safetensors  
model-00002-of-00002.safetensors  
model.safetensors.index.json  
special\_tokens\_map.json  
tokenizer.json  
tokenizer.model  
tokenizer\_config.json

config.json:

```auto
{
    "architectures": [
        "Mamba2ForCausalLM"
    ],
    "bos_token_id": 0,
    "chunk_size": 256,
    "conv_kernel": 4,
    "eos_token_id": 0,
    "expand": 2,
    "head_dim": 64,
    "hidden_act": "silu",
    "hidden_size": 4096,
    "initializer_range": 0.1,
    "intermediate_size": 8192,
    "layer_norm_epsilon": 1e-05,
    "model_type": "mamba2",
    "n_groups": 8,
    "norm_before_gate": true,
    "num_heads": 128,
    "num_hidden_layers": 64,
    "pad_token_id": 0,
    "quantization": {
        "group_size": 64,
        "bits": 8
    },
    "quantization_config": {
        "group_size": 64,
        "bits": 8
    },
    "rescale_prenorm_residual": false,
    "residual_in_fp32": true,
    "rms_norm": true,
    "state_size": 128,
    "tie_word_embeddings": false,
    "time_step_floor": 0.0001,
    "time_step_init_scheme": "random",
    "time_step_limit": [
        0.0,
        Infinity
    ],
    "time_step_max": 0.1,
    "time_step_min": 0.001,
    "time_step_rank": 256,
    "time_step_scale": 1.0,
    "torch_dtype": "bfloat16",
    "transformers_version": "4.44.0.dev0",
    "use_bias": false,
    "use_cache": true,
    "use_conv_bias": true,
    "vocab_size": 32768
}

```

Dockerfile:

```auto
# Use a base image with PyTorch and GPU support
FROM huggingface/transformers-pytorch-gpu:latest

# Set working directory inside the container
WORKDIR /workspace

# Install necessary packages
RUN apt-get update && \
    pip3 install mistral_inference>=1 mamba-ssm causal-conv1d && \
    pip3 install langchain torch accelerate flask pydantic uvicorn mlx-lm && \
    pip3 install python-dotenv langchain-community langchain-huggingface llama-index llama-index-embeddings-huggingface && \
    pip3 install peft auto-gptq optimum bitsandbytes sentence-transformers numpy fastapi ...

```

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [November 6, 2024, 1:01am UTC](https://discuss.huggingface.co/t/loading-model-from-local-disk/115880/2 "2024-11-06T01:01:26Z")

</div>

> The Model [mlx-community/Mamba-Codestral-7B-v0.1-8bits](https://huggingface.co/mlx-community/Mamba-Codestral-7B-v0.1-8bits) was converted to MLX format from [mistralai/Mamba-Codestral-7B-v0.1](https://huggingface.co/mistralai/Mamba-Codestral-7B-v0.1) using mlx-lm version **0.18.2**.

So, let’s try installing the latest version first.

```auto
pip install -U mlx-lm

```

---

<div class="post-metadata">

**Author:** ![miloskec](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/miloskec/32/33628_2.png) [@miloskec](https://discuss.huggingface.co/u/miloskec)\
**Post date:** [November 6, 2024, 3:12pm UTC](https://discuss.huggingface.co/t/loading-model-from-local-disk/115880/3 "2024-11-06T15:12:51Z")

</div>

Thanks John6666,

I initially tried installing the latest version as you suggested:

```auto
pip install -U mlx-lm

```

Unfortunately, this didn’t resolve the issue.

_ERROR:root:Model type mamba2 not supported._  
_codestral-mistral-models | Error loading model mlx-community/Mamba-Codestral-7B-v0.1-8bit: Model type mamba2 not supported._  
_codestral-mistral-models | ERROR:root:Error checking if model is saved: Model type mamba2 not supported._  
_codestral-mistral-models | ERROR:root:Error loading model: Model type mamba2 not supported._