# Inference Endpoint Fails to Start

**URL:** <https://discuss.huggingface.co/t/inference-endpoint-fails-to-start/65809>\
**Category:** Inference Endpoints on the Hub\
**Created:** [December 14, 2023, 10:42am UTC](https://discuss.huggingface.co/t/inference-endpoint-fails-to-start/65809 "2023-12-14T10:42:16Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![alexsafayan](https://avatars.discourse-cdn.com/v4/letter/a/278dde/32.png) [@alexsafayan](https://discuss.huggingface.co/u/alexsafayan)\
**Post date:** [December 14, 2023, 10:42am UTC](https://discuss.huggingface.co/t/inference-endpoint-fails-to-start/65809/1 "2023-12-14T10:42:16Z")

</div>

Whenever I start to start an Inference Endpoint, it fails.

# Setup

aws-dolphin-2-5-mixtral-8x7b-934

AWS

us-east-1

GPU · Nvidia Tesla T4 · 4x GPU · 64 GB

# Log:

```auto
66d6896fchcwbw 2023-12-14T10:25:02.055Z INFO | Repository ID: ehartford/dolphin-2.5-mixtral-8x7b
66d6896fchcwbw 2023-12-14T10:25:02.055Z INFO | Repository Revision: cbf205ee8b82ec81cec217b124e5d57804c3139a

66d6896fchcwbw 2023-12-14T10:26:53.761Z Error: DownloadError
66d6896fchcwbw 2023-12-14T10:26:53.761Z {"timestamp":"2023-12-14T10:26:53.761152Z","level":"ERROR","fields":{"message":"Download encountered an error: Traceback (most recent call last):\n... huggingface_hub.utils._validators.HFValidationError: Repo id must use alphanumeric chars or '-', '_', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96: '/repository'... ValueError: Can't find 'adapter_config.json' at '/repository'"}} [...]

```

---

<div class="post-metadata">

**Author:** ![Moose4400](https://avatars.discourse-cdn.com/v4/letter/m/ac8455/32.png) [@Moose4400](https://discuss.huggingface.co/u/Moose4400)\
**Post date:** [December 14, 2023, 2:01pm UTC](https://discuss.huggingface.co/t/inference-endpoint-fails-to-start/65809/2 "2023-12-14T14:01:13Z")

</div>

Running into this error as well. Contacted support.

---

<div class="post-metadata">

**Author:** ![alexsafayan](https://avatars.discourse-cdn.com/v4/letter/a/278dde/32.png) [@alexsafayan](https://discuss.huggingface.co/u/alexsafayan)\
**Post date:** [December 20, 2023, 11:38pm UTC](https://discuss.huggingface.co/t/inference-endpoint-fails-to-start/65809/3 "2023-12-20T23:38:59Z")

</div>

Did support fix this issue?

---

<div class="post-metadata">

**Author:** ![Moose4400](https://avatars.discourse-cdn.com/v4/letter/m/ac8455/32.png) [@Moose4400](https://discuss.huggingface.co/u/Moose4400)\
**Post date:** [December 21, 2023, 4:07pm UTC](https://discuss.huggingface.co/t/inference-endpoint-fails-to-start/65809/4 "2023-12-21T16:07:29Z")

</div>

Yes. Everything is working for me now.

---

<div class="post-metadata">

**Author:** ![calvinball](https://avatars.discourse-cdn.com/v4/letter/c/22d042/32.png) [@calvinball](https://discuss.huggingface.co/u/calvinball)\
**Post date:** [December 23, 2023, 10:52am UTC](https://discuss.huggingface.co/t/inference-endpoint-fails-to-start/65809/5 "2023-12-23T10:52:02Z")

</div>

Hi, how did you fix this? What steps did support told you? I’m also facing the same error currently…

---

<div class="post-metadata">

**Author:** ![Moose4400](https://avatars.discourse-cdn.com/v4/letter/m/ac8455/32.png) [@Moose4400](https://discuss.huggingface.co/u/Moose4400)\
**Post date:** [December 23, 2023, 4:42pm UTC](https://discuss.huggingface.co/t/inference-endpoint-fails-to-start/65809/6 "2023-12-23T16:42:01Z")

</div>

They didn’t provide me with any steps. It was an internal error that they fixed.  
You should contact support with your details.

---

<div class="post-metadata">

**Author:** ![meganariley](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/meganariley/32/56966_2.png) [@meganariley](https://discuss.huggingface.co/u/meganariley)\
**Post date:** [January 3, 2024, 4:34pm UTC](https://discuss.huggingface.co/t/inference-endpoint-fails-to-start/65809/7 "2024-01-03T16:34:14Z")

</div>

Hi @alexsafayan! Thanks for reporting this issue. We’re currently investigating this issue and I’ll update you as soon as I have more information or there’s a fix.

---

<div class="post-metadata">

**Author:** ![jostelo](https://avatars.discourse-cdn.com/v4/letter/j/f6c823/32.png) [@jostelo](https://discuss.huggingface.co/u/jostelo)\
**Post date:** [January 3, 2024, 4:44pm UTC](https://discuss.huggingface.co/t/inference-endpoint-fails-to-start/65809/8 "2024-01-03T16:44:03Z")

</div>

Hello @meganariley. Same problem here for the same VM but different models. E.g. [LeoLM/leo-hessianai-7b-chat](https://huggingface.co/LeoLM/leo-hessianai-7b-chat).

Thanks in advance!

---

<div class="post-metadata">

**Author:** ![meganariley](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/meganariley/32/56966_2.png) [@meganariley](https://discuss.huggingface.co/u/meganariley)\
**Post date:** [January 3, 2024, 4:46pm UTC](https://discuss.huggingface.co/t/inference-endpoint-fails-to-start/65809/9 "2024-01-03T16:46:22Z")

</div>

Thanks @jostelo! I’ll keep you updated as well. 🤗

---

<div class="post-metadata">

**Author:** ![GowthamYarlagadda](https://avatars.discourse-cdn.com/v4/letter/g/b19c9b/32.png) [@GowthamYarlagadda](https://discuss.huggingface.co/u/GowthamYarlagadda)\
**Post date:** [January 3, 2024, 7:26pm UTC](https://discuss.huggingface.co/t/inference-endpoint-fails-to-start/65809/10 "2024-01-03T19:26:28Z")

</div>

Hello @meganariley. Could you resolve my issue too? While trying to create an Inference Endpoint, I get this error: “Server message: Endpoint failed to start. Endpoint failed”

[nonprof/llama2-7b-finetuned-counsellor-full-model] ([nonprof/llama2-7b-finetuned-counsellor-full-model · Hugging Face](https://huggingface.co/nonprof/llama2-7b-finetuned-counsellor-full-model)).

In the logs:  
HFValidationError(\n\nhuggingface\_hub.utils._validators.HFValidationError: Repo id must use alphanumeric chars or ‘-’, '_', ‘.’, ‘–’ and ‘…’ are forbidden, ‘-’ and ‘.’ cannot start or end the name, max length is 96: ‘/repository’.\n\n\nDuring handling of the above exception, another exception occurred:\n\n\nTraceback (most recent call last):\n\n File "/opt/conda/bin/text-generation-server", line 8, in \n sys.exit(app())\n\n File "/opt/conda/lib/python3.10/site-packages/text\_generation\_server/cli.py", line 204, in download\_weights\n utils.download\_and\_unload\_peft(\n\n File "/opt/conda/lib/python3.10/site-packages/text\_generation\_server/utils/peft.py", line 24, in download\_and\_unload\_peft\n model = AutoPeftModelForSeq2SeqLM.from\_pretrained(\n\n File "/opt/conda/lib/python3.10/site-packages/peft/auto.py", line 69, in from\_pretrained\n peft\_config = PeftConfig.from\_pretrained(pretrained\_model\_name\_or\_path, \*\*kwargs)\n\n File "/opt/conda/lib/python3.10/site-packages/peft/utils/config.py", line 121, in from\_pretrained\n raise ValueError(f"Can’t find ‘{CONFIG\_NAME}’ at ‘{pretrained\_model\_name\_or\_path}’")\n\nValueError: Can’t find ‘adapter\_config.json’ at ‘/repository’\n\n"},“target”:“text\_generation\_launcher”,“span”:{“name”:“download”},“spans”:[{“name”:“download”}]}

Thanks in advance! You could be my lifesaver!

---

<div class="post-metadata">

**Author:** ![IBLLMTST](https://avatars.discourse-cdn.com/v4/letter/i/7feea3/32.png) [@IBLLMTST](https://discuss.huggingface.co/u/IBLLMTST)\
**Post date:** [January 4, 2024, 3:17am UTC](https://discuss.huggingface.co/t/inference-endpoint-fails-to-start/65809/11 "2024-01-04T03:17:18Z")

</div>

I am having the same (or at least very similar) problem. I am trying to create an Inference Endpoint and it fails to start.  
The model I am using is: [tiiuae/falcon-40b-instruct](https://huggingface.co/tiiuae/falcon-40b-instruct)  
The configuration is: AWS us-east-1 GPU · Nvidia Tesla T4 · 4x GPU · 64 GB  
The complete log output is below, but the relevant part seems to be the “HFValidationError: Repo id must use alphanumeric” as in the post above.

Any help would be greatly appreciated!

Here’s the complete log:

```auto

2024/01/03 21:52:54 ~ INFO | Start loading image artifacts from huggingface.co
2024/01/03 21:52:54 ~ INFO | Used configuration:
2024/01/03 21:52:54 ~ INFO | Repository ID: tiiuae/falcon-40b-instruct
2024/01/03 21:52:54 ~ INFO | Repository Revision: ecb78d97ac356d098e79f0db222c9ce7c5d9ee5f
2024/01/03 21:52:54 ~ INFO | Ignore regex pattern for files, which are not downloaded: *tflite, flax*, *ckpt, tf*, *onnx*, *tar.gz, *safetensors, *mlmodel, rust*, *openvino*
2024/01/03 21:54:05 ~ Token will not been saved to git credential helper. Pass `add_to_git_credential=True` if you want to set the git credential as well.
2024/01/03 21:54:05 ~ Login successful
2024/01/03 21:54:05 ~ Token is valid.
2024/01/03 21:54:05 ~ Your token has been saved to /root/.cache/huggingface/token
2024/01/03 21:54:39 ~ {"timestamp":"2024-01-04T02:54:39.200748Z","level":"INFO","fields":{"message":"Starting download process."},"target":"text_generation_launcher","span":{"name":"download"},"spans":[{"name":"download"}]}
2024/01/03 21:54:39 ~ {"timestamp":"2024-01-04T02:54:39.200602Z","level":"INFO","fields":{"message":"Args { model_id: \"/repository\", revision: None, validation_workers: 2, sharded: None, num_shard: None, quantize: Some(Bitsandbytes), speculate: None, dtype: None, trust_remote_code: false, max_concurrent_requests: 128, max_best_of: 2, max_stop_sequences: 4, max_top_n_tokens: 5, max_input_length: 1024, max_total_tokens: 1512, waiting_served_ratio: 1.2, max_batch_prefill_tokens: 2048, max_batch_total_tokens: None, max_waiting_tokens: 20, hostname: \"e-8a68-aws-falcon-40b-instruct-1221-fdd998944-hr86z\", port: 80, shard_uds_path: \"/tmp/text-generation-server\", master_addr: \"localhost\", master_port: 29500, huggingface_hub_cache: Some(\"/data\"), weights_cache_override: None, disable_custom_kernels: false, cuda_memory_fraction: 1.0, rope_scaling: None, rope_factor: None, json_output: true, otlp_endpoint: None, cors_allow_origin: [], watermark_gamma: None, watermark_delta: None, ngrok: false, ngrok_authtoken: None, ngrok_edge: None, env: false }"},"target":"text_generation_launcher"}
2024/01/03 21:54:39 ~ {"timestamp":"2024-01-04T02:54:39.200639Z","level":"INFO","fields":{"message":"Sharding model on 4 processes"},"target":"text_generation_launcher"}
2024/01/03 21:54:44 ~ {"timestamp":"2024-01-04T02:54:44.400443Z","level":"INFO","fields":{"message":"Loading the model it might take a while without feedback\n"},"target":"text_generation_launcher"}
2024/01/03 21:54:44 ~ {"timestamp":"2024-01-04T02:54:44.400411Z","level":"INFO","fields":{"message":"Peft model detected.\n"},"target":"text_generation_launcher"}
2024/01/03 21:54:45 ~ Error: DownloadError
2024/01/03 21:54:45 ~ {"timestamp":"2024-01-04T02:54:45.007548Z","level":"ERROR","fields":{"message":"Download encountered an error: Traceback (most recent call last):\n\n File \"/opt/conda/lib/python3.10/site-packages/peft/utils/config.py\", line 117, in from_pretrained\n config_file = hf_hub_download(\n\n File \"/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py\", line 110, in _inner_fn\n validate_repo_id(arg_value)\n\n File \"/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py\", line 164, in validate_repo_id\n raise HFValidationError(\n\nhuggingface_hub.utils._validators.HFValidationError: Repo id must use alphanumeric chars or '-', '_', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96: '/repository'.\n\n\nDuring handling of the above exception, another exception occurred:\n\n\nTraceback (most recent call last):\n\n File \"/opt/conda/lib/python3.10/site-packages/text_generation_server/utils/peft.py\", line 16, in download_and_unload_peft\n model = AutoPeftModelForCausalLM.from_pretrained(\n\n File \"/opt/conda/lib/python3.10/site-packages/peft/auto.py\", line 69, in from_pretrained\n peft_config = PeftConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)\n\n File \"/opt/conda/lib/python3.10/site-packages/peft/utils/config.py\", line 121, in from_pretrained\n raise ValueError(f\"Can't find '{CONFIG_NAME}' at '{pretrained_model_name_or_path}'\")\n\nValueError: Can't find 'adapter_config.json' at '/repository'\n\n\nDuring handling of the above exception, another exception occurred:\n\n\nTraceback (most recent call last):\n\n File \"/opt/conda/lib/python3.10/site-packages/peft/utils/config.py\", line 117, in from_pretrained\n config_file = hf_hub_download(\n\n File \"/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py\", line 110, in _inner_fn\n validate_repo_id(arg_value)\n\n File \"/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py\", line 164, in validate_repo_id\n raise HFValidationError(\n\nhuggingface_hub.utils._validators.HFValidationError: Repo id must use alphanumeric chars or '-', '_', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96: '/repository'.\n\n\nDuring handling of the above exception, another exception occurred:\n\n\nTraceback (most recent call last):\n\n File \"/opt/conda/bin/text-generation-server\", line 8, in <module>\n sys.exit(app())\n\n File \"/opt/conda/lib/python3.10/site-packages/text_generation_server/cli.py\", line 204, in download_weights\n utils.download_and_unload_peft(\n\n File \"/opt/conda/lib/python3.10/site-packages/text_generation_server/utils/peft.py\", line 24, in download_and_unload_peft\n model = AutoPeftModelForSeq2SeqLM.from_pretrained(\n\n File \"/opt/conda/lib/python3.10/site-packages/peft/auto.py\", line 69, in from_pretrained\n peft_config = PeftConfig.from_pretrained(pretrained_model_name_or_path,**kwargs)\n\n File \"/opt/conda/lib/python3.10/site-packages/peft/utils/config.py\", line 121, in from_pretrained\n raise ValueError(f\"Can't find '{CONFIG_NAME}' at '{pretrained_model_name_or_path}'\")\n\nValueError: Can't find 'adapter_config.json' at '/repository'\n\n"},"target":"text_generation_launcher","span":{"name":"download"},"spans":[{"name":"download"}]}
2024/01/03 21:54:46 ~ {"timestamp":"2024-01-04T02:54:46.323170Z","level":"INFO","fields":{"message":"Starting download process."},"target":"text_generation_launcher","span":{"name":"download"},"spans":[{"name":"download"}]}
2024/01/03 21:54:46 ~ {"timestamp":"2024-01-04T02:54:46.323061Z","level":"INFO","fields":{"message":"Sharding model on 4 processes"},"target":"text_generation_launcher"}
2024/01/03 21:54:46 ~ {"timestamp":"2024-01-04T02:54:46.323014Z","level":"INFO","fields":{"message":"Args { model_id: \"/repository\", revision: None, validation_workers: 2, sharded: None, num_shard: None, quantize: Some(Bitsandbytes), speculate: None, dtype: None, trust_remote_code: false, max_concurrent_requests: 128, max_best_of: 2, max_stop_sequences: 4, max_top_n_tokens: 5, max_input_length: 1024, max_total_tokens: 1512, waiting_served_ratio: 1.2, max_batch_prefill_tokens: 2048, max_batch_total_tokens: None, max_waiting_tokens: 20, hostname: \"e-8a68-aws-falcon-40b-instruct-1221-fdd998944-hr86z\", port: 80, shard_uds_path: \"/tmp/text-generation-server\", master_addr: \"localhost\", master_port: 29500, huggingface_hub_cache: Some(\"/data\"), weights_cache_override: None, disable_custom_kernels: false, cuda_memory_fraction: 1.0, rope_scaling: None, rope_factor: None, json_output: true, otlp_endpoint: None, cors_allow_origin: [], watermark_gamma: None, watermark_delta: None, ngrok: false, ngrok_authtoken: None, ngrok_edge: None, env: false }"},"target":"text_generation_launcher"}
2024/01/03 21:54:51 ~ {"timestamp":"2024-01-04T02:54:51.392245Z","level":"INFO","fields":{"message":"Peft model detected.\n"},"target":"text_generation_launcher"}
2024/01/03 21:54:51 ~ {"timestamp":"2024-01-04T02:54:51.392299Z","level":"INFO","fields":{"message":"Loading the model it might take a while without feedback\n"},"target":"text_generation_launcher"}
2024/01/03 21:54:51 ~ Error: DownloadError
2024/01/03 21:54:51 ~ {"timestamp":"2024-01-04T02:54:51.929184Z","level":"ERROR","fields":{"message":"Download encountered an error: Traceback (most recent call last):\n\n File \"/opt/conda/lib/python3.10/site-packages/peft/utils/config.py\", line 117, in from_pretrained\n config_file = hf_hub_download(\n\n File \"/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py\", line 110, in _inner_fn\n validate_repo_id(arg_value)\n\n File \"/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py\", line 164, in validate_repo_id\n raise HFValidationError(\n\nhuggingface_hub.utils._validators.HFValidationError: Repo id must use alphanumeric chars or '-', '_', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96: '/repository'.\n\n\nDuring handling of the above exception, another exception occurred:\n\n\nTraceback (most recent call last):\n\n File \"/opt/conda/lib/python3.10/site-packages/text_generation_server/utils/peft.py\", line 16, in download_and_unload_peft\n model = AutoPeftModelForCausalLM.from_pretrained(\n\n File \"/opt/conda/lib/python3.10/site-packages/peft/auto.py\", line 69, in from_pretrained\n peft_config = PeftConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)\n\n File \"/opt/conda/lib/python3.10/site-packages/peft/utils/config.py\", line 121, in from_pretrained\n raise ValueError(f\"Can't find '{CONFIG_NAME}' at '{pretrained_model_name_or_path}'\")\n\nValueError: Can't find 'adapter_config.json' at '/repository'\n\n\nDuring handling of the above exception, another exception occurred:\n\n\nTraceback (most recent call last):\n\n File \"/opt/conda/lib/python3.10/site-packages/peft/utils/config.py\", line 117, in from_pretrained\n config_file = hf_hub_download(\n\n File \"/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py\", line 110, in _inner_fn\n validate_repo_id(arg_value)\n\n File \"/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py\", line 164, in validate_repo_id\n raise HFValidationError(\n\nhuggingface_hub.utils._validators.HFValidationError: Repo id must use alphanumeric chars or '-', '_', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96: '/repository'.\n\n\nDuring handling of the above exception, another exception occurred:\n\n\nTraceback (most recent call last):\n\n File \"/opt/conda/bin/text-generation-server\", line 8, in <module>\n sys.exit(app())\n\n File \"/opt/conda/lib/python3.10/site-packages/text_generation_server/cli.py\", line 204, in download_weights\n utils.download_and_unload_peft(\n\n File \"/opt/conda/lib/python3.10/site-packages/text_generation_server/utils/peft.py\", line 24, in download_and_unload_peft\n model = AutoPeftModelForSeq2SeqLM.from_pretrained(\n\n File \"/opt/conda/lib/python3.10/site-packages/peft/auto.py\", line 69, in from_pretrained\n peft_config = PeftConfig.from_pretrained(pretrained_model_name_or_path,**kwargs)\n\n File \"/opt/conda/lib/python3.10/site-packages/peft/utils/config.py\", line 121, in from_pretrained\n raise ValueError(f\"Can't find '{CONFIG_NAME}' at '{pretrained_model_name_or_path}'\")\n\nValueError: Can't find 'adapter_config.json' at '/repository'\n\n"},"target":"text_generation_launcher","span":{"name":"download"},"spans":[{"name":"download"}]}
2024/01/03 21:55:07 ~ {"timestamp":"2024-01-04T02:55:07.040546Z","level":"INFO","fields":{"message":"Starting download process."},"target":"text_generation_launcher","span":{"name":"download"},"spans":[{"name":"download"}]}
2024/01/03 21:55:07 ~ {"timestamp":"2024-01-04T02:55:07.040419Z","level":"INFO","fields":{"message":"Args { model_id: \"/repository\", revision: None, validation_workers: 2, sharded: None, num_shard: None, quantize: Some(Bitsandbytes), speculate: None, dtype: None, trust_remote_code: false, max_concurrent_requests: 128, max_best_of: 2, max_stop_sequences: 4, max_top_n_tokens: 5, max_input_length: 1024, max_total_tokens: 1512, waiting_served_ratio: 1.2, max_batch_prefill_tokens: 2048, max_batch_total_tokens: None, max_waiting_tokens: 20, hostname: \"e-8a68-aws-falcon-40b-instruct-1221-fdd998944-hr86z\", port: 80, shard_uds_path: \"/tmp/text-generation-server\", master_addr: \"localhost\", master_port: 29500, huggingface_hub_cache: Some(\"/data\"), weights_cache_override: None, disable_custom_kernels: false, cuda_memory_fraction: 1.0, rope_scaling: None, rope_factor: None, json_output: true, otlp_endpoint: None, cors_allow_origin: [], watermark_gamma: None, watermark_delta: None, ngrok: false, ngrok_authtoken: None, ngrok_edge: None, env: false }"},"target":"text_generation_launcher"}
2024/01/03 21:55:07 ~ {"timestamp":"2024-01-04T02:55:07.040456Z","level":"INFO","fields":{"message":"Sharding model on 4 processes"},"target":"text_generation_launcher"}
2024/01/03 21:55:12 ~ {"timestamp":"2024-01-04T02:55:12.260841Z","level":"INFO","fields":{"message":"Loading the model it might take a while without feedback\n"},"target":"text_generation_launcher"}
2024/01/03 21:55:12 ~ {"timestamp":"2024-01-04T02:55:12.260808Z","level":"INFO","fields":{"message":"Peft model detected.\n"},"target":"text_generation_launcher"}
2024/01/03 21:55:12 ~ Error: DownloadError
2024/01/03 21:55:12 ~ {"timestamp":"2024-01-04T02:55:12.847314Z","level":"ERROR","fields":{"message":"Download encountered an error: Traceback (most recent call last):\n\n File \"/opt/conda/lib/python3.10/site-packages/peft/utils/config.py\", line 117, in from_pretrained\n config_file = hf_hub_download(\n\n File \"/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py\", line 110, in _inner_fn\n validate_repo_id(arg_value)\n\n File \"/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py\", line 164, in validate_repo_id\n raise HFValidationError(\n\nhuggingface_hub.utils._validators.HFValidationError: Repo id must use alphanumeric chars or '-', '_', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96: '/repository'.\n\n\nDuring handling of the above exception, another exception occurred:\n\n\nTraceback (most recent call last):\n\n File \"/opt/conda/lib/python3.10/site-packages/text_generation_server/utils/peft.py\", line 16, in download_and_unload_peft\n model = AutoPeftModelForCausalLM.from_pretrained(\n\n File \"/opt/conda/lib/python3.10/site-packages/peft/auto.py\", line 69, in from_pretrained\n peft_config = PeftConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)\n\n File \"/opt/conda/lib/python3.10/site-packages/peft/utils/config.py\", line 121, in from_pretrained\n raise ValueError(f\"Can't find '{CONFIG_NAME}' at '{pretrained_model_name_or_path}'\")\n\nValueError: Can't find 'adapter_config.json' at '/repository'\n\n\nDuring handling of the above exception, another exception occurred:\n\n\nTraceback (most recent call last):\n\n File \"/opt/conda/lib/python3.10/site-packages/peft/utils/config.py\", line 117, in from_pretrained\n config_file = hf_hub_download(\n\n File \"/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py\", line 110, in _inner_fn\n validate_repo_id(arg_value)\n\n File \"/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py\", line 164, in validate_repo_id\n raise HFValidationError(\n\nhuggingface_hub.utils._validators.HFValidationError: Repo id must use alphanumeric chars or '-', '_', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96: '/repository'.\n\n\nDuring handling of the above exception, another exception occurred:\n\n\nTraceback (most recent call last):\n\n File \"/opt/conda/bin/text-generation-server\", line 8, in <module>\n sys.exit(app())\n\n File \"/opt/conda/lib/python3.10/site-packages/text_generation_server/cli.py\", line 204, in download_weights\n utils.download_and_unload_peft(\n\n File \"/opt/conda/lib/python3.10/site-packages/text_generation_server/utils/peft.py\", line 24, in download_and_unload_peft\n model = AutoPeftModelForSeq2SeqLM.from_pretrained(\n\n File \"/opt/conda/lib/python3.10/site-packages/peft/auto.py\", line 69, in from_pretrained\n peft_config = PeftConfig.from_pretrained(pretrained_model_name_or_path,**kwargs)\n\n File \"/opt/conda/lib/python3.10/site-packages/peft/utils/config.py\", line 121, in from_pretrained\n raise ValueError(f\"Can't find '{CONFIG_NAME}' at '{pretrained_model_name_or_path}'\")\n\nValueError: Can't find 'adapter_config.json' at '/repository'\n\n"},"target":"text_generation_launcher","span":{"name":"download"},"spans":[{"name":"download"}]}
2024/01/03 21:55:41 ~ {"timestamp":"2024-01-04T02:55:41.040752Z","level":"INFO","fields":{"message":"Starting download process."},"target":"text_generation_launcher","span":{"name":"download"},"spans":[{"name":"download"}]}
2024/01/03 21:55:41 ~ {"timestamp":"2024-01-04T02:55:41.040611Z","level":"INFO","fields":{"message":"Args { model_id: \"/repository\", revision: None, validation_workers: 2, sharded: None, num_shard: None, quantize: Some(Bitsandbytes), speculate: None, dtype: None, trust_remote_code: false, max_concurrent_requests: 128, max_best_of: 2, max_stop_sequences: 4, max_top_n_tokens: 5, max_input_length: 1024, max_total_tokens: 1512, waiting_served_ratio: 1.2, max_batch_prefill_tokens: 2048, max_batch_total_tokens: None, max_waiting_tokens: 20, hostname: \"e-8a68-aws-falcon-40b-instruct-1221-fdd998944-hr86z\", port: 80, shard_uds_path: \"/tmp/text-generation-server\", master_addr: \"localhost\", master_port: 29500, huggingface_hub_cache: Some(\"/data\"), weights_cache_override: None, disable_custom_kernels: false, cuda_memory_fraction: 1.0, rope_scaling: None, rope_factor: None, json_output: true, otlp_endpoint: None, cors_allow_origin: [], watermark_gamma: None, watermark_delta: None, ngrok: false, ngrok_authtoken: None, ngrok_edge: None, env: false }"},"target":"text_generation_launcher"}
2024/01/03 21:55:41 ~ {"timestamp":"2024-01-04T02:55:41.040648Z","level":"INFO","fields":{"message":"Sharding model on 4 processes"},"target":"text_generation_launcher"}
2024/01/03 21:55:46 ~ {"timestamp":"2024-01-04T02:55:46.095858Z","level":"INFO","fields":{"message":"Peft model detected.\n"},"target":"text_generation_launcher"}
2024/01/03 21:55:46 ~ {"timestamp":"2024-01-04T02:55:46.095902Z","level":"INFO","fields":{"message":"Loading the model it might take a while without feedback\n"},"target":"text_generation_launcher"}
2024/01/03 21:55:46 ~ Error: DownloadError
2024/01/03 21:55:46 ~ {"timestamp":"2024-01-04T02:55:46.647317Z","level":"ERROR","fields":{"message":"Download encountered an error: Traceback (most recent call last):\n\n File \"/opt/conda/lib/python3.10/site-packages/peft/utils/config.py\", line 117, in from_pretrained\n config_file = hf_hub_download(\n\n File \"/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py\", line 110, in _inner_fn\n validate_repo_id(arg_value)\n\n File \"/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py\", line 164, in validate_repo_id\n raise HFValidationError(\n\nhuggingface_hub.utils._validators.HFValidationError: Repo id must use alphanumeric chars or '-', '_', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96: '/repository'.\n\n\nDuring handling of the above exception, another exception occurred:\n\n\nTraceback (most recent call last):\n\n File \"/opt/conda/lib/python3.10/site-packages/text_generation_server/utils/peft.py\", line 16, in download_and_unload_peft\n model = AutoPeftModelForCausalLM.from_pretrained(\n\n File \"/opt/conda/lib/python3.10/site-packages/peft/auto.py\", line 69, in from_pretrained\n peft_config = PeftConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)\n\n File \"/opt/conda/lib/python3.10/site-packages/peft/utils/config.py\", line 121, in from_pretrained\n raise ValueError(f\"Can't find '{CONFIG_NAME}' at '{pretrained_model_name_or_path}'\")\n\nValueError: Can't find 'adapter_config.json' at '/repository'\n\n\nDuring handling of the above exception, another exception occurred:\n\n\nTraceback (most recent call last):\n\n File \"/opt/conda/lib/python3.10/site-packages/peft/utils/config.py\", line 117, in from_pretrained\n config_file = hf_hub_download(\n\n File \"/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py\", line 110, in _inner_fn\n validate_repo_id(arg_value)\n\n File \"/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py\", line 164, in validate_repo_id\n raise HFValidationError(\n\nhuggingface_hub.utils._validators.HFValidationError: Repo id must use alphanumeric chars or '-', '_', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96: '/repository'.\n\n\nDuring handling of the above exception, another exception occurred:\n\n\nTraceback (most recent call last):\n\n File \"/opt/conda/bin/text-generation-server\", line 8, in <module>\n sys.exit(app())\n\n File \"/opt/conda/lib/python3.10/site-packages/text_generation_server/cli.py\", line 204, in download_weights\n utils.download_and_unload_peft(\n\n File \"/opt/conda/lib/python3.10/site-packages/text_generation_server/utils/peft.py\", line 24, in download_and_unload_peft\n model = AutoPeftModelForSeq2SeqLM.from_pretrained(\n\n File \"/opt/conda/lib/python3.10/site-packages/peft/auto.py\", line 69, in from_pretrained\n peft_config = PeftConfig.from_pretrained(pretrained_model_name_or_path,**kwargs)\n\n File \"/opt/conda/lib/python3.10/site-packages/peft/utils/config.py\", line 121, in from_pretrained\n raise ValueError(f\"Can't find '{CONFIG_NAME}' at '{pretrained_model_name_or_path}'\")\n\nValueError: Can't find 'adapter_config.json' at '/repository'\n\n"},"target":"text_generation_launcher","span":{"name":"download"},"spans":[{"name":"download"}]}
2024/01/03 21:56:35 ~ {"timestamp":"2024-01-04T02:56:35.052474Z","level":"INFO","fields":{"message":"Sharding model on 4 processes"},"target":"text_generation_launcher"}
2024/01/03 21:56:35 ~ {"timestamp":"2024-01-04T02:56:35.052425Z","level":"INFO","fields":{"message":"Args { model_id: \"/repository\", revision: None, validation_workers: 2, sharded: None, num_shard: None, quantize: Some(Bitsandbytes), speculate: None, dtype: None, trust_remote_code: false, max_concurrent_requests: 128, max_best_of: 2, max_stop_sequences: 4, max_top_n_tokens: 5, max_input_length: 1024, max_total_tokens: 1512, waiting_served_ratio: 1.2, max_batch_prefill_tokens: 2048, max_batch_total_tokens: None, max_waiting_tokens: 20, hostname: \"e-8a68-aws-falcon-40b-instruct-1221-fdd998944-hr86z\", port: 80, shard_uds_path: \"/tmp/text-generation-server\", master_addr: \"localhost\", master_port: 29500, huggingface_hub_cache: Some(\"/data\"), weights_cache_override: None, disable_custom_kernels: false, cuda_memory_fraction: 1.0, rope_scaling: None, rope_factor: None, json_output: true, otlp_endpoint: None, cors_allow_origin: [], watermark_gamma: None, watermark_delta: None, ngrok: false, ngrok_authtoken: None, ngrok_edge: None, env: false }"},"target":"text_generation_launcher"}
2024/01/03 21:56:35 ~ {"timestamp":"2024-01-04T02:56:35.052600Z","level":"INFO","fields":{"message":"Starting download process."},"target":"text_generation_launcher","span":{"name":"download"},"spans":[{"name":"download"}]}

```

---

<div class="post-metadata">

**Author:** ![Vsauv](https://avatars.discourse-cdn.com/v4/letter/v/ec9cab/32.png) [@Vsauv](https://discuss.huggingface.co/u/Vsauv)\
**Post date:** [January 4, 2024, 6:39pm UTC](https://discuss.huggingface.co/t/inference-endpoint-fails-to-start/65809/12 "2024-01-04T18:39:02Z")

</div>

Hi @meganariley. I have the same issue when I try to create an inference endpoint. I receive this message error “erver message:Endpoint failed to start. Endpoint failed”

Setup :  
Model : mixtral-8x7b-instruct-v0-1-7  
Configuration : GPU · Nvidia A10G · 1x GPU · 24 GB

Any help would incredibly help! You could save my life

I can provide the logs is necessary !

---

<div class="post-metadata">

**Author:** ![eugenecamus](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/eugenecamus/32/22814_2.png) [@eugenecamus](https://discuss.huggingface.co/u/eugenecamus)\
**Post date:** [January 8, 2024, 4:55pm UTC](https://discuss.huggingface.co/t/inference-endpoint-fails-to-start/65809/13 "2024-01-08T16:55:19Z")

</div>

Hello @meganariley,

I ran into the same problem with the following setup:  
defog/sqlcoder2  
AWS  
us-east-1  
nvidia A10G

Thanks

---

<div class="post-metadata">

**Author:** ![meganariley](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/meganariley/32/56966_2.png) [@meganariley](https://discuss.huggingface.co/u/meganariley)\
**Post date:** [January 9, 2024, 5:17pm UTC](https://discuss.huggingface.co/t/inference-endpoint-fails-to-start/65809/14 "2024-01-09T17:17:46Z")

</div>

@eugenecamus @Vsauv @IBLLMTST @GowthamYarlagadda @jostelo @alexsafayan

Thanks for your patience while we looked into this! We just deployed a fix, so you should be all set now. Please make sure to recreate the endpoint, and let us know if you run into any issues.

---

<div class="post-metadata">

**Author:** ![jostelo](https://avatars.discourse-cdn.com/v4/letter/j/f6c823/32.png) [@jostelo](https://discuss.huggingface.co/u/jostelo)\
**Post date:** [January 12, 2024, 9:12am UTC](https://discuss.huggingface.co/t/inference-endpoint-fails-to-start/65809/15 "2024-01-12T09:12:54Z")

</div>

Works! Thank you very much!

---

<div class="post-metadata">

**Author:** ![IBLLMTST](https://avatars.discourse-cdn.com/v4/letter/i/7feea3/32.png) [@IBLLMTST](https://discuss.huggingface.co/u/IBLLMTST)\
**Post date:** [January 12, 2024, 3:46pm UTC](https://discuss.huggingface.co/t/inference-endpoint-fails-to-start/65809/16 "2024-01-12T15:46:36Z")

</div>

Thank you, Megan!

---

<div class="post-metadata">

**Author:** ![collinbarnwell](https://avatars.discourse-cdn.com/v4/letter/c/46a35a/32.png) [@collinbarnwell](https://discuss.huggingface.co/u/collinbarnwell)\
**Post date:** [February 9, 2024, 3:52pm UTC](https://discuss.huggingface.co/t/inference-endpoint-fails-to-start/65809/17 "2024-02-09T15:52:07Z")

</div>

Hi @meganariley - I’m still experiencing this issue. I’ve recreated my endpoint but I get the same error.

I’m using a fork of the pyannote/speaker-diarization-3.1 model so that I can create my own handler.

- I’ve tried using `path` in the ` __init__ ` function, which is set to “/repository”
- I’ve tried making my fork public and private
- I’ve tried using “pyannote/speaker-diarization-3.1” instead of path
- I’ve tried making my fork public and using that model name instead of path

But I always get this error:

`Server message:Endpoint failed to start. new fontManager /opt/conda/lib/python3.9/site-packages/pyannote/audio/core/io.py:43: UserWarning: torchaudio._backend.set_audio_backend has been deprecated. With dispatcher enabled, this function is no-op. You can remove the function call. torchaudio.set_audio_backend("soundfile") Traceback (most recent call last): File "/opt/conda/lib/python3.9/site-packages/starlette/routing.py", line 705, in lifespan async with self.lifespan_context(app) as maybe_state: File "/opt/conda/lib/python3.9/site-packages/starlette/routing.py", line 584, in __aenter__ await self._router.startup() File "/opt/conda/lib/python3.9/site-packages/starlette/routing.py", line 682, in startup await handler() File "/app/webservice_starlette.py", line 57, in some_startup_task inference_handler = get_inference_handler_either_custom_or_default_handler(HF_MODEL_DIR, task=HF_TASK) File "/app/huggingface_inference_toolkit/handler.py", line 41, in get_inference_handler_either_custom_or_default_handler custom_pipeline = check_and_register_custom_pipeline_from_directory(model_dir) File "/app/huggingface_inference_toolkit/utils.py", line 190, in check_and_register_custom_pipeline_from_directory custom_pipeline = handler.EndpointHandler(model_dir) File "/repository/handler.py", line 31, in __init__ self._pipeline = Pipeline.from_pretrained(path) File "/opt/conda/lib/python3.9/site-packages/pyannote/audio/core/pipeline.py", line 88, in from_pretrained config_yml = hf_hub_download( File "/opt/conda/lib/python3.9/site-packages/huggingface_hub/utils/_validators.py", line 110, in _inner_fn validate_repo_id(arg_value) File "/opt/conda/lib/python3.9/site-packages/huggingface_hub/utils/_validators.py", line 164, in validate_repo_id raise HFValidationError( huggingface_hub.utils._validators.HFValidationError: Repo id must use alphanumeric chars or '-', '_', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96: '/repository'. Application startup failed. Exiting.`

This is my handler:

```auto
from pyannote.audio import Pipeline, Audio
import torch

class EndpointHandler:
    def __init__ (self, path=""):
        # initialize pretrained pipeline
        self._pipeline = Pipeline.from_pretrained(path) # I've tried varations on this line as described above
        HYPER_PARAMETERS = {
            "segmentation": {
                "min_duration_off": 3.0,
            }
        }
        self._pipeline.instantiate(HYPER_PARAMETERS)

        # send pipeline to GPU if available
        if torch.cuda.is_available():
            self._pipeline.to(torch.device("cuda"))

        # initialize audio reader
        self._io = Audio()

```

I’ve tried this on a GPU and CPU and I get the same error. I’ve tried recreating the endpoint multiple times.

Please let me know what I can do.

Thanks,  
Collin