# Stuck starting inference model **URL:** \ **Category:** Inference Endpoints on the Hub\ **Created:** [August 16, 2023, 10:08pm UTC](https://discuss.huggingface.co/t/stuck-starting-inference-model/51018 "2023-08-16T22:08:09Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![vadman97](https://avatars.discourse-cdn.com/v4/letter/v/13edae/32.png) [@vadman97](https://discuss.huggingface.co/u/vadman97)\ **Post date:** [August 16, 2023, 10:08pm UTC](https://discuss.huggingface.co/t/stuck-starting-inference-model/51018/1 "2023-08-16T22:08:09Z")
Seeing message `[Server message]Load balancer not ready yet` for a while now on [Inference Endpoints - Hugging Face](https://ui.endpoints.huggingface.co/vadman97/endpoints/aws-gte-large-7774). Have tried to recreate the inference endpoint, change the instance size, but all without luck. Could I get some help please? ---
**Author:** ![ncolina](https://avatars.discourse-cdn.com/v4/letter/n/4491bb/32.png) [@ncolina](https://discuss.huggingface.co/u/ncolina)\ **Post date:** [August 17, 2023, 12:24am UTC](https://discuss.huggingface.co/t/stuck-starting-inference-model/51018/2 "2023-08-17T00:24:19Z")
I’m experiencing the same issue hosting llama2. I’ve even tried switching out different versions of the llama2 model 7b and 13b. These are the logs that get generated before it gets stuck. ````auto 2023/08/16 17:19:01 ~ INFO | Repository ID: meta-llama/Llama-2-7b-chat-hf 2023/08/16 17:19:01 ~ INFO | Repository Revision: 08751db2aca9bf2f7f80d2e516117a53d7450235 2023/08/16 17:19:01 ~ INFO | Start loading image artifacts from huggingface.co 2023/08/16 17:19:01 ~ INFO | Ignore regex pattern for files, which are not downloaded: tf*, flax*, rust*, *onnx, *safetensors, *mlmodel, *tflite, *tar.gz, *ckpt 2023/08/16 17:19:01 ~ INFO | Used configuration: 2023/08/16 17:20:27 ~ {"timestamp":"2023-08-17T00:20:27.894791Z","level":"INFO","fields":{"message":"Args { model_id: \"/repository\", revision: None, validation_workers: 2, sharded: None, num_shard: None, quantize: None, dtype: None, trust_remote_code: false, max_concurrent_requests: 128, max_best_of: 2, max_stop_sequences: 4, max_input_length: 1024, max_total_tokens: 1512, waiting_served_ratio: 1.2, max_batch_prefill_tokens: 2048, max_batch_total_tokens: None, max_waiting_tokens: 20, hostname: \"ackplain9886-aws-llama-2-7b-chat-hf-9863-67bf75fb89-6gn6n\", port: 80, shard_uds_path: \"/tmp/text-generation-server\", master_addr: \"localhost\", master_port: 29500, huggingface_hub_cache: Some(\"/data\"), weights_cache_override: None, disable_custom_kernels: false, cuda_memory_fraction: 1.0, rope_scaling: None, rope_factor: None, json_output: true, otlp_endpoint: None, cors_allow_origin: [], watermark_gamma: None, watermark_delta: None, ngrok: false, ngrok_authtoken: None, ngrok_edge: None, env: false }"},"target":"text_generation_launcher"} 2023/08/16 17:20:27 ~ {"timestamp":"2023-08-17T00:20:27.894934Z","level":"INFO","fields":{"message":"Starting download process."},"target":"text_generation_launcher","span":{"name":"download"},"spans":[{"name":"download"}]} 2023/08/16 17:20:30 ~ {"timestamp":"2023-08-17T00:20:30.179786Z","level":"WARN","fields":{"message":"No safetensors weights found for model /repository at revision None. Converting PyTorch weights to safetensors.\n"},"target":"text_generation_launcher"} 2023/08/16 17:21:36 ~ {"timestamp":"2023-08-17T00:21:36.274320Z","level":"INFO","fields":{"message":"Convert: [1/2] -- Took: 0:01:06.093642\n"},"target":"text_generation_launcher"} 2023/08/16 17:21:41 ~ {"timestamp":"2023-08-17T00:21:41.052882Z","level":"INFO","fields":{"message":"Convert: [2/2] -- Took: 0:00:04.778280\n"},"target":"text_generation_launcher"} 2023/08/16 17:21:41 ~ {"timestamp":"2023-08-17T00:21:41.653348Z","level":"INFO","fields":{"message":"Successfully downloaded weights."},"target":"text_generation_launcher","span":{"name":"download"},"spans":[{"name":"download"}]} 2023/08/16 17:21:41 ~ {"timestamp":"2023-08-17T00:21:41.653577Z","level":"INFO","fields":{"message":"Starting shard"},"target":"text_generation_launcher","span":{"rank":0,"name":"shard-manager"},"spans":[{"rank":0,"name":"shard-manager"}]} 2023/08/16 17:21:49 ~ {"timestamp":"2023-08-17T00:21:49.039672Z","level":"INFO","fields":{"message":"Server started at unix:///tmp/text-generation-server-0\n"},"target":"text_generation_launcher"} 2023/08/16 17:21:49 ~ {"timestamp":"2023-08-17T00:21:49.060661Z","level":"INFO","fields":{"message":"Shard ready in 7.406491831s"},"target":"text_generation_launcher","span":{"rank":0,"name":"shard-manager"},"spans":[{"rank":0,"name":"shard-manager"}]} 2023/08/16 17:21:49 ~ {"timestamp":"2023-08-17T00:21:49.159300Z","level":"INFO","fields":{"message":"Starting Webserver"},"target":"text_generation_launcher"} 2023/08/16 17:21:49 ~ {"timestamp":"2023-08-17T00:21:49.215664Z","level":"WARN","message":"no pipeline tag found for model /repository","target":"text_generation_router","filename":"router/src/main.rs","line_number":191} 2023/08/16 17:21:49 ~ {"timestamp":"2023-08-17T00:21:49.220354Z","level":"INFO","message":"Warming up model","target":"text_generation_router","filename":"router/src/main.rs","line_number":210} 2023/08/16 17:21:50 ~ {"timestamp":"2023-08-17T00:21:50.528107Z","level":"INFO","message":"Connected","target":"text_generation_router","filename":"router/src/main.rs","line_number":244} 2023/08/16 17:21:50 ~ {"timestamp":"2023-08-17T00:21:50.528113Z","level":"WARN","message":"Invalid hostname, defaulting to 0.0.0.0","target":"text_generation_router","filename":"router/src/main.rs","line_number":249} 2023/08/16 17:21:50 ~ {"timestamp":"2023-08-17T00:21:50.528075Z","level":"INFO","message":"Setting max batch total tokens to 15440","target":"text_generation_router","filename":"router/src/main.rs","line_number":243}``` ```` ---
**Author:** ![vadman97](https://avatars.discourse-cdn.com/v4/letter/v/13edae/32.png) [@vadman97](https://discuss.huggingface.co/u/vadman97)\ **Post date:** [August 17, 2023, 12:40am UTC](https://discuss.huggingface.co/t/stuck-starting-inference-model/51018/3 "2023-08-17T00:40:02Z")
same here @ncolina, i’m using gte-large and the model seems to start, but the load balancer never does ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/b/9/b99097dc2545e260b1b7143accbcdb72482f6709.png) ---
**Author:** ![michellehbn](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/michellehbn/32/58964_2.png) [@michellehbn](https://discuss.huggingface.co/u/michellehbn)\ **Post date:** [August 18, 2023, 12:13pm UTC](https://discuss.huggingface.co/t/stuck-starting-inference-model/51018/4 "2023-08-18T12:13:36Z")
Hi @vadman97, @ncolina - Thanks for reporting and sorry for the wait! This load balancer issue should now be resolved. Please let us know though if you continue to encounter an issue when deploying the endpoint. Thanks again! ---
**Author:** ![floripaolo](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/floripaolo/32/21438_2.png) [@floripaolo](https://discuss.huggingface.co/u/floripaolo)\ **Post date:** [November 23, 2023, 9:28am UTC](https://discuss.huggingface.co/t/stuck-starting-inference-model/51018/5 "2023-11-23T09:28:25Z")
It’s been more than half an hour now that the load balancer is stuck… Server message:Load balancer not ready yet ![Screenshot 2023-11-23 at 10.27.55](https://us1.discourse-cdn.com/hellohellohello/original/3X/3/f/3fdd3778ab13e2abde05ff3551c80945e2177db5.png) ---
**Author:** ![michellehbn](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/michellehbn/32/58964_2.png) [@michellehbn](https://discuss.huggingface.co/u/michellehbn)\ **Post date:** [November 23, 2023, 3:24pm UTC](https://discuss.huggingface.co/t/stuck-starting-inference-model/51018/6 "2023-11-23T15:24:11Z")
Hi @floripaolo, Thanks for letting us know. We’re taking a look and we’ll get back to you soon. ---
**Author:** ![floripaolo](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/floripaolo/32/21438_2.png) [@floripaolo](https://discuss.huggingface.co/u/floripaolo)\ **Post date:** [November 23, 2023, 4:55pm UTC](https://discuss.huggingface.co/t/stuck-starting-inference-model/51018/7 "2023-11-23T16:55:00Z")
Restarting solved the problem.