# TGI doesn't like quantization option on aws inf2 **URL:** \ **Category:** Amazon Inferentia & Trainium\ **Created:** [August 8, 2024, 9:15pm UTC](https://discuss.huggingface.co/t/tgi-doesnt-like-quantization-option-on-aws-inf2/101581 "2024-08-08T21:15:54Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![arisin](https://avatars.discourse-cdn.com/v4/letter/a/a87d85/32.png) [@arisin](https://discuss.huggingface.co/u/arisin)\ **Post date:** [August 8, 2024, 9:15pm UTC](https://discuss.huggingface.co/t/tgi-doesnt-like-quantization-option-on-aws-inf2/101581/1 "2024-08-08T21:15:54Z")
Hi All, Facing this error when running TGI on inf2 Error: `2024-08-08T21:14:29.441172Z INFO text_generation_launcher: Default`max\_batch\_prefill\_tokens` to 32768 2024-08-08T21:14:29.441176Z INFO text\_generation\_launcher: Using default cuda graphs [1, 2, 4, 8, 16, 32] 2024-08-08T21:14:29.441256Z INFO download: text\_generation\_launcher: Starting download process. 2024-08-08T21:14:29.514298Z WARN text\_generation\_launcher: ‘extension’ argument is not supported and will be ignored. 2024-08-08T21:14:32.644926Z INFO download: text\_generation\_launcher: Successfully downloaded weights. 2024-08-08T21:14:32.645248Z INFO shard-manager: text\_generation\_launcher: Starting shard rank=0 2024-08-08T21:14:32.746522Z ERROR shard-manager: text\_generation\_launcher: Shard complete standard error output: Usage: text-generation-server serve [OPTIONS] MODEL\_ID Try ‘text-generation-server serve --help’ for help. Error: No such option: --quantize rank=0 Error: ShardCannotStart 2024-08-08T21:14:32.845269Z ERROR text\_generation\_launcher: Shard 0 failed to start 2024-08-08T21:14:32.845293Z INFO text\_generation\_launcher: Shutting down shards ` Command to start: `docker run -p 8000:80 -v $(pwd)/data:/data --privileged ghcr.io/huggingface/neuronx-tgi:latest --model-id /data/meta/ --quantize awq` Appreciate any guidance to get this to work.