# How to push on hub a quantized model **URL:** \ **Category:** 🤗Hub\ **Created:** [July 13, 2024, 7:51am UTC](https://discuss.huggingface.co/t/how-to-push-on-hub-a-quantized-model/97113 "2024-07-13T07:51:50Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![Sohaibsoussi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sohaibsoussi/32/27805_2.png) [@Sohaibsoussi](https://discuss.huggingface.co/u/Sohaibsoussi)\ **Post date:** [July 13, 2024, 7:51am UTC](https://discuss.huggingface.co/t/how-to-push-on-hub-a-quantized-model/97113/1 "2024-07-13T07:51:50Z")
I quantized my model using torch but when I try to push model\_quantized\_qint8 to the hub it don’t work from torch.quantization import quantize\_dynamic model\_ckpt = “Sohaibsoussi/distilbert-base-uncased-distilled-finetuned-clinc” tokenizer = AutoTokenizer.from\_pretrained(model\_ckpt) model = AutoModelForSequenceClassification.from\_pretrained(model\_ckpt).to(“cpu”) # Dynamic Quantization for INT8 model\_quantized\_qint8 = quantize\_dynamic(model, {nn.Linear}, dtype=torch.qint8) { “name”: “TypeError”, “message”: “\_batch\_encode\_plus() got an unexpected keyword argument ‘dtype’”, “stack”: "--------------------------------------------------------------------------- TypeError Traceback (most recent call last) Cell In[169], line 5 2 optim\_type = "Distillation + Static Quantization" 4 pb = PerformanceBenchmark(pipe, clinc["test"], optim\_type=optim\_type) ----\> 5 perf\_metrics.update(pb.run\_benchmark()) Cell In[3], line 20, in PerformanceBenchmark.run\_benchmark(self) 18 metrics = {} 19 metrics[self.optim\_type] = self.compute\_size() —\> 20 metrics[self.optim\_type].update(self.time\_pipeline()) 21 metrics[self.optim\_type].update(self.compute\_accuracy()) 22 return metrics Cell In[17], line 8, in time\_pipeline(self, query) 6 # Warmup 7 for \_ in range(10): ----\> 8 \_ = self.pipeline(query) 9 # Timed Run 10 for \_ in range(100): File ~/miniconda3/envs/langchain/lib/python3.8/site-packages/transformers/pipelines/text\_classification.py:155, in TextClassificationPipeline. **call** (self, \*args, \*\*kwargs) 121 def **call** (self, \*args, \*\*kwargs): 122 """ 123 Classify the text(s) given as inputs. 124 (…) 153 If `top_k` is used, one such dictionary is returned per label. 154 """ → 155 result = super(). **call** (\*args, \*\*kwargs) 156 # TODO try and retrieve it in a nicer way from \_sanitize\_parameters. 157 \_legacy = "top\_k" not in kwargs File ~/miniconda3/envs/langchain/lib/python3.8/site-packages/transformers/pipelines/base.py:1196, in Pipeline. **call** (self, inputs, num\_workers, batch\_size, \*args, \*\*kwargs) 1188 return next( 1189 iter( 1190 self.get\_iterator( (…) 1193 ) 1194 ) 1195 else: → 1196 return self.run\_single(inputs, preprocess\_params, forward\_params, postprocess\_params) File ~/miniconda3/envs/langchain/lib/python3.8/site-packages/transformers/pipelines/base.py:1202, in Pipeline.run\_single(self, inputs, preprocess\_params, forward\_params, postprocess\_params) 1201 def run\_single(self, inputs, preprocess\_params, forward\_params, postprocess\_params): → 1202 model\_inputs = self.preprocess(inputs, \*\*preprocess\_params) 1203 model\_outputs = self.forward(model\_inputs, \*\*forward\_params) 1204 outputs = self.postprocess(model\_outputs, \*\*postprocess\_params) File ~/miniconda3/envs/langchain/lib/python3.8/site-packages/transformers/pipelines/text\_classification.py:179, in TextClassificationPipeline.preprocess(self, inputs, \*\*tokenizer\_kwargs) 173 elif isinstance(inputs, list): 174 # This is likely an invalid usage of the pipeline attempting to pass text pairs. 175 raise ValueError( 176 "The pipeline received invalid inputs, if you are trying to send text pairs, you can try to send a" 177 ’ dictionary `{\"text\": \"My text\", \"text_pair\": \"My pair\"}` in order to send a text pair.’ 178 ) → 179 return self.tokenizer(inputs, return\_tensors=return\_tensors, \*\*tokenizer\_kwargs) File ~/miniconda3/envs/langchain/lib/python3.8/site-packages/transformers/tokenization\_utils\_base.py:2829, in PreTrainedTokenizerBase. **call** (self, text, text\_pair, text\_target, text\_pair\_target, add\_special\_tokens, padding, truncation, max\_length, stride, is\_split\_into\_words, pad\_to\_multiple\_of, return\_tensors, return\_token\_type\_ids, return\_attention\_mask, return\_overflowing\_tokens, return\_special\_tokens\_mask, return\_offsets\_mapping, return\_length, verbose, \*\*kwargs) 2827 if not self.\_in\_target\_context\_manager: 2828 self.\_switch\_to\_input\_mode() → 2829 encodings = self.\_call\_one(text=text, text\_pair=text\_pair, \*\*all\_kwargs) 2830 if text\_target is not None: 2831 self.\_switch\_to\_target\_mode() File ~/miniconda3/envs/langchain/lib/python3.8/site-packages/transformers/tokenization\_utils\_base.py:2935, in PreTrainedTokenizerBase.\_call\_one(self, text, text\_pair, add\_special\_tokens, padding, truncation, max\_length, stride, is\_split\_into\_words, pad\_to\_multiple\_of, return\_tensors, return\_token\_type\_ids, return\_attention\_mask, return\_overflowing\_tokens, return\_special\_tokens\_mask, return\_offsets\_mapping, return\_length, verbose, \*\*kwargs) 2915 return self.batch\_encode\_plus( 2916 batch\_text\_or\_text\_pairs=batch\_text\_or\_text\_pairs, 2917 add\_special\_tokens=add\_special\_tokens, (…) 2932 \*\*kwargs, 2933 ) 2934 else: → 2935 return self.encode\_plus( 2936 text=text, 2937 text\_pair=text\_pair, 2938 add\_special\_tokens=add\_special\_tokens, 2939 padding=padding, 2940 truncation=truncation, 2941 max\_length=max\_length, 2942 stride=stride, 2943 is\_split\_into\_words=is\_split\_into\_words, 2944 pad\_to\_multiple\_of=pad\_to\_multiple\_of, 2945 return\_tensors=return\_tensors, 2946 return\_token\_type\_ids=return\_token\_type\_ids, 2947 return\_attention\_mask=return\_attention\_mask, 2948 return\_overflowing\_tokens=return\_overflowing\_tokens, 2949 return\_special\_tokens\_mask=return\_special\_tokens\_mask, 2950 return\_offsets\_mapping=return\_offsets\_mapping, 2951 return\_length=return\_length, 2952 verbose=verbose, 2953 \*\*kwargs, 2954 ) File ~/miniconda3/envs/langchain/lib/python3.8/site-packages/transformers/tokenization\_utils\_base.py:3008, in PreTrainedTokenizerBase.encode\_plus(self, text, text\_pair, add\_special\_tokens, padding, truncation, max\_length, stride, is\_split\_into\_words, pad\_to\_multiple\_of, return\_tensors, return\_token\_type\_ids, return\_attention\_mask, return\_overflowing\_tokens, return\_special\_tokens\_mask, return\_offsets\_mapping, return\_length, verbose, \*\*kwargs) 2998 # Backward compatibility for ‘truncation\_strategy’, ‘pad\_to\_max\_length’ 2999 padding\_strategy, truncation\_strategy, max\_length, kwargs = self.\_get\_padding\_truncation\_strategies( 3000 padding=padding, 3001 truncation=truncation, (…) 3005 \*\*kwargs, 3006 ) → 3008 return self.\_encode\_plus( 3009 text=text, 3010 text\_pair=text\_pair, 3011 add\_special\_tokens=add\_special\_tokens, 3012 padding\_strategy=padding\_strategy, 3013 truncation\_strategy=truncation\_strategy, 3014 max\_length=max\_length, 3015 stride=stride, 3016 is\_split\_into\_words=is\_split\_into\_words, 3017 pad\_to\_multiple\_of=pad\_to\_multiple\_of, 3018 return\_tensors=return\_tensors, 3019 return\_token\_type\_ids=return\_token\_type\_ids, 3020 return\_attention\_mask=return\_attention\_mask, 3021 return\_overflowing\_tokens=return\_overflowing\_tokens, 3022 return\_special\_tokens\_mask=return\_special\_tokens\_mask, 3023 return\_offsets\_mapping=return\_offsets\_mapping, 3024 return\_length=return\_length, 3025 verbose=verbose, 3026 \*\*kwargs, 3027 ) File ~/miniconda3/envs/langchain/lib/python3.8/site-packages/transformers/tokenization\_utils\_fast.py:576, in PreTrainedTokenizerFast.\_encode\_plus(self, text, text\_pair, add\_special\_tokens, padding\_strategy, truncation\_strategy, max\_length, stride, is\_split\_into\_words, pad\_to\_multiple\_of, return\_tensors, return\_token\_type\_ids, return\_attention\_mask, return\_overflowing\_tokens, return\_special\_tokens\_mask, return\_offsets\_mapping, return\_length, verbose, \*\*kwargs) 554 def \_encode\_plus( 555 self, 556 text: Union[TextInput, PreTokenizedInput], (…) 573 \*\*kwargs, 574 ) → BatchEncoding: 575 batched\_input = [(text, text\_pair)] if text\_pair else [text] → 576 batched\_output = self.\_batch\_encode\_plus( 577 batched\_input, 578 is\_split\_into\_words=is\_split\_into\_words, 579 add\_special\_tokens=add\_special\_tokens, 580 padding\_strategy=padding\_strategy, 581 truncation\_strategy=truncation\_strategy, 582 max\_length=max\_length, 583 stride=stride, 584 pad\_to\_multiple\_of=pad\_to\_multiple\_of, 585 return\_tensors=return\_tensors, 586 return\_token\_type\_ids=return\_token\_type\_ids, 587 return\_attention\_mask=return\_attention\_mask, 588 return\_overflowing\_tokens=return\_overflowing\_tokens, 589 return\_special\_tokens\_mask=return\_special\_tokens\_mask, 590 return\_offsets\_mapping=return\_offsets\_mapping, 591 return\_length=return\_length, 592 verbose=verbose, 593 \*\*kwargs, 594 ) 596 # Return tensor is None, then we can remove the leading batch axis 597 # Overflowing tokens are returned as a batch of output so we keep them in this case 598 if return\_tensors is None and not return\_overflowing\_tokens: TypeError: \_batch\_encode\_plus() got an unexpected keyword argument ‘dtype’" }