# Return all class labels from SageMaker invoke\_endpoint

**URL:** <https://discuss.huggingface.co/t/return-all-class-labels-from-sagemaker-invoke-endpoint/13669>\
**Category:** Amazon SageMaker\
**Created:** [January 13, 2022, 9:39pm UTC](https://discuss.huggingface.co/t/return-all-class-labels-from-sagemaker-invoke-endpoint/13669 "2022-01-13T21:39:12Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![willivz](https://avatars.discourse-cdn.com/v4/letter/w/ba8739/32.png) [@willivz](https://discuss.huggingface.co/u/willivz)\
**Post date:** [January 13, 2022, 9:39pm UTC](https://discuss.huggingface.co/t/return-all-class-labels-from-sagemaker-invoke-endpoint/13669/1 "2022-01-13T21:39:12Z")

</div>

I’ve deployed the Hate-speech-CNERG/bert-base-uncased-hatexplain multi-class text classification model (along with others) to a SageMaker endpoint. Using the TextClassificationPipeline you’re able to pass in the return\_all\_scores=True parameter to see all class labels for the input.

However when using the SageMakerRuntime.invoke\_endpoint function, I can only get one class per input. Anyone have any thoughts on an equivalent to the return\_all\_scores parameter?

---

<div class="post-metadata">

**Author:** ![marshmellow77](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marshmellow77/32/3279_2.png) [@marshmellow77](https://discuss.huggingface.co/u/marshmellow77)\
**Post date:** [January 14, 2022, 6:18am UTC](https://discuss.huggingface.co/t/return-all-class-labels-from-sagemaker-invoke-endpoint/13669/2 "2022-01-14T06:18:24Z")

</div>

Hi Will, when deploying a model to a Sagemaker endpoint you can provide a custom inference script in which you can control the behaviour of the model for inference requests. You can find the documentation on how to do this here: [Deploy models to Amazon SageMaker](https://huggingface.co/docs/sagemaker/inference#user-defined-code-and-modules)

In your particular scenario I would think hat you can override the `predict_fn` and/or the `output_fn` methods to get to the desired result. You can find an example for an `inference.py` file [here](https://github.com/marshmellow77/text-summarisation-project/blob/main/inference_code/inference.py) (note that this particular file is for text summarization, so it won’t apply for your use case).

To see what code you need to get all scores returned you can refer back to the code from the TextClassification Pipeline, that you have already mentioned: [transformers/text\_classification.py at master · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/master/src/transformers/pipelines/text_classification.py#L163). If you implement this code into your inference script you should be all set.

Hope that helps, let me know if any questions.

Cheers  
Heiko

---

<div class="post-metadata">

**Author:** ![lamwilton](https://avatars.discourse-cdn.com/v4/letter/l/7ab992/32.png) [@lamwilton](https://discuss.huggingface.co/u/lamwilton)\
**Post date:** [January 20, 2022, 9:28pm UTC](https://discuss.huggingface.co/t/return-all-class-labels-from-sagemaker-invoke-endpoint/13669/3 "2022-01-20T21:28:55Z")

</div>

I would like to do the same too. Has anyone tried it yet?

---

<div class="post-metadata">

**Author:** ![marshmellow77](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marshmellow77/32/3279_2.png) [@marshmellow77](https://discuss.huggingface.co/u/marshmellow77)\
**Post date:** [January 21, 2022, 5:29am UTC](https://discuss.huggingface.co/t/return-all-class-labels-from-sagemaker-invoke-endpoint/13669/4 "2022-01-21T05:29:06Z")

</div>

I just had a look into the code of the Huggingface Sagemaker Inference Toolkit and found an even easier way to accomplish this: You can pass `parameters` in your inference request: [sagemaker-huggingface-inference-toolkit/handler\_service.py at main · aws/sagemaker-huggingface-inference-toolkit · GitHub](https://github.com/aws/sagemaker-huggingface-inference-toolkit/blob/main/src/sagemaker_huggingface_inference_toolkit/handler_service.py#L152)

That’s awesome, because it means you don’t even have to provide an `inference.py` script! 🤗

All you need to do is:

```auto
predictor.predict({"inputs": "I love using the new Inference DLC.", "parameters": {"return_all_scores": True}})

```

 ![Capture](https://us1.discourse-cdn.com/hellohellohello/original/2X/a/aa5d5dc13903aa7178d2b905ccf07b561946f606.png)

Hope that helps!

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [January 21, 2022, 7:34am UTC](https://discuss.huggingface.co/t/return-all-class-labels-from-sagemaker-invoke-endpoint/13669/5 "2022-01-21T07:34:27Z")

</div>

We have also a documentation section available for how the API works: [Reference](https://huggingface.co/docs/sagemaker/reference#inference-toolkit-api)

> The Inference Toolkit accepts inputs in the `inputs` key, and supports additional [`pipelines`](https://huggingface.co/transformers/main_classes/pipelines.html) parameters in the `parameters` key. You can provide any of the supported `kwargs` from `pipelines` as `parameters` .

```python
{
  "inputs": "This sound track was beautiful! It paints the senery in your mind so well I would recomend it
  even to people who hate vid. game music!", 
 "parameters": {
        "return_all_scores": True
  }
}

```

---

<div class="post-metadata">

**Author:** ![lamwilton](https://avatars.discourse-cdn.com/v4/letter/l/7ab992/32.png) [@lamwilton](https://discuss.huggingface.co/u/lamwilton)\
**Post date:** [January 21, 2022, 11:37am UTC](https://discuss.huggingface.co/t/return-all-class-labels-from-sagemaker-invoke-endpoint/13669/6 "2022-01-21T11:37:00Z")

</div>

Thanks both of you for your help! I will go ahead and try that

---

<div class="post-metadata">

**Author:** ![lamwilton](https://avatars.discourse-cdn.com/v4/letter/l/7ab992/32.png) [@lamwilton](https://discuss.huggingface.co/u/lamwilton)\
**Post date:** [January 22, 2022, 1:19am UTC](https://discuss.huggingface.co/t/return-all-class-labels-from-sagemaker-invoke-endpoint/13669/7 "2022-01-22T01:19:33Z")

</div>

@marshmellow77  
I tried it on Sagemaker studio and it still gives only one label. May I ask you to please look at the screenshot to see if anything is not right? Thank you

 ![2022-01-21_201649](https://us1.discourse-cdn.com/hellohellohello/original/2X/6/6fae00df65ff22c37fa3d43a6e1c4d7543c46d54.jpeg)

---

<div class="post-metadata">

**Author:** ![marshmellow77](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marshmellow77/32/3279_2.png) [@marshmellow77](https://discuss.huggingface.co/u/marshmellow77)\
**Post date:** [January 22, 2022, 4:46am UTC](https://discuss.huggingface.co/t/return-all-class-labels-from-sagemaker-invoke-endpoint/13669/8 "2022-01-22T04:46:56Z")

</div>

All looking good except that you are using an old version of the transformers library. Try `transformers_version=“4.12.3”`, `pytorch_version=“1.9.1”`, and `py_version=“py38”`. That should work 🙂

Just for reference, you can find the latest version of the DLCs here: [Reference](https://huggingface.co/docs/sagemaker/reference#inference-dlc-overview)

Cheers  
Heiko

---

<div class="post-metadata">

**Author:** ![lamwilton](https://avatars.discourse-cdn.com/v4/letter/l/7ab992/32.png) [@lamwilton](https://discuss.huggingface.co/u/lamwilton)\
**Post date:** [January 22, 2022, 5:01am UTC](https://discuss.huggingface.co/t/return-all-class-labels-from-sagemaker-invoke-endpoint/13669/9 "2022-01-22T05:01:30Z")

</div>

It finally worked. Thanks Marshmellow!

 ![2022-01-22_000035](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/df3483cae4c921620b74dd49ca37ea4b2fb6d500.jpeg)