# Payload too large for Async Inference on Sagemaker

**URL:** <https://discuss.huggingface.co/t/payload-too-large-for-async-inference-on-sagemaker/40717>\
**Category:** Amazon SageMaker\
**Created:** [May 23, 2023, 9:51am UTC](https://discuss.huggingface.co/t/payload-too-large-for-async-inference-on-sagemaker/40717 "2023-05-23T09:51:12Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![philhd](https://avatars.discourse-cdn.com/v4/letter/p/ea5d25/32.png) [@philhd](https://discuss.huggingface.co/u/philhd)\
**Post date:** [May 23, 2023, 9:51am UTC](https://discuss.huggingface.co/t/payload-too-large-for-async-inference-on-sagemaker/40717/1 "2023-05-23T09:51:12Z")

</div>

In order to transcribe audio files withe the whisper model I would assume that the asyn inference option on AWS sagemaker might be the right choice for long audio files (1 hour, around 5-50mb).

According to the docs it should be possible to have [payload](https://docs.aws.amazon.com/sagemaker/latest/dg/deploy-model.html) sizes up to 1gb

I followed philipp Schmids article here

> **[Asynchronous Inference with Hugging Face Transformers and Amazon SageMaker](https://www.philschmid.de/sagemaker-huggingface-async-inference#create-inference-huggingfacemodel-for-the-asynchronous-inference-endpoint)**
>
> Learn how to deploy an Asynchronous Inference model with Hugging Face Transformers and Amazon SageMaker, with autoscaling to zero.

but I do get the following error which is surprising to me, since my payload is around 11mb.

> Received client error (413) from primary and could not load the entire response body

```auto

hub = {
        'HF_MODEL_ID': 'openai/whisper-base',
        'HF_TASK': 'automatic-speech-recognition'
    }

    # create Hugging Face Model Class
    huggingface_model = HuggingFaceModel(
        env=hub, # configuration for loading model from Hub
        role=role, # iam role with permissions to create an Endpoint
        transformers_version="4.26", # transformers version used
        pytorch_version="1.13", # pytorch version used
        py_version='py39', # python version used
    )

    # create async endpoint configuration
    async_config = AsyncInferenceConfig(
        output_path=s3_path_join("s3://", sagemaker_session_bucket, "async_inference/output"),
        # Where our results will be stored
        # notification_config={
        # "SuccessTopic": "arn:aws:sns:us-east-2:123456789012:MyTopic",
        # "ErrorTopic": "arn:aws:sns:us-east-2:123456789012:MyTopic",
        # }, # Notification configuration
    )

    # deploy the endpoint
    huggingface_model.deploy(
        initial_instance_count=1,
        instance_type="ml.m5.xlarge", # ml.g4dn.xlarge,
        async_inference_config=async_config
    )

def predict():
    session = boto3.session.Session()
    sagemaker_session = sagemaker.Session(session)

    predictor = HuggingFacePredictor(endpoint_name=endpoint_name,
                                     sagemaker_session=sagemaker_session,
                                     serializer=audio_serializer
                                     )
    async_predictor = AsyncPredictor(predictor)

    ASYNC_S3_PATH = "s3://async-inf/async-distilbert"

    with open(audio_path, "rb") as data_file:
        audio_data = data_file.read()

        data = {
            "s3_file": "s3://async-inf/async-distilbert"
            # "language": "pl"
        }
        res = async_predictor.predict_async(input_path="s3://async-inf/async-distilbert")
        # res = async_predictor.predict_async(data=audio_data, input_path=ASYNC_S3_PATH)
        config = WaiterConfig(
            max_attempts=5, # number of attempts
            delay=10 # time in seconds to wait between attempts
        )

        res.get_result(config)
        print(res)

```

@philschmid Any idea about how to post large payloads to the async endpoint?  
Anyhow thanks a lot for your tireless support. Very much appreciated.

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [May 23, 2023, 10:24am UTC](https://discuss.huggingface.co/t/payload-too-large-for-async-inference-on-sagemaker/40717/2 "2023-05-23T10:24:49Z")

</div>

Hey @philhd,

There might be a minor error in the inference code, it seems that `"s3://async-inf/async-distilbert"` is not pointing to a “file” only to a “directory”

---

<div class="post-metadata">

**Author:** ![philhd](https://avatars.discourse-cdn.com/v4/letter/p/ea5d25/32.png) [@philhd](https://discuss.huggingface.co/u/philhd)\
**Post date:** [May 23, 2023, 10:29am UTC](https://discuss.huggingface.co/t/payload-too-large-for-async-inference-on-sagemaker/40717/3 "2023-05-23T10:29:02Z")

</div>

Hey @philschmid , thanks for the swift reply. This `s3://async-inf/async-distilbert` is actually the file. I missed the file ending. I tried to rename it in the bucket and the code accordingly `input_path="s3://async-inf/async-distilbert.mp3"` but no change .

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [May 25, 2023, 12:27pm UTC](https://discuss.huggingface.co/t/payload-too-large-for-async-inference-on-sagemaker/40717/4 "2023-05-25T12:27:10Z")

</div>

Have you tried creating a custom `inference.py` script to log some information, e.g. if the data file gets correct passed into the handler?

---

<div class="post-metadata">

**Author:** ![philhd](https://avatars.discourse-cdn.com/v4/letter/p/ea5d25/32.png) [@philhd](https://discuss.huggingface.co/u/philhd)\
**Post date:** [May 25, 2023, 12:35pm UTC](https://discuss.huggingface.co/t/payload-too-large-for-async-inference-on-sagemaker/40717/5 "2023-05-25T12:35:55Z")

</div>

I got it up and running by doing it slightly different

```auto
def infer_async():
    sagemaker_runtime = boto3.client("sagemaker-runtime")

    # Specify the location of the input. Should be JSON with the input audion file (example in 02_deploy_whisper-Async.ipynb notebook)
    input_location = "s3://async-inf/input.json"

    # The name of the endpoint. The name must be unique within an AWS Region in your AWS account.

    # After you deploy a model using SageMaker hosting
    # services, your client applications use this API to get inferences
    # from the model hosted at the specified endpoint.
    response = sagemaker_runtime.invoke_endpoint_async(
        EndpointName=endpoint_name,
        # ContentType='audio/mpeg',
        InputLocation=input_location,
    )
    print(response)

```

---

<div class="post-metadata">

**Author:** ![xolisani](https://avatars.discourse-cdn.com/v4/letter/x/ea666f/32.png) [@xolisani](https://discuss.huggingface.co/u/xolisani)\
**Post date:** [June 6, 2023, 1:37pm UTC](https://discuss.huggingface.co/t/payload-too-large-for-async-inference-on-sagemaker/40717/6 "2023-06-06T13:37:21Z")

</div>

Whats the structure of input.json

---

<div class="post-metadata">

**Author:** ![xolisani](https://avatars.discourse-cdn.com/v4/letter/x/ea666f/32.png) [@xolisani](https://discuss.huggingface.co/u/xolisani)\
**Post date:** [June 7, 2023, 1:58pm UTC](https://discuss.huggingface.co/t/payload-too-large-for-async-inference-on-sagemaker/40717/7 "2023-06-07T13:58:49Z")

</div>

Whats the structure of input.json. I get ann error saying “No such file or directory: \u0027s3://”

---

<div class="post-metadata">

**Author:** ![philhd](https://avatars.discourse-cdn.com/v4/letter/p/ea5d25/32.png) [@philhd](https://discuss.huggingface.co/u/philhd)\
**Post date:** [June 9, 2023, 7:00am UTC](https://discuss.huggingface.co/t/payload-too-large-for-async-inference-on-sagemaker/40717/8 "2023-06-09T07:00:10Z")

</div>

depends on your inference script. you can try

```auto
"s3_location" : "path_to_s3"

```

---

<div class="post-metadata">

**Author:** ![philhd](https://avatars.discourse-cdn.com/v4/letter/p/ea5d25/32.png) [@philhd](https://discuss.huggingface.co/u/philhd)\
**Post date:** [June 9, 2023, 7:04am UTC](https://discuss.huggingface.co/t/payload-too-large-for-async-inference-on-sagemaker/40717/9 "2023-06-09T07:04:01Z")

</div>

For AsyncInference there is another very important configuration required to prevent the 413 error.

```auto

env={
        'MMS_MAX_REQUEST_SIZE': '2000000000',
        'MMS_MAX_RESPONSE_SIZE': '2000000000',
        'MMS_DEFAULT_RESPONSE_TIMEOUT': '900'
    }

```

HuggingFaceModel(env=env …)

@philschmid  
would be nice to have it mentioned in the documentation