# Batch\_transform Pipeline?

**URL:** <https://discuss.huggingface.co/t/batch-transform-pipeline/10275>\
**Category:** Amazon SageMaker\
**Created:** [September 24, 2021, 12:56pm UTC](https://discuss.huggingface.co/t/batch-transform-pipeline/10275 "2021-09-24T12:56:01Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![marlon89](https://avatars.discourse-cdn.com/v4/letter/m/6de8d8/32.png) [@marlon89](https://discuss.huggingface.co/u/marlon89)\
**Post date:** [September 24, 2021, 12:56pm UTC](https://discuss.huggingface.co/t/batch-transform-pipeline/10275/1 "2021-09-24T12:56:01Z")

</div>

Hello,

I fine tuned 2 BERT models which (1.) classify different customer reviews and tag them with different labels and (2.) detect the sentiment in each text. I process these texts in a weekly batch on AWS Sagemaker. Right now I am writing two different batch transform jobs which (1.) predict the class & (2.) predict the text. My question now if it is possible to integrate both models to one batchtransform job. My fine tuned models are in my S3 bucket in the tar.gz format and my code currently looks like this:

```auto
# package the inference scrip and pre-trained classifier model into .tar.gz format
!cd model_token && tar zcvf model.tar.gz * 
!mv model_token/model.tar.gz ./model.tar.gz

# upload pre-trained classifier model to s3 bucket
model_url = s3_path_join("s3://",sagemaker_session_bucket,"batch_transform/model")
print(f"Uploading Model to {model_url}")
model_uri = S3Uploader.upload('model.tar.gz',model_url)
print(f"Uploaded model to {model_uri}")

#same procedure for sentiment model
!cd sentiment_token && tar zcvf sentiment.tar.gz * 
!mv sentiment_token/sentiment.tar.gz ./sentiment.tar.gz
model_url = s3_path_join("s3://",sagemaker_session_bucket,"batch_transform/model")
print(f"Uploading Model to {model_url}")
sentiment_uri = S3Uploader.upload('sentiment.tar.gz',model_url)
print(f"Uploaded model to {sentiment_uri}")

from sagemaker.huggingface.model import HuggingFaceModel

# create Hugging Face Model Class for classifier
huggingface_model = HuggingFaceModel(
   model_data=model_uri, # configuration for loading model from Hub
   role=role, # iam role with permissions to create an Endpoint
   transformers_version="4.6", # transformers version used
   pytorch_version="1.7", # pytorch version used
   py_version='py36', # python version used
)

# create Transformer to run our batch job
batch_job = huggingface_model.transformer(
    instance_count=1,
    instance_type='ml.g4dn.xlarge',
    output_path=output_s3_path, # we are using the same s3 path to save the output with the input
    strategy='SingleRecord')

# starts batch transform job and uses s3 data as input
batch_job.transform(
    data=s3_file_uri,
    content_type='application/json',    
    split_type='Line')

#same for sentiment 
huggingface_model = HuggingFaceModel(
   model_data=sentiment_uri, # configuration for loading model from Hub
   role=role, # iam role with permissions to create an Endpoint
   transformers_version="4.6", # transformers version used
   pytorch_version="1.7", # pytorch version used
   py_version='py36', # python version used
)

# create Transformer to run our batch job
batch_job = huggingface_model.transformer(
    instance_count=1,
    instance_type='ml.g4dn.xlarge',
    output_path=output_s3_path, # we are using the same s3 path to save the output with the input
    strategy='SingleRecord')

# starts batch transform job and uses s3 data as input
batch_job.transform(
    data=s3_file_uri,
    content_type='application/json',    
    split_type='Line')

```

Thanks in advance!

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [September 27, 2021, 7:52am UTC](https://discuss.huggingface.co/t/batch-transform-pipeline/10275/2 "2021-09-27T07:52:42Z")

</div>

Hey @marlon89,

Currently, does Batch transform doesn’t support multi-model endpoints. But what you could do is create a customer `inference.py` which contains two models and runs prediction against both.  
Or you could use something like [SageMaker Pipelines](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-transform), AWS Lambda function to create an automated pipeline that takes care of it so you don’t need to run your batch transform jobs.

Or you could add `wait=False` to `batch_job.transform` which tells the SDK not to wait until the job completes and your jobs would run in “parallel”.

The probably easiest way to start and automate is to create an AWS Lambda function with a cloudwatch trigger (run every week on day X) and start your batch transform jobs with it.

---

<div class="post-metadata">

**Author:** ![marlon89](https://avatars.discourse-cdn.com/v4/letter/m/6de8d8/32.png) [@marlon89](https://discuss.huggingface.co/u/marlon89)\
**Post date:** [September 28, 2021, 8:48am UTC](https://discuss.huggingface.co/t/batch-transform-pipeline/10275/3 "2021-09-28T08:48:46Z")

</div>

Hey Phil,

so I have written to predictors in one notebook to run the different predictions on the same text. So I have 2 HuggingFaceModels with 2 BatchTransformjobs in one notebook. The last issue I am facing here is that in each of those two batch jobs I have to define the output path:

```auto
batch_job = huggingface_model.transformer(
    instance_count=1,
    instance_type='ml.g4dn.xlarge',
    output_path=output_s3_path, 
    strategy='SingleRecord')

```

So I am getting two output file when I just want the predictions of both models in one. Is that somehow possible without a workaround?

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [September 28, 2021, 11:31am UTC](https://discuss.huggingface.co/t/batch-transform-pipeline/10275/4 "2021-09-28T11:31:49Z")

</div>

Hey,

batch transform offers something called `join_source`, where you can join input and output files.

> - **join\_source** ([_str_](https://docs.python.org/3/library/stdtypes.html#str)) – The source of data to be joined to the transform output. It can be set to ‘Input’ meaning the entire input record will be joined to the inference result. You can use OutputFilter to select the useful portion before uploading to S3. (default: None). Valid values: Input, None.

But I am not sure if this works with the `jsonl` and `json` structure we need for `HuggingFace`. But you can find more about it here: [Associate Prediction Results with Input Records - Amazon SageMaker](https://docs.aws.amazon.com/sagemaker/latest/dg/batch-transform-data-processing.html#batch-transform-data-processing-workflow)

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/4/42b1a67cee66b9ec1e574b2512ad7819e22dac89.png)

The easiest might be to write a custom python function, which post-process and merges your data files after the batch transform job is finished. If you use `SageMaker Pipelines` you could a lambda step for this.

---

<div class="post-metadata">

**Author:** ![marlon89](https://avatars.discourse-cdn.com/v4/letter/m/6de8d8/32.png) [@marlon89](https://discuss.huggingface.co/u/marlon89)\
**Post date:** [September 28, 2021, 11:41am UTC](https://discuss.huggingface.co/t/batch-transform-pipeline/10275/5 "2021-09-28T11:41:31Z")

</div>

Seems not to work with .json files:

```auto
ClientError: An error occurred (ValidationException) when calling the CreateTransformJob operation: ContentType ("application/json") in TransformInput has to be the same as Accept ("null") in TransformOutput when JoinSource is "Input".

```

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [September 28, 2021, 11:48am UTC](https://discuss.huggingface.co/t/batch-transform-pipeline/10275/6 "2021-09-28T11:48:02Z")

</div>

I think the error is saying something different.  
Could you try adding the `accept` parameter to

```auto
# create Transformer to run our batch job
batch_job = huggingface_model.transformer(
    instance_count=1,
    instance_type='ml.g4dn.xlarge',
    output_path=output_s3_path, # we are using the same s3 path to save the output with the input
    accept="application/json",
    strategy='SingleRecord')

```

---

<div class="post-metadata">

**Author:** ![marlon89](https://avatars.discourse-cdn.com/v4/letter/m/6de8d8/32.png) [@marlon89](https://discuss.huggingface.co/u/marlon89)\
**Post date:** [September 28, 2021, 11:50am UTC](https://discuss.huggingface.co/t/batch-transform-pipeline/10275/7 "2021-09-28T11:50:56Z")

</div>

I hope I didn’t mess anything around. This is my code now:

```auto
# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
   model_data=model_uri, # configuration for loading model from Hub
   role=role, # iam role with permissions to create an Endpoint
   transformers_version="4.6", # transformers version used
   pytorch_version="1.7", # pytorch version used
   py_version='py36', # python version used
)

# create Transformer to run our batch job
batch_job = huggingface_model.transformer(
    instance_count=1,
    instance_type='ml.g4dn.xlarge',
    output_path=output_s3_path, # we are using the same s3 path to save the output with the input
    strategy='SingleRecord',accept="application/json")

# starts batch transform job and uses s3 data as input
batch_job.transform(
    data=s3_file_uri,
    content_type='application/json',    
    split_type='Line',join_source='Input')

```

And this is my output:

```auto
ClientError: An error occurred (ValidationException) when calling the CreateTransformJob operation: SplitType ("Line") in TransformInput has to be the same as AssembleWith ("null") in TransformOutput when JoinSource is "Input".

```

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [September 28, 2021, 11:55am UTC](https://discuss.huggingface.co/t/batch-transform-pipeline/10275/8 "2021-09-28T11:55:59Z")

</div>

As the errors says you need to adjust `AssembleWith` to be the same.  
=\>

```python
# create Transformer to run our batch job
batch_job = huggingface_model.transformer(
   instance_count=1,
   instance_type='ml.g4dn.xlarge',
   output_path=output_s3_path, # we are using the same s3 path to save the output with the input
   accept="application/json",
   assemble_with="Line",
   strategy='SingleRecord')

```

You can find the documentation here [Transformer — sagemaker 2.198.0 documentation](https://sagemaker.readthedocs.io/en/stable/api/inference/transformer.html)

> - **assemble\_with** ([_str_](https://docs.python.org/3/library/stdtypes.html#str)) – How the output is assembled (default: None). Valid values: ‘Line’ or ‘None’.

---

<div class="post-metadata">

**Author:** ![marlon89](https://avatars.discourse-cdn.com/v4/letter/m/6de8d8/32.png) [@marlon89](https://discuss.huggingface.co/u/marlon89)\
**Post date:** [September 28, 2021, 12:05pm UTC](https://discuss.huggingface.co/t/batch-transform-pipeline/10275/9 "2021-09-28T12:05:02Z")

</div>

Thank you for your support @philschmid! I am new to Sagemaker and somehow its not that easy for me. One more question: The HuggingFace models just accept json files on Sagemaker and no csv, correct?

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [September 28, 2021, 12:39pm UTC](https://discuss.huggingface.co/t/batch-transform-pipeline/10275/10 "2021-09-28T12:39:02Z")

</div>

Yes and No, So it accepts `CSV` files, but only if you provide the Header and the complete file than otherwise, you can determine the fields. Also, not all tasks are supported.  
But for Batch Transform it is only `JSON` if you want to support another format you need to create a custom `inference.py` and overwrite the methods.