# Monitoring Metric "Transform Fn"

**URL:** <https://discuss.huggingface.co/t/monitoring-metric-transform-fn/9361>\
**Category:** Amazon SageMaker\
**Created:** [August 20, 2021, 1:12am UTC](https://discuss.huggingface.co/t/monitoring-metric-transform-fn/9361 "2021-08-20T01:12:25Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![dzorlu](https://avatars.discourse-cdn.com/v4/letter/d/f6c823/32.png) [@dzorlu](https://discuss.huggingface.co/u/dzorlu)\
**Post date:** [August 20, 2021, 1:12am UTC](https://discuss.huggingface.co/t/monitoring-metric-transform-fn/9361/1 "2021-08-20T01:12:25Z")

</div>

Hi- It’s me again.

I want to get a better sense of the system latency for the speech model I have deployed.  
AWS provides [Invocation Endpoint Metrics](https://docs.aws.amazon.com/sagemaker/latest/dg/monitoring-cloudwatch.html#cloudwatch-metrics-endpoint-invocation) like `ModelLatency` but that is more end-to-end. I am particularly interested in how much time is spent with the forward pass. I think I have two options:

1- HuggingFace model logs `preprocess`,`predict`, and `postprocess` times [here](https://github.com/aws/sagemaker-huggingface-inference-toolkit/blob/main/src/sagemaker_huggingface_inference_toolkit/handler_service.py#L196) but there is a bug as below where predict time is not captured correctly.

 ![Screen Shot 2021-08-19 at 5.23.12 PM](https://us1.discourse-cdn.com/hellohellohello/original/2X/6/668e6eb82265248127643c71f12ff3c07137bec5.png)

2- [Here](https://github.com/aws/sagemaker-huggingface-inference-toolkit/blob/main/src/sagemaker_huggingface_inference_toolkit/handler_service.py#L236), the metrics `Transform Fn` is added to the context using the API described [here](https://github.com/awslabs/multi-model-server/blob/master/docs/metrics.md).

My question is where does the `Transform Fn` go? I have hard time finding it on CloudWatch. I will also submit a PR to fix the predict time logging.

Thanks so much!  
Deniz

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [August 20, 2021, 1:47pm UTC](https://discuss.huggingface.co/t/monitoring-metric-transform-fn/9361/2 "2021-08-20T13:47:04Z")

</div>

Hey @dzorlu,

thank you for finding the bug, please let me know if you not manage to open PR, then I would take care of it.

Maybe @dan21c can tell more about the `transform_fn` and where is stored?

```python
context.metrics.add_time("Transform Fn", round((predict_end - predict_start) * 1000, 2))

```

---

<div class="post-metadata">

**Author:** ![dzorlu](https://avatars.discourse-cdn.com/v4/letter/d/f6c823/32.png) [@dzorlu](https://discuss.huggingface.co/u/dzorlu)\
**Post date:** [August 20, 2021, 8:02pm UTC](https://discuss.huggingface.co/t/monitoring-metric-transform-fn/9361/3 "2021-08-20T20:02:47Z")

</div>

Thanks @philschmid. [Here](https://github.com/aws/sagemaker-huggingface-inference-toolkit/pull/27) is the PR.