# Formatting Inference API call for LLama 2

**URL:** <https://discuss.huggingface.co/t/formatting-inference-api-call-for-llama-2/54901>\
**Category:** Inference Endpoints on the Hub\
**Created:** [September 13, 2023, 7:49pm UTC](https://discuss.huggingface.co/t/formatting-inference-api-call-for-llama-2/54901 "2023-09-13T19:49:20Z")\
**Posts on this page:** 1\
**Showing post:** 3

<div class="post-metadata">

**Author:** ![peteceptron](https://avatars.discourse-cdn.com/v4/letter/p/eada6e/32.png) [@peteceptron](https://discuss.huggingface.co/u/peteceptron)\
**Post date:** [September 28, 2023, 7:36pm UTC](https://discuss.huggingface.co/t/formatting-inference-api-call-for-llama-2/54901/3 "2023-09-28T19:36:34Z")

</div>

I solved it by inputting a single string using the official Llama 2 format (see [Llama 2 is here - get it on Hugging Face](https://huggingface.co/blog/llama2#how-to-prompt-llama-2)). I don’t know why the default Sagemaker Llama endpoint doesn’t work that way. But this works for me:

```python
import json
import requests
API_URL = "https://api-inference.huggingface.co/models/meta-llama/Llama-2-7b-chat-hf"
headers = {"Authorization": f"Bearer hf_XXXXXXXXXXXXXXXXXX",
        "Content-Type": "application/json",}

def query(payload):
    json_body = {
        "inputs": f"[INST] <<SYS>> Your job is to talk like a pirate. Every reponse must sound like a pirate. <<SYS>> {payload} [/INST] ",
                "parameters": {"max_new_tokens":256, "top_p":0.9, "temperature":0.7}
        }
    data = json.dumps(json_body)
    response = requests.request("POST", API_URL, headers=headers, data=data)
    try:
        return json.loads(response.content.decode("utf-8"))
    except:
        return response

data = query("Just say hi!")
print(data[0]['generated_text'].split('[/INST] ')[1])

```

---

_[View the full topic](https://discuss.huggingface.co/t/formatting-inference-api-call-for-llama-2/54901)._