# meta-llama/Llama-2-7b-chat-hf weird responses, compared to the ones returned by the HF API

**URL:** <https://discuss.huggingface.co/t/meta-llama-llama-2-7b-chat-hf-weird-responses-compared-to-the-ones-returned-by-the-hf-api/138456>\
**Category:** 🤗Transformers\
**Created:** [February 1, 2025, 2:31pm UTC](https://discuss.huggingface.co/t/meta-llama-llama-2-7b-chat-hf-weird-responses-compared-to-the-ones-returned-by-the-hf-api/138456 "2025-02-01T14:31:15Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![bstoilov](https://avatars.discourse-cdn.com/v4/letter/b/a183cd/32.png) [@bstoilov](https://discuss.huggingface.co/u/bstoilov)\
**Post date:** [February 1, 2025, 2:31pm UTC](https://discuss.huggingface.co/t/meta-llama-llama-2-7b-chat-hf-weird-responses-compared-to-the-ones-returned-by-the-hf-api/138456/1 "2025-02-01T14:31:15Z")

</div>

Hello, I can’t say I know much about how the models work internally, but I am trying to get the same behaviour I see [on the Llama-2-7b-chat-hf example section to the right](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf)

I am running the model locally and give both the web API version and mine the following prompt

```auto
<<SYS>>
<large chukn of text with facts about apples>
<</SYS>>

[INST]
User: What are the nutrition facts about Apples
[/INST]

```

The one from the API looks very nice and is exactly what I want, but the local one responds with weird source links and overall is not good.

This is my code

```auto
# Load the tokenizer and model
model_name = "meta-llama/Llama-2-7b-chat-hf"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

prompt = get_file_contents("prompt.txt") # same prompt
# Tokenize the input conversation
inputs = tokenizer(prompt, return_tensors="pt")

# Generate response from the model
outputs = model.generate(
    **inputs,
    max_new_tokens=200,
    temperature=0.7,
    repetition_penalty=1.5,
    top_p=0.9,
    top_k=50,
)

# Decode the model's output and return the response
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

```

What am I doing wrong, the resources on my PC don’t allow it to reach 32 GB of ram, it caps at 25-27GB could that be the issue, or that affects only speed

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [February 2, 2025, 6:07am UTC](https://discuss.huggingface.co/t/meta-llama-llama-2-7b-chat-hf-weird-responses-compared-to-the-ones-returned-by-the-hf-api/138456/2 "2025-02-02T06:07:35Z")

</div>

> or that affects only speed

Basically, this should be the case, and there are few cases where you get half-baked results due to insufficient hardware performance. It’s either it works or it doesn’t, and it’s either fast or slow.

I found the official HF implementation for Llama2. It may be that **tokenizer.use\_default\_system\_prompt = False** is meaningful.

Since Llama2 has been around for a long time, it has been affected by various HF specification changes, so there is likely to be some confusion about how to use it.

> **[app.py · huggingface-projects/llama-2-7b-chat at main](https://huggingface.co/spaces/huggingface-projects/llama-2-7b-chat/blob/main/app.py)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

> [@Issue with LlaMA-2 Chat Template (and out of date documentation)](https://discuss.huggingface.co/t/issue-with-llama-2-chat-template-and-out-of-date-documentation/61645):
>
> Hi @Rocketknight1 is see that you added the chat\_template data for the LlaMA-2 models. There appears to be a bug in that logic where if you only pass in a system prompt, formatting the template returns an empty string/list. For example, the below code results in printing an empty string: chat = [{"role": "system", "content": "You are a helpful and honest assistant."},] print(tokenizer.apply\_chat\_template(chat, tokenize=False)) However, if you edit it to have an empty user object, then it w…