# Isn't KV cache influenced by position encoding in inference?

**URL:** <https://discuss.huggingface.co/t/isnt-kv-cache-influenced-by-position-encoding-in-inference/63225>\
**Category:** 🤗Transformers\
**Created:** [November 23, 2023, 3:46am UTC](https://discuss.huggingface.co/t/isnt-kv-cache-influenced-by-position-encoding-in-inference/63225 "2023-11-23T03:46:54Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![anon58558744](https://avatars.discourse-cdn.com/v4/letter/a/d9b06d/32.png) [@anon58558744](https://discuss.huggingface.co/u/anon58558744)\
**Post date:** [November 23, 2023, 3:46am UTC](https://discuss.huggingface.co/t/isnt-kv-cache-influenced-by-position-encoding-in-inference/63225/1 "2023-11-23T03:46:54Z")

</div>

The KV cache can speed up inference because after the first iteration, the KV of the post sentance will not change, therefore we can store it.  
However, every iteration will append the input sentance with the new token, and in my opinion, that means the position encoding should be changed, and thus the KV value of the old part is changed too, which makes the KV cache become useless.  
So which part of my describition is wrong, and how exactly the KV cache work?

---

<div class="post-metadata">

**Author:** ![OrfeasFil](https://avatars.discourse-cdn.com/v4/letter/o/5f9b8f/32.png) [@OrfeasFil](https://discuss.huggingface.co/u/OrfeasFil)\
**Post date:** [May 15, 2024, 4:18pm UTC](https://discuss.huggingface.co/t/isnt-kv-cache-influenced-by-position-encoding-in-inference/63225/2 "2024-05-15T16:18:16Z")

</div>

Hello there,

Did you find any answer? I have the same question

---

<div class="post-metadata">

**Author:** ![RaushanTurganbay](https://avatars.discourse-cdn.com/v4/letter/r/848f3c/32.png) [@RaushanTurganbay](https://discuss.huggingface.co/u/RaushanTurganbay)\
**Post date:** [May 16, 2024, 8:42am UTC](https://discuss.huggingface.co/t/isnt-kv-cache-influenced-by-position-encoding-in-inference/63225/3 "2024-05-16T08:42:59Z")

</div>

Hey!

Yes, KV cache works as you described by storing previous keys and values. But I don’t understand why position embeddings for older cache should be changed for correct functioning.

Here’s how it works in `transformers`:

When we are generating autoregressively, we can add a cache of 5 tokens with positions as `[0,1,2,3,4]`. And every time we generate the next token, it gets its own position id, naturally following on from the previous ones. . In other words, the new token will have a position `[5]`, then `[6]` and so on.

---

<div class="post-metadata">

**Author:** ![OrfeasFil](https://avatars.discourse-cdn.com/v4/letter/o/5f9b8f/32.png) [@OrfeasFil](https://discuss.huggingface.co/u/OrfeasFil)\
**Post date:** [May 16, 2024, 1:57pm UTC](https://discuss.huggingface.co/t/isnt-kv-cache-influenced-by-position-encoding-in-inference/63225/4 "2024-05-16T13:57:42Z")

</div>

Wow, you are right, i don’t know how i missed that.

Thank you so much!!