# KV Cache size shrinks during Inference instead of growing. Can someone explain why?

**URL:** <https://discuss.huggingface.co/t/kv-cache-size-shrinks-during-inference-instead-of-growing-can-someone-explain-why/73104>\
**Category:** 🤗Transformers\
**Created:** [February 14, 2024, 5:58am UTC](https://discuss.huggingface.co/t/kv-cache-size-shrinks-during-inference-instead-of-growing-can-someone-explain-why/73104 "2024-02-14T05:58:06Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![gopikrishnajha](https://avatars.discourse-cdn.com/v4/letter/g/90db22/32.png) [@gopikrishnajha](https://discuss.huggingface.co/u/gopikrishnajha)\
**Post date:** [February 14, 2024, 5:58am UTC](https://discuss.huggingface.co/t/kv-cache-size-shrinks-during-inference-instead-of-growing-can-someone-explain-why/73104/1 "2024-02-14T05:58:06Z")

</div>

This is w.r.t inference using models like mistral7b or llama.

In my understanding, KV cache size should grow as we process more tokens, however I see in the code that it shrinks as more tokens are processed. For example, in transformers/src/transformers/models/mistral/modeling\_mistral.py, see the following code.

`key_states, value_states = past_key_value.update(key_states, value_states, self.layer_idx, cache_kwargs)`

The cache `past_key_value` instead of growing, shrinks in size. Initial size is `[batch_size, num_heads, seq_len, head_dim]` and with increasing iterations, while `batch_size, num_heads and head_dim` remain the same, `seq_len` decreases. This results in a shrinking cache.

Can anyone explain why is the cache shrinking instead of growing in size?