# Problem in dynamicCache: index -1 is out of bounds for dimension 0 with size 0 in cache\_position\[-1\] **URL:** \ **Category:** 🤗Transformers\ **Created:** [February 21, 2025, 9:48pm UTC](https://discuss.huggingface.co/t/problem-in-dynamiccache-index-1-is-out-of-bounds-for-dimension-0-with-size-0-in-cache-position-1/142285 "2025-02-21T21:48:21Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![hmishfaq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/hmishfaq/32/40624_2.png) [@hmishfaq](https://discuss.huggingface.co/u/hmishfaq)\ **Post date:** [February 21, 2025, 9:48pm UTC](https://discuss.huggingface.co/t/problem-in-dynamiccache-index-1-is-out-of-bounds-for-dimension-0-with-size-0-in-cache-position-1/142285/1 "2025-02-21T21:48:21Z")
I am trying to use DynamicCache with an input prompt. After caching the prompt I want to generate one output at a time (and append it to previous prompt) and generate again. I am running the following code (example toy code to reproduce the error): ```auto import os import copy from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, DynamicCache max_length = 6 prompt_cache = DynamicCache() INITIAL_PROMPT = "I have a dream" inputs_initial_prompt = tokenizer(INITIAL_PROMPT, return_tensors="pt", padding=True).to("cuda") # This is the common prompt cached, we need to run forward without grad to be able to copy with torch.no_grad(): prompt_cache = model(**inputs_initial_prompt, past_key_values = prompt_cache).past_key_values.to("cuda") input_text = INITIAL_PROMPT # input_text = INITIAL_PROMPT + " " responses = [] for _ in range(max_length): new_inputs = tokenizer(input_text, return_tensors="pt", padding=True).to("cuda") past_key_values = copy.deepcopy(prompt_cache) outputs = model.generate(**new_inputs, past_key_values=past_key_values,max_new_tokens=20,pad_token_id=tokenizer.eos_token_id, do_sample=True, temperature=0.7) output_text = tokenizer.decode(outputs[0]) print(output_text[len(input_text):]) input_text = output_text print("#"*24) ``` The above code gives the following error: ```auto --------------------------------------------------------------------------- IndexError Traceback (most recent call last) Cell In[61], line 20 18 new_inputs = tokenizer(input_text, return_tensors="pt", padding=True).to("cuda") 19 past_key_values = copy.deepcopy(prompt_cache) ---> 20 outputs = model.generate(**new_inputs, past_key_values=past_key_values,max_new_tokens=20,pad_token_id=tokenizer.eos_token_id, do_sample=True, temperature=0.7) 21 output_text = tokenizer.decode(outputs[0]) 22 print(output_text[len(input_text):]) File /media/data1/haque/.conda/envs/llm/lib/python3.12/site-packages/torch/utils/_contextlib.py:116, in context_decorator..decorate_context(*args, **kwargs) 113 @functools.wraps(func) 114 def decorate_context(*args, **kwargs): 115 with ctx_factory(): --> 116 return func(*args, **kwargs) File /media/data1/haque/.conda/envs/llm/lib/python3.12/site-packages/transformers/generation/utils.py:2252, in GenerationMixin.generate(self, inputs, generation_config, logits_processor, stopping_criteria, prefix_allowed_tokens_fn, synced_gpus, assistant_model, streamer, negative_prompt_ids, negative_prompt_attention_mask, **kwargs) 2244 input_ids, model_kwargs = self._expand_inputs_for_generation( 2245 input_ids=input_ids, 2246 expand_size=generation_config.num_return_sequences, 2247 is_encoder_decoder=self.config.is_encoder_decoder, 2248 **model_kwargs, 2249 ) 2251 # 12. run sample (it degenerates to greedy search when `generation_config.do_sample=False`) -> 2252 result = self._sample( 2253 input_ids, 2254 logits_processor=prepared_logits_processor, 2255 stopping_criteria=prepared_stopping_criteria, 2256 generation_config=generation_config, 2257 synced_gpus=synced_gpus, 2258 streamer=streamer, 2259 **model_kwargs, 2260 ) 2262 elif generation_mode in (GenerationMode.BEAM_SAMPLE, GenerationMode.BEAM_SEARCH): 2263 # 11. prepare beam search scorer 2264 beam_scorer = BeamSearchScorer( 2265 batch_size=batch_size, 2266 num_beams=generation_config.num_beams, (...) 2271 max_length=generation_config.max_length, 2272 ) File /media/data1/haque/.conda/envs/llm/lib/python3.12/site-packages/transformers/generation/utils.py:3244, in GenerationMixin._sample(self, input_ids, logits_processor, stopping_criteria, generation_config, synced_gpus, streamer, **model_kwargs) 3239 is_prefill = True 3240 while self._has_unfinished_sequences( 3241 this_peer_finished, synced_gpus, device=input_ids.device, cur_len=cur_len, max_length=max_length 3242 ): 3243 # prepare model inputs -> 3244 model_inputs = self.prepare_inputs_for_generation(input_ids, **model_kwargs) 3246 # prepare variable output controls (note: some models won't accept all output controls) 3247 model_inputs.update({"output_attentions": output_attentions} if output_attentions else {}) File /media/data1/haque/.conda/envs/llm/lib/python3.12/site-packages/transformers/generation/utils.py:388, in GenerationMixin.prepare_inputs_for_generation(self, input_ids, past_key_values, attention_mask, inputs_embeds, cache_position, **kwargs) 384 if past_key_values is not None: 385 model_inputs["past_key_values"] = past_key_values 386 if ( 387 inputs_embeds is not None # Exception 1 --> 388 or (is_torchdynamo_compiling() or cache_position[-1] >= input_ids.shape[1]) # Exception 3 389 ): 390 input_ids = input_ids[:, -cache_position.shape[0] :] 391 elif input_ids.shape[1] != cache_position.shape[0]: # Default case (the "else", a no op, is Exception 2) IndexError: index -1 is out of bounds for dimension 0 with size 0 ``` But when I instead use ` input_text = INITIAL_PROMPT + " "` it works fine. What is the actually issue with cache\_position and how can I resolve it without adding extra string (here `" "`) in the input\_text? ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [February 22, 2025, 2:22pm UTC](https://discuss.huggingface.co/t/problem-in-dynamiccache-index-1-is-out-of-bounds-for-dimension-0-with-size-0-in-cache-position-1/142285/2 "2025-02-22T14:22:37Z")
It seems to also happen with StaticCache(). It’s similar to the following issue… > > > \### System Info > > \- \`transformers\` version: 4.46.3 > \- Platform: Linux-6.8.0-48-gen…eric-x86\_64-with-glibc2.17 > \- Python version: 3.8.20 > \- Huggingface\_hub version: 0.26.1 > \- Safetensors version: 0.4.5 > \- Accelerate version: 1.0.1 > \- Accelerate config: not found > \- PyTorch version (GPU?): 2.4.1+cu121 (True) > \- Tensorflow version (GPU?): not installed (NA) > \- Flax version (CPU?/GPU?/TPU?): not installed (NA) > \- Jax version: not installed > \- JaxLib version: not installed > \- Using distributed or parallel set-up in script?: no > \- Using GPU in script?: yes > \- GPU type: NVIDIA RTX A6000 > > \### Who can help? > > @gante > > \### Information > > \- \[\] The official example scripts > \- \[x\] My own modified scripts > > \### Tasks > > \- \[\] An officially supported task in the \`examples\` folder (such as GLUE/SQuAD, ...) > \- \[x\] My own task or dataset (give details below) > > \### Reproduction > > I am currently rewriting the generate\_progressively function for my custom model class. My goal is to enable the model to generate results progressively by concatenating the initial input\_ids with each element of the compress\_outputs sequence in turn. Specifically: > 1. In the first iteration, the model generates results by concatenating input\_ids with the first element of compress\_outputs. > 2. In the second iteration, it concatenates input\_ids with the first and second elements of compress\_outputs (the first two elements) to generate results. > 3. This process continues until the last element of the compress\_outputs sequence is included. > > To improve efficiency, I want to leverage caching, as the majority of the concatenated input in each iteration has already been used to compute past\_key\_values. Below is the code snippet for the function I implemented. In this context, self.model refers to mistral-7b-chat-v0.2. > \`\`\` > @torch.no\_grad() > def generate\_progressively( > self, > input\_ids, > attention\_mask, > compress\_outputs, > \*\*kwargs, > ): > results = \[\] > compress\_output\_count = compress\_outputs.size(1) > batch\_size = input\_ids.size(0) > > inputs\_embs = self.base.model.embed\_tokens(input\_ids) > prompt\_cache = DynamicCache() > outputs = self.model( > input\_ids=input\_ids, > attention\_mask=attention\_mask, > use\_cache=True, > past\_key\_values=prompt\_cache, > ) > prompt\_cache = outputs.past\_key\_values > > for compress\_ind in range(compress\_output\_count): > current\_compress\_outputs = compress\_outputs\[:, compress\_ind: compress\_ind+1, :\].type\_as(input\_ids) > outputs = self.model( > input\_ids=None, > inputs\_embeds=current\_compress\_outputs, > use\_cache=True, > past\_key\_values=prompt\_cache, > ) > prompt\_cache = outputs.past\_key\_values > > inputs\_embs = torch.cat(\[inputs\_embs, current\_compress\_outputs\], dim=1) > attention\_mask = torch.cat(\[attention\_mask, torch.ones(batch\_size, 1, device=input\_ids.device)\], dim=1) > > generated\_outputs = self.base.generate( > inputs\_embeds=inputs\_embs, > attention\_mask=attention\_mask, > use\_cache=True, > past\_key\_values=prompt\_cache, > return\_dict\_in\_generate=True, > \*\*kwargs, > ) > results.append(generated\_outputs.sequences) > return results > \`\`\` > > When I execute this code, the program throws an error during execution. The error occurs at line 393 in transformers/generation/utils.py, specifically in the prepare\_inputs\_for\_generation function. > The problematic line of code is: > \`\`\` > if inputs\_embeds is not None and cache\_position\[0\] == 0: > \`\`\` > The error message is: IndexError: index 0 is out of bounds for dimension 0 with size 0. > > I track the excution of the code and here’s a detailed breakdown of the issue: > The error occurs in transformers/generation/utils.py. Initially, the program enters the self.\_sample function and then proceeds to the self.\_get\_initial\_cache\_position function. > Within this function, the following line: > \`\`\` > if not is\_torchdynamo\_compiling(): > cache\_position = cache\_position\[past\_length:\] > \`\`\` > causes the correct cache\_position slice to become empty, resulting in an IndexError in subsequent steps. > > Even if I manage to fix the issue with cache\_position, another problem arises later in the self.prepare\_inputs\_for\_generation function. > The relevant code is as follows: > \`\`\` > if not self.config.is\_encoder\_decoder: > if inputs\_embeds is not None and cache\_position\[0\] == 0: > model\_inputs\[input\_ids\_key\] = None > model\_inputs\["inputs\_embeds"\] = inputs\_embeds > else: > model\_inputs\[input\_ids\_key\] = input\_ids.clone(memory\_format=torch.contiguous\_format) > model\_inputs\["inputs\_embeds"\] = None > \`\`\` > In my case, I provide only inputs\_embeds and past\_key\_values, and since cache\_position\[0\] is not 0, the code attempts to set model\_inputs\[input\_ids\_key\] using input\_ids. However, since input\_ids is None, this results in further issues. > > Under the current implementation of the generate function in transformers, is it possible to use only inputs\_embeds and past\_key\_values for generation? How can I modify my implementation to achieve progressive generation with caching as intended? Are there specific guidelines for correctly managing cache\_position and ensuring compatibility with inputs\_embeds? > > \### Expected behavior > > My primary objective is to progressively generate outputs by leveraging caching (past\_key\_values) to improve efficiency.