# AutoModel from\_pretrained not releasing memory and causing a memory leak **URL:** \ **Category:** 🤗Transformers\ **Created:** [December 2, 2022, 7:39pm UTC](https://discuss.huggingface.co/t/automodel-from-pretrained-not-releasing-memory-and-causing-a-memory-leak/27151 "2022-12-02T19:39:29Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![farazk86](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/farazk86/32/922_2.png) [@farazk86](https://discuss.huggingface.co/u/farazk86)\ **Post date:** [December 2, 2022, 7:39pm UTC](https://discuss.huggingface.co/t/automodel-from-pretrained-not-releasing-memory-and-causing-a-memory-leak/27151/1 "2022-12-02T19:39:29Z")
Hi, I am using `transformers pipeline` for `token-classification`. ```auto tokenizer = AutoTokenizer.from_pretrained("./modelfiles") model = AutoModelForTokenClassification.from_pretrained("./modelfiles") nlp = pipeline("token-classification", model=model, tokenizer=tokenizer, aggregation_strategy="simple") ner_results = nlp(text) ``` The problem here is that on the first call to the function that implements the above lines the memory is released, then on the second call onwards the memory is not released as can be seen from the screenshot : ![Screenshot_2022-12-02_19-07-06](https://us1.discourse-cdn.com/hellohellohello/original/2X/1/10f578816077064b3d7af36aa341f0ffdab9e38f.png) The first peak is me calling the function and on return it frees up the memory but then on second call onwards it does not… this eventually leads to a crash. A memory profiler suggests the line `model = AutoModelForTokenClassification.from_pretrained("./modelfiles")` to be the problem. As can be seen from the screenshot below: ![Screenshot_2022-12-02_19-36-27](https://us1.discourse-cdn.com/hellohellohello/original/2X/0/060dde2479a3a24a144df20c07b56d25cc7fad31.png) I have tried setting `model = None` before the `return` statement and also called `gc.collect()` but the problem persists. Can someone please help me with this as this is always leading to a crash of the application. ☹ Thank you ---
**Author:** ![farazk86](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/farazk86/32/922_2.png) [@farazk86](https://discuss.huggingface.co/u/farazk86)\ **Post date:** [December 5, 2022, 2:37am UTC](https://discuss.huggingface.co/t/automodel-from-pretrained-not-releasing-memory-and-causing-a-memory-leak/27151/2 "2022-12-05T02:37:02Z")
Sorry for the bump, but would appreciate some help on this please as I just cant find the source of the problem. ☹ ---
**Author:** ![3553NC3](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/3553nc3/32/12914_2.png) [@3553NC3](https://discuss.huggingface.co/u/3553NC3)\ **Post date:** [December 6, 2022, 7:20am UTC](https://discuss.huggingface.co/t/automodel-from-pretrained-not-releasing-memory-and-causing-a-memory-leak/27151/3 "2022-12-06T07:20:40Z")
I faced the same issue and was able to avoid it when I defined a class that inhireted the `BertModel` class instead of immediately using `AutoModel` ---
**Author:** ![farazk86](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/farazk86/32/922_2.png) [@farazk86](https://discuss.huggingface.co/u/farazk86)\ **Post date:** [December 6, 2022, 2:49pm UTC](https://discuss.huggingface.co/t/automodel-from-pretrained-not-releasing-memory-and-causing-a-memory-leak/27151/4 "2022-12-06T14:49:27Z")
Thank you for your answer, but I managed to solve this problem by not loading the model at every call and have a global/app level reference to the model. More details and minimal solution code posted in my issue here: [Transformers model inference via pipeline not releasing memory after 2nd call. Leads to memory leak and crash in Flask web app · Issue #20594 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/issues/20594) ---
**Author:** ![ndvb](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ndvb/32/17239_2.png) [@ndvb](https://discuss.huggingface.co/u/ndvb)\ **Post date:** [May 23, 2023, 1:42pm UTC](https://discuss.huggingface.co/t/automodel-from-pretrained-not-releasing-memory-and-causing-a-memory-leak/27151/5 "2023-05-23T13:42:17Z")
I can’t free the GPU memory. Here’s a minimal example: ```auto import os os.environ["CUDA_VISIBLE_DEVICES"] = "6" import torch from transformers import LlamaForCausalLM model = LlamaForCausalLM.from_pretrained( pretrained_model_name_or_path='decapoda-research/llama-7b-hf', load_in_8bit=True, device_map={'': 0}, ) del model torch.cuda.empty_cache() print('breakpoint here - is memory freed?') ``` ---
**Author:** ![JB28666](https://avatars.discourse-cdn.com/v4/letter/j/35a633/32.png) [@JB28666](https://discuss.huggingface.co/u/JB28666)\ **Post date:** [June 25, 2023, 12:51am UTC](https://discuss.huggingface.co/t/automodel-from-pretrained-not-releasing-memory-and-causing-a-memory-leak/27151/6 "2023-06-25T00:51:04Z")
Hello, I am facing the same problem, have you solved it? ---
**Author:** ![ndvb](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ndvb/32/17239_2.png) [@ndvb](https://discuss.huggingface.co/u/ndvb)\ **Post date:** [June 26, 2023, 6:23am UTC](https://discuss.huggingface.co/t/automodel-from-pretrained-not-releasing-memory-and-causing-a-memory-leak/27151/7 "2023-06-26T06:23:03Z")
Yes. Do cpu garbage collection before the cuda free mem ---
**Author:** ![pribadihcr](https://avatars.discourse-cdn.com/v4/letter/p/bcef8e/32.png) [@pribadihcr](https://discuss.huggingface.co/u/pribadihcr)\ **Post date:** [February 7, 2024, 9:07am UTC](https://discuss.huggingface.co/t/automodel-from-pretrained-not-releasing-memory-and-causing-a-memory-leak/27151/8 "2024-02-07T09:07:35Z")
@ndvb, What are the script? like this ? gc.collect() torch.cuda.empty\_cache()