# Create an Assistant to be used via Python scripts **URL:** \ **Category:** Beginners\ **Created:** [September 20, 2024, 3:35pm UTC](https://discuss.huggingface.co/t/create-an-assistant-to-be-used-via-python-scripts/107505 "2024-09-20T15:35:12Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![alp82](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/alp82/32/31845_2.png) [@alp82](https://discuss.huggingface.co/u/alp82)\ **Post date:** [September 20, 2024, 3:35pm UTC](https://discuss.huggingface.co/t/create-an-assistant-to-be-used-via-python-scripts/107505/1 "2024-09-20T15:35:12Z")
Hey there, I’m new to the community. So far, I played around with [HuggingChat](https://huggingface.co/chat/) and created an Assistant for a specific use case. I want to be able to create such an Assistant to be available via API calls. So basically: - Choose a model (my preference atm is “CohereForAI/c4ai-command-r-plus-08-2024”) - Give it initial instructions - Publish it And then, probably via the “transformers” library I want to be able to write messages to that model. I’m happy to pay for the Pro subscription if that’s necessary. Can you please give me some pointers on how to achieve this? ---
**Author:** ![alp82](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/alp82/32/31845_2.png) [@alp82](https://discuss.huggingface.co/u/alp82)\ **Post date:** [September 21, 2024, 5:53am UTC](https://discuss.huggingface.co/t/create-an-assistant-to-be-used-via-python-scripts/107505/2 "2024-09-21T05:53:54Z")
Never mind, I just found out about Spaces which seems the way to go. Experimenting! ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [September 21, 2024, 7:54am UTC](https://discuss.huggingface.co/t/create-an-assistant-to-be-used-via-python-scripts/107505/3 "2024-09-21T07:54:46Z")
Although it is relatively unprofessional (I’m not an expert on anything to begin with), I think it is this function that is closest to your use case. Any well-known model on HF can be called with a reasonable probability. I heard that even with a free account you can do 300 requests per hour. There are some large models you can only call with a Pro subscription. Without an account, I heard it’s 1 request per hour… > **[Inference](https://huggingface.co/docs/huggingface_hub/package_reference/inference_client)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. > **[Inference](https://huggingface.co/docs/huggingface_hub/package_reference/inference_client#huggingface_hub.InferenceClient.chat_completion)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. > **[Run Inference on servers](https://huggingface.co/docs/huggingface_hub/guides/inference#openai-compatibility)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. This is an example of actual use and a sample of a usable LLM. The models used in this space are available candidates. > **[HF's Missing Inference Widget - a Hugging Face Space by featherless-ai](https://huggingface.co/spaces/featherless-ai/try-this-model)** > > Discover amazing ML apps made by the community It is also a good idea to get code from an app with a similar use case that someone else has created, rather than building it from scratch. > **[Spaces for LLM / VLM / NLP - a John6666 Collection](https://huggingface.co/collections/John6666/spaces-for-llm-vlm-nlp-66999a0f61278f96d85256f4)** > > Unlock the magic of AI with handpicked models, awesome datasets, papers, and mind-blowing Spaces from John6666 ---
**Author:** ![alp82](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/alp82/32/31845_2.png) [@alp82](https://discuss.huggingface.co/u/alp82)\ **Post date:** [September 21, 2024, 10:03am UTC](https://discuss.huggingface.co/t/create-an-assistant-to-be-used-via-python-scripts/107505/4 "2024-09-21T10:03:56Z")
Thanks a lot for your detailed answer! I tried to create a space and added this to `app.py`: ```auto import gradio as gr from transformers import AutoModelForCausalLM, AutoTokenizer model_name = "CohereForAI/c4ai-command-r-plus-08-2024" tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForCausalLM.from_pretrained(model_name) def generate_response(prompt): inputs = tokenizer(prompt, return_tensors="pt") outputs = model.generate(**inputs, max_new_tokens=100) return tokenizer.decode(outputs[0], skip_special_tokens=True) interface = gr.Interface(fn=generate_response, inputs="text", outputs="text") interface.launch() ``` After that, the space rebuilds the container but I get this error: ```auto OSError: You are trying to access a gated repo. Make sure to have access to it at https://huggingface.co/CohereForAI/c4ai-command-r-plus-08-2024. 401 Client Error. (Request ID: Root=1-66ee97d2-2830f72d701dd7eb35d0359f;5d578753-6b5d-40e5-b832-6dc565ed94e8) Cannot access gated repo for url https://huggingface.co/CohereForAI/c4ai-command-r-plus-08-2024/resolve/main/config.json. Access to model CohereForAI/c4ai-command-r-plus-08-2024 is restricted. You must have access to it and be authenticated to access it. Please log in. ``` I know how to create a token, but don’t know how I can assign one to this space. Can you help me with that? ---
**Author:** ![alp82](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/alp82/32/31845_2.png) [@alp82](https://discuss.huggingface.co/u/alp82)\ **Post date:** [September 21, 2024, 10:10am UTC](https://discuss.huggingface.co/t/create-an-assistant-to-be-used-via-python-scripts/107505/5 "2024-09-21T10:10:29Z")
I found the answer myself: Gated models need additional permissions to be used. I just went to [CohereForAI/c4ai-command-r-plus-08-2024 · Hugging Face](https://huggingface.co/CohereForAI/c4ai-command-r-plus-08-2024), filled out the form and got access. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [September 21, 2024, 10:23am UTC](https://discuss.huggingface.co/t/create-an-assistant-to-be-used-via-python-scripts/107505/6 "2024-09-21T10:23:49Z")
> I know how to create a token, but don’t know how I can assign one to this space. Can you help me with that? You can use it directly in your code, but it is dangerous because the tokens are fully visible, so Space has a **Secret** environment variable setting facility. After setting the token in the **Secret** , it is useful to retrieve it as follows. Be aware that normal environment variables are still fully visible. ```py import os hf_token = os.environ.get("HF_TOKEN") ``` ---
**Author:** ![alp82](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/alp82/32/31845_2.png) [@alp82](https://discuss.huggingface.co/u/alp82)\ **Post date:** [September 21, 2024, 10:43am UTC](https://discuss.huggingface.co/t/create-an-assistant-to-be-used-via-python-scripts/107505/7 "2024-09-21T10:43:53Z")
Yes, I did that and it retrieves the secret properly from the system’s env vars. Now my problem is which model to pick that fits into ZeroGPU. `CohereForAI/c4ai-command-r-plus-08-2024` is too big. I’d like to find something with similar capabilities because my initial instructions are quite complex. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [September 21, 2024, 10:52am UTC](https://discuss.huggingface.co/t/create-an-assistant-to-be-used-via-python-scripts/107505/8 "2024-09-21T10:52:17Z")
Do you load on the VRAM of your own space? If so, you can use a BNB with 4-bit quantisation and you only need a quarter of the VRAM. You can use a larger model for that. > **[bitsandbytes](https://huggingface.co/docs/transformers/main/quantization/bitsandbytes)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. My actual code ```py from transformers import BitsAndBytesConfig nf4_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16) text_model = AutoModelForCausalLM.from_pretrained(model_name, quantization_config=nf4_config, device_map=device, torch_dtype=torch.bfloat16).eval() ``` ---
**Author:** ![alp82](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/alp82/32/31845_2.png) [@alp82](https://discuss.huggingface.co/u/alp82)\ **Post date:** [September 21, 2024, 11:02am UTC](https://discuss.huggingface.co/t/create-an-assistant-to-be-used-via-python-scripts/107505/9 "2024-09-21T11:02:05Z")
> [@John6666](#): > > Do you load on the VRAM of your own space? I don’t fully understand what you mean by that. The space is running on Hugging Face infrastructure. The code suggests that it’s using the space’s VRAM, but I’m not sure: ```auto tokenizer = AutoTokenizer.from_pretrained(model_id, token=hf_token) model = AutoModelForCausalLM.from_pretrained(model_id, token=hf_token) def generate_response(prompt): inputs = tokenizer(prompt, return_tensors="pt") outputs = model.generate(**inputs, max_new_tokens=100) return tokenizer.decode(outputs[0], skip_special_tokens=True) ``` I will try the 4-bit quantisation, thanks for the code example! ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [September 21, 2024, 11:14am UTC](https://discuss.huggingface.co/t/create-an-assistant-to-be-used-via-python-scripts/107505/10 "2024-09-21T11:14:00Z")
Models that are too big might be easier this way: if you only have HF, a token and an internet connection, it will even work on a smartphone. But you can do a lot more if you load it on hand, like your code. If you do it via an API, the functionality is inevitably limited. ```py from huggingface_hub import InferenceClient client = InferenceClient("meta-llama/Meta-Llama-3.1-70B-Instruct", token=hf_token) system_message = "You are a helpful assistant. Try your best to give the best response possible to the user." user_message = f"{base_prompt}\nDescription: {input_text}" messages = [ {"role": "system", "content": system_message}, {"role": "user", "content": user_message} ] response = client.chat_completion( model="meta-llama/Meta-Llama-3.1-70B-Instruct", max_tokens=1024, temperature=0.7, top_p=0.95, messages=messages, ) ``` ---
**Author:** ![alp82](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/alp82/32/31845_2.png) [@alp82](https://discuss.huggingface.co/u/alp82)\ **Post date:** [September 21, 2024, 11:35am UTC](https://discuss.huggingface.co/t/create-an-assistant-to-be-used-via-python-scripts/107505/11 "2024-09-21T11:35:45Z")
I was able to find a great space and duplicated it. The copy works but I reached my usage limit pretty quickly while using the text box in the space UI: ```auto You have exceeded your GPU quota (60s requested vs. 40s left). Create a free account to get more usage quota. ``` I am already logged in and just subscribed to Pro. Is this normal? I will try your suggestion with the InferenceClient now. ---
**Author:** ![alp82](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/alp82/32/31845_2.png) [@alp82](https://discuss.huggingface.co/u/alp82)\ **Post date:** [September 21, 2024, 11:56am UTC](https://discuss.huggingface.co/t/create-an-assistant-to-be-used-via-python-scripts/107505/12 "2024-09-21T11:56:11Z")
The InferenceClient is working perfectly and is exactly what I needed. Do you anything about rate limits for Pro accounts? ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [September 21, 2024, 12:03pm UTC](https://discuss.huggingface.co/t/create-an-assistant-to-be-used-via-python-scripts/107505/13 "2024-09-21T12:03:20Z")
> I am already logged in and just subscribed to Pro. Is this normal? I was also mistaken until recently, but it seems that **“sign in”** in this case refers to the following functions. It seems to be different from a login or a token. Well, if you change one line in app.py and one line in README.md, you can sign in anyway. Can’t help if it’s someone else’s space… > **[Sign in with Hugging Face](https://huggingface.co/docs/hub/oauth)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. I heard that Pro accounts have 5x Quota, but I don’t know if this is true. ---
**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\ **Post date:** [September 22, 2024, 12:03am UTC](https://discuss.huggingface.co/t/create-an-assistant-to-be-used-via-python-scripts/107505/14 "2024-09-22T00:03:57Z")
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.