# Making local LLM + TTS work for a Persian voice assistant on a GTX 1650 Ti (4GB VRAM) — looking for advice **URL:** \ **Category:** Beginners\ **Created:** [September 17, 2026, 8:19pm UTC](https://discuss.huggingface.co/t/making-local-llm-tts-work-for-a-persian-voice-assistant-on-a-gtx-1650-ti-4gb-vram-looking-for-advice/180568 "2026-09-17T20:19:40Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![mahnazDEV](https://avatars.discourse-cdn.com/v4/letter/m/e36b37/32.png) [@mahnazDEV](https://discuss.huggingface.co/u/mahnazDEV)\ **Post date:** [September 17, 2026, 8:19pm UTC](https://discuss.huggingface.co/t/making-local-llm-tts-work-for-a-persian-voice-assistant-on-a-gtx-1650-ti-4gb-vram-looking-for-advice/180568/1 "2026-09-17T20:19:40Z")
Hi everyone, I’m building a real-time, two-way Persian voice conversation pipeline for a small robot project (Reachy Mini) with my professor. Current architecture: - **ASR** : Shenava-Rizeh (~32M params, Persian streaming ASR) via sherpa-onnx — fully local, CPU only, offline-capable - **VAD** : Silero VAD — fully local, CPU only - **LLM** : Qwen2.5-72B-Instruct via Hugging Face Inference API — cloud-only - **TTS** : edge-tts (Microsoft Edge’s cloud service, `fa-IR-FaridNeural` voice) — cloud-only **Why LLM/TTS ended up cloud-based:** I initially tried `qwen2.5:3b` locally via Ollama (CPU-only, since my venv’s torch build is CPU-only), but the output quality in Persian was poor — frequent code-switching into English/other languages, incoherent responses. Switching to Qwen2.5-72B via HF Inference API fixed this completely; responses are now coherent and consistently Persian. edge-tts also gives good voice quality. **The problem:** My internet connection is very slow and unstable (~0.7 Mbps down), so I’m getting periodic `ConnectTimeout` errors on the API calls (I’ve added retry logic, which helps but doesn’t eliminate the issue). For a robot that needs to work reliably, I’d like to move both LLM and TTS to fully local inference if possible. **My hardware:** - GPU: GTX 1650 Ti, 4GB VRAM - CPU-only PyTorch currently installed in the project venv (I haven’t set up CUDA there yet) - Very limited/unstable bandwidth for downloading large model weights **What I’m trying to figure out:** 1. For the LLM stage: with only 4GB VRAM, what’s the best quantized model (Qwen2.5 7B? something Persian-tuned like Dorna-Llama3-8B?) that would fit and give reasonable Persian instruction-following quality, ideally without needing to download tens of GB? 2. For TTS: are there any local/offline Persian TTS engines that get close to the naturalness of edge-tts? I found Piper has one Persian voice (“Amir”), but I’m not sure how it compares quality-wise to a commercial cloud TTS voice. 3. More generally — **has anyone gone through this same cloud-to-local migration for a low-resource language + constrained GPU setup, and what approach worked for you?** Any pointers to models, quantization strategies, or alternative local Persian TTS projects would be hugely appreciated. Happy to share more details (code, benchmarks) if useful. Thanks in advance! ---
**Author:** ![negdev](https://avatars.discourse-cdn.com/v4/letter/n/a6a055/32.png) [@negdev](https://discuss.huggingface.co/u/negdev)\ **Post date:** [September 18, 2026, 5:48am UTC](https://discuss.huggingface.co/t/making-local-llm-tts-work-for-a-persian-voice-assistant-on-a-gtx-1650-ti-4gb-vram-looking-for-advice/180568/2 "2026-09-18T05:48:31Z")
Really interesting setup — you’re basically doing everything right on the ASR/VAD side and only getting hurt by the cloud dependency on the LLM/TTS stages. Given your hardware (GTX 1650 Ti, 4GB VRAM, unstable bandwidth), I’d think in terms of a constrained but robust local pipeline rather than trying to replicate the exact behavior of Qwen2.5-72B + edge-tts. A few practical points: 1. LLM choice and quantization On 4GB VRAM, you’re realistically looking at: - 3B–4B class models in higher precision - 7B–8B class models only if heavily quantized (Q4\_K, Q5\_K, etc.) and with small context For Persian, I would look at: - Qwen2.5-7B or 3B in a good quantized format (GGUF) via llama.cpp / ollama - any Persian-tuned Llama3 or Qwen variant (like Dorna-Llama3-8B) but only if there is a solid quantized build and you accept slower generation The key is: don’t chase “perfect” quality like 72B; aim for “good enough + stable + local”. You can compensate a slightly weaker model with: - tighter prompts - shorter, more focused responses - a conversation manager that keeps context small and clean If you can set up CUDA in your venv, even a 3B–7B quantized model on GPU will feel much better than a big cloud model that times out. 1. VRAM strategy On a 4GB card, you’ll want: - low-rank quantization (Q4/Q5) for the LLM - small context window (e.g. 1–2k tokens) - aggressive offloading of non-critical parts to CPU if your framework supports it Think of it as a “tight real-time assistant” rather than a general-purpose chat model. 1. Local Persian TTS You already found Piper’s Persian voice (“Amir”). It won’t match a commercial cloud voice like edge-tts in naturalness, but it has three big advantages: - fully offline - predictable latency - no dependency on your unstable connection For a robot, consistency and reliability often matter more than absolute voice quality. You can also: - tune prosody via SSML-like controls if the engine supports it - pre-generate some common phrases to avoid real-time synthesis for everything If you need something closer to edge-tts quality, you might look for: - any open Persian TTS based on VITS / Glow-TTS / FastPitch - community projects trained on Persian audiobooks or news datasets But Piper is a very reasonable starting point for an offline pipeline. 1. Cloud-to-local migration mindset You’re already thinking in terms of a pipeline, which is good. I’d suggest: - lock ASR/VAD as they are (they’re already local and solid) - pick ONE local LLM that you can actually run reliably on your 4GB GPU - pick ONE local TTS (Piper/Amir to start) - then iterate on prompts, latency, and UX rather than constantly swapping models If you share some logs (latency, VRAM usage, token throughput) for your local tests, people here can give more targeted suggestions (e.g. “this quantization is too heavy”, “context is too large”, etc.). You’re on the right track — the fact that you already have a working cloud pipeline means you just need to “shrink” it into something that fits your hardware and bandwidth, not reinvent everything from scratch. By the way, I’m working on a local GPU utility specifically designed for low‑VRAM cards like the GTX 1650 Ti (4GB). I have the same GPU, so I know exactly the limitations you’re dealing with. My tool focuses on stabilizing VRAM usage for AMD/Intel/NVIDIA cards during AI workloads (DirectML, ROCm, SD, local LLMs). It prevents memory fragmentation, stuck contexts, and OOM issues — basically the problems that make small GPUs unreliable for real‑time assistants. I’m preparing a public demo release soon. If you want, I can ping you when it’s available so you can test it on your setup and give feedback. It might help you get more stable local inference without depending on cloud APIs. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [September 18, 2026, 8:43am UTC](https://discuss.huggingface.co/t/making-local-llm-tts-work-for-a-persian-voice-assistant-on-a-gtx-1650-ti-4gb-vram-looking-for-advice/180568/3 "2026-09-18T08:43:31Z")
For now, I tried a few models: * * * I think your current architecture is actually a good starting point. You already have the difficult low-resource-language pieces partly solved: Silero VAD and Shenava are local, and moving to Qwen2.5-72B + `edge-tts` showed that the remaining problem is mostly **deployment fit** rather than the overall design. For your three questions, my current answer would be: 1. **LLM:** I would not start by downloading a 7B/8B model. On this particular 4 GB machine, I would first verify that Ollama is actually using the GTX 1650 Ti, then try one modern ~4B-class Q4 model. My first test would currently be [Gemma 4 E2B IT QAT Q4\_0](https://huggingface.co/google/gemma-4-E2B-it-qat-q4_0-gguf). 2. **TTS:** the first local Persian TTS I would try is [Ava-82M](https://huggingface.co/xmanii/Ava-82M). For a much smaller fallback, [Mana-Persian-Piper](https://huggingface.co/MahtaFetrat/Mana-Persian-Piper) is interesting. 3. **Migration:** I would keep your working Silero + Shenava path, move the LLM local first, then move TTS local, and only after that optimize streaming/overlap. Changing everything simultaneously makes it much harder to tell what is actually failing. The most important first check may actually be this: ```powershell ollama ps ``` Your project’s CPU-only PyTorch build does **not** imply that Ollama was also CPU-only. Ollama has its own runtime, and the GTX 1650 Ti is explicitly listed in its current [NVIDIA GPU support table](https://github.com/ollama/ollama/blob/main/docs/gpu.mdx). `ollama ps` shows whether a loaded model is `100% GPU`, `100% CPU`, or split between CPU and GPU; see the [Ollama FAQ](https://github.com/ollama/ollama/blob/main/docs/faq.mdx). So I would treat these as two separate questions: ```plaintext PyTorch in the project venv != Ollama / llama.cpp GPU support ``` For the first experiment I would also keep the context small, around 2K. Larger context consumes more memory; Ollama documents that relationship explicitly in its [context-length notes](https://github.com/ollama/ollama/blob/main/docs/context-length.mdx). The goal of the first run is not to maximize context — it is to find out whether a useful Persian model can run with acceptable TTFT/tok/s on the actual laptop. A practical first path would be: ```plaintext Silero VAD (CPU) -> Shenava / sherpa-onnx (CPU) -> local ~4B Q4 LLM (GPU-first) -> first safe sentence/clause -> Ava TTS -> audio ``` And for the first half hour, roughly: ```plaintext 1. Load the local model you already have and run `ollama ps`. 2. Confirm whether the GTX 1650 Ti is actually being used. 3. Try exactly one modern ~4B Q4 model at about 2K context. 4. If Persian quality is acceptable, stop model-shopping and integrate it. 5. Try Ava locally; keep Mana-Piper as the small fallback. 6. Measure: VAD end -> ASR final -> first LLM token -> first TTS audio ``` I would avoid downloading several multi-GB models before the first one answers the main question, especially on a ~0.7 Mbps connection. > **Why I would start around 4B rather than 7B/8B** > > The 7B/8B direction is reasonable, but I do not think it is the cheapest _first_ experiment on this hardware. > > For example, a Q4\_K\_M build of Qwen2.5-7B is around **4.68 GB**. A Dorna-Llama3-8B Q4\_K\_M build is around **4.92 GB** ; the Dorna GGUF repo also has smaller Q3 variants around **3.66–4.02 GB**. > > Useful links: > > - [Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) > - [Qwen2.5-7B-Instruct GGUF example](https://huggingface.co/bartowski/Qwen2.5-7B-Instruct-GGUF) > - [Dorna-Llama3-8B-Instruct](https://huggingface.co/PartAI/Dorna-Llama3-8B-Instruct) > - [Dorna-Llama3-8B-Instruct GGUF](https://huggingface.co/QuantFactory/Dorna-Llama3-8B-Instruct-GGUF) > > Those file sizes do **not** mean “it cannot run on 4 GB”. GGUF file size is not a one-to-one VRAM number, and llama.cpp/Ollama can partially offload layers to the GPU and leave the rest in system RAM. > > But starting that close to or above the VRAM capacity leaves less headroom for the KV cache and runtime buffers, and it also makes each failed experiment more expensive over your connection. > > That is why I would use a smaller model as a discriminator first. > > The official Gemma 4 E2B Q4\_0 repo contains: > > ```plaintext > gemma-4-E2B_q4_0-it.gguf 3.35 GB > gemma-4-E2B-it-mmproj.gguf ~987 MB > > ``` > > For a text-only voice assistant you do not need the multimodal projector, so the first download can stay at the 3.35 GB text GGUF. > > The model page also exposes a direct llama.cpp path: > > ```powershell > winget install llama.cpp > > llama serve ^ > -hf google/gemma-4-E2B-it-qat-q4_0-gguf:Q4_0 ^ > -c 2048 > > ``` > > See [Gemma 4 E2B IT QAT Q4\_0](https://huggingface.co/google/gemma-4-E2B-it-qat-q4_0-gguf). > > I would not interpret “3.35 GB” as proof that it will fully reside in 4 GB VRAM. I would interpret it only as: **this is a much cheaper first experiment than starting with a ~4.7–4.9 GB 7B/8B quant**. > > If E2B runs well but its Persian behavior is still not good enough, then I would compare another 4B-class model before increasing the parameter count: > > ```plaintext > Gemma 4 E2B > | > +-- Persian good + speed good > | -> integrate it > | > +-- Persian weak, runtime fit good > | -> compare Gemma 3 4B / Qwen3 4B > | > +-- GPU placement / latency bad > | -> reduce context / inspect offload > | > +-- 4B class consistently not good enough > -> then try 7B/8B partial offload or keep a cloud fallback > > ``` > > Dorna is still interesting because it is Persian-oriented. I just would not let “Persian-tuned” override the deployment cost before checking whether a modern ~4B model is already sufficient. > **Small Persian LLM tests I ran** > > I did some small qualitative tests on Colab/T4. These are **not** GTX 1650 Ti benchmarks, and the prompt sets are too small to call any model a winner. I mainly wanted to see whether the ~4B range was worth investigating at all. > > An early fixed-prompt pass looked roughly like this: > > | Model | Observed loaded VRAM on T4 | What I saw | > | --- | --- | --- | > | IbnSina 1.5B Q4 | ~1.08 GB | Very small, but degenerated under the fixed sampler | > | Qwen2.5 3B Q4 | ~2.17 GB | Completed, but still had repetition/language/task issues | > | Qwen3 4B Q4 | ~2.92 GB | Usable control, still imperfect | > | Gemma 3 4B Q4 | ~3.01 GB | Stronger in that tiny set, still imperfect | > > Again, those T4 memory observations should **not** be copied over to a GTX 1650 Ti. The interesting result was simply that ~4B models were not obviously too weak to investigate. > > I also did a same-base comparison between vanilla Gemma 3 4B and a Persian SFT of Gemma 3 4B. Runtime cost was effectively tied, but the Persian SFT did not show a convincing assistant-quality advantage in that small A/B. > > So I would not assume: > > ```plaintext > Persian fine-tuned = automatically better Persian assistant > > ``` > > It may be better for a particular task or dataset, but I would test the exact assistant behavior you need. > > Later I tested the official Gemma 4 E2B/E4B Q4\_0 artifacts: > > | Model | GGUF | Observed T4 VRAM @ ~2K | Mean TTFT | Approx. generation speed | > | --- | --- | --- | --- | --- | > | Gemma 4 E2B QAT Q4\_0 | 3.35 GB | ~1.75 GB | ~0.076 s | ~92.7 tok/s | > | Gemma 4 E4B QAT Q4\_0 | 5.15 GB | ~3.19 GB | ~0.132 s | ~52.7 tok/s | > > These are T4/llama.cpp observations, not expected numbers for your laptop. > > E4B did not clearly dominate E2B in my ten-prompt set, so E2B became the more interesting target-machine test. > > One reason I would keep the evaluation application-specific is that even the apparently stronger models made ordinary mistakes. For example: > > ```plaintext > 08:20 + 1:45 = 10:05 > > ``` > > Vanilla Gemma 3 4B returned `10:05` in that test, while the Persian SFT returned `10:55`, and the tested Gemma 4 E2B/E4B runs returned `09:05`. > > That is not evidence that Gemma 3 is globally “better”; it is evidence that ten prompts are nowhere near enough to establish general quality. > > For your use case I would make a tiny smoke set around the errors that actually matter: > > ```plaintext > - normal Persian conversation > - colloquial Persian > - one noisy / ASR-like sentence > - one clarification-required request > - one date/time/number question > - one "answer only in Persian" instruction > > ``` > > I would watch for: > > ```plaintext > language switching > repetition / loops > instruction-following failures > date/time arithmetic failures > register instability > claims that an action was executed when no tool exists > > ``` > > And record only: > > ```plaintext > model / quant / context > CPU/GPU placement > VRAM > TTFT > tok/s > Persian stability > obvious task failures > > ``` > > That gives you much more useful information than downloading several models and asking each one a couple of generic questions. > **Local Persian TTS options I tested** > > For TTS, I ended up testing more candidates than I originally expected: > > ```plaintext > Ava-82M > Mana-Persian-Piper > Gooya + Negara G2P > Pocket-TTS Farsi v2 > MOSS-TTS-Nano-Persian > > ``` > > The two I would keep at the front of the queue are **Ava** and **Mana**. > > ### Ava-82M > > [Ava-82M](https://huggingface.co/xmanii/Ava-82M) is a compact Persian model (~82M class; current repository ~329 MB) and, importantly for Persian, it includes more than just the acoustic model. > > Its frontend currently includes: > > ```plaintext > number/date normalization > contextual Persian G2P > pronunciation overrides > Ezafe-related repair > long-text splitting > > ``` > > That is attractive because Persian TTS quality is not only an acoustic-model problem. If the frontend sends the wrong pronunciation, a good acoustic model will pronounce the wrong thing cleanly. > > The model card also makes an important limitation explicit: it is still a research release, and a formal MOS / Persian intelligibility benchmark has not yet been completed. > > So I would call Ava a **promising first local candidate** , not “as good as FaridNeural” or “the best Persian TTS”. > > ### Mana-Persian-Piper > > [MahtaFetrat/Mana-Persian-Piper](https://huggingface.co/MahtaFetrat/Mana-Persian-Piper) is much smaller. The main Piper ONNX file is only about **63.5 MB**. > > There is also an interesting companion project: > > [Piper + Lightweight Context-Aware Phonemizer](https://github.com/MahtaFetrat/Piper-with-LCA-Phonemizer) > > It adds context-sensitive Persian phonemization, especially around ambiguous phenomena such as Ezafe. > > For a first deployment I would still start with the simpler path: > > ```plaintext > Ava first > Mana raw Piper as very small fallback > > ``` > > and only add the LCA layer if real pronunciation errors make it worthwhile. > > ### Automatic comparison > > One complication on my side is that I do not speak Persian, so I did **not** pretend to do a native-listener ranking. > > Instead I used: > > - two different Persian ASR families as intelligibility proxies; > - disagreement between those ASRs as an uncertainty signal; > - UTMOS separately as an acoustic-quality proxy. > > I deliberately did not combine those into one arbitrary “overall TTS score”. > > On 5 shared utterances per model, the aggregate proxy results were: > > | Model | Mean ASR CER ↓ | ASR-family disagreement CER ↓ | Mean UTMOS ↑ | > | --- | --- | --- | --- | > | **Ava-82M** | **0.0355** | **0.0708** | **3.731** | > | Mana-Piper raw | 0.0568 | 0.1074 | 3.197 | > | Gooya + Negara | 0.1268 | 0.1886 | 3.058 | > | Pocket-TTS Farsi v2 | 0.2176 | 0.1736 | 3.465 | > | MOSS Nano Persian | 0.2726 | 0.2479 | 3.051 | > > I would interpret that as: > > ```plaintext > Ava = strongest automatic shortlist candidate > Mana = very small fallback with surprisingly strong intelligibility proxy > > ``` > > I would **not** interpret it as: > > ```plaintext > Ava sounds best to native Persian listeners > Ava == edge-tts quality > CER/UTMOS == human MOS > > ``` > > The ASR models have their own errors, UTMOS measures acoustic quality rather than Persian pronunciation correctness, and Ezafe is especially difficult to evaluate from standard Persian orthography because it is often spoken but not explicitly written. > > So this is useful for narrowing the search, not for replacing a native listening test. > > Given that you already like `fa-IR-FaridNeural`, I would keep your cloud TTS as a temporary quality/reference fallback while integrating Ava locally. You do not need to delete the working path on day one. > **Pipeline layout and latency notes** > > I would leave your VAD and ASR alone initially. > > [Shenava-Rizeh v1.0 for sherpa-onnx](https://huggingface.co/Reza2kn/Shenava-Rizeh-v1.0-sherpa-onnx) is already a compact ~32M Persian ASR deployment. Its model card explicitly supports offline sherpa-onnx inference and separates spoken-number output from the bundled Persian ITN/post-processing layer. > > That separation is useful: > > ```plaintext > audio > -> ASR text > -> Persian normalization / ITN > -> LLM > > ``` > > rather than baking every normalization decision into the recognizer itself. > > For a 4 GB GPU, my baseline resource split would be: > > ```plaintext > CPU: > Silero VAD > Shenava ASR > Persian normalization / ITN > orchestration > > GPU-first: > local LLM > > TTS: > start CPU-side if needed to protect LLM VRAM > try CUDA only if first-audio latency justifies it > > ``` > > The important point is that: > > ```plaintext > model choice != device placement > > ``` > > You should be able to move a component between CPU/GPU without redesigning the logical pipeline. > > ### Streaming > > For voice interaction, I would not wait for the entire LLM answer before starting TTS. > > Something like: > > ```plaintext > LLM stream > -> accumulate a semantically safe clause/sentence > -> synthesize it > -> continue generating the next chunk > > ``` > > But I would also avoid extremely tiny chunks, because TTS prosody gets worse if you synthesize fragments without enough linguistic context. > > ### Small integrated timing experiment > > I put the pipeline together once as: > > ```plaintext > Shenava-Rizeh > -> > Gemma 4 E2B Q4_0 > -> > first speakable response chunk > -> > Ava-82M > > ``` > > The cleaned Colab notebook is here: > > [Persian ASR → LLM → TTS timing notebook](https://huggingface.co/datasets/John6666/forum3/blob/main/persian_voice_pipeline/persian_voice_pipeline_timing_colab_t4_public.ipynb) > > In the latest successful T4 run, the warm means were roughly: > > | Stage | Mean | > | --- | --- | > | Shenava ASR decode | ~0.238 s | > | LLM TTFT | ~0.054 s | > | first token → first speakable chunk | ~0.129 s | > | Ava first-chunk synthesis | ~0.645 s | > | serial end-to-first-audio proxy | **~1.066 s** | > > Median serial proxy was about **0.992 s**. > > Gemma 4 E2B showed about **1.75 GB of observed T4 GPU memory** in that run, and Ava selected CUDA. > > Those numbers have a lot of boundaries: > > - Colab T4, not GTX 1650 Ti; > - synthetic fixed Persian input, not microphone speech; > - fixture-generation time excluded; > - warm models; > - simple serial addition, not a fully overlapped streaming service; > - no VAD end-of-speech wait; > - no microphone or playback-device buffering; > - no IPC/server overhead; > - first-audio timing depends on the chunking rule. > > So I would **not** predict “~1 second on your laptop” from this. > > The useful result was something else: in that pipeline, the LLM TTFT was tiny compared with first-chunk TTS. That is a good reminder to measure the complete voice timeline before spending all the optimization effort on LLM tok/s. > > On your machine, I would log: > > ```plaintext > t0 = VAD says utterance ended > t1 = ASR final transcript ready > t2 = first LLM token > t3 = first complete speakable chunk > t4 = first TTS audio ready > t5 = playback actually begins > > ``` > > Then you immediately know which component deserves optimization. > > ### One integration gotcha > > I also hit an unrelated but useful correctness issue: an LLM can say things like “I turned it off”, “I sent it”, or “I set the reminder” even when no real action tool exists. > > For a robot I would keep that boundary explicit: > > ```plaintext > speech / text > -> > LLM proposes intent > -> > orchestrator / real tool executes > -> > real tool result > -> > LLM may report success > > ``` > > In other words, do not let the language model’s sentence itself be evidence that a physical or software action happened. > > You do not need a large agent framework for this. A small explicit dispatcher is enough. > > If you later want an OpenAI-compatible local HTTP boundary around llama.cpp, its server is documented here: > > [llama.cpp server](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md) > **With a ~0.7 Mbps connection, I would treat downloads as part of the architecture** > > With normal broadband, downloading five candidate models and comparing them is annoying. > > At ~0.7 Mbps, it becomes part of the engineering problem. > > I would therefore use a deliberately boring strategy: > > ```plaintext > download one discriminating artifact > -> > measure it > -> > make the next decision > > ``` > > rather than: > > ```plaintext > download E2B > download E4B > download Qwen 7B > download Dorna 8B > download several quantizations > -> > compare later > > ``` > > A few things that help: > > - keep the HF/Ollama/llama.cpp caches; > - do not repeatedly remove working model files; > - use exact model filenames/revisions when possible; > - do not download Gemma 4’s ~987 MB multimodal projector for a text-only assistant; > - if an already-cached model can answer the deployment question, benchmark it before replacing it; > - prefer one decisive A/B over a large model collection. > > This is also why I would not make a 7B/8B download the first step. > > The larger model may eventually be the right answer, but finding out that a 3–4B model is sufficient saves download time, memory pressure, and integration complexity at the same time. > **What I would measure on the GTX 1650 Ti** > > If you want one compact target-machine test, I would collect only: > > ```plaintext > 1. runtime + exact model/quant > 2. context size > 3. `ollama ps` CPU/GPU placement > 4. loaded VRAM / observed split > 5. LLM TTFT > 6. generation tok/s > 7. 5-10 application-shaped Persian prompts > 8. VAD end -> ASR final -> first token -> first audio > > ``` > > Then I would use this decision tree: > > ```plaintext > Gemma 4 E2B @ ~2K context > | > +-- Persian quality OK, latency OK > | -> stop model shopping and integrate > | > +-- Persian quality weak, runtime fit OK > | -> compare Gemma 3 4B / Qwen3 4B > | > +-- model is CPU-heavy / split and latency is bad > | -> verify GPU path, reduce context, inspect offload > | > +-- 4B class consistently not good enough > | -> try 7B/8B with partial offload > | or retain cloud LLM as fallback > | > +-- local LLM is good, local TTS is weak > -> keep local LLM and temporarily retain edge-tts > > ``` > > That last branch is important: “fully local” does not have to be one atomic migration. > > You can move: > > ```plaintext > cloud LLM + cloud TTS > -> > local LLM + cloud TTS > -> > local LLM + local TTS > > ``` > > and keep the old path available until the replacement is actually good enough. So, if this were my machine, my default experiment would be: ```plaintext keep Silero + Shenava -> confirm GTX 1650 Ti use with `ollama ps` -> Gemma 4 E2B Q4_0, ~2K context -> 5-10 Persian assistant prompts -> Ava-82M -> measure end-to-first-audio ``` If E2B is not good enough in Persian, I would compare another ~4B model before paying the cost of 7B/8B. If Ava is not good enough acoustically, I would keep `edge-tts` temporarily rather than making TTS block the local-LLM migration. The main thing I would avoid is assuming that the failed `qwen2.5:3b` experiment already proved the GTX 1650 Ti cannot help. First check the actual runtime placement; that one observation changes the rest of the decision tree. ---
**Author:** ![mahnazDEV](https://avatars.discourse-cdn.com/v4/letter/m/e36b37/32.png) [@mahnazDEV](https://discuss.huggingface.co/u/mahnazDEV)\ **Post date:** [September 18, 2026, 4:57pm UTC](https://discuss.huggingface.co/t/making-local-llm-tts-work-for-a-persian-voice-assistant-on-a-gtx-1650-ti-4gb-vram-looking-for-advice/180568/4 "2026-09-18T16:57:10Z")
Thanks a lot, this is exactly the kind of practical breakdown I needed! A few follow-ups: 1. I’ll set up CUDA in my venv first (currently running CPU-only torch there) and test `qwen2.5:3b`/`qwen2.5:7b` (Q4\_K\_M GGUF) through Ollama on the GPU. Will report back with latency, VRAM usage, and token throughput once I have numbers. 2. For context size — since this is a real-time voice assistant with fairly short back-and-forth turns, would you say keeping context around 1-2k tokens is enough, or should I budget more if I want the model to remember earlier parts of the conversation (not just the last turn)? 3. I’ll give Piper/Amir a try for TTS and compare it side-by-side with edge-tts for naturalness. If anyone’s aware of a VITS/Glow-TTS/FastPitch-based Persian model trained on cleaner data (audiobooks/news), I’d love a pointer — happy to test and share results here. Appreciate the offer on the VRAM stabilization tool — I’ll keep an eye out for the release, but for now I want to get a baseline working with the standard Ollama/llama.cpp setup first, so I have something to compare against. Will post my benchmarks once I’ve run the GPU tests. ---
**Author:** ![mahnazDEV](https://avatars.discourse-cdn.com/v4/letter/m/e36b37/32.png) [@mahnazDEV](https://discuss.huggingface.co/u/mahnazDEV)\ **Post date:** [September 18, 2026, 4:59pm UTC](https://discuss.huggingface.co/t/making-local-llm-tts-work-for-a-persian-voice-assistant-on-a-gtx-1650-ti-4gb-vram-looking-for-advice/180568/5 "2026-09-18T16:59:53Z")
This is incredibly helpful, thank you — the point about Ollama’s runtime being separate from my venv’s PyTorch build is something I completely missed. I was assuming the failed 3B test meant the GPU couldn’t help, but I never actually checked whether Ollama was using it. I’ll follow your step-by-step plan exactly: 1. Running `ollama ps` first, right now, to see if the GTX 1650 Ti shows up as the processor. 2. Then testing Gemma 4 E2B IT QAT Q4\_0 at ~2K context before touching anything bigger. 3. If Persian quality is acceptable, I’ll stop shopping for models and integrate it rather than chasing marginal gains. 4. For TTS, I’ll try Ava-82M first and keep Mana-Persian-Piper as the lightweight fallback. 5. I like the “one variable at a time” approach — keeping Silero + Shenava untouched, moving LLM local first, then TTS, then optimizing latency/overlap last. That makes a lot more sense than what I was doing (changing multiple things at once and not being able to tell what fixed or broke what). I’ll measure the full chain like you suggested: VAD end → ASR final → first LLM token → first TTS audio, and post those numbers here once I have them. One question: for Gemma 4 E2B, do you know if it’s well-supported for Persian specifically, or is it more that the 4B class in general tends to handle Persian better than the older 3B options I tried? Also, if E2B doesn’t hold up in Persian, what would be your next ~4B pick to try before considering 7B/8B? Given my ~0.7 Mbps connection, I’ll only pull one model at a time and validate it before downloading anything else — appreciate the reminder on that, it would’ve cost me days otherwise. ---
**Author:** ![Jtxhob](https://avatars.discourse-cdn.com/v4/letter/j/c37758/32.png) [@Jtxhob](https://discuss.huggingface.co/u/Jtxhob)\ **Post date:** [September 21, 2026, 7:14am UTC](https://discuss.huggingface.co/t/making-local-llm-tts-work-for-a-persian-voice-assistant-on-a-gtx-1650-ti-4gb-vram-looking-for-advice/180568/6 "2026-09-21T07:14:12Z")
I find your entire project extremely fascinating. Admittedly, I am only an absolute beginner when it comes to AI and the different systems that work in the background to bring us the things that we typically see every day. But, I am interested in your project because of its significant usefulness. It’s bridging a crucial gap between Persian-speaking people and the the usefulness of AI as a general-purpose assistant or explainer. This connection is significantly boosted by the fact that the world is currently watching a country where the majority of the world’s Persian-speakers live. I can only imagine that these people have many questions, but not enough answers (or answers in Persian, at least). This tool will only add, not detract, from the usefulness of AI. I would love to discuss some more about how to build a similar Persian-speaking LLM that incorporates the nuances of the Persian culture. I don’t believe I have hardware that is capable of feasibly hosting something like this locally, but I’d certainly like to see if it’s possible as some sort of cloud or web-based service. Finally, please update us all on how the project is going. ---
**Author:** ![Bidram](https://avatars.discourse-cdn.com/v4/letter/b/71e660/32.png) [@Bidram](https://discuss.huggingface.co/u/Bidram)\ **Post date:** [September 21, 2026, 3:53pm UTC](https://discuss.huggingface.co/t/making-local-llm-tts-work-for-a-persian-voice-assistant-on-a-gtx-1650-ti-4gb-vram-looking-for-advice/180568/7 "2026-09-21T15:53:11Z")
Hello! I am not an expert, but would like to help. I had a somewhat related project before: [Fine-tuning an SLM for a low-resource language](https://discuss.huggingface.co/t/fine-tuning-an-slm-for-a-low-resource-language/176467). It didn’t really succeed, but I learned a few things that might be useful here. - **LLM:** I think the model you use matters a lot. If you mainly need simple responses and basic tasks, very small models such as [Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B) can actually be useful, although they can struggle with grammar and more complex Persian. You could also try larger multilingual models in the 2–4B range and compare them on your actual tasks rather than judging only by parameter count. - **TTS:** I found some small Persian TTS models that may be worth testing. [Piper Persian voices](https://huggingface.co/rhasspy/piper-voices/tree/main/fa/fa_IR) are quite lightweight and decent, while [Ava-82M](https://huggingface.co/xmanii/Ava-82M) is another option. In my experience, I also tried [OmniVoice](https://huggingface.co/k2-fsa/OmniVoice) (quantized version) myself and got good results, although it is considerably heavier and needs more compute / memory. One other approach you could consider is making the language part hybrid. use a very small model such as Qwen3.5-0.8B for easy requests and route more complicated requests to a larger local or cloud model. That could reduce resource usage while keeping better quality when it matters. I also wanted to point out that if the ASR (as an example) produces an incorrect or low-quality output for the next phase, it could lead to unexpected behavior downstream. Because of this, I recommend checking the inputs and outputs of every phase individually. This would make it easier to identify where problems originate and understand which parts of the architecture are actually weak. Although, looking at the replies, John and negdev have already covered a lot of this much better than I have, so consider my message more as additional help rather than a primary reference. ---
**Author:** ![negdev](https://avatars.discourse-cdn.com/v4/letter/n/a6a055/32.png) [@negdev](https://discuss.huggingface.co/u/negdev)\ **Post date:** [September 21, 2026, 5:18pm UTC](https://discuss.huggingface.co/t/making-local-llm-tts-work-for-a-persian-voice-assistant-on-a-gtx-1650-ti-4gb-vram-looking-for-advice/180568/8 "2026-09-21T17:18:26Z")
He’s saying that he finds your project fascinating and important, especially because it fills a real gap for Persian-speaking users. He shares his experience with fine-tuning small models for low-resource languages and gives practical advice: - Very small models (like Qwen3.5-0.8B) can work for simple tasks but struggle with proper Persian. - Models in the 2–4B range may perform better, but you should evaluate them on real tasks. - For Persian TTS, he suggests Piper voices, Ava-82M, or OmniVoice (heavier but good). He also recommends a hybrid architecture: use a tiny model for easy queries and route complex ones to a larger local or cloud model. Finally, he advises checking each stage of the pipeline separately (ASR → LLM → TTS), because errors in one stage propagate downstream. He notes that John and negdev already covered most of the topic in more detail, so his message is meant as additional help rather than the main reference.