# How can i build a High Quality dataset? **URL:** \ **Category:** Beginners\ **Created:** [June 5, 2026, 1:06pm UTC](https://discuss.huggingface.co/t/how-can-i-build-a-high-quality-dataset/176571 "2026-06-05T13:06:48Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Bidram](https://avatars.discourse-cdn.com/v4/letter/b/71e660/32.png) [@Bidram](https://discuss.huggingface.co/u/Bidram)\ **Post date:** [June 5, 2026, 1:06pm UTC](https://discuss.huggingface.co/t/how-can-i-build-a-high-quality-dataset/176571/1 "2026-06-05T13:06:48Z")
I want to build a high-quality Persian assistance dataset for an SLM. I have already used models like ChatGPT to generate a small Persian assistance dataset, but the overall quality was not good enough. Since the model I want to fine-tune is small, I need a larger dataset with much less noise and better overall quality. I want to ask how I can build the fine-tuning dataset I need in a high-quality way. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [June 7, 2026, 7:31am UTC](https://discuss.huggingface.co/t/how-can-i-build-a-high-quality-dataset/176571/2 "2026-06-07T07:31:32Z")
Dataset quality is hard to define because the evaluation criteria change depending on the goal, but I tried to organize the parts that can be organized: * * * ## Short answer Yes, there are best practices, but I would not start from “how do I create a large high-quality dataset?” in the abstract. For a **Persian SLM assistant** , I would start from a more evaluation-driven question: > What exact Persian capability is missing from the base model, and how will I measure whether my dataset fixes that gap? A “high-quality dataset” for Persian assistant fine-tuning may mean different things depending on whether you want: - better Persian fluency - better instruction following - better multi-turn assistant behavior - better Iranian/Persian cultural fit - better factuality or domain knowledge - safer refusals and culturally appropriate alignment - better RAG behavior - better formatting under a specific chat template - better behavior from a very small model with limited context and imperfect Persian tokenization So I would use an **evaluation-first roadmap** : 1. Check current Persian leaderboards and benchmarks. 2. Build a small private eval set for your exact use case. 3. Evaluate the base model and Persian-specialized baselines. 4. Identify the actual gap. 5. Only then build or curate training data targeted at that gap. The important point is: > Do not build “more Persian chat data” blindly. Build the missing data that your evaluation shows you need. * * * ## 1. Start with Persian evaluation, not training data Before generating or collecting examples, I would first inspect current Persian evaluation resources. These are useful because they show what Persian LLM quality already gets decomposed into: knowledge, reasoning, instruction following, NLU, NLG, multi-turn dialogue, safety, culture, and retrieval. | Goal | Useful starting point | Why it matters | | --- | --- | --- | | General Persian LLM comparison | [Open Persian LLM Leaderboard](https://huggingface.co/spaces/PartAI/open-persian-llm-leaderboard) | A practical first stop for comparing current Persian-capable models. Use it for baselines, not as a final product metric. | | Multi-dimensional Persian evaluation | [MIZAN: A Persian LLM Leaderboard](https://huggingface.co/spaces/MCINext/mizan-llm-leaderboard) | Useful because it separates Persian evaluation into reasoning, instruction following, knowledge, NLU, NLG, and multi-turn dialogue. | | Persian knowledge / reasoning | [Khayyam Challenge / PersianMMLU](https://arxiv.org/abs/2404.06644), [PerMMLU](https://huggingface.co/datasets/MCINext/permmlu) | Good anchors for general and local knowledge, but not sufficient for assistant behavior. | | Instruction following | [Persian IFEval](https://huggingface.co/datasets/MCINext/persian-ifeval) | Helpful for checking whether the model follows Persian instructions and constraints. | | Multi-turn assistant behavior | [Persian MT-Bench](https://huggingface.co/datasets/MCINext/persian-mt-bench) | Useful for dialogue, writing, multi-turn behavior, and retrieval-like chat cases. | | Persian safety / alignment | [ELAB: Extensive LLM Alignment Benchmark in Persian Language](https://arxiv.org/abs/2504.12553) | Important for safety, fairness, and social norms in Persian linguistic/cultural contexts. | | Persian embeddings / RAG | [FaMTEB](https://arxiv.org/abs/2502.11571), [PTEB Leaderboard](https://huggingface.co/spaces/PartAI/pteb-leaderboard) | If the assistant uses retrieval, do not evaluate only generation. Evaluate embeddings, retrieval, reranking, grounding, and citation behavior. | | Custom local evaluation | [ParsBench](https://github.com/ParsBench/ParsBench), [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) | Useful if you want repeatable private evaluation rather than only leaderboard screenshots. | ### Why this matters If the model fails on Persian IFEval-style tasks, you probably need better instruction-following data. If it fails on PerMMLU/PersianMMLU-style tasks, you may need domain or knowledge data, or maybe RAG rather than SFT. If it fails on ELAB-style tasks, you need safety/alignment data, not generic QA. If it fails because Persian inputs become very long under the tokenizer, more SFT examples may not be enough; you may need to think about tokenizer efficiency, model choice, or continued pretraining. * * * ## 2. Do not treat “Persian” as one fully specified target It is useful to state the target variety explicitly. Persian/Farsi can mean Iranian Persian, but Persian is also a pluricentric language with varieties such as Dari and Tajik. If your target is an Iranian Persian assistant, say that. If you want Dari, Tajik, code-switching, Arabic-script Persian only, Latin transliteration, or mixed Persian-English technical support, those become different data and evaluation problems. A simple target statement can prevent many later mistakes: > Target: Iranian Persian/Farsi assistant for \, mostly formal/semi-formal register, Arabic-script Persian, occasional English technical terms, no Dari/Tajik coverage for now. Or: > Target: general Persian assistant covering Iranian Persian plus common code-switching in technical conversations. This affects: - source selection - spelling normalization - register - cultural assumptions - evaluation examples - safety examples - tokenizer measurements - what counts as a natural answer * * * ## 3. Define “quality” by capability, not by dataset size A large dataset can still be low quality if it is repetitive, mistranslated, inconsistent, contaminated, or irrelevant to the target model’s failure modes. For a Persian SLM assistant, I would split quality into dimensions like this: | Quality dimension | What to check | Persian SLM-specific note | | --- | --- | --- | | Persian fluency | Natural Persian, spelling, orthography, punctuation, register | Translated English data alone may sound unnatural. Native or fluent review matters. | | Instruction following | Constraints, requested format, multi-step tasks, refusal when needed | Use Persian IFEval-like and private instruction tests. | | Multi-turn ability | Context tracking, follow-up answers, corrections, clarification | Do not evaluate only single-turn QA. | | Factuality | Verified answers, domain correctness, local knowledge | Use PersianMMLU/PerMMLU/PARSE-like tests, but do not train on their test items. | | Cultural fit | Iranian/Persian norms, idioms, local references, etiquette | Use culturally grounded resources; translation-only data can miss this. | | Safety/alignment | Refusal behavior, safe alternatives, fairness, privacy, social norms | ELAB-like evaluation is relevant here. | | Domain fit | Medical/legal/education/support correctness | Domain data may need expert validation. | | Retrieval behavior | Finds correct docs, cites evidence, avoids unsupported claims | Evaluate embedding/retrieval separately with FaMTEB/PTEB-like resources. | | Format consistency | Stable schema, roles, chat template, answer style | Bad formatting can ruin otherwise good data. | | Tokenization cost | Tokens per word/sentence, truncation rate, context waste | Especially important for 0.5B–3B SLMs. | | Provenance/license | Source, generation method, license, redistribution rights | Required if you publish the dataset. | | Contamination control | No benchmark leakage into train data | Critical if you use public Persian benchmarks to guide development. | A good Persian assistant dataset is not just “lots of Persian conversations.” It is a dataset that targets one or more of these dimensions clearly. * * * ## 4. Compare against Persian-specialized baselines before training Before spending time building a dataset, evaluate your base SLM against existing Persian-specialized or Persian-adapted baselines. Examples worth inspecting include: - [Dorna-Llama3-8B-Instruct](https://huggingface.co/PartAI/Dorna-Llama3-8B-Instruct) - [PersianMind](https://huggingface.co/universitytehran/PersianMind-v1.0) - [Persian-Phi](https://arxiv.org/abs/2512.07454) - current models listed on [Open Persian LLM Leaderboard](https://huggingface.co/spaces/PartAI/open-persian-llm-leaderboard) - current models listed on [MIZAN](https://huggingface.co/spaces/MCINext/mizan-llm-leaderboard) The goal is not necessarily to use those models directly. The goal is to learn what type of gap you are dealing with. | Observation | Likely implication | | --- | --- | | Your SLM is weak at basic Persian fluency | SFT alone may not be enough; consider model choice, continued pretraining, or tokenizer issues. | | It is fluent but bad at following instructions | SFT data may help. | | It follows instructions but lacks local facts | Use RAG or domain knowledge data; do not expect generic chat data to fix this reliably. | | It gives unsafe or culturally odd answers | You need alignment/safety/culture data, not just more QA. | | It fails long Persian inputs because of token length | Measure tokenizer fertility and truncation before scaling the dataset. | | Larger Persian models are much better | Maybe the task is too hard for the chosen SLM size, or needs a narrower scope. | * * * ## 5. Inspect existing Persian resources before generating from scratch Before creating 50k–100k synthetic examples, inspect existing Persian datasets and corpora. Useful examples: | Resource | How I would use it | | --- | --- | | [FarsInstruct](https://huggingface.co/datasets/ParsiAI/FarsInstruct), [paper](https://arxiv.org/abs/2407.11186), [GitHub](https://github.com/Hojjat-Mokhtarabadi/FarsInstruct) | A major Persian instruction-following resource. Useful because it is not just generic chat data; it covers multiple Persian NLP task types and instruction templates. | | [MatinaAI instruction tuning / alignment datasets](https://huggingface.co/datasets/MatinaAI/instruction_tuning_datasets) | Useful for cultural-alignment and Persian-focused instruction data patterns. Check access, license, and intended use carefully. | | [Matina Persian Text Corpus](https://arxiv.org/abs/2502.09188) | More relevant to language adaptation / continued pretraining than ordinary SFT. Useful if the model’s Persian base ability is weak. | | [ParsBench datasets/models](https://huggingface.co/ParsBench) | Useful for Persian task/evaluation exploration and possible private benchmark inspiration. | | [PQuAD](https://arxiv.org/abs/2202.06219) | Persian reading comprehension / QA resource. Good example of task-specific Persian data. | | [FarsTail](https://github.com/dml-qom/FarsTail) | Persian textual entailment resource. Useful for NLI-style evaluation or training inspiration. | | Community Persian SFT/COT collections, e.g. [xmanii Persian SFT/COT collection](https://huggingface.co/collections/xmanii/persian-sft-cot-fine-tuning-datasets) | Potential bootstrapping material, but inspect carefully. Do not assume translation-based community datasets are automatically high quality. | For community SFT datasets, I would check: - Is it native Persian, translated Persian, or synthetic Persian? - What model generated it? - What was the source dataset? - Is the license compatible with your use? - Are there duplicate examples? - Does it contain benchmark items? - Does it use the schema you need? - Are the answers natural in Persian? - Are there hidden English artifacts? - Does it match your target register and domain? * * * ## 6. Decide which layer you actually need to improve For a Persian SLM assistant, there are several different intervention layers. SFT is only one of them. | If the problem is… | Better first intervention | | --- | --- | | Poor basic Persian language modeling | Persian continued pretraining / language adaptation / better base model | | Bad instruction following | SFT on instruction-following data | | Bad multi-turn chat | Multi-turn SFT examples and multi-turn eval | | Bad local/cultural behavior | Culturally grounded Persian examples and human/native review | | Bad safety/refusal behavior | Safety/alignment dataset and Persian alignment eval | | Bad factual/domain answers | RAG, domain corpus, expert-validated QA, or domain SFT | | Bad retrieval | Better Persian embeddings/rerankers, FaMTEB/PTEB-like eval | | Bad formatting | Schema cleanup, chat template alignment, loss masking | | Too many tokens for Persian inputs | Tokenizer/model choice, shorter examples, context-aware data design | This is why evaluation first is useful: it tells you which layer to work on. * * * ## 7. A practical dataset-building pipeline A practical workflow could look like this. | Stage | Output | Notes | | --- | --- | --- | | 1. Define target | One short target statement | Example: “Iranian Persian customer-support assistant for \.” | | 2. Choose public eval anchors | 3–5 public benchmarks/leaderboards | Use them for orientation, not for training data. | | 3. Build private eval | 100–500 examples | Include the actual tasks users will ask. Keep it held out. | | 4. Evaluate base model | Failure report | Measure before training. | | 5. Compare baselines | Baseline table | Compare against Persian-specialized models if possible. | | 6. Inspect existing data | Dataset inventory | FarsInstruct, Matina, ParsBench, domain datasets, etc. | | 7. Build seed set | 200–2,000 high-quality examples | Manual/native-reviewed examples are valuable. | | 8. Expand synthetically | Larger candidate pool | Use teacher LLMs carefully; do not blindly trust outputs. | | 9. Filter | Clean training set | Fluency, correctness, diversity, safety, format, license. | | 10. Deduplicate/decontaminate | Train/dev/test split | Remove duplicates and eval leakage. | | 11. Train | SFT/LoRA/QLoRA run | Use correct chat template and loss masking. | | 12. Re-evaluate | New failure report | Add targeted examples based on failures. | The key loop is: > evaluate → inspect failures → add targeted data → train → re-evaluate not: > generate a huge dataset → train once → hope quality improves * * * ## 8. Build a small private eval set Public leaderboards are useful, but they will not perfectly match your application. I would create a small private eval set early. Even 100–300 examples can be very useful if they are well chosen. Suggested categories: | Category | Example | | --- | --- | | Basic Persian assistant | “Explain \ in simple Persian.” | | Instruction constraints | “Answer in exactly 3 bullet points.” | | Multi-turn follow-up | User corrects or narrows the previous request. | | Local/cultural knowledge | Iranian holidays, etiquette, food, education, bureaucracy, etc. | | Domain task | Your real target domain. | | Safety/refusal | Harmful, privacy-sensitive, or ethically sensitive requests. | | Formatting | JSON, markdown table, numbered list, citation format. | | RAG/grounding | Answer using provided documents only. | | Robustness | Ambiguous questions, typos, mixed Persian-English, informal spelling. | A simple private eval JSONL schema: ```json {"id":"eval_0001","category":"instruction_following","language":"fa","input":"","expected_behavior":"Follow all constraints; answer in Persian; no extra sections.","must_include":[],"must_not_include":[],"notes":"Private held-out eval. Do not train on this."} {"id":"eval_0002","category":"safety","language":"fa","input":"","expected_behavior":"Refuse briefly in Persian and offer a safe alternative.","must_include":[],"must_not_include":[],"notes":"Private held-out eval. Do not train on this."} ``` Do not overcomplicate the first version. A small, stable, private eval is better than no eval. * * * ## 9. Do not train on your evaluation data This is important. If you use PersianMMLU, MIZAN, ELAB, Persian IFEval, Persian MT-Bench, or leaderboard samples to guide your work, do not copy those examples into your SFT data. Also avoid putting held-out benchmark examples into prompts for synthetic data generation. Bad pattern: ```plaintext Take these PersianMMLU examples and generate many similar examples. ``` Better pattern: ```plaintext We need examples that test high-school-level Persian science explanations, but do not copy from any benchmark. Generate new questions from independently sourced material that is not in the held-out eval set. ``` Use public benchmarks to define **categories and failure modes** , not to create near-duplicate training examples. This matters because contamination can make a model look better on a leaderboard without actually becoming more useful. * * * ## 10. Synthetic data can help, but only after the seed set is clear Synthetic data is useful, especially for low-resource languages, but it needs quality control. Useful references and tools: - [Self-Instruct](https://arxiv.org/abs/2212.10560) - [WizardLM / Evol-Instruct](https://arxiv.org/abs/2304.12244) - [distilabel](https://github.com/argilla-io/distilabel) - [Bonito](https://arxiv.org/abs/2402.18334) - [Aya Dataset](https://arxiv.org/abs/2402.06619) - [Bactrian-X](https://arxiv.org/abs/2305.15011) - [MURI](https://arxiv.org/abs/2409.12958) For Persian, I would be careful about: | Risk | What to do | | --- | --- | | Translationese | Have native/fluent reviewers check samples. | | Cultural mismatch | Include Persian-local examples, not only translated English tasks. | | Repetition | Deduplicate prompts and answers. | | Overly generic assistant style | Add real target-domain tasks. | | Wrong facts | Verify factual/domain answers. | | Unsafe completions | Add safety filters and alignment eval. | | Teacher model bias | Use multiple teacher models or human review for important categories. | | Benchmark contamination | Keep eval examples out of generation prompts. | A safe synthetic expansion pattern: 1. Write 200–500 excellent seed examples. 2. Define task categories and style rules. 3. Generate candidate examples. 4. Filter automatically for schema/length/language. 5. Review samples manually. 6. Deduplicate. 7. Train a small run. 8. Evaluate. 9. Add more only where eval shows a gap. * * * ## 11. Data selection: quality, complexity, diversity A useful mental model is to select data using three axes: | Axis | Meaning | | --- | --- | | Quality | Is the answer correct, natural, safe, and useful? | | Complexity | Does it teach the model nontrivial behavior? | | Diversity | Does it cover enough tasks, domains, styles, and difficulty levels? | Relevant papers: - [LIMA](https://arxiv.org/abs/2305.11206): small, carefully curated instruction data can be surprisingly effective. - [AlpaGasus](https://arxiv.org/abs/2307.08701): filtering noisy instruction data can improve results. - [What Makes Good Data for Alignment? / Deita](https://arxiv.org/abs/2312.15685): quality, complexity, and diversity are useful selection dimensions. - [Long Is More for Alignment](https://arxiv.org/abs/2402.04833): simple selection heuristics such as response length can be strong baselines. - [Large-Scale Data Selection for Instruction Tuning](https://arxiv.org/abs/2503.01807): automatic data selection methods do not always scale cleanly, so evaluation and ablation matter. For a Persian SLM, I would avoid both extremes: - only tiny “perfect” examples with no diversity - huge synthetic data dumps with no review A balanced dataset is usually better. * * * ## 12. Measure tokenizer cost for an SLM For a small model, tokenization can matter a lot. If Persian text becomes much longer than equivalent English text under your tokenizer, the model uses more context and compute just to represent the input. That can hurt training efficiency and inference quality. Measure this before scaling your dataset. Example: ```python # pip install transformers from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("") samples = [ "این یک جملهٔ نمونه به زبان فارسی است.", "لطفاً این متن را در سه bullet point خلاصه کن.", "کاربر می‌خواهد دربارهٔ توضیح ساده‌ای دریافت کند.", ] for s in samples: ids = tokenizer(s, add_special_tokens=False)["input_ids"] words = s.split() print({ "text": s, "chars": len(s), "words": len(words), "tokens": len(ids), "tokens_per_word": len(ids) / max(1, len(words)), "tokens_per_char": len(ids) / max(1, len(s)), }) ``` Also check: | Metric | Why | | --- | --- | | average tokens per Persian example | Cost and context length | | truncation rate | Whether long examples are being cut | | tokens per word | Rough tokenization fertility | | Persian vs English equivalent length | Whether the tokenizer is inefficient for Persian | | mixed Persian-English examples | Important for technical assistants | If tokenizer cost is bad, possible responses include: - choose a better base model/tokenizer - shorten examples - reduce unnecessary verbosity - use narrower tasks - consider language adaptation / continued pretraining - avoid assuming SFT alone will solve everything PersianMind is an example where vocabulary adaptation was part of the Persian model-building story, so this is not just a theoretical concern. * * * ## 13. HF / TRL formatting matters Good content can become bad training data if the format is wrong. For Hugging Face TRL, check: - [TRL SFTTrainer](https://huggingface.co/docs/trl/en/sft_trainer) - [TRL dataset formats](https://huggingface.co/docs/trl/main/en/dataset_formats) - [Transformers chat templates](https://huggingface.co/docs/transformers/main/chat_templating) Common formats include: ### Conversational format ```json {"messages":[{"role":"system","content":"You are a helpful Persian assistant."},{"role":"user","content":""},{"role":"assistant","content":""}]} ``` ### Prompt-completion format ```json {"prompt":"","completion":""} ``` Things to verify: | Issue | Why it matters | | --- | --- | | Correct chat template | Chat models expect specific control tokens and role formatting. | | Assistant-only loss | You may want loss only on assistant responses, not user prompts. | | Completion-only loss | For prompt-completion SFT, train on completions rather than prompts. | | Multi-turn formatting | Roles and turn boundaries must be unambiguous. | | System prompt consistency | Random system messages can create unstable behavior. | | Persian punctuation/normalization | Inconsistency can add avoidable noise. | | Train/inference match | The fine-tuning template should match how you will call the model later. | A dataset can look good in a spreadsheet and still fail because the model was trained on the wrong serialized chat format. * * * ## 14. Minimal training data schema For a Persian assistant SFT dataset, I would keep metadata. It makes filtering and later analysis much easier. Example JSONL: ```json {"id":"sft_000001","messages":[{"role":"system","content":"You are a helpful Persian assistant."},{"role":"user","content":""},{"role":"assistant","content":""}],"source":"manual_seed","language":"fa","variety":"iranian_persian","domain":"general","quality_checked":true,"reviewer_type":"native_or_fluent","license":"","notes":"Do not include eval examples."} {"id":"sft_000002","messages":[{"role":"system","content":"You are a helpful Persian assistant."},{"role":"user","content":""},{"role":"assistant","content":""}],"source":"synthetic_reviewed","teacher_model":"","language":"fa","variety":"iranian_persian","domain":"","quality_checked":true,"reviewer_type":"fluent","license":"","notes":"Generated from category spec, not from benchmark examples."} ``` Useful metadata fields: | Field | Purpose | | --- | --- | | `id` | Deduplication and audit trail | | `source` | manual, translated, synthetic, scraped, domain expert, etc. | | `teacher_model` | Needed if synthetic | | `language` | Persian/Farsi, Dari, Tajik, mixed, etc. | | `variety` | Iranian Persian, Dari, Tajik, mixed | | `domain` | general, medical, legal, education, support, etc. | | `quality_checked` | Whether reviewed | | `reviewer_type` | native, fluent, expert, automatic only | | `license` | Reuse constraints | | `eval_overlap_checked` | Whether contamination check was done | | `notes` | Known caveats | * * * ## 15. Dataset publication quality If you publish the dataset on Hugging Face, the dataset card is part of the quality. Use: - [Hugging Face Dataset Cards](https://huggingface.co/docs/hub/datasets-cards) - [Create a dataset card](https://huggingface.co/docs/datasets/dataset_card) At minimum, document: | Dataset card item | What to write | | --- | --- | | Intended use | SFT, evaluation, DPO, RAG, domain adaptation, etc. | | Language scope | Iranian Persian, Dari, Tajik, code-switching, etc. | | Data sources | Manual, translated, synthetic, scraped, domain documents | | Generation process | Prompts, teacher models, translation method | | Human review | Who reviewed, how much, what criteria | | Filtering | Dedup, language ID, safety filters, length filters | | Splits | Train/dev/test, held-out eval, no overlap policy | | Contamination policy | What benchmarks were excluded | | License | Dataset license and inherited source licenses | | Limitations | Translation artifacts, domain gaps, safety gaps | | Ethical notes | Bias, harmful content handling, privacy considerations | A dataset without documentation may be hard for others to trust, even if the examples look good. * * * ## 16. Special case: domain assistants If the assistant is for a domain, general Persian SFT data is not enough. | Domain | Extra requirement | | --- | --- | | Medical | Expert validation, conservative answers, disclaimers, Persian medical QA eval | | Legal | Jurisdiction-specific knowledge, refusal boundaries, citations | | Education | Curriculum alignment, grade level, step-by-step explanations | | Customer support | Company/product-specific data, policy consistency | | Religious/cultural | Careful cultural review, sensitivity, source grounding | | News/current events | RAG and source freshness, not static SFT | For domain assistants, I would usually prefer: - small expert-reviewed SFT set - strong RAG pipeline - domain-specific private eval - refusal/uncertainty examples - citations or source-grounded answers rather than a huge generic Persian chat dataset. * * * ## 17. What I would avoid I would avoid these patterns: | Pattern | Why risky | | --- | --- | | “Generate 100k Persian conversations with ChatGPT and fine-tune.” | Likely repetitive, translation-like, weakly targeted, and hard to audit. | | Training on benchmark examples | Contamination and misleading scores. | | Using only translated English instruction data | Persian cultural and linguistic naturalness may be weak. | | No private eval set | You cannot tell what improved. | | No baseline comparison | You may build data for a model that is simply the wrong base model. | | No tokenizer check | SLM context/efficiency problems may be misdiagnosed as data quality problems. | | Ignoring chat template | Bad formatting can erase the value of good examples. | | No dataset card | Others cannot assess provenance, license, or limitations. | | Treating leaderboard rank as product readiness | Leaderboards are useful signals, not deployment guarantees. | * * * ## 18. A compact roadmap Here is the full process in one table. | Step | Action | Output | | --- | --- | --- | | 1 | Define target assistant | Scope statement | | 2 | Pick public Persian eval anchors | Benchmark list | | 3 | Build private eval set | 100–500 held-out examples | | 4 | Evaluate base model | Failure categories | | 5 | Compare Persian baselines | Model comparison | | 6 | Measure tokenizer cost | Tokens/example, truncation rate | | 7 | Inspect existing datasets | Data inventory | | 8 | Decide missing layer | SFT, CPT, RAG, safety, domain, etc. | | 9 | Build manual seed data | High-quality seed examples | | 10 | Expand synthetically if useful | Candidate pool | | 11 | Filter and review | Clean training set | | 12 | Deduplicate/decontaminate | Safe train/dev/test split | | 13 | Train with correct format | SFT/LoRA/QLoRA run | | 14 | Re-evaluate | Failure report | | 15 | Add targeted data | Next iteration | | 16 | Document dataset | Dataset card | * * * ## 19. Final practical checklist Before calling the dataset “high quality,” I would want to answer these questions: - What exact Persian variety/register is targeted? - Which public benchmarks did you inspect? - What is your private held-out eval set? - What are the base model’s main failures? - Did you compare against Persian-specialized baselines? - Is the data native, translated, synthetic, or mixed? - Who reviewed Persian fluency? - Are answers factually checked where needed? - Is there enough diversity of task type and difficulty? - Are safety/refusal examples included? - Are cultural/local examples included? - Are benchmark examples excluded from training? - Are duplicates removed? - Is the dataset formatted for the exact chat template? - Is assistant-only/completion-only loss handled correctly? - Have you measured tokenization cost? - Is the license clear? - Is the dataset card complete? If most of these are answered, the dataset is much closer to “high quality” in a practical sense. * * * ## Bottom line For a Persian SLM assistant, I would define dataset quality like this: > A high-quality dataset is a documented, decontaminated, eval-driven Persian dataset that targets the measured weaknesses of the chosen model: fluency, instruction following, multi-turn behavior, factuality, cultural fit, safety, domain knowledge, retrieval behavior, formatting, or tokenizer efficiency. So the best first move is not to generate a huge dataset. The best first move is to build the evaluation map, measure the current model, and then create only the data that addresses the measured gaps. ---
**Author:** ![Bidram](https://avatars.discourse-cdn.com/v4/letter/b/71e660/32.png) [@Bidram](https://discuss.huggingface.co/u/Bidram)\ **Post date:** [June 7, 2026, 6:55pm UTC](https://discuss.huggingface.co/t/how-can-i-build-a-high-quality-dataset/176571/3 "2026-06-07T18:55:44Z")
I am targeting Iranian Persian only. By high-quality, I mean a natural Persian dataset with fluent grammar, correct syntax, and student/teacher usage and later when the SLM was good enough, a Voice assistant for teacher/students with low-end devices. I am using Qwen 3.5 0.8B as the base model. I also tested Qwen 3 0.6B, but its tokenizer was very inefficient for Persian. Even with Qwen 3.5 0.8B, the tokenizer is still not very good for Persian, and the model struggles with simple Persian assistance tasks, grammar, and syntax. Because of that, I do not think this problem can be fixed with an SFT dataset alone. The model needs continued pretraining (CPT). I can run CPT with LoRA rank 64, and it works well enough in practice. I tested this before with Qwen 3 0.6B trained on about 800 MB of cleaned Persian Wikipedia, and the model improved noticeably. Before that, it could barely write even two Persian words correctly. For training data, I selected: - **Persian Wikipedia** (~2 GB) - **Persian OSCAR** (~3 GB, but very noisy) - **Persian Aya dataset** (~600 MB) Right now, I am trying to train an n-gram model to identify whether a given text is natural and correct Persian. If the text does not meet that standard, I reject it. I should also mention that I clean the text as much as possible before sending it to the n-gram model. At the moment, this is the best approach I can think of. I do not believe a simple scripted pipeline can reliably identify natural Persian quality on its own. Because of my hardware limitations, I cannot use local LLMs for filtering. Smaller LLMs are not good enough at Persian, and larger ones are too expensive to run. For example, I get around **70 tokens/sec** with **Gemma E2B Q4** and around **40 tokens/sec** with **Gemma E4B Q4**. And I have two questions: 1. In continued pretraining (CPT), the model learns grammar, syntax, and additional knowledge, right? My understanding is that CPT changes the model more deeply, in a way that affects its outputs broadly, while SFT mainly teaches the model how to respond in the desired format and follow patterns more reliably. In other words, CPT builds the underlying language ability, while SFT guides that ability toward better answers. Is that understanding correct? 2. If I want the model to gain a larger amount of knowledge ( not at a GPT level, but enough to remember or recall information from the CPT data ) do I need to include everything I want it to remember in the CPT dataset? I should also mention that API costs for AI models are very high in my country. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [June 8, 2026, 1:21am UTC](https://discuss.huggingface.co/t/how-can-i-build-a-high-quality-dataset/176571/4 "2026-06-08T01:21:16Z")
Oh. That clarification narrows the focus quite a lot: * * * ## Short answer Your understanding is mostly correct. If the target is **Iranian Persian only** , the model is **Qwen3.5-0.8B** , and the model still struggles with basic Persian grammar/syntax, then I agree that this is probably **not an SFT-only problem**. A useful mental model is: | Stage | Best for | Not ideal for | | --- | --- | --- | | CPT / continued pretraining | Persian language grounding, grammar, syntax, orthography, style, broad domain familiarity | Teaching exact assistant behavior or reliable factual recall | | SFT | Instruction following, teacher/student answer style, dialogue format, refusals, formatting, persona | Repairing weak base Persian language ability | | RAG / retrieval | Reliable factual recall, changing knowledge, exact textbook/document facts | Improving the model’s internal Persian grammar | So I would summarize it like this: > CPT gives the model more Persian language mass. SFT tells it how to behave as an assistant. RAG or a small local knowledge base is better when exact knowledge must be recalled reliably. Your experience with ~800 MB cleaned Persian Wikipedia improving Qwen3-0.6B is consistent with that. * * * ## 1. CPT vs SFT: your understanding is mostly right I would phrase it slightly more carefully: ### CPT Continued pretraining is still next-token prediction. It can move the model’s internal distribution toward Persian: - grammar - syntax - punctuation - orthography - style - common expressions - common factual associations - domain familiarity - local writing conventions This is why CPT is often used for language adaptation. Meta’s overview of LLM adaptation also describes continued pretraining as useful when the goal is to add capabilities such as multilingual ability, while noting that it is more expensive and can risk forgetting: [Adapting Large Language Models](https://ai.meta.com/blog/adapting-large-language-models-llms/). For low-resource language adaptation, a similar staged pattern appears in some work like: - CPT for language grounding - SFT for task/instruction specialization Example: [TibetanLLM: CPT + SFT for Tibetan language adaptation](https://arxiv.org/abs/2512.03976). ### SFT SFT is better for teaching the model: - how to answer as a teacher - how to follow instructions - how to produce student-friendly explanations - how to ask clarifying questions - how to refuse unsafe requests - how to use a particular chat format - how verbose or concise it should be - what style of Persian answer you want But if the base model cannot produce stable Persian sentences, SFT often becomes inefficient. You may end up teaching answer patterns on top of a weak language foundation. So I would agree with your main diagnosis: > If the model cannot reliably handle basic Persian grammar and syntax, CPT or language adaptation should come before serious SFT. * * * ## 2. Consider whether to CPT the Base model or the post-trained model Qwen provides both: - [Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B) - [Qwen3.5-0.8B-Base](https://huggingface.co/Qwen/Qwen3.5-0.8B-Base) If possible, I would consider this order: ```plaintext Qwen3.5-0.8B-Base -> Persian CPT / language adaptation -> SFT for teacher/student assistant behavior -> optional preference tuning / DPO later ``` Why? Because CPT on an already post-trained/instruct model can still work, but it may partially degrade instruction-following behavior. If you CPT the instruct/post-trained model, I would keep a small instruction-following regression eval and check it after every CPT run. Practical rule: | If you use… | Watch for… | | --- | --- | | Base model | You need SFT afterward before it behaves like an assistant | | Instruct/post-trained model | CPT may damage some instruction-following behavior | | LoRA CPT | Safer/cheaper, but limited capacity compared with full CPT | | Full CPT | More capacity, more cost, more forgetting risk | Since you can run LoRA rank 64 and it already helped in practice, it sounds like a reasonable constraint-aware approach. * * * ## 3. Does the model need to see everything in CPT to remember it? Partly yes, but with an important caveat. If you want the model to become broadly familiar with some knowledge, the model must see that knowledge during training somehow. CPT exposure can help the model internalize patterns and associations. But CPT is **not a reliable database**. For exact knowledge recall, especially facts, dates, school content, rules, or domain-specific material, I would not rely only on parametric memory. A relevant comparison is [Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs](https://arxiv.org/abs/2312.05934), which found that retrieval-augmented generation often outperforms unsupervised fine-tuning for knowledge-intensive tasks and that learning new factual information through unsupervised fine-tuning can be difficult. A practical split: | Knowledge type | Better method | | --- | --- | | General Persian grammar and syntax | CPT | | Common Iranian Persian writing patterns | CPT | | General educational style | CPT + SFT | | Teacher/student answer behavior | SFT | | Exact textbook facts | RAG or local knowledge base | | Changing/current facts | RAG/search | | Small set of very frequent facts | CPT/SFT may be acceptable | | High-stakes facts | Retrieval + citations + conservative answer style | For a low-end-device voice assistant, full RAG may be hard, but you can still think in layers: ```plaintext Very common knowledge -> CPT/SFT Exact or large knowledge -> compressed local KB / retrieval if possible Teacher behavior -> SFT Voice interface -> later ASR/TTS/latency problem ``` If you try to put all knowledge into CPT, the model may remember some of it, but recall will not be perfectly reliable, especially at 0.8B scale. * * * ## 4. Your selected data sources make sense, but they have different roles Your sources: - Persian Wikipedia - Persian OSCAR - Persian Aya are not equivalent. I would not mix them blindly. | Source | Best use | Main risk | | --- | --- | --- | | Persian Wikipedia | Clean-ish formal Persian, encyclopedic facts, stable style | Too encyclopedic; not conversational or teacher/student by itself | | Persian OSCAR | Broader web Persian, more style diversity | Very noisy, mixed language, boilerplate, duplicates, spam | | Persian Aya | Instruction-following data | More SFT-like than CPT-like; may not be ideal as raw CPT text | ### Wikipedia Good for: - formal grammar - basic factual associations - clean-ish prose - general encyclopedic style Risk: - the model may become too encyclopedia-like - not enough student/teacher dialogue - not enough conversational assistant style ### OSCAR OSCAR is useful, but I would treat it as **raw material** , not clean training data. The OSCAR 23.01 documentation mentions metadata such as KenLM-based harmful-content perplexity, TLSH hashes for near deduplication, sentence-level language identification, and quality warnings: [OSCAR 23.01 docs](https://oscar-project.github.io/documentation/versions/oscar-2301/). That supports your instinct: OSCAR can be valuable, but only after strong cleaning. ### Aya [Aya](https://huggingface.co/datasets/CohereLabs/aya_dataset) is more instruction-oriented. It may be useful for SFT or for a small instruction mixture, but I would not treat it the same way as Wikipedia/OSCAR for CPT. For CPT, I would prefer raw fluent Persian prose. For SFT, I would prefer instruction/response examples. * * * ## 5. A better CPT mixture might be staged Instead of one big mixture immediately, I would test small stages. Example: | Stage | Data mixture | Goal | | --- | --- | --- | | CPT-1 | Clean Persian Wikipedia | Basic grammar, syntax, formal Persian | | CPT-2 | Wikipedia + filtered OSCAR | Broader style and web Persian | | CPT-3 | Add educational Persian prose if available | Teacher/student domain adaptation | | SFT-1 | Small high-quality teacher/student examples | Assistant behavior | | SFT-2 | More instruction/multi-turn examples | Dialogue robustness | I would avoid making noisy OSCAR too large early. A safe first ratio could be something like: ```plaintext 70-90% clean Persian Wikipedia / curated Persian prose 10-30% strongly filtered OSCAR 0-10% instruction-like data, if converted carefully ``` This is not a universal ratio. It is just a safe starting point. If filtered OSCAR improves eval, increase it. If it makes outputs noisier, reduce it. * * * ## 6. The n-gram filter idea is good, but use it as one signal I think your n-gram idea is practical under your constraints. [KenLM](https://aclanthology.org/W11-2123/) is a good fit for this kind of low-cost filtering because it is fast and small compared with neural LLM filtering. But I would not use a single rule like: ```plaintext low perplexity = good Persian high perplexity = bad Persian ``` That can fail. Why? - very repetitive boilerplate can have low perplexity - Wikipedia-like prose may be favored too much - short junk text can be unstable - copied templates may look fluent but be useless - unnatural but common web spam can get through - good informal Persian may be rejected if your n-gram LM was trained only on formal text Instead, use n-gram scoring as one filter in a pipeline. * * * ## 7. Good-LM / Bad-LM filtering may be stronger than one LM A useful cheap approach: ```plaintext Good Persian LM: trained on clean Persian Wikipedia + curated high-quality Persian text Bad Persian LM: trained on rejected OSCAR samples, spam, boilerplate, malformed text, mixed-language junk Score each candidate text: good_lm_score bad_lm_score difference_or_ratio = bad_score - good_score ``` Then select text that looks good under the good LM and not good under the bad LM. This is often more useful than a single perplexity threshold. Rough idea: ```plaintext accept if: good_perplexity is reasonable bad_perplexity is worse text length is reasonable Persian script ratio is high repetition is low duplicate score is low ``` Do not choose thresholds blindly. Sample 100 accepted and 100 rejected texts, read them, and adjust. * * * ## 8. Cheap filtering pipeline under hardware/API constraints Given your constraints, I would use a classical pipeline first. Something like: ```plaintext raw text -> normalization -> language/script filtering -> length filtering -> boilerplate removal -> repetition filtering -> exact dedup -> near dedup -> n-gram LM scoring -> optional fastText classifier -> manual sample audit -> CPT shard ``` ### Step 1: Normalize Persian For Persian preprocessing, [Hazm](https://www.roshan-ai.ir/hazm/content/hazm/index.html) is useful. It provides Persian normalization, tokenization, lemmatization, and related tools. Normalize things like: - Arabic/Persian variants of letters - spacing - half-space / ZWNJ issues - punctuation - repeated characters - strange Unicode artifacts ### Step 2: Script/language ratio Use cheap rules: ```plaintext Persian/Arabic-script character ratio Latin character ratio digit ratio symbol ratio average line length number of URLs number of repeated lines ``` Reject obvious junk before expensive scoring. ### Step 3: Deduplicate Do both: - exact dedup - near dedup For OSCAR 23.01, the documentation mentions TLSH hashes for exact and near deduplication. If you are using a different OSCAR version, you may need your own MinHash/SimHash/TLSH pipeline. ### Step 4: KenLM score Use KenLM perplexity as a quality signal. Train on your best available clean Persian text. Then score candidate documents. ### Step 5: Optional small classifier If you manually label examples as good/bad Persian, you can train a cheap classifier. [fastText](https://fasttext.cc/) is useful for this kind of lightweight text classification. It is much cheaper than LLM filtering. Example labels: ```plaintext __label__ good __label__ bad ``` This can become surprisingly useful after a few thousand labeled examples. * * * ## 9. What I would evaluate after each CPT run Do not wait until the final model. After each CPT run, check a small fixed eval set. | Eval | Why | | --- | --- | | Persian perplexity on held-out clean text | Did CPT improve Persian modeling? | | Tokenization stats | Are examples being truncated? | | Basic grammar prompts | Can it produce correct Persian sentences? | | Teacher/student prompts | Did educational explanation improve? | | Instruction-following prompts | Did CPT damage instruction-following? | | Repetition tests | Did it become repetitive? | | Mixed Persian-English prompts | Useful for technical/student settings | | Safety/refusal sanity checks | Make sure it did not become less safe | | Small factual probes | Did knowledge improve at all? | I would keep a small frozen eval like: ```json {"id":"fa_grammar_001","type":"grammar","prompt":"","expected_behavior":"Produce fluent Iranian Persian."} {"id":"teacher_001","type":"teacher","prompt":"","expected_behavior":"Explain simply, step by step, in Persian."} {"id":"if_001","type":"instruction_following","prompt":"","expected_behavior":"Exactly 3 bullets, no extra text."} {"id":"regression_001","type":"regression","prompt":"","expected_behavior":"Should not degrade after CPT."} ``` The regression part is important if you CPT an already post-trained/instruct model. * * * ## 10. About LoRA CPT rank 64 LoRA CPT with rank 64 can be a reasonable compromise. It probably will not have the same capacity as full CPT, but your empirical result matters: if it noticeably improved Qwen3-0.6B after Persian Wikipedia CPT, that is evidence that it is useful in your setup. I would just watch for: - overfitting to Wikipedia style - loss of instruction-following - repetition - catastrophic forgetting - too much formal/encyclopedic tone - weak conversational style - weak teacher/student style If you can afford it, run small ablations: ```plaintext A: Wikipedia only B: Wikipedia + filtered OSCAR C: Wikipedia + filtered OSCAR + educational prose D: same as B but fewer steps E: same as B but different LoRA rank ``` Even small ablations can teach you more than one big run. * * * ## 11. One important warning: do not overfit to “clean Persian” only Your n-gram filter may become too strict. If the filter only accepts very formal Wikipedia-style Persian, the model may become better at formal prose but not better as a student/teacher assistant. For your target, you probably need at least three Persian styles: | Style | Example source | | --- | --- | | Formal Persian | Wikipedia, books, formal articles | | Educational Persian | textbooks, explanations, lessons, student-facing content | | Conversational Persian | teacher/student dialogue, Q&A, simple explanations | If you only use formal text, SFT will have to fight the CPT style later. So I would keep a small amount of high-quality conversational/educational Persian, even if it is much smaller than the formal corpus. * * * ## 12. SFT after CPT After CPT, I would do SFT with a small, clean dataset. Do not start with huge SFT. Start with examples like: - explain a concept to a student - correct a student’s grammar - simplify a paragraph - ask a clarifying question - answer with examples - answer in short teacher style - refuse unsafe requests politely - handle mixed Persian-English technical terms - multi-turn follow-up For TRL, check the dataset format and loss masking carefully: - [TRL SFTTrainer](https://huggingface.co/docs/trl/en/sft_trainer) - [TRL dataset formats](https://huggingface.co/docs/trl/main/en/dataset_formats) - [Transformers chat templates](https://huggingface.co/docs/transformers/main/chat_templating) Important: ```plaintext CPT teaches language distribution. SFT teaches assistant behavior. Wrong chat template or wrong loss masking can waste good data. ``` If using chat data, make sure the model is trained on assistant outputs, not just random serialized conversations. * * * ## 13. My practical recommendation Given your constraints, I would do this: ### Phase 1: CPT data cleaning ```plaintext Persian Wikipedia -> clean -> normalize -> dedup -> train Good KenLM OSCAR Persian -> clean -> normalize -> language/script filter -> remove boilerplate -> dedup / near-dedup -> score with Good KenLM -> optionally score with Bad KenLM -> sample audit -> keep only high-confidence Persian ``` ### Phase 2: Small CPT runs ```plaintext Run 1: Wikipedia only Run 2: Wikipedia + filtered OSCAR Run 3: add educational Persian prose if available ``` Compare them with the same held-out eval. ### Phase 3: SFT ```plaintext Small teacher/student Persian SFT -> 500 to 5,000 excellent examples first -> then expand only if eval shows benefit ``` ### Phase 4: Knowledge ```plaintext For common background knowledge: CPT exposure is useful. For exact educational facts: use RAG / local KB if possible. For teacher style: SFT. ``` * * * ## Bottom line I think your direction is reasonable. Your two core ideas are right: 1. If the model cannot write Persian well, CPT is probably needed before SFT. 2. A cheap n-gram-based quality filter is a practical idea under hardware/API constraints. I would only refine the plan like this: > Use CPT for Persian language grounding, SFT for teacher/student assistant behavior, and RAG or a small local knowledge base for reliable factual recall. And for filtering: > Use the n-gram model as one signal, not the only signal. Combine it with normalization, script/language filters, deduplication, repetition filters, OSCAR metadata when available, and manual sampling. ---
**Author:** ![Bidram](https://avatars.discourse-cdn.com/v4/letter/b/71e660/32.png) [@Bidram](https://discuss.huggingface.co/u/Bidram)\ **Post date:** [June 8, 2026, 5:29am UTC](https://discuss.huggingface.co/t/how-can-i-build-a-high-quality-dataset/176571/5 "2026-06-08T05:29:06Z")
Thank you 🙏 But I still have one more question: how should I prepare the SFT dataset? For example, I need natural Persian SFT data, but I have not been able to find a dataset that feels truly natural in Persian. Also, as you said, I need to cover all the capabilities the model is expected to have, and writing everything by hand would take too much time. The model needs capabilities such as: - tool calling - school-grade math reasoning - strong Persian academic/helpful support for students - grammar help in Farsi, such as فاعل، مفعول، متمم, and similar concepts If the right approach is something like: CPT → SFT → Hand-Written SFT and I can keep the final hand-written SFT dataset small, that is completely fine for me, I am willing to write that part manually. However, I still do not understand how to generate SFT data efficiently from the raw text I already have. Assume the text has already been cleaned using the pipeline we discussed earlier. How can I turn that cleaned raw text into a useful SFT dataset in an efficient way? I remember you mentioned using a larger LLM to help generate SFT data from raw text. If possible, could you explain the best workflow for that? Also, if you can give me an example of the dataset size usually needed for SFT, I think I could make a much better plan. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [June 8, 2026, 7:06am UTC](https://discuss.huggingface.co/t/how-can-i-build-a-high-quality-dataset/176571/6 "2026-06-08T07:06:22Z")
Since this is already quite concrete, I looked into it directly: * * * ## Short answer Yes, your proposed high-level order is reasonable: ```plaintext CPT -> SFT -> small final hand-written / polished SFT ``` But I would slightly modify it: ```plaintext CPT -> small hand-written seed + private eval -> synthetic / generated SFT from cleaned raw text -> filtering + verification -> SFT -> small final hand-written / polished SFT ``` The important change is that the hand-written data should not only be at the end. It should also be used at the beginning as: - examples for the large teacher model - style guide - private evaluation set - final correction / polish set The main idea is: > Cleaned raw text is not an SFT dataset. It is source material. You turn it into SFT by designing task families, generating instruction-answer pairs, verifying them, and mixing them deliberately. So I would not try to generate one universal Persian SFT dataset. I would generate several smaller datasets, each targeting one capability. For example: | Capability | Dataset family | | --- | --- | | Persian grammar help | explanation, correction, quiz, role identification | | school-grade math | verified problem-solution pairs | | student support | simplify, explain, summarize, give examples | | tool calling | tool schemas + Persian user requests + tool-call traces | | knowledge QA | passage-grounded QA from source texts | | multi-turn assistant | document-to-dialogue or scenario-to-dialogue | | natural Persian style | small native/polished final SFT set | * * * ## 1. Raw text is source material, not SFT A cleaned Persian paragraph like this: ```plaintext ``` is good for CPT. But SFT needs something like: ```json {"messages":[{"role":"user","content":""},{"role":"assistant","content":""}]} ``` or: ```json {"prompt":"","completion":""} ``` That means you need a conversion step: ```plaintext cleaned raw text -> passage selection -> task type selection -> instruction generation -> answer generation -> verification -> formatting -> SFT dataset ``` This is the same general idea behind work such as [Bonito](https://arxiv.org/abs/2402.18334), which converts unannotated text into task-specific instruction tuning datasets, and [Raw Text is All You Need](https://arxiv.org/abs/2407.03040), which studies generating knowledge-intensive multi-turn dialogues from raw documents. You do not need to copy those exact methods. The useful idea is: > First decide what task the raw text should become. * * * ## 2. Use capability-specific datasets, not one giant mixed dataset For your case, I would split SFT data by target capability. ### Recommended split | Capability | Should it come from raw text? | Best construction method | | --- | --- | --- | | Persian grammar help | partly | grammar passages + hand-written seed + generated correction/explanation tasks | | student support | yes | educational passages → explanations, summaries, examples, exercises | | school math reasoning | not mainly | generated/translated problems + verified solutions | | tool calling | no, mostly separate | tool schema + user intent + tool call + tool result + final answer | | general helpful Persian | partly | seed examples + self-instruct-style generation + filtering | | factual Persian QA | yes | passage-grounded QA from cleaned text | | multi-turn dialogue | yes | document/scenario → multi-turn conversation | | final natural style | mostly hand-written/polished | small native final SFT set | The mistake to avoid is: ```plaintext I have clean Persian text, so I will ask an LLM to make random Q&A pairs from all of it. ``` That usually creates a noisy, repetitive, shallow SFT dataset. A better approach is: ```plaintext I need 7 capability buckets. For each bucket, I will design task templates. Then I will generate and verify examples for that bucket. Then I will mix the buckets intentionally. ``` * * * ## 3. Better training order Your proposed order: ```plaintext CPT -> SFT -> Hand-Written SFT ``` is basically good. I would expand it like this: | Stage | Purpose | | --- | --- | | CPT | Teach Persian distribution: grammar, syntax, style, general domain familiarity | | private eval | Keep a small set of Persian tasks that never enters training | | hand-written seed | Show the teacher LLM what “good Persian assistant data” looks like | | synthetic SFT generation | Scale from raw text and task templates | | filtering/verification | Remove wrong, unnatural, repetitive, or badly formatted examples | | first SFT | Teach assistant behavior | | final hand-written/polished SFT | Fix style, grammar tutor behavior, safety, and final answer tone | The seed set and final polish set can be small. They just need to be high quality. * * * ## 4. A practical raw-text-to-SFT workflow Here is a concrete workflow. ```plaintext 1. Split cleaned corpus into passages. 2. Classify passages by usefulness. 3. Assign each useful passage to one or more task families. 4. Generate instruction-answer pairs with a teacher LLM. 5. Verify automatically where possible. 6. Review samples manually. 7. Deduplicate. 8. Convert to TRL/HF chat format. 9. Train a small SFT. 10. Evaluate. 11. Add more data only where the model fails. ``` ### Passage classification Not every cleaned text is useful for SFT. Classify each passage: | Passage type | Use | | --- | --- | | grammar explanation | grammar tutor data | | educational explanation | student support data | | factual article | grounded QA / summarization | | math problem | math reasoning if answer is verifiable | | tool documentation | tool calling data | | noisy opinion/comment | maybe conversational style, but risky | | list/table/boilerplate | usually reject or handle separately | | very short text | often reject | | very long text | split or summarize first | You can use simple tags: ```json {"id":"p_0001","text":"","tags":["grammar","education"],"quality":"high"} {"id":"p_0002","text":"","tags":["factual","qa"],"quality":"medium"} {"id":"p_0003","text":"","tags":["boilerplate"],"quality":"reject"} ``` * * * ## 5. Use a teacher LLM with task templates For each task family, write a generation prompt. Do not simply say: ```plaintext Make SFT data from this text. ``` That is too vague. Instead, make the task explicit. ### Passage-grounded QA ```plaintext You are creating Persian SFT data for a small Iranian Persian assistant. Source passage: Create 3 Persian user questions that can be answered only from the passage. For each question, write a concise Persian answer. Do not add facts that are not in the passage. Use natural Iranian Persian. Return JSONL with fields: id, task_type, messages, source_passage_id. ``` ### Student explanation ```plaintext You are creating Persian SFT data for a student-support assistant. Source passage: Create: 1. one student question, 2. one simple explanation, 3. one example, 4. one short follow-up question the student might ask, 5. one assistant follow-up answer. The assistant should sound like a patient Persian teacher. Avoid English unless the source requires it. Return JSON. ``` ### Grammar tutor ```plaintext You are creating Persian grammar tutor SFT data. Grammar topic: Source passage: Create examples for: - explaining the concept - identifying the concept in a sentence - correcting a student's wrong answer - giving 3 new examples - making a mini quiz Use Iranian Persian. Keep explanations short and clear. Return JSONL. ``` ### Simplification ```plaintext Convert the passage into a student-friendly explanation. Source: Create: - user instruction - assistant answer - difficulty level: elementary / middle_school / high_school - key concepts - possible student confusion Use natural Persian. ``` This is much better than unrestricted generation. * * * ## 6. Grammar help in Farsi: build it as a separate dataset family For things like: - فاعل - مفعول - متمم - مسند - فعل - قید - صفت - مضاف و مضاف‌الیه - نقش‌های دستوری - جمله ساده / مرکب I would not rely only on raw text. You need targeted examples. Suggested grammar task families: | Task type | Example | | --- | --- | | concept explanation | “فاعل چیست؟ با مثال توضیح بده.” | | role identification | “در جمله زیر فاعل و مفعول را مشخص کن.” | | error correction | “دانش‌آموز گفته X مفعول است. آیا درست است؟” | | contrastive explanation | “فرق متمم و مفعول چیست؟” | | example generation | “برای متمم سه مثال ساده بساز.” | | mini quiz | “سه سوال کوتاه درباره فاعل و مفعول بساز.” | | step-by-step analysis | “این جمله را از نظر نقش‌های دستوری تحلیل کن.” | | student misconception | “چرا این پاسخ اشتباه است؟ ساده توضیح بده.” | A useful schema: ```json { "id": "grammar_0001", "task_type": "grammar_role_identification", "topic": "فاعل", "messages": [ {"role": "system", "content": "You are a helpful Persian grammar tutor."}, {"role": "user", "content": "در جمله «علی کتاب را خواند»، فاعل را مشخص کن و کوتاه توضیح بده."}, {"role": "assistant", "content": "در این جمله «علی» فاعل است، چون انجام‌دهندهٔ عمل خواندن است. «کتاب» مفعول است، چون عمل خواندن روی آن انجام شده است."} ], "source": "manual_seed", "language": "fa", "variety": "iranian_persian" } ``` For grammar tutoring, I would make at least a small hand-written seed set. Something like: ```plaintext 100-300 excellent grammar tutor examples ``` Then use those as few-shot examples for the teacher LLM. * * * ## 7. Student support data For student support, raw educational text is very useful. Convert passages into tasks like: | Task | Input | Output | | --- | --- | --- | | explain simply | passage | simple explanation | | summarize | passage | short Persian summary | | give examples | concept | 2-3 examples | | make exercise | passage | student exercise | | answer misconception | wrong student answer | correction | | scaffold | hard question | step-by-step hint | | quiz | lesson passage | short quiz | | compare concepts | two concepts | explanation of difference | Example generation prompt: ```plaintext You are building SFT data for an Iranian Persian student assistant. Source passage: Create 5 examples: 1. simple explanation 2. summary 3. example-based explanation 4. student misconception correction 5. mini quiz Rules: - Use natural Iranian Persian. - Do not add facts outside the passage unless clearly common knowledge. - Keep answers short enough for a small model. - Do not create very long chain-of-thought. - Return JSONL. ``` The “short enough for a small model” rule matters. A 0.8B model may not benefit from very long, complex answers early. * * * ## 8. Math reasoning: do not rely only on raw Persian text Math is different. For math SFT, correctness matters more than naturalness. I would not generate math examples only from raw Persian text. Instead: 1. generate or translate school-grade math problems, 2. solve them with a reliable solver or teacher model, 3. verify the answer, 4. write a Persian explanation, 5. sample-review. Relevant examples: - [GSM8K](https://huggingface.co/datasets/openai/gsm8k) is a classic school-grade math word-problem dataset. - [OpenMathInstruct](https://arxiv.org/abs/2402.10176) generated large math instruction data with code-interpreter-style solutions. - [MAmmoTH / MathInstruct](https://arxiv.org/abs/2309.05653) is another important math instruction tuning example. For your use case, I would make Persian school-grade math as a separate dataset family. Example schema: ```json { "id": "math_0001", "task_type": "school_math_reasoning", "grade_level": "middle_school", "messages": [ {"role": "system", "content": "You are a helpful Persian math tutor."}, {"role": "user", "content": "اگر علی ۳ دفتر بخرد و قیمت هر دفتر ۴۰ هزار تومان باشد، او در مجموع چقدر پرداخت می‌کند؟"}, {"role": "assistant", "content": "قیمت هر دفتر ۴۰ هزار تومان است و علی ۳ دفتر می‌خرد. پس باید ۳ را در ۴۰ هزار ضرب کنیم: ۳ × ۴۰٬۰۰۰ = ۱۲۰٬۰۰۰. بنابراین علی ۱۲۰ هزار تومان پرداخت می‌کند."} ], "answer": "120000", "verified": true } ``` For math, add automatic checks when possible. For arithmetic, you can verify with Python. ```python def check_answer(problem_id, predicted_answer, gold_answer): return str(predicted_answer).replace(",", "").strip() == str(gold_answer).strip() ``` If you generate 10,000 math examples but 20% are wrong, that can damage the model. Better: ```plaintext 2,000 verified math examples > 20,000 unverified math examples ``` * * * ## 9. Tool calling should be a separate dataset Tool calling does not naturally come from Persian raw text. You need a tool-use dataset with: - tool schema - Persian user request - assistant tool call - tool output - final Persian answer The TRL docs include tool-calling SFT support and describe using tool schemas and tool calls in the dataset: [TRL SFTTrainer](https://huggingface.co/docs/trl/en/sft_trainer). Tool-use work such as [ToolLLM / ToolBench](https://arxiv.org/abs/2307.16789) is useful conceptually because it treats tool use as its own data construction problem. A simple tool example: ```json { "tools": [ { "type": "function", "function": { "name": "calculator", "description": "Perform basic arithmetic.", "parameters": { "type": "object", "properties": { "expression": {"type": "string"} }, "required": ["expression"] } } } ], "messages": [ {"role": "user", "content": "حاصل ۲۳ ضربدر ۴۷ چقدر است؟"}, { "role": "assistant", "tool_calls": [ { "type": "function", "function": { "name": "calculator", "arguments": "{\"expression\":\"23*47\"}" } } ] }, {"role": "tool", "content": "1081"}, {"role": "assistant", "content": "حاصل ۲۳ ضربدر ۴۷ برابر با ۱۰۸۱ است."} ] } ``` You also need negative examples. | Example type | Why | | --- | --- | | tool needed | model learns when to call | | no tool needed | model does not call tools unnecessarily | | ambiguous request | model asks clarification | | unavailable tool | model refuses or explains limitation | | invalid arguments | model learns valid JSON/schema | | tool result summarization | model explains result in Persian | For Persian tool calling, I would start small: ```plaintext 200-500 high-quality tool examples ``` Then expand only after testing. * * * ## 10. A large LLM workflow for generating SFT data If you can use a larger LLM as a teacher, use it as a generator, but also as a critic/verifier. A simple pipeline: ```plaintext passage -> generator LLM creates candidate SFT examples -> verifier LLM checks faithfulness, Persian naturalness, format -> rule-based filters check JSON/schema/length -> sample manual review -> accepted examples enter SFT dataset ``` ### Generator prompt ```plaintext You are generating Persian SFT data for a small Iranian Persian assistant. Target capability: Source passage: Generate examples. Each example must include: - task_type - messages - source_passage_id - difficulty - verification_notes Rules: - Use natural Iranian Persian. - Keep answers concise. - Do not invent facts outside the passage. - Do not include unsafe content. - Do not use English unless needed. - Return valid JSONL only. ``` ### Verifier prompt ```plaintext You are verifying Persian SFT examples. Source passage: Candidate example: Check: 1. Is the Persian natural? 2. Is the answer faithful to the passage? 3. Is the instruction clear? 4. Is the answer useful for the target capability? 5. Is the format valid? 6. Is there any hallucinated fact? 7. Should this be accepted, rejected, or edited? Return: {"decision":"accept|edit|reject","reason":"...","fixed_example":...} ``` Do not accept everything from the teacher LLM. The teacher LLM is a data generator, not a guarantee of quality. * * * ## 11. You can implement this with simple scripts first You do not need a complex framework at the beginning. A simple Python pipeline is enough: ```plaintext input passages -> prompt templates -> teacher LLM calls -> JSON parser -> filters -> output JSONL ``` If the pipeline becomes large, tools like [distilabel](https://github.com/argilla-io/distilabel) can help organize synthetic data generation, AI feedback, judging, filtering, and dataset export. But I would start simple. A minimal folder structure: ```plaintext data/ raw_clean/ passages/ generated_candidates/ rejected/ accepted/ eval_private/ prompts/ grammar_generation.txt student_support_generation.txt math_generation.txt tool_calling_generation.txt verifier.txt scripts/ split_passages.py generate_candidates.py verify_candidates.py filter_jsonl.py dedup.py build_train_mix.py ``` * * * ## 12. Suggested SFT dataset sizes There is no universal number. The needed size depends on: - model size - base model quality - CPT quality - task complexity - data diversity - data correctness - evaluation target But for planning, I would use this rough scale: | Dataset part | Suggested starting size | | --- | --- | | private eval | 100-500 examples | | hand-written seed | 200-1,000 examples | | first SFT experiment | 1,000-5,000 examples | | grammar tutor data | 1,000-5,000 examples | | student support data | 2,000-10,000 examples | | math reasoning data | 2,000-20,000 verified examples | | tool calling data | 200-2,000 high-quality examples | | broad useful SFT | 5,000-30,000 examples | | large synthetic SFT | 50,000-100,000+ examples, only after filtering is mature | Some relevant scale references: - [LIMA](https://arxiv.org/abs/2305.11206): around 1,000 carefully curated examples showed that small, high-quality data can matter. - [AlpaGasus](https://arxiv.org/abs/2307.08701): selected around 9k high-quality examples from Alpaca-style data. - [SmolTalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk): 1M synthetic SFT samples used for SmolLM2-Instruct. - [SmolLM2](https://arxiv.org/abs/2502.02737): small-model training can still rely heavily on careful data design and staged training. But I would not start with 1M examples. For your project, a more realistic first target might be: ```plaintext CPT first private eval: 200-300 hand-written seed: 300-800 first generated SFT: 3k-10k final polished SFT: 500-2k ``` Then expand based on evaluation. * * * ## 13. Example final mixture for your case A possible first real SFT mixture: | Component | Examples | Notes | | --- | --- | --- | | Persian grammar tutor | 2,000 | include فاعل، مفعول، متمم, correction, quiz | | student support | 3,000 | explanations, examples, simplification | | passage-grounded QA | 3,000 | from cleaned Persian educational/factual text | | summarization/simplification | 1,500 | useful for student assistant | | school math | 2,000 | verified answers only | | tool calling | 500 | high quality, schema-valid | | safety/refusal | 500 | simple Persian safety cases | | final hand-written polish | 500 | naturalness and style | Total: ```plaintext ~13k examples ``` That is already enough for a serious first SFT experiment. If it works, expand the weak categories. * * * ## 14. How to mix generated and hand-written data I would not put hand-written data only at the end. Use it in three places: | Hand-written data role | Purpose | | --- | --- | | seed examples | show the teacher LLM what good data looks like | | private eval | measure model improvement | | final polish | fix the model’s final behavior/style | Workflow: ```plaintext write 300 excellent examples -> use 100 as private eval -> use 100 as few-shot generation examples -> keep 100 for final polish / style correction ``` Do not train on the private eval examples. * * * ## 15. Quality filters for generated SFT For each generated example, check: | Check | Method | | --- | --- | | valid JSON | parser | | correct language | language/script ratio | | not too long | token length filter | | not too short | length filter | | no duplicate | exact/near dedup | | source faithfulness | verifier LLM or rules | | natural Persian | sample review / classifier | | no benchmark leakage | decontamination | | correct math answer | calculator / solver | | valid tool schema | JSON schema validation | | useful task | category-specific review | | safe answer | safety filter / manual review | The most important filter depends on the task. | Task | Most important filter | | --- | --- | | grammar tutor | correctness and natural explanation | | math | answer verification | | tool calling | schema validity | | passage QA | source faithfulness | | student support | helpfulness and natural Persian | | safety | refusal quality | * * * ## 16. Example schema for mixed SFT I would keep metadata. It will help you debug later. ```json { "id": "sft_grammar_000001", "task_family": "persian_grammar_tutor", "source_type": "manual_seed", "source_passage_id": null, "difficulty": "middle_school", "language": "fa", "variety": "iranian_persian", "messages": [ {"role": "system", "content": "You are a helpful Persian tutor."}, {"role": "user", "content": "در جمله «مریم به مدرسه رفت»، متمم را مشخص کن."}, {"role": "assistant", "content": "در این جمله «به مدرسه» متمم است، چون با حرف اضافهٔ «به» آمده و معنی فعل «رفت» را کامل می‌کند."} ], "verified": true, "notes": "manual seed" } ``` For generated passage-grounded data: ```json { "id": "sft_qa_000001", "task_family": "passage_grounded_qa", "source_type": "generated_from_clean_text", "source_passage_id": "passage_12345", "language": "fa", "variety": "iranian_persian", "messages": [ {"role": "system", "content": "You are a helpful Persian assistant. Answer only from the provided passage."}, {"role": "user", "content": ""}, {"role": "assistant", "content": ""} ], "verified": true, "verifier": "teacher_llm_plus_manual_sample" } ``` For tool calling, keep it separate because the schema is different. * * * ## 17. What not to do I would avoid: | Bad approach | Why | | --- | --- | | Generate random Q&A from all raw text | Usually shallow and repetitive | | Mix tool calling, grammar, math, and chat without labels | Hard to debug | | Trust teacher LLM outputs without verification | Wrong answers enter training | | Generate huge data before first eval | You will not know what helped | | Use only passage-grounded QA | The model may not learn teacher behavior | | Use only final hand-written data | Too small and slow | | Use only synthetic data | Persian naturalness may be weak | | Ignore math verification | Bad math data is harmful | | Treat tool calling like normal chat | Tool use needs schema-valid traces | | Train on private eval examples | Contamination | * * * ## 18. Practical recommendation For your exact case, I would do this: ```plaintext 1. Finish CPT on clean Persian text. 2. Build 200-300 private eval examples. 3. Write 300-800 high-quality seed SFT examples manually. 4. Split cleaned text into passages. 5. Classify passages into task families. 6. Generate SFT candidates with a larger teacher LLM. 7. Verify generated examples by task type. 8. Build a first 5k-15k SFT mixture. 9. Fine-tune. 10. Evaluate. 11. Add data only for the categories that fail. 12. Finish with 500-2k hand-written/polished examples. ``` If I had to choose the first SFT categories, I would start with: ```plaintext Persian grammar tutor student support / explanation passage-grounded QA math with verified answers small tool-calling set ``` This is more realistic than trying to build every possible assistant capability immediately. * * * ## Bottom line Your plan is good, but I would refine it like this: ```plaintext CPT -> private eval + hand-written seed -> capability-specific synthetic SFT generation -> verification/filtering -> SFT -> small final hand-written polish ``` The key point is: > Do not generate one universal SFT dataset from raw text. Generate several small datasets for separate capabilities, verify each one differently, and then mix them intentionally. For your target model, I would rather have: ```plaintext 10k carefully categorized and verified Persian SFT examples ``` than: ```plaintext 100k random Persian Q&A pairs generated from raw text ``` ---
**Author:** ![Bidram](https://avatars.discourse-cdn.com/v4/letter/b/71e660/32.png) [@Bidram](https://discuss.huggingface.co/u/Bidram)\ **Post date:** [June 10, 2026, 10:40am UTC](https://discuss.huggingface.co/t/how-can-i-build-a-high-quality-dataset/176571/7 "2026-06-10T10:40:44Z")
Hello again! Sorry for the late reply. I’ve been cleaning my Persian CPT data and iterating through a **`[CPT clean -> review]`** loop, and overall things are going very well. But I’ve had one question on my mind for a long time. In the local models I tested, even relatively strong ones like **`Gemma 4 E4B UD-Q4`** struggled with noisy or unclear prompts. The model would often ignore parts of my instructions or fail to follow everything I said consistently. That made me wonder: if even large companies like Google could not make an `8B` model that reliably handles noisy prompts, how could I expect a `0.8B` model to do so while also maintaining attention across long, messy prompts? **So my question is:** is it really true that only larger models can maintain attention well and follow instructions more reliably? ---
**Author:** ![Bidram](https://avatars.discourse-cdn.com/v4/letter/b/71e660/32.png) [@Bidram](https://discuss.huggingface.co/u/Bidram)\ **Post date:** [June 10, 2026, 6:01pm UTC](https://discuss.huggingface.co/t/how-can-i-build-a-high-quality-dataset/176571/8 "2026-06-10T18:01:57Z")
I have already cleaned my Persian Wikipedia data with a script. I removed duplicates, normalized the text, and stripped boilerplate, HTML, and other Wikipedia-related garbage. Now I want to split the text into smaller chunks, label them as good or bad, and then train an n-gram model on the good data. However, I ran into an issue. For example, I found a passage like this in my dataset: مصدق از طرفداران انقلاب مشروطه ایران بود و معتقد بود که مقام سلطنت مقام عالی تشریفاتی است. او در دوران نخست‌وزیری کوشید تا قدرت شاه را محدود به چارچوب مشخص‌شدهٔ آن در قانون اساسی مشروطه کند و نهادهای مدنی را تقویت کرد. در زمان نخست‌وزیری او، خانوادهٔ سلطنتی (به خصوص اشرف پهلوی) از مخالفان سرسخت مصدق بودند. پس از کودتای ۲ ۸ مرداد، مصدق همچنان منتقد دخالت شاه در امر حکومت بود. اگرچه محمدرضاشاه کودتای ۲ ۸ مرداد را «قیام مردم» و «رستاخیز ملی ایران» می‌نامید، اما ۶ ۰ سال پس از کودتا، سیا با انتشار اسنادی، به دخالت مستقیمِ خود در ایران معترف شد.سیا به نقش خود در اجرای کودتای ۲ ۸ مرداد اعتراف کرد افکارنیوزاعتراف سازمان سیا رسماً به رهبری کودتای 28 مرداد پس از 60 سال عصر ایران BBC Persian برخی محققان معتقدند در جستجوی ریشهٔ اصلی انقلاب ۱ ۳ ۵ ۷ ایران، می‌توان به کودتای ۲ ۸ مرداد در سال ۱ ۳ ۳ ۲ بازگشت.همایون کاتوزیان، مصدق و کودتا (فصل اول: «دکتر محمد مصدق در تاریخ ایران») ص ۳ ۹–۴ ۵Abrahamian, Modern Iran, p۱ ۲ ۲ گازیوروسکی، مصدق و کودتا (فصل آخر: «نتیجه‌گیری») ص ۲ ۸ ۳ The beginning of the text looks fine, but the ending contains noisy and broken fragments. This kind of garbage appears often, especially near the ends of passages. Should I let the n-gram model handle this noise by itself, or should I remove it beforehand with a cleaning script? If cleaning is better, how should I approach it? ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [June 11, 2026, 10:27am UTC](https://discuss.huggingface.co/t/how-can-i-build-a-high-quality-dataset/176571/9 "2026-06-11T10:27:40Z")
Hmm, after looking into it, it seems to be something like this: * * * ## Short answer I would separate the two questions. For the first question: > Is it true that only larger models can maintain attention and follow noisy/long instructions reliably? Mostly yes, in practice. Larger models usually handle messy, long, multi-constraint prompts better. But I would not phrase it as “only large models can do it.” A 0.8B model can still be useful if the product design reduces the burden on the model. For the second question: > Should the n-gram model handle noisy Wikipedia tails, or should I clean them first? Clean them first. The n-gram model should be a **quality scorer** , not a **garbage collector**. If the text has good Persian prose followed by reference/citation garbage, that is usually a **boundary/truncation problem** , not necessarily a reason to reject the whole document. * * * ## 1. About 0.8B models and noisy prompts Your intuition is reasonable. A 0.8B model should not be expected to handle the same prompt complexity as a strong 7B, 8B, 14B, or frontier model. The problem is not only Persian. Even much larger models can fail when prompts are: - long - messy - ambiguous - internally contradictory - full of irrelevant context - full of embedded instructions - multi-step - multi-constraint - noisy or poorly formatted This is related to several known issues: - [Lost in the Middle](https://arxiv.org/abs/2307.03172): models may fail to use information reliably when it appears in the middle of long contexts. - [Instruction-following robustness / prompt injection](https://aclanthology.org/2024.emnlp-main.33/): models may struggle to distinguish which instructions to follow and which to ignore. - [Instruction-following survey](https://arxiv.org/abs/2303.10475): instruction following is a broad and still nontrivial problem. - [Long-context instruction following](https://aclanthology.org/2026.findings-eacl.254.pdf): longer context windows do not automatically solve instruction adherence. So I would agree with your concern: > If an 8B model struggles with noisy, unclear, long prompts, a 0.8B model will probably struggle more. But the practical answer is not simply “give up.” The answer is: > Do not design the system so that the 0.8B model has to solve everything inside one messy prompt. * * * ## 2. Reduce the burden on the 0.8B model For a small model, the system design matters a lot. Instead of asking the model to handle this: ```plaintext long noisy user prompt + mixed task + unclear context + multiple instructions + irrelevant text + long document + expected structured answer ``` try to convert it into this: ```plaintext short clean task + one clear instruction + limited relevant context + simple expected format ``` A 0.8B assistant can become much more useful if you do preprocessing before the prompt reaches the model. ### Practical design | Problem | Better design | | --- | --- | | User gives long messy text | clean/split/summarize before model call | | User asks multiple things | split into subtasks | | Prompt contains irrelevant context | retrieve/select only relevant spans | | Prompt is unclear | ask a clarification question | | Prompt has many constraints | use a template with explicit fields | | Long document QA | use short retrieved chunks | | Math | use calculator/tool when possible | | Tool calling | use strict schema and small examples | | Grammar help | classify the grammar task first, then answer | For example, instead of: ```plaintext User gives a long messy paragraph and asks the model to understand everything, correct grammar, summarize, answer questions, and explain. ``` you can make a pipeline: ```plaintext input -> normalize -> detect task type -> remove irrelevant noise -> split into smaller chunks -> select the useful part -> send short structured prompt to model ``` For a small model, this kind of pipeline is often more important than trying to make the model “smart enough” to handle arbitrary mess. * * * ## 3. Train on realistic noise, not arbitrary garbage There is a difference between useful robustness data and garbage data. ### Good noisy SFT examples Good noisy examples teach the model to handle realistic user input: ```plaintext typos informal Persian missing punctuation mixed Persian-English terms short unclear question student misconception slightly messy formatting ``` ### Bad noisy examples Bad examples teach the model to imitate broken data: ```plaintext citation fragments broken references HTML leftovers random source names duplicated lines malformed dates garbled bibliography text OCR garbage truncated sentences ``` The first kind can be useful for SFT. The second kind should usually be removed from CPT data and from the “good” n-gram corpus. So I would use this rule: | Noise type | Use in training? | | --- | --- | | natural human typo | maybe yes | | informal Persian | yes, if target includes it | | student mistake | yes, for tutor SFT | | unclear user question | yes, if assistant learns to clarify | | Wikipedia reference tail | no, remove or use as bad data | | broken source list | no | | malformed citation numbers | no | | duplicated boilerplate | no | | random mixed-language bibliography | no | * * * ## 4. About your Wikipedia example The example you showed looks like this: 1. The beginning is normal Persian prose. 2. The ending becomes reference/citation garbage. 3. There are broken spaced numbers like `۲ ۸`, `۱ ۳ ۵ ۷`, `۱ ۳ ۳ ۲`. 4. Source names and article titles are glued into the sentence. 5. The passage boundary seems wrong. So I would not treat that as a pure “bad document” problem. It is more like: ```plaintext good prose + bad trailing span ``` That means the best operation is often: ```plaintext keep the good part truncate the bad tail ``` not: ```plaintext reject the whole passage ``` and not: ```plaintext let the n-gram model figure it out ``` A good cleaning system should detect that the text changes from normal prose into citation/reference fragments. * * * ## 5. Clean before training the Good n-gram model For the Good n-gram model, only train on text that you would be happy for the model to imitate. If the Good n-gram model sees citation tails, it may learn that citation tails are normal Persian. So I would do this: ```plaintext raw Wikipedia text -> markup/HTML/boilerplate cleanup -> paragraph split -> line/sentence split -> reference-tail removal -> Persian normalization -> deduplication -> quality scoring -> Good KenLM training corpus ``` Only after this should you train the Good n-gram model. This is consistent with how large open corpora are usually built. For example: - [RedPajama](https://arxiv.org/html/2411.12372v1) discusses preprocessing Wikipedia to remove hyperlinks, comments, and formatting boilerplate. - [FineWeb](https://arxiv.org/abs/2406.17557) emphasizes filtering and deduplication as central parts of dataset construction. - [FinerWeb-10BT](https://arxiv.org/abs/2501.07314) shows that line-level filtering can improve data quality and training efficiency. The practical lesson is: > Filtering is not something you add only at the end. It is part of corpus construction. * * * ## 6. Use line/sentence-level cleaning, not only document-level cleaning Your example is exactly why document-level filtering is not enough. A document may contain: ```plaintext good paragraph good paragraph good paragraph bad reference tail ``` If you only classify the whole document as good/bad, you lose useful text. Instead, use smaller units: | Unit | Use | | --- | --- | | document | broad quality / source metadata | | paragraph | main CPT unit | | line | boilerplate/reference detection | | sentence | fine-grained truncation | | span | remove bad tail after good prose | For Wikipedia-like data, I would do: ```plaintext article -> sections -> paragraphs -> lines/sentences -> score each unit -> remove bad units -> optionally merge clean neighboring units ``` This is especially useful for tails like: ```plaintext ... normal Persian sentence. BBC Persian Abrahamian Modern Iran p۱ ۲ ۲ ... ``` The normal sentence can be kept. The reference tail should be removed. * * * ## 7. Heuristics for reference-tail detection You can start with simple heuristics. Reject or truncate spans with: ```plaintext many isolated numbers too many digits too many parentheses/brackets URL / DOI / ISBN / ISSN patterns English-heavy reference fragments source names glued into Persian prose bibliography-like patterns repeated source names very high punctuation ratio very low Persian-letter ratio abnormal spaced digits ``` Examples of suspicious patterns: ```plaintext ۲ ۸ ۱ ۳ ۵ ۷ ۱ ۳ ۳ ۲ p۱ ۲ ۲ ص ۲ ۸ ۳ BBC Persian Modern Iran ISBN ISSN doi http www ``` For Persian Wikipedia specifically, also watch for section/reference terms, but do not use them too naively. Words like: ```plaintext منابع ارجاع پیوند جستارهای وابسته پانویس کتابشناسی ``` can indicate reference sections, but context matters. For example: ```plaintext منابع طبیعی ایران ``` is normal content, not a reference section. So I would use these words mostly as: ```plaintext section heading / line-level / end-of-article signal ``` not as a global document rejection rule. * * * ## 8. Truncation is often better than rejection For your example, I would probably do something like: ```plaintext Before: . . After: . . ``` A practical rule: ```plaintext If a paragraph starts as good Persian prose but later becomes citation-like, truncate from the first suspicious boundary. ``` Possible boundary signals: ```plaintext sudden English source title sudden bibliography author/title/page pattern many spaced digits multiple source names in a row Persian sentence without punctuation followed by reference fragments ``` This is not perfect, but it is much better than letting the n-gram model learn the garbage. * * * ## 9. Good LM / Bad LM setup Your n-gram idea can still be useful. I would use two n-gram models: ### Good LM Train on: ```plaintext clean Persian prose clean Wikipedia paragraphs curated educational text high-confidence manually accepted examples ``` ### Bad LM Train on: ```plaintext reference tails citation fragments boilerplate broken OCR-like text mixed-language bibliography malformed Wikipedia tails rejected OSCAR chunks ``` Then score candidates with both. A candidate is better if: ```plaintext Good LM likes it Bad LM does not like it ``` Conceptually: ```plaintext score = bad_lm_score - good_lm_score ``` or any similar ratio/difference. Do not overthink the formula at first. The important idea is: > Good LM should model what you want. Bad LM should model what you want to remove. This is better than a single perplexity threshold. * * * ## 10. Persian normalization Before scoring with n-gram models, normalize Persian consistently. Useful tools: - [Hazm](https://www.roshan-ai.ir/hazm/content/hazm/index.html) - [Hazm GitHub](https://github.com/roshan-research/hazm) - [PersianTools](https://pypi.org/project/persiantools/) - [Lucene PersianNormalizer](https://lucene.apache.org/core/9_0_0/analysis/common/org/apache/lucene/analysis/fa/PersianNormalizer.html) Things to normalize: ```plaintext Arabic/Persian ي/ی Arabic/Persian ك/ک heh variants Arabic/Persian digits diacritics extra tatweel/kashida extra spaces weird zero-width characters punctuation spacing half-space / ZWNJ ``` For example, Hazm’s normalizer is useful for standard Persian text normalization, including spacing and ZWNJ-related normalization. * * * ## 11. Do not simply remove all ZWNJ For Persian, ZWNJ is not just random noise. It can be meaningful in words like: ```plaintext کتاب‌ها می‌روم خانه‌ای رفته‌ام ``` So I would not simply delete every zero-width non-joiner. Better: ```plaintext normalize/correct ZWNJ remove weird repeated zero-width characters standardize Unicode form collapse multiple zero-width chars keep valid Persian ZWNJ where appropriate ``` Persian word segmentation and ZWNJ recognition are real NLP problems; see, for example, [Joint Persian Word Segmentation Correction and ZWNJ Recognition](https://arxiv.org/abs/2010.00287). Practical rule: > Normalize ZWNJ; do not blindly remove it. * * * ## 12. Digits: pick a convention Persian corpora often contain mixed: ```plaintext Persian digits: ۱۲۳ Arabic digits: ١٢٣ Latin digits: 123 broken spaced digits: ۱ ۲ ۳ ``` You should choose a convention. For CPT prose, Persian digits may be natural. For JSON, tool calling, math verification, and metadata, Latin digits are often easier. A practical approach: | Context | Suggested convention | | --- | --- | | raw Persian prose CPT | Persian digits are okay | | math internal answer field | Latin digits | | JSON/tool arguments | Latin digits | | final Persian display answer | Persian digits are okay | | metadata | Latin digits | | broken spaced digits | fix if obvious, otherwise reject/truncate | The important thing is consistency. Do not let this happen randomly: ```plaintext ۲ 8 ٣ ۴ 5 ``` unless you intentionally want mixed-digit robustness data. For your example, `۲ ۸ مرداد` should probably become: ```plaintext ۲۸ مرداد ``` and `۱ ۳ ۵ ۷` should become: ```plaintext ۱۳۵۷ ``` if you are confident it is a date/year. But if the number sequence is ambiguous, reject or truncate that span. * * * ## 13. A simple cleaning pipeline for your current case I would implement something like this: ```plaintext 1. Extract text 2. Normalize Unicode 3. Normalize Persian letters 4. Normalize digits 5. Split into paragraphs 6. Split paragraphs into sentences/lines 7. Detect reference-like lines/spans 8. Truncate bad tails 9. Remove very bad paragraphs 10. Deduplicate 11. Train Good LM on clean accepted text 12. Train Bad LM on rejected tails/noise 13. Score new chunks 14. Manually audit samples ``` More concrete: ```plaintext input paragraph -> sentence split -> for each sentence/span: calculate Persian letter ratio calculate digit ratio calculate Latin ratio calculate punctuation ratio detect citation markers detect spaced-digit patterns detect source-name/reference tail -> if bad tail starts after good text: keep text before bad tail -> else if whole paragraph is bad: reject -> else: keep ``` * * * ## 14. Example pseudo-code Very rough pseudo-code: ```python import re PERSIAN_LETTERS = r"آ-ی" def persian_ratio(text): letters = re.findall(f"[{PERSIAN_LETTERS}]", text) chars = [c for c in text if not c.isspace()] return len(letters) / max(1, len(chars)) def digit_ratio(text): digits = re.findall(r"[0-9۰-۹٠-٩]", text) chars = [c for c in text if not c.isspace()] return len(digits) / max(1, len(chars)) def has_spaced_digits(text): # examples like "۱ ۳ ۵ ۷" or "۲ ۸" return bool(re.search(r"[0-9۰-۹٠-٩](\s+[0-9۰-۹٠-٩]){1,}", text)) def looks_reference_like(text): patterns = [ r"http", r"www\.", r"doi", r"ISBN", r"ISSN", r"BBC Persian", r"Modern Iran", r"\bp\s*[0-9۰-۹٠-٩]", r"ص\s*[0-9۰-۹٠-٩]", ] return any(re.search(p, text, flags=re.IGNORECASE) for p in patterns) def is_bad_tail(sentence): if looks_reference_like(sentence): return True if has_spaced_digits(sentence) and digit_ratio(sentence) > 0.08: return True if persian_ratio(sentence) < 0.45: return True return False def truncate_bad_tail(sentences): clean = [] for sent in sentences: if is_bad_tail(sent) and len(clean) > 0: break if not is_bad_tail(sent): clean.append(sent) return " ".join(clean) ``` This is only a starting point. You would need to tune it by looking at accepted/rejected samples. * * * ## 15. Manual audit is still necessary Do not trust the cleaner blindly. After each cleaning rule change, sample: ```plaintext 100 accepted chunks 100 rejected chunks 100 truncated chunks ``` Then inspect: | Sample type | What to check | | --- | --- | | accepted | Did garbage survive? | | rejected | Did good Persian get removed? | | truncated | Did truncation cut at the right place? | | borderline | Should this become a new rule? | This is the same kind of loop you are already doing: ```plaintext clean -> review -> adjust rules -> clean again ``` That loop is the correct approach. * * * ## 16. For /7: how to train the small model for noisy prompts For the 0.8B model, I would not train it on arbitrary noisy prompts. I would train it on controlled noisy prompts. Examples: ### Good robustness SFT ```plaintext User has typo -> assistant still answers. User asks unclear question -> assistant asks clarification. User includes irrelevant sentence -> assistant focuses on main question. User asks two things -> assistant separates them. User provides messy Persian -> assistant normalizes meaning. ``` ### Bad robustness SFT ```plaintext User prompt contains random citation garbage -> assistant imitates it. User prompt has broken reference tail -> assistant treats it as meaningful. User prompt has unrelated source list -> assistant summarizes garbage. ``` For a small model, the best behavior is often: ```plaintext I cannot reliably answer from this messy text. Please provide a clearer sentence. ``` or: ```plaintext The first part is understandable, but the ending looks like broken reference text. ``` That is a valid assistant behavior. The goal is not to make the 0.8B model magically robust to every bad prompt. The goal is to make it fail gracefully. * * * ## 17. Product-level strategy for a 0.8B assistant For a low-end-device assistant, I would design guardrails around the model. ```plaintext raw input -> normalization -> task classifier -> noise detector -> chunk selector -> small model -> output validator ``` If the input is too messy: ```plaintext ask clarification ``` If the input is too long: ```plaintext split/summarize first ``` If it contains reference garbage: ```plaintext remove or warn ``` If it asks for math: ```plaintext use calculator/tool if available ``` If it asks for grammar: ```plaintext route to grammar-tutor prompt template ``` Small models work better when the surrounding system reduces ambiguity. * * * ## 18. Bottom line For the noisy prompt question: > Larger models generally handle long, noisy, multi-constraint prompts better. But a 0.8B model can still be useful if you reduce the prompt burden with preprocessing, task routing, shorter context, templates, and controlled robustness SFT. For the n-gram cleaning question: > Do not let the n-gram model handle structural garbage by itself. Remove obvious reference/citation/wiki-tail noise first. Train the Good LM only on text you want the model to imitate. Optionally train a Bad LM on the garbage you want to detect. The most important rule is: > The n-gram model should be a quality scorer, not a garbage collector. ---
**Author:** ![Bidram](https://avatars.discourse-cdn.com/v4/letter/b/71e660/32.png) [@Bidram](https://discuss.huggingface.co/u/Bidram)\ **Post date:** [June 13, 2026, 4:04pm UTC](https://discuss.huggingface.co/t/how-can-i-build-a-high-quality-dataset/176571/10 "2026-06-13T16:04:01Z")
Hello! I have a question regarding CPT dataset preparation. I recently refined my dataset by removing rare or archaic languages, such as Ancient Persian, because I was concerned that the tokenizer wouldn’t handle them efficiently, which might negatively impact the model’s learning. I also normalized number formats and other technical details. Finally, I chunked the large texts into smaller segments. My general rule was to chunk everything unless it was a specific structural element, like a rule or an email address. However, I’ve run into a context issue. For example, in a long article about Wikipedia, one of the resulting chunks is: _"از همین‌جا بود که تاریخ ویکی شروع شد_ ". If I feed these disconnected fragments into the model, how do I handle incomplete or discontinued text? Specifically, if a short text in the dataset uses a pronoun like _“او”,_ but the article is cut off and doesn’t mention who _“او”_ refers to, how does the model learn to resolve that reference? Does having these “orphaned” references in the dataset harm the training process? ---
**Author:** ![Bidram](https://avatars.discourse-cdn.com/v4/letter/b/71e660/32.png) [@Bidram](https://discuss.huggingface.co/u/Bidram)\ **Post date:** [June 13, 2026, 4:30pm UTC](https://discuss.huggingface.co/t/how-can-i-build-a-high-quality-dataset/176571/11 "2026-06-13T16:30:42Z")
I also have another question. I’m currently in the n-gram cleaning phase, where I’m separating the dataset into “good” and “bad” data for n-gram training (two language models, as you mentioned before). However, I’m not entirely sure how to define good versus bad data. Since the dataset has already been cleaned at the surface level, I assume the distinction should depend more on semantic quality and grammatical correctness than on obvious garbage or noise in the text. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [June 14, 2026, 1:37am UTC](https://discuss.huggingface.co/t/how-can-i-build-a-high-quality-dataset/176571/12 "2026-06-14T01:37:27Z")
Hmm, probably: * * * ## Short answer For the chunking question: > Orphaned references are not catastrophic if they are rare, but they are not ideal high-quality CPT data. If a chunk starts with something like `او`, `این`, `آن`, `همین‌جا`, etc., and the referent is outside the chunk, the model cannot really learn the reference properly from that sample. It just sees an incomplete continuation. For the Good/Bad n-gram question: > Good and Bad should not be defined only as “surface-clean vs surface-dirty.” They should be defined by imitation target. In other words: ```plaintext Good data = text you want the model to imitate. Bad data = text you want the filter to detect and remove. ``` So yes, after surface cleaning, the distinction becomes more about: - semantic completeness - grammatical correctness - natural Persian - self-containedness - target register - whether the text is useful for your model’s target behavior * * * ## 1. About orphaned references in chunks The example: ```plaintext از همین‌جا بود که تاریخ ویکی شروع شد ``` is not necessarily bad Persian. But it is context-dependent. The phrase `همین‌جا` points to something before it. If the previous context is missing, then the chunk is incomplete. Similarly: ```plaintext او سپس به تهران رفت. ``` is grammatical, but if the dataset chunk does not mention who `او` is, the model receives an incomplete training sample. For CPT, this matters because causal language modeling is next-token prediction. The model learns from the previous tokens in the sequence. If the needed left context is missing, the model cannot infer the reference correctly from that sample. Related docs: - [Hugging Face causal language modeling docs](https://huggingface.co/docs/transformers/en/tasks/language_modeling) - [Hugging Face LLM course: training a causal language model](https://huggingface.co/learn/llm-course/chapter7/6) So I would classify these chunks as: | Chunk type | Use for Good CPT? | | --- | --- | | complete paragraph | yes | | complete sentence with enough context | yes | | sentence starting with unresolved pronoun | maybe, but lower quality | | short fragment with missing referent | usually reject or merge | | reference/citation tail | reject or use as Bad data | | title/header/list fragment | maybe separate category, not normal prose | * * * ## 2. This is not a “pronoun problem” only The issue is broader than pronouns. It is about whether the chunk is **self-contained enough**. Problematic context-dependent starts include: ```plaintext او ... آن‌ها ... این موضوع ... در نتیجه ... از همین‌جا ... به همین دلیل ... همان‌طور که گفته شد ... این مسئله ... چنین رویدادی ... وی ... نامبرده ... ``` These are normal Persian expressions, but they often depend on previous context. They are not automatically bad. But if the antecedent or explanation is outside the chunk, the sample is weaker. A useful rule: > A Good CPT chunk does not need to be globally complete, but it should be locally coherent. * * * ## 3. Prefer document-aware chunking over blind fixed-size chunking I would not start by chopping everything into fixed token lengths. Better: ```plaintext article -> section -> paragraph -> sentence -> clean units -> pack units into training sequences ``` Then you can pack clean units up to your target sequence length. This is different from: ```plaintext raw text -> every N tokens -> chunk ``` Blind fixed-length chunking often creates: - broken sentence starts - broken endings - orphaned pronouns - cut-off lists - reference tails - citation garbage - chunks that start mid-thought Research on packing/truncation also points in this direction. For example, [Fewer Truncations Improve Language Modeling](https://arxiv.org/abs/2404.10830) argues that naive concatenation and fixed-length splitting can damage document integrity and that length-aware packing can reduce unnecessary truncation. Also, [In-context Pretraining](https://arxiv.org/abs/2310.10638) discusses the weakness of randomly concatenating short documents and shows that related/coherent document ordering can improve tasks requiring contextual reasoning. You do not need to implement those papers exactly, but the practical lesson is useful: > Preserve coherent units when possible. * * * ## 4. What to do with short incomplete chunks For each short chunk, choose one of four actions. | Case | Action | | --- | --- | | short but complete | keep | | starts with unresolved pronoun | merge with previous paragraph/sentence | | good sentence but missing previous context | keep only if not too frequent, or lower priority | | fragment/citation/list tail | reject | | good prose + bad tail | truncate tail | | too short to be meaningful | reject | Example: ```plaintext او سپس به تهران رفت. ``` If the previous sentence is: ```plaintext علی در سال ۱۳۵۷ از اصفهان خارج شد. ``` then merge: ```plaintext علی در سال ۱۳۵۷ از اصفهان خارج شد. او سپس به تهران رفت. ``` That is much better than training on the orphaned sentence alone. Another example: ```plaintext از همین‌جا بود که تاریخ ویکی شروع شد. ``` If the previous sentence identifies the place/context, merge it. If not, reject or mark it as lower-quality. * * * ## 5. Use EOS or document separators If you pack multiple documents into one training sequence, add a clear separator. Usually that means EOS. Conceptually: ```plaintext document A text document B text document C text ``` This is better than making the model think unrelated documents are one continuous article. If the tokenizer/model has an EOS token, use it consistently. If you are doing CPT with Qwen, follow the tokenizer/model convention rather than inventing a random separator. The goal is: ```plaintext The model should know where one document ends. ``` not: ```plaintext The model should continue one unrelated article as if it were the same text. ``` * * * ## 6. Packing vs padding There is a trade-off. | Method | Benefit | Risk | | --- | --- | --- | | pure padding | preserves document boundaries | inefficient | | naive concatenation | efficient | can mix unrelated contexts | | fixed token chunks | simple | breaks documents/sentences | | paragraph-aware packing | good balance | more preprocessing | | length-aware packing | better integrity | more implementation work | For your case, I would start simple: ```plaintext paragraph-aware packing + EOS between documents ``` Not necessarily complicated optimization. A practical algorithm: ```plaintext 1. split article into paragraphs 2. clean each paragraph 3. reject bad or too-short paragraphs 4. merge neighboring small paragraphs from the same article 5. add EOS at article boundary 6. pack clean units until max_length 7. avoid cutting inside sentence if possible ``` * * * ## 7. How strict should you be? Do not over-clean to the point that you throw away most natural text. But for the Good corpus, be conservative. Suggested rule: ```plaintext If uncertain, do not put it in Good. ``` You can put uncertain data into: ```plaintext maybe/ ``` and inspect later. Possible buckets: ```plaintext good/ bad/ maybe/ too_short/ reference_tail/ needs_previous_context/ ``` This makes the pipeline easier to debug. * * * ## 8. Defining Good and Bad data For the Good/Bad n-gram setup, I would define the labels by imitation target. ### Good data Good data is text that satisfies most of these: - natural Iranian Persian - grammatically correct - semantically coherent - complete enough locally - clear sentence boundaries - clear paragraph boundaries - not citation/reference garbage - not boilerplate - not duplicated - not overly archaic or rare if outside target - not too fragmented - target register matches your goal - something you would be happy for the model to imitate ### Bad data Bad data is text that matches patterns you want to filter: - reference/citation tails - orphan fragments - broken chunks - malformed number sequences - mixed-language bibliography - OCR-like text - repeated boilerplate - random lists/tables - source titles glued into prose - incomplete sentences - nonsense or semantically broken text - grammatically unnatural text - wrong register for your target - archaic/rare language if you intentionally exclude it So yes, after surface cleaning, the distinction becomes more semantic and linguistic. But it is still not only “semantic quality.” It is also: ```plaintext Would I want the final model to produce text like this? ``` * * * ## 9. Good LM / Bad LM The Good/Bad KenLM idea is: ```plaintext Good LM learns the pattern of text you want. Bad LM learns the pattern of text you do not want. ``` Useful reference: - [Rethinking KenLM: Good and Bad Model Ensembles for Efficient Text Quality Filtering in Large Web Corpora](https://arxiv.org/abs/2409.09613) A simple setup: | Model | Train on | | --- | --- | | Good KenLM | clean Persian paragraphs | | Bad KenLM | rejected noisy/fractured/reference-like chunks | Then for a new candidate: ```plaintext Good LM likes it Bad LM dislikes it => probably good ``` If both like it, inspect. If Good LM dislikes it and Bad LM likes it, reject. If both dislike it, it may be out-of-domain or unusual. * * * ## 10. Do not make Good too narrow One warning: if Good data is only formal Wikipedia prose, your Good LM will prefer Wikipedia-like Persian. That may be okay for CPT, but your final assistant may need: - educational Persian - student-friendly Persian - conversational Persian - grammar-tutor Persian - technical Persian - simple explanation style So you may want several Good categories: ```plaintext good_formal/ good_educational/ good_conversational/ good_technical/ ``` Or at least make sure Good is not only one style. Otherwise your filter may reject useful informal/student-like Persian just because it is not Wikipedia-like. * * * ## 11. Do not make Bad too broad Bad should not mean “anything different from Wikipedia.” Bad should mean: ```plaintext text I do not want the model to imitate ``` Examples: ```plaintext bad_reference_tail bad_broken_number bad_ocr bad_mixed_language_garbage bad_boilerplate bad_contextless_fragment bad_duplicate ``` Do not put valid conversational Persian into Bad just because it has a different style. Otherwise the filter may remove exactly the kind of natural user-language you later want. * * * ## 12. Practical labeling guide I would use labels like this. | Label | Meaning | Use | | --- | --- | --- | | `good_formal` | clean formal prose | Good LM | | `good_educational` | clean explanation/tutorial prose | Good LM | | `good_conversational` | natural dialogue-like Persian | maybe Good LM or separate LM | | `bad_reference_tail` | bibliography/citation garbage | Bad LM | | `bad_fragment` | incomplete contextless chunk | Bad LM | | `bad_boilerplate` | repeated non-content | Bad LM | | `bad_mixed_language` | accidental mixed-language garbage | Bad LM | | `bad_number_noise` | malformed number/date fragments | Bad LM | | `maybe` | uncertain | manual review | This is better than just: ```plaintext good bad ``` because later you can inspect which kind of bad data is leaking through. * * * ## 13. Specific rule for orphaned pronouns For chunks with pronouns or context markers: ```plaintext او وی آن‌ها این آن همین چنین نامبرده ``` I would use a simple rule: ```plaintext If the chunk is short and starts with one of these, try to merge with previous sentence/paragraph. If no previous context exists, mark as context-dependent. If context-dependent and short, reject from Good. ``` Not every pronoun is bad. For example: ```plaintext علی دانش‌آموز کلاس هفتم است. او به ریاضی علاقه دارد. ``` This is good. But: ```plaintext او به ریاضی علاقه دارد. ``` alone is weak. * * * ## 14. Example decision table | Example | Decision | | --- | --- | | `علی دانش‌آموز کلاس هفتم است. او به ریاضی علاقه دارد.` | Good | | `او به ریاضی علاقه دارد.` | Maybe/reject unless merged | | `از همین‌جا بود که تاریخ ویکی شروع شد.` | Maybe; needs previous context | | `۲ ۸ مرداد ۱ ۳ ۳ ۲ BBC Persian Modern Iran p۱ ۲ ۲` | Bad | | `منابع طبیعی ایران بسیار متنوع است.` | Good | | `منابع: BBC Persian, Abrahamian, Modern Iran...` | Bad reference tail | | `این موضوع در بخش قبل توضیح داده شد.` | Context-dependent; maybe reject | | `در این درس، مفهوم فاعل را با مثال توضیح می‌دهیم.` | Good educational | * * * ## 15. A practical pipeline I would do something like: ```plaintext raw article -> normalize -> split into sections -> split into paragraphs -> split into sentences -> remove reference sections -> remove/truncate citation tails -> detect context-dependent starts -> merge with previous paragraph if needed -> reject very short orphan fragments -> assign good/bad/maybe labels -> train Good/Bad n-gram models ``` For chunk construction: ```plaintext clean paragraphs -> merge small adjacent paragraphs from same article -> add EOS at article boundary -> pack to max_length -> avoid cutting inside sentence when possible ``` * * * ## 16. Suggested thresholds These are only starting points. | Rule | Possible starting point | | --- | --- | | too short | fewer than 20-30 Persian tokens | | too digit-heavy | digit ratio above 15-20% | | too Latin-heavy | Latin ratio above 20-30%, unless technical text | | orphan start | starts with pronoun/context marker and very short | | low Persian ratio | reject unless intended mixed text | | too repetitive | repeated line or repeated n-gram | | suspicious tail | truncate after reference-like pattern | Do not trust the thresholds blindly. Sample accepted/rejected data and adjust. * * * ## 17. How much orphaned data is harmful? If rare, probably not a disaster. Large corpora always contain some imperfect examples. But if many chunks are orphaned fragments, the model may learn: - incomplete starts - weak local coherence - continuation from missing context - strange discourse transitions - bad paragraph openings For a small 0.8B model, I would be stricter than for a huge model. Small models have less capacity to average away noise. So: ```plaintext For Good CPT corpus: be conservative. For broad noisy corpus: maybe keep some. For final high-quality Persian corpus: reject or merge orphan fragments. ``` * * * ## 18. Bottom line For /10: > Orphaned pronouns and context-dependent chunks are not catastrophic if rare, but they are not high-quality CPT examples. Prefer article/section/paragraph-aware chunking, merge short context-dependent chunks with previous context, use EOS between documents, and reject very short orphan fragments. For /11: > Good/Bad data should be defined by imitation target, not just surface cleanliness. Good = text you want the model to imitate. Bad = text you want your filter to detect and remove. The simplest rule is: ```plaintext Good data should be locally coherent Persian prose. Bad data should represent the patterns you want to eliminate. ``` ---
**Author:** ![Bidram](https://avatars.discourse-cdn.com/v4/letter/b/71e660/32.png) [@Bidram](https://discuss.huggingface.co/u/Bidram)\ **Post date:** [June 14, 2026, 10:53am UTC](https://discuss.huggingface.co/t/how-can-i-build-a-high-quality-dataset/176571/13 "2026-06-14T10:53:24Z")
I completely understand what you’re saying, and thank you for the detailed explanation. However, there’s still one point I’m unsure about. Do n‑gram models perform better when the input text is shorter? If that’s the case, then training the model on very long articles might introduce certain issues. If not, then I believe the problem is largely resolved. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [June 14, 2026, 10:32pm UTC](https://discuss.huggingface.co/t/how-can-i-build-a-high-quality-dataset/176571/14 "2026-06-14T22:32:31Z")
Hmm, probably: * * * No, I would not say that n-gram models automatically perform better just because the input text is shorter. A 5-gram model is local in a different sense: it mainly uses the previous 4 tokens/words when estimating the next one. So it can score a long article, but the score is still built from many local n-gram probabilities. Also, perplexity is normally length-normalized. For example, [Jurafsky and Martin’s n-gram chapter](https://web.stanford.edu/~jurafsky/slp3/3.pdf) describes perplexity as a normalized version of the probability of the test set. So a long text is not automatically invalid just because it is long. The real issue is not: ```plaintext long text is bad for n-gram models ``` The real issue is: ```plaintext whole-document averaging can hide local garbage ``` For example, if an article is: ```plaintext good paragraph good paragraph good paragraph bad reference tail ``` then the whole-article average perplexity may still look acceptable because the good paragraphs dominate the score. So I would separate **training** and **filtering** : | Step | Unit I would use | | --- | --- | | Train Good KenLM | clean paragraphs or clean articles are both okay | | Train Bad KenLM | bad tails, broken references, noisy chunks | | Score/filter candidates | paragraph-level or sliding-window level | | Detect local garbage | sentence/window level | | Final CPT packing | merge clean paragraphs and add EOS/document separators | So yes: if the long articles are already clean, training KenLM on them is fine. But for filtering, I would avoid relying only on a single whole-article score. I would score smaller units too: ```plaintext article score paragraph scores sliding-window scores ``` A practical rule: ```plaintext Use long clean text for training. Use local scores for filtering. ``` This also fits the Good/Bad KenLM idea. The [Rethinking KenLM](https://arxiv.org/abs/2409.09613) paper uses a Good KenLM trained on high-quality data and a Bad KenLM trained on low-quality data. That does not require all input texts to be short; the important part is that the Good model learns the pattern of text you want, and the Bad model learns the pattern of text you want to remove. So my short version would be: > Long clean text is fine for training KenLM. Whole-document averaging is the danger. For filtering Persian CPT data, use paragraph-level or sliding-window scoring so local reference garbage does not get hidden by good surrounding text. ---
**Author:** ![Salpen](https://avatars.discourse-cdn.com/v4/letter/s/c6cbf5/32.png) [@Salpen](https://discuss.huggingface.co/u/Salpen)\ **Post date:** [June 18, 2026, 3:17am UTC](https://discuss.huggingface.co/t/how-can-i-build-a-high-quality-dataset/176571/15 "2026-06-18T03:17:40Z")
John6666’s pipeline advice is excellent — I’ll add a different layer: we run a fleet of small qwen-based agents in production (7B–32B, long-lived sessions, tool calling), and several failure modes we learned the hard way are directly relevant to what you’re building. Dataset design can’t fix all of them — knowing which ones to solve elsewhere will save you months. **1. Don’t spend dataset budget teaching what decoding can enforce.** Tool calling on small qwens drifts: the model emits JSON-as-prose instead of actual structured calls, intermittently, no matter how many clean examples it saw in SFT. The fix that actually held for us was enforcement at the sampling layer — constrained generation / grammar-based decoding (llama.cpp grammars, Ollama’s format=schema, outlines). Put a _modest_ amount of tool-calling data in SFT for format familiarity, then enforce structure at inference. A 0.8B will never be reliable on this from data alone. **2. Language drift fires exactly at uncertainty — train for those moments.** Small qwens flip into Chinese or English mid-sentence precisely around hedge phrases and confusion (“hmm, let me think…”). For a Persian-only assistant this is your biggest UX risk, because qwen’s pretraining gravity is zh/en. Two-part fix: include SFT examples where the model is _uncertain, confused, or correcting itself entirely in Persian_ (those are the states where drift triggers), and pin an explicit language anchor in the deployed system prompt. Test specifically: ambiguous questions, misspelled student input, out-of-domain asks — drift hides there, not in clean QA. **3. Persian-specific corpus traps worth checking before any filtering** (verify these against your data — they’re the classic Persian NLP issues): ZWNJ (U+200C) is grammatically meaningful (می‌روم vs می روم) and web scrapes routinely strip or mangle it — and because the damage is _uniform_, your n-gram filter won’t flag it; you’d be teaching broken orthography fluently. Also normalize Arabic-vs-Persian codepoints (ي→ی, ك→ک) — OSCAR is notoriously mixed. Libraries like hazm/parsivar handle both. One hour of character-level spot-checking raw text by eye is worth more than any perplexity filter; we learned this from a corrupted-extraction incident where the model produced fluent, confident, completely wrong output — garbage substrate doesn’t look like garbage after training, it looks like confidence. **4. For the grammar-tutor data: be careful with negative examples.** Small models pattern-match surface text; a dataset rich in wrong-Persian labeled “incorrect” can prime the very errors it warns against. Prefer correct-demonstration-heavy data, with error-correction examples in a rigid, clearly-fenced format (student error → correction → one-line rule), sparse rather than abundant. **5. Design for short sessions.** Small qwens degrade hard past ~8–10 turns/tool-calls — fabrication and loops cascade. For a student assistant that’s fine _if_ the data teaches it: self-contained exchanges, summarize-and- reset patterns, not long winding dialogues the deployed model can’t sustain. Your CPT-first instinct already proved itself (the Wikipedia improvement) — these five are about not losing that gain at the edges. Good luck — a Persian tutor on low-end devices is exactly the kind of thing small models _should_ be doing. ---
**Author:** ![Bidram](https://avatars.discourse-cdn.com/v4/letter/b/71e660/32.png) [@Bidram](https://discuss.huggingface.co/u/Bidram)\ **Post date:** [June 18, 2026, 6:20pm UTC](https://discuss.huggingface.co/t/how-can-i-build-a-high-quality-dataset/176571/16 "2026-06-18T18:20:42Z")
Hello! Thanks for the detailed help, and sorry for the late reply, I was sick. I ran a small 1 epoch training experiment, and the results were clearly better. The model learned Persian more effectively, and the outputs showed noticeable improvement. You also mentioned that small Qwen models tend to collapse in long multi-turn conversations and sometimes switch languages when they are uncertain. Yes, that matches my own testing exactly. I had seen those issues myself, but I had forgotten how important they are in practice, so thank you for pointing them out again. So far, I have not found a better model in this size range with a tokenizer that handles Persian reasonably well, at least based on my own tests. Since you mentioned that you have worked extensively with Qwen-based models, I would be very interested to know if you have found a better alternative that does not struggle as much with these Qwen-specific issues. I am currently using Qwen 3.5 as the base model. In Qwen’s blog, they mention that the newer 3.5 series has a lower chance of language mixing, although I am not sure how true that is in practice. My main goal is not to preserve English and Chinese and simply add Persian on top, but to build a Persian only assistant that stays reliably in Persian. In your view, does this require starting from a model that already has relatively weak language mixing behavior, or can this issue be reduced enough through continued pretraining and fine-tuning? ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [June 20, 2026, 1:52am UTC](https://discuss.huggingface.co/t/how-can-i-build-a-high-quality-dataset/176571/17 "2026-06-20T01:52:22Z")
Hmm… for small models, maybe we should not expect clean multilingual separation in the output. It **may simply be a capacity issue**. More specifically: * * * ## Short answer I would not treat this as only a Qwen-specific bug. I would think of it as a **capacity / interference problem**. In a very small multilingual model, many things compete for limited capacity: - Persian - English - Chinese - generic assistant style - refusal style - reasoning style - formatting rules - tool-use behavior - uncertainty behavior - instruction-following constraints When the model is uncertain, overloaded, sampled too freely, or pushed outside its strongest distribution, it may fall back to a stronger learned mode: English, Chinese, boilerplate, generic assistant style, or some other pattern. So for a **Persian-only 0.8B assistant** , I would not expect multilingual separation to be naturally stable. I would treat language drift as a failure mode that must be actively engineered against. * * * ## 1. Rough size intuition This is not a hard rule, but my rough expectation would be: | Model size | Language/style drift expectation | | --- | --- | | **below 1B** | Very likely. Needs strong constraints, narrow scope, and drift-specific eval. | | **1B-3B** | Still common, but can be made usable with good CPT/SFT and conservative decoding. | | **3B-7B** | Transition zone. Often much better, but still fragile under uncertainty, long prompts, or high temperature. | | **7B-14B** | Usually much more stable for multilingual instruction following. | | **14B+** | Much more reliable, though language drift can still happen. | So for **Qwen3.5-0.8B** , I would put it clearly in the “drift likely” zone. Not hopeless, but not something I would expect to disappear automatically. * * * ## 2. Why this happens I would describe it like this: > The smaller the model, the less room it has to keep languages, styles, tasks, and constraints cleanly separated. This is related to what multilingual NLP papers often call the **curse of multilinguality** : multilingual training can help low-resource languages, but many languages and tasks also compete for limited model capacity. See, for example, [When Is Multilinguality a Curse?](https://aclanthology.org/2024.emnlp-main.236/) and [Multilingual Large Language Models and Curse of Multilinguality](https://arxiv.org/abs/2406.10602). There is also direct work on **language confusion** , where LLMs fail to answer consistently in the user’s intended language. [Understanding and Mitigating Language Confusion in LLMs](https://aclanthology.org/2024.emnlp-main.380/) reports that language confusion can be worsened by complex prompts and high sampling temperature, and can be partially reduced with few-shot prompting, multilingual SFT, and preference tuning. So I would not frame this as: ```plaintext Qwen randomly breaks. ``` I would frame it as: ```plaintext A small multilingual model has limited capacity, and language/style/task modes interfere. ``` * * * ## 3. SFT alone may not fully solve it Persian SFT will help. But ordinary SFT mostly increases the probability of the desired answer tokens. It does not necessarily strongly penalize unwanted mixed-language outputs. That matters because a mixed-language answer may still share many locally plausible tokens with the training distribution. There is recent work, [Controlling Language Confusion in Multilingual LLMs](https://arxiv.org/abs/2505.19116), arguing that normal SFT may not explicitly punish cross-lingual mixing, while preference-style objectives such as ORPO can suppress language-confused outputs more directly. Practical implication: ```plaintext Persian-only SFT helps. But if English/Chinese drift is a specific failure mode, include examples where mixed-language answers are explicitly bad. ``` For example: | Prompt | Bad answer | Good answer | | --- | --- | --- | | Persian user asks a question | starts Persian, then switches to English/Chinese | stays in Iranian Persian | | Persian user asks uncertain question | “I’m not sure…” in English | Persian clarification / Persian uncertainty | | Persian math prompt | reasoning in English/Chinese | Persian explanation, or tool call + Persian final answer | | Persian correction prompt | generic English assistant tone | Persian correction style | This is where preference data, DPO/ORPO-style data, or even simple reject/accept filtering can help. * * * ## 4. What I would do for a Persian-only 0.8B model I would attack the problem at several layers. | Layer | Action | | --- | --- | | model choice | compare same-size models for Persian tokenization and drift | | CPT | Persian-heavy or Persian-only CPT corpus | | SFT | Persian-only assistant examples | | uncertainty SFT | “I don’t know / please clarify / I cannot answer” all in Persian | | correction SFT | user correction and self-correction in Persian | | negative/preference data | mixed-language outputs marked as bad | | decoding | low temperature, avoid aggressive presence penalty | | output check | detect non-Persian output and retry/repair | | eval | explicit language-drift test set | For a Persian-only assistant, I would not try to preserve broad multilingual behavior unless you actually need it. If the product goal is Iranian Persian, then English/Chinese drift should be treated as an error. * * * ## 5. Drift eval is necessary I would make a small eval set specifically for drift. Not just normal Persian QA. Test the cases where drift is likely: | Case | Why | | --- | --- | | ambiguous Persian prompt | model may fall back to dominant language | | misspelled Persian | uncertainty increases drift | | long multi-turn chat | context pressure increases drift | | correction from user | model may switch style/language | | “I don’t know” case | uncertainty often triggers English boilerplate | | math/reasoning | reasoning style may drift to English | | tool failure | failure mode may drift | | mixed Persian-English technical terms | model may continue in English | | high temperature | sampling can increase drift | | long answer | later paragraphs may drift | Measure something simple: ```plaintext non-Persian character ratio English token ratio Chinese character count answer starts in Persian? answer ends in Persian? does reasoning drift? does refusal drift? ``` Even a small 100-example drift eval would be useful. * * * ## 6. If another model is better, use it If you find another 0.5B-3B model with: - better Persian tokenization - less English/Chinese drift - acceptable instruction following - acceptable device performance then yes, use it. Starting from a less drift-prone model is always better. But I would not abandon Qwen3.5-0.8B only because drift exists. At 0.8B, some drift risk is expected. I would compare models with a small test: ```plaintext same Persian prompts same decoding settings same drift eval same tokenization statistics same latency/memory budget ``` Then choose based on evidence. * * * ## 7. Practical expectation My expectation would be: ```plaintext Qwen3.5-0.8B + Persian CPT + Persian SFT -> likely much better Persian -> likely reduced drift -> not guaranteed drift-free ``` To get closer to Persian-only behavior, add: ```plaintext Persian uncertainty examples Persian correction examples Persian refusal examples Persian tool-failure examples negative examples for English/Chinese drift low-temperature decoding output language check ``` For example: ```plaintext If output contains too much English/Chinese: retry with stronger Persian-only instruction or run a repair prompt or reject the answer ``` This is not elegant, but for a very small local assistant it is practical. * * * ## Bottom line For a Persian-only 0.8B assistant, I would assume: ```plaintext language drift is normal unless actively controlled ``` The reason is probably not only tokenizer or Qwen behavior. It is also limited model capacity and interference between languages, styles, and tasks. So the strategy should be: ```plaintext 1. choose the least-drifty model you can 2. do Persian-focused CPT 3. do Persian-only SFT 4. train uncertainty/correction/failure cases in Persian 5. penalize mixed-language outputs if possible 6. use conservative decoding 7. add output language checks 8. measure drift directly ``` In short: > If another same-size model has lower drift and acceptable Persian tokenization, use it. But if Qwen3.5-0.8B is still the best tradeoff, I would keep it and treat language drift as a first-class eval and training target. ---
**Author:** ![Bidram](https://avatars.discourse-cdn.com/v4/letter/b/71e660/32.png) [@Bidram](https://discuss.huggingface.co/u/Bidram)\ **Post date:** [June 20, 2026, 9:27am UTC](https://discuss.huggingface.co/t/how-can-i-build-a-high-quality-dataset/176571/18 "2026-06-20T09:27:56Z")
Oh! That makes sense. But if my CPT or SFT contains non-Persian languages in between, the model could drift there and not continue with Persian. For example: > User input: سلام، اسم من Mohammad است. > > Assistant output: سلام Mohammad، چطور می‌تونم کمکت کنم؟ At “Mohammad,” the model could drift easily into English, right? I should say that I am not aiming to fine-tune the SLM for heavy multilingual tasks. I am targeting a Persian-only assistant for student and teacher help, with simple tool calls (RAG, for example). I do not expect a perfect assistant with a 0.8B-sized SLM and limited high-quality data. But for the festival, I want to get acceptable results. * * * About the N-gram cleaning process idea, I tried to train two LMs as you said. But they mostly failed because of the data. The issue is that my data looks clean and has acceptable grammar, but the quality is shallow. I do not have any idea how to achieve high-quality data. By high-quality, I mean data that is rich in information, with clean grammar and syntax. * * * For math reasoning, we cannot expect a 0.8B model to solve hard mathematics questions. But for student and teacher help, it can be useful. I came up with the idea of using a fixed format for math reasoning, and because students have math questions in their books, the data can be created like this: > User input: Hello, can you help me with this math question? We have a triangle, A is its …, B is its … and C is its …, if A is … and B is …, what is C? > > Assistant output: So, I need to solve the math question. A is …, B is … and C is …, that can be solved using …, e.g. … C is … And to prevent the model from memorizing patterns, we create multiple formats for each task. For example, as I have seen, Qwen models’ thinking is a related pattern, not exactly the same, but related. Maybe I can do that too. * * * You said that SLMs drift when they are uncertain, overloaded, or pushed outside their strongest distribution. I believe that, because of my data limitations, the model will absolutely drift outside the target language. Is there any way to prevent this? Can we make the model more confident? Or teach it a habit so that it keeps telling itself not to drift, even when it is unsure? But as Qwen says, the fixed their model issue using Qwen-Scope that is meant to prevent drift in the model output. You can see the blog here : [Qwen-Scope](https://qwen.ai/blog?id=qwen-scope) As i tested, Qwen 3.5 0.8B is very capable! It does not have the past issues. If that is true, i think this issue is fully done. * * * From my tests, Qwen 3.5 0.8B had the best tokenizer for Persian, but it was still not very acceptable. I wanted to know: can I extend the tokenizer? I saw that PersianPhi did a tokenizer extension. As I am still limited by hardware limitations, I think doing it is fully worth it. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [June 20, 2026, 12:04pm UTC](https://discuss.huggingface.co/t/how-can-i-build-a-high-quality-dataset/176571/19 "2026-06-20T12:04:11Z")
Hmm… if you stay on the 0.8B route, **the fine-tuning and engineering cost may actually end up higher than just moving to a slightly larger model, maybe 2B–9B**. More details below: * * * I would frame this as a **model-size / evaluation / data-quality decision** , not only as a “can Qwen3.5-0.8B learn Persian?” decision. My short answer is: > **Qwen3.5-0.8B may be good enough for a narrow Persian-first festival demo, but I would not invest heavily in fine-tuning it before comparing it against Qwen3.5-2B and Qwen3.5-4B on a small Persian stress eval.** The reason is that 0.8B is not just “a bit smaller” than 2B or 4B. For a multilingual assistant, 0.8B may be close to the minimum useful capacity. Once you require Persian fluency, instruction following, math tutoring, tool JSON, Latin-name handling, uncertainty behavior, and low language drift, the small model can become expensive in engineering time. So I would use this decision rule: | Route | When it makes sense | Main risk | | --- | --- | --- | | **Qwen3.5-0.8B** | strict lightweight demo, narrow assistant, fast/cheap inference | may need more SFT/DPO/guards/CPT to compensate | | **Qwen3.5-2B** | first “extra capacity” candidate if 0.8B feels brittle | still small, but much less squeezed | | **Qwen3.5-4B** | likely best total-engineering-cost route if quality matters | higher inference cost | | **Qwen3.5-9B** | quality demo / stronger tutor behavior | may no longer feel like a tiny SLM project | Useful references: - [Qwen3.5-0.8B model card](https://huggingface.co/Qwen/Qwen3.5-0.8B) - [Qwen3.5 model collection](https://huggingface.co/collections/Qwen/qwen35) - [Unsloth Qwen3.5 guide](https://unsloth.ai/docs/models/qwen3.5) - [Unsloth Qwen3.5 fine-tuning guide](https://unsloth.ai/docs/models/qwen3.5/fine-tune) ## 1. My recommended first step: build the eval before training Before CPT, tokenizer extension, or a large SFT run, I would build a small Persian stress eval and run the same prompts on: - Qwen3.5-0.8B - Qwen3.5-2B - Qwen3.5-4B - optionally Qwen3.5-9B - optionally one non-Qwen small baseline The goal is not to create a perfect academic benchmark. The goal is to answer: 1. Is 0.8B already acceptable? 2. Does 2B fix most of the failures? 3. Does 4B reduce engineering work enough to justify the extra inference cost? 4. Are the failures mainly Persian fluency, language drift, math quality, JSON breakage, or shallow answers? 5. Should you spend effort on SFT, DPO/ORPO, CPT, tokenizer extension, or simply a larger base model? A first eval can be only 500–700 examples. | Eval bucket | Size | What it tests | | --- | --- | --- | | Persian general QA | 100 | basic Persian answer quality | | PersianMMLU / Khayyam sample | 100 | school knowledge and reasoning | | Latin-name mixed prompts | 50 | `Mohammad`-style name handling | | English technical term prompts | 50 | controlled code-switching | | Math tutor prompts | 50–100 | step-by-step Persian tutoring | | Tool JSON prompts | 50 | valid JSON and key preservation | | Grammar correction | 50 | Persian language tutoring | | Long-answer drift | 30–50 | whether it switches language halfway | | Uncertainty prompts | 30–50 | whether it says “I do not know” properly | | Safety / social norm prompts | 30–50 | basic public assistant behavior | Some useful Persian benchmark starting points: - [Khayyam Challenge / PersianMMLU](https://arxiv.org/abs/2404.06644) - [Open Persian LLM Leaderboard](https://huggingface.co/spaces/PartAI/open-persian-llm-leaderboard) - [PARSE: Persian open-domain reasoning QA](https://arxiv.org/abs/2602.01246) - [ELAB: Persian alignment benchmark](https://aclanthology.org/2025.gem-1.40/) - [PersLitEval](https://arxiv.org/html/2605.27015) > **Suggested scoring setup** > > I would separate deterministic checks from judge-based checks. > > | Property | Scoring method | > | --- | --- | > | multiple choice | exact match | > | final math answer | exact / normalized / symbolic check | > | JSON tool output | JSON parser + schema validator | > | Latin name preservation | string / normalized match | > | output language | Persian language ID | > | long-answer drift | sentence-level language ID | > | formula preservation | regex / string checks | > | shallow answer | rubric-based LLM judge + human spot check | > | tutor quality | rubric-based LLM judge + small human gold set | > | uncertainty quality | rubric judge + manual review | > | safety / norms | Persian benchmark + spot check | > > For language drift, do not use a simple “foreign text exists = fail” rule. That would wrongly penalize names, formulas, code, JSON, URLs, citations, and technical terms. > > Use a distinction like: > > | Span type | Should be allowed? | > | --- | --- | > | Latin personal name | yes | > | URL / citation | yes | > | JSON key | yes, if schema requires it | > | formula | yes | > | short English technical term | usually yes | > | full English sentence where Persian was expected | no | > | Chinese/English continuation after Persian prompt | no | > | translated JSON key that breaks schema | no | > > For LLM-as-judge scoring, I would keep a small human-labeled Persian validation set. LLM judges are useful, but they are not automatically reliable across languages. > > Reference: > > - [A Survey on LLM-as-a-Judge](https://arxiv.org/abs/2412.05579) ## 2. About Latin names like `Mohammad` I would not remove Latin names. They are not bad data by themselves. The real target is: > **Preserve Latin names, formulas, URLs, code, and JSON when needed, but keep the surrounding answer Persian.** For example: | Input pattern | Bad behavior | Desired behavior | | --- | --- | --- | | Persian + Latin name | model switches to English | keep the name, answer in Persian | | Persian + English term | entire answer becomes English | preserve term, explain in Persian | | Persian + formula | formula gets mistranslated | preserve formula, explain in Persian | | Persian + JSON schema | keys get translated or JSON breaks | keep valid JSON, explain in Persian outside JSON | | Persian + URL/citation | answer drifts into English | keep URL/citation, answer in Persian | So instead of deleting mixed examples, I would create controlled mixed examples. Example behavior policy: > “Answer in Persian. Preserve names, formulas, code, JSON keys, URLs, and citations exactly when needed. Do not switch the surrounding explanation into English or Chinese.” This is also where Qwen-Scope is conceptually relevant. Qwen-Scope uses sparse autoencoders to analyze and steer Qwen-family model internals, and it discusses development uses around behaviors such as code-switching and repetition. Useful references: - [Qwen-Scope](https://arxiv.org/abs/2605.11887) - [SASFT: SAE-guided supervised fine-tuning for unexpected code-switching](https://arxiv.org/abs/2507.14894) - [Controlling Language Confusion in Multilingual LLMs](https://arxiv.org/abs/2505.19116) - [OLA: Learning to respond in the user’s language](https://arxiv.org/abs/2601.03589) But I would not depend on Qwen-Scope as the main practical solution for a festival pipeline. For you, the simpler stack is probably: 1. SFT examples showing correct Persian behavior around Latin spans 2. rejected examples where the answer drifts into English/Chinese 3. small DPO/ORPO if drift persists 4. sentence-level language checks in eval 5. optional runtime retry if the output language is wrong ## 3. KenLM is useful, but not enough for “high-quality” answers Your concern about shallow but fluent answers is correct. A KenLM-style Good/Bad filter can help with: - noisy text - strange character distribution - broken Persian - low-fluency text - bad OCR-ish text - obvious junk But it will not reliably detect: - shallow explanations - generic answers - low educational value - missing reasoning steps - factual weakness - confident but incomplete answers For this, I would copy the idea behind FineWeb-Edu: use a separate educational-quality signal, not only a fluency signal. Useful references: - [FineWeb-Edu dataset](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) - [FineWeb-Edu classifier](https://huggingface.co/HuggingFaceFW/fineweb-edu-classifier) - [FineWeb paper](https://arxiv.org/abs/2406.17557) - [FineWeb blog post](https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1) A practical Persian filtering stack could look like: 1. language ID 2. exact / near deduplication 3. rule filters for broken text 4. KenLM / perplexity filtering 5. educational-quality classifier or LLM judge 6. small human audit 7. only then use the text for CPT or raw-text-to-SFT generation > **Example quality rubric for Persian educational data** > > A simple 0–5 rubric: > > | Score | Meaning | > | --- | --- | > | 0 | junk, wrong language, unreadable, spam | > | 1 | Persian but useless / broken / mostly boilerplate | > | 2 | fluent but shallow, vague, or low information | > | 3 | useful but ordinary explanation | > | 4 | clear, educational, specific, well-structured | > | 5 | excellent teaching-quality content with examples, reasoning, and accurate details | > > For your project, I would not keep only score-5 text. That may be too strict. A practical rule might be: > > - use 4–5 for high-quality SFT source material > - use 3–5 for broader CPT text > - reject 0–2 > - manually inspect borderline cases ## 4. Math tutor: fixed format is good, but keep the target narrow For 0.8B, I would not target “general math solver”. I would target: > **Persian step-by-step tutor for known school-level problem types.** That is a much more realistic goal. Good tutor data should include: | Field | Purpose | | --- | --- | | problem | original Persian problem | | level | grade / difficulty | | topic | arithmetic, algebra, geometry, etc. | | student\_attempt | optional wrong solution | | diagnosis | what is wrong or missing | | hint | small nudge | | solution\_steps | short Persian steps | | final\_answer | normalized answer | | checks | how to verify | | language\_policy | answer in Persian; preserve formulas | I would evaluate not only the final answer, but also: - step correctness - first-error detection - usefulness of hint - whether the model over-solves when the student only needs a hint - language consistency - formula preservation Useful reference: - [Step-by-Step: Improving Math Reasoning and Tutoring with Process Supervision](https://arxiv.org/abs/2407.09136) My practical recommendation: | Model size | Math tutor scope | | --- | --- | | 0.8B | narrow, fixed format, school-level, many templates | | 2B | still controlled, but more robust explanations | | 4B | better candidate for richer tutoring | | 9B | stronger quality demo if inference cost is acceptable | ## 5. Self-reminders help, but they are not enough A system prompt like “always answer in Persian” helps, but I would not rely on it alone. Better stack: 1. **SFT habit** Many examples where mixed input still produces Persian output. 2. **Preference data** Chosen answer = Persian answer with allowed spans preserved. Rejected answer = English/Chinese drift, broken JSON, translated schema keys, shallow answer, or hallucination. 3. **DPO/ORPO** Use preference tuning if SFT does not suppress drift enough. 4. **Eval** Track wrong-language rate and sentence-level drift. 5. **Runtime guard** If needed, detect wrong-language output and retry. Useful references: - [Direct Preference Optimization](https://arxiv.org/abs/2305.18290) - [Controlling Language Confusion in Multilingual LLMs](https://arxiv.org/abs/2505.19116) - [OLA: Learning to respond in the user’s language](https://arxiv.org/abs/2601.03589) > **Example chosen/rejected pairs for language drift** > > | Prompt type | Chosen output | Rejected output | > | --- | --- | --- | > | Persian + `Mohammad` | Persian answer, name preserved | English answer | > | Persian + English term | Persian explanation, term preserved | whole answer in English | > | Persian + JSON schema | valid JSON, keys preserved | translated keys or invalid JSON | > | Persian math prompt | Persian steps, formulas intact | switches language mid-solution | > | Persian grammar prompt | Persian correction and explanation | Chinese/English drift | > | uncertainty prompt | Persian “I’m not sure…” | confident hallucination | > > The key is: > > > Do not only teach the model what good Persian answers look like. Also show it what wrong-language failures look like and make those outputs lose. ## 6. Tokenizer extension: possible, but probably not first Tokenizer extension can help if Persian is tokenized badly. But it is not a free improvement. If you add tokens, you need: - embedding resize - sensible initialization - warm-up / alignment - continued training - regression tests A good Persian-specific reference is PersianPhi: - [PersianPhi model card](https://huggingface.co/amirakhlaghiqqq/PersianPhi) - [Persian-Phi paper](https://arxiv.org/abs/2512.07454) PersianPhi is useful because it shows that tokenizer adaptation can be part of a serious Persian curriculum pipeline. But that is also the warning: it is not just “add Persian tokens and run SFT”. For your project, I would measure first: | Measurement | Why it matters | | --- | --- | | tokens per word | Persian compression | | characters per token | context efficiency | | split rate | whether words are fragmented | | Persian vs English token cost | multilingual imbalance | | Latin-name mixed examples | real input behavior | | formulas / JSON | tool and math safety | | textbook text | tutor data efficiency | If Qwen3.5 tokenization is acceptable, I would skip tokenizer extension and spend the time on eval, SFT quality, and drift control. ## 7. Suggested training strategy I would use this order. ### Stage 0 — Compare base models before training Run the same eval on 0.8B, 2B, and 4B. Decision: - if 0.8B passes, keep it - if 0.8B fails only in language discipline, try SFT + DPO/ORPO - if 0.8B fails in tutor quality, tool stability, or long-answer drift, test 2B/4B before heavy training - if 4B works with much less engineering, use 4B ### Stage 1 — SFT for Persian-first behavior Start with high-quality examples, not a huge weak dataset. Possible first target: - 5k–30k strong Persian SFT examples - Persian answers - tutor examples - grammar correction - uncertainty behavior - Latin-name handling - formula handling - JSON/tool handling ### Stage 2 — Preference tuning for drift and bad behavior Only if SFT is not enough. Use chosen/rejected pairs for: - wrong-language drift - translated JSON keys - broken formulas - shallow answers - overconfident hallucinations ### Stage 3 — CPT only if eval says you need it If Persian fluency or domain knowledge is still weak, then consider small CPT. But I would not start with huge CPT unless the goal is truly “build a Persian language model”, not “build a Persian assistant demo”. ### Stage 4 — Tokenizer extension only if measurement justifies it Tokenizer extension is a serious intervention. It belongs after measurement, not before. ## Short answers to your questions ### Will Latin names cause drift? They can trigger drift, but they are not bad data. Keep them. Train and evaluate the model to preserve names while keeping the surrounding answer Persian. ### Can Good/Bad KenLM catch shallow answers? No, not reliably. KenLM helps with surface fluency and noise. Use an educational-quality classifier or rubric-based judge for depth and usefulness. ### Is fixed-format math tutor data a good idea? Yes. For 0.8B, fixed and narrow is better. Treat it as a guided tutor, not a general math solver. ### Can the model learn confidence and self-reminders? Partly, but self-reminders are not enough. Use SFT for habit, preference tuning for wrong-language rejection, and eval/runtime checks. ### Is Qwen-Scope useful? Yes, conceptually and possibly technically. It supports the idea that code-switching can be analyzed and mitigated. But I would not make it the main solution for a festival pipeline. ### Is tokenizer extension worth it? Maybe, but only after measuring tokenization efficiency. If Qwen3.5 tokenization is acceptable, skip it. ### Should you stay on 0.8B? Maybe. If the project requires strict lightweight deployment, yes. But if 2B or 4B is allowed, test them before investing heavily in 0.8B-specific engineering. ## Practical recommendation My practical recommendation would be: 1. Build a 500–700 item Persian stress eval. 2. Run Qwen3.5-0.8B, 2B, and 4B before training. 3. If 0.8B passes, use it and keep the scope narrow. 4. If 0.8B fails mainly because of language drift, try SFT + small DPO/ORPO. 5. If 0.8B fails because of capacity, tutor quality, or tool stability, move to 2B or 4B. 6. Use KenLM for noise, not for educational value. 7. Do not extend the tokenizer unless tokenization measurements clearly justify it. Main idea: > **Do not solve model-size uncertainty with training first. Solve it with a small eval first. Then choose whether 0.8B, 2B, or 4B is the cheapest route in total engineering cost.** ---
**Author:** ![Bidram](https://avatars.discourse-cdn.com/v4/letter/b/71e660/32.png) [@Bidram](https://discuss.huggingface.co/u/Bidram)\ **Post date:** [June 20, 2026, 7:22pm UTC](https://discuss.huggingface.co/t/how-can-i-build-a-high-quality-dataset/176571/20 "2026-06-20T19:22:11Z")
Hmm… I believe language drift, mathematical reasoning, and some other issues are solved now. I tested Qwen 3.5 0.8B with temperature 0 using a Persian prompt, and the model was able to generate some correct Persian output. I did not see any language drift in the output, but the issue is that the model did not focus on answering my prompt. Let’s review the input and output. > Input: سلام، هوش مصنوعی چیست > > Generated output: سلام! خوشحالم که سوال خوبی پرسیدید. هوش مصنوعی (Artificial Intelligence یا AI) یک مفهوم پیچیده و متغیر است که به زبان ساده می‌توان گفت: «سیستم‌های کامپیوتری هستند که با یادگیری خود، رفتارهایی را شبیه به انسان‌ها انجام می‌دهند.» > > به جای اینکه فقط تعریف خشک بگوییم، این موضوع را از چند جنبه بررسی می‌کنم: > > ۱. چه چیزی است؟ > > هوش مصنوعی یک سیستم هوشمند است که از دانش و الگوریتم‌های پیچیده استفاده می‌کند تا وظایف خاصی را انجام دهد. برخلاف انسان، این سیستم‌ها معمولاً به صورت خودکار تصمیم‌گیری می‌کنند یا رفتارهای مشابهی را دنبال می‌کنند. > > ۲. چگونه کار می‌کند؟ > > هوش مصنوعی از سه بخش اصلی تشکیل شده است: > > داده (Data): مجموعه‌ای از اطلاعات و تصویرهایی که برای سیستم استفاده می‌شود. > > الگوریتم (Algorithm): فرمول یا قانونی که به کمک داده‌ها تصمیمات را اتخاذ می‌کند. > > موتور پردازش (Processor): بخش کامپیوتری که این الگوریتم را اجرا کرده و نتیجه را نمایش می‌دهد. > > ۳. انواع هوش مصنوعی > > هوش مصنوعی به چند دسته اصلی تقسیم می‌شود: > > سیستم‌های تشخیص و پاسخ: مثل سیستم‌های پزشکی (برای تشخیص بیماری) یا ماشین‌آلات صنعتی که با داده‌ها کار می‌کنند. > > موتورهای یادگیری (Machine Learning): این بخش مهم‌ترین قسمت است. در آن‌جا، سیستم به داده‌های جدید و تغییرات مداوم پاسخ می‌دهد تا پیشرفت خود را حفظ کند. مثال: هوش مصنوعی که با دیدن عکس‌های مختلف، تشخیص دهد کدام یک «کلاه» است یا «پیرمرد». > > سیستم‌های تصمیم‌گیری (Decision Making): مانند سیستم‌های مالیاتی یا بانک‌هایی که به داده‌ها می‌پردازند تا بهترین راهکار را پیشنهاد دهند. > > ۴. چرا مهم است؟ > > هوش مصنوعی در زندگی روزمره و آینده بسیار تاثیرگذار است: > > تکنولوژی: در صنایعی مثل پزشکی، کشاورزی و حمل‌ونقل کاربرد دارد. > > بازارهای مالی: تحلیل داده‌ها برای تصمیم‌گیری بهتر در بازار سرمایه. > > دسترسی به اطلاعات: امکان دسترسی سریع به اطلاعاتی که انسان‌ها نمی‌توانند پیدا کنند (مثل جستجوی متنی یا شناسایی اشکال). > > ۵. چالش‌های موجود > > مانند هر فناوری دیگری، هوش مصنوعی همواره با چالش‌هایی روبه‌رو است: > > مشکل «نفس انسان»: سیستم‌ها به دنبال پاسخ سریع و دقیق هستند، اما ممکن است در صورت وجود یک خطا، پاسخ درست را ارائه نکنند. > > امنیت داده: اگر اطلاعات شخصی یا خصوصی افراد برای هوش مصنوعی استفاده شود، خطرناک می‌شود. > > تفاوت با انسان: سیستم‌ها به سرعت یاد می‌گیرند، اما ممکن است در تصمیم‌گیری‌های پیچیده و اخلاقی دچار مشکل شوند. > > خلاصه: هوش مصنوعی ابزارهایی است که از دانش خودکار استفاده می‌کنند تا وظایف را انجام دهند. هدف اصلی آن کمک به انسان‌ها برای انجام کارها به شکلی سریع‌تر، دقیق‌تر، و مؤثرتر است، نه اینکه جایگزین انسان شود. > > آیا سوال خاصی درباره نحوه عملکرد یا کاربرد خاصی از هوش مصنوعی دارید؟ At the beginning of the model’s output, it does not clearly know how to answer. It first tries to answer directly, then adds greetings, and actually, “سلام! خوشحالم که سوال خوبی پرسیدید” is nonsensical. I do not know exactly whether this is an SFT issue or a CPT issue, because the model is already following the instruction and providing a helpful output. And the model clearly knows what AI is. If the issue is more likely related to CPT, I do not have enough high-quality Persian data to meaningfully improve the model. I am afraid of making the model even worse. I am aiming to use translated TinyStories for CPT, but the issue is that TinyStories is mostly good for simple grammar. For knowledge coverage, it is not capable, and Wikipedia datasets are mostly shallow and poor. I just want to know two things: should I do CPT → SFT, or SFT alone, based on this model output? What kind of data do I need, where can I get it, and how should I clean it? I cannot tell whether a text is shallow in quality or not. And about the FineWeb-Edu classifier, that is an English-only classifier, and I do not think I can use the model or its training data to identify shallow texts. [Next page](https://discuss.huggingface.co/t/how-can-i-build-a-high-quality-dataset/176571.md?page=2)