# Hinduja Swiss IT-Professional-How do I preprocess or tokenize data for training language models?

**URL:** <https://discuss.huggingface.co/t/hinduja-swiss-it-professional-how-do-i-preprocess-or-tokenize-data-for-training-language-models/172778>\
**Category:** 🤗Datasets\
**Created:** [January 24, 2026, 12:04pm UTC](https://discuss.huggingface.co/t/hinduja-swiss-it-professional-how-do-i-preprocess-or-tokenize-data-for-training-language-models/172778 "2026-01-24T12:04:52Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![raghavhinduja26](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/raghavhinduja26/32/57488_2.png) [@raghavhinduja26](https://discuss.huggingface.co/u/raghavhinduja26)\
**Post date:** [January 24, 2026, 12:04pm UTC](https://discuss.huggingface.co/t/hinduja-swiss-it-professional-how-do-i-preprocess-or-tokenize-data-for-training-language-models/172778/1 "2026-01-24T12:04:52Z")

</div>

Hi everyone, I am Raghav Hinduja Swiss based IT-professional. I’m looking for suggestions on how to preprocess or tokenize data for training language models using Hugging Face.  
What tools, workflows, or best practices have worked well for you?

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [January 25, 2026, 2:02am UTC](https://discuss.huggingface.co/t/hinduja-swiss-it-professional-how-do-i-preprocess-or-tokenize-data-for-training-language-models/172778/2 "2026-01-25T02:02:09Z")

</div>

If you’re unsure about what to fine-tune, I think starting with either [`LLM Course` or `a smol course`](https://huggingface.co/learn) will help you avoid confusion.

* * *

## 1) Start by choosing your training objective (it determines the “right” preprocessing)

### Causal LM (CLM, decoder-only; “next token prediction”)

- You typically **tokenize → concatenate → chunk into fixed-length blocks** (`block_size`) for efficient training.
- This is the pattern used in HF’s canonical CLM example script (`run_clm.py`). ([GitHub](https://github.com/huggingface/transformers/issues/23198 "Should group\_text in run\_clm.py separate documents with ..."))

### Chat / instruction SFT (still CLM under the hood, but formatted as messages)

- Your biggest risk is **formatting + special tokens + label masking** , not raw tokenization.
- The safe default is to use **chat templates** correctly (details below). ([Hugging Face](https://huggingface.co/docs/transformers/en/chat_templating "Chat templates"))

### MLM (BERT-style)

- Tokenization is similar, but masking is usually applied by a **data collator** at batch time.

* * *

## 2) Core tools in the Hugging Face stack (and what each is for)

### 🤗 `datasets` (I/O + transformations)

- Load data from files or Hub, transform with `map()`, filter, shuffle, stream big corpora.
- If your dataset is too large to store locally, load in **streaming** mode to get an `IterableDataset`. ([Hugging Face](https://huggingface.co/docs/datasets/en/stream "Stream"))

### 🤗 `transformers` tokenizers (text → token IDs)

- Prefer **Fast** tokenizers (Rust-backed) for speed and consistent behavior. ([Hugging Face](https://huggingface.co/docs/transformers/en/main_classes/tokenizer "Tokenizer"))

### Optional: large-scale data pipelines (dedup/filtering)

- For web-scale preprocessing (filtering, dedup, etc.), HF’s **DataTrove** provides reference pipelines (e.g., the FineWeb processing script). ([GitHub](https://github.com/huggingface/datatrove/blob/main/examples/fineweb.py "datatrove/examples/fineweb.py at main"))

* * *

## 3) Data cleaning & quality filtering (what matters most before tokenization)

This step often dominates downstream model quality.

### Minimum “always do it” cleaning

- Normalize whitespace / remove null bytes / fix obvious encoding issues.
- Drop pathological samples (extremely short, extremely long, repetitive junk).
- Remove markup if your source is HTML.

### Deduplicate (especially for pretraining / continued pretraining)

Duplicate data wastes compute and can leak evaluation examples into training.

- FineWeb explicitly documents a pipeline of **cleaning + dedup** , and points to a working script for the full process. ([Hugging Face](https://huggingface.co/datasets/HuggingFaceFW/fineweb/blob/main/README.md "README.md · HuggingFaceFW/fineweb at main"))
- The DataTrove repository includes an example script used to create FineWeb. ([GitHub](https://github.com/huggingface/datatrove/blob/main/examples/fineweb.py "datatrove/examples/fineweb.py at main"))

If you’re not operating at web scale, even **exact-match dedup** (hash the normalized text) gives a meaningful win.

* * *

## 4) Tokenizer strategy: reuse vs train a new one

### Fine-tuning an existing model

Use the model’s tokenizer as-is. Changing vocab has knock-on effects and usually isn’t worth it.

### Pretraining from scratch (or new language/domain where the tokenizer is a bad fit)

Train a tokenizer on a representative slice of your corpus.

- HF’s LLM course shows `train_new_from_iterator()` as a practical approach (works with fast tokenizers). ([Hugging Face](https://huggingface.co/learn/llm-course/en/chapter6/2 "Training a new tokenizer from an old one"))
- The Transformers tokenizer docs explain fast vs slow tokenizers and expected capabilities. ([Hugging Face](https://huggingface.co/docs/transformers/en/main_classes/tokenizer "Tokenizer"))
- HF also published a late-2025 overview of tokenization for LLMs (useful for updated mental models and API direction). ([Hugging Face](https://huggingface.co/blog/tokenizers "Tokenization in Transformers v5: Simpler, Clearer, and ..."))

* * *

## 5) Tokenize efficiently with `datasets.map()` (speed + reproducibility)

### Use batch mapping (`batched=True`)

Batch mapping is explicitly designed to speed up tokenization because tokenizers run faster on batches. ([Hugging Face](https://huggingface.co/docs/datasets/en/about_map_batch "Batch mapping"))

```python
from datasets import load_dataset
from transformers import AutoTokenizer

ds = load_dataset("json", data_files={"train": "train.jsonl"})
tok = AutoTokenizer.from_pretrained("gpt2", use_fast=True)

def tokenize(batch):
    return tok(batch["text"], truncation=False)

tokenized = ds["train"].map(
    tokenize,
    batched=True,
    remove_columns=ds["train"].column_names,
)

```

### Caching and saving artifacts

- HF Datasets uses caching; if caching is disabled, your transforms can be recomputed and then deleted at session end unless you explicitly save the result. ([Hugging Face](https://huggingface.co/docs/datasets/en/about_cache "The cache"))

```python
tokenized.save_to_disk("./tokenized_train")

```

### When `map()` slows down near the end

This is a common report in real workflows (often due to I/O, cache writes, or skewed example sizes). A typical mitigation is to shard, reduce output columns, and ensure fast local storage. ([Hugging Face Forums](https://discuss.huggingface.co/t/how-to-handle-big-data/40143 "How to handle big data? - 🤗Datasets"))

* * *

## 6) CLM preprocessing: packing (concatenate + chunk) and boundary handling

### The standard “group\_texts” approach

The canonical CLM recipe is: tokenize → concatenate → slice into `block_size` chunks (often with `labels = input_ids`). This is the approach discussed around `run_clm.py`. ([GitHub](https://github.com/huggingface/transformers/issues/23198 "Should group\_text in run\_clm.py separate documents with ..."))

### Boundary pitfall: “Should I insert EOS between documents?”

This is a frequently debated detail; there’s a dedicated issue asking whether `run_clm.py` should separate documents with a special token. ([GitHub](https://github.com/huggingface/transformers/issues/23198 "Should group\_text in run\_clm.py separate documents with ..."))

**Practical guidance**

- If your samples are independent documents, append an **EOS** to each doc before concatenation to prevent unnatural “doc bleed”.
- If your data is already a continuous stream (e.g., book text split into lines), you may choose not to.

### Block-size pitfall: remainder handling

A known failure mode is producing chunks that aren’t exactly `block_size`, causing training errors. There’s an issue specifically about `group_texts` needing to drop incorrect-length sequences. ([GitHub](https://github.com/huggingface/transformers/issues/18167 "Group\_texts in run\_clm.py will add shorter than block\_size ..."))

* * *

## 7) Chat / instruction SFT: use chat templates correctly (most important for your case)

### Recommended default: `apply_chat_template(..., tokenize=True)`

Transformers explicitly warns that chat templates generally already include the special tokens; templating into text and then tokenizing “normally” can insert special tokens twice and degrade performance. ([Hugging Face](https://huggingface.co/docs/transformers/en/chat_templating "Chat templates"))

```python
def chat_to_features(example, tokenizer):
    # example["messages"] = [{"role": "system"/"user"/"assistant", "content": "..."}]
    return tokenizer.apply_chat_template(
        example["messages"],
        tokenize=True,
        add_generation_prompt=False,
        return_dict=True,
    )

```

### If you do template → tokenize in two steps

Set `add_special_tokens=False` when tokenizing the rendered string, exactly as the docs recommend. ([Hugging Face](https://huggingface.co/docs/transformers/en/chat_templating "Chat templates"))

This issue shows a concrete example where templating then encoding results in duplicated BOS. ([GitHub](https://github.com/huggingface/transformers/issues/37686 "Tokenizing with `apply\_chat\_template` behaves differently ..."))

* * *

## 8) Labels and loss masking (assistant-only / completion-only training)

If you want loss only on the assistant output (common in instruction tuning):

- TRL documents `DataCollatorForCompletionOnlyLM` and states it works only when `packing=False`. ([Hugging Face](https://huggingface.co/docs/trl/v0.7.1/en/sft_trainer "Supervised Fine-tuning Trainer"))
- There’s also an explicit TRL issue asking if you can combine packing with completion-only training (short answer: not directly “as-is”). ([GitHub](https://github.com/huggingface/trl/issues/772 "Can I use example packing and train on completions only ..."))

**Practical recommendation**

- Start with correctness: **completion-only + no packing** (simple, reliable).
- Only introduce packing after you have tests that confirm label masking does not cross sample boundaries.

* * *

## 9) Large datasets: when to stream instead of materialize

If the corpus is too large for local disk/RAM, use streaming:

- `streaming=True` yields an `IterableDataset` you can iterate without downloading everything. ([Hugging Face](https://huggingface.co/docs/datasets/en/stream "Stream"))
- Be aware: streaming has different performance characteristics, and there are ongoing questions/issues about throughput and how it compares to map-style datasets. ([GitHub](https://github.com/huggingface/datasets/issues/7680 "Question about iterable dataset and streaming · Issue #7680"))

A common production pattern is:

1. stream + light filtering →
2. write cleaned shards (e.g., parquet/jsonl) →
3. train on the stable shards with map-style datasets for speed.

* * *

## 10) A “best-practice checklist” (what tends to work well)

### Tokenization & formatting

- Use **fast** tokenizers (`use_fast=True`). ([Hugging Face](https://huggingface.co/docs/transformers/en/main_classes/tokenizer "Tokenizer"))
- Use `Dataset.map(..., batched=True)` for tokenization speed. ([Hugging Face](https://huggingface.co/docs/datasets/en/about_map_batch "Batch mapping"))
- For chat SFT: prefer `apply_chat_template(tokenize=True)`; if not, set `add_special_tokens=False`. ([Hugging Face](https://huggingface.co/docs/transformers/en/chat_templating "Chat templates"))

### CLM packing

- Ensure chunking outputs exactly `block_size` (drop remainder). ([GitHub](https://github.com/huggingface/transformers/issues/18167 "Group\_texts in run\_clm.py will add shorter than block\_size ..."))
- Decide and document whether you insert EOS between documents (and keep it consistent). ([GitHub](https://github.com/huggingface/transformers/issues/23198 "Should group\_text in run\_clm.py separate documents with ..."))

### Dataset ops & reproducibility

- Remove unused columns early (`remove_columns=...`) to reduce I/O and cache size. ([Hugging Face](https://huggingface.co/docs/datasets/v1.4.0/processing.html "Processing data in a Dataset"))
- If caching is disabled, `save_to_disk()` or you’ll lose results at session end. ([Hugging Face](https://huggingface.co/docs/datasets/en/about_cache "The cache"))

### Scaling

- Stream very large corpora, and materialize only cleaned/filtered shards you intend to train on. ([Hugging Face](https://huggingface.co/docs/datasets/en/stream "Stream"))
- For web-scale, follow a pipeline-style approach with filtering + dedup (FineWeb + DataTrove are good reference points). ([Hugging Face](https://huggingface.co/datasets/HuggingFaceFW/fineweb/blob/main/README.md "README.md · HuggingFaceFW/fineweb at main"))

* * *

## Recommended “reading order” (fast path)

1. Batch mapping (`datasets.map` with `batched=True`). ([Hugging Face](https://huggingface.co/docs/datasets/en/about_map_batch "Batch mapping"))
2. Chat templating (and the special-token pitfall). ([Hugging Face](https://huggingface.co/docs/transformers/en/chat_templating "Chat templates"))
3. Completion-only SFT constraints in TRL (packing vs masking). ([Hugging Face](https://huggingface.co/docs/trl/v0.7.1/en/sft_trainer "Supervised Fine-tuning Trainer"))
4. Streaming docs for big data. ([Hugging Face](https://huggingface.co/docs/datasets/en/stream "Stream"))
5. FineWeb/DataTrove pipeline as a reference for real-world filtering/dedup. ([Hugging Face](https://huggingface.co/datasets/HuggingFaceFW/fineweb/blob/main/README.md "README.md · HuggingFaceFW/fineweb at main"))

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [January 27, 2026, 9:46am UTC](https://discuss.huggingface.co/t/hinduja-swiss-it-professional-how-do-i-preprocess-or-tokenize-data-for-training-language-models/172778/3 "2026-01-27T09:46:38Z")

</div>

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.