# Preprocessing data for text classification, HF dataset

**URL:** <https://discuss.huggingface.co/t/preprocessing-data-for-text-classification-hf-dataset/23926>\
**Category:** 🤗Datasets\
**Created:** [October 3, 2022, 9:31am UTC](https://discuss.huggingface.co/t/preprocessing-data-for-text-classification-hf-dataset/23926 "2022-10-03T09:31:47Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![jorgenhw](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jorgenhw/32/5005_2.png) [@jorgenhw](https://discuss.huggingface.co/u/jorgenhw)\
**Post date:** [October 3, 2022, 9:31am UTC](https://discuss.huggingface.co/t/preprocessing-data-for-text-classification-hf-dataset/23926/1 "2022-10-03T09:31:47Z")

</div>

I have this dataset

```auto
text label
lorem ipsum positive
lorem ipsum positive
lorem ipsum positive
lorem ipsum positive

```

which I tokenize like this

```auto
def tokenize_function(example):
  return tokenizer(example["text"], padding="max_length", truncation=True, max_length=128)

tokenized_datasets = raw_dataset.map(tokenize_function, batched=True)

```

and now it looks like this:

```auto
Dataset({
    features: ['text', 'label', 'input_ids', 'token_type_ids', 'attention_mask'],
    num_rows: 1024
})

```

Questions:  
1: Should the label column be tokenized? The task is multiclass classification  
2: Should I remove the ‘text’ variable from the dataset (which I use for training?  
3: Other things I haven’t thought about?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [October 3, 2022, 4:23pm UTC](https://discuss.huggingface.co/t/preprocessing-data-for-text-classification-hf-dataset/23926/2 "2022-10-03T16:23:09Z")

</div>

Hi !

1. Usually the “label” column must be an integer that is passed to the model. You can use

```nohighlight
tokenized_datasets = tokenized_datasets.class_encode_column("label")

```

to automatically convert the column to integers. The mapping string\<-\>integer can be found then at `tokenized_datasets.features["label"]`

1. In general, models accept tokens as input (input\_ids, token\_type\_ids, attention\_mask), so you can drop the “text” column