# Converting string label to int **URL:** \ **Category:** 🤗Datasets\ **Created:** [December 14, 2020, 6:49am UTC](https://discuss.huggingface.co/t/converting-string-label-to-int/2816 "2020-12-14T06:49:16Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![spentaur](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/spentaur/32/825_2.png) [@spentaur](https://discuss.huggingface.co/u/spentaur)\ **Post date:** [December 14, 2020, 6:49am UTC](https://discuss.huggingface.co/t/converting-string-label-to-int/2816/1 "2020-12-14T06:49:17Z")
I’m using a custom dataset from a CSV file where the labels are strings. I’m curious what the best way to encode these labels to integers would be. Sample code: ```auto datasets = load_dataset('csv', data_files={ 'train': 'train.csv', 'test': 'test.csv' } ) def tokenize(batch): return tokenizer(batch['text'], padding=True, truncation=True, max_length=128) datasets = datasets.map(tokenize, batched=True) datasets.set_format('torch', columns=['input_ids', 'attention_mask', 'labels']) ``` ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\ **Post date:** [December 14, 2020, 2:47pm UTC](https://discuss.huggingface.co/t/converting-string-label-to-int/2816/2 "2020-12-14T14:47:04Z")
In your `tokenize` function, you can also add a line to convert your labels to ints: ```auto def tokenize(batch): tokenized_batch = tokenizer(batch['text'], padding=True, truncation=True, max_length=128) tokenized_batch["labels"] = [str_to_int[label] for label in batch["labels"]] return tokenized_batch ``` with `str_to_int` your correspondence string label to int label. ---
**Author:** ![spentaur](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/spentaur/32/825_2.png) [@spentaur](https://discuss.huggingface.co/u/spentaur)\ **Post date:** [December 14, 2020, 8:11pm UTC](https://discuss.huggingface.co/t/converting-string-label-to-int/2816/3 "2020-12-14T20:11:10Z")
Thank you for your reply. I ended up using your approach. ```auto from datasets import ClassLabel labels = ClassLabel(names_file='labels.txt') datasets = load_dataset('csv', data_files={ 'train': 'train.csv', 'test': 'test.csv' } ) def tokenize(batch): tokens = tokenizer(batch['text'], padding=True, truncation=True, max_length=128) tokens['labels'] = labels.str2int(batch['labels']) return tokens datasets = datasets.map(tokenize, batched=True) datasets.set_format('torch', columns=['input_ids', 'attention_mask', 'labels']) ``` ---
**Author:** ![loretoparisi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/loretoparisi/32/27841_2.png) [@loretoparisi](https://discuss.huggingface.co/u/loretoparisi)\ **Post date:** [April 29, 2022, 4:52pm UTC](https://discuss.huggingface.co/t/converting-string-label-to-int/2816/4 "2022-04-29T16:52:26Z")
It seems I’m running in the same issue: [https://github.com/huggingface/transformers/issues/16975](https://github.com/huggingface/transformers/issues/16975) Doing as suggested ```auto from transformers import AutoTokenizer #https://huggingface.co/microsoft/xtremedistil-l6-h256-uncased model_name = 'microsoft/xtremedistil-l6-h256-uncased' tokenizer = AutoTokenizer.from_pretrained(model_name) def tokenize(batch): tokens = tokenizer(batch['text'], padding="max_length", truncation=True, max_length=128) tokens['label'] = features["label"].str2int(batch['label']) if batch["label"] is not None else None return tokens tokenized_datasets = sentences.map(tokenize, batched=True) ``` I’m getting ```auto --------------------------------------------------------------------------- KeyError Traceback (most recent call last) in () 14 tokens['label'] = features["label"].str2int(batch['label']) if batch["label"] is not None else None 15 return tokens ---> 16 tokenized_datasets = sentences.map(tokenize, batched=True) 10 frames /usr/local/lib/python3.7/dist-packages/datasets/features/features.py in str2int(self, values) 852 if value not in self._str2int: 853 value = str(value).strip() --> 854 output.append(self._str2int[str(value)]) 855 else: 856 # No names provided, try to integerize KeyError: 'None' ``` ---
**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\ **Post date:** [May 2, 2022, 12:27pm UTC](https://discuss.huggingface.co/t/converting-string-label-to-int/2816/5 "2022-05-02T12:27:26Z")
Hi! It seems like there are `None` values in the `label` column. To avoid the error, you can either drop such examples with `.filter(..)` or modify the tokenize function as follows: ```python def tokenize(batch): tokens = tokenizer(batch['text'], padding="max_length", truncation=True, max_length=128) tokens["label"] = [features["label"].str2int(label) if label is not None else None for label in batch["label"]] return tokens ``` ---
**Author:** ![dronych](https://avatars.discourse-cdn.com/v4/letter/d/ce7236/32.png) [@dronych](https://discuss.huggingface.co/u/dronych)\ **Post date:** [August 8, 2023, 10:35am UTC](https://discuss.huggingface.co/t/converting-string-label-to-int/2816/6 "2023-08-08T10:35:38Z")
As opposed to using some pre-made encoders like those found in sklearn, for example which I found to be slow, I wrote a simple class like this: ```auto class LabelEncoder: def __init__ (self): self.labels_to_int = {} def encode(self, labels): prev_label = 0 encoded_labels = [] for label in labels: if label not in self.labels_to_int: self.labels_to_int[label] = prev_label encoded = prev_label prev_label += 1 else: encoded = self.labels_to_int[label] encoded_labels.append(encoded) return encoded_labels label_encoder = LabelEncoder() tokenized_dataset = tokenized_dataset.map( lambda batch: { 'labels_encoded': label_encoder.encode(batch['labels']) }, load_from_cache_file=False, batched=True ) ```