# Sequence features - Class Label Cast\_

**URL:** <https://discuss.huggingface.co/t/sequence-features-class-label-cast/44638>\
**Category:** 🤗Datasets\
**Created:** [June 26, 2023, 6:48pm UTC](https://discuss.huggingface.co/t/sequence-features-class-label-cast/44638 "2023-06-26T18:48:58Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![R0m4ntic](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/r0m4ntic/32/16535_2.png) [@R0m4ntic](https://discuss.huggingface.co/u/R0m4ntic)\
**Post date:** [June 26, 2023, 6:48pm UTC](https://discuss.huggingface.co/t/sequence-features-class-label-cast/44638/1 "2023-06-26T18:48:58Z")

</div>

Hello,

I am having trouble with the ClassLabel features for Token Classification. I am working via Pandas data frame for my dataset. And I am loading the data frame with the dataset. I cannot see the 9 custom IOB labels inside ClassLabel.

```auto
df = pd.DataFrame(df)
dataset = Dataset.from_pandas(df)
dataset = dataset.train_test_split(test_size=0.1)

Output:
DatasetDict({
    train: Dataset({
        features: ['tokens', 'labels', 'id'],
        num_rows: 10000
    })
    test: Dataset({
        features: ['tokens', 'labels', 'id'],
        num_rows: 1000
    })
})

Output:
{'tokens': Value(dtype='string', id=None),
 **'labels': Value(dtype='string', id=None),**
 'id': Value(dtype='int64', id=None)}

```

I already tried the **“cast” method** → dataset.cast\_column(“labels”  
here … [ClassLabel Error · Issue #5737 · huggingface/datasets · GitHub](https://github.com/huggingface/datasets/issues/5737)

And the **“new\_features”** in the package reference.  
here… [Main classes](https://huggingface.co/docs/datasets/package_reference/main_classes)

@mariosasko ? 😇

Thank you guys!

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [June 27, 2023, 2:09pm UTC](https://discuss.huggingface.co/t/sequence-features-class-label-cast/44638/2 "2023-06-27T14:09:16Z")

</div>

Try this:

```python
df = pd.DataFrame(df)
dataset = Dataset.from_pandas(df)
dataset = dataset.class_encode_column("labels")
dataset = dataset.train_test_split(test_size=0.1)

```

---

<div class="post-metadata">

**Author:** ![R0m4ntic](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/r0m4ntic/32/16535_2.png) [@R0m4ntic](https://discuss.huggingface.co/u/R0m4ntic)\
**Post date:** [June 28, 2023, 4:42pm UTC](https://discuss.huggingface.co/t/sequence-features-class-label-cast/44638/3 "2023-06-28T16:42:41Z")

</div>

Thank you for your help @mariosasko

It’s correctly creating a ClassLabel(names=  
Unfortunately, it’s appending all the labels on each row…

Column df[“labels”] with many rows as follow [‘O,O,B-DRUG,O,B-HOSPITAL,O’]

With encode\_column, my results is a very long ClassLabel(names= [‘O,O,B-DRUG,O,B-HOSPITAL,O,B-HOSPITAL,O,B-DATE,O,O,O’]

Maybe the labels shouldn’t be a Pandas Series?

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [June 28, 2023, 6:09pm UTC](https://discuss.huggingface.co/t/sequence-features-class-label-cast/44638/4 "2023-06-28T18:09:35Z")

</div>

Oh, I missed the “Token Classification” part.

Then this should work:

```python
df = pd.DataFrame(df)
dataset = Dataset.from_pandas(df)
dataset = dataset.map(lambda ex: {"labels": ex["labels"].split(",")})

def get_label_list(labels):
    # copied from https://github.com/huggingface/transformers/blob/66fd3a8d626a32989f4569260db32785c6cbf42a/examples/pytorch/token-classification/run_ner.py#L320
    unique_labels = set()
    for label in labels:
        unique_labels = unique_labels | set(label)
    label_list = list(unique_labels)
    label_list.sort()
    return label_list

all_labels = get_label_list(dataset["labels"])

dataset = dataset.cast_column("labels", datasets.Sequence(datasets.ClassLable(names=all_labels)))
dataset = dataset.train_test_split(test_size=0.1)

```

---

<div class="post-metadata">

**Author:** ![R0m4ntic](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/r0m4ntic/32/16535_2.png) [@R0m4ntic](https://discuss.huggingface.co/u/R0m4ntic)\
**Post date:** [June 28, 2023, 6:58pm UTC](https://discuss.huggingface.co/t/sequence-features-class-label-cast/44638/5 "2023-06-28T18:58:03Z")

</div>

Perfect! Thank you @mariosasko

Have a good evening!

---

<div class="post-metadata">

**Author:** ![R0m4ntic](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/r0m4ntic/32/16535_2.png) [@R0m4ntic](https://discuss.huggingface.co/u/R0m4ntic)\
**Post date:** [July 4, 2023, 12:16pm UTC](https://discuss.huggingface.co/t/sequence-features-class-label-cast/44638/6 "2023-07-04T12:16:54Z")

</div>

Sorry to bother you again @mariosasko

I am trying to create a “ner\_tags” column with integers corresponding to “labels” to follow the HF tutorial on Token Classification.

I tried that…

```auto
tags = dataset.features[f"labels"].feature.names
print(tags)

def create_tag_names(batch):
    return {"ner_tags_str": [tags.str2int(idx) for idx in batch["labels"]]}
dataset = dataset.map(create_tag_names) 

```

Any ideas? Thx!

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [July 4, 2023, 1:04pm UTC](https://discuss.huggingface.co/t/sequence-features-class-label-cast/44638/7 "2023-07-04T13:04:16Z")

</div>

> I am trying to create a “ner\_tags” column with integers corresponding to “labels” to follow the HF tutorial on Token Classification.

Can you provide a link to the tutorial? We already store class labels as integers, so I’m not sure I understand what you want to do.

---

<div class="post-metadata">

**Author:** ![R0m4ntic](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/r0m4ntic/32/16535_2.png) [@R0m4ntic](https://discuss.huggingface.co/u/R0m4ntic)\
**Post date:** [July 4, 2023, 1:18pm UTC](https://discuss.huggingface.co/t/sequence-features-class-label-cast/44638/8 "2023-07-04T13:18:12Z")

</div>

Oops, sorry!

> **[Token classification](https://huggingface.co/docs/transformers/tasks/token_classification)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

The tutorial is working with the column ner\_tags made of numbers mapping to the corresponding labels…  
Thx!

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [July 4, 2023, 2:12pm UTC](https://discuss.huggingface.co/t/sequence-features-class-label-cast/44638/9 "2023-07-04T14:12:06Z")

</div>

You can rename the `labels` column to `ner_tags` to have the same structure:

```python
dataset = dataset.rename_column("labels", "ner_tags")

```

And apply the rest of the processing.

---

<div class="post-metadata">

**Author:** ![R0m4ntic](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/r0m4ntic/32/16535_2.png) [@R0m4ntic](https://discuss.huggingface.co/u/R0m4ntic)\
**Post date:** [July 4, 2023, 3:23pm UTC](https://discuss.huggingface.co/t/sequence-features-class-label-cast/44638/10 "2023-07-04T15:23:17Z")

</div>

@mariosasko Thank you for your help!

I found the solution to my stupid bug on “Tokens”. I transformed my sentences in “Tokens” into a list of tokens by adding the following line:

dataset = dataset.map(lambda ex: {“tokens”: ex[“tokens”].split(“,”)})

Thx again! : )