# Token Classification run\_NER.py AttributeError **URL:** \ **Category:** Models\ **Created:** [July 7, 2022, 8:06am UTC](https://discuss.huggingface.co/t/token-classification-run-ner-py-attributeerror/20089 "2022-07-07T08:06:58Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![dineshmane](https://avatars.discourse-cdn.com/v4/letter/d/e495f1/32.png) [@dineshmane](https://discuss.huggingface.co/u/dineshmane)\ **Post date:** [July 7, 2022, 8:06am UTC](https://discuss.huggingface.co/t/token-classification-run-ner-py-attributeerror/20089/1 "2022-07-07T08:06:59Z")
Hi, I am fine tuning "bert-base-uncased’ model for NEE using “wnut\_17”. For finetuning, I am saving the train and test dataset using .to\_json() method of datasets library. However while loading that locally saved file using load\_datasets() method, the structure of the attribute are changing. ```auto def tokenize_and_align_labels(examples): tokenized_inputs = tokenizer(examples["tokens"], truncation=True, is_split_into_words=True) labels = [] for i, label in enumerate(examples[f"ner_tags"]): word_ids = tokenized_inputs.word_ids(batch_index=i) # Map tokens to their respective word. previous_word_idx = None label_ids = [] for word_idx in word_ids: # Set the special tokens to -100. if word_idx is None: label_ids.append(-100) elif word_idx != previous_word_idx: # Only label the first token of a given word. label_ids.append(label[word_idx]) else: label_ids.append(-100) previous_word_idx = word_idx labels.append(label_ids) tokenized_inputs["labels"] = labels return tokenized_inputs wnut = load_dataset("wnut_17") tokenized_wnut = wnut.map(tokenize_and_align_labels, batched=True) features = tokenized_wnut['train'].features print("output1-->") print(features) tokenized_wnut["train"].to_json("train_ner1.json") raw_datasets = load_dataset("json", data_files={'train': 'train_ner1.json'}) re_features = raw_datasets['train'].features print("output2-->") print(re_features) ``` > output1–\> > {‘id’: Value(dtype=‘string’, id=None), ‘tokens’: Sequence(feature=Value(dtype=‘string’, id=None), length=-1, id=None), **‘ner\_tags’: Sequence(feature=ClassLabel(num\_classes=13, names=[‘O’, ‘B-corporation’, ‘I-corporation’, ‘B-creative-work’, ‘I-creative-work’, ‘B-group’, ‘I-group’, ‘B-location’, ‘I-location’, ‘B-person’, ‘I-person’, ‘B-product’, ‘I-product’], id=None)**, length=-1, id=None), ‘input\_ids’: Sequence(feature=Value(dtype=‘int32’, id=None), length=-1, id=None), ‘attention\_mask’: Sequence(feature=Value(dtype=‘int8’, id=None), length=-1, id=None), ‘labels’: Sequence(feature=Value(dtype=‘int64’, id=None), length=-1, id=None)} > > output2–\> > {‘id’: Value(dtype=‘string’, id=None), ‘tokens’: Sequence(feature=Value(dtype=‘string’, id=None), length=-1, id=None), **‘ner\_tags’: Sequence(feature=Value(dtype=‘int64’, id=None)**, length=-1, id=None), ‘input\_ids’: Sequence(feature=Value(dtype=‘int64’, id=None), length=-1, id=None), ‘attention\_mask’: Sequence(feature=Value(dtype=‘int64’, id=None), length=-1, id=None), ‘labels’: Sequence(feature=Value(dtype=‘int64’, id=None), length=-1, id=None)} I am not understanding why ner\_tags attribute format is changed while exporting dataset into json and importing back. Model expects labels to be string but importing json converts them into integer and getting below error: > if label.startswith(“B-”) and label.replace(“B-”, “I-”) in label\_list: > AttributeError: ‘int’ object has no attribute ‘startswith’ -Thanks, Dinesh ---
**Author:** ![dineshmane](https://avatars.discourse-cdn.com/v4/letter/d/e495f1/32.png) [@dineshmane](https://discuss.huggingface.co/u/dineshmane)\ **Post date:** [July 8, 2022, 8:26pm UTC](https://discuss.huggingface.co/t/token-classification-run-ner-py-attributeerror/20089/2 "2022-07-08T20:26:27Z")
I noticed that storing train and test dataset in csv/json and reloading it, not giving me the original dataset . In reloaded dataset, **ner\_tags** feature isn’t instance of ClassLabel. However, when I am saving the train/test dataset in arrow format and re-loading it. The reloaded dataset is same as original one with label feature being instance of ClassLabel. I have modified the [run\_NER.py](https://github.com/huggingface/transformers/blob/main/examples/pytorch/token-classification/run_ner.py) file to consume train/test/validation dataset in following way: ```auto if data_args.dataset_name is not None: # Downloading and loading a dataset from the hub. raw_datasets = load_dataset( data_args.dataset_name, data_args.dataset_config_name, cache_dir=model_args.cache_dir, use_auth_token=True if model_args.use_auth_token else None, ) if "train" in raw_datasets: train_dataset = raw_datasets['train'] if "test" in raw_datasets: test_dataset = raw_datasets['test'] if "validation" in raw_datasets: validation_dataset = raw_datasets['validation'] else: # data_files = {} # if data_args.train_file is not None: # data_files["train"] = data_args.train_file # if data_args.validation_file is not None: # data_files["validation"] = data_args.validation_file # if data_args.test_file is not None: # data_files["test"] = data_args.test_file # extension = data_args.train_file.split(".")[-1] # raw_datasets = load_dataset(extension, data_files=data_files, cache_dir=model_args.cache_dir) train_dataset = load_from_disk(data_args.train_file) test_dataset = load_from_disk(data_args.test_file) if data_args.validation_file is not None: validation_dataset = load_from_disk(data_args.validation_file) ``` in rest of the code of run\_NER.py, i just replaced raw\_datasets with appropriate train\_dataset/test\_dataset/validation\_dataset. Note: If anyone knows how to club the two dataset loaded from arrow format files, please feel free to drop the solution… 🙂 Until then this hack works 😉 -Thanks, Dinesh