# Tutorial: Fine-tuning with custom datasets – sentiment, NER, and question answering **URL:** \ **Category:** 🤗Transformers\ **Created:** [August 17, 2020, 2:01pm UTC](https://discuss.huggingface.co/t/tutorial-fine-tuning-with-custom-datasets-sentiment-ner-and-question-answering/733 "2020-08-17T14:01:20Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![joeddav](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/joeddav/32/53881_2.png) [@joeddav](https://discuss.huggingface.co/u/joeddav)\ **Post date:** [August 17, 2020, 2:01pm UTC](https://discuss.huggingface.co/t/tutorial-fine-tuning-with-custom-datasets-sentiment-ner-and-question-answering/733/1 "2020-08-17T14:01:20Z")
Interested in fine-tuning on your own custom datasets but unsure how to get going? I just added a tutorial to the docs with several examples that each walk you through downloading a dataset, preprocessing & tokenizing, and training with either Trainer, native PyTorch, or native TensorFlow 2. Examples include: 1. Sequence classification (sentiment) – [IMDb](http://ai.stanford.edu/~amaas/data/sentiment/) 2. Token classification (NER) – [W-NUT Emerging and Rare entities](http://noisy-text.github.io/2017/emerging-rare-entities.html) 3. Question answering (span selection) – [SQuAD 2.0](https://rajpurkar.github.io/SQuAD-explorer/) Click the `Open in Colab` button at the top to open a colab notebook in either TF or PT. This tutorial demonstrates one workflow for working with custom datasets, but there are many valid ways to accomplish the same thing. The intention is to be demonstrative rather than definitive. Also, we highly recommend you check out and contribute to our [NLP datasets & metrics library](https://github.com/huggingface/nlp) for easy access 150+ datasets. Tutorial: [https://huggingface.co/transformers/master/custom\_datasets.html](https://huggingface.co/transformers/master/custom_datasets.html) Feedback and questions welcome! ---
**Author:** ![rbint](https://avatars.discourse-cdn.com/v4/letter/r/85e7bf/32.png) [@rbint](https://discuss.huggingface.co/u/rbint)\ **Post date:** [August 17, 2020, 8:17pm UTC](https://discuss.huggingface.co/t/tutorial-fine-tuning-with-custom-datasets-sentiment-ner-and-question-answering/733/2 "2020-08-17T20:17:40Z")
I spotted a minor typo. “…which we can use for for evaluation and tuning without _ **taining** _ our test set results.” I believe you meant to say tainting. Otherwise, great tutorial. I’m looking forward to digging in more. ---
**Author:** ![smalltoken](https://avatars.discourse-cdn.com/v4/letter/s/c67d28/32.png) [@smalltoken](https://discuss.huggingface.co/u/smalltoken)\ **Post date:** [August 17, 2020, 9:07pm UTC](https://discuss.huggingface.co/t/tutorial-fine-tuning-with-custom-datasets-sentiment-ner-and-question-answering/733/3 "2020-08-17T21:07:36Z")
**thanks for contributing!** May I ask if I don’t have any label or relationship between sentences, could I fine tune a bert model by masked language model task? ---
**Author:** ![joeddav](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/joeddav/32/53881_2.png) [@joeddav](https://discuss.huggingface.co/u/joeddav)\ **Post date:** [August 17, 2020, 10:01pm UTC](https://discuss.huggingface.co/t/tutorial-fine-tuning-with-custom-datasets-sentiment-ner-and-question-answering/733/4 "2020-08-17T22:01:30Z")
Of course! At the bottom of the tutorial we actually link to a blog post that shows you how to do just that [https://huggingface.co/blog/how-to-train](https://huggingface.co/blog/how-to-train) ---
**Author:** ![stefan-jo](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/stefan-jo/32/64_2.png) [@stefan-jo](https://discuss.huggingface.co/u/stefan-jo)\ **Post date:** [August 18, 2020, 3:19pm UTC](https://discuss.huggingface.co/t/tutorial-fine-tuning-with-custom-datasets-sentiment-ner-and-question-answering/733/5 "2020-08-18T15:19:04Z")
Thanks a lot for creating the tutorial @joeddav! I ran into an issue with `tokenizer`. It seems like I cannot just pass my list of texts to tokenizer like in the tutorial. Am I doing something wrong? ``` tokenizer = DistilBertTokenizerFast.from_pretrained('distilbert-base-uncased') train_encodings = tokenizer(train_texts, truncation=True, padding=True) val_encodings = tokenizer(test_texts, truncation=True, padding=True) --------------------------------------------------------------------------- TypeError Traceback (most recent call last) in ----> 1 train_encodings = tokenizer(train_texts, truncation=True, padding=True) 2 val_encodings = tokenizer(test_texts, truncation=True, padding=True) TypeError: 'DistilBertTokenizerFast' object is not callable ``` ---
**Author:** ![rgwatwormhill](https://avatars.discourse-cdn.com/v4/letter/r/a9adbd/32.png) [@rgwatwormhill](https://discuss.huggingface.co/u/rgwatwormhill)\ **Post date:** [August 18, 2020, 3:38pm UTC](https://discuss.huggingface.co/t/tutorial-fine-tuning-with-custom-datasets-sentiment-ner-and-question-answering/733/6 "2020-08-18T15:38:13Z")
Hi stefan-jo what version of transformers are you using? You might need version 3. See this issue [https://github.com/huggingface/transformers/issues/5931](https://github.com/huggingface/transformers/issues/5931) , which says that transformers 2.3.0 does not have callable tokenizers. ---
**Author:** ![stefan-jo](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/stefan-jo/32/64_2.png) [@stefan-jo](https://discuss.huggingface.co/u/stefan-jo)\ **Post date:** [August 18, 2020, 3:50pm UTC](https://discuss.huggingface.co/t/tutorial-fine-tuning-with-custom-datasets-sentiment-ner-and-question-answering/733/7 "2020-08-18T15:50:11Z")
Yes, that was it. I installed from source, restarted the kernel and now it’s working 🙂 Thanks @rgwatwormhill ---
**Author:** ![stefan-jo](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/stefan-jo/32/64_2.png) [@stefan-jo](https://discuss.huggingface.co/u/stefan-jo)\ **Post date:** [August 19, 2020, 9:53am UTC](https://discuss.huggingface.co/t/tutorial-fine-tuning-with-custom-datasets-sentiment-ner-and-question-answering/733/9 "2020-08-19T09:53:50Z")
I have two (very basic) questions: 1. I suppose in the tutorial the entire model is being fine-tuned at once. Is there an easy way to first train only the classification head and only then unfreeze the entire model? 2. Is the classification head in `BertForSequenceClassification` pre-trained or initialized randomly on top of `BertModel`? If pre-trained, which task/dataset has been used for pre-training? Note: I’ve been using BERT instead of DistilBERT, but I guess the same applies to both. ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\ **Post date:** [August 19, 2020, 12:37pm UTC](https://discuss.huggingface.co/t/tutorial-fine-tuning-with-custom-datasets-sentiment-ner-and-question-answering/733/10 "2020-08-19T12:37:19Z")
For 1, you can look in the [training tutorial](https://huggingface.co/transformers/training.html#freezing-the-encoder) where there is an example in PyTorch. For 2, the head is initialized randomly since we are using a checkpoint of the base model, it would be pretrained if we used a checkpoint that has been fine-tuned for sequence classification like `distilbert-base-uncased-finetuned-sst-2-english`. ---
**Author:** ![stefan-jo](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/stefan-jo/32/64_2.png) [@stefan-jo](https://discuss.huggingface.co/u/stefan-jo)\ **Post date:** [August 19, 2020, 12:58pm UTC](https://discuss.huggingface.co/t/tutorial-fine-tuning-with-custom-datasets-sentiment-ner-and-question-answering/733/11 "2020-08-19T12:58:09Z")
Thank you for your answer! I’ll check out the tutorial. ---
**Author:** ![abdallah197](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/abdallah197/32/3973_2.png) [@abdallah197](https://discuss.huggingface.co/u/abdallah197)\ **Post date:** [November 19, 2020, 3:50pm UTC](https://discuss.huggingface.co/t/tutorial-fine-tuning-with-custom-datasets-sentiment-ner-and-question-answering/733/12 "2020-11-19T15:50:49Z")
Hi @sgugger, @joeddav. In case using the Trainer class in the NER task along with using a compute\_metrics function ```auto # def compute_metrics(pred): # labels = pred.label_ids # preds = pred.predictions.argmax(-1) # print(labels.shape == preds.shape) # precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average='binary') # # acc = accuracy_score(labels, preds) # return { # 'f1': f1, # 'precision': precision, # 'recall': recall # } ``` I always run through an error `ValueError: Classification metrics can't handle a mix of multiclass-multioutput and multilabel-indicator targets` solutions online said that the labels and predictions might not be the same shape, but this is not the case here.the rest of the code is exactly like the tutorial. ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\ **Post date:** [November 19, 2020, 4:00pm UTC](https://discuss.huggingface.co/t/tutorial-fine-tuning-with-custom-datasets-sentiment-ner-and-question-answering/733/13 "2020-11-19T16:00:31Z")
Hi there. You should use metrics designed for NER. For instance the package `seqeval` has some that will work directly. Check the [run\_ner](https://github.com/huggingface/transformers/blob/master/examples/token-classification/run_ner.py) script to see how it’s used in `compute_metrics`. ---
**Author:** ![g3casey](https://avatars.discourse-cdn.com/v4/letter/g/e19b73/32.png) [@g3casey](https://discuss.huggingface.co/u/g3casey)\ **Post date:** [April 8, 2021, 2:44am UTC](https://discuss.huggingface.co/t/tutorial-fine-tuning-with-custom-datasets-sentiment-ner-and-question-answering/733/15 "2021-04-08T02:44:34Z")
Thanks for this tutorial @joeddav. I have reviewed your W-NUT example a few times. I was wondering if you could point me to an similar example that demonstrates how to add new labels to the classification. For example, I would like to classify address information. ---
**Author:** ![abercher](https://avatars.discourse-cdn.com/v4/letter/a/e495f1/32.png) [@abercher](https://discuss.huggingface.co/u/abercher)\ **Post date:** [April 17, 2021, 12:54pm UTC](https://discuss.huggingface.co/t/tutorial-fine-tuning-with-custom-datasets-sentiment-ner-and-question-answering/733/16 "2021-04-17T12:54:54Z")
Hello everyone, I’m trying to reproduce the IMDB sentiment analysis model of the tutorial. I already had the data on my personal machine under a slight different form, but after transforming it into a list of strings, each one containing a review, it should be the same. But I get an error. I don’t know if I did something wrong or if the library changed since the tutorial was created. My code looks like this (the script isn’t finished as it doesn’t contain the evaluation part but the training is already failing): ``` import os import pandas as pd import pickle from transformers import DistilBertTokenizerFast, DistilBertForSequenceClassification, Trainer, TrainingArguments import torch from torch import cuda from torch.utils.data import Dataset import time from transformers.integrations import TensorBoardCallback from train_and_eval_lstm import print_evaluation_scores device = 'cuda' if cuda.is_available() else 'cpu' def main(): clean_text_train_fn = os.path.join(os.getcwd(), "Transformed_data/clean_text_train.csv") df_clean_text_train = pd.read_csv(clean_text_train_fn) clean_text_train = df_clean_text_train["clean_text"].tolist() clean_text_valid_fn = os.path.join(os.getcwd(), "Transformed_data/clean_text_valid.csv") df_clean_text_valid = pd.read_csv(clean_text_valid_fn) clean_text_valid = df_clean_text_valid["clean_text"].tolist() clean_text_test_fn = os.path.join(os.getcwd(), "Transformed_data/clean_text_test.csv") df_clean_text_test = pd.read_csv(clean_text_test_fn) clean_text_test = df_clean_text_test["clean_text"].tolist() ## Load binary labels y_binary_train_fn = os.path.join(os.getcwd(), 'Transformed_data/Labels/y_binary_train.pkl') with open(y_binary_train_fn, mode='rb') as f: y_binary_train = pickle.load(f) y_binary_valid_fn = os.path.join(os.getcwd(), 'Transformed_data/Labels/y_binary_valid.pkl') with open(y_binary_valid_fn, mode='rb') as f: y_binary_valid = pickle.load(f) y_binary_test_fn = os.path.join(os.getcwd(), 'Transformed_data/Labels/y_binary_test.pkl') with open(y_binary_test_fn, mode='rb') as f: y_binary_test = pickle.load(f) ## Using pretrained Tokenizer model_name = 'distilbert-base-uncased' tokenizer = DistilBertTokenizerFast.from_pretrained(model_name) start = time.time() train_encodings = tokenizer(clean_text_train, truncation=True, padding=True) stop = time.time() print(f"Time to tokenize training set: {stop - start}") val_encodings = tokenizer(clean_text_valid, truncation=True, padding=True) test_encodings = tokenizer(clean_text_test, truncation=True, padding=True) class IMDbDataset(torch.utils.data.Dataset): def __init__ (self, encodings, labels): self.encodings = encodings self.labels = labels def __getitem__ (self, idx): item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()} item['labels'] = torch.tensor(self.labels[idx]) return item def __len__ (self): return len(self.labels) n_toy = 500 toy_dataset = IMDbDataset(train_encodings[:n_toy], y_binary_train[:n_toy]) train_dataset = IMDbDataset(train_encodings, y_binary_train) val_dataset = IMDbDataset(val_encodings, y_binary_valid) test_dataset = IMDbDataset(test_encodings, y_binary_test) training_args = TrainingArguments( output_dir='./results', # output directory num_train_epochs=1, # total number of training epochs per_device_train_batch_size=16, # batch size per device during training per_device_eval_batch_size=64, # batch size for evaluation warmup_steps=500, # number of warmup steps for learning rate scheduler weight_decay=0.01, # strength of weight decay logging_dir='./logs', # directory for storing logs logging_steps=10, ) model = DistilBertForSequenceClassification.from_pretrained("distilbert-base-uncased") trainer = Trainer( model=model, # the instantiated 🤗 Transformers model to be trained args=training_args, # training arguments, defined above train_dataset=toy_dataset, # training dataset eval_dataset=val_dataset, # evaluation dataset callbacks=[TensorBoardCallback] ) start = time.time() trainer.train() stop = time.time() print(f"Time to train the model: {stop-start}") model_dir = os.path.join(os.getcwd(), "Saved_models") model.save_pretrained(model_dir) if __name__ == " __main__": main() ``` And if I execute it, I receive the following error: ``` File "/home/me/Documents/CS_Programming_Machine_Learning/Projects/IMDB_sentiment_analysis_2/Comparison_models/train_and_eval_DistilBERT.py", line 61, in __getitem__ item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()} AttributeError: 'list' object has no attribute 'items' ``` If I use debugging, I see that indeed, self.encodings is a python list. I guess that I can fix it on my own, but I was wondering if I did something wrong or if the docs are outdated (I use version 4.5.1 of HF Transformers). ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\ **Post date:** [April 17, 2021, 1:28pm UTC](https://discuss.huggingface.co/t/tutorial-fine-tuning-with-custom-datasets-sentiment-ner-and-question-answering/733/17 "2021-04-17T13:28:54Z")
The problem lies in your added line: ```auto toy_dataset = IMDbDataset(train_encodings[:n_toy], y_binary_train[:n_toy]) ``` The `train_encodings` is a dictionary (with some added properties that let you take a slice like this) so you should do some thing like ```auto toy_encodings = {k: v[:n_toy] for k, v in train_encodings.items()} ``` to keep a dictionary. ---
**Author:** ![abercher](https://avatars.discourse-cdn.com/v4/letter/a/e495f1/32.png) [@abercher](https://discuss.huggingface.co/u/abercher)\ **Post date:** [April 17, 2021, 7:11pm UTC](https://discuss.huggingface.co/t/tutorial-fine-tuning-with-custom-datasets-sentiment-ner-and-question-answering/733/18 "2021-04-17T19:11:33Z")
Thank you very much for your answer sgugger. Your suggestion solved my problem! I’m sorry I didn’t realize the source of this issue myself 😅 Have a nice day! ---
**Author:** ![dineshmane](https://avatars.discourse-cdn.com/v4/letter/d/e495f1/32.png) [@dineshmane](https://discuss.huggingface.co/u/dineshmane)\ **Post date:** [July 7, 2022, 6:53am UTC](https://discuss.huggingface.co/t/tutorial-fine-tuning-with-custom-datasets-sentiment-ner-and-question-answering/733/19 "2022-07-07T06:53:25Z")
@joeddav, could you please update the tutorial link? I’m getting 404 error. Thank you 🙂 ---
**Author:** ![akshay0710](https://avatars.discourse-cdn.com/v4/letter/a/49beb7/32.png) [@akshay0710](https://discuss.huggingface.co/u/akshay0710)\ **Post date:** [August 19, 2022, 12:03am UTC](https://discuss.huggingface.co/t/tutorial-fine-tuning-with-custom-datasets-sentiment-ner-and-question-answering/733/20 "2022-08-19T00:03:22Z")
@joeddav Thank you for the tutorial. I was trying to replicate the finetuning code with a different dataset and it worked. But when I changed the pretrainedmodel from Distilbert to something else like Roberta or XlNet, I got an error in the encoding function. This is the encoding function: ```auto def encode_tags(tags, encodings): labels = [[tag2id[tag] for tag in doc] for doc in tags] encoded_labels = [] for doc_labels, doc_offset in zip(labels, encodings.offset_mapping): # create an empty array of -100 doc_enc_labels = np.ones(len(doc_offset),dtype=int) * -100 arr_offset = np.array(doc_offset) # set labels whose first offset position is 0 and the second is not 0 doc_enc_labels[(arr_offset[:,0] == 0) & (arr_offset[:,1] != 0)] = doc_labels encoded_labels.append(doc_enc_labels.tolist()) return encoded_labels ``` It didn’t throw an error if I use BERT or DISTILBERT as the pretrained model and tokenizer, but if I use some other model in its place - This was the error that I got: Traceback (most recent call last): File “huggingFace\_NER.py”, line 187, in train\_labels = encode\_tags(train\_tags, train\_encodings) File “huggingFace\_NER.py”, line 70, in encode\_tags doc\_enc\_labels[(arr\_offset[:,0] == 0) & (arr\_offset[:,1] != 0)] = doc\_labels ValueError: NumPy boolean array indexing assignment cannot assign 100 input values to the 80 output values where the mask is true ---
**Author:** ![jaymojnidar](https://avatars.discourse-cdn.com/v4/letter/j/58956e/32.png) [@jaymojnidar](https://discuss.huggingface.co/u/jaymojnidar)\ **Post date:** [September 15, 2023, 5:34pm UTC](https://discuss.huggingface.co/t/tutorial-fine-tuning-with-custom-datasets-sentiment-ner-and-question-answering/733/21 "2023-09-15T17:34:27Z")
The link “[https://huggingface.co/transformers/master/custom\_datasets.html](https://huggingface.co/transformers/master/custom_datasets.html)” is giving me 404. Is the tutorial removed for some reason? ---
**Author:** ![armagetiton](https://avatars.discourse-cdn.com/v4/letter/a/ecd19e/32.png) [@armagetiton](https://discuss.huggingface.co/u/armagetiton)\ **Post date:** [February 12, 2024, 5:12pm UTC](https://discuss.huggingface.co/t/tutorial-fine-tuning-with-custom-datasets-sentiment-ner-and-question-answering/733/22 "2024-02-12T17:12:01Z")
I have the same question. Where did the tutorial go?