# Using EXTREMELY small dataset to finetune BERT **URL:** \ **Category:** 🤗Transformers\ **Created:** [July 30, 2021, 1:37pm UTC](https://discuss.huggingface.co/t/using-extremely-small-dataset-to-finetune-bert/8847 "2021-07-30T13:37:48Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![JoshuaP](https://avatars.discourse-cdn.com/v4/letter/j/49beb7/32.png) [@JoshuaP](https://discuss.huggingface.co/u/JoshuaP)\ **Post date:** [July 30, 2021, 1:37pm UTC](https://discuss.huggingface.co/t/using-extremely-small-dataset-to-finetune-bert/8847/1 "2021-07-30T13:37:48Z")
Hi, I have a domain-specific language classification problem that I am attempting to use a bert model for. My approach has been to take the standard pretrained bert model and run further unsupervised learning using domain-specific language corpora (using [TSDAE training from the Sentence-Transformer framework](https://www.sbert.net/examples/unsupervised_learning/TSDAE/README.html)). I am now trying to take this domain-trained model and finetune it for a classification task. The problem is I only have an extremely small labelled dataset (~1000 samples), I have been running a few training experiments and surprisingly have received very good results that I am very sceptical of. The task is to take natural language text and classify it to 1 of 5 classes. Here is my training setup: ```auto criterion = nn.CrossEntropyLoss() optimizer = optim.SGD(model.parameters(), lr=0.0001, momentum=0.9) ``` ```auto class BertClassification(nn.Module): def __init__ (self): super(BertClassification, self). __init__ () self.bert = BertModel.from_pretrained("TSDAE_model/0_Transformer") self.to_class = nn.Linear(768, 5) def forward(self, x): x = self.bert(x)[0][:,0,:] x = self.to_class(x) return x ``` And here are the training results: ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/f/fe4d9f468cbf93b8842f37cbc3681c326da0685e.png) I am very unsure of how trust worthy these results are as the dataset is so small. I have also tried freezing the bert weights and just training the `self.to_class` linear layer (~4000 params) but the model peaks at only about 50% accuracy. I was hoping someone may be able to help me decide if this is an appropriate training strategy for this dataset or if maybe I should look at alternatives. Thanks! ---
**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\ **Post date:** [July 31, 2021, 12:41am UTC](https://discuss.huggingface.co/t/using-extremely-small-dataset-to-finetune-bert/8847/2 "2021-07-31T00:41:13Z")
hey @JoshuaP, are your 5 classes equally balanced? if not, you might be better off charting a metric like the f1-score which tends to be less biased by cases where you have a lot of examples in just a few classes. another idea would be to implement a baseline (e.g. the classic naive bayes 😃) and see how that compares against your transformer model. finally you could try cross-validation (with a stratified split if your classes aren’t balanced) to mitigate some of the problems that come from doing a train/test split with small datasets ---
**Author:** ![JoshuaP](https://avatars.discourse-cdn.com/v4/letter/j/49beb7/32.png) [@JoshuaP](https://discuss.huggingface.co/u/JoshuaP)\ **Post date:** [August 6, 2021, 4:55pm UTC](https://discuss.huggingface.co/t/using-extremely-small-dataset-to-finetune-bert/8847/3 "2021-08-06T16:55:46Z")
Hi @lewtun my classes aren’t balanced but i’m using a sample to get an even 20% per class in each batch. Thanks for the suggestions, I will try and implement a simple baseline and use cross-validation! ---
**Author:** ![LadyHangaku](https://avatars.discourse-cdn.com/v4/letter/l/71c47a/32.png) [@LadyHangaku](https://discuss.huggingface.co/u/LadyHangaku)\ **Post date:** [October 28, 2021, 10:02am UTC](https://discuss.huggingface.co/t/using-extremely-small-dataset-to-finetune-bert/8847/4 "2021-10-28T10:02:23Z")
I got no suggestion but rather a question 😅 Would you mind showing me your whole code? I am currently trying to write a token classifier / NER with also very little data. But I’m not quite sure how to define the optimizer only with the linear layers and freezing all BERT weights. ---
**Author:** ![rosenjcb](https://avatars.discourse-cdn.com/v4/letter/r/e79b87/32.png) [@rosenjcb](https://discuss.huggingface.co/u/rosenjcb)\ **Post date:** [October 28, 2021, 5:07pm UTC](https://discuss.huggingface.co/t/using-extremely-small-dataset-to-finetune-bert/8847/5 "2021-10-28T17:07:33Z")
~1000 samples is tiny? I’ve had a good experience training classification models with 1k samples honestly - but nothing below that. That accuracy/loss score looks a lot like our results as well. ---
**Author:** ![savasci](https://avatars.discourse-cdn.com/v4/letter/s/e9c0ed/32.png) [@savasci](https://discuss.huggingface.co/u/savasci)\ **Post date:** [January 7, 2023, 10:53pm UTC](https://discuss.huggingface.co/t/using-extremely-small-dataset-to-finetune-bert/8847/6 "2023-01-07T22:53:21Z")
Hi, I have a similar problem and it seems u can help me! My data set has 500 samples (with 250 samples for each class). Surprisingly, I trained a lightgbm classifier on the dataset and it scored 1 for all binary classification metrics!! (precision, recall, and f1 score). Though its scores on the valid set were not good enough. On the other hand, I trained a Roberta classifier and after 50 epochs, it reached a score of 0.5 !!! In other words, my transformer couldn’t learn as much as a lightgbm model!! I think there is a subtle mistake that I can’t figure out! Could you please give me suggestions? ---
**Author:** ![BbKWlf](https://avatars.discourse-cdn.com/v4/letter/b/c67d28/32.png) [@BbKWlf](https://discuss.huggingface.co/u/BbKWlf)\ **Post date:** [February 1, 2023, 3:24am UTC](https://discuss.huggingface.co/t/using-extremely-small-dataset-to-finetune-bert/8847/7 "2023-02-01T03:24:09Z")
Hello, I want to use k-fold cross validation, but I’m not sure how to do that. Can you help me or point me in a right direction? Thank you.