# Bert ner classifier **URL:** \ **Category:** Beginners\ **Created:** [April 29, 2021, 9:02am UTC](https://discuss.huggingface.co/t/bert-ner-classifier/5847 "2021-04-29T09:02:22Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![yucheng](https://avatars.discourse-cdn.com/v4/letter/y/eada6e/32.png) [@yucheng](https://discuss.huggingface.co/u/yucheng)\ **Post date:** [April 29, 2021, 9:02am UTC](https://discuss.huggingface.co/t/bert-ner-classifier/5847/1 "2021-04-29T09:02:23Z")
hi, I fine-tune the bert on NER task, and huggingface add a linear classifier on the top of model. I want to know more details about classifier architecture. e.g. fully connected + softmax… thank you for your help ---
**Author:** ![sbmaruf](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sbmaruf/32/1528_2.png) [@sbmaruf](https://discuss.huggingface.co/u/sbmaruf)\ **Post date:** [April 29, 2021, 3:47pm UTC](https://discuss.huggingface.co/t/bert-ner-classifier/5847/2 "2021-04-29T15:47:10Z")
Hi! Can you be a little bit more specific about your query? Just to give you a head start, In general, NER is a sequence labeling (a.k.a token classification) problem. The additional stuff you may have to consider for NER is, for a word that is divided into multiple tokens by bpe or sentencepiece like model, you use the first token as your reference token that you want to predict. Since all the tokens are connected via self-attention you won’t have problem not predicting the rest of the bpe tokens of a word. In PyTorch, you can ignore [computing loss](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html) (see _ignore\_index_ argument) of those tokens by providing -100 as a label to those tokens (life is so easy with pytorch 😉 ). Apart from that, I didn’t find any more additional complexity in the training NER model. Some other implementation details you need to check, 1. One important Note: So far I remember (please verify), In conll, german or dutch dataset there are 2-3 long sentences in the test dataset. Sequence labeling doesn’t work like sentiment analysis. You need to make sure your sentence is not cut down by the max\_sequence\_len argument of the Language Model’s tokenizer. Otherwise, you will see a little bit of discrepency in your test F1 sore. An easier hack for this problem is to divide the sentence into smaller parts and predict them one by one and finally merge them. 2. Imo Self-attention and CRF layer is theoretically different but in application some of the problem that CRF solved in prior model, self-attention can also solve them (because they create a fully connected graph). So using softmax is more preferable than a CRF layer. 3. The score that the original [BERT paper](https://arxiv.org/pdf/1810.04805.pdf) reported are not reproducible and comparable with most of the papers since they used document level NER fine-tuning. If you still have query about the architecture you can follow this, > **[Sequence Tagging with Tensorflow](https://guillaumegenthial.github.io/sequence-tagging-with-tensorflow.html)** > > GloVe + character embeddings + bi-LSTM + CRF for Sequence Tagging (Named Entity Recognition, NER, POS) - NLP example of bidirectionnal RNN and CRF in Tensorflow you only have to replace hierarchical rnn with transformer as the encoder. You can check the following paper’s for more info, 1. [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT - ACL Anthology](https://www.aclweb.org/anthology/D19-1077/) 2. [https://arxiv.org/pdf/2007.07683.pdf](https://arxiv.org/pdf/2007.07683.pdf) 3. [https://arxiv.org/pdf/2004.12440.pdf](https://arxiv.org/pdf/2004.12440.pdf) 4. [https://arxiv.org/pdf/2004.13240.pdf](https://arxiv.org/pdf/2004.13240.pdf) (beware, this is my publication. I may be biased, still not accepted) Please let me know if you have more queries. ---
**Author:** ![yucheng](https://avatars.discourse-cdn.com/v4/letter/y/eada6e/32.png) [@yucheng](https://discuss.huggingface.co/u/yucheng)\ **Post date:** [April 30, 2021, 2:57am UTC](https://discuss.huggingface.co/t/bert-ner-classifier/5847/3 "2021-04-30T02:57:34Z")
Thank you very much for your explanation. It let me learning a lot. I print my model ,then I find it has a classifier. I want to know what is the architecture. ![擷取.PNG](https://us1.discourse-cdn.com/hellohellohello/original/2X/b/bf3429cb1eff3ea82b6161f36d17784cb7231192.jpeg) ---
**Author:** ![sbmaruf](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sbmaruf/32/1528_2.png) [@sbmaruf](https://discuss.huggingface.co/u/sbmaruf)\ **Post date:** [April 30, 2021, 3:14am UTC](https://discuss.huggingface.co/t/bert-ner-classifier/5847/4 "2021-04-30T03:14:22Z")
As you can see, the classifier is a single dense layer. It is probably pointing out from here if you are using `BertForSequenceClassification`, [transformers/modeling\_bert.py at b29eb247d39b56d903ea36c4f6c272a7bb0c0b4c · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/b29eb247d39b56d903ea36c4f6c272a7bb0c0b4c/src/transformers/models/bert/modeling_bert.py#L1462) If you are using `BertForTokenClassification`, it is pointing out here, > For setting up num\_of\_label, please change the variable in your config file. ---
**Author:** ![yucheng](https://avatars.discourse-cdn.com/v4/letter/y/eada6e/32.png) [@yucheng](https://discuss.huggingface.co/u/yucheng)\ **Post date:** [May 3, 2021, 7:21am UTC](https://discuss.huggingface.co/t/bert-ner-classifier/5847/5 "2021-05-03T07:21:08Z")
hi, can i think self.classifier = nn.Linear(config.hidden\_size, config.num\_labels) as a fully-connected layer. input dimension is config.hidden\_size and out dinension is config.num\_labels. as shown ![tfdl_0401](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/d97e02671bfe7dbf9e59efdcedeb301ae75e3b6a.png) ---
**Author:** ![sbmaruf](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sbmaruf/32/1528_2.png) [@sbmaruf](https://discuss.huggingface.co/u/sbmaruf)\ **Post date:** [May 3, 2021, 11:54am UTC](https://discuss.huggingface.co/t/bert-ner-classifier/5847/6 "2021-05-03T11:54:43Z")
yes, this is just a linear layer.