# BERT (CamemBERT) for Sequence Classification maps any sequence to the exact same encoding

**URL:** <https://discuss.huggingface.co/t/bert-camembert-for-sequence-classification-maps-any-sequence-to-the-exact-same-encoding/46012>\
**Category:** Models\
**Created:** [July 7, 2023, 1:52pm UTC](https://discuss.huggingface.co/t/bert-camembert-for-sequence-classification-maps-any-sequence-to-the-exact-same-encoding/46012 "2023-07-07T13:52:03Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![pltrdy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/pltrdy/32/2806_2.png) [@pltrdy](https://discuss.huggingface.co/u/pltrdy)\
**Post date:** [July 7, 2023, 1:52pm UTC](https://discuss.huggingface.co/t/bert-camembert-for-sequence-classification-maps-any-sequence-to-the-exact-same-encoding/46012/1 "2023-07-07T13:52:03Z")

</div>

Hi,

I’m trying to fine-tune sequence regression using BERT in French i.e. [CamembertForSequenceClassification](https://huggingface.co/docs/transformers/model_doc/camembert#transformers.CamembertForSequenceClassification)

I simply load the model and tokenizer passing `num_labels=1` to actually run a regression task (my labels are actually either 0 or 1, I tried either as classification task with 2 labels or 1 label regression with the same issue)

```plaintext
import transformers

model = transformers.CamembertForSequenceClassification.from_pretrained("camembert-base", num_labels=1)
tokenizer = transformers.AutoTokenizer.from_pretrained("camembert-base", num_labels=1)

def tokenize_bert(texts):
    self = model
    tokenizer_kwargs = {
        "truncation": True,
        "max_length": 512,
        "padding": True,
        "return_tensors": "pt",
    }

    return tokenizer(
        texts,
        **tokenizer_kwargs,
    )

```

But when I’m generating, all output logits are exactly the same, and I figured out the encoder was outputting the exact same representation for any sequence:

1. I check that tokenized sequence are not the same:  

2. I check that embeddings are actually different:  
 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/5/2/52ca0531da96d397920e3f3bea25ea5a2a3062d1.png)

3. But then I notice that the encoder output is exactly the same for all input.  
 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/d/2/d2339ec971943db438811683eb4267ed8bfa057a.png)

Any idea how I could fix my training (more than 50k steps, with mostly default parameters) to make it work?