# Sentence Pair Classification

**URL:** <https://discuss.huggingface.co/t/sentence-pair-classification/17540>\
**Category:** Intermediate\
**Created:** [May 4, 2022, 10:51pm UTC](https://discuss.huggingface.co/t/sentence-pair-classification/17540 "2022-05-04T22:51:05Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![xap](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/xap/32/4537_2.png) [@xap](https://discuss.huggingface.co/u/xap)\
**Post date:** [May 4, 2022, 10:51pm UTC](https://discuss.huggingface.co/t/sentence-pair-classification/17540/1 "2022-05-04T22:51:05Z")

</div>

I am doing a sentence pair classification where based on two sentences I have to classify the label of the sentence.

After I created my train and test data I converted both the sentences to a list and applied BERT tokenizer as

```auto
train_encode = tokenizer(train1, train2,padding="max_length",truncation=True)
test_encode = tokenizer(test1, test2,padding="max_length",truncation=True)

```

After that, I extracted the only required things as

```auto
train_seq = torch.tensor(train_encode['input_ids'])
train_mask = torch.tensor(train_encode['attention_mask'])
train_token = torch.tensor(train_encode['token_type_ids'])

```

I did the same for the test data and created a data loader as

```auto
train_data = TensorDataset(train_seq, train_mask, train_token, train_y)
train_dataloader = DataLoader(train_data, batch_size=32,shuffle=True)

```

Created model as

```auto
model = BertForSequenceClassification.from_pretrained(checkpoint, num_labels=5)

```

and the training loop is

```auto
train_losses = []
num_mb_train = len(train_dataloader)

import torch.nn as nn
import numpy as np

EPOCHS = 5
criterion = nn.CrossEntropyLoss()

# empty list to save model predictions
total_preds=[]

for epoch in range(EPOCHS):
  train_loss = 0

  
  for step,batch in enumerate(train_dataloader):
    optimizer.zero_grad()
    model.train()

    batch = [r.to(device) for r in batch]
    input_id,attention_mask,token_type_id,y = batch
    

    
    prediction = model(input_id,attention_mask= attention_mask,token_type_ids=token_type_id, labels=y)

    loss = prediction[0]
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    optimizer.step()
    scheduler.step()
        
    train_loss += loss.data / num_mb_train

    print("\nTrain loss after itaration %i: %f" % (epoch+1, train_loss))
    train_losses.append(train_loss.cpu())

```

Once I run my training loop it says

`RuntimeError: CUDA out of memory.`

I am not sure where I am missing and what should I do? Any help would be much appreciated.

@sgugger

---

<div class="post-metadata">

**Author:** ![phosseini](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/phosseini/32/26690_2.png) [@phosseini](https://discuss.huggingface.co/u/phosseini)\
**Post date:** [May 4, 2022, 11:50pm UTC](https://discuss.huggingface.co/t/sentence-pair-classification/17540/2 "2022-05-04T23:50:20Z")

</div>

> [@xap](#):
>
> `batch_size`

Have you tried reducing your `batch_size` and/or `max_length`? It looks like you’re running out of memory.