# AttributeError: 'str' object has no attribute 'item' - Bert Fine tuning

**URL:** <https://discuss.huggingface.co/t/attributeerror-str-object-has-no-attribute-item-bert-fine-tuning/13884>\
**Category:** 🤗Transformers\
**Created:** [January 19, 2022, 6:22am UTC](https://discuss.huggingface.co/t/attributeerror-str-object-has-no-attribute-item-bert-fine-tuning/13884 "2022-01-19T06:22:29Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![ayush488](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ayush488/32/2561_2.png) [@ayush488](https://discuss.huggingface.co/u/ayush488)\
**Post date:** [January 19, 2022, 6:22am UTC](https://discuss.huggingface.co/t/attributeerror-str-object-has-no-attribute-item-bert-fine-tuning/13884/1 "2022-01-19T06:22:29Z")

</div>

Hi, I am working on a intent classification problem, so I am fine tuning bert for it,

here is my code:-

```auto
import random
import numpy as np

# This training code is based on the `run_glue.py` script here:
# https://github.com/huggingface/transformers/blob/5bfcd0485ece086ebcbed2d008813037968a9e58/examples/run_glue.py#L128

# Set the seed value all over the place to make this reproducible.
seed_val = 42

random.seed(seed_val)
np.random.seed(seed_val)
torch.manual_seed(seed_val)
torch.cuda.manual_seed_all(seed_val)

# We'll store a number of quantities such as training and validation loss, 
# validation accuracy, and timings.
training_stats = []

# Measure the total training time for the whole run.
total_t0 = time.time()

# For each epoch...
for epoch_i in range(0, epochs):
    
    # ========================================
    # Training
    # ========================================
    
    # Perform one full pass over the training set.

    print("")
    print('======== Epoch {:} / {:} ========'.format(epoch_i + 1, epochs))
    print('Training...')

    # Measure how long the training epoch takes.
    t0 = time.time()

    # Reset the total loss for this epoch.
    total_train_loss = 0

    # Put the model into training mode. Don't be mislead--the call to 
    # `train` just changes the *mode*, it doesn't *perform* the training.
    # `dropout` and `batchnorm` layers behave differently during training
    # vs. test (source: https://stackoverflow.com/questions/51433378/what-does-model-train-do-in-pytorch)
    model.train()

    # For each batch of training data...
    for step, batch in enumerate(train_dataloader):

        # Progress update every 40 batches.
        if step % 40 == 0 and not step == 0:
            # Calculate elapsed time in minutes.
            elapsed = format_time(time.time() - t0)
            
            # Report progress.
            print(' Batch {:>5,} of {:>5,}. Elapsed: {:}.'.format(step, len(train_dataloader), elapsed))

        # Unpack this training batch from our dataloader. 
        #
        # As we unpack the batch, we'll also copy each tensor to the GPU using the 
        # `to` method.
        #
        # `batch` contains three pytorch tensors:
        # [0]: input ids 
        # [1]: attention masks
        # [2]: labels 
        b_input_ids = batch[0].to(device)
        b_input_mask = batch[1].to(device)
        b_labels = batch[2].to(device)

        # Always clear any previously calculated gradients before performing a
        # backward pass. PyTorch doesn't do this automatically because 
        # accumulating the gradients is "convenient while training RNNs". 
        # (source: https://stackoverflow.com/questions/48001598/why-do-we-need-to-call-zero-grad-in-pytorch)
        model.zero_grad()        

        # Perform a forward pass (evaluate the model on this training batch).
        # The documentation for this `model` function is here: 
        # https://huggingface.co/transformers/v2.2.0/model_doc/bert.html#transformers.BertForSequenceClassification
        # It returns different numbers of parameters depending on what arguments
        # arge given and what flags are set. For our useage here, it returns
        # the loss (because we provided labels) and the "logits"--the model
        # outputs prior to activation.
        loss, logits = model(b_input_ids, 
                             token_type_ids=None, 
                             attention_mask=b_input_mask, 
                             labels=b_labels)

        # Accumulate the training loss over all of the batches so that we can
        # calculate the average loss at the end. `loss` is a Tensor containing a
        # single value; the `.item()` function just returns the Python value 
        # from the tensor. 
        print(loss)
        total_train_loss += loss.item()

        # Perform a backward pass to calculate the gradients.
        loss.backward()

        # Clip the norm of the gradients to 1.0.
        # This is to help prevent the "exploding gradients" problem.
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)

        # Update parameters and take a step using the computed gradient.
        # The optimizer dictates the "update rule"--how the parameters are
        # modified based on their gradients, the learning rate, etc.
        optimizer.step()

        # Update the learning rate.
        scheduler.step()

    # Calculate the average loss over all of the batches.
    avg_train_loss = total_train_loss / len(train_dataloader)            
    
    # Measure how long this epoch took.
    training_time = format_time(time.time() - t0)

    print("")
    print(" Average training loss: {0:.2f}".format(avg_train_loss))
    print(" Training epcoh took: {:}".format(training_time))
        
    # ========================================
    # Validation
    # ========================================
    # After the completion of each training epoch, measure our performance on
    # our validation set.

    print("")
    print("Running Validation...")

    t0 = time.time()

    # Put the model in evaluation mode--the dropout layers behave differently
    # during evaluation.
    model.eval()

    # Tracking variables 
    total_eval_accuracy = 0
    total_eval_loss = 0
    nb_eval_steps = 0

    # Evaluate data for one epoch
    for batch in validation_dataloader:
        
        # Unpack this training batch from our dataloader. 
        #
        # As we unpack the batch, we'll also copy each tensor to the GPU using 
        # the `to` method.
        #
        # `batch` contains three pytorch tensors:
        # [0]: input ids 
        # [1]: attention masks
        # [2]: labels 
        b_input_ids = batch[0].to(device)
        b_input_mask = batch[1].to(device)
        b_labels = batch[2].to(device)
        
        # Tell pytorch not to bother with constructing the compute graph during
        # the forward pass, since this is only needed for backprop (training).
        with torch.no_grad():        

            # Forward pass, calculate logit predictions.
            # token_type_ids is the same as the "segment ids", which 
            # differentiates sentence 1 and 2 in 2-sentence tasks.
            # The documentation for this `model` function is here: 
            # https://huggingface.co/transformers/v2.2.0/model_doc/bert.html#transformers.BertForSequenceClassification
            # Get the "logits" output by the model. The "logits" are the output
            # values prior to applying an activation function like the softmax.
            (loss, logits) = model(b_input_ids, 
                                   token_type_ids=None, 
                                   attention_mask=b_input_mask,
                                   labels=b_labels)
            
        # Accumulate the validation loss.
        total_eval_loss += loss.item()

        # Move logits and labels to CPU
        logits = logits.detach().cpu().numpy()
        label_ids = b_labels.to('cpu').numpy()

        # Calculate the accuracy for this batch of test sentences, and
        # accumulate it over all batches.
        total_eval_accuracy += flat_accuracy(logits, label_ids)
        

    # Report the final accuracy for this validation run.
    avg_val_accuracy = total_eval_accuracy / len(validation_dataloader)
    print(" Accuracy: {0:.2f}".format(avg_val_accuracy))

    # Calculate the average loss over all of the batches.
    avg_val_loss = total_eval_loss / len(validation_dataloader)
    
    # Measure how long the validation run took.
    validation_time = format_time(time.time() - t0)
    
    print(" Validation Loss: {0:.2f}".format(avg_val_loss))
    print(" Validation took: {:}".format(validation_time))

    # Record all statistics from this epoch.
    training_stats.append(
        {
            'epoch': epoch_i + 1,
            'Training Loss': avg_train_loss,
            'Valid. Loss': avg_val_loss,
            'Valid. Accur.': avg_val_accuracy,
            'Training Time': training_time,
            'Validation Time': validation_time
        }
    )

print("")
print("Training complete!")

print("Total training took {:} (h:mm:ss)".format(format_time(time.time()-total_t0)))

```

I am getting a error:-

```auto
AttributeError Traceback (most recent call last)
<ipython-input-27-a6f23d2754c8> in <module>()
     92 # from the tensor.
     93 print(loss)
---> 94 total_train_loss += loss.item()
     95 
     96 # Perform a backward pass to calculate the gradients.

AttributeError: 'str' object has no attribute 'item'

```

---

<div class="post-metadata">

**Author:** ![ayush488](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ayush488/32/2561_2.png) [@ayush488](https://discuss.huggingface.co/u/ayush488)\
**Post date:** [January 19, 2022, 6:57am UTC](https://discuss.huggingface.co/t/attributeerror-str-object-has-no-attribute-item-bert-fine-tuning/13884/2 "2022-01-19T06:57:37Z")

</div>

Here is the collab:- [Google Colab](https://colab.research.google.com/drive/1kyh0kLD806q6PFSIlpllV37DNnzqYYct?usp=sharing)

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [January 19, 2022, 1:44pm UTC](https://discuss.huggingface.co/t/attributeerror-str-object-has-no-attribute-item-bert-fine-tuning/13884/3 "2022-01-19T13:44:29Z")

</div>

You can’t write

```python
loss, logits = model(...)

```

since v4, as the output of the model is a dictionary, so this will give you the keys (hence the fact your loss is a string) and not the values.

You can do:

```python
loss, logits = model(...).to_tuple()

```

or

```python
outputs = model(...)
loss = outputs.loss
logits = outputs.logits

```

---

<div class="post-metadata">

**Author:** ![ayush488](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ayush488/32/2561_2.png) [@ayush488](https://discuss.huggingface.co/u/ayush488)\
**Post date:** [January 19, 2022, 7:09pm UTC](https://discuss.huggingface.co/t/attributeerror-str-object-has-no-attribute-item-bert-fine-tuning/13884/4 "2022-01-19T19:09:39Z")

</div>

Thanks, @sgugger ,

I am having another question, I have trained my model, Now I trained as well, Now I need to make predictions,

```auto
sents = ["how would you say fly in italian"]

encoded_dicts = tokenizer.encode_plus(
                        sents, # Sentence to encode.
                        add_special_tokens = True, # Add '[CLS]' and '[SEP]'
                        max_length = 64, # Pad & truncate all sentences.
                        pad_to_max_length = True,
                        return_attention_mask = True, # Construct attn. masks.
                        return_tensors = 'pt', # Return pytorch tensors.
                   )
    
    # Add the encoded sentence to the list.    
one_in = encoded_dicts['input_ids'].to(device)
    
    # And its attention mask (simply differentiates padding from non-padding).
atten_in = encoded_dicts['attention_mask'].to(device)
outputs = model(one_in, token_type_ids=None, 
                attention_mask=atten_in)

logits = outputs[0]
logits = logits.detach().cpu().numpy()

```

I am using above code to make predictions, How do I know which is my category as I have 151 categories to be predicted, I used LabelEncoder to encode the intent column, so how do I decide which Intent it is?

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/8/8e71afff45f3ba7fa033a80e88a315014a6ac541.png)

Colab is here:- [Google Colab](https://colab.research.google.com/drive/1kyh0kLD806q6PFSIlpllV37DNnzqYYct?usp=sharing)

I have another question, I was using PyTorch Lightning, so below is the code,

```auto
import numpy as np
import pandas as pd

from tqdm import tqdm
import torch
from transformers import BertTokenizer
from torch.utils.data import Dataset, DataLoader
import pytorch_lightning as pl
from pytorch_lightning.callbacks import ModelCheckpoint
from pytorch_lightning.callbacks.early_stopping import EarlyStopping

from transformers import BertForSequenceClassification, AdamW, BertConfig

```

I am getting

```auto
ImportError: cannot import name 'TFPreTrainedModel' from 'transformers' (E:\Anaconda\lib\site-packages\transformers\ __init__.py)

```

---

<div class="post-metadata">

**Author:** ![ayush488](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ayush488/32/2561_2.png) [@ayush488](https://discuss.huggingface.co/u/ayush488)\
**Post date:** [January 20, 2022, 7:14pm UTC](https://discuss.huggingface.co/t/attributeerror-str-object-has-no-attribute-item-bert-fine-tuning/13884/5 "2022-01-20T19:14:42Z")

</div>

@sgugger Can u take a look at the problem, It is not yet solved!