# Run detectron2 for feature extraction in SageMaker notebook

**URL:** <https://discuss.huggingface.co/t/run-detectron2-for-feature-extraction-in-sagemaker-notebook/15569>\
**Category:** Amazon SageMaker\
**Created:** [March 10, 2022, 10:49am UTC](https://discuss.huggingface.co/t/run-detectron2-for-feature-extraction-in-sagemaker-notebook/15569 "2022-03-10T10:49:20Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Petrus](https://avatars.discourse-cdn.com/v4/letter/p/ecd19e/32.png) [@Petrus](https://discuss.huggingface.co/u/Petrus)\
**Post date:** [March 10, 2022, 10:49am UTC](https://discuss.huggingface.co/t/run-detectron2-for-feature-extraction-in-sagemaker-notebook/15569/1 "2022-03-10T10:49:21Z")

</div>

Dear HuggingFace/SageMaker pros,

I am trying to produce visual embeddings to fine-tune a VisualBERT model: [VisualBERT](https://huggingface.co/docs/transformers/model_doc/visual_bert)

There is one example “Generate Embeddings for VisualBERT” that uses detectron2 to do this: [VisualBERT](https://huggingface.co/docs/transformers/model_doc/visual_bert)

However, this notebook is for Colab. I can run inside Colab, but when I download the ipynb and try to run it in SageMaker I cannot download detectron2…

I try to run:  
python -m pip install ‘git+https://github.com/facebookresearch/detectron2.git’

As well as:  
git clone [GitHub - facebookresearch/detectron2: Detectron2 is a platform for object detection, segmentation and other visual recognition tasks.](https://github.com/facebookresearch/detectron2.git)  
python -m pip install -e detectron2

But both fails to install detectron2…

Would anyone have an idea on how I can run the “Generate Embeddings for VisualBERT” example inside SageMaker? (Either how to install detectron2, or a way to bypass this.)

Or would you know some other way to successfully extract the visual embedding in a way that VisualBERT would accept?

Many thanks for your help! Best, Petrus

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [March 10, 2022, 1:46pm UTC](https://discuss.huggingface.co/t/run-detectron2-for-feature-extraction-in-sagemaker-notebook/15569/2 "2022-03-10T13:46:34Z")

</div>

@Petrus you can take a look at [Sagemaker Serverless Inference for LayoutLMv2 model - #15 by mansimov](https://discuss.huggingface.co/t/sagemaker-serverless-inference-for-layoutlmv2-model/14186/15)  
to see how to install detectron2 properly.

Are you trying to fine-tune VisualBERT in a SageMaker Notebook or using the `HuggingFace.fit()` estimator=

---

<div class="post-metadata">

**Author:** ![Petrus](https://avatars.discourse-cdn.com/v4/letter/p/ecd19e/32.png) [@Petrus](https://discuss.huggingface.co/u/Petrus)\
**Post date:** [March 11, 2022, 9:39am UTC](https://discuss.huggingface.co/t/run-detectron2-for-feature-extraction-in-sagemaker-notebook/15569/3 "2022-03-11T09:39:02Z")

</div>

@philschmid Thanks a lot for your reply!

That thread was very informative! I am now able to successfully download detectron2 inside my SageMaker notbook.

What eventually worked for me to successfully download detectron2 inside SageMaker was the following:

1. Use a regular ‘python3’ version notebook.
2. Run `pip3 install torch torchvision`
3. Follow the detectron2 installation shown in this example: [Google Colab](https://colab.research.google.com/drive/16jcaJoc6bCFAQ96jDe2HwtXj7BMD_-m5#scrollTo=FsePPpwZSmqt)

I ran the following:

```auto
!pip install pyyaml==5.1

import torch
TORCH_VERSION = ".".join(torch. __version__.split(".")[:2])
CUDA_VERSION = torch. __version__.split("+")[-1]
print("torch: ", TORCH_VERSION, "; cuda: ", CUDA_VERSION)

!pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/$CUDA_VERSION/torch$TORCH_VERSION/index.html

```

1. After that I could successfully import detectron2

---

<div class="post-metadata">

**Author:** ![Petrus](https://avatars.discourse-cdn.com/v4/letter/p/ecd19e/32.png) [@Petrus](https://discuss.huggingface.co/u/Petrus)\
**Post date:** [March 11, 2022, 9:42am UTC](https://discuss.huggingface.co/t/run-detectron2-for-feature-extraction-in-sagemaker-notebook/15569/4 "2022-03-11T09:42:25Z")

</div>

@philschmid To answer your second point on whether I want to just fine-tune inside a SM notebook, or specifically use an estimator object for training, I would say that either is fine - I want to try both and go with the one that quickest helps me to have a model that I can experiment with. Given that, any particular advice you would give me on my VisualBERT model building journey?

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [March 14, 2022, 8:34am UTC](https://discuss.huggingface.co/t/run-detectron2-for-feature-extraction-in-sagemaker-notebook/15569/5 "2022-03-14T08:34:41Z")

</div>

The easiest is to experiment in the Notebook and then move to a managed Training Job when training e2e on the full dataset.

---

<div class="post-metadata">

**Author:** ![Petrus](https://avatars.discourse-cdn.com/v4/letter/p/ecd19e/32.png) [@Petrus](https://discuss.huggingface.co/u/Petrus)\
**Post date:** [March 16, 2022, 8:28am UTC](https://discuss.huggingface.co/t/run-detectron2-for-feature-extraction-in-sagemaker-notebook/15569/6 "2022-03-16T08:28:41Z")

</div>

@philschmid thank you, I took your advice and made plain experiments in the Notebook.

However, I am experiencing some troubles on putting together my multimodal text and image input such that is accepted by the model. I am now just making a small-scale test on ten data points to properly prepare the inputs.

This is what I have done:

1. Generated tokens of my text: `input_ids, token_type_ids, attention_mask`  
I used the “bert.base-cased” tokenizer for this.
2. Generated visual embeddings of my images: `visual_embeds, visual_token_type_ids, visual_attention_mask`  
I followed this example and updated it with my data to generate the visual embeddings (using the detectron2 library): [Google Colab](https://colab.research.google.com/drive/1bLGxKdldwqnMVA5x4neY7-l_8fKGWQYI?usp=sharing)
3. I put all these six tensors and the labels (also a tensor) into a dictionary and transformed it into a `Dataset`. Further, I split it into a train and test Dataset.  
Here is what my training data set looks like:

```auto
Dataset({
    features: ['labels', 'input_ids', 'token_type_ids', 'attention_mask', 'visual_embeds', 'visual_token_type_ids', 'visual_attention_mask'],
    num_rows: 8
})

```

1. I then perform training as per the below:

```auto
from transformers import BertTokenizer, VisualBertForMultipleChoice
from transformers import TrainingArguments
from transformers import Trainer
import torch

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
model = VisualBertForMultipleChoice.from_pretrained("uclanlp/visualbert-vcr")

training_args = TrainingArguments(
    output_dir="output_dir/",
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train,
    eval_dataset=test,
)

trainer.train()

```

The issue is that when I run `trainer.train()` I receive the following error message:

```auto
/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/transformers/optimization.py:309: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning
  FutureWarning,
*****Running training*****
  Num examples = 8
  Num Epochs = 3
  Instantaneous batch size per device = 8
  Total train batch size (w. parallel, distributed & accumulation) = 8
  Gradient Accumulation steps = 1
  Total optimization steps = 3
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-25-3435b262f1ae> in <module>
----> 1 trainer.train()

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/transformers/trainer.py in train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)
   1398 tr_loss_step = self.training_step(model, inputs)
   1399 else:
-> 1400 tr_loss_step = self.training_step(model, inputs)
   1401 
   1402 if (

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/transformers/trainer.py in training_step(self, model, inputs)
   1982 
   1983 with self.autocast_smart_context_manager():
-> 1984 loss = self.compute_loss(model, inputs)
   1985 
   1986 if self.args.n_gpu > 1:

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/transformers/trainer.py in compute_loss(self, model, inputs, return_outputs)
   2014 else:
   2015 labels = None
-> 2016 outputs = model(**inputs)
   2017 # Save past state if it exists
   2018 # TODO: this needs to be fixed and made cleaner later.

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725 result = self._slow_forward(*input, **kwargs)
    726 else:
--> 727 result = self.forward(*input, **kwargs)
    728 for hook in itertools.chain(
    729 _global_forward_hooks.values(),

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/transformers/models/visual_bert/modeling_visual_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, visual_embeds, visual_attention_mask, visual_token_type_ids, image_text_alignment, output_attentions, output_hidden_states, return_dict, labels)
   1143 output_attentions=output_attentions,
   1144 output_hidden_states=output_hidden_states,
-> 1145 return_dict=return_dict,
   1146 )
   1147 

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725 result = self._slow_forward(*input, **kwargs)
    726 else:
--> 727 result = self.forward(*input, **kwargs)
    728 for hook in itertools.chain(
    729 _global_forward_hooks.values(),

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/transformers/models/visual_bert/modeling_visual_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, visual_embeds, visual_attention_mask, visual_token_type_ids, image_text_alignment, output_attentions, output_hidden_states, return_dict)
    821 visual_embeds=visual_embeds,
    822 visual_token_type_ids=visual_token_type_ids,
--> 823 image_text_alignment=image_text_alignment,
    824 )
    825 

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725 result = self._slow_forward(*input, **kwargs)
    726 else:
--> 727 result = self.forward(*input, **kwargs)
    728 for hook in itertools.chain(
    729 _global_forward_hooks.values(),

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/transformers/models/visual_bert/modeling_visual_bert.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds, visual_embeds, visual_token_type_ids, image_text_alignment)
    140 )
    141 
--> 142 visual_embeds = self.visual_projection(visual_embeds)
    143 visual_token_type_embeddings = self.visual_token_type_embeddings(visual_token_type_ids)
    144 

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725 result = self._slow_forward(*input, **kwargs)
    726 else:
--> 727 result = self.forward(*input, **kwargs)
    728 for hook in itertools.chain(
    729 _global_forward_hooks.values(),

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/nn/modules/linear.py in forward(self, input)
     91 
     92 def forward(self, input: Tensor) -> Tensor:
---> 93 return F.linear(input, self.weight, self.bias)
     94 
     95 def extra_repr(self) -> str:

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/nn/functional.py in linear(input, weight, bias)
   1690 ret = torch.addmm(bias, input, weight.t())
   1691 else:
-> 1692 output = input.matmul(weight.t())
   1693 if bias is not None:
   1694 output += bias

RuntimeError: mat1 dim 1 must match mat2 dim 0

```

The issue seems to origin from `image_text_allignment` and `visual_embeds`.

- Any ideas on ways to solve this error message?
- Does the overall steps sound like a good approach to prepare the data? Or would you try something else?

Thanks!

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [March 16, 2022, 9:00am UTC](https://discuss.huggingface.co/t/run-detectron2-for-feature-extraction-in-sagemaker-notebook/15569/7 "2022-03-16T09:00:14Z")

</div>

@Petrus would you mind opening a new thread in [Beginners - Hugging Face Forums](https://discuss.huggingface.co/c/beginners/5). This topic is not really SageMaker specific and might be pretty interesting for other community members.

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [March 16, 2022, 9:02am UTC](https://discuss.huggingface.co/t/run-detectron2-for-feature-extraction-in-sagemaker-notebook/15569/8 "2022-03-16T09:02:00Z")

</div>

Also, i am not sure if you have done this but we provide nice documentation for all models including `VisualBertForMultipleChoice` with example on how to do a forward pass.

> **[VisualBERT](https://huggingface.co/docs/transformers/model_doc/visual_bert#transformers.VisualBertForMultipleChoice)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

---

<div class="post-metadata">

**Author:** ![Petrus](https://avatars.discourse-cdn.com/v4/letter/p/ecd19e/32.png) [@Petrus](https://discuss.huggingface.co/u/Petrus)\
**Post date:** [March 16, 2022, 10:20am UTC](https://discuss.huggingface.co/t/run-detectron2-for-feature-extraction-in-sagemaker-notebook/15569/9 "2022-03-16T10:20:01Z")

</div>

@philschmid I am happy to make a post in that thread.

But just to respond on your second point: Yes, I have seen that example and tried to follow it. But am I right when I say that their way of concatenating the inputs as a dictionary only works for making predictions? While, for training, you need to use a pytorch Dataset, right?

Thank you Philipp!