# Convert Conv1D to nn.Linear

**URL:** <https://discuss.huggingface.co/t/convert-conv1d-to-nn-linear/36760>\
**Category:** 🤗Transformers\
**Created:** [April 16, 2023, 4:45pm UTC](https://discuss.huggingface.co/t/convert-conv1d-to-nn-linear/36760 "2023-04-16T16:45:46Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![IdoAmit198](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/idoamit198/32/12110_2.png) [@IdoAmit198](https://discuss.huggingface.co/u/IdoAmit198)\
**Post date:** [April 16, 2023, 4:45pm UTC](https://discuss.huggingface.co/t/convert-conv1d-to-nn-linear/36760/1 "2023-04-16T16:45:47Z")

</div>

Hi everyone,

I’m working on a research regarding GPT2 and want to test few ideas which apply for linear layers.

What’s the problem? GPT2 model consist of Conv1D layers instead of nn.Linear.

Now I know that Conv1D is said to be just like linear, but transposed, and still I’d love if someone can help me out to convert Huggingface pre-trained GPT2 model (the one with the Conv1D) to equivalent GPT2 model with nn.Linear instead.

Thanks for any coming help!

#transformers #models #research

---

<div class="post-metadata">

**Author:** ![NamburiSrinath](https://avatars.discourse-cdn.com/v4/letter/n/49beb7/32.png) [@NamburiSrinath](https://discuss.huggingface.co/u/NamburiSrinath)\
**Post date:** [January 20, 2024, 6:34pm UTC](https://discuss.huggingface.co/t/convert-conv1d-to-nn-linear/36760/2 "2024-01-20T18:34:07Z")

</div>

Hi @IdoAmit198,

Did you figure out a way? Because I also want to apply some function I’ve written on this model but the function currently works only on Linear classes!

Any help is appreciated

---

<div class="post-metadata">

**Author:** ![meztech](https://avatars.discourse-cdn.com/v4/letter/m/e19b73/32.png) [@meztech](https://discuss.huggingface.co/u/meztech)\
**Post date:** [May 12, 2024, 6:39pm UTC](https://discuss.huggingface.co/t/convert-conv1d-to-nn-linear/36760/3 "2024-05-12T18:39:44Z")

</div>

This is as simple as it seems. Just set the weights of a linear layer to the transposed weights of a Conv1D layer and set the bias of the linear layer to be the same as the bias of the Conv1D.

It’s as simple as:

test\_in = torch.rand((1,1024))  
test\_c1d = \<some pretrained Conv1D layer of size (1024,1024)\>  
test\_lin = torch.nn.Linear(1024,1024)  
test\_lin.weight = torch.nn.Parameter(test\_c1d.weight.T)  
test\_lin.bias = test\_c1d.bias

test\_out\_c1d = test\_c1d(test\_in)  
test\_out\_lin = test\_lin(test\_in)

# Show that results are equivalent:

test\_out\_c1d == test\_out\_lin