# What is the input vector size for a BERT and Transformer-XL?

**URL:** <https://discuss.huggingface.co/t/what-is-the-input-vector-size-for-a-bert-and-transformer-xl/965>\
**Category:** 🤗Transformers\
**Created:** [September 2, 2020, 3:00pm UTC](https://discuss.huggingface.co/t/what-is-the-input-vector-size-for-a-bert-and-transformer-xl/965 "2020-09-02T15:00:29Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![gibbo0789](https://avatars.discourse-cdn.com/v4/letter/g/e47c2d/32.png) [@gibbo0789](https://discuss.huggingface.co/u/gibbo0789)\
**Post date:** [September 2, 2020, 3:00pm UTC](https://discuss.huggingface.co/t/what-is-the-input-vector-size-for-a-bert-and-transformer-xl/965/1 "2020-09-02T15:00:29Z")

</div>

Does anyone know what size vectors the BERT and Transformer-XL models take and output?

For example, I know that bert-large is 24-layer, 1024-hidden, 16-heads per block, 340M parameters. (bert-base is 12 heads per block) does that mean it takes a vector size of [24,1024,16]? or am I miss understanding?

Any help is much appreciated

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [September 2, 2020, 3:38pm UTC](https://discuss.huggingface.co/t/what-is-the-input-vector-size-for-a-bert-and-transformer-xl/965/2 "2020-09-02T15:38:29Z")

</div>

No the inputs are usually a tensor of ints of size batch\_size x sequence\_length (with integers between 0 and the vocabulary size of the model -1). You can find all expected shapes and output shapes in the documentation, for instance [here](https://huggingface.co/transformers/model_doc/bert.html#bertmodel) for BERT.