# My input sentence is very long(more than 512). What should I do when I want to fintune model about classify?Thanks

**URL:** <https://discuss.huggingface.co/t/my-input-sentence-is-very-long-more-than-512-what-should-i-do-when-i-want-to-fintune-model-about-classify-thanks/9688>\
**Category:** Beginners\
**Created:** [September 2, 2021, 10:41am UTC](https://discuss.huggingface.co/t/my-input-sentence-is-very-long-more-than-512-what-should-i-do-when-i-want-to-fintune-model-about-classify-thanks/9688 "2021-09-02T10:41:19Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ccfeidao](https://avatars.discourse-cdn.com/v4/letter/c/f08c70/32.png) [@ccfeidao](https://discuss.huggingface.co/u/ccfeidao)\
**Post date:** [September 2, 2021, 10:41am UTC](https://discuss.huggingface.co/t/my-input-sentence-is-very-long-more-than-512-what-should-i-do-when-i-want-to-fintune-model-about-classify-thanks/9688/1 "2021-09-02T10:41:19Z")

</div>

I know I can intercept the first 512 lengths.But I don’t want to do this. This task can understand the classification of texts.

AutoModelForSequenceClassification can be directly used for classification, but the question is whether the input can be some 512 length sentences, and use pooler layer to classify ?

Or what should I do? Thanks

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\
**Post date:** [September 2, 2021, 1:03pm UTC](https://discuss.huggingface.co/t/my-input-sentence-is-very-long-more-than-512-what-should-i-do-when-i-want-to-fintune-model-about-classify-thanks/9688/2 "2021-09-02T13:03:46Z")

</div>

Hey @ccfeidao you might want to try one of the dedicated models like LongFormer or BigBird which have a longer context size of around 4,096 tokens. See [this thread](https://discuss.huggingface.co/t/fintuning-transformer-on-clef-dataset/9584/2) for more details 🙂

---

<div class="post-metadata">

**Author:** ![ccfeidao](https://avatars.discourse-cdn.com/v4/letter/c/f08c70/32.png) [@ccfeidao](https://discuss.huggingface.co/u/ccfeidao)\
**Post date:** [September 2, 2021, 1:39pm UTC](https://discuss.huggingface.co/t/my-input-sentence-is-very-long-more-than-512-what-should-i-do-when-i-want-to-fintune-model-about-classify-thanks/9688/3 "2021-09-02T13:39:35Z")

</div>

Thanks ，I will try

---

<div class="post-metadata">

**Author:** ![ccfeidao](https://avatars.discourse-cdn.com/v4/letter/c/f08c70/32.png) [@ccfeidao](https://discuss.huggingface.co/u/ccfeidao)\
**Post date:** [September 3, 2021, 4:49am UTC](https://discuss.huggingface.co/t/my-input-sentence-is-very-long-more-than-512-what-should-i-do-when-i-want-to-fintune-model-about-classify-thanks/9688/4 "2021-09-03T04:49:20Z")

</div>

I want to user BigBird to try my task, but have a new [problem](https://discuss.huggingface.co/t/why-bigbirdtokenizer-can-t-load-my-own-vocab-or-trained-bpe-resultsr/9700), I am very Confusion.