# Automatic sentence segmentation and encoding

**URL:** <https://discuss.huggingface.co/t/automatic-sentence-segmentation-and-encoding/1479>\
**Category:** 🤗Tokenizers\
**Created:** [October 12, 2020, 9:43am UTC](https://discuss.huggingface.co/t/automatic-sentence-segmentation-and-encoding/1479 "2020-10-12T09:43:56Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](https://discuss.huggingface.co/u/vblagoje)\
**Post date:** [October 12, 2020, 9:43am UTC](https://discuss.huggingface.co/t/automatic-sentence-segmentation-and-encoding/1479/1 "2020-10-12T09:43:56Z")

</div>

Hello everyone,

Is there a way to segment sentences and encode them using the tokenizers library automatically? I am currently using spacy and/or nltk for the sentence segmentation part and then tokenizers to encode the sentences.

Many thanks.