# Sentence splitting **URL:** \ **Category:** 🤗Tokenizers\ **Created:** [April 9, 2021, 3:02pm UTC](https://discuss.huggingface.co/t/sentence-splitting/5393 "2021-04-09T15:02:03Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![carschno](https://avatars.discourse-cdn.com/v4/letter/c/c57346/32.png) [@carschno](https://discuss.huggingface.co/u/carschno)\ **Post date:** [April 9, 2021, 3:02pm UTC](https://discuss.huggingface.co/t/sentence-splitting/5393/1 "2021-04-09T15:02:03Z")
I am following the [Trainer example](https://huggingface.co/transformers/training.html#trainer) to fine-tune a Bert model on my data for text classification, using the pre-trained tokenizer (`bert-base-uncased`). In all examples I have found, the input texts are either single sentences or lists of sentences. However, my data is one string per document, comprising multiple sentences. When I inspect the tokenizer output, there are no `[SEP]` tokens put in between the sentences, e.g.: This is how I tokenize my dataset: ```auto def encode(examples): return tokenizer(examples['text'], truncation=True, padding='max_length') train_dataset = train_dataset.map(encode, batched=True) ``` And this is an example result of the tokenization: ```auto tokenizer.decode(train_dataset[0]["input_ids"]) [CLS] this is the first sentence . this is the second sentence. [SEP] ``` Given the special tokens in the beginning and the end, and that the output is lower-cased, I see that the input has been tokenized as expected. However, I was expecting to see a `[SEP]` between each sentence, as is the case when the input comprises a list of sentences. What is the recommended approach? Should I split the input documents into sentences, and run the tokenizer on each of them? Or does the Transformer model handle the continuous stream of sentences? I have seen posts like this: > [@Split document into sentences for sentence embedding](https://discuss.huggingface.co/t/split-document-into-sentences-for-sentence-embedding/3553/2): > > Just use a parser like stanza or spacy to tokenize/sentence segment your data. This is typically the first step in many NLP tasks. And: > [@Summarization on long documents](https://discuss.huggingface.co/t/summarization-on-long-documents/920/24): > > The disadvantage is that there is no sentence boundary detection. You can theoretically solve that with the NLTK (or SpaCy) approach and splitting sentences. However, it is not clear to me if this applied for a standard pipeline. ---
**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\ **Post date:** [April 9, 2021, 4:08pm UTC](https://discuss.huggingface.co/t/sentence-splitting/5393/2 "2021-04-09T16:08:04Z")
hey @carschno, how long are your documents (on average) and what kind of performance do you get with the current approach (i.e. tokenizing + truncating the whole document)? if performance is a problem then since you’re doing text classification, you could try chunking each document into smaller passages with a sliding window (see this [tutorial](https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/question_answering.ipynb) for details on how that’s done), and then aggregate the `[CLS]` representations for each window in a manner similar to [this](https://arxiv.org/pdf/1910.10781.pdf) paper. it’s not the most memory efficient strategy, but if your documents are not super long it might be a viable alternative to simple truncation ---
**Author:** ![carschno](https://avatars.discourse-cdn.com/v4/letter/c/c57346/32.png) [@carschno](https://discuss.huggingface.co/u/carschno)\ **Post date:** [April 10, 2021, 12:00pm UTC](https://discuss.huggingface.co/t/sentence-splitting/5393/3 "2021-04-10T12:00:43Z")
Hi @lewtun , Thanks for the suggestion, I think the sliding window approach looks very promising indeed. My documents are body texts from scraped web pages, so the length varies widely, from a few tokens till very long texts. Current performance of the fine-tuned classifier is around 0.8-0.9 accuracy (2 classes). I’ll have to look deeper into whether this is good enough for my application, and do more data analysis. Anyway for clarification: i understand the tokenizer behavior I described above is expected and the Bert model is supposed to handle input texts with multiple sentences in a single string well, right? ---
**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\ **Post date:** [April 12, 2021, 8:57pm UTC](https://discuss.huggingface.co/t/sentence-splitting/5393/4 "2021-04-12T20:57:05Z")
> [@carschno](#): > > Anyway for clarification: i understand the tokenizer behavior I described above is expected and the Bert model is supposed to handle input texts with multiple sentences in a single string well, right? Yes you’re totally right 🙂. From the tokenizer’s perspective, it doesn’t matter if the input string is composed of one or more sentences - it will split it into words/subwords according to the underlying tokenization algorithm (WordPiece in BERT’s case). In case you want to see the tokens directly, you can use the tokenizer’s `convert_ids_to_tokens` function on the `input_ids` returned by the tokenizer ---
**Author:** ![carschno](https://avatars.discourse-cdn.com/v4/letter/c/c57346/32.png) [@carschno](https://discuss.huggingface.co/u/carschno)\ **Post date:** [April 13, 2021, 7:14am UTC](https://discuss.huggingface.co/t/sentence-splitting/5393/5 "2021-04-13T07:14:55Z")
I actually used the tokenizer’s `decode` method on the tokenized dataset: ``` tokenizer.decode(train_dataset[0]["input_ids"]) [CLS] <...> [SEP] [SEP] [PAD] [PAD] <...> ``` I suppose both are the same. ---
**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\ **Post date:** [April 13, 2021, 9:31am UTC](https://discuss.huggingface.co/t/sentence-splitting/5393/6 "2021-04-13T09:31:16Z")
they’re similar, but `convert_ids_to_tokens` let’s you see the subwords, e.g.: ```auto tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased") txt = "The Higgs field gives mass to subatomic particles" txt_enc = tokenizer(txt) tokenizer.convert_ids_to_tokens(txt_enc.input_ids) # returns ['[CLS]', 'the', 'hi', '##ggs', 'field', 'gives', 'mass', 'to', 'sub', '##ato', '##mic', 'particles', '[SEP]'] tok.decode(txt_enc.input_ids) # returns "[CLS] the higgs field gives mass to subatomic particles [SEP]" ``` the `decode` method combines the subwords to produce a single string and adds the special tokens as in your example. i find both method quite handy for debugging! ---
**Author:** ![hhaven](https://avatars.discourse-cdn.com/v4/letter/h/8edcca/32.png) [@hhaven](https://discuss.huggingface.co/u/hhaven)\ **Post date:** [June 3, 2021, 1:49pm UTC](https://discuss.huggingface.co/t/sentence-splitting/5393/7 "2021-06-03T13:49:55Z")
> From the tokenizer’s perspective, it doesn’t matter if the input string is composed of one or more sentences - it will split it into words/subwords according to the underlying tokenization algorithm (WordPiece in BERT’s case). Am I correct that if I have data that doesn’t exceed `max_seq_length` but each observation consists of 2 or more sentences, establishing the sentence separation isn’t necessary (BERT, DistilBERT)? I guess I’m confused about `add_special_tokens=True` option in case if sentence separation doesn’t matter. ---
**Author:** ![Kwame](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/kwame/32/6078_2.png) [@Kwame](https://discuss.huggingface.co/u/Kwame)\ **Post date:** [September 15, 2022, 9:57pm UTC](https://discuss.huggingface.co/t/sentence-splitting/5393/8 "2022-09-15T21:57:04Z")
Sorry for necro posting but just wanted to point out that [pyBSD is a great library for sentence tokenization](https://github.com/nipunsadvilkar/pySBD). Be sure to set clean=True when using the Segmenter class.