Link

Sentence Splitting

Table of contents


Description

Sentence splitting is the process of dividing text into sentences. For instance the document Hello world. Hello world again. would be split into the sentences Hello world. and Hello world again. CoreNLP splits documents into sentences via a set of rules.

As of 4.5.0, the ssplit annotator is automatically included as part of the tokenize annotator. It is no longer necessary to specify ssplit as part of the annotators list. If using an older version, it would still be necessary to incude ssplit in the annotators list.

NameAnnotator class nameRequirementGenerated AnnotationDescription
ssplitWordToSentencesAnnotatorTokensAnnotationSentencesAnnotationSplits text into sentences.

Options

Option nameTypeDefaultDescription
ssplit.eolonlyBooleanfalseSplit sentences at and only at newlines. Suitable for input such as many machine translation datasets which are already formatted to be treated as strictly one sentence per line. Works well in conjunction with -tokenize.whitespace true, in which case StanfordCoreNLP will treat the input as already tokenized and one sentence per line, only separating words on whitespace.
ssplit.isOneSentenceBooleanfalseEach document is to be treated as one sentence, with no sentence splitting at all.
ssplit.newlineIsSentenceBreakEnum { always, never, two }neverWhether to treat newlines as sentence breaks. This property has 3 legal values. “always” means that a newline is always a sentence break (but there still may be multiple sentences per line). This is usually appropriate for texts with soft line breaks. “never” means to ignore newlines for the purpose of sentence splitting. This is appropriate for continuous text with hard line breaks, when just the non-whitespace characters should be used to determine sentence breaks. “two” means that two or more consecutive newlines will be treated as a sentence break. This option is appropriate when dealing with text with hard line breaks and a blank line between paragraphs. Note: A side-effect of setting ssplit.newlineIsSentenceBreak to “two” or “always” is that the tokenizer will tokenize newlines.
ssplit.boundaryMultiTokenRegexRegexnullIf non-null, value is a multi-token regex, that is, a tokensregex expression, that will match something to be treated as a sentence boundary. For example, ssplit.boundaryMultiTokenRegex = /(?:\\n|\\*NL\\*)/{2,} is basically equivalent to ssplit.newlineIsSentenceBreak = two. The matched tokens will be treated as not part of the following sentence. They will be discarded if and only if they also match ssplit.boundariesToDiscard
ssplit.boundaryTokenRegexRegex\\.|[!?]+If non-null, value is a regex for regular sentence boundary tokens; otherwise the default is used. For example, for Chinese, a possible setting might be: ssplit.boundaryTokenRegex = [.。]|[!?!?]+.
ssplit.boundariesToDiscardList(String)nullIf non-null value is a String which contains a comma-separated list of String tokens that will be treated as sentence boundaries (when matched with String equality) and then discarded. For example, it might be ssplit.boundariesToDiscard = <p>,<P>,</p>,</P>.
ssplit.htmlBoundariesToDiscardStringnullIf non-null, value is a String which contains a comma-separated list of XML element names that will be treated as sentence boundaries (when matched with String equality), and then discarded. For example, it might be: htmlBoundariesToDiscard = p,text,post,postdate,poster,turn,speaker,quote. Note that this functionality overlaps, but is less flexible than, functionality that is available in the cleanxml annotator.
ssplit.tokenPatternsToDiscardList(Regex)nullIf non-null, value is a comma-separated list of regex for tokens to discard without marking them as sentence boundaries.
ssplit.boundaryFollowersRegexRegexnullIf non-null, value is a regex for tokens to allow to be part of the preceding sentence following something that matches ssplit.boundaryTokenRegex (but not something after a newline or XML sentence break).

Sentence Splitting From The Command Line

This command will take in the text of the file input.txt and produce a human readable output of the sentences:

java edu.stanford.nlp.pipeline.StanfordCoreNLP -annotators tokenize -file input.txt

Other output formats include conllu, conll, json, and serialized.

The following command shows an example of customizing sentence splitting. Here we will tell the annotator to only split on newlines, meaning the file is one sentence per line:

java edu.stanford.nlp.pipeline.StanfordCoreNLP -annotators tokenize -ssplit.eolonly -file input.txt

Sentence Splitting From Java

package edu.stanford.nlp.examples;

import edu.stanford.nlp.pipeline.*;

import java.util.*;

public class SentenceSplittingExample {

  public static String text = "Hello world. Hello world again.";

  public static void main(String[] args) {
    // set up pipeline properties
    Properties props = new Properties();
    // set the list of annotators to run
    props.setProperty("annotators", "tokenize");
    // build pipeline
    StanfordCoreNLP pipeline = new StanfordCoreNLP(props);
    // create a document object
    CoreDocument doc = new CoreDocument(text);
    // annotate
    pipeline.annotate(doc);
    // display sentences
    for (CoreSentence sent : doc.sentences()) {
        System.out.println(sent.text());
    }
  }
}

This demo code will produce this output:

Hello world.
Hello world again.