Pipeline and Processors

Table of contents

Pipeline

For basic end to end examples, please see Getting Started.

To start annotating text with Stanza, you would typically start by building a Pipeline that contains Processors, each fulfilling a specific NLP task you desire (e.g., tokenization, part-of-speech tagging, syntactic parsing, etc). The pipeline takes in raw text or a Document object that contains partial annotations, runs the specified processors in succession, and returns an annotated Document (see the documentation on Document for more information on how to extract these annotations).

To build and customize the pipeline, you can specify the options in the table below:

Processors

Processors are units of the neural pipeline that perform specific NLP functions and create different annotations for a Document. The neural pipeline supports the following processors:

Processor variants

New in v1.1

Sometimes you might want to build your own models to perform the tasks that existing processors already handle in the Stanza neural pipeline, or simply experiment with alternative toolkits for a specific task. Processor variants are here to help with that.

One example use case is using your own tokenizer for tokenization. Previously we have added support for popular tokenizers like spaCy for English and jieba for Chinese, and now we have made it much easier to add your own. You simply need to implement a ProcessorVariant and register it with Stanza using the @register_processor_variant decorator. For instance, this is how our spaCy tokenizer is implemented

from stanza.pipeline.processor import ProcessorVariant, register_processor_variant

@register_processor_variant('tokenize', 'spacy')
class SpacyTokenizer(ProcessorVariant):
    def __init__(self, config):
        # initialize spacy

    def process(self, text):
        # tokenize text with spacy

This allows the user to set tokenize_with_spacy as True (or processors={"tokenize": "spacy"}) when instantiating the pipeline to use it. In the case of the tokenizer, the TokenizeProcessor handles options such as pre-tokenization and pre-sengmentation, and only passes text to the variants when tokenization from raw text is needed.

Alternatively, one can also implement a processor variant as a drop-in replacement for a processor by setting OVERRIDE as True in the ProcessorVariant class, for instance

@register_processor_variant("lemma", "cool")
class CoolLemmatizer(ProcessorVariant):
    ''' An alternative lemmatizer that lemmatizes every word to "cool". '''

    OVERRIDE = True

    def __init__(self, lang):
        pass

    def process(self, document):
        for sentence in document.sentences:
            for word in sentence.words:
                word.lemma = "cool"

        return document

This lemmatizer will replace all of the functionality of the Stanza lemmatizer when it’s used in the pipeline, and lemmatize every single word to “cool”.

Building your own Processors and using them in the neural pipeline

New in v1.1

If you’re looking to implement annotation capabilities that don’t currently exist in Stanza and want to use it in the neural pipeline, it hasn’t been easier. You can simply implement a Processor class and register it with Stanza using the @register_processor decorator, and then it’s easy to use it in your project, and/or publish it for other Stanza users to use.

Here is an example:

@register_processor("lowercase")
class LowercaseProcessor(Processor):
    ''' Processor that lowercases all text '''
    _requires = set(['tokenize'])
    _provides = set(['lowercase'])

    def __init__(self, device, config, pipeline):
        pass

    def _set_up_model(self, *args):
        pass

    def process(self, doc):
        doc.text = doc.text.lower()
        for sent in doc.sentences:
            for tok in sent.tokens:
                tok.text = tok.text.lower()

            for word in sent.words:
                word.text = word.text.lower()

        return doc

Once registered, you can use this processor in the pipeline as if it were one of Stanza’s standard processors

nlp = stanza.Pipeline(dir=TEST_MODELS_DIR, lang='en', processors='tokenize,lowercase')

and in this case the processor will lowercase all text in the document.

Advanced Pipeline Options

Build Pipeline from a Config Dictionary

When there are many options you want to configure, or even set programmatically, it might not be convenient to set them one by one using keyword arguments to instantiate the Pipeline. In these cases, alternatively, you can build the desired pipeline with a config dictionary, allowing maximum customization for the pipeline:

import stanza

config = {
        # Comma-separated list of processors to use
	'processors': 'tokenize,mwt,pos',
        # Language code for the language to build the Pipeline in
        'lang': 'fr',
        # Processor-specific arguments are set with keys "{processor_name}_{argument_name}"
        # You only need model paths if you have a specific model outside of stanza_resources
	'tokenize_model_path': './fr_gsd_models/fr_gsd_tokenizer.pt',
	'mwt_model_path': './fr_gsd_models/fr_gsd_mwt_expander.pt',
	'pos_model_path': './fr_gsd_models/fr_gsd_tagger.pt',
	'pos_pretrain_path': './fr_gsd_models/fr_gsd.pretrain.pt',
        # Use pretokenized text as input and disable tokenization
	'tokenize_pretokenized': True
}
nlp = stanza.Pipeline(**config) # Initialize the pipeline using a configuration dict
doc = nlp("Van Gogh grandit au sein d'une famille de l'ancienne bourgeoisie .") # Run the pipeline on the pretokenized input text
print(doc) # Look at the result

Here, we can specify the language, processors, and paths for many Processor models all at once, and pass that to the Pipeline initializer. Note that config dictionaries and keyword arguments can be combined as well, to maximize your flexibility in using Stanza’s neural pipeline.