Combined models

Table of contents

Combined models

The default models for some languages are “combined”. The goal is to get better coverage of the language, hopefully without sacrificing consistency in the annotation scheme. In each case, the data used to train the models is a combination of multiple UD datasets.

Other data sets would be added, or combined models for other languages created, but there are often data problems preventing that. For example, English Lines uses a different xpos and feature scheme. Spanish GSD and AnCora use the same general annotation scheme, but sentence splits are annotated differently, and some of the features are noticeably different. Hopefully over time we can resolve some of those issues and expand the models.

Whether or not this was a good idea was explored in a GURT paper from Georgetown

Lemma dictionaries

The English, German, and Slovenian lemmatizers also use a large external lexicon, to cover words which never appear in the treebanks. Each lexicon is converted to a list of (word, UPOS, lemma) entries, and those entries go into the lemmatizer’s dictionary. Where a treebank and a lexicon disagree on the lemma for the same word and UPOS, the treebank’s lemma is the one in the dictionary.

The seq2seq model also trains on the lexicon, at a lower rate than the treebank data, so that it learns the inflection patterns as well as the specific words.

See the lemmatizer page for how to rebuild these dictionaries when training a model.

SPECIALIST Lexicon

The English lemma dictionary is built from the inflection table (inflVars.data) of the 2026 release of the SPECIALIST Lexicon, part of the SPECIALIST NLP Tools from the Lister Hill National Center for Biomedical Communications, National Library of Medicine. The SPECIALIST Lexicon is a United States Government product, and its use and redistribution are subject to the following conditions, from the SPECIALIST NLP Tools Terms and Conditions:

• Redistributions of the SPECIALIST NLP Tools in source or binary form must include this list of conditions in the documentation and/or other materials provided with the distribution.
• In any publication or distribution of all or any portion of the SPECIALIST NLP Tools (1) you must attribute the source of the tools as the SPECIALIST NLP Tools with the release number and date; (2) you must state any modifications made to the SPECIALIST NLP Tools along with a complete description of the modifications, which may be in the form of patch files.
• You shall not assert any proprietary rights to any portion of the SPECIALIST NLP Tools, nor represent the SPECIALIST NLP Tools or any part thereof to anyone as other than a United States Government product.
• The name of the U.S. Department of Health and Human Services, National Institutes of Health, National Library of Medicine, Lister Hill National Center for Biomedical Communications may not be used to endorse or promote products derived from the SPECIALIST NLP Tools without specific prior written permission.
• Neither the United States Government, U.S. Department of Health and Human Services, National Institutes of Health, National Library of Medicine, Lister Hill National Center for Biomedical Communications, nor any of its agencies, contractors, subcontractors or employees of the United States Government make any warranties, expressed or implied, with respect to the SPECIALIST NLP Tools, and, furthermore, assume no liability for any party’s use, or the results of such use, of any part of these tools.

The modifications made to the lexicon are:

• Only the noun, verb, adjective, and adverb rows are kept, mapped to the UPOS tags NOUN, VERB, ADJ, and ADV. The lexicon has no category for proper nouns, so a noun whose lemma has a capital letter, such as Alzheimer, is tagged PROPN.
• The lemma is the uninflected form of each row, which keeps the spelling of the inflected form (colours → colour), rather than the citation form.
• Rows are dropped if the form or lemma contains whitespace, if the form is a negated contraction such as didn't, or if the form or lemma has characters other than letters, digits, -, ', and ..
• A (word, UPOS) pair with more than one lemma, such as axes → axe / axis, is dropped.
• The remaining entries are written as one-word CoNLL-U sentences.

The complete conversion is stanza/utils/datasets/lemma/convert_en_specialist.py.

The English dictionary also has 10,000 generated numbers written with commas, such as 1,234,567 → 1234567, which follows the UD English treebanks. These are not from the SPECIALIST Lexicon. They teach the seq2seq model to copy long numbers without changing or repeating digits.

German Wiktionary

The German lemma dictionary is built from German Wiktionary, which is licensed under CC BY-SA 4.0. The converted lemmas are in handparsed-treebank.

The forms of nouns, verbs, adjectives, and adverbs are taken from the inflection tables of their entries, along with the pages for conjugated verb forms. Names are tagged PROPN. A noun with adjectival declension, such as Abgeordneter, has its weak form (Abgeordnete) as the lemma. Every form with ß also gets a version with ss, for Swiss spelling, and a form with more than one lemma is left out. The conversion is stanza/utils/datasets/lemma/convert_de_wiktionary.py.

Sloleks

The Slovenian lemma dictionary is built from Sloleks 3.1 (Čibej, Gantar, Dobrovoljc, Krek, Holozan, Erjavec, Romih, Arhar Holdt, Krsnik, and Robnik-Šikonja, 2026, Slovenian language resource repository CLARIN.SI), which is licensed under CC BY-SA 4.0.

The lexicon is converted to (word, UPOS, lemma) entries, keeping the orthographic forms of each headword. Conjunctions, abbreviations, and residuals are skipped, and the other categories are mapped to UPOS. The conversion is stanza/utils/datasets/lemma/convert_sl_sloleks.py.

Data augmentation

In general, the models also use various sorts of text “data augmentation” of the original data.

For example, in a language where all of the sentences in the POS training data end with punctuation, we remove the trailing punctuation from some fragment of sentences to train the model to handle sentences which don’t end with sentence final punctuation. Otherwise, we would frequently get issues such as a model tagging This is an unfinished sentence_PU

The tokenization models also use a couple different versions of this. For example, most of the tokenization datasets have just one or a couple forms of quotes, but we replace some fraction of quotes with different types so that each model has a chance of correctly tokenizing «data augmentation», "data augmentation", etc. Also, sentence final punctuation will often have spaces added or removed to make the model more robust to typos.