Model Performance

System Performance on UD Treebanks

Here we report the performance of Stanza’s pretrained models on all supported languages. Again, performances of models for tokenization, multi-word token (MWT) expansion, lemmatization, part-of-speech (POS) and morphological features tagging and dependency parsing are reported on the Universal Dependencies (UD) treebanks, while performances of the NER models are reported separately.

System Performance on UD Treebanks

In the table below you can find the performance of Stanza’s pretrained UD models as of version 1.5.1 on UD 2.12.

Older versions of this chart are also available:

Stanza v1.0 on UD 2.5.

Stanza v1.3 on UD 2.8.

All models are trained and tested with the Universal Dependencies v2.12 treebanks. The following scores are reported for models with charlm, when possible, but not with transformers.

Note that all scores reported are from an end-to-end evaluation on the official test sets (from raw text to the full CoNLL-U file), and are generated with the CoNLL 2018 UD shared task official evaluation script. For detailed interpretation of these scores and the evaluation scripts we used, please refer to the CoNLL 2018 UD Shared Task Evaluation page. For details on how we handled treebanks with no training data, please refer to our CoNLL 2018 system description paper.

If the models for a particular dataset are the default for that language, it is marked with a green checkmark: For languages which have a “combined” family of models, that will be the default, and those models are not represented in this table.

License information for models built from the UD data is unclear, but users are encouraged to click on the git links below and check the license for the relevant data if they are unsure of the licenses for the data used to build these models. To the extent that The Trustees of Leland Stanford Junior University have ownership and rights over these language packs, all these Stanza language packs are made available under the Open Data Commons Attribution License v1.0.

Language	Treebank	lcode	Tokens	Sentences	Words	UPOS	XPOS	UFeats	AllTags	Lemmas	UAS	LAS	CLAS	MLAS	BLEX
Macro Avg			99.50	89.23	98.95	94.68	94.39	91.66	89.10	94.28	83.68	79.02	74.81	67.47	71.29
Afrikaans	AfriBooms	af	99.99	99.65	99.99	98.60	95.77	98.41	95.66	98.29	89.54	86.14	80.10	76.71	77.53
Ancient Greek	PROIEL	grc	100.00	52.00	100.00	97.42	97.82	92.30	91.06	97.18	82.77	79.02	72.47	63.94	70.00
Ancient Greek	Perseus	grc	99.99	98.93	99.99	92.41	85.13	91.11	84.81	87.86	79.46	73.97	67.99	53.93	57.20
Ancient Hebrew	PTNK	hbo	100.00	98.18	93.60	90.20	90.53	85.40	84.63	92.14	76.12	72.72	63.10	54.54	61.98
Arabic	PADT	ar	99.98	83.56	98.03	95.18	92.40	92.52	92.18	93.54	83.99	79.77	76.72	71.10	73.25
Armenian	ArmTDP	hy	99.92	99.46	99.48	83.05	99.48	66.22	64.30	91.35	74.04	61.80	52.52	29.58	49.05
Armenian	BSUT	hy	99.91	98.23	99.86	95.94	99.86	89.16	88.29	94.99	84.31	77.20	71.56	60.29	67.48
Basque	BDT	eu	99.98	100.00	99.98	96.07	99.98	92.95	91.16	96.38	86.65	83.20	80.80	73.42	77.71
Belarusian	HSE	be	98.43	88.39	98.43	96.72	96.10	92.12	91.15	94.59	85.16	82.68	79.28	72.70	75.20
Bulgarian	BTB	bg	99.96	96.50	99.96	99.16	96.97	97.84	96.62	97.28	93.98	91.09	88.00	84.88	84.59
Buryat	BDT	bxr	100.00	100.00	100.00	75.16	100.00	74.51	67.97	77.78	56.21	40.52	31.58	24.88	25.84
Catalan	AnCora	ca	99.97	99.57	99.95	98.70	95.93	98.27	95.67	99.06	93.55	91.66	87.24	84.72	86.41
Chinese (Simplified)	GSDSimp	zh-hans	92.14	99.10	92.14	88.82	88.61	91.47	87.95	92.07	73.41	70.65	67.39	63.96	67.35
Chinese (Traditional)	GSD	zh-hant	92.79	99.10	92.79	89.06	88.80	92.01	88.07	92.70	72.94	69.98	66.65	62.60	66.62
Classical Chinese	Kyoto	lzh	97.85	49.57	97.85	89.16	88.17	91.38	85.89	97.63	71.32	65.80	64.54	61.94	64.43
Coptic	Scriptorium	cop	100.00	34.09	88.63	85.77	85.69	86.37	84.77	85.95	66.37	64.43	56.39	53.08	55.19
Croatian	SET	hr	99.96	97.08	99.96	98.21	94.89	95.27	94.60	96.60	90.29	86.95	84.14	78.19	80.25
Czech	CAC	cs	99.82	100.00	99.57	98.52	94.59	93.64	92.92	97.46	91.09	88.71	86.78	80.27	84.87
Czech	CLTT	cs	99.23	98.23	99.23	97.38	88.25	88.22	87.42	95.78	83.53	79.71	76.52	66.32	74.12
Czech	FicTree	cs	99.98	99.07	99.97	98.45	95.15	96.00	94.77	98.32	93.28	90.80	88.55	82.85	86.53
Czech	PDT	cs	99.97	94.62	99.97	98.88	95.17	95.84	94.65	98.73	92.22	90.18	88.52	83.49	87.26
Danish	DDT	da	99.97	91.81	99.97	98.30	99.97	97.87	97.14	97.88	88.61	86.61	83.80	80.06	81.27
Dutch	Alpino	nl	99.87	90.08	99.87	96.75	95.50	97.00	94.97	95.33	91.42	88.67	83.94	79.65	78.12
Dutch	LassySmall	nl	99.87	78.61	99.87	96.67	95.92	96.83	95.20	95.86	88.25	85.32	80.00	76.72	75.15
English	Atis	en	100.00	87.55	100.00	99.15	100.00	98.74	98.19	99.65	94.45	92.61	90.08	87.49	89.72
English	ESLSpok	en	100.00	92.81	100.00	98.28	98.32	100.00	97.88	100.00	93.12	90.73	89.43	87.45	89.43
English	EWT	en	99.17	89.16	98.92	96.20	95.75	96.39	94.64	96.63	88.90	86.77	84.11	80.27	81.54
English	GUM	en	99.78	95.34	99.69	97.59	97.42	97.52	96.52	98.22	90.49	88.24	84.52	81.71	82.68
English	LinES	en	99.96	88.63	99.96	97.42	96.57	97.04	94.52	98.26	88.67	85.04	81.02	76.57	79.11
English	ParTUT	en	99.72	100.00	99.63	96.70	96.40	95.76	94.50	97.31	91.15	88.36	83.62	78.22	81.20
Erzya	JR	myv	99.63	96.92	99.08	84.84	83.29	75.88	69.52	84.72	70.08	57.35	52.10	36.09	44.27
Estonian	EDT	et	99.97	92.88	99.97	97.19	97.94	95.64	94.39	95.93	86.60	83.81	82.23	77.02	78.05
Estonian	EWT	et	98.79	79.57	98.79	93.14	95.02	92.11	89.35	90.77	77.27	72.94	70.34	63.87	63.75
Faroese	FarPaHC	fo	99.65	93.94	99.64	96.51	91.56	92.92	90.86	99.64	84.39	80.07	72.28	64.48	72.28
Finnish	FTB	fi	100.00	90.13	99.98	96.88	96.09	97.12	95.29	97.10	90.99	88.63	86.24	82.60	84.20
Finnish	TDT	fi	99.73	90.70	99.70	97.51	98.09	96.13	95.39	95.98	90.37	88.31	87.15	83.01	83.42
French	GSD	fr	99.71	95.03	99.50	97.63	99.50	97.74	97.05	98.20	93.27	91.28	87.44	84.45	85.78
French	ParTUT	fr	99.88	100.00	99.42	96.92	96.58	94.73	93.96	97.35	91.93	89.58	85.48	78.46	82.69
French	ParisStories	fr	99.76	92.80	99.56	97.08	99.56	94.99	93.40	97.66	78.11	73.95	69.67	62.76	67.87
French	Rhapsodie	fr	99.81	99.82	99.52	97.31	99.52	96.08	95.06	97.74	85.22	80.72	74.73	69.63	72.74
French	Sequoia	fr	99.88	87.46	99.64	98.80	99.64	97.87	97.52	98.50	93.22	91.57	87.74	84.88	86.36
Galician	CTG	gl	99.91	99.30	99.37	97.11	96.90	99.19	96.53	98.01	84.58	81.95	75.72	69.72	74.39
Galician	TreeGal	gl	99.68	88.02	98.53	93.89	91.51	93.11	90.64	94.52	77.27	71.86	64.38	58.14	60.30
German	GSD	de	99.62	83.84	99.60	95.61	97.36	89.78	87.19	97.23	85.80	81.80	77.08	62.81	74.04
German	HDT	de	100.00	97.36	100.00	98.30	98.20	92.57	92.17	98.04	95.59	93.56	89.88	78.81	87.18
Gothic	PROIEL	got	100.00	40.33	100.00	96.06	96.62	90.43	88.63	96.10	79.34	72.87	67.49	60.64	65.16
Greek	GDT	el	99.90	93.66	99.90	97.71	97.71	94.72	94.01	96.05	91.22	88.87	84.19	77.99	79.08
Greek	GUD	el	99.89	98.16	99.89	96.52	95.60	92.50	89.00	92.63	89.81	85.33	80.28	68.93	69.75
Hebrew	HTB	he	99.98	100.00	92.29	89.99	89.99	88.68	88.03	89.71	79.38	77.18	71.95	65.35	68.81
Hebrew	IAHLTwiki	he	99.64	96.78	94.61	91.69	91.69	86.88	86.14	92.51	83.23	81.19	75.58	64.88	73.14
Hindi	HDTB	hi	100.00	99.52	100.00	97.95	97.53	94.41	92.74	96.88	95.26	92.46	89.38	80.26	88.12
Hungarian	Szeged	hu	99.86	97.21	99.86	95.80	99.86	93.54	92.74	93.81	83.57	78.61	77.02	68.19	70.84
Icelandic	GC	is	99.70	95.80	99.70	92.68	76.46	81.10	74.89	91.06	80.41	75.45	71.17	52.55	63.45
Icelandic	IcePaHC	is	99.90	93.07	99.84	96.36	92.27	90.42	85.48	95.78	86.79	82.85	77.27	64.78	73.63
Icelandic	Modern	is	99.24	94.40	99.24	96.65	93.95	85.87	83.03	95.34	82.70	79.40	73.76	58.26	69.99
Indonesian	CSUI	id	99.65	90.76	99.26	95.90	96.12	96.66	95.50	96.36	83.22	78.55	76.07	73.23	73.39
Indonesian	GSD	id	99.88	93.04	99.64	94.16	94.69	95.58	89.38	98.28	87.05	80.83	76.98	71.78	76.03
Irish	IDT	ga	99.75	97.24	99.75	96.01	94.55	89.41	86.52	95.32	86.81	81.51	76.34	63.38	71.94
Irish	TwittIrish	ga	98.51	56.98	98.51	88.59	98.51	98.51	88.59	87.96	75.83	67.60	60.05	53.42	52.16
Italian	ISDT	it	99.93	99.07	99.79	98.41	98.32	98.14	97.69	98.23	93.29	91.61	87.32	85.05	85.17
Italian	MarkIT	it	100.00	99.12	99.67	96.29	96.34	94.28	92.34	87.08	85.58	80.63	74.48	65.35	70.41
Italian	ParTUT	it	99.81	100.00	99.82	98.42	98.37	98.04	97.51	97.71	93.53	91.47	86.62	84.28	84.06
Italian	ParlaMint	it	99.59	95.83	99.01	96.83	95.96	95.28	93.74	95.64	85.19	81.45	74.76	68.88	70.30
Italian	PoSTWITA	it	99.61	61.71	99.34	96.37	96.15	96.32	95.05	96.63	83.60	79.30	73.66	70.14	71.68
Italian	TWITTIRO	it	99.18	57.78	98.91	94.70	94.22	93.82	91.73	92.91	79.21	73.55	65.15	59.58	59.88
Italian	VIT	it	99.94	95.93	99.61	97.82	97.00	97.61	96.06	98.35	90.40	87.40	82.33	78.73	80.70
Japanese	GSDLUW	ja	96.32	99.72	96.32	95.10	94.67	96.32	94.64	94.67	88.26	87.36	80.74	78.71	78.14
Japanese	GSD	ja	97.37	99.72	97.37	96.38	95.72	97.36	95.45	96.02	90.01	89.29	84.99	83.86	83.34
Kazakh	KTB	kk	99.42	100.00	96.85	85.58	86.15	78.70	74.50	89.21	60.55	51.38	49.45	38.40	44.48
Korean	GSD	ko	99.90	96.29	99.90	95.98	90.42	99.72	88.16	93.55	86.56	82.48	80.79	79.07	75.39
Korean	Kaist	ko	100.00	99.93	100.00	95.74	85.81	100.00	85.81	94.10	87.63	85.50	83.65	80.93	78.78
Kurmanji	MG	kmr	99.38	100.00	99.38	91.96	92.37	83.30	79.59	92.78	73.40	68.45	64.38	46.58	57.53
Kyrgyz	KTMU	ky	99.06	98.14	99.06	87.65	87.94	60.22	57.91	88.09	74.95	58.77	52.60	28.47	47.22
Latin	ITTB	la	99.99	80.66	99.99	98.76	95.72	96.69	94.63	99.09	88.82	86.80	85.71	81.47	85.31
Latin	LLCT	la	99.99	99.32	99.99	99.57	96.65	96.79	96.47	98.09	96.13	94.88	93.53	89.49	91.50
Latin	PROIEL	la	99.98	43.10	99.98	96.95	97.15	91.58	90.59	96.86	77.98	73.99	68.89	61.96	67.12
Latin	Perseus	la	99.97	98.93	99.97	91.40	78.09	81.83	76.37	83.57	72.07	63.16	58.27	45.14	48.05
Latin	UDante	la	99.66	98.22	99.30	90.08	74.55	81.64	73.07	86.95	68.00	58.84	48.74	37.46	42.52
Latvian	LVTB	lv	99.81	98.15	99.81	96.70	89.72	94.73	89.29	96.12	88.91	85.77	83.37	76.71	80.17
Ligurian	GLT	lij	100.00	100.00	98.00	89.14	98.00	88.25	85.59	92.68	77.61	69.18	60.33	51.24	53.72
Lithuanian	ALKSNIS	lt	99.94	89.93	99.94	93.71	86.48	88.16	85.54	92.95	77.82	72.75	70.23	60.53	65.21
Lithuanian	HSE	lt	97.86	52.75	97.86	82.30	81.16	70.91	66.82	76.79	50.21	38.25	33.79	21.90	26.25
Maghrebi Arabic French	Arabizi	qaf	96.91	8.44	93.77	77.40	70.73	83.75	70.27	51.03	56.50	47.88	39.84	33.23	23.73
Maltese	MUDT	mt	99.85	85.35	99.85	95.72	95.61	99.85	95.35	99.85	83.62	78.22	70.76	67.20	70.76
Manx	Cadhan	gv	99.87	99.74	97.55	93.82	97.55	95.63	92.76	92.67	88.55	84.79	81.48	77.69	76.15
Marathi	UFAL	mr	97.99	89.36	92.89	85.78	92.89	73.53	71.57	82.84	68.87	59.56	54.73	37.04	46.09
Naija	NSC	pcm	99.99	100.00	99.99	97.55	99.99	98.84	97.03	99.14	92.39	89.56	88.60	86.43	88.21
North Sami	Giella	sme	99.71	99.65	99.71	90.73	92.77	88.08	83.69	88.54	72.26	66.63	64.19	57.21	56.46
Norwegian	Bokmaal	nb	99.88	97.70	99.88	98.38	99.04	97.09	96.38	98.61	93.26	91.85	89.11	85.13	87.53
Norwegian Nynorsk	Nynorsk	nn	99.97	95.48	99.97	98.13	98.91	96.87	95.93	97.87	91.63	89.94	86.84	82.33	84.30
Old Church Slavonic	PROIEL	cu	100.00	40.27	100.00	96.14	96.45	89.68	88.18	94.65	78.87	74.64	70.00	62.04	66.83
Old East Slavic	Birchbark	orv	99.93	26.85	99.93	83.89	99.12	67.88	65.34	73.69	60.02	50.82	40.18	27.23	29.53
Old East Slavic	RNC	orv	99.14	89.29	99.14	91.04	87.53	72.39	64.40	81.04	68.10	60.81	51.90	33.25	40.36
Old East Slavic	TOROT	orv	100.00	40.67	100.00	95.12	95.15	89.06	87.22	92.79	78.66	74.06	68.86	60.63	64.19
Old French	SRCMF	fro	99.89	100.00	99.89	96.64	96.56	97.64	95.72	99.85	90.80	87.01	84.11	80.72	84.07
Persian	PerDT	fa	99.96	99.73	99.46	97.17	97.14	97.31	95.35	98.55	92.90	90.80	88.87	85.48	87.83
Persian	Seraji	fa	100.00	99.25	99.72	97.69	97.63	97.70	97.31	98.18	90.98	87.96	85.43	83.76	83.76
Polish	LFG	pl	99.92	99.91	99.92	98.97	94.73	95.80	94.27	97.24	96.71	95.24	93.79	88.67	90.38
Polish	PDB	pl	99.85	98.19	99.83	98.63	94.59	94.78	93.89	97.47	93.73	91.73	89.82	83.50	86.95
Pomak	Philotis	qpm	100.00	95.05	100.00	98.76	100.00	95.37	95.13	97.90	87.88	82.87	77.20	70.29	74.96
Portuguese	Bosque	pt	99.85	93.10	99.76	97.45	99.76	96.44	95.33	98.04	90.59	87.75	82.66	77.07	80.59
Portuguese	CINTIL	pt	100.00	85.34	99.93	97.19	95.74	95.48	92.94	98.56	84.08	80.09	74.75	68.90	73.19
Portuguese	PetroGold	pt	99.71	95.83	99.66	98.59	99.66	98.51	97.94	99.02	94.48	93.28	90.26	88.10	89.57
Romanian	Nonstandard	ro	98.96	97.53	98.96	95.60	90.85	89.71	88.32	94.96	88.97	84.88	80.51	67.17	76.58
Romanian	RRT	ro	99.73	95.72	99.73	97.50	96.80	97.10	96.74	97.42	90.70	86.86	82.99	79.90	80.83
Romanian	SiMoNERo	ro	99.77	100.00	99.77	97.98	97.55	97.19	96.99	98.69	93.45	91.44	88.50	84.36	87.34
Russian	GSD	ru	99.58	97.33	99.58	97.63	97.41	93.63	92.76	95.51	89.56	85.89	83.91	77.00	79.48
Russian	SynTagRus	ru	99.74	98.43	99.74	98.06	99.74	92.78	92.44	97.65	93.26	90.90	89.47	80.36	87.17
Russian	Taiga	ru	99.21	89.67	99.21	96.34	99.21	93.28	92.20	94.41	82.36	78.53	75.91	69.20	71.30
Sanskrit	Vedic	sa	100.00	32.93	100.00	89.19	100.00	80.70	76.91	88.92	62.52	50.64	49.90	43.27	46.84
Scottish Gaelic	ARCOSG	gd	99.65	64.54	98.86	94.62	90.24	91.91	89.42	95.81	82.73	77.81	72.86	66.53	70.20
Serbian	SET	sr	99.99	98.46	99.99	98.44	94.16	94.31	93.76	96.30	91.44	88.36	85.92	78.82	81.74
Slovak	SNK	sk	99.96	84.31	99.96	96.34	88.19	90.95	87.35	94.08	89.13	86.58	84.43	74.72	78.62
Slovenian	SSJ	sl	99.93	99.26	99.93	98.41	95.88	96.09	95.37	97.69	92.45	90.80	88.28	83.80	85.67
Slovenian	SST	sl	99.97	26.62	99.97	94.15	87.92	88.17	85.82	94.32	61.73	54.75	49.24	42.41	46.63
Spanish	AnCora	es	99.97	98.49	99.96	98.96	96.10	98.54	95.65	99.24	93.09	91.30	87.80	85.49	86.94
Spanish	GSD	es	99.94	95.60	99.80	97.07	99.80	96.42	94.77	98.42	92.01	89.57	84.56	76.86	82.66
Swedish	LinES	sv	99.94	87.91	99.94	97.34	95.07	90.68	87.95	97.19	89.02	85.25	82.28	69.82	79.13
Swedish	Talbanken	sv	99.97	97.49	99.97	98.28	97.13	97.29	96.24	97.92	90.32	87.85	85.91	81.88	83.42
Tamil	TTB	ta	99.27	98.74	89.61	78.51	77.53	79.63	73.03	82.55	58.39	50.97	48.82	41.40	43.87
Telugu	MTG	te	99.79	98.62	99.79	93.97	93.97	99.10	93.97	99.79	90.09	81.64	78.13	74.50	78.13
Turkish	Atis	tr	100.00	86.93	100.00	98.77	100.00	98.57	98.34	98.98	89.03	86.91	86.31	84.60	85.56
Turkish	BOUN	tr	99.58	90.08	96.32	89.92	85.74	81.66	73.03	90.37	71.53	65.17	63.79	48.49	58.96
Turkish	FrameNet	tr	100.00	100.00	100.00	95.71	100.00	91.89	91.14	95.16	92.50	83.78	81.37	71.69	76.07
Turkish	IMST	tr	99.86	98.16	97.75	93.44	93.27	91.06	89.07	95.55	73.39	64.07	60.96	54.70	59.51
Turkish	Kenet	tr	99.99	98.45	99.99	93.60	99.99	91.27	90.35	93.22	82.98	70.55	69.62	60.84	64.00
Turkish	Penn	tr	100.00	94.52	100.00	96.24	100.00	94.37	93.46	94.58	85.09	71.95	68.83	61.15	64.27
Turkish	Tourism	tr	100.00	100.00	100.00	99.08	100.00	94.90	94.64	98.90	96.99	91.31	88.74	81.60	87.58
Turkish German	SAGT	qtd	99.95	99.44	98.99	85.78	98.99	75.28	69.83	90.24	65.11	52.74	47.26	34.13	43.41
Ukrainian	IU	uk	99.79	97.21	99.76	97.52	93.01	93.11	92.07	96.72	88.60	85.87	82.77	75.63	79.55
Upper Sorbian	UFAL	hsb	100.00	86.36	100.00	95.87	100.00	78.48	77.17	93.70	79.57	74.13	65.71	45.36	60.36
Urdu	UDTB	ur	100.00	98.88	100.00	94.78	92.98	84.06	80.49	95.58	87.96	82.76	77.13	59.12	74.79
Uyghur	UDT	ug	99.42	84.69	99.42	89.26	91.19	87.58	80.06	95.30	74.80	63.42	58.96	48.01	55.81
Vietnamese	VTB	vi	88.02	99.25	88.02	81.48	80.61	88.02	80.56	81.66	58.82	51.49	46.60	42.58	42.47
Welsh	CCG	cy	99.90	97.79	99.68	95.86	94.60	90.11	87.83	94.85	86.32	80.25	74.85	62.81	69.32
Western Armenian	ArmTDP	hyw	99.95	99.56	99.91	96.42	99.91	92.70	92.04	96.70	87.73	82.83	77.33	68.89	74.38
Wolof	WTB	wo	99.95	94.33	99.35	94.36	94.16	93.19	91.38	94.63	83.28	77.32	71.22	64.89	67.65

Model Performance

Table of contents

System Performance on UD Treebanks