System Performance in CoNLL 2018 Shared Task

In the table below you can find the performance of the version 0.2.0 models of StanfordNLP’s neural pipeline, which is our best attempt at replicating our final system in the full PyTorch pipeline. The scores shown are from an end-to-end evaluation on the official test sets (from raw text to the full CoNLL-U file), and the scores are generated with the official evaluation script. For how we handled treebanks with no training data, please refer to the system description paper for details.

Note that these results are “unofficial” in the sense that these don’t reflect the systems’ state at the time of submission to the Shared Task (which had an unfortunate bug that we thereafter have fixed, as is also described in our paper).

Treebank Tokens Sentences Words UPOS XPOS UFeats AllTags Lemmas UAS LAS CLAS MLAS BLEX
Macro Avg 98.64 86.81 98.43 90.99 87.88 87.71 82.03 90.51 79.87 74.82 70.91 63.26 66.04
UD_Afrikaans-AfriBooms 99.67 100 99.67 97.51 94.26 97.25 94.25 96.78 87.05 84.12 78.21 74.25 74.11
UD_Ancient_Greek-Perseus 99.97 98.81 99.97 92.86 85.47 91.06 85.29 87.8 79.4 74.14 68.49 54.86 57.28
UD_Ancient_Greek-PROIEL 100 51.98 100 97.42 97.72 92.23 91.05 96.44 80.73 76.92 72.23 61.49 69.54
UD_Arabic-PADT 99.98 80.4 96.5 93.47 90.83 90.92 90.54 91.55 81.45 77.14 74.21 68.96 70.47
UD_Armenian-ArmTDP 95.66 87.45 95.09 72.09 95.09 55.37 52.93 67.84 50.12 32.7 25.13 12.44 15.17
UD_Basque-BDT 99.99 99.92 99.99 96.21 99.99 92.95 91.2 95.32 86.27 82.76 81.22 73.07 76.7
UD_Breton-KEB 91.66 89.54 90.9 33.49 0 23.28 0 50.12 33.45 10.94 7.16 0.02 1.87
UD_Breton-KEB 99.93 97.12 99.93 98.73 96.68 97.65 96.08 96.56 93.2 90.1 86.81 83.58 82.58
UD_Buryat-BDT 96.45 90.62 96.45 38.96 96.45 34.78 24.96 53.19 28.49 13.49 8.22 1.36 3.24
UD_Catalan-AnCora 99.99 99.84 99.98 98.66 98.66 98 97.45 97.71 92.8 90.48 86.27 83.11 84.13
UD_Chinese-GSD 92.6 99.1 92.6 88.84 88.7 91.87 87.91 92.6 73.99 70.73 67.01 62.22 67.01
UD_Croatian-SET 99.91 97.59 99.91 97.93 99.91 91.7 91.14 95.4 90.93 86.31 83.46 73.17 78.56
UD_Czech-CAC 99.99 100 99.97 98.78 95.16 93.6 92.87 97.44 91.65 89.19 86.84 79.82 84.06
UD_Czech-FicTree 99.98 99.19 99.97 98.3 95.15 95.9 94.56 97.91 93.26 90.35 87.72 81.85 85.07
UD_Czech-PDT 99.98 94.06 99.97 98.6 95.42 94.66 93.84 98.2 91.85 89.6 87.84 81.56 85.88
UD_Czech-PUD 99.15 95.5 99 96.06 92.18 91.39 89.79 95.47 89.24 84.17 81.41 73.02 78.8
UD_Danish-DDT 99.96 93.06 99.96 97.82 99.96 97.53 96.61 95.63 86.33 83.86 80.69 76.54 75.6
UD_Dutch-Alpino 99.96 90.86 99.96 95.99 94.33 95.88 93.65 96.27 90.05 86.94 81.71 75.31 77
UD_Dutch-LassySmall 99.92 78.85 99.92 96.01 94.9 96.19 94.26 96.27 86.6 83.06 77.16 73.08 73.5
UD_English-EWT 99.16 81.27 99.16 95.62 95.36 96.18 94.15 96.52 86.22 83.69 80.36 75.91 77.63
UD_English-GUM 99.81 81.76 99.81 95.54 95.45 96.68 94.58 96.14 85.44 81.99 76.64 72.18 72.32
UD_English-LinES 99.92 88.03 99.92 96.71 95.39 96.79 93.23 96.29 83.51 78.65 76.07 71.39 72.04
UD_English-PUD 99.86 96.15 99.86 95.79 94.82 95.64 92.04 95.26 88.6 85.86 83.12 75.4 77.91
UD_Estonian-EDT 99.95 93.82 99.95 97.13 98.03 95.73 94.37 94.99 87.15 84.11 82.45 77.25 77.19
UD_Faroese-OFT 99.54 84.4 99.54 59.82 0 31.35 0 37.45 51.14 41.54 34.32 0.26 13.37
UD_Finnish-FTB 100 90.24 99.97 95.92 95.22 96.51 94.27 95.22 89.57 86.96 84.04 80.08 79.91
UD_Finnish-PUD 99.46 91.25 99.46 97.19 0 96.52 0 92.55 90.09 88.05 86.24 82.26 78.46
UD_Finnish-TDT 99.76 91.82 99.76 96.86 97.62 95.26 94.37 93.18 88.92 86.44 84.94 79.77 77.85
UD_French-GSD 99.67 95.4 99.37 96.97 99.37 96.61 95.77 96.97 89.21 86.38 82.43 78.08 79.54
UD_French-Sequoia 99.85 90.61 99.65 98.43 99.65 97.8 97.36 97.42 90.42 88.51 85.13 82.27 82.33
UD_French-Spoken 100 19.59 100 95.99 97.47 100 93.62 94.74 75.23 69.56 61.03 58.24 57.57
UD_Galician-CTG 99.89 98.26 99.37 97.25 96.96 99.2 96.68 97.45 85.31 82.69 77.31 71.19 75.04
UD_Galician-TreeGal 99.46 88.5 98.22 93.87 91.42 92.78 90.52 92.93 76.91 71.63 64.16 57.93 58.36
UD_German-GSD 99.55 85.07 99.55 93.8 96.75 89.31 83.95 95.95 83.13 78.61 74.55 57.43 70.05
UD_Gothic-PROIEL 100 40.54 100 95.97 96.64 90.74 89 95.71 76.89 70.56 68.08 59.35 65.53
UD_Greek-GDT 99.91 92.74 99.92 97.86 97.86 94.86 94.22 95.6 90.91 88.48 84.03 77.5 77.91
UD_Hebrew-HTB 99.98 100 93.32 90.59 90.59 89.3 88.62 89.96 79.57 75.6 69 63.22 64.95
UD_Hindi-HDTB 100 99.35 100 97.45 96.97 93.81 91.91 96.42 94.92 91.79 88.18 78.31 86.59
UD_Hungarian-Szeged 99.8 96.54 99.8 95.97 99.8 93.66 92.92 91.66 83.24 78.18 76.87 69.03 68.14
UD_Indonesian-GSD 100 93.94 100 93.7 94.53 95.87 88.91 99.63 85.22 79.17 77 68.53 76.66
UD_Irish-IDT 99.71 96.69 99.71 92.52 90.99 82.84 78.96 89.12 79.6 70.34 61.11 45.28 52.27
UD_Italian-ISDT 99.9 98.76 99.77 97.92 97.86 97.84 97.19 97.34 92.96 91.12 86.83 83.9 83.2
UD_Italian-PoSTWITA 99.73 62.97 99.41 95.99 95.84 96.08 94.78 95.43 82.3 78.13 71.84 68.33 68.29
UD_Japanese-GSD 92.24 94.92 92.24 90.81 92.24 92.24 90.81 91.55 80.14 78.48 68.58 66.93 68.03
UD_Japanese-Modern 70.39 0 70.39 52.66 0 68.36 0 57.84 34.93 27.79 14.78 10.86 13.81
UD_Kazakh-KTB 95.13 81.4 95.67 57.9 58.22 43.68 37.17 55.29 45.05 26.25 20.62 8.22 9.58
UD_Korean-GSD 99.9 96.59 99.9 96.17 90.41 99.7 88.29 92.24 87.6 83.8 81.64 79.44 74.93
UD_Korean-Kaist 100 99.93 100 95.46 86.09 100 86.08 92.48 88.39 86.48 84.1 80.62 77.03
UD_Kurmanji-MG 94.12 88.61 93.79 54.63 53.8 39.34 33.36 56.24 33.15 24.15 18.03 3.99 8.15
UD_Latin-ITTB 99.69 79.95 99.69 98.16 94.77 96.29 93.65 98.38 87.97 85.59 84.32 80.02 83.53
UD_Latin-Perseus 100 99.31 100 91.01 78.3 82.22 77.66 80.03 71.89 62.17 57.77 45.23 44.09
UD_Latin-PROIEL 100 42.57 100 96.74 96.93 91.25 90.38 95.85 75.85 71.89 69.27 60.38 66.76
UD_Latvian-LVTB 99.71 98.38 99.71 94.89 86.89 91.93 86.2 92.2 85.45 81.19 78.16 69.09 70.93
UD_Naija-NSC 95.27 2.03 95.27 56.9 0 34.26 0 89.21 37.46 23.08 24.35 3.42 22.54
UD_North_Sami-Giella 99.76 99.65 99.76 91.97 93.29 88.18 84.73 88.07 76.63 70.99 68.07 60.62 58.8
UD_Norwegian-Bokmaal 99.85 97.78 99.85 98.11 99.85 97.05 96.28 97.47 92.08 90.35 88.16 84.04 85.03
UD_Norwegian-Nynorsk 99.95 95 99.95 97.84 99.95 96.69 95.85 97.03 91.61 89.58 87.34 82.38 83.58
UD_Norwegian-NynorskLIA 100 100 100 91.87 100 90.25 86.46 93.27 66.45 58.11 52.4 44.78 48.01
UD_Old_Church_Slavonic-PROIEL 100 48.48 100 96.39 96.73 90.77 89.78 94.98 80.85 75.65 74.57 65.7 71.52
UD_Old_French-SRCMF 100 100 100 96.07 96.09 97.75 95.6 100 91.56 86.67 83.77 80.31 83.77
UD_Persian-Seraji 100 99.25 99.66 97.13 97.16 97.33 96.69 97.43 89.41 86.17 82.91 80.73 80.78
UD_Polish-LFG 99.94 99.91 99.94 98.43 94.49 95.5 93.77 95.94 96 94.09 92.38 86.92 87.31
UD_Polish-SZ 100 99.46 99.9 97.87 92.44 92.51 91.35 94.64 93 90.66 88.96 80.44 82.43
UD_Portuguese-Bosque 99.71 93.14 99.55 96.71 99.55 96.41 94.34 96.88 90.25 87.98 84 77.26 80.15
UD_Romanian-RRT 99.79 95.28 99.79 97.49 96.89 97.14 96.69 96.98 90.99 86.38 82.29 78.32 79.08
UD_Russian-SynTagRus 99.61 99.1 99.61 98.29 99.61 96.16 95.8 97.5 92.81 91.14 89.51 85.44 87.18
UD_Russian-Taiga 97.33 77.58 97.33 91.55 97.29 84.24 82.56 84.35 69.07 62.26 58.18 46.96 48.11
UD_Serbian-SET 99.96 95.32 99.96 98.16 99.96 94.03 93.71 95.03 92.12 88.49 86.19 78.14 80.23
UD_Slovak-SNK 99.98 90.34 99.98 96.45 86.66 91.42 86.13 93.36 89.92 86.87 84.69 75.06 77.34
UD_Slovenian-SSJ 99.9 98.86 99.9 98.25 95.38 95.66 94.84 95.79 92.92 91.02 88.49 83.71 83.12
UD_Slovenian-SST 100 26.8 100 94.21 87.74 87.73 85.48 92.14 63.62 56.76 51.66 44.68 47.29
UD_Spanish-AnCora 99.96 99.39 99.95 98.71 98.71 98.28 97.69 98.08 91.88 89.72 85.74 83.19 83.53
UD_Swedish-LinES 99.93 85.85 99.93 96.67 94.66 89.52 86.81 95.33 84.78 80.3 78.54 65.63 73.42
UD_Swedish-PUD 98.94 89.38 98.94 93.57 91.66 77.99 75.98 84.27 82.53 78.39 75.75 50.91 61.16
UD_Swedish-Talbanken 99.97 98.11 99.97 97.6 96.48 96.62 95.46 97 88.84 85.69 83.29 78.49 79.81
UD_Thai-PUD 8.56 0.39 8.56 5.86 0.02 5.67 0 8.56 0.84 0.7 0.42 0.03 0.42
UD_Turkish-IMST 99.85 98.11 97.87 94.13 93.3 92.39 90.66 93.89 70.62 64.43 61.54 56.67 59.18
UD_Ukrainian-IU 99.7 97.58 99.7 96.63 92.1 91.96 90.87 94.29 88.17 84.86 81.43 73.57 75.39
UD_Upper_Sorbian-UFAL 92.7 84.17 92.7 54.54 92.7 36.91 34.95 55.67 34.16 22.97 16.29 2.83 6.95
UD_Urdu-UDTB 99.98 98.88 99.98 94.51 92.75 83.89 80.32 95.48 88.44 82.58 76.34 58.51 73.97
UD_Uyghur-UDT 99.82 85.4 99.82 89.43 91.96 88.15 80.48 95.65 76.04 64.23 57.73 46.25 54.57
UD_Vietnamese-VTB 87.18 93.15 87.18 79.55 77.93 86.95 77.9 87.12 53.41 48.01 44.7 41.72 44.65