Link

Past Performance 1.3

Table of contents


Here we report the performance of Stanza’s pretrained models on all supported languages as of version 1.3. Please see the newer performance for an updated chart. Performances of models for tokenization, multi-word token (MWT) expansion, lemmatization, part-of-speech (POS) and morphological features tagging and dependency parsing are reported on the Universal Dependencies (UD) treebanks as of UD 2.8.

System Performance on UD Treebanks

In the table below you can find the performance of Stanza’s pretrained UD models. All models are trained and tested with the Universal Dependencies v2.8 treebanks.

Note that all scores reported are from an end-to-end evaluation on the official test sets (from raw text to the full CoNLL-U file), and are generated with the CoNLL 2018 UD shared task official evaluation script. For detailed interpretation of these scores and the evaluation scripts we used, please refer to the CoNLL 2018 UD Shared Task Evaluation page. For details on how we handled treebanks with no training data, please refer to our CoNLL 2018 system description paper.

Between v1.0 and v1.3, some data augmentation was added, especially for POS to avoid POS errors with comment patterns such as no punct at the end of a sentence. In general, this did not harm or help performance by much. Results can be compared to the older results as of v1.0 on UD 2.5.

TreebankTokensSentencesWordsUPOSXPOSUFeatsAllTagsLemmasUASLASCLASMLASBLEX
Macro Avg99.5288.1199.1394.8194.4092.7190.0195.0583.6279.0475.0868.2271.83
UD_Afrikaans-AfriBooms99.93100.0099.9397.9793.9797.2293.8897.3987.3883.5776.6672.0773.00
UD_Ancient_Greek-PROIEL100.0052.02100.0097.3897.7492.2091.0397.2181.2777.3872.6062.2270.27
UD_Ancient_Greek-Perseus99.9998.9399.9992.3684.9591.0384.7587.5879.2673.6567.8353.8656.75
UD_Arabic-PADT99.9882.8798.6195.6192.4992.6492.3294.1684.4079.9476.8771.1573.54
UD_Armenian-ArmTDP99.9299.4699.4895.7099.4891.6690.6394.9183.8277.9573.5065.8869.97
UD_Basque-BDT99.99100.0099.9996.0999.9992.9391.1996.2686.1982.7681.1273.2877.95
UD_Belarusian-HSE98.4888.1298.4896.7594.9091.9789.9894.0785.2982.8079.4372.7974.80
UD_Bulgarian-BTB99.9396.2999.9398.8396.3497.5795.8897.3193.3090.1686.7683.4283.58
UD_Catalan-AnCora99.9799.5999.9598.7799.9598.3897.9598.5793.1790.9786.2083.8485.38
UD_Chinese-GSD93.9998.8093.9990.2690.0593.1889.2393.9974.2970.9868.3164.4168.31
UD_Chinese-GSDSimp93.8998.8093.8990.0489.8093.0688.9893.8975.1471.9169.3165.3369.31
UD_Classical_Chinese-Kyoto97.1741.7397.1787.8086.6990.6284.4396.9368.2162.7361.3659.0361.26
UD_Coptic-Scriptorium100.0034.8387.1383.9683.8484.8682.9784.0164.4562.4453.7850.6552.46
UD_Croatian-SET99.9497.6199.9497.8294.6895.0893.9696.5990.4486.7983.9277.4380.09
UD_Czech-CAC99.92100.0099.8198.5394.3493.0892.1997.6591.1188.6486.4279.3984.40
UD_Czech-CLTT99.1997.7999.1298.1391.4391.6090.8096.7086.4583.9381.0572.6278.98
UD_Czech-FicTree99.9898.9599.9798.3195.1296.0194.6098.2793.2690.7388.2582.8886.15
UD_Czech-PDT99.9894.9999.9898.6095.3994.6393.6898.4392.2590.1888.4682.1086.82
UD_Danish-DDT99.9091.6599.9097.6499.9097.3596.3696.9786.6384.3681.3377.3078.09
UD_Dutch-Alpino99.5288.0799.5295.8794.4896.0093.7494.0389.4386.4181.7376.3574.28
UD_Dutch-LassySmall99.8880.7799.8895.9895.1196.3394.3095.2487.0683.3477.2673.2072.12
UD_English-EWT99.3689.0999.1195.6495.4096.2394.0296.9187.4184.9181.6077.1879.35
UD_English-GUM99.2588.9899.4296.1596.1595.3994.0097.5588.3085.6881.5776.6079.49
UD_English-LinES99.9690.0599.9696.9296.1796.7194.0798.2186.8082.9278.5173.8176.71
UD_English-ParTUT99.66100.0099.5796.1195.7995.2693.9797.2089.0986.3681.0975.3478.61
UD_Estonian-EDT99.9593.1099.9597.2198.0795.8694.5195.9187.1584.4382.9377.7878.75
UD_Estonian-EWT98.7878.9698.7892.5594.4591.4388.5089.8176.9771.9469.0162.1561.63
UD_Faroese-FarPaHC99.7294.4599.7097.0091.7693.1591.2099.7084.0479.6871.7464.5771.74
UD_Finnish-FTB100.0090.3899.9895.4795.1596.5293.9495.9989.5586.7484.0379.5580.97
UD_Finnish-TDT99.8091.2999.7696.9497.7695.4194.4594.7088.8986.5485.0979.9780.28
UD_French-GSD99.7395.7799.4697.4299.4597.5596.7897.8891.9889.6085.1881.9883.45
UD_French-ParTUT99.88100.0099.4296.5496.1594.0093.3196.0491.4288.9684.1875.6579.95
UD_French-Sequoia99.8690.9999.5598.3999.5597.7197.2598.2291.1488.9085.3582.6883.92
UD_French-Spoken99.9723.4499.3895.6396.8392.8288.4795.7877.2370.9562.2156.1160.61
UD_Galician-CTG99.8698.7899.2597.1496.9199.0996.6397.8585.0682.5377.1270.8775.77
UD_Galician-TreeGal99.6188.8198.5194.1391.8693.2090.9794.6478.0672.9965.7258.9962.03
UD_German-GSD99.4982.8199.4794.4096.9389.2884.6896.2285.2980.6675.5059.2071.21
UD_German-HDT100.0097.10100.0097.9597.8391.3290.8797.5094.7192.2688.2075.9785.16
UD_Gothic-PROIEL100.0040.62100.0096.1096.6990.6988.9396.0275.2169.4566.8257.9564.51
UD_Greek-GDT99.9493.0899.9397.6497.6494.9394.2796.2891.5389.1484.4078.0779.63
UD_Hebrew-HTB100.00100.0092.8290.1590.1588.7888.0589.9379.6276.9370.2963.8467.25
UD_Hindi-HDTB100.0099.29100.0097.5597.0793.8791.9896.8094.7491.6788.1278.2986.82
UD_Hungarian-Szeged99.9297.4599.9296.0099.9293.6292.7394.1984.1979.2377.2368.8271.61
UD_Icelandic-IcePaHC99.9193.1899.8796.2392.0289.7284.5995.6287.0983.1477.6464.1873.82
UD_Icelandic-Modern99.9798.8399.9798.5997.5398.0197.3298.7093.4291.9189.5487.2788.29
UD_Indonesian-CSUI99.5892.9099.0895.6996.0996.6595.3996.1281.4076.3973.1970.4270.58
UD_Indonesian-GSD99.9894.1399.9893.8594.5495.7589.1299.6586.5380.3677.4069.0377.06
UD_Irish-IDT99.8896.9199.8896.0894.8789.3486.5595.2387.2081.6676.4063.2471.74
UD_Italian-ISDT99.9499.0799.8397.9397.8897.7997.1598.0192.7890.9986.7183.8084.33
UD_Italian-ParTUT99.81100.0099.7997.7997.6297.4696.6397.6091.9989.5783.8980.5581.27
UD_Italian-PoSTWITA99.6366.7299.3696.1695.9796.2594.9096.4983.2478.8573.1169.6171.00
UD_Italian-TWITTIRO99.2355.2498.9194.4693.8693.2591.1692.9879.1472.5563.1057.5357.34
UD_Italian-VIT99.9895.9999.5397.4196.4997.2795.5597.9989.2586.0280.3876.6178.57
UD_Japanese-GSD96.7999.7296.7995.5895.0196.7694.7796.2888.5087.6482.8081.4082.42
UD_Korean-GSD99.8595.7499.8596.0990.7999.5988.5293.6587.4683.7181.6179.2776.22
UD_Korean-Kaist100.0099.93100.0095.5886.27100.0086.2794.2588.5186.5384.1580.8279.07
UD_Latin-ITTB99.9980.7399.9997.5595.7895.7393.5799.0788.7686.6685.4780.4985.06
UD_Latin-LLCT100.0099.49100.0099.5096.7396.7996.4198.0895.9794.8193.7189.7391.72
UD_Latin-PROIEL100.0042.28100.0096.9897.1391.2690.4696.7176.9972.8669.7360.9567.82
UD_Latin-Perseus100.0098.99100.0090.9778.1882.4477.5382.7071.2461.5857.0644.5545.88
UD_Latin-UDante99.9698.4599.6489.5875.0679.5372.9287.3468.3058.7748.6135.3741.99
UD_Latvian-LVTB99.8398.9799.8396.4288.5893.6987.9995.6189.0985.8983.6375.4079.82
UD_Lithuanian-ALKSNIS99.9489.3899.9493.5485.8887.1584.5592.7878.8173.6671.0259.6965.84
UD_Lithuanian-HSE98.8751.1198.8782.8981.8574.3969.0077.0350.6639.4134.5724.4027.31
UD_Maltese-MUDT99.8385.3899.8395.7995.7099.8395.3999.8383.2377.9770.4166.6170.41
UD_Marathi-UFAL97.5992.6394.8576.2394.8564.2259.8076.7269.1257.8450.5229.6541.34
UD_Naija-NSC99.97100.0099.9797.5999.9798.9397.1999.0292.4389.4988.5286.5088.09
UD_North_Sami-Giella99.7199.8399.7190.4492.4687.1082.8088.1272.8766.6664.1457.5156.48
UD_Norwegian-Bokmaal99.8797.6599.8798.0499.8797.0796.2998.1092.1690.3587.8683.8085.69
UD_Norwegian-Nynorsk99.9694.2999.9697.7099.9696.7795.8297.7391.4989.5987.0982.3384.41
UD_Norwegian-NynorskLIA100.0099.69100.0096.09100.0095.2693.1197.5578.0573.2568.1062.1465.76
UD_Old_Church_Slavonic-PROIEL100.0050.67100.0096.5196.8390.5989.6095.6279.6175.1074.5465.9571.79
UD_Old_East_Slavic-RNC97.8781.6097.8786.8587.1166.3059.3676.2060.8651.8640.8225.8030.41
UD_Old_East_Slavic-TOROT100.0032.78100.0093.4093.5586.4384.2890.8372.4766.4063.0752.4658.51
UD_Old_French-SRCMF100.00100.00100.0096.0695.8997.6195.31100.0090.6885.7181.8679.0081.86
UD_Persian-PerDT99.9699.8699.6397.0396.9697.2995.1198.6492.6790.3988.2984.5787.21
UD_Persian-Seraji100.0099.2599.6797.1997.2697.3696.8097.8189.8086.5183.5181.4881.82
UD_Polish-LFG99.9199.9199.9198.4394.3595.4293.6796.5595.7593.7892.0086.6987.98
UD_Polish-PDB99.8798.2699.8498.2694.1094.3893.3397.2893.2890.9788.8682.2085.85
UD_Portuguese-Bosque99.8191.5999.7497.0799.7496.1494.7397.7991.0987.9983.0576.4380.71
UD_Portuguese-GSD99.9297.8499.8298.1298.1099.6698.0399.1491.5987.8583.7880.0082.96
UD_Romanian-Nonstandard98.8697.5398.8695.2890.5789.4288.0394.8188.7184.4780.1266.4576.35
UD_Romanian-RRT99.7395.4099.7397.6396.9597.1696.8797.5290.5686.5182.6079.1980.45
UD_Romanian-SiMoNERo99.62100.0099.6297.8697.4197.0596.8598.5792.9791.1088.2784.2887.14
UD_Russian-GSD99.6597.1699.6597.3897.1893.0892.1995.0389.0184.8182.5275.2177.41
UD_Russian-SynTagRus99.5998.9599.5998.2799.5995.9995.6797.4692.7991.1389.5185.2287.15
UD_Russian-Taiga99.0088.3099.0095.7899.0092.7391.5394.1680.9076.4873.2666.8468.74
UD_Sanskrit-Vedic100.0033.21100.0088.88100.0080.4677.0188.5261.5949.3348.6542.4545.49
UD_Scottish_Gaelic-ARCOSG99.7265.3798.8594.2789.1290.8688.1195.3382.8477.6171.4764.5268.34
UD_Serbian-SET99.9999.3399.9998.3294.0894.4193.6496.3291.8488.9286.5079.2682.36
UD_Slovenian-SSJ99.9098.1099.9098.3395.1395.4394.6297.0792.7290.9788.3783.2184.98
UD_Slovenian-SST100.0027.46100.0093.2687.6787.6484.6394.1562.6055.2649.5142.5246.72
UD_Spanish-AnCora99.9798.3299.9598.7599.9598.3597.8999.2092.1389.9485.6283.1984.86
UD_Spanish-GSD99.9195.6099.8296.6199.8296.4394.4498.3689.7887.0781.9374.1079.79
UD_Swedish-LinES99.9287.2099.9296.6594.4190.0987.2696.6488.0683.8680.4867.8777.11
UD_Swedish-Talbanken99.9898.0399.9897.6696.4696.6895.5997.5188.6485.5183.0878.5380.31
UD_Swedish_Sign_Language-SSLC100.003.39100.0077.6678.01100.0074.82100.0023.0513.1213.6510.8413.65
UD_Tamil-TTB99.3298.7692.1279.9279.3181.5974.6084.4864.4756.0653.5245.7348.60
UD_Telugu-MTG99.7998.9799.7993.1493.1498.9693.1499.7990.6481.2276.1271.9376.12
UD_Turkish-BOUN99.3890.2598.6290.7091.6492.1887.1093.5675.9569.6467.5258.3063.37
UD_Turkish-FrameNet100.00100.00100.0094.89100.0092.3091.4893.0590.9380.8578.2369.8171.27
UD_Turkish-IMST99.9497.8798.2392.4993.3490.8688.9195.1771.7465.4661.8455.9060.04
UD_Turkish-Kenet99.9998.8799.9993.4499.9991.4390.5292.7182.7269.6868.9060.0263.10
UD_Turkish-Penn100.0089.84100.0094.93100.0092.6091.9393.5886.3476.7074.7264.1268.53
UD_Turkish-Tourism100.0099.93100.0099.01100.0094.9794.7898.9595.9090.2887.6480.4986.63
UD_Turkish_German-SAGT99.9699.0799.0485.7199.0475.4869.8890.6464.7952.3046.7033.9442.83
UD_Ukrainian-IU99.7897.4899.7596.7492.4092.4791.2996.2387.8384.7181.4073.8477.77
UD_Urdu-UDTB100.0098.88100.0094.6192.7484.4380.5995.6488.1382.5376.8859.1874.59
UD_Uyghur-UDT99.4684.7299.4688.7691.1787.5279.8795.1674.7462.9156.6645.4053.67
UD_Vietnamese-VTB89.2295.4589.2281.6579.9189.0379.8889.2256.0150.4147.5044.4847.50
UD_Welsh-CCG99.8497.8999.2993.9792.5889.0686.2592.5683.2176.1669.7758.4562.76
UD_Western_Armenian-ArmTDP99.8198.1199.6992.3899.6985.2184.1293.3378.8370.1160.8947.6256.10
UD_Wolof-WTB99.9392.3199.3793.8693.6192.8390.7693.9683.5277.4171.4764.4167.16