Model performance

News: Trankit v1.0.0 is out. The new version has pretrained pipelines using XLM-Roberta Large which is much better than our previous version with XLM-Roberta Base. Checkout the performance comparison below on this page. Trankit v1.0.0 also provides a brand new command-line interface that helps users who are not familiar with Python can use Trankit more easily. Finally, the new version has a brand new Auto Mode in which a language detector is used to activate the language-specific models for processing the input, avoiding switching back and forth between languages in a multilingual pipeline.

Universal Dependencies v2.5

The following table shows the performance comparison between Trankit large (using XLM-Roberta large) and base (using XLM-Roberta base), spaCy v2.3, UDPipe v1.2, and Stanza v1.1.1 on the test treebanks of the 5 major languages on the Universal Dependencies v2.5 corpora. Performance for each model are F1 scores obtained by using the official evaluation script of the CoNLL 2018 Shared Task. The meanings of the columns in the following tables are provided by the evaluation script as follows:

  • Tokens: how well do the gold tokens match system tokens.

  • Sents.: how well do the gold sentences match system sentences.

  • Words: how well can the gold words be aligned to system words.

  • UPOS: using aligned words, how well does UPOS match.

  • XPOS: using aligned words, how well does XPOS match.

  • UFeats: using aligned words, how well does universal FEATS match.

  • AllTags: using aligned words, how well does UPOS+XPOS+FEATS match.

  • Lemmas: using aligned words, how well does LEMMA match.

  • UAS: using aligned words, how well does HEAD match.

  • LAS: using aligned words, how well does HEAD+DEPREL(ignoring subtypes) match.

  • CLAS: using aligned words with content DEPREL, how well does HEAD+DEPREL(ignoring subtypes) match.

  • MLAS: using aligned words with content DEPREL, how well does HEAD+DEPREL(ignoring subtypes)+UPOS+UFEATS+FunctionalChildren(DEPREL+UPOS+UFEATS) match.

  • BLEX: using aligned words with content DEPREL, how well does HEAD+DEPREL(ignoring subtypes)+LEMMAS match.

Treebank

System

Tokens

Sents.

Words

UPOS

XPOS

UFeats

Lemmas

UAS

LAS

Arabic-PADT

Trankit[large]

99.95

96.79

99.39

96.8

94.93

94.99

94.92

89.74

85.98

Trankit[base]

99.93

96.59

99.22

96.31

94.08

94.28

94.65

88.39

84.68

Stanza

99.98

80.43

97.88

94.89

91.75

91.86

93.27

83.27

79.33

UDPipe

99.98

82.09

94.58

90.36

84.00

84.16

88.46

72.67

68.14

Chinese-GSD

Trankit[large]

97.75

99.4

97.75

95.13

94.94

97.28

97.75

87.38

84.82

Trankit[base]

97.01

99.7

97.01

94.21

94.02

96.59

97.01

85.19

82.54

Stanza

92.83

98.80

92.83

89.12

88.93

92.11

92.83

72.88

69.82

UDPipe

90.27

99.10

90.27

84.13

84.04

89.05

90.26

61.60

57.81

English-EWT

Trankit[large]

98.67

90.49

98.67

96.65

96.47

96.75

97.25

91.29

89.4

Trankit[base]

98.48

88.35

98.48

95.95

95.71

96.26

96.84

90.14

87.96

Stanza

99.01

81.13

99.01

95.40

95.12

96.11

97.21

86.22

83.59

UDPipe

98.90

77.40

98.90

93.26

92.75

94.23

95.45

80.22

77.03

spaCy

97.30

61.19

97.30

86.72

90.83

87.05

French-GSD

Trankit[large]

99.77

98.8

99.72

97.89

97.34

97.9

94.97

93.37

Trankit[base]

99.7

96.63

99.66

97.85

97.16

97.80

94.00

92.34

Stanza

99.68

94.92

99.48

97.30

96.72

97.64

91.38

89.05

UDPipe

99.68

93.59

98.81

95.85

95.55

96.61

87.14

84.26

spaCy

98.34

77.30

94.15

86.82

87.29

67.46

60.60

Spanish-Ancora

Trankit[large]

99.91

99.39

99.91

99.06

99

98.85

99.15

94.77

93.29

Trankit[base]

99.94

99.13

99.93

99.02

98.94

98.80

99.17

94.11

92.41

Stanza

99.98

99.07

99.98

98.78

98.67

98.59

99.19

92.21

90.01

UDPipe

99.97

98.32

99.95

98.32

98.13

98.13

98.48

88.22

85.10

spaCy

99.47

97.59

98.95

94.04

79.63

86.63

84.13

As can be seen from the table, both the large and base version of Trankit outperform other toolkits over different tasks (e.g., POS and morphological tagging) in which the improvement boost is substantial and significant for sentence segmentation and dependency parsing. For example, English enjoys a 9.36% improvement for sentence segmentation, a 5.07% and 5.81% improvement for UAS and LAS in dependency parsing. For Arabic, Transit has a remarkable improvement of 16.36% for sentence segmentation while Chinese observes 14.50% and 15.0% improvement of UAS and LAS for dependency parsing.

Next, we show the detailed performance comparison of Trankit large (using XLM-Roberta large), Trankit base (using XLM-Roberta base), and Stanza v1.1.1 on 90 Universal Dependencies v2.5 treebanks. Over all 90 treebanks, both versions of Trankit outperform the previous state-of-the-art framework Stanza in most of the tasks, particularly for sentence segmentation (+4.28%), POS tagging (+2.00% for UPOS and +2.14% for XPOS), morphological tagging (+2.18%), and dependency parsing (+5.61% for UAS and +6.81% for LAS) while maintaining the competitive performance on tokenization, multi-word token expansion, and lemmatization.

Tokens

Sentences

Words

UPOS

XPOS

UFeats

AllTags

Lemmas

UAS

LAS

CLAS

MLAS

BLEX

Overall performance

Trankit[large]

99.32

92.86

99.15

96.22

94.64

93.93

91.19

94.47

88.67

85.49

82.97

76.17

78.45

Trankit[base]

99.23

91.82

99.02

95.65

94.05

93.21

90.29

94.27

87.06

83.69

80.88

73.57

76.53

Stanza

99.26

88.58

98.90

94.21

92.50

91.75

88.62

94.15

83.06

78.68

74.65

67.83

71.28

Trankit[large] - Trankit[base]

0.09

1.04

0.13

0.56

0.59

0.72

0.91

0.20

1.61

1.80

2.10

2.60

1.92

Trankit[large] - Stanza

0.07

4.28

0.26

2.00

2.14

2.18

2.57

0.32

5.61

6.81

8.32

8.34

7.17

Afrikaans-AfriBooms

Trankit[large]

99.84

100.00

99.84

98.87

96.63

98.68

96.54

97.47

91.61

89.35

85.30

82.64

81.65

Trankit[base]

99.72

100.00

99.72

98.55

96.10

98.26

95.89

97.39

91.03

88.79

84.46

81.31

80.91

Stanza

99.75

99.65

99.75

97.56

94.27

97.03

94.24

97.48

87.51

84.45

78.58

74.70

75.39

Ancient_Greek-Perseus

Trankit[large]

99.71

98.70

99.71

93.97

87.25

91.66

86.88

88.52

83.48

78.56

73.79

60.72

61.97

Trankit[base]

99.66

98.70

99.66

92.72

84.89

89.95

84.30

88.18

80.95

75.57

70.26

55.75

58.84

Stanza

99.98

98.85

99.98

92.54

85.22

91.06

84.98

88.26

78.75

73.35

67.88

54.22

57.54

Ancient_Greek-PROIEL

Trankit[large]

99.91

67.60

99.91

97.86

97.93

93.03

92.02

97.50

85.63

82.31

78.16

68.27

75.76

Trankit[base]

99.83

58.62

99.83

97.30

97.37

91.59

90.33

97.31

83.21

79.68

74.96

64.13

72.80

Stanza

100.00

51.65

100.00

97.38

97.75

92.09

90.96

97.42

80.34

76.33

71.37

61.23

69.23

Arabic-PADT

Trankit[large]

99.95

96.79

99.39

96.80

94.93

94.99

94.61

94.92

89.74

85.98

83.92

78.47

79.96

Trankit[base]

99.93

96.59

99.22

96.31

94.08

94.28

93.69

94.65

88.39

84.68

82.35

76.46

78.46

Stanza

99.98

80.43

97.88

94.89

91.75

91.86

91.51

93.27

83.27

79.33

76.24

70.58

72.79

Armenian-ArmTDP

Trankit[large]

98.52

99.46

97.26

95.03

97.26

90.42

89.56

93.45

86.47

83.01

80.91

72.45

77.07

Trankit[base]

98.41

98.92

97.23

94.36

97.23

88.89

87.84

93.28

84.22

80.14

77.47

67.92

73.94

Stanza

98.96

95.22

96.58

92.49

96.58

88.19

86.94

92.27

78.18

72.46

68.50

60.39

65.88

Basque-BDT

Trankit[large]

99.89

100.00

99.89

96.95

99.89

93.96

92.30

96.69

90.22

87.47

86.54

78.10

83.21

Trankit[base]

99.81

99.92

99.81

96.23

99.81

92.35

90.40

96.46

88.24

85.25

84.29

74.20

80.92

Stanza

100.00

100.00

100.00

96.23

100.00

93.09

91.34

96.52

86.19

82.76

81.29

73.56

78.26

Belarusian-HSE

Trankit[large]

99.81

82.28

99.81

96.46

31.77

82.82

27.96

80.37

77.48

73.36

70.18

51.71

52.39

Trankit[base]

99.53

84.68

99.53

91.58

36.18

78.47

27.19

79.48

69.40

65.02

64.51

46.24

48.41

Stanza

99.38

78.24

99.38

91.92

31.34

77.73

26.31

79.48

69.28

63.88

58.49

41.88

44.05

Bulgarian-BTB

Trankit[large]

99.78

98.79

99.78

99.29

97.75

98.30

97.29

97.37

96.30

94.21

92.19

89.64

88.79

Trankit[base]

99.84

98.25

99.84

99.18

97.42

98.10

96.91

97.36

95.81

93.47

91.23

88.28

87.80

Stanza

99.93

97.27

99.93

98.68

96.35

97.59

95.75

97.29

93.37

90.21

86.84

83.71

83.62

Catalan-AnCora

Trankit[large]

99.94

99.76

99.94

99.11

99.07

98.66

98.24

98.49

95.15

93.83

91.41

89.31

90.27

Trankit[base]

99.94

100.00

99.93

99.02

98.97

98.57

98.09

98.46

94.61

93.01

90.09

87.89

88.99

Stanza

99.99

99.84

99.98

98.75

98.66

98.29

97.74

98.47

92.84

90.56

86.25

84.07

85.31

Chinese-GSD

Trankit[large]

97.75

99.40

97.75

95.13

94.94

97.28

94.45

97.75

87.38

84.82

83.44

79.82

83.44

Trankit[base]

97.01

99.70

97.01

94.21

94.02

96.59

93.56

97.01

85.19

82.54

80.91

77.42

80.91

Stanza

92.83

98.80

92.83

89.12

88.93

92.11

88.18

92.83

72.88

69.82

66.81

63.26

66.81

Classical_Chinese-Kyoto

Trankit[large]

99.70

70.58

99.70

92.89

91.90

94.43

89.98

99.67

82.03

76.82

75.60

73.16

75.60

Trankit[base]

99.63

61.82

99.63

92.07

91.03

93.88

88.92

99.60

78.79

73.23

72.18

69.27

72.18

Stanza

99.47

46.95

99.47

90.25

89.64

92.68

87.34

99.45

71.81

66.08

64.54

62.61

64.54

Croatian-SET

Trankit[large]

99.93

99.08

99.93

98.58

96.55

96.85

95.99

96.71

93.86

89.74

87.77

82.19

83.56

Trankit[base]

99.92

99.16

99.92

98.38

96.08

96.52

95.44

96.60

93.34

89.36

87.16

81.12

82.91

Stanza

99.96

98.15

99.96

97.88

94.86

95.32

94.22

96.67

90.27

85.56

82.43

76.37

78.78

Czech-CAC

Trankit[large]

99.99

100.00

99.98

99.64

98.17

98.01

97.65

98.30

95.48

93.97

93.06

90.13

90.92

Trankit[base]

99.96

100.00

99.95

99.42

97.40

97.12

96.68

98.13

94.97

93.27

92.08

88.13

89.85

Stanza

99.99

100.00

99.97

98.76

94.79

93.52

92.65

98.00

91.70

89.19

86.84

80.14

84.89

Czech-CLTT

Trankit[large]

99.89

98.89

99.85

99.17

93.80

94.02

93.53

97.81

92.53

90.76

88.90

81.07

86.35

Trankit[base]

99.82

100.00

99.76

98.93

93.36

93.68

93.21

97.66

90.20

88.01

85.13

77.66

82.80

Stanza

99.93

100.00

99.84

98.92

91.89

91.97

91.28

97.48

86.67

83.38

79.35

70.70

77.56

Czech-FicTree

Trankit[large]

99.98

99.50

99.98

99.11

97.26

97.99

97.06

98.64

96.50

94.85

93.44

89.79

91.46

Trankit[base]

99.97

99.38

99.97

98.94

96.47

97.09

96.12

98.61

95.85

93.86

92.10

87.13

90.16

Stanza

99.97

98.60

99.96

98.31

95.23

96.01

94.58

98.43

92.69

89.81

87.30

81.94

85.42

Czech-PDT

Trankit[large]

99.95

97.87

99.95

99.32

98.19

98.19

97.77

98.54

95.24

93.65

92.79

90.18

91.01

Trankit[base]

99.94

97.85

99.94

99.23

97.81

97.77

97.34

98.49

94.81

93.18

92.09

89.11

90.33

Stanza

99.97

94.14

99.97

98.50

95.38

94.61

93.67

98.55

91.00

88.64

86.91

81.12

85.45

Danish-DDT

Trankit[large]

99.81

95.46

99.81

98.71

99.81

98.33

97.87

97.35

91.75

90.33

88.81

85.66

85.42

Trankit[base]

99.79

95.19

99.79

98.35

99.79

97.79

97.19

97.18

90.41

88.78

86.94

82.58

83.38

Stanza

99.96

93.57

99.96

97.75

99.96

97.38

96.45

97.32

86.83

84.19

81.20

77.13

78.46

Dutch-Alpino

Trankit[large]

99.43

90.65

99.43

96.67

95.01

96.55

94.75

96.39

94.41

92.49

89.56

84.22

85.29

Trankit[base]

99.22

89.88

99.22

96.55

94.92

96.22

94.56

96.23

93.28

91.28

87.88

82.58

83.86

Stanza

99.96

89.98

99.96

96.33

94.76

96.28

94.13

96.97

89.56

86.44

81.22

75.76

77.80

Dutch-LassySmall

Trankit[large]

99.36

92.60

99.36

96.52

95.57

96.63

94.99

97.37

92.25

89.68

86.51

82.56

84.19

Trankit[base]

99.21

91.09

99.21

96.20

95.18

96.25

94.53

97.19

91.09

88.18

84.63

80.47

82.44

Stanza

99.90

77.95

99.90

95.97

94.87

96.22

94.05

97.59

85.34

81.93

75.54

71.98

73.49

English-EWT

Trankit[large]

98.67

90.49

98.67

96.65

96.47

96.75

95.49

97.25

91.29

89.40

87.41

83.45

85.73

Trankit[base]

98.48

88.35

98.48

95.95

95.71

96.26

94.62

96.84

90.14

87.96

85.75

81.02

83.83

Stanza

99.01

81.13

99.01

95.40

95.12

96.11

93.90

97.21

86.22

83.59

80.21

76.02

78.50

English-GUM

Trankit[large]

99.52

91.60

99.52

96.66

96.51

97.47

95.77

96.63

91.61

89.09

85.58

81.29

81.80

Trankit[base]

99.45

91.63

99.45

96.39

96.24

97.19

95.46

96.55

91.04

88.43

84.80

80.19

80.81

Stanza

99.82

86.35

99.82

95.89

95.91

96.87

94.99

96.80

87.06

83.57

78.42

74.68

74.97

English-LinES

Trankit[large]

99.46

91.87

99.46

97.31

95.91

96.96

93.91

97.98

89.99

87.00

85.67

80.34

83.77

Trankit[base]

99.53

93.01

99.53

97.14

95.54

96.67

93.17

98.01

89.39

86.32

84.75

78.96

82.78

Stanza

99.95

88.49

99.95

96.88

95.18

96.76

93.11

98.32

85.82

81.97

79.04

74.47

77.31

English-ParTUT

Trankit[large]

99.71

100.00

99.65

96.86

96.65

95.77

94.63

97.71

93.51

91.10

87.37

81.25

85.06

Trankit[base]

99.66

100.00

99.60

96.79

96.55

95.94

94.67

97.64

93.15

90.95

87.21

81.37

84.96

Stanza

99.68

100.00

99.59

96.15

95.83

95.21

93.92

97.45

90.31

87.35

82.56

76.19

80.53

Estonian-EDT

Trankit[large]

99.75

96.58

99.75

97.87

98.35

97.10

96.04

96.09

91.71

89.52

88.57

84.68

84.10

Trankit[base]

99.72

96.55

99.72

97.53

98.13

96.56

95.37

95.98

90.65

88.31

87.15

82.82

82.81

Stanza

99.96

93.32

99.96

97.19

98.04

95.77

94.43

96.05

86.68

83.82

82.41

77.63

78.32

Estonian-EWT

Trankit[large]

97.76

82.58

97.76

94.26

94.93

91.49

89.91

85.71

82.18

78.49

76.41

69.32

64.00

Trankit[base]

96.96

83.72

96.96

92.07

93.16

89.17

86.89

84.65

78.21

73.79

71.59

62.91

59.90

Stanza

99.20

67.14

99.20

88.86

91.70

87.16

83.43

85.62

67.23

60.07

56.21

48.32

47.38

Finnish-FTB

Trankit[large]

99.84

97.36

99.83

98.32

97.29

98.09

96.87

96.94

95.84

94.53

93.54

90.68

90.39

Trankit[base]

99.75

95.83

99.74

97.46

96.23

97.22

95.61

96.58

94.17

92.43

90.84

87.09

87.79

Stanza

100.00

89.59

99.97

95.50

95.12

96.51

93.92

96.16

89.09

86.39

83.80

79.90

81.02

Finnish-TDT

Trankit[large]

99.71

97.22

99.72

98.48

98.78

96.84

96.33

95.59

94.98

93.77

92.92

88.97

88.02

Trankit[base]

99.62

95.98

99.62

97.99

98.44

96.52

95.76

95.39

93.47

91.94

90.78

86.55

86.00

Stanza

99.77

93.05

99.73

96.97

97.72

95.36

94.44

94.98

88.62

86.18

84.66

79.73

80.24

French-GSD

Trankit[large]

99.77

98.80

99.72

97.89

99.72

97.34

96.66

97.90

94.97

93.37

90.17

86.35

87.73

Trankit[base]

99.70

96.63

99.66

97.85

99.66

97.16

96.60

97.80

94.00

92.34

88.66

84.76

86.08

Stanza

99.68

94.92

99.48

97.30

99.47

96.72

96.05

97.64

91.38

89.05

84.38

80.30

82.40

French-ParTUT

Trankit[large]

99.76

98.63

99.65

97.66

97.35

94.55

93.82

96.09

95.05

93.32

90.65

81.49

84.88

Trankit[base]

99.74

98.63

99.69

97.77

97.54

94.20

93.66

96.01

94.20

92.67

89.26

78.71

83.56

Stanza

99.82

100.00

99.37

96.60

96.37

93.98

93.41

95.48

90.71

88.37

83.37

74.41

77.88

French-Sequoia

Trankit[large]

99.81

94.07

99.78

99.22

99.78

98.43

98.13

98.64

95.70

94.85

92.95

90.47

91.15

Trankit[base]

99.73

94.36

99.73

98.90

99.73

97.98

97.57

98.47

94.68

93.59

91.26

88.27

89.44

Stanza

99.90

88.79

99.58

98.19

99.58

97.58

96.94

98.25

90.47

88.34

84.71

81.77

83.31

French-Spoken

Trankit[large]

99.36

53.06

99.19

96.80

96.91

99.19

94.66

96.34

85.70

81.84

76.29

73.65

73.78

Trankit[base]

99.38

39.39

99.18

96.73

96.73

99.18

94.43

96.38

82.40

78.35

71.68

69.01

69.49

Stanza

100.00

22.09

99.45

95.49

97.06

99.45

93.23

96.53

75.82

70.71

62.13

59.57

60.44

Galician-CTG

Trankit[large]

99.76

98.44

99.31

97.30

97.05

99.17

96.77

98.07

85.70

83.14

78.24

72.41

76.90

Trankit[base]

99.76

98.09

99.38

97.17

96.83

99.23

96.54

98.06

85.51

82.81

77.50

71.49

76.20

Stanza

99.89

99.13

99.32

97.21

96.99

99.14

96.71

97.94

85.22

82.66

77.24

71.13

75.96

Galician-TreeGal

Trankit[large]

99.47

95.52

99.22

97.62

95.68

96.50

94.97

91.08

87.17

83.90

80.31

74.67

68.39

Trankit[base]

99.47

94.60

99.06

97.06

94.90

95.89

94.08

90.91

85.38

81.96

77.96

71.57

66.32

Stanza

99.59

89.17

98.41

94.29

91.81

93.36

90.88

94.39

78.04

72.94

65.61

59.06

61.49

German-GSD

Trankit[large]

99.71

89.72

99.72

95.23

97.68

91.68

87.21

96.58

89.01

85.20

81.49

65.82

77.20

Trankit[base]

99.75

92.72

99.75

95.04

97.57

91.51

86.86

96.60

88.73

84.77

80.78

64.76

76.58

Stanza

99.53

85.79

99.53

94.07

96.98

89.52

84.51

96.37

85.39

80.61

75.38

58.57

71.24

German-HDT

Trankit[large]

99.92

99.67

99.92

98.44

98.41

94.05

93.70

97.36

97.63

96.86

95.14

85.73

91.67

Trankit[base]

99.90

99.50

99.90

98.42

98.37

93.95

93.52

97.35

97.38

96.51

94.63

85.02

91.18

Stanza

100.00

97.41

100.00

98.04

97.94

91.77

91.34

97.48

94.91

92.59

88.73

77.26

85.63

Greek-GDT

Trankit[large]

99.85

93.50

99.85

98.41

98.41

96.34

95.84

96.73

95.25

93.87

91.74

85.40

86.54

Trankit[base]

99.75

93.57

99.75

98.04

98.04

95.41

94.73

96.55

94.16

92.80

89.84

82.39

84.83

Stanza

99.88

93.18

99.89

97.84

97.84

94.94

94.33

96.49

91.12

88.78

84.12

78.00

79.48

Hebrew-HTB

Trankit[large]

99.81

99.69

96.31

94.32

94.32

93.03

92.39

93.48

88.41

86.04

82.23

74.92

78.35

Trankit[base]

99.79

100.00

96.03

93.75

93.75

91.96

91.24

93.21

87.02

84.55

80.34

72.38

76.52

Stanza

99.98

99.69

93.19

90.46

90.46

89.24

88.45

90.27

79.18

76.60

71.05

64.51

67.79

Hindi-HDTB

Trankit[large]

99.88

99.91

99.88

98.01

97.70

93.91

92.38

96.54

95.95

92.96

89.79

79.69

88.47

Trankit[base]

99.89

99.64

99.89

97.77

97.38

94.03

92.33

96.54

95.68

92.70

89.59

79.60

88.28

Stanza

100.00

99.44

100.00

97.59

97.08

94.03

92.11

96.66

94.80

91.74

88.20

78.73

87.01

Hungarian-Szeged

Trankit[large]

99.59

99.33

99.59

97.49

99.59

95.23

94.40

94.45

91.31

87.78

86.83

78.95

80.31

Trankit[base]

99.41

98.00

99.41

96.97

99.41

94.47

93.47

94.28

89.43

85.70

85.08

76.13

78.73

Stanza

99.87

97.00

99.87

96.03

99.87

93.76

92.94

94.25

83.62

78.86

77.14

69.46

71.87

Indonesian-GSD

Trankit[large]

99.89

95.54

99.89

93.39

95.06

96.11

89.22

99.53

86.33

79.81

77.99

69.08

77.60

Trankit[base]

99.86

95.37

99.86

93.57

94.18

95.67

88.65

99.49

86.55

80.28

78.64

69.42

78.26

Stanza

99.99

93.78

99.99

93.68

94.79

96.00

89.17

99.61

85.17

79.19

77.04

68.86

76.68

Irish-IDT

Trankit[large]

99.47

98.24

99.47

94.72

93.74

80.90

77.94

92.64

83.47

76.86

70.64

48.55

64.06

Trankit[base]

99.32

97.25

99.32

93.88

92.46

80.36

76.72

92.34

82.52

74.91

67.96

46.29

61.34

Stanza

99.76

95.93

99.76

93.90

92.43

78.19

75.00

91.79

82.65

74.03

66.11

42.98

59.09

Italian-ISDT

Trankit[large]

99.88

99.07

99.86

98.72

98.63

98.32

97.79

98.33

95.73

94.45

91.97

89.08

89.45

Trankit[base]

99.88

98.76

99.87

98.58

98.46

98.20

97.60

98.23

95.31

93.87

90.93

87.81

88.45

Stanza

99.91

98.76

99.76

98.01

97.91

97.72

97.11

98.10

92.79

90.84

86.43

83.60

84.23

Italian-ParTUT

Trankit[large]

99.81

100.00

99.79

98.58

98.42

98.15

97.54

97.84

96.19

94.11

90.65

87.75

87.86

Trankit[base]

99.82

100.00

99.81

98.63

98.41

98.16

97.47

97.94

95.38

93.32

89.28

86.22

86.55

Stanza

99.81

100.00

99.77

97.82

97.76

97.79

96.94

97.57

92.24

90.01

84.39

81.77

82.05

Italian-PoSTWITA

Trankit[large]

99.34

73.95

99.18

96.60

96.43

96.52

95.31

96.41

86.33

82.54

78.49

74.27

75.83

Trankit[base]

99.29

69.95

99.07

96.10

95.91

95.87

94.53

96.30

84.19

80.32

75.33

71.09

72.98

Stanza

99.71

63.70

99.46

96.19

96.04

96.28

95.01

96.70

82.67

78.27

72.20

68.55

70.35

Italian-TWITTIRO

Trankit[large]

99.15

65.72

98.89

95.47

94.90

94.09

91.98

93.12

84.73

79.86

73.94

66.46

66.58

Trankit[base]

99.22

56.00

99.01

95.31

94.74

93.83

91.68

92.96

83.44

78.30

70.79

63.25

63.81

Stanza

99.34

52.40

98.76

94.41

94.01

93.34

91.45

93.17

78.87

72.85

64.64

58.67

59.35

Italian-VIT

Trankit[large]

99.97

98.18

99.84

98.07

97.29

97.76

96.16

98.42

93.02

90.44

86.85

82.53

84.91

Trankit[base]

99.99

96.52

99.81

97.82

97.02

97.39

95.74

98.31

92.39

89.60

85.59

80.70

83.64

Stanza

99.98

94.92

99.49

97.21

96.23

96.79

94.99

98.01

89.32

85.87

80.26

76.16

78.61

Japanese-GSD

Trankit[large]

95.25

95.88

95.25

93.66

93.47

95.23

93.44

94.68

86.67

85.56

78.00

76.02

77.62

Trankit[base]

94.57

95.49

94.57

92.86

92.44

94.56

92.42

93.99

84.58

83.38

75.60

73.67

75.14

Stanza

92.67

94.57

92.67

91.16

90.84

92.66

90.84

92.02

81.20

80.16

71.39

69.85

71.01

Kazakh-KTB

Trankit[large]

95.98

81.71

95.37

77.94

77.47

63.01

55.46

50.60

47.46

37.98

36.01

19.00

12.05

Trankit[base]

94.48

90.00

93.62

75.94

75.67

62.28

54.51

49.76

46.42

36.84

34.72

18.65

11.81

Stanza

93.46

88.56

94.16

56.23

56.10

42.73

36.96

52.12

44.33

25.21

20.28

7.63

10.01

Korean-GSD

Trankit[large]

98.57

98.08

98.57

95.71

90.88

98.35

88.90

91.93

89.87

87.22

85.97

83.63

79.65

Trankit[base]

98.63

97.67

98.63

95.63

90.32

98.43

88.26

91.96

88.48

85.77

84.26

81.98

78.08

Stanza

99.88

96.65

99.88

96.18

90.14

99.66

88.00

92.69

87.29

83.53

81.34

79.29

75.31

Korean-Kaist

Trankit[large]

98.70

99.87

98.70

95.13

88.07

98.70

88.07

92.36

90.00

88.22

86.37

83.56

80.16

Trankit[base]

98.79

99.14

98.79

94.99

87.62

98.79

87.62

92.44

88.72

86.96

84.99

81.84

78.90

Stanza

100.00

99.93

100.00

95.45

86.31

100.00

86.31

93.02

88.41

86.38

83.95

80.63

77.57

Kurmanji-MG

Trankit[large]

94.95

91.50

94.63

75.07

74.16

57.15

52.27

57.63

37.12

29.89

25.57

9.04

10.16

Trankit[base]

94.52

80.56

94.20

74.33

73.44

56.54

51.38

57.61

35.65

28.58

25.35

8.88

10.76

Stanza

94.81

87.43

94.49

57.17

55.91

43.02

38.41

56.13

32.01

21.91

16.35

3.84

5.84

Latin-ITTB

Trankit[large]

100.00

94.54

100.00

98.97

97.29

97.98

96.41

99.13

93.25

91.87

90.75

87.86

90.00

Trankit[base]

100.00

94.57

100.00

98.76

96.74

97.54

95.68

99.07

92.42

90.91

89.45

86.12

88.71

Stanza

99.99

80.66

99.99

98.09

95.38

96.43

93.80

98.90

87.61

85.36

84.23

80.28

83.60

Latin-Perseus

Trankit[large]

99.60

97.93

99.60

92.84

83.33

86.79

82.33

70.34

83.50

76.76

73.58

60.70

44.41

Trankit[base]

99.45

97.87

99.45

90.15

77.12

81.12

75.64

69.95

78.01

69.58

65.24

49.58

40.23

Stanza

100.00

98.24

100.00

90.63

78.42

82.42

77.74

83.08

71.94

61.99

57.89

45.28

47.28

Latin-PROIEL

Trankit[large]

99.85

66.10

99.85

97.79

97.75

93.22

92.53

97.21

86.43

83.33

81.62

73.62

79.55

Trankit[base]

99.82

58.16

99.82

96.80

96.83

91.28

90.27

96.88

82.23

78.58

76.36

67.10

74.43

Stanza

100.00

43.04

100.00

96.92

97.10

91.24

90.32

96.78

76.55

72.37

70.06

61.28

68.19

Latvian-LVTB

Trankit[large]

99.73

98.69

99.73

97.61

91.22

95.18

90.72

95.83

93.63

91.25

89.78

82.69

85.58

Trankit[base]

99.71

99.10

99.71

97.16

90.24

94.47

89.62

95.61

92.05

89.44

87.73

79.78

83.52

Stanza

99.82

99.01

99.82

96.03

88.25

93.46

87.73

95.55

87.84

84.44

82.16

73.91

78.25

Lithuanian-ALKSNIS

Trankit[large]

99.84

95.72

99.84

97.45

93.98

94.46

93.30

94.30

90.48

87.67

86.66

79.86

80.20

Trankit[base]

99.82

95.10

99.82

97.03

92.35

93.00

91.54

94.05

88.30

84.96

83.59

75.11

77.35

Stanza

99.87

88.79

99.87

93.37

85.67

87.84

84.84

92.51

78.54

73.11

70.66

60.81

65.53

Lithuanian-HSE

Trankit[large]

97.71

100.00

97.71

90.59

89.85

79.64

75.90

80.02

71.41

62.05

59.15

41.13

44.82

Trankit[base]

98.22

94.55

98.22

90.46

89.71

77.92

74.18

80.07

66.70

58.47

55.18

36.60

40.03

Stanza

97.53

51.11

97.53

81.08

80.04

70.72

66.44

76.90

48.10

37.45

32.37

21.10

24.86

Marathi-UFAL

Trankit[large]

99.20

69.31

97.22

87.79

97.22

70.62

67.47

81.50

72.79

63.36

59.67

36.63

46.50

Trankit[base]

99.20

60.87

95.25

82.83

95.25

69.43

66.02

79.17

60.90

54.08

52.19

28.81

40.50

Stanza

98.00

76.40

92.25

77.24

92.25

60.27

58.55

75.77

66.42

52.64

42.80

24.15

33.90

Norwegian_Nynorsk-Nynorsk

Trankit[large]

99.84

98.97

99.84

98.52

99.84

97.79

97.13

98.01

95.23

93.82

92.35

89.01

89.79

Trankit[base]

99.81

98.71

99.81

98.20

99.81

97.20

96.48

97.89

94.15

92.58

90.70

86.61

88.15

Stanza

99.97

94.85

99.97

97.92

99.97

96.88

96.03

97.90

91.87

89.73

87.28

82.86

84.78

Norwegian_Nynorsk-NynorskLIA

Trankit[large]

99.76

99.53

99.76

96.48

99.76

95.59

93.57

97.49

81.96

77.85

73.57

67.19

71.00

Trankit[base]

99.74

99.53

99.74

96.31

99.74

95.41

93.29

97.50

80.86

76.44

71.96

65.82

69.71

Stanza

100.00

99.69

100.00

95.92

100.00

94.82

92.70

97.72

77.82

72.94

67.56

61.32

65.54

Norwegian-Bokmaal

Trankit[large]

99.88

98.89

99.88

98.85

99.88

98.07

97.61

98.40

95.54

94.33

92.82

90.15

90.84

Trankit[base]

99.88

99.20

99.88

98.66

99.88

97.60

97.02

98.34

94.78

93.47

91.77

88.29

89.72

Stanza

99.99

97.17

99.99

98.29

99.99

97.17

96.41

98.36

92.57

90.69

88.32

84.41

86.33

Old_French-SRCMF

Trankit[large]

99.91

100.00

99.91

96.96

96.83

98.32

96.45

99.91

94.30

90.75

88.69

85.43

88.69

Trankit[base]

99.84

100.00

99.84

96.36

96.21

97.75

95.72

99.84

92.82

88.76

86.12

82.63

86.12

Stanza

100.00

100.00

100.00

96.05

96.09

97.74

95.56

100.00

91.38

86.35

83.39

80.05

83.39

Old_Russian-TOROT

Trankit[large]

98.87

51.91

98.87

94.70

94.63

89.61

88.02

90.87

78.64

74.60

71.82

63.30

66.49

Trankit[base]

98.44

42.22

98.44

92.63

92.66

86.75

84.52

90.00

74.14

68.92

65.57

55.81

60.56

Stanza

100.00

35.69

100.00

93.63

93.83

86.76

84.80

91.35

72.94

67.00

63.60

54.13

59.18

Persian-Seraji

Trankit[large]

99.26

99.25

99.20

97.78

97.67

97.70

97.35

97.35

92.24

89.58

86.86

84.97

84.90

Trankit[base]

99.22

99.25

99.11

97.35

97.24

97.36

96.90

97.29

91.38

88.68

85.92

83.86

84.08

Stanza

100.00

99.25

99.65

97.29

97.30

97.37

96.86

97.73

89.45

86.06

82.78

81.00

81.08

Polish-LFG

Trankit[large]

98.34

99.57

98.34

97.84

95.52

96.00

95.05

95.48

93.90

93.04

92.58

89.18

88.61

Trankit[base]

98.32

99.91

98.32

97.66

94.59

95.05

94.00

95.37

93.31

92.17

91.43

86.88

87.55

Stanza

99.95

99.83

99.95

98.55

94.66

95.84

94.07

96.86

95.80

93.94

92.35

87.62

88.64

Polish-PDB

Trankit[large]

99.93

98.71

99.92

99.16

96.92

97.11

96.54

97.56

96.43

94.88

93.77

89.78

90.53

Trankit[base]

99.91

98.53

99.89

99.06

96.29

96.44

95.77

97.52

95.52

93.86

92.50

87.67

89.34

Stanza

99.87

98.39

99.83

98.31

94.04

94.27

93.13

97.29

92.68

90.40

88.35

81.69

85.42

Portuguese-Bosque

Trankit[large]

99.75

97.18

99.67

97.52

99.67

96.50

95.17

97.97

93.31

90.91

87.75

81.09

85.36

Trankit[base]

99.70

97.48

99.65

97.27

99.65

96.50

94.95

97.89

92.76

90.25

86.96

80.03

84.52

Stanza

99.77

94.30

99.67

97.04

99.67

96.36

94.91

97.80

90.67

87.57

82.59

76.78

80.30

Portuguese-GSD

Trankit[large]

99.81

97.10

99.72

98.43

98.43

99.61

98.41

99.20

95.35

94.37

92.23

90.46

91.47

Trankit[base]

99.82

96.76

99.71

98.30

98.30

99.61

98.28

99.19

94.92

93.95

91.65

89.58

90.89

Stanza

99.96

98.00

99.87

98.18

98.18

99.79

98.17

95.83

92.83

91.36

87.44

85.87

86.75

Romanian-Nonstandard

Trankit[large]

98.74

98.00

98.74

96.23

91.64

90.51

89.07

94.66

90.68

86.89

83.13

70.10

78.50

Trankit[base]

98.68

98.57

98.68

96.04

91.48

90.33

88.89

94.57

90.14

86.40

82.40

69.46

77.93

Stanza

98.96

97.53

98.96

95.40

90.73

89.79

88.19

94.63

87.24

82.71

77.60

65.24

73.52

Romanian-RRT

Trankit[large]

99.60

98.49

99.60

97.90

97.35

97.43

97.11

97.98

93.60

89.50

86.61

82.78

84.60

Trankit[base]

99.72

97.67

99.72

97.87

97.25

97.44

97.01

98.05

93.14

89.04

85.93

82.02

84.01

Stanza

99.77

96.64

99.77

97.54

96.97

97.13

96.75

97.95

90.66

85.85

81.49

77.94

79.84

Russian-GSD

Trankit[large]

99.79

99.25

99.79

98.25

97.89

95.78

94.73

95.75

92.70

89.75

88.71

82.92

83.66

Trankit[base]

99.63

98.25

99.63

97.96

97.65

94.86

93.83

95.50

91.86

88.62

87.41

80.83

82.36

Stanza

99.65

97.16

99.65

97.38

97.18

93.11

92.22

95.34

88.97

84.83

82.37

75.16

77.75

Russian-SynTagRus

Trankit[large]

99.71

99.45

99.71

99.06

99.71

98.20

97.99

97.99

95.66

94.65

93.78

91.68

91.51

Trankit[base]

99.71

99.14

99.71

98.94

99.71

97.85

97.59

97.89

95.19

94.08

93.13

90.59

90.77

Stanza

99.57

98.86

99.57

98.20

99.57

95.91

95.59

97.51

92.38

90.60

89.01

85.04

86.78

Russian-Taiga

Trankit[large]

98.90

92.32

98.90

96.16

96.62

91.02

87.69

91.90

85.25

81.73

80.06

68.48

71.59

Trankit[base]

98.77

92.60

98.77

95.50

97.27

89.42

86.58

91.46

83.08

79.15

76.91

64.25

68.53

Stanza

97.11

85.79

97.11

92.25

94.70

85.76

82.61

89.28

72.09

66.00

61.80

51.94

55.64

Scottish_Gaelic-ARCOSG

Trankit[large]

99.43

57.46

99.42

94.70

88.65

90.71

87.13

95.48

81.60

76.46

70.62

61.71

66.95

Trankit[base]

99.26

54.10

99.25

92.98

85.47

88.25

83.78

95.06

79.48

73.09

66.41

56.27

62.83

Stanza

99.48

55.35

99.47

92.50

84.89

87.99

83.93

95.51

77.90

70.81

62.63

54.00

59.74

Serbian-SET

Trankit[large]

99.91

100.00

99.91

99.06

96.22

96.40

95.84

96.90

95.57

93.13

91.92

85.76

87.71

Trankit[base]

99.91

99.71

99.91

98.97

95.82

95.96

95.32

96.90

95.24

92.94

91.53

84.84

87.46

Stanza

100.00

99.33

100.00

98.44

94.26

94.55

93.86

96.34

91.79

88.78

86.50

79.48

82.38

Simplified_Chinese-GSDSimp

Trankit[large]

97.66

98.20

97.66

94.99

94.78

97.24

94.32

97.66

87.01

84.63

83.23

79.84

83.23

Trankit[base]

96.94

99.70

96.94

94.17

93.98

96.51

93.52

96.94

84.64

81.96

80.14

76.30

80.14

Stanza

92.92

99.10

92.92

89.05

88.84

92.12

88.03

92.92

73.44

70.44

67.69

64.07

67.69

Slovak-SNK

Trankit[large]

99.94

98.49

99.94

97.90

90.82

95.22

90.20

94.79

96.97

95.60

95.04

87.64

88.39

Trankit[base]

99.93

98.07

99.93

97.80

89.02

94.00

88.38

94.66

95.72

93.97

93.19

84.33

86.63

Stanza

99.97

90.93

99.97

96.34

87.15

91.59

86.34

94.73

89.96

86.82

84.74

75.39

79.35

Slovenian-SSJ

Trankit[large]

99.97

100.00

99.97

99.24

97.83

97.99

97.60

97.55

96.91

96.06

94.88

91.78

91.37

Trankit[base]

99.93

99.81

99.93

99.03

96.70

96.97

96.38

97.49

95.94

94.99

93.53

89.09

90.12

Stanza

99.91

91.60

99.91

98.29

95.08

95.37

94.56

97.34

91.63

89.60

87.18

82.35

84.37

Slovenian-SST

Trankit[large]

99.84

33.03

99.84

95.81

91.74

91.86

89.70

88.72

74.86

71.02

66.85

59.58

56.40

Trankit[base]

99.79

31.96

99.79

94.90

90.27

90.37

87.92

88.66

71.15

66.65

61.94

54.19

52.26

Stanza

100.00

26.59

100.00

93.66

88.09

88.06

85.27

94.78

63.13

56.50

51.34

44.81

48.96

Spanish-AnCora

Trankit[large]

99.91

99.39

99.91

99.06

99.00

98.85

98.36

99.15

94.77

93.29

91.10

89.23

90.14

Trankit[base]

99.94

99.13

99.93

99.02

98.94

98.80

98.27

99.17

94.11

92.41

89.66

87.60

88.71

Stanza

99.98

99.07

99.98

98.78

98.67

98.59

97.97

99.19

92.21

90.01

86.05

84.22

85.20

Spanish-GSD

Trankit[large]

99.93

99.30

99.88

97.38

99.88

96.96

95.23

98.59

93.47

91.56

88.58

80.91

86.50

Trankit[base]

99.91

98.94

99.88

97.41

99.88

96.88

95.23

98.62

92.66

90.50

87.01

79.83

85.04

Stanza

99.96

95.97

99.87

96.69

99.87

96.40

94.44

98.44

89.61

86.73

81.22

73.96

79.19

Swedish-LinES

Trankit[large]

99.89

90.64

99.89

98.05

95.77

90.98

88.52

96.94

91.74

88.73

87.89

74.52

83.97

Trankit[base]

99.73

90.57

99.73

97.60

95.23

90.50

87.93

96.72

90.45

87.36

86.11

72.78

82.24

Stanza

99.94

86.99

99.94

96.97

94.58

90.11

87.33

96.79

87.10

83.06

80.76

67.97

77.44

Swedish-Talbanken

Trankit[large]

99.91

99.26

99.91

99.06

98.09

98.04

97.43

97.87

94.41

92.97

92.08

88.67

89.28

Trankit[base]

99.87

99.38

99.87

98.76

97.77

97.73

97.03

97.82

93.61

91.87

90.72

86.97

88.03

Stanza

99.97

98.85

99.97

97.65

96.57

96.70

95.63

97.51

88.96

85.91

83.59

79.17

80.78

Tamil-TTB

Trankit[large]

98.33

100.00

94.44

87.64

83.92

87.19

82.56

88.85

72.34

67.66

66.19

58.16

61.43

Trankit[base]

98.02

100.00

93.64

86.18

82.09

86.43

80.27

88.09

68.37

63.67

61.78

52.73

57.32

Stanza

99.58

95.08

91.42

82.60

78.80

81.89

78.10

85.14

61.23

55.76

53.43

46.40

49.61

Telugu-MTG

Trankit[large]

98.89

98.62

98.89

94.31

94.31

98.33

94.31

98.89

93.19

87.08

84.46

80.65

84.46

Trankit[base]

98.89

98.62

98.89

94.32

94.32

97.92

94.32

98.89

91.97

84.35

81.10

78.44

81.10

Stanza

100.00

97.95

100.00

92.93

92.93

99.17

92.93

100.00

89.32

79.89

74.88

71.25

74.88

Turkish-IMST

Trankit[large]

99.84

98.32

98.91

96.20

95.33

92.78

91.19

96.21

78.94

73.14

71.15

63.81

69.11

Trankit[base]

99.86

98.18

98.68

95.15

94.35

92.02

89.94

95.80

76.59

70.75

68.28

60.61

66.24

Stanza

99.89

97.62

98.07

94.21

93.43

92.08

90.27

94.92

70.78

64.50

61.62

56.04

59.60

Ukrainian-IU

Trankit[large]

99.77

97.55

99.76

98.50

96.32

96.09

95.35

96.99

94.63

93.16

91.78

86.22

88.03

Trankit[base]

99.78

97.72

99.76

98.33

94.96

94.94

93.86

96.98

93.44

91.69

89.89

83.20

86.33

Stanza

99.81

96.65

99.79

96.77

92.49

92.53

91.31

96.49

87.11

83.86

80.51

73.38

77.28

Urdu-UDTB

Trankit[large]

99.75

97.67

99.75

94.43

92.77

81.91

78.35

95.44

88.52

83.26

78.30

57.41

75.87

Trankit[base]

99.66

98.32

99.66

94.15

92.66

83.04

79.29

95.33

87.81

82.51

77.31

57.57

74.83

Stanza

100.00

98.88

100.00

94.42

92.62

84.21

80.36

95.62

88.30

82.78

77.06

59.48

74.75

Uyghur-UDT

Trankit[large]

97.95

88.95

97.95

88.50

91.34

86.56

79.14

94.45

80.19

70.14

66.06

51.63

62.99

Trankit[base]

97.63

88.51

97.63

87.47

90.37

85.31

77.28

94.26

78.36

68.24

63.89

48.42

60.70

Stanza

99.79

86.90

99.79

89.45

91.92

87.92

80.54

96.16

75.55

63.61

57.00

46.06

54.39

Vietnamese-VTB

Trankit[large]

94.88

96.63

94.88

89.70

88.14

94.64

88.07

94.88

71.07

65.37

63.67

60.22

63.67

Trankit[base]

95.22

96.25

95.22

89.40

87.85

95.03

87.82

95.22

70.96

64.76

62.72

58.51

62.72

Stanza

87.25

93.15

87.25

79.50

77.90

87.02

77.87

87.20

53.63

48.16

44.88

42.17

44.85

Performance for Stanza, UDPipe, and spaCy is obtained using their public pretrained models. The overall performance for Trankit and Stanza is computed as the macro-averaged F1 over 90 treebanks using the official evaluation script of the CoNLL 2018 Shared Task.

Named Entity Recognition

Performance comparison between Trankit large, Trankit base, and Stanza v1.1.1 on the test sets of 11 public NER datasets. Performance is based on entity micro-averaged F1.

Language

Corpus

Trankit[large]

Trankit[base]

Stanza v1.1.1

Arabic

AQMAR

76.1

74.8

74.3

Chinese

OntoNotes

80.5

80

79.2

Dutch

CoNLL02

93.8

91.8

89.2

WikiNER

95.0

94.8

94.8

English

CoNLL03

92.5

92.1

92.1

OntoNotes

90.2

89.6

88.8

French

WikiNER

92.9

92.3

92.9

German

CoNLL03

85.3

84.6

81.9

GermEval14

89.4

86.9

85.2

Russian

WikiNER

93.2

92.8

92.9

Spanish

CoNLL02

89.2

88.9

88.1