Framework and Resources for Natural Language Parser Evaluation

Abstract

Because of the wide variety of contemporary practices used in the automatic syntactic parsing of natural languages, it has become necessary to analyze and evaluate the strengths and weaknesses of different approaches. This research is all the more necessary because there are currently no genre- and domain-independent parsers that are able to analyze unrestricted text with 100% preciseness (I use this term to refer to the correctness of analyses assigned by a parser). All these factors create a need for methods and resources that can be used to evaluate and compare parsing systems. This research describes: (1) A theoretical analysis of current achievements in parsing and parser evaluation. (2) A framework (called FEPa) that can be used to carry out practical parser evaluations and comparisons. (3) A set of new evaluation resources: FiEval is a Finnish treebank under construction, and MGTS and RobSet are parser evaluation resources in English. (4) The results of experiments in which the developed evaluation framework and the two resources for English were used for evaluating a set of selected parsers.

PhD dissertation. 264 pages

References in corpus (14)

• An Efficient Probabilistic Context-Free Parsing Algorithm that Computes Prefix Probabilities
• Natural Language Parsing as Statistical Pattern Recognition
• Exploiting Diversity in Natural Language Processing: Combining Parsers
• ATLAS: A flexible and extensible architecture for linguistic annotation
• A Robust Parsing Algorithm For Link Grammars
• Corpus Annotation for Parser Evaluation
• An Efficient Implementation of the Head-Corner Parser
• Efficient Algorithms for Parsing the DOP Model
• Towards an implementable dependency grammar
• A Linguistically Interpreted Corpus of German Newspaper Text
• The Complexity of Recognition of Linguistically Adequate Dependency Grammars
• Directed Replacement
• An Empirically Motivated Reinterpretation of Dependency Grammar
• Measuring efficiency in high-accuracy, broad-coverage statistical parsing