QA4MRE at CLEF2013



The main objective of this exercise is to develop a methodology for evaluating Machine Reading systems through Question Answering and Reading Comprehension Tests.
Systems should be able to extract knowledge from large volumes of text and use this knowledge to answer questions. This methodology should allow the comparison of systems' performance and the study of the best approaches.

 

TASK OVERVIEW

The Machine Reading task addresses the problem of building a bridge between knowledge encoded as natural text and the formal reasoning systems that need such knowledge. The knowledge contained in naturally occurring texts should be made available in forms that machines can use to perform some kind of reasoning and expand the system's inference capabilities. In contrast to text mining (or text harvesting, sometimes also called macro-reading), where the system reads and combines evidence from hundreds or thousands of texts, MR is the task of obtaining an in-depth understanding of just one, or a small number, of texts.  In fact, the task will focus on the reading of single documents, where correct answers require some inference and the consideration of previously acquired background knowledge.

 

THREE TASKS

Beside the Main Task, also two pilot tasks are offered this year at QA4MRE

  1. Machine Reading of Biomedical Texts about Alzheimer's Disease

    It is aimed at setting questions in the Biomedical domain with a special focus on one disease, namely Alzheimer. This pilot task will explore the ability of a system to answer questions using scientific language. Texts will be taken from Medline abstracts. MEDLINE (Medical Literature Analysis and Retrieval System Online) is a bibliographic database of life sciences and biomedical information. It was compiled by the United States National Library of Medicine (NLM), and is freely available on the Internet. In order to keep the task reasonably simple for systems, participants will be given the background collection already processed with Tok, Lem, POS, NER, and Dependency parsing. A development set will also be provided to participants.


    The task will be offered in English only and will be coordinated by the University of Antwerp, Belgium.


  2. Entrance Exams

    Japanese University Entrance Exams include questions formulated at various levels of complexity and test a wide range of capabilities. The challenge of "Entrance Exams" aims at evaluating systems under the same conditions humans are evaluated to enter the University. In this first campaign we will reduce the challenge to Reading Comprehension exercises contained in the English exams. More types of exercises will be included in subsequent campaigns (2014–2016) in coordination with the "Entrance Exams" task at NTCIR. Exams are created by the Japanese National Center for University Admissions Tests. The "Entrance Exams" corpus is provided by NII's Todai Robot Project and NTCIR.