MACHINE READING OF BIOMEDICAL TEXTS ABOUT THE ALZHEIMER DISEASE
This task aims at exploring the ability of a machine reading system to answer questions about a scientific topic, namely Alzheimer's disease. As in the main QA4MRE task, this task focuses on the reading of single documents and the identification of the answers to a set of questions about information that is stated or implied in the text. Questions are in the form of multiple choice, each having five options, and only one correct answer. The detection of correct answers is specifically designed to require various kinds of inference and the consideration of previously acquired background knowledge from reference document collections provided by the organization. Although the additional knowledge obtained through the background collection may be used to assist with answering the questions, the principal answer is to be found among the facts contained in the test documents given.
Participants will be provided with a background collection, the Alzheimer's Disease Literature Corpus, and test documents about Alzheimer's disease. To solve the task, participants can make use of existing resources, such as ontologies or databases, and tools, such as named entity taggers, event extractors, parsers, etc. In order to keep the task reasonably simple for systems, the organisation will provide the texts of the background collection and the test documents processed at several levels of linguistic analysis (lemmas, part-of-speech, named entities, chunking, dependency parsing). Publicly available state of the art tools will be used for this purpose.
This is the second edition of the task, which was run as a pilot task of the QA4MRE Lab at CLEF 2012.
The background collection is a reference corpus consisting of documents related to the topic. As in the main task, the background collection should be used by the systems to acquire the reading capabilities and the knowledge needed to answer questions about the test documents. The collection consists of abstracts and full articles about Alzheimer's Disease: (i) Around 66,000 abstracts from PubMed. PubMed (http://www.ncbi.nlm.nih.gov/pubmed/) is a free resource that is developed and maintained by the National Center for Biotechnology Information (NCBI), at the U.S. National Library of Medicine (NLM), located at the National Institutes of Health (NIH). (ii) Around 8,000 Open Access full articles from PubMed Central. PubMed Central (http://www.ncbi.nlm.nih.gov/pmc/) is a free full-text archive of biomedical and life sciences journal literature at the U.S. National Institutes of Health's National Library of Medicine (NIH/NLM). (iii) Full articles about the key hypotheses in Alzheimer Disease published by Elsevier.
The test set will be composed of 4 reading tests. Each reading test will consist of one single document, with 10 questions and a set of five choices per question. So, there will be in total 40 questions and 200 choices/options.
Participating systems will be required to answer these 40 questions by choosing in each case one answer from the five alternatives. There will always be one and only one correct option. Systems will also have the chance to leave some questions unanswered if they are not confident about the correctness of their response.
Document collections and reading tests will be available in English.
This pilot task will be evaluated following the same criteria as the main task. Evaluation will be performed automatically by comparing the answers given by systems to the ones given by humans. No manual assessment will be required. Each test will receive an evaluation score between 0 and 1 using c@1.
An example test will be provided so that participants can form an idea about the type of texts and questions that will be provided as evaluation set. A preprocessed version of the test set will also provided, so that participants can form an idea of the format and annotations that will be provided. The example test will be available from the download site of the main task.