ARE - Anaphora Resolution Exercise

The general objective of ARE is to develop discourse anaphora resolution methods and to evaluate them in a common and consistent manner. In this first edition, the evaluation is organised only for English and tackles pronominal anaphora and NP coreference. If successful, in future editions it can be extended to include other types of referential expressions and anaphoric relations (e.g. bridging references) as well as other languages. The first Anaphora Resolution Exercise (ARE) is organised in conjunction with the 6th Discourse Anaphora and Anaphor Resolution Colloquium (DAARC2007).

Tasks

Four tasks will be organised:

  1. Pronominal resolution on pre-annotated texts: in this task, participants will be provided with documents in which the NPs are annotated. Among these NPs, some of them will have an attribute which indicates that they are referential and have to be resolved. In this task only referential pronouns will have this attribute (i.e. pleonastic pronouns will not be annotated). The participants will have to identify for each pronominal referential expression any correct antecedent from the list of annotated NPs.
  2. Coreferential chains resolution on pre-annotated texts: in this task participants will be provided with documents in which the NPs are annotated. The participants will have to identify full coreferential chains which contain the annotated NPs.
  3. Pronominal resolution on raw texts: in this task, participants will have to resolve referential pronouns in unannotated texts (i.e. the referential expressions and antecedent candidates are not marked as in task 1)
  4. Coreferential chains resolution on raw texts: in this task, participants will have to identify full coreferential chains in unannotated texts (i.e. the NPs are not pre-marked)

Tentative timeline
Data

The format of the input and output, as well as training data is available for each task. They can be viewed here:

The data was inially annotated using the these guidelines. In order to keep the task simple, the only relations kept in the data are those of IDENTITY, SYNONYMY, GENERALIZATION and SPECIALIZATION. In this first edition of the competition, participants are not required to make a distinction between the relations. They are only required to identify that there is a relation between two entities.
Evaluation

The evaluation metrics can be seen on this page. There are also program written in .NET which implement the measures. They can be downloaded from here.

Organising committee
Contact details

To indicate your intention to participate or to find out more please contact Constantin Orasan.

(c) 2006 - 2014 Research Group in Computational Linguistics
Last modified: August 13 2007