Referring Expression Generation Challenges 2008

Referring Expression Generation Challenges 2008 (REG'08) was the second shared-task evaluation challenge (STEC) in the field of Natural Language Generation (NLG) and took place between September 2007, when REG'08 was first announced, and May 2008, when the evaluation of the participating systems was completed.

REG'08 followed on from the Attribute Selection for Referring Expression Generation Challenge in 2007(ASGRE'07) which was conceived as a pilot event for shared-task evaluation in Natural Language Generation (NLG). Support from the ESPSRC allowed for the delivery and expansion of the challenge.

The GREC Task at REG’08 required participating systems to select coreference chains to the main subject of short encyclopaedic texts collected from Wikipedia. Three teams submitted a total of 6 systems, and we additionally created four baseline systems. Systems were tested automatically using a range of existing intrinsic metrics. We also evaluated systems extrinsically by applying coreference resolution tools to the outputs and measuring the success of the tools. In addition, systems were tested in a reading/comprehension experiment involving human subjects.

The TUNA Challenge was a set of three shared tasks at REG’08, all of which used data from the TUNA Corpus. The three tasks covered attribute selection for referring expressions (TUNA-AS), realisation (TUNA-R) and end-to-end referring expression generation (TUNAREG). Eight teams submitted a total of 33 systems to the three tasks, with an additional submission to the Open Track. The evaluation used a range of automatically computed measures. In addition, an evaluation experiment was carried out using the peer outputs for the TUNAREG task.

A report describes the GREC Task and the evaluation methods, gives brief descriptions of the participating systems, and presents the evaluation results.