Assessing the trade-off between system building cost and output quality in data-to-text generation

Anja Belz, Eric Kow

Research output: Chapter in Book/Conference proceeding with ISSN or ISBNChapter

Abstract

Data-to-text generation systems tend to be knowledge-based and manually built, which limits their reusability and makes them time and cost-intensive to create and maintain. Methods for automating (part of) the system building process exist, but do such methods risk a loss in output quality? In this paper, we investigate the cost/quality trade-off in generation system building. We compare six data-to-text systems which were created by predominantly automatic techniques against six systems for the same domain which were created by predominantly manual techniques. We evaluate the systems using intrinsic automatic metrics and human quality ratings. We find that there is some correlation between degree of automation in the system-building process and output quality (more automation tending to mean lower evaluation scores). We also find that there are discrepancies between the results of the automatic evaluation metrics and the human-assessed evaluation experiments. We discuss caveats in assessing system-building cost and implications of the discrepancies in automatic and human evaluation.
Original languageEnglish
Title of host publicationEmpirical Methods in Natural Language Generation
EditorsE. Krahmer, M. Theune
Place of PublicationBerlin, Heidelberg
PublisherSpringer-Verlag
Pages180-200
Number of pages21
Volume5790
ISBN (Print)9783642155727
DOIs
Publication statusPublished - 1 Jan 2010

Publication series

NameLecture Notes in Computer Science

Fingerprint Dive into the research topics of 'Assessing the trade-off between system building cost and output quality in data-to-text generation'. Together they form a unique fingerprint.

  • Cite this

    Belz, A., & Kow, E. (2010). Assessing the trade-off between system building cost and output quality in data-to-text generation. In E. Krahmer, & M. Theune (Eds.), Empirical Methods in Natural Language Generation (Vol. 5790, pp. 180-200). (Lecture Notes in Computer Science). Springer-Verlag. https://doi.org/10.1007/978-3-642-15573-4_10