System building cost vs. output quality in data-to-text generation

Data-to-text generation systems tend tobe knowledge-based and manually built,which limits their reusability and makes them time and cost-intensive to createand maintain. Methods for automating(part of) the system building process exist,but do such methods risk a loss inoutput quality? In this paper, we investigatethe cost/quality trade-off in generation system building. We comparefour new data-to-text systems which were created by predominantly automatic techniques against six existing systems for the same domain which were created by predominantlymanual techniques. We evaluate the ten systems using intrinsic automatic metrics and human quality ratings.We find that increasing the degree towhich system building is automated doesnot necessarily result in a reduction in outputquality. We find furthermore that standardautomatic evaluation metrics underestimate the quality of handcrafted systems and over-estimate the quality of automatically created systems.