Discrete vs. continuous rating scales for language evaluation in NLP

Anja Belz; Eric Kow

Discrete vs. continuous rating scales for language evaluation in NLP

Anja Belz, Eric Kow

University of Brighton

Research output: Chapter in Book/Conference proceeding with ISSN or ISBN › Conference contribution with ISSN or ISBN › peer-review

Abstract

Studies assessing rating scales are very common in psychology and related fields, but are rare in NLP. In this paper we assess discrete and continuous scales used for measuring quality assessments of computer-generated language. We conducted six separate experiments designed to investigate the validity, reliability, stability, interchangeability and sensitivity of discrete vs. continuous scales. We show that continuous scales are viable for use in language evaluation, and offer distinct advantages over discrete scales.

Original language	English
Title of host publication	The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies
Place of Publication	Stroudsburg, PA, USA
Publisher	Association for Computational Linguistics
Pages	230-235
Number of pages	6
ISBN (Print)	9781932432886
Publication status	Published - 1 Jan 2011
Event	The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Portland, Oregon, USA, 19-24 June, 2011 Duration: 1 Jan 2011 → …

Conference

Conference	The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies
Period	1/01/11 → …

Access to Document

http://www.aclweb.org/anthology-new/P/P11/P11-1000.pdfLicence: Unspecified

Cite this

@inproceedings{f8c65f2851dd4733bf0e6289ce6d7a36,

title = "Discrete vs. continuous rating scales for language evaluation in NLP",

abstract = "Studies assessing rating scales are very common in psychology and related fields, but are rare in NLP. In this paper we assess discrete and continuous scales used for measuring quality assessments of computer-generated language. We conducted six separate experiments designed to investigate the validity, reliability, stability, interchangeability and sensitivity of discrete vs. continuous scales. We show that continuous scales are viable for use in language evaluation, and offer distinct advantages over discrete scales.",

author = "Anja Belz and Eric Kow",

year = "2011",

month = jan,

day = "1",

language = "English",

isbn = "9781932432886",

pages = "230--235",

booktitle = "The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies",

publisher = "Association for Computational Linguistics",

note = "The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies ; Conference date: 01-01-2011",

}

Belz, A & Kow, E 2011, Discrete vs. continuous rating scales for language evaluation in NLP. in The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Stroudsburg, PA, USA, pp. 230-235, The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, 1/01/11. <http://www.aclweb.org/anthology-new/P/P11/P11-1000.pdf>

Discrete vs. continuous rating scales for language evaluation in NLP. / Belz, Anja; Kow, Eric.
The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg, PA, USA: Association for Computational Linguistics, 2011. p. 230-235.

Research output: Chapter in Book/Conference proceeding with ISSN or ISBN › Conference contribution with ISSN or ISBN › peer-review

TY - GEN

T1 - Discrete vs. continuous rating scales for language evaluation in NLP

AU - Belz, Anja

AU - Kow, Eric

PY - 2011/1/1

Y1 - 2011/1/1

N2 - Studies assessing rating scales are very common in psychology and related fields, but are rare in NLP. In this paper we assess discrete and continuous scales used for measuring quality assessments of computer-generated language. We conducted six separate experiments designed to investigate the validity, reliability, stability, interchangeability and sensitivity of discrete vs. continuous scales. We show that continuous scales are viable for use in language evaluation, and offer distinct advantages over discrete scales.

AB - Studies assessing rating scales are very common in psychology and related fields, but are rare in NLP. In this paper we assess discrete and continuous scales used for measuring quality assessments of computer-generated language. We conducted six separate experiments designed to investigate the validity, reliability, stability, interchangeability and sensitivity of discrete vs. continuous scales. We show that continuous scales are viable for use in language evaluation, and offer distinct advantages over discrete scales.

M3 - Conference contribution with ISSN or ISBN

SN - 9781932432886

SP - 230

EP - 235

BT - The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies

PB - Association for Computational Linguistics

CY - Stroudsburg, PA, USA

T2 - The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies

Y2 - 1 January 2011

ER -

Discrete vs. continuous rating scales for language evaluation in NLP

Abstract

Conference

Access to Document

Fingerprint

Cite this