Evaluating guidelines for reporting empirical software engineering studies

Barbara Kitchenham, M. Al-Khilidar, M. Ali Babar, M. Berry, Karl Cox, J. Keung, F. Kurniawati, M. Staples, J. Zhang, L. Zhu

Research output: Contribution to journalArticle

Abstract

Several researchers have criticized the standards of performing and reporting empirical studies in software engineering. In order to address this problem, Jedlitschka and Pfahl have produced reporting guidelines for controlled experiments in software engineering. They pointed out that their guidelines needed evaluation. We agree that guidelines need to be evaluated before they can be widely adopted. The aim of this paper is to present the method we used to evaluate the guidelines and report the results of our evaluation exercise. We suggest our evaluation process may be of more general use if reporting guidelines for other types of empirical study are developed. We used a reading method inspired by perspective-based and checklist-based reviews to perform a theoretical evaluation of the guidelines. The perspectives used were: Researcher, Practitioner/Consultant, Meta-analyst, Replicator, Reviewer and Author. Apart from the Author perspective, the reviews were based on a set of questions derived by brainstorming. A separate review was performed for each perspective. The review using the Author perspective considered each section of the guidelines sequentially. The reviews detected 44 issues where the guidelines would benefit from amendment or clarification and 8 defects. Reporting guidelines need to specify what information goes into what section and avoid excessive duplication. The current guidelines need to be revised and then subjected to further theoretical and empirical validation. Perspective-based checklists are a useful validation method but the practitioner/consultant perspective presents difficulties.
Original languageEnglish
Pages (from-to)97-121
Number of pages25
JournalEmpirical Software Engineering
Volume13
Issue number1
DOIs
Publication statusPublished - 1 Feb 2008

Fingerprint

Software engineering
Defects
Experiments

Keywords

  • Controlled experiments
  • Software engineering
  • Guidelines
  • Perspective-based reading
  • Checklist-based reviews

Cite this

Kitchenham, B., Al-Khilidar, M., Ali Babar, M., Berry, M., Cox, K., Keung, J., ... Zhu, L. (2008). Evaluating guidelines for reporting empirical software engineering studies. Empirical Software Engineering, 13(1), 97-121. https://doi.org/10.1007/s10664-007-9053-5
Kitchenham, Barbara ; Al-Khilidar, M. ; Ali Babar, M. ; Berry, M. ; Cox, Karl ; Keung, J. ; Kurniawati, F. ; Staples, M. ; Zhang, J. ; Zhu, L. / Evaluating guidelines for reporting empirical software engineering studies. In: Empirical Software Engineering. 2008 ; Vol. 13, No. 1. pp. 97-121.
@article{5fe591f3c625491abed68ee503897809,
title = "Evaluating guidelines for reporting empirical software engineering studies",
abstract = "Several researchers have criticized the standards of performing and reporting empirical studies in software engineering. In order to address this problem, Jedlitschka and Pfahl have produced reporting guidelines for controlled experiments in software engineering. They pointed out that their guidelines needed evaluation. We agree that guidelines need to be evaluated before they can be widely adopted. The aim of this paper is to present the method we used to evaluate the guidelines and report the results of our evaluation exercise. We suggest our evaluation process may be of more general use if reporting guidelines for other types of empirical study are developed. We used a reading method inspired by perspective-based and checklist-based reviews to perform a theoretical evaluation of the guidelines. The perspectives used were: Researcher, Practitioner/Consultant, Meta-analyst, Replicator, Reviewer and Author. Apart from the Author perspective, the reviews were based on a set of questions derived by brainstorming. A separate review was performed for each perspective. The review using the Author perspective considered each section of the guidelines sequentially. The reviews detected 44 issues where the guidelines would benefit from amendment or clarification and 8 defects. Reporting guidelines need to specify what information goes into what section and avoid excessive duplication. The current guidelines need to be revised and then subjected to further theoretical and empirical validation. Perspective-based checklists are a useful validation method but the practitioner/consultant perspective presents difficulties.",
keywords = "Controlled experiments, Software engineering, Guidelines, Perspective-based reading, Checklist-based reviews",
author = "Barbara Kitchenham and M. Al-Khilidar and {Ali Babar}, M. and M. Berry and Karl Cox and J. Keung and F. Kurniawati and M. Staples and J. Zhang and L. Zhu",
year = "2008",
month = "2",
day = "1",
doi = "10.1007/s10664-007-9053-5",
language = "English",
volume = "13",
pages = "97--121",
journal = "Empirical Software Engineering",
issn = "1382-3256",
number = "1",

}

Kitchenham, B, Al-Khilidar, M, Ali Babar, M, Berry, M, Cox, K, Keung, J, Kurniawati, F, Staples, M, Zhang, J & Zhu, L 2008, 'Evaluating guidelines for reporting empirical software engineering studies', Empirical Software Engineering, vol. 13, no. 1, pp. 97-121. https://doi.org/10.1007/s10664-007-9053-5

Evaluating guidelines for reporting empirical software engineering studies. / Kitchenham, Barbara; Al-Khilidar, M.; Ali Babar, M.; Berry, M.; Cox, Karl; Keung, J.; Kurniawati, F.; Staples, M.; Zhang, J.; Zhu, L.

In: Empirical Software Engineering, Vol. 13, No. 1, 01.02.2008, p. 97-121.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Evaluating guidelines for reporting empirical software engineering studies

AU - Kitchenham, Barbara

AU - Al-Khilidar, M.

AU - Ali Babar, M.

AU - Berry, M.

AU - Cox, Karl

AU - Keung, J.

AU - Kurniawati, F.

AU - Staples, M.

AU - Zhang, J.

AU - Zhu, L.

PY - 2008/2/1

Y1 - 2008/2/1

N2 - Several researchers have criticized the standards of performing and reporting empirical studies in software engineering. In order to address this problem, Jedlitschka and Pfahl have produced reporting guidelines for controlled experiments in software engineering. They pointed out that their guidelines needed evaluation. We agree that guidelines need to be evaluated before they can be widely adopted. The aim of this paper is to present the method we used to evaluate the guidelines and report the results of our evaluation exercise. We suggest our evaluation process may be of more general use if reporting guidelines for other types of empirical study are developed. We used a reading method inspired by perspective-based and checklist-based reviews to perform a theoretical evaluation of the guidelines. The perspectives used were: Researcher, Practitioner/Consultant, Meta-analyst, Replicator, Reviewer and Author. Apart from the Author perspective, the reviews were based on a set of questions derived by brainstorming. A separate review was performed for each perspective. The review using the Author perspective considered each section of the guidelines sequentially. The reviews detected 44 issues where the guidelines would benefit from amendment or clarification and 8 defects. Reporting guidelines need to specify what information goes into what section and avoid excessive duplication. The current guidelines need to be revised and then subjected to further theoretical and empirical validation. Perspective-based checklists are a useful validation method but the practitioner/consultant perspective presents difficulties.

AB - Several researchers have criticized the standards of performing and reporting empirical studies in software engineering. In order to address this problem, Jedlitschka and Pfahl have produced reporting guidelines for controlled experiments in software engineering. They pointed out that their guidelines needed evaluation. We agree that guidelines need to be evaluated before they can be widely adopted. The aim of this paper is to present the method we used to evaluate the guidelines and report the results of our evaluation exercise. We suggest our evaluation process may be of more general use if reporting guidelines for other types of empirical study are developed. We used a reading method inspired by perspective-based and checklist-based reviews to perform a theoretical evaluation of the guidelines. The perspectives used were: Researcher, Practitioner/Consultant, Meta-analyst, Replicator, Reviewer and Author. Apart from the Author perspective, the reviews were based on a set of questions derived by brainstorming. A separate review was performed for each perspective. The review using the Author perspective considered each section of the guidelines sequentially. The reviews detected 44 issues where the guidelines would benefit from amendment or clarification and 8 defects. Reporting guidelines need to specify what information goes into what section and avoid excessive duplication. The current guidelines need to be revised and then subjected to further theoretical and empirical validation. Perspective-based checklists are a useful validation method but the practitioner/consultant perspective presents difficulties.

KW - Controlled experiments

KW - Software engineering

KW - Guidelines

KW - Perspective-based reading

KW - Checklist-based reviews

U2 - 10.1007/s10664-007-9053-5

DO - 10.1007/s10664-007-9053-5

M3 - Article

VL - 13

SP - 97

EP - 121

JO - Empirical Software Engineering

JF - Empirical Software Engineering

SN - 1382-3256

IS - 1

ER -