Evaluating portfolio assessment systems: what are the appropriate criteria?

Purpose. The purpose of this paper is to discuss how portfolio assessment processes should be evaluated. Background. Articles in the nursing literature discuss the use of validity and reliability as criteria for evaluating portfolio assessment processes, and recommendations include tighter specification of grading criteria, a standardized national approach to assessing clinical competence in nursing students, and inter-rater reliability checks. On the other hand, some general practitioner educators suggest that these may not be the appropriate criteria because the nature of the evidence in portfolios is descriptive and judgement-based rather than quantifiable. Method. Drawing on multi-method case study data from a recent study evaluating the use of portfolios in the assessment of learning and competence in nursing education in England, we suggest that criteria developed to evaluate qualitative research may be more appropriate for evaluating portfolio assessment processes. Discussion. Multiple sources of evidence from the varied perspectives of students, teachers, practice assessors and external examiners are tapped as part of the portfolio assessment process. Tripartite meetings between students, teachers and clinical assessors to review placements are crucial in verifying both the written evidence and students’ ability to communicate and critically analyse their performance. The variety of evidence collected would potentially allow monitoring, using qualitative research evaluation criteria, both of the portfolios themselves and the systems by which they are monitored and evaluated. However, not all this information is collected consistently and systematically, as called for in curriculum documents. Conclusions. Use of qualitative research evaluation criteria offers a potentially productive way forward in evaluating portfolio assessment processes but some aspects of current practice need to be tightened, particularly double marking, internal moderation and external examining.