An Assessment of Inter‑Observer Agreement in Water Source Classification and Sanitary Risk Observations

Joseph Okotto‑Okotto Okotto‑Okotto, Peggy Wanza, Emmah Kwoba, Weiyu Yu, Mawuli Dzodzomenyo, S.M. Thumbi, Diogo Trajano Gomes da Silva, Jim A. Wright

    Research output: Contribution to journalArticlepeer-review


    Sanitary risk inspection, an observation protocol for identifying contamination hazards around water sources, is promoted for managing rural water supply safety. However, it is unclear how far different observers consistently identify contamination hazards and consistently classify water source types using standard typologies. This study aimed to quantify inter-observer agreement in hazard identification and classification of rural water sources. Six observers separately visited 146 domestic water sources in Siaya County, Kenya, in wet and dry seasons. Each observer independently classified the source type and conducted a sanitary risk inspection using a standard protocol. Water source types assigned by an experienced observer were cross-tabulated against those of his colleagues, as were contamination hazards identified, and inter-observer agreement measures calculated. Agreement between hazards observed by the most experienced observer versus his colleagues was significant but low (intra-class correlation = 0.49), with inexperienced observers detecting fewer hazards. Inter-observer agreement in classifying water sources was strong (Cohen’s kappa = 0.84). However, some source types were frequently misclassified, such as sources adapted to cope with water insecurity (e.g. tanks drawing on both piped and rainwater). Observers with limited training and experience thus struggle to consistently identify hazards using existing protocols, suggesting observation protocols require revision and their implementation should be supported by comprehensive training. Findings also indicate that field survey teams struggle to differentiate some water source types based on a standard water source classification, particularly sources adapted to cope with water insecurity. These findings demonstrate uncertainties underpinning international monitoring and analyses of safe water access via household surveys.
    Original languageEnglish
    JournalExposure and Health
    Publication statusPublished - 24 Dec 2019

    Bibliographical note

    This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit


    • Water safety
    • Rainwater
    • Inter-observer agreement
    • Sanitary inspection
    • Sustainable development goals


    Dive into the research topics of 'An Assessment of Inter‑Observer Agreement in Water Source Classification and Sanitary Risk Observations'. Together they form a unique fingerprint.

    Cite this