Gesture Me: A Machine Learning Tool for Designers to Train Gesture Classifiers

This paper contributes to the body of work examining how designers can be sup-ported in integrating machine learning (ML) capabilities into their designs for novel applications and services. It presents an online tool enabling designers and other non-specialist audiences to define body gestures, interactively and iterative-ly train and test a classifier to recognise these gestures, and integrate the trained classifier into a template web application. An empirical evaluation with MSc User Experience Design students and practitioners, all of whom had previous experi-ence in web development but not in ML, found that the tool enables them to de-fine, train and test a gesture recognition classifier with little or no help, and that engagement with the tool advances their understanding of the capabilities, limita-tions and operational aspects of ML. The evaluation confirmed the value of visu-alising the ML perspective and encouraging designers to experiment with ML to support their experiential learning. The study led to design recommendations that can inform the development of tools supporting designers to ideate and prototype ML-enhanced applications.