So I’m making my robot move his head to the music in response to social and musical events and I’m going to do a study to research the effects of this in about two months. To find out when it should move its head and why I’ve done some reading and written it up for you lovely folks.
First the social.
Fong et al. cited realistic facial expression as a key design factor in the social robots, especially in the demonstration of affective behaviour . They report that this is often not done well and describe mechanical approaches as often clunky and abrupt.
Although 90% of gestures happen during conversation and are redundant , Breazeal and Fitzpatrick suggest that all bodily movements are perceived as semantically rich, even if this is not their intention. Head movements can also provide attentional cues that make up our sense of engagement with another . Further, Macdorman and Cowley demonstrated that attentive head movements are sufficient to elicit the perception of what they call personhood, a concept that we have shown to have large overlaps with social presence and believability . Head movements have also be used by Weinberg et al. in their musical robot Shimon in order to increase its social presence within an ensemble .
Now onto music.
In almost all acoustic music performance, the body, and in some cases, the head and face, are to some extent coupled to the generation of sound [3, 6–8]. However, they are also used as cues, sometimes intentionally, sometimes not, to augment the performance and to anticipate or accentuate important events. For example, in an analysis of an improvising jazz guitarist, Gratier demonstrates that musicians may use their bodily movements to convey the structure and meaning of the music . Similarly, Vines et al. discovered that the perceived tension of a performance is most effected by visual cues, rather than auditory . Further, that it is a combination of sound and visual stimulus that effects audience’s perception of phrasing in a musical performance. This is supported by their observation that the contours of the performers body movement tended to align with the phrasing of the music. Similarly, Thompson et al. also suggest that facial expressions are used to convey timing events, thus increasing musical intelligibility .
They also find that facial expressions can be used to make music sound more or less dissonant or to make musical intervals sound further apart or closer together.
Gratier suggests that facial displays of affect may serve the purpose of grounding between improvisers. For example, a musician may smile at a mistake or a particularly satisfying lick . Further, whilst drawing comparisons between improvised music and conversation, she reports that mutual gaze is much less constant in the former. Though less frequent, it tends to occur during moments of structural change or importance in the music.
In a study of a performance by blues guitarist BB King, Thompson et al. find he often used facial expressions to display affect. For example, in moments of tension he takes on an introspective demeanour, looking down and shaking his head. A musicologist inter- prets this as him signally he feels the emotion but will not submit to it. Alternatively, in moments of release he opens his mouth towards the audience as if in wander. Similarly, Vines et al. found that more expressive performances will create more tension . Again using nonverbal cues in relation to rhythmic timing, King’s head movements tend to re- act to individual notes and licks and he tends to reflect only his performance, rather than that of his band. Conversely, a study by the same authors of a Judy Garland performance shows that she uses hand gestures in a more illustrative fashion, literally reflecting the lyrics of the song.
 Pierce Brennand and Cheng Gordon. Automatic Face Replacement for Humanoid Robot with 3D Face Shaped Display. In ICSR’12, pages 469–474, 2012.
 Terrence Fong, Illah Nourbakhsh, and Kerstin Dautenhahn. A survey of socially interactive robots. Robotics and Autonomous Systems, 42:143–166, 2003.
 Maya Gratier. Grounding in musical interaction: Evidence from jazz performances. Musicae Scien- tiae, 12(1 suppl):71–110, 2008.
 K.F Macdorman and S.J Cowley. Long-term relationships as a benchmark for robot personhood. ROMAN ’06, pages 378–383, 2006.
 M.P. Michalowski, S Sabanovic, and R Simmons. A spatial model of engagement for a social robot. In Proc. 2006 Advanced Motion Control Workshop, pages 762–767, Istanbul, 2006.
 WF Thompson and P Graham. Seeing music performance: Visual influences on perception and experience. Semiotica, (156):203–227, 2005.
 Bradley W Vines, Carol L Krumhansl, Marcelo M Wanderley, and Daniel J Levitin. Cross-modal interactions in the perception of musical performance. Cognition, 101(1):80–113, 2006.
 Marcelo M Wanderley. Quantitative analysis of non-obvious performer gestures. pages 241–253, 2002.
 G. Weinberg, A. Raman, and T. Mallikarjuna. Interactive jamming with Shimon: a social robotic
musician. In Proc. 2009 Human Robot Interaction Conf., pages 233–234, San Diego, CA, 2009.