Dr Simon Emmerson

Simon Emmerson Professor of Music, Technology and Innovation Faculty of Humanities De Montfort University Leicester LE1 9BH

Background remarks

The term ?interaction? is one which is much used but little understood. A musician routinely ?acts? with other musicians in the course of performance. They ?read? the actions and reactions of fellow musicians and adjust their responses accordingly. This is extended to ?reacting? to audiences, venue, acoustic of the space. In more improvised musical genres this becomes more substantive in that reactions become more unpredictable; new material may be created which has a direct influence (unmediated by any written ?score?) and the musician may be expected to respond directly. This crosses an unclear threshold to true interactivity. The borderline between ?mutual influence? and ?interactivity? is clearly not fixed.

Musicians know and deeply value this feeling of mutual influence. The problem is that an audience may not. With the advent of technological applications to sound and music production the source of the sound ? the performative gesture ? may (and usually does) become dislocated from the sound as heard (usually from a loudspeaker). In the first 20 years of live computer music in which ideas of interaction have played a substantial part, much work has been done which has played scant attention to this dislocation: the performers may believe they have created human-computer interaction but this is seldom perceived by the audience.

It is evident both from declared intentions that this project seeks as a core objective to address this issue: how to (re)establish perceptible relationships between the performance elements. Furthermore the aim is to do this at the material level ? that is not just performers influencing ?interpretation? but that musical material is actually determined through the analysis of sound and body gesture in live performance.

There is a second element of core concern in this project which is easy to take for granted. While possible before the advent of technological mediation, the technology allows and encourages the interaction of different art forms to a much greater extent. The addition of a dance component which proactively influences musical material can be substantially enhanced using technological mediation. A dance movement and a violinist?s movement may be tracked by technology to produce a similar kind of ?control information? ? although the tracking mechanisms may, for sure, be specific to the type of body movement involved. In this project the technology seemed to be unobtrusive and did not inhibit in any way the performers? actions ? in earlier technologies this has not always been the case.

The identity of the work

Comparing the performance I attended (15th February) with the video (made on the 17th February) the identity of the work was clearly defined: while divided into three ?movements?, each was itself divided into a number of sections ? effectively small ?scenes?. A series of solos, duos and ensembles presented nearly every possible combination of musician, dancer, robot and animation. Without describing each in detail we can assess their general characteristics; the whole was coherent and it is a paradoxical tribute to the structuring of the show that the division of the whole according to the programme note was not always clear in the result, there were no obvious hitches or hiatuses and the whole moved steadily without disjunction. While the individual scenes were clearly differentiated with respect to the forces at work the overall narrative continuity of the show was maintained. Although using the term narrative there was no ?story? line inherent in the sequence. There were moments of humour, emotional intensity (especially between human and non-human elements) and lyrical expressionism (sometimes bordering on the melancholic). The use of the technology was ? in this sense ? quite remote from the emotional disengagement of ?hard line modernism?. The electronic vistas created were sometimes stark and hard edged but always spoke of a feeling or state of being as much as a ?structure or form?.

Sound world - sources

The musical character of each section was defined by the type of performance gesture which in general also closely defined the sound type. These followed the classic series of ?progressive remoteness? away from what we could directly perceive to be the source of the sound:

Violin or voice produced (clear origin); Probably violin or voice produced (surmised) ? processed electronically; Probably not instrumental or vocal (or at least so highly processed we would never know).

Stevie Wishart balanced these sound types with flair and sensitivity; she retained a strongly melodic approach in many sections; from modal to ?late expressionist?. It is clear that she often deliberately triggers ?subversive? material which counters this initial melody and sets up quite stark contrasts within the musical flow. The vocal material contrasted between the two performances I had access to which gave an interesting insight into the degree of definition of material types she had set herself in the plans for the work.

Gesture world ? controls

Generally the relationship of gesture to result was commendably clear. The relationships covered all possible combinations of physical gesture, sound quality (analysed), real dance, animated dance, robot gesture movement.

Focusing on two which involved music and the gestures of its production: Physical gesture (Wishart) to sound: this was the clearest to perceive as a member of the audience and clearly reengaged the pre-technological physical world of physical gesture as sound cause; Sound gesture (that is the envelope of the sound quality) to live dance, animated dance, robot: this is a more intimate relationship, movement is not dictated by the sound but the dancers (real or animated) ?respond?. This response was inevitably more subtle with respect to the human. The ?expressivity? of the animated dancer was somewhat limited to limb movement and overall body shape. Future developments of this software will need to embody some very advanced concepts of muscle tension, posture and the like to capture this advanced body expression. (For example, that of the athlete ?on the blocks?, stationary yet we sense the stored energy ready to spring and the attendant emotional tension.)

Questions raised and issues to be developed

The development of such a rich system of audio visual dance interactivity has made a very real contribution to this field both artistically and technically (the two cannot be separated in this context). The refining of questions for research (in the broadest sense to include creation and performance) is a key function of research itself:

Complexity and Polyphony This project focused on the interaction of soloists ? single ?voices?. Technology can produce polyphony from these with ease but risks losing the ?connection? of gesture to sound which is key to the approach of this project. Nonetheless could systems involving more human performers be developed?

Causality and time delay This project commendably developed a sensible relationship with what is known in sound for animation as ?Mickey mousing? ? the crude coordination of ?cause and (sound) effect?. Perhaps influenced by this background there has been much live electronic music which has positively tried to obscure cause/effect relations. This can lead to much audience alienation ? the listener has no idea what action (if any) of a performer produces a particular result. This performance in re-engaging clear cause/effect relationships suggests there is more to be done to explore and exploit cause/effect delays (longer term memory) and how these are perceived.

Causality, interactivity, predictability Especially in improvised musics there are issues of predictability versus surprise. In this show truly surprising sounds and music were actually quite rare (the occasional ?subversive? sound referred to above). This is ?not surprising? if a model of interactivity takes a close adherence to literal cause/effect ? a present result is bound to be based on something which has recently occurred. Throwing the unexpected into interactive performance will be an interesting investigation.

Conclusion

The project is a major achievement and a substantial success. While the questions asked have been around for a decade this is one of the first really focused attempts to bring these worlds together. In engaging the world of movement and dance, primal gesture/movement types have been used to ensure that the audience sees and hears cause/effect relations and hence interactivity more clearly. Any shortcomings (for example in the expressivity of the real-time animation) were clearly those of a technique at the earliest stage of its development. The technology enabled an extraordinary range of music performer-sound interactions. While the sound world was not in itself extended from that previously available, the audience could easily apprehend a substantial increase in performer control over that environment.