Roberto Bresin: complete list of publications
| 2012 (2012). Evaluation of computer systems for expressive music performance. In Kirke, A., & Miranda, E. (Eds.), Guide to Computing for Expressive Music Performance. Berlin: Springer-Verlag (in press). (2012). Interactive sonification. Journal on Multimodal User Interfaces, 5(3), 85-86. [link] (2012). Systems for interactive control of computer generated music performance. In Kirke, A., & Miranda, E. (Eds.), Guide to Computing for Expressive Music Performance. Berlin: Springer-Verlag (in press). (2012). Experimental methods for the perceptual evaluation of sound-producing objects and interfaces. In Franinovic, K., & Serafin, S. (Eds.), Sonic Interaction Design.. Boston, MA: MIT Press. (2012). Interactive sonification of synchronisation of motoric behaviour in social active listening of music with mobile devices. Journal on Multimodal User Interfaces, 5(3), 157-173. [abstract] [link]Abstract: This paper evaluates three different interactive sonifications of dyadic coordinated human rhythmic activity. An index of phase synchronisation of gestures was chosen as coordination metric. The sonifications are implemented as three prototype applications exploiting mobile devices: Sync'n'Moog, Sync'n'Move, and Sync'n'Mood. Sync'n'Moog sonifies the phase synchronisation index by acting directly on the audio signal and applying a nonlinear time-varying filtering technique. Sync'n'Move intervenes on the multi-track music content by making the single instruments emerge and hide. Sync'n'Mood manipulates the affective features of the music performance. The three sonifications were also tested against a condition without sonification.2011 (2011). Emotion rendering in music: Range and characteristic values of seven musical variables. Cortex, 47(9), 1068-1081. [abstract] [link]Abstract: Many studies on the synthesis of emotional expression in music performance have focused on the effect of individual performance variables on perceived emotional quality by making a systematical variation of variables. However, most of the studies have used a predetermined small number of levels for each variable, and the selection of these levels has often been done arbitrarily. The main aim of this research work is to improve upon existing methodologies by taking a synthesis approach. In a production experiment, 20 performers were asked to manipulate values of 7 musical variables simultaneously (tempo, sound level, articulation, phrasing, register, timbre, and attack speed) for communicating 5 different emotional expressions (neutral, happy, scary, peaceful, sad) for each of 4 scores. The scores were compositions communicating four different emotions (happiness, sadness, fear, calmness). Emotional expressions and music scores were presented in combination and in random order for each performer for a total of 5 × 4 stimuli. The experiment allowed for a systematic investigation of the interaction between emotion of each score and intended expressed emotions by performers. A two-way analysis of variance (ANOVA), repeated measures, with factors emotion and score was conducted on the participants’ values separately for each of the seven musical factors. There are two main results. The first one is that musical variables were manipulated in the same direction as reported in previous research on emotional expressive music performance. The second one is the identification for each of the five emotions the mean values and ranges of the five musical variables tempo, sound level, articulation, register, and instrument. These values resulted to be independent from the particular score and its emotion. The results presented in this study therefore allow for both the design and control of emotionally expressive computerized musical stimuli that are more ecologically valid than stimuli without performance variations. (2011). Sonification of physical quantities throughout history: a meta-study of previous mapping strategies. In Proceedings of the 17th International Conference on Auditory Display (ICAD 2011). Budapest, Hungary: OPAKFI Egyesület. [abstract] [pdf]Abstract: We introduce a meta-study of previous sonification designs taking physical quantities as input data. The aim is to build a solid foundation for future sonification works so that auditory display researchers would be able to take benefit from former studies, avoiding to start from scratch when beginning new sonification projects. This work is at an early stage and the objective of this paper is rather to introduce the methodology than to come to definitive conclusions. After a historical introduction, we explain how to collect a large amount of articles and extract useful information about mapping strategies. Then, we present the physical quantities grouped according to conceptual dimensions, as well as the sound parameters used in sonification designs and we summarize the current state of the study by listing the couplings extracted from the article database. A total of 54 articles have been examined for the present article. Finally, a preliminary analysis of the results is performed. (2011). Interactive sonification of expressive hand gestures on a handheld device. Journal on Multimodal User Interfaces, 1-9. [abstract] [link]Abstract: We present here a mobile phone application called MoodifierLive which aims at using expressive music performances for the sonification of expressive gestures through the mapping of the phone’s accelerometer data to the performance parameters (i.e. tempo, sound level, and articulation). The application, and in particular the sonification principle, is described in detail. An experiment was carried out to evaluate the perceived matching between the gesture and the music performance that it produced, using two distinct mappings between gestures and performance. The results show that the application produces consistent performances, and that the mapping based on data collected from real gestures works better than one defined a priori by the authors. (2011). MoodifierLive: interactive and collaborative music performance on mobile devices. In Proceedings of the International Conference on New Interfaces for Musical Expression (NIME11) (pp. 116-119). Oslo, Norway: University of Oslo and Norwegian Academy of Music. [abstract] [pdf]Abstract: This paper presents Moodi erLive, a mobile phone application for interactive control of rule-based automatic music performance. Five different interaction modes are available, of which one allows for collaborative performances with up to four participants, and two let the user control the expressive performance using expressive hand gestures. Evaluations indicate that the application is interesting, fun to use, and that the gesture modes, especially the one based on data from free expressive gestures, allow for performances whose emotional content matches that of the gesture that produced them. (2011). Ljudskrapan/The Soundscraper: Sound exploration for children with complex needs, accommodating hearing aids and cochlear implants. In Zanolla, S., Avanzini, F., Canazza, S., & de Götzen, A. (Eds.), Proceedings of the Sound and Music Computing Conference (pp. 70-76). Padova, Italy: Padova University Press. [abstract] [pdf]Abstract: This paper describes a system for accommodating active listening for persons with hearing aids or cochlear implants, with a special focus on children with complex needs, for instance at an early stage of cognitive development and with additional physical disabilities. The system is called Ljudskrapan (or the Soundscraper in English) and consists of a software part in Pure data and a hardware part using an Arduino microcontroller with a combination of sensors. For both the software and hardware development, one of the most important aspects was to always ensure that the system was flexible enough to cater for the very different conditions that are characteristic of the intended user group. The Soundscraper has been tested with 25 children with good results. An increased attention span was reported, as well as surprising and positive reactions from children where the caregivers were unsure whether they could hear at all. The sound generating models, the sensors and the parameter mapping were simple, but provided a controllable and complex enough sound environment even with limited interaction. (2011). Analysis of the acoustics and playing strategies of turntable scratching. Acta Acustica united with Acustica, 97(2), 303-314. [abstract] [link]Abstract: Scratching performed by a DJ (disk jockey) is a skillful style of playing the turntable with complex musical output. This study focuses on the description of some of the acoustical parameters and playing strategies of typical scratch improvisations, and how these parameters typically are used for expressive performance. Three professional DJs were instructed to express different emotions through improvisations, and both audio and gestural data were recorded. Feature extraction and analysis of the recordings are based on a combination of audio and gestural data, instrument characteristics, and playing techniques. The acoustical and performance parameters extracted from the recordings give a first approximation on the functional ranges within which DJs normally play. Results from the analysis show that parameters which are important for other solo instrument performances, such as pitch, have less influence in scratching. Both differences and commonalities between the DJs' playing styles were found. Impact that the findings of this work may have on constructing models for scratch performances are discussed.2010 (2010). Expressive sonification of footstep sounds. In Bresin, R., Hermann, T., & Hunt, A. (Eds.), Proceedings of the Interaction Sonification workshop (ISon) 2010 (pp. 51-54). Stockholm, Sweden: KTH Royal Institute of Technology. [abstract] [pdf]Abstract: In this study we present the evaluation of a model for the interactive sonification of footsteps. The sonification is achieved by means of specially designed sensored-shoes which control the expressive parameters of novel sound synthesis models capable of reproducing continuous auditory feedback for walking. In a previous study, sounds corresponding to different grounds were associated to different emotions and gender. In this study, we used an interactive sonification actuated by the sensored-shoes for providing auditory feedback to walkers. In an experiment we asked subjects to walk (using the sensored-shoes) with four different emotional intentions (happy, sad, aggressive, tender) and for each emotion we manipulated the ground texture sound four times (wood panels, linoleum, muddy ground, and iced snow). Preliminary results show that walkers used a more active walking style (faster pace) when the sound of the walking surface was characterized by an higher spectral centroid (e.g. iced snow), and a less active style (slower pace) when the spectral centroid was low (e.g. muddy ground). Harder texture sounds lead to more aggressive walking patters while softer ones to more tender and sad walking styles. (2010). Communication of Musical Expression by Means of Mobile Robot Gestures. Journal of Multimodal User Interfaces, 3(1-2), 109-118. [abstract] [link]Abstract: We developed a robotic system that can behave in an emotional way. A 3-wheeled simple robot with limited degrees of freedom was designed. Our goal was to make the robot displaying emotions in music performance by performing expressive movements. These movements have been compiled and programmed based on literature about emotion in music, musicians’ movements in expressive performances, and object shapes that convey different emotional intentions. The emotions happiness, anger, and sadness have been implemented in this way. General results from behavioral experiments show that emotional intentions can be synthesized, displayed and communicated by an artificial creature, also in constrained circumstances. (2010). User-centric context-aware mobile applications for embodied music listening. In Daras, P., Ibarra, O. M., Akan, O., Bellavista, P., Cao, J., Dressler, F., Ferrari, D., Gerla, M., Kobayashi, H., Palazzo, S., Sahni, S., Shen, X. (., Stan, M., Xiaohua, J., Zomaya, A., & Coulson, G. (Eds.), User Centric Media (pp. 21-30). Springer Berlin Heidelberg. [abstract] [link]Abstract: This paper surveys a collection of sample applications for networked user-centric context-aware embodied music listening. The applications have been designed and developed in the framework of the EU-ICT Project SAME (www.sameproject.eu) and have been presented at Agora Festival (IRCAM, Paris, France) in June 2009. All of them address in different ways the concept of embodied, active listening to music, i.e., enabling listeners to interactively operate in real-time on the music content by means of their movements and gestures as captured by mobile devices. In the occasion of the Agora Festival the applications have also been evaluated by both expert and non-expert users (2010). Sonification of sculler movements, developement of preliminary methods. In Bresin, R., Hermann, T., & Hunt, A. (Eds.), Proceedings of ISon 2010, 3rd Interactive Sonification Workshop (pp. 39-43). Stockholm, Sweden: KTH Royal Institute of Technology. [abstract] [pdf]Abstract: Sonification is a widening field of research with many possibilities for practical applications in various scientific domains. The rapid development of mobile technology capable of efficiently handling numerical information offers new opportunities for interactive auditory display. In this scope, the SONEA project (SONification of Elite Athletes) aims at improving performances of Olympic-level athletes by enhancing their training techniques, taking advantage of both the strong coupling between auditory and sensorimotor systems, and the efficient learning and memorizing abilities pertaining the sense of hearing. An application to rowing is presented in this article. Rough estimates of the position and mean velocity of the craft are given by a GPS receiver embedded in a smartphone taken onboard. An external accelerometer provides boat acceleration data with higher temporal resolution. The development of preliminary methods for sonifying the collected data has been carried out under the specific constraints of a mobile device platform. The sonification is either performed by the phone as a real-time feedback or by a computer using data files as input for an a posteriori analysis of the training. In addition, environmental sounds recorded during training can be synchronized with the sonification to perceive the coherence of the sequence of sounds throughout the rowing cycle. First results show that sonification using a parameter-mapping method over (2010). Improving running mechanics by use of interactive sonification. In Bresin, R., Hermann, T., & Hunt, A. (Eds.), Proceedings of the Interaction Sonification workshop (ISon) 2010 (pp. 95-98). Stockholm, Sweden: KTH Royal Institute of Technology. [abstract] [pdf]Abstract: Running technique has a large effect on running economy in terms of consumed amount of oxygen. Changing the natural running technique, though, is a difficult task. In this paper, a method based on sonification is presented, that will assist the runner in obtaining a more efficient running style. The system is based on an accelerometer sending data to a mobile phone. Thus the system is non-obtrusive and possible to use in the everyday training. Specifically, the feedback given is based on the runner’s vertical displacement of the center of mass. As this is the main source of energy expenditure during running, it is conjectured that a reduced vertical displacement should improve running economy. (2010). Interactive sonification of emotionally expressive gestures by means of music performance. In Bresin, R., Hermann, T., & Hunt, A. (Eds.), Proceedings of ISon 2010, 3rd Interactive Sonification Workshop (pp. 113-116). Stockholm, Sweden: KTH Royal Institute of Technology. [abstract] [pdf]Abstract: This study presents a procedure for interactive sonification of emotionally expressive hand and arm gestures by affecting a musical performance in real-time. Three different mappings are described that translate accelerometer data to a set of parameters that control the expressiveness of the performance by affecting tempo, dynamics and articulation. The first two mappings, tested with a number of subjects during a public event, are relatively simple and were designed by the authors using a top-down approach. According to user feedback, they were not intuitive and limited the usability of the software. A bottom-up approach was taken for the third mapping: a Classification Tree was trained with features extracted from gesture data from a number of test subject who were asked to express different emotions with their hand movements. A second set of data, where subjects were asked to make a gesture that corresponded to a piece of expressive music they just listened to, were used to validate the model. The results were not particularly accurate, but reflected the small differences in the data and the ratings given by the subjects to the different performances they listened to. (2010). The Skipproof Virtual Turntable for High-level Control of Scratching. Computer Music Journal, 34(2), 39-50. [abstract] [39]Abstract: Skipproof is an application that emulates a typical disc jockey (DJ) setup of turntable plus mixer and also allows high-level control of the playing style known as scratching. High-level control in this case means performing with modeled, complex DJ gestures through simplified actions: For instance, letting a single movement produce a sound that normally would require precisely synchronized rightand left-hand gestures. The performer controls Skipproof either with the software interface or through hardware devices connected to the computer. The hardware devices become alternative performance interfaces to the standard turntable, controlling the Skipproof application with both low-level gestures and high-level control actions in real time. The mapping between hardware input and Skipproof output is freely adaptable. Skipproof is in the prototype phase, but it has already been used in several projects in recent years.2009 (2009). Embodied music listening and making in context-aware mobile applications: the EU-ICT SAME Project. In Gesture Workshop 2009. Bielefeld, Germany. [pdf] (2009). Gestures in performance. In Godøy, R. I., & Leman, M. (Eds.), Musical Gestures: Sound, Movement, and Meaning (pp. 36-68). New York: Routledge. [abstract] [link]Abstract: We experience and understand the world, including music, through body movement–when we hear something, we are able to make sense of it by relating it to our body movements, or form an image in our minds of body movements. Musical Gestures is a collection of essays that explore the relationship between sound and movement. It takes an interdisciplinary approach to the fundamental issues of this subject, drawing on ideas, theories and methods from disciplines such as musicology, music perception, human movement science, cognitive psychology, and computer science. (2009). Sound design and perception in walking interactions. International Journal of Human-Computer Studies, 67(11), 947-959. [abstract] [link]Abstract: This paper reviews the state of the art in the display and perception of walking generated sounds and tactile vibrations, and their current and potential future uses in interactive systems. As non-visual information sources that are closely linked to human activities in diverse environments, such signals are capable of communicating about the spaces we traverse and activities we encounter in familiar and intuitive ways. However, in order for them to be effectively employed in human–computer interfaces, significant knowledge is required in areas including the perception of acoustic signatures of walking, and the design, engineering, and evaluation of interfaces that utilize them. Much of this expertise has accumulated in recent years, although many questions remain to be explored. We highlight past work and current research directions in this multidisciplinary area of investigation, and point to potential future trends.2008 (2008). Timing in piano music - Testing a model of melody lead. In Proc. of the 10th International Conference on Music Perception and Cognition. Sapporo, Japan. [pdf] (2008). Auditory feedback through continuous control of crumpling sound synthesis. In Proceedings of Sonic Interaction Design: Sound, Information and Experience. A CHI 2008 Workshop organized by COST Action IC0601 (pp. 23-28). IUAV University of Venice. [abstract] [pdf]Abstract: A realtime model for the synthesis of crumpling sounds is presented. By capturing the statistics of short sonic transients which give rise to crackling noise, it allows for a consistent description of a broad spectrum of audible physical processes which emerge in several everyday interaction contexts. The model drives a nonlinear impactor that sonifies every transient, and it can be parameterized depending on the physical attributes of the crumpling material. Three different scenarios are described, respectively simulating the foot interaction with aggregate ground materials, augmenting a dining scenario, and affecting the emotional content of a footstep sequence. Taken altogether, they emphasize the potential generalizability of the model to situations in which a precise control of auditory feedback can significantly increase the enactivity and ecological validity of an interface. (2008). Influence of Acoustic Cues on the Expressive Performance of Music. In Proc. of the 10th International Conference on Music Perception and Cognition. Sapporo, Japan (abstract only). (2008). Controlling sound production. In Polotti, P., & Rocchesso, D. (Eds.), Sound to Sense - Sense to Sound: A state of the art in Sound and Music Computing (pp. 447-486). Berlin: Logos Verlag. [pdf] (2008). Real-time control of music performance. In Polotti, P., & Rocchesso, D. (Eds.), Sound to Sense - Sense to Sound: A state of the art in Sound and Music Computing (pp. 279-302). Berlin: Logos Verlag. [pdf] (2008). Sense in expressive music performance: Data acquisition, computational studies, and models. In Polotti, P., & Rocchesso, D. (Eds.), Sound to Sense - Sense to Sound: A state of the art in Sound and Music Computing (pp. 195-242). Berlin: Logos Verlag. [pdf] (2008). Verbal Description of DJ Recordings. In Proc. of the 10th International Conference on Music Perception and Cognition (pp. 20). Sapporo. [abstract]Abstract: In a recent pilot study, DJs were asked to perform the same composition using different intended emotional expression (happiness, sadness etc). In a successive test, these intentions could not be matched by listeners' judgement. One possible explanation is that DJs have a different vocabulary when describing expressivity in their performances. We designed an experiment to understand how DJs and listeners describe the music. The experiment was aimed at identifying a set of descriptors used mainly with scratch music, but possibly also with other genres. In a web questionnaire, subjects were presented with sound stimuli from scratch music recordings. Each participant described the music with words, phrases and terms in a free labelling task. The resulting list of responses was analyzed in several steps and condensed to a set of about 10 labels. Important differences were found between describing scratch music and other Western genres such as pop, jazz or classical music. For instance, labels such as cocky, cool, amusement and skilled were common. These specific labels seem mediated from the characteristic hip-hop culture. The experiment offered some explanation to the problem of verbally describing expressive scratch music. The set of labels found can be used for further experiments, for example when instructing DJs in performances. (2008). Describing the emotional content of hip-hop DJ recordings. In The Neurosciences and Music III (pp. 565). Montreal. (2008). Sonic Interaction Design: Sound, Information and Experience. In CHI '08 extended abstracts on Human factors in computing systems (pp. 3969-3972). New York, NY, USA: ACM. [abstract] [pdf]Abstract: Sonic Interaction Design (SID) is an emerging field that is positioned at the intersection of auditory display, ubiquitous computing, interaction design, and interactive arts. SID can be used to describe practice and inquiry into any of various roles that sound may play in the interaction loop between users and artifacts, services, or environments, in applications that range from the critical functionality of an alarm, to the artistic significance of a musical creation. This field is devoted to the privileged role the auditory channel can assume in exploiting the convergence of computing, communication, and interactive technologies. An overemphasis on visual displays has constrained the development of interactive systems that are capable of making more appropriate use of the auditory modality. Today the ubiquity of computing and communication resources allows us to think about sounds in a proactive way. This workshop puts a spotlight on such issues in the context of the emerging domain of SID. (2008). Emotional cues in knocking sounds. In Proc. of the 10th International Conference on Music Perception and Cognition (pp. 276). Sapporo, Japan. [pdf]2007 (2007). Displaying expression in musical performance by means of a mobile robot. In Paiva, A., Prada, R., & Picard, R. W. (Eds.), Affective Computing and Intelligent Interaction (pp. 753-754). Berlin / Heidelberg: Springer. [abstract] [link]Abstract: In recent times several attempts have been made to give a robot or broader spoken a computer some kind of feelings in order to understand and model human capacities. The main idea of our work was the design of expressive robot movements for the display of emotional content embedded in the audio layer in both live and recorded music performance. Starting from results in studies on musicians’ body in emotional expressive music performance (see [3]), we tried to map different movement cues (e.g. speed, fluency) to movements of a small mobile robot. The robot had constraints of sensors and motors, so the emotions were implemented taking into account only the main characteristics of musicians’ movements. We implemented movements for the three emotions happiness, anger and sadness. Subjects were asked to judge in a perceptual test which emotional intentions were communicated by the movements. (2007). Expressive Control of Music and Visual Media by Full-Body Movement. In Proceedings of the 7th international conference on New interfaces for musical expression (pp. 390-391). New York, NY, USA: ACM. [abstract] [pdf]Abstract: In this paper we describe a system which allows users to use their full-body for controlling in real-time the generation of an expressive audio-visual feedback. The system extracts expressive motion features from the user’s full-body movements and gestures. The values of these motion features are mapped both onto acoustic parameters for the real-time expressive rendering of a piece of music, and onto real-time generated visual feedback projected on a screen in front of the user. (2007). User-Centered Control of Audio and Visual Expressive Feedback by Full-Body Movements. In Paiva, A., Prada, R., & Picard, R. W. (Eds.), Affective Computing and Intelligent Interaction (pp. 501-510). Berlin / Heidelberg: Springer. [abstract] [link]Abstract: n this paper we describe a system allowing users to express themselves through their full-body movement and gesture and to control in real-time the generation of an audio-visual feedback. The systems analyses in real-time the user’s full-body movement and gesture, extracts expressive motion features and maps the values of the expressive motion features onto real-time control of acoustic parameters for rendering a music performance. At the same time, a visual feedback generated in real-time is projected on a screen in front of the users with their coloured silhouette, depending on the emotion their movement communicates. Human movement analysis and visual feedback generation were done with the EyesWeb software platform and the music performance rendering with pDM. Evaluation tests were done with human participants to test the usability of the interface and the effectiveness of the design. (2007). Sound design for affective interaction. In Paiva, A., Prada, R., & Picard, R. W. (Eds.), Affective Computing and Intelligent Interaction (pp. 523-533). Berlin / Heidelberg: Springer. [abstract] [link]Abstract: Different design approaches contributed to what we see today as the prevalent design paradigm for Human Computer Interaction; though they have been mostly applied to the visual aspect of interaction. In this paper we presented a proposal for sound design strategies that can be used in applications involving affective interaction. For testing our approach we propose the sonification of the Affective Diary, a digital diary with focus on emotions, affects, and bodily experience of the user. We applied results from studies in music and emotion to sonic interaction design. This is one of the first attempts introducing different physics-based models for the real-time complete sonification of an interactive user interface in portable devices. (2007). A virtual head driven by music expressivity. IEEE Transactions on Audio, Speech and Language Processing, 15(6), 1833-1841. [abstract] [link]Abstract: In this paper, we present a system that visualizes the expressive quality of a music performance using a virtual head. We provide a mapping through several parameter spaces: on the input side, we have elaborated a mapping between values of acoustic cues and emotion as well as expressivity parameters; on the output side, we propose a mapping between these parameters and the behaviors of the virtual head. This mapping ensures a coherency between the acoustic source and the animation of the virtual head. After presenting some background information on behavior expressivity of humans, we introduce our model of expressivity. We explain how we have elaborated the mapping between the acoustic and the behavior cues. Then, we describe the implementation of a working system that controls the behavior of a human-like head that varies depending on the emotional and acoustic characteristics of the musical execution. Finally, we present the tests we conducted to validate our mapping between the emotive content of the music performance and the expressivity parameters. (2007). Emerging sounds for disappearing computers. In Streitz, N., Kameas, A., & Mavrommati, I. (Eds.), The Disappearing Computer (pp. 233-254). Berlin / Heidelberg: Springer. [abstract] [link]Abstract: Try this simple experiment one day: wear a couple of earplugs and try to conduct your regular everyday activities, for a couple of hours. How would you describe your feelings in that deafened state? You would probably feel a sense of isolation, from the world and from other people. So, absence of sound induces perceived isolation which may turn into felt oppression in some environments, such as an anechoic chamber. One may think that if silence induces isolation, sound induces presence, but unfortunately this is not the case. We know sensitive souls that have difficulties falling asleep because they live in noisy neighborhoods. One solution that may work in this case is to play loud noise through the hi-fi loudspeakers to mask the noise from the environment. Again, isolation (e.g. from street noise) is the result, but the means to achieve it is loud noise, the opposite of silence. And how would you describe all those people that experience modern city life being shielded by earphones that play music from their walkmans or mp3 players? They look rather isolated from each other, don’t they? In some circumstances there might be a need of concentration (e.g., in studying), or people want to tune their mood (Brodsky 2002). In all those cases sounds may be the appropriate mean, as it was well known even to Thomas Edison, who used to accompany commercialization of his phonograph with a “mood change chart” aimed at surveying users reactions to that new technology. So, it seems that sounds have the potential to modulate human engagement (from isolation to arousal) in everyday environments, and this is an aspect that should be seriously considered when designing the artefacts that will populate the environments of the future, likely to be pervaded by Ambient Intelligence (AmI) in its various facets. (2007). Sound and Music Computing: Challenges and Strategies. Journal of New Music Research, 36(3), 185-190. [abstract] [link]Abstract: Based on the current context of the Sound and Music Computing (SMC) field, the state of the art in research and the open issues that have been identified and described in other articles of this journal issue, in this article we make a step forward and try to identify the broad SMC challenges and we propose strategies with which to tackle them. On the research side we identify a clear need for designing better sound objects and environments and for promoting research to understand, model, and improve human interaction with sound and music. In the education domain we feel the need for better training our multidisciplinary researchers and to make sure that they can contribute to the multicultural society we live in. There is also a clear need for improving the transferring of the knowledge and technologies generated by our community. Finally we claim that the SMC field should be very much concerned with its social context and that a number of current social concerns should be addressed. We accompany each of these challenges with strategies that should help researchers, educators and policy makers take specific actions to advance in the proposed SMC roadmap.2006 (2006). Overview of the KTH rule system for musical performance. Advances in Cognitive Psychology, Special Issue on Music Performance, 2(2-3), 145-161. [abstract] [pdf]Abstract: The KTH rule system models performance principles used by musicians when performing a musical score, within the realm of Western classical, jazz and popular music. An overview is given of the major rules involving phrasing, micro-level timing, metrical patterns and grooves, articulation, tonal tension, intonation, ensemble timing, and performance noise. By using selections of rules and rule quantities, semantic descriptions such as emotional expressions can be modeled. A recent real-time implementation provides the means for controlling the expressive character of the music. The communicative purpose and meaning of the resulting performance variations are discussed as well as limitations and future improvements. (2006). Walking and playing: What's the origin of emotional expressiveness in music?. In Baroni, M., Addessi, A. R., Caterina, R., & Costa, M. (Eds.), ICMPC9 - 9th International Conference on Music Perception & Cognition (pp. 149). Bologna: Bonomia University Press (abstract). [pdf] (2006). Mapping strategies in DJ scratching. In Proc. of the Conference on New Interfaces for Musical Expression (pp. 188-191). Paris, France: IRCAM, Centre Pompidou. [abstract] [pdf]Abstract: For 30 years Disc Jockeys have been expressing their musical ideas with scratching. Unlike many other popular instruments, the equipment used for scratching is not built as one single unit, and it was not intended to be a musical instrument. This paper gives an overview of how DJs use their turntable, vinyl record and audio mixer in junction to produce scratch music. Their gestural input to the instrument is explained by looking at the mapping principles between the controller parameters and the audio output parameters. Implications are discussed for the design of new interfaces with examples of recent innovations and experiments in the field. (2006). Principles for expressing emotional content in turntable scratching. In Baroni, M., Addessi, A. R., Caterina, R., & Costa, M. (Eds.), Proc. 9th International Conference on Music Perception & Cognition (pp. 532-533). Bologna: Bonomia University Press. [abstract] [pdf]Abstract: Background: Scratching is a novel musical style that introduces the turntable as a musical instrument. Sounds are generated by moving vinyl records with one or two hands on the turntable and controlling amplitude with the crossfader with one hand. With this instrument mapping, complex gestural combinations that produce unique 'tones' can be achieved. These combinations have established a repertoire of playing techniques, and musicians (or DJs) know how to perform most of them. Scratching is normally not a melodically based style of music. It is very hard to produce tones with discrete and constant pitch. The sound is always strongly dependent on the source material on the record, and its timbre is not controllable in any ordinary way. However, tones can be made to sound different by varying the speed of the gesture and thereby creating pitch modulations. Consequently timing and rhythm remain as important candidates for expressive playing when compared to conventional musical instruments, and with the additional possibility to modulate the pitch. Aims: The experiment presented aims to identify acoustical features that carry emotional content in turntable scratching performances, and to find relationships with how music is expressed with other instruments. An overall aim is to investigate why scratching is growing in popularity even if it a priori seems ineffective as an expressive interface. Method: A number of performances by experienced DJs were recorded. Speed of the record, mixer amplitude and the generated sounds were measured. The analysis focuses on finding the underlying principles for expressive playing by examining musician's gestures and the musical performance. The found principles are compared to corresponding methods for expressing emotional intentions used for other instruments. Results: The data analysis is not completed yet. The results will give an indication of which acoustical features DJs use to play expressively on their instrument with musically limited possibilities. Preliminary results show that the principles for expressive playing are in accordance with current research on expression. Conclusions: The results present some important features in turntable scratching that may help explain why it remains a popular instrument despite its rather unsatisfactory playability both melodically and rhythmically. (2006). Rencon 2005. In Proceeding of the 20th Annual Conference of the Japanese Society for Artficial Intelligence (pp. 1D2-1). [abstract] [pdf]Abstract: Contest for performance rendering systems, Rencon, was held concurrently with the panel session entitled "Software Tools for Expressive Music Performance" in International Computer Music Conference (ICMC) 2005. In this paper, we describe the contest and the panel session. The contest consisted of the compulsory section where Mozart's Minuette KV 1 (1e) was the compulsory music. The contest winner was decided according to the voting prior to the panel session. Five panelists in the panel session introduced Rencon and research on expressive music performance. The panel session was exciting by the active discussion with the full audiences. (2006). Affective diary: designing for bodily expressiveness and self-reflection. In CHI '06 extended abstracts on Human factors in computing systems (pp. 1037-1042). New York, NY, USA: ACM. [abstract] [pdf]Abstract: A diary provides a useful means to express inner thoughts and record experiences of past events. In re-readings, it also provides a resource for reflection, allowing us to re-experience, brood over or even shed the thoughts and feelings we've associated with events or people. To expand on the ways in which we creatively engage in diary-keeping, we have designed an affective diary that captures some of the physical, bodily aspects of experiences and emotions--what we refer to as "affective body memorabilia". The affective diary assembles sensor data, captured from the user and uploaded via their mobile phone, to form an ambiguous, abstract colourful body shape. With a range of other materials from the mobile phone, such as text and MMS messages, photographs, etc., these shapes are made available to the user. Combining these materials, the diary is designed to invite reflection and to allow the user to piece together their own stories. (2006). Influence of expressive music on the perception of short text messages. In Baroni, M., Addessi, A. R., Caterina, R., & Costa, M. (Eds.), ICMPC9 -- 9th International Conference on Music Perception & Cognition (pp. 739). Bologna: Bonomia University Press (abstract). [pdf] (2006). From Acoustic Cues to an Expressive Agent. In Gibet, S., Courty, N., & Kamp, J-F. (Eds.), Gesture in Human-Computer Interaction and Simulation: 6th International Gesture Workshop, GW 2005, Berder Island, France, May 18-20, 2005, Revised Selected Papers (pp. 280-291). Berlin / Heidelberg: Springer. [abstract] [pdf]Abstract: This work proposes a new way for providing feedback to expressivity in music performance. Starting from studies on the expressivity of music performance we developed a system in which a visual feedback is given to the user using a graphical representation of a human face. The first part of the system, previously developed by researchers at KTH Stockholm and at the University of Uppsala, allows the real-time extraction and analysis of acoustic cues from the music performance. Cues extracted are: sound level, tempo, articulation, attack time, and spectrum energy. From these cues the system provides an high level interpretation of the emotional intention of the performer which will be classified into one basic emotion, such as happiness, sadness, or anger. We have implemented an interface between that system and the embodied conversational agent Greta, developed at the University of Rome “La Sapienza” and “University of Paris 8”. We model expressivity of the facial animation of the agent with a set of six dimensions that characterize the manner of behavior execution. In this paper we will first describe a mapping between the acoustic cues and the expressivity dimensions of the face. Then we will show how to determine the facial expression corresponding to the emotional intention resulting from the acoustic analysis, using music sound level and tempo characteristics to control the intensity and the temporal variation of muscular activation. (2006). Automatic characterization of ornamentation from bassoon recordings for expressive synthesis. In Baroni, M., Addessi, A. R., Caterina, R., & Costa, M. (Eds.), ICMPC9 - 9th International Conference on Music Perception & Cognition (pp. 1533-1538). Bologna: Bonomia University Press. [abstract] [pdf]Abstract: Expressive performance characterization is traditionally based on the analysis of the main differences between performances, players, playing styles and emotional intentions. This work addresses the characterization of expressive bassoon ornaments by analyzing audio recordings played by a professional bassoonist. This characterization is then used to generate expressive ornaments from symbolic representations by means of Machine Learning2005 (2005). What is the color of that music performance?. In Proceedings of the International Computer Music Conference - ICMC 2005 (pp. 367-370). Barcelona. [abstract] [pdf]Abstract: The representation of expressivity in music is still a fairly unexplored field. Alternative ways of representing musical information are necessary when providing feedback on emotion expression in music such as in real-time tools for music education, or in the display of large music databases. One possible solution could be a graphical non-verbal representation of expressivity in music performance using color as index of emotion. To determine which colors are most suitable for an emotional expression, a test was run. Subjects rated how well each of 8 colors and their 3 nuances corresponds to each of 12 music performances expressing different emotions. Performances were played by professional musicians with 3 instruments, saxophone, guitar, and piano. Results show that subjects associated different hues to different emotions. Also, dark colors were associated to music in minor tonality and light colors to music in major tonality. Correspondence between spectrum energy and color hue are preliminary discussed. (2005). Touch and temporal behavior of grand piano actions. Journal of the Acoustical Society of America, 118(2), 1154–1165. [abstract] [pdf]Abstract: This study investigated the temporal behavior of grand piano actions from different manufacturers under different touch conditions and dynamic levels. An experimental setup consisting of accelerometers and a calibrated microphone was used to capture key and hammer movements, as well as the sound signal. Five selected keys were played by pianists with two types of touch (“pressed touch” versus “struck touch”) over the entire dynamic range. Discrete measurements were extracted from the accelerometer data for each of the over 2300 recorded tones (e.g., finger-key, hammer-string, and key bottom contact times, maximum hammer velocity). Travel times of the hammer (from finger-key to hammer-string) as a function of maximum hammer velocity varied clearly between the two types of touch, but only slightly between pianos. A travel time approximation used in earlier work [Goebl W., (2001). J. Acoust. Soc. Am. 110, 563–572] derived from a computer-controlled piano was verified. Constant temporal behavior over type of touch and low compression properties of the parts of the action (reflected in key bottom contact times) were hypothesized to be indicators for instrumental quality. (2005). A dimensional approach to vocal expression of emotion. Cognition and Emotion, 19(5), 633-653. [abstract] [link]Abstract: This study explored a dimensional approach to vocal expression of emotion. Actors vocally portrayed emotions (anger, disgust, fear, happiness, sadness) with weak and strong emotion intensity. Listeners (30 university students and 6 speech experts) rated each portrayal on four emotion dimensions (activation, valence, potency, emotion intensity). The portrayals were also acoustically analysed with respect to 20 vocal cues (e.g., speech rate, voice intensity, fundamental frequency, spectral energy distribution). The results showed that: (a) there were distinct patterns of ratings of activation, valence, and potency for the different emotions; (b) all four emotion dimensions were correlated with several vocal cues; (c) listeners' ratings could be successfully predicted from the vocal cues for all dimensions except valence; and (d) the intensity dimension was positively correlated with the activation dimension in the listeners' ratings. (2005). Greta Listening to Expressive Music. In Ruttkay, Z. (Ed.), Gathering of Animated Lifelike Agents - GALA 2005. [abstract] [link]Abstract: The synthetic head Greta reacts with emotional facial expressions to music, based on studies on the expressivity of music performance. (2005). Book Review: Freedom and Constraints in Timing and Ornamentation: Investigations of Music Performance. Psychology of Music, 33(1), 122-128. [pdf]2004 (2004). Real-time visualization of musical expression. In Proceedings of Network of Excellence HUMAINE Workshop "From Signals to Signs of Emotion and Vice Versa" (pp. 19-23). Santorini, Greece, Institute of Communication and Computer Systems, National Technical University of Athens. [abstract] [pdf]Abstract: A system for real-time feedback of expressive music performance is presented. The feedback is provided by using a graphical interface where acoustic cues are presented in an intuitive fashion. The graphical interface presents on the computer screen a three-dimensional object with continuously changing shape, size, position, and colour. Some of the acoustic cues were associated with the shape of the object, others with its position. For instance, articulation was associated with shape, staccato corresponded to an angular shape and legato to a rounded shape. The emotional expression resulting from the combination of cues was mapped in terms of the colour of the object (e.g., sadness/blue). To determine which colours were most suitable for respective emotion, a test was run. Subjects rated how well each of 8 colours corresponds to each of 12 music performances expressing different emotions. (2004). Once again: The perception of piano touch and tone. Can touch audibly change piano sound independently of intensity?. In In Proceedings of the International Symposium on Musical Acoustics - ISMA'04 (pp. 332–335). Nara, Japan: The Acoustical Society of Japan, CD-ROM. [abstract] [pdf]Abstract: This study addresses the old question of whether the timbre of isolated piano tones can be audibly varied independently of their hammer velocities—only through the type of touch. A large amount of single piano tones were played with two prototypical types of touch: depressing the keys with the finger initially resting on the key surface (pressed), and hitting the keys from a certain distance above (struck). Musicians were asked to identify the type of touch of the recorded samples, in a first block with all attack noises before the tone onsets included, in a second block without them. Half of the listeners could correctly identify significantly more tones than chance in the first block (up to 86% accuracy), but no one in block 2. Those who heard no difference tended to give struck ratings for louder tones in both blocks. (2004). Analysis of a genuine scratch performance. In Camurri, A., & Volpe, G. (Eds.), Gesture-Based Communication in Human-Computer Interaction, 5th International Gesture Workshop, GW 2003 (pp. 519-528). Genova, Italy: Springer Verlag. [abstract] [pdf]Abstract: The art form of manipulating vinyl records done by disc jockeys (DJs) is called scratching, and has become very popular since its start in the seventies. Since then turntables are commonly used as ex- pressive musical instruments in several musical genres. This phenomenon has had a serious impact on the instrument-making industry, as the sales of turntables and related equipment have boosted. Despite of this, the acoustics of scratching has been barely studied until now. In this paper, we illustrate the complexity of scratching by measuring the gestures of one DJ during a performance. The analysis of these measurements is important to consider in the design of a scratch model. (2004). Rencon 2004: Turing Test for Musical Expression. In Lyons, M. J. (Ed.), Proceedings of the 4th international conference on New interfaces for musical expression (pp. 120-123). Hamamatsu, Shizuoka, Japan: National University of Singapore. [abstract] [pdf]Abstract: Rencon is an annual international event that started in 2002. It has roles of (1) pursuing evaluation methods for systems whose output includes subjective issues, and (2) providing a forum for researches of several fields related to musical expression. In the past, Rencon was held as a workshop associated with a musical contest that provided a forum for presenting and discussing the latest research in automatic performance rendering. This year we introduce new evaluation methods of performance expression to Rencon: a Turing Test and a Gnirut Test, which is a reverse Turing Test, for performance expression. We have opened a section of the contests to any instruments and genre of music, including synthesized human voices. (2004). Contact sounds for continuous feedback. In Hunt, A., & Hermann, T. (Eds.), Proceedings of International Workshop on Interactive Sonification (Human Interaction with Auditory Displays). Bielefeld University, Germany. [abstract] [pdf]Abstract: The role of continuous auditory feedback in multimodal embodied interfaces is advocated. Examples of physics based cartoon sound models (rolling and friction) are used to display deviation from equilibrium and exerted effort in manipulative interfaces.2003 (2003). Experiments on gestures: walking, running, and hitting. In Rocchesso, D., & Fontana, F. (Eds.), The Sounding Object (pp. 111-136). Florence, Italy: Mondo Estremo. [pdf] (2003). Controlling the virtual bodhran - the vodhran. In Bresin, R. (Ed.), Proceedings of SMAC 2003, Stockholm Music Acoustics Conference (pp. 685-688). [pdf] (2003). The Radio Baton as configurable musical instrument and controller. In Bresin, R. (Ed.), Proceedings of SMAC 2003, Stockholm Music Acoustics Conference (pp. 689-691). [abstract] [pdf]Abstract: The Max Mathews’ Radio Baton (RB) has usually been applied as an orchestra conducting system, as interactive music composition controller using typical percussionist gestures, and as a controller for sound synthesis models. In the framework of the Sounding Object EU founded project, the RB has found new applications scenarios. Three applications were based on this controller. This was achieved by changing the gesture controls. Instead of the de- fault batons, a new radio transmitter based on a thimble that fits the fingertips was developed. This new transmitter allows musicians’ interaction based on hand gestures and it can also fit different devices. The RB and the thimble-based transmitter are used for the control of apd model implementing DJ scratching techniques. This new configuration of the RB allows DJs a direct control of sampled sounds maintaining hand gestures similar to those used on vinyl. In another application the sound model of a bodhran was controlled with a traditional playing approach. In this case the RB can be controlled with a traditional bodhran double beater with one thimble radio transmitter fitted at each end. This allows detection of the beater position on the RB surface, the surface corresponding to the membrane of the bodhran in the sound model. In a third application the thimble controller was used to move a virtual ball rolling along the elastic surface of a box placed over the receiving antennae of the RB. (2003). Devices for manipulation and control of sounding objects: The Vodhran and the Invisiball. In Rocchesso, D., & Fontana, F. (Eds.), The Sounding Object (pp. 271-295). Florence, Italy: Mondo Estremo. [pdf] (2003). Real-time visualization of musical expression. In Proceedings of ESCOM 5. Hanover, Germany (abstract). (2003). Physics-based sound synthesis and control: crushing, walking and running by crumpling sounds. In XIV Colloquium on Musical Informatics, XIV CIM 2003 (pp. 109--114). Florence, Italy. [pdf] (2003). Measurement and reproduction accuracy of computer-controlled grand pianos. Journal of the Acoustical Society of America, 114(4), 2273-2283. [pdf] (2003). Measurement and reproduction accuracy of computer-controlled grand pianos. In Bresin, R. (Ed.), Proceedings of SMAC 03, Stockholm Music Acoustics Conference (pp. 155-158). [pdf] (2003). The piano action as the performer's interface: Timing properties, dynamic behaviour and the performer's possibilities. In Bresin, R. (Ed.), Proceedings of SMAC 03, Stockholm Music Acoustics Conference (pp. 159-162). [pdf] (2003). Complex Gestural Audio Control: The Case of Scratching. In Rocchesso, D., & Fontana, F. (Eds.), The Sounding Object (pp. 221-269). Florence, Italy: Mondo Estremo. [pdf] (2003). DJ scratching performance techniques: Analysis and synthesis. In Bresin, R. (Ed.), Proc. Stockholm Music Acoustics Conference (pp. 693-696). Stockholm. [abstract] [pdf]Abstract: Scratching is a popular way of making music, turning the DJ into a musician. Normally scratching is done using a vinyl record, a turntable and a mixer. Vinyl manipulation is built up by a number of specialized techniques that have been analysed in a previous study. The present study has two main objectives. First is to better understand and model turntable scratching as performed by DJs. Second is to design a gesture controller for physical sound models, i.e. models of friction sounds. We attached sensors to a DJ equipment set-up. Then a DJ was asked to perform typical scratch gestures both isolated and in a musical context, i.e. as in a real performance. He also was asked to play with different emotions: sad, angry, happy and fearful. A model of the techniques used by the DJ was built based on the analysis of the collected data. The implementation of the model has been done in pd. The Radio Baton, with specially adapted gesture controllers, has been used for controlling the model. The system has been played by professional DJs in concerts. (2003). After the first year of RENCON. In Proceedings of the International Computer Music Conference - ICMC 2003 (pp. 135-138). San Francisco: ICMA. [pdf] (2003). Rencon in 2002. In Proceedings of IJCAI-03 Rencon Workshop (pp. 59–64). Acapulco, MEXICO. [pdf] (2003). Expressivity comes from within your soul: A questionnaire study of music students' perspectives on expressivity. Research Studies in Music Education, 20(1), 23-47. [abstract] [pdf]Abstract: Much has been written about expressivity by philosophers, composers, musicologists, and psychologists, but little is known about how the musicians of tomorrow — music students — approach this subject. This paper reports an exploratory study in which 135 students from music conservatories in three countries (England, Italy, Sweden) filled out a questionnaire that addressed four themes: (a) conceptualizing expressivity, (b) expressivity in everyday practice, (c) expressivity in music teaching, and (d) novel teaching strategies. The results suggest that students define expressivity mainly in terms of communicating emotions and `playing with feeling'. Expressive skills are regarded as highly important by students, and they would like to practice more on expressivity than is currently the case. However, most students are skeptical toward using computers in teaching of expressivity since they cannot see how such applications could work. The results suggest that expressivity deserves more attention in music education than has hitherto been the case. (2003). Sounding object. IEEE Multimedia Magazine, 10(2), 42-52. [abstract] [link]Abstract: Interactive systems, virtual environments, and information display applications need dynamic sound models rather than faithful audio reproductions. This implies three levels of research: auditory perception, physics-based sound modeling, and expressive parametric control. Parallel progress along these three lines leads to effective auditory displays that can complement or substitute visual displays. This article aims to shed some light on how psychologists, computer scientists, acousticians, and engineers can work together and address these and other questions arising in sound design for interactive multimedia systems. (2003). Attempts to reproduce a pianist's expressive timing with Director Musices performance rules. Journal of New Music Research, 32(3), 317-325. [abstract] [link]Abstract: The Director Musices generative grammar of music performance is a system of context dependent rules that automatically introduces expressive deviation in performances of input score files. A number of these rule concern timing. In this investigation the ability of such rules to reproduce a professional pianist's timing deviations from nominal note inter-onset-intervals is examined. Rules affecting tone inter-onset-intervals were first tested one by one for the various sections of the excerpt, and then in combinations. Results were evaluated in terms of the correlation between the deviations made by the pianist and by the rule system. It is found that rules reflecting the phrase structure produced high correlations in some sections. On the other hand, some rules failed to produce significant correlation with the pianist's deviations, and thus seemed irrelevant to the particular performance analysed. It is concluded that phrasing was a prominent principle in this performance and that rule combinations have to change between sections in order to match this pianist's deviations. (2003). Musician's and computer's tone inter-onset-interval in Mozart's Piano Sonata K 332, 2nd mvt, bar 1-20. TMH-QPSR, 45(1), 047-059. [pdf]2002 (2002). Color feedback in expressive music performance. In Proceedings of ICMPC7 - 7th International Conference on Music Perception & Cognition (pp. 14). Sydney (abstract). (2002). Importance of note-level control in automatic music performance. In Proceedings of RENCON 2002 - performance RENdering piano CONtest (pp. 1-6). Kyoto. [pdf] (2002). Displaying the expressive parameters of music performance. In Proceedings of ICMPC7 - 7th International Conference on Music Perception & Cognition (pp. 6). Sydney (abstract). (2002). Director musices: The KTH performance rules system. In Proceedings of SIGMUS-46 (pp. 43-48). Kyoto: Information Processing Society of Japan,. [pdf] (2002). Automatic real-time extraction of musical expression. In Proceedings of the International Computer Music Conference, ICMC 2002 (pp. 365-367). Göteborg. [pdf] (2002). Computational modelling of different aspects of expressivity: The GERM model. In Proceedings of ICMPC7 - 7th International Conference on Music Perception & Cognition (pp. 13). Sydney (abstract). (2002). Toward a computational model of expression in performance: The GERM model. Musicae Scientiae, Special issue 2001-2002, 63-122. [abstract] [pdf]Abstract: Presents a computational model of expression in music performance: the GERM model. The purpose of the GERM model is to (1) describe the principal sources of variability in music performance, (2) emphasize the need to integrate different aspects of performance in a common model, and (3) provide some preliminaries for a computational model that simulates the different aspects. Drawing on previous research on performance, the authors propose that performance expression derives from 4 main sources of variability: (1) Generative Rules, which function to convey the generative structure in a musical manner; (2) Emotional Expression, which is governed by the performer's expressive intention; (3) Random Variations, which reflect internal timekeeper variance and motor delay variance; and (4) Movement Principles, which prescribe that certain features of the performance are shaped in accordance with biological motion. A preliminary version of the GERM model was implemented by means of computer synthesis. Synthesized performances were evaluated by musically trained participants in a listening test. The results from the test support a decomposition of expression in terms of the GERM model. Implications for future research on music performance are discussed. (2002). The teacher's perspective on expression in music performance: Preliminary evidence from music conservatories. In Proceedings of ICMPC7 - 7th International Conference on Music Perception & Cognition (pp. 13). Sydney (abstract). (2002). Students' views on expressivity in music performance: A questionnaire study. In Proceedings of ICMPC7 - 7th International Conference on Music Perception & Cognition (pp. 12-13). Sydney (abstract). (2002). The Sounding Object. IEEE Computer Graphics and Applications, 22(4), Flash presentation included in the premier CG&A CD-ROM. (2002). A system for improving the communication of emotion in music performance by feedback learning, (abstract). Journal of the Acoustical Society of America, 111(5), 2471.2001 (2001). Articulation rules for automatic music performance. In Schloss, A., Dannenberg, R., & Driessen, P. (Eds.), Proceedings of the International Computer Music Conference - ICMC 2001 (pp. 294-297). San Francisco: ICMA. [pdf] (2001). Expressive musical icons. In Hiipakka, J., Zakarov, N., & Takala, T. (Eds.), Proceedings of the International Conference on Auditory Display - ICAD 2001 (pp. 141-143). Espoo, Finland. [pdf] (2001). Toward a new model for sound control. In Fernström, M., Brazil, E., & Marshall, M. (Eds.), Proc DAFx01 (pp. 45-49). Limerick, Ireland. [pdf] (2001). Is the player more influenced by the auditory than the tactile feedback from the instrument?. In Fernström, M., Brazil, E., & Marshall, M. (Eds.), Proceedings of DAFx01 (pp. 194-197). Limerick, Ireland. [pdf] (2001). Are computer-controlled pianos a reliable tool in music performance research? Recording and reproduction precision of a Yamaha Disklavier grand piano. In MOSART workshop on current research directions in computer music (pp. 45-50). Barcelona. [pdf]2000 (2000). Virtual Virtuosity. Studies in Automatic Music Performance. Doctoral dissertation. [pdf] (2000). Articulation strategies in expressive piano performance. Analysis of legato, staccato, and repeated notes in performances of the Andante movement of Mozart's sonata in G major (K 545). Journal of New Music Research, 29(3), 211-224. [abstract] [link]Abstract: Articulation strategies applied by pianists in expressive performances of the same score are analysed. Measurements of key overlap time and its relation to the inter-onset-interval are collected for notes marked legato and staccato in the first sixteen bars of the Andante movement of W. A. Mozart's Piano Sonata in G major, K 545. Five pianists played the piece nine times. First, they played in a way that they considered (optimal). In the remaining eight performances they were asked to represent different expressive characters, as specified in terms of different adjectives. Legato , staccato , and repeated notes articulation applied by the right hand were examined by means of statistical analysis. Although the results varied considerably between pianists, some trends could be observed. The pianists generally used similar strategies in the renderings intended to represent different expressive characters. Legato was played with a key overlap ratio that depended on the inter-onset-interval (IOI). Staccato tones had approximate duration of 40% of the IOI. Repeated notes were played with a duration of about 60% of the IOI. The results seem useful as a basis for articulation rules in grammars for automatic piano performance. (2000). Legato, staccato, and repeated tones in expressive piano performance. TMH-QPSR, 41(1), 61-72. [pdf] (2000). Emotional coloring of computer controlled music performance. Computer Music J, 24(4), 44-63. [pdf] (2000). Rule-based emotional colouring of music performance. In Zannos, I. (Ed.), Proceedings of the International Computer Music Conference - ICMC 2000 (pp. 364-367). San Francisco: ICMA. (2000). Software tools for musical expression. In Zannos, I. (Ed.), Proceedings of the International Computer Music Conference - ICMC 2000 (pp. 499-502). San Francisco: ICMA. (2000). Production of staccato articulation in Mozart sonatas played on a grand piano. Preliminary results. TMH-QPSR, 41(4), 1-6. [pdf]1999 (1999). An artificial neural network model for analysis and synthesis of pianists' performance styles (abstract only). Journal of the Acoustical Society of America, 105(3), 1056. (1999). Japer and Pann: Two java-applets for music performance included in the cd-rom ''Midishare: Operating system for musical applications''. National Center of Contemporary Music - GRAME. (1999). Synthesis and decoding of emotionally expressive music performance. In Proceedings of the IEEE 1999 Systems, Man and Cybernetics Conference - SMC'99 (pp. 317–322). Tokyo. [abstract] [link]Abstract: A recently developed application of Director Musices (DM) is presented. The DM is a rule-based software tool for automatic music performance developed at the Speech Music and Hearing Dept. at the Royal Institute of Technology, Stockholm. It is written in Common Lisp and is available both for Windows and Macintosh. It is demonstrated that particular combinations of rules defined in the DM can be used for synthesizing performances that differ in emotional quality. Different performances of two pieces of music were synthesized so as to elicit listeners' associations to six different emotions (fear, anger, happiness, sadness, tenderness, and solemnity). Performance rules and their parameters were selected so as to match previous findings about emotional aspects of music performance. Variations of the performance variables IOI (Inter-Onset Interval), OOI (Offset-Onset Interval) and L (Sound Level) are presented for each rule-setup. In a forced-choice listening test 20 listeners were asked to classify the performances with respect to emotions. The results showed that the listeners, with very few exceptions, recognized the intended emotions correctly. This shows that a proper selection of rules and rule parameters in DM can indeed produce a wide variety of meaningful, emotional performances, even extending the scope of the original rule definition1998 (1998). Artificial neural networks based models for automatic performance of musical scores. Journal of New Music Research, 27(3), 239-270. [abstract] [link]Abstract: This article briefly summarises the author's research on automatic performance, started at CSC (Centro di Sonologia Computazionale, University of Padua) and continued at TMH-KTH (Speech, Music Hearing Department at the Royal Institute of Technology, Stockholm). The focus is on the evolution of the architecture of an artificial neural networks (ANNs) framework, from the first simple model, able to learn the KTH performance rules, to the final one, that accurately simulates the style of a real pianist performer, including time and loudness deviations. The task was to analyse and synthesise the performance process of a professional pianist, playing on a Disklavier. An automatic analysis extracts all performance parameters of the pianist, starting from the KTH rule system. The system possesses good generalisation properties: applying the same ANN, it is possible to perform different scores in the performing style used for the training of the networks. Brief descriptions of the program Melodia and of the two Java applets Japer and Jalisper are given in the Appendix. In Melodia, developed at the CSC, the user can run either rules or ANNs, and study their different effects. Japer and Jalisper, developed at TMH, implement in real time on the web the performance rules developed at TMH plus new features achieved by using ANNs. (1998). Emotional expression in music performance: synthesis and decoding. TMH-QPSR, 39(4), 085-094. [html] (1998). Musical punctuation on the microlevel: Automatic identification and performance of small melodic units. Journal of New Music Research, 27(3), 271-292. [abstract] [pdf]Abstract: In this investigation we use the term musical punctuation for the marking of melodic structure by commas inserted at the boundaries that separate small structural units. Two models are presented that automatically try to locate the positions of such commas. They both use the score as the input and operate with a short context of maximally five notes. The first model is based on a set of subrules. One group of subrules mark possible comma positions, each provided with a weight value. Another group alters or removes these weight values according to different conditions. The second model is an artificial neural network using a similar input as that used by the rule system. The commas proposed by either model are realized in terms of micropauses and of small lengthenings of interonset durations. The models are evaluated by using a set of 52 musical excerpts, which were marked with punctuations according to the preference of an expert performer. * Sound examples are available in the JNMR Electronic Appendix (EA), which can be found on the WWW at http://www.swets.nl/jnmr/jnmr.html1997 (1997). A multimedia environment for interactive music performance. In Camurri, A. (Ed.), Proceedings of KANSEI - The Technology of Emotion, AIMI International Workshop (pp. 64-67). Genoa, Italy. (1997). A multimedia environment for interactive music performance. TMH-QPSR, 38(2-3), 029-032. [pdf] (1997). Automatic musical punctuation: A rule system and a neural network approach. In Camurri, A. (Ed.), Proceedings of KANSEI - The Technology of Emotion, AIMI Intl Workshop (pp. 159-163). Genoa, Italy.1995 (1995). A Neural Networks Based System for Automatic Performance of Musical Scores. In La Motte-Haber, H. d., & Kopiez, R. (Eds.), Der Hörer als Interpret (pp. 99-112). Frankfurt am Main: Peter Lang GmbH. (1995). Real-time musical rhythm tapping. In Finarelli, L., & Regazzi, F. (Eds.), Colloquio di Informatica Musicale - XI CIM (pp. 185-188). Bologna: AIMI Associazione di Informatica Musicale Italiana. [pdf] (1995). A fuzzy formulation to KTH performance rules system. In Proceedings of the 2nd International Conference on Acoustic and Musical Research - CIARM 95 (pp. 433–438). Ferrara. (1995). Fuzzy performance rules. In Friberg, A., & Sundberg, J. (Eds.), Proceedings of the KTH Symposium on Grammars for music performance (pp. 15–36). Stockholm: Department of Speech Communication and Music Acoustics, KTH. (1995). Neural networks play Schumann. Analysis and synthesis of the performing action of a real pianist. In Friberg, A., & Sundberg, J. (Eds.), Proceedings of the KTH Symposium on Grammars for music performance (pp. 5–14). Stockholm: Department of Speech Communication and Music Acoustics, KTH. (1995). A fuzzy approach to performance rules. In Finarelli, L., & Regazzi, F. (Eds.), Colloquio di Informatica Musicale - XI CIM (pp. 163-166). Bologna: AIMI Associazione di Informatica Musicale Italiana. [pdf] (1995). A gesture interface controlled by the vocal tract. In Finarelli, L., & Regazzi, F. (Eds.), Colloquio di Informatica Musicale - XI CIM (pp. 159-162). Bologna: AIMI Associazione di Informatica Musicale Italiana. [pdf]1994 (1994). Analysis by synthesis in piano performance: A study on the theme of the Brahms' ''Variations on a theme of Paganini'' op. 35. In Friberg, A., Iwarsson, J., Jansson, E., & Sundberg, J. (Eds.), Proceedings of the Stockholm Music Acoustic Conference - SMAC 1993 (pp. 69–73). Stockholm. (1994). Neural networks vs. Rules system: Evaluation tests of automatic performance of musical scores. In Proceedings of the International Computer Music Conference - ICMC 1994 (pp. 109–113). San Francisco: ICMA. (1994). Performance of musical scores by means of neural networks. Progress and status report. In Sundberg, J. (Ed.), Proceedings of the Symposium on Generative Grammars for Musical Performance - Aarhus (pp. 3–6). Stockholm: Department of Speech Communication and Music Acoustics, KTH. (1994). A neural networks based system for automatic performance of musical scores. In Friberg, A., Iwarsson, J., Jansson, E., & Sundberg, J. (Eds.), Proceedings of the Stockholm Music Acoustic Conference - SMAC 1993 (pp. 74–78). Stockholm. (1994). Analysis and synthesis of the performing action of a real pianist by means of artificial neural networks. In Proceedings of ICMPC3 - 3rd International Conference on Music Perception & Cognition (pp. 353–354). Liege. (1994). Neural networks for musical tones compression, control, and synthesis. In Proceedings of the International Computer Music Conference - ICMC 1994 (pp. 368–371). Aarhus. (1994). Neural networks for the compression of musical tones and for the control of their resynthesis. In Proceedings of the IEEE-SP International Symposium on Time-Frequency and Time-Scale Analysis (pp. 628–631). Philadelphia. [abstract] [link]Abstract: Presents a hybrid system: a neural network for data compression and a generic algorithm for sound synthesis. The goal is the reduction of the number of parameters associated with a sound. Because of the growth of electronic technology, actual synthesisers are very complex systems in which the user cannot move without difficulty, and expensive loss of time. The basic ideas are reduction of the parameters made by the neural network (NN), and the freedom to choose any algorithm for sound synthesis. For example, using a machine based on FM synthesis, one can change a subset of the proper FM parameters, while in a sample-based keyboard one can change the gain value, some envelope parameters, etc. Such a system is very easy to use, flexible, and requires only a little more hardware for its realisation in an electronic synthesiser. The major work is for the developer, but this is made only once. The user just moves some controllers, i.e. some sliders; the neural network computes the output which is the set of the parameters for the synthesiser1993 (1993). Automatic performance of musical scores by means of neural networks: Evaluation with listening tests. In Haus, G., & Pighi, I. (Eds.), Colloquio di Informatica Musicale - X CIM (pp. 97-101). Milano: AIMI Associazione di Informatica Musicale Italiana. [pdf] (1993). Melodia: A program for performance rules testing, teaching, and piano score performance. In Haus, G., & Pighi, I. (Eds.), Colloquio di Informatica Musicale - X CIM (pp. 325-327). Milano: AIMI Associazione di Informatica Musicale Italiana. [pdf]1992 (1992). A connectionist approach to timing deviation control in musical performance. In Dalmonte, R., & Baroni, M. (Eds.), Second European Conference of Musical Analysis, 1991 (pp. 635-638). Trento, Italy. (1992). Symbolic and sub-symbolic rules system for real time score performance. In Proceedings of the International Computer Music Conference - ICMC 1992 (pp. 211–214). San Francisco: ICMA.1991 (1991). Applicazione delle reti neurali alla classificazione dei registri dell'organo a canne. In Camurri, A., & Canepa, C. (Eds.), Colloquio di Informatica Musicale - IX CIM (pp. 112-114). Genova: AIMI Associazione di Informatica Musicale Italiana. [pdf] (1991). Reti neurali per il controllo di deviazioni temporali nell'esecuzione musicale. In Camurri, A., & Canepa, C. (Eds.), Colloquio di Informatica Musicale - IX CIM (pp. 88-102). Genova: AIMI Associazione di Informatica Musicale Italiana. [pdf]1989 (1989). Una sorgente di melodie con controllo di entropia. In Colloquio di Informatica Musicale - VIII CIM (pp. 213-215). Cagliari: AIMI Associazione di Informatica Musicale Italiana. [pdf] |
|
KTH - Royal Institute
of Technology |
phone: +46 (8) 790 78 76 |
|
|
Updated 2010.09.04 by Roberto Bresin This is a personal homepage. More information here. |
||