A multimedia environment for interactive music performance

Roberto Bresin and Anders Friberg
Royal Institute of Technology
Speech, Music and Hearing
S-100 44 Stockholm, Sweden
e.mail: roberto@speech.kth.se, andersf@speech.kth.se


We propose a music performance tool based on the Java programming language. This software runs in any Java applet viewer (i.e. a WWW browser) and interacts with the local Midi equipment by mean of a multi-task software module for Midi applications (MidiShare).
Two main ideas are at the base of our project: one is to realise an easy, intuitive, hardware and software independent tool for performance, and the other is to achieve an easier development of the tool itself.
At the moment there are two projects under development: a system based only on a Java applet, called Japer (Java performer), and a hybrid system based on a Java user interface and a Lisp kernel for the development of the performance tools. In this paper the first of the two projects is presented.


After many years of research in computer generated performance, there is the need to collect all the most promising results in a hardware/software environment that allows musicians (both professional and not) to use and test them.
Such an environment would be useful for studying music performance, and to perform music otherwise impossible to be performed in a different way. In the first case, music students could take advantage of such a system: they can use it as a tool to better understand the mechanisms of music performance, and/or to build their own new performance rules. In the second case, if we look at computer-music composers, who want to perform their compositions, they would benefit from our system since it let them concentrate more on composition then on performance, and also a new role for the interactive computer-music performer could arise.

Software systems developed at KTH-TMH, Stockholm and at CSC, Padova University

In the last years two different software packages were developed separately at the Speech, Music and Hearing Department of KTH, Stockholm (TMH-KTH), and at Centro di Sonologia Computazionale (CSC), Padova University.

The first one, Director Musices, is a stand-alone program written in Common Lisp for Macintosh computers (a porting to the Allegro Common Lisp environment for Windows 95 is under development). Director Musices contains the basic set of rules described in Friberg (1991; 1995a) as well as the more recent rules Punctuation (Friberg et al., 1997) and Phrase-arch (Friberg,1995b). It is also used for the development of rules; new rules can easily be added using the rule definition utilities. It features Midi/Midi file input/output and parameters, such as duration and sound level, represented in terms of physical units.

The software developed at CSC is called Melodia: it runs under Windows, and allows the user to perform files of different formats (Midi, Csound, Adagio, Melodia). This software can perform music in two different ways: using a rule system based on that developed at TMH-KTH, with some added feature and differences (Battel & Bresin 1993), or using a Neural Network (NN) based system, that uses already trained NNs (Battel & al. 1994, Bresin et al. 1992, 1993, 1995).

Both programs are continuously under development and modification.

A machine independent environment

The problem of portability of software and of its continuos update lend us to think to re-design our systems, so to separate the research on performance from the user interface design and evolution. The basic idea is to have a Lisp kernel for the performance tools (rules, NNs, and other) communicating with the graphic user interface (GUI). The latter is written in Java language and it takes advantage of the Midishare operating system developed at GRAME (Orlarey and Lequay, 1989; Fober 1994; Orlarey 1994).

We chose Java mainly for the following reasons:

Midishare allows a Java applet to exchange real-time Midi messages with any Midi device attached to the client machine.
The system can run both on a computer network or a stand alone machine. The Java program will send/receive parameters to/from the Lisp program: in this way it is possible to have different developments both for the Lisp and the Java code. In particular we expect users of the system to suggest us both new improvements for the GUI, and further adjustments of the performance rules and NNs. In this way the whole system is easy to upgrade and develop.

Japer: Java Performer

In this paper we present JAPER (Java Performer): it is a complete Java version of the "merge" between Director Musices and Melodia. The reason of such a choice is to distribute a complete version of the performance program also to people who do not have a Lisp environment and want to use the software under different operating systems.

Figure 1 shows a general scheme of the global system: in the present work "Host" stands both for a WWW server or for a local host (it is the place where the applets are hosted); the box "Java applets" is represented by Japer; "MidiShare" is the real time multi-tasks software module for Midi applications, it acts as a Midi interface and communicates with other compatible applications as well with any Midi devices of the local host.

JAPER flow chart

In the following lines a description of the system running is given.

The user can run Japer in a WWW page or in an applet viewer, and do the following actions:

  1. Load a score. The input score can be written in different formats (Midi, Csound, Melodia) and can be loaded both from a server on the Internet or from the user's computer. The Melodia format provides a meta score in which information related to performance actions and score description are inserted (i.e. phrase limits, bar boundaries, tonality changes).
  2. Set the performance parameters/weights (cf. the k values in Friberg 1991). Each performance command (rule, NN, or other) has one or more sliders to set the values of some parameters in order to achieve the desired performance.
  3. Process the score. The score is performed according to the settings at point 2.
  4. Play the score (both in its dead pan version and in its precessed version). The user can listen to the performance using any Midi device connected at her/his machine thanks to the MidiShare system.
  5. Save the performance and/or interact with other MidiShare applications.
  6. Go back to point 1, 2 or 3.


The system has an important application in the pedagogical field, since can be used as a mean to study the different behaviours of performance rules and performance actions.
JAPER will be part of the "Garden of Knowledge", a project under development at CID-NADA, KTH Stockholm. The Garden of Knowledge is a knowledge tool developed in order to experience, explore and experiment with connections between mathematics, music and art.
Furthermore a Java applet implementing performance capabilities is useful to perform scores stored in music databases (i.e. in Internet it is possible to find very large music files databases like the Classical Midi Archive, http://www.prs.net/Midi.html, which contains over 3600 classical music files in Midi format). A performance tool would be very useful since most of the files in these databases are not performed (otherwise they would not be printable in a useful form using a commercial score editor, and it would also take a lot of time to obtain a good performance adding nuances by hand!), or are poorly performed.


We proposed a music performance tool based on the Java programming language, running within a Web page and interacting with the local Midi equipment by mean of a multi-task software module for Midi applications (MidiShare).
The system is hardware and software independent and provides an intuitive tool for music performance.
The modularity of the architecture as shown in figure 1 provided an easier development of the software tool.

Future versions of the system will include a Java score viewer and editor as well as the interaction with new Java modules that will be developed; and a Lisp kernel for the formulation of new performance commands.

Resources available on the Internet

JAPER, a JAva PERformer:

Java applet for rules testing:

KTH performance rules description:


Melodia sw (Windows OS):


Director Musices sw (Mac OS):





This work was supported and by EU TMR program (Training and Mobility of Researchers), and by the Swedish National Council for Research in the Humanities and Social Sciences, by the Bank of Sweden Tercentenary Foundation.


Updated 2002.12.04 by roberto@speech.kth.se