Creating musical compositions with Java

Transcription

1 Creating musical compositions with Java by Michael Newton 2nd May 2001 Supervisor: Dr. Tony Simons This report is submitted in partial fulfilment of the requirement for the degree of Bachelor of Science with Honours in Computer Science by Michael Paul Newton. I

2 All sentences or passages quoted in this dissertation from other people's work have been specifically acknowledged by clear cross-referencing to author, work and page(s). Any illustrations which are not the work of the author of this dissertation, have been included with the explicit permission of the originator and are specifically acknowledged. I understand that failure to do this amounts to plagiarism and will be considered grounds for failure in this dissertation and the degree examination as a whole. Name: Michael Newton Signature: Date: 2nd May 2001 II

3 ABSTRACT Java provides support for Musical Instrument Digital Interface (MIDI) programming with the Java Sound API. This study investigates the facilities that Java provides for MIDI programmers and looks at using other Java packages such as Java Swing for interface design and implementing user interaction. Java's suitability for building MIDI musical compositions will be demonstrated by producing a Simple Music Composer Application allowing basic facilities for music construction, playback using MIDI synthesis and the saving of musical compositions. The investigation highlights design issues that are involved with interface design and MIDI programming. These include constructing musical notation, printing a musical score to paper, conforming to MIDI standards to save and load Standard MIDI Files and producing musical notation from a given MIDI file. The results of the development work and the research behind the Simple Music Composer is presented together with the specification, design and implementation details in this report. III

4 Acknowledgements I would like to thank the following people who have lent their time, materials and support to this project: Dr. Tony Simons Dr. Guy Brown Mark Bunney Alex Newton Ken Newton Joe Stevens IV

5 Contents ABSTRACT...III ACKNOWLEDGEMENTS... IV 1. INTRODUCTION AIMS CONSTRAINTS FURTHER MATERIAL STRUCTURE OF THE REPORT LITERATURE REVIEW A HISTORY OF MIDI GENERAL MIDI SYSTEM LEVEL REAL-TIME MIDI V MIDI PLAYBACK THE GENERAL MIDI SYSTEM COMMUNICATION BETWEEN MIDI DEVICES SAVING MIDI PERFORMANCES MUSIC THEORY SOFTWARE REVIEW APPLICATION USABILITY AND USER INTERFACE DESIGN THE USE OF JAVA IN THE SIMPLE MUSIC COMPOSER JAVA S AUDIO SUPPORT THE JAVA SOUND API JAVA SWING AND AWT SYSTEM DESIGN & IMPLEMENTATION THE FINAL SPECIFICATION DESIGN ISSUES USER INTERACTION...29 V

6 4.4 PROGRAMMING THE SIMPLE MUSIC COMPOSER RESULTS AND DISCUSSION SYSTEM TESTING THE PRODUCT THE SIMPLE MUSIC COMPOSER DISCUSSION OF THE PROJECT RESULTS REFERENCES...50 APPENDIX THE OPEN() METHOD THE ADDNOTE() METHOD THE PLAYBACK FUNCTION THE SAVE TO MIDI FILE FUNCTION THE LOAD FROM A MIDI FILE FUNCTION...53 VI

7 1. Introduction 1.1 Aims The Musical Instrument Digital Interface (MIDI) offers a standardised method of communication between electronic instruments. MIDI can also be used to generate note sounds through a synthesizer using a process called MIDI synthesis. This project looks at how Java allows programmers to conform to the MIDI standard and assesses how useful the Java facilities are that handle MIDI data. The product of this project is a Simple Music Composer application, which demonstrates how Java can be used for MIDI programming to provide a useful MIDI tool. The client for the Simple Music Composer is a hobby musician, who requires an application which automates his current transcription process (which is done manually at present). The application should therefore allow a musical score to be constructed from various musical symbols and notation. The application will implement a playback facility which uses the Java Sound API and the use of this will be evaluated in terms of the functionality provided and the ease in which it can be used. The playback facility will play the musical representation of the musical score written on-screen accurately, with the correct timing values. Figure 1.1 The functions of a composer application Figure 1.1 shows the functions of the Simple Music Composer, represented informally, to give an outline of the tasks involved in this project. These are high level descriptions and more detail will be given in the design section. The application will use MIDI in three ways; loading and saving compositions using the MIDI file format and playing back a composed tune. The user can also edit a musical score or create one from scratch. The products of the Simple Music Composer are: 1

8 A printout of the musical score An on-screen representation of the musical score A MIDI file containing the saved data representing the score constructed by the user A performance of the composition using MIDI synthesis These allow the user to use the printouts for reading whilst playing an instrument, to import their MIDI files into other applications and to able to change any performance created by a MIDI file. Each task shown in figure 1.1 will be explained in greater detail in the design and development section in chapter 4. This report aims to give background information on the topics covered by this project, to give a detailed description of the how the Simple Music Composer was designed and implemented and to discuss the resulting application and the value of Java Sound to MIDI programmers. 1.2 Constraints The composer system shown in figure 1.1 is quite complex and will require a large development project. As there are tight time constraints on the development of the Simple Music Composer, limits must be placed upon the application s functionality. This will allow more time for the important MIDI functions and for researching the Java Sound API. The initial system, which can be thought of as a 'blue sky' scenario (a perfect implementation), would contain the following features: 1. A method of constructing musical notation on-screen. 2. A function allowing the user to load and save compositions created in the Simple Music Composer adhering to the standard MIDI file format. 3. A function allowing the user to print a musical score to paper. 4. A playback function allowing users to listen to the composition that they have created. 5. Bars should be of variable length, so that when a note is added, the other notes shift up and so that: bars containing few notes are not too small bars containing many notes are not too large. 6. On loading, the Simple Music Composer should parse a MIDI file and reconstruct the musical score representing the file. 7. The playback function should be polyphonic, allowing many notes to be played at the same time. 8. Users should be able to cut, paste and delete items from the musical notation. Realistically, the development of such an application takes teams of developers longer than the lifetime of this project. Due to this a simpler version of the application will be implemented. This can be outlined as: 2

9 1. A subset of musical notation will be constructed on-screen. This will consist of a set of notes, rest, clefs, accidentals and staves. This will provide enough functionality to compose basic compositions to a suitable standard for a hobby musician. 2. The composition created can be played back using MIDI synthesis on one instrument and will reflect the notation shown on screen. 3. Compositions will be saved as standard MIDI files. 4. MIDI files will be imported into the Simple Music Composer and be heard through MIDI playback. Parsing the MIDI file into musical notation will only be implemented as part of the basic system if all other functions are fully working.. 5. The score will be printed to paper giving sheet music. This is the basic system and any spare time once this is completed will be spent looking at the extra features. For features that are not implemented, the approach to implementing the feature will be discussed in chapter 4. The most important tasks are providing a automated version of the current process and in addition implementing MIDI synthesis techniques to play the constructed composition. This will help evaluate the use of the Java Sound API in producing MIDI applications. 1.3 Further Material This report assumes that the reader has a basic understanding of the Java programming language and concepts associated with it. If this is not the case, the Core Java books [21, 22] are a useful introduction to the ideas and features of Java, and cover all of the basic Java used in this project, such as features of Java Swing and Java AWT. The Java Sound API is relatively new and therefore there is little literature available which describes the features of the API. However, the Java Sound API Programmers Guide [14] can be downloaded from the Java Sound API homepage at Documentation is available to accompany the packages provided and explain the uses of each function.. In this report it is assumed that the reader has little knowledge of Java Sound and this will be introduced from the basics through to more complex features. It is also assumed that the reader has some grounding in Music theory and will understand the musical concepts discussed. A short section is included in the literature review (section 2.7) introducing the different notational symbols that are commonly used. It is presumed that the musical terms used in this report will be understood by the reader. The Musical Instrument Digital Interface (MIDI) is a concept for which there is a wide variety of material available. Literature is available to explain many details of MIDI and its different uses. Much of the information available is about handling real-time MIDI data. However many of these concepts given can be applied to MIDI playback. A basic understanding of MIDI can be obtained by reading some of the literature sources used by this report [1, 8, 11, 13, 14, 18, 20, 23]. 3

10 1.4 Structure of the report The body of this report is structured into chapters and each chapter into the following subsections describing the different topics covered: Chapter 2: Literature Review A literature review looks at the different areas encapsulated in the task of building the Simple Music Composer and studies work that has been completed in these areas. Different literature sources are used to give a wider picture of these areas and the relevance of this work is discussed in relation to the Simple Music Composer. In this section the main topics are MIDI, musical notation, application design and a review of existing software packages that offer similar functionality to the Simple Music Composer. Chapter 3: The use of Java in the Simple Music Composer An outline of the use of Java in this project will be given in chapter three, explaining the different packages to be used in the Simple Music Composer and how these packages plug-in. The main focus of this chapter will be on Java Sound and the functionality that it provides, but other packages such as Java Swing and the Java Abstract Window Toolkit (AWT) will also be explained. Chapter 4: Design and Implementation Chapter four describes the design and implementation stages of the Simple Music Composer development. This gives a description of the system specification together with design diagrams showing how program behaviour and human interaction is modelled and how the graphical user interface was designed. Issues that arose during the development work and how these were resolved are also included in this section. Chapter 5: Results and Discussion The final product, the Simple Music Composer, is assessed and discussed in chapter five, as is the ease of use and usefulness of Java Sound in the development work. This gives an evaluation of the project and of issues that have arisen during its lifecycle. Future work on the application is discussed along with other extensions of the project. Chapter 6: Conclusions The final chapter draws conclusions about the design and implementation of composer applications and the use of the Java programming language in building them. An evaluation of Java Sound is given and conclusions are drawn from the developer s experiences in building the Simple Music Composer. Finally, the appendices include segments of program code that illustrate how the MIDI functions in the Simple Music Composer were implemented. The references are also given from which information has been taken to aid the project and the background to MIDI, Java and application design. 4

11 2. Literature Review This chapter provides background information about the different areas encapsulated within MIDI programming and in the development of the Simple Music Composer tool. This will set the context for the remainder of the project and show the sources that were used in researching the topics of MIDI and Java. A brief history of MIDI will be followed by a description of a standard MIDI system and the components from which it is built. This in turn is followed by an overview of Java and the Java facilities that will be needed in the implementation stages of the project. This subject will be dealt with in greater detail in chapter A History of MIDI In the 1960s and 70s the electronic synthesizer became an increasingly common piece of musical equipment. These used electrical voltages to create sounds. A high voltage gave a high note and low voltages were used to create low notes [23]. The synthesizer used electrical wires called patch cords to manipulate the characteristics of a sound. These devices were large boxes which were generally difficult to configure and use. By the mid-1970s brands such as Moog and ARP were in widespread use by musicians. However these were monophonic; only one note could be played at any one time until the Oberheim company introduced the first commercial polyphonic keyboard synthesizer. This was more portable, simpler to use and allowed four notes to be played at once. The sounds could be changed by setting switches and turning knobs which was much more efficient than previously. Many other companies soon released their own polyphonic electronic instruments [23]. The next advance in synthesizer technology was programmable memory, which allowed the musician to save sounds by storing the positions of the knobs and switches. This meant that sounds could be recalled at the press of a button rather than the time consuming method previously used. The next development was driven by the growing need of layering sounds upon other sounds. Many musicians were using two synthesizers and playing the same note on both with each hand. This was until in 1979, when the first keyboards were released equipped with computer interfaces by companies such as Oberheim and Roland [23]. This allowed two identical keyboards to be connected so that the musician could play on the master keyboard and the slave keyboard would automatically play the same notes in whatever sound the keyboard was programmed to make. Digital sequencers also began to appear on the market, allowing musicians to sequence and playback all parts of a song on a group of synthesizers. These were improvements, however all electronic instruments still had to be of the same make in order to communicate, and increasing pressure was being placed on the instrument makers to make their instruments compatible with other brands. 5

12 At the convention of the National Association of Music Merchandisers (NAMM) in 1982, a small group of synthesizer manufacturers congregated to discuss a proposal for adopting a universal standard for transmitting and receiving musical performance data between all types of electronic instrument. The proposed standard was called UMI, the Universal Musical Interface [23]. As the original proposal for UMI underwent a series of revisions, it was finally renamed MIDI, standing for Musical Instrument Digital Interface. In 1983, Sequential Circuits and Roland introduced the first keyboards which were MIDI compatible. This was soon followed by the rest of the industry. To this day, MIDI has remained the industry standard and has undergone extensions to incorporate new features into the language. MIDI s uses had reached not just music but also areas such as theatrical lighting and computer games [23]. In September of 1991, the bodies responsible for MIDI standards, the MIDI Manufacturers Association and Japan MIDI Standards Committee, adopted the General MIDI Specification Level 1 (explained in the next section). This was aimed at providing a minimum level of performance compatibility among MIDI instruments. This is the standard used today by all MIDI compatible instruments [11]. 2.2 General MIDI System Level 1 Today s standard for MIDI is the General MIDI System Level 1 (GM1) which was created by the MIDI Manufacturers Association (MMA) and Japan MIDI Standards Committee (JMSC). The aim of the specification was to set standards for a minimum level of performance offered by any MIDI compatible instruments. In order for an instrument to conform to GM1 the following features must be provided: [11] 1) A minimum of 24 dynamically allocated voices are available simultaneously for melodic and percussive sounds, or 16 dynamically allocated voices are available for melody and 8 for percussion. All voices must respond to velocity. 2) 16 MIDI channels must be supported with each channel being polyphonic. Each channel can play a different instrument. 3) There must be a minimum of 16 simultaneous and potentially different instruments which can be played at once. There must also be a minimum of 128 preset instruments which conform to the GM1 Instrument Patch Map and 47 percussive sounds which conform to the GM1 Percussion Key Map. 4) A variety of MIDI messages must be supported for actions such as note-on, noteoff and relating to channel pressure and pitch bend. This specification defined a minimum et of functions to be met by any GM1 instrument such as a synthesizer, sequencer or equivalent computer software or hardware. The components of a MIDI system are of little use individually but when combined provide a powerful system used to create and control musical performances. MIDI is a powerful tool as the strict GM1 standard means that it is extremely portable and 6

13 universal. The way that MIDI was specified means that writing software to conform to the GM1 standard and to use MIDI features is relatively simple. 2.3 Real-time MIDI V MIDI Playback Before the MIDI system can be described in detail, it is important to distinguish between the two types of MIDI performance; MIDI playback and real-time MIDI. MIDI performances can be produced in real-time, where an electronic instrument or software package is used to play the notes and decide the timing values of each note depending on how long the note is played for. This can be thought of as recording the performance. An alternative type of performance is that of MIDI playback which uses a pre-constructed MIDI performance to invoke the synthesizer to play the given notes. Real-time MIDI is useful when recording a composition from an instrument or electronic keyboard, but not if the MIDI performance is to be created using musical notation [14]. MIDI playback is more useful for this purpose as the MIDI sequence can be constructed as notation is placed upon the screen and the sequence of MIDI data that is created can be played upon request. 2.4 The General Midi System A MIDI System usually consists of a high-level controller, a sequencer and a synthesizer. The high-level controller is usually incorporates some method of tracking user actions such as with a piano keyboard. In a composer software tool it would be driven by the on screen musical notes being placed on a stave. The high-level controller sends messages to the sequencer and synthesizer which represent the notes being played, the style in which the note is played and the instrument (also referred to as a patch or timbre) which plays the note [1, 4, 8, 11, 13, 14, 18]. MIDI System Synthesizer Sequencer Sound bank Channels patch 0 patch 1 patch n Figure 2.1. The structure of a MIDI system [10] 7

14 Figure 2.1 shows the hierarchy of components in a midi system in terms of the client and server relationships encapsulated in the system. The MIDI System invokes the sequencer and synthesizer. The sequencer passes data to the synthesizer which passes data through its channels and these tell the soundbank which patches to play. The Sequencer A sequencer is responsible for playing the musical performance that it receives [1, 8, 11, 13]. The sequencer stores each musical action in a sequence and then sends messages to the synthesizer containing data relating to which notes to play and when to play them [18]. The Simple Music Composer will need to use a sequencer, as this controls the timing of the performance and will ensure that the performance relates to the note lengths displayed in the on-screen representation. The sequencer supports the loading and saving of MIDI data to MIDI File Format data files, which allows the musician to save, restore and edit their composition at a later time [18]. The Simple Music Composer will take advantage of this facility to allow the user to save and load their MIDI performance created by the onscreen notation. Sequencers come in many forms both hardware and software with many different features and varying complexity[1, 4, 13]. Modern commercial sequencing software uses patterns to organise and control the events to be played. One such example is Cubase (see figure 2.2) which uses patterns to graphically display the events to be played, set the times for each sound to be played and gives a more intuitive and useable application than more traditional sequencers. This project will not use a graphical representation of a sequencer but the music musical notation will show similar information about the MIDI notes to be played.. Figure 2.2 Cubase a pattern based sequencer application. [18] The Synthesizer A synthesizer plays the actual sounds sent to it in the form of MIDI messages by the sequencer. This is done by accessing a bank of instruments (soundbank) and playing the desired note on the requested instrument [8, 11, 13]. The synthesizer 8

15 uses MIDI channels which allow each instrument to play a unique part in a composition. Each channel effectively allows a different instrument to be played and controlled [13]. A synthesizer can contain up to sixteen MIDI channels, which enables up to sixteen different instruments to be played at the same time in a composition [8]. This process of playing a musical composition using MIDI channels and MIDI messages is often called MIDI synthesis [11]. The Simple Music Composer will use MIDI synthesis to play the sounds relating to the musical notation constructed on screen, with the sequencer controlling the notes played by the synthesizer. Computer synthesizers can take the form of either software or hardware, with hardware synthesizers being contained within computer soundcards. The Soundbank A soundbank is a collection of instruments identified by a unique program number [14, 20]. There are usually up to 128 instrument sounds (timbres) contained in a soundbank, but this number varies dramatically depending on its purpose. Different uses of a soundbank require different numbers of instruments and different levels of sound quality. Soundbanks can also be implemented in hardware or software. Hardware soundbanks can be found in computer soundcards and often use a synthesis technique known as wavetable synthesis [11]. The resulting sounds produced, often vary slightly depending on the manufacturer of the soundcard. Software soundbanks are a collection of read-only files, which provide the necessary data about each program and each timbre to allow the desired MIDI notes to be played. The Simple Music Composer will use a software soundbank to ensure consistency across platforms and between individual computers. Each of these components add some aspect of functionality to the MIDI system. The sequencer controls the performance and the synthesizer uses MIDI channels and the soundbank to play musical notes when it is told to do so. 2.5 Communication between MIDI devices As MIDI devices are most powerful and useful when combined together to form a MIDI system, a simple but effective method of communication is needed to allow MIDI devices to work together. This is done using MIDI messages. All MIDI messages are three bytes long (to simplify decoding) and different messages invoke different actions in MIDI devices [13]. Messages are sent from the sequencer to the synthesizer and from the synthesizer to its channels. This provides all of the communication necessary to allow the MIDI devices to work together as a system [14]. Before MIDI messages can be explained in more detail, the concept of MIDI events, tracks and sequences must be introduced [14]. A MIDI event is the most basic action in MIDI and is used to describe, for example sounding a single note. A collection of MIDI events can be stored by the sequencer to form a track. The events in a track usually 9

16 contain some common features such as the instrument used to play the notes [20]. A number of tracks form a sequence which contains all of the MIDI information required for the entire composition. For MIDI playback, the sequencer contains the sequence relating to the performance and passes MIDI messages to the synthesizer to invoke notes to be played [18]. These MIDI messages can be classified into four distinct types: Channel voice messages Channel voice messages are the most common commands passed around in MIDI. They control instruments, which are played down a specified MIDI channel. The most common of these messages are note on and note off. When a note-on message is sent to the synthesizer in a MIDI system, the synthesizer starts to play the specified note. The note on message consists of a status byte specifying the channel the note is to played through and two data bytes. These two bytes specify the key number of the note to be played and the velocity with which the key is to be played [8, 11, 13]. In MIDI, musical notes are represented using MIDI key numbers, which range from 0 to 127. A typical piano keyboard ranges between notes 24 to 127 with middle C being note number 60 [11]. A note s velocity represents the loudness with which the note is to be played and ranges from 1 (representing pianissimo in musical terms), and the highest is 127 (representing fortissimo) [13]. A note off message tells the synthesizer to cease playing a specified note. This has the same effect as changing a note s velocity to 0. The note off message, similarly to note on, contains a status byte and two data bytes representing the note number and the release velocity. The release velocity can often be ignored, depending on the MIDI devices being used [10, 13]. Another channel voice message is the program change message. This instructs the synthesizer to change the playback instrument for a particular channel [13]. The current instrument will change to the new instrument, which corresponds to the MIDI program number contained in the message. The MIDI program numbers can range from 0 to 127 and map onto the instruments available in the soundbank (so program 0 will always represent the first instrument in the soundbank) [14]. Other channel voice messages include messages for aftertouch and for pitch bend. These are outside the scope of the Simple Music Composer as pitch bend and polyphonic key pressure will be ignored for the purposes of this project [13]. Channel mode messages These messages allow the behaviour of a channel to be explicitly defined by using messages such as all notes off and all sounds off. These messages can be used to switch off all MIDI messages on a specified channel and to turn off all notes which are currently playing on a particular channel respectively [1, 8, 13]. 10

17 The other channel mode messages tend to be more relevant to real-time MIDI performances and are not applicable to the Simple Music Composer. System common messages System common messages support MIDI functions, but create no sound on their own [13]. An example of a system common message is the song position pointer. The song position pointer keeps track of the position reached in a sequence. The start command causes all clock-based MIDI devices to play from the beginning of the song stored in memory. The stop command causes these devices to stop playing and hold their place in the music. The continue command makes the device play from the held position. The system common message may be used during playback of a musical composition to start playback from a given point in a sequence. Other messages include song select and tune request, which will not be used in this project [13]. System real-time messages System real-time messages are used to synchronise components of a MIDI system, ensuring that notes are played at the same time on all MIDI devices so that they are in phase [8, 11,13]. These messages are more suited to controlling real-time MIDI performances and will not feature in the Simple Music Composer. MIDI messages are used for all communication between MIDI devices. The Simple Music Composer will mainly use channel voice messages to invoke the synthesizer to play musical notes for a given time period. 2.6 Saving MIDI performances Unless MIDI performances can be saved, a lot of time and effort would be needed to create a performance and the performance would need to be created every time it was required this would be back to the pre-midi days. To ensure that this is not the case, musical compositions that are created using MIDI can be stored in files conforming to the Standard MIDI File (SMF) format. A MIDI file in SMF format saves the MIDI messages needed to create the musical composition plus any tempo and time signature settings. This allows any composition saved in SMF format to be replayed exactly as it was recorded. There are three variants of SMF file; Type 0, Type 1 and Type 2 [18] Each format saves the same information but the method of saving it varies between them [18]: Type 0 The original and most common MIDI format. Type 0 takes all recorded tracks and mixes them onto one single track without altering any channel settings. Type 1 A more recent format which preserves the structure of the data. Type 1 saves a collection of tracks as separate tracks with data for different channels on different tracks. It also allows names to be stored for each track. 11

18 Type 2 An even more recent file type, which offers a format which is easier to edit than Type 1. It saves the file in the same way as Type 1 but also stores pattern-data for each track. This can be used with pattern-based sequencers such as Cubase (see fig2). Type 2 is rarely used. All files in the SMF format record the MIDI messages, timing information about when each message is sent and information about the tempo and timing of the whole composition. Standard MIDI files can be loaded into a sequencer, where a sequence is created by parsing the file. This sequence can then be treated like any other, allowing editing and playing. MIDI files are created by a sequencer, which supports the SMF format and give the option to save compositions to this format. 2.7 Music Theory Musical notation is a formal language used by musicians to produce universally understood instructions for the performance of a musical composition. It is important that any tool used for composing musical compositions adheres to the standards of musical notation so that any musician can read it. In order to provide a useful yet realistic set of musical notation for the Simple Music Composer, it is necessary to study the musical symbols which are commonly used and decide which symbols are essential, which are desirable and which are rarely used. This review will now look at the different types of notation available and rate how useful the symbols are to the Simple Music Composer: Notes & Note Lengths The most important musical features are the musical notes themselves, as these dictate the pitch of the note to be played, as well as the length for which the sound is to be sustained. Figure 2.3 shows the set of notes that correspond to different length tones. Figure 2.3 Musical notes and rests 12

19 The Stave and Clefs Depending on the instrument on which the composition is to be played, the layout of the stave and the clefs provided vary. For piano music, a base and treble clef are usually given on a stave with appearance as in Figure 2.4a. Violin arrangements differ, in that a treble clef is given on a single stave and all notes hang from this clef. These differences occur since the number of different pitch notes that can be created on a violin is much smaller than the number of possible pitch notes created on a piano. Figure 2.4b shows a violin stave to demonstrate how the notation changes when the music is being written for different instruments. Figure 2.4a. An extract from the Beatles song when I m sixty four arranged for piano and guitar. Figure 2.4b. An extract from Air composed by J.S Bach and arranged for the violin. For music written for the guitar, it is common to use a single stave to display the melody line and to use letters (as in figure 2.4a) to display the chords that accompany each part of the melody. In Figure 2.4a the B flat chord is played from the first three bars on the guitar. Key signatures The key signature specifies the overall pitch of the composition and shows the note pitches which fit in with the other notes in that key. This automatically sharpens or flattens some notes in that key, so they do not need to be explicitly stated as being sharps or flats in the actual piece as the key signature is stated at the beginning with the time signature. Figure 2.5 shows some of the most common key signatures and the notes that are contained within each key. 13

20 Key Note 1 Note 2 Note 3 Note 4 Note 5 Note 6 Note 7 Note 8 C Major C D E F G A B C D Major D E F# G A B C# D E Major E F# G# A B C# D# E F Major F G A B flat C D E F G Major G A B C D E F# G A Major A B C# D E F# G# A B Major B C# D# E F# G# A# B F# Major B flat C D E flat F G A B flat Figure 2.5 A table showing the most common key signatures Accidentals Sometimes, when writing a piece of music, a composer wants to specify sharps or flats which are not included in the key signature for the composition. When this happens, the composer explicitly sharpens or flattens a note as in figure 2.6. Figure 2.6. The accidental symbols The three main types of accidental are sharps, flats and naturals. Naturals are used to unsharpen or unflatten a note that had been previously been sharpened or flattened. Time Signatures The timing of a performance can alter the entire feeling of the composition. For example, a slow love song would not sound right if played at a lightening pace. Therefore, a composer uses time signatures to specify the speed at which a composition should be played. The most common time signatures used are 2/2, 2/4, 3/4, 4/4 and 6/8. These will be provided for compositions created in the Simple Music Composer. Lyrics and Musical Commands As music is a very expressive medium and musical notation itself does not convey the style in which the composition is to be played, explicit terms are used to instruct the musician on how the piece is to be played. Lyrics are another way of adding expression to a composition. These can be displayed alongside the musical notation to show where lyrics fit in with the melody. 14

21 Lyrics are usually displayed between the two staves on a double stave and beneath the stave if a single stave is being used. 2.8 Software Review There are many computer packages which provide the functionality to construct musical notation in an interactive manner. This section studies two different composer packages and looks at different features that each provide and different ways of interaction and controlling the construction of musical notation. The two applications reviewed are ABC Composer and Noteworthy Composer. These both produce musical notation in different ways, and this review will identify possible features to implement the most effective and intuitive methods for producing a musical score. The ABC composer uses a text-based ASCII approach to construct musical notation. A file, such as that shown in Figure 2.7a, is parsed and produces an output as shown in figure 2.7b. The ABC language for constructing music notation was introduced in 1991 and now several tools are available to produce a score (such as the ABC composer) and other tools to play the notes represented in the parsed file through a computer using MIDI synthesis. Figure 2.7a The input file for the ABC Composer where the text specifies how the notation will be written. Figure 2.7b The output created by parsing the file shown above. 15

22 The text-based approach is simple to design and means that development effort is not spent creating a graphical user interface. However, it is not intuitive to use and requires the user to understand the different characters used to represent the symbols of music. To produce music of any complexity, a wide range of these characters need to be learnt. Another disadvantage is that the user cannot see the music notation on screen until the file is completed and parsed. This makes composing a musical score quite difficult as the user cannot visualise the composition in real-time. The Noteworthy Composer is an application which, like the Simple Music Composer, allows sheet music to be produced using a graphical interaction method and allows the composed score to be played back using MIDI. Figure 2.8 shows the user interface from the Noteworthy Composer and highlights the features of the application: The Noteworthy Composer is not written in the Java Programming language but still offers many of functions that are required of the Simple Music Composer. The main features of the application are: 1. The user can write musical scores and songs using musical notation. Lyrics to be written alongside the score using the notation editor. 2. Compositions can be played back using a soundbank (hardware). The editor highlights each note as it is played. 3. The application supports external MIDI devices so that real-time MIDI input can be used. 4. The musical score can be printed, either as a whole or as selected parts. 5. Standard MIDI files can be imported and exported, converting imported files into musical notation so that they can be viewed, rearranged and edited as desired. 6. Parts of the composition can be transposed into different key signatures. The Graphical User Interface (GUI) uses toolbars containing icons representing the various music symbols. To place a note on the stave a cursor system is used where the desired note is selected and the position on screen selected by moving the cursor. The enter key is pressed to place the note. The playback facility allows the composer to audio-visualise the tune that they have created to see if it matches the tune that they intended. 16

23 Figure 2.8 The Noteworthy composer GUI 2.9 Application Usability and User Interface design The Graphical User Interface (GUI) is the component of the Simple Music Composer that the user will see and will interact with. This means that the layout of the GUI needs to be intuitive and efficient so that the application can be as user friendly as possible. There have been many studies into human-computer interaction and software usability. Nielsen's usability heuristics were obtained from a study of software usability. The main areas for which usability of an application is important are speed of navigation, ease of navigation and clarity of how-to-use each function. The user interface should fulfil the following criteria: [12] Keep the user informed of the state that the system is in. Appear as similar to the real world as possible using the same notation and language. Support ways of correcting mistakes. Use consistent language and layout. Provide help functions to guide a user if needed. Use a minimalist design so as not to confuse the user. Position features that need to be selected near the edges of the screen. Features that are commonly selected should be selectable in one click. Features that are rarely used should be hidden from the main screen so as not to clutter the user environment. 17

24 When designing the user interface, factors such as the screen size, screen resolution and the size and position of interactive features, such as buttons and text fields, need to be taken into account. These are crucial if the final GUI is going to meet the demands of the user [12]. The GUI for the Simple Music Composer will need menus to enable options to be selected. The features which allow symbols to be placed should be available alongside the musical notation, and features such as application controls (quit, load, save etc) can be hidden in menus that are shown when needed. Information boxes can be shown to the user to inform him of any changes in state [21]. For example, the successful loading of a MIDI file can be indicated by a message box appearing telling the user of the results of his request. Edit functions such as cut, paste and delete should be provided to ensure that mistakes are easy to correct. The user will also experience consistency throughout the application with language, layout and look and feel. The layout can include selectable buttons near the edge of the screen for ease operation [12]. A consistent look and feel for each session in which the application is used must be given [12] A key issue in the GUI layout is the representation of musical notation. The main options are to have a piano keyboard shown on-screen or have the user construct a composition with musical symbols. With the keyboard approach, the user can press the desired piano notes to create the musical notation. The musical score construction approach involves the user building a composition be placing musical symbols on a stave. The most usable solution to this representation is to construct musical notation itself, since using a piano keyboard representation would be assume that the composer can play the piano. This may not be the case as the Simple Music Composer is not specifically aimed at piano players. A musical notation construction approach using musical symbols offers a more universal notation and does not assume that the composer can play any instrument. This fits in with the usability heuristic which states that the on screen representation should be as similar to the real world as possible. In this case, simply automating the current process of drawing musical notation is most appropriate. The usability features will be incorporated into the design of the Simple Music Composer, ensuring that the user can operate the application with ease and in an efficient manner. 18

25 3. The Use of Java in the Simple Music Composer Java is to be used to implement the Simple Music Composer due its compatibility across platforms and the functionality offered by the Java Sound API. This chapter looks at the different Java packages that will be used in the development of the Simple Music Composer. The functionality provided by these packages is described and an explanation about how the package would be useful to the Simple Music Composer is included. A brief description about Java s background in audio programming follows which is in turn followed by information about each of the standard Java packages to be imported to the Simple Music Composer, describing the functionality that each provide. 3.1 Java s audio support Java was produced by Sun Microsystems and since its release, many different Application Programming Interfaces (APIs) have been released to assist programmers who use the Java language. APIs offer classes and interfaces that perform common functions and provides interfaces with other pieces of hardware and software installed on a computer. One such package is the Java Sound API, which was released in 2000 to assist with adding sounds to computer applications [20]. In the pre-java Sound days, most multimedia programs were developed using the Java Media Framework (JMF) but this had very limited support for audio and little support for MIDI programming in particular. The only real audio support enabled audio files (*.au files) and MIDI files of type 1 and 2 to be played on 32 bit Windows environments and Solaris [3, 9]. The Java Sound API offers new functionality and support for MIDI programming and for sampled audio [14]. The package contains two related APIs; javax.sound.sampled and javax.sound.midi. For this project, we will concentrate on the javax.sound.midi package as sampled audio will not be used in the Simple Music Composer. The Java Sound API is not distributed as part of the standard Java Development Kit (JDK). 3.2 The Java Sound API The javax.sound.midi package contains all of the support needed for complex MIDI programming, with the capability to create MIDI sequences both through real-time recording and through the concatenation of MIDI events (creating a sequence from scratch) [14]. We will ignore the content which is aimed specifically at real-time recording from MIDI devices as this will not be relevant to the Simple Music Composer. 19

26 The components of the Simple Music Composer which will need to import packages from the Java Sound API are: 1. The MIDI playback utility Creating sequences, tracks, MIDI messages and MIDI events Accessing and controlling MIDI devices Timing of the composition. 2. The saving and loading of MIDI files These functions have already been explained in chapter two where the MIDI system has been described. The Java Sound packages which can be used to help fulfil these requirements will be described together with how they can be used to create a MIDI performance. The Java Sound API provides a MidiSystem class which can be used to access all MIDI devices installed on a system [14]. This class provides methods to call MIDI devices available on the computer, such as the getsequencer() and getsynthesizer() methods [20]. The MidiSystem basically represents the overall system consisting of many MIDI devices. A MIDI device is modelled by the MidiDevice class. This is the superclass of objects representing the individual MIDI systems components, such as the Sequencer, Synthesizer and Soundbank. The MidiSystem class provides methods to list available MIDI devices, so that the user can specify which devices to use if more than one are available [14]. Each MIDI device relevant to this project will now be described in terms of the Java representation of itself: The Sequencer The Sequencer class is a subclass of the MidiDevice interface. Its task is to concatenate MidiMessage objects to represent a musical composition. These form a Sequence object, which is a data structure that contains a number of MidiEvents. The MidiEvent class is the superclass of MidiMessage. A Sequencer object contains the following types of method that are relevant to the Simple Music Composer [14]: 1. Methods to load data to and from standard MIDI file. 2. Methods to set the timing parameters of a composition. The Sequencer object provides the functionality to allow the Simple Music Composer to load and save sequences to a standard MIDI file (see section 3). The timing of a performance is specified in the Sequence object representing the composition. A Sequence can be instantiated by making a new Sequence call. In Java two timing mechanisms can be used to time the MIDI performance; Pulses Per Quarter note (PPQ) or ticks per frame [14]. Ticks per frame is the standard adopted by the Society of Motion Picture and Television Engineers. This is generally used when sound is synchronized with video frames, so sound can be played at specific points in the picture. PPQ note will be used as the timing mechanism as this 20

27 allows a time period to be specified and used to represent one musical note. This can then be the basis unit to calculate the other note lengths. For example if one beat is equal to 100 milliseconds, a quaver s timing value will be equal to 50 milliseconds and a minim 200 milliseconds. This policy will provide the timing values for playback in the Simple Music Composer. To give a series of notes the correct timing values in a sequence, Java can use a count which begins at zero, representing the start of the composition. The timing is given in MIDI ticks (1 unit in PPQ timing). The cumulative timing will be given to each note, so that it starts sounding the note at the correct time in the composition and the note off message will be given after the note has lasted its duration. Each MidiMessage specifies a time in MIDI ticks, which represents the timing in respect to the remainder of the composition in which the message should be sent to the synthesizer [20]. The Synthesizer The synthesizer is represented by the Synthesizer class and receives MidiEvents from the Sequencer. The Synthesizer s responsibility is to access a soundbank and the Instrument objects contained within. A Synthesizer can be instantiated by calling the MidiSystem to return the default synthesizer. The Synthesizer object uses classes that implement the MidiChannel interface to represent the synthesizer using MIDI channels. MidiEvents can then be passed along to the MidiChannels via the Synthesizer. The MIDI channels of a synthesizer can be returned by calling the Synthesizer.getChannels() method [20]. The Synthesizer also accesses a soundbank to get different instruments for the synthesized sound. Methods are given allowing different instruments to be loaded and unloaded from the Synthesizer [14]. The Soundbank The Java Sound soundbanks can contain up to 128 x 128 instruments as they can contain up to 128 MIDI banks [20]. The synthesizer object is responsible for obtaining a default Soundbank [14]. This can be done using the synthesizer s getdefaultsoundbank() method. A Soundbank contains many Instrument objects which emulate musical instruments in the real world [14, 20]. The Instrument object is a MIDI synthesis algorithm which produces the sound of the instrument. The Instrument class contains a method which returns the patch number and bank number of itself encapsulated within a Patch object [20]. A Soundbank interface is provided which contains methods to obtain details about the default soundbank and two other methods; getinstruments() and getinstrument(patch p). The getinstruments() method returns an array of Instrument objects contained in the soundbank. The getinstrument(patch p) method returns an Instrument object which corresponds to the patch number sent as a parameter [20]. 21

28 All of these MidiDevice objects help the programmer to : access the system s MIDI devices. create MIDI playback. save and load MIDI data as standard MIDI files In order to allow a user to playback a composition using MIDI synthesis, Java needs some representation of the MIDI messages sent between MIDI devices to invoke actions in other devices. Java Sound provides the Sequence, Track, MidiEvent and MidiMessage classes where a Sequence consists of a set of Track objects. A Track is a sequence of MidiEvents. A MidiEvent is the most basic object type and models any simple MIDI event like a note on call. Java Sound provides a MidiMessage class which represents the concept of a MIDI message. This is a direct subclass of MidiEvent and there are three further subclasses; ShortMessage, SysexMessage and MetaMessage [14, 20]. The ShortMessage class represents channel voice, channel mode, system common, and system real-time messages. These all have the same format of a status byte followed by two data bytes, as this is easier for the Java compiler to handle. The class encapsulates a series of static integer variables which correspond to a command number (for example 144 represents the note on command). The ShortMessage class gives access to a Channel object with the getchannel() method and also provides accessors for each databyte. SysexMessage represents system exclusive messages. The class contains an accessor method giving the data contained in the message and a number of mutator methods to set the data in the message. The MetaMessage class is concerned with representing meta-events such as tempo indicators. MetaMessage contains methods to get the type of message and to access the data contained in the message. The MidiSystem class provides the functionality to store a sequence as a Standard MIDI file. This adheres to the standards set in the GM1 Specification and therefore means that these files can be imported into other MIDI devices and MIDI software products. The method: MidiSystem.write(sequence, filetype[0], file) is used to write a given sequence to the given file which conforms to the standard represented by filetype[0], which is the first in the list of MIDI file types [14]. For the Simple Music Composer filetype[0] will be used to represent Standard MIDI files of Type 0. The write method can throw SecurityExceptions and IOExceptions. The file variable passed to the above method is an instance of the Java IO File object. This represents an empty file to which data can be written. This method can be used to 22

29 provide the save function on the Simple Music Composer as the created sequence can be written to a file in SMF format. To restore a sequence that has been saved as a Standard Midi File, several accessor methods can be used to retrieve the data. The MidiSystem.getSequence(file) method returns the sequence represented in the MIDI file. The Sequence.getTracks() method returns the track contained in the sequence [20]. This will provide the basis for the loading function in the Simple Music Composer. 3.3 Java Swing and AWT In the presentation of the Graphical User Interface, Java Swing will be a useful package as it gives tools with which to construct and organise user interface environments. The Swing API is a set of lightweight components that are built upon the heavyweight Abstract Windowing Toolkit (AWT) components. The term heavyweight is used to describe the fact that the AWT components contain some native code for each platform and therefore were limited in their use across platforms. A lightweight component uses no native code and is therefore able to look consistent and feel like native applications due to the relaxation of constraints which were imposed by use of the platforms with poorer facilities for computer graphics [15]. The paintcomponent(graphics g) method is provided to create a graphics rendering pipeline. This is updated by calling the repaint() method. All musical notation will be drawn by the paintcomponent(graphics g) method [21]. Swing also gives objects that can be placed into a GUI to show the required information and give the user a platform to interact with. The Simple Music Composer will be a JFrame object (showing as a window) which contains a canvas area (a JPanel) and some menus with which the user can interact. The JMenuBar gives a text menu bar that can contain JMenu items (other smaller menus). These can display menus like the File, Edit and View menus that are often found in applications [15, 21]. Information boxes can be created in Java either to tell the user that an event has occurred, or to gather some information from the user. The JOptionPane object is a dialogue box that contains a message and in some instances the user can enter data into the box which can then be used in the main program [21]. The Simple Music Composer will use these objects to allow the user to set and change key and time signatures. Swing gives a set of buttons which can contain text or an image label to help the button be identified. A menu that contains buttons (JButton objects) can be presented in different ways using the layout managers and borders that Swing supplies. These improve the usability of the application. The buttons allow the user to register the actions that they wish to invoke. To respond to user actions, Java AWT provides an ActionListener interface. An actionperformed(event e) method can be used to locate the source of a detected action and an algorithm can be specified to invoke the necessary functions as a result of the action [15, 21]. 23

30 Swing also provides a JScrollBar object which can be placed on a panel, allowing a panel which is larger than the onscreen viewable area to be viewed by scrolling down or across the page [22]. This is done by using a JScrollbar and an AdjustmentListener which can be implemented and a adjustmentvaluechanged(adjustmentevent evt) method can invoke a translation so that a different part of the panel can be seen. This will be used in the Simple Music Composer to allow a complete composition to be shown. To draw musical symbols onscreen, either the Graphics package needs to be used to draw a shape from basic lines or the Graphics2D package to draw symbols from basic two dimensional shapes or by placing ready-drawn images on the screen. If the symbols are too complex or difficult to draw using the Graphics classes provided by AWT, Swing gives a way in which a ready drawn image (saved as a GIF file) can be placed at a specified point on the screen. This method could be used for symbols such as the treble clef whereas Graphics2D could be used to draw a crotchet as this just consists of an ellipse and a line. Graphics2D offers methods to draw most two-dimensional shapes and curves. The Java.awt.Print package provides the necessary functionality to print an area of the screen to paper through a printer. The object that is to be printed needs to implement the Printable interface and a print method needs to be included in the class [22]. A PrinterJob object represents one instance of a printing action and allows an object to be set as Printable, such as the canvas area on the Simple Music Composer. If multiple pages are to be printed, a Book object can be used to represent a set of pages, this can then be printed [22]. A common way of allowing the user to interact with an application is to register mouse events on the screen. In the case of the Simple Music Composer, the user will want to select the position where a symbol is to be placed or select a symbol that has already been placed. Java AWT offers the MouseListener interface which can implemented to register and act upon mouse events [21]. This is similar to the ActionListener class but has more methods that need to be defined. The mouseclicked(mouseevent e) method is the most commonly used as it allows the programmer to specify the action to take when the mouse has been clicked in a certain position. Other methods act when the mouse is dragged, pressed, released or moved. All of these features of Java Swing can be combined to build a usable and attractive looking Graphical User Interface, with which the user can interact. Java AWT also provides many components that can be used for GUI implementation, but these are heavyweight components and are less portable than Swing components [15]. When compared to the AWT, Swing components take longer to appear onscreen and do not automatically refresh[15]. This is done explicitly with the repaint() method. Swing has a larger and more convenient set of interface elements and the users will experience a consistent display across platforms[15]. 24

31 25

32 4. System Design & Implementation The development of the Simple Music Composer followed a strict development plan which included exactly specifying a basic system with a set of requirements, a design phase and an implementation stage. The design phase involved modelling user interaction, program behaviour and working out where software functions needed to be written or where an Java API could plug-in and provide the required function. The implementation stage involved writing the Java code for the Simple Music Composer and tackling issues that arose along the way. This chapter will describe each stage of the system specification and design and with the use of diagrams, show the final designs for the Simple Music Composer. The implementation stage will be described, giving an explanation of the components that were produced and how issues that arose during development were addressed. 4.1 The Final Specification The Simple Music Composer s main aim is to demonstrate Java s suitability as a tool for MIDI programmers. This means that the application will be completely written in Java using the Java Sound, Swing and AWT APIs. This ensures that the application will be portable between all Microsoft Windows platforms in addition to Unix platforms and the Apple Macintosh environment. In an ideal scenario, a large proportion of musical notation would be provided in the construction techniques for creating sheet music interactively. However, as stated in section 1.2, the time constraints of this project make this impractical. A smaller subset of musical notation must be provided. This subset is as follows: Notes = {semibreve, dotted minim, minim, crotchet, quaver, semiquaver, demisemiquaver} Rests = {semibreve, dotted minim, minim, crotchet, quaver} Accidentals = {sharp, flat, natural} Clefs = {bass, treble} Time Signatures = {2/2, 2/4. 3/4, 4/4, 6/8} Key Signatures = {C, G} Staves = {Piano} Figure 4.1 The subset of notation provided in the Simple Music Composer This set will be sufficient to provide the basic functionality for the music composer and allow an expressive composition of reasonable complexity to be created. 26

33 The following specification for the basic system was devised which met all client requirements and added conditions that have been discussed in the literature and software reviews: 1. The subset of musical notation shown in figure 4.1 will be available to construct the notation. 2. An intuitive mechanism will be used to select a symbol and place it on the musical stave. 3. The application will be compatible with all Microsoft windows platforms. 4. The application will use a variety of Java Sound classes to demonstrate the use and functionality provided by the API. 5. The user will be able to save a composition as a standard MIDI file of Type 0 (*.MID file). 6. The user can load standard MIDI files of Type 0 into the application. 7. The user will be able to hear a composition that has been loaded or created through a playback facility that uses MIDI synthesis which represents the exact timing and note pitches specified on screen. 8. The user can print a musical score on paper to create sheet music. 9. The time signature will be constant for a complete composition. 10. The key signature will be constant for a complete composition. 11. The user should be able to edit the musical score ( cut, paste and delete symbols) This specification describes the basic system to be implemented. The details of how this specification is to be met is discussed later in this chapter. When the Simple Music Composer was designed, several issues were raised, mostly concerning the construction of musical notation. These issues will be discussed before the behaviour of the Simple Music Composer is specified. 4.2 Design Issues There are several design issues that have arisen where there are choices between alternative ways of implementing functions. Many of these are to do with the amount of automation involved in constructing the musical score and the complexity of the facilities provided by the Simple Music Composer. The issues will be discussed and alternative approaches put forward with the justification for the chosen approach being given. 1. The symbol positioning policy The first issue that arises is what policy the Simple Music Composer will take regarding how the user will select the position of the symbol to be placed. There are two basic options; a cursor based approach or a click and drag from a menu approach. The advantages of a click and drag approach are that the user can place a note in one direct action and it is a very intuitive system to understand. The cursor approach is simpler to implement as a cursor object can be created which can be placed in the desired position and can be called to find the position for the symbol. The cursor approach is also easy to use and gives a good on-screen guide to where the symbol is to be placed. A click and 27

34 drag menu would be more difficult to visualise and it would be hard to accurately indicate the exact position of the symbol. For example, the difference between two adjacent notes will be quite small and the note could inadvertently be placed in the wrong position quite easily. For ease of implementation and simplicity the cursor approach will be used for the Simple Music Composers symbol positioning policy. 2. Recreating Musical Notation from a MIDI file When a MIDI file is loaded into the Simple Music Composer, the musical notation should be reconstructed on screen to represent the MIDI sequence currently loaded into the application. This complex procedure involves breaking the sequence up bit by bit until all of the MidiEvent objects have been extracted. When this stage is completed, the positions of the notes can be calculated since the timing values give the order of the notes along the x axis and the note number gives the y-axis position. This enables the music to be written. The rests will be calculated by finding positions in the music which a note is not sounding. Java Sound provides all of the functions needed to do this; it is just a matter of using loops to get each MIDI message and then decomposing each message to find the message type, timing value and note number [14]. This can then be used to construct the notation onscreen. Unfortunately, it is unlikely that this will be implemented fully due to the time constraints on the project. 3. Automatic bar line insertion and variable bar lengths A desirable feature would be to add bar lines when the maximum number of beats in a bar has been placed on the stave. This requires the application to shift the bar line if more notes are added to the bar. For example if the user went back and changed a crotchet to two quavers, the bar line would need to shift right by the space of one note. It is desirable that bars containing just a semibreve or few notes should be of a reasonable size and not squashed in between two bar lines. The task of making the size of the bars vary is not easy as the sizes can vary greatly depending on the duration of the notes. Four beats can be represented by 32 demisemiquavers or one semibreve. This highlights the variation in the number of notes in a bar. Due to the constraints on this project in terms of time and the fact that the one of the original aims was to automate the process of manually constructing music notation, the bar lines will not appear automatically when a bar has been completed. Instead the bar lines will be placed just like any other symbols by moving the cursor to the desired position and clicking on insert bar line button. The bar lines visually pleasing but not vital to project. 3. Stave creation and selection Compositions vary greatly in length, some are pages long and others are just a few bars long. This means that it is not possible to give a number of staves on screen and just let the user place notes on them. Instead, a policy will be used which detects when the end 28

35 of the line has been reached and creates a new stave beneath. This can be programmed by throwing an exception when the cursor detects that it has been moved outside the limit of the x axis for the stave. This exception can be caught by the ComposerCanvas and a new stave drawn as a result. This gives a efficient and automatic stave creation technique which means that staves are drawn as they are required so compositions do not contain blank staves. 4. Adding Accidentals to Notes Accidentals are an important feature as they allow the composer to deviate from the key signature and place notes that are out of key. These will be available in the musical symbols to place sharps, flats and naturals on notes on the stave. The Simple Music Composer will provide a set of buttons which can be selected so that any note placed whilst the selection is made carries that accidental. 5. Setting Key and Time Signatures The key and time signatures are an important feature of musical notation. The major key signatures will be available for selection by the user. When a key signature is selected the appropriate sharps or flats symbols will be shown in the correct position between the clef and the time signature. The common time signatures will be available for the user to select and will immediately change on screen when a new time signature has been selected. The facility to change these values will be provided in a information box that appears on screen and contains a list of available options for the user to select. 4.3 User Interaction To meet the aims stated in section 1.1 the user needs to be able to interact with the Simple Music Composer to produce sheet music. This automates the current manual process of writing the musical score by hand. Additional features such as the ability to save, load and playback a composition make the application more desirable for the hobby musician. This makes the automation of the process worthwhile and takes advantage of the processing power and audio support offered by modern desktop computers. 29

36 Figure 4.2 The Task Model diagram for the Simple Music Composer The user interaction between the hobby musician and the Simple Music Composer can be modelled by the Discovery Method s Task Modelling stage. This includes a set of task sketches, which outline the basic functions and a high level Task Model that can be decomposed into a series of lower level task models, which show how each function is performed. Figure 4.2 shows the task models demonstrating how a hobby musician can interact with the Simple Music Composer. The task model illustrates the different tasks involved in building a musical composition using a software tool and shows the functionality which is required of the Simple Music Composer. The arrows in the diagram represent input to and output from the application. For the Simple Music Composer, the outputs are a printout of the composition and a standard MIDI file containing saved work. Each of the tasks in the diagram above can be broken down into a set of subtasks at a lower level to show in more detail how each task is to be completed. The sole interaction between the hobby musician and the Simple Music Composer will consist of the user interacting with the on-screen display (the GUI). To monitor and respond to user actions, event listeners will be a key part of the GUI, tracking the selections the user makes to invoke internal actions within the application itself. The GUI will be described in terms of its features and the ways in which the user interacts with them. In order to make user interaction efficient, intuitive and simple, a menu structure needs to be devised to allow notational symbols to be selected and allow the user to select the action that he requires. According to usability heuristics, selectable items should be placed in functional groups, ensuring that if the user knows what function he requires, he will be able to find the appropriate selection within the right group. The Simple Music Composer includes the following functional groups: 30

37 Notes Rests Accidentals Stave controls MIDI controls File controls (ie save, load, exit) Time and Key signatures To make item selection as simple as possible, the controls that need to be selected frequently should be available at the edge of the screen. This fits in with Nielsen s usability heuristics that state that items around the screen edges are easier to select than those closer to the centre. This also reduces the number of clicks needed for the most common actions as each item can be directly selected. The most commonly used buttons will be those that select each musical symbol and construct the notation. This means that all notes, rests, accidentals and stave controls should be provided in an on screen menu. It is conventional to give file commands in a dropdown File menubar at the top of the screen. This replicates most popular applications such as the Microsoft Office range. This means that the user is likely to be familiar with the selection process to save and open files and to exit the application. Due to the simple nature of the application, the time and key signatures are only able to be set once in the Simple Music Composer changes. This means that at most, the option to change these parameters will only be selected once per composition and therefore the items to change these can be selectable from the Settings menu bar. This means that like the options contained in the File menu, an extra click is required to select the right menu. Due to the low frequency of these actions this is negligible. The button that commences the MIDI playback of a composition will be contained in the menubar under the menu name of MIDI. MIDI playback will only be required by the user at intervals in the completion of the composition. The requirement to use two clicks to start the playback is therefore acceptable in terms of usability. 31

38 Figure 4.3 The menu systems & GUI layout Figure 4.3 shows the layout of the menu systems in the Simple Music Composer, which form a usable and direct way of registering user actions. User actions can be used to invoke internal actions in the application. The GUI will provide the look and feel that is common to most windows based applications, implementing common usability features such as tooltips and mnemonics to assist users of all levels of expertise. As figure 4.3 shows there will be the two menu bars and a canvas area where the actual notation will be shown. The next design issue is that of how musical notation will be constructed. As section 2.8 Software Review, shows there are several approaches to be taken. The two most feasible are a click and drag approach and a cursor approach. The click and drag approach involves the user selecting the desired musical symbol from a menu and dragging the shape to the position on the canvas where it is to be placed. The cursor method involves positioning a cursor in the place that the symbol is to be placed and then clicking a button in the menu to specify which symbol is wanted. Both approaches are 32

39 adequate for the simple music composer but in the case where the whole canvas is not in view (scrollbars are required to move the viewport down the page) the cursor approach would be preferable. This is because it is quite difficult to drag items and scroll down at the same time. For this reason, a cursor mechanism was adopted so that the user could select a note, position the cursor and then place the note. Now that user interaction with the system has been modelled the user will be able to perform all of the functions that they require to construct musical notation. The external behaviour of the application when interacting with the user has been shown, but the internal behaviour caused by user actions needs to be modelled. The next section looks at the Simple Music Composer s internal behaviour and at how external actions invoke changes in the application s internal state. 4.4 Programming the Simple Music Composer As the Simple Music Composer will be written in the Java programming language, the design is object-oriented. This means that the program will be written as a set of classes representing different objects. This class structure is displayed in figure 4.4. This section describes the outline of how the code and abstract data types (ADTs) were structured as well as giving a brief description of how the components communicate. Figure 4.4 The class structure of the Simple Music Composer The ComposerFrame class encapsulates all of the data about the GUI and provides all of the facilities for user interaction. This class instantiates a ComposerCanvas class which represents the panel showing the musical notation. The ComposerFrame will implement the ActionListener interface and will call methods in the ComposerCanvas object to react to user events (button presses and menu selections). The ComposerFrame will also extend the JFrame class, which gives the frame for the user environment and provides the ability to close and minimize the application. 33

40 The ComposerCanvas implements both the Printable interface and the MouseListener interface. It encapsulates all of the methods which control the construction of the notation and is responsible for drawing the notation on-screen and printing the area encompassed by the canvas object. The MouseListener interface gives methods which allow responses to mouse clicks, so that the position of the note to be placed can be specified by the user. There are Note and Rest classes, which represent the musical symbols and encapsulate the timing value of the symbol, as well as data about the position that the symbol occupies on screen. A Stave class represents each musical staff that has been constructed. Instances of these classes are stored in arrays which can be iterated through drawing features in the rendering pipeline. The ComposerCanvas class instantiates a MidiPlayer object which is responsible for creating the MIDI system and playing back the sequence that represents the musical score displayed on screen. This class is also responsible for writing a created sequence to a file in the SMF format and for retrieving a sequence from a standard MIDI file. In order to select the position in which symbols are to be placed during notation construction, a cursor will be used. This is created by the Cursor class instantiated by the ComposerCanvas class. This class draws the cursor on the canvas area and is responsible for moving the cursor to the required position. To provide the necessary functionality, packages will be imported from the Java libraries. As described in chapter 3, the Java Sound, Swing and AWT APIs will be used, together with Java IO to provide the underlying functionality of the Simple Music Composer. Figure 4.5 shows how the class structure of the Simple Music Composer and how the classes plug in to the Java libraries. 34

41 FIGURE 4.5 How the Java libraries communicate with the Simple Music Composer Figure 4.5 shows the different APIs that the Simple Music Composer uses. The design of the class structure means that each class of the Simple Music Composer deals with a different task and therefore needs different imports. Chapter 3 describes some of the functionality that is offered by each API. For this project, the most important class is the MidiPlayer, which gives the functionality of producing sound by MIDI synthesis and for using MIDI files to save and restore compositions. A look at how the MidiPlayer class was programmed will follow to identify the parts of Java Sound used and how the functions are performed: Initialising the MIDI system The open() method configures and initialises the MIDI system represented by the MidiSystem class. The elements of the MIDI System that are needed for the MidiPlayer class are a sequencer, a synthesizer, a soundbank and a sequence. Appendix 2.1 displays the open() method. All of the attempts to connect to MIDI devices must be tested in try loops and any exceptions that arise from the connection attempt can be caught and handled. It can be seen how the MidiSystem class is used to call the getmididevice() methods which creates gives the programmer a handle on controlling the MidiDevice [14, 20]. 35

42 Creating and playing a sequence Every time a note is placed on the musical stave, two MidiMessage objects are added to the Track. By adding the MidiMessages to the track, they are being added to the Sequence. The Java code which implements the MidiPlayer.addNote() method can be seen in Appendix 2.2. Two MidiMessage objects are instantiated. The first message is given a status byte identifying the message as a note on. The channel number is stated together with the number of the note to be played and the velocity. The MidiMessage is encapsulated in a MidiEvent along with the time (in MIDI ticks) when the message is to be sent. The message is then added to the Track. The second message is a note off to stop the note sounding. This is created in the same way as the note on message with a different timing value and a different status byte [14]. The Sequence object, that contains the Track captures all of the MidiEvents and can be played back using MIDI synthesis by using the Sequencer.setSequence(Sequence s) method to load the sequence into the sequencer s memory [14]. The Sequencer.start() method can be used to invoke the sequencer to start sending MIDI messages to the synthesizer. The playback() function in MidiPlayer can be seen in Appendix 2.3. Saving the MIDI composition The Simple Music Composer s MidiPlayer class supplies the savemidifile(file file) method. This calls the Java Sound s MidiSystem.write(Sequence s, File file) method which writes a given sequence to the file parameter [14]. This is stored as a MIDI file of type 0. The method is implemented inside a try loop in case SecurityExceptions or IOExceptions are thrown. The savemidifile(file file) method can be viewed in Appendix 2.4. Loading a MIDI composition. The MidiPlayer class provides a loadmidifile(file file) method which loads a sequence from a MIDI file. The MidiSystem.getSequence(File file) provides the algorithm to obtain a Sequence object from a standard MIDI file [14]. The loadmidifile(file file) method can be seen in Appendix 2.5. These functions of the MidiPlayer class in the Simple Music Composer, have been coded using the Java Sound API to give the functionality needed for MIDI Synthesis. The Java code was written for all functions and the resulting application is presented in chapter 5. 36

43 5. Results and Discussion This chapter presents the Simple Music Composer the product of this project, and explains the results of the system testing. Following this, a discussion will evaluate the work completed in this project and give an analysis of the Java Sound API, which has been used to implement the application. The chapter concludes by outlining further work that can be undertaken to extend the project and improve the Simple Music Composer. 5.1 System Testing The testing of the Simple Music Composer was conducted in small parts upon the completion of each functional unit. This ensured that each function worked. The composition of the functional units will contain very few errors and be significantly less likely to fail. It is difficult to specify a single testing strategy which can be applied to all functional units, so each was tested using a different method. The complete system was tested by transcribing well-known songs and studying the outputs from the application. The functional testing during the implementation stage minimised the time needed for final testing. Each testing method will be explained and some examples of the test results will be given: Testing the notation construction mechanisms Writing short musical scores where the scores were already known tested the notation construction capabilities of the Simple Music Composer. Together these incorporated all musical symbols provided, and also tested the different key and time signature settings. In writing the musical notation, checks were made for consistency across staves, a test of constructing a very large composition and to ensure that meaningful notation could be constructed which is similar or superior in quality to a hand written score. This test was passed as all notation provided, could be placed in a meaningful and consistent composition which was displayed on screen. Testing the MIDI Playback function The MIDI Playback function was tested by comparing the output of an electronic keyboard to the output of the Simple Music Composer. The composition was played on the keyboard and then the Simple Music Composer played the same tune. As both are MIDI devices, the same instrument was used and it was expected that the output for both would be the same. This proved to be correct, showing that this function that uses MIDI synthesis, works as expected. Further testing of the playback facility was carried out in the testing of the load and save functions. Testing the Saving and Loading functions The two functions that handle MIDI files were the most difficult to test. As the function that constructs musical notation from a MIDI file was not implemented, it was difficult to tell whether the correct data had been stored in the file. To combat 37

44 this problem, two other applications were used to test the files created by the Simple Music Composer. The save function was tested using the Microsoft Windows 95 Media Player and the Noteworthy Composer. A melody was written on the Simple Music Composer and saved as a MIDI file. This MIDI file was imported into the Media Player, so that the output could be compared with the Simple Music Composer s playback. This tested the portability of the produced MIDI file and ensured that the two tunes were the same. The second test imported the MIDI file into the Noteworthy Composer, and the notation and playback produced by Noteworthy were compared to the Simple Music Composer s output. Figure 5.1 shows the output from both applications for a simple note sequence. The output from the Simple Music Composer The output from the Noteworthy Composer when passed the MIDI file created in the Simple Music Composer Figure 5.1 Testing the MIDI File save function. The load function is tested by simply loading a MIDI file into the Simple Music Composer, and using the playback function. This output was tested using the same techniques, described in testing the MIDI playback function. The comparison shows that exactly the same notation is produced by both applications from the same MIDI file. This shows that the Simple Music Composer can accurately represent the musical notation on screen with a MIDI file that contains the same note and timing information. 38

45 The print method The print method was tested by printing compositions using the Simple Music Composer. The printouts were then compared to the on-screen representation. On initial tests after implementation, the test was passed. However, when tested with the complete system, the print failed due to exceptions being thrown by Java. Due to time constraints, the print function will not be rewritten. The testing techniques were adequate to ensure that the basic functionality was error free. They considerably reduced the time needed for final testing. Final testing was completed and the only problems experienced were with the print function as explained above. The technique used was to compose a musical score and test each function systematically. This revealed no further errors. 39

46 5.2 The Product The Simple Music Composer Having finished the implementation of the project, the Simple Music Composer has been built to a degree where most of the functions described in the specification have been completed. Unfortunately, due to the tight time constraints on the project some of the non-vital functions have been omitted, but these are concerned with constructing musical notation rather than the MIDI system. This means that an evaluation of the uses of Java Sound in the Simple Music Composer can still take place and the MIDI functions involved be explained. Figure 5.2 shows the GUI of the Simple Music Composer. Figure 5.2 The GUI of the implemented Simple Music Composer The key features that were implemented in the final application are: A composition can be constructed so that the musical notation for the composition is displayed on screen. 40