Interactive Music Instruction with Java Objects Paul E. Dworak College of Music University of North Texas Denton, TX 76203

Size: px

Start display at page:

Download "Interactive Music Instruction with Java Objects Paul E. Dworak College of Music University of North Texas Denton, TX 76203"

Norma Lee
5 years ago
Views:

1 Interactive Music Instruction with Java Objects Paul E. Dworak College of Music University of North Texas Denton, TX When microcomputers became available for computer based instruction in the early 1980s, courseware authors were able to develop instructional materials for students in their college or university. With the current interest in distance learning, these same instructors need tools that enable them to provide the same interactive instruction remotely. This paper describes a set of Java classes that support applets for music computer based instruction. These classes enable students at any location to complete drills served from remote hosts, with the same interactivity that is possible if they were at a microcomputer at the host institution. Serving instructional materials developed in Java over the internet has several indisputable advantages: 1. It enables students with different platforms to use the same materials, since the student's web browser will interpret the Java code and control presentation of the materials. 2. It enables students in many locations to take advantage of pedagogical methods developed and used at other colleges and universities. Both are extremely important. Although most music instruction has historically been developed on the Apple, the Macintosh, and the PowerPC, more and more music applications are implemented on PC platforms using Windows 95. Developing new instructional applications for each platform, of course, takes a great deal of time, due to the idiosyncrasies of each operating system. Java handles the graphic user interface for each platform on which it runs. Consequently, an application developed in Java will look more or less the same on any platform. This saves the author substantial development time and makes software readily available to a larger number of users. Advantages of the Java Music Toolkit Applications served over the internet must be small to run quickly and effectively. Of course, music applications contain graphical images of scores, as well as sound files.

2 Both these elements can be quite large. Graphics ideally should be less than 30 KB to load quickly, but even at this size, the student will notice a significant delay if the network is busy. Since sampled sound files can be extraordinarily long, transporting them over the internet on demand is not much of an option for music CBI at present. Music applications can currently be served by preparing the score files in Finale and exporting the graphical image as a ".tif" file. This format can then be converted to a ".gif" file that can be served. A corresponding ".mid" file can contain the sounds represented by the score. An application that has many musical examples will need to serve each of these images and MIDI files separately. The Java Music Toolkit is a set of classes that reads compressed score files and converts them both to the graphical image and to a MIDI file that can be played. These files are very small--a set of five harmonic dictation exercises, each four measures long, fits in less than 4 KB. Consequently, an entire drill with all its component exercises, can be served in one step. The client will decode each score in the drill as the student needs it. The toolkit also enables the student to create scores that are solutions to dictation exercises. The classes that display scores are the same ones that enable the student to create a new score. The student could not interact in the same way with drills prepared using Finale. Disadvantages of Development with Java Although Java has been widely used, it still is not a mature product for music instruction. Java 1.02 was the standard until recently, and most web browsers supported applications written in this version. Sun Microsystems recently released version 1.1.4, which significantly improves event handling and provides a much larger suite of tools for the application developer. Unfortunately, existing web browsers cannot all run applets developed with this new version. Sun Microsystems also has not yet released the Macintosh version 1.1. Consequently, applications developed in this new release will not be widely useable until all web browsers support it. Part of the reason why Java can run on various platforms is that it transports its classes in unicode, and then interprets this code on each platform. This interpretation process is slow, and in some of the longer examples I developed, it took just as long to decode a score as it would to load its graphical image over the internet. Vendors are beginning to advertise "just in time" compilers that create native code from unicode. When these are widely available, the student will likely not experience delays in running Java applets. 2

3 Java currently only supports sound files in the Sun ".au" format. Sun Microsystems will soon be releasing a Media Player class set that supports both MIDI files and sampled sound files. The Java Music Toolkit has the capability of decoding score files into MIDI files that can be played with a web browser, but I chose to delay implementation of this feature until the Java Media Player becomes available. Classes in the Java Music Toolkit The Java Music Toolkit is divided into five groups of classes: 1. Score definitions: constants that represent types of staves, ties, notes, etc. 2. Score types: data structures for notes, chords, voices, etc. 3. Score drawing: the classes that draw images, staves, clefs, notes, etc. 4. Score canvas: the graphical object onto which items are drawn 5. Score data file: the file that stores the binary representation of scores in a drill The total size of all these classes, uncompressed, is less than 200 KB. Ideally, users of the Java Music Toolkit would download and install the classes before accessing any applet that used them. Structure of the Score Data File Scores are arrays of data structures, organized hierarchically, as shown below: Figure 1. Score data structures. Array of Scores Array of Systems Array of Staves Array of Voices Array of Staff Data Array of Voice Data Clefs, Keys, etc. Notes, Rests, etc. Each item in this data structure has a class associated with it. The class knows how to read the data it needs from the file and to convert the information into drawing commands and MIDI data. The ScoreTable is the top level object. public class ScoreTable extends Object public short NumberOfScores; // Total number of score public Score[] Scores; // Array of Score data public short ScoreDataAmount; // Number of score data entries 3

4 Each score in the array of scores contains the following data. public class Score extends Object public ScoreTraits Traits; // Score traits public short CurrentSystem; // Current system public short CurrentStaff; // Current staff public short CurrentVoice; // Current voice public int When; // Current time position in score public Point Where; // Current place position in score public int SystemTableOffset; // Offset to system table public class ScoreTraits extends ScoreData public boolean Wraps; // true if score wraps, false if it scrolls public boolean PlayChord; // true if chord played before score public boolean PlayPulses; // true if meter pulses played before score public boolean PlayTriad; // true if triad played before score public boolean PlayBass; // true if bass note played before score public boolean PlayRoot; // true if root played before score public boolean KeySignatureGiven; // true if key signature given (if shown or not) public boolean Modulates; // true if score modulates The score traits represent various features of the score: whether it scrolls or wraps at the edge of the display window, whether the tempo is set with metric pulses before the score is played, whether a tonic triad is first played, and so forth. The next five fields identify the position of the cursor in the score and, as a result, which system, staff, and voice have been selected. The SystemTableOffset is an offset to the location of the system table, which contains one or more systems resources and their data: public class SystemTable extends Object public short NumberOfSystems; // Number of SystemResource's in this table public SystemResource[] SystemResources; // Array of SystemResource data public short SystemDataAmount; // Number of system data entries Each system resource contains general information about the system: the rectangle that defines its position in the score; its traits, and offsets to the staff table and voice table. These are respectively, arrays of data that describe the score's background (its staff, clefs, 4

5 keys, meters, etc.) and arrays of data that describe the score's foreground (its beats, notes, rests, and chord symbols). public class SystemResource extends Object public Position SystemRect; // System rectangle's top and bottom public SystemTraits Traits; // Current attributes public int StaffTableOffset; // Offset to staff table public int VoiceTableOffset; // Offset to voice table public class SystemTraits extends ScoreData public byte Brace; // Type of brace for the system public boolean Invisible; // TRUE if system is NOT visible Background Data for the Score. The next data structure, StaffTable, is defined much like the previous tables. It includes the number of staves that are represented and an array of resources that contain general information about each staff in the system. public class StaffTable extends Object public short NumberOfStaves; // Number of StaffResource's in this table public StaffResource[] StaffResources; // Array of StaffResource data public short StaffDataAmount; // Number of staff data entries public class StaffResource extends Object public short StaffType; // Type of staff public Position StaffRect; // Staff rectangle's top and bottom public short NumberOfMeasures; // Number of measures on this staff public short Anacrusis; // Duration of pickup, if any public StaffTraits Traits; // Staff traits public int StaffDataTableOffset; // Offset to staff data table public class StaffTraits extends ScoreData public byte Beam; // Current type of beam public byte Stem; // Current type of stem The first general feature of a staff is its type: five line staff, one line staff for rhythms only, and roman numeral staves, which display chord analyses. By simply changing the staff type from FiveLineStaff to OneLineStaff, a melodic drill can be changed to a rhythm drill: only the rhythm of the score is displayed, without affecting the pitches that will be heard when the score is played. 5

6 The staff also has a rectangle that defines its location within its system and within the score. The staff resource also specifies the number of measures on the staff and the duration of the anacrusis, if any. Finally, the resource specifies various staff traits and contains an offset to the staff's data table. This table include the number of data items and an array of data items. public class StaffDataTable extends Object public short NumberOfItems; // Number of staff data items public StaffData[] StaffData; // The staff data items Each item of staff data includes a type, a timepoint (when it occurs in the score), and the data that defines the type. public class StaffData extends Object public short TypeID; // Identifies clef, key or meter data public int When; // Timepoint for this data public ScoreData KindData; // Key, clef, meter, tempo, mindur, barline data public class ScoreData extends Object public short Data; KeyData is one type of staff data. It describes the name class and accidental class of the key. This data structure also defines seven modes, and has a flag that indicates whether the key signature is shown or suppressed. public class KeyData extends ScoreData public byte NameClass; // Number representing the key's name class. // Same as those used for NameClass in the // TNoteData type public byte AccdClass; // Number representing the key's accidental. // Same as those used for AccdClass in the // TNoteData type public byte Mode; // Number representing the key's mode: // Ionian 0 Mixolydian 4 // Dorian 1 Aeolian 5 // Phrygian 2 Locrian 6 // Lydian 3 public boolean KeySignatureFlag; // 0 = no key signature // 1 = key signature public byte Number; // AccdClass << 3 + NameClass 6

7 Foreground Data for the Score. Each system in a score includes a voice table that identifies the number of voices in the system and includes an array of voice resources. Each voice resource includes the voice's traits and an offset to the beats that constitute the voice. public class VoiceTable extends Object public short NumberOfVoices; // Number of TVoiceResource's in this table public VoiceResource[] VoiceResources; // Array of CVoiceResource data public short VoiceDataAmount; // Number of voice data entries public class VoiceResource extends Object public VoiceTraits Traits; // Voice traits public int VoiceDataTableOffset; // Offset to VoiceDataTable The voice traits are flags that indicate whether beats and pitches are locked. Locking and unlocking voices or individual beats of a voice makes it possible to control what the student sees and what he or she must complete. public class VoiceTraits extends ScoreData public boolean BeatsLocked; // TRUE if beats are locked public boolean PitchesLocked; // TRUE if pitches are locked Like all other data tables, the VoiceDataTable includes the number of beats in a voice and an array of beats. public class VoiceDataTable extends Object public short NumberOfBeats; // Number of beats in the voice public BeatData[] Beats; // The beat data Each beat contains a variety of data: its duration, whether the duration value or pitch is given (and therefore locked), the type of stem and tie, the system and staff on which the beat is drawn, and the type of beat: note, rest, or chord analysis. public class PackedBeatData extends Object public short Data; public short Duration; // Beat's duration public ScoreData KindData; // The note, rest, and chord data 7

8 public class BeatData extends PackedBeatData public byte Stem; // The type of stem // 0 = down, 1 = up, 2 = default, 3 = none public boolean BeatGiven; // TRUE is beat is given public boolean NoteGiven; // TRUE if Note or Chord is given, // irrelevant if a rest public byte Tie; // The type of tie // 0 = none, 1 = tie, 3 = flipped tie public byte Kind; // Type of beat // 0 = note, 1 = chord analysis, // 2 = visible rest, 3 = invisible rest public byte SystemNumber; // The system on which the beat is to be // displayed. Ranges from 0 to MaxNumSystems. public byte StaffNumber; // The staff on which the beat is to be // displayed. Ranges from 0 to MaxNumStaves. Note data includes the note's name, accidental, and octave. In this particular implementation, it also includes a field that identifies the nonharmonic tone assigned to the pitch. Rests require only one field, which specifies the rest's position. Chord analyses include a field that specifies the roman numeral and one that identifies the figured bass symbol. public class NoteData extends ScoreData public byte NoteHead; // The type of notehead for durations // 0 = beat's value, 2 = black, 3 = white public byte NameClass; // Number representing the note name class. // This mod 7 representation for note names // assigns numbers to names based on their // location in the circle of fifths, thus: // F C G D A E B // public byte AccdClass; // Number representing the note's accidental class: // bb b n # x // public byte Octave; // The note's octave -- middle C begins octave 5. public byte NHT; // The nonharmonic tone: // 0 = None, 1 = UPT, 2 = UN, 3 = LN, 4 = APT, // 6 = Susp, 7 = Ret, 8 = Ant, 9 = ET, 10 = CT Displaying a Score Every score contains elements that constitute its foreground and background. Each element (clef, note, rest, etc.) has a timepoint in the score. This timepoint is associated with its physical location, or placepoint, on the screen. The placepoint of any timepoint will vary, depending on the contents of the score. A note preceded by an accidental 8

9 requires more space on the screen than does a note without an accidental. When chords are drawn, notes forming the interval of a second must be offset from one another, and accidentals also must be offset if they occur within the interval of the sixth. Consequently, the ScoreCanvas class must calculate the placepoint of each element of a score with respect to the context in which it occurs. To prepare for drawing the score, ScoreCanvas creates a list of ScoreEvents, which associates the timepoints and placepoints of each element of a score. Each score event also has a type, which ScoreCanvas uses to draw the element. To draw background elements, ScoreCanvas sends a draw message to a Staff object, identifying the type of element (clef, key signature, or meter) that this object should draw on the staff. To draw foreground elements, ScoreCanvas sends a draw message to a Voice object, requesting that it draw beats on the staff at the appropriate locations. These actions are summarized in the following code excerpt. To draw the current score events, ScoreCanvas iterates a list of score events to determine whether each event is a system event (a brace), a background event, or a foreground event. After identifying the event type, it calls the appropriate drawing routine. public void DrawCurrentEvents() throws IOException ScoreEvent ascoreevent; for (Enumeration seiter = ScoreEventList.elements(); seiter.hasmoreelements(); ) ascoreevent = (ScoreEvent) seiter.nextelement(); if (ascoreevent.geteventtype() < Opcode.CLEF) DrawSystems(aScoreEvent); else if (ascoreevent.geteventtype() < Opcode.BEAT) DrawScoreBackground(aScoreEvent, Color.black); else DrawScoreForeground(aScoreEvent, Color.black); Drawing the score background entails iterating the list of systems and staves that constitute the background and determining the physical location of each on the screen. DrawStaffData then invokes the appropriate drawing routine. ScoreCanvas has the ability to transpose the key signature of a score, so that one model score can be used to create several different examples, each in a different key. To draw the score foreground, ScoreCanvas initializes voice data based on the timepoint of the current score event. After calculating note offsets and accidental offsets for chords, and aligning notes vertically, it then invokes DrawVoiceData. public void DrawScoreForeground(ScoreEvent ascoreevent, Color color) throws IOException SetVoiceData(aScoreEvent); SetNoteOffsets(aScoreEvent); SetAccidentalOffsets(aScoreEvent); DrawVoiceData(aScoreEvent, color); 9

10 DrawVoiceData draws the score foreground by iterating the list of systems and voices that constitute the foreground. If a voice contains a beat at the timepoint specified by the score event, ScoreCanvas asks the voice to draw its beat. If not, the voice determines if its last sounding note needs a beam or tie, and if so, it draws the appropriate shape. By segmenting the score into a series of score events, ScoreCanvas is able to draw the score one beat at a time. Formatting the Screen for Music CBI Drills Java enables the application developer to specify various layouts for a window, so that its components look more or less the same on different platforms. Components should not be placed at specific coordinates, but instead, relative to the top or bottom of a window. The demonstration screens in this presentation use the following layout: Figure 2. Sample Java Applet Layout. In this layout, the score that a student sees in a drill will be at the top of the applet window. The toolbar, which allows score editing, will be at the bottom. The controls that select the exercise to hear and the type of exercise to complete are located in the 10

11 center of the window. Certainly, any layout is possible. Java simply provides the author with the ability to specify the relative location of applet components. Graphic Java Toolkit Although Java provides classes for windows, buttons, list boxes, menus, and other graphical user interface components, it lacks the higher level components that simplify application development. David Geary has developed a class library called the Graphic Java Toolkit. This library enables the author to develop toolbars and image buttons that a student can use to enter select note durations, chords, and accidentals. These choices provide an intuitive means for students to select the components of a musical score that is their solution to a dictation drill. Java Music Toolkit Samples The following examples show how the Java Music Toolkit can be used for CBI application development. The applet window contains the solution for a rhythmic dictation drill. It also includes buttons that enable the student to select the next or previous example in a set of drills. The choice box enables the student to select rhythmic, melodic, or harmonic drills. Figure 3 shows a rhythm displayed on a one-line staff. 11

12 Figure 3. Rhythm Drill. Figure 4 shows the result of selecting the next rhythm drill, and it also shows the three types of drills that the student sees when selecting the choice box. 12

13 Figure 4. Types of Drills. The next figure shows two different melodies. Notice that beams are drawn horizontally to minimize aliasing. 13

14 Figure 5. Two melodies. 14

15 Figure 6 shows a harmonic example. Notice that each chord is identified with a roman numeral. Figure 6. Harmonic Example. The final example shows how a student might use the toolbar to identify chords. Clicking on one of the images in the toolbar makes that chord the current selection. Notice also that Java has balloon help, which enables the student to see what each entry in the toolbar represents. The applet window also can provide feedback on what action has just been completed. 15

16 Figure 7. Use of the Toolbar. Summary The Java Music Toolkit enables an author to create a suite of exercises that can be used by a student at a remote location. This set of tools not only makes it possible to deliver traditional instructional materials remotely, but it also enables a student to complete drills that he or she can to an instructor. This will be an important requirement for theory classes delivered over the internet. In addition, this toolkit enables an instructor to collect sets of student solutions that can be presented in the classroom, so that students can observe multiple solutions for a particular problem set and the advantages of each. The Java environment will certainly begin to mature during the next year, and as it does, the concepts discussed in this paper will become feasible for widespread implementation. Reference Geary, David. Graphic Java: Mastering the AWT. Mountain View, CA: Sun Microsystems, Inc.,

What s New in Finale 2008a for Windows

What s New in Finale 2008a for Windows Finale 2008a is a free maintenance update available to Finale 2008 Windows registered users. Full Read Me documentation with more information on fixes in the initial