Communication Architectures and Algorithms for Media Mixing in Multimedia Conferences

Transcription

1 Communication Architectures and Algorithms for Media Mixing in Multimedia Conferences P. Venkat Rangan, Harrick M. Vin and Srinivas Ramanathan Multimedia Laboratory Department of Computer Science and Engineering University of California at San Diego La Jolla, CA fvenkat, vin, Phone: (619) , Fax: (619) Abstract Advances in computer and communication technologies have stimulated the integration of digital video and audio with computing, leading to the development of computer-assisted multimedia conferencing. We address the problem of media mixing which arises in tele-conferencing applications such as tele-orchestra. We present a mixing algorithm which minimizes the difference between generation times of the media packets that are being mixed together in the absence of globally synchronized clocks, but in the presence of jitter in communication delays on packet switched networks. The algorithm is shown to be complete: given that there are no other message exchanges except media data between mixers and media sources, there cannot be any other algorithm that succeeds when our algorithm fails. Mixing can be accomplished by several different communication architectures. In order to support applications such as tele-orchestra, which involve a large number of participants, we propose hierarchical mixing architectures, and show that they are an order of magnitude more scalable compared to purely centralized or distributed architectures. Furthermore, we present mechanisms for minimizing the delays incurred by mixing in various communication architectures. We have implemented the mixing algorithms on a network of workstations connected by Ethernets, and have experimentally evaluated the performance of various mixing architectures. These experiments revealed interesting results such as the maximum number of participants that can be supported in a conference. To appear in IEEE/ACM Transactions on Networking, Vol. 1, No. 1, February, 1993 i

2 1 1 Introduction 1.1 Motivation Until recently, voice, video, and data communications have been handled by different communication networks. The primary carrier of voice has been the public switched telephone network, video has been transmitted by cable TV, and data has been handled by specialized computer networks (LANs and WANs). This historical separation can be attributed to fundamental differences in characteristics of voice, video and data traffic. Voice and video characteristics such as sensitivity to delay, high bandwidth requirement, and the ability to tolerate relatively high error rates are in complete contrast with requirements of data. Hence, voice and video transmission use circuit switching of analog signals, whereas data transmission typically uses digital packet-switched networks. Interest in integrating these different media has been stimulated by advances in communication and computer technologies. Advances in communication technology have made available large bandwidth at modest cost, whereas advances in computer technology have resulted in the development of high-performance workstations with audio and video capabilities. These advances have led to the feasibility of supporting many multimedia applications on computer systems. One such class of applications is multimedia conferencing, which permits individuals to carry out collaborative activities by exchanging audio, video, and text information through multimedia workstations. There are various kinds of conferences, among which the most demanding ones (in terms of performance) are those in which video and voice streams are being continuously transmitted by individuals in the conference, and the individual video images and voice streams have to be mixed together to obtain a composite image and audio stream. An example of such an application is a tele-orchestra, in which a large number of performers are continuously playing their respective musical instruments, each at his or her own multimedia workstation. The audio streams being received from all the performers need to be continuously mixed so as to yield a coherent musical composition, and then broadcast to the entire audience (which may also consist of individuals receiving audio at their respective workstations). The techniques for mixing depend upon the type of the media as well as the application. In the case of audio, mixing multiple streams involves digitally summing the audio samples and then normalizing the result. In order to eliminate one s own feedback, each participant, on receiving a mixed audio packet, removes his own contribution before playing it back. In the case of video, whereas it is possible to simply juxtapose the display of multiple video streams without having to mix them, applications such as tele-virtual conferencing may benefit from the mixing of individual video streams to synthesize the image of a virtual meeting room. The need to continuously mix media streams from several sources, coupled with the real-time nature of the media poses special requirements for media transmission protocols. The design of algorithms, protocols, and communication architectures for supporting media mixing in multimedia conferences is the subject matter of this paper.

3 2 1.2 Related Work In recent years, there have been many efforts towards integrating multimedia conferencing into computer systems. Lantz [3] proposes a text/graphics conferencing architecture where the basic goal is to fit into existing systems with minimal impact on the rest of the system. Sarin and Greif [6] have studied conferencing architectures for text and graphics, but their system does not include audio. Aguilar et al. at SRI [1] have proposed architectures for person-to-person voice conferencing. Ziegler et al. [8] present an analytical performance evaluation of voice conferencing on broadcast and ring networks. Ahuja et al at Bell Laboratories [2], and Swinehart et al. at Xerox PARC [4] have proposed architectures for video conferencing. However, the emphasis of all of most of these efforts has been person-to-person telephony-type conferences, and high performance applications such as teleorchestra which require continuous time-critical mixing of media streams from several sources are not supported. Algorithms, protocols, and communication architectures for supporting media mixing have not received much attention. 1.3 Research Contributions of This Paper In this paper, we address the problem of mixing media streams transmitted by multiple sources in a multimedia conference on a packet-switched network. Owing to the real-time nature of the media being mixed, packets from different sources generated at about the same time are required to be mixed together. More precisely, for coherent reproduction of audio and video in applications such as tele-orchestra, or for maintaining consistent ordering of user-triggered events in distributed real-time games, the difference between the generation times of packets being mixed must be minimized. Reduction in generation time differences also reduces the waiting time of packets during mixing, leading to lower buffering requirements. We present a mixing algorithm which minimizes the difference between generation times of the media packets that are being mixed together in the absence of globally synchronized clocks 1, but in the presence of jitter in transmission delays. The algorithm is shown to be complete: given that there are no other message exchanges except media data between a mixer and the media sources, there cannot be any other algorithm that succeeds when our algorithm fails. Mixing can be accomplished by several different communication architectures. In order to support applications such as tele-orchestra, which involve a large number of participants, we propose hierarchical mixing architectures, and show that they are an order of magnitude more scalable compared to purely centralized or distributed architectures. Furthermore, live performances such as tele-orchestra require that each performer receives the audio played by other performers almost instantaneously. Towards this end, we present mechanisms to minimize the real-time delays incurred by mixing in various communication architectures. We have implemented the mixing algorithms on a network of workstations connected by Ethernets, and have experimentally evaluated the performance of various mixing architectures. These experiments revealed interesting results such as the maximum number of participants that can be supported in a conference. 1 In future integrated networks, media capture and display sites that digitize and packetize media data may be connected directly (as opposed to via a host computer) to the network, e.g., digital HDTV, ISDN Telephone, etc. Such devices may not be capable of running sophisticated time synchronization protocols.

4 3 The rest of the paper is organized as follows: In Section 2, we formulate the problem of mixing. In Section 3, we present algorithms for mixing two media sources, and in Section 4, we extend them to more than two sources. In Section 5, we analyze various communication architectures for mixing, and in Section 6, we present mechanisms for minimizing real-time delays due to mixing. Section 7 presents our experience with implementing the mixing algorithms, and finally, Section 8 concludes the paper. 2 Problem of Media Mixing in Conferencing In its simplest form, a multimedia conference is established by setting up logical media communication channels among a group of participants. Once a conference is established, media information can be exchanged among its participants. In a multi-party conference, each participant must receive the composite media stream obtained by mixing the media streams transmitted by all the other participants. In a packet switched network environment, an important problem is to determine the set of packets from different sources that are to be mixed to form a composite media packet. We call such a set of packets as the Fusion Set. If each source generates media packets at a constant rate with a period of p, then the smallest time window during which two independent sources are guaranteed to generate packets is p 2. In general, when there are m sources, the smallest time window during which all the m sources are guaranteed to generate packets is p? p. (The proof is by a simple induction on the number of sources). If m is unbounded, the smallest 2 (m?1) window during which we are guaranteed to have packets from each of the sources is p. This is formalized by the following mixing rule for a multi-party conference: Mixing Rule: Packets n 1 and n 2 can belong to a Fusion Set (i.e., can be mixed together) if and only if jg(n 1 )? g(n 2 )j p (1) where p is the period of generation of media packets at a source, and g(n 1 ) and g(n 2 ) are the generation times of packets n 1 and n 2 at their respective sources. The fusion set can be determined in a straight-forward manner when the clocks of the generating sources are globally synchronized. However, clock synchronization requires sophisticated protocols for negotiation and agreement among media sources. We assume that the media sources can digitize, packetize, and transmit data on the network, but may not have the sophistication to exchange control messages and run complex protocols. Hence, any protocol that requires the exchange of messages other than media data is out of question. Furthermore, the overhead of clock synchronization protocols may be excessive compared to simpler protocols that are sufficient for determining a fusion set. Hence, determination of a fusion set in the absence of globally synchronized clocks is an important problem that needs to be solved in multimedia conferencing.

5 4 3 Algorithms for Media Mixing Consider a multimedia conference in which there are m sources S 1, S 2,..., S m (see Figure 1). Each source generates media packets at a constant rate with period p, and transmits them to a mixer M. Let the communication delay from a source to a mixer, which includes transmission times on the network and queuing delays at intermediate nodes, be bounded between min and max. If s is the arrival time at the mixer of a packet numbered n s generated by a source S s, then the earliest and the latest possible generation times (denoted by gs e (n s) and gs l (n s), respectively) of packet n s can be derived as: g e s (n s) = s? max (2) gs l (n s) = s? min (3) Measured on the mixer s clock, the generation time g s (n s ) of a media packet n s at source S s belongs to the interval [gs e (n s); gs l (n s)], which is referred to as the generation interval of packet n s. Arrival times Packet numbers τ1 τ 2 M τs τm n1 S 1 n2 S2 n s Ss n m S m Figure 1: Mixing in a multimedia conference. Mixer M mixes packets from sources S 1 ; :::; S m A straightforward mechanism for the mixer M to carry out mixing is to buffer the media packets it receives from sources S 1, S 2,..., S m, until at least one packet is received from each of them, at which time, M can combine all the packets and transmit the mixed packet. By the Mixing Rule, the separation in the generation times of packets (of different sources) being mixed together can be at most p. In the presence of communication delay jitter, two packets generated p apart (at two different sources) may suffer delays of max and min, respectively, and arrive at the mixer p + ( max? min ) apart, which then is the maximum waiting time of a packet in the mixer s buffers before being mixed. However, such a buffer and mix scheme has several disadvantages: Due to a quirk of the effect of jitter, two packets generated as far apart as p + 2 ( max? min ) may also arrive at the mixer within a window of p + ( max? min ), thereby misleading the mixer to combine those two packets in violation of the mixing rule. Higher the jitter (as in wide area networks), greater is the extent of the violation. This problem gets further compounded in the presence of packet losses. For instance, while mixing packets from three sources S 1, S 2, and S 3, packet loss may cause the mixer to derive fusion sets in a pairwise manner for

6 5 fs 1 ; S 2 g and fs 2 ; S 3 g, each of which may contain packets maximally separated in their generation times (i.e., by p + 2 ( max? min )). In order to mix packets from all three together, if the mixer takes a union of these pairwise fusion sets, packets separated as far apart as 2 (p + 2 ( max? min )) may get mixed together, and this separation may increase linearly with the number of sources. On the other hand, if the mixer decides to derive a new fusion set containing more closely generated packets from all the three sources, it may be flouting the requirement of maintaining consistent relative ordering of packets in successive fusion sets. We now propose a mixing algorithm which guarantees that the packets being mixed together always satisfy the Mixing Rule, first for the case when there are only two media sources (i.e., a binary mixing), and then extend it to a case when there are multiple media sources (i.e., a M-ary mixing). The algorithm is robust towards packet losses on the network. Hence, even in the absence of arrival of a current packet from a source, the mixer can still derive a fusion set with the help of any earlier packet received from that source. Thus, the algorithm effectively decouples the determination of fusion sets from live mixing of packets. An extreme variation of this situation is the determination of fusion sets from packets transmitted serially (rather than, concurrently) by the various sources, which finds an important application in live performances. In particular, a live performance generally has an initial sound-check period preceding its commencement, during which a mixer can sequence through the various performers/performing instruments prompting each of them to transmit a audio packet, one at a time. At the end of the sound-check, the mixer computes the fusion set (in addition to, of course, carrying out routine adjustments such as volume level, tone frequency, etc.), and keeps it for later use during the actual performance. 3.1 Binary Mixing Algorithm Let the arrival times at a mixer of packets n 1 and n 2 from two sources S 1 and S 2 be 1 and 2. The earliest and the latest possible generation times of packets n 1 and n 2 are: g e 1 (n 1) = 1? max g l 1 (n 1) = 1? min g e 2 (n 2) = 2? max g l 2 (n 2) = 2? min Since the generation times of successive packets from the same source are displaced by p, the generation interval of packet n s + k from source S s can be estimated as: g e s (n s + k) = g e s (n s) + k p (4) g l s (n s + k) = g l s (n s) + k p (5) Since the exact generation times of packets are unknown at a mixer, determination of a binary fusion set will have to be based on the above estimates of the earliest and the latest packet generation times. If g l max =

7 6 max(g l 1 (n 1); g l 2 (n 2)) and g e min = min(ge 1 (n 1); g e 2 (n 2)), then g e min g 1 (n 1 ); g 2 (n 2 ) g l max and jg 2 (n 2 )? g 1 (n 1 )j gmax l? gmin e. Consequently, if g l max? g e min p (6) then, the generation times of packets n 1 and n 2 are guaranteed to satisfy the Mixing Rule, and hence, packets n 1 and n 2 belong to a binary fusion set. Equation (6) defines a stronger condition than the Mixing Rule for determining a fusion set. Nevertheless, since the exact generation times of packets are unknown at a mixer, determination of a fusion set will have to be based on Equation (6) rather than on the Mixing Rule. Hence, in the remainder of this paper, we use Equation (6), referred to as the Strong Mixing Rule, as the condition for determining a fusion set. The maximum uncertainty, gmax l? gmin e in estimated generation times is at least as large as the generation intervals of n 1 and n 2, each of which equals max? min in duration. Hence, g l max? ge min max? min Clearly, if the jitter in communication latency, max? min exceeds p, (gmax l? ge min ) will also exceed p, and the generation intervals of n 1 and n 2 cannot satisfy the Strong Mixing Rule (Equation (6)). In fact, since the generation interval of every packet equals the transmission jitter in duration, if this jitter exceeds p, no two packets from S 1 and S 2 can satisfy the Strong Mixing Rule. Hence, max? min p is a necessary condition for determining a binary fusion set. This is the first result of importance, and is formally stated as Proposition 1 in Table 1. On the other hand, suppose that the jitter max? min p. If it is also the case that gmax l? gmin e p, then packets n 1 and n 2 form a binary fusion set. However, if max? min p, but gmax l?gmin e > p, then it may seem likely that a binary fusion set cannot be determined. Surprisingly, this is not true; even when g l max? ge min > p, there are situations in which a mixer can determine a binary fusion set. This is because, even though the generation intervals of n 1 and n 2 do not satisfy the Strong Mixing Rule, those of two other packets n1 0 and n0 2 may do so. However, notice that if packets n 0 1 and n0 2 satisfy the Strong Mixing Rule, so will packets n 1 and n (n 1? n 0 1 ) (because, the generation intervals of successive packets from the same source are displaced by p). Hence, in order to determine a binary fusion set, it suffices to search for a packet from S 2 that can form a binary fusion set with n 1. We shall now derive exactly those cases in which a mixer can determine a binary fusion set even when g l max? g e min > p. Without loss of generality, let us assume that g l 1 (n 1) > g l 2 (n 2) and g e 1 (n 1) > g e 2 (n 2). Depending on the relative positions of the generation intervals of n 1 and n 2, two possible scenarios arise: The generation intervals of n 1 and n 2 overlap (see Figure 2): In such a case, no matter which packet (n 2 + i); i 2 [?1; 1] from S 2 is considered, by comparing the generation intervals of n 2 + i (derived by substituting s = 2 and k = i in Equations (4) and (5)) and n 1, it can be shown that n 2 + i and n 1 do not satisfy the Strong Mixing Rule. This is the second result of importance, and is formalized as Proposition 2 in Table 1.

8 7 g e (n ) l 1 1 g (n ) 1 1 g e 2 (n 2 ) g l 2 (n 2 ) Time Figure 2: Packets with overlapped generation intervals g e 2 (n ) 2 gl (n 2 2 ) ge (n 1 1) g l 1 (n 1 ) < p < k * p < p Time < (k + 2) * p Figure 3: Mixing of packets with disjoint generation intervals The generation intervals of n 1 and n 2 are disjoint (see Figure 3): In this case, their minimum separation, g1 e (n 1)?g2 l (n 2) is bound to be positive, and let k 1 be the smallest positive integer such that (k? 1) p g1 e (n 1)? g2 l (n 2) < k p. Their maximum separation, on the other hand, is given by: g1 l (n 1)? g2 e (n 2) = (g1 l (n 1)? g1 e (n 1)) + (g1 e (n 1)? g2 l (n 2)) + (g2 l (n 2)? g2 e (n 2)) Since g1 l (n 1)? g1 e (n 1) = g2 l (n 2)? g2 e (n 2) = max? min, we obtain, g1 l (n 1)? g2 e (n 2) = 2 ( max? min ) + (g1 e (n 1)? g2 l (n 2)) Since max? min p, we get: (k? 1) p g1 l (n 1)? g2 e (n 2) < (k + 2) p (7) Elaboration of Equation (7) yields three sub-scenarios: 1. (k?1)p g1 l (n 1)?g2 e (n 2) < k p: In this case, by comparing the generation intervals of n 2 +k?1 and n 2 +k (derived using Equations (4) and (5)) with that of n 1, it can be shown that both (n 2 +k?1) and (n 2 + k) form fusion sets with n 1. We would be utilizing this result in deriving a comprehensive binary mixing algorithm, and is formally stated as Proposition 3 in Table k p g1 l (n 1)? g2 e (n 2) < (k + 1) p: In this case, by comparing the generation interval of n 2 + k (derived again by using Equations (4) and (5)) with that of n 1, it can be shown that n 2 + k forms a fusion set with n 1. This will also be utilized in the binary mixing algorithm, and is formalized as Proposition 4 in Table (k +1)p g1 l (n 1)?g2 e (n 2) < (k +2)p: In this case, no matter which packet (n 2 +i); i 2 [?1; 1] from S 2 is considered, a comparison of the generation intervals of n 2 + i and n 1 shows that they do not satisfy the Strong Mixing Rule. This impossibility result is formally stated as Proposition 5 in Table 1.

9 8 Proposition 1 If the jitter in communication latency, max? min > p, then it is not possible for a mixer to derive a binary fusion set. Proposition 2 If the generation intervals of n 1 and n 2 as estimated by the mixer are such that they are overlapping and gmax l? gmin e > p, then it is impossible for the mixer to determine a binary fusion set. Proposition 3 If the generation intervals of packets n 1 and n 2 as estimated by a mixer are such that, for some integer k, k 1, (k? 1) p g e 1 (n 1)? g l 2 (n 2) < k p and (k? 1) p g l 1 (n 1)? g e 2 (n 2) < k p n 1 can form a binary fusion set with either packet (n 2 + k? 1) or packet (n 2 + k). We refer to fn 1 ; n 2 + k? 1g as the lower binary fusion set and fn 1 ; n 2 + kg as the upper binary fusion set. Proposition 4 If the generation intervals of packets n 1 and n 2 as estimated by a mixer are such that, for some integer k 1, (k? 1) p g e 1 (n 1)? g l 2 (n 2) < k p and k p g l 1 (n 1)? g e 2 (n 2) < (k + 1) p then n 1 can form a binary fusion set with packet (n 2 + k). Proposition 5 If the generation intervals of packets n 1 and n 2 as estimated by a mixer are such that, for some integer k 1, (k? 1) p g e 1 (n 1)? g l 2 (n 2) < k p and (k + 1) p g l 1 (n 1)? g e 2 (n 2) < (k + 2) p and then, it is impossible for the mixer to determine a binary fusion set of packets from the two media streams. Table 1: Binary mixing propositions; rigorous mathematical proofs of these propositions have been omitted for the sake of brevity, and can be found in [5].

10 9 Propositions 1-5 (see Table 1) yield powerful techniques which can be combined together into the following algorithm that a mixer can execute to determine a binary fusion set when it receives packets from two media sources. Binary Mixing Algorithm: Suppose that packets n 1 and n 2 arrive at the mixer at times 1 and 2, respectively. Let max and min be the maximum and minimum possible transmission delays of packets on the network between a media source and the mixer. 1. If max? min > p, then a binary fusion set cannot be determined (Proposition 1). 2. If max? min p, compute the generation intervals of packets n 1 and n 2 to be [g1 e (n 1); g1 l (n 1)] and [g2 e (n 2); g2 l (n 2)], respectively, where, gs e (n s) = s? max, gs l (n s) = s? min, and s 2 [1; 2]. Without loss of generality, let gmin e = min(ge 1 (n 1); g2 e (n 2)) = g2 e (n 2) and gmax l = max(gl 1 (n 1); g2 l (n 2)) = g1 l (n 1). (a) If g1 l (n 1)? g2 e (n 2) p, then the binary fusion set is fn 1 ; n 2 g (by Equation (6). (b) If the generation intervals of packets n 1 and n 2 overlap, and g1 l (n 1)? g2 e (n 2) > p, then it is impossible for the mixer to determine a binary fusion set (by Proposition 2). (c) If the generation intervals of packets n 1 and n 2 are disjoint, and there exists an integer k 1 such that (k? 1) p g1 e (n 1)? g2 l (n 2) < k p, then: i. If (k?1)p g1 l (n 1)?g2 e (n 2) < kp, then two binary fusion sets are possible: fn 1 ; n 2 +k?1g, which is the lower binary fusion set, and fn 1 ; n 2 + kg, which is the upper binary fusion set (by Proposition 3). ii. If k p g1 l (n 1)?g2 e (n 2) < (k +1)p, then the binary fusion set is fn 1 ; n 2 +kg (by Proposition 4). iii. If (k + 1) p g1 l (n 1)? g2 e (n 2) < (k + 2) p, then it is impossible to determine a binary fusion set (by Proposition 5). 3.2 Completeness of Binary Mixing Algorithm The effectiveness (i.e., the success) of the above algorithm in determining a binary fusion set critically depends on the relative values of the transmission jitter (i.e., max? min ), and the packet duration p. This is quantified in the following Theorem. Theorem 1 When a mixer executes the binary mixing algorithm on two packets n 1 and n 2 received at times 1 and 2, respectively, (1) if max? min p=2, the algorithm always produces a binary fusion set. (2) If p=2 < max? min p, the algorithm may succeed or fail in determining a binary fusion set. (3) If max? min > p, the algorithm fails to determine a binary fusion set. (4) The algorithm is complete: given that there are no other message exchanges except media data between a mixer and the media sources, there cannot be any other algorithm that succeeds when our algorithm fails.

11 10 Proof: Let the earliest and latest generation times of packets n 1 and n 2 be computed as: gs e (n s) = s? max, and gs l (n s) = s? min, where s 2 [1; 2]. Again, without loss of generality, let g2 e (n 2) g1 e (n 1), and g2 l (n 2) g1 l (n 1). Hence, if the generation intervals of n 1 and n 2 overlap, then g2 e (n 2) < g1 e (n 1) < g2 l (n 2) < g1 l (n 1). Similarly, if the intervals are disjoint, then g2 e (n 2) < g2 l (n 2) < g1 e (n 1) < g1 l (n 1). The four assertions of the above theorem can be proved as follows: 1. max? min p=2: It may be observed that the lengths of the generation intervals (of packets n 1 and n 2 ), g1 l (n 1)? g1 e (n 1) and g2 l (n 2)?g2 e (n 2) are each equal to max? min, and hence neither of them exceeds p=2. Therefore, (g l 1 (n 1)? g e 1 (n 1)) + (g l 2 (n 2)? g e 2 (n 2)) p (8) The generation intervals can be either overlapped or disjoint. If the generation intervals overlap, we obtain that, gmax l? ge min (gl 1 (n 1)? g1 e (n 1)) + (g2 l (n 2)? g2 e (n 2)) p (9) Thus, a binary fusion set mixing can be derived using Equation (6). Suppose that, on the other hand, the generation intervals of packets n 1 and n 2 are disjoint; that is, g l 1 (n 1)? g e 2 (n 2) = (g l 1 (n 1)? g e 1 (n 1)) + (g e 1 (n 1)? g l 2 (n 2)) + (g l 2 (n 2)? g e 2 (n 2)) (10) Let k be an integer such that: Using Equations (8), (10), and (11) we obtain that, (k? 1) p g e 1 (n 1)? g l 2 (n 2) < k p (11) (k? 1) p g1 l (n 1)? g2 e (n 2) < (k + 1) p (12) Hence conditions of either Proposition 3 or Proposition 4 are satisfied, leading to the determination of a binary fusion set. 2. p=2 max? min < p: The lengths of the generation intervals of packets n 1 and n 2 will each lie between p=2 and p, and the generation intervals can be overlapped or disjoint. If they are overlapped, g1 l (n 1)? g2 e (n 2) will not exceed 2p, but can lie either between 0 and p in which case Equation (6) yields the binary fusion set, or between p and 2p in which case Proposition 2 shows that it is impossible to determine a binary fusion set. Suppose that, on the other hand, the generation intervals of packets n 1 and n 2 are disjoint, that is, g l 1 (n 1)? g e 2 (n 2) = (g l 1 (n 1)? g e 1 (n 1)) + (g e 1 (n 1)? g l 2 (n 2)) + (g l 2 (n 2)? g e 2 (n 2)) (13)

12 11 Let k 1 be an integer such that, (k? 1) p < g e 1 (n 1)? g l 2 (n 2) k p (14) Given that the lengths of generation intervals are no less than p=2 but are definitely less than p, Equation (13) and (14) yield: k p g l 1 (n 1)? g e 2 (n 2) < (k + 2) p Hence, either Proposition 4 is satisfied in which case a binary fusion set can be determined, or Proposition 5 is satisfied in which case it is impossible to determine a binary fusion set. Thus, the algorithm may succeed or fail in determining a binary fusion set. 3. max? min > p: By Proposition 1, it is impossible to derive a binary fusion set. 4. Completeness: The algorithm fails to determine a binary fusion set only when (a) max? min > p, or (b) p=2 max? min < p and, either the generation intervals are overlapped and g l max?ge min > p, or they are disjoint but (k + 1) p g l 1 (n 1)? g e 2 (n 2) < (k + 2) p, where k 1 is an integer such that, (k? 1) p < g e 1 (n 1)? g l 2 (n 2) k p. In case (a), Proposition 1 shows that it is impossible to determine a binary fusion set, and in case (b), Propositions 2 and 5 show the impossibility. Hence, there cannot be any other algorithm that succeeds in such cases, which goes to show that our binary mixing algorithm is complete under the networking assumptions stated in Section 2. 2 In the above theorem, the limitation to the effectiveness of the binary mixing algorithm arises due to the uncertainty in generation intervals, which is at most the jitter in communication delay. In practice, however, a mixer may be able to reduce the uncertainty (so as to be much below the jitter) by iteratively refining its estimates of the generation intervals, every time a new packet arrives. To see how, suppose that I = [g e (n s + i); g l (n s + i)]; 8i 2 [?1; 1] denote the generation intervals of packets n s + i from source S s, predicted from that of n s using Equations (4) and (5). Arrival of a subsequent media packet n 0 s from S s enables the mixer to recompute the generation intervals of n s +i from that of n 0 s to be: I 0 = [g e (n 0 s +i0 ); g l (n 0 s +i0 )], i 0 = (n s?n 0 s +i)), again using Equations (4) and (5). The actual generation instant of packet (n s + i) is guaranteed to be in both I and I 0, and hence, in? I \ I 0. Therefore,? I \ I 0 represents a revised and possibly more precise estimate of the generation interval for packet (n s + i). Surprisingly, the intersection is small when media packets (on which estimates are based) undergo widely different communication delays. In the limit, the exact generation time can be determined if the intersection reduces to a single point, which can occur when two media packets transmitted by a source suffer network delays of max and min, respectively.

13 12 4 M-ary Mixing In multimedia applications such as a tele-orchestra, there may, in general, be more than two media sources. In this section, we extend the binary algorithm to handle mixing of media packets received from more than two sources. The resulting M-ary mixing algorithm yields fusion sets such that the entire range of generation times of all the packets in a fusion set are bounded within a window of p (which is the shortest window possible, as shown in Section 2). Let us suppose that a mixer has to determine a fusion set of packets n 1, n 2,..., n m being received from sources S 1, S 2,..., S m. In a straight forward extension of the binary algorithm, the mixer can determine the fusion set for each of the pairs (S 1, S 2 ), (S 1, S 3 ),..., (S 1, S m ), and take the union of all of those (m? 1) binary fusion sets. If the maximum of the latest generation times, and the minimum of the earliest generation times of all the packets in the union are within a window of p, then the union constitutes a M-ary fusion set. However, recall that in the binary algorithm, there are scenarios when two binary fusion sets (lower and upper) are possible. It is likely that, choosing one of the two binary fusion sets can lead to the formation of a M-ary fusion set, whereas choosing the other binary fusion set does not. In the worst case, given that there are two alternatives for each binary fusion set, there are 2 m?1 alternatives for the unions of m? 1 binary fusion sets that may need to be computed in the determination of a M-ary fusion set, yielding exponential execution times. We will now present an algorithm that uses a window sliding technique to obtain a M-ary fusion set in a linear number of executions of the binary algorithm. 4.1 Algorithm for M-ary Mixing Let [g e 1 ; gl 1 ], [ge 2 ; gl 2 ],..., [ge m; g l m ] be the generation intervals of packets n 1, n 2,..., n m generated by m sources, respectively. Let g e min = g l max = min gs e s2[1;m] max gs l s2[1;m] Without loss of generality, let the generation interval of packet n 1 be disjoint from those of all the other packets (we can always replace packet n 1 by n 0 1, where (n0 1? n 1) is an integer sufficiently large such that the earliest generation time of packet n 0 1 exceeds the maximum latest generation time of packets n 2, n 3,..., n m ). Using the binary mixing algorithm, compute the binary fusion sets of packets from pairs of sources (S 1, S 2 ), (S 1, S 3 ),..., (S 1, S m ). Clearly, there must exist binary fusion sets for each of the above pairs of sources, if there exists a M-ary fusion set for the m media streams. Since the generation interval of n 0 1 is disjoint from those of all others, the determination of each of the binary fusion sets would require the use of Propositions 3 or 4. Let F = fn 0 1 ; n0 2 ; n0 3 ; :::; n0 mg be the set obtained as the union of binary fusion sets corresponding to packet n 0 1 from source S 1 using the propositions of Section 3, with the lower binary fusion set always being chosen in case of choice. If set F does not turn out to be a M-ary fusion set, it is only because the choice of lower binary fusion set for some packet n 0 s in F is incorrect; it must be set right by replacing packet n 0 s with (n 0 s + 1). We

14 13 refer to this correction technique as window sliding. There can be at most (m? 1) incorrect choices in the set F. The window sliding operation starts by considering the packet with the lowest generation interval, and terminates when either (1) a M-ary fusion set is formed, or (2) a packet for which there is no choice of binary fusion sets is encountered, or (3) all the (m? 1) choices have been corrected. The exact algorithm is as follows: M-ary Mixing Algorithm: 1. Let a mixer receive packets fn 1 ; n 2 ; ::::; n m g from m media sources. Let: g l max = Let k 1 be the smallest positive integer such that max gs l (n s) s2[2::m] g e 1 (n 1) + k 1 p > g l max Replace packet n 1 by packet n 0 1 = n 1 + k 1. Mark packet n 0 1 as FIXED. 2. 8s 2 [2::m] do: (a) Using the binary mixing algorithm, compute a binary fusion set for sources S 1 and S i, choosing the lower binary fusion set in case of a choice. If a binary fusion set cannot be determined, terminate the algorithm and report failure. (b) Let the binary fusion set be fn 0 1, n0 sg. Mark n 0 s as CHOOSABLE if there had been a choice of lower binary fusion set, else mark it FIXED. 3. (a) Compute gmax l and gmin e as follows: g l max = g e min = max gs l (n0 s ) s2[1::m] min gs e (n0 s ) s2[1::m] (b) If g l max? ge min p, then terminate the algorithm successfully and report the union fn0 1 ; n0 2 ; :::; n0 m g as a M-ary fusion set of the m media streams. (c) If gmax l? ge min > p, then perform the following window sliding operation: i. Without loss of generality, let g e min = ge s (n0 s ). If packet n0 s is marked FIXED, then terminate the algorithm and report failure. ii. If packet n 0 s is marked CHOOSABLE, then replace it by packet (n0 s + 1) in the union, and mark it as FIXED. iii. Recompute g e min and gl max for the union. If gl max? ge min successfully and report the union as the M-ary fusion set. p, then terminate the algorithm iv. Otherwise, go to step (i).

15 Completeness of M-ary Algorithm The correctness and completeness of the above algorithm in determining a M-ary fusion set is proved by the following theorem: Theorem 2 Suppose that a mixer receives packets n 1, n 2,..., n m from m media sources. The M-ary Mixing Algorithm produces a fusion set that satisfies the mixing rule (i.e. it is correct). Given that there are no other message exchanges between media sources and the mixer, there cannot be any other algorithm that produces a M-ary fusion set when our algorithm fails (i.e. it is complete). Proof: 1. Correctness: This is self-evident from the computation of the M-ary fusion set in which, the earliest and latest estimates of generation times of packets are guaranteed to satisfy gmax l? gmin e p, and hence satisfy Equation (6). 2. Completeness: Suppose that there is an algorithm that determines a fusion set, fn 0 1 ; n00 2 ; ::::; n00 m g containing packet n1 0. We will now show that our M-ary algorithm will produce the above fusion set. Since the new algorithm produces a fusion set, it can also produce binary fusion sets for pairs of sources, (S 1, S 2 ), (S 1, S 3 ),..., (S 1, S m ). By the completeness property of our binary mixing algorithm, our algorithm will also succeed in determining binary fusion sets. Let the union of all these binary fusion sets as determined by our algorithm be F = fn 0 1 ; n0 2 ; ::::; n0 mg. Since our binary algorithm always chooses lower binary fusion sets, we obtain that, 8s 2 [2; m], either n 0 s = n00 s or n 0 s = n00 s? 1. p g e s (n s ) e g (n ) 1 1 e g (n ) s s Union of binary fusion sets M-ary fusion set Figure 4: Window sliding in the M-ary algorithm If n 0 s = n00 s? 1, then it must be the case that n 0 s forms a lower binary fusion set with n 0 1 and is marked CHOOSABLE in our M-ary algorithm (on the other hand, n 00 s must form a upper binary fusion set with n1 0 ). Since their binary fusion sets are all lower, the earliest generation time of all such n0 s is guaranteed to occur below that of n 0 1, which is marked FIXED. Every window slide will replace a n0 s by n 00 s, and when the window slide operation reaches g1 e (n0 1 ) which is marked FIXED, the algorithm terminates after having (1) replaced all n 0 ss that are marked CHOOSABLE, and (2) performed at most m?1 window slides, yielding the set fn1 0 ; n00 2 ; ::::; n00 mg. Hence, if there is any algorithm that produces a M-ary fusion set, so will our M-ary mixing algorithm, which goes to show that it is complete. 2

16 15 Theorem 2 can be used to determine the media packet size p such that the mixing algorithm is effective in a given network environment: p 2 ( max? min ) However, in practice, p cannot be chosen to be very large, because, feedback delays in applications such as tele-orchestra are required to be small, and are bounded by the human response time. 5 Communication Architectures for Media Mixing The process of mixing passes through a sequence of two phases: (1) a transient phase, during which a fusion set is determined, and (2) a steady phase, in which media packets are mixed using the fusion set. Typically, the process of mixing enters the transient phase whenever a new source joins a conference, requiring that the fusion set that had been computed earlier for the older sources be changed to accommodate the new source. If the fusion set determined during a transient phase contains packets n 1 and n 2 from sources S 1 and S 2, respectively, then during the steady phase, 8k 0, packets (n 1 + k) and (n 2 + k) are mixed together. The architecture for communication among participants of a conference during the steady phase can be centralized at one end of the spectrum, or fully distributed at the other end of the spectrum. The centralized architecture requires that each participant in a conference transmit media information to a central mixer. The mixer receives packets from all the participants, creates a composite packet by mixing the received packets, and then transmits it to all the participants. Each participant, on receiving the composite media packet, may have to perform some media dependent processing of the composite packet (such as removing his own contribution in the case of audio) before scheduling it for playback. At the other end of the spectrum is the distributed architecture, which requires that each participant in a conference transmit media information to each of the other participants. Mixing is performed by each participant independently. However, it is possible that the fusion sets derived by the participants may be different. To resolve this conflict, one of the participants is designated as the master mixer, and the fusion set computed by this master is propagated to all the participants. Whereas the centralized architecture is simple to implement but inflexible (i.e., does not provide features such as autonomous volume control of each media stream), the distributed architecture is flexible but incurs duplication of mixing computation and bandwidth usage. Neither architecture scales well (with either the number of participants or the geographical separation between participants) if the network, the network interface, or the processing power at the mixer is the bottleneck. By clustering together closely situated participants, and using a hierarchical mixing architecture (see Figure 5), we can bound the bandwidth and processing requirements at the mixers [7]. In a mixing hierarchy, participants constitute the leaf nodes, and the mixers constitute non-leaf nodes. During the transient phase, the root mixer computes the fusion set (since it is the only node that receives packet information from all the participants), and propagates it to each of the intermediate mixers. During the steady phase, each mixer receives media packets from its children, mixes them, and sends the composite packet to its parent. The mixer that is at the root of the

17 16 hierarchy forwards the final mixed packet to each of the leaf nodes. The bandwidth required for packet reception at each mixer is proportional to the number of its children, whereas the bandwidth for packet transmission is that of sending to just one parent. (Even though the root mixer has to send a mixed packet to each of the participants, since the mixed packet is common to all the participants, the root mixer needs to make only one packet transmission by using multicasting). Thus, by increasing only the height of the hierarchy while bounding the number of children of each mixer, the hierarchical architectures can be made highly scalable. Root mixer : M Mixers : M1 Multicast to all participants M2 Participants : P1 P2 P3 P4 P5 Figure 5: A hierarchical architecture for mixing A special case of a hierarchical architecture is a directed ring, which can be thought of as a mixing tree in which each node has exactly one child. Such a configuration is appropriate for token ring based networks, and is analyzed by Ziegler et al. [8]. A generalization of the hierarchical architecture yields a graph-structured mixing architecture. In a nonhierarchical graph, there may be multiple paths between a participant and a mixer. Hence, a mixer may receive multiple mixed packets containing the same participant s packet. To eliminate the duplication, the participant s packet may have to be transmitted in addition to the mixed packet, leading to wastage of bandwidth. Since graphstructured architectures do not afford any special advantages over hierarchical ones, they are not very interesting for mixing. 6 Real-Time Performance of Mixing The interactive and real-time nature of collaborative applications require that the end-to-end delay experienced by media packets be bounded. When a mixer receives the first packet from one of the sources that goes to form a mixed packet, it has to delay the completion of the mixing process until it receives all other packets that constitute the mixed packet. However, if the network is unreliable, some of the packets that go to form a mixed packet may not arrive at the mixer. Hence, an important question is: how long should a mixer wait for packets from sources before deciding to transmit a partially mixed packet (which is not fully mixed because of the unavailability of packets from some of the sources)? A simple solution is for the mixer to transmit a partially mixed packet when a media packet that goes to form a subsequent mixed packet is received from one of the sources. However, this

18 17 causes mixing delays of the order of packet duration (p) at the mixer. Extension of this solution to a hierarchical mixing architecture of height H results in a mixing delay of p at each level, leading to an overall mixing delay of H p, and end-to-end delay of H p+(h +1) max (H max for the transmission delay from a leaf to the root up the hierarchy, and an additional max for the multicast of the final mixed packet from the root back to the leaf). The packet duration p cannot be chosen to be very small (typical values for voice packets on Ethernet are 20 to 150 ms) mainly to keep the packet transmission overhead low. Hence, even in small mixing hierarchies, the mixing delay will turn out to be unacceptably large for supporting interactive and real-time multimedia applications. We now present a simple algorithm that removes the proportional dependence of mixing delay on p in a general mixing hierarchy (of which, other mixing architectures are special cases). In this algorithm, each mixer maintains information about the expected generation times of packets at its children. When a new source S s joins the conference, it sends a probe packet n s up the hierarchy to enable each intermediate mixer to compute the earliest and latest generation times for the packet. If the packet reaches a mixer at height h at time (n s ), the mixer computes its earliest and latest generation times as follows: g e s (n s) = (n s )? h max g l s (n s) = (n s )? h min Consider the process of forming the kth mixed packet at the mixer. Since packets are generated regularly at an interval of p, the mixer can estimate the earliest and the latest generation times of packet (n s + k) from source S s as follows: given by: g e s (n s + k) = g e s (n s) + k p g l s (n s + k) = g l s (n s) + k p The minimum earliest and maximum latest generation times of packets constituting the kth mixed packet are g e min (k) = g l max (k) = min gs e (n s + k) s2f1;2;:::;mg max gs l (n s + k) s2f1;2;:::;mg The earliest and latest arrival times of packets constituting the kth mixed packet can be pre-computed as follows: e min (k) = h min + g e min (k) max l (k) = h max + gmax l (k) Hence, all packets constituting the kth mixed packet must arrive at a mixer at height h within the interval [min e (k); max l (k)]. At the root mixer, h = H (height of the entire tree), and max l (k) = gl max + H max. The maximum aggregate end-to-end delay suffered by a packet from a leaf to the root is given by, max l (k)? ge min (k). Adding max for the transmission of the final mixed packet from the root to participants, we obtain the maximum aggregate end-to-end delay of a packet as:

19 18 Aggregate End-to-End Delay = ((g l max + H max)? g e min ) + max p + (H + 1) max The above derivation of mixing delay has ignored the computational overhead of mixing at each mixer, which is assumed to be small compared to the communication and waiting delays. 7 Implementation and Experience We have implemented the mixing algorithms on a network of multimedia stations, each consisting of a computing workstation, a PC-AT, a video camera, and a TV monitor (see Figure 6). The workstation and PC-ATs are connected via Ethernets. The PC-ATs are equipped with digital video processing hardware that can digitize and compress motion video at real-time rates with a resolution of 480x200 pixels and 12 bits of color information per pixel, and audio hardware that can digitize voice at 8 KBytes/sec. Multimedia Station Multimedia Station Video Monitor Camera Video Monitor Camera Workstation PC-AT Workstation PC-AT GATEWAY ETHERNETS Figure 6: Hardware configuration We carried out several experiments to evaluate the performance limits of mixing in audio conferencing applications. Audio samples are packetized and transmitted on the Ethernet. In order to strike a balance between network transmission overhead (which favors large packet sizes), and the packetization delay (which favors small packet sizes), the audio packet size was chosen to be 512 samples, yielding p = ms. The maximum communication delay per packet is within 10 ms. We observed the performance of centralized and distributed mixing architectures, and experimentally measured the maximum number of participants that they can support. Figure 7 shows the variation of the fraction of packets reaching a mixer in a centralized architecture with increase in the size of a conference. When that fraction goes below 98%, there is a rapid deterioration of voice quality, and the mixing architecture breaks down. The break down point yields a maximum conference size of 20 in the presence of multicasting, and 12 in its absence for centralized architecture. In the presence of multicasting, distributed and centralized architectures behave in a similar fashion. However, in the absence of multicasting, their performance will be poorer due to the growth of bandwidth consumption as the square of the number of participants. For hierarchical architectures, given a maximum allowable end-to-end delay of about 100 ms, the analysis in Section 6 yields H 2, showing that they can support conferences with up to 20 2 = 400 participants.