TABLE OF CONTENTS Exhibit List... iv I. Mandatory Notices... 1 A. Counsel and Service Information... 1 B. Real Parties-in-Interest... 2 C. Related Mat

Transcription

1 UNITED STATES PATENT AND TRADEMARK OFFICE BEFORE THE PATENT TRIAL AND APPEAL BOARD FRIENDFINDER NETWORKS INC., STREAMRAY INC., WMM, LLC, WMM HOLDINGS, LLC, MULTI MEDIA, LLC, AND DUODECAD IT SERVICES LUXEMBOURG S.À.R.L. Petitioners v. WAG ACQUISITION, LLC Patent Owner Patent No. 8,122,141 Issue Date: February 21, 2012 Title: STREAMING MEDIA BUFFERING SYSTEM PETITION FOR INTER PARTES REVIEW UNDER 35 U.S.C AND 37 C.F.R ET SEQ.

2 TABLE OF CONTENTS Exhibit List... iv I. Mandatory Notices... 1 A. Counsel and Service Information... 1 B. Real Parties-in-Interest... 2 C. Related Matters... 2 II. Certification of Grounds for Standing... 3 III. Statement of Precise Relief Requested... 3 IV. Threshold Requirement for Inter Partes Review... 4 V. Statement of Reasons for Relief Requested... 4 A. Technical Introduction The 141 Patent State of the Prior Art... 7 B. Patents and Printed Publications Relied Upon Herein Hollfelder et al., Transparent Integration of Continuous Media Support into a Multimedia DBMS ( Hollfelder ) (Ex. 1002) Su, Continuous Media Support for Multimedia Databases ( Su ) (Ex. Case IPR ) i

3 3. International Standard ISO/IEC 11172, Information Technology Coding of Moving Pictures and Associated Audio for Digital Storage Media at Up to About 1,5 Mbit/s ( ISO ) (Exs ) U.S. Patent No. 6,389,473 to Carmel et al. ( Carmel ) (Ex. 1007) U.S. Patent No. 6,336,143 to Diedrich et al. ( Diedrich ) (Ex. 1012) C. Claim Construction Claims 1, 10, 19 and 24 Preambles Claims 1, 10, and 24 rate more rapid than the rate at which said streaming media is played back by a user Claims 6 and 15 said server does not maintain a pointer into a buffer established within said server Claims 8, 17 and 21 streaming media from a live source Claims 1 and 24 a sufficient number of media data elements in the media player for uninterrupted playback Claim 19 format capable of being served to users by said server D. Level of Ordinary Skill in the Art ii

4 E. Claim-By-Claim Explanation of Grounds for Unpatentability Ground 1: Claims 1-2, 5-7, 9-11, 14-16, 18-20, 24 and are unpatentable under 35 U.S.C. 103(a) as obvious over Hollfelder in view of Su Ground 2: Claims 3-4, 12-13, and are unpatentable under 35 U.S.C. 103(a) over Hollfelder in view of Su and ISO Ground 3: Claims 8, 17 and 21 are unpatentable under 35 U.S.C. 103(a) over Hollfelder in view of Su and Carmel Ground 4: Claims 19 is unpatentable under 35 U.S.C. 102(a)/(e) over Diedrich VI. Conclusion iii

5 EXHIBIT LIST Case IPR Exhibit Number Description 1001 U.S. Patent No. 8,122,141 to Price 1002 Hollfelder et al., Transparent Integration of Continuous Media Support into a Multimedia DBMS 1003 Su, Continuous Media Support for Multimedia Databases 1004 Information Technology Coding of Moving Pictures and Associated Audio for Digital Storage Media at Up to About 1,5 Mbit/s, Part 1: Systems 1005 Information Technology Coding of Moving Pictures and Associated Audio for Digital Storage Media at Up to About 1,5 Mbit/s, Part 2: Video 1006 Information Technology Coding of Moving Pictures and Associated Audio for Digital Storage Media at Up to About 1,5 Mbit/s, Part 3: Audio 1007 U.S. Patent No. 6,389,473 to Carmel 1008 Declaration of Dr. Nathaniel Polish in Support of Inter Partes Review of U.S F. Kozamernik: Webcasting The Broadcasters Perspective EBU Technical Review No. 282, March Prosecution history for U.S. Patent No. 8,122, Library and Archives Canada Theses Canada 1012 U.S. Patent No. 6,336,143 to Diedrich 1013 Curriculum vitae of Dr. Nathaniel Polish 1014 U.S. Patent No. 5,822,524 to Chen 1015 U.S. Patent No. 6,430,620 to Omura 1016 U.S. Patent No. 6,625,656 to Goldhor 1017 S. Boll et al., Intelligent Prefetching and Buffering for Interactive Streaming of MPEG Videos 1018 N. Polish, The Burstware Family of Protocols iv

6 I. MANDATORY NOTICES Case IPR A. Counsel and Service Information Counsel: Frank M. Gasparo (Reg. No. 44,700) (Lead) Jeffri A. Kaminski (Reg. No. 42,709) (Backup) Address: Venable LLP 1270 Avenue of the Americas, 24 th Floor New York, NY Phone and Fax: P: (212) , F: (212) Counsel: Kevin M. O Brien (Reg. No. 30,578) (Backup) Richard V. Wells (Reg. No. 53,757) (Backup) Matthew S. Dushek (Reg. No. 61,921) (Backup) Address: Baker & McKenzie LLP 815 Connecticut Ave NW, Washington, DC Phone and Fax: P: , F: Please send all correspondence to the lead counsel at the address shown above. Petitioners consent to service by at: FMGasparo@Venable.com, JAKaminski@Venable.com, and Duodecad_WAG@bakermckenzie.com. A Power of Attorney is filed concurrently herewith under 37 C.F.R (b). The Office is authorized to charge the fee set forth in 37 C.F.R (a) to Deposit Account No , and any other fees that might be due in connection with this Petition. 1

7 B. Real Parties-in-Interest Case IPR The real parties-in-interest for this Petition are: FriendFinder Networks Inc., StreamRay Inc., WMM, LLC, WMM Holdings, LLC, Multi Media, LLC, Various, Inc., Interactive Network, Inc., DataTech Global, LLC, DataTech Systems, LLC, Docler USA, LLC, Duodecad IT Services Luxembourg S.à r.l., Docler Holding S.à r.l., Gattyàn Family Irrevocable Trust (including Mr. György Gattyàn in his capacity as Grantor and Investment Advisor), Duodecad IT Services Hungary KFT, and Gattyàn Group S.à r.l. C. Related Matters WAG Acquisition, LLC has asserted U.S. Patent No. 8,122,141 (the 141 Patent ) (the subject of the present Petition) as well as U.S. Patent No. 8,327,011 (the 011 Patent ), U.S. Patent No. 8,364,839 (the 839 Patent ) and U.S. Patent No. 8,185,611 (the 611 Patent ) in the District of New Jersey as follows: WAG Acquisition, LLC v. Sobonito Investments, Ltd., No. 2:14-cv-1661 (D.N.J.) ( 141 Patent; 011 Patent; 611 Patent; and 839 Patent); WAG Acquisition, LLC v. Multi Media, LLC, No. 2:14-cv-2340 (D.N.J.) ( 141 Patent; and 011 Patent); WAG Acquisition, LLC v. Data Conversions, Inc., No. 2:14-cv-2345 (D.N.J.) ( 141 Patent; and 011 Patent); WAG Acquisition, LLC v. Flying Crocodile, Inc., 2:14-cv-2674 (D.N.J.) ( 141 Patent; 011 Patent; 611 Patent; and 839 Patent); WAG Acquisition, LLC v. Gattyàn Group S.à r.l, No. 2:14- cv-2832 (D.N.J.) ( 141 Patent; 011 Patent; 611 Patent; and 839 Patent); WAG Acquisition, LLC v. MFCXY, Inc., No. 2:14-cv (D. N.J.) ( 141 Patent; 011 2

8 Patent; 611 Patent; and 839 Patent); WAG Acquisition, LLC v. FriendFinder Networks Inc., No. 2:14-cv-3456 (D.N.J.) ( 141 Patent; 011 Patent; 611 Patent; and 839 Patent); and WAG Acquisition, LLC v. Vubeology, Inc., No. 2:14-cv-4531 (D.N.J.) ( 141 Patent; and 011 Patent). Contemporaneously with this petition for the 141 Patent, Petitioner is also filing a petition for Inter Partes Review of the 011 Patent, 611 Patent, and 839 Patent. II. CERTIFICATION OF GROUNDS FOR STANDING Petitioners hereby certify that the patent for which review is sought is available for Inter Partes Review and that petitioners are not barred or estopped from requesting an Inter Partes Review challenging the patent claims on the Grounds identified in the petition. III. STATEMENT OF PRECISE RELIEF REQUESTED Petitioners respectfully request that Claims 1-28 of the 141 Patent (Ex. 1001) be canceled based on the following Grounds of Unpatentability, set forth and explained in detail in the following sections: Ground 1: Claims 1-2, 5-7, 9-11, 14-16, 18-20, 24 and are unpatentable under 35 U.S.C. 103(a) as obvious over Hollfelder et al., Transparent Integration of Continuous Media Support into a Multimedia DBMS ( Hollfelder ) (Ex. 1002) in view of Su, Continuous Media Support for Multimedia Databases ( Su ) (Ex. 1003); 3

9 Ground 2: Claims 3-4, 12-13, and are unpatentable under 35 U.S.C. 103(a) as obvious over Hollfelder in view of Su and International Standard ISO/IEC ( ISO ) (Exs ); and Ground 3: Claims 8, 17 and 21 are unpatentable under 35 U.S.C. 103(a) over Hollfelder in view of Su and U.S. Patent No. 6,389,473 to Carmel et al. ( Carmel ) (Ex. 1007). Ground 4: Claim 19 is unpatentable under 35 U.S.C. 102(b) over U.S. Patent No. 6,336,143 to Diedrich ( Diedrich ) (Ex. 1012). IV. THRESHOLD REQUIREMENT FOR INTER PARTES REVIEW A petition for Inter Partes Review must demonstrate a reasonable likelihood that the petitioner would prevail with respect to at least 1 of the claims challenged in the petition. 35 U.S.C. 314(a). This Petition meets this threshold. All the elements of claims 1-28 of the 141 Patent are rendered obvious by the prior art as explained below in the proposed Grounds of Unpatentability. Specific motivation or suggestion to combine the references is provided. V. STATEMENT OF REASONS FOR RELIEF REQUESTED A. Technical Introduction The following technical introduction is supported by the Declaration of Dr. Nathaniel Polish ( Polish Decl. ; Ex. 1008) and as indicated. 1. The 141 Patent 4

10 The 141 Patent issued from U.S. Patent Application No. 12/800,152, filed on May 10, 2010 (the 152 Application ). The 152 Application claims priority to Provisional Application No. 60/231,997, filed on September 12, The effective filing date of the claims of the 141 Patent is no earlier than September 12, The 141 Patent is directed to methods and systems for buffering streaming media data over the Internet. 141 Patent at 1: The 141 Patent was subject to an Office Action during prosecution a Non-Final Rejection on November 24, Ex The examiner rejected the following claims: rejected Claims under 35 USC 101 as directed to nonstatutory subject matter; provisionally rejected Claims 1 and 10 for obviousness-type double patenting over Claims 1 and 8; rejected Claims 1, 2, 10, 11 and 24 as indefinite by reason of the use of the term identifiers; and rejected Claims 1-9, and under 35 USC 103(a) as unpatentable over the prior art. Ex The applicant responded by revising Claims 19-27, submitting a Terminal Disclaimer, and adding Claim 28. Ex The 141 Patent admits that sending audio and video files via a network was known in the art. Ex. 1001, 141 Patent at 4:1-2. The 141 Patent also admits that it was known for media data stored in a server to be sent over networks to a client buffer to assure a continuous stream of audio and video. Id. at 2: The 141 Patent further admits that it was known to use a pre-buffering technique so that the audio and video can be played with a minimum of dropouts, and that it was known to 5

11 transmit audio and video at the rate it is to be played back on the associated media player. Id. at 2: Indeed, the invention presumes the existence of a data communications transport mechanism, such as the TCP protocol, for the reliable delivery of data in an ordered sequence from the source of the media data to the server, or from the server to the media player software of the user computer. Thus, the delivery of data in the proper sequence is outside the scope of this invention. Id. at 5:5-11. Finally, the patent notes the existence of two types of encoding schemes Variable Bit Rate VBR, and Constant Bit Rate CBR... [where] [t]he standard encoding scheme used for streaming media is CBR.... Id. at 5: Against this backdrop of well-known techniques, the 141 Patent asserts that there was a need for improved systems and methods to afford immediate and uninterrupted listening/viewing of streaming media by the user. Id. at 4: According to the 141 Patent, these objectives are addressed by transmitting data from the server more rapidly than it is played out by the user system under conditions wherein the user s computer buffer is not full, at a rate faster than the playback rate. Id. at 9: In this way, the 141 Patent claims that the client s buffer will always be full and ready for uninterrupted playback to the user. Id. at 9: However, this description amounts to no more than simply using the wellknown pre-buffering technique to buffer media data elements at a faster rate than they are played back to the user. Id. at 2: In other words, this purported innovation 6

12 merely fills the buffer as quickly as possible so that playback can start or continue as soon as possible. As will be shown below, this approach to filling and maintaining a buffer was well-known in the prior art. The 141 Patent includes four independent claims (1, 10, 19, and 24), which are similar. Claims 1 and 10 recite that media data elements are assigned serial identifiers. A client sends requests to the server for the media data elements specifying the serial identifier of the requested element. The server provides the requested media data elements to the client. Claim 1 recites that the requests for the media data elements are made in order to maintain a sufficient number of media data elements in the media player for uninterrupted playback. Claim 10 recites that the server to send media data elements to the user system responsive to said requests, at a rate more rapid than the rate at which said streaming media is played back by a user. Claim 19 simply recites that the serial identified media data elements are stored in a format capable of being served to users by the server. Claim 24 recites that a media player maintains a record of the last media data element received and requests the next media data element in order to maintain a sufficient number of media data elements in the media player for uninterrupted playback. 2. State of the Prior Art Since at least the beginning of the 1990 s, digital transmission technologies had been introduced for digital audio broadcasting and digital video broadcasting. Ex. 1009, Kozamernik at pp The Internet was recognized as a de facto worldwide 7

13 network important for broadcasting activities having achieved more than 50% penetration in five major American cities and 50 million users in four years. Id. at pp. 3, 5. It was recognized that in contrast to conventional broadcasting, the Internet allowed the audience to interact with the originator and shape the content that is delivered. Id. at p. 5. Several protocols, including Real Time Streaming Protocol ( RTSP ) a popular application-level protocol known to enable controlled ondemand delivery of real-time audio and video stream were in use for providing realtime services via the Internet. Id. at p. 13. While download-first-and-then-play technology was considered acceptable for short program clips, streaming technology, which allowed for immediate playback, was preferred for online radio listening and watching video clips. Id. at p. 6. During media streaming, a media player, such as RealPlayer TM, read the media file stream as it was arriving from the network and began playing it before the rest of the file arrived. Id. In order to make the playback smooth, the player used a process of buffering. Id. As the player played out the file, it continued to collect packets in reserve so that if there were minor delays in receiving the packets, playback was still continuous. Id. Buffering was commonly known in the context of media streaming over the Internet. Transmission over the Internet may not be fully reliable and may suffer from delay jitter. Id. at pp Delay jitter is the variance in delay from end to end of one packet as compared to the subsequent packet. Id. Buffering was known to smooth out jitter to enhance playback quality. Id. at p.12. 8

14 A potential side effect of buffering is a short delay while the buffer fills. The 141 Patent contends that the user must wait for the length of the buffer. Ex. 1001, 141 Patent at 3:8-32. The 141 Patent specifically states that if the user waits ten seconds before playback starts, then there is enough buffer for ten seconds of interruptions and the buffer cannot increase during playback. Id. But this was not the state of the art. As explained by Kozamernik, streaming technology existing at the time of invention allowed for immediate playback. Ex. 1009, Kozamernik at p. 6. It was well-known to increase the transmission rate above the playback rate to fill the buffer in the first instance, and to refill the buffer to provide uninterrupted playback. See Polish Decl., Ex at Additionally, Hollfelder, published in 1997, describes a system where the server sends streaming media data at a rate faster than playback to the user, which is received by a client-side buffer. Ex. 1002, Hollfelder at 2.1 2, The client maintains a record of the data retrieved and placed into the buffer, and periodically requests additional data in order to maintain a continuous streaming presentation to the user. Id. at This is functionally identical to the method and system described in the 141 Patent and summarized above. Thus, as the prior art and the Declaration of Dr. Polish show, the purported problem and solution of the 141 Patent were already known and obvious to a person of skill in the art at the time of the alleged invention. B. Patents and Printed Publications Relied Upon Herein 9

15 1. Hollfelder et al., Transparent Integration of Continuous Media Support into a Multimedia DBMS ( Hollfelder ) (Ex. 1002) Hollfelder was published in GMD Technical Report, Nr. 1104, St. Augustin, Germany, in December 1997 and is therefore prior art under at least 35 U.S.C. 102(b). Hollfelder describes a system of continuous multimedia data delivery and media-oriented improvement of presentation quality according to user requirements. It was not cited during original prosecution. 2. Su, Continuous Media Support for Multimedia Databases ( Su ) (Ex. 1003) Su is a thesis that was submitted to the Department of Computing and Information Science at Queen s University, Ontario, Canada in September It was also published by and available at the National Library of Canada in 1999 and is therefore prior art under at least 35 U.S.C. 102(a). Su at p.1-2; see also Library and Archives Canada, Ex In the thesis, Su references Hollfelder. Su at p.86. Su describes a system of buffer management for continuous delivery of multimedia data from a database server to multiple destinations over a network. It was not cited during original prosecution. 3. International Standard ISO/IEC 11172, Information Technology Coding of Moving Pictures and Associated Audio for Digital Storage Media at Up to About 1,5 Mbit/s ( ISO ) (Exs ) ISO was published on August 1, 1993 and is therefore prior art under at least 35 U.S.C. 102(b). ISO specifies the international standard for the 10

16 encoding and decoding of audio and video streams, commonly known as MPEG-1. ISO was published as a three part document: Part 1: Systems ( ISO ) (Ex. 1004); Part 2: Video ( ISO ) (Ex. 1005); and Part 3: Audio ( ISO ) (Ex. 1006). It was not cited during original prosecution. 4. U.S. Patent No. 6,389,473 to Carmel et al. ( Carmel )(Ex. 1007) Carmel was filed on March 24, 1999 and issued on May 14, It is thus prior art under at least sections 102(a) and (e) and was not cited during prosecution. 5. U.S. Patent No. 6,336,143 to Diedrich et al. ( Diedrich ) (Ex. 1012) Diedrich was filed on Sept. 27, 1993 and issued on Jan. 1, It is therefore prior art under at least sections 102(a) and (e) was not cited during prosecution. C. Claim Construction In an Inter Partes Review ( IPR ), a claim receives the broadest reasonable construction in light of the specification of the patent in which it appears. 37 C.F.R (b). Petitioners propose, for purposes of this IPR only, that the claim terms of the 141 Patent take on their ordinary and customary meaning that the terms would have to one of ordinary skill in the art, unless specifically discussed below. This claim construction analysis is not a concession by Petitioners as to the proper scope of any claim term in any litigation. These assumptions are not a waiver of any argument in any litigation that claim terms in the 141 Patent are indefinite or otherwise invalid. 11

17 The district court in the above-mentioned lawsuits has not received any claim construction briefing. The constructions discussed below are for the Board s consideration and for purposes of this Petition only, in the manner outlined below Claims 1, 10, 19 and 24 Preambles Whether the preamble is determined to be limiting or not, Petitioner has specifically shown in the chart further below where in the prior art it exists. 2. Claims 1, 10, and 24 rate more rapid than the rate at which said streaming media is played back by a user Petitioner submits that the broadest reasonable construction of a rate more rapid than the rate at which said streaming media is played back by a user is a transmission rate faster than the rate at which the media is played by the user. Polish Decl., Ex at 32. This is supported in the specification: Connections from the server 12 through the Internet 10 commonly are much faster than the data rate required for audio or video playback..... * * * 1 Petitioners have argued in the litigations that Claims 1-18 and of the 141 Patent are unenforceable because those claims contain language such as the Internet struck by the patentee during prosecution. WAG Acquisition, LLC v. Multi Media, LLC, No. 2:14-cv-2340, ECF No at 6-8 (D.N.J. Jan. 15, 2015) (Brief in Support of Joint Motion to Dismiss Plaintiff s Complaints). Specifically, in response to an indefiniteness rejection by the Examiner directed at that phrase, the patentee agreed to delete such as the Internet to obtain allowance of the application leading to the 141 Patent. Ex. 1010, Examiner-Initiated Interview Summary (Sept. 29, 2011), Detailed Action Examiner s Amendment (Sept. 29, 2011). However, the 141 Patent still issued with that language, making the patent unenforceable. H-W Tech., LC v. Overstock.com, Inc., 758 F.3d 1329, 1335 (Fed. Cir. 2014). 12

18 Since the connection from the Internet to the user is faster than that required for media playback, audio/video data is transmitted from the server faster than it is played out by the user system, thus building up audio/video data in the user buffer. 141 Patent at 11:17-30, 11: Claims 6 and 15 said server does not maintain a pointer into a buffer established within said server There is insufficient written description to determine the scope of these claims that require the negative limitation: said server does not maintain a pointer into a buffer. Polish Decl., Ex at 33. The only mention of this phrase in the specification describes it as an equivalent to the media player maintaining a record of the identifier, the same language already stated in Claim 1. The server buffer manager does not maintain a pointer into the server buffer for each user. Instead, the media player buffer manager in the user computer maintains a record of the serial number of the last data element that has been received. Via the use of standard data communications protocol techniques such as TCP, the user computer transmits a request to the server to send one or more data elements, specifying the serial numbers of the data elements. 141 Patent at 8: To the extent the Board determines that this phrase can be construed for the purposes of this proceeding, Petitioner submits that the broadest reasonable construction is that the media player maintains a record of the identifier. 13

19 4. Claims 8, 17 and 21 streaming media from a live source Claim 8 recites streaming media from a live source. Under its broadest reasonable interpretation, media data elements provided from a recording of a live broadcast, including from a CD, would satisfy this limitation. 141 Patent, Ex at 10:60-65 (describing live source 26 as announcer speaking into a microphone, or playing a CD, the media source 26 ); see also Polish Decl., Ex at 34. Thus, this limitation cannot exclude media read from a storage medium as this would improperly exclude the embodiment of a live source discussed in the specification. See id. Accordingly, Petitioners submit that the broadest reasonable construction of media from a live source is media that originated from a broadcast. 5. Claims 1 and 24 a sufficient number of media data elements in the media player for uninterrupted playback The 141 Patent is silent as to what is or is not a sufficient amount of streaming media data elements for uninterrupted playback. Polish Decl., Ex at 35. The prosecution history similarly fails to describe any upper or lower bound for sufficient. Id.. Without such bounds, one of ordinary skill in the art would not have any way of determining whether a particular instrumentality is within the scope of the claims. See id. Accordingly, Petitioner submits that these claims, and all that depend on them, are indefinite. See, e.g., Nautilis v. Biosig, 134 S.Ct (June 2, 2014). 14

20 To the extent the Board determines that this phrase can be construed for the purposes of this proceeding, Petitioner submits that it should be given the broadest reasonable construction. 6. Claim 19 format capable of being served to users by said server The 141 Patent specification is silent as to what formats would be capable or incapable of being served to the user. Polish Decl., Ex at 36. The prosecution history similarly fails to describe any upper or lower bound for format capable of. Id. The specification merely states that there may be a means for: formatting media data according to the requirements of [the] buffer 14, without giving any description or example of an acceptable format. Ex. 1001, 141 Patent at 10:42-44, 12: Without such bounds, one of ordinary skill in the art would not have any way of determining whether a particular instrumentality is within the scope of the claims. See also Polish Decl., Ex at 37. Accordingly, Petitioner submits that these claims, and all that depend on them, are indefinite. See, e.g., Nautilis v. Biosig, 134 S.Ct (June 2, 2014). To the extent the Board determines that this phrase can be construed for the purposes of this proceeding, Petitioner submits that it should be given the broadest reasonable construction. 15

21 D. Level of Ordinary Skill in the Art Case IPR Petitioners submit that a person of ordinary skill in the art would have had the level of skill described in the Polish Decl., Ex at 23. Specifically, one of ordinary skill would have had a B.S. degree in computer science or electrical engineering (or comparable degree) and two years of experience in networking or streaming media, or a M.S. in computer science or electrical engineering (or comparable degree). Such descriptions are approximate, and a higher level of education or specific skill might make up for less experience, and vice-versa. E. Claim-By-Claim Explanation of Grounds for Unpatentability Claims 1-28 are unpatentable as shown in the detailed explanation below. 1. Ground 1: Claims 1-2, 5-7, 9-11, 14-16, 18-20, 24 and are unpatentable under 35 U.S.C. 103(a) as obvious over Hollfelder in view of Su. Claims 1-2, 5-7, 9-11, 14-16, 18-20, 24 and are unpatentable as obvious over Hollfelder in view of Su as shown in the text and chart below. Hollfelder discloses a server-client architecture in which streaming media data is delivered from a server to a client buffer over a network in the form of units (i.e. media data elements ) called Continuous Object Presentation Units (COPUs). Ex. 1002, Hollfelder , The client makes requests to the server and receives COPUs in response, which it stores in a client-side buffer. Id. at The associated media player application, loaded onto the user s system, displays the streaming media data for the user by retrieving the transmitted COPUs from the 16

22 client buffer. Id. at The server sends COPUs to the client at a rate faster than the media player presents the data to the user, ensuring continuous presentation through preliminary storage of upcoming COPUs in the buffer. Id. at 2.1 2, The client maintains a record of the COPUs retrieved and placed into the buffer, in order to determine which COPUs to request in order to continue buffering the data stream. Id. at 2.3 3, Fig. 2, ; Polish Decl., Ex at 62. The media data sent by the server may initially be stored in a relational database system, which encompasses storage in the disk file of Claims 9, 18 and 20. Id. at 2.3 3, Hollfelder s client-pull mechanism supports continuous presentation of the multimedia data by maintaining the size of the buffer, which corresponds to the recitation in Claims 1 and 24 that requests are made in order to maintain a sufficient number of media data elements in the media player for uninterrupted playback. Id. at 2.1 2, Fig. 2, Claims 1, 10 and 19 further require that each media data element be associated with a serial number, which the client uses for its request according to the record of media data elements already played and/or buffered. Claims 2 and 11 specify that these serial numbers must be sequential. The client-pull system of Su assigns all the COPUs of a continuous object indexes numbered from 0 to n-1, where n denotes the total number of COPUs. Ex. 1003, Su at p.15. Su s indexes are functionally equivalent to a sequential serial number. Polish Decl., Ex at Hollfelder teaches that the atomic unit requested during presentation is the COPU. 17

23 Ex. 1002, Hollfelder at Su further teaches that in the context of MPEG video presentation, the client sends a request to the server that specifies the frame number. Ex. 1003, Su at p Thus, to one of ordinary skill in the art, the combined teachings indicate that a COPU may be a frame in an MPEG stream, as each may be requested as a unit. Polish Decl., Ex at 59. The requests by the client in view of the combined teachings of Hollfelder and Su disclose the limitation of sending or receives a request for media data elements corresponding to specified serial identifiers in two ways. Id. First, a request for a COPU having a particular index discloses this limitation. Id. Second, a request for a media data element by frame number discloses this feature. Id. It would have been obvious for one of ordinary skill in the art to apply the sequential identifiers of Su to the COPUs in Hollfelder, in order to improve the efficiency of the client-side record of retrieved COPUs. Id. at 60. Sequential number values are an easily implemented and intuitive type of identifier, and are an ideal fit for the system of Hollfelder. Id. Claims 10 and 28 further require that the server assigns serial identifiers to the sequential media data elements. Su discusses that IBM DB2/UDB [IBAI97b] is used to store MPEG video at the server side. Ex. 1003, Su at 5, 22 (discussing how MPEG-1 video is composed of group-of-pictures (GoP)). The client initializes all its processes first. Then it sends a request to the server to initiate a presentation. Id. at 44. For the presentation, the server assign[s] a sequential number, called a GoP number, to each GoP. A sequential number is also assigned to each frame and is 18

24 denoted as a frame number. Id. at 43; Polish Decl., Ex at 61. The client then [t]o retrieve a frame sends a request to the server that specifies the frame number. The server retrieves the desired frame from its local buffer. If the desired frame is not in its local buffer, the server retrieves it from the database. The server then delivers the frame to the client over network. Ex. 1003, Su at It would have been obvious for a person of ordinary skill to combine Su s server-side assignment of identifiers with the server in Hollfelder, since the data elements must be assigned identifiers for the client-pull system to function, and it would be efficient to do so where the data is initially stored (the server). Polish Decl., Ex at 61. To the extent that the language such as the Internet in the preamble is considered limiting, Hollfelder teaches that the streaming media data are delivered from the server to a client over a network. Su describes a media player in its clientserver architecture which receives data via the Internet. Ex. 1003, Su at p.21. Su also teaches a client-pull mechanism in which COPUs are delivered from a server to a client. Id. at pp.15, 21. It would have been obvious for a person of ordinary skill in the art at the relevant time to combine the teachings of Hollfelder and Su so that the client-server network transactions described in Hollfelder occur specifically on the Internet. Polish Decl., Ex at 67. This combination would have been desirable to increase the range of practical applications, given the prevalence of the Internet as a computer network for media data transfer. Id. 19

25 Claim 10 additionally requires a server programmed with machine-readable instructions to apply the method described in Claim 1, and specifies that it contains a storage device, memory, CPU, OS, connection to the medium and a communications system. The system of Hollfelder also contains a server equipped with a storage device, memory, CPU, OS, connection to the medium and a communications system, all of which are essential to the stated function of the server. Ex. 1002, Hollfelder 3.3 1, Fig. 4. Moreover, it is obvious to one of ordinary skill in the art at the relevant time that the client-server architecture in Hollfelder would contain them. Polish Decl., Ex at 56. For the server to receive requests and send corresponding data elements (that is, an integrated storage device and memory to contain the data, a CPU and OS to process the system functions, and a connection to the medium and a communications system to transmit the data to the client). See also id. Claim 19 requires a machine-readable medium containing a computer program which prepares streaming media content for transmission. One of ordinary skill in the art would understand that Hollfelder also employs such a program within its client-server architecture. Ex. 1002, Hollfelder 3.3 1, Fig. 4. The server in Hollfelder requires a software component to receive and process requests from the client and prepare stored media data for transmission. Id. It would not be possible for the server to send the requested data elements to the client if it did not contain such software. Polish Decl., Ex at

26 Claim 24 requires a machine-readable medium containing a computer program which operates a media player for receiving and playing the streaming media. Hollfelder likewise discloses a media player application, loaded onto the user s system, which displays the streaming media data for the user by retrieving the transmitted COPUs from the client buffer. Ex. 1002, Hollfelder at Claims 1 and 24 recite that the media player maintains a record of the last media data element that has been received by the player. As illustrated in Figure 2 and accompanying disclosures, Hollfelder through presentation points maintains a record of the last COPU received and stored. Id. at Fig. 2, Hollfelder discloses an algorithm on the client side that contains values for each COPU such that COPUs with least relevance are replaced first and those with highest relevance value are preloaded first.... For example the COPUs following the presentation point in presentation direction have the highest value, decreasing with the distance from the presentation point. If, for example, a change of the presentation direction is likely in the application, the COPUs preceding the presentation point, that are typically already loaded in the buffer.... Id. at It uses the function getnextcopu which retrieves the next COPU in presentation direction. Id. at p. 14, 7 n. 1 ( In our client-pull system, dropping COPUs means that they are not requested by the client. ) Thus, Hollfelder maintain[s] a record of the identifier of the last data element that has been received. Polish Decl., Ex at 62. Su similarly maintains a record of the last data element received through storing the frame number. Id. at 61. The 21

27 client first retrieves N GoPs into its local buffer.... [T]he client can continuously retrieve frames from the server and the next frames to be presented are always loaded in the local buffer. To retrieve a frame the client sends a request to the server that specifies the frame number. Ex. 1003, Su at pp ( If there are already enough frames in local buffer to ensure continuous presentation, that is more than N GoPs, the Buffer Manager simply loads the following frames in sequence. ) One of ordinary skill in the art would be motivated to record the serial number, as taught by Su, of the last COPU received and stored so the client could, for example, request the next COPU required. Polish Decl., Ex at 61. Claim 19 recites that the media data elements are in a format capable of being served to user by the server. As discussed previously, the client-server architecture described in Hollfelder is designed for streaming multimedia presentations transmitted from server to client, so the data to be transmitted must be in a format suitable for such transmission. Polish Decl., Ex at 64. For example, the COPUs described in Hollfelder are designed to facilitate effective streaming transmission. Hollfelder at 2.1, 1; Polish Decl., Ex at 64. Claim 5 recites a system that distributes streaming media using the method of Claim 1 to a plurality of users. Similarly, Claim 14 recites a server adapted to distribute streaming media to a plurality of simultaneous users. Su discloses distributing buffered media streams from a server to multiple users through multiple clients. Ex. 1003, Su at p. 2, 29, 81. Methods for streaming media data to a plurality 22

28 of users were well known to one of ordinary skill in the art at the time of invention, so one would have been motivated to apply those methods in their intended manner to the method of Hollfelder to distribute media to multiple users. Polish Decl., Ex at 65. While it is possible to construct a network with a single client, such a system would severely limit the number of practical applications. Id. Thus a person of ordinary skill would have been motivated to combine the teachings of Hollfelder and Su so that the system is adapted to serve multiple users. Id. Claims 6 and 15 recite distribution of media streams to multiple users wherein the server does not maintain a pointer into a server-side buffer for each user. Hollfelder describes a client-side buffer which operates without reference to pointers in a server-side buffer. As previously discussed, Hollfelder maintains a record of the COPU on the client-side. Ex. 1002, Hollfelder at 2.3 3, ; Polish Decl., Ex at 62. Claims 7, 16 and 27 describe the use of standard communications protocol techniques such as TCP. Su discloses a client-server architecture where transfers of video data from server to client are governed by the UDP protocol, while transfers of control data are governed by the TCP protocol. Ex. 1003, Su at p.40. Communications between a client and server in a system such as that described in Hollfelder must employ a network protocol, such as the standard UDP or TCP. Polish Decl., Ex at 68. Accordingly, it would have been obvious to one of ordinary skill in the art combine the teachings of Su regarding UDP and TCP to the 23

29 network interactions in Hollfelder, owing to the widespread use of UDP and TCP for network communications. Id. The following claim chart shows where the teachings of Hollfelder and Su lead one of ordinary skill in the art to all the limitations of Claims 1-2, 5-7, 9-11, 14-16, 18-20, 24 and of the 141 Patent, thus rendering those claims obvious over these prior art references. 141 Patent Hollfelder and Su 1[a]. A method for distributing streaming media via a data communications medium such as the Internet to at least one user system of at least one user, Hollfelder, 2.1, 2: For continuous presentation our approach supports a client-pull mechanism, i.e., the client continuously requests chunks of data which are provided in our system by the server in a best-effort manner. Since requests at the granularity level of CODUs are not very effective and presentation engines, like, e.g., MPEG players or other video and audio devices, handle larger units of data, a second abstraction is introduced to support presentation. An atomic unit requested during synchronous presentation is called a Continuous Object Presentation Unit (COPU). One COPU may consist of several CODUs as illustrated in Figure 1. By this generic concept, the buffer manager is able to handle any kind of continuous media in the same way by means of generic COPU requests. Hollfelder, 2.2, 1: An intelligent buffering mechanism for continuous media streams has not only to optimize transport of data from server to client but also to take into account the presentational requirements of the media stream. Since the size of continuous objects it too large to store them in the client buffer as a whole, single COPUs are requested continuously by the buffer manager. We use an algorithm called Least/Most 24

30 141 Patent Hollfelder and Su Relevant for Presentation (L/MRP) [MKK95] for preloading and replacing COPUs in the client buffer. Its goal is to support a synchronized presentation of the media stream. Su, p.21 Shanwei Cen et al. at Oregon Graduate Institute of Science and Technology have designed and implemented a distributed, real-time MPEG video and audio player [CPS95]. The player is designed for use across the Internet, a shared environment with variable traffic and with great diversity in network bandwidth and host processing speed. Su, pp : The Video Decoder gets data from the Buffer Manager, decodes it and passes the decoded pictures to the application for display. The UDP protocol is used to transfer video data from the server to the client, and the TCP/IP protocol is used to transfer control data between client and server. The Communication Manager can also be easily modified to work over an ATM network. 1[b]. the streaming media comprising a plurality of sequential media data elements for a digitally encoded audio or video program, comprising Hollfelder, 2.1, 1: We provide a data type Continuous Long Field (CLF) as generic, abstract representation for any kind of continuous media, like audio, video or animations. This datatype supports operations for editing and presentation, like insert, request, or delete. By modelling the generic concept CLF, we combine the concepts of streams and long fields. While streams are only consumed, e.g., read by presentation entities, the concept of CLFs allows consumption as well as any access to the data. Structural meta data, like the format and other recording parameters, are stored with every CLF object. Multiple media encoded in a single stream (like, 25

31 141 Patent Hollfelder and Su e.g., with MPEG-encoded audio and video) are modelled as separate CLF objects, to enable the individual retrieval and manipulation of each of the media parts. For editing purposes, CLF objects are segmented into a sequence of manipulation units, called Continuous Object Data Units (CODU). In consequence, a CLF is a sequence of CODUs. The granularity of these units is determined individually for each type of media, e.g. Motion-JPEG frames, MPEG-1 frames or audio samples. At this level of granularity, it is possible to manipulate the data stream, e.g., by inserting, deleting or appending CODUs of a Continuous Long Field. Hollfelder, 2.1, 2: For continuous presentation our approach supports a client-pull mechanism, i.e., the client continuously requests chunks of data which are provided in our system by the server in a best-effort manner. Since requests at the granularity level of CODUs are not very effective and presentation engines, like, e.g., MPEG players or other video and audio devices, handle larger units of data, a second abstraction is introduced to support presentation. An atomic unit requested during synchronous presentation is called a Continuous Object Presentation Unit (COPU). One COPU may consist of several CODUs as illustrated in Figure 1. By this generic concept, the buffer manager is able to handle any kind of continuous media in the same way by means of generic COPU requests. 1[c]. providing a server programmed to receive requests from the user system for media data elements corresponding to specified serial identifiers and Hoffelder, Fig. 1. See above citation to Fig. 1 for 1[a]. Hollfelder, 2.1, 2: For continuous presentation our approach supports a client-pull mechanism, i.e., the client continuously requests chunks of data which are provided in our system by the server in a best-effort manner. Since requests at the granularity level of CODUs are not very effective and presentation engines, like, e.g., MPEG players or other video and audio devices, handle larger units of data, a second abstraction is introduced to support presentation. An atomic unit requested during synchronous presentation is called a Continuous Object Presentation Unit (COPU). One COPU may consist of several CODUs as illustrated in Figure 1. By this generic concept, the buffer manager is able to handle any kind 26

32 141 Patent Hollfelder and Su of continuous media in the same way by means of generic COPU requests. Hoffelder, Fig. 1. See above citation to Fig. 1 for 1[a]. Hollfelder, 3.1, 3: Object-relational DBMS combine the positive aspects of both system alternatives. They store data in tables and provide the powerful querying of relational DBMS, but the data types used in the tables are not limited to a predefined set. New types can be created and used in the database just like any predefined type. Functions can be defined to support content-based operations on new datatypes or special purpose functionality to the database. Hollfelder, 3.3, 1: In the following, we describe the system architecture that integrates creation, editing, continuous presentation support, and content-based annotation of Continuous Long Fields in the OR-DBMS Informix Universal Server. Our system is based on a client/server architecture where the server is responsible for the storage of continuous and discrete data. The client is responsible for requesting data from the server, for presenting them and for handling interactions with the user. The architecture is composed of four parts, the IUS, the VF DataBlade, the external storage manager and the CLF DataBlade. The CLF DataBlade in turn consists of three parts, the DBMS functionality extending the OR-DBMS, the presentation support module located on the client, and the continuous transport module connecting the presentation support module and the external storage manager during a presentation. Figure 4 displays the system components and the different connections that exist between them. Additionally, an application is connected to outline the interfaces to the system. Hollfelder, 3.3, 5: The Transport Module connects the COM with the external storage manager. It implements fast data access, fast delivery over the network, and transport of data requests from the client to the server and vice versa. The Transport Module sets up a continuous data connection to the external storage manager utilizing an appropriate network. The data connection and a corresponding control connection work 27

33 141 Patent Hollfelder and Su in parallel to the other components of the DBMS and the application program. One data connection can consist of several logical channels to support the presentation of composed media. Using this connection, time-dependent data are transferred from the server to client buffers. The control connection is responsible for transferring user commands as well as system control commands. Su, p Figure 2.4 shows the sets of History, Referenced and Skip COPUs for the given presentation point (508) and a skip value of +2. Each COPU, identified by the index on the x- axis, is associated with a relevance value reflecting the importance of the COPU with respect to the specific interaction set. Su, Fig See above citation to Fig. 2.3 for 1[c]. Su, p.18 Each set As has an associated criteria that is used to decide whether or not a COPU belongs to the set at a specific point of a presentation, and to specify the relevance value of the COPU with respect to s. Hence, an interaction set As is defined as a binary relation relating a COPU to a relevance value. 1[d]. to send media data elements to the user system responsive to said requests, at a rate more rapid than the rate at which said streaming media is played Su, p To retrieve a frame the client sends a request to the server that specifies the frame number. The server retrieves the desired frame from its local buffer. If the desired frame is not in its local buffer, the server retrieves it from the database. Each time the server goes to the database it retrieves a block of video data that includes the desired frame. The server then delivers the frame to the client over network. Hollfelder, 2.1, 2: For continuous presentation our approach supports a client-pull mechanism, i.e., the client continuously requests chunks of data which are provided in our system by the server in a best-effort manner. Since requests at the granularity level of CODUs are not very effective and presentation engines, like, e.g., MPEG players or other video and audio devices, handle larger units of data, a second abstraction is introduced to support presentation. An atomic unit requested during synchronous presentation is called a 28

34 141 Patent Hollfelder and Su back by a user; and Case IPR Continuous Object Presentation Unit (COPU). One COPU may consist of several CODUs as illustrated in Figure 1. By this generic concept, the buffer manager is able to handle any kind of continuous media in the same way by means of generic COPU requests. Hollfelder, 2.3, 3: In Figure 2, the architecture of the adaptation feedback and its interaction with the buffering mechanism, described in Section 2.2, is illustrated. The presentation process starts with a filled client buffer. The presentation engine continuously requests COPUs for their synchronized presentation and handles user interactions. While the COPUs are presented, the preloader asynchronously prefetches COPUs from the server in the client buffer. If prefetching from the server is faster than or as fast as the presentation of COPUs, the buffer utilization should never decrease. If server CPU, server disk or network bandwidth have temporary bottlenecks, the client s buffer utilization starts to decrease. In this case the presentation quality is gracefully degraded. The adaptation control module permanently controls the buffer size, and, based on the adaptation specification, determines which quality has to be requested. In case of dropping the request of COPUs, logically empty COPUs are inserted into the buffer to increase buffer utilization. Hollfelder, 3.2, 2: The VFDB application scenario s software architecture consists of three main components: The IUS, an external storage manager and the application program. The IUS stores all meta data. It manages and controls the access to the external storage managers and devices. The external storage 29

35 141 Patent Hollfelder and Su managers handle the storage of the various media streams. Whenever an application wants to access continuous objects, it can access them indirectly through the IUS VFDB schema and its access functions. If the access is time-critical, a direct connection to the responsible storage manager (through an appropriate client-server connection) can be mediated through the IUS and the VFDBs meta data as well. If the application requests only meta data, the IUS will serve the request on the basis of the VFDB schema. 1[e]. providing a machine-readable medium accessible to said user, on which there has been recorded software for implementing a media player for receiving and playing the streaming media on said user system, Hollfelder, 2.3, 3: In Figure 2, the architecture of the adaptation feedback and its interaction with the buffering mechanism, described in Section 2.2, is illustrated. The presentation process starts with a filled client buffer. The presentation engine continuously requests COPUs for their synchronized presentation and handles user interactions. While the COPUs are presented, the preloader asynchronously prefetches COPUs from the server in the client buffer. If prefetching from the server is faster than or as fast as the presentation of COPUs, the buffer utilization should never decrease. If server CPU, server disk or network bandwidth have temporary bottlenecks, the client s buffer utilization starts to decrease. In this case the presentation quality is gracefully degraded. The adaptation control module permanently controls the buffer size, and, based on the adaptation specification, determines which quality has to be requested. In case of dropping the request of COPUs, logically empty COPUs are inserted into the buffer to increase buffer utilization. Hollfelder, 3.3, 1: In the following, we describe the system architecture that integrates creation, editing, continuous presentation support, and content-based annotation of Continuous Long Fields in the OR-DBMS Informix Universal Server. Our system is based on a client/server architecture where the server is responsible for the storage of continuous and discrete data. The client is responsible for requesting data from the server, for presenting them and for handling interactions with the user. The architecture is composed of four parts, the IUS, the VF DataBlade, the external storage manager and the CLF DataBlade. The CLF DataBlade in turn consists of three parts, the DBMS functionality extending the OR-DBMS, 30 Case IPR

36 141 Patent Hollfelder and Su the presentation support module located on the client, and the continuous transport module connecting the presentation support module and the external storage manager during a presentation. Figure 4 displays the system components and the different connections that exist between them. Additionally, an application is connected to outline the interfaces to the system. 1[f]. said software being programmed to cause the media player to maintain a record of the identifier of the last data element that has been received; and Case IPR Hollfelder, 2.2, 1-2: We use an algorithm called Least/Most Relevant for Presentation (L/MRP) [MKK95] for preloading and replacing COPUs in the client buffer. Its goal is to support a synchronized presentation of the media stream. The algorithm considers the current presentation state and likely interactions when deciding which COPUs need to be preloaded and replaced in the client buffer. For that purpose, it assigns a relevance value to each COPU, such that COPUs with least relevance are replaced first and those with highest relevance value are preloaded first. Dynamic relevance values are related to the current presentation state. For example the COPUs following the presentation point in presentation direction have the highest value, decreasing with the distance from the presentation point. If, for example, a change of the presentation direction is likely in the application, the COPUs preceding the presentation point, that are typically already loaded in the buffer, obtain relatively high relevance, too. Hollfelder, p. 14: The class MDDA models the MDDAprotocol interface for a presentation. It offers the function getnextcopu which retrieves the next COPU in presentation direction. ; Hollfelder, p. 7 f.n. 1 ( In our client-pull system, dropping 31

37 141 Patent Hollfelder and Su COPUs means that they are not requested by the client. ) Su, p. 43 We discussed in section how the group of pictures (GoP) structure enables random access within a MPEG video stream. We assign a sequential number, called a GoP number, to each GoP. A sequential number is also assigned to each frame and is denoted as a frame number. Alternatively a frame can be referred to using a frame position, which is a GoP number and the frame s relative frame position within the GoP. Su, p. 44 A frame is the basic data unit in our system. Client requests one frame at a time, and the server sends one frame at a time. By doing so the system can avoid wasting network bandwidth in a highly interactive presentation environment. The default is that the client must send requests for each frame it wants. Su, pp : The client first retrieves N GoPs into its local buffer. The number N represents the number of GoPs that can be delivered in the amount of time it takes to retrieve a frame from the server. In our case, N is 4 and it takes just under 2 seconds to retrieve a frame. Thus, in the absence of user interactions, the client can continuously retrieve frames from the server and the next frames to be presented are always loaded in the local buffer. To retrieve a frame the client sends a request to the server that specifies the frame number. The 32

38 141 Patent Hollfelder and Su server retrieves the desired frame from its local buffer. If the desired frame is not in its local buffer, the server retrieves it from the database. Each time the server goes to the database it retrieves a block of video data that includes the desired frame. The server then delivers the frame to the client over network. The three components at the client side, namely the Buffer Manager, Decoder and Presenter, work independently to process the same data flow as shown in Figure [g]. to transmit requests to the server to send one or more data elements, specifying the identifiers of the data elements, Hollfelder, 2.1, 2: For continuous presentation our approach supports a client-pull mechanism, i.e., the client continuously requests chunks of data which are provided in our system by the server in a best-effort manner. Since requests at the granularity level of CODUs are not very effective and presentation engines, like, e.g., MPEG players or other video and audio devices, handle larger units of data, a second abstraction is introduced to support presentation. An atomic unit requested during synchronous presentation is called a Continuous Object Presentation Unit (COPU). One COPU may consist of several CODUs as illustrated in Figure 1. By this generic concept, the buffer manager is able to handle any kind of continuous media in the same way by means of generic COPU requests. Hoffelder, Fig. 1. See above citation to Fig. 1 for 1[a]. Hollfelder, 2.2, 1: An intelligent buffering mechanism for continuous media streams has not only to optimize transport of data from server to client but also to take into account the presentational requirements of the media stream. Since the size of continuous objects it too large to store them in the client buffer as a whole, single COPUs are requested continuously by the buffer manager. We use an algorithm called Least/Most Relevant for Presentation (L/MRP) [MKK95] for preloading and replacing COPUs in the client buffer. Its goal is to support a synchronized presentation of the media stream. Hollfelder, 2.3, 3: In Figure 2, the architecture of the adaptation feedback and its interaction with the buffering mechanism, described in Section 2.2, is illustrated. The presentation process starts with a filled client buffer. The 33 Case IPR

39 141 Patent Hollfelder and Su presentation engine continuously requests COPUs for their synchronized presentation and handles user interactions. While the COPUs are presented, the preloader asynchronously prefetches COPUs from the server in the client buffer. If prefetching from the server is faster than or as fast as the presentation of COPUs, the buffer utilization should never decrease. If server CPU, server disk or network bandwidth have temporary bottlenecks, the client s buffer utilization starts to decrease. In this case the presentation quality is gracefully degraded. The adaptation control module permanently controls the buffer size, and, based on the adaptation specification, determines which quality has to be requested. In case of dropping the request of COPUs, logically empty COPUs are inserted into the buffer to increase buffer utilization. Su, p Figure 2.4 shows the sets of History, Referenced and Skip COPUs for the given presentation point (508) and a skip value of +2. Each COPU, identified by the index on the x- axis, is associated with a relevance value reflecting the importance of the COPU with respect to the specific interaction set. Su, Fig See above citation to Fig. 2.3 for 1[c]. Su, p.18 Each set As has an associated criteria that is used to decide whether or not a COPU belongs to the set at a specific point of a presentation, and to specify the relevance value of the COPU with respect to s. Hence, an interaction set As is defined as a binary relation relating a COPU to a relevance value. 34

40 141 Patent Hollfelder and Su Case IPR [h]. as said media player requires in order to maintain a sufficient number of media data elements in the media player for uninterrupted playback. Su, p To retrieve a frame the client sends a request to the server that specifies the frame number. The server retrieves the desired frame from its local buffer. If the desired frame is not in its local buffer, the server retrieves it from the database. Each time the server goes to the database it retrieves a block of video data that includes the desired frame. The server then delivers the frame to the client over network. Hollfelder, 2.1, 2: For continuous presentation our approach supports a client-pull mechanism, i.e., the client continuously requests chunks of data which are provided in our system by the server in a best-effort manner. Since requests at the granularity level of CODUs are not very effective and presentation engines, like, e.g., MPEG players or other video and audio devices, handle larger units of data, a second abstraction is introduced to support presentation. An atomic unit requested during synchronous presentation is called a Continuous Object Presentation Unit (COPU). One COPU may consist of several CODUs as illustrated in Figure 1. By this generic concept, the buffer manager is able to handle any kind of continuous media in the same way by means of generic COPU requests. Hollfelder, 2.2, 1: An intelligent buffering mechanism for continuous media streams has not only to optimize transport of data from server to client but also to take into account the presentational requirements of the media stream. Since the size of continuous objects it too large to store them in the client buffer as a whole, single COPUs are requested continuously by the buffer manager. We use an algorithm called Least/Most Relevant for Presentation (L/MRP) [MKK95] for preloading and replacing COPUs in the client buffer. Its goal is to support a synchronized presentation of the media stream. Hollfelder, 2.3, 3: In Figure 2, the architecture of the adaptation feedback and its interaction with the buffering mechanism, described in Section 2.2, is illustrated. The presentation process starts with a filled client buffer. The presentation engine continuously requests COPUs for their synchronized presentation and handles user interactions. 35

41 141 Patent Hollfelder and Su 2. The method of claim 1, wherein said serial identifiers are sequential. Hollfelder, 3.4, 17: For the purpose of clarity and compactness, the example was stripped from all irrelevant code. The code fragment initializes the CAPI functionality by creating an instance of class ContinuousObjectManager. The meta data for the continuous object to present (its object_name is vid1 ) is fetched from the IUS by means of the C++ interface of the Universal Server. An instance of class CODUString is created using the value of the row type CODUSTRING_Type. A presentation with the id video is initialized and the corresponding MDDA instance is fetched from the MDDA_Manager. The presentation loop requests all COPUs in normal order. For each COPU the pointer to the actual data is passed to the function senttodisplay and upon return from that function the COPU is dropped, meaning the system can replace it in the client buffer. Su, p Figure 2.4 shows the sets of History, Referenced and Skip COPUs for the given presentation point (508) and a skip value of +2. Each COPU, identified by the index on the x- axis, is associated with a relevance value reflecting the importance of the COPU with respect to the specific interaction set. Su, Fig See above citation to Fig. 2.3 for 1[c]. Case IPR Su, pp The client first retrieves N GoPs into its local buffer.... [T]he client can continuously retrieve frames from the server and the next frames to be presented are always loaded in the local buffer. To retrieve a frame the client sends a request to the server that specifies the frame number. 5. The method of claim 1, further comprising distributing said streaming media to a plurality of users. If there are already enough frames in local buffer to ensure continuous presentation, that is more than N GoPs, the Buffer Manager simply loads the following frames in sequence. Su, p.2 Multimedia presentation demands specific support from a MM-DBMS. They require the delivery of continuous media data from a database server to multiple destinations over a network. Su, p.29 A Session Manager-Client pair is created for each 36

42 141 Patent Hollfelder and Su application, which contains multiple media streams and deals with a specific multimedia application, such as Video-on- Demand, News-on-Demand, Media Editing Workbenches, and so on. The function and configuration of each Session Manager-Client pair are specific to the media they support. The system, therefore can be extended to support new media types and new multimedia applications. 6. The method of claim 5, wherein said server does not maintain a pointer into a buffer established within said server, for each said user. 7. The method of claim 1, wherein said server is adapted to receive said requests from said user system via standard communications protocol techniques such as Su, p.81 In our system, the server can support multiple clients. The data rates at the server side are so high that despite the development of an efficient retrieval strategy, database I/O can still be the potential bottleneck. This problem limits the number of concurrent clients that can be supported by the system. A media sharing technique [KRT95] can reserve the buffers that have been played by a user in controlled fashion for use by subsequent users requesting the same data. We can implement this technique at the server side to improve the performance of the system. See claim element 1[f]. Hollfelder: 2.3 1: Since we do not assume the availability of reservation mechanisms for stream delivery, bottlenecks on server or network may occur during presentation and lead to delay of presentation. Therefore, we use a client-based adaptation mechanism to dynamically change the presentation quality in order to reduce the data volume that needs to be delivered from the server to the client. In this way, intra-media synchronization can be maintained by reducing disk utilization, memory consumption or required network bandwidth. Su, p.40 The UDP protocol is used to transfer video data from the server to the client, and the TCP/IP protocol is used to transfer control data between client and server. 37

43 141 Patent Hollfelder and Su TCP. 9. The method of claim 1, wherein said server obtains said streaming media from a disk file. Case IPR Hollfelder, 2.3, 3: In Figure 2, the architecture of the adaptation feedback and its interaction with the buffering mechanism, described in Section 2.2, is illustrated. The presentation process starts with a filled client buffer. The presentation engine continuously requests COPUs for their synchronized presentation and handles user interactions. While the COPUs are presented, the preloader asynchronously prefetches COPUs from the server in the client buffer. If prefetching from the server is faster than or as fast as the presentation of COPUs, the buffer utilization should never decrease. If server CPU, server disk or network bandwidth have temporary bottlenecks, the client s buffer utilization starts to decrease. In this case the presentation quality is gracefully degraded. The adaptation control module permanently controls the buffer size, and, based on the adaptation specification, determines which quality has to be requested. In case of dropping the request of COPUs, logically empty COPUs are inserted into the buffer to increase buffer utilization. 10[a]. A server for Hollfelder, 3.3, 1: In the following, we describe the system architecture that integrates creation, editing, continuous presentation support, and content-based annotation of Continuous Long Fields in the OR-DBMS Informix Universal Server. Our system is based on a client/server architecture where the server is responsible for the storage of continuous and discrete data. The client is responsible for requesting data from the server, for presenting them and for handling interactions with the user. The architecture is composed of four parts, the IUS, the VF DataBlade, the external storage manager and the CLF DataBlade. The CLF DataBlade in turn consists of three parts, the DBMS functionality extending the OR-DBMS, the presentation support module located on the client, and the continuous transport module connecting the presentation support module and the external storage manager during a presentation. Figure 4 displays the system components and the different connections that exist between them. Additionally, an application is connected to outline the interfaces to the system. Hollfelder, 2.1, 2: For continuous presentation our approach 38

44 141 Patent Hollfelder and Su distributing streaming media via a data communications medium such as the Internet to at least one user system of at least one user, supports a client-pull mechanism, i.e., the client continuously requests chunks of data which are provided in our system by the server in a best-effort manner. Since requests at the granularity level of CODUs are not very effective and presentation engines, like, e.g., MPEG players or other video and audio devices, handle larger units of data, a second abstraction is introduced to support presentation. An atomic unit requested during synchronous presentation is called a Continuous Object Presentation Unit (COPU). One COPU may consist of several CODUs as illustrated in Figure 1. By this generic concept, the buffer manager is able to handle any kind of continuous media in the same way by means of generic COPU requests. Hoffelder, Fig. 1. See above citation to Fig. 1 for 1[a]. Hollfelder, 2.2, 1: An intelligent buffering mechanism for continuous media streams has not only to optimize transport of data from server to client but also to take into account the presentational requirements of the media stream. Since the size of continuous objects it too large to store them in the client buffer as a whole, single COPUs are requested continuously by the buffer manager. We use an algorithm called Least/Most Relevant for Presentation (L/MRP) [MKK95] for preloading and replacing COPUs in the client buffer. Its goal is to support a synchronized presentation of the media stream. Su, p.21 Shanwei Cen et al. at Oregon Graduate Institute of Science and Technology have designed and implemented a distributed, real-time MPEG video and audio player [CPS95]. The player is designed for use across the Internet, a shared environment with variable traffic and with great diversity in network bandwidth and host processing speed. Su, pp.39-40: The Video Decoder gets data from the Buffer Manager, decodes it and passes the decoded pictures to the application for display. The UDP protocol is used to transfer video data from the server to the client, and the TCP/IP protocol is used to transfer control data between client and server. The Communication Manager can also be easily 39 Case IPR

45 141 Patent Hollfelder and Su modified to work over an ATM network. 10[b]. the streaming media comprising a plurality of sequential media data elements for a digitally encoded audio or video program, 10[c] said user system being assumed to have a media player for receiving and playing the streaming media on said user system, 10[d] which is operable to obtain media data elements from said server by transmitting requests to said server to send one or more specified media data elements, said server comprising: 40 See above citations for Claim 1[b]: Hollfelder 2.1, 1; 2.1, 2; Fig. 1 See above citations for Claim 1[e]: Hollfelder 2.3, 3; 3.3, 1 Hollfelder, 2.1, 2: For continuous presentation our approach supports a client-pull mechanism, i.e., the client continuously requests chunks of data which are provided in our system by the server in a best-effort manner. Since requests at the granularity level of CODUs are not very effective and presentation engines, like, e.g., MPEG players or other video and audio devices, handle larger units of data, a second abstraction is introduced to support presentation. An atomic unit requested during synchronous presentation is called a Continuous Object Presentation Unit (COPU). One COPU may consist of several CODUs as illustrated in Figure 1. By this generic concept, the buffer manager is able to handle any kind of continuous media in the same way by means of generic COPU requests. Hoffelder, Fig. 1. See above citation to Fig. 1 for 1[a]. Case IPR Hollfelder, 2.2, 1: An intelligent buffering mechanism for continuous media streams has not only to optimize transport of data from server to client but also to take into account the presentational requirements of the media stream. Since the size of continuous objects it too large to store them in the client buffer as a whole, single COPUs are requested continuously by the buffer manager. We use an algorithm called Least/Most Relevant for Presentation (L/MRP) [MKK95] for preloading and replacing COPUs in the client buffer. Its goal is to support a synchronized presentation of the media stream. Hollfelder, 2.3, 3: In Figure 2, the architecture of the adaptation feedback and its interaction with the buffering mechanism, described in Section 2.2, is illustrated. The

46 141 Patent Hollfelder and Su presentation process starts with a filled client buffer. The presentation engine continuously requests COPUs for their synchronized presentation and handles user interactions. While the COPUs are presented, the preloader asynchronously prefetches COPUs from the server in the client buffer. If prefetching from the server is faster than or as fast as the presentation of COPUs, the buffer utilization should never decrease. If server CPU, server disk or network bandwidth have temporary bottlenecks, the client s buffer utilization starts to decrease. In this case the presentation quality is gracefully degraded. The adaptation control module permanently controls the buffer size, and, based on the adaptation specification, determines which quality has to be requested. In case of dropping the request of COPUs, logically empty COPUs are inserted into the buffer to increase buffer utilization. 10[e] at least one data storage device, memory for storing machine-readable executable routines and for providing a working memory Su, p To retrieve a frame the client sends a request to the server that specifies the frame number. The server retrieves the desired frame from its local buffer. If the desired frame is not in its local buffer, the server retrieves it from the database. Each time the server goes to the database it retrieves a block of video data that includes the desired frame. The server then delivers the frame to the client over network. Hollfelder, 3.3, 1: In the following, we describe the system architecture that integrates creation, editing, continuous presentation support, and content-based annotation of Continuous Long Fields in the OR-DBMS Informix Universal Server. Our system is based on a client/server architecture where the server is responsible for the storage of continuous and discrete data. The client is responsible for requesting data 41

47 141 Patent Hollfelder and Su area for routines executing on the server, a central processing unit for executing the machine-readable executable routines, an operating system, at least one connection to the communications medium, and a communications system providing a set of communications protocols for communicating through said at least one connection; 10[f]. a machinereadable, executable routine containing instructions to cause the server to assign serial identifiers to the sequential media data elements comprising the program; 10[g]. a machinereadable, from the server, for presenting them and for handling interactions with the user. The architecture is composed of four parts, the IUS, the VF DataBlade, the external storage manager and the CLF DataBlade. The CLF DataBlade in turn consists of three parts, the DBMS functionality extending the OR-DBMS, the presentation support module located on the client, and the continuous transport module connecting the presentation support module and the external storage manager during a presentation. Figure 4 displays the system components and the different connections that exist between them. Additionally, an application is connected to outline the interfaces to the system. Su, pp. 5, 22: IBM DB2/UDB [IBAI97b] is used to store MPEG video at the server side. Su, p. 44: The client initializes all its processes first. Then it sends a request to the server to initiate a presentation. Su, p. 43: assign[s] a sequential number, called a GoP number, to each GoP. A sequential number is also assigned to each frame and is denoted as a frame number. Su, pp : To retrieve a frame sends a request to the server that specifies the frame number. The server retrieves the desired frame from its local buffer. If the desired frame is not in its local buffer, the server retrieves it from the database. The server then delivers the frame to the client over network. Hollfelder, 2.1, 2: For continuous presentation our approach supports a client-pull mechanism, i.e., the client continuously 42 Case IPR

48 141 Patent Hollfelder and Su executable routine containing instructions to cause the server to receive requests from the user system for one or more media data elements specifying the identifiers of the requested data elements; and requests chunks of data which are provided in our system by the server in a best-effort manner. Since requests at the granularity level of CODUs are not very effective and presentation engines, like, e.g., MPEG players or other video and audio devices, handle larger units of data, a second abstraction is introduced to support presentation. An atomic unit requested during synchronous presentation is called a Continuous Object Presentation Unit (COPU). One COPU may consist of several CODUs as illustrated in Figure 1. By this generic concept, the buffer manager is able to handle any kind of continuous media in the same way by means of generic COPU requests. Hoffelder, Fig. 1. See above citation to Fig. 1 for 1[a]. Hollfelder, 3.1, 3: Object-relational DBMS combine the positive aspects of both system alternatives. They store data in tables and provide the powerful querying of relational DBMS, but the data types used in the tables are not limited to a predefined set. New types can be created and used in the database just like any predefined type. Functions can be defined to support content-based operations on new datatypes or special purpose functionality to the database. Hollfelder, 3.3, 1: In the following, we describe the system architecture that integrates creation, editing, continuous presentation support, and content-based annotation of Continuous Long Fields in the OR-DBMS Informix Universal Server. Our system is based on a client/server architecture where the server is responsible for the storage of continuous and discrete data. The client is responsible for requesting data from the server, for presenting them and for handling interactions with the user. The architecture is composed of four parts, the IUS, the VF DataBlade, the external storage manager and the CLF DataBlade. The CLF DataBlade in turn consists of three parts, the DBMS functionality extending the OR-DBMS, the presentation support module located on the client, and the continuous transport module connecting the presentation support module and the external storage manager during a presentation. Figure 4 displays the system components and the 43 Case IPR

49 141 Patent Hollfelder and Su different connections that exist between them. Additionally, an application is connected to outline the interfaces to the system. Hollfelder, 3.3, 5: The Transport Module connects the COM with the external storage manager. It implements fast data access, fast delivery over the network, and transport of data requests from the client to the server and vice versa. The Transport Module sets up a continuous data connection to the external storage manager utilizing an appropriate network. The data connection and a corresponding control connection work in parallel to the other components of the DBMS and the application program. One data connection can consist of several logical channels to support the presentation of composed media. Using this connection, time-dependent data are transferred from the server to client buffers. The control connection is responsible for transferring user commands as well as system control commands. 10[h]. a machinereadable, executable routine containing instructions to cause the server to send media data elements to the user system responsive to said requests, at a rate more rapid than the rate at which said streaming media is played Su, p To retrieve a frame the client sends a request to the server that specifies the frame number. The server retrieves the desired frame from its local buffer. If the desired frame is not in its local buffer, the server retrieves it from the database. Each time the server goes to the database it retrieves a block of video data that includes the desired frame. The server then delivers the frame to the client over network. Hollfelder, 2.1, 2: For continuous presentation our approach supports a client-pull mechanism, i.e., the client continuously requests chunks of data which are provided in our system by the server in a best-effort manner. Since requests at the granularity level of CODUs are not very effective and presentation engines, like, e.g., MPEG players or other video and audio devices, handle larger units of data, a second abstraction is introduced to support presentation. An atomic unit requested during synchronous presentation is called a Continuous Object Presentation Unit (COPU). One COPU may consist of several CODUs as illustrated in Figure 1. By this generic concept, the buffer manager is able to handle any kind of continuous media in the same way by means of generic COPU requests. 44

50 141 Patent Hollfelder and Su back by a user. Case IPR Hollfelder, 2.3, 3: In Figure 2, the architecture of the adaptation feedback and its interaction with the buffering mechanism, described in Section 2.2, is illustrated. The presentation process starts with a filled client buffer. The presentation engine continuously requests COPUs for their synchronized presentation and handles user interactions. While the COPUs are presented, the preloader asynchronously prefetches COPUs from the server in the client buffer. If prefetching from the server is faster than or as fast as the presentation of COPUs, the buffer utilization should never decrease. If server CPU, server disk or network bandwidth have temporary bottlenecks, the client s buffer utilization starts to decrease. In this case the presentation quality is gracefully degraded. The adaptation control module permanently controls the buffer size, and, based on the adaptation specification, determines which quality has to be requested. In case of dropping the request of COPUs, logically empty COPUs are inserted into the buffer to increase buffer utilization. 11. The server of claim 10, wherein said serial identifiers are sequential. See above citations for Claim 2: Hollfelder 3.4, 17; Su p.16-17, Fig The server of claim 10, wherein said server is adapted to distribute said streaming media to a plurality of simultaneous users. 15. The server of claim 10, wherein said server does not maintain a pointer into a buffer established within said server, for each said user. 45 See above citations for Claim 5: Su 2, 29, 81. See claim element 1[f] and claim The server of claim 10, wherein said operating Su, p.40 The UDP protocol

51 141 Patent Hollfelder and Su system further comprises a protocol to receive said requests from said user system via standard communications protocol techniques such as TCP. 18. The server of claim 10, wherein said server obtains said streaming media from a disk file. 19. A nontransitory machinereadable medium on which there has been recorded a computer program for use in operating a computer to prepare streaming media content for transmission by a server wherein said server responds to user requests for media data elements identified by a serial identifier, said program recorded on said nontransitory machine readable medium comprising a routine to store and serially identify sequential data elements comprising said streaming media content, in a format capable of being served to 46 is used to transfer video data from the server to the client, and the TCP/IP protocol is used to transfer control data between client and server. See claim 9. Case IPR Hollfelder, 2.1, 2: For continuous presentation our approach supports a client-pull mechanism, i.e., the client continuously requests chunks of data which are provided in our system by the server in a best-effort manner. Since requests at the granularity level of CODUs are not very effective and presentation engines, like, e.g., MPEG players or other video and audio devices, handle larger units of data, a second abstraction is introduced to support presentation. An atomic unit requested during synchronous presentation is called a Continuous Object Presentation Unit (COPU). One COPU may consist of several CODUs as illustrated in Figure 1. By this generic concept, the buffer manager is able to handle any kind of continuous media in the same way by means of generic COPU requests. Hollfelder, 2.3, 3: In Figure 2, the architecture of the adaptation feedback and its interaction with the buffering mechanism, described in Section 2.2, is illustrated. The presentation process starts with a filled client buffer. The presentation engine continuously requests COPUs for their synchronized presentation and handles user interactions. While the COPUs are presented, the preloader asynchronously prefetches COPUs from the server in the client buffer. If prefetching from the server is faster than or as fast as the presentation of COPUs, the buffer utilization should never decrease. If server CPU, server disk or network bandwidth have temporary bottlenecks, the client s buffer utilization starts to decrease. In this case the presentation quality is gracefully degraded. The adaptation control module permanently controls the buffer size, and, based on the adaptation specification, determines which quality has to be requested. In case of dropping the request of COPUs, logically empty COPUs are

52 141 Patent Hollfelder and Su inserted into the buffer to increase buffer utilization. users by said server. Hollfelder, 3.3, 1: In the following, we describe the system architecture that integrates creation, editing, continuous presentation support, and content-based annotation of Continuous Long Fields in the OR-DBMS Informix Universal Server. Our system is based on a client/server architecture where the server is responsible for the storage of continuous and discrete data. The client is responsible for requesting data from the server, for presenting them and for handling interactions with the user. The architecture is composed of four parts, the IUS, the VF DataBlade, the external storage manager and the CLF DataBlade. The CLF DataBlade in turn consists of three parts, the DBMS functionality extending the OR-DBMS, the presentation support module located on the client, and the continuous transport module connecting the presentation support module and the external storage manager during a presentation. Figure 4 displays the system components and the different connections that exist between them. Additionally, an application is connected to outline the interfaces to the system. Su, p. 43: assign[s] a sequential number, called a GoP number, to each GoP. A sequential number is also assigned to each frame and is denoted as a frame number. p To retrieve a frame the client sends a request to the server that specifies the frame number. The server retrieves the desired frame from its local buffer. If the desired frame is not in its local buffer, the server retrieves it from the database. Each time the server goes to the database it retrieves a block of video data that includes the desired frame. The server then delivers the frame to the client over network. The client first retrieves N GoPs into its local buffer.... [T]he client can continuously retrieve frames from the server and the next frames to be presented are always loaded in the local buffer. To retrieve a frame the client sends a request to the server that specifies the frame number. If there are already enough frames in local buffer to ensure 47 Case IPR

53 141 Patent Hollfelder and Su continuous presentation, that is more than N GoPs, the Buffer Manager simply loads the following frames in sequence. 20. The nontransitory machine readable medium of claim 19, wherein said streaming media content is obtained by said computer from a disk file. Case IPR Hollfelder, 2.3, 3: In Figure 2, the architecture of the adaptation feedback and its interaction with the buffering mechanism, described in Section 2.2, is illustrated. The presentation process slivetarts with a filled client buffer. The presentation engine continuously requests COPUs for their synchronized presentation and handles user interactions. While the COPUs are presented, the preloader asynchronously prefetches COPUs from the server in the client buffer. If prefetching from the server is faster than or as fast as the presentation of COPUs, the buffer utilization should never decrease. If server CPU, server disk or network bandwidth have temporary bottlenecks, the client s buffer utilization starts to decrease. In this case the presentation quality is gracefully degraded. The adaptation control module permanently controls the buffer size, and, based on the adaptation specification, determines which quality has to be requested. In case of dropping the request of COPUs, logically empty COPUs are inserted into the buffer to increase buffer utilization. Hollfelder, 3.3, 1: In the following, we describe the system architecture that integrates creation, editing, continuous presentation support, and content-based annotation of Continuous Long Fields in the OR-DBMS Informix Universal Server. Our system is based on a client/server architecture where the server is responsible for the storage of continuous and discrete data. The client is responsible for requesting data from the server, for presenting them and for handling interactions with the user. The architecture is composed of four parts, the IUS, the VF DataBlade, the external storage manager and the CLF DataBlade. The CLF DataBlade in turn consists of three parts, the DBMS functionality extending the OR-DBMS, the presentation support module located on the client, and the continuous transport module connecting the presentation support module and the external storage manager during a presentation. Figure 4 displays the system components and the different connections that exist between them. Additionally, an application is connected to outline the interfaces to the system. 48

54 141 Patent Hollfelder and Su 24[a]. A nontransitory machinereadable medium on which there has been recorded a computer program for use in operating a media player for receiving and playing streaming media comprising a plurality of sequential media data elements, said elements being available on request by said player via a data communications medium such as the Internet, from a server assumed to be capable of sending streaming media elements at a rate more rapid than the rate at which said streaming media is played back by a user, each said data element having a serial identifier, said program recorded on said machine readable medium comprising: Hollfelder, 2.3, 3: In Figure 2, the architecture of the adaptation feedback and its interaction with the buffering mechanism, described in Section 2.2, is illustrated. The presentation process starts with a filled client buffer. The presentation engine continuously requests COPUs for their synchronized presentation and handles user interactions. While the COPUs are presented, the preloader asynchronously prefetches COPUs from the server in the client buffer. If prefetching from the server is faster than or as fast as the presentation of COPUs, the buffer utilization should never decrease. If server CPU, server disk or network bandwidth have temporary bottlenecks, the client s buffer utilization starts to decrease. In this case the presentation quality is gracefully degraded. The adaptation control module permanently controls the buffer size, and, based on the adaptation specification, determines which quality has to be requested. In case of dropping the request of COPUs, logically empty COPUs are inserted into the buffer to increase buffer utilization. Hollfelder, 3.3, 1: In the following, we describe the system architecture that integrates creation, editing, continuous presentation support, and content-based annotation of Continuous Long Fields in the OR-DBMS Informix Universal Server. Our system is based on a client/server architecture where the server is responsible for the storage of continuous and discrete data. The client is responsible for requesting data from the server, for presenting them and for handling interactions with the user. The architecture is composed of four parts, the IUS, the VF DataBlade, the external storage manager and the CLF DataBlade. The CLF DataBlade in turn consists of three parts, the DBMS functionality extending the OR-DBMS, the presentation support module located on the client, and the continuous transport module connecting the presentation support module and the external storage manager during a presentation. Figure 4 displays the system components and the different connections that exist between them. Additionally, an application is connected to outline the interfaces to the system. Su, p.21 Shanwei Cen et al. at Oregon Graduate Institute of Science and Technology have designed and implemented a 49 Case IPR

55 141 Patent Hollfelder and Su distributed, real-time MPEG video and audio player [CPS95]. The player is designed for use across the Internet, a shared environment with variable traffic and with great diversity in network bandwidth and host processing speed. Su, pp.39-40: The Video Decoder gets data from the Buffer Manager, decodes it and passes the decoded pictures to the application for display. The UDP protocol is used to transfer video data from the server to the client, and the TCP/IP protocol is used to transfer control data between client and server. The Communication Manager can also be easily modified to work over an ATM network. Su, p To retrieve a frame the client sends a request to the server that specifies the frame number. The server retrieves the desired frame from its local buffer. If the desired frame is not in its local buffer, the server retrieves it from the database. Each time the server goes to the database it retrieves a block of video data that includes the desired frame. The server then delivers the frame to the client over network. 24[b]. a routine that maintains a record of the identifier of the last sequential media data element that has been received by said player; Hollfelder, 2.1, 1: We provide a data type Continuous Long Field (CLF) as generic, abstract representation for any kind of continuous media, like audio, video or animations. This datatype supports operations for editing and presentation, like insert, request, or delete. By modelling the generic concept CLF, we combine the concepts of streams and long fields. While streams are only consumed, e.g., read by presentation entities, the concept of CLFs allows consumption as well as any access to the data. Structural meta data, like the 50

56 141 Patent Hollfelder and Su format and other recording parameters, are stored with every CLF object. Multiple media encoded in a single stream (like, e.g., with MPEG-encoded audio and video) are modelled as separate CLF objects, to enable the individual retrieval and manipulation of each of the media parts. For editing purposes, CLF objects are segmented into a sequence of manipulation units, called Continuous Object Data Units (CODU). In consequence, a CLF is a sequence of CODUs. The granularity of these units is determined individually for each type of media, e.g. Motion-JPEG frames, MPEG-1 frames or audio samples. At this level of granularity, it is possible to manipulate the data stream, e.g., by inserting, deleting or appending CODUs of a Continuous Long Field. Hollfelder, 2.3, 3: In Figure 2, the architecture of the adaptation feedback and its interaction with the buffering mechanism, described in Section 2.2, is illustrated. The presentation process starts with a filled client buffer. The presentation engine continuously requests COPUs for their synchronized presentation and handles user interactions. While the COPUs are presented, the preloader asynchronously prefetches COPUs from the server in the client buffer. If prefetching from the server is faster than or as fast as the presentation of COPUs, the buffer utilization should never decrease. If server CPU, server disk or network bandwidth have temporary bottlenecks, the client s buffer utilization starts to decrease. In this case the presentation quality is gracefully degraded. The adaptation control module permanently controls the buffer size, and, based on the adaptation specification, determines which quality has to be requested. In case of dropping the request of COPUs, logically empty COPUs are inserted into the buffer to increase buffer utilization. 51

57 141 Patent Hollfelder and Su Case IPR Hollfelder, 3.4, 17: For the purpose of clarity and compactness, the example was stripped from all irrelevant code. The code fragment initializes the CAPI functionality by creating an instance of class ContinuousObjectManager. The meta data for the continuous object to present (its object_name is vid1 ) is fetched from the IUS by means of the C++ interface of the Universal Server. An instance of class CODUString is created using the value of the row type CODUSTRING_Type. A presentation with the id video is initialized and the corresponding MDDA instance is fetched from the MDDA_Manager. The presentation loop requests all COPUs in normal order. For each COPU the pointer to the actual data is passed to the function senttodisplay and upon return from that function the COPU is dropped, meaning the system can replace it in the client buffer. Su, p Figure 2.4 shows the sets of History, Referenced and Skip COPUs for the given presentation point (508) and a skip value of +2. Each COPU, identified by the index on the x- axis, is associated with a relevance value reflecting the importance of the COPU with respect to the specific interaction set. Su, Fig See above citation to Fig. 2.3 for 1[c]. Su, p.18 Each set As has an associated criteria that is used to decide whether or not a COPU belongs to the set at a specific point of a presentation, and to specify the relevance value of the COPU with respect to s. Hence, an interaction set As is defined as a binary relation relating a COPU to a relevance 52