Remote display of large raster images using JPEG2000 and the rectangular FishEye-View

Transcription

1 Remote dsplay of large raster mages usng JPEG2000 and the rectangular FshEye-Vew René Rosenbaum* Insttute of Computer Graphcs Unversty of Rostock Rostock Germany Davd S. Taubman School of Electr. Engneerng&Telecommuncaton Unversty of New South Wales Sydney 2052, N.S.W. Australa ABSTRACT In moble envronments, the sze of large raster mages often exceeds the dsplay area on small output devces. The rectangular FshEye-Vew as a Focus&Context technque can be used to provde an nteractve vew of the mage wthout zoomng and pannng. Ths paper presents the applcaton of JPEG2000 for the demand-drven transmsson of mage data optmzed for ths dsplay technque. The proposed method s fully complant to the JPEG2000 standard and enhances ts statc concept for regons of nterest to be fully dynamc. Due to lmted processng power at clent sde almost all calculatons belongng to the dynamc RoI-codng are executed at server sde. Keywords JPEG2000, regons of nterest (RoI), rectangular FshEye-Vew, mage codng, mage transmsson. 1. INTRODUCTION In moble envronments, the sze of large mages often exceeds the dsplay area of the user's output devce. The method commonly used to present such mages s to allow nteractve pan and zoom by the vewer. An mportant drawback of ths conventonal paradgm s that the dsplay cannot smultaneously provde nformaton about the user's regon of nterest, as well as surroundng areas. Whle the user can pan to new locatons, contextual nformaton, dentfyng the poston of the vew wthn the mage, s lost. These dffcultes become partcularly apparent when workng wth the small dsplays typcally avalable to moble devces. To overcome these drawbacks, dsplay technques from nformaton vsualzaton can also be appled to mage data. One of these methods s the FshEye- Permsson to make dgtal or hard copes of all or part of ths work for personal or classroom use s granted wthout fee provded that copes are not made or dstrbuted for proft or commercal advantage and that copes bear ths notce and the full ctaton on the frst page. To copy otherwse, or republsh, to post on servers or to redstrbute to lsts, requres pror specfc permsson and/or a fee. Journal of WSCG, Vol.11, No.1., ISSN WSCG 2003, February 3-7, 2003, Plzen, Czech Republc. Copyrght UNION Agency Scence Press Vew ntroduced by Furnas [Fur81] to dsplay large structures effcently. Ths general approach can also be appled to raster mages [Rau99]. It belongs to the group of Focus&Context technques, whch dsplay the user's current man area of nterest n an undstorted focus regon. Ths focus regon s embedded n a spatally dstorted representaton of the remanng mage areas. In ths way, the user receves both the nformaton whch s of prncple nterest and useful context nformaton for orentaton and further navgaton. Ths strategy substantally facltates nteractve navgaton wthn large mages. To dstngush between the mportance of dfferent mage regons, the concepts of regons of nterest (RoI) and level of detal (LoD), as proposed n [RS99], are used. In moble envronments the mage data must be often transmtted n compressed form before t can be dsplayed. Progressve refnement of the compressed representaton s a partcularly useful feature, snce t provdes an nteractve user at any gven pont n the transmsson wth a prevew of the whole mage nformaton. As explaned n Secton 2, the proposed rectangular FshEye-Vew s coupled wth a transmsson system whch supports compresson and *Thanks to German Academc Exchange Servce and Unversty of New South Wales for supportng the research.

2 Fgure 1. The rectangular FshEye-Vew. an adapted progressve refnement. Secton 3 serves to famlarze the reader wth relevant aspects of the JPEG2000 compresson standard, adopted for our work. Secton 4 ntroduces a new approach for dynamc regon of nterest codng n JPEG2000, and descrbes how t can be used especally for the rectangular FshEye-Vew. Our proposal s completed by presentng results and a concluson n Sectons 5 and THE RECTANGULAR FISHEYE- VIEW Focus&Context technques are based on the reasonable assumpton that a user's degree of nterest decreases wth dstance from the focus regon. Accordngly, the rectangular FshEye-Vew apples varyng degrees of dstorton to dfferent parts of the mage, startng wth less dstorton near the focus regon, and progressng to strong dstorton near the borders of the avalable dsplay. These dstortons amount to selectve scalng (compacton) of rectangular regons from the orgnal mage, as llustrated n Fgures 1 and 2. Specfcally, the orgnal mage data s classfed nto separate RoI's each of whch s assgned a par of scalng factors, denoted H:V. Therefore, each orgnal mage regon s scaled horzontally by 1/H and vertcally by 1/V. As may be seen from the fgures, the choce of scalng factors s not completely arbtrary, snce the overall transformaton from orgnal mage to dsplay must preserve contnuty at the regon boundares. In addton to scalng parameters, each rectangular RoI may also be assgned a separate prorty value, whch s used to determne the order n whch compressed data wll be transmtted from server to clent. In order to maxmze the perceved mpact of progressve refnement, the underlyng transmsson system delvers hgher prorty mage data (e.g., the focus regon) earler than lower prorty data (context data). As the transmtted data arrves, t can be ncrementally decompressed and rendered to the user's dsplay. The exact nature of prorty values, and the way n whch they are used to control the sequence of transmtted data, s descrbed later n Secton 4. Snce some of the regons wll be scaled at clent sde, pror to dsplay n the rectangular FshEye-Vew, not all of the orgnal mage nformaton assocated wth those regons can be used. As an example, the upper left hand regon shown n Fgure 2 has scalng factors 4:4, meanng that only one quarter of the orgnal resoluton can actually be used, n each drecton. As we shall see, the multresoluton propertes of the Dscrete Wavelet Transform (DWT) employed by JPEG2000 may be exoted to transmt only the nformaton whch s actually relevant to the dstorted FshEye-Vew. 3. JPEG2000 In ths secton, we descrbe some of the most relevant features of the JPEG2000 mage compresson standard [J2kBA]. Whle JPEG2000 does generally provde superor compresson effcency to that of JPEG, ts most sgnfcant advantages rest wth ts support for progressve refnement of the mage nformaton n both resoluton and qualty, along wth ts support for regon of nterest access to the mage. Fgure 2. Appled scalng to the dfferent regons of nterest n X and Y drecton.

3 3.1 Multresoluton Decomposton and Compresson The JPEG2000 standard employs a dyadc multresoluton transform, known as the Dscrete Wavelet Transform (DWT) to analyze the mage nto a collecton of subband mages, each of whch contans nformaton from dfferent spatal frequency bands. Each successve resoluton level n the mage representaton conssts of three detal subbands, whch augment the nformaton avalable from the prevous resoluton level, havng half the number of samples n each drecton. One detal subband contans the nformaton requred to double the mage resoluton n the horzontal drecton; another contans the nformaton requred to double the resoluton vertcally; whle the thrd completes the nformaton requred to double the resoluton n both drectons. Fgure 3 llustrates the detal bands assocated wth two resoluton levels, together wth a separate lowest resoluton mage (top left n the fgure). The DWT coeffcents n each subband are further sub-dvded nto code-blocks, each of whch s ndependently encoded. The encoded code-block btstreams are then grouped nto contaners called precncts. Each precnct contans a number of correspondng code-blocks from every subband of a sngle resoluton level (Fg.3). Precncts form the base for the creaton of the fnal compressed btstream, whch s explaned n the next subsecton. 3.2 JPEG2000 Btstream Formaton Precnct structures play an mportant role n enablng resoluton- and regon-based access to JPEG2000 compressed mages. The precncts can have dfferent dmensons n each resoluton level and are used to lay down rules for the creaton of packets, whch form the fnal btstream. To allow progressve refnement of the mage representaton, the encoded data assocated wth each code-block s spread over a number of layers. Ths s generally done by an optmzaton procedure, whch ams to unformly ncrease the mage qualty by smlar perceptual ncrements n each successve layer. The fnal JPEG2000 btstream s constructed by concatenatng a lst of so-called packets, where each packet represents the compressed data contrbutons to a sngle layer, from code-blocks belongng to a sngle precnct. Thus, each precnct contrbutes one packet for each layer. Fgure 3. Multresoluton decomposton n JPEG2000 usng dfferent scalng values (blue: subband areas belongng to a sngle precnct). In the next secton, we brefly revew exstng proposed methods for adaptng the mage data transmsson, based on RoI's. 3.3 Regons of Interest n JPEG2000 In most of the proposed schemes for RoI encodng n JPEG2000, the supported regon has to be defned durng the mage encodng process [J2kBA,J2k2A,WBEB02,GCT00]. Ths has the drawback that the RoIs cannot easly be defned dynamcally, n accordance wth a user's nterests. One scheme has been proposed n [RSI02] to support dynamc regons of nterest. Here, JPEG2000's flexble packet syntax s used to adjust the sequence n whch compressed data packets are transmtted, so as to reflect the relevance of dfferent precncts to a regon of nterest. Even though the order of the compressed data s determned dynamcally, the transmtted nformaton always consttutes a complant JPEG2000 btstream, whch can be reconstructed and rendered progressvely usng a complant JPEG2000 decoder. No further RoI handlng s requred at the clent sde, and the decompressed data s sutable for dsplay wthn a rectangular FshEye-Vew. The next secton s concerned wth partcular consderatons for enhancng the effcency and sequencng of transmtted nformaton to be used wth the FshEye- Vew.

4 4. DYNAMIC ROIS FOR THE FISHEYE-VIEW The basc dea of our approach s the prortzed transmsson of JPEG2000 packets, so as to mprove the effcency and rate of refnement of relevant regons n the FshEye-Vew. It s helpful to dfferentate between refnement n resoluton and refnement n qualty. Wth regard to resoluton, we take advantage of the fact that not all of the orgnal mage resoluton wll actually be dsplayed n nonfocal regons -- those wth scalng factors H > 1or V > 1. Ths means that some orgnal mage subbands can be dscarded n such regons, wthout any loss of nformaton. We use the term Level of Detal (LoD) and refer to the collecton of DWT subbands whch are relevant to the FshEye-Vew n any gven regon as the fnal LoD. These are the subbands, b, whose scalng factors H b : V (see Fg.3) satsfy b H b H or V b V, where H : V are the scalng factors for the regon. By prunng the transmtted nformaton to avod data whch les beyond the fnal LoD, valuable transmsson bandwdth can be saved. Wth regard to qualty, we take advantage of the fact that the perceved mpact of any gven layer on vsual qualty s generally lower n the context regons than the focus regon. Equvalently, unform perceved mage qualty generally nvolves a smaller number of layers for precncts whch contrbute only to the context regon than for precncts whch contrbute to the focus regon. One reason for ths s smply that the focus regon s by defnton more mportant to the nteractve user. Another reason s that non-focus regons are scaled. Scalng an mage regon by factors H : V reduces the mpact of quantzaton on mean squared error n the rendered mage by the factor HV. Although formal means may be derved for accommodatng ths later phenomenon through a resequencng of the orgnal packets, t s less clear how the former aspect should be accommodated n a rgorous manner. For the purpose of the presented study, a smple prortzaton scheme s employed to determne the number of layers from a context regon whch should be consdered perceptually smlar to a number of layers from the focus regon. To create a complant RoI-enhanced btstream whch can be easly decoded at clent-sde, our approach s based on [RSI02]. The orgnal mage s compressed only once, nto a JPEG2000 compressed btstream, whch can be used to serve clents wth dfferent focus regons. For any gven rectangular FshEye- Vew, specfed by the remote clent, a server determnes the RoIs, ther scalng values and ther fnal LoDs. For each RoI and each resoluton, the server determnes the set of precncts from that regon whch contrbute (ether completely, or n part) to the RoI. A precnct contrbutes to the RoI f ts spatal regon of support overlaps wth that of the RoI and ts resoluton contans one or more subbands whch belong to the RoI's fnal LoD. The set of contrbutng precncts determnes the collecton of packets from the orgnal compressed mage btstream whch are relevant to the FshEye- Vew. An RoI-scheduler then determnes the order n whch these packets are to be transmtted. The RoIenhanced btstream s created layer-wse, startng wth the lowest layer. To ensure that the sequence of constructed layers forms a complant JPEG2000 btstream, each layer must contan at least a place holder for every packet n the orgnal mage. However, only the contrbutng precncts are assgned non-empty packets n the current layer. All other precncts are assgned empty packets, whch contrbute no compressed data from ther code-blocks to the layer beng constructed. In order to explot the dfferent prorty assocated wth precncts whch contrbute to dfferent RoIs, the scheduler generally creates a transmtted btstream whch contans more layers than the orgnal btstream. The frst layer created by the scheduler contans all frst packets from those precncts whch contrbute to the hghest prorty RoI (the focus regon), wth all other packets empty. Subsequent layers created by the scheduler contan later packets from the precncts whch contrbute to the hghest prorty RoI and earler packets from the precncts whch contrbute only to lower prorty RoIs (context). For our current work, a smple prortzaton scheme s mplemented, wth each precnct,, assgned an nteger prorty ndex, p. Suppose that the ndex of the frst layer not yet transmtted from precnct s l { 1 l < L}, where L represents the number of layers n the orgnal JPEG2000 data stream. When the scheduler creates a new layer for transmsson, t ncludes a new packet for precnct only f ths s a contrbutng precnct and l p l p for all contrbutng precncts, j. Ths polcy gves a regon r, havng hgher prorty, a compettve edge of a pa p b layers over a regon r, havng lower b prorty. As descrbed n Secton 3.1 every packet contans mage nformaton from every subband of a resoluton level. Thus, a certan packet may contan more mage data than necessary to refne a scaled regon. Ths happens n every RoI where the fnal LoD contans some but not all subbands from a j j

5 resoluton level. These RoIs have scalng factors H : V wth H V. To avod such redundant transmsson, t s necessary to generate new compressed data packets, whch contan non-empty contrbutons only from the code-blocks whch belong to subbands wthn the fnal LoD. The server can perform the necessary repacketzaton of codeblock contrbutons at some expense n computatonal complexty and sgnalng overhead wthn the JPEG2000 packet headers. As we shall see n the next secton, however, ths repacketzaton can have a sgnfcant benefcal mpact on the effcency wth whch data s transmtted for the FshEye-Vew. The dynamcally created packets can be scheduled n the same manner descrbed prevously. Note, however, that the RoI-schedulng task becomes more complex, snce t must keep track of both orgnal mage packets whch have been fully transmtted, and also orgnal packets whch have been only partally transmtted. Later changes n the user's focus regon may requre these partal packets to be completed by the transmsson of addtonal complementary packets. 5. RESULTS To demonstrate the effectveness of the proposed approach, we have mplemented a clent/server mage transmsson archtecture wth a b-drectonal communcaton. The clent smply conssts n a GUI, to defne and dsplay the rectangular FshEye-Vew, a JPEG2000-complant decoder to convert ncomng mage data, and a communcaton component to handle data transmsson from the server and the delvery of request parameters to the server. The server ncludes a transcodng element, whch dynamcally transcodes an orgnal btstream nto one wth the RoI-enhanced layerng, as descrbed above. The transcoded RoI-enhanced layers are transmtted to the clent packet by packet, n a sequence whch s compatble wth the JPEG2000 data stream syntax. We use the Kakadu JPEG2000 tools [Kak02] to accomplsh ths dynamc transcodng, ncludng, where necessary, the repacketzaton operatons descrbed n the prevous secton. Fgures 4 and 5 present numercal results whch ndcate the mpact of the proposed dynamc RoIs on transmsson effcency. The upper curve, marked orgnal n each of the fgures, corresponds to the orgnal transmsson scheme proposed n [RSI02]. In ths scheme, all nformaton relevant to the FshEye- Vew must eventually be transmtted to the clent. A second curve n each fgure dentfes the performance of our proposed transmsson scheme, where orgnal packets are not sub-dvded to accommodate dfferences n the horzontal and Fgure 4. The length of a layer ncreases wth the number of non-empty packets to nclude. vertcal scalng factors. The thrd, lowest curve, n each fgure corresponds to the proposed scheme, usng repacketzaton to avod the transmsson of subbands whch do not belong to the fnal LoD of any RoI. Both fgures correspond to the same mage, RoI parameters, and transmsson sequence, wth Fgure 4 dentfyng the sze of each transmtted RoI-enhanced layer, and Fgure 5 dentfyng the cumulatve sze of the layers. The orgnal mage consdered here has a Fgure 5. Comparson of the dfferent methods usng the length of the fnal btstream.

6 sze of , and a compressed representaton of length 727 kb, consstng of 9 layers, wth 2 levels of DWT decomposton, as n Fgure 3. Addtonally, we use precncts of sze n the spatal doman and shrnk ths sze progressvely for precncts n lower resoluton levels, by factor 2 n each drecton. Thus, precncts n every resoluton level contrbute to a spatal area of approxmately the same sze. Ths polcy for szng precncts lmts the extent of ther spatal contrbuton and helps to speed up our mplementaton. The cumulatve bt-rate assocated wth each successve layer n the orgnal btstream s 2 tmes that of the prevous layer. The focus regon for the FshEye-Vew has been placed n the center of the mage, havng 1/8 of the full mage wdth and heght. A prortzaton ndex, p, of 3 s assgned to the precncts whch contrbute to the focus regon. The prortzaton ndex s reduced to 1 n the frst context belt (regons scaled by at most 2 n ether drecton), and 0 n the outer belt (regons scaled by at most 4 n any drecton). Accordngly, the fnal RoI-enhanced btstream contans 12 layers. The results presented n Fgures 4 and 5 are easly understood. The orgnal scheme produces the largest layers, and hence has the lowest transmsson effcency, because all subbands from all resolutons must be ncluded n both focus and context regons. The performance of the FshEye-adapted transmsson scheme s clearly superor, and most notably so when repacketzaton s used to avod redundant transmsson of subbands whch are not relevant to the scaled RoIs. From Fgure 4 we see that the rato between the most and least effcent transmsson schemes can be as large as 4.5:1. Ths s because RoIs whch have large scalng factors generally represent the largest porton of the orgnal mage, and these are the very RoIs whch stand to beneft most from the proposed FshEye-adapted transmsson scheme. Evdently, the effect of FshEye adaptaton s sgnfcant only n the later layers. Ths s because the prortzaton scheme delays the schedulng of context nformaton untl later layers; FshEye adaptaton affects only the packets whch belong to the context regon, not the unscaled focus regon. The strong ncrement n transmtted length observed at layer 4 corresponds to the frst appearance of packets from the second belt; these represent the largest regon on the orgnal mage. The cumulatve layer szes presented n Fgure 5 are perhaps more ndcatve of the overall mpact of our proposed transmsson scheme, revealng a maxmum overall gan of approxmately 3:1 n transmsson effcency relatve to the orgnal approach. Wth regard to objectve and subjectve qualty, the decoded and dsplayed FshEye-Vew mage appears to be almost dentcal (cp. Fgure 6) usng all three methods compared n Fgures 4 and 5. Ths confrms our asserton that subbands not belongng to a regon's fnal LoD can be safely omtted. (orgnal) (non-dvded) (dvded) Fgure 6. The scaled mage content appears almost dentcal for all three methods f dsplayed wthn FshEye-Vew. Dvdng packets nto ther subband contrbutons places ncreased demands on the server's computatonal resources. The addtonal state memory requred to keep track of these subband contrbutons, also places demands on the server's resources. Nevertheless, even low powered servers can beneft from our proposed approach to delverng orgnal undvded packets, snce fewer packets must be handled than n the orgnal case. Regardng processng at the clent sde, both the proposed approach wth undvded packets and the proposed approach wth full repacketzaton tend to reduce the clent's complexty, snce they reduce the amount of data whch must be decoded. Wth the orgnal scheme, a consderable amount of data may be decoded whch does not substantally contrbute to the rendered FshEye-Vew. In fact, the lowest clent complexty s acheved when the server repacketzes all orgnal packets whch contrbute only partally to the fnal LoD. Even though the results presented here suggest that sgnfcant gans n transmsson effcency can be obtaned by a FshEye-adapted transmsson scheme, t s worth notng that we are unable to fully elmnate the transmsson of redundant nformaton for regons

7 wth certan scalng parameters. As shown n Fgure 2, the FshEye-Vew nvolves some RoIs whose horzontal and vertcal scalng parameters can dffer by a factor greater than 2. Unfortunately, the JPEG2000 decomposton scheme (see Fg.3) supports dfferent horzontal and vertcal resolutons only up to a factor of 2, so we are forced to transmt redundant nformaton for some regons. Consder, for example, a regon wth scalng parameters 4:1, as shown n Fgure 2. Such a regon must be reconstructed at 1/4 the orgnal resoluton n the horzontal drecton, but at full resoluton vertcally. Ths suggests that only the subbands marked 4:4, 4:2 and 2:1 n Fgure 3 need actually be sent to the decoder. The total number of transmtted subband samples n ths case s 3/8 of the orgnal mage samples, whle a perfectly effcent scheme would send only 1/4 the orgnal samples. Ths necesstates a transmsson of 1/8 more mage samples, but ensures that our approach s stll JPEG2000-complant. 6. CONCLUSIONS In ths paper, we have proposed new methods for transmttng a JPEG2000-complant btstream, so as to allow effcent remote browsng of large mages, usng a rectangular FshEye-Vew. Ths method s based on the successve creaton of new JPEG2000 layers, usng already encoded packets from an orgnal compressed mage. After each layer has been dynamcally created, t can be transmtted mmedately to the clent. Ths can be done wthout regard for the effect whch future clent nteracton may have on the packets to be ncluded n subsequent layers. The resultng stream of transmtted data can be decoded by a complant JPEG2000 decoder, at any pont n the transmsson process, allowng progressve refnement of the dsplayed FshEye- Vew. To avod the transmsson of redundant mage data, we assgn a fnal LoD to each regon of the FshEye- Vew, based on the scalng dstortons assocated wth that regon. Only mage data whch contrbutes to the refnement of ths LoD s consdered for transmsson. Ths strategy saves sgnfcant transmsson bandwdth, wthout damagng the vsual propertes of the mage, rendered wthn the FshEye-Vew. The JPEG2000 standard further allows an ntellgent server to repacketze the orgnal compressed data on the fly, so as to elmnate specfc subbands whch do not contrbute to any fnal LoDs. Our experments show that ths can lead to further mprovements n transmsson effcency, at the expense of some addtonal computaton and state management memory n the server. 7. REFERENCES [Fur81] Furnas, G. W.: "The FISHEYE vew: a new look at structured fles", Bell Laboratores Tech. Report, Murray Hll, New Jersey, [Rau99] Rauschenbach, U. The rectangular FshEye Vew as an effcent method for the transmsson and dsplay of large mages, n Proceedngs of IEEE-ICIP'99, Kobe, Japan, [RS99] Rauschenbach, U., Schumann, H. Demanddrven Image Transmsson wth Levels of Detal and Regons of Interest, Computers and Graphcs, Vol23/6, December, [J2kBA] JPEG2000 mage codng system, part1, Fnal Publcaton Draft, ISO/IEC JTC 1/SC29/WG 1 N2678, 19, July [J2k2A] JPEG2000 mage codng system, part2, Fnal Publcaton Draft, ISO/IEC JTC 1/SC29/WG 1 N2679, 19 July [WBEB02] Wang, Z., Banerjee, S., Evans, B., Bovk, A. Generalzed btplane-by-btplane shft method for jpeg2000 RoI codng, n Proc. IEEE- ICIP 02, Rochester, N.Y., September [GCT00] Grosbos, R., Santa Cruz, D., Ebrahm, T., New approach to jpeg2000 complant regon of nterest codng, SPIE 46th annual meetng, Applcatons of Dgtal Image Processng, July [RSI02] Rosenbaum, R., Schumann, H., Flexble, dynamc and complant regon of nterest codng n JPEG2000, n Proc. IEEE-ICIP02, Rochester, N.Y., September [Kak02] KAKADU, a comprehensve, heavly optmzed, fully complant software toolkt for JPEG2000 developers, Verson 3.3,