Real Time File System

Transcription

1 Real Time File System For Content Distribution Heiko Sparenberg (Fraunhofer IIS)

2 Contents Introduction Composition of Scalable Media Substitution Strategy Reading Scalable Media Caching Scalable Media Applying User Rights to Scalable Media Conclusion

3 Introduction Various aspects must be taken into account for content distribution Display Size, Aspect Ratio Network Bandwidth Access Rights Available processing resources => Generation of many proxy files

4 Introduction Today, the use of scalable media is marginal Technology offers exiting opportunities for Real-time capability Access management when used on file system layer

5 Composition of Scalable Media L=Layer (Quality) R=Resolution C=Component

6 Substitution Strategy Due to the structure of JPEG 2000 files it is possible to substitute information with a series of 0 values. Substitution Strategy leads to a standard compliant code stream comprising The file size of the original file The internal structure of the original file!!! Of course!!! The new code stream contains less information; quality or resolution may be reduced.

7 Reading Scalable Media P0 P1 P2 P3 P4 P0 P1 P2 P3 P

8 Reading Scalable Media Ancillary Conditions Real-time playback is required (e.g. 24fps, 2k) Scalable media files are available on the storage device Used file system is able to recognize scalable media data Read time for each image: 1/24s Not interruptible when using non-scalable media files Interruptible when using scalable media files

9 Reading Scalable Media Operating System Storage Playback Software

10 Example #1 File request without Substitution Strategy

11 Reading Scalable Media Operating System Storage Playback Software

12 Reading Scalable Media Operating System Storage Playback Software

13 Reading Scalable Media Operating System Storage Playback Software

14 Example #2 File request with Substitution Strategy

15 Reading Scalable Media On top of this strategy an algorithm for filling the cache can be applied as well: All data will be stored in the Cache (even incomplete data portions) Read-technique will continue to read where it stopped on a further request Frequently requested images will increase in quality Real-time conditions are still valid

16 Example #3 File request with advanced Cache Strategy

17 Eviction Strategies Eviction Strategies for non-scalable media do not use the advantages of scalable coding: First In First Out (FiFo) Last In Last Out (LiFo) Least Recently Used (LRU) Development of special eviction strategies for scalable media seems appropriate >>> Due to the structure of scalable files certain components can be skipped rather than skipping whole files <<<

18 Eviction according to scalability Packets can be evicted using their scalability Resolution Quality Component

19 Eviction according to several criteria Example: Cache runs full for the first time: resolution-components shall be reduced Cache runs full again: components for the image quality shall be removed As a last resort, a file-based strategy is necessary, if the cache-limit is reached again and no packet-based strategy can free more space.

20 User Rights Management Scalable media allows for granting user rights to certain parts of a media file:

21 User Rights Management

22 User Rights Management File system Media Repackaging Component Database User-Mode Kernel-Mode Application Filesystem- Application Operating System File System Driver

23 User Rights Management DRIVE Media Repackaging Component VIRTUAL DRIVE res=50%

24 User Rights Management File system Media Repackaging Component Database User-Mode Kernel-Mode Application Filesystem- Application Operating System File System Driver

25 Conclusion Use of scalable media files can be used for speeding-up Existing caching algorithms Existing eviction strategies Read procedures A new user-rights management may reduce proxy generation

26 Conclusion Presented algorithms are backward compatible for non-scalable (media) files No need for special encoded (proprietary) JPEG 2000 files Besides of JPEG 2000 other scalable media files shall be considered in future (H.264 SVC)

27 Q&A Thank you very much for your attention

28 Conclusion Use of scalable media files can be used for speeding-up Existing caching algorithms Existing eviction strategies Read procedures A new user-rights management may reduce proxy generation 2012 SMPTE e 2012 Annual Technical Conference & Exhibition

29 Conclusion Presented algorithms are backward compatible for non-scalable (media) files No need for special encoded (proprietary) JPEG 2000 files Besides of JPEG 2000 other scalable media files shall be considered in future (H.264 SVC) 2012 SMPTE e 2012 Annual Technical Conference & Exhibition

30 Q&A Thank you very much for your attention 2012 SMPTE e 2012 Annual Technical Conference & Exhibition

31 SMPTE Meeting Presentation Real Time File System for Content Distribution Heiko Sparenberg Fraunhofer Institute for Integrated Circuits, Germany, Siegfried Foessel Fraunhofer Institute for Integrated Circuits, Germany, Written for presentation at the SMPTE Annual Tech Conference 2012 Abstract. This presentation gives a deep view into the development of a file system, especially designed for scalable media files including JPEG 2000 and H.264 SVC. By applying specially developed techniques, including the Substitution Strategy, a real-time capable file-system can be built, even if the mass storage, or the interface to it, is too slow to deliver the data in the desired time. Rather than skipping whole files, new caching strategies will be shown that again, take advantage of the file-inherent scalability. The presented system also comprises an advanced user-rights-management that allows for granting access-rights to certain parts of a scalable file, rather than granting rights to whole files. Users will therefore get a different version of an image or video, dependent on their current access-rights. Keywords. real-time, virtual file system, caching for scalable media, JPEG 2000, image/video data compression, content distribution, eviction policy (The SMPTE disclaimer is on a footer on this page, and will show in Print Preview or Page Layout view.) The authors are solely responsible for the content of this technical presentation. The technical presentation does not necessarily reflect the official position of the Society of Motion Picture and Television Engineers (SMPTE), and its printing and distribution does not constitute an endorsement of views which may be expressed. This technical presentation is subject to a formal peer-review process by the SMPTE Board of Editors, upon completion of the conference. Citation of this work should state that it is a SMPTE meeting paper. EXAMPLE: Author's Last Name, Initials Title of Presentation, Meeting name and location.: SMPTE. For information about securing permission to reprint or reproduce a technical presentation, please contact SMPTE at jwelch@smpte.org or (3 Barker Ave., White Plains, NY 10601). Copyright 2012 Society of Motion Picture and Television Engineers. All rights reserved.

32 Introduction For content distribution, various proxy files are created and stored on servers in order to supply data for several devices, differing in display size and storage capabilities. Moreover, these proxies are used to apply different access rights, e.g. for resolution and quality. For remote playback systems, it is usually most important to function in real time, whereby it is not necessarily important that these sequences are shown in full resolution and best quality. Rather, it is important to provide image and sound data in real-time using existing storage systems and data networks, ideally if these system components are even too slow to do so. In such scenarios, the use of scalable media is marginal, although this technology offers exciting opportunities to develop real-time playback systems, post-production systems with low latency as well as new distribution platforms. This paper provides an overview of the current research in the field of real-time data processing using scalable media. For this, various aspects of file processing - independent from the operating system - are examined and shown, how the application of scalable media - with particular focus on image and video data can help to improve the existing processes. Composition of scalable media with reference to JPEG 2000 [1] A JPEG 2000 code-stream is organized into data units. Each data unit contains information for a scaling level, such as resolution, quality or colors. With this structure, it is possible for a JPEG 2000 decoder, to read for example a lower resolution or quality of a scalable image. By simply skipping the respective data unit, a new variation of the JPEG2000 image - lacking the omitted dimension - is obtained. Here, the data units have a certain dependence on one another. Figure 1a shows a JPEG 2000 comprising three colors, two quality layers as well as three resolution levels. [3] Figure 1: a) JPEG 2000 encoded image with progression order LRCP comprising two quality layer (L), three resolution layer (R) and three color components (C) b) New version derived from original image with reduced quality (L) c) New version derived from original image with reduced resolution (R) d) New version derived from original image with reduced quality and reduced resolution. [3] Copyright 2012 Society of Motion Picture and Television Engineers. All rights reserved. 2

33 New methods for handling scalable media files Replacing missing data units using the Substitution Strategy When running applications in a real-time scenario, several processes will be confronted with the situation that certain data is not available at a certain time in the right place or that data cannot be processed fast enough. Due to a variety of new technologies, real-time processing of JPEG 2000 has been significantly improved in recent years and is therefore hardly a problem any more. The so-called Substitution Strategy [2] can counteract the problem regarding missing data - at least to a certain extent. With this strategy, data units of a JPEG 2000 image - that are nonavailable or incomplete while a certain file is requested get replaced with corresponding zero values 1. These zero-values do not take up space in the memory or cache and can be generated by the processor during a file request. By this method, an original image I n gets transformed by the Substitution Strategy to I n *. The structure S(x) of both images is identical, it applies that S(I n ) = S(I n * ) - this is important for the requesting process, since it would not expect spontaneous changes in the file structure and/or a file size nor would it be able to react on theses changes. Rather, the substitution method is an abstract operation for the inquiring application, which retrieves a standard-compliant JPEG 2000 image, which it can decode (see Figure 2). Filling the file cache by taking advantage of the scalability The request of a file is triggered by a process, and sent to the operating system. It is the task of the operating system to manage the connected hardware logically and to forward incoming requests to the appropriate devices. The use of cache memory is common practice, reasonable and proven for decades. Most cache strategies store data that has been read from the massstorage - in a specific area of the main memory, so that this buffer can serve additional requests to the same data if requested again. When using non-scalable media, the files are fully read from slow storage and delivered to the calling process. In parallel, the data is also stored in the operating systems file-cache. The required time for such an access is hardly predictable and therefore only partially useful for real-time applications. Nevertheless, the approach is useful here, since the calling process can only interpret the whole file. When using scalable files, the existing method can be optimized, so that parts of a file can already be sent to the calling process, even if missing data units could not yet be read from the mass memory. The calling application can then process a reduced variant (e.g. in terms of resolution or quality) of the original image. Especially for real-time systems, this approach is advantageous since the read-routine can be interrupted if the available time has been exceeded. For this, the file system needs information about how much time is available for the read-process. By this method, a response time of the file system can be predicted very accurately and real-time requirements can be met. As with the use of non-scalable data, other requests to the same file can be served from the cache, if the data has not been previously removed by an eviction policy. 1 The reason why the data units may not be available, is not relevant at this point Copyright 2012 Society of Motion Picture and Television Engineers. All rights reserved. 3

34 P0 P1 P2 P3 P4 P0 P1 P2 P3 P Figure 2: Comparison between the original JPEG 2000 image comprising five quality packets (left image) and the resulting JPEG 2000 image after applying the Substitution Strategy (right image) where three of the five packets (P2-P4) have been replaced by zero-values. However, it turns out that the presented method comprises a further state: Usually, a file resides completely in cache (Cache Hit) or not (Cache Miss).The presented method offers a new state of partially cached files. Since enough space for all data units of a certain image was reserved during the first file-access, the strategy starts to transfer additional data from the mass storage to the cache if a process calls the same image again. Of course, the subsequent accesses are performed under the same real-time requirements stated above. Eviction Policies for Scalable Media In addition to the read-procedure also eviction policies can be optimized, in order to use the scalability of media. The aim is to increase the hit rate of caching strategies and to store more images in the cache, as it would be possible with existing eviction policies - such as First In First Out (FIFO), Last In Last Out (LIFO) or Least Recently Used (LRU). The mentioned methods have the characteristic to consider cached files of an image as a whole and decide whether a file has to be removed from the cache or not. Decisions always apply for the entire file (filebased characteristics). According to [3], more choices are available when using scalable media: Before a file is removed from the cache, it would also be possible, only to discard the highest level of resolution of all images residing in cache, thereby deallocating memory for new files (data unit-based characteristics). The following list shows a collection of data unit-based eviction policies that have been developed during the research on a file system for scalable media: 1. Eviction of incomplete data units. Due to the described read-procedure incomplete data units may reside in cache. These data units have been replaced with zero-values using the Substitution Strategy when a process Copyright 2012 Society of Motion Picture and Television Engineers. All rights reserved. 4

35 requested the file. Also these fragments are ideal candidates for eviction when the cache reaches its memory limit. 2. Eviction as a function of scalability Data units can be evicted as a function of a scaling dimension. For this purpose, a reduction of quality and/or resolution seems to be appropriate. This strategy allows for setting a limit, defining how much a certain dimension can be reduced. The current implementation rejects the respective data units of all images that are currently present in the cache. 3. Eviction as a function of the data unit size This method calculates separately for each image - which dimension shall be removed, in order to free the maximum (or minimum) space in cache. The size of the data units varies from image to image. So it may be useful for some images in the cache, to discard the maximum quality, as this frees most space, while for other images the highest resolution level must be discarded, in order to achieve the maximum eviction. 4. Combination of several eviction policies Moreover, there is the possibility to combine different policies. For example, the quality can be reduced to 80%, before the resolution can be reduced to a 75% when the cache runs full again. Subsequently, the quality of the images can be reduced again to 55% and so on. Assuming, of course, that the source images comprise enough scaling components to apply these policies. If a process attempts to read a file from the cache, after one of the policies has been executed, the system will try to read missing data units from the mass- storage, taking the real-time condition into account. Completed data units that were taken from the cache, or read from the mass-storage, are then passed to the calling process. Again, incomplete or missing data units will be replaced using the Substitution Strategy. The described method, however, cannot free cache memory in any case. Rather, in the event that no more data can be removed by the data unit-based methods - for example, once all images have been reduced to its minimum quality and resolution - a file-based policy has to be applied. Thereby, memory is released for sure. Advanced user-rights management for scalable media Also in the area of rights management used in file systems, the scalability can be utilized effectively: In a conventional file system certain access rights - such as Read, Write, Edit or Create are granted to complete files and folders. When using scalable media such access rights can be granted more finely. This makes it possible to provide the full quality and resolution of an image-sequence to one user, whereas another user only has access to a smaller version of the same images. In contrast to the proposed methods for filling a cache as well as the application of eviction policies for scalable media - a different approach has to be chosen here: As shown in [4], it should be transparent to users that a plurality of variants are provided (and that they are not allowed to access the best version of the file) above all. Instead, the file system must keep this process abstract. If a user has the appropriate access rights to access the data without restrictions (full resolution/best quality), the additional user rights layer will simply forward the file to the calling process. If the user has limited access rights, an interposed module creates a new variant of each requested image, when a user requests it. Fortunately this is no problem at all Copyright 2012 Society of Motion Picture and Television Engineers. All rights reserved. 5

36 since compression methods, such as JPEG 2000, support the mechanisms for the reorganization of the data units. It is worth to mention that this process does not decode and scale the original JPEG 2000 file, before it encodes it again to a new JPEG 2000 file. Rather, all the steps will be applied by simply reordering the data units - a process that can be executed very fast (> 1000 fps 2 ) by today's computers. Before the new variant can be sent to the calling process, a new header needs to be generated, which describes the properties of the new, virtual file (e.g. file size, dimension). The advantage of this implementation is obvious. Instead of creating many different versions of a file in advance and to distribute these files to the appropriate users, detailed access rights are assigned to scalable (media-) files. Depending on the assignment of rights, the user can then call up their version of the scalable media. This is done by simply rearranging the JPEG 2000 data units and adding a new header in front of these data units. Current Implementation The current version of the file system has been realized using a virtual file system, which runs in the so-called User-mode of the operating system. This has the advantage that the development of programs in User-mode is much easier compared to the development of software in the socalled Kernel-mode. In contrast to the development of modules running in Kernel-mode (e.g. drivers), there is no need to care about specific implementation details of the used operating system. However, a process running in User-mode that accesses a lot files from a mass storage needs more time to execute, since the operating system must repeatedly switch between Userand Kernel-mode - a process that takes a while. Fortunately, there are several open-source libraries that can be used for the development of virtual file systems. In addition to the Filesystem in USErspace (FUSE) library [2] for Unix operating systems, the Windows variant Dokan [3] is used for the current demonstrator. Thus, the virtual file system can be started and operated using Windows, but can then be mounted as a network drive using any other operating system. The assignment of access rights is represented by a graphical user interface, which also allows for selecting the used eviction strategy. During execution, the current cache level can be determined. Conclusion The use of the scalability of images in media files can be used effectively in many areas of computer science, in order to speed up systems in different scenarios. The data unit-based cache-filling methods can be useful, to ensure real-time processing of a file system even if the drive or the interface is too slow for real-time applications. Of course, one could also reach this behavior even by using faster storage systems and interfaces, but not in all areas of digital image processing it is possible to deploy such technologies. Thus, many of todays Digital Cinema Packages (DCP) are sent with hard drives that only support USB 2.0 [7]. Real-time playback without copying the data to a faster storage is not possible. The same can be said for slow USB sticks or DVDs that are currently used for the distribution of trailers and advertisements to digital equipped cinemas. Further it can be expected, that data rates, frame rates, image sizes, as well as used components within one picture will increase in the future. Today, the industry talks about higher frame rates in digital cinema and current interfacing technology, such as Thunderbold [8] and USB 3.0 [9] can deliver these data rates easily. But in future, applications like real 3D, Lightfield, 2 Among other variables this value depends on the file size of he managed JPEG 2000 images. Copyright 2012 Society of Motion Picture and Television Engineers. All rights reserved. 6

37 8k and others will become reality and drastically raise the amount of data needed per frame. In contrast, real-time playback will always be a basic requirement - in production, post-production, and of course at the screening facility. In addition, even very fast storage devices will turn slow if the number of users - and thus the number of file requests - increases. Granting access-rights to parts of a scalable media file can help to accelerate workflows and to reduce the number of proxy files. The expansion of the existing system will be part of further research. Especially the development of more eviction policies - as well as a port into a non-virtual file system will be focused in the future. References [1] ISO/IEC , Information technology JPEG 2000 image coding system Part 1: Core coding system, 2000 [2] H. Sparenberg, M. Martin, S. Foessel, Real-time Capable File System for Scalable Media, Picture Coding Symbosium, May 2012 [3] H. Sparenberg, M. Martin, S. Foessel, Introduction of Eviction Strategies for Caching Scalable Media Files, International Conference of Digital Information Management ICDIM, August 2012 [4] H. Sparenberg, A. Schmitt, R. Scheler, S. Foessel, K. Brandenburg, Virtual File System for Scalable Media Formats, 14th ITG Conference - Dortmunder Fernsehseminar, 2011 [5] H. Csaba, S. Miklos, Filesystem in Userspace, 2004, 23 th February 2011, < [6] Dokan Project, 2007, 23 th February 2011, < [7] Universal Serial Bus Specification Revision 2.0, 23 th April 2000, < [8] Thunderbold TM Technology, 28 th August 2012, < [9] Universal Serial Bus Specification Revision 3.0, 6 th June 2011, < Copyright 2012 Society of Motion Picture and Television Engineers. All rights reserved. 7