Procedural Compression: Efficient, Low Bandwidth Remote Android Graphics

Transcription

1 : Efficient, Low Bandwidth Remote Android Graphics Joel Isaacson. Copyright 2014 Joel Isaacson

2 of GUI Rendered Video Streams The compression system described allows high performance interactive graphics with very low network bandwidth The video compression system is designed for procedurally generated graphic streams rather than more general photographically generated video streams. The higher performance of the video compression enables many use cases that are otherwise impractical using standard video codecs.

3 provides: 1.Low bandwidth - typically 30 kbytes/sec 2.High frame rate - 60 Frame/Sec 3.Lossless compression - no artifacts 4.Low latency 5.Low computation complexity 6.No need to render pixels on the remote host

4 of Video Streams is based on a priori knowledge of the ensemble (collection) of material being compressed. Consider a hierarchy of video stream collections. 1. Arbitrary video streams. 2. Photographically generated (e.g. MPEG, H.264, JP9, Hollywood) video streams. 3. Computer graphics rendered (OpenGL, SKIA) video streams. 4. Computer GUI generated (e.g. Android graphics, OpenGLRenderer) video streams.

5 Entropy of Video Streams Each subsequent ensemble in the hierarchy is a subset of the previous ensemble of video streams - like a set of nested Russian dolls. The entropy, the logarithm of the volume of the collection space, gives a lower limit of the average data compression. The smaller the volume of the collection space the greater the compression ratio possible.

6 Video Stream Ensemble Space

7 Video Stream Ensemble Space

8 Visually Relevant Ensemble Space

9 Computer Rendered Ensemble Space

10 Source Modeling Being able to calculate, or estimate, the entropy of the source data gives an lower bound on the achievable compression. Unfortunately it does not simply indicate how to design an efficient compression algorithm. Normally the design of a compression scheme starts with modeling the data creation process. An understanding of both the source model and target use of the data to be compressed is necessary.

11 Visually Relevant Ensemble Space MPEG Video streams, photographically taken or photorealistically synthesized, can be compressed with MPEG standard codecs. Even though no specific technique used in MPEG compression is applicable for the compression of our problem domain (rendering streams), some of the assumptions about the source model and target use is similar.

12 MPEG Stream Assumptions The source material consists of a large number of sequential pixel images (frames). The target of the video images is the human visual system and consists of moving images. The subject matter is a product of our everyday visual world and not a series of random images. Apparent smooth motion depends on visual continuity between frames.

13 MPEG Techniques The conversion of RGB images to YUV and subsampling is motivated by the color physiological opponency theory model of human vision. Within each frame, the accuracy of spacial changes with shorter wavelengths are less important than the accuracy of longer wavelengths. Interframe compression is based on finding motion vectors of the current frame based on previous and subsequent frames.

14 Rendering Stream Assumptions The rendered material consists of a large number of sequential images (frames). The source material is generated frame after frame by repeated invocation of GUI procedures. The target of the GUI images is the human visual system. The subject matter while not being a product of our everyday visual world is modeled on the causal physical processes of this world.

15 Rendering Techniques In practice, the number of unique sequences of rendering functions in execution paths taken within the code are bounded. This is because the rendering commands are generated by a fixed number of GUI functions and an application running a bounded amount of code. The execution paths can be incrementally learned and entered into a procedure dictionary as the rendering commands are streamed.

16 Rendering Techniques Even if the sequences of rendering functions themselves are in the dictionary, the data arguments associated with these functions might be quite different from one another. Therefore, we keep a dictionary of the data arguments previously encountered for this particular sequence. As a rendering procedural sequence with associated data is encountered, the data sequence dictionary for this procedural sequence is searched for - the closest match to the current data sequence.

17 Android Contact List

18 Android Contact

19 Structured A careful examination of the SKIA rendering stream generated by the Android GUI reveals additional structural information that can be used to improve the data model. The rendering stream has balanced save() restore() pairs within each frame of the rendering stream. Each save() is found at the beginning of a GUI function and a restore() is found at the end of each GUI function.

20 Structured This information can be used to reverse engineer individual GUI and application procedures. It will also reveal the call-graph of these procedures. Using this information, the rendering code dictionary becomes a rendering procedure dictionary. The call graph data is best embedded in the per-procedure data dictionary.

21 Structured The server (encoder) constructs a procedure dictionary that is identical to the dictionary that is constructed by the client (decoder). Similar of LZW compression. The procedures in this dictionary, surprisingly, are reversed engineered toolkit and application level routines. Amazingly: we in effect have ended up exporting graphics at the toolkit-application level rather than the rendering level.

22 Structured Statistics Our compression algorithm was tested on the rendering trace of a 60 frame sequence. There were rendering commands for an average of 228 rendering commands per frame. Of the rendering commands, there were 2691 functions (save/restore pairs). Of these, only 47 were unique. This gives a compression rate of 1.75% (about 1:57).

23 Structured Statistics Of the rendering commands only 354 had completely unique data parameter sets and 203 had data sets which are partially different. Only these 557 data sets must be transmitted, thereby giving data compression of 4.06% (about 1:25). If the partially different data sets are differentially transmitted, a data compression of 3.3% (about 1:30) is obtained.

24 Structured Entropy Encoding For the last stage of compression, general techniques such as run length encoding (RLE) and entropy encoding (Huffman or Arithmetic) are used to produce a minimum bit count representation of the compressed material. This additional phase should be expected to reduce the number of bits by a factor of 2-3.