MRR (Multi Resolution Raster) Revolutionizing Raster

Transcription

1 MRR (Multi Resolution Raster) Revolutionizing Raster Praveen Gupta Pitney Bowes, Noida, India T M Pitney Bowes, pitneybowes.com/in 5 th Floor, Tower B, Logix Park, C28/29, Sector 62, NOIDA (UP), , INDIA 1

2 Abstract Pitney Bowes has developed a new raster data format called Multi-Resolution Raster (MRR) as part of a new raster visualization and processing package for MapInfo Professional (64-bit edition). MRR is a new raster data format for the modern computing age. For all classes of raster data, MRR maximizes storage flexibility, minimizes storage requirements and guarantees high performance data access at any scale. The MRR format supports storage of image data, classified (or thematic) data, as well as continuous and discrete data. It supports rasters of virtually unlimited size and extends the concept of a raster from a simple 2D array of cells to an extensible sparse matrix of tiles of cells. A raster in MRR format will contain a pyramid of overviews which enable high performance data access at any scale and high quality visualization. MRR achieves efficient storage using lossless and lossy compression techniques via industry standard compression codecs. It also supports data storage in the temporal dimension, allowing data to be accumulated and accessed by time. MRR supports a wide and extensible number of data types. It stores one or more multi-banded fields. It supports local registration for each field, and tile decimation. MRR contains high quality statistics and is contained within a single file on disk. Keywords: Raster; MRR, scalable; GIS; Performance; 2

3 Introduction Yes raster is faster, but raster is vaster and vector just seems more corrector. - C. Dana Tomlin Spatial analysis is about releasing insights from data and communicating those insights to stakeholders in an intuitive and easily comprehended way. Before you get to the maps, it s all about the data. Geographic Information Systems (GIS) consume data in two principle forms: vector and raster. Vector data is used for display of discrete objects (points, lines and polygons/regions). Raster datasets are able to represent data that continuously changes spatially and/or temporally - like elevation or aerial imagery. The ability to rapidly display and process raster data has been limited by the capabilities of available GIS applications and the various raster data formats. Typically, users have had to accept a trade-off between raster resolution and spatial extent. Due to performance and resource limitations, large high resolution rasters have been impractical to work with. The availability of large, high resolution raster datasets to GIS professionals is growing rapidly. For example, the resolution and coverage of remote sensing platforms increases with every new generation of hardware. The physical number of satellites and other sensing platforms in operation is increasing year on year. Storing, managing, visualizing and processing this data has become increasingly challenging for data providers and consumers. Although there are already more than a hundred raster data formats, Pitney Bowes saw a requirement for a new, unifying format to handle all types of raster data and to provide new capabilities to improve the user experience when visualizing and processing raster data. Pitney Bowes have responded to the need for large, high resolution raster datasets by introducing an innovative new raster format called Multi-Resolution Raster (MRR). MRR is a completely new raster data format that reimagines not only how raster data is stored, but what data can be stored in a raster. It is an integral part of the new MapInfo Pro Advanced raster visualization and processing solution. It provides a flexible solution for the entire spectrum of industry requirements. It is a unifying and enabling technology. It unifies the storage of all kinds of raster data: imagery, spectral imagery, continuous gridded data and thematic data. It removes the barriers to working with different kinds of raster data in the same context. It enables the highest quality visualization and processing of raster data at any scale and for a raster of any size. So, what s so special about MRR? Below are just some of the attributes that differentiate the MRR format from other existing raster formats: Supports image data, classified (or thematic) data and continuous and discrete data. Supports raster datasets of virtually unlimited size. 3

4 Extends the concept of a raster from a simple 2D array of cells to an extensible sparse matrix of tiles of cells. Efficient display of data at all zoom levels. Contains a data overview pyramid enabling data access at any scale. Achieves efficient storage using lossless and lossy compression techniques via industry standard compression codecs. Supports the temporal dimension, allowing data to be accumulated and accessed by time. Supports a wide and extensible number of data types. Stores one or more multi-banded fields. Supports local registration for each field, and tile decimation. Contains high quality summary and spatial statistics. Is contained within a single file on disk. Regardless of the size or complexity of the raster whether it contains one million or one trillion cells and whether it contains one band or 256 bands opening and displaying an MRR is virtually instantaneous in MapInfo Pro Advanced. MRR lifts the restrictions on raster spatial size, resolution and complexity and allows users to realize huge efficiency and productivity gains. Efficient storage is a pre-requisite enabling technology for large raster files. MRR provides a variety of techniques that users can employ to minimize storage requirements. These include sparse storage, flexible data type support, data transformations, control over decimal precision, predictive encoding and a variety of industry standard compression techniques using lossless and lossy encoders. MRR also allows advanced users to employ tile based decimation to reduce the resolution of the data locally, where appropriate, and further reduce storage requirements. MRR achieves this flexibility and efficiency through a compound file system which allows it to be stored on disk as a single file. An MRR is easy to share! Structure of MRR An MRR is stored in a single file on disk but internally, this file is a file system within a file. This system provides the flexibility required to build huge rasters and to enable future editing and additions to the raster. The MRR file will contain all of the metadata information, statistics, spatial maps and raster data required to display, interrogate and process the raster. An MRR consists of one or more fields and each field contains raster data that is of a particular, user designated type. There are four different raster types - Image, Image Palette, Classified and Continuous. A field of type Image contains a single-color data band. If this is a composite color data type (like RGB) then each component will also be represented as a virtual band. A field of type Image Palette contains a color palette and a single data band which stores indices into that color palette. The palette will contain a color data type which is exposed as a virtual band. If this is a composite color data type (like RGB) then each component will also be represented as a virtual band. 4

5 A field of type Classified contains a classification table and a single data band which stores indices into that classification table. The classification table fields are exposed as virtual bands. A table may contain virtually any kind of data including strings and there are no structural requirements or limits imposed except that the number of entries in the table is limited to approximately four billion. A field of type Continuous contains one or more bands of data. Each band may have a unique data type. Virtual bands are automatically available for the components of data types such as complex numbers or multi-component color. While all fields in a raster share a coordinate system, each field has its own registration transform. In practical terms, this provides an ability to specify a raster cell size for each field individually. Data for a field is stored in one or more events which are used by MRR to record data in the temporal dimension. Each event has a designated time of occurrence. In an event, the user can add new data in any spatial configuration. Hyper-spectral surveying methods create raster datasets with a large number of bands. MRR is a suitable storage format for raster data with hundreds of bands. Datasets such as these do not compromise the performance of MRR data access. Capabilities 1. Unified storage and data MRR unifies the storage of raster data. It is capable of storing all types of raster and grid data in common use. It is a particularly good vessel for hyper-spectral imagery which may contain hundreds of bands of spectral data. It can store classified data where each cell is linked to a flexible table of data. In all cases, MRR removes traditional barriers that limit the size and complexity of the raster database. It supports image and image palette data, classified (or thematic) data and continuous or discrete data. It provides a common interface to all these different types of raster. The raster is stored within a single file on disk. 2. Virtually unlimited size MRR is designed to store raster datasets of virtually unlimited size. It imposes no performance penalty when working with increasingly larger file sizes. A MRR of any size can be visualized almost instantly in MapInfo Pro Advanced at any scale. The size, resolution and complexity of the rasters you can create are limited by your available storage, and by practical considerations. Display performance is not determined by the size of the file. Opening, panning, zooming, changing the colors and other display operations are nearly instantaneous. 5

6 3. Broad data type support MRR Supports a wide and extensible number of data types including: Boolean Bit (1, 2, 4 bit) Integer (1, 2, 4, 8 byte) Real (4, 8 byte) Complex numbers (integer and real) Time String Blob objects with fixed/variable length Color 4. Sparse tile storage MRR extends the concept of a raster from a simple 2D array of cells to an extensible sparse matrix of tiles of cells. A data tile is a multi-banded array of MxN cells. An MRR is a sparse collection of data tiles, stored at multiple levels of resolution. Utilizing this sparse tile arrangement, MRR can reduce overall storage requirements by not storing regions of a raster that are empty. For example, some rasters can be extremely large but only sparsely populated with data. This model is far better suited to modern surveying techniques - like LIDAR - which may collect data over a highly irregular coverage polygon. One possible example of such a dataset is a LIDAR survey which is designed to survey a highway. Such a survey may be orders of magnitude longer than it is wide. In cases such as these, the MRR format does not store data in the empty regions, but still remains flexible enough to add additional data to the source file any time. This allows extremely sparse datasets to be stored efficiently on disk, be rapidly accessed and easily updated. 5. Extendibility An MRR can be extended as new data is acquired it does not have a fixed size. It can be extended by adding tiles anywhere and at any time. The MRR format provides a capability to store changes in raster data over time. Using the MRR format you can store and then view changes in a dataset for a point in time or over a given period of time. 6. Temporal events MRR supports the temporal dimension, allowing data to be accumulated and accessed by time. Each event has a designated time of occurrence. These temporal events are stored separately within the MRR file. In an event, the user can add new data in any spatial configuration. Events are complimentary to pre-existing data in the raster. This history of data acquisition is available to users who can visualize and process raster data at a specific time, or over a period of time. 6

7 7. Storage efficiency via compression Rasters are growing huge and it is vital to use efficient storage techniques that compress the data on disk. MRR supports a variety of compression codecs both lossless and lossy. In general, the lossless techniques are used for thematic and non-thematic data and the lossy techniques are used for image data. For image data, industry standard compressors such as PNG and JPEG can be used and new codecs can be added in the future. Through careful design, users can employ other techniques like transformation, data type selection, predictive encoding and decimal restriction to improve the compressibility of non-thematic data and minimize storage requirements for huge rasters. Typically, an MRR will achieve compression factors of between 4x and 1000x. As an example, consider the Danish 1.6-meter Digital Surface Model. This raster is supplied as 657 ESRI Arc Grid files which consume approximately 500 GB. In contrast, the same data merged into a single MRR using lossless compression consumes only 39 GB. 8. Storage efficiency via raster design MRR provides a number of other features that allow users to design an efficient and appropriate storage model for their raster data. When used appropriately, these features may help to further reduce the storage requirements. MRR allows users to choose the data type of every band in the raster individually. For any data, the most compact and appropriate data type can be selected. By choosing wisely, users can reduce both the in-memory and on-disk storage requirements. Further to this, a data band in a continuous field can have two defined data types the user visible data type and a storage data type. This can be used in conjunction with a data transformation comprising an offset and scaling factor. Before being written to the MRR, data will be transformed from the visible data type to the storage data type via offset and scaling. For example, a data band that contains floating point data can be stored as integer data via a suitable transformation. Because this will restrict the resolution and range of the data, it should only be applied when judged to be appropriate. This technique can significantly reduce storage requirements. If the stored data type of a band is integer, then a technique called predictive encoding can be applied in conjunction with a lossless compression codec. Predictive encoding is a reversible process that is applied to the data to convert it from a series of absolute values to a series of differences from a predicted value. The data, after this transformation, may be more compressible and this may reduce storage requirements. MRR provides a number of different methods to determine the prediction. Predictive encoding can also be applied to floating point data, but in this case is not perfectly reversible. For floating point data, decimal restriction can be employed to reduce the number of decimal points stored for each data value. For example, you might choose to restrict elevation data to two decimal places (one centimeter). When combined with a lossless compression codec, this can significantly reduce storage requirements. 7

8 9. Efficient memory management The MapInfo pro Advanced SDK APIs provide intelligent data caching which automatically manage the use of system memory for the user. This automatic caching system simplifies the programmer s tasks when working with extremely large datasets. 10. Variable resolution via tile decimation Many remote sensors return data that is at varying resolutions. For example, the WorldView2 satellite simultaneously records data at 0.31m cell size as well as 1.24m and 3.72m. MRR can efficiently store raster data such as this by using fields with different cell sizes. But in addition, MRR allows you to vary the cell size within individual tiles to appropriately match the spatial resolution of your source data. For example, if you have elevation data that is of high resolution in some parts and low resolution in some other places, you might want to combine the data in a single raster to perform better analysis of the area. This multi-resolution capability, which is achieved by decimation within tiles, reduces storage requirements and improves the performance of your analysis. 11. High quality statistics An MRR stores and maintains high quality summary and spatial statistics within the file. The statistics computed can include: Count - Sample count and validity statistics; Summary - Count + Summary statistics; Distribution - Count + Summary + Histogram; Spatial - Count + Summary + Histogram + cell to cell statistics 12. NULL handling MRR does not use a magic number to represent null values. Instead, a bit mask is maintained for all tiles. This mask identifies a cell as containing valid or invalid data. The mask can apply to all bands in a field, or each band can have its own mask. If the cell is invalid, then the value of the cell can be used to classify why or how it is invalid. For example, we distinguish between empty cells and null cells. An empty cell does not overprint a cell with valid data, but a null cell does. 13. Overview pyramid MRR format minimizes storage requirements and provides efficient data access to extremely large raster files at any resolution. It has been designed to deliver an enhanced visualization experience by maintaining an overview data pyramid. This guarantees high quality and efficient data visualization at any scale and that opening, panning and zooming of MRR files is nearinstantaneous. A raster has a base resolution level from which the source data is accessible. An MRR also stores a pyramid of overview levels - each at half the resolution of the previous. Data can be accessed from throughout the pyramid, allowing applications to access data at a resolution suitable for their needs. For example, when rendering to the screen or hardcopy device, we access data from a level in the pyramid where the raster cell size is close to the size of a pixel. 8

9 14. Under-views MRR also provides underviews. An under-view level contains data at a higher resolution than the base level data. MRR uses a variety of interpolation techniques such as nearest neighbor, bilinear and bicubic to provide under-views on the fly at any resolution level. Taken together, overviews and under-views provide MRR the ability to guarantee high quality, high performance visualization at any scale. 15. Thread safe, high performance SDK The MapInfo Pro Advanced SDK allows you to interact with it, and with any supported raster formats, in thread-safe manner. You can open, close, read and write rasters in parallel - safely and with the highest possible performance. Users can write their own multi-threaded code, allowing them to utilize hardware optimally. 16. Simple to use and integrate Pitney Bowes provides an SDK that can be used to develop custom applications for MapInfo Pro Advanced users. This provides access to the features of the MRR format and to supported legacy raster formats. It provides a.net/c API for application programming apart from normal MapInfo UI interface. With the MRR I/O API, users can build their own custom functionality to support specific markets. These capabilities will be expanded in future across the windows and Linux clients/servers. Conclusion With the renaissance in Earth observation satellites and venture funded Earth imaging companies, the decreasing cost of acquiring data and the emergence of unmanned aerial surveying platforms, the availability and quantity of raster data will continue to explode. In recognition of this trend, the MRR storage format has been developed to provide an efficient mechanism to store, access, edit and process modern raster data. 9