The Power of In-Memory Computing for Intelligence Missions WHITE PAPER

Transcription

1 The Power of In-Memory Computing for Intelligence Missions WHITE PAPER

2 OVERVIEW Intelligence missions increasingly require near-instant analysis of large data sets from a huge variety of sources. SAP has collaborated with Intel Corporation, leveraging technology advances in main memory, multicore processing, and data management, to deliver radically better computing performance at less cost. This revolutionary new approach delivers significant performance increases for US national security missions. A NEW PARADIGM IN DATA MANAGEMENT: SAP HANA TM SAP, the worldwide leader in enterprise software, has pioneered a game-changing approach to the problem of managing and analyzing big data. SAP s investment of more than 5,000 man-years by more than 1,500 engineers has resulted in what is now the fastest-growing solution in the company s 40-year history. Coinnovation with Intel Corporation which wrote HANA-specific instruction sets into their newest generation of chips, allocating 400 engineers to the project means that software and hardware are optimized to deliver an unparalleled increase in speed for complex analysis of multi-billion-record data sets. 1 Because the SAP HANA software was written from inception to leverage the new Intel chipsets and main memory architecture, it delivers a level of performance that significantly exceeds that of traditional database management systems. SAP HANA is a unique approach to the problem of operating on very large data sets, because the logic to be applied to the data, and all of the data that will be subject to processing, are both contained in main memory at all times. Because external memory either in solid state or spinning disk drives is not part of the computational path, SAP HANA is a High Performance Computing environment that can achieve speeds many thousands of times faster than traditional databases. Yet because it is based on open standards, SAP HANA can deliver this acceleration to any existing tool or user interface that already is in use in your enterprise. The skills needed to operate the SAP HANA environment and configure an analytical process are standard and readily available. Structured Query Language, Regular Expressions, the R library of predictive analysis algorithms, and JDBC are ubiquitous skills that already are extant in your enterprise. The power of SAP HANA is the unparalleled ability to combine transactional and analytical applications, structured and unstructured data, real-time streaming and historical data in one platform, unleashing the ability to do work that was previously impossible. 1 The SAP HANA software's data page, pointer strategy, vector processing, L3 cache-line size management are all optimized for the Intel chipset architecture and Intel's Streaming SIMD Extensions. SAP HANA is in the regression suite for Intel chips, so future chips will continue to support the HANA-specific instruction sets. Because the HANA software was written from inception to leverage this new hardware architecture, it delivers a level of performance that exceeds that of traditional database management systems, even when they are accelerated by caching of tables. SAP National Security Services TM SAP NS2 TM 2

3 OPTIMIZING THE ARCHITECTURE FOR THE MISSION: SAP HANA AND HADOOP A few years ago, the Hadoop Distributed File System was created to speed work on very large data files. The innovation of Hadoop was to bring the logic to the data by splitting up large data files into small chunks spread across hundreds or thousands of computers. 2 This is an excellent way of doing batch processing of large, unstructured files. But the read-only, batch-processing nature of Hadoop is a limitation if the goal is to deliver iterative, ad hoc analyses to a mission or business analyst. So organizations that need to perform real-time ad hoc analyses have continued to struggle with Big Data, because the repetitive, iterative queries that analysts need to perform as part of their natural pace of work requires a great deal of attention from highly skilled data scientists to write and revise the Map/Reduce programs in Hadoop. SAP HANA, in contrast, is optimized for real-time analyses of massive data sets. Trillion-record scans can be performed very quickly. Any analysis front-end tool can invoke the high performance queries in SAP HANA using standard interfaces. 3 Data scientists do not need to write complex programs to do this work. In addition, SAP HANA is a fully ACID-compliant 4 columnar data base, so queries and algorithmic computations can be performed with direct access of selected data elements instead of having to read the entirety of a large data file over and over. SAP HANA combined with Hadoop offers the advantages of both these new paradigms: the ability to absorb any sort of data needed for the mission or the business into a large unstructured bit-bucket file in Hadoop, where it can be accessed with transparent data federation from queries in the SAP HANA platform. This gives users the ability to visualize and analyze big data at the speed of thought using SAP HANA s in-memory processing. SAP HANA performs analyses for predictive algorithms, link analysis, unstructured text analysis, streaming event processing, and geospatial queries on multi-terabytes to petabytes of data--all in one unified platform. In addition, in order to optimize the costs and benefits for the mission, the SAP HANA platform works well in an integrated architecture (shown below) that can include storage of "hot" data in the in-memory HANA database; "warm" data on-disc in the proven SAP Sybase IQ high performance columnar data base; 5 and "cool" data in the Hadoop Distributed File System. In this way, the platform is able to provide userdriven analytics across the variety of data available in modern environments. Federated queries across the three "temperatures" of data simplify the operation of the platform. The tremendous computing power of this combined platform is made available to any application in your enterprise, and cross-agency data sharing is facilitated because the platform uses standard interfaces. SAP HANA differs from other in-memory approaches in that HANA is not merely caching a sub-set of database tables in DRAM, which then need to be recached whenever an update is performed. SAP HANA uses sophisticated dictionary encoding and differential buffering to leverage the full potential of the newest developments in multi-core processing, larger cache lines, and huge amounts of main memory available from hardware manufacturers such as Cisco, Dell, Hitachi, HP, IBM and others. 2 Also, no structure or context is imposed upon the data until it is read and organized by the MapReduce program (this makes Hadoop schema-on-read ). 3 SAP HANA supports SQL, MDX/XMLA, BICS, JDBC/ODBC, JSON/XML and other open standards. 4 ACID (Atomicity, Consistency, Isolation, and Durability) is a set of properties that guarantee that database transactions are processed reliably. Both disaster recovery and fault tolerance are supported. 5 SAP HANA smart data access provides data virtualization capabilities that expedite dynamic data queries across heterogeneous relational and non-relational database systems such as Hadoop, SAP Sybase Adaptive Server Enterprise (SAP Sybase ASE), SAP Sybase IQ, Teradata, and SAP HANA itself. SAP National Security Services TM SAP NS2 TM 3

4 Fig 1. The SAP Real-Time Data Platform, powered by SAP HANA, integrates with all data sources and with any front-end tool using standard interfaces. SAP National Security Services TM SAP NS2 TM 4

5 THE SAP REAL-TIME DATA PLATFORM, POWERED BY SAP HANA Delivered as an integrated platform, the SAP Real-Time Data Platform combines the in-memory data base with industry-leading capabilities such as event stream processing and data integration needed for national security missions. Capabilities include: A graph engine to enable link analysis. Sophisticated natural language processing (NLP) text analysis capability for reading unstructured text and extracting entities to create context and meaning. Geospatial capabilities for querying inside a polygon on a map or up to a boundary. The R library of predictive analysis algorithms. Running predictive models on HANA s high-speed platform can identify in real time patterns of behavior that are indicative of activities such as attack planning, money laundering, insider threats, etc. Because of this integrated platform, SAP HANA is able to process blends of spatial, predictive, and text analysis results within one SQL statement, providing simplified development of intelligent and intuitive location-based solutions that combine geo-spatial data with other intelligence. Event Stream Processing for streaming data such as machinegenerated or sensor data, which allows streaming data to be integrated with historical data to detect anomalies from normal behavior in real time. A click and drag data integration tool that allows new data to be included quickly, and without the need for specialized skillsets. Integrating data from other systems inside or outside the enterprise multiplies the value of data sets by unlocking previously undiscoverable analytical insights. BENEFITS OF SAP HANA A remarkable increase in processing speed: Even on very large data sets, customers report that SAP HANA speeds query and analysis of large data sets by factors ranging from 60x to over 100,000x faster than their traditional computing architectures. SAP HANA allows sub-second searching, filtering, and aggregation of massive databases (trillions of rows of data), even when the system is stressed by thousands of users conducting simultaneous complex queries. Analysis at the speed of thought: The dramatic processing speed of SAP HANA allows analysts to work iteratively, asking questions of the data and receiving responses as fast as they can click their mouse, which enables them to quickly refine their queries until they arrive at the answers they seek. Complex predictive analysis on detail-level data: SAP HANA s in-memory architecture enables a new level of functionality because it actually embraces table scans scanning every row of the data so that complex queries can be performed on all of the data available. In contrast, traditional databases are actually designed to prevent users from conducting a table scan of all of the data. To avoid unacceptably poor performance, these databases use complex, high-maintenance data models and pre-built aggregations to create a summary of the data for users. This means that complex analysis can only be performed on summary-level data, not on all of the data available. Ad hoc queries of large data sets: Traditional data models require building indexes and summary aggregations of data that are predicated on the likely questions analysts will ask. With SAP HANA, however, it is not necessary to pre-conceive an appropriate structure or index strategy for the data, or to anticipate the future analytical uses of the data. Because of the speed of in-memory processing, SAP HANA is aggregate-onquery. That means that users are free to ask any new ad hoc query that is required by the changing needs of the mission or the business. Analysts can more easily pull operationally relevant knowledge from large, SAP National Security Services TM SAP NS2 TM 5

6 diverse data sets. And they are free to explore data as the mission evolves, without having to reach back to scarce resources of data scientists and data modelers. SAP HANA leverages your existing investments: It does not take esoteric skill sets to deliver the power of in-memory processing to your mission. SQL, JDBC/ODBC, JSON, and XML are used to connect the HANA platform to your enterprise. The power and speed of in-memory computing can accelerate your existing desktop tools (ESRI, I2 Analyst Notebook, HTML5 or Ozone Widget Framework widgets), or you can use SAP s industry-leading tools with HANA. SAP HANA s computing power can be used to conduct high-speed in-memory analysis of multiple data sources, including existing databases (Oracle, Sybase, MS SQL server, Teradata, DB2 and others), applications, messaging systems, unstructured text, and the Hadoop Distributed File System. The SAP HANA architecture radically accelerates big data analyses in the enterprise, including data stored in Hadoop. High Availability: SAP HANA supports both synchronous and asynchronous replication to enable Continuity of Operations. It is ACID-compliant, which translates to being fully resilient against hardware failure. Integration with the NIEM model: SAP HANA is available with the National Information Exchange Model (NIEM) hierarchy already instantiated as a data model in the database. The SAP Real-Time Situational Awareness Rapid Deployment Solution can accelerate your time-to-value for an analytical system that leverages the data definitions of NIEM. COST ADVANTAGES OF SAP HANA Reduced data footprint: SAP HANA s in-memory columnar data base provides 4- to 20-times data compression of your raw data, and it eliminates the need for aggregation and index tables, which again sharply reduces total data storage needed (as well as eliminating the time and cost of designing those data structures). Simplified system landscape: Data contained in SAP HANA can be exploited as both a transactional (OLTP) and high-performance analytical (OLAP) database simultaneously. This eliminates the O&M costs of maintaining separate optimized data environments. Reduced ETL (extract, transform, and load) processes between systems: Because a single inmemory data repository can be used both for analytical and transactional applications simultaneously, data movement, synchronization and latency differences are ameliorated. Reduction in specialized skills needed to operate and maintain the overall system Multiply the Value of Existing Investments: SAP HANA s data virtualization capabilities expedite dynamic data queries across the in-memory data storage in SAP HANA with bulk on-disk data storage in Hadoop, Teradata, or Sybase data stores, which allows organizations to optimize the use of their infrastructure resources. In addition, standard interfaces allow SAP HANA to multiply the speed and processing power of existing front-end user tools in the organization. Less Intensive Administration: SAP HANA does not require the constant tuning that database administrators have had to do with traditional database management systems that depend on complex data modeling, aggregation tables, and indexing driven by analytical needs of the mission. Cost-Effective Education and Training: SAP has re-thought education via the SAP HANA Academy, which has 250 tutorials of five-minute videos. This allows enterprises to save money spent on education, classes and conferences. SAP National Security Services TM SAP NS2 TM 6

7 FOR MORE INFORMATION: Contact your account manager or call us at SAPNS2 ( ) WEB: About SAP National Security Services (SAP NS2 ) SAP National Security Services, Inc. TM (SAP NS2 TM ) is an independent U.S. subsidiary of the global enterprise software company SAP, offering a full suite of world-class enterprise applications, analytics, database, cyber security, cloud, and mobility software solutions from SAP, with specialized levels of security and support to meet the unique mission requirements of US national security and critical infrastructure customers. In addition to software solutions, SAP NS2 offers consulting and support services from credentialed experts in the national security space. See how SAP NS2 can help your organization run faster, smarter, leaner in the most secure environments at You may follow us on LinkedIn, and we re also on Twitter at (SAP National Security Services and SAP NS2 are trademarks owned by SAP National Security Services, Inc.) 2013 by SAP National Security Services, Inc. All rights reserved. May not be copied or redistributed without permission. SAP, R/3, xapps, xapp, SAP NetWeaver, Duet, SAP Business ByDesign, ByDesign, PartnerEdge and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. Business Objects and the Business Objects logo, BusinessObjects, Crystal Reports, Crystal Decisions, Web Intelligence, Xcelsius and other Business Objects products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of Business Objects S.A. in the United States and in several other countries. Business Objects is an SAP Company. All other product and service names mentioned and associated logos displayed are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary. The information in this document is proprietary to SAP. This document is a preliminary version and not subject to your license agreement or any other agreement with SAP. This document contains only intended strategies, developments, and functionalities of the SAP product and is not intended to be binding upon SAP to any particular course of business, product strategy, and/or development. Please note that this document is subject to change and may be changed by SAP at any time without notice. SAP assumes no responsibility for errors or omissions in this document. SAP does not warrant the accuracy or completeness of the information, text, graphics, links, or other items contained within this material. This document is provided without a warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability, fitness for a particular purpose, or noninfringement. SAP shall have no liability for damages of any kind including without limitation direct, special, indirect, or consequential damages that may result from the use of these materials. This limitation shall not apply in cases of intent or gross negligence.