AnyMiner 3.0, Real-time Big Data Analysis Solution for Everything Data Analysis. Mar 25, TmaxSoft Co., Ltd. All Rights Reserved.

Transcription

1 AnyMiner 3.0, Real-time Big Analysis Solution for Everything Analysis Mar 25, TmaxSoft Co., Ltd. All Rights Reserved.

2 Ⅰ Ⅱ Ⅲ Platform for Net IT AnyMiner, Real-time Big Analysis Solution AnyMiner Utilization

3 Interest and Questions about Big Although interest in big data is ever growing and corporations have employed big data for various business purposes, lingering questions remain. Growing Interest in Big Lingering Questions 1, Number of searches for 네이버 big data 뉴스 from 빅데이터 Naver news 검색수 Can data of this size and type be analyzed? Q 2Q 3Q 4Q Q 2Q 3Q 4Q Big data: IT term common noun Big Market in Korea (Unit: Million USD) Big data has become an actual area of business. Isn t it too difficult to analyze big data? Is data analyzed only after an problem occurs? Isn t there a way to predict an error and automatically handle it? * Source: Korea Institute of Science and Technology Information (KISTI) Recomposed in a Dailian news article 2/21

4 Issues and Direction for Resolution Intelligent solutions have emerged that overcome the limitations of existing solutions and prevent problems in advance by providing real-time and integrated analysis. Issues of Existing Analysis Passive Response Focused on Response AFTER an Error has Occurred Direction for Resolution Early Prevention Historical Analysis Focused on Causal Analysis after Error Occurrence No Ability to Respond Focused on Analysis of Individual Issues Integrated Analysis Distributed and Logs in Various Systems Analysis Focused on Units Insufficient Integrated Analysis Real-time Analysis Real-time Analysis Limitations Inability to Process of all Types and Size Big Processing 3/21

5 Solution for a New IT Environment There is a growing need for a new concept big data solution that meets the requirements of new IT environments. Requirements for New IT Environments Big Solution for New IT Implementation Early Prevention Rule Engine for pattern definitions Business Administrator IT Administrator Prediction algorithms for business insights Action Plan execution Analysis Engine High-performance engine for real-time analysis Real-time Analysis Integrated Analysis Early Prevention Automated data analysis algorithm Integrated data analysis Collection Storage Analysis Search Response Big Processing Infrastructure 실행력 Collection of various types of data for analysis 사전조치 Massive data collection and processing Storage for massive data integration Logs System Information Events Contents 4/21

6 AnyMiner, Real-time Big Analysis Solution AnyMiner is a solution that can be utilized in various areas (Any Task) because it collects data from various sources (Any Source), analyzes massive data (Any ), and prevents errors in advance (Any Way). Any Way Bi-directional Communication with Real-time Big Analysis Solution Any Task Utilized in Various Areas Any Source Collects Various Types of Any Massive Analysis 5/21

7 AnyMiner Architecture AnyMiner is composed of the Agent,, and Search Head layers. It includes LogMiner, which collects logs, and SysMiner, which collects and responds to system information. HTTP Rest AnyMiner Area Early Prevention Real-time Analysis Integrated Analysis Prediction Search Head Search Head Layer User interface Configuration management for systems in all sections Real-time Server LogMiner System Log Transaction Log Network Batch Agent Application Sensor Terminal SysMiner Terminal Information S/W Installation Information Configuration Information Agent Response Any Machine Layer LogMiner Real-time/batch collection of logs SysMiner Status information collection Real-time Analysis Agent Layer Load balancing between Agent and layers Minimized volume of pre-processing logs 6/21

8 AnyMiner Key Features While general methods require a complex set of steps to find value from data, AnyMiner processes all steps in a one-stop procedure to help achieve early prevention, which is the fundamental issue. [AnyMiner s Analysis Method] Integrated Analysis Parsing Sending Saving Response Analysis Schematizing Value Real-time Analysis Various Collection Early Prevention VS [General Analysis Method] Parsing Sending Saving Schematizing Analysis Response Value 7/21

9 Various Collection AnyMiner Agent can easily collect various types of data from various sources. syslog Agent s Collection Log Parsing HW Log TBR Network ERRPT get Error Java.lang.Exception Router AnyMiner Remote Collection (Agentless) Local Collection Known Log Various Pre-defined Parsers New Log Convenient Parsing Framework Provided Scripts Scripted Input File/directory Windows registry Unix log and metric Network device log Apache log WebSphere log base log W3C log Creating a GROK rule is easier than writing regular expressions Windows Input Collection from Various Sources Convenient Log Parsing 8/21

10 Real-time Analysis AnyMiner provides real-time analysis by processing complex events in a few seconds and by taking appropriate action through real-time event processing technology. Time Real-time Real-time Analysis AnyMiner 1 Window Slides Real-time Analysis of Streaming Rule Application - Time - Count - Order - Direction Search Pattern Registration Complex Event Processing Real-time Event Processing Handles an event within seconds Early Prevention through Real-time Analysis Action 9/21

11 Integrated Analysis AnyMiner can detect abnormal symptoms before an error occurs through integrated analysis, not independent system analysis. Individual Base Line for a Normal Situation Normal Exceed Unstructured Abnormal Situation Base Line for Normal Situation DB User Program Network Server DB Unstructured Dynamic Log - System Log - Transaction Log DB Static Asset - IP Information - S/W Installation Information Integrated Analysis Server Network Early Prevention via Integrated Analysis LogMiner SysMiner Program Real-time Stream Analysis Topology Analysis User Integrated Analysis Result 10/21

12 Prediction AnyMiner can predict future events by analyzing the patterns of previous events in real time. Pattern Analyzer up-down-up-up A B down-down-up-down Real-time Analysis LogMiner! A Pattern Stream Collection A Pattern B Pattern B Pattern A Pattern B Pattern Recognition SysMiner Action Logging Alert Control Time Current Point in Time Early Prevention through Prediction 11/21

13 Early Prevention AnyMiner facilitates the prevention of events by collecting various data and providing real-time and integrated analysis. Search Head Integrated Analysis Application Example Excel file includes employee numbers Severity WARNING Event Detection Copy to a shared folder Decision and Response Warning-level logging Real-time alert! A B Real-time Stream Analysis Topology Analysis Excel file includes client phone numbers CRITICAL Extension modification Critical-level logging Real-time alert Internet block Real-time Analysis Early Prevention Logging LogMiner SysMiner Log File Index System Log System Information Various Collection Notification Agent Agent Agent SMS Control Scripts Script Execution Block 12/21

14 AnyMiner Features AnyMiner provides various functions for data collection, storage, and analysis. High Performance and High Availability Various Analysis Functions Index Sharding and Parallel Query Mirroring and failover Efficient Management Massive Processing Various user interfaces Korean language morpheme analysis Indexing data materialization Powerful and user-intuitive analysis language Big data lifecycle management Robust security architecture Parallel and distributed processing Rapid and optimal search 13/21

15 High Performance and High Availability AnyMiner uses multi node distributed processing and failover to guarantee high performance and high availability. Index Sharding and Parallel Query Mirroring and Failover Index Sharding Transaction failover when an error occurs Error Parallel processing Mirroring Transaction recovery through mirrored data High performance through distributed processing for all nodes High availability for multiple nodes 14/21

16 Efficient Management AnyMiner guarantees efficient lifecycle management for massive data and robust data privacy. Hot Massive Lifecycle Management Write Enabled Warm Search Enabled Cold Write Disabled Saves an indexing result file to a directory (bucket). Saves storage space and related costs through automated management Frozen Search Disabled Establishes a data lifecycle management policy according to the storage period. Modifies the rolling policy by configuring indexes. User Control Robust Security Architecture User Access, Login, Authentication, and Information Built-in Security AnyMiner Masking Access Control Mirroring Encrypted Communication Channel Monitoring Permission Network Security SSL/TLS Targets for Collection Big data platform that guarantees data privacy All users have access after authentication. Use of data capsules according to permissions. Manages and traces data access, modifications, and usage. Provides reliable communication through SSL (Secure Socket Layer) and TLS. 15/21

17 Massive Processing AnyMiner guarantees high performance processing of massive data through parallel and distributed processing and rapid and optimal search. Parallel and Distributed Processing Structure Rapid and Optimal Search Search Head LogMiner Clustering Index Index Index Indexing Parsing Indexing Parsing Indexing Parsing FILE FILE FILE Collection Collection Collection Key: Value type Index Agent Agent Agent Saves inverted Indexes. Uses archived index data for historical data queries. Multi-node scalability High-performance processing of real-time and massive batch tasks 16/21

18 Various Analysis Functions (1/2) AnyMiner provides web-based analysis screens to allow users to analyze data. AnyMiner also supports Korean indexing and search through a Korean morpheme analyzer. Various User Interfaces Korean Morpheme Analysis Korean Analyzer Structure Source Reader Korean Tokenizer Korean Filter Korean Indexing Korean Search Result 5 Stop Filter 4 Lowercase Filter Report Various massive log analysis reports Dashboard Time-series data search and analysis dashboards Customized Screen Customized reports and dashboards 1. Reads collected data. 2. Breaks down a sentence when whitespace or punctuation is encountered. 3. Classifies the separated sentence into Korean type (morpheme and compound noun analysis) and other types. 4. Changes all English text to lowercase. 5. Eliminates stop words. Convenient web-based data analysis Korean language analysis to boost utilization in the Korean market 17/21

19 Various Analysis Functions (2/2) AnyMiner enables intuitive analysis of massive and unstructured data by providing rapid analysis of archived and real-time data and a powerful user-intuitive analysis language. Indexing Materialization Powerful and User-intuitive Analysis Language Time T1 T2 T3 T4 Search index= A region= asia Indexing Indexing Indexing Analysis join ip [ search index= B age < 20 ] Query Result T1 Query Result T2 Query Result T3 Stream A asia top limit=10 join top Materialized T1+T2+T3 Stream B age Materializes log indexing results in advance. Integrated analysis of structured and unstructured data. High performance for archived data analysis Materialized + Indexed Log = T1+T2+T3+T4 Index Search Language Provides an easy-touse language. Executes join for multiple streams and indexes. Powerful Pipeline Provides a pipeline optimized for processing stream data. Easy Analysis 18/21

20 AnyMiner Utilization AnyMiner is a big data solution that can offer value in any area. Business Area U-City: Analyzes and utilizes sensor data for traffic control and crime prevention. Fraud Prevention: Detects and blocks abnormal electronic financial transactions. Disaster Management: Predicts disasters and takes action in advance. Applied in Various Businesses as a Plug-In type Security Area Private Leakage Prevention: Blocks terminals that attempt abnormal data access. Real-time Security Control: Checks vulnerable points and takes preventative action. IT Management Area Error Prevention: Prevents errors by analyzing operating logs. IT Asset Management: Integrates and analyzes system logs. 19/21

21 Tmax s Suggestion for a New IT Environment AnyMiner, the real-time big data analysis solution for everything data analysis, will lead the way to the new future. Traditional Analysis Existing Solutions Tmax s New IT Environment Store. Listen to your. Communicate with Any. SQL (Structured Query Language) Search Language Early Prevention Batch Real-time Indexing Integrated Analysis Structured Stores structured data using a RDBMS, processing it through a batch ETL method, and executes a SQL query to use it. Unstructured Real-time indexing and search for structured and unstructured data. Any Collects all data for analysis and early prevention. 20/21

22 Conclusion AnyMiner, Tmax s big data analysis solution, holds a bright future and is expected to be quickly adopted by a variety of businesses. Cost Reductions through Innovative Productivity Improvements Greater Profits Thanks to Better Informed Decisions Greater Adoption by Various Businesses Means a More Efficient Business Support Real-time Big Analysis Solution for Everything Analysis Real-time Analysis Early Prevention Integrated Analysis Existing Analysis Requirements 21/21

23 Total enterprise solution provider, TmaxSoft Thank you! 22/21