SEIZE THE DATA SEIZE THE DATA. 2015

Transcription

1 1 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

2 Machine Data Log Text Search Malu Castellanos & Jörn Schimmelpfeng August 11, 2015

3 Forward-looking statements This is a rolling (up to three year) Roadmap and is subject to change without notice. This document contains forward looking statements regarding future operations, product development, product capabilities and availability dates. This information is subject to substantial uncertainties and is subject to change at any time without prior notification. Statements contained in this document concerning these matters only reflect Hewlett Packard's predictions and / or expectations as of the date of this document and actual results and future plans of Hewlett-Packard may differ significantly as a result of, among other things, changes in product strategy resulting from technological, internal corporate, market and other changes. This is not a commitment to deliver any material, code or functionality and should not be relied upon in making purchasing decisions. 3 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

4 HP confidential information This is a rolling (up to three year) Roadmap and is subject to change without notice. This Roadmap contains HP Confidential Information. If you have a valid Confidential Disclosure Agreement with HP, disclosure of the Roadmap is subject to that CDA. If not, it is subject to the following terms: for a period of 3 years after the date of disclosure, you may use the Roadmap solely for the purpose of evaluating purchase decisions from HP and use a reasonable standard of care to prevent disclosures. You will not disclose the contents of the Roadmap to any third party unless it becomes publically known, rightfully received by you from a third party without duty of confidentiality, or disclosed with HP s prior written approval. 4 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

5 What is new? Single-system solution for text search on machine log data High performance Customizable Enables queries that combine log text search data with structured data No need to move data around 5 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

6 Machine data valuable asset of IT management and analytics Application logs Clickstream Data Web Access Logs Database logs File system logs OS logs Network logs Wire data Packet Data Web Proxy Logs... Mobile devices logs Call Detail Records Configurations Diagnostics output Change Events Log analysis is critical to understand, diagnose, and deploy systems in large operational environments Source of the truth of what is happening in the IT infrastructure User activity analysis Digital marketing Fraud detection App performance and availability Data modification history I Infrastructure set-up Failures debug Backdoor attacks Read access audit DNS lookups Threats detection Timeouts Bottlenecks Suspicious activity Monitor terms of service Investigate data leakage incidents Proof of compliance with regulatory and corporate governance mandates 6 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

7 Machine data text How does it look? Alert(id:1f407960afe#4d9fdbdb078a6108edc42 customer id: 1 name: AST-EL-YGA Eagle QA GMAM_Portal: Login-transaction-response-time is greater than 20 seconds) was sent :00: %ASA : Built outbound TCP connection for outside: /443 ( /443) Lightweight, this laptop weighs 2 something pounds and is easy to pick up and hold in one hand. Keyboard, the keys are great and feel a lot like a Mac keyboard they are also easy to type on. It took me probably less than 5 minutes to get accustomed to the size of the keyboard and where the keys are. Size, I like that it is the same size as a Notebook, but it's totally a laptop. 7 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

8 Text index support for log search Text indices are essential to accelerate text/log search Indices are maintained transactionally consistent with their base tables Ability to search the index with regular SQL Ability to join search result with base table (and other tables) to retrieve other data Load data Log messages & other data Log table Search query result join tokenize Search matches Index table Search query: Retrieve the unique_id, message, host and time of the log records containing transaction response time Tokenization is the process of breaking a stream of text up into words, phrases, symbols, or other meaningful elements called tokens. The list of tokens becomes input for further processing such as parsing or text mining 8 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

9 Tokenization for text indexing :00: %ASA : Built outbound TCP connection for outside: /443 ( /443) Tokenization Vertica basic tokenizer can separate strings by space only Indexing Token Doc_id :00: %ASA : 1 Built 1 outbound 1 TCP 1 connection for 1 outside: /443 1 ( /443) 1 Text/log table Index table SQL Query SELECT count(*) FROM public.la_unique_msg WHERE unique_id IN (SELECT doc_id FROM public.whitespace_index WHERE token = outside ) No match found 9 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

10 Advanced tokenization* and indexing (1/5) Configurable tokenizers Vertica basic tokenizer separates tokens by space only Advanced configurable tokenizer: o Which special characters to use as token separators 2 tier separators o Which words to ignore stop words o What are the size limits that a token can have min and max length Supports UTF-8 encoding Supports English but can be extended to other languages * Tokenization is the process of breaking a stream of text up into words, phrases, symbols, or other meaningful elements called tokens. The list of tokens becomes input for further processing such as parsing or text mining. 10 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

11 Advanced tokenization and indexing (2/5) Configurable tokenizers with two-tier separators Text to tokenize: login url ( Basic tokenizer: login url /57625( Issue: special characters glued to words, leads to mismatch With two-tier separator tokenizers: o Tier 1 Major separators - Default: [ ] < > ( ) { }! ;, ' " * \n \r \s \t &? + o Tokens: login url o Tier 2 Minor separators - Default: / : - $ # % \ _ o Tokens: http www hp com Benefit: Improved search accuracy and search performance o E.g., both tokens hp and are indexed and fast searchable 11 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

12 Extended tokenization and indexing (3/5) With tier 1 & 2 (major + minor) Larger indices with two-tier separator tokenizers With only tier 1 (major separators) :00: %ASA : Built outbound TCP connection for outside: /443 ( /443) index has 11 rows Index has 23 rows 12 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

13 Advanced tokenization and indexing (4/5) Multi-column Indices with Position Info Indices with position support phrasal and proximity search: Token and its position (i.e., first token will have position 0, second token position 1, etc.) Example: Doc 1: transaction exceeded required time -> transaction 1, exceeded 2, required 3, time 4 Search string transaction required time exceeded -> transaction 1, required 2, time 3, exceeded 4 No match found: keywords match but the sequence given by their relative positions don t! Benefit improve performance of phrasal search 13 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

14 What s new: Advanced tokenization and indexing (5/5) N-gram Tokenizers N-gram indices are available (in addition to word indices) Indices with ngram tokens can support regex and wildcard search: o Example string: transaction-code o N-gram (tri-gram) tokens: tra, ran, ans, nsa, sac, act, cti, tio, ion, on-, n-c, -co, cod, ode If tri-gram index exists, text search can find matching strings that contain a partial word tra Benefit: accelerate wildcard and regex search. * An n-gram is a contiguous sequence of n items from a given sequence of text or speech. An n-gram of size 1 is referred to as a "unigram"; size 2 is a "bigram" (or, less commonly, a "digram"); size 3 is a "trigram". Larger sizes are sometimes referred to by the value of n, e.g., "four-gram", "five-gram", and so on. 14 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

15 Create a text index using pre-configured tokenizer Given a preconfigured tokenizer instance advancedlogtokenizer, a user creates a text index using the following vsql Statement: CREATE TEXT INDEX myconfigurable_basic_lower ON la_unique_msg (unique_id, message) TOKENIZER (public.advancedlogtokenizer) STEMMER (v_txtindx.caseinsensitivenostemming); 15 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

16 Advanced log search supported by advanced text indices Text Search Expressions response time Keyword Search: Find text that contain both words response and time transaction response time Phrasal search: Find text that contain the word transaction immediately followed by the word response and then word time REGEX (www www8)\.hp\..*?\.com greater than * seconds Keyword with wildcard search: Find text that contain words of greater than, seconds, and any characters in between transaction response time 2* Phrasal with wildcard search : Find text that contain the word response followed by the word time followed by any word that starts with the number 2 help ( error & REGEX www[8].hp.*.com ) Regular expression search: Find text that contain a substring that starts with www or www8, follows by hp with any numbers of characters, ends by.com Boolean search: Find text that either contain the keyword help or contain both the phrase error code and a substring that satisfies the regex pattern 16 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

17 Keyword search basics Single keyword: brian Base table Primary key of base table SELECT count(*) FROM public.la_unique_msg WHERE unique_id IN (SELECT doc_id FROM public.whitespace_index_lower WHERE token = v_txtindex.caseinsensitivenostemming('brian')) text index In general there are multiple ways to formulate the SQL query for a given search expression Built-in strategies to generate the one with best performance are on the roadmap 17 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

18 Different query shapes for search expressions Instead of referencing index m times, reference it 1 time then use a count the number of times the doc-id appears (i.e. use count/group-by to verify) 3 keywords: brian time. AST-EL-YGA SELECT count(*) FROM public.la_unique_msg WHERE (unique_id IN (SELECT doc_id FROM (SELECT doc_id, count(*) AS count FROM public.whitespace_index_lower WHERE token IN (v_txtindex.caseinsensitivenostemming( Brian ),v_txtindex.caseinsensitivenostemming(time.'),v_txtindex.caseinsensitivenostemming( AST-EL-YGA')) GROUP BY doc_id) r WHERE count>=4)) In general there are multiple ways to formulate the SQL query for a given search expression It is important to choose the one with best performance 18 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

19 Demo 19 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

20 Machine Data Log Text Search Use Case HP Operations Analytics - Log Search and Analysis Jörn Schimmelpfeng 20 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

21 You are challenged to stay out of the headlines: But how do you? Ensure the technology can sustain the business Understand what is happening when things begin to go wrong Triage and remediate before the business is affected Add value to your business 1 Minyanville 2 - UsaToday 21 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

22 The answer lies in your data but how do you make sense of it? Data Immense volumes of data (up to 2Gb/server/day) Thousands of servers Many different types (log data, events, topology, etc.) Numerous siloed data sources (apps, storage, servers, networks, mobile, etc.) And the complexity is growing 22 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

23 Introducing HP Operations Analytics Application HP Operations Analytics Standalone, scalable platform Mobile app Storage Network Metrics Events Intelligent search Guided troubleshooting Fewer outages Faster problem resolution System Cloud Third-party tools Non IT data Topology Logs Big Data Store HP Vertica Visual analytics Predictive analytics Automated log analytics Real time alerting Optimized resources Higher staff productivity LOB data 23 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

24 Unique self learning machine analytics HP Operations Analytics Clustering Text relevance Anomaly detection not complete login customer ID severity error exception ID exceeded SME input keywords 24 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

25 Machine analytics drives log and predictive analytics Find the root cause faster and prevent issues from happening Behavior assessment Clustering Text relevance SME input Log analytics Pattern detection Data seasonality Predictive analytics 25 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

26 Troubleshooting with Ops Analytics All relevant data in a single dashboard Near real time data collection Data easily viewed in visual analytics Historical view of data instantly available Faster time to identify root cause with fewer people involved Application ecosystem Physical or virtual server Network Business application Metrics Events Operations Analytics Logs Topology Operation support Storage 26 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

27 Analyze database metrics to isolate problems Correlate related metrics in a common sleeve of time Correlate related metrics in a common sleeve of time 27 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

28 Log analytics used to reveal the root cause Automatically reveals time and count of most significant log events. 28 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

29 Drill down to the actual logs View the log message content to identify root cause Automatically reveals time and count of most significant log events. Number of occurrences of significant log data over time 29 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

30 Summary HP IT order management system outage Before HP Ops Analytics After HP Ops Analytics Time to reveal the root cause 36 hours 30 minutes Resources engaged 5 experts from Application, Database, UNIX, Network, and Storage 1 expert Revenue impact (to clear order backlog) 2 weeks 4.5 hours 99 % 30 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

31 Vertica combines the world of metrics and logs! 31 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

32 From noise to value Start to make meaning out of your logs Who are your users? Where do they connect from? What are the common errors? No problem with HP Operations Analytics and Vertica! 32 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

33 Fast search and field extraction New User First query language Very powerful but easy to learn Language translated into Vertical SQL (vsql) This guarantees the power and speed of Vertica for analytical oprations 33 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

34 Next generation log search Simple Search AccountService Status lasthour Advanced Search transactionid: (.+) failed hostname=myhost AND (severity=major OR severity=critical) Charting error lasthour movingcount Log Processing error countby hostname where count> 20 Field Extraction failed to login rex user:%{username} failed to login top USERNAME Command List: top, bottom, movingcount, movingaverage, rex, sort, head, tail, between, dedup, lasthour, count, countby, distinctcount, genericsql 34 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

35 HP Operations Analytics architecture Monitored nodes Databases Mail server Corporate network Vertica nodes Opsa Server Log ingestion Operator / data scientist Network Erp Log analytics Vsql Gen. Log Search Web shop Vertica Text Index 35 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

36 36 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

37 37 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

38 38 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

39 39 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

40 40 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

41 41 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

42 42 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

43 43 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

44 HP Vertica First choice DB for Operations Analytics HP Vertica 44 Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

45