TESTING BIG DATA WORLD RIGA. by Konstantin Pletenev OCTOBER, 2017, TAPOST GROW CONFIDENTLY

Transcription

1 RIGA TESTING BIG DATA WORLD by Konstantin Pletenev OCTOBER, 2017, TAPOST GROW CONFIDENTLY

2 BIG DATA IS NOT ABOUT THE DATA THE REVOLUTION IS NOT THAT THERE S MORE DATA AVAILABLE THE REVOLUTION IS THAT WE KNOW WHAT TO DO WITH THIS DATA NOW GARY KING PROFESSOR OF HARVARD UNIVERSITY

3 65 MILLION SOCIAL MEDIA POSTS PER DAY 1 HOUR, 16 MINUTES EACH DAY WATCHING VIDEO ON DIGITAL DEVICES 100 MILLIONS NETFLIX STREAMING SUBSCRIBERS

4 WHAT IS BIG DATA

5 WHAT BIG DATA IS NOT TRADITIONAL DATA LIKE DOCUMENTS AND DATABASES JUST A SYNONYM FOR LOTS OF DATA IT S NOT ABOUT THE DATA

6 WHAT BIG DATA IS BIG DATA IS THE LARGE VOLUME OF DATA ON A DAY-TO- DAY BASIS WHAT DO WITH THE DATA MATTERS BIG DATA FOR MADE A BETTER DECISIONS AND STRATEGIC BUSINESS MOVES

7 3 BIG DATA PROJECTS IN ACCENTURE ON LAST 2 YEARS 1. USER DATA PROCESSING AND SERIALIZATION 2. VIDEO ON DEMAND DATA PROCESSING AND BUSINESS INTELLIGENCE DASHBOARDS 3. VIDEO ON DEMAND DATA PROCESSING FOR DATA SCIENTISTS

8 BIG DATA V 3 VOLUME VELOCITY BIG DATA VARIETY

9 BIG DATA V 3 VOLUME VARIETY VELOCITY

10 V 1 - VOLUME FACEBOOK STATISTIC 100 PB DISK SPACE ON SINGLE HDFS CLUSTER 150 TB DATA SCANNING IN HIVE EACH 30 MINUTES 500 TB NEW DATA INGESTED PER DAY

11 V 2 - VARIETY FACEBOOK STATISTIC 300 MB IMAGES UPLOADED EACH MINUTE 4.7 BILLION STATUS UPDATES EACH DAY 1 BILLION VIDEO WATCH'S PER DAY

12 V 3 - VELOCITY NETFLIX STATISTIC 37% OF ALL DOWNSTREAM INTERNET BANDWIDTH IN NORTH AMERICA REAL TIME DATA ANALYSIS PROCESS 100,000 SERVER INSTANCES

13 HOW BIG DATA IS PROCESSED Web site Validation Mobile app 3 rd app Data storage 3 rd app 3 rd app FTP S3 Hadoop HDD Processing MapR Hive Pig Data Scientist/Analytics Data Base Oracle RedShift MongoDb BI tools QlikView QlikSance Splunk

14 HOW BIG DATA IS PROCESSED Web site Validation Mobile app 3 rd app Data storage 3 rd app 3 rd app FTP S3 Hadoop HDD Processing MapR Hive Pig Data Scientist/Analytics Data Base Oracle RedShift MongoDb BI tools QlikView QlikSance Splunk

15 MAP REDUCE EXAMPLE MAPPING SHUFFLING REDUCING

16 BIG DATA TESTING VS TRADITIONAL TESTING

17 Properties Traditional testing Big data testing Data Testers works with structured data Testing approachs defined and time-tested Tester has the option of "Sampling" strategy doing manually Testers works with both structured and unstructured data Testing approach requires focused R&D efforts "Sampling" strategy in Big data is a challenge

18 Properties Traditional testing Big data testing Infrastructure Validation Tools It does not require special test environment as the file size is limited Tester uses either the Excel based macros or UI based automation tools Testing Tools can be used with basic operating knowledge and less training. It requires special test environment due to large data size and files (HDFS) No defined tools, the range is vast from programming tools like MapReduce to HIVEQL It requires a specific set of skills and training to operate tools. Copyright 2014 Accenture All rights reserved. 18

19 TESTING CHALLENGES 1. UNDERSTAND YOUR DATA 2. UNDERSTAND YOUR PROCESS (BATCH/REAL) 3. AUTOMATE AS MUCH AS POSSIBLE 4. READY TO A LOT OF PERFORMANCE TESTING 19

20 UNDERSTAND YOUR DATA

21 BATCH / REAL TIME BATCH SYSTEM 1. DATA GENERATION 2. DATA STAGING VALIDATION 3. DATA PROCESSING VALIDATION 4. OUTPUT VALIDATION REAL TIME SYSTEM 1. ACTIVITY SIMULATION 2. VELOCITY ANALYSIS 3. METRIC ANALYSIS 4. OUTPUT VALIDATION 21

22 DATA STAGING VALIDATION Web site Validation Mobile app 3 rd app Data storage 3 rd app 3 rd app FTP S3 Hadoop HDD Processing MapR Hive Pig Data Scientist/Analytics Data Base Oracle RedShift MongoDb BI tools QlikView QlikSance Splunk

23 DATA PROCESSING VALIDATION Web site Validation Mobile app 3 rd app Data storage 3 rd app 3 rd app FTP S3 Hadoop HDD Processing MapR Hive Pig Data Scientist/Analytics Data Base Oracle RedShift MongoDb BI tools QlikView QlikSance Splunk

24 OUTPUT VALIDATION Web site Validation Mobile app 3 rd app Data storage 3 rd app 3 rd app FTP S3 Hadoop HDD Processing MapR Hive Pig Data Scientist/Analytics Data Base Oracle RedShift MongoDb BI tools QlikView QlikSance Splunk

25 TEST AUTOMATION TOOLS DATA DRIVEN TESTING TALEND OPEN STUDIO DATA GENERATORS MAP REDUCE ACCEPTANCE TESTING TALEND OPEN STUDIO QUERY SURGE STUDIO PERFORMANCE TESTING GATLING TOOL DATA PRODUCERS OUTPUT TESTING BDD ACCEPTANCE SCENARIO PREPARED STATIC DATA

26 TALEND OPEN STUDIO

27 TALEND OPEN STUDIO MAP REDUCE

28 QUESTION TIME

29 THANK YOU