Big Data analytics in insurance

Transcription

1 Big Data analytics in insurance

2 Who we are Experts At Your Service > Over 50 specialists in IT infrastructure > Certified, experienced, passionate Based In Switzerland > 100% self-financed Swiss company > Over CHF 8 mio. Turnover Leading In Infrastructure Services > More than 150 customers in CH, D & F > Over 50 SLAs dbi FlexService contracted Big Data analytics in insurance Page 2

3 About us Yann Neuhaus / David Hueber Chief Sales Officer / CEO yann.neuhaus[at]dbi-services.com / david.hueber[at]dbi-services.com Big Data analytics in insurance Page 3

4 Agenda 1.Introduction 2.Relational data model 3.Testing infrastructures 4.Big Data process 5.Conclusion Big Data analytics in insurance Page 4

5 Introduction > Big Data processes > Objectives > High level architecture Big Data analytics in insurance Page 5

6 Introduction Big Data Analytics Business Intelligence process Business requirements Old model Rigid model, Alteration, Long time Implementation Data source Data assessment Data cleansing Analyze Scope Collect Clean Visualization Page 6

7 Introduction Big Data Analytics Big Data process Business requirements New model Fast and Flexible process Data Data source source Data integration Data assessment Data exploration Data cleansing Business validation Analyze Process repetition Identification Semi/unstructured data Analyze, Patterns Implement solutions New sources, patterns Page 7

8 Introduction Objectives Analyzing the risks of an insurance portfolio Big Data Analytics for insurance Business Comparison between relational and Big Data data model > Implementation of relational data model > Implementation of a Big Data process > Choosing the right Big Data tools Page 8

9 Introduction Objectives DATA Data consolidation from multiple sources DATA DATA STATISTICS Analyze the risks of insurance customer BUSINESS Generation of new business use cases BUSINESS STATISTICS Big Data = Get the true about all data Page 9

10 Introduction Disclaimer BE CARFUL WE ARE NOT >Insurance Business Analyst >Data scientists It s a demonstration use case Not intended to production environments Big Data infrastructure project is first of all about choosing the right tool for your use case > Large ecosystem Page 10

11 Introduction High level architecture Customer Data - Exploration - Analyze - Processing - Manipulation Assess the risk of an insurance portfolio Page 11

12 Relational data model > Architecture > Accident fact table > Challenges Big Data analytics in insurance Page 12

13 Relational data model Architecture DB1 an OLTP database DB2 a DataWareHouse VEH Ref Tables VEH (tmp) Ref Tables CSV CSV CSV CUST CONTR Replication CUST (tmp) CONTR (tmp) Processing Data Load STG_ACC MV_CONTR FACT_ACC REPORTING Page 13

14 Relational data model Accident fact table Age range Gender Accident fact table Type vehicle Brand vehicle New dimension? Page 14

15 Relational data model Challenges Relational data model offer 3 main challenges for Big Data Performance > Data loading is slow with relational model (ACID vs CAP) Modeling > Adding a new dimension in the fact table consumes performances and leads to fragmentation Infrastructure > Increasing the data volume needs to scale up the infrastructure Page 15

16 Testing infrastructures 1 > Infrastructures comparison Big Data analytics in insurance Page 16

17 Testing infrastructures Infrastructures comparison Relation data model 2 nodes (1 OLTP 1 DWH) 4 GB Memory per nodes 2 vcpu per nodes 100 GB per nodes Oracle 12.2 Enterprise Edition Cloudera On-Premise cluster 5 nodes 8 GB Memory Cloudera Manager 4 GB Memory data nodes 1 vcpu per nodes 100 GB per nodes Cloudera EDH edition Oracle Big Data Appliance 6 nodes 64 GB Memory 2 * 22 CPU per nodes 96 TB per nodes Cloudera EDH edition Cloudera Cloud cluster 5 nodes EC2 15 GB Memory per nodes 4 vcpu per nodes 500 GB per nodes Cloudera EDH edition Page 17

18 Big Data process > Reference architecture > Data ingestion > Data analysis > Data processing > Data visualization Start Demo (on Cloudera master):./data_analysis.bash 5 Big Data analytics in insurance Page 18

19 Big Data process Reference architecture Data Sources Data Ingestion Data Analysis Data Processing Data Visualization CSV Customer Data CSV Page 19

20 Big Data process Data Ingestion Importing data from multiple sources into HDFS Data Lake Data sources > Customer csv (extracted from relational database) > Open Data csv (1 file per year in our business case) HDFS import methods > WebHDFS > HTTPFS > HDFS copy from local Page 20

21 Big Data process Data Ingestion > OpenData import ~]$ curl -i -X PUT -L " centos" -T /home/centos/opendata2009.csv ~]$ curl -i -X PUT -L " centos" -T /home/centos/opendata2010.csv > Customer data import ~]$ curl -i -X PUT -L " -T /home/centos/customer.csv Page 21

22 Big Data process Data ingestion Time results for data import 12 GB Data import Time (minutes) Page 22

23 Big Data process Data analysis Creation of Hive meta-data > Hive DDL > Creation of insurance database > Creation of tables > Accident data > customer Pre-analysis the data > Data manipulation with SQL queries External tables for data processing > Creation of external tables with transformations > customer1 & customer2 (2 transformation steps) > opendata ( transformed accident data) Page 23

24 Big Data process Data analysis Time results for data analysis Data analysis Time (minutes) Page 24

25 Big Data process Follow up Demo :./data_processing.bash Big Data analytics in insurance Page 25

26 Big Data process Data processing Retrieving external tables from HDFS > customer2 > opendata Usage of Spark for data processing > Spark SQL queries > Creation of Spark dataframes Implementation of python scripts for data counting > Persist results into HDFS in csv format Page 26

27 Big Data process Data processing Time results for data processing Data processing Time (minutes) Page 27

28 Big Data process Data visualization Creation of Hive external tables for OpenData and customer > per age range > per gender > per vehicle type > per vehicle brand ODBC connection for retrieving data from Hive > Extract tables from Hive insurance database ODBC connection with QlikView or Excel for reporting Follow up Demo :./data_reporting.bash Page 28

29 Conclusion > Big Data advantages > Advantages vs Drawbacks Big Data analytics in insurance Page 29

30 Conclusion Big Data advantages Flexible Architecture > Standard Big Data architecture > Lambda architecture Schemaless data model > No integrity constraints > Several way of doing Adding a new dimension in the fact table > Same data model > Same architecture > No performances decrease Page 30

31 Conclusion Advantages vs Drawbacks Flexibility for data modeling Performances New technologies Infrastructures are not easy to deploy New business generation Compatibility with common B.I tools Lack of human resources Page 31

32 Basel Delémont Zürich Any questions? Please do ask! Nyon We would love to boost }5 your IT-Infrastructure How about you? Big Data analytics in insurance Page 32