Data Engineering for Data Science

Transcription

1 Engineering for Science Arup Nanda VP, Services Priceline

2 booking.com priceline.com kayak.com agoda.com rentalcars.com opentable.com 2

3 Science and Machine Learning Customer Segmentation Prediction of Behavior 3

4 Science and Machine Learning Machine Behavior Predictive Setting of System Properties 4

5 Bots Demand Technology Supply 5

6 WWW Providers Affiliates GDS White labels Affiliates Inventory 250K Hotels X 200 rooms X 365 days X. Demand Technology Supply 6

7 WWW Providers Affiliates GDS White labels Cache Affiliates Demand Technology Supply 7

8 8

9 9

10 Cache Demand Technology Supply Smart Cache Management 10

11 Bots MySQL MySQL MySQL MySQL Shards 11

12 Inventory and Rates stored as blob Bots MySQL MySQL MySQL MySQL Shards 12

13 Moved to Hadoop Bots ETL MySQL MySQL MySQL MySQL Shards 13

14 Bots Pipe based on Kafka named Mississipi has APIs and common libraries MySQL MySQL MySQL MySQL Shards 14

15 Bots Pipe MySQL MySQL MySQL MySQL Shards 15

16 Bots Pipe Oracle MySQL MySQL MySQL MySQL Shards 16

17 APIs Transformation 17

18 APIs Machine Learning Step 1 Step 2 Step 3 18

19 Deep Learning Applications Scientists Analysts Simple ML, Personalization, Recommendation, Ranking Training Preparation Query Testing, Experimentation Engineering Platform Analytics, Dashboards, Metrics, KPIs Summarization, Aggregation, Classification Cleaning, Anomaly Detection, Preparation Signals, Logs, External, User Content System of Record (Customers, Bookings) Transient (Cache, Session State) 19

20 Deep Learning Applications Scientists Analysts Signals, Logs, External, User Content Simple ML, Personalization, Recommendation, Ranking Training Preparation Query Testing, Experimentation Analytics, Dashboards, Metrics, KPIs Summarization, Aggregation, Classification Cleaning, Anomaly Detection, Preparation System of Record (Customers, Bookings) Engineering Platform Transient (Cache, Session State) Engineering Wishlist 1.Repeatable Code 2.Easy to transform 3.Quick to market 4.Scheduling built in 5.Customer can monitor 6.Analyze the past metrics FREE 20

21 Apache Airflow 21

22 Directed Acyclic Graphs (DAG) Activity 1 Activity 2 Activity 3 Activity 4 Directed = In one direction only. Can t go back Acyclic = Must finish. Must be Idempotic 22

23 Directed Acyclic Graphs (DAG) 23

24 24

25 25

26 26

27 27

28 Website Python Code Source, Destination, Transformation Shell Script DAG ZipLine 28

29 Deep Learning Applications Scientists Analysts Simple ML, Personalization, Recommendation, Ranking Training Preparation Query Testing, Experimentation Engineering Platform ZIPLINE Analytics, Dashboards, Metrics, KPIs Summarization, Aggregation, Classification Cleaning, Anomaly Detection, Preparation Signals, Logs, External, User Content System of Record (Customers, Bookings) Transient (Cache, Session State) 29

30 Activity Before After Engineering 1 Quarter 1 week After ZIPLINE Machine Learning Experiments 1/month 1/day Visibility by Customers Granular Debugging No No Yes Yes APIs Ste p1 Machine Learning Ste p2 Ste p3 30

31 After Machine Learning for Cache Cache hit ratio up from 61% to 96% Parity up from 82% to 98% No need to add more shards to MySQL cluster Bots Docker Containers MySQL MySQL MySQL MySQL 31

32 Summary Machine Learning is useful for smart machine usage Use data for event based data collection and manipulation Think pipe ; not ETL Engineering is the foundation for Science Airflow can be used to automate and quicken data engineering The python code can be generated easily by shell scripts Quicker to market means Scientists can experiment more frequently 32

33 Thank You! Blog: arup.blogspot.com Facebook.com/ArupKNanda 33