Data processing. Intelligent Data Analysis Fault Tolerant Systems Research Group

Transcription

1 Data processing Intelligent Data Analysis Budapest University of Technology and Economics Fault Tolerant Systems Research Group Budapest University of Technology and Economics Department of Measurement and Information Systems 1

2 Outline Data format/representation Data processing ETL, workflow support Outlook: OLAP Case studies 2

3 Data science process 3

4 DATA FORMAT 4

5 Tidy data 3 Simple rules to facilitate statistics and visualization One variable one column One observation one row Each type of observational unit one table seems to be trivial not true in most practical cases and even for staitstical tools (e.g. output of R packages) Wickham, H. (2014). Tidy data. Journal of Statistical Software, 59(10),

6 Data originally: long/wide 6

7 How to use these formats? 7 Sparse Screening for Exact Data Reduction. Jieping Ye, Arizona State University

8 Examples for tidy data 8 R dataframe representation:

9 tidying 9 R: spread(data,key,value)

10 tidying 10 R: spread(data,key,value) Generalization?

11 Data restructuring examples ( in R) 11

12 DATA STORAGE 12

13 .CSV Common data storage techniques o Majority of inputs o Length? Header? Encoding? DB with a schema (in memory?) Graph databases, ontologies, RDF Key-value stores (redis) Time series databases (opentsdb, influxdb) o Time series + metadata Data in motion o Streams as input for processing/analysis 13

14 Time series example: influxdb Data: measurement o Fields, tags, timestamp 14

15 Dashboards (e..g Grafana) 15

16 16 DATA PROCESING WORKFLOW & TOOLS

17 ETL Extract-Transform-Load Originally: to fill a snowflake/star schema In data science: create dataframes Cleaning tasks o Standardization o Normalization o Deduplication o Enrichment o Clear/fill NAs 17

18 Example data processing workflow (KNIME) 18 Steps: reading, filtering/aggregation, transformation, plotting, Status of the concrete execution KNIME

19 Measurement processing: RapidMiner 19 Read CSV Format conversion Identifying source node Filter to cpu.usage.average Calculating averages (interval) Add machine information Delete unnecessary attribute

20 20 CASE STUDY Processing of telco data

21 SOME BACKGROUND OLAP 21

22 22 On-Line Analytical Processing (img: Business intelligence approach Extensively used since early 2000s o Still! (although not that popular as it was at least in academic research) Features o Multi-dimensional analysis o Fast query execution o Exploratory analysis of data Support ad-hoc queries o Report generation o (Visualization) snowplowanalytics.com)

23 On-Line Analytical Processing (img: snowplowanalytics.com) Central concept: OLAP cube o Multi-dimensional array: set of separate data Dimensionality >3 technically a hypercube ~ a multi-dimensional spreadsheet o Slicer: dimension held constant For a given query (e.g. sales in a particular year) 23

24 24 OLAP process (img: Pranav Joshi)

25 Operations o Slicing & dicing o Drill up & down o Pivoting OLAP operations Easy to visualize by the cube itself 25

26 26 Slicing (img: Wikipedia)

27 27 Dicing (img: Wikipedia)

28 28 Drill up & down (img: Wikipedia)

29 29 Pivoting (img: Wikipedia)

30 OLAP vs. regular/modern data analysis OLAP cube: like a set of spreadsheets o multi-dimensional o interlinked Modern data analysis: flat data frames o Modern machine learning algorithms: require (?) single dataframes Operations: basically the same (slicing, dicing, drill up & down, pivoting) 30

31 CASE STUDY3 Deep insights from observations with the help of modern data analysis tools CECRIS IAPP project Railway accidents: casualties by type of accident, Department for Transport Statistics, Rail Statistics, Table TSGB0805 (RAI0501) ( Analysis: next class, now let us process the data 31

32 PowerBI data import 32

33 Load data to PowerQuery 33

34 Remove unnecessary top rows 34

35 Remove unnecessary bottom rows 35

36 Remove blank rows 36

37 Remove columns 37

38 Promote first row to header 38

39 Filter total and all rows 39

40 Split first column 40

41 Replace empty values to null in first column 41

42 Replace empty values to null in second column 42

43 Remove colon character from first column 43

44 Automation: RapidMiner process 44

45 Read Excel 45 Read measurements

46 Filter rows 46

47 Split 47

48 Rename attributes 48

49 Loop attributes 49

50 Replace spaces 50

51 Replace colon character 51

52 Header row problem 52 To be removed (derived) To be kept