Data Mining. ❷Chapter 2 Basic Statistics. Asso.Prof.Dr. Xiao-dong Zhu. Business School, University of Shanghai for Science & Technology

Transcription

1 ❷Chapter 2 Basic Statistics Business School, University of Shanghai for Science & Technology nd Semester, Spring2017

2 Contents of chapter 1 1 recording data using computers

3 some famous database management systems some famous data analysis software Some famous programming language for data analysis 1 recording data using computers

4 DBMS recording data using computers some famous database management systems some famous data analysis software Some famous programming language for data analysis

5 DBMS recording data using computers some famous database management systems some famous data analysis software Some famous programming language for data analysis 1 Oracle company. Oracle 12c (recent version)

6 DBMS recording data using computers some famous database management systems some famous data analysis software Some famous programming language for data analysis 1 Oracle company. Oracle 12c (recent version) 2 Microsoft company. Microsoft SQL server 2014.

7 DBMS recording data using computers some famous database management systems some famous data analysis software Some famous programming language for data analysis 1 Oracle company. Oracle 12c (recent version) 2 Microsoft company. Microsoft SQL server IBM company. IBM DB (recent version)

8 DBMS recording data using computers some famous database management systems some famous data analysis software Some famous programming language for data analysis 1 Oracle company. Oracle 12c (recent version) 2 Microsoft company. Microsoft SQL server IBM company. IBM DB (recent version) 4 MySQL (developed by MySQL AB company, Sweden, now belong to Oracle, 5.5 recent version)

9 DBMS recording data using computers some famous database management systems some famous data analysis software Some famous programming language for data analysis 1 Oracle company. Oracle 12c (recent version) 2 Microsoft company. Microsoft SQL server IBM company. IBM DB (recent version) 4 MySQL (developed by MySQL AB company, Sweden, now belong to Oracle, 5.5 recent version) 5 Excel, which is a good and simple office software for recording data)

10 DBMS recording data using computers some famous database management systems some famous data analysis software Some famous programming language for data analysis 1 Oracle company. Oracle 12c (recent version) 2 Microsoft company. Microsoft SQL server IBM company. IBM DB (recent version) 4 MySQL (developed by MySQL AB company, Sweden, now belong to Oracle, 5.5 recent version) 5 Excel, which is a good and simple office software for recording data) 6 others.

11 some famous data analysis software some famous database management systems some famous data analysis software Some famous programming language for data analysis 1 SPSS 2 Matlab 3 WEKA 4 Tableau 5 others...(excel is a good and simple office software for analyzing data)

12 some famous data analysis software some famous database management systems some famous data analysis software Some famous programming language for data analysis 1 SPSS 2 Matlab 3 WEKA 4 Tableau 5 others...(excel is a good and simple office software for analyzing data) we will learning above four tools in later classes and experiments

13 Some famous programming language some famous database management systems some famous data analysis software Some famous programming language for data analysis

14 Some famous programming language some famous database management systems some famous data analysis software Some famous programming language for data analysis 1 Pythorn

15 Some famous programming language some famous database management systems some famous data analysis software Some famous programming language for data analysis 1 Pythorn 2 R

16 Some famous programming language some famous database management systems some famous data analysis software Some famous programming language for data analysis 1 Pythorn 2 R 3 Matlab scripts

17 Some famous programming language some famous database management systems some famous data analysis software Some famous programming language for data analysis 1 Pythorn 2 R 3 Matlab scripts 4 Java

18 Some famous programming language some famous database management systems some famous data analysis software Some famous programming language for data analysis 1 Pythorn 2 R 3 Matlab scripts 4 Java 5 C#

19 Some famous programming language some famous database management systems some famous data analysis software Some famous programming language for data analysis 1 Pythorn 2 R 3 Matlab scripts 4 Java 5 C# 6 others.

20 Starting from buying a bicycle

21 Starting from buying a bicycle What would you consider in buying a second hand bike?

22 Starting from buying a bicycle What would you consider in buying a second hand bike? 1 Brand (Trek, Raleigh)

23 Starting from buying a bicycle What would you consider in buying a second hand bike? 1 Brand (Trek, Raleigh) 2 Type (road, mountain, racer)

24 Starting from buying a bicycle What would you consider in buying a second hand bike? 1 Brand (Trek, Raleigh) 2 Type (road, mountain, racer) 3 Components (Shimano, no name)

25 Starting from buying a bicycle What would you consider in buying a second hand bike? 1 Brand (Trek, Raleigh) 2 Type (road, mountain, racer) 3 Components (Shimano, no name) 4 Age

26 Starting from buying a bicycle What would you consider in buying a second hand bike? 1 Brand (Trek, Raleigh) 2 Type (road, mountain, racer) 3 Components (Shimano, no name) 4 Age 5 Condition (Excellent, good, poor)

27 Starting from buying a bicycle What would you consider in buying a second hand bike? 1 Brand (Trek, Raleigh) 2 Type (road, mountain, racer) 3 Components (Shimano, no name) 4 Age 5 Condition (Excellent, good, poor) 6 Price

28 Starting from buying a bicycle What would you consider in buying a second hand bike? 1 Brand (Trek, Raleigh) 2 Type (road, mountain, racer) 3 Components (Shimano, no name) 4 Age 5 Condition (Excellent, good, poor) 6 Price 7 Frame size

29 Starting from buying a bicycle What would you consider in buying a second hand bike? 1 Brand (Trek, Raleigh) 2 Type (road, mountain, racer) 3 Components (Shimano, no name) 4 Age 5 Condition (Excellent, good, poor) 6 Price 7 Frame size 8 Number of gears

30 Table: data field(data variable) Brand (Trek, Raleigh) Type (road, mountain, racer) Components (Shimano, no name) Age Condition (Excellent, good, poor) Price Frame size Number of gears

31 variables recording data using computers Samples are made up of individuals, all individuals have characteristics. Members of a sample will differ on certain characteristics. Hence, we call this variation amongst individuals variable characteristics or variables for short.

32 types of scales Table: data field(data variable) Brand (Trek, Raleigh) Type (road, mountain, racer) Components (Shimano, no name) Age Condition (Excellent, good, poor) Price Frame size Number of gears Trek road Shimano 22 Excellent 500$ 45cm 21

33 types of scales Table: data field(data variable) Brand (Trek, Raleigh) Type (road, mountain, racer) Components (Shimano, no name) Age Condition (Excellent, good, poor) Price Frame size Number of gears Trek road Shimano 22 Excellent 500$ 45cm 21

34 types of scales Nominal objects or people are categorized according to some criterion (gender, job category) Ordinal categories which are ranked according to characteristics (income- low, moderate, high) Interval contain equal distance between units of measure- but no zero (calendar years, temperature) Ratio has an absolute zero and consistent intervals (distance, weight)

35 Scales in variable view of SPSS Figure: Variable View in SPSS tool

36 1 recording data using computers

37 Population VS Sample Figure: census and sample

38 Population VS Sample Population A population refers to all the cases to which a researcher wants his estimates to apply to Examples: White mice, light bulb life, students

39 Population VS Sample Population A population refers to all the cases to which a researcher wants his estimates to apply to Examples: White mice, light bulb life, students Sample A sample is used because it is normally impossible to study all the members of a population Descriptive stats simply summarize a sample Inferential stats generalize from a sample to the wider population

40 Population & Sample

41 variance standard deviation kurtosis skewness example of 1 recording data using computers

42 variance standard deviation kurtosis skewness example of VS Example I cycle about 50 km per week on average. We can expect a lot of rain this time of year.inferential statistics

43 variance standard deviation kurtosis skewness example of VS Example I cycle about 50 km per week on average. statistics description We can expect a lot of rain this time of year.inferential statistics

44 terms of variance standard deviation kurtosis skewness example of minimum maximum sum mean variance range standard deviation distribution: kurtosis distribution: skewness

45 variance, σ 2 recording data using computers variance standard deviation kurtosis skewness example of (1)variance is the expectation of the squared deviation of a random variable from its mean, and it informally measures how far a set of (random) numbers are spread out from their mean. (2)it is often represented by σ 2

46 standard deviation, SD, σ variance standard deviation kurtosis skewness example of is a measure that is used to quantify the amount of variation or dispersion of a set of data values. p A low standard deviation indicates that the data points tend to be close to the mean (also called the expected value) of the set a high standard deviation indicates that the data points are spread out over a wider range of values.

47 kurtosis recording data using computers variance standard deviation kurtosis skewness example of Figure: the Pearson type VII distribution with excess kurtosis of infinity (red); 2 (blue); and 0 (black)

48 skewness recording data using computers variance standard deviation kurtosis skewness example of Figure: Comparison of mean, median and mode of two log-normal distributions with different skewness.

49 variance standard deviation kurtosis skewness example of

50 variance standard deviation kurtosis skewness example of

51 Bayes theorem hypothesis testing Chi Square statistic regression correlation 1 recording data using computers

52 Bayes theorem hypothesis testing Chi Square statistic regression correlation Bayes theorem hypothesis testing Chi square testing Regression correlation...

53 basic concepts and formulas Bayes theorem hypothesis testing Chi Square statistic regression correlation Posterior Probability: P(h 1 x i ) Prior Probability: P(h 1 )

54 basic concepts and formulas Bayes theorem hypothesis testing Chi Square statistic regression correlation Posterior Probability: P(h 1 x i ) Prior Probability: P(h 1 ) Bayes theorem m P(x i ) = P(x i h j )P(h j ) (1) j=1 P(h 1 x i ) = P(x i h 1 )P(h 1 ) P(x i ) (2)

55 Bayes Example recording data using computers Bayes theorem hypothesis testing Chi Square statistic regression correlation Credit authorizations (hypotheses): 1 h1 = authorize purchase, 2 h2 = authorize after further identification, 3 h3 = do not authorize, 4 h4 = do not authorize but contact police.

56 Bayes theorem hypothesis testing Chi Square statistic regression correlation Bayes Example(Cont d):assign twelve data values for all combinations of credit and income Table: Add caption Excellent x 1 x 2 x 3 x 4 Good x 5 x 6 x 7 x 8 Bad x 9 x 10 x 11 x 12

57 Bayes theorem hypothesis testing Chi Square statistic regression correlation Bayes Example(Cont d):10 training data Table: Add caption ID Income Credit Class x i 1 4 Excellent h 1 x Good h 1 x Excellent h 1 x Good h 1 x Good h 1 x Excellent h 1 x Bad h 2 x Bad h 2 x Bad h 3 x Bad h 4 x 9

58 Bayes theorem hypothesis testing Chi Square statistic regression correlation Bayes Example(Cont d):10 training data Table: Add caption ID Income Credit Class x i 1 4 Excellent h 1 x Good h 1 x Excellent h 1 x Good h 1 x Good h 1 x Excellent h 1 x Bad h 2 x Bad h 2 x Bad h 3 x Bad h 4 x 9 From training data: P(h 1 ) = 60%; P(h 2 ) = 20%; P(h 3 ) = 10%; P(h 4 ) = 10%.

59 Bayes Example(Cont d): Bayes theorem hypothesis testing Chi Square statistic regression correlation Calculate P(x i h j ) and P(x i ) P(x 7 h 1 ) = 2/6; P(x 4 h 1 ) = 1/6; P(x 2 h 1 ) = 2/6; P(x 8 h 1 ) = 1/6; P(x i h 1 ) = 0 for all other x i. Predict the class for x 4 Calculate P(h j x 4 ) for all h j. Place x p 4 in class with largest value. Ex: P(h 1 x 4 )= (P(x 4 h 1 )(P(h 1 ))/P(x 4 ) = (1/6)(0.6)/0.1 = 1. x 4 in class h 1.

60 Bayes Example(Cont d): Bayes theorem hypothesis testing Chi Square statistic regression correlation Calculate P(x i h j ) and P(x i ) P(x 7 h 1 ) = 2/6; P(x 4 h 1 ) = 1/6; P(x 2 h 1 ) = 2/6; P(x 8 h 1 ) = 1/6; P(x i h 1 ) = 0 for all other x i. Predict the class for x 4 Calculate P(h j x 4 ) for all h j. Place x p 4 in class with largest value. Ex: P(h 1 x 4 )= (P(x 4 h 1 )(P(h 1 ))/P(x 4 ) = (1/6)(0.6)/0.1 = 1. x 4 in class h 1.

61 what is hypothesis testing? Bayes theorem hypothesis testing Chi Square statistic regression correlation hypothesis testing Find model to explain behavior by creating and then testing a hypothesis about the data. H 0 Null hypothesis; Hypothesis to be tested. H 1 Alternative hypothesis

62 Chi Square Statistic Bayes theorem hypothesis testing Chi Square statistic regression correlation formula O - observed value E - expected value based on hypothesis (usually denoted by h0). χ 2 = (O E) 2 E (3) example: O = 50, 93, 67, 78, 87 E = 75 χ 2 = and therefore significant

63 Principle of Chi Square Statistic Bayes theorem hypothesis testing Chi Square statistic regression correlation 1 (1) (O E) is a residual. 2 (2) Obviously, (O E) can represent a deviation degree of observation value from the theoretical value. It has shortcoming, (O E) maybe appear negative number, then the sum maybe leadto zero. So we using (O E) 2 3 (3) On the other hand, (O E) has relative property. If E is 10, then it is intolerable and too big that residual being 20. However, if E is 1000, then it looks very small that residual being 20. So before summation, using (Op E) 2 /E

64 Bayes theorem hypothesis testing Chi Square statistic regression correlation Principle of Chi Square Statistic(Cont d) 1 So, if χ 2 is less, we tend to accept H 0. 2 if χ 2 is large, we tend to reject H 0. 3 to what degree χ 2 is large? we need using χ 2 distribution.

65 Bayes theorem hypothesis testing Chi Square statistic regression correlation Principle of Chi Square Statistic(Cont d) 1 So, if χ 2 is less, we tend to accept H 0. 2 if χ 2 is large, we tend to reject H 0. 3 to what degree χ 2 is large? we need using χ 2 distribution. So χ 2 statistic can be applied into hypothesis testing

66 Regression recording data using computers Bayes theorem hypothesis testing Chi Square statistic regression correlation 1 Predict future values based on past values 2 Linear Regression assumes linear relationship exists. y = c 0 + c 1 x c n x n 3 Find values to best fit the data

67 Regression recording data using computers Bayes theorem hypothesis testing Chi Square statistic regression correlation 1 Predict future values based on past values 2 Linear Regression assumes linear relationship exists. y = c 0 + c 1 x c n x n 3 Find values to best fit the data

68 Correlation recording data using computers Bayes theorem hypothesis testing Chi Square statistic regression correlation Correlation Examine the degree to which the values for two variables behave similarly. Correlation coefficient r :

69 Correlation recording data using computers Bayes theorem hypothesis testing Chi Square statistic regression correlation Correlation Examine the degree to which the values for two variables behave similarly. Correlation coefficient r : 1 1 = perfect correlation 2-1 = perfect but opposite correlation 3 0 = no correlation 4 0 < r < 1 or 1 < r < 0, denote the correlation degree formula r = (xi X )(y i Ȳ ) (xi X ) 2 (y i Ȳ ) 2 (4)

70 Demo recording data using computers Bayes theorem hypothesis testing Chi Square statistic regression correlation In the next class. Demonstrate SPSS operation.

71 Homework Find an article (a paper, or a book) from websites, download it and read it.