Beer, Farms, and Peas

Transcription

1 Sir Francis Galton Karl Pearson William Sealy Gosset Beer, Farms, and Peas Applied Statistics & 석사과정이상열 Sir Ronald Fisher

2 Descriptive statistics is the discipline of quantitatively describing the main features of a collection of data, [1] or the quantitative description itself. Descriptive statistics are distinguished from inferential statistics (Wikipedia) 1. Central tendency(arithmetic mean, median, mode, Geometric mean ) 2. Dispersion(Sample standard deviation, Range, variance, mean difference ) 3. Minimum and Maximum values 4. Kurtosis, skewness R graphics : low-level graphics function high-level graphics function (R Package ggplot2, igraph) FF / Bigmemory : Large data processing package 01 Challenge 02 Descriptive statistics 03 R graphics 04 FF / Bigmemory

3 Challenge

4 Pareto Distribution( Generate 51 random numbers. Create a histogram of these random numbers hist(rnorm(51, , )) hist(rpareto(51,1,1))

5 Random numbers Pareto Distribution Code ( ppareto <- function(x, scale, shape) { ifelse(x > scale, 1 - (scale / x) ^ shape, 0) } - CDF qpareto <- function(y, scale, shape) { ifelse( y >= 0 & y <= 1, scale * ((1 - y) ^ (-1 / shape)), NaN ) } inverse cdf rpareto <- function(n, scale, shape, lower_bound = scale, upper_bound = Inf) { quantiles <- ppareto(c(lower_bound, upper_bound), scale, shape) uniform_random_numbers <- runif(n, quantiles[1], quantiles[2]) qpareto(uniform_random_numbers, scale, shape) }

6 Descriptive statistics

7 1. Data acquisition 서울열린데이터광장 ( 생필품가격 ( 2. Descriptive statistics pp <- read.csv("c:/users/user/desktop/korea/2013 년 2 학기 /datascience/ 발표 1/ 생필품가격.csv", header=true) dim(pp) [1] ppta <- split(pp$ 가격, list(pp$ 품목. 이름, pp$ 년도. 월, pp$ 시장유형. 구분. 시장. 마트.. 이름 )) ppta$` 고등어 ( 생물, 국산 ).Apr-13. 대형마트 ` [1] [24] ppta$` 고등어 ( 생물, 국산 ).Apr-13. 전통시장 ` [1] mean(ppta$` 사과 ( 부사, 300g).Sep-13. 전통시장 `, trim=0.05) [1] mean(ppta$` 사과 ( 부사, 300g).Sep-13. 대형마트 `, trim=0.05) [1] median(ppta$` 사과 ( 부사, 300g).Sep-13. 전통시장 `) [1] 2000 median(ppta$` 사과 ( 부사, 300g).Sep-13. 대형마트 `) [1] 2175

8 2. Descriptive statistics median(ppta$` 쇠고기 ( 한우, 불고기 ).Sep-13. 대형마트 `) [1] median(ppta$` 쇠고기 ( 한우, 불고기 ).Sep-13. 전통시장 `) [1] sd(ppta$` 쇠고기 ( 한우, 불고기 ).Sep-13. 전통시장 `) [1] sd(ppta$` 쇠고기 ( 한우, 불고기 ).Sep-13. 대형마트 `) [1] ( 한우, 불고기 ).Sep-13. 전통시장 `) [1] lapply(ppta, summary) $` 배추 ( 중간 ).Sep-13. 전통시장 ` Min. 1st Qu. Median Mean 3rd Qu. Max $` 사과.Sep-13. 전통시장 ` Min. 1st Qu. Median Mean 3rd Qu. Max $` 사과 ( 부사 ).Sep-13. 전통시장 ` Min. 1st Qu. Median Mean 3rd Qu. Max

9 # Split 함수이외의그룹핑하는방법 1. By (tapply 와비슷한역할이지만벡터대신객체를사용 ) 1. Aggregate ( 그룹의각변수별로 tapply 를한번식호출 )

10 3. Histogram 4. Boxplot

11 R graphics

12 1. High-level graphics function plot (generic graph function) boxplot Hist qqnorm curve 2. Low-level graphics function points lines abline segments polygon text High-level graphics function 를사용하면새로운그래프생성, low-level graphics function 는우선고수준그래픽을불러온뒤에추가호출

13 3. 9 월달생필품가격차이 ( 전통시장, 대형마트 ) plyr, ggplot2 package 사용하여생필품의품목별, 시장별가격차이를알수있었다. ( 코드는 에서참조 )

14 4. ggplot2 ggplot example iris data R 기반의그래픽패키지 The Grammar of Graphcis(Wikinson, 2005) 기반그래프객체를사용 유연한플로팅환경을제공. 그래프를프로그래밍화함. qplot 빠른플로팅을위한함수 ggplot 문법기준, 상세설정을위한함수 ggplot(iris,aes(sepal.length,sepal.width)) + geom_point(aes(colour=species)) + geom_smooth(aes(colour=species), method=lm) Referenece (

15 5. Network graph 활용사례 : 시계열데이터간상관계수값을이용하여네트워크그래프생성 library(igraph) gd <- graph(c(1,2, 2,3, 2,4, 1,4, 5,5, 3,6)) plot(gd) Label 사이의이름을추가하거나거리, 색깔, 사이즈모두바꿀수있음. 네트워크그래프에대한자세한설명은아래사이트에나와있음. R Graphics Cookbook ( urce=web&cd=1&ved=0cdiqfjaa&url=http%3a%2f%2fdeca.c uc.edu.cn%2fcommunity%2fmedia%2fp%2f23508%2fdownloa d.aspx&ei=vem9uu6- O8W1iQeFrIHgAg&usg=AFQjCNHJNZvnRcHXqfmXzNGuwmVRsE ZD1A&sig2=xNEX- Z9Eu6S4qsEyyHw7aQ&bvm=bv ,d.aGc&cad=rjt) : 개별 : 그룹화

16 FF / Bigmemory

17 Big data in the limit of R R works only on RAM (R 은 바이트의객체크기제한이있다. 객체가메모리에저장되기때문에 ) File.txt (big data) Data = read.table( File.txt, ) 1. 로딩이되어도오래걸리거나 2. 메모리에저장이안될정도로크거나 해결책 1. 청킹 (Chunking) 2. 메모리관리용 R package (ff, bigmemory, RevoScaleR) 3. 병렬실행 (Parallel execution)

18 1. Chunking read.table(file=, header = FALSE, nrow =?, skip =?, ) File.txt Data1 = read.table( File.txt, nrow = , skip = 0, ) Stat1 = summary(data1) Data2 = read.table( File.txt, nrow = , skip = , ) Stat2 = summary(data2) Data3 = read.table( File.txt, nrow = , skip = , ) Stat3 = summary(data3) Data4 = read.table( File.txt, nrow = , skip = , ) Stat4 = summary(data4) Aggregate(Stat1, Stat2, Stat3, Stat4) 데이터를분할하여분석가능하지만이것조차크다면처리가불가능하고분할및순차계산으로의해계산속도가느려질수있음.

19 2. 메모리관리용패키지 설명장점단점 ff 메모리를디스크에저장 Clean system few examples Bigmemory 메모리를디스크뿐만아니라메인메모리에도저장 쉬운사용방법 ff 보다널리쓰임 병렬패키지와의연동 (SNOW) 윈도우에서사용불가, 문자가들어있는데이터는전처리작업필요데이터자료형태를통일해야함 (like double, integer, short, char) RevoScaleR Revolution R Enterprise 상용프로그램으로 XDF 데이터형식을사용해서메모리한계를극복 C++ 프로그래머가외부메모리알고리즘을작성할수있도록확장가능한프레임워크제공 상용제품이기때문에아카데미버전만사용가능 ( 학생 ) 빠른연산속도

20 2. ff package ( 관련패키지 : ff, ffbase, biglm, biglars, speedglm)

21 2. ff example library(ff) rff <- read.table.ffdf(file="e:/dataset/3d_spatial_network.txt", sep=",",header=false) rff Data (3D Road Network Data set) V1 : ID V2 : LONGITUDE( 경도 ) V3 : LATITUDE( 위도 ) V4 : ALTITUDE( 높이 ) Data UCI machine Learning (

22 2. ff example library(ffbase) - Basic statistical functions for ff dim(rff) [1] sum(rff$v2) [1] mean(rff$v2) [1] min(rff$v2) [1] max(rff$v2) [1] range(rff$v2) [1] quantile(rff$v2) 0% 25% 50% 75% 100% hist(rff$v2)

23 2. ff example library(biglm) - Bounded memory linear regression dim(rff) [1] rfflm <- biglm(v1 ~ V2 + V3 + V4, data=rff) rfflm Large data regression model: biglm(v1 ~ V2 + V3 + V4, data = rff) Sample size = summary(rfflm) Large data regression model: biglm(v1 ~ V2 + V3 + V4, data = rff)

24 2. Bigmemory package( bigalgebra (linear algebra function) biganalytics (big.matrix such as GLM, bigkmeans) bigmemory bigtabulate (table, tapply, split 와같은함수를제공 ) synchronicity (data streaming, shared-memory capabilities) library(bigmemory) X <- read.big.matrix("/media/604e93df4e93ac72/dataset/3d_spatial_network.txt", header = FALSE, sep=",", type = "double", backingfile ="BigMem.bin", descriptorfile = "BigMem.desc", shared = TRUE) X An object of class "big.matrix" Slot "address": <pointer: 0xad5c470> > summary(x) Length Class Mode big.matrix S4 Data UCI machine Learning (

25 2. Bigmemory example > head(x) y x1 x2 x3 [1,] [2,] [3,] [4,] [5,] [6,] > dim(x) [1] Library(biganalytics) > sum(x) [1] e+13 > mean(x) [1] > colmax(x) y x1 x2 x e e e e+02 > colsd(x) y x1 x2 x e e e e+01 > colrange(x) min max y e e+08 x e e+01 x e e+01 x e e+02 > colsum(x) y x1 x2 x e e e e+06

26 2. Bigmemory example library(biganalytics) xlm <- biglm.big.matrix(y ~ x1 + x2 + x3, data=x) xlm Large data regression model: biglm(formula = formula, data = data,...) Sample size = summary(xlm) Large data regression model: biglm(formula = formula, data = data,...) Sample size = Coef (95% CI) SE p (Intercept) x x x bigkmeans(x, 3, iter.max=0, nstart=1) K-means clustering with 3 clusters of sizes , , Cluster means: [,1] [,2] [,3] [,4] [1,] [2,] [3,] Clustering vector: [1] [67] [133]