Big Data with rubygems.org Download Data. Aja Hammerly

Transcription

1 Big Data with rubygems.org Download Data Aja Hammerly

2 Aja Hammerly

3

4 Lawyer Cat Says: Any code is copyright Google and licensed Apache

5 @thagomizer_rb Big Data

6 @thagomizer_rb DATA

7 @thagomizer_rb Big Data

8 @thagomizer_rb Storage is Cheap

9 @thagomizer_rb Intimidating

10 @thagomizer_rb OMG Statistics

11 @thagomizer_rb

12 @thagomizer_rb Machine Learning

13 @thagomizer_rb Exploratory

14 Rubygems Download

15 @thagomizer_rb Overview

16 @thagomizer_rb rubygems

17 Column Name id name created_at updated_at slug Type integer varchar datetime datetime

18 Column Name id name created_at updated_at slug Type integer varchar datetime datetime

19 @thagomizer_rb 126,007

20 @thagomizer_rb gem_downloads

21 Column Name Type id integer rubygem_id integer version_id integer count

22 @thagomizer_rb 883,848

23 @thagomizer_rb dependencies

24 Column Name id requirements rubygem_id version_id scope created_at updated_at unresolved_name Type integer varchar integer integer varchar datetime datetime

25 Column Name id requirements rubygem_id version_id scope created_at updated_at unresolved_name Type integer varchar integer integer varchar datetime datetime

26 @thagomizer_rb 3,638,968

27 @thagomizer_rb linksets

28 Column Name id rubygem_id home wiki docs mail code bugs created_at updated_at Type integer integer varchar varchar varchar varchar varchar varchar datetime

29 @thagomizer_rb 125,932

30 @thagomizer_rb versions

31 Column Name Type Column Name Type id integer authors text rubygem_id integer description text size integer summary text position integer requirements text number varchar platform varchar indexed boolean full_name varchar prerelease boolean licenses varchar latest boolean required_ruby_version varchar yanked_at datetime required_rubygems_version varchar built_at datetime info_checksum varchar updated_at datetime metadata hstore created_at datetime sha256

32 Column Name Type Column Name Type id integer authors text rubygem_id integer description text size integer summary text position integer requirements text number varchar platform varchar indexed boolean full_name varchar prerelease boolean licenses varchar latest boolean required_ruby_version varchar yanked_at datetime required_rubygems_version varchar built_at datetime info_checksum varchar updated_at datetime metadata hstore created_at datetime sha256

33 Column Name Type Column Name Type id integer authors text rubygem_id integer description text size integer summary text position integer requirements text number varchar platform varchar indexed boolean full_name varchar prerelease boolean licenses varchar latest boolean required_ruby_version varchar yanked_at datetime required_rubygems_version varchar built_at datetime info_checksum varchar updated_at datetime metadata hstore created_at datetime sha256

34 Column Name Type Column Name Type id integer authors text rubygem_id integer description text size integer summary text position integer requirements text number varchar platform varchar indexed boolean full_name varchar prerelease boolean licenses varchar latest boolean required_ruby_version varchar yanked_at datetime required_rubygems_version varchar built_at datetime info_checksum varchar updated_at datetime metadata hstore created_at datetime sha256

35 Column Name Type Column Name Type id integer authors text rubygem_id integer description text size integer summary text position integer requirements text number varchar platform varchar indexed boolean full_name varchar prerelease boolean licenses varchar latest boolean required_ruby_version varchar yanked_at datetime required_rubygems_version varchar built_at datetime info_checksum varchar updated_at datetime metadata hstore created_at datetime sha256

36 Column Name Type Column Name Type id integer authors text rubygem_id integer description text size integer summary text position integer requirements text number varchar platform varchar indexed boolean full_name varchar prerelease boolean licenses varchar latest boolean required_ruby_version varchar yanked_at datetime required_rubygems_version varchar built_at datetime info_checksum varchar updated_at datetime metadata hstore created_at datetime sha256

37 @thagomizer_rb 757,920

38 @thagomizer_rb Asking Questions

39 @thagomizer_rb Domain Knowledge

40 @thagomizer_rb Hypothesis

41 @thagomizer_rb Examples

42 The gem with the most downloads is

43 MiniTest is more popular than

44 Gems released in the last year require ruby >

45 Rails 3 is still more popular than rails

46 Fewer gems are released during

47 @thagomizer_rb Largish Data

48 @thagomizer_rb BigQuery

49 @thagomizer_rb What

50 @thagomizer_rb Why

51 @thagomizer_rb How

52 @thagomizer_rb I BigQuery

53 @thagomizer_rb SQL

54 @thagomizer_rb Fast

55 @thagomizer_rb Scales

56 @thagomizer_rb Complex Enough

57 @thagomizer_rb Demo

58 @thagomizer_rb Vocabulary

59 @thagomizer_rb Dataset

60 @thagomizer_rb Table

61 @thagomizer_rb Import

62 @thagomizer_rb Streaming

63 @thagomizer_rb gcloud

64 @thagomizer_rb pg

65 require 'pg' require 'gcloud' ENV["GOOGLE_CLOUD_PROJECT"] = "rubygems-bigquery" ENV["GOOGLE_CLOUD_KEYFILE"] =

66 gcloud bigquery = Gcloud.new = gcloud.bigquery bq_database = bigquery.dataset

67 @thagomizer_rb postgres = PG.connect dbname: "rubygems"

68 bq_table = bq_database.create_table("gems") do s s.integer s.string "id" "name" end s.timestamp "created_at" s.timestamp

69 @thagomizer_rb columns = %w[id name created_at updated_at]

70 postgres.exec("select * FROM rubygems") do pg_table pg_table.each do row hashed_row = Hash[columns.zip(row.values)] bq_table.insert(data) end

71 postgres.exec("select * FROM rubygems") do pg_table pg_table.each do row hashed_row = Hash[columns.zip(row.values)] bq_table.insert(data) end

72 postgres.exec("select * FROM rubygems") do pg_table pg_table.each do row hashed_row = Hash[columns.zip(row.values)] bq_table.insert(data) end

73 postgres.exec("select * FROM rubygems") do pg_table pg_table.each do row hashed_row = Hash[columns.zip(row.values)] bq_table.insert(hashed_row) end

74 @thagomizer_rb Zip & Hash[]

75 [ key1, key2, key3, key4 ] [ val1, val2, val3, val4

76 @thagomizer_rb zip

77 [ key1, key2, key3, key4 ] [ val1, val2, val3, val4 ] [[, ], [, ], [, ], [,

78 [ key1, key2, key3, key4 ] [ val1, val2, val3, val4 ] [[ key1, val1], [ key2, val2], [ key3, val3], [ key4,

79 [[key1, val1], [key2, val2], [key3, val3], [key4,

80 @thagomizer_rb Hash::[]

81 Hash[[key1, val1], [key2, val2], [key3, val3], [key4,

82 { key1 => val1, key2 => val2, key3 => val3, key4 => val4

83 @thagomizer_rb Hash[keys.zip(values)]

84 postgres.exec("select * FROM rubygems") do pg_table pg_table.each do row hashed_row = Hash[columns.zip(row.values)] bq_table.insert(hashed_row) end

85 @thagomizer_rb Batch

86 @thagomizer_rb Formats

87 @thagomizer_rb CSV

88 @thagomizer_rb JSON

89 @thagomizer_rb Avro

90 @thagomizer_rb CSV

91 require 'pg' require 'csv' require

92 postgres = PG.connect dbname: "rubygems" cols = %w[id requirements created_at updated_at rubygem_id version_id

93 query = "SELECT #{cols.join(',')} FROM dependencies" CSV.open(csv_path, "wb") do csv postgres.exec(query) do pg_table pg_table.each do row csv << row.values end end

94 storage = Gcloud.new.storage bucket = storage.bucket "goruco2016-bg-files" bucket.create_file csv_path,

95 @thagomizer_rb Import

96

97

98 @thagomizer_rb What Now?

99 @thagomizer_rb rubygems

100 @thagomizer_rb Simple

101 @thagomizer_rb Rails has the most downloads.

102 Which gem has the most

103 SELECT name, count FROM [rubygems.downloads] JOIN rubygems.gems ON rubygems.gems.id = rubygems.downloads.rubygem_id ORDER BY count DESC LIMIT

104 name count rake 107,076,261 rack 100,955,906 multi_json 100,171,080 json 95,715,131 bundler

105 SELECT name, sum(count) as total FROM [rubygems.downloads] JOIN rubygems.gems ON rubygems.gems.id = rubygems.downloads.rubygem_id GROUP BY name ORDER BY total DESC LIMIT

106 name count rake 214,152,212 rack 201,911,759 multi_json 200,342,260 json 191,430,173 bundler

107 How many downloads does Rails

108 SELECT name, sum(count) as total FROM [rubygems.downloads] JOIN rubygems.gems ON rubygems.gems.id = rubygems.downloads.rubygem_id WHERE name =

109 name total rails

110 Minitest is more popular than

111 SELECT name, sum(count) as total FROM [rubygems.downloads] JOIN rubygems.gems ON rubygems.gems.id = rubygems.downloads.rubygem_id GROUP BY name HAVING name IN ('minitest',

112 name total minitest rspec

113 Gems released in the last year require ruby >

114 SELECT required_ruby_version, COUNT(*) AS total FROM rubygems.versions WHERE created_at > DATE_ADD(CURRENT_TIMESTAMP(), -1, "YEAR") GROUP BY required_ruby_version ORDER BY total

115 name total >= 0 95,857 >= ,069 >= ,624 >= 2.0 1,648 >=

116 @thagomizer_rb Complex

117 Rails 3 has more downloads than the other Rails major

118 SELECT name, REGEXP_EXTRACT(number,r'(\d\.)') AS major, sum(rubygems.downloads.count) AS total FROM [rubygems.versions] JOIN rubygems.gems ON rubygems.gems.id = rubygems.versions.rubygem_id JOIN rubygems.downloads ON rubygems.versions.rubygem_id = rubygems.downloads.rubygem_id WHERE rubygems.gems.name = 'rails' GROUP BY name, major ORDER BY

119 SELECT name, REGEXP_EXTRACT(number,r'(\d\.)') as major, sum(rubygems.downloads.count) as total FROM [rubygems.versions] JOIN rubygems.gems ON rubygems.gems.id = rubygems.versions.rubygem_id JOIN rubygems.downloads ON rubygems.versions.rubygem_id = rubygems.downloads.rubygem_id WHERE rubygems.gems.name = 'rails' GROUP BY name, major order by

120 @thagomizer_rb REGEXP_EXTRACT(number,r'(\d\.)') as major

121 version downloads 0 2,890,350, ,064,535, ,991,436, ,378,651, ,662,487,252 5

122 version downloads 0 2, , , , ,662 5

123 Gems released in the last year require ruby >

124 SELECT required_ruby_version, COUNT(*) AS total FROM rubygems.versions WHERE created_at > DATE_ADD(CURRENT_TIMESTAMP(), -1, "YEAR") GROUP BY required_ruby_version ORDER BY total

125 SELECT REGEXP_EXTRACT(required_ruby_version, r'(.*?\d\.?)') AS version, COUNT(*) AS total FROM rubygems.versions WHERE created_at > DATE_ADD(CURRENT_TIMESTAMP(), -1, "YEAR") GROUP BY version ORDER BY total

126 name total >= 0 95,851 >= 1 13,080 >= 2 12,944 ~> 2 2,040 > 2

127 @thagomizer_rb Thank You

128 @thagomizer_rb