DataCube Data Analysis With Mongodb.

Transcription

1 DataCube Data Analysis With Mongodb

2 datacube.taobao.com

3 shu.taobao.com

4 Project 数据化运营平台 首页 客户信息 类目信息 潜在用户 产品监控 运营工具

5 Agenda Offline Data Analysis Realtime Data Analysis New Aggregation Framework

6 Offline Data Analysis 异构数据源的 ETL 过程 基于 map/reduce 数据处理 CRUD create/read/update/delete

7 异构数据源的 ETL 过程

8 mongodb mysql 云梯

9 CRUD

10 try.mongodb.org

11 e.g: meta_user_info { } id : 1471, last_login_ip: , userversion: 2

12 Create mysql insert into meta_user_info(id, last_login_ip, userverion) values(1471, , 2) mongodb db.meta_user_info.save({ id: 1471, last_login_ip: , userversion: 2 }); db.meta_user_info.insert({id: 1471, last_login_ip: , userversion: 2 }); Any different?

13 Read mysql select last_login_ip, userversion from meta_user_info where id = 1471 mongodb db.meta_user_info.find({id: 1471}, {last_login_ip: 1, userversion: 1, _id: 0}) More info? sure

14 Enable Query Index db.user.ensureindex({id: 1})

15 query options Field Selection db.meta_user_info.find({id: 1471}) Sorting db.meta_user_info.find().sort({id: 1}) Skip and Limit db.meta_user_info.find().skip(1).limit(10) Ref:

16 Update mysql update meta_user_info set ip = where id = 1471 mongodb db.meta_user_info.update({id: 1471}, {$set: {ip: }}) Syntax: db.collection.update( criteria, objnew, upsert, multi ) Ref:

17 More about update In place update Read before update Care about multi option

18 delete mysql delete from meta_user_info where id = 1471 Monogdb db.meta_user_info.remove({id: 1471}) Ref:

19 Optimizition Enabling Profiling setprofilinglevel/getprofilingstatus 0 - off 1 - log slow operations (by default, >100ms is considered slow) 2 - log all operations Checking with explain

20 基于 map/reduce 数据处理

21 mapreduce map reduce query sort limit out finalize scope Syntax

22 e.g: topic { } title : "datacube data analysis using mongodb", speaker : { name : " SongheYang", address : Hangzhou" }, datetime : new Date(), ppt_pages : 30, tags : [ "mongodb2012", "admaster", "beijing" ]

23 Distinct var userlog_distinct_map = function() { }; emit(this.username, 1); var userlog_distinct_reduce = function(key, values) { }; return values[0];

24 Group var userlog_group_map = function() { emit(this.username, 1); }; var userlog_group_reduce = function(key, values) { var count = 0; for (index in values) { count += values[index]; } return count; };

25 var command = { mapreduce : in_coll_name, map : userlog_map.tostring(), reduce : userlog_reduce.tostring(), out : {replace : out_coll_name} };

26 Realtime Data Analysis Capped Collection a. fixed size b. fifo c. no _id index d. no delete, can drop it first Storm a. distributed b. realtime compute system c. fault-tolerant

27 Data Processing Code MongoSpout PvBolt/UvBolt OutputMongoBolt

28 New Aggregation Framework pipeline a. sequence of operators b. process a stream of documents expression a. calculate values

29 Pipeline Operators $project $match $limit $skip $unwind $group $sort

30 Expressions Comparison Operators $cmp, $eq Boolean Operators $and, $or Arithmetic Operators $add, $mod String Operators $substr, $toupper Date Operators $dayofmonth, $hour

31 $match db.runcommand({ aggregate : "topic", pipeline : [ { $match : { "speaker.name" : "Songhe Yang" } }] });

32

33 $project db.runcommand({ aggregate : "topic", pipeline : [ { $project : { _id : 0, title : 1, datetime : 1 } }] });

34

35 $project db.runcommand({ aggregate : "topic", pipeline : [ { $project : { _id : 0, title : 1, datetime : 1, name: "$speaker.name" } }] });

36

37 $unwind db.runcommand({ aggregate : "topic", pipeline : [ { $unwind : "$tags" } ] });

38

39 $group db.runcommand({ aggregate : "topic", pipeline : [ { $project: { _id : 0, speaker : 1, tags : 1 } }, { $unwind : "$tags" }, { $group : { _id : "$tags", count : { $sum : 1 }, authors : { $addtoset : "$speaker.name" } } }] });

40

41