How$to$integrate$Hadoop$ with$your$nosql$database?

Transcription

1

2 How$to$integrate$Hadoop$ with$your$nosql$database? Tugdual$ Tug $Grall Technical$Evangelist

3 About$Me$ Tugdual$ Tug $Grall Couchbase Technical.Evangelist exo CTO Oracle Developer/Product.Manager Mainly.Java/SOA hap://blog.grallandco.com tgrall NantesJUG$coIfounder Pet$Project$: hap://

4 Big.Data High$Data$Variety$and$Velocity 2.00 Trillions.of.Gigabytes.(ZeKabytes) Structured$Data Unstructured$and$SemiB Structured$Data Text,,Log,Files,,Click, Streams,,Blogs,, Tweets,,Audio,, Video,,etc Source:.IDC.2011.Digital.Universe.Study.(hKp:// More$Flexible$Data$Model$Required 4

5 $30B.Database.Market.Being.Disrupted 95% RelaOonal$Technology Other <50%? RelaOonal$Technology RelaOonal$ Technology RelaOonal$ Technology NoSQL Technology All$new$database$growth$will$be$NoSQL

6 RealBOme,$ InteracOve$Databases AnalyOc Databases NoSQL Fast$access$ to$data Get$insights$from$ data Couchbase Mongo Cloudera Hortonworks

7 49 % 35 % 29 % 16 % 12 % 11 % Lack,of,flexibility/ rigid,schemas Inability,to,scale, Performance,challenges Cost All,of,these Other out,data Source:.Couchbase.Survey,.December.2011,.n.=.1351.

8 Hadoop

9 What$is$Hadoop? Highly$scalable Unstructured$data Open$source Big$Data$OperaOng$System Changing$the$World$One$Petabyte$at$a$Time

10 What$is$Hadoop? Simplest$unit$of$compute$and$storage Disks Application CPU Data

11 What$is$Hadoop? And$when$it$grows? Application Data

12 What$is$Hadoop? And$when$it$grows$more?

13 What$is$Hadoop? NoSQL$to$the$rescue Application Data

14 What$is$Hadoop? Hadoop$is$a$different$paradigm Application Data

15

16 Hadoop$and$NoSQL

17 Ad$and$offer$targeOng 40$milliseconds$to.respond.with. the.decision. 3 profiles,.real.@me.campaign. sta@s@cs 2 1 events profiles,.campaigns

18 Moving$Parts Ad Targeting Platform Couchbase Server Cluster sqoop export Logs Logs Logs Logs Logs sqoop import flume flow Hadoop Cluster

19 Content$&$RecommendaOon$TargeOng 3& make&& recommenda2ons& Content Oriented Site 1& events& Legacy Relational Database 2& user&profiles&

20 Moving$Parts In order to keep up with changing needs on richer, more targeted content that is delivered to larger and larger audiences very quickly, data behind content driven sites is shifting to Couchbase. Content Driven Web Site Couchbase Server Cluster Original RDBMS sqoop import sqoop export Logs Logs Logs Logs Logs flume flow Hadoop excels at complex analytics which may involve multiple steps of processing which incorporate a number of different data sources. sqoop import Hadoop Cluster

21 What$is$Sqoop? Sqoop is a tool designed to transfer data between Hadoop and relational databases. You can use Sqoop to import data from a relational database management system (RDBMS) such as MySQL or Oracle into the Hadoop Distributed File System (HDFS), transform the data in Hadoop MapReduce, and then export the data back into an RDBMS. sqoop.apache.org

22 What$is$Sqoop? Traditional ETL T Data Application Data

23 What$is$Sqoop? A different paradigm Data Applicatio n Data

24 What$is$Sqoop? A very scalable different paradigm Application Data Application Data Application Data Data

25 What$is$Sqoop? Where did the Transform go? TTT TTT TTT TTT Application Data

26 What$is$Sqoop? Sqoop$ SQLBHadoop Lots$of$custom$connectors Teradata,.Netezza Oracle,.MySQL,.Postgres

27 Sqoop$:$Import sqoop import --connect jdbc:mysql://rdbms1.demo.com/crm --table customers

28 Sqoop$:$Export sqoop export --connect jdbc:mysql://rdbms1.demo.com/analytics --table sales --export-dir /user/hive/warehouse/zip_profits --input-fields-terminated-by '\0001'

29 Sqoop$:$Import sqoop import -connect --table DUMP

30 Sqoop$:$Import Metadata Launches Sqoop$ Client Map Map Map HDFS HDFS HDFS MapReduceJob

31 Sqoop$:$Export sqoop export --connect --table DUMP --export-dir /user/hive/profiles/recommendation --username social

32 Sqoop$:$Export MapReduceJob HDFS HDFS HDFS Map Map Map Launches Sqoop$ Client Metadata

33 DemonstraOon

34 Couchbase

35 Couchbase$Server$Core$Principles Easy$ Scalability PERFORMANCE Consistent$High$ Performance a.single.click Consistent.subNmillisecond. with.consistent.high.throughput Always$On$ 24x365 JSON JSON JSON JSON Flexible$Data$ Model upgrades,.hardware.maintenance,. etc. JSON.document.model.with.no. fixed.schema.

36 Couchbase$Handles$Real$World$Scale

37 Couchbase$Server$ Query.API Memcapable Memcapable..2.0 Moxi Query$Engine Memcached Couchbase$EP$Engine Data$Manager storage$interface New$Persistence$Layer REST$management$API/Web$UI Heartbeat Process$monitor ConfiguraQon$manager Global$singleton$supervisor Rebalance$orchestrator Node$health$monitor vbucket$state$and$replicaqon$manager Cluster$Manager hdp on$each$node one$per$cluster Erlang/OTP HTTP 8091 Erlang.port.mapper 4369 Distributed.Erlang 21100,Q,21199

38 Couchbase$Server$ Query.API Memcapable Memcapable..2.0 Moxi Query$Engine Memcached Couchbase$EP$Engine storage$interface New$Persistence$Layer REST$management$API/Web$UI hdp Heartbeat Process$monitor ConfiguraQon$manager Global$singleton$supervisor on$each$node Rebalance$orchestrator Node$health$monitor vbucket$state$and$replicaqon$manager one$per$cluster Erlang/OTP HTTP 8091 Erlang.port.mapper 4369 Distributed.Erlang 21100,Q,21199

39 The$Classic$Order$Entry$Structure Rela%onal(databases(were(not(designed(with(clusters(in(mind,(which(is(why(people( have(cast(around(for(an(alterna%ve.(storing(aggregates(as(fundamental(units(makes( a(lot(of(sense(for(running(on(a(cluster.( 39

40 Aggregate$by$Comparison o::1001 { uid:. ji22jd, customer:. Ann, line_items:.[. {.sku: ,.quan:.3,..unit_price:.48.0.}, {.sku: ,.quan:.1,.unit_price:.39.0.}, {.sku: ,.quan:.1,.unit_price:.51.0.}. ], payment:.{....type:. Amex,...expiry:. 04/2001,. last5: } Easy$to$distribute$data Makes$sense$to$applicaQon$programmers 40

41 Basic$OperaOons APP,SERVER,1 APP,SERVER,2 COUCHBASE$Client$Library CLUSTER$MAP COUCHBASE$Client$Library CLUSTER$MAP READ/WRITE/UPDATE SERVER$1 ACTIVE SERVER$2 ACTIVE SERVER$3 ACTIVE s$distributed$evenly$across$ servers$ $5 $2 $4 $7 $1 $2 Each$server$stores$both$acOve$and$ replica$docs Only$one$server$acQve$at$a$Qme $9 $8 $6 Client$library$provides$app$with$ simple$interface$to$database $4 REPLICA $6 REPLICA $7 REPLICA Cluster$map$provides$map$ to$which$server$doc$is$on App$never$needs$to$know $1 $3 $9 App$reads,$writes,$updates$docs $8 $2 COUCHBASE,SERVER,,CLUSTER $5 MulOple$app$servers$can$access$same$ document$at$same$ome User.Configured.Replica.Count.=.1

42 Indexing APP$SERVER$1 APP$SERVER$2 COUCHBASE$Client$Library CLUSTER$MAP COUCHBASE$Client$Library CLUSTER$MAP Query SERVER$1 ACTIVE $5 SERVER$2 ACTIVE $5 SERVER$3 ACTIVE $5 Indexing,work,is,distributed,amongst, nodes Large,data,set,possible $2 $2 $2 Parallelize,the,effort $9 $9 $9 Each,node,has,index,for,data,stored,on,it REPLICA REPLICA REPLICA Queries,combine,the,results,from,required, nodes $4 $4 $4 $1 $1 $1 $8 $8 $8 COUCHBASE$SERVER$$CLUSTER

43 DemonstraOon

44 Map$Reduce$... Deal$with$ Big$Data More $is$beder$than$ Faster Batch$Oriented Usually$used$to$ extract/transform $data Fully$distributed Map,.Shuffle,.Reduce Distributed$ Executed$where$the$document$is Deal$with$ indexing $data$ As$fast$as$possible Use$to$query$the$data$in$the$Database

45 Conclusion Big$Data$and$Big$Users$working$together Use$Hadoop$to$store$ everything Batch.oriented Complex.data.processing MapReduce Expose$a$subset$of$the$dataset$to$your$applicaOon Low.latency

46 We re$hiring!$couchbase.com/careers

47 Q&A