Realtime visitor analysis with Couchbase and Elasticsearch

Transcription

1 Realtime visitor analysis with Couchbase and Elasticsearch Jeroen #nosql13

2 About me Jeroen Reijn Software engineer

3 About Hippo

4 Visitor Analysis Goto

5 Goto

6 Goto

7 Journey based Targeting

8 How we analyse Goto Hippo

9 Registration Visitor - entity making HTTP requests Collector - records data about a visitor or his behaviour Example: location collector (GeoIPCollector) Targeting Data - all data about a specific visitor Example: IP address is located in Amsterdam

10 Matching Characteristic - a type of fact about visitors Example: "comes from a city", "experiences a type of weather" Target Group - the specification of a Characteristic Example: "comes from a European city", "comes from Amsterdam" Persona - one or more target groups that describe a certain type of visitor Example: "Jim, the European urban consumer", "Alice, the Pet owner"

11 What do we store? Request log! Targeting data! Statistics Averages, e.g. how many visitors became which persona

12 Real-time analysis

13 How about YOU? Do you analyse your visitors? Do you do it realtime?

14 Architecture Goto

15 JSON XML (X)HTML App server Hippo Delivery Tier Hippo Repository RDBMS

16 Request Delivery Tier URL Matching Fetch content Compose output Response

17 Request NoSQL Matters 2013 Delivery Tier URL Matching Collect data Scoring Fetch content Compose output Response

18 Scaling Goto

19 Scaling out App server App server Hippo Delivery Tier Hippo Delivery Tier Hippo Repository Hippo Repository RDBMS

20 Scaling out App server App server Delivery Tier Targeting Datastore Delivery Tier Repository Repository RDBMS

21 What kind of storage? Goto

22 Typical Data Access Pattern Several reads Goto Single write Writer Datastore

23 Analytics Data Access Pattern Several writes Single read Datastore CMS user Writers

24 Targeting Data Access Pattern Several writes Single read Several reads Datastore CMS user Visitors

25 Distributed Cache

26 Requirements Goto change!

27 NoSQL? Goto

28 Suitable types Key-value store Document database Column oriented store

29 Assessment Criteria Maturity Data model Scalability Replication Performance Reliability Caching model Query model Support Consistency model Monitoring

30 Selection Criteria Performance Scalability Schema flexibility Simplicity

31 Couchbase Goto

32 Why Couchbase? Drop-in replacement for memcached Read/Write-through cache High throughput Easily scalable Schema flexibility Low latency

33 Couchbase Open Source Document-oriented Easy Scalable Consistent High Performance Apache licensed

34 Performance Object managed cache Write Queue to disk

35 Easy scalable Auto sharding Cross cluster replication (XDCR) Master - Master replication

36 Flexible data model Native JSON support Incremental Map Reduce Gives power to the developer

37 How we run Goto Hippo

38 Load Balancer Hippo Delivery Tier Couchbase cluster Database cluster Request log data Targeting data Statistics data

39 Analysis capabilities Querying via views Secondary indexes via views Views based on Map - Reduce Limited ad-hoc query capabilities

40 Elasticsearch Apache Lucene Designed to be distributed Schema free Apache license RESTful API

41 Added value Unstructured search Structured search Faceted search Geo spatial search Combinate all All in (near) real-time

42 Replication Hippo Delivery Tier Java API Read / Query Write Read Couchbase Server Cluster Elasticsearch Server Cluster XDCR Couchbase Transport plugin

43 What s Next? Goto

44 Advanced analytics

45 { Demo } Goto

46 ! Thanks! Goto