Common JBoss Data Grid Architectures

Size: px

Start display at page:

Download "Common JBoss Data Grid Architectures"

Archibald Banks
5 years ago
Views:

1 Common JBoss Data Grid Architectures Ray Tsang Solution Architect, Red Hat June 10, 2013

3 Lots of Reads Web Service Web Service Web Service Application

4 Use Design Patterns Application Service Layer DAO Web Service Web Service Web Service

5 Easy! Web Service Web Service Web Service Application DAO Service Layer

6 System of Record Application 3. Store in Service / DAO 1. Cached? 2. If not in Cache, Load It Web Web Web Service Service Service

7 System of Record Application Service / DAO 2. Update 1. Write Data to System or Record Web Web Web Service Service Service

9 JVM Application Service Layer DAO Embedded in your application Same JVM Faster access to cached objects

10 JVM JVM Heap Application Service Layer DAO Application Cache?

11 Manage Size of Eviction Max number of elements Expiration Expire after lifespan, or max idle Hit Ratio

12 Use Cases! Application Data JPA 2LC Non- Java Advanced API Transaction See Documentation for Embedded Library Mode vs Remote Server mode differences Share Data

14 JDG Server JVM Application DAO Client Service Layer Hot Rod REST Memcached

15 JDG Server CRM Application Hot Self Service Mobile PHP Application Rod Hot Rod REST hed Memcac

16 JDG Server Web Application Hot Rod External Loader Hot R od

17 Use Cases! Share Data Big Data Non- Java Transaction Advanced API See Documentation for Embedded Library Mode vs Remote Server mode differences

19 Embedded Replicated Distributed Remote

20 Node 2 Node 1 Node N Replicated

21 App Server Web App Web Content DAO App Server Replicated Load Balancer Service Layer Web App Service Layer DAO Web Content

22 Use Cases! Heavy Reads Extreme Redundancy Application Data Big Data Heavy Write Scaling

23 Node 2 Node 1 B A Node 3 C

24 Node 2 Node 1 A C B Node 3 B A C N- Copies for Redundancy

25 Node 1 Node 2 Node 3 Node 4 A B B C B A More Capacity!

26 Node 1 Node 2 Node 3 Node 4 A B C B A Rebalanced! C

27 I need A HTTP Load Balancer Node 1 Node 2 Node N A Oh No! B C B A C

28 I need A Node 1 Node 2 Node N A B C B A C

29 I need C Node 1 Node 2 Node N A B C B A C

30 Use Cases! Good Reads Good Writes Great Redundancy Big Data Scaling Non- Hot Rod

32 I need A REST Load Balancer Node 1 Node 2 Node N A Oh No! B C B A C

33 I need A REST Node 1 A B Copy to L1 Load Balancer Node 2 Node N A C B A C

34 I need A REST Load Balancer Node 1 Node 2 Node N A A B C B A C

35 Use Cases! Distributed Not Using Hot Rod Memory Consumption Mostly Reads High Saturation

36 App Server JDG Cluster (Distributed) Web App Hot R od Load Balancer Client Cache Service Layer JDG Server 1 DAO A B JDG Server 2 App Server B C Web App DAO Cache Client Service Layer od Hot R JDG Server N A C

37 I need A App Server JDG Cluster (Distributed) Cache Service Layer Load Balancer JDG Server 1 Client HTTP Web App DAO A B JDG Server 2 App Server B C Web App DAO Cache Client Service Layer JDG Server N Hot Rod A C

38 I need A App Server JDG Cluster (Distributed) Cache Service Layer Load Balancer JDG Server 1 Client HTTP Web App DAO A B JDG Server 2 App Server B C Web App A DAO Cache Client Service Layer JDG Server N A Retrieved he From Remote Cac C

39 Save A App Server JDG Cluster (Distributed) Cache Service Layer Load Balancer DAO A JDG Server 1 Client HTTP Web App A B JDG Server 2 App Server B C Web App A DAO Cache JDG Server N Client Service Layer Update Copy A C

40 Save A App Server JDG Cluster (Distributed) Cache Service Layer Load Balancer DAO A JDG Server 1 Client HTTP Web App A B JDG Server 2 Invalidate A App Server B C Web App A DAO Cache Client Service Layer JDG Server N A C

41 Use Cases! Scaling Heavy Read Heavy Write Share Data Keep Data Close

42 Configure Remote <server> <profile> <subsystem xmlns="urn:infinispan:server:core:5.2" default- cache- container="clustered > <cache- container name="clustered" default- cache="default"> <cache- container> <distributed- cache name= RemoteUsersCache" mode="sync" start="eager"/> </cache- container> </subsystem> </profile> </server>

43 Embedded + Remote <?xml version="1.0" encoding="utf- 8"?> <infinispan > <namedname= EmbeddedUsersCache > <clustering mode="invalidation"> </clustering> <loaders> <remotestore shared= true remotecache= RemoteUserCache" rawvalues="true"> <servers> <server host= remoteserver /> </servers> <connectionpool maxactive="10" exhaustedaction="create_new" /> <async enabled="true" /> <! or false, depending on your use case - - > </remotestore> </loaders> </namedcache> </infinispan>

44 JDG Cluster (New York) JDG Cluster (Los Angeles) JDG Server 1 JDG Server 1 A A B Cross Site Replication JDG Server 2 B C B JDG Server 2 JDG Server N B C JDG Server N A A C C

45 Use Cases! Share Data Multiple Data Centers Keep Data Close Online DR

47 System of Record Application 3. Store in Service / DAO 1. Cached? 2. If not in Cache, Load It Web Web Web Service Service Service

48 System of Record Application Service / DAO Get From JDG Not in Load It Web Web Web Service Service Service

49 System of Record Application Service / DAO 2. Update 1. Write Data to System or Record Web Web Web Service Service Service

50 System of Record Application Service / DAO Update Persist Data Web Web Web Service Service Service

51 Stores JDBC Filesystem Open Source JPA MongoDB Cassandra Write Your Own! Map to your own data

52 System of Record Application Service / DAO JDG Server 1 JDG Server 2 B Cache C JDG Server N B C B C

53 App Server JDG Cluster (Distributed) Web App Hot R od Load Balancer Client Cache Service Layer JDG Server 1 DAO A B JDG Server 2 App Server B C Web App DAO Cache Client Service Layer od Hot R JDG Server N A C

54 Application select * from where and lots and lots of where conditions

55 select * from where Application Query Tech Preview

56 Annotated public class Book String Date Set<Author> authors = new HashSet<Author>(); } Tech Preview

57 Query API SearchManager searchmanager = Search.getSearchManager(cache); Query query = //any Apache Lucene Query // convert the Lucene query to a CacheQuery: CacheQuery cachequery = searchmanager.getquery( query ); // get the results List<Object> found = cachequery.list(); Tech Preview

58 Use Cases! Full Text Query Sorting Big Data Query Heavy Write Pagination

59 Bring your code to the data Distributed Execution Map Reduce

60 Map Reduce public class WordCountMapper implements Mapper<String,String,String,Integer> { public void map(string key, String value, Collector<String, Integer> collector) { StringTokenizer tokens = new StringTokenizer(value); while (tokens.hasmoreelements()) { String s = (String) tokens.nextelement(); collector.emit(s, 1); } } } }

61 Map Reduce public class WordCountReducer implements Reducer<String, Integer> { public Integer reduce(string key, Iterator<Integer> iter) { int sum = 0; while (iter.hasnext()) { Integer i = (Integer) iter.next(); sum += i; } return sum; } }

62 Use Cases! Analytics Statistics Distribute Work Heavy Write

63 Embedded Library Mode! Map Reduce Transaction Query (Technical Preview) Must Use Embedded Library Mode

65 App Server EJB Data Service EJB Store DAO Application EJB Client Clustered - Distributed EJB Cal l App Server EJB Service EJB DAO Data Store

66 Enterprise Java Bean public interface CrudGridService<K, V> { public void put(v object); public void remove(k key); public V findbykey(k key); public Page<V> findall(pagerequest pagerequest); public Page<V> findall(pagerequest pagerequest, GenericQuery genericquery); public Long count(); public void clear(); }

67 Enterprise Java Bean protected Page<V> executepaginatedquery(query query, PageRequest pagerequest, Class<?>... classes) { List<V> items = null; int totalresults = 0; CacheQuery cq = searchmanager.getquery(query, classes); if (pagerequest!= null) { totalresults = cq.getresultsize(); cq.firstresult((pagerequest.getpage() - 1) * pagerequest.getpagesize()); cq.maxresults(pagerequest.getpagesize()); if (pagerequest.getsort()!= null) { cq.sort(pagerequest.getsort()); } List<Object> result = cq.list(); return new Page<V>(pageRequest, (List<V>) (List<?>) result, totalresults); } else { items = (List<V>) (List<?>) cq.list(); return new Page<V>(new PageRequest(1, items.size()), items, items.size()); } }

68 Enterprise ) public class UserGridServiceBean extends AbstractCrudGridServiceBean<String, User> implements UserGridService { }

69 Client UserGridService gridservice = (UserGridService ) context.lookup( ejb:grid- service- ear/grid- service- ejb/usergridservice! + UserGridService.class.getName()); User user = gridservice.findbykey( raytsang ); Page<User> users = gridservice.findall(new PageRequest(1, 10)); Page<User> users = gridservice.findall(new UserQuery( ), new PageRequest(1, 10)); UserTransaction txn = ; txn.begin(); gridservice.put( raytsang, updateduser1); gridservice.put( jane, updateduser2); gridservice.remove( charlie ); txn.commit();

70 Other Possibilities Create your own Data Grid Service Analytics with Map Reduce operations Query Multiple Intakes EJB service, Web Service, or Ingest data steam from a queue

71 JBoss Tusk Smart Data Created by Justin Hayes JBoss Reference Architecture Suitable for Addressing Big Data Integration Use Cases TB of Log Data, Activities, etc Need to swap out RDBMS and expensive/laborious process with more scalable, cost effective one

72 U se r App Server App Server App Server Acti v ity S EJB tream JDG Service EJB MRGM / A- MQ vice e / De m l i b o M Strea Data Data Transformation / Business Rules Appliance Data Stream Ingester MDB Real Time Analytics Long Te Stora rm ge Red Hat Storage

75 Links and Materials JBoss Tusk Presentation hayes_jboss_enterprise_middleware_and_big_data1.pdf JBoss Tusk Git Hub

Our Index. Searching in Infinispan. Infinispan Query engine Clustering a Lucene index Cloud deployed applications Future. Map/Reduce Fulltext indexing

Our Index. Searching in Infinispan. Infinispan Query engine Clustering a Lucene index Cloud deployed applications Future. Map/Reduce Fulltext indexing Who am I? Sanne Grinovero Software Engineer at Red Hat Hibernate, especially Search Infinispan, focus on Query and Lucene Hibernate OGM Apache Lucene JGroups Our Index Searching in Infinispan Map/Reduce