Rapid Large-Scale SOA - Connected Products at Leapfrog Enterprises A little bit about myself Jason Whaley Web Infrastructure Engineer Leapfrog Enterprises jwhaley@leapfrog.com What Will be Covered Overview of Leapfrog s Connected Products SOA implementation How we implemented and deployed in under a year Technologies used: What Why How What You Should Walk Away With Knowledge of what can go right and wrong with a fast paced SOA project Ideas from our project that you can apply and problems you can avoid in your own SOA project How to take advantage of the same technologies found in our SOA ecosystem Challenges and lessons learned The Problem to be Solved Four new Connected toys to be released in 08 with potentially more to come Uploading of tons of game play data that we want to use in diverse ways Integration with desktop applications and several web applications How do we collect that data and then expose it for consumption in diverse ways? Usages such as Game modification Game play rewards Learning progress reporting to parents Recommendation of new titles and/or toys and just about anything else our business can come up with The Solution? A Service Oriented Architecture Logically defined groups of loosely coupled services representing an abstraction of heterogeneous sources of data and and business logic that work together All Java-based Everything is exposed as a service Device uploading Data reporting Content (text and media) [Insert next requirement from our business here] 1
Rewind to early 07 Leapfrog does not host any externally exposed web applications itself All web applications and e-commerce outsourced and hosted by a third party Very few developers focused on the web and services located inhouse Connected products being developed but none in the marketplace No SOA pieces in place Little to no requirements for new upcoming products available The First Steps - Second Half of 07 Recruiting and contracting talent Team grew from single digits to 30+ members Bringing the website in-house Remove dependence on our third party providers Manage our website, content, and media with a content management system - Day Communiqué (more later ) The First Steps - Second Half of 07 End of 07 - What We Ended Up With Beginning of initial SOA Architecture Atlassian Crowd used for single sign-on and identity management Mule ESB to abstract and handle routing for communications between Legacy CRM System Third party e-commerce provider Crowd identity management Ad-hoc processes Begin planning and architecture for new products First Half of 08 - The Rubber Meets the Road Release date of our SOA platform established in late May - Five months for actual development and testing SOAP + CXF chosen for service and service client development Development begins on new CRM backend to be exposed by some services First Half of 08 - The Rubber Meets the Road (Cont.) Services to be hosted and wired together using: A set of Mule servers as a typical ESB Another set of Mule servers as an application container CMS (Communiqué) content exposed via its own RESTful service layer Web Applications used by consumers developed using Apache Wicket with a dependency on the new services 2
First Half of 08 - The Rubber Meets the Road (Cont.) The New Architecture: Bird s Eye View Porting integration with e-commerce provider to new architecture No changes needed by the client! All handled at the ESB level through endpoint abstraction and payload transformation Preparing to Scale - Late 08 Endpoint abstraction allowed easy horizontal scaling But many services were synchronous or too granular E.g. Device log data uploading - large payloads processed synchronously Preparing to Scale - Late 08 (Cont.) Introduce asynchronous queuing to device uploading and processing In the ESB Backbone - passing a device log for object transformation Cache responses for services that do not require the newest customer data Take advantage of ability to horizontally scale by adding more hardware Store often used media that gets referenced in services on Amazon S3 3
Device log processing object transformation and queuing Device log processing object transformation and queuing Device log processing - polling from the queue. Processing now asynchronous - will scale to high traffic Queuing and routing logic is separated from the actually processing logic Platform Selection Mule - Service Bus and Services Container Protocol abstraction Messaging and routing through configuration, not code Implementation code written in POJOs Negatives we ve experienced Not a completely mature product (Version 1.x). at the time of implementation, although much better now Internal developer adoption Sparse documentation Platform Selection Day Communiqué / CRX - Web and Content Management Agility for our business users Better solution for content management than alternatives Vendor supplied caching layer very beneficial for horizontal scaling Underlying CRX repository (JSR-170 compliant) easy to write services for Negatives we ve experienced VERY difficult for builds and deployment Difficult API (ContentBus) 4
Platform Selection Atlassian Crowd - SSO Solution Easy to use identity management system SOAP based Very stable under normal load Negatives we ve experienced Scalability under high load Platform Selection Apache Wicket - Web App Framework of Choice Component based web development Outstanding separation of presentation and logic No xml configuration Speedy to develop with (components encourage re-use) Development challenges Several geographically dispersed development teams. Several concurrent projects with multiple dependencies on each other Rapid releases for internal QA ing (2 to 3 releases required a week!) Solutions to Development Challenges Continuous Integration Began with Cruise Control, moving to Atlassian Bamboo Emphasis on keeping working builds at all times Apache Maven Dependency management enforced at build time Convention over configuration - allowed any developer to quickly and easily view and work on another project Synchronous Communication with each other at all working hours Lessons Learned A single source (group or person) of architecture needed Address scalability up front Turn synchronous services to asynchronous where possible. Consider transport overhead when writing your services. Keep your services decoupled as much as possible Future Work Mule 2.1 Communiqué 5 Crowd Clustering with Terracotta DSO Using leading edge technologies means owning their flaws - be prepared 5
Questions 6