The Platform as a Service of Google. Vincenzo Gianferrari Pini

Size: px

Start display at page:

Download "The Platform as a Service of Google. Vincenzo Gianferrari Pini"

Wilfred Briggs
6 years ago
Views:

1 The Platform as a Service of Google Vincenzo Gianferrari Pini June 11, 2012

2 Agenda General Architecture The Infrastructure as a Service of Google The Software as a Service of Google: Services and APIs The Platform as a Service of Google Google App Engine Scalability Restrictions Services The supported Application Environments The Java Runtime Environment The Database Support Google Web Toolkit SDK, IDE Support and deployment Secure Data Connector Pricing Case Study 29/06/2010 2

3 General Architecture 29/06/2010 2

4 The Infrastructure as a Service (IaaS) of Google The Google infrastructure is not directly accessible, so that it's not correct to speak about Infrastructure as a Service "Giant" infrastructure, spread over several datacenters in several countries based on the cooperation of a very large number of physical servers, not using common virtualization techniques but proprietary mechanisms instead the technical details are confidential Allows a large automatic scalability, by means of a deep parallelism and of large data redundancy for physical security Informations about Google security practices: 29/06/2010 2

5 The Software as a Service (SaaS) of Google Google Apps Personal Productivity and Collaboration Environment , Calendar, Drive, Sites, Video,... Google Maps many other well known services... Functional and administrative access by means of Web Interface rich set of APIs based on HTTP + REST protocols 29/06/2010 2

6 La Platform as a Service (PaaS) di Google 1. Google App Engine (GAE): runtime execution environment 2. Google Web Toolkit: for the development of complex client components Javascriptbased (optional) 3. Software Development Kit (SDK) and Google Plugin for Eclipse: for the development, full local debug and "automatic" application deployment 29/06/2010 2

7 Google App Engine PaaS runtime execution environment for both the Google SaaS and the custom applications Web access through HTTP protocol Manages scalability automatically, activating (Java) virtual machines on-the-fly on a very high number of physical servers Load Balancing Can quickly and automatically manage an increase from few to millions of users Fine tuning options High data redundancy hence there is no need to backup data (other than for logical recovery) Functionally rich Administration Console 29/06/2010 2

8 GAE Administration Console 29/06/2010 2

9 Google App Engine: The Sandbox Applications run inside a Sandbox with some limitations enforcing security and "good behaviour": No thread control No file system writes No socket opens 29/06/2010 2

10 Google App Engine: the Services Single Sign-on for authentication, integrated with all Google services 2-step authentication integration with corporate Single Sign-on Systems (SAML-based) Integration with Provisioning Systems (LDAP), like MS Active Directory URL Fetch Mail Memcache Image Manipulation Scheduled Tasks e Task Queues Database/Datastore... 29/06/2010 2

11 Google App Engine: Database and Datastore support App Engine Datastore, a NoSQL schemaless object datastore Query engine, with SQL interface High Replication Distributed Atomic Transactions Very high Scalability Google Cloud SQL, relational database based on MySQL RDBMS Fully Managed High Availability: automatic data replicas "across multiple geographic regions" Google Cloud Storage, for very large objects and files, size up to terabytes 29/06/2010 2

12 Google App Engine: The Application Environments Python Runtime Environment "GO" Runtime Environment Java Runtime Environment Java 6 Java 5 compatible Web Container (Servlet Container) Java Servlet Standard (including JSP) Using normal Java Web Development tools and standard APIs Supporting Java Data Objects (JDO) Java Persistence API (JPA) JavaMail java.net HTTP API 29/06/2010 2

13 Google Web Toolkit (GWT) Toolkit for developing and optimizing the GUI side of complex AJAX browser-based applications Code writing in Java, with automatic Javascript code generation optimized code the generated code is portable across the most popular browsers, including mobile browsers for iphone and Android Simplifying a lot the programmer's job Many Widgets available Visual interactive debug directly in Java Can be used also for developing the GUI side of non- GAE Web applications (is independent of GAE) 29/06/2010 2

14 SDK and IDE plugins SDK for compiling, debugging and deploying GAE and/or GWT applications Google Plugin for Eclipse GAE and/or GWT Visual interactive debug in local-mode (running offline) before deployment, providing a local emulator of the GAE cloud environment also using Database e/o Datastore A plugin for NetBeans is also available (provided by the open source community) "One Click" deploy versioning team development 29/06/2010 2

15 "One Click" deploy 29/06/2010 2

16 Secure Data Connector (SDC) Builds a secure (SSL) and controlled channel between on-premises data/applications and the Cloud-based Google SaaS and GAE Applications SDC Server resides behind the firewall SSL connection is initially started by the SDC server, after that communications flow is bi-directional Open Source Software in order to allow control and customization by the Corporation 29/06/2010 2

17 Secure Data Connector 29/06/2010 2

18 Pricing Quotas: safety and billable per day and per minute Pricing types: Free Accounts Within the billable quotas Paid Applications and Premier Accounts 99,95% SLA Monthly fixed fee (Paid $9 / app, Premier $500 / account) + variable fees Weekly billing for Paid, and monthly for Premier Several items considered fro the variable billing part, when the billable quotas are exceeded adjustable daily maximum budget 29/06/2010 2

19 Pricing: some billing quotas Instances Max 28 CPU h / day Outgoing Bandwidth: 1 GB / day Stored Data: 1 GB 200 indexes 29/06/2010 2

20 Instances dynamic class 1 (128 MB, 600 MHz): $0,08 / h class 2 (256 MB, 1,2 GHz): $0,16 / h class 4, classe 8 resident: $0,05 / h Outgoing Bandwidth: $0,12 / GB Stored Data: Blobstore: $0,13 / (GB x Month) Datastore: $0,24 / (GB x Month) Low-level Datastore operations Write: $0,10 / 100k_ops Read: $0,07 / 100k_ops Small: $0,01 / 100k_ops Channels: $0,00001 / channels_opened Recipients ed: $0,0001 / Pricing: main variable fees 29/06/2010 2

21 Via Grancini Milano

22 Intranet: Gestione Intranet: Gestione Intranet: Repository Extranet: Gestione Extranet: Repository Extranet: Gestione reclutamento richieste d'acquisto permessi e ferie documenti pratiche anomale documenti candidati Totali numero di utenti dimensioni database (GB) [1] [1] numero pratiche giornaliere gestite (su 10 ore lavorative?) numero pratiche di picco (all'ora?) dimensione delle pratiche movimentate (KB) richieste / pratica MHz / istanza Mcycles / richiesta Max CPU Utilization richieste /secondo richieste / secondo picco numero istanze medio numero istanze picco totale ore CPU / giorno Outgoing Bandwidth + 20% (GB / mese lavorativo) numero read per pratica numero read / mese lavorativo 240, , , ,000 35, ,000 1,835,000 Costo fisso mensile $9.00 $9.00 $9.00 $9.00 $9.00 $9.00 $54.00 Costo mensile istanze $0.00 $0.00 $0.00 $0.00 $0.00 $0.00 $0.00 Costo mensile Bandwidth $0.00 $0.00 $1.92 $0.00 $0.00 $0.00 $1.92 Costo mensile Storage $0.96 $3.36 $12.87[2] $9.36 $16.77[2] $18.96 $62.28

23 Intranet: Gestione richieste d'acquisto Intranet: Gestione permessi e ferie Intranet: Repository documenti Extranet: Gestione pratiche anomale Extranet: Repository documenti Extranet: Gestione reclutamento candidati Totali Costo mensile Datastore Operations $0.17 $0.21 $0.21 $0.35 $0.02 $0.32 $1.28 Totale mensile $10.13 $12.57 $24.00 $18.71 $25.79 $28.28 $119.48

24 1. Blobstore 2. Pricing Blobstore

25 Luca Zanconato

26 GoCloud GeoPlanning GoCloud GeoPlanning is an unique and innovative solution of Sales Force Automation, Optimized Mobility and Sales Process Optimization, oriented to rapidly increase sales and enhance scouting for new opportunities in the area, making the most of existing resources. Entirely developed using Google App Engine and GWT, uses the power of the PaaS to serve multiple users simultaneously and multitenancy to manage all the customers on the same application. GoCloud GeoPlanning uses: Google Web Toolkit to create user interfaces Google Maps to display data and calculate mobility optimization Google Latitude to locate user's position Google Apps to schedule events and communicate via Database.com to store data

27 Demo: Slots of appointments

com task queue task task Compute distances task Google Calendar

28 Slots request for a Customer Slots request and events sync Load campaign data Load Agent events Load Agent private events DB.com task queue task task Compute distances task Google Calendar Compute optimal slots... Fill empty days and sort Google Maps Slots response

29 Via Grancini Milano

30 Google App Engine Luca Zanconato

31 Introduction SDK and Google Eclipse Plugin Storing data DataStore Services URL Fetch Task Queues Mail Channel GWT Entry Point Communications with backend Practice Agenda

32 Introduction App Engine is the Google Platform as a Service (PaaS) offer, a distributed server with cloud technology able to run custom applications and let them scale automatically. It supports three development languages: Java Python Go (experimental) App Engine resides on the same cloud infrastructure that runs Google services (like GMail, Calendar etc...)

33 WAR AppEngine management Introduction WAR WAR Servlet Container DataStore WAR Servlet Container DataStore Servlet Container DataStore

34 Browser 1 request Introduction Application URL response 4 2 request response 3 Servlet Container DataStore Servlet Container DataStore Servlet Container DataStore

35 SDK and Eclipse Plugin App Engine SDK: GWT SDK: Eclipse plugin:

36 Storing Data App Engine provides three different ways for storing data: AppEngine DataStore Google Cloud SQL Google Cloud Storage DataStore, which we will use, is a NoSQL schemaless object database. Cloud SQL provides a MySQL database accessible via web services and hosted in the cloud. Cloud Storage is like a distributed filesystem where storing large objects and data chunks (even terabytes of data).

37 Storing Data: DataStore App Engine DataStore is a highly scalable Persistence Manager accessible by all the application instances and able to store objects. It is schema less, and the objects themselves (Entities), connected eachother by using instance variables, generate the structure of the database. There exist two kinds of relationship between objects, Owned and Unowned; objects that are connected with an Owned relationship belong to the same Entity Group.

38 Storing Data: DataStore Entity object public class = IdGeneratorStrategy.IDENTITY) private Key private String private String private Date hiredate; public Employee(String firstname, String lastname, Date hiredate) { this.firstname = firstname; this.lastname = lastname; this.hiredate = hiredate; } //...

39 Storing Data: DataStore Owned relationship example: Employee.java import java.util.list; = "employee") private List<ContactInfo> contactinfosets; ContactInfo.java import Employee; private Employee employee;

40 Storing Data: DataStore Unowned relationship example: public class = IdGeneratorStrategy.IDENTITY) private Key private Key favoritefood; public class = IdGeneratorStrategy.IDENTITY) private Key key;

41 Storing Data: DataStore GQL query example: Query query = pm.newquery(employee.class); query.setfilter("lastname == lastnameparam"); query.setordering("hiredate desc"); query.declareparameters("string lastnameparam"); try { List<Employee> results = (List<Employee>) query.execute("smith"); if (!results.isempty()) { for (Employee e : results) { //... } } else { //... no results... } } finally { query.closeall(); }

42 Services App Engine SDK provides a set of services via API, in order to communicate with other platforms or to handle large quantities of data by using small blocks. The main services are: URL Fetch Mail Task Queues Memcache Channel

43 Services: URL Fetch The URL Fetch service allows the application to call external servers by executing HTTP requests: it is possible to connect different cloud systems or to access Google services by using RSS feeds or REST/JSON APIs. try { URL url = new URL(" BufferedReader reader = new BufferedReader( new InputStreamReader(url.openStream())); String line; while ((line = reader.readline())!= null) { //... } reader.close(); } catch (MalformedURLException e) { //... } catch (IOException e) { //... }

44 Services: Mail The Mail service allows the application to send messages to public addresses. Session session = Session.getDefaultInstance(new Properties(), null); try { Message msg = new MimeMessage(session); msg.setfrom(new InternetAddress("admin@example.com", "Example.com Admin")); msg.addrecipient(message.recipienttype.to, new InternetAddress("user@example.com", "Mr. User")); msg.setsubject("your Example.com account has been activated"); msg.settext(msgbody); Transport.send(msg); } catch (AddressException e) { //... } catch (MessagingException e) { //... } It is possible to set as sender one of the developers' addresses associated to the application or an application-specific address.

45 Services: Task Queues The Task Queues service allows the application to enqueue operations (Tasks) on queues managed automatically by the server; it is very useful for elaborating a huge amount of data. import com.google.appengine.api.taskqueue.queue; import com.google.appengine.api.taskqueue.queuefactory; import static com.google.appengine.api.taskqueue.taskoptions.builder.*; //... Queue queue = QueueFactory.getDefaultQueue(); queue.add(url("/worker").param("key", key)); Every worker calls a Servlet passing to it HTTP parameters referring to the subset of data to process.

46 Services: Memcache The Memcache service provides a distributed in-memory data cache that can be used to speed up the application when the same data is requested by different instances. import com.google.appengine.api.memcache.memcacheservicefactory; import com.google.appengine.api.memcache.memcacheservice; //... MemcacheService cache = MemcacheServiceFactory.getMemcacheService(); Integer testvalue = 150; cache.put("value", testvalue); //... Integer val = (Integer) cache.get("value");

47 Services: Channel The Channel service allows to establish a persistent tunnel between the application and the JavaScript code, in order to manage events generated from the server to the client.

48 GWT Google Web Toolkit is an SDK that allows the developer to write HTML pages with AJAX support using the Java programming language. The Java classes are automatically compiled in JavaScript files and included in the associated web page.

49 GWT: Entry Point The main class that generates the JavaScript content of a page is called Entry Point, and it is always associated to a Module. A Module is an XML file containing dependencies and settings for the Java->JavaScript compiler. All the application Modules must be connected each-other if there exists a common Java code used by all the classes referenced into the modules themselves. After the compilation process, the concepts of Entry Point and in general of Class decay in favor of a pure prototype-oriented JavaScript code.

50 GWT: Entry Point Entry Point example: public class Test implements EntryPoint { public void onmoduleload() { final Button sendbutton = new Button("Send"); final TextBox namefield = new TextBox(); namefield.settext("gwt User"); final Label errorlabel = new Label(); RootLayoutPanel.get("nameFieldContainer").add(nameField); RootLayoutPanel.get("sendButtonContainer").add(sendButton); RootLayoutPanel.get("errorLabelContainer").add(errorLabel); //...

51 GWT: Entry Point Module example: <?xml version="1.0" encoding="utf-8"?> <module rename-to='test'> <inherits name='com.google.gwt.user.user'/> <inherits name='com.google.gwt.user.theme.standard.standard'/> <entry-point class='it.gocloud.test.client.test'/> <source path='client'/> <source path='shared'/> </module>

52 GWT: Communication with backend GWT provides native APIs to communicate with backend by serializing a Java object (which becomes a JavaScript object once compiled) and using the HTTP protocol. On the client-side for every service the developer must create two interfaces: an interface containing the methods an interface containing the methods with asynchronous callbacks On the server-side for every service the developer creates a Servlet that implements the interface containing the methods.

53 GWT: Communication with backend public interface GreetingService extends RemoteService { } String greetserver(string name) throws IllegalArgumentException; GreetingServiceAsync.java public interface GreetingServiceAsync { } void greetserver(string input, AsyncCallback<String> callback) throws IllegalArgumentException;

54 GWT: Communication with backend public class GreetingServiceImpl extends RemoteServiceServlet GreetingService { public String greetserver(string input) throws IllegalArgumentException { } } //...

55 GWT: Communication with backend Entry Point private final GreetingServiceAsync greetingservice = GWT.create(GreetingService. class); //... greetingservice.greetserver(texttoserver, new AsyncCallback<String>() { public void onfailure(throwable caught) { } //... public void onsuccess(string result) { }); } //...

56 Practice Create a project based on AppEngine-GWT with the following characteristics: an Entry Point with a textbox and a button an Entry Point with a label and a button a service that stores a string into the DataStore a service that returns the stored string The button in the first Entry Point will call the service that stores the string, passing to it the content of the textbox. The button of the second Entry Point will call the service that returns the string and will display it using the label.

57 Via Grancini Milano

INTRODUCTION TO GOOGLE APP ENGINE

INTRODUCTION TO GOOGLE APP ENGINE Sam Guinea guinea@elet.polimi.it http://servicetechnologies.wordpress.com/exercises/ A special thanks to my friends at GoCloud Outline General Architecture Software as