COMP6511A: Large-Scale Distributed Systems. Windows Azure. Lin Gu. Hong Kong University of Science and Technology Spring, 2014

Size: px

Start display at page:

Download "COMP6511A: Large-Scale Distributed Systems. Windows Azure. Lin Gu. Hong Kong University of Science and Technology Spring, 2014"

Virgil Skinner
5 years ago
Views:

1 COMP6511A: Large-Scale Distributed Systems Windows Azure Lin Gu Hong Kong University of Science and Technology Spring, 2014

2 Cloud Systems Infrastructure as a (IaaS): basic compute and storage resources E.g., Amazon AWS/EC2, VMWare vcloud Platform as a (PaaS): cloud application infrastructure E.g. Google App Engine, Salesforce.com, Windows Azure Software as a (SaaS): cloud applications E.g. Google Docs, Microsoft Office Web Companions, Office 365

3 The Microsoft Cloud Categories of s s Software s Platform s Infrastructure s

4 Windows Azure - Overview Windows Azure presents itself as a platform for cloudbased application development For the datacenter: parallel processing, resource management, provisioning, and monitoring For users: cloud-based applications, integrated development environment

5 Cloud Based Computation with Windows Azure Web Role Worker Role HTTP Load Balancer IIS ASP.NET, WCF, etc. main() { } Agent Agent Fabric Compute Storage Fabric VM

6 Model Cloud apps are constructed in role-instance Model A role instance is a set of code, configuration, and local data, deployed in a dedicated VM At runtime each Role executes on one or more instances

Windows Azure Web & Worker Roles Worker Role

Background processing Web Role Customized

role with the addition of IIS (i.e., it can

Windows Azure currently supports two types

A service must include at least one role of

7 Windows Azure Web & Worker Roles Worker Role For general development and computation Background processing Web Role Customized for web applications A web role is a worker role with the addition of IIS (i.e., it can do everything a worker role can do as well) Windows Azure currently supports two types of roles: worker role and web role. A service must include at least one role of either type, but may consist of any number of web or worker roles.

8 Azure Node Structure A node is a management unit of FC Contains an FC Agent in Hyper-V root partition Each role instance runs in Guest OS with GA FA delegates GAs to handle VM status

Enterprise Virtualization Stack (VSC) s Guest OS Server Enterprise

9 High-Level Architecture Host Partition Guest Partition Guest Partition Host OS Server Core Virtualization Stack (VSP) s Guest OS Server Enterprise Virtualization Stack (VSC) s Guest OS Server Enterprise Virtualization Stack (VSC) Drivers VMBUS VMBUS VMBUS Hypervisor NIC Disk1 Disk2 Hardware CPU

10 Windows Azure Compute Virtual Machines The VMs are provided by a cloud-optimized hypervisor s see a 64-bit Windows Server 2008 or Windows Server 2012 interface Development fabric: a desktop replica of Windows Azure

Node Health Monitoring Based on heartbeats, which are typically sent in 15 second intervals Once the index falls below zero, FC attempts to heal node For example, host agent timeout is 10 minutes

11 Node Health Monitoring Based on heartbeats, which are typically sent in 15 second intervals Once the index falls below zero, FC attempts to heal node For example, host agent timeout is 10 minutes Timeouts vary depending on node state and operation Worst-case reaction time is timeout interval + heartbeat interval Missing Heartbeats Missed Heartbeat Recovery Initiated Node Health Index Heartbeat Interval Heartbeat Timeout

12 Windows Azure Storage Types of Storage HTTP Blobs Tables Queues Compute Storage Fabric

13 Windows Azure Storage The Windows Azure storage services provide storage for binary and text data, messages, and structured data Blob service: storing binary and text data Queue service: storing messages that may be accessed by a client Table service: structured storage for non-relational data Windows Azure drives: mounting an NTFS volume accessible to code running in your Windows Azure service Azure provides scalable, durable cloud storage, backup and recovery solutions for any data, large and small. --

14 Windows Azure Storage Tables Table Table Table... Entity Entity Entity... Storage Accounts Property Property Property Name Type Value

15 Windows Azure Storage Tables: Strengths Massively scalable Perspective: Applied to the right problem, Windows Azure Tables are a beautiful thing But they re not the optimal solution for many other applications Amazon, Google, and others provide similar abstractions

16 Windows Azure Storage Tables: Limitation Access via REST You can t use ordinary ADO.NET No SQL A new approach for developers to learn No aggregates You can t easily move relational data to or from it No schema Entities and properties are not exactly rows (tuples) and columns (attributes)

17 SQL Azure (a.k.a. SQL s) SQL Azure Database Others (Future).NET s s SQL Azure Windows Azure s Windows Server Windows Vista/XP Windows Mobile Others

18 SQL Azure Database SQL Azure Formerly known as SQL Data s (SDS) and SQL Server Data s (SSDS) Huron Data Hub Built on the Microsoft Sync Framework, which is a comprehensive synchronization platform that enables collaboration and offline access for applications, services, and devices with support for any data type, any data store, any transfer protocol, and any network topology. Limitations in functionality and scalability In the future: reporting, analysis, extract/transform/load (ETL) services,

19 Using Queues An application model To scale, add more of either 1) Receive work Web Role ASP.NET, WCF, etc. Worker Role main() { } 4) Do work 2) Put work in queue 3) Get work from queue Queue

20 Azure Apps - Scale out

Roles Step 5: Maintain desired number of roles Failed roles automatically restarted Node failure results

21 Windows Azure Deployment Step 1: Allocate nodes Across fault domains, across update domains Step 2: Place OS and role images on nodes Step 3: Configure settings Step 4: Configure loadbalancers, Start Roles Step 5: Maintain desired number of roles Failed roles automatically restarted Node failure results in new nodes automatically allocated Fault-tolerant high-performance distributed computing fabric Load-balancers

Image-Based Deployment Host Partition Guest Partition Guest

Package App3 Package App2 Package Host partition differencing

differencing VHD Guest partition differencing VHD HV-enabled

22 Image-Based Deployment Host Partition Guest Partition Guest Partition Maintenance OS Guest Partition VHD VHD VHD App1 Package App3 Package App2 Package Host partition differencing VHD Guest partition differencing VHD Guest partition differencing VHD Guest partition differencing VHD HV-enabled Server Core base VHD Server Enterprise Server base Enterprise base VHD VHD Server Core base VHD

23 How Azure runs an app The Fabric Controller Process app definition files to create VHDs (Virtual Hard Disks) Place role VHDs on compute nodes Boot and configure role instance VMs Start role host processes (IIS, WCF ) Run role instances Response to HTTP request Maintain service health

24 Bus 2) Discover endpoints 3) Access application Bus Registry Endpoints 1) Register endpoints Organization X Organization Y Access Control Bus

25 Bus Exposing internal applications on the Internet isn t a comfortable solution Firewalls get in the way, may be slow, expensive Bus: provides a cloud-based intermediary between clients and internal applications Direct or queued It also provides a service registry that clients can use to find the services they need

26 Azure Apps Overview The Internet The Internet via TCP or HTTP L B L B L B Web Web Site Site (ASPX, ASMX, (ASPX, ASMX, Role WCF) WCF) IIS as Host Tables Queues Storage Blobs Worker Worker Role Managed Interface Call Windows Azure Data Center

Web Patterns User Web Browser Mobile Browser Silverlight WPF Enterprise Enterprise Web Svc ASP.NET ASP.NET (Web ASP.NET Role) (Web ASP.NET Role) (Web ASP.NET Role) (Web Role) (Web Role) Web Svc ASP.

27 Web Patterns User Web Browser Mobile Browser Silverlight WPF Enterprise Enterprise Web Svc ASP.NET ASP.NET (Web ASP.NET Role) (Web ASP.NET Role) (Web ASP.NET Role) (Web Role) (Web Role) Web Svc ASP.NET (Web ASP.NET Role) (Web ASP.NET Role) (Web ASP.NET Role) (Web Role) (Web Role) Jobs (Worker ASP.NET ASP.NET (Web Role) ASP.NET Role) (Web ASP.NET Role) (Web Role) (Web Role) Data Enterprise Data Table Storage Blob Storage Queue Storage Enterprise Identity Private Cloud User Data Data Reference Data Bus Access Control Workflow Identity Public Cloud s

Patterns Grid / Parallel Computing User Web Browser Mobile Browser Silverlight WPF Enterprise

NET Role) (Web ASP.NET Role) (Web Role) (Web Role) Web Svc ASP.NET (Web ASP.NET Role) (Web ASP.NET Role) (Web ASP.NET Role) (Web Role) (Web Role) Jobs (Worker ASP.

NET Role) (Web Role) (Web Role) Data Enterprise Data Table Storage Blob Storage Queue Storage

28 Patterns Grid / Parallel Computing User Web Browser Mobile Browser Silverlight WPF Enterprise Enterprise Web Svc ASP.NET ASP.NET (Web ASP.NET Role) (Web ASP.NET Role) (Web ASP.NET Role) (Web Role) (Web Role) Web Svc ASP.NET (Web ASP.NET Role) (Web ASP.NET Role) (Web ASP.NET Role) (Web Role) (Web Role) Jobs (Worker ASP.NET ASP.NET (Web Role) ASP.NET Role) (Web ASP.NET Role) (Web Role) (Web Role) Data Enterprise Data Table Storage Blob Storage Queue Storage Enterprise Identity Private Cloud User Data Data Reference Data Bus Access Control Workflow Identity Public s

Introduction to Windows Azure Cloud Computing Futures Group, Microsoft Research Roger Barga, Jared Jackson, Nelson Araujo, Dennis Gannon, Wei Lu, and

Introduction to Windows Azure Cloud Computing Futures Group, Microsoft Research Roger Barga, Jared Jackson, Nelson Araujo, Dennis Gannon, Wei Lu, and Jaliya Ekanayake Range in size from edge facilities