DRYAD / DRYADLINQ OVERVIEW. Xavier Pillons, Principal Program Manager, Technical Computing Customer Advocate Team

Transcription

1 DRYAD / DRYADLINQ OVERVIEW Xavier Pillons, Principal Program Manager, Technical Computing Customer Advocate Team

2 Data Intensive Scalable Computing (DISC) Market Customer needs for DISC lie on a spectrum One extreme is analytics targeting a traditional data warehouse. The analyst knows the cube he or she wants to build, and the analyst knows the data sources. Another extreme is analyzing raw unstructured data. The analyst does not know exactly what the data contains, nor what cube would be justified. The analyst needs to do ad-hoc analyses that may never be run again. HCP+Dryad targets the raw unstructured data extreme.

3 HPC+Dryad compliments Microsoft s existing data platform assets Microsoft already has great data platform assets PowerPivot, SQL Server Integration Services (SSIS), Parallel Data Warehouse (PDW), HPC+Dryad s focus on raw unstructured data analytics enables new solutions that incorporate multiple assets E.g., analyze raw unstructured data using HPC+Dryad then pipe it to SSIS and apply rest of BI stack

4 How is Microsoft responding?

5 Dryad is a technology MS uses internally to address the need Hadoop addresses for Yahoo Dryad has powered Microsoft Search for 5 years on thousands of servers Dryad is at the cutting edge of research on distributed computing Multiple publications at top-tier research conferences Dryad is a proven MS internal technology and HPC is making it available to customers Dryad is already available as a CTP

6 Dryad / DryadLinq / DSC Tools Languages and Libraries Visual Studio, Excel, etc. Visual Studio for C#/LINQ C#, C++, C#/LINQ for Dryad Distributed runtimes MPI SOA Dryad Cluster and cloud services HPC provisioning, management, etc. Distributed Storage Catalog Platform Windows Server Azure Bind individual NTFS shares together to support the Dryad distributed runtime

7 Dryad allows HPC to solve a new class of problems MPI Optimize CPU utilization for tightly coupled problems like climate modeling, car crash simulation, etc. SOA Optimize CPU utilization for loosely coupled problems like financial product pricing, etc. CPU Oriented Data Oriented Dryad Optimize for data locality rather than CPU utilization to support jobs that are primarily bound on disk I/O.

8 2-D Piping Unix Pipes: 1-D grep sed sort awk perl Dryad: 2-D grep 1000 sed 500 sort 1000 awk 500 perl 50

9 A Dryad Job Directed acyclic graph (DAG) Inputs Processing vertices Channels (file, fifo, pipe) Outputs

10 HPC + Dryad Job Overview 2a A Dryad job starts 1 basic task assigning a node as the DGM 3a Dryad Graph Manager starts/stops Dryad Vertices Application that calls Dryad APIs 1 1 Submit Dryad Job DSC HPC Head Node 2a 2b 2b The Dryad job also starts a set of parametric sweep tasks across the rest of the nodes as DVH 3b 3a HPC Compute Nodes 3b Dryad Graph Manager Dryad Vertex Host Dryad Vertices read and write files

11 Explaining DryadLINQ by example: extract Ulfar s favorite web pages from many web log files LINQ query transformed into computation graph var logentries = Input from line in logs where!line.startswith("#") Compute select new LogEntry(line); var user = from access in logentries where access.user.endswith(@"\ulfar") select access; var accesses = from access in user group access by access.page into pages select new UserPageCount("ulfar", pages.key, pages.count()); var htmaccesses = from access in accesses where access.page.endswith(".htm") orderby access.count descending select access; Compute and resort Compute and resort Output

12 More on HPC + Dryad mechanics 2a A Dryad job starts 1 basic task assigning a node as the DGM 3a DGM reads XML description of Dryad graph from share, calls DSC to locate files referenced in XML Application that calls Dryad APIs 1 DSC HPC Head Node 1 Publish to share: 1. binaries for Dryad job 2. XML description of Dryad graph 2a 2b 3b 2b The Dryad job also starts a set of parametric sweep tasks across the rest of the nodes as DVH 3a HPC Compute Nodes 3b Dryad Graph Manager Dryad Vertex Host DVH loads binaries for this Dryad job from share, executes them according to commands from DGM 4/14/

13 Takeaways Microsoft will compete with Hadoop using HPC+Dryad in the Data Intensive Scalable Computing market on both the cluster and cloud. Microsoft s DISC value proposition is about the overall stack, not about Dryad alone Dryad name subject to change before commercial release DryadLinq/DSC available as a CTP

14 Questions?

15 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.