I D C T E C H N O L O G Y S P O T L I G H T Orchestration S i m p l i f i es and Streamlines V i r t u a l and Cloud D a t a Center Management January 2013 Adapted from Systems Management Software Purchasing Priorities Must Focus on Productivity and Simplification in 2013 by Mary Johnston Turner, IDC #239056 Sponsored by IBM Virtualization, cloud computing, and the consumerization of IT continue to drive fundamental shifts in data center management priorities. Many organizations are implementing multi-hypervisor architectures and hybrid public and private cloud strategies. Converged systems integrate servers, storage, networks, and software to optimize the performance of specific workloads and simplify system deployment and operations. Yet, traditional data center architectures continue to support many mission-critical business applications. This Technology Spotlight examines how advanced automation and orchestration solutions are helping IT data center operations teams effectively manage increasingly complex enterprise computing environments. It also looks at IBM SmartCloud Orchestrator as an example of a solution that is addressing these emerging IT management priorities. Industry Trends: Dynamic, Heterogeneous Data Centers Demand Advanced Management Solutions IDC believes enterprise IT is in the midst of a highly disruptive transition away from client/serverbased architectures toward much more dynamic and agile environments enabled by cloud, big data, social technologies, and mobility. IDC's research indicates that the typical North American IT decision maker expects the combined cost of buying and managing public and private cloud resources will consume approximately half of his or her IT budget by 2016. The average number of virtual machines (VMs) per physical server doubled from 5 6 VMs in 2008 to 9 10 VMs in 2012 in mature customer accounts. Multi-hypervisor infrastructures are becoming more common. Simultaneously, converged systems that combine servers, storage, networks, and software are gaining traction. IDC expects the market for converged systems to grow at a compound annual growth rate (CAGR) of 54.7% from $2.0 billion in 2011 to $17.8 billion in 2016. The software-defined network ecosystem, including the associated network infrastructure, will grow at an exponential pace as well. The resulting enterprise data centers will be not only more agile and dynamic but also more complex and challenging to manage as mainframe and client/server-based applications sit side by side with workloads running on virtual servers, converged systems, and dynamic public, private, and hybrid clouds. IDC expects that in response to this complexity, the drive to improve IT staff productivity and operational efficiency will be the number 1 IT management software priority in 2013 (see Figure 1). Implementing common processes and tools across cloud and non-cloud environments and reducing the number of disconnected management tools in use are also high priorities. Collectively, these drivers point to IT environments under pressure to make dramatic improvements in staff productivity, time to value, cost control, and business risk management. IDC 1446
F i g u r e 1 Top IT Management Software Priorities for 2013 Improve IT staff productivity and operational efficiency 48.8 Implement common management software, services, and processes across cloud and non-cloud environments Reduce number of management software vendors and tools 40.3 39.3 Aggressively invest in automation and orchestration Shift from on-premise to SaaS-based management 22.9 22.4 Shift performance and availability monitoring from system level to APM Rely more on management software embedded in converged systems or offered as part of a cloud service Accelerate use of usage-based chargeback to LOBs 17.4 16.9 20.4 Emphasize free open source management software over paid software Shift routine provisioning and service management tasks to end users via self-service tools 15.9 15.4 Other 1.0 0 10 20 30 40 50 60 (% of respondents) n = 201 Note: Multiple responses were allowed. Source: IDC's IT Management Quickpoll Survey, January 2013 Automation and orchestration solutions will be critical technologies to address this set of requirements in 2013 and beyond. IT teams will need tools that can standardize, streamline, and optimize resource provisioning, workload placement, real-time resource scaling, and capacity utilization across the development, test, release, and production life cycles of mission-critical applications. To effectively address the complex requirements of today's data centers, IT organizations need orchestration solutions that can consistently implement service models, governance, and policies across complex, heterogeneous environments including cloud, virtual, and legacy infrastructure. 2 2013 IDC
Unlike earlier-generation tools that enabled task-based server configuration automation or self-service VM provisioning, emerging data centers need automation tools that can react rapidly to real-time performance and capacity requirements, scale resources on the fly, and integrate across applications, infrastructure, and management resources. Rather than rely on traditional, brittle script-based automation linked to specific hardware and application resource configurations, tools that are optimized for dynamic virtual and cloud data centers rely on templates or patterns to define service configuration and policies that can be consistently applied across heterogeneous platforms. By taking an approach that uses open standards or APIs to abstract service provisioning and management automation from infrastructure configuration control, advanced orchestration solutions can more easily scale and adapt to changing infrastructure and applications. In evaluating orchestration solutions, organizations need to consider how to best accommodate today's environment while also maintaining the ability to evolve that environment over time. To ensure rapid time to value, IT buyers should look for solutions that can integrate with existing management tools and data center resources rather than require IT to implement islands of automation in isolation from the broader operational environment. Particularly for organizations that expect to operate hybrid virtual, cloud, and legacy environments for a number of years, it is important that automation and orchestration solutions enable unified workflows and help integrate configuration, provisioning, capacity utilization, and resource utilization across all available resources and infrastructure. Many organizations start with fairly limited automation and orchestration programs and then extend use of those technologies as virtualization and cloud architectures become more widely used. Important attributes that help deliver value quickly while ensuring long-term agility and flexibility include: Sophisticated multi-tier template and service design tools to enable IT staff to create and maintain infrastructure, middleware, and application patterns that best support specific workloads across a range of in-house and public cloud infrastructure options Built-in discovery and dependency mapping capabilities to streamline service design and provisioning activities and to support real-time migration and optimization activities Integrated monitoring and analytics to support real-time capacity optimization, provisioning, and workload management automation across heterogeneous hardware platforms, multi-hypervisor environments, and hybrid public/private clouds Role-based service catalogs and self-service provisioning capabilities that manage and enforce business and configuration compliance and utilization policies while enabling both IT staff and end users to execute routine tasks more quickly and efficiently The ability to consistently define and apply policies and governance across groups, users, workloads, and platforms Scalable plug-and-play architecture that can expand as needed over time Ease of integration with existing management tools, processes, and workflows The ability to implement increasingly sophisticated levels of automation and orchestration over time as the organization becomes comfortable with this type of highly integrated approach to day-to-day configuration, deployment, migration, and optimization activities Whether the initial use case for orchestration focuses on automating virtual machine and private cloud deployments for application developers or aims to dynamically burst workload placements across in-house and public cloud resources, any orchestration tool should be able to be continuously extended across a wide range of infrastructure, middleware, and application profiles over time. As applications evolve over the next several years to become more mobile and cloud enabled, state-of-the-art orchestration solutions should be able to evolve with the needs of the organization by 2013 IDC 3
supporting an ever-increasing array of standards, service templates, patterns, and analytics. By investing in a scalable solution today, IT organizations will be able to continue to leverage current pattern, workflow, and service designs over the long term. Together, the capabilities listed previously will allow IT management teams to realize the benefits of automation and orchestration immediately in those areas where they have the most current need while enabling them to scale out use of the technology as requirements change over time. Benefits of Orchestration in Complex Virtual and Cloud Data Centers As enterprise IT environments become more densely virtualized and look to dynamically scale across in-house and external public cloud resources, traditional manual processes and tools for tracking VM and associated workloads will no longer be adequate to support the real-time needs of the enterprise. Similarly, application and middleware configuration, update, deployment, and migration requirements need to move beyond traditional, manual patching and change control processes. Orchestration solutions that enable IT staff to create, maintain, and reuse infrastructure, middleware, and application patterns across a range of in-house and public cloud infrastructure options will allow for faster and more consistent deployments and migrations. When paired with open standards based infrastructure APIs and third-party and community-provided best practices, these solutions will enable enterprise IT teams to better maintain consistent service levels, reduce costs and errors, and adjust more quickly to new requirements. These types of solutions can also help reduce the cost and complexity of the enterprise systems management software environment by enabling IT teams to consolidate the number of tools and reduce the number of different products that need to be licensed and maintained. Reliance on a single pane of glass to design and deploy services, monitor capacity and performance, control updates and migrations, and recover unused resources as needed can help make IT staff more efficient. Self-service functionality can help IT deliver resources to end users more quickly and consistently. Overall, this type of comprehensive orchestration solution promises to reduce configuration and deployment problems caused by human error, speed up resource and application delivery timelines, and better align the utilization of resources with real-time and rapidly changing business requirements. It also helps reduce business risk and unplanned downtime by creating more standardized and compliant environments where changes can be fully documented and tracked using policy-based systems to enforce business requirements. Considering IBM SmartCloud Orchestrator IBM SmartCloud Orchestrator is designed to address many of the emerging virtual and cloud data center management requirements discussed previously. Specifically, IBM aims to help IT organizations design and implement a full life cycle of automated pattern-based provisioning, configuration management, and policy-based optimization across heterogeneous infrastructure, middleware, and application resources including initial application test, release, and runtime operation. To do so, SmartCloud Orchestrator uses a distributed, loosely coupled plug-in based architecture that segregates service management activities from specific infrastructure dependencies and allows IT to rely on a unified interface and automation to: Build new cloud services in minutes by combining pattern-based cloud delivery, with a graphical orchestrator for simple composition of cloud automation Reduce operational costs by automating cloud application deployment and life-cycle management across compute, storage, and network management tasks including integration with third-party tools 4 2013 IDC
Simplify end-user consumption of cloud services via support for a self-service portal Accelerate the delivery of heterogeneous cloud environments via support for a broad range of hypervisors and cloud standards, including OpenStack Enable deployments on public clouds such as Amazon Elastic Compute Cloud Measure the cost of cloud services with metering and chargeback capabilities Keep track of cloud service health and perform capacity management of cloud resources As Figure 2 shows, resource-specific plug-ins provide integrations between the underlying resources and the orchestration engine. The orchestration layer provides graphical, object-oriented service modeling tools; discovers available resources, topologies, and dependencies; maintains best practice service patterns and policies; monitors resource utilization; and executes workflows to deploy, configure, and maintain services as needed, depending on resource availability and service policies. It also provides IT staff with a single interface to monitor and control service performance and the configuration and behavior of the underlying infrastructure resources, including public cloud as well as in-house resources. F i g u r e 2 IBM SmartCloud Orchestrator Architecture Source: IBM, 2013 SmartCloud Orchestrator is designed to scale over time as customers look to place additional services and resources under its control. In addition to standard patterns provided out of the box, IBM provides a graphical, object-based pattern design capability to support custom pattern developments 2013 IDC 5
and updates. Role-based service catalogs are targeted at the needs of different IT roles to support self-service and routine automation, deployment, and migration activities across complex tier 3 applications as well as infrastructure resource specific tasks. SmartCloud Orchestrator is part of an IBM solution set that allows progressive adoption of cloud from entry capabilities to complete orchestration based on a common architecture to enable a smooth transition. SmartCloud Orchestrator leverages IBM investments in developing best practice pattern libraries, image management technologies, and the IBM Business Process Manager orchestration engine. The majority of currently available plug-ins are based on IBM-developed vendor-specific integrations with a wide array of systems, storage, and network hardware as well as management software tools and multiple hypervisors. Over time, IBM expects to rely primarily on OpenStack APIs to enable plug-in integrations. IBM is also contributing to the TOSCA and OSLC standard initiatives to enable more flexible and effective topology analysis, pattern design, and workload monitoring using open standards. SmartCloud Orchestrator comes pre-integrated with IBM monitoring, usage, capacity, and accounting analytics and reporting capabilities to further streamline day-to-day cloud and virtual data center management and chargeback. Challenges Many organizations are willing to experiment with pockets of automation and orchestration but resist committing to broad-based use of patterns and automated policy enforcement. Despite the fact that many IT organizations have years of experience using scripting and automation server and infrastructure provisioning tools, broad deployment of integrated automated orchestration across multi-tier architectures and the full DevOps life cycle is still fairly uncommon. This is due to a number of challenges, including organization, security, compliance, and governance considerations. Extensive use of pattern-based automation requires IT teams, spanning multiple technology domains, to agree on best practices and to support consistent use of pre-defined standards, policies, and priorities. Not all IT decision makers are convinced that extensive use of orchestration will deliver on the promises of reduced costs and downtime and the ability to provide more scalable and reliable services. Rather, they fear that standardized, automated environments will not be able to adequately serve the needs of mission-critical business applications. Conclusion Sophisticated automation and orchestration solutions will be critical to the effective management of large-scale, dynamic virtual and cloud data centers going forward. Without this type of capability, IT teams will struggle to keep up with rapidly changing business needs and will find it difficult to control costs or meet service-level agreements. With SmartCloud Orchestrator, IBM has the opportunity to help many customers rapidly implement more scalable and cost-effective data center management solutions across diverse, heterogeneous application and infrastructure life cycles. A B O U T T H I S P U B L I C A T I ON This publication was produced by IDC Go-to-Market Services. The opinion, analysis, and research results presented herein are drawn from more detailed research and analysis independently conducted and published by IDC, unless specific vendor sponsorship is noted. IDC Go-to-Market Services makes IDC content available in a wide range of formats for distribution by various companies. A license to distribute IDC content does not imply endorsement of or opinion about the licensee. C O P Y R I G H T A N D R E S T R I C T I O N S Any IDC information or reference to IDC that is to be used in advertising, press releases, or promotional materials requires prior written approval from IDC. For permission requests, contact the GMS information line at 508-988-7610 or gms@idc.com. Translation and/or localization of this document requires an additional license from IDC. For more information on IDC, visit www.idc.com. For more information on IDC GMS, visit www.idc.com/gms. Global Headquarters: 5 Speen Street Framingham, MA 01701 USA P.508.872.8200 F.508.935.4015 www.idc.com 6 2013 IDC