ARC BRIEF JANUARY 2012 Application Downtime, Your Productivity Killer By John Blanchard and Greg Gorbach Overview Today, manufacturing enterprises are faced with intense competitive pressure, limited IT resources, and increasing IT costs. They increasingly More and more manufacturers are recognize that modern automation technology and services, properly selected and deployed, can help them deploying new uptime assurance solutions that combine hardware, improve productivity, better ensure product quality, software, and services designed to and improve margins. They are automating and integrating most every aspect of manufacturing operations. prevent computer outages. This includes moving from dedicated to consolidated computer resources, which is critical to controlling IT hardware, software, staffing, and facilities cost to reduce total cost of ownership (TCO). Consolidation also means that even a minor computer outage can have severe and immediate consequences. As a result, more and more manufacturers are deploying new uptime assurance solutions that combine hardware, software, and services designed to prevent computer outages. Trends in Plant Application Adoption In today s increasingly competitive global manufacturing environment, manufacturing enterprises are deploying an increasing number of missioncritical applications in production operations to improve manufacturing efficiency and effectiveness. This is true across every industry and every geographic region of the world, but particularly so in both the fast and slow moving consumer goods manufacturing industries, as well as many consumer facing industries. These applications include materials management, finite scheduling, quality and performance monitoring and reporting, plant historians, asset lifecycle management, facilities management, procurement, order management, other ERP functions, as well as computer aided manufacturing (CAM) and process planning. Some industry trends also help explain why we expect requirements for uptime assurance to continue to increase into the foreseeable future. Many VISION, EXPERIENCE, ANSWERS FOR INDUSTRY
ARC Brief, Page 2 companies face a lack of available manpower with the proper skill sets. In fact, about 5 percent of skill jobs in the US remain unfilled. Over 50 percent of manufacturers surveyed by ARC further indicated that they felt this shortage of skilled workers would continue to grow worse. Companies are also finding ways to do more with less. They are automating more and more traditionally semi-manual and manual operations. Even the practice of standardizing on hardware from a single vendor to reduce Traditional manufacturing organizational structures, costs is being challenged, as companies best practices, and automation architectures are being look to reduce total cost of ownership transformed to meet the needs of the new real-time, (TCO) and use technology to gain performance improvements. on-demand and highly competitive global manufacturing environment. The traditional siloed organizational structure is inefficient. The days of having a department head be responsible for determining best practices, standard operating procedures, and automation technology within his or her own department can no longer meet today s business requirements. It s now necessary to consolidate these systems and applications. Even the practice of standardizing on hardware from a single vendor to reduce costs is being challenged, as companies look to reduce total cost of ownership (TCO) and use technology to gain performance improvements. Why is Application Uptime Important? As companies consolidate disparate departmentalized systems, they have less tolerance for downtime. In addition, manufacturers are using an increasing number of applications to help minimize production costs and product variability on a weekly and even daily basis. Many of these applications work with real-time data, rather than analyzing historic data offline. With fewer production facilities, many operations run around the clock. This has produced a proliferation of productivity-enhancing applications in manufacturing, which are often woven together into an integrated, mutually dependent solution. Production machinery such as CNC machines, robotic packaging machines, and other processes are also increasingly becoming mission critical - where traditional process controls are no longer sufficient to meet uptime needs. Even slight interruptions to any of these applications and equipment controls can have severe and immediate consequences. ARC research shows that direct downtime costs vary from several thousand to tens of thousands USD per hour, dependent
ARC Brief, Page 3 upon the site, product, etc. This does not even take into account lost revenues from customers who decide to take their business elsewhere, never to return. Production Management Software is Mission Critical at KapStone Paper Loss of MES Applications Can Shut Down Operations at Kraft Paper Mill KapStone Paper and Packaging provides a good example of the need for uptime assurance and the cost of unscheduled downtime caused by the lack of availability of its mission-critical production management software. The company produces unbleached kraft paper that goes into the packages, wraps, and surfaces that are a part of everyday life. Emerson Beach, Kapstone s Director of IT Operations, explained, If the MES system stops, we can make reels of paper coming off the paper machines for a little while. But we can t cut the rolls to match our orders because the size of the pieces comes from the production management system. So we only have a little time before we would have to shut down the paper machines. Since the company operates 24/7, that translates into about $33,000 per hour per machine. Loss or Lack of Data Can Result in Unsalable Product and High Inventory Costs Loss of data can also have considerable financial impact. Information about the components or ingredients that went into the manufacture of the product, as well as information about the actual manufacturing of the product, is often required to be able to sell the product. Loss of information on a single batch of product in the pharmaceutical industry can translate into millions of dollars in raw material costs and lost revenue. In this case uptime assurance requires not just a fault-tolerant server, but also fault-tolerant communications and a fault-tolerant data storage system. Many companies also lack visibility to important information due to the many disparate legacy systems that are remnants of traditional organizational structures and practices. For example, one key to lowering the cost of goods sold is to lower inventory carrying costs. However, few companies have accurate information on raw materials, in-process, and finished goods inventories. Consolidation into mutually interdependent integrated applications brings greater data visibility and greater need for uptime assurance.
ARC Brief, Page 4 Evaluating Infrastructure Options Using TCO As legacy mission critical hardware and software reach the end of their lifecycles, manufacturers must evaluate technology options to replace them. Most legacy production-level systems rely upon a single high-reliability server and backup data storage. IT departments commonly use cluster computing and virtual environments for their business systems. The financial services and banking industries typically use high-availability software and physically remote, fault-tolerant servers with dedicated high-speed Ethernet connections. This architecture ensures uninterrupted continuation of services in case of a disaster. Specially designed fault-tolerant computers and subsystems, with closely coupled fully redundant components sourced from a single supplier, provide the next highest availability option. The optimum uptime assurance solution is integrated hardware, software, and services designed to prevent failure, and provided by a single supplier. Hot Backup or Clustering Solutions vs. Fault-Tolerant Servers While traditional clustering or hot back up approaches to uptime provide enhanced availability, they also pose significant challenges. In contrast to true hardware fault tolerance, which prevents server failure in a manner that is both automatic and transparent to the user, hot back up and cluster solutions rely on complex "failure recovery" mechanisms that incur varying degrees of downtime. During these failover/recovery periods (that can range from a few seconds to minutes) a backup system automatically restarts the applications and logs on users. Performance and throughput may be compromised and all in-memory content is lost. Built from combinations of conventional servers, software and enabling technologies, clusters require failover scripting and testing that must be repeated whenever changes are made to the environment. Clustering may also require licensing and installing multiple copies of software as well as software upgrades and application modifications. Hot backup solutions and clusters typically provide 99.9 percent availability or more. Actual availability levels depend on the environment and the sophistication and skill set of the team responsible for setup, administration, and management. In contrast, fault-tolerant servers inherently offer 99.999 percent and greater availability and are as easy to install, use, and maintain as a single conventional system.
ARC Brief, Page 5 Fault-Tolerant Virtualization Many manufacturing applications are now certified for virtualization. Virtualization has become an important technology in efforts to reduce automation and plant IT costs. It increases server utilization, enables server consolidation, simplifies the migration of legacy applications to higher performance servers, and enables better disaster recovery. It provides an environment that is much more efficient for IT staff to manage. However, placing more applications on a single server increases the impact of a server failure. This creates even greater business risk if these are mission critical applications. With fault-tolerant servers, the risk of virtualizing mission critical applications is eliminated. Determining Total Cost of Ownership Each manufacturing operation has its own unique requirements in terms of uptime assurance, acceptable degree of business risk, available IT skill sets, etc. Since this is an investment over the lifecycle of the technology, it should be evaluated based upon TCO and any anticipated changes in business requirements and risk, available IT skill sets, etc. TCO is the sum of the installed cost plus the net present value of the recurring costs and the Given today s intensely competitive cost of retirement. What is the full cost of downtime? Will there be sufficient IT skill sets in the manufacturing environment, limited resources, and increasing business risks, coming years or will much of it be outsourced? many manufacturers are selecting single What support services does the technology supplier supplier-sourced, fault-tolerant provide? What is the cost/value of this service? computing platforms and services for Given today s intensely competitive manufacturing their plant applications. environment, limited resources, and increasing business risks, many manufacturers are selecting single-supplier sourced, fault-tolerant computing platforms and services. As we move more and more toward supplier-agnostic automation and plant IT architectures, smart technology, and increased use of Internet services, companies should continuously reevaluate their IT best practices. Conclusion To improve efficiency and effectiveness in today s intensely competitive global environment, manufacturing is becoming more and more automated. As a result, manufacturers have increasingly less tolerance for downtime or data loss. Uptime assurance has become critical to the success of business operations. The need for greater uptime assurance continues to
ARC Brief, Page 6 grow and it will be a critical component to the future success of manufacturing enterprises. This paper was written by ARC Advisory Group on behalf of Stratus Technologies. The opinions and observations stated are those of ARC Advisory Group. For further information or to provide feedback on this paper, please contact the authors at ggorbach@arcweb.com or jblanchard@arcweb.com. ARC Briefs are published and copyrighted by ARC Advisory Group. The information is proprietary to ARC and no part of it may be reproduced without prior permission from ARC Advisory Group.
ARC Brief, Page 7 John Blanchard: John is part of the manufacturing automation consulting group at ARC covering the food & beverage, life sciences, fine chemicals, and CPG industries. He concentrates on batch process automation, governmental regulations such as US FDA 21 CFR Part 11, industry automation requirements, and evolving issues and technologies affecting these industries. John has over 25 years of experience as both a user and a supplier in the food, beverage, and pharmaceutical industries as a manufacturing engineer, project engineer, project manager, industry marketing manager, and automation consultant. Greg Gorbach: As ARC s Service Director for Collaborative Manufacturing and Architecture, Greg Gorbach is a thought leader in collaborative manufacturing and provides clients in a number of manufacturing vertical markets with strategic advice in dealing with boundarycrossing business processes. Greg s primary areas of focus are collaborative manufacturing, production management, business process management, manufacturing performance services, and the synchronization of plant systems with CRM, ERP, PLM, supply chain and other business systems. He brings over twenty years of hands-on experience to ARC, with direct experience within manufacturing organizations, as well as extensive experience with suppliers to manufacturers.