White Paper. FOX Audience Network Transition from NetApp to Oracle ZFS Storage Appliance Case Study

Transcription

1 89 Fifth Avenue, 7th Floor New York, NY White Paper FOX Audience Network Transition from NetApp to Oracle ZFS Storage Appliance Case Study March 2011

2 Printed in the United States of America. Copyright 2011 Edison Group, Inc. New York. Edison Group offers no warranty either expressed or implied on the information contained herein and shall be held harmless for errors resulting from its use. All products are trademarks of their respective owners. First Publication: January 2011, March 2011 Produced by: Barry Cohen, Senior Analyst; Craig Norris, Editor

3 Table of Contents Executive Summary... 1 Introduction... 1 Objective... 2 Audience... 2 Background - The Challenge... 2 Company FOX Audience Network... 2 Problem Inadequate Performance and High Operating Costs... 3 Solution - Sun ZFS Storage Appliance... 4 Conclusions... 6

4 Executive Summary FOX Audience Network (FAN), the fifth largest provider of online advertising services, requires high-performance storage systems to support their service. The storage system they were using, a NetApp FAS6070 Filer, could not keep up with their performance demands. FAN had to spend a lot of time carefully tuning and redesigning their application, with NetApp support, to obtain even marginal performance improvements. The tuning efforts were hampered by limitations in the system monitoring capabilities of the NetApp filer, to such an extent that FAN had all but given up on getting the performance they needed. In addition, in an attempt to achieve better performance, FAN had to acquire many more drives for performance reasons than they would otherwise require. This resulted in an implementation that utilized six racks in their data center. Maintaining this large installation resulted in very high annual maintenance fees from NetApp. To address this problem, FAN tested an Oracle Sun ZFS Storage 7140 appliance, which met all their performance requirements with capacity and performance to spare. Not only did the Sun ZFS Storage Appliance achieve FAN's performance goals, it also reduced the datacenter footprint from six racks to one, which resulted in lower annual maintenance fees. Most importantly, the integrated Sun ZFS Storage Appliance software management console provides FAN with critically important operational transparency, allowing both developers and storage administrators to quickly identify and address any application performance issues that might arise as the applications in use evolve. Study Page 1

5 Introduction Objective This case study describes the issues FAN was facing with the NetApp FAS6070 Filer, how these issues were being addressed (whether successfully or not), and how the switch to the Sun 0 ZFS 7140 appliance enabled FAN to resolve the issues and move forward. Audience The audience for this case study is anyone concerned about how both storage system architecture and manageability can affect application performance and how choosing the right storage platform can address those concerns. Background - The Challenge Company FOX Audience Network The FOX Audience Network (FAN), a unit of News Corporation, supports monetization efforts across the company's online content portfolio, as well as third-party publisher sites. FAN leverages proprietary advertising technology to create highly targeted advertising campaigns for a wide range of marketers. It also delivers cutting-edge tools and services to third-party publisher partners. FAN works directly with hundreds of advertisers to develop customized marketing programs that optimize both branded and performancebased strategies. The company also developed and manages a self-serve advertising platform, which is currently utilized by more than 30,000 advertisers. In all, FAN reaches over 150 million consumers each month and has built relationships with more than 600 online publishers. Study Page 2

6 Problem Inadequate Performance and High Operating Costs The large-scale volume and performance demands of the services FAN offers its advertisers and publishers (over seven billion ads per day) was causing operational performance issues on the advertising servers. The existing NetApp FAS6070 storage system had been identified as the main cause of these performance issues. The monitoring tools available with the NetApp FAS6070 delivered "poor visibility" into the operations of the system. From the perspective of a developer or storage administrator, monitoring the arrays was like looking at a "black box." In order to address the storage-related performance issues, FAN storage managers and developers needed to work together with NetApp technical support in order to identify specific factors causing performance problems. Identifying performance issues required downloading various logs from the filer and sending them to NetApp for analysis. NetApp would reply after the analysis, which could take anywhere from a few hours to a day or more, with information and recommendations related to the causes of the issue. FAN developers or storage administrators would then modify filer configuration settings or application code based on these recommendations. Performance would once again be monitored. If performance targets were not reached, additional logs would have to be sent to NetApp, repeating the cycle to reach current performance requirements. This was also an ongoing process as the application evolved to meet the needs of FAN s customers. 1 It was a never-ending cycle of "Dump, Send, and Redo." Among the recommendations from NetApp, technical support suggested the standard approach of adding more and more drives or spindles to the array in order to improve performance by greater parallelism. This resulted in a storage system consisting of six racks of drives and controllers. A result of this approach was higher cost for electrical power, floor space, and licensing and maintenance fees. This relatively larger and more complex hardware configuration also caused greater management complexity. In addition, the slow iterative approach to technical support meant that application tuning that should have taken hours or days, took days or weeks. For an evolving set of applications, this approach was untenable, since each change restarted the cycle. 1 FAN did not criticize the quality of NetApp technical support, which was considered excellent. The issues were with the lack of system state transparency and the overly complex methodology required for addressing performance issues. Study Page 3

7 In summary, FAN s NetApp FAS6070 filer was causing application performance problems. Addressing these problems resulted in very high costs for storage hardware, software, and system and software maintenance, as well as data center space. In addition, the applications for which the storage was used required constant redesign and tuning, using a methodology that was slow and dependent upon NetApp for resolution. Study Page 4

8 Solution Oracle s Sun ZFS Storage Appliance FOX Audience Network was already a Sun storage customer, so their Oracle Sun account team recommended testing a Sun 7140 ZFS Appliance. These tests were so successful that the NetApp FAS6070 was taken offline and replaced by the ZFS Appliance. FAN s testing revealed that not only could performance requirements be met, but also there was performance and storage capacity to spare. It is important to note that this performance gain was obtained by switching from six racks of storage with several hundred drives to a single rack with five shelves of twelve drives each, plus solid-state storage devices. With the Sun ZFS Storage appliance, performance averaged a bit less than 3,000 NFS IOPS, peaking as high as 175,000 IOPS. Network throughput averaged 60 Mb/second, peaking at 894 MB/second. This performance was far better than what had previously been experienced. 2 Figure 1: Six NetApp Filer Cabinets - One Oracle Sun ZFS Storage appliance Cabinet (For illustration only. This is not a photograph of an actual installation.) 2 In fairness to NetApp, the replacement ZFS Appliance did use hybrid storage pools, including solid-state storage devices that greatly improve performance, while movable disk storage alone had been in use on the FAS6070. NetApp also offers solid-state storage, which would undoubtedly have improved the performance of that platform. Adding solid-state storage on a NetApp filer not only adds the costs for the hardware, it also adds more licensing and maintenance costs to the system. Study Page 5

9 It is obvious that a storage system with five shelves of drives, even with solid-state devices, will cost much less to acquire than a system with six racks of disks. In addition to the purchase price, annual maintenance fees are a significant cost component over time. According to FAN, the difference in maintenance fees charged by NetApp for the six racks of equipment and required software, compared with those for the new Sun ZFS Storage 7140 appliance, was enough to cover the cost of the new storage system. This difference alone could justify the acquisition. More than purchase price and support fees are involved in acquiring a storage system. FAN s management experience using the NetApp filer, especially in the area of monitoring and troubleshooting, was one of the key drivers for looking at an alternative storage platform. The need to send data to NetApp for analysis so that performancetuning recommendations could be obtained, and then repeating that process each time a change was made to either the applications or the filer, was more than onerous. It was ultimately unsupportable. The integrated ZFS Storage Appliance software for managing the ZFS Storage Appliance provided FAN with "Operational Transparency" using the analytics tools provided by the DTrace Analytics feature of the Solaris operating system. Using these analytic tools, storage administrators and developers could monitor live performance data across every aspect of the storage system, drilling down to the processes accessing the storage on the server side as well as the interfaces, caches, and drives on the storage side. The ability to monitor live data has allowed FAN to tune their software interactively, with no delay for outside analysis. The savings in administrator and developer time have been huge, making what had been a near impossible task on the NetApp filer an activity readily undertaken. The performance gains FAN has experienced have had additional cost-saving ramifications. The original FAS6070 essentially supported 250 to 300 servers, accessing the same data on the storage system: text files that contain the content used in the banner ads that FAN hosts. With the new system, there is more than enough surplus performance and space capacity to support additional existing or new applications, all with a much smaller physical footprint. Study Page 6

10 Conclusions Faced with performance issues affecting their ability to provide services to their customers, FOX Audience Network replaced an underperforming and expensive to maintain NetApp FAS6070 storage system with Oracle s Sun ZFS Storage 7140 appliance. The Oracle product provided superior operational transparency into the state of the system, greatly simplifying monitoring and tuning for both developers and storage administrators. A single, partially filled rack containing the Sun ZFS Storage 7140 appliance, as well as both spinning and solid state drives, replaced the six racks of drives and controllers required with the NetApp FAS6070, resulting in considerable savings in floor space, power utilization, and cabling. Additionally, the hardware maintenance fee savings were equivalent to the cost of the new storage array, making the acquisition almost CAPEX-free. Replacing the NetApp Data ONTAP, plus several additional management and monitoring software packages, with the integrated Sun ZFS Storage Appliance software delivered the operational transparency needed for tuning storage to meet the needs of applications as they evolved. The more efficient user interfaces provided by the ZFS Storage Appliance system served to lower administration costs still further. Study Page 7