Technical Report Test Report: Digital Rapids Transcode Manager Application with NetApp Media Content Management Solution Jim Laing, NetApp July 2012 TR-4084
TABLE OF CONTENTS 1 Executive Summary... 3 2 Introduction... 3 3 NetApp Media Content Management Solution... 3 3.1 Optimized for Consolidation...3 4 Digital Rapids Transcoding Workflow... 3 4.1 Software Components...3 5 Test Environment... 4 5.1 Test Equipment: Hardware Environment...4 5.2 Test Equipment: Software Environment...4 5.3 File System Configuration...5 5.4 Transcode Manager (Media Manager) and Agent/Engine Configuration...6 6 Test Results... 7 6.1 Test 1: Transcode 12 Uncompressed Files to AVC-HD Format...8 6.2 Test 2: 12 Simultaneous Encodes from Uncompressed HD to MPEG-2... 10 6.3 Test 3: Transcode from Uncompressed Format to WMV Files... 12 7 Summary... 14 References... 14 LIST OF FIGURES Figure 1) Transcode Manager overview....4 Figure 2) Test setup layout....5 Figure 3) Logical view of volume groups and volumes....6 Figure 4) Transcode Manager showing three servers acting as transcode agents....7 Figure 5) Digital Rapids Media Manager Console: 12 engines on 3 agent servers (resources in a single group)....7 Figure 6) Digital Rapids Media Manager Console job view during simple 12-file transcode job....7 Figure 7) Test 1: graphical view of throughput (MB/sec) and I/O operations per second (IOPS) from SANtricity ES....9 Figure 8) Test 1: table view of performance statistics from SANtricity ES....9 Figure 9) Test 1: Digital Rapids Media Manager showing jobs complete.... 10 Figure 10) Test 2: graph showing throughput and IOPS from SANtricity ES.... 11 Figure 11) Test 2: table view from SANtricity ES.... 11 Figure 12) Test 2: Digital Rapids Media Manager, showing job completion statistics.... 12 Figure 13) Test 2: Digital Rapids Media Manager, showing resource allocation.... 12 Figure 14) Test 3: graphical view showing throughput and IOPS from SANtricity ES.... 13 Figure 15) Test 3: table view from SANtricity ES.... 13 Figure 16) Test 3: Digital Rapids Media Manager showing job completion statistics.... 14 2 Test Report: Digital Rapids Transcode Manager Application with NetApp Media Content Management Solution
1 Executive Summary The purpose of this document is to highlight the results of testing the Digital Rapids Transcode Manager application with the NetApp Media Content Management (MCM) Solution, based on NetApp E-Series storage and the Quantum StorNext file system. The document describes the lab testing environment, including hardware and software setup. An overview of the tests performed is included to show that this application ran successfully on the MCM solution. System sizing can be extrapolated from these test results. 2 Introduction Transcoding is the data conversion of a media file from one digital format to another. Often, a media file (video, audio, film transfer, and so on) is created or edited in one format but is needed in one or many other formats for further production or distribution, such as for broadcast, Web, mobile platforms, television, or digital cinema. Transcoding is an important stage in many media workflows, often taking place during ingest, after postproduction or prior to program playout or delivery. Although transcoding is thought of as a CPUintensive activity, the transcoding of many large files simultaneously also demands a storage system with high-throughput (bandwidth) capabilities, making the NetApp MCM Solution an ideal infrastructure for transcoding. Digital Rapids offers hardware and software solutions that include ingest, encoding, and transcoding workflows. Digital Rapids is a well-established supplier of such solutions within the media industry. 3 NetApp Media Content Management Solution NetApp has created the MCM solution to address the specific bandwidth and capacity requirements for efficient media workflows and huge digital libraries. This solution helps media organizations improve the production and delivery of television, movie, social media, interactive game, music, and photography content. The solution can be configured for all sizes of media operations, ranging from corporate media and local broadcast news operations to 4K film postproduction and large cable and Internet delivery services. The solution is architected with modular components that can be configured to meet the specific requirements across each step of your operations. By deploying components optimized for each stage of the media workflow, you get a solution that is tailored for your unique needs and can scale in nondisruptive granular increments to meet your growing needs. 3.1 Optimized for Consolidation The NetApp E-Series optimizes storage density to consolidate all six stages of the media workflow (ingest, manage, produce, process, deliver, and archive) into a single, centralized repository. By consolidating siloed storage, you benefit from an infrastructure that delivers maximum storage capacity with excellent performance. Using SAS disk options, the MCM solution delivers superior bandwidth cost compared to solutions using SATA drives. 4 Digital Rapids Transcoding Workflow 4.1 Software Components Digital Rapids uses the following components as part of its transcode workflow: Transcode Manager, which controls and manages transcoding functions; it is usually a dedicated server 3 Test Report: Digital Rapids Transcode Manager Application with NetApp Media Content Management Solution
Media Manager Console, the user interface for Transcode Manager Watch folders, into which files are placed for transcoding Agents, which run on each hardware server Engines, which are units of transcode function; one or more engines run on each agent server Jobs, which are specifications for the type of transcode operation and its technical parameters Figure 1 shows a typical Digital Rapids configuration. Figure 1) Transcode Manager overview. 5 Test Environment NetApp has set up a lab testing environment in which to test the Digital Rapids transcode workflow. This section details the components and configuration of this testing environment. 5.1 Test Equipment: Hardware Environment NetApp E-Series 5460 with 60 2TB drives 4 Fujitsu Primergy Rx300 S6 servers, 8 cores at 2.4GHz (16 virtual), 48GB memory 2 Brocade 300 Fibre Channel switches Cisco Gigabit Ethernet switch Quantum M330 metadata controller appliance 5.2 Test Equipment: Software Environment Windows Server 2008 R2 Digital Rapids Stream 3.7.2.25 install on all servers Digital Rapids Transcode Manager 1.6.1 (build 16 rev 62898) installed on one server QuickTime (for its codecs including Apple ProRes) installed on all servers Calibrated{Q} XD decode (for Sony XD codecs) A variety of video source files, uncompressed and with various codecs NetApp SANtricity ES 10.80.G4.51 used to configure and manage the E-Series storage Quantum StorNext file system, 4.2.0 4 Test Report: Digital Rapids Transcode Manager Application with NetApp Media Content Management Solution
Figure 2) Test setup layout. 5.3 File System Configuration The E-Series storage system was set up according to NetApp best practices as documented in NVA- 0001: NetApp Media Content Management Solution. A brief overview is presented here. The E5460 used for testing has 60 2TB drives, which are configured as six 8+2 RAID 6 arrays (volume groups). One volume is created for each volume group, as shown in Figure 3. The preferred controller assignment has been set appropriately so as to share the I/O workload evenly across controllers A and B. 5 Test Report: Digital Rapids Transcode Manager Application with NetApp Media Content Management Solution
Figure 3) Logical view of volume groups and volumes. The StorNext file system has been configured according to NetApp best practices. The six volumes created on the E5460 each include a stripe group with a breadth of 1MB (128kB x 8). The result is a single mount point storage system presented to each Windows host server as a drive letter. 5.4 Transcode Manager (Media Manager) and Agent/Engine Configuration One server acted as the Digital Rapids Transcode Manager; the other three servers acted as agents with multiple engines. The licensing setup was for 12 engines: each of the 3 agent servers was set up with 4 engines. Media Manager Console was then configured with a single group containing all these engine resources. Video files for testing were placed in a subfolder on the file system. Several Digital Rapids Stream projects were created, each representing a sample workflow (transcodes to a specific codec or codecs with various settings). Various watch folders were then created. All project files, source files, watch folders, and output files were stored in locations on the E-Series/StorNext file system shared storage. 6 Test Report: Digital Rapids Transcode Manager Application with NetApp Media Content Management Solution
The following figures show the overall configuration of the test environment. Details of specific sample test runs follow. Figure 4) Transcode Manager showing three servers acting as transcode agents. Figure 5) Digital Rapids Media Manager Console: 12 engines on 3 agent servers (resources in a single group). Figure 6) Digital Rapids Media Manager Console job view during simple 12-file transcode job. Source (test) video files were placed into the watch folders to initiate transcode jobs. Job completion was monitored to verify that all transcodes were processed to successful completion. The E-Series storage was monitored using SANtricity ES to determine that the file system was active and that no errors occurred. The performance characteristics of several such tests are shown later in this report. 6 Test Results Installation and configuration of the Digital Rapids application went smoothly and as documented. All mount points were set to the single-namespace StorNext mount, which for Windows Server appears as 7 Test Report: Digital Rapids Transcode Manager Application with NetApp Media Content Management Solution
a single drive letter. The Digital Rapids application worked smoothly throughout the test. Because transcoding is a CPU-intensive activity, the servers are the bottleneck for these tests. For this reason, performance levels achieved in these tests were well under the documented maximum throughput potential of E-Series with StorNext. General performance testing done by the NetApp Solution Integration Group showed the E5460 capable of at least 2.8GB/sec sustained I/O with a mixed read/write workload for up to several hundred threads. A watch folder workflow was used whereby input (source) files were placed into previously configured watch folders to be transcoded. The Digital Rapids Stream application was used to create a variety of transcode jobs to run by means of the Transcode Manager, using various codecs and settings. Some jobs were simple processes taking input files and transcoding to a single video format. Others involved transcoding each input file through several codecs to several output video formats. The application behaved as expected, starting up jobs so that the various input files were simultaneously transcoded. All jobs completed without error, writing transcoded files to the appropriate directory in the file system. Although the tests were not designed primarily to maximize throughput and performance, some performance data was collected during several test runs. The details of the jobs run and data collected are listed later in this document. In all cases, all 12 engines were kept busy by jobs that involved 12 simultaneous transcode operations. For these tests, the source (input) files were 12 different uncompressed video files with the following characteristics. The parameters were chosen to create files that would result in high storage read activity. Transcodes to various video formats were run. Input files were: Uncompressed HD video, 1920x1080, created in Final Cut Pro QuickTime MOV format 32 bits per pixel (24 bits per pixel for color information, and 8 bpp for alpha/mask channel) Millions of color+ format Two channels of 48kHz audio, uncompressed 6 minutes 33 seconds duration File size 91.1GB, or 97,850,020,572 bytes Bandwidth required to play this video in real time: 248MB/sec QuickTime inspector shows audio at 1.53Mb/sec and video at 1.99Gb/sec The tests documented in the following subsections were done using these input files. Examples of some of the tests are described and shown in the following sections. Because the test environment included only four servers (one as manager and three for transcode engines), the tests are limited by the CPU resources in the test environment. 6.1 Test 1: Transcode 12 Uncompressed Files to AVC-HD Format This test was designed to run a simple job, maximizing reads from storage while using the Transcode Manager to initiate 12 simultaneous encodes. This test was designed to minimize CPU utilization so as to maximize overall I/O activity given the limited number of engines (and servers) in the test environment. Hence, the input files were as previously described, and the output files were very small AVC files (192x108 at 256kb/sec). Results are shown in Figure 7, Figure 8, and Figure 9. 8 Test Report: Digital Rapids Transcode Manager Application with NetApp Media Content Management Solution
Figure 7) Test 1: graphical view of throughput (MB/sec) and I/O operations per second (IOPS) from SANtricity ES. Figure 8) Test 1: table view of performance statistics from SANtricity ES. 9 Test Report: Digital Rapids Transcode Manager Application with NetApp Media Content Management Solution
Figure 9) Test 1: Digital Rapids Media Manager showing jobs complete. As can be seen in the preceding three figures, test 1 achieved approximately 1.8GB/sec and about 4,000 IOPS and took about 10 minutes to transcode the 12 6.5-minute HD files. 6.2 Test 2: 12 Simultaneous Encodes from Uncompressed HD to MPEG-2 Unlike test 1, this test was designed to perform a transcode job that utilized the CPUs of the engines more heavily and wrote larger output files, so that the ratio of reads to writes was different than in test 1. This test was designed to result in large read activity and moderate write activity. Hence, the input files were as previously described, and the output files were MPEG-2 files at 1,920x1,080 resolution at a bit rate of 17MB/sec. As can be seen in Figure 10 through Figure 13, test 2 achieved approximately 1.7GB/sec and about 4,000 IOPS and took just over 10 minutes to transcode the 12 6.5-minute uncompressed HD input files. 10 Test Report: Digital Rapids Transcode Manager Application with NetApp Media Content Management Solution
Figure 10) Test 2: graph showing throughput and IOPS from SANtricity ES. Figure 11) Test 2: table view from SANtricity ES. 11 Test Report: Digital Rapids Transcode Manager Application with NetApp Media Content Management Solution
Figure 12) Test 2: Digital Rapids Media Manager, showing job completion statistics. Figure 13) Test 2: Digital Rapids Media Manager, showing resource allocation. 6.3 Test 3: Transcode from Uncompressed Format to WMV Files In test 3 the transcode was from the uncompressed format described earlier (input files) to AVI files, which were also uncompressed. The goal of this test was to show how the test environment behaved when significant read and write activity was occurring. To achieve this, the input files were unchanged from earlier tests, while the output files were larger uncompressed AVI files at 1,280x720 resolution, resulting in a bit rate of about 880Mb/sec for the transcoded (output) files. As can be seen in Figure 14 through Figure 16, test 3 achieved approximately 2.1GB/sec and about 10,000 IOPS and took about 13 minutes to transcode the 12 6.5-minute uncompressed HD files. The higher IOPS value is likely due to the much larger output file sizes than in the previous two tests. The results of this test are what would be expected for an optimal Digital Rapids installation. 12 Test Report: Digital Rapids Transcode Manager Application with NetApp Media Content Management Solution
Figure 14) Test 3: graphical view showing throughput and IOPS from SANtricity ES. Figure 15) Test 3: table view from SANtricity ES. 13 Test Report: Digital Rapids Transcode Manager Application with NetApp Media Content Management Solution
Figure 16) Test 3: Digital Rapids Media Manager showing job completion statistics. 7 Summary The Digital Rapids application ran well with the NetApp MCM Solution based on NetApp E-Series with Quantum StorNext. The shared, single mount point made creation of Digital Rapids Stream jobs easy, and configuration of inputs, outputs, and watch folders worked as it should. With a single drive letter representing a very high-capacity and high-performing storage system, it was easy to use Digital Rapids Transcode Manager to manage files and projects. Given the number of CPUs in the test environment, the E-Series storage performed very well and achieved the high throughput expected with this solution. These test results confirm the sizing data and assumptions made in the MCM sizing guide for video read/write workloads. Therefore, prospective workload sizing can be extrapolated using the results shown in the previous sections. Detailed sizing information for the MCM solution is available in TR-3952: NetApp Media Content Management Sizing Guide. The NetApp Media Content Management solution is scalable according to your media workflow needs. It enables more efficient automated workflows because file transfers between storage silos are eliminated or at least greatly reduced. Bandwidth to support concurrent ancillary processes, such as quality checking and file delivery, can be easily provisioned. References The following references were used in this technical report: NetApp Media Content Management Solution NVA-0001: NetApp Media Content Management Solution NetApp Media Content Management Sizing Guide TR-3952: NetApp Media Content Management Sizing Guide 14 Test Report: Digital Rapids Transcode Manager Application with NetApp Media Content Management Solution
Refer to the Interoperability Matrix Tool (IMT) on the NetApp Support site to validate that the exact product and feature versions described in this document are supported for your specific environment. The NetApp IMT defines the product components and versions that can be used to construct configurations that are supported by NetApp. Specific results depend on each customer's installation in accordance with published specifications. NetApp provides no representations or warranties regarding the accuracy, reliability, or serviceability of any information or recommendations provided in this publication, or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS, and the use of this information or the implementation of any recommendations or techniques herein is a customer s responsibility and depends on the customer s ability to evaluate and integrate them into the customer s operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document. 2012 NetApp, Inc. All rights reserved. No portions of this document may be reproduced without prior written consent of NetApp, Inc. Specifications are subject to change without notice. NetApp, the NetApp logo, Go further, faster, and SANtricity are trademarks or registered trademarks of NetApp, Inc. in the United States and/or other countries. Apple, Final Cut Pro, and QuickTime are registered trademarks of Apple, Inc. Windows and Windows Server are registered trademarks of Microsoft Corporation. All other 15 Test Report: Digital Rapids Transcode Manager Application with NetApp Media Content Management Solution brands or products are trademarks or registered trademarks of their respective holders and should be treated as such. TR-4084-0712