A Robust Cloud-based Service Architecture for Multimedia Streaming Using Hadoop

Transcription

1 A Robust Cloud-based Service Architecture for Multimedia Streaming Using Hadoop Myoungjin Kim 1, Seungho Han 1, Jongjin Jung 3, Hanku Lee 1,2,*, Okkyung Choi 2 1 Department of Internet and Multimedia Engineering, Konkuk University, 1 Hwayang-dong, Gwangjin-gu, Seoul , Republic of Korea {touhg105, shhan87, hlee}@konkuk.ac.kr 2 Center for Social Media Cloud Computing, Konkuk University, 1 Hwayang-dong, Gwangjin-gu, Seoul , Republic of Korea hlee@konkuk.ac.kr, okwow2@gmail.com 3 Digital Media Research Center, Korea Electronics Technology Institute, Electronics Center #1599, Sangam-dong, Seoul , Republic of Korea mozzalt@keti.re.kr Abstract. Delivering scalable rich multimedia applications and services on the Internet requires sophisticated technologies for transcoding, distributing, and streaming content. Although cloud computing provides an infrastructure for such technologies, the specific challenges of task management, load balancing, and fault tolerance remain. To address these issues, we propose a cloud-based distributed multimedia streaming service, or CloudDMSS. The system is designed to run on all major cloud computing services, and is highly adapted to the structure and policies of Hadoop, which give it additional capabilities for transcoding, task distribution, load balancing, content replication and distribution. Keywords: Streaming Service, Mobile Media Service, Cloud Computing, Media Transcoding 1 Introduction With the recent proliferation of rich social media across a variety of personal devices, considerable attention has shifted to the challenge of adaptively distributing and streaming multimedia content over the Internet. Among the technologies that have emerged, cloud-based media streaming, transcoding, and distributed storage have been the most noteworthy and influential. The reason why cloud-based technologies have emerged in this regard can be discerned from the features of recent multimedia services: media heterogeneity, Quality of Service (QoS) heterogeneity, network heterogeneity, and device heterogeneity [1]. To support such features, the streaming, transcoding, and distribution of media must depend on massive and massively scalable computational resources, i.e., CPUs, memory, network bandwidth, and storage. * Corresponding Author 348

2 Though cloud computing can provide these resources, in doing so, it also introduce a heavy burden on existing Internet infrastructure and cloud resources, and it introduces a host of new challenges (e.g., cluster rebalancing, namespace management, data distribution/replication, auto-recovery, and fault tolerance), all of which are intensified under the massive swings in traffic associated with rich media streaming. These challenges have proven difficult for developers and service vendors alike, and continue to trouble current media delivery systems. To address these challenges, we herein propose a cloud-based distributed multimedia streaming service (CloudDMSS) system designed to run on current cloud computing infrastructure. CloudDMSS capabilities include (1) Transcoding of large amounts of media into the MPEG-4 video format for delivery to a variety of devices, including PCs, smart pads, and phones (2) Exponential reduction in transcoding time through incorporation of the Hadoop file system (HDFS) for storage of multimedia data and MapReduce for distributed parallel processing (3) Reduction in content delays and traffic bottlenecks using streaming job distribution algorithms (4) Improvement in overall performance using dual-hadoop clustering per physical cluster (5) Efficient content distribution and improved scalability through adherence to Hadoop policies The remainder of this paper is organized as follows: section 2 discusses relevant research on cloud-based streaming services; section 3 describes the core architecture of CloudDMSS with respect to transcoding, job distribution, content replication and distribution, etc.; section 4 presents our prototype of the proposed system and its configuration; and section 5 offers concluding remarks and plans for future work. 2 Related Work In recent years, many researchers have applied cloud computing technologies to rich media services, in response to the explosion of demand for such services. This section presents the research most relevant to our CloudDMSS system. Hui et al. in [5] proposed MediaCloud, a layered architecture that defines a new paradigm for dealing with multimedia applications and services. The architecture comprises three layers a Media Service Layer, a Media Overlay Layer, and a Resource Management Layer and addresses such key challenges as heterogeneity, scalability, and QoS provisioning. However, this architecture is treated mainly at the conceptual level, leaving most of the challenges of real-world implementation to future work [5]. In contrast, Luo et al. addressed the implementation challenge of QoS provision over virtualized infrastructure by presenting a practical architecture and mechanism for a private media cloud [3]. They describe their system in terms of four major components: monitoring, load balancing, traffic management, and security. 349

3 In regard to cloud-based streaming, Lee et al. in [4] have proposed a configuration scheme for connectivity-aware P2P networks with algorithms for connectivity-aware mobile P2P network configuration and connectivity-aware P2P network reconfiguration. In [6], Chang et al. described a cloud-based media streaming architecture that dynamically adjusts streaming services in response to mobile device resources, multimedia codec features, and network environment. They also presented a design for the stream dispatcher component, including real-time adaptation of codecs in response to client device profiling, and a dynamic adjustment of multimedia streaming (DAMS) algorithm. In [9], Huang et al. presented CloudStream, a cloudbased video proxy capable of delivering high-quality video streams by transcoding the original video in real time to a scalable codec, which in turn allows adaptation of the stream to various network dynamics. They also proposed a multi-level transcoding parallelization framework with two mapping options: hallsh-based mapping and lateness-first mapping. 3 Proposed System Architecture Fig.1 shows an overview of the CloudDMSS architecture, highlighting three main modules: the Hadoop-based distributed multimedia transcoding module (HadoopDMT), and the Hadoop-based distributed multimedia streaming module (HadoopDMS), and the cloud multimedia management module (CMM). The HadoopDMT transcodes a variety of multimedia data into MPEG4, a standard format in which media can be streamed and played on a variety of devices. Quality and speed are improved by adopting the Hadoop distributed file system (HDFS) [2] for storing video data from many sources, MapReduce [2] for distributed parallel processing of this data, and Xuggler [7] for transcoding the data. Once transcoded, media contents are automatically moved and stored in HDFS of the HadoopDMS module. The contents are split into blocks of configurable size and distributed across the system. When a block is distributed, it is also replicated at three data nodes managed by NameNode, which constructs a directory tree of all transcoded contents according to the Hadoop distribution policy. This virtually guarantees content availability even under system or node failure. Furthermore, by conforming to the Hadoop policy, HadoopDMS automatically benefits from Hadoop s distributed processing capabilities, as well as its facilities for data replication, file splitting and merging, load balancing, and fault tolerance. The role of the CMM module is to manage jobs such as streaming and transcoding tasks, and to balance the load on streaming servers with media stream scheduling. The module s streaming job distribution algorithm is called streaming resource-based connection (SRC). SRC optimally distributes streaming jobs among streaming servers in HadoopDMS based on CPU usage rates and the currently streaming traffic. 350

4 Fig. 1. Architectural overview of the CloudDMSS system 4 Implementation and Prototype For our prototype implementation of CloudDMSS, we constructed our own cloud computing servers comprising 28 nodes in total. Each node consisted of Linux OS (Ubuntu LTS) running on two Intel Xeon quad-core 2.13 GHz processors with 4 GB registered ECC DDR memory and 1 TB SATA-2 disk storage. All nodes were interconnected through 100 Mbps Ethernet adapters. To implement the CMM module, one node was designated as our management server, running Tomcat, and a second node was designated as the content DB server, running MySQL. The HadoopDMS module was composed of 1 NameNode and 12 DataNodes running on HDFS. The HadoopDMS module consisted of 3 streaming servers based on NginX and 10 content storage servers running on HDFS. We used a dual Hadoop cluster in one physical cluster to distribute load between transcoding and streaming tasks. Our software specification included Java 1.6.0_39 (64-bit), Hadoop-1.0.4, Xuggler (64- bit) for video transcoding, H.264 streaming module-2.2.7, and fuse_dfs_

5 (a) (b) Fig. 2. (a) Web-based dashboard for streaming transcoded contents (b) Web-based dashboard for transcoding tasks and resources The output from the prototype system is provided in Fig. 2. Fig. 2(a) shows a selection of streamed content offered through a web-based dashboard, running commodity PC hardware. Fig. 2(b) shows a screenshot of the web page for managing transcoding tasks. Using this page, users and administrators can upload original content, select transcoding options (e.g., resolution, format, and codec) and stream the content to other users. Users can also monitor MapReduce-based transcoding processes and the remaining HDFS storage capacity. 352

6 5 Conclusion and Future Works In this paper, we proposed the CloudDMSS system for efficient cloud-based streaming of rich social media. Our system addresses a number of pressing issues related to distributed media streaming, including transcoding for heterogeneous devices, job distribution, and content replication/distribution under HDFS. Our current plan is to implement a fully functional CloudDMSS system, and to conduct thorough quantitative performance analysis of the system on a variety of cloud computing infrastructures, including Amazon EC2 and Rackspace Compute. Acknowledgements. This research was supported by the MSIP (Ministry of Science, ICT & Future Planning) of Korea under the C-ITRC (Convergence Information Technology Research Center) support program (NIPA-2013-H ) supervised by the NIPA (National IT Industry Promotion Agency). References 1. Zue, W., Luo, C., Wang, J., Li, S.: Multimedia Cloud Computing. IEEE Signal Processing Magazine 28, (2011) 2. Dean, J., Ghemawat, S.: MapReduce: Simplified Data Processing on Large Clusters. Communication of the ACM 51, (2008) 3. Luo, H., Egbert, A., Stahlhut, T.: QoS Architecture for Cloud-based Media Computing. In: IEEE 3 rd International Conference on Software Engineering and Service Science, pp IEEE press, Beijing (2012) 4. Lee, H.S., Lim K.H., Kim, S.J.: A Configuration Scheme for Connectivity-aware Mobile P2P Networks for Efficient Mobile Cloud-based Video Streaming Services. Cluster Computing (2013) 5. Hui, W., Lin, C., Yang, Y.: MediaCloud: A New Paradigm of Multimedia Computing. KSII Transactions on Internet and Information Systems 6, (2012) 6. Chang, S.Y., Lai, C.F., Huang Y.M.: Dynamic Adjustable Multimedia Streaming Service Architecture over Cloud Computing. Computer Communications 35, (2012) 7. Xuggler Java Library, 353