3 Convergence of broadcasting and telecommunications

Transcription

1 We continued researching and developing technologies and services that will lead to the convergence of broadcasting and telecommunications, targeting new broadcasting services for the era in which broadcasting and telecommunications will be tightly integrated. In our research on Hybridcast, we studied recording and playback functionality, support for non-broadcast-oriented managed applications that will enable service providers other than broadcasters to provide content services, and technologies for highly precise synchronization between broadcasting program and broadband content. We contributed to the establishment of Hybridcast technical specifications ver. 2.0 at the IPTV Forum. We also investigated new multiplexing schemes and coding technologies for closed captions and superimposition based on the Hybridcast technical specifications that will enable provision of sophisticated multimedia services for Super Hi-Vision (SHV) broadcasting. We contributed to the establishment of the ARIB STD-B62 standard specifying multimedia broadcasting for SHV. For device linkage services, we prototyped the TV note service as a non-broadcast-oriented managed application for Hybridcast. The service enables viewers to record subjects of interest by simply tapping on a tablet device screen and to interact with external Internet services. We also developed the Augmented TV service, which offers a new viewing experience: capturing images on a TV screen by using the camera in a mobile device and then displaying them with the related virtual objects on the device. We developed a technology for estimating the position and rotation of the mobile device relative to the TV screen and conducted implementation evaluations. In our research on networks for broadcasting programs, we developed technologies for utilizing program-related for Internet services and for analyzing data collected by social networking services (SNSs). We prototyped a program data hub that provides related to broadcaster programs in the Linked Open Data (LOD) format and studied ways to analyze and utilize various types of external big data such as data collected by SNSs. In our research on program viewing analysis, we developed technologies for determining the viewing status and estimating the viewer s interests. We prototyped a system for estimating keywords representing viewer interests on the basis of their TV viewing behavior and for displaying the keywords on a tablet device. We also developed ways to generate user profiles based on the estimated keywords and recommend some related programs to viewers. Regarding security technologies for more secure and reliable services, we prototyped a searchable encryption system that can search encrypted data on the cloud without decrypting them. We also developed a watermarking algorithm for embedding a receiver s identity into compressed data streams as a way of preventing illegal distribution of content. For IP content delivery, we began developing a platform that can deliver video efficiently to diverse viewing terminals including Hybridcast receiver-equipped TVs and mobile devices. We also made progress in our research on a distributed server-organized broadcasting system for easy viewing of past programs. We prototyped and evaluated a time-shifting system that enables zapping by tags such as cast member name and on-air time. 3.1 Hybridcast We are continuing our research on Hybridcast, which is a service platform for flexibly providing extensible new services that take advantage of the characteristics of broadcasting and telecommunications networks. In FY 2014, we made progress on R&D aimed at making the platform more efficient and on implementing multimedia broadcasting for Super Hi-Vision, including 8K Hybridcast (SHV multimedia broadcasting). We also continued our standardization activities. To further promote the Hybridcast service we launched in September 2013, we showed the appeal of Hybridcast by demonstrating it to visitors at the NHK STRL Open House and other exhibitions at home and abroad. such as support for video on demand (VOD) and recorder functionality, development of non-broadcast-oriented managed applications, and high-precision synchronization of broadcasting Advanced Hybridcast The IPTV Forum published the first edition of the Hybridcast Technical Specifications in In FY 2014, we studied elemental technologies for advanced services, conducted experimental verifications, and promoted the establishment of the second edition of the specifications. Version 2.0 of the specifications includes additional features Figure 1. Example program guide that enables direct transition to VOD NHK STRL ANNUAL REPORT

2 and broadband content. At the STRL Open House, we exhibited our prototype receiver in cooperation with receiver manufacturers. The support for VOD and recorder functionality includes VOD playback from an application, recording and playback of broadcast programs, and launch of cooperative applications during viewing of recorded programs. We fabricated a receiver to verify these functions (1) and prototyped and exhibited a program guide application that enables direct transition to VOD (Figure 1). We are also studying MPEG-DASH (Dynamic Adaptive Streaming over HTTP) as a VOD content delivery scheme. We conducted technical verifications by creating test streams and reproducing them on the TVs of various manufacturers, aiming at inclusion of MPEG-DASH in the Hybridcast operation rules and application for actual services. Non-broadcast-oriented managed applications are provided by service providers other than broadcasters and are available regardless of broadcast channel. They enable new services and business models. We studied the management of program-related data distributed to service providers (2) (Figure 2) and the push transmission system. We also prototyped a receiver model (3) and applications and examined an overall system model for using the services (4). An example of our prototype non-broadcast-oriented managed applications is shown in Figure 3. This application, which receives and displays in real time the data distributed by push technology, provides viewers useful and timely such as train operation status and weather reports. High-precision synchronization is the display of content delivered by broadcasting and broadband networks in a synchronized manner in video frame units using time. This technology is indispensible for multi-angle services, which enable viewing of images from different angles. In FY 2014, we developed a basic technology for reproducing video on mobile devices such as tablets and smartphones in synchronization Figure 2. Prototype management system for distributing program-related data with the main broadcasting program displayed on the TV screen (5). We will continue to test the feasibility of this technology, which is applicable to various purposes including synchronization with data. Promotion of Hybridcast We are providing technical support for NHK s program production departments to enable wider use of the program-linked Hybridcast service, which was launched in December We provided prototype applications for the Hybridcast service for additional programs that started in September 2014, thereby contributing to its smooth launch. We also exhibited examples of Hybridcast services offered by commercial broadcasters at the STRL Open House and demonstrated the growing popularity of Hybridcast. Effort for SHV multimedia broadcasting Using the Hybridcast technologies, we prototyped sample 8K Hybridcast services that take advantage of large, ultra-high-resolution screens, aiming at the realization of SHV multimedia broadcasting (Figure 4). Displayed on the screen along with the golf program are multiple images provided by broadcasting and broadband networks, multilingual closed-captions, posts on social networking services (SNSs), and various other types of. We exhibited this example at the STRL Open House. Activities related to standardization Domestic standardization activities included promoting the establishment of Technical Specifications ver. 2.0 (6) at the IPTV Forum. We also contributed to the establishment of HTML5 Browser Specifications ver. 2.1 and the Hybridcast Operation Rules ver. 2.0 for a new VOD system supporting HTML5. We worked on standardization of technologies related to SHV multimedia broadcasting in cooperation with relevant organizations. We examined such technologies as MPEG Media Transport (MMT)-based transmission schemes, browser functionality and data coding, and a Timed Text Markup Language (TTML)-based coding method for closed captions and character superimposition (7). Our efforts led to the establishment of the ARIB STD-B62 standard (8) in July 2014, which specifies SHV multimedia broadcasting covering all of these technologies. Regarding closed-captioning and character superimposition technologies, we participated in discussions on exchange formats and operation rules, which are indispensable for facility preparation, and created test data required for receiver development. International standardization activities included incorporating Hybridcast into a draft recommendation for hybrid systems at the International Telecommunication Union, Radiocommunication Sector (ITU-R) and Telecommunication Standardization Figure 3. Example non-broadcast-oriented managed application (useful and timely ) Figure 4. Example 8K Hybridcast service 22 NHK STRL ANNUAL REPORT 2014

3 Sector (ITU-T). Moreover, Technical Specifications ver. 2.0 and ARIB STD-B62 were reflected in a report on hybrid systems. At the Technical Plenary/Advisory Committee (TPAC) Meeting of the World Wide Web Consortium (W3C) working on HTML5 standardization, we presented an overview of the functions specified in Technical Specifications ver. 2.0 to promote understanding and diffusion of Hybridcast worldwide. (1) H. Furumiya, M. Takechi, H. Ohmata, K. Matsumura, S. Sunasaki and K. Kai: Support for VOD and recorder function on Hybridcast, ITE Annual Conference, 24-5 (2014) (in Japanese) (2) Y. Hironaka, K. Majima, K. Kai and S. Sunasaki: An Application Distribution Management System for Hybridcast, IEEE ICCE 2015 (2015) (3) H. Endo, Y. Hasegawa, H. Ohmata, A. Baba, K. Matsumura and K. Kai: Implementation Study of a Hybridcast Receiver for Non-broadcast-oriented Managed Application, ITE Annual Conference, 24-2 (2014) (in Japanese) (4) H. Ohmata, H. Endo, Y. Hasegawa, A. Baba, K. Matsumura and K. Kai: System Architecture for Cross-Channel Application Services on Hybridcast, IEEE ICCE 2015 (2015) (5) M. Onishi, H. Ohmata and K. Matsumura: A Method for Accurate Synchronization between Broadcast and Broadband on Wi-Fi Connected Multi-screen Devices, ITE Annual Conference, 24-7 (2014) (in Japanese) (6) IPTV Forum Japan: Integrated Broadcast-Broadband System Specification Version 2.0, IPTVFJ STD-0010 (2014) (in Japanese) (7) A. Baba, S. Takeuchi, M. Takechi and K. Kai: A Study of Schemes for Multimedia Services on Super Hi-Vision Satellite Broadcasting System, ITE Technical Report, Vol. 38, No. 35, pp (2014) (in Japanese) (8) ARIB: Multimedia Coding Specification for Digital Broadcasting (2nd Generation), ARIB STD-B62, Ver. 1.0 (2014) 3.2 Device linkage services For more sophisticated services that take advantage of broadcasting and telecommunications, we are researching ways to make use of user data as well as device-linked system technology. Technology for utilization of user data To promote non-broadcast-oriented managed applications for Hybridcast, we are developing the TV note system (1), which supports viewers in their search, purchase, and social networking service (SNS) activities on the Internet by efficiently collecting on their interests from broadcast programs and commercials. This system records the viewer s interests, which the viewer indicates by simply tapping on the screen of a tablet device linked with a TV on which various subjects of interest are displayed. It then stores the recorded as authenticated viewer interest data. It can then guide the viewer to Internet services matching his/her interests after the program. We developed a method that makes it easier to collect from the Internet by linking metadata provided by external providers with the subjects of viewer interest. We developed a prototype system as a non-broadcast-oriented managed application that can be shared among multiple broadcasting channels and exhibited it at the STRL Open House (Figure 1). Device-linked system technology We are proceeding with R&D on a device linkage system that will provide a new viewing experience ( Augmented TV ). It will capture images displayed on a TV screen with the camera in a mobile device such as a smartphone or tablet and then display them on the device. This system will enable broadcast content to be shown on devices other than receiver equipped TVs, with 3DCGs and other telecommunications content overlaid on the captured images of the TV screen on the mobile device. We developed a method for quickly estimating the position and rotation of the mobile device relative to the TV screen (2). Figure 1. TV note system Tap on tablet device screen Pointer to indicate subject of interest Hybridcast TV The display location of the telecommunications content can be easily adjusted by using the gyro sensor of the mobile device and a simple frame that shows the area of the TV screen. Implementation evaluations using commercial tablet devices showed that capturing the TV screen just once to estimate the position and rotation is sufficient to maintain the mobile device s position relative to the TV screen and that the device can keep displaying telecommunications content naturally and continuously even when capturing areas off the TV screen. These results confirmed the effectiveness of this method. (1) C. Yamamura, H. Ohmata and T. Nakagawa: TV note program viewing annotation and its utilization with Hybridcast -, ITE Annual Conference, 24-6 (2014) (in Japanese) (2) H. Kawakita, T. Nakagawa and M. Sato: Estimation of TV Screen Position and Rotation using Camera and Gyro Sensor, ITE Annual Conference, 13-1 (2014) (in Japanese) 3.3 Content utilization and program viewing analysis To provide more attractive services that exploit the advantages of broadcasting and telecommunications, we researched and developed technologies for making use of program-related for Internet services, analyzing data collected by NHK STRL ANNUAL REPORT

4 social networking services (SNSs), and monitoring viewer viewing behaviors and estimating their interests. Information network for broadcasting programs To explore the possibility of broadcaster Internet services we are studying ways to enable related to broadcast programs to be used for various services provided by broadcasters and service providers. We prototyped a program data hub that structures broadcaster program into computer-readable Linked Open Data (LOD) format and provides it through application programming interfaces (APIs). We designed a database structure for describing and sharing program guide and related on the basis of program schedule and implemented automatic linkage with the vocabulary of the Japanese version of DBpedia, which is the Internet encyclopedia Wikipedia structured in LOD format. This has enabled the data hub to be connected with external open data (1). We also examined individual data structures and APIs for each program genre, assuming specific ways program-related can be used for various Internet services. For example, we linked health-related programs with a concept map and content map, which present data for program-related knowledge, that we have been researching. This has enabled advanced search using knowledge data. For education-related programs, we studied a method for systematizing science video content by linking it with structures extracted from Wikipedia (2). This program data hub can facilitate service cooperation with broadcasters and third-party providers who wish to use program-related. For example, a provider offering cooking and living can enrich its services by utilizing from broadcast health programs. Diverse new Internet services making use of program can be expected in various areas. We also studied ways of analyzing and utilizing a wide variety of big data besides those of broadcasters such as SNS big data and external open data. From the analysis of tweets on Twitter at the time of the Great East Japan Earthquake, we found that messages during a large-scale disaster contain a significantly higher frequency of use of proper nouns such as names of places than in ordinary times. This makes it impossible to use feature extraction by using such indicators as word significance, which is a common technique for text summarization. We therefore developed a message categorization method that is effective for particular situations like an earthquake (3). This method should be helpful for appropriate and effective message extraction for regional disaster prevention and disaster reporting. Determination and analysis of viewing behaviors We are researching technologies for automatically determining and analyzing viewing status and for estimating viewer interests, with a view to developing broadcasting and life services matching viewer preferences and viewing styles. Using the analysis results of TV viewing experiments we conducted in FY 2013, we developed a system for estimating keywords related to viewer interests from their TV viewing status and for displaying the keywords on a tablet device (4). The system, which was exhibited at the STRL Open House in 2014 (Figure 2), collects keywords from the program guide and subtitle in advance and calculates the significance of each keyword on the basis of appearance frequency and other indicators. During program viewing, the system calculates the degree of viewer interest from his/her viewing status, that is, gaze (head pose) (5) and facial expression. The system then estimates candidate keywords related to the viewer s interests from the time average of the product of the degree of interest and the significance of keywords appearing in the subtitles and displays the keywords in rank order on the tablet in real time. The results of verification using the viewing experiment data showed that the higher ranked keywords were consistent with the keywords the test participants had listed in the questionnaire, demonstrating the effectiveness of this method. This technology was used for application development by participants at the NHK HACKATHON event held in May (6). For determination of viewing behaviors, research on gaze behavior estimation and on facial expression recognition was conducted in cooperation with Tokyo City University and the University of Tokyo, respectively. We also developed a method for generating a user profile based on program viewing behavior. The proposed method categorizes the keywords of interest stored during TV viewing and generates a user profile automatically. More specifically, it Program description schema DBpedia Program ontology Program schedule, etc. Existing program DB Link to program sites and to moving and still images Related-content Program data hub Linked Open Data (LOD) format data Cross-sectional use of data External service applications Broadcaster service applications Concept map Content map External LOD data External knowledge Figure 1. Conceptual diagram of program data hub 24 NHK STRL ANNUAL REPORT 2014

5 reflects the weight of the category of the interest keyword on a tree structure graph built from the categories registered in Wikipedia. We collected viewing behavior data from ten test participants in general households for two weeks and generated their user profiles with the proposed method. Comparison of the profiles with the in the questionnaire they answered showed that the category distribution generated automatically was consistent with their genres of interest. We also found that using interest keywords for each participant increased the accuracy of estimating the programs that are likely to arouse the participant s interest. Determined viewing status Interest keywords displayed on tablet device Figure 2. Example of determining viewing status and displaying interest keywords (1) M. Miyazaki, M. Urakawa, I. Yamada, K. Miura, H. Sumiyoshi, H. Fujisawa and T. Nakagawa: Study on Program Information Database as Linked Open Data, The Japanese Society of Artificial Intelligence SIG-SWO-A , pp (2014) (in Japanese) (2) M. Urakawa, M. Miyazaki, I. Yamada, T. Nakagawa and H. Fujisawa: A study on systematical view of video content based on biological ontology from Wikipedia, The 77th National Convention of IPSJ (2015) (in Japanese) (3) K. Ariyasu and H. Fujisawa: SNS message analysis for regional prevention at a large-scale disaster, ITE Winter Annual Conference, 3-2 (2014) (in Japanese) (4) Y. Yamanouchi, M. Okuda, M. Takahashi, S. Clippingdale and M. Naemura: A Prototype System for Estimating TV Viewers Interests Based on Attentional Behavior, ITE Annual Conference, 7-6 (2014) (in Japanese) (5) M. Takahashi, S. Clippingdale, M. Naemura and M. Shibata: Estimation of Viewers ratings of TV programs based on behaviors in home environments, Multimedia Tools and Applications, DOI / s (2014) (6) Security technologies To support the security and reliability of the convergence of broadcasting and telecommunications, we are researching cryptography and watermarking technologies. Broadcast station Provider A Part of Part of Part of Provider B Cloud server TV program Encrypted viewer Recommended Recommended tion Recommended ation Viewer Figure 1. Provision of personalized services that match viewers preferences Cryptography and authentication algorithm for hybrid services We researched cryptography technologies that can be used to provide secure and reliable hybrid services that take advantage of broadcasting and telecommunications. Service providers need about viewers in order to personalize services while viewers expect amazing services from providers that they have not previously accessed. An efficient way to meet these requirements is to store viewer in the cloud and allow providers to freely access the (Figure 1). From the perspective of preserving viewers privacy, however, it is best to keep the accessible to the providers to the minimum necessary. We therefore studied a system that will allow providers to view only the minimum necessary data. The system stores encrypted (1), searches encrypted data in the cloud without decrypting them, and provides the provider with only the data matching the search condition. We verified the basic operation and effectiveness of the system using a prototype. Watermarking algorithm for HEVC-compressed streams For copyright protection of content, we researched a watermarking technology that can embed to identify a receiver into streams compressed with a new coding scheme, High Efficiency Video Coding (HEVC). An effective way to protect content from illegal distribution is to identify who is distributing content illegally. At the same time, various measures must be taken for HEVC-compressed content. We therefore developed a watermarking method in which micro auxiliary sent from content providers and the receiver s identity are embedded into HEVC-compressed streams within the receiver (2). (1) G. Ohtake and K. Ogawa: Privacy Preserving System for Integrated Broadcast-broadband Services, IEEE ICCE 2015, pp (2015) (2) K. Ogawa and G. Ohtake: Watermarking for HEVC/H.265 Stream, IEEE ICCE 2015, pp (2015) NHK STRL ANNUAL REPORT

6 3 Convergence of broadcasting and telecommunications 3.5 IP content delivery technology With the aim of distributing programs over a network, we are researching a video delivery technology that uses the Internet and a distributed server-organized broadcasting system to provide broadcast programs over a network. Video delivery technology using the Internet To support large-scale delivery of live video, we began our development of a delivery platform that can support diverse viewing terminals including Hybridcast receiver-equipped TVs and mobile devices. We released a prototype player that runs on an HTML5 browser and verified its operation on TV receivers, mobile devices, and PCs. These results contributed to the establishment of relevant technical specifications(1) at the IPTV Forum Japan. We also conducted delivery experiments with test participants in various regions throughout Japan by using a player capable of measuring the video delivery quality. We collected basic data for use in further stabilizing large-scale delivery. We also studied a way of using a distributed processing technology for generating streams for Internet video delivery. We examined the processing performance of a method for generating video streams from an original video file by using distributed processing technology that can process a large amount of data at high speed using multiple servers. We obtained data on the relationship between the number of servers used for stream generation and the processing speed as well as on the extensibility of input/output video formats(2). IP transmission equipment We developed stored video transmission equipment that can extract an image captured at a certain time specified by the broadcaster from accumulated images recorded at reporting sites and transmit it over an IP network. This equipment divides time-continuous video data into multiple files with a length of a few seconds before storing them. It can also transmit multiple files simultaneously so that stable transmission can be maintained even if the bandwidth of the IP network fluctuates. We also developed underwater wireless IP transmission equipment using blue LEDs for stably transmitting live video captured by an underwater camera without using a cable. Distributed server-based broadcasting system We are developing a time-shift viewing system that will enable program zapping by tags such as cast member name and on-air time so that users can view any of the large number of past programs. To develop this system, we researched a distributed server-organized broadcasting system for accumulating, reprocessing, and distributing broadcast content using multiple servers on the network. We improved tag search functionality and the interfaces of the time-shift viewing system we prototyped in FY 2013 and conducted system evaluation experiments. The speed of tag search was increased by sorting tag data by on-air time at the time of registration into the database and by creating a list of tags for the same time in advance(3). We also created a keyword dictionary focusing on personal names from the past four years 26 NHK STRL ANNUAL REPORT 2014 Figure 1. Time-shift viewing system for tag-based program zapping worth of program guide and subtitle. Using the dictionary, we built a database containing over ten million records of program-related tags. For the interface, we upgraded the system to enable easy zapping with a remote-control device (Figure 1). We conducted evaluation experiments of the upgraded system and verified its operability and zapping functionality(4). Through the experiments, we found that tag zapping gives viewers a chance to find a program they had never viewed and helps them view it. We also developed a system for visualizing viewing logs for use in analyzing viewing behavior. The system can calculate the amount of viewing for each specified tag in video frame units from the collected viewing logs. We developed a viewing system based on cloud computing that runs the program viewing player for the time-shift viewing system on a server instead of the viewing terminal, so it does not depend on the processing performance of the viewing terminal. Only the reproduction of video and sound is handled on the viewing terminal; the other processes requiring more CPU power are run on a cloud server. This enables the usage of various types of program viewing software even on a low-end terminal. (1) IPTV Forum Japan: HTML5 Browser Specification Version 2.1, IPTVFJ STD-0011 (2014) (in Japanese) (2) S. Oda, M. Kurozumi and Y. Endo: Verification of ultra-high definition video coding on distributed processing framework, 77th National Convention of IPSJ, 2E-02 (2015) (in Japanese) (3) Y. Kaneko, S. Takeuchi, M. Fan and M. Naemura: A Consideration of Tag Retrieval Including On-air Time Range Field, FIT 2014, No. 2, pp (2014) (in Japanese) (4) S. Takeuchi, K. Hiramatsu, Y. Kaneko and M. Naemura: An Experiment of Time-shift Viewing System with Zapping along the On-air Time Axis and Tag, Proc. of the IEICE General Conference, D-9-29 (2015) (in Japanese)