Overview Video Indexer overview Concepts Quickstarts Get started How to Use Video Indexer API Examine Video Indexer output Find exact moments within

Transcription

1 Table of Contents Overview Video Indexer overview Concepts Quickstarts Get started How to Use Video Indexer API Examine Video Indexer output Find exact moments within videos View and edit Video Indexer insights Create video insights from existing videos Embed widgets into your application Reference REST Resources Azure Roadmap

2 Video Indexer (preview) 1/9/ min to read Edit Online Video Indexer is a cloud service that enables you to extract the following insights from your videos using artificial intelligence technologies: Audio Transcription: Video Indexer has speech-to-text functionality, which enables customers to get a transcript of the spoken words. Supported languages include English, Spanish, French, German, Italian, Chinese (Simplified), Portuguese (Brazilian), Japanese and Russian (with many more to come in the future). Face tracking and identification: Face technologies enable detection of faces in a video. The detected faces are matched against a celebrity database to evaluate which celebrities are present in the video. Customers can also label faces that do not match a celebrity. Video Indexer builds a face model based on those labels and can recognize those faces in videos submitted in the future. Speaker indexing: Video Indexer has the ability to map and understand which speaker spoke which words and when. Visual text recognition: With this technology, Video Indexer service extracts text that is displayed in the videos. Voice activity detection: This enables Video Indexer to separate background noise and voice activity. Scene detection: Video Indexer has the ability to perform visual analysis on the video to determine when a scene changes in a video. Keyframe extraction: Video Indexer automatically detects keyframes in a video. Sentiment analysis: Video Indexer performs sentiment analysis on the text extracted using speech-to-text and optical character recognition, and provide that information in the form of positive, negative of neutral sentiments, along with timecodes. Translation: Video Indexer has the ability to translate the audio transcript from one language to another. The following languages are supported: English, Spanish, French, German, Italian, Chinese-Simplified, Portuguese- Brazilian, Japanese, and Russian. Once translated, the user can even get captioning in the video player in other languages. Visual content moderation: This technology enables detection of adult and/or racy material present in the video and can be used for content filtering. Keywords extraction: Video Indexer extracts keywords based on the transcript of the spoken words and text recognized by visual text recognizer. Annotation: Video Indexer annotates the video based on a pre-defined model of 2000 objects. Once Video Indexer is done processing and analyzing, you can review, curate, and publish the video insights. Whether your role is a content manager or a developer, the Video Indexer service is able to address your needs. Content managers can use the Video Indexer web portal to consume the service without writing a single line of code, see Get started using the Video Indexer portal. Developers can take advantage of APIs to process content at scale, see Use Video Indexer REST API. The service also enables customers to use widgets to publish video streams and extracted insights in their own applications, see Embed visual widgets in your application. You can sign up for the service using existing AAD, LinkedIn, Facebook, Google, or MSA account. For more information, see the getting started topic. Scenarios Below are a few scenarios where Video Indexer can be very useful

3 Search Insights extracted from the video can be used to enhance the search experience across a video library. For example, indexing spoken words and faces can enable the search experience of finding moments in a video where a particular person spoke certain words or when two people were seen together. Search based on such insights from videos is applicable to news agencies, educational institutes, broadcasters, entertainment content owners, enterprise LOB apps and in general to any industry that has a video library that users need to search against. Monetization Video Indexer can help improve the value of videos. As an example, industries that rely on ad revenue (for example, news media, social media, etc.), can deliver more relevant ads by using the extracted insights as additional signals to the ad server (presenting a sports shoe ad is more relevant in the middle of a football match vs. a swimming competition). User engagement Video insights can be used to improve user engagement by positioning the relevant video moments to users. As an example, consider an educational video that explains spheres for the first 30 minutes and pyramids in the next 30 minutes. A student reading about pyramids would benefit more if the video is positioned starting from the 30-minute marker. For more information, see this blog. Next steps You are ready to get started with Video Indexer. For more information, see the following articles: Get started with the Video Indexer portal Process content with Video Indexer REST API Embed visual widgets in your application

4 Video Indexer concepts 1/9/ min to read Edit Online This topic describes some concepts of the Video Indexer service. Breakdowns The breakdowns element has the full list and details of everything. For example, it has a list of blocks where each block has its transcript lines, OCR lines, faces. In addition, transcript lines, OCR lines, faces have their timestamps and other details. This is where the full transcript comes from. However, the breakdown can be too detailed and too long for most users, in which case you would look at Summarized insights. Summarized insights Summarized insights contain an aggregated view of the data: faces, keywords, sentiments. For example, instead of going over each of the thousands of time ranges and checking which faces are in it, the summarized insights contains all the faces and for each one, the time ranges it appears in and the % of the time it is shown. Topics/keywords Topics/keywords are in the list of key phrases that Video Indexer extracts from the text. For example, a Scott Guthrie video might contain the following topics/keywords: Security, Azure, Microsoft Cloud, Revenue. Sentiments When Video Indexer analyzes transcripts, it detects sentiments as well. For example, "this is a very exciting event" is a positive sentiment. time range vs. adjusted time range TimeRange is the time range in the original video. AdjustedTimeRange is the time range relative to the current playlist. Since you can create a playlist from different lines of different videos, you can take a 1-hour video and use just 1 line from it, for example, 10:00-10:15. In that case, you will have a playlist with 1 line, where the time range is 10:00-10:15 but the adjustedtimerange is 00:00-00:15. Blocks Blocks are meant to make it easier to go through the data. For example, block might be broken down based on when speakers change or there is a long pause. Next steps For information about how to get started, see How to sign up and upload your first video. See also Video Indexer overview

5 How to sign up and upload your first video 7/20/ min to read Edit Online This short getting started tutorial shows how to sign in to Visual Indexer and how to upload your first video. To read a detailed overview of the Video Indexer service, see the overview article. Sign up and sign in To start developing with Video Indexer, you must first Sign up and sign in to the Video Indexer portal. You can sign up for the service using existing AAD, LinkedIn, Facebook, Google, or MSA account. If signing in with an AAD account (for example, alice@contoso.onmicrosoft.com) you must go through two preliminary steps: 1. Contact Microsoft to register your AAD organization s domain (contoso.onmicrosoft.com). 2. Your AAD organization s admin must first sign in to grant the portal permissions to your org. Upload a video To upload video press the Upload button. Once your video has been uploaded, Video Indexer starts indexing and analyzing the video. Once Video Indexer is done analyzing, you will get a notification with a link to your video and a short description of what was found in your video. For example: people, topics, OCRs. Next steps You can now use the Video Indexer portal or the API to see the insights of the video. See also

6 Video Indexer overview

7 Use Azure Video Indexer API 1/9/ min to read Edit Online Video Indexer consolidates various audio and video artificial intelligence (AI) technologies offered by Microsoft in one integrated service, making development simpler. The APIs are designed to enable developers to focus on consuming Media AI technologies without worrying about scale, global reach, availability, and reliability of cloud platform. You can use the API to upload your files, get detailed video insights, get URLs of insight and player widgets in order to embed them into your application, and other tasks. This article shows how the developers can take advantage of the Video Indexer API. To read a more detailed overview of the Video Indexer service, see the overview article. Subscribe to the API 1. Sign in. To start developing with Video Indexer, you must first Sign In to the Video Indexer portal. If signing in with an AAD account (for example, alice@contoso.onmicrosoft.com) you must go through two preliminary steps: a. Contact us to register your AAD organization s domain (contoso.onmicrosoft.com). b. Your AAD organization s admin must first sign in to grant the portal permissions to your org. To do this, the organization's admin must navigate to client_id=e97b04d3-3e10-41fb-926f-b3c868fef6df&redirect_uri= sign in and give consent. 2. Subscribe. Select the Products tab. Then, select Production and subscribe.

8 Once you subscribe, you will be able to see your subscription and your primary and secondary keys. The keys should be protected. The keys should only be used by your server code. They should not be available on the client side (.js,.html, etc.). 3. Start developing. You are ready to start integrating with the API. Find the detailed description of each Video Indexer REST API. Recommendations This section lists some recommendations when using Video Indexer API. If you are planning to upload a video, it is recommended to place the file in some public network location (for example, OneDrive). Get the link to the video and provide the URL as the upload file param. When you call the API that gets video breakdowns for the specified video, you get a detailed JSON output as the response content. See details about the returned JSON in this topic. Code sample The following C# code snippet demonstrates the usage of all the Video Indexer APIs together.

9 var apiurl = " var client = new HttpClient(); client.defaultrequestheaders.add("ocp-apim-subscription-key", "..."); var content = new MultipartFormDataContent(); Console.WriteLine("Uploading..."); var videourl = " var result = client.postasync(apiurl + "? name=some_name&description=some_description&privacy=private&partition=some_partition&videourl=" + videourl, content).result; var json = result.content.readasstringasync().result; Console.WriteLine(); Console.WriteLine("Uploaded:"); Console.WriteLine(json); var id = JsonConvert.DeserializeObject<string>(json); while (true) { Thread.Sleep(10000); result = client.getasync(string.format(apiurl + "/{0}/State", id)).result; json = result.content.readasstringasync().result; Console.WriteLine(); Console.WriteLine("State:"); Console.WriteLine(json); } dynamic state = JsonConvert.DeserializeObject(json); if (state.state!= "Uploaded" && state.state!= "Processing") { break; } result = client.getasync(string.format(apiurl + "/{0}", id)).result; json = result.content.readasstringasync().result; Console.WriteLine(); Console.WriteLine("Full JSON:"); Console.WriteLine(json); result = client.getasync(string.format(apiurl + "/Search?id={0}", id)).result; json = result.content.readasstringasync().result; Console.WriteLine(); Console.WriteLine("Search:"); Console.WriteLine(json); result = client.getasync(string.format(apiurl + "/{0}/InsightsWidgetUrl", id)).result; json = result.content.readasstringasync().result; Console.WriteLine(); Console.WriteLine("Insights Widget url:"); Console.WriteLine(json); result = client.getasync(string.format(apiurl + "/{0}/PlayerWidgetUrl", id)).result; json = result.content.readasstringasync().result; Console.WriteLine(); Console.WriteLine("Player token:"); Console.WriteLine(json); Next steps Examine details of the output JSON.

10 See also Video Indexer overview

11 Examine the Video Indexer output 1/9/ min to read Edit Online When you call the Get Breakdowns API and the response status is OK, you get a detailed JSON output as the response content. The JSON content contains details of the specified video insights including (transcript, OCRs, people). The details include keywords (topics), faces, blocks. Each block includes time ranges, transcript lines, OCR lines, sentiments, faces, and block thumbnails. You can use the Get Breakdowns API to get the full breakdown of a video as a JSON content. GET HTTP/1.1 Host: videobreakdown.azure-api.net Ocp-Apim-Subscription-Key: You can also visually examine the video's summarized insights by pressing the Play button on the video in the Video Indexer portal. For more information, see View and edit video insights. This topic examines the JSON content returned by the Get Breakdowns API. NOTE Make sure to review the concepts topic. Root elements id The id of this video. For example, "63c6d532ff".

12 partition name description username A logical partition that the user can specify in upload in order to search for it later. The name of the video. For example, "Azure Monitor". Description of the video. For example, "John Kemnetz joins Scott Hanselman to show how to unlock the power of Azure monitoring data with Azure Monitor. " The creator of the video. For example, "Channel9 Videos". createtime Time created. For example, " T16:36: :00". privacymode isowned isbase durationinseconds summarizedinsights breakdowns social Your video can have one of the following modes: Private, Public. Public - the video is visible to everyone in your account and anyone that has a link to the video. Private - the video is visible to everyone in your account. True, if the current user owns the video. Otherwise, false. True, if the breakdown is based on a source video. False, if the breakdown is of a playlist that is derived from another breakdown. Duration of the video. Contains one summarizedinsights. May contain one or more breakdowns Contains one social element that describes number of likes and views of the video. summarizedinsights This section shows the summary of the insights. name shortid privacymode duration The name of the video. For example, "Azure Monitor". The id of the video. For example, "63c6d532ff". Your breakdown can have one of the following modes: Private, Public. Public - the video is visible to everyone in your account and anyone that has a link to the video. Private - the video is visible to everyone in your account. Contains one duration that describes the time an insight occurred. Duration is in seconds.

13 thumbnailurl faces topics sentiments audioeffects The video's thumbnail full URL. For example, " fac5-e8ae-40d9-a04a-6b2928fb5d10? accesstoken=eyj0exaioijkv1qilcjhbgcio...". Notice that if the video is private, the URL contains a 1 hour access token. After 1 hour, the URL will no longer be valid and you will need to either get the breakdown again with a new url in it, or call GetAccessToken to get a new access token and construct the full url manually (' nailid]?accesstoken=[accesstoken]'). May contain one or more faces May contain one or more topics May contain one or more sentiments May contain one or more audioeffects breakdowns This section shows the details of the insights. id state The breakdown id. For example, "63c6d532ff". The processing state of the given breakdown id. Could be one of the following: Uploaded, Processing, Processed, Failed. processingprogress The progress. For example, "10%". externalid externalurl metadata insights thumbnailurl You can set externalid during upload. For example, "4f9c3500-eca7-4ab3-987e-a745017af698". You can later search for your videos by this external id. You can set externalurl during upload. You can set metadata during upload. May contain one or more insights The video's thumbnail full URL. For example, " fac5-e8ae-40d9-a04a-6b2928fb5d10? accesstoken=eyj0exaioijkv1qilcjhbgcio...". Notice that if the video is private, the URL contains a 1 hour access token. After 1 hour, the URL will no longer be valid and you will need to either get the breakdown again with a new url in it, or call GetAccessToken to get a new access token and construct the full url manually (' nailid]?accesstoken=[accesstoken]').

14 publishedurl viewtoken sourcelanguage language The published URL. For example, " 48e2-4fbc ea6335da563/Azure%20Monitor%20%20Azure%20Friday.is m/manifest". The bearer token The source language. The following are supported: Chinese, English, French, German, Italian, Japanese, Portuguese, Russian, Spanish. The language of the transcript. insights transcriptblocks topics faces participants contentmoderation audioeffectscategories May contain one or more transcriptblocks May contain one or more topics May contain one or more faces May contain one or more participants May contain one contentmoderation May contain one or more audioeffectscategories faces summarizedinsights faces that appear under summarizedinsights, show a summary of each face found in the video. id The id of a person. For example, shortid name description The short id. Because a playlist may be derived from several breakdowns, this id is needed to find out which of these breakdowns is the origin of each face. If the face is recognized, the name of the person is added. For example, "Scott Hanselman". If the face is unknown, "Unknown #" is added. If the face is recognized, the description is populated based on the Bing API search. Otherwise, the description is null.

15 title thumbnailfullurl appearances seenduration If the face is recognized, the description is populated based on the Bing API search. Otherwise, the title is null. The face's thumbnail full URL. For example, " fac5-e8ae-40d9-a04a-6b2928fb5d10? accesstoken=eyj0exaioijkv1qilcjhbgcio...". Notice that if the video is private, the URL contains a 1 hour access token. After 1 hour, the URL will no longer be valid and you will need to either get the breakdown again with a new url in it, or call GetAccessToken to get a new access token and construct the full url manually (' nailid]?accesstoken=[accesstoken]'). May contain one or more appearances For how long the face was seen (in seconds). seendurationratio Presence relative to the video duration (0-1). breakdown insights faces that appear under breakdowns, describe details about each face found in the video. id The id of a person. For example, bingid name thumbnailid description title imageurl If the face is recognized, the name of the person is added. For example, "Scott Hanselman". If the face is unknown, "Unknown #" is added. For example, "616468f efa-94e7-262f2e575059". If the face is recognized, the description is populated based on the Bing API search. Otherwise, the description is null. If the face is recognized, the description is populated based on the Bing API search. Otherwise, the title is null. This URL points to an image that is taken from the source video. confidence knownpersonid The id of a known person (for example, celebrity). If a person is not known, the id contains zeros. For example, "e3eaff5fee1b-4eac-80ce-ebac47aadf64". topics

16 summarizedinsights topics that appear under summarizedinsights, show a summary of each topic found in the video. name appearances istranscript The topic name (for example, "Azure"). May contain one or more appearances. True, if found in a transcript. False, if found in an OCR. breakdown insights topics that appear under breakdowns, describe details about each topic found in the video. id name stem words rank sentiments sentimentkey appearances Currently, the following sentiments are supported: Positive, Neutral, Negative. May contain one or more appearances seendurationratio Presence relative to the video duration (0-1). audioeffects audioeffectkey appearances Valid values are: Speech, Silence, HandClaps. May contain one or more appearances seendurationratio Presence relative to the video duration (0-1). seenduration For how long the audio effect was present (in seconds). appearances

17 starttime endtime startseconds endseconds Time value. Time value. Time value. Time value. participants id The id of the participant. name The name of the participant. For example, "Speaker #1". pictureurl The pictureurl attribute is reserved for future use. contentmoderation adultclassifiervalue bannedwordscount bannedwordsratio issuspectedasadult isadult audioeffectscategories type key Id of the category. One of the following: Speech, Silence, HandClaps. transcriptblocks id lines Id of the block. May contain one or more lines

18 sentimentids thumbnailids sentiment faces ocrs audioeffectinstances scenes annotations The sentimentids attribute is reserved for future use. The thumbnailids attribute is reserved for future use. The sentiment in the block (0-1, negative to positive). May contain one or more faces. May contain one or more ocrs. May contain one or more audioeffectinstances. May contain one or more scenes. May contain zero or more annotations. ocrs Describes at what point in the video the text content was found. timerange adjustedtimerange lines The time range in the original video. AdjustedTimeRange is the time range relative to the current playlist. Since you can create a playlist from different lines of different videos, you can take a 1-hour video and use just 1 line from it, for example, 10:00-10:15. In that case, you will have a playlist with 1 line, where the time range is 10:00-10:15 but the adjustedtimerange is 00:00-00:15. May contain one or more lines. lines transcriptblocks lines that appear under transcriptblocks, describe lines of transcripts found in the video. id timerange adjustedtimerange The id of the line. The time range in the original video. AdjustedTimeRange is the time range relative to the current playlist. Since you can create a playlist from different lines of different videos, you can take a 1-hour video and use just 1 line from it, for example, 10:00-10:15. In that case, you will have a playlist with 1 line, where the time range is 10:00-10:15 but the adjustedtimerange is 00:00-00:15.

19 participantid text isincluded The id of the speaker of this line. The transcript. In base breakdowns always true. In derived playlists, the lines that were included in the source video, are set to isincluded=true. All other lines are false. ocrs lines that appear under ocrs, describe lines of OCRs found in the video. id The OCR id. width height language textdata The OCR language. The OCR text. confidence scenes id timerange keyframe shots The scene id. Contains one timerange. The time of the key frame. May contain one or more shots. shots id timerange keyframe Contains one timerange. The time of the key frame. audioeffectinstances

20 type ranges May contain one or more ranges. ranges ranges that appear under audioeffectinstances, describe audio effects in those ranges. timerange adjustedtimerange The time range in the original video. AdjustedTimeRange is the time range relative to the current playlist. Since you can create a playlist from different lines of different videos, you can take a 1-hour video and use just 1 line from it, for example, 10:00-10:15. In that case, you will have a playlist with 1 line, where the time range is 10:00-10:15 but the adjustedtimerange is 00:00-00:15. Next steps For information about how to create your own breakdown, see View and edit Video Indexer insights. For information about how to embed widgets in your application, see Embed Video Indexer widgets into your applications. See also Video Indexer API Video Indexer overview

21 Find exact moments within videos 1/9/ min to read Edit Online This topic shows you the search options that enable you to find exact moments within videos. 1. Sign in to your Video Indexer account. 2. Search among all videos in your account. In the following example, we searched for all videos created by Channel9 with Scott Hanselman. 3. Search the summarized insights of the video. You can then search within a video by clicking Play on the video. Then, you can search within the video by selecting the Search tab. For example, we searched for all the places where the "identity protection" text is used.

22 If you click one of the results, the player brings you to that moment in the video. You can achieve the player/insights view and synchronization in your application. For more information, see Embed Video Indexer widgets into your application. 4. Search the detailed breakdown of the video. If you want to create your own breakdown based on the video that you found, press the Edit button. This page shows you the full breakdown of a video. You can search within the breakdown to only show the lines you are interested in. For more information, see View and edit Video Indexer insights. In this example, we searched the "identity protection" text. We also applied additional filters, as shown in the screen below.

23 Next steps Once you find the video you want to work with, you can continue processing the video, as described in one of these topics: Create new video insights based on existing video Process content with Video Indexer REST API Embed visual widgets in your application See also Video Indexer overview

24 View and edit Video Indexer insights 1/9/ min to read Edit Online This topic shows you how to view and edit the Video Indexer insights of a video. 1. Sign in to your Video Indexer account. 2. Find a video from which you want to create your Video Indexer insights. For more information, see Find exact moments within videos. 3. Press Play. The page shows the video's summarized insights. 4. View the summarized insights of the video. Summarized insights show an aggregated view of the data: faces, keywords, sentiments. For example, you can see the faces of people and the time ranges each face appears in and the % of the time it is shown. The player and the insights are synchronized. For example, if you click a keyword or the transcript line, the player brings you to that moment in the video. You can achieve the player/insights view and synchronization in your application. For more information, see Embed Azure Indexer widgets into your application. 5. Edit the Video Indexer insights. Press Edit under the video. The page that shows you the full breakdown of a video is displayed. The breakdown is broken into blocks. Blocks are here to make it easier to go through the data. For example, block might be broken down based on when speakers change or there is a long pause. You can create your own playlist that contains only lines that you want. To show only specific parts of the source video, you can filter by topics/keywords, sentiments, people, speakers. You can choose to only view the video's transcript or OCR.

25 Next steps Learn how to create your own Video Indexer insights based on some other video. See also Video Indexer overview

26 Create video insights from existing videos 7/20/ min to read Edit Online This topic shows you how to create and publish video insights based on some other video. 1. Sign in to your Video Indexer account. 2. Find a video from which you want to create your video insights. 3. Press Play. The page shows the video's summarized insights. 4. Press the Edit button. This page shows you the full breakdown of a video. The breakdown is broken into blocks. Blocks are here to make it easier to go through the data. For example, block might be broken down based on when speakers change or there is a long pause. You can create your own playlist that contains only lines that you want. To show only specific parts of the source video, you can filter by topics/keywords, sentiments, people, speakers. You can choose to only view the video's transcript or OCR. 5. Create your playlist. To add or remove lines to/from your playlist, press +/-.

27 6. Preview your playlist. Once you are done creating the playlist, press Preview. 7. Publish the playlist. After you preview the playlist, you can publish it. Once you publish the playlist, it is added to the list of your video insights. Next steps Once you create the new playlist, you can continue processing it, as described in one of these topics: Process content with Video Indexer REST API Embed visual widgets in your application See also Video Indexer overview

28 Embed Video Indexer widgets into your applications 1/9/ min to read Edit Online Video Indexer supports embedding two types of widgets into your application: Cognitive Insights and Player. A Cognitive Insights widget includes all visual insights that were extracted from your video indexing process. A Player widget enables you to stream the video using adaptive bit rate. Embedding public content 1. Sign in to your Video Indexer account. 2. Click the "embed" button that appears below the video. After clicking the button, an embed modal will appear on the screen where you can choose what widget you want to embed in your application. Selecting a widget (Player or Cognitive Insights), generates the embedded code for you to paste in your application. 3. Choose the type of widget you want (Cognitive Insights or Player). 4. Copy the embed code, and add to your application. Embedding private (or Org) content You can get embed codes from embed popups (as shown in the previous section) for Public videos only. If you want to embed a Private video, you have to pass an access token in the iframe's src attribute: player]/<videoid>/?accesstoken=<accesstoken>

29 Use the GetInsightsWidgetUrl API to get a URL for a Cognitive Insights widget. The URL includes the access token. Specify this URL as the iframe's src value. If you want to provide editing insights capabilities (like we have in our web application) in your embedded widget, you will have to call Get Insights Widget Url or Get Insight Widget By External Id and add &allowedit=true. Widgets interaction The Cognitive Insights widget can interact with a video on your application. This section shows how to achieve this interaction. Cross-origin communications

30 To get Video Indexer widgets to communicate with other components, the Video Indexer service does the following: Uses the cross-origin communication HTML5 method postmessage and Validates the message across videobreakdown.com origin. If you choose to implement your own player code and do the integration with Cognitive Insights widgets, it is your responsibility to validate the origin of the message that comes from videobreakdown.com. Embed both types of widgets in your application / blog (recommended) This section shows how to achieve interaction between two Video Indexer widgets so when a user clicks the insight control on your application, the player jumps to the relevant moment. <script src=" 1. Copy the Player widget embed code. 2. Copy the Cognitive Insights embed code. 3. Add the file to handle the communication between the two widgets. Now when a user clicks the insight control on your application, the player jumps to the relevant moment. For more information, see this demo. Embed the Cognitive Insights widget and use Azure Media Player to play the content This section shows how to achieve interaction between a Cognitive Insights widget and an Azure Media Player instance using the AMP plugin. 1. Add a Video Indexer plugin for the AMP player. <script src=" 2. Instantiate Azure Media Player with the Video Indexer plugin.

31 // Init Source function initsource() { var tracks = [{ kind: 'captions', // Here is how to load vtt from VI, you can replace it with your vtt url. src: this.getsubtitlesurl("c4c1ad4c9a", "English"), srclang: 'en', label: 'English' }]; myplayer.src([ { "src": "//amssamples.streaming.mediaservices.windows.net/ c b-ba01- faba6c2206a2/azuremediaservicespromo.ism/manifest", "type": "application/vnd.ms-sstr+xml" } ], tracks); } // Init your AMP instance var myplayer = amp('vid1', { /* Options */ "nativecontrolsfortouch": false, autoplay: true, controls: true, width: "640", height: "400", poster: "", plugins: { videobreakedown: {} } }, function () { // Activate the plugin this.videobreakdown({ videoid: "c4c1ad4c9a", synctranscript: true, synclanguage: true }); // Set the source dynamically initsource.call(this); }); 3. Copy the Cognitive Insights embed code. You should be able now to communicate with your Azure Media Player. For more information, see this demo. Embed Video Indexer Cognitive Insights widget and use your own player (could be any player) If you use your own player, you have to take care of manipulating your player yourself in order to achieve the communication. 1. Insert your video player. For example, a standard HTML5 player <video id="vid1" width="640" height="360" controls autoplay preload> <source src="//breakdown.blob.core.windows.net/public/microsoft%20hololens-%20roboraid.mp4" type="video/mp4" /> Your browser does not support the video tag. </video> 2. Embed the Cognitive Insights widget.

32 3. Implement communication for your player by listening to the "message" event. For example: <script> (function(){ // Reference your player instance var playerinstance = document.getelementbyid('vid1'); function jumpto(evt) { var origin = evt.origin evt.originalevent.origin; // Validate that event comes from the videobreakdown domain. if ((origin === " && evt.data.time!== undefined){ // Here you need to call your player "jumpto" implementation playerinstance.currenttime = evt.data.time; // Confirm arrival to us if ('postmessage' in window) { evt.source.postmessage({confirm: true, time: evt.data.time}, origin); } } } // Listen to message event window.addeventlistener("message", jumpto, false); }()) </script> For more information, see this demo. Adding subtitles If you embed Video Indexer insights with your own AMP player, you can use the GetVttUrl method to get closed captions (subtitles). You can also call a javascript method from the Video Indexer AMP plugin getsubtitlesurl (as shown earlier). Customizing embeddable widgets Cognitive insights widget You can choose the types of insights you want by specifying them as a value to the following URL parameter added to the the embed code you get (from API or from the web application): &widgets= The possible values are: people, keywords, sentiments, transcript, search. For example, if you want to embed a widget containing only people and search insights the iframe embed URL will look like this: The title of the iframe window can also be customized by providing &title= to the iframe url. (It will customize the html