Computer Vision and Media Analytics

Size: px

Start display at page:

Download "Computer Vision and Media Analytics"

Domenic Collins
5 years ago
Views:

1 1 Computer Vision and Media Analytics Creating New Opportunities for Value-Add Connected TV Services Tony Emerson Managing Director Worldwide Media and Cable

2 Our OTT Vision Capitalize on existing content rights Unlock new niche markets Create a better viewing experience Create a closer connection to the audience Grow the business

4 Platform Services Security & Management Portal Active Directory Cloud Services Batch Service Fabric Remote App Web Apps Mobile Apps API Apps Logic Apps API Management Notification Hubs Visual Studio Team Project Azure SDK Application Insights Hybrid Operations Azure AD Connect Health AD Privileged Identity Management Multi-Factor Authentication Backup Automation HDInsight Machine Learning VoD/Live Transcoding Media Insights SQL Database SQL Data Warehouse Operational Insights Key Vault Data Factory Event Hubs Multi DRM VoD/Live Channel Streaming Redis Cache Search Import/Export Store / Marketplace VM Image Gallery & VM Depot Stream Analytics Mobile Engagement Azure Media Player Content Delivery Network (CDN) DocumentDB Tables Site Recovery StorSimple Infrastructure Services

5 Rio Olympics most successful in history

6 Rio 2016 by the Numbers

7 Microsoft Azure and Partners deliver globally

8 Azure Media Services Scalable components for building custom media workflows in the cloud Plus a growing ecosystem of value-add third party partner components Cloud Upload & Storage Encoding & Media Analytics Content Protection Live & On Demand Streaming with integrated CDN Player Clients

9 What will you do to differentiate your service? How can you increase the value of content? Can you make it easier to search for text, faces, logos and images, specific actions? How can you pull more data out of the content to enhance discoverability, viewability? How can you deal more efficiently with legal and regulatory compliance?

10 Azure Media Analytics Enhancing Your Content Make Video and Audio Searchable Creating a database of rich metadata pulled directly out of the video and audio content itself Powerful new media processors Speech-to-Text Facial and Emotion Detection & Facial Redaction Motion Detection, Stabilization, and Acceleration Object, Character, and Logo Recognition Automated Video Summarization

11 Azure Media Analytics Speech and Vision Services at enterprise scale, compliance, security and global reach 3 rd Party industry specific SaaS solutions Cloud Upload & Storage Encoding Computer Vision Speech Analytics Clients AND MORE.

12 Indexer Enables speech to text conversion Languages supported English & Spanish as GA German, French, Italian, Chinese, Portuguese, Arabic as Preview Use cases Deep Search & First-pass captions Capable of custom vocabulary adaptation User provides list of words related to video to improve speech recognition

AZURE MEDIA INDEXER TECHNICAL DETAILS Azure

(TTML/WebVTT/SAMI) Audio or Video MP4, WMV,

Vocabulary Adaptation Segmentation Speech

13 AZURE MEDIA INDEXER TECHNICAL DETAILS Azure Media Indexer Closed captions (TTML/WebVTT/SAMI) Audio or Video MP4, WMV, MP3, M4A, AAC, WAV, WMA Audio Decoding Vocabulary Adaptation Segmentation Speech Recognition Caption Alignment Audio Indexing Blob (AIB) for use with SQL Server and custom Ifilter add-on (link) Flexible metadata files (keywords, word info)

14 Hyperlapse Transforms first-person videos into smooth time-lapses Designed for forward-moving camera scenarios (action sports) dash camera)

15 Motion Detection Reduce False Positives Receive alerts for motion events in video Look for motion in specified parts of the frame Advanced processing to detect false positives better than camera s built in motion detection Ignore false positives such as light changes, shadows, small insects, and others.

16 Face Detection Detect faces that appear in your videos Track faces as they move around the frame Output Metadata with face locations and timestamps Age Detection Gender Detection Facial Recognition

Emotion Recognition Recognize the emotion of a person or crowd over time based on the facial expressions in the video Designed for real emotions in-the-wild.

17 Emotion Recognition Recognize the emotion of a person or crowd over time based on the facial expressions in the video Designed for real emotions in-the-wild. Identifies emotions based on expressions that psychological research has identified as universal A solution for personalizing experiences, analyzing responses to media and products, and crowd analytics Recognize: happiness, sadness, surprise, anger, contempt, fear, digest, neutral Use cases Audience Analytics, Personalization etc.

18 Video Summarization Creates an automatic summary for videos to let people see a preview or snapshot of their video Frames are selected based off of video quality, diversity, and stability of the footage original video

Video OCR Extract typeset words from video content Select your own sampling rate to balance performance and quality Specify where in the video to look (e.g. bottom third for captions) Output describes text with location Text: Who are we?

19 Video OCR Extract typeset words from video content Select your own sampling rate to balance performance and quality Specify where in the video to look (e.g. bottom third for captions) Output describes text with location Text: Who are we? Location: (200,100,250,50) Time: 0:45:02 Text: Who are you and who is the person sitting next to you? Location: (100,250,350,90) Time: 0:45:02

20 Content Moderation Detect video content policy violations Save time and money spent manually reviewing your content for offensive, illicit and inappropriate material Currently supports adult content classification Coming - Detection of profanity, violence etc.

21 Storage Example customers Users Administrators Surveillance cameras Ingestion Live media Indexer Media Analytics technologies (combination of 1 st and 3 rd party) Hyperlapse Face Detector Motion Detector Content Moderator Face recognition Media devices Media file Media metadata Audio Processor Video Stabilization Face Redaction Logo Detector Video tagging More. IOT Sensors Industry specific data Event metadata Other data VOD encoding Live encoding Media Metadata Events Insights Streaming Dynamic packaging Dynamic encryption

GrayMeta - The video & metadata experts behind MetaFarm MetaFarm is a

business s time and money and bringing insight to the data that is already

Connect to dispersed and siloed data for the right reasons, enabling easier

metadata from any file type across all file systems, databases and data

machine learning & AI services powered by Azure and the Cortana Suite Easy

22 GrayMeta - The video & metadata experts behind MetaFarm MetaFarm is a powerful platform that tackles big data and metadata problems, saving business s time and money and bringing insight to the data that is already there. Connect to dispersed and siloed data for the right reasons, enabling easier migration and adoption of Azure and other services Extract embedded metadata from any file type across all file systems, databases and data feeds Create new metadata leveraging the exponential growth of cognitive, machine learning & AI services powered by Azure and the Cortana Suite Easy Upload to Azure (Signiant) Powerful Search & Discovery Bring Cognitive, AI & Machine Learning to the data in a easy to digest way Review, Consume, Share and Take Action

23 Azure Consumption Storage Compute Cognitive Services - Vision - Speech - Language - Knowledge - Search

SORT, ORGANIZE & ACCESS What data do I really have? How many duplicates? Find and organize all assets / data across all departments, which also helps prior to data migration.

24 SORT, ORGANIZE & ACCESS What data do I really have? How many duplicates? Find and organize all assets / data across all departments, which also helps prior to data migration. SEARCH & DISCOVERY ACROSS ENTERPRISE DATA SILO s I need the same asset another group has, why do I need to create or source it again? Leverage data / content across Linear, VOD and OTT within broadcaster s multiple data locations and increase efficiencies across the enterprise while driving cost down. TIME & COST EFFICENCIES I need more data quicker and the FTE approach doesn t have the ROI. Bring efficiencies to the workplace, reducing the need for manual tagging and labeling of content and reducing the time to access the right data by 10x.

VSNEXPLORER and Azure Media Services MICROSOFT AZURE AZURE MEDIA SERVICES Integration with Speech to Text and Translation VSN is a leading End-to-End IT developer company for the Broadcast and the

25 VSNEXPLORER and Azure Media Services MICROSOFT AZURE AZURE MEDIA SERVICES Integration with Speech to Text and Translation VSN is a leading End-to-End IT developer company for the Broadcast and the M&E Industries, with over 1000 clients in more than 100 countries. VSNEXPLORER provides corporations a secure, always-on media asset management solution that allows companies and users to collaborate with their media archive, optimizing their processes and enhancing their capabilities from any location VSNEXPLORER working with Azure Media Services Video Stream Networks S.L. Copyright

26 Celebrity Search Input Entertainment videos Output Search index based on celebrities in videos User should be able to search for videos where Celebrity X and Y were sighted together Celebrity X said certain words or phrases.. Media Analytics Cognitive Vision API Azure Search

27 Face recognition Input Videos of any type (entertainment, surveillance etc.) Output Search index based on list of known faces User should be able to search for videos where Person X and Y were sighted together Person X said certain words or phrases.. Media Analytics Cognitive Face API Azure Search

28 Audio redaction Input Videos of any type Output Videos with keywords redacted Useful in following scenarios Identity protection (security videos) Applying censorship (broadcasting on public channels).. Media Indexer Transcript Filter Redaction timecodes AMS Audio Overlays

29 Video Tagging Identifies objects and categories that are within a video frame Uses a trained model with over 2000 tags Output metadata with video tags by frames

30 Action Recognition Identifies the actions taking place within a sequence of frames Starting with 61 categories (46 sports + 15 daily activities) Output metadata with action and time stamp

31 Redaction Protect the Identities of individuals by blurring the video Automatically detect and redact faces Tag and blur identifiable information in dynamic settings such as License Plates

Storage Industry specific SaaS solutions

Ingestion Live media Indexer Media Analytics

Media devices Media file Media metadata Audio

32 Storage Industry specific SaaS solutions Consumers Administrators Surveillance cameras Ingestion Live media Indexer Media Analytics technologies (combination of 1 st and 3 rd party) Hyperlapse Face Detector Motion Detector Content Moderator Face recognition Media devices Media file Media metadata Audio Processor Video Stabilization Face Redaction Logo Detector Video tagging More. IOT Sensors Industry specific data Event metadata Other data VOD encoding Live encoding Media Metadata Events Insights Streaming Dynamic packaging Dynamic encryption

33 Computer Vision and Media Analytics Imagine what you can do with these services?

34 37

Vishesh Oberoi Seth Reid Technical Evangelist, Microsoft Software Developer, Intergen

Vishesh Oberoi Seth Reid Technical Evangelist, Microsoft Software Developer, Intergen Vishesh Oberoi Technical Evangelist, Microsoft VishO@microsoft.com @ovishesh Seth Reid Software Developer, Intergen contact@sethreid.co.nz @sethreidnz Vishesh Oberoi Technical Evangelist, Microsoft VishO@microsoft.com