Wonder of Mobile Social Multimedia. Shih-Fu Chang June 2013, New York

Transcription

1 Wonder of Mobile Social Multimedia Shih-Fu Chang June 2013, New York

2 First Digital Camera in film-less photography New York Times Bits, 8/26/2010 by Steve Sassan of Kodak CCD array, A/D converter, 16 batteries 23 seconds to record a photo to cassette customized reader on a B/W TV for viewing Q: quality, size, cost, store, share?

3 , 1 st camera phone, 0.3Mp 1 billion camera phones sold in , Apple iphone ios 2011, Galaxy S II, 8Mp Android, GPS 2012, Lumia 800 Windows 7 Mobile hongkiat.com/blog/evolution-of-mobile-phones/

4 From Photo Shoebox to Social Multimedia Images 300 million photos uploaded to Facebook every day. Videos 4 billion videos watched per month on YouTube. Social media 30 billion content shared on Facebook per month.

5 Explosion of Mobile Apps July million apps downloaded in the first weekend May billion apps downloaded

6 Wonders of New Mobile Multimedia Recognize music you hear Find objects you see Shazam App, Prof. Ellis Watch diet for you LeafSnap Prof. Belhumeur Tell me what foods are in this picture

7 How Does Automatic Recognition Work? database picture in field Find Important Feature Points in Pictures, Compute Similarity, and Match Layout Figure: David Lowe 7

8 Deploy this over Mobile Networks 4. Visual matching with database images Image Database Send image features Send via mobile networks 5. Send results back Take a picture

9 Technical Challenges Fast response (< 1 2 seconds) Gigantic Database (Billions) 0.5 Limited bandwidth 4. Visual matching with database images Image Database 0.4 Limited power/memory/ 1. Take speed a picture Image feature extraction Send via mobile networks 5. Send results back

10 Columbia Mobile Product Search System: using Advanced Hash Coding Columbia DVMM Lab, 2011 Database: 400,000 product images of Amazon, ebay, and Zappos Performance: Feature extraction: ~1s Compact code: 1K bytes/image Large database search: ~0.4s video demo

11 Augmenting Physical Senses MIT Six Sense Project (Pranav Mistry and Pattie Maes) Camera and display Visual recognition Gesture interaction Mobile wearable computer Pre cursor Concept of Google Glass

12 Mobile Augmented Reality Create virtual worlds at finger tip and interact Examples: Smart AR from Qualcomm and SONY Easy creation of 3D virtual space Real time interaction between characters in physical & virtual worlds tech.philbuzz.com, bookmarkblogs.com,

13 Multi Player Augmented Reality Several students go to a coffee shop. They use their smartphones to turn their table into an Augmented Reality (AR) sandbox! What would be possible if the system could automatically perform 3D scene modeling and semantic labeling? 3D Scene Model + =? Semantic Labels Joint work with Prof. Steve Feiner, CS, Columbia U.

14 Mobile Multimedia Social

15 Social Behavior Crandall et al, 2009, studied 35 million Flickr photos, 300,000 users, photographer movement paths Lens free camera New York San Francisco Blinks&Buttons Load others same location/time

16 Social Sharing > Opinion Expression (socialdefender.com)

17 Tracking the Sentiment on Social Media SGI/Uni. Illinois Global Twitter Heartbeat presidential

18 Social Media Reveal Many Emotions Search for Happiness

19 Emotions across Cultures Search for 幸福 Much more visual on the Chinese site

20 The Power of Social Multimedia A picture is worth one thousand Four more years. Example Rollercoaster at Ru: Queen of the far far away land. 20

21 Research: Which 1000 sentimental concepts? data mining to discover visual sentiments in social media MISTY WOODS Build Sentiment Ontology Psychology emotion wheel (24 emotions) Discover sentiment words SAD EYES Select Adj Noun Pairs 21

22 Sad Beautiful Misty Happy Eyes Woods Face Sky Flower Emotions Expressed And Many in More Social Media Photos 22

23 Research: Teach Machine to Recognize Visual Sentiments MISTY WOODS Build Sentiment Ontology Train Classifiers Psychology emotion wheel (24 emotions) Discover sentiment words SAD EYES Select Adj Noun Pairs Performance Filtering Sentiment Prediction SentiBank (1200 Detectors) S.F. Chang 23

24 Beautiful Sad Happy EyeFace Flower Sky Machine Detected Visual Sentiments More Green: than 600 Correct classifiers Red: Incorrect > 78% accuracy 24

25 Columbia Visual SentiBank System 1200 Classifiers PhotoTweet Stream: True stuff. I have mad respect for all the ladies that DO NOT give in to abortion. #groundzero #newjersey Ouch mr police Predict Sentiment Sentiment Prediction Accuracy Text 0.61 Visual 0.65 Text Visual (Joint)

26 Other Impacts of Social-Mobile Media Where did you get your news yesterday? News consumption is moving mobile and social Access on any device, time, place Social sharing Source: PEW RESEARCH CENTER 2012 News Consumption Survey

27 How will we watch news in the future? NBA finals tech elections elections olympics Greek riots Personalized Multiple channels in one stream Device agnostic Uses viewing history and personal data from social media posts Videos analyzed and processed in the cloud, then consumed anywhere

28 Supported in part by Brown Institute for Media Innovations, in collaboration with Stanford University

29 Research: Linked Personalized News Social Media Posts TV Viewing History Social Media Network 2 Last Monday 7:03pm - 7:49pm Last Thursday 7:02am - 7:10am News Topics Personalized Stream A joint project between Columbia Engineering/Journalism and Stanford University

30 Research on Deep Information Extraction e.g., Detect Who & What in News Video Feature Modes Text Visual Audi o Speech Segmentation Audiotra ck: Segmentati on: Face Tracking & Clustering Name Extraction from Aligned CC- ASR Edit Distance Alignment Speaker Diarization Visual Speaker Detection Name Extraction from OCR Segmented Audiotrack: MFCC Speaker Gender Classification Male GMM Female GMM Classifier Anchor Detection Name Gender Classification Gender

31 Lean Forward User: Active Social Exploration

32 Lean Backward User: Show What s New Today Players of Interest Topical Concepts

33 Wonders of Social Mobile Media Mobil Social Multimedia + Data Science Engineering Smart Living, Entertainment, News, Health, Education