Pixelsior: Photo Management as a Platform Service for Mobile Apps

Transcription

1 Pixelsior: Photo Management as a Platform Service for Mobile Apps Kyungho Jeon, Sharath Chandrashekhara, Karthik Dantu and Steven Y. Ko University at Buffalo, The State University of New York

2 Photo data is exploding 2 billion mobile devices with cameras were sold in 2014 [Evans 2014] 3+ billion photos were shared on Facebook, Snapchat, Instagram, and WhatsApp every day in 2015 Source: KPCB Internet Trends

3 Photo apps are so popular Many popular apps have photorelated features Google Photos, Snapchat, Instagram, Facebook, WhatsApp,... Popular features including... Sharing, Editing, Sync, Image analysis 3

4 Popular feature #1: Sharing photos over messaging apps Alice Bob 4

5 Popular feature #2: Editing a photo and synchronizing Smartphone Cloud Applying Monochrome filter Sync 5

6 Popular feature #3: Image analysis E.g., Face recognition Use the information to organize and search photos Managed as metadata Source: Google Photos 6

7 How do current apps provide these features and what are the problems? 7

8 Sharing photos over multiple messaging apps Alice Bob Chris Resize 8

9 Sharing photos over multiple messaging apps Alice Bob Chris Resize 9

10 Sharing photos over multiple messaging apps Alice Bob Resize Resize Chris 10

11 Sharing photos over multiple messaging apps Alice Bob Resize Resize Chris 11

12 Sharing photos over multiple messaging apps Alice Bob Repeated resizing and duplicate resized image files Resize Resize Chris 12

13 Editing synchronized photos with content adaptation Smartphone Cloud "Adapted : Reduced size image Synchronized Original size image 13

14 Editing synchronized photos with content adaptation Smartphone I want to apply a filter! Cloud "Adapted : Reduced size image Synchronized Original size image 14

15 Editing and synchronizing with content adaptation Smartphone I want to apply a filter! Cloud Reduced size Original size Download Original size image 15

16 Editing and synchronizing with content adaptation Smartphone I want to apply a filter! Cloud Reduced size Original size Original size image 16

17 Editing and synchronizing with content adaptation Smartphone I want to apply a filter! Cloud Reduced size Original size Sync Original size image 17

18 Editing and synchronizing with content adaptation Smartphone I want to apply a filter! Cloud Reduced size Original size Sync Original size image 18

19 Editing and synchronizing with content adaptation Smartphone I want to apply a filter! Cloud Reduced size Original size Original size image 19

20 Editing and synchronizing with content adaptation Smartphone Done! Cloud Reduced size Original size image 20

21 Editing and synchronizing with content adaptation Smartphone Done! Cloud Unnecessary network and computing operations Original size image 21

22 Editing and synchronizing with content adaptation Smartphone Done! Cloud Impossible to edit a photo without a network connectivity Original size image 22

23 Image analysis: managing semantic information App A runs face recognition algorithm App B runs face recognition algorithm Per-app Metadata File Per-app Metadata 23

24 Image analysis: managing semantic information App A runs face recognition algorithm Repeated image analyses and individual management of metadata App B runs face recognition algorithm Per-app Metadata File Per-app Metadata 24

25 Summary of current problems Sharing photos Repeated resizing computation and redundant resized images Editing synchronized, adapted photos Propagating updates on adapted photos requires careful implementation Image analysis and metadata management Repeated analyses from multiple apps and individual metadata management 25

26 Summary of current problems Sharing photos Repeated resizing computation and redundant resized images Editing Every app synchronized, implements adapted photo related photosfeatures individually! Propagating updates on adapted photos requires careful implementation Image analysis and metadata management Repeated analyses from multiple apps and individual metadata management 26

27 Research question What is a better way to manage mobile photo data? 27

28 A consolidated shared service, provided by a Pixelsior A new platform for mobile photo data mobile platform An efficient mechanism for photo management functionality 28

29 Before Image processing Image loading (Meta) Data management Image processing Image loading (Meta) Data management 29

30 Before After Image processing Image loading (Meta) Data management Image processing Image loading (Meta) Data management Pixelsior Image processing Image loading (Meta) Data management 30

31 Sharing photos over messaging apps Alice Bob Request: width 800px Chris Pixelsior 31

32 Sharing photos over messaging apps Alice Bob Chris Pixelsior 32

33 Sharing photos over messaging apps Alice Bob Chris Pixelsior 33

34 Sharing photos over messaging apps Alice Bob Request: width 800px Chris Pixelsior 34

35 Sharing photos over messaging apps Alice Bob Chris Pixelsior 35

36 Sharing photos over messaging apps Alice Bob Chris Pixelsior 36

37 Editing synchronized photos with content adaptation Smartphone Cloud Pixelsior Pixelsior Reduced size image Full size image 37

38 Editing synchronized photos with content adaptation Smartphone Cloud Pixelsior Pixelsior Reduced size image Grayscale filter Full size image 38

39 Editing synchronized photos with content adaptation Smartphone Cloud Pixelsior Pixelsior Reduced size image Grayscale filter Apply Grayscale Full size image 39

40 Editing synchronized photos with content adaptation Smartphone Cloud Pixelsior Pixelsior Apply Grayscale Reduced size image Grayscale filter Full size image Grayscale filter 40

41 Semantic information management App A runs face detection algorithm and updates metadata App B retrieves not only photo but also its metadata Pixelsior metadata 41

42 Design Space Data model API: a developer s point of view Transformation management 42

43 Design: data model Pixelsior Photo store Pixelsior API 43

44 Design: APIs read and write Retrieve a resize depending on app-specific policy (qufile, FAST 10) search and query Organize and search photos with attributes (Semantic File Systems, SOSP 91) Query a photo s metadata and transformation history 44

45 Design: transformation management Built-in primitive image transformations e.g., blur, brightness, contrast, grayscale,... Customized transformations Can be developed as a sequence of primitive transformations Can be plugged into the platform so that other apps can use 45

46 Design: transformation management Tablet Brightness & Contrast: -40% Phone 46

47 Design: transformation management Tablet Brightness & Contrast: -40% Download Phone Resize 47

48 Design: transformation management Tablet Brightness & Contrast: -40% Brightness & Contrast: -40% Phone 48

49 Open questions and thoughts (1/2) Access control Letting apps access a View, e.g., Earp (NSDI 16) Legacy apps Leveraging application rewriting frameworks e.g. BlueMountain (HotStorage 15) 49

50 Open questions and thoughts (2/2) Other data types, e.g., video, voice mail,... Provide a layered design: Consistency Primitive interface layer Data type specific layer Operation-based synchronization can result in inconsistency Handle each transformation case by case Phan et al., MobiSys 04 50

51 Summary We identified mobile photo management problems To address the problems, a new platform service, Pixelsior, is proposed Currently, we are implementing it! 51