Next-generation Annotation Management. Brian Schlining

Transcription

1 Nextgeneration Annotation Management Brian Schlining

2 A Little History

3 A Little History circa 2005 Video Annotation and Reference System RDBMS Crossplatform

4 A Little History... r o f d e sign e d s a w VARS circa 2005 Frame capture hardware videotape RDBMS Crossplatform Deck control via RS422

5 VARS Today for d e s u s i RS A V y a d To ats e v a c e with som.. videotape Video Files RDBMS Crossplatform Image Collections

6 Image collection at MBARI

7 Lessons Learned Spelling Matters Constrained vocabulary required!

8 Lessons Learned Spelling Matters Constrained vocabulary required! Database >>> Files No files to manage Data always in sync Easy to share data

9 Lessons Learned Spelling Matters Database >>> Files Constrained vocabulary required! No files to manage Data always in sync Easy to share data One tool to rule them all Researchers have different needs

10 Lessons Learned Spelling Matters Database >>> Files No files to manage Data always in sync Easy to share data One tool to rule them all Constrained vocabulary required! Researchers have different needs Power users need power tools

11 Lessons Learned Spelling Matters Database >>> Files No files to manage Data always in sync Easy to share data One tool to rule them all Constrained vocabulary required! Researchers have different needs Power users need power tools Digital video is hard Many formats Large file sizes Many, many files to track

12 Lessons Learned Spelling Matters Database >>> Files No files to manage Data always in sync Easy to share data One tool to rule them all Constrained vocabulary required! Researchers have different needs Power users need power tools Digital video is hard Many formats Large file sizes Many, many files to track

13 Software Design Requirements Goal: Continue to enable scientists to use MBARI collected video for qualitative and quantitative science Accommodate future changes to digital video (futureproofing). Efficiently deliver video. Track the locations of videos. Manage videos annotations.

14 From This... videotape Video Files RDBMS Crossplatform Image Collections

15 To This... W eb Video asset manager pro Crossplatform toc ol videotape Annotation service Knowledgebase W eb pro toc ol Sharktopoda Operatingsystem specific video player Video Files Image Collections

16 Our Nearterm Goal W eb Video asset manager Annotation service Knowledgebase pro toc ol Crossplatform Discrete services A service provides a web API videotape owns its own data does one thing and does it well W eb pro toc ol Sharktopoda Operatingsystem specific video player Video Files Image Collections

17 Our Nearterm Goal W eb Video asset manager pro Crossplatform toc ol videotape Annotation service Knowledgebase W eb pro toc ol Sharktopoda Operatingsystem specific video player Video Files Image Collections

18 Our Nearterm Goal W eb Video asset manager pro Crossplatform toc ol videotape Annotation service Knowledgebase W eb pro toc ol Sharktopoda Operatingsystem specific video player Video Files Image Collections

19 Our Nearterm Goal W eb Video asset manager Annotation service Knowledgebase pro Crossplatform toc ol Apps use only services needed Programming language agnostic videotape UDP/JSON W eb pro toc ol REST/JSON Sharktopoda Operatingsystem specific video player Video Files Image Collections

20 Using a Service Python import requests import json r = requests.get(' j = json.loads(r.text) j[ name ] j[ alternatenames ] j[ rank ] j[ media ]

21 Using a Service Matlab j = webread(' j.description j.alternatenames j.rank j.media

22 Using a Service R install.packages('rjson') library('rjson') j < fromjson(readlines(' j$name j$alternatenames j$rank j$media

23 Using a Service Perl $j==0&&($c++,$c>=$a&&($c=0));$j==2&&($c ($C=$a),$C); $j==3&&($r++,$r>=@b&&($r=0));$j==1&&($r ($R=@B),$R);} KP($){push@S,shift}KJ(){pop@S 0}KX(){@S[1,2]=@S[2,1]}KR() {push@s,$s[1]}kw($$){"z".$_[0]."z,k{".$_[1]."},"}kg($){($_)=($a)=@_; y/\`/>/;w$a," X;P(J$_ J)"}KD($){($a)=@_;W$a,ZP Z.$a}KE($){($_)=($a) =@_;y/0123/>^<v/;w $_,"N $a"}';y/z/\'/;s/k/s$l /g;my($r,$c,@s); eval$_;}$_=$x=w '_','N(J?2:0)';y/_02/ 31/;$x.=$_ #r^>\"j eg"1+t,,,,l#

24 Our Nearterm Goal W eb Video asset manager Annotation service Knowledgebase vampiresquid pro Crossplatform toc ol annosaurus videotape W eb pro toc ol Sharktopoda Operatingsystem specific video player Video Files Image Collections

25 vampire squid Efficiently Deliver Video to Users Data Collection Deployment 8 hour dive is ~1TB Video Segments 32x 15min videos

26 vampire squid Efficiently Deliver Video to Users Data Collection Master ~40GB Mezzanine ~4GB Proxy ~400MB Deployment 8 hour dive is ~1TB Video Segments 32x 15min videos

27 vampire squid Efficiently Deliver Video to Users Data Archiving Capture resolution Minimal compression For detailed annotations Master ~40GB Mezzanine ~4GB Proxy ~400MB

28 vampire squid Efficiently Deliver Video to Users Data Archiving Capture resolution Minimal compression For detailed annotations Master ~40GB Compressed For generating proxies Mezzanine ~4GB Proxy ~400MB

29 vampire squid Efficiently Deliver Video to Users Data Archiving Capture resolution Minimal compression For detailed annotations Master ~40GB Compressed For generating proxies Mezzanine ~4GB Highly compressed at various resolutions For outline annotations Proxy ~400MB

30 Efficiently Deliver Video to Users Data Delivery Physical delivery mechanism Prefetch via web Master ~40GB Streaming via web Mezzanine ~4GB Streaming via web Proxy ~400MB

31 vampire squid 96 videos / deployment X 300 deployments /year ~30,000 videos / year

32 Allows Applications to Ask: 96 videos / deployment X 300 deployments /year ~30,000 videos / year vampire squid For a given deployment and/or moment in time: What videos are available? Where are they? What are their: codecs containers resolution framerate fingerprint (SHA512)

33 vampiresquid vampire squid A simple custom video asset manager. Open source docker run p 8080:8080 hononuuli/vampiresquid

34 Annotation API anno sauru s Goal: Make it easy for researchers to create and edit annotations from their own applications An evolved VARS annotation data model. Rich, flexible data model

35 Annotation API anno sauru s Goal: Make it easy for researchers to create and edit annotations from their own applications An evolved VARS annotation data model. Rich, flexible data model Want to search for any kind of squid eating any kind of Myctophidae between 200 and 400 meters in June 2007?

36 Image Annotation Supported Index is timestamp T16:11:09Z anno sauru s

37 Image Annotation Supported anno sauru s Index is timestamp T16:11:09Z Can have multiple images at one index. e.g. left/right stereo image raw/color corrected image

38 Annotations apply across all versions of a video annosaurus + vampiresquid Index into Media (Elapsedtime and/or Timecode) millis / 01:05:43:22 Master Mezzanine Proxy

39 Annotation Software Infrastructure Video and Image annotation service Annosaurus Open source docker run p 8080:8080 hohonuuli/annosaurus

40 Status Video asset manager Annotation service Knowledgebase Sharktopoda

41 Status Video asset manager Annotation service Knowledgebase Alpha ready for integration Sharktopoda Completed In Production Web API is Alpha Rewrite required Existing app to be rewritten to new APIs

42 Potential Applications Automation Video asset manager Annotation service Knowledgebase Automated Image Processing W Color correct, remove marine snow etc. eb pro toc ol Sharktopoda Register corrected images with annosaurus Operatingsystem specific video player

43 Potential Applications Automation Video asset manager Annotation service Knowledgebase Automated Detector and Classifier W Store classifier output in annosaurus eb pro toc ol Sharktopoda Operatingsystem specific video player

44 Potential Applications Extract training sets W eb Video asset manager Annotation service Knowledgebase pro toc ol Extract training sets of images from annotated video Extract Training Set (App) Sharktopoda Operatingsystem specific video player

45 Potential Applications Realtime multiusers W eb Video asset manager Annotation service Knowledgebase Video stream pro toc ol

46 Ecosystem of Tools W eb Video asset manager Annotation service Knowledgebase pro toc ol Automated Detector and Classifier Develop special tools while using existing applications.

47

48 github.com/underwatervideo underwatervideo.slack.com

49 The end