Henry Chu. Professor, School of Computing and Informatics Executive Director, Informatics Research Institute. University of Louisiana at Lafayette

Transcription

1 Henry Ch Professor, School of Compting and Informatics Exective Director, Informatics Research Institte University of Loisiana at Lafayette

2 Informatics Research Institte We condct research in data science to nleash the potential of Big Data for the benefit of society in sch areas as health, crisis response, commnity secrity & resiliency, and smart & connected commnity Smart and Connected Commnity Crisis Research Pblic Safety Open Data Predictive Analytics Information Exchange Health Informatics Big Data Platform Cyber Physical Systems Data Science and Big Data Analytics

3 Leveraging Data for Health Clinical Data for Research Trials Collect, Connect, Aggregrate, and Analyze Clinical Data Registry Pblic Health Data for Analytics

4 Intelligent Infrastrctre Fondation for increased safety and resilience Improved efficiencies and civic services Broader economic opportnities and job growth Deep embedding of sensing, compting, and commnications capabilities into traditional rban and rral physical infrastrctres sch as roads, bildings, and bridges

5 Intelligent Pblic Safety and Secrity Real time crowd analysis Threat detection; dispatch pblic safety officers Anticipate vlnerable settings and events New commnication and coordination response approaches

6 Intelligent Disaster Response Real time water levels in flood prone areas Timely levee management and evacations as needed Anticipate flood inndation with low-cost digital terrain maps Inform vlnerable poplations

7 Crisis Research Points of Distribtion and Spply Chain Optimization Hman Geography Mapping Loisiana Hazard Information Portal Bsiness Emergency Operations Center Geo-Referenced Wireless Emergency Alerting Conseqence Analysis of Natral Gas Pipeline Disrptions Fel Demand & Spply Prediction for Regional Evacation

8 Big Data Modeling frameworks, Analytics and Tools for Disaster Prediction and Management Probabilistic modeling of complex events to develop predictive analytics and enhance the capabilities for appropriate and adaptive response, and to refine response planning. Mltilevel, mltiscale modeling methods for nderstanding factors that contribte to or ndermine commnity resilience Captre and visalize data elements reflecting different aspects of a commnity, from physical geography to bilt infrastrctre to activities, entities, events, and processes on the infrastrctre Research into protocols and methods for ensring both reliability and privacy of data collection and analytics dring emergency sitations, disasters, and crises.

9 Virtal Reality Content Creation by Deep Learning of Video Clips Joe Reed The NeMachine Presented by Henry Ch University of Loisiana at Lafayette The NeMachine LLC

10 Motivation Emergencies that impact bildings Fire Mass killings Floods Hrricanes Tornadoes Toxic gas releases Hostage sitations Chemical spills Explosions Civil distrbances Utility failres EMS calls Atomatic fire/secrity alarms Active threat policy/protocol for Dispatch Sorce: Eagle View Technologies

11 Motivation: State-of-the-art Soltion Professional captre of interior imagery and LiDAR, or laser scanning, data Post-process data with 360 panoramic imagery and LiDAR data point clod Generation of 3D floor plan models with room attribte data Links to MSDS sheets, images and URLs, if available WHAT IF WE CANNOT DEPLOY A LiDAR UNIT? Sorce: Eagle View Technologies

12 Research challenges in creating virtal objects, hmans and environments especially for enhancing physical and interactive realism High-fidelity, intractable 3D content, sch as intelligent virtal hmans and interactive virtal environments, drives the creation of compelling graphics innovations sch as agmented reality (AR) and virtal reality (VR) applications. Creating sch interactive, smart virtal content goes beyond the traditional graphics goal of attaining visal realism, giving rise to a new wave of exciting opportnities in compter graphics research. This new research frontier aims to close the loop between 3D scanning and content creation, 3D scene and object nderstanding, virtal hman modeling, physical simlations, 3D graphics researchers, as well as experts in AR/VR, compter vision, robotics and artificial intelligence

13 With the rapid changes occrring in the field there needs to be a framework for incorporating different modal data into the development pipeline. To redce cost and man power, we believe that a tool agmented with Deep Learning can learn tasks needed to create VR content and can learn to do it faster and more efficiently than today s hand crafted algorithms.

14 Topics that need to be addressed in the evoltion of VR technology Affordance analysis of scenes and objects Physically-gronded scene interpretation Physics-based design of objects cost effectively to provide haptic feel (e.g., 3D printing of special objects, treadmills, moving walls or stairs, terrain like water, rocks, grass, wind) Cognitive, perceptal and behavioral modeling of virtal hmans Virtal hman interaction and hman perception Biomechanics modeling and simlation of hman body Artificial life and crowd simlations Novel applications of AR/VR/haptic devices

15 3D Scene Reconstrction from Video Clip Handcrafted soltions extensively stdied Typically rely on featre detection, featre matching (typically poor accracies), matched pair prning, soltions of transformation parameters, and stratified reconstrction

16 3D Scene Reconstrction from Video Clip Handcrafted soltions typically based on Featre points detection and matching, sally very error-prone Use 3D parameters to eliminate mis-matched pairs Stratified reconstrction to create sparse and dense data points

17 Deep Learning Deep learning spported by GPU processing power has led to classification, detection, and segmentation of image and video data with spectaclar reslts in the past few years

18 Pilot Work We hypothesize that sing a Deep Learning soltion, we can recover sfficient information labeled image regions with srface normals and depth information to enable s to recover a 3D scene that can be sed in a virtal reality rendering sing digital assets Deep Learning Analysis 3D Synthesis

19 Qick Example Individal frames are grabbed and resized to 320 by 240 still images

20 From Deep Learning VGG Network Color coded srface normal VGG Network RGB still frame Color coded depth map Color coded segmentation otpt

21 From Deep Learning VGG Network Color coded srface normal VGG Network RGB still frame Color coded depth map Color coded segmentation otpt

22 Key Frame Video Clip

23 Sample Frame Otpt Color-coded distances from camera Labels: Floor Spport Frnitre Props Color-coded srface normal vectors

24 Key Frame Video Clip Analysis Reslts

25 Sample Frame Otpt Color-coded distances from camera Labels: Floor Spport Frnitre Props Color-coded srface normal vectors

26 From Image to 3D Planes Srface normals and depth maps are qite accrate Labels of floor and spport are sally correct Large horizontal srface sometimes mistaken as floor Horizontal srface normals seem to be more accrate than those of other srfaces

27 From Image to 3D Planes Horizontal plane [ , , ], [ , , ], [ , , ], [ , , ], [ , , ], [ , , ] Clstering of all srface normal vectors sing k- means with k = 6

28 From Image to 3D Planes Vertical planes [ , , ], [ , , ], [ , , ], [ , , ], [ , , ], [ , , ] Clstering of all srface normal vectors sing k- means with k = 6

29 From Images to 3D Planes We go back to the srface normal map and label each point with the clster id (0, 1, 2,, 5) that it belongs to Use the clster id label ( horizontal, vertical, etc) to label each point

30 3D Planes from Images Goal is to extract these parameters of each plane in the scene being imaged Orientation Srface normal in world coordinates Position Scale Up to scale Up to scale

31 3D Planes from Images Goal is to extract these parameters of each plane in the scene being imaged Orientation Position Scale We rotate all points (planes) so that the floor (horizontal) plane points p, as the z-axis We can arbitrarily rotate all points so that one of the walls (vertical spport) points as the x- or y- axis This rotates the scene to align with the world coordinate from the camera coordinate

32 3D Planes from Images Goal is to extract these parameters of each plane in the scene being imaged Orientation Position Scale Camera Recovered depth data We se the depth information to position the plane in the scene, p to scale x Srfaces identified by srface normals z

33 3D Planes from Images Goal is to extract these parameters of each plane in the scene being imaged Orientation Position Scale Camera How do we find x, y, z? Grid the space and find the set that agrees with the recovered depth data We se the depth information to position the plane in the scene, p to scale x Hypothesized plane that is not consistent with depth data z

34 3D Planes from Images Goal is to extract these parameters of each plane in the scene being imaged Orientation Position Scale Camera Recovered depth data We se the depth information to establish the size of the plane in the scene, p to scale, in different directions s x Srfaces identified by srface normals

35 Reconstrcted 3D Srfaces Two views of srfaces in the 3D scene that are consistent with the reslts obtained from the Deep Learning networks

36 Ongoing Work Fine tne the reconstrction process for one frame Rectify reconstrction from different viewpoints obtained by different images Connect asset data to insert into the scene (replacing the placeholding srfaces) Use an initial classification step to identify scene category (indoor, office, bedroom, etc) to constrain the deep learning network for better accracy and inference efficiency

37 How to drive the synthesis of objects? Deep Learning Analysis 3D Synthesis Originally we planned to se a classify-and-pick approach Use DL to perform object detection Use object label to search a 3D parts database Pivoted to se a Generative Adversarial Network approach

38 Generative Learning Generative learning is a theory that involves the active integration of new ideas with the learner's existing schemata. The main idea of generative learning is that, in order to learn with nderstanding, a learner has to constrct meaning actively -Wikipedia A classifier tries to determine the best p(y x) where y is the label and x is the inpt A generative learning system tries to determine the best p(y,x) where y is the label and x is the inpt

39 Generative Adversarial Network (GAN) Goal is to estimate the nderlying probability density of pdata so that the system can generate any data that are consistent with the original pdata Image synthesis from an image collection

40 GAN: Simple Example Original data density

41 GAN: Simple Example Generator Discriminator Real? Train both the discriminator and generator networks together Learning converges when the discriminator chooses 0.5

42 GAN: Simple Example Generator Discriminator Real?

43 3D Synthesis GAN

44 3D Synthesis GAN

45 3D Synthesis GAN Arithmetic in Latent Space

46 3D Synthesis GAN Arithmetic in Latent Space

47 How to drive the synthesis of objects? Deep Learning Analysis 3D Synthesis Latent variables Train GAN for 3D Synthesis

48 Or Ultimate Goal To establish a framework to be sed for a work flow pipeline arond creating VR content from 2D legacy movies and their accompanying assets (i.e. script, adio descriptor, closed caption, CG/VFX files) Use deep learning tools to be more efficient in time and workforce With more training, adding newer models, and scaling p the GPU compte we can achieve a prodct soltion that can integrate with the existing pipeline sed for CG Movies and VR content

49 The Challenge The main challenge is for the compter to recognize low-level and high-level activities in the context of a scene Factors that create the challenge are accrate depth estimation, video segmentation into scenes, then into objects both rigid and non-rigid which are frther segmented and classified into data strctres that can be then sed to generate the desired reslt Advances in compter vision and machine learning techniqes and algorithms have improved over years, inclding more accrate eye, face and head tracking and motion captre Video recording of hman activities can be of se for potential marketing research (for example, how do consmers move in a store and where they stay the most), bsiness srveillance, robotic assembly or as datasets for biologists, sociologists and psychologists to observe hman motion and overall action

50 Or Soltion An ensemble of neral nets will be sed for the prodction soltion of inferencing the immersive experience from the legacy video We also se a few commercial software and open-sorce software applications: Unreal Engine 4, Poser Pro 11, and Blender+Lxrender, & Nvidia s Deep Learning SDK, Ato Desk Maya, Micro Soft Kinect SDK, VR Works, Game Works, all rnning on Nvidia TITAN X Pascal GPUs. We have created a framework for the workflow pipeline: the UI of the system is bilt from Unreal Engine 4 with the deep learning embedded into the engine pipeline The pipeline takes the video and other inpt in and sing well known pblished algorithms and models for varios tasks creates new data strctres to be sed in the generation of the immersive content The workflow consist of a well defined ensemble of neral nets that each prodce one set of the data that is needed for the following steps in the process After each neral net otpts its data it is sent into a semi-spervised neral net that allows a hman to perform a gided qality assrance process to correct any errors with a series of mose clicks, spoken words, or mose or pen drawn lines

51 Or Ensemble Applications Or ensemble wold allow for the creations of VR cinematic experiences and as new VR hardware comes on the market allow for a gamification of VR cinematic content not nlike what is described as Sync Sims in the book Ready Player One We se the Unreal Engine cstomized with plgins to enable atomation to agment the hman work flow pipeline We chose UE4 becase of its open sorce and the ble printing, cinematic seqencer, and VR editor We chose Poser Pro 11 for its Unreal Portal Development Environment

52 UE4 Pipeline FBX is an Atodesk file format that provides interoperability between digital content creation applications sch as Atodesk Motion Bilder, Atodesk Maya, and Atodesk 3ds Max Atodesk Motion Bilder software spports FBX natively, while Atodesk Maya and Atodesk 3ds Max software inclde FBX plg-ins. Unreal Engine featres an FBX import pipeline which allows simple transfer of content from any nmber of digital content creation applications that spport the format. The advantages of the Unreal FBX Importer over other importing methods are: Static Mesh, Skeletal Mesh, animation, and morph targets in a single file format. Mltiple assets/content can be contained in a single file. Import of mltiple LODs and Morphs/Blendshapes in one import operation. Materials and textres imported with and applied to meshes. Poser's Unreal Portal Development Environment

53 Ble Print Editor

54 Poser's Unreal Portal Development Environment

55 FAIR Facebook AI Research We were able to replicate this research from FAIR inclding UR torch and also UnrealCV which both interface with Unreal Engine 4 and serve as an interface to deep learning and open CV libraries and we fond that we cold achieve similar reslts which we will provide screen shots of or findings The algorithm DeepMask1 segmentation framework copled with the newwsharpmask2 segment refinement modle. Together, they have enabled FAIR s machine vision systems to detect and precisely delineate every object in an image. The final stage of their recognition pipeline ses a specialized convoltional net, which they call MltiPathNet3, to label each object mask with the object type it contains (e.g. person, dog, sheep).

56 Or Ensemble Applications Movie Scene Object Segmentation Captred in Unreal Engine with Embedded Deep Learning Movie Scene Example

57 Or Ensemble Application Reconstrcted VR Scene sing Unreal Engine with the Torch Plgin with embedded Deep Learning

58 Virtal Reality of Rendered Scene UnrealCV is a project to help compter vision researchers bild virtal worlds sing Unreal Engine 4 (UE4).

59 Virtal Reality of Rendered Rendering UE4

60

61

62 Three consective frames of a cat video: Deepmask from FAIR reslts

63 Or Ultimate Goal Establish a DL-based framework to be sed for a work flow pipeline arond creating VR content from 2D video With more training, adding newer models, and scaling p the GPU compte we can achieve a prodct soltion that can integrate with the existing pipeline sed for CG Movies and VR content Althogh the focs of this presentation is on the inpt of 2D video into an ensemble of neral nets to create an immersive experience it can easily be adapted for other application Some of them are video srveillance, video retrieval and hman-machine interaction

64 For More Information Henry Ch Informatics Research Institte University of Loisiana at Lafayette