Getting Started with Bellus3D Face Camera SDK

Transcription

1 Getting Started with Bellus3D Face Camera SDK Revision History Version Date Description Author a 2017/12/20 1. Use Service App to support Bellus3D camera connection 2. Recommend to use new Synchronized method for DepthCamera and DeviceCamera class a 2018/02/07 1.Fixed generated model tilt bug 2.Face model cut on the chin bug 3.Texture off bug Please Note: current library doesn t support Android a 2018/03/20 1.Extended expiration date for Face Camera Prototype 2.Fixed sample app open device camera failed bug Please Note: Face Camera Prototype developer must upgrade to this version before April 1 st a 2018/04/27 1.Updated camera driver to support Android 8 2.Support.stl 3d format and.zip output 3. Improved scan results when using registration bite plates for Dental scanning 4.Improved sample app error handling 5.Added sample code for output jialiu@bellus3d.com jialiu@bellus3d.com jialiu@bellus3d.com jialiu@bellus3d.com

2 Table of Contents 1. FOUR MAJOR COMPONENTS DEPTHCAMERA DEVICECAMERA HEADSCANNER HEADPROCESSOR COMPONENT STATES CHANGE DEPTH CAMERA STATE DEVICE CAMERA STATE HEAD SCANNER STATE HEAD PROCESSOR STATE COMPLETE PROCESS OF GETTING A FACE MODEL CONNECT DEPTH CAMERA AND OPEN DEVICE CAMERA TRACKING SCANNING PROCESSING CLOSE SESSION SETTINGS DEVICE CAMERA HEAD SCANNER HEAD PROCESSOR OUTPUT HEADMESH CLASS DIFFERENT FORMAT UPDATE WINDOWS SDK NEW DEVELOPER EXISTING DEVELOPER WEBCAM CONNECTIONS USING EMBEDDED WEBCAM USING EXTERNAL WEBCAM...9

3 FOUR MAJOR COMPONENTS 1.1. DepthCamera Controls the Bellus3D depth camera DeviceCamera Controls Controls PC embeded Device Camera or external Device Camera HeadScanner Performs 3D face tracking and scanning. It operates DepthCamera for capturing depth images, color camera on PC or external device for color images HeadProcessor Performs 3D face model building. It processes images captured by HeadScanner and generates a 3D face model. COMPONENT STATES CHANGE Make sure that you call each function at the correct state Depth Camera State Disconnected State Initial State of Depth Camera when it s unplugged Connecting State Intermediate state when depth camera is connecting Connected State Happens when DepthCamera.connect() successfully Opening State Intermediate state when depth camera is opening Open State Happens when DepthCamera.open() successfully Starting State Intermediate state when depth camera is connecting Steaming State Happens when DepthCamera.startStream() successfully Stopping State Intermediate state when depth camera is stopping Closing State

4 Intermediate state when depth camera is closing Disconnecting State Currently not used Disconnected Connecting DepthCamera.connect() Steaming Connected DepthCamera.startStream() Starting Opening DepthCamera.open() DepthCamera.stopStream() Stopping Closing Open DepthCamera.close() 2.2. Device Camera State Closed State Happens when DeviceCamera.closeSync() successfully Opening State Intermediate state when device camera is opening Open State Happens when DeviceCamera.openSync() successfully Starting State Intermediate state when device camera is connecting Streaming State Happens when DeviceCamera.startStream() successfully Stopping State Intermediate state when device camera is stopping Closing State Intermediate state when device camera is closing.

5 Closed Closing Opening DeviceCamera.openSync() DeviceCamera.startStream() Steaming Starting Open DeviceCamera.closeSync() Stopping DeviceCamera.stopStream() 2.3. Head Scanner State Inactive State Initial state, happens after scanning completed as well Tracking State Happens after HeadScanner.startTracking() Scanning State Happens after HeadScanner.startscanning() Stopping State Intermediate state when HeadScanner is stopping, happens after HeadScanner.stop() HeadScanner.startScanning() Stopping HeadScanner.stop() Inactive Stopping HeadScanner.startTracking() HeadScanner.stop() Scanning Tracking 2.4. Head Processor State Inactive State

6 Initial state, happens after processing completed / cancelled as well Processing State Happens after HeadProcessor.startProcessing() HeadProcessor.startProcessing() Inactive Processing HeadProcessor.cancelProcessing() COMPLETE PROCESS OF GETTING A FACE MODEL 3.1. Connect Depth Camera and Open Device Camera If user choose to use middle camera from Bellus3D Camera, don t open DeviceCamera Tracking Make sure the following before calling HeadScanner.startTracking() a. DepthCamera is in Open state b. DeviceCamera is in Open state c. HeadScanner is in Inactive state d. HeadProcessor is in Inactive state Tracking State Information o FacePosition represents current face position Scanning Make Sure TrackingState s FacePosition is READY_TO_SCAN before calling HeadScanner.startScanning() Scanning State Information o targetprogress represents where your face should be instructively [0, 4]: 0 is in the middle, 1 is most left, 2 is in the middle, 3 is most right, 4 is back to the middle currentposition represents your face rotation angle [-90, 90 ] corresponding to [-1, 1] 3.4. Processing Make Sure HeadScanner InactiveState s sessiondatacompleted is READY_TO_SCAN before calling HeadScanner.startScanning() Processing State Information o progress represents the percentage of current processing progress. o processingstage represents if current state is processing data or recalibrating camera

7 3.5. Close Session SETTINGS Clean up resources for each component. Depending on their State, do following as shown in sample app HeadScanner.stop(); HeadProcessor.cancelProcessing(); DeviceCamera.stopPreview(); DeviceCamera.closeSync(); DepthCamera.stopStreamSync(); Make sure to set up setting to each component before start capture, tracking or scanning by e.g. HeadScanner.setHeadScannerSettings(HeadScannerSettings); 4.1. Device Camera CameraID represents if you are using front or back camera in your device 4.2. Head Scanner CaptureInMemory: true means never save depth images, processing in memory ScanningMode: capture time is increasing for 180, 270, and 270 ext. o 180 means scanning from [-90, 90] o o 4.3. Head Processor OUTPUT 270 [-135, 135], it would capture more data behind your ears 270ext is more for using back camera to do hand holding capture. MeshEnhancement: Choose to enhance or not by color, landmarks or both MeshResolution: Process output to LD, SD or HD resolution MeshSmoothness: How much smooth enhancement to apply to the output model, range from[0, 10] RemoveRawImages: choose if removing captured raw images after processing, default is true TextureSource: choose texture source for texture from Bellus3D color camera or color camera on your device EnableAutoOutput: choose auto output 3D models in format obj. By default, it s enabled EnableRecalibration: choose if doing calibration before processing the model. By default, it s 5.1. HeadMesh Class disabled. Used when you see very bumpy scan Landmarks Contains 77-point ASM facial landmarks Landmarks2d is a Point2f Array. Each Point2f has x and y coordinates.

8 Landmarks3d is a Point3f Array. Each Point3f has x, y and z coordinates Polygons Faces is a float array that contains indices info. Every 3 forms a triangle Vertices is a float array that contains vertices info. Every 8 forms one vertex info. First 3 are x, y and z position coordinates. Following 3 are normal coordinates. And last 2 are texture UV coordinates. getmeshpolys() function include simplify process when gets called at the first time. Keep in mind that it takes some time to finish FrontalImage This is the first captured color image from device camera. You may use it as a thumbnail to for your face model 5.2. Different Format b3d file Use write() to output.b3d file. It includes all the information for HeadMesh obj file Use export() with ***.obj as filename input to export obj file. By default, it would also output material file ***.mtl and texture file ***.jpg Use writelandmarks() with ***.obj as landmarkfilename input to export landmarks file stl file Use export() with ***.stl as filename input to export stl file. UPDATE WINDOWS SDK 6.1. New Developer Follow README file inside your WindowSDK package. Install Inuitive Drivers Setup Environment Variables 6.2. Existing Developer Reinstall Inuitive Driver InuDev_SP_ _X64.exe Make sure your INUITIVE_PATH is still correct in your environment variables Read sample app source code carefully since error handling was improved a lot WEBCAM CONNECTIONS 7.1. Using Embedded Webcam the USB port of Laptop can be either USB 2.0 or USB 3.0

9 the Type-C connector of B3D3 can be either USB 2.0 side or USB 3.0 side 7.2. Using External Webcam Connect webcam and B3D3 to Laptop/PC directly o the USB port of Laptop/PC can be either USB 2.0 or USB 3.0 o the Type-C connector of B3D3 can be either USB 2.0 side or USB 3.0 side Not recommend to connect webcam and B3D3 to Laptop/PC via a USB hub.