P I X E V I A : A I B A S E D, R E A L - T I M E C O M P U T E R V I S I O N S Y S T E M F O R D R O N E S

Transcription

1 P I X E V I A : A I B A S E D, R E A L - T I M E C O M P U T E R V I S I O N S Y S T E M F O R D R O N E S Mindaugas Eglinskas, CEO at PIXEVIA

2 Origins in R&D projects for Lithuanian MoD. Autonomous systems research at Vilnius University ( )

3

4 4

5 5

6 PIXEVIA FUSION Information / sensor fusion NAV OBJECTS Navigation AI based object recognition CORE Real-time imaging CORE X1 Hardware interfaces

7 Integrates AI technology in daily commercial drone operations. Uses NVIDIA Jetson TX1 and PIXEVIA for: First responder - Person detection & tracking - Vehicle detection & tracking - License plate recognition - Privacy masking Inspection - Object detection / Classification - Recognizing condition deviations - Determine fault location - Distance measurement Real-time onboard mapping

8 Drones with AI for security applications. Uses NVIDIA Jetson TX1 and PIXEVIA system: carrierboard, pipelines and object classification.

9 Fully autonomous control Main driving forces Fully autonomous information processing powered by machine learning Fusion of information from different machines and data sources

10 U S E C A S E S F O R A I P O W E R E D D R O N E S

11 Surveillance: defence / law enforcement / private security cars (with number plate recognition) trucks boats people heavy machinery

12 Automated infrastructure inspections Automated inspection: power lines utility poles insulators foreign objects

13 Inventory management C o n t a i n e r s W a g o n s L o c o m o t i v e s C a r s M a t e r i a l s P a c k a g e s

14 Smart city Parkings Car flows Persons Security

15 S O F T W A R E A R C H I T E C T U R E / D E C I S I O N S / L E S S O N S L E A R N E D

16 CORE Intelligent real-time imaging: collection, transformation, communication Interfaces with sensors and file formats (input) On-board image processing. Image processing pipelines Image processing modules Geographical metadata Distributed processing Industry standards

17 OBJECTS Object detection, properties of objects (size, speed, coordinates) Cars / Trucks License plate recognition People Face detection Other objects: boats, environment, etc.

18 NAVIGATION Visual position estimation Visual odometry Image-map matching Foreign object detection on the landing site

19 INFORMATION FUSION Fusion of information in real-time Information fusion from different sources (drones, cameras) Vizualization on 3D map

20 1. Interfaces USB3 / USB2 SD-card UART HDMI GPIO CAN minipcie CSI I2C SPI PWM Accel, gyro, compass, barometer

21 2. Sensors I n d u s t r i a l b l o c k c a m e r a L o n g r a n g e d i g i t a l d a t a l i n k ( M i c r o h a r d ) T h e r m a l i m a g i n g c a m e r a G P R S d a t a l i n k G i m b a l c o n t r o l W i f i, l o c a l c o m m u n c a t i o n s U l t r a f a s t c a m e r a f o r v i s u a l o d o m e t r y

22 3. Modular architecture Spaghetti type of integration will kill any bigger project OpenVX / Gstreamer NVIDIA provides accelerated Gstreamer modules for encoding DDS for real time communication Processing can be changed before the mission or during the flight

23 4. Simple description of image processing pipeline and tools

24 5. Geographical metadata Every video frame contains geographical information: - image corners with coordinates, position of drone/camera, angles of sensors, camera geometry. Allows integration with GIS systems, provides data for later learning.

25 6. Hardware and software frameworks used HARDWARE NVIDIA Frameworks Other frameworks

26 7. GPU based moving object detection VisionWorks (OpenVX graph) CUDA OpenCV with CUDA optimization Caffe cudnn

27 8. Region detection with convolutional neural networks cudnn Caffe Fully convolutional network Single shot detection Filtering after detection

28 9. Self-adaptation mode - sleep Real-time Sleep

29 Self-adaptation during the sleep Slow models Information from the fast neural models (big neural nets, other computer vision algorithms, physics)

30 10. Simulation and learning

31 31 MobilEye 600 people doing labeling MobilEye photo

32 AI (neural networks / SGD) limitations - It can win all games (if can play more than human plays through all life) - Recognize images (if can see more than human sees through all life) Terrible performance with small datasets

33 Simulated data from unity3d

34 Simulation for data fusion

35 11. 3D reconstructions during the flight points, 90 seconds on Jetson TX1, 4 images

36 12. Visual position estimation Visual odometry Terrain segmenation Deep neural networks training item dataset. Convolutional neural network, cudnn Image - map matching Multiple hyphothesis tracking

37 13. User interfaces in the embedded system Web / Qt via HDMI and datalinks

38 Current status PIXEVIA version 0.5 Technology preview

39 AI for autonomous systems FUSION Information / sensor fusion OBJECTS AI based object recognition NAV Navigation CORE Real-time imaging CORE X1 Hardware interfaces

40 P I X E V I A : A I B A S E D, R E A L - T I M E C O M P U T E R V I S I O N S Y S T E M F O R D R O N E S Mindaugas Eglinskas, CEO at PIXEVIA