Automated Driving System Toolbox 소개

Transcription

1 1

2 Automated Driving System Toolbox 소개 이제훈차장 2017 The MathWorks, Inc. 2

3 Common Questions from Automated Driving Engineers vehicle How can I Visualize Sensor data? How can I design and verify Perception algorithms? How can I design and verify Sensor fusion? 3

4 Common Questions from Automated Driving Engineers vehicle How can I Visualize Sensor data? How can I design and verify Perception algorithms? How can I design and verify Sensor fusion? 4

5 Automated Driving Sensor data Camera Radar Object Detection Lidar Sensor fusion & Tracking IMU IMU: Inertial Measurement Unit 5

6 Automated Driving Sensor data Camera (640 x 480 x 3) Radar Detector SensorID = 2; Timestamp = ; Vision Detector NumDetections = 23; SensorID = 2541; Detections(1) Timestamp = ; TrackID: Lidar ( x 3) NumDetections = 2546; TrackStatus: Detections(1) Position: [ ] TrackID: Lane Detector Velocity: [ ] Classification: Left 5 Amplitude: Position: IsValid: [ ] Detections(2) Velocity: 255 Confidence: [ ] TrackID: Size: 255 BoundaryType: [ ] TrackStatus: Detections(2) Offset: Position: [ ] TrackID: Vision based 253 HeadingAngle: Inertial Measurement Velocity: [-8.02 Unit ] Classification: 253 Curvature: lane 253 detector Timestamp: Amplitude: Position: [ ] Right Velocity: Detections(3) Velocity: [ ] TrackID: IsValid: YawRate: TrackStatus:

7 7

8 Visualize Sensor data in vehicle coordinates ISO 8855 vehicle axis coordinate system Positive x is forward Positive y is left y x %% Plot in vehicle coordinates ax2 = axes(... 'Position',[ ]); bep = birdseyeplot(... 'Parent',ax2,... 'Xlimits',[0 45],... 'Ylimits',[-10 10]); legend('off'); Plot in vehicle coordinates with birdseyeplot 8

9 Visualize Sensor data - expected coverage area %% Create coverage area plotter covplot = coverageareaplotter(bep,... 'FaceColor','blue',... 'EdgeColor','blue'); %% Update coverage area plotter plotcoveragearea(covplot,... [sensorparams(1).x... % Position x sensorparams(1).y],... % Position y sensorparams(1).range,... sensorparams(1).yawangle,... sensorparams(1).fov(1)) % Field of view Plot sensor coverage area with coverageareaplotter 9

10 Visualize Sensor data - detected objects (vehicle coordinates) %% Create detection plotter detplot = detectionplotter(bep,... 'MarkerEdgeColor','blue',... 'Marker','^'); %% Update detection plotter n = round(currenttime/0.05); numdets = vision(n).numobjects; pos = zeros(numdets,3); vel = zeros(numdets,3); labels = repmat({''},numdets,1); for k = 1:numDets pos(k,:) = vision(n).object(k).position; vel(k,:) = vision(n).object(k).velocity; labels{k} = num2str(... vision(n).object(k).classification); end plotdetection(detplot,pos,vel,labels); Plot vision detections with detectionplotter detectionplotter can be used to visualize vision detector, radar detector, and lidar point cloud 10

11 Visualize Sensor data - detected objects (image coordinates) %% Bounding box positions in image coordinates imboxes = zeros(numdets,4); for k = 1:numDets if vision(n).object(k).classification == 5 vehposlr = vision(n).object(k).position(1:2)'; imposlr = vehicletoimage(sensor, vehposlr); boxheight = 1.4 * 1333 / vehposlr(1); boxwidth = 1.8 * 1333 / vehposlr(1); imboxes(k,:)=[imposlr(1) - boxwidth/2,... imposlr(2) - boxheight,... boxwidth, boxheight]; end end %% Draw bounding boxes on image frame frame = insertobjectannotation(frame,... 'Rectangle', imboxes, labels,... 'Color','yellow','LineWidth',2); im.cdata = frame; Draw boxes with insertobjectannotation Transform vehicle to image coordinates with vehicletoimage 11

12 Learn more about visualizing vehicle data by exploring examples in the Automated Driving System Toolbox Plot object detectors in vehicle coordinates Vision & radar detector Lane detectors Detector coverage areas Transform between vehicle and image coordinates Plot lidar point cloud 12

13 Common Questions from Automated Driving Engineers vehicle How can I Visualize Sensor data? How can I design and verify Perception algorithms? How can I design and verify Sensor fusion? 13

14 Automated Driving Perception Algorithms Pedestrian Detection Vehicle Detection Object Detection: Locate and classify object in image 14

15 MATLAB Tools to Train Detectors Images Label Ground Truth Ground Truth Train detector Object detector imageds = imagedatastore(dir) Easily manage large sets of images - Single line of code to access images - Operates on disk, database, big-data file system 15

16 MATLAB Tools to Train Detectors Images Label Ground Truth Ground Truth Train detector Object detector Automate Labeling of Ground Truth 16

17 MATLAB Tools to Train Detectors Images Label Ground Truth Ground Truth Train detector Object detector Design object detectors with the Computer Vision System Toolbox Machine Learning Deep Learning Aggregate Channel Feature Cascade R-CNN (Regions with Convolutional Neural Networks) Fast R-CNN Faster R-CNN trainacfobjectdetector traincascadeobjectdetector trainrcnnobjectdetector trainfastrcnnobjectdetector trainfasterrcnnobjectdetector 17

18 Designing Perception Algorithms Computer Vision Algorithms for Automated Driving Vehicle Detection Deep learning and ACF based (pre-trained) Pedestrian Detection ACF and HOG/SVM based (pre-trained) 18

19 Designing Perception Algorithms Additional Computer Vision Algorithms for Automated Driving Vehicle detection with distance estimation using mono-camera Lane Detection and Classification RANSAC-based lane boundary fitting Lane boundary visualization 19

20 Designing Perception Algorithms LiDAR Processing Algorithms 20

21 Example of Vision System Detection How can I verify this detection is correct? 21

22 Ground truth labeling to Train Detectors Images Label Ground Truth Ground Truth Train detector Object detector Ground truth labeling to Evaluate Detectors Images Label Ground Truth Ground Truth Evaluate detections Detections 22

23 23

24 Learn more about verifying perception algorithms by exploring examples in the Automated Driving System Toolbox Train object detector using deep learning and machine learning techniques Label detections with Ground Truth Labeler App Extend connectivity of Ground Truth Labeler App 24

25 Common Questions from Automated Driving Engineers vehicle How can I Visualize Sensor data? How can I design and verify Perception algorithms? How can I design and verify Sensor fusion? 25

26 Automated Driving Sensor fusion with radar and vision Can we fuse detections to better track the vehicle? 26

27 27

28 Sensor fusion framework Sensor fusion Framework Object Detections Track Manager Tracking Filter Tracks Time Measurement Measurement Noise Assigns detections to tracks Creates new tracks Updates existing tracks Removes old tracks Predicts and updates state of track Supports linear, extended, and unscented Kalman filters Time State State Covariance Track ID Age Is Confirmed Is Coasted 28

29 Sensor fusion - Data Association Radar Object Vision Object Radar Vision V 1 V 2 R 1 R 2 R m Pairs of visions and associated radars Assignments V 1 + R 2 V 2 + R 1 V n + R m Fusion f(v 1 ) + f(r 2 ) f(v 2 ) + f(r 1 ) f(v n ) + f(r m ) Fused Object List V n costmatrix [assignments, unassignedvisions, unassignedradars] =... assigndetectionstotracks(costmatrix, param.costofnonassignment); 29

30 Sensor fusion - Kalman Filter Initial state & covariance ˆx 0 P 0 Output of updated state xˆ k P k Previous state & covariance ˆ 1 x k P k 1 k k 1 Current becomes previous Time Update ( Predict ) (1) Predict state based on physical model and previous state xˆ k Axˆ k 1 Buk wk (2) Predict error covariance matrix P k AP k 1 A T Q Measurement Update ( Correct ) (1) Compute Kalman gain K k P k H T (HP k H T R) 1 (2) Update estimate state with measurement?x k?x k K k (z k H?x k ) (3) Update the error covariance matrix P k (I K k H) P k K : Kalman gain R : Sensor noise covariance matrix (measurement error) Measurement z k v H u w P Q E[ ww k ek : Control variable matrix : Process (state) noise Hx E[ e x k k k ek ] xˆ T minimize P k T v : Measurement noise ] k k : Process (state) covariance matrix (estimation error) : Process noise covariance matrix A : State matrix relates the state at the previous, k-1 to the state at the current, k : Output matrix relates the state to the measurement From sensor spec or experiment 30

31 Sensor fusion - Kalman Filter Initial state & covariance ˆx 0 P 0 Previous state & covariance ˆ 1 x k P k 1 k k 1 Current becomes previous Time Update ( Predict ) [z_pred,x_pred,p_pred] = predict(obj) z_pred : prediction of measurement x_pred : prediction of state P_pred : state estimation error covariance at the next time step Predicted state x_pred Output of updated state x_corr P_corr Measurement Update ( Correct ) Current Measurement Linear KF (trackingkf) [z_corr,x_corr,p_corr] = correct(obj,z) Extended KF (trackingekf) z_corr : correction of measurement x_corr : correction of state P_corr : state estimation error covariance zunscented KF (trackingukf) Constant velocity initcvkf initcvekf initcvukf Constant acceleration initcakf initcaekf initcaukf Constant turn Not applicable initctekf initctukf 31

32 Synthesize Driving Scenario for Sensor fusion Simulated data for worst-case scenarios Environment Models (Roads and actors) OEM specific test scenarios Sensor Models Fail Operation test scenarios Scenarios identified from real world test drive data 32

33 Synthesize Driving Scenario for Sensor fusion %% Create a new scenario s = drivingscenario('sampletime', 0.05); %% Create road road(s, [ 0 0;... % Centers [x,y] (m) 45 0],... 5); % Width (m) road(s, [35 20; ],... 5); Specify road centers and width as part of a drivingscenario %% Plot scenario p1 = uipanel('position',[ ]); a1 = axes('parent',p1); plot(s,'parent',a1,... 'Centerline','on','Waypoints','on') a1.xlim = [0 45]; a1.ylim = [-6 20]; 33

34 Synthesize Driving Scenario for Sensor fusion %% Add ego vehicle egocar = vehicle(s); waypoints = [ ;... % [x y] (m) ; ; ; ; ]; speed = 13.89; % (m/s) = 50 km/hr path(egocar, waypoints, speed); %% Play scenario while advance(s) pause(s.sampletime); end Specify ego vehicle path using waypoints and speeds 34

35 Synthesize Driving Scenario for Sensor fusion %% Add child pedestrian actor child = actor(s,'length',0.24,... 'Width',0.45,... 'Height',1.7,... 'Position',[40-5 0],... 'Yaw',180); path(child,... [30 15; 40 15],... % Waypoints (m) 1.39); % Speed (m/s) = 5 km/hr Specify pedestrian actor size and path %% Add Target vehicle targetvehicle = vehicle(s); Specify target vehicle with varying speed path(targetvehicle,... [44 1; -4 1],... % Waypoints (m) [5 ; 14]); % Speeds (m/s) 35

36 Synthesize Driving Scenario for Sensor fusion Measurement rate Mounting position on car Most common params, e.g. coverage More params (resolution, bias, etc.) 36

37 Euro NCAP TEST PROTOCOL AEB VRU systems Car-to-VRU Nearside Child (CVNC) Vehicle Travels towards a VRU 5 km/h VRU A child pedestrian crossing the road km/h Scenario Pedestrian`s path running from behind and obstruction vehicle strikes the pedestrian at 50% of the vehicle's width when no braking action is applied. 37

38 Learn more about sensor fusion by exploring examples in the Automated Driving System Toolbox Design multi-object tracker based on logged vehicle data Generate C/C++ code from algorithm which includes a multi-object tracker Synthesize driving scenario to test multi-object tracker 38

39 Common Questions from Automated Driving Engineers vehicle How can I Visualize Sensor data? How can I design and verify Perception algorithms? How can I design and verify Sensor fusion? 39

40 % Thank you 40