Interaction Technology

Transcription

1 Faculty of Science Information and Computing Sciences 2017 Introduction Computer Vision Coert van Gemeren 8 maart 2017

2 Information and Computing Sciences TODAY 1.Computer Vision 2.Programming C/C++ OpenCV 3.Installation & Tutorial OpenCV 8 maart

3 Information and Computing Sciences COMPUTER VISION 8 maart

4 Information and Computing Sciences COMPUTER VISION Definition Computer Vision is the AUTOMATIC ANALYSIS and INTERPRETATION of IMAGES or image sequences (VIDEO). Computer Graphics Image Processing 3D Modeling C.G. C.S. Cognitive Science Vis. Perception Semantics M.L. Computer Vision Machine Learning Search Problems Classification Problems 8 maart

5 Information and Computing Sciences COMPUTER VISION Relation to subfields Image Computer vision Metadata / Signaling interface 3D model 8 maart

6 COMPUTER VISION Example pipeline: Hawkeye goal decision system 6 high speed cameras (200+ fps) around the pitch, aimed at the goal Via Invoegen Koptekst en Voettekst invoegen Subafdeling<2spaties> <2spaties>Titel van de presentatie 8 maart

7 COMPUTER VISION Example pipeline: Detection (CV + trigonometry) Process the images to find (part of) the ball in at least 3 camera viewpoints Use triangulation + timing data to estimate the 3D spatial location of the ball and predict its future position Via Invoegen Koptekst en Voettekst invoegen Subafdeling<2spaties> <2spaties>Titel van de presentatie 8 maart

8 COMPUTER VISION Example pipeline: Triangulation & interfacing Estimate if the position of the ball is in the goal-box Send signal to the referee s watch Via Invoegen Koptekst en Voettekst invoegen Subafdeling<2spaties> <2spaties>Titel van de presentatie 8 maart

9 COMPUTER VISION Example pipeline: Animation (graphics) Generate animation of the ball and the players with respect to the goal-box Via Invoegen Koptekst en Voettekst invoegen Subafdeling<2spaties> <2spaties>Titel van de presentatie 8 maart

10 Information and Computing Sciences COMPUTER VISION Input & Output Input to Computer Vision Algorithm: Image (single or multiple) Video (sequence of images) Depth images Additional information: Metadata Image captions Calibration data etc. 8 maart

11 Information and Computing Sciences person person COMPUTER VISION Input & Output Output from a Computer Vision Algorithm: Localization (person detection) Classification (eg.: female, male, ~25 y/o ) Identification (eg.: Merel, René ) 3Hoog/DUB Segmentation 3D Shape Pose Reconstruction Scene Information (depth, lighting, etc.) Camera Information (zoom, viewpoint) 8 maart

12 Information and Computing Sciences APPLICATIONS 8 maart

13 Information and Computing Sciences APPLICATIONS Process Automation License plate recognition Handwriting recognition/verification Product verification Rail track inspection Measuring distances 8 maart

14 Information and Computing Sciences APPLICATIONS Big Data Face recognition in social networks Automatic tagging of images Finding duplicate movies Recognizing where photos have been taken Reconstructing the world 8 maart

15 Information and Computing Sciences APPLICATIONS Security & Surveillance Finding left luggage Localizing threats Crowd control Monitoring of elderly homes Drowning warning systems

16 Information and Computing Sciences APPLICATIONS Human Computer Interaction Gaming Social robots Gesture control

17 Information and Computing Sciences APPLICATIONS (Serious) Gaming Body movement: Wii, Move, Kinect Facial expression: affect, control, playful chat (avateering) Scene recognition/augmented reality

18 Information and Computing Sciences APPLICATIONS Medical Rehabilitation Visual assistance Anomaly detection

19 Information and Computing Sciences APPLICATIONS Automotive Industry Automatic collision avoidance Pedestrian detection Lane assist Autonomous driving Driver attention detection

20 Information and Computing Sciences APPLICATIONS Face detection Surveillance Consumer products Social networks

21 CHALLENGES Information and Computing Sciences

22 Information and Computing Sciences CHALLENGES Digitization Information is in the pixels (more about that next week) But pixels can vary a lot

23 Information and Computing Sciences CHALLENGES Projection Depth is lost Relative sizes Deformation

24 Information and Computing Sciences CHALLENGES Projection Honda Illusions, An Impossible Made Possible

25 Information and Computing Sciences CHALLENGES Low resolution & noise Limited image resolution Limited dynamic sensor range Weather

26 Information and Computing Sciences CHALLENGES Lighting Directed light Amount of lighting Shadows

27 CHALLENGES Lighting Information and Computing Sciences

28 CHALLENGES Lighting Information and Computing Sciences

29 Information and Computing Sciences CHALLENGES Variation in shape and size Inter-class variation Intra-class variation Viewpoint variation

30 Information and Computing Sciences CHALLENGES Clutter and occlusion Distractions due to cluttered environment Incomplete observation due to partial occlusions

31 Information and Computing Sciences CHALLENGES Context Other objects Object use Prior knowledge

32 Information and Computing Sciences CHALLENGES Context Other objects Object use Prior knowledge Light

33 QUESTIONS? Information and Computing Sciences

34 OpenCV Information and Computing Sciences

35 OpenCV OpenCV - Computer Vision Package: C/C++ (Python / Java) Many tutorials available to get you started: All provided code and tutorials will be C++!

36 OpenCV OpenCV Low level image manipulations

37 OpenCV 1 Images consist of pixels Pixels consist of numbers

38 OpenCV 1 Images consist of pixels Each cell encodes a pixel intensity value as an unsigned char value (uchar). Which is 8 bit (0 255) scalar values Gray images are encoded by a single scalar value uchar gray_value = 127; Color is encoded in a vector of 3 scalars of which the order is BGR (vs. RGB) cv::vec3b color_value = cv::vec3b(0, 127, 255); cv::vec3b is an alias of cv::vec<unsigned char, 3>

39 OpenCV 1 Images consist of pixels Each cell encodes one pixel intensity value as a scalar value Common primitive types for scalar values in OpenCV are unsigned 8 bit char values (unsigned char) o [0 255] signed 16 bit integer values (integer) o [ ] signed 32 bit floating point values (float) o [-inf inf ] signed 64 bit double precision floating point values (double) o [-inf inf ] Many more types exist of course

40 OpenCV 2 The cells in a rectangular matrix make up the image A 2 dimensional matrix with cells a 11 to a mn A single channel uchar matrix encodes gray images Colors are interleaved in the matrix through the 3-channel vectors a 11 = cv::vec3b(0, 127, 255), a 12 = cv::vec3b(0, 127, 255), etc. BGR BGR BGR (vs. BBB GGG RRR) As the image are matrices, all standard matrix operations can be applied to them: add, subtract, multiply, etc. A video (sequence of images) can either be a vector of matrices, or a 3 dimensional matrix

41 OpenCV 2 The specifications of an image matrix A matrix can have different cell types which have to be assigned on definition uchar cv::mat(rows, cols, CV_8U); // 8 bit, 1 chan cv::vec3b cv::mat(rows, cols, CV_8UC3); // 8 bit, 3 chan int cv::mat(rows, cols, CV_32S); // 32 bit, 1 chan float cv::mat(rows, cols, CV_32F); // 32 bit, 1 chan double cv::mat(rows, cols, CV_64F); // 64 bit, 1 chan A data type is given by the CV_-macro (blue), but it can also be retrieved from the DataType class: CV_32F == cv::datatype<float>::type; What is the definition of the data type for CV_32FC3?

42 OpenCV 2 The specifications of an OpenCV matrix Matrices in OpenCV can become very large Copying cv::mat objects around does not copy all data each time: cv::mat A = cv::mat::zeros(1920, 1080, CV_64FC3); // ~50MB of memory cv::mat B(A); // copy constructor cv::mat C = A; // assignment cv::mat D = functionreturningmata(a); // return by value Doing anything to the data of B and C will also change the data in A! This is because the matrix object has a reference counter, to which 1 is added in each of these cases. The matrix data is not freed from memory until this reference counter reaches 0, meaning that the last object has ceased to exist. This keeps your program fast and your memory footprint low! What about matrix D?

43 OpenCV 2 The specifications of an OpenCV matrix For clarity most OpenCV functions have a source and destination parameter used as reference variables cv::mat D; functionturingaintod(a, D); void functionturingaintod(const cv::mat &src, cv::mat &dst); The following operations copy every value of a cv::mat object: cv::mat E = A.clone(); // deep copy each value of entire matrix cv::mat F; A.copyTo(F, mask); // copy data marked by the mask Use these with care! They are more time and memory consuming Once copied, doing something to E or F does not change A!

44 Height (M) OpenCV 3 Low level image manipulations Width (N) The size of the matrix is given by a number of columns and rows, or a cv::size object defining width and height Matrix dimensions: MxN (Rows x Columns) const int r = 480, c = 640; cv::mat(r, c, CV_8U); Image dimensions: WxH (Columns x Rows) cv::size size(640, 480); cv::mat(size, CV_8U); A matrix can be resized to a given cv::size cv::resize(i_mat, o_mat, cv::size(100, 50)); by a given factor const double fx = 0.5, fy = 0.5; cv::resize(i_mat, o_mat, cv::size(), fx, fy); With different pixel interpolation schemes (def. cv::inter_linear) cv::resize(,,, fy, cv::inter_nearest); cv::resize(,,, fy, cv::inter_cubic);

45 OpenCV 3 Low level image manipulations A matrix can be cropped to a Region Of Interest with a cv::rect cv::mat lena = ; cv::rect roi(300, 300, 106, 106); // x, y, w, h cv::mat lenas_face = lena(roi); The ROI must be within the bounds of the image, an error is thrown otherwise. Bound the roi to the image size by first taking the intersection of the ROI and the image: cv::rect bounds = lena.size() + cv::rect(0,0,0,0); cv::rect crop_box = roi & bounds; cv::mat slice = lena(crop_box); lena lenas_face slice

46 OpenCV 3 Low level image manipulations A matrix can be rotated using a rotation matrix and an affine transformation cv::mat src =, dst; const double diag = sqrt(src.cols*src.cols+src.rows*src.rows); const int size = std::max<int>((diag src.cols)/2, (diag src.rows)/2)); cv::copymakeborder(src, src, size,size,size,size, cv::border_constant); // top,bottom,left,right cv::point2f center(src.cols / 2.f, src.rows / 2.f); const double angle = 45.0, scale = 1.0; cv::mat rot_m = cv::getrotationmatrix2d(center, angle, scale); cv::warpaffine(src, dst, rot_m, src.size());

47 OpenCV 3 Low level image manipulations A matrix can be flipped cv::flip(lena, lena, 0) cv::flip(lena, lena, 1) cv::flip(lena, lena, -1)

48 OpenCV 3 Low level image manipulations A color image matrix can be split into separate channels std::vector<cv::mat> channels; cv::split(lena, channels); 3 color channels reds greens blues

49 OpenCV 3 Low level image manipulations A vector of image matrices can be merged back together cv::mat merged; channels[1] = cv::mat::zeros(lena.size(), CV_8U); channels[2] = cv::mat::zeros(lena.size(), CV_8U); cv::merge(channels, merged); reds greens blues merged

50 OpenCV 4 Drawing functions are available All very straight forward cv::rectangle, cv::circle, cv::line, etc.

51 OpenCV OpenCV High level image manipulations

52 OpenCV 5 Optical flow Look at the motion between two images Translation (planar directions) 2D movement direction Per keypoint (sparse, left clip) Per pixel (dense, right clip)

53 OpenCV 5 Tracking Actively keep track of the location of a shape or object in a video Usually based on low level feature information such as color Can be based on high level information: tracking by detection

54 OpenCV 5 Object detection Person detection Locate (an upper) body in the image Face detection Location of a face Location of parts of a face person person

55 OpenCV 6 Computer Vision is an Engineering approach to Visual Perception: do smart things with the output of your image processing steps Count pixels Count pixels per region Count pixels with certain direction Ignore pixels with certain color Etc. Your solution depends strongly on your creativity and observation skills!

56 Questions?

57 Code and tutorial Visit: Download the tutorial PDF The tutorial goes over some of the things shown here today The tutorial is not mandatory Download the tutorial code and get it to run on your own system!

58 Assignment 2 Recognize a gesture or sign with a hand to make the system spray once or twice. Use a laptop with a webcam + OpenCV The system must recognize whether or not hands are washed. If not, let the LCD screen show an appropriate message. Interface between the Arduino and your laptop (via USB) The system must be evaluated with a user point of view. 3/8/

59 Assignment 2: deliverables The annotated code of your project (code only please:.cpp and.h files, no binaries or other stuff). Use Submit! Be concise Describe you classes, and describe the input and output of your functions/methods A short (max 1:00 minute) video that demonstrates your implementation. Don t , use YouTube, Vimeo, WeTransfer, etc. Mention a link in your report. User-evaluation report Details: 3/8/

60 Questions?