Computer and Machine Vision Lecture Week 12 Part-1 Additional Programming Considerations March 29, 2014 Sam Siewert
Outline of Week 12 Computer Vision APIs and Languages Alternatives to C++ and OpenCV API for Top- Down Halide MATLAB with CV Toolbox Mathematica Additional 3D Scene Considerations Camera Depth of Field Scene Complexities Receiver Operator Curves for Recognition Performance 3D Scene Capture to Point-Cloud-Models Structure from Motion Methods Sam Siewert 2
Major Concepts Post Production Digital Video Frame Processing Pipeline Movie! 1. Acquire, Decode, or Read Frame Files 2. Apply Transforms Simple (Brightness), Convolution (Edge Enhancement), Color Edit, 3. Add CGI Generate Computer Generated Imagery with Rendering 4. Composite Frame with Alpha Blending or Green Screen Human Characters into CGI Background 5. Encode Frames into MPEG Linux Tools Used Today ffmpeg a.k.a avconv, OpenCV, Pixie (Renderman), Python Scripting to Coordinate file-to-tool-to-file RenderMan Pixie MPEG ffmpeg Raw Frames OpenCV Composite Frames ffmpeg Sam Siewert 3
Major Concepts Video Analytics Computer Vision Pipeline Find Bad Guy! 1. Acquire, Decode, or Read Frame Files 2. Apply Transforms Simple (Brightness), Convolution (Edge Enhancement), False Color, Segment, 3-D Correspondence, SIFT, AdaBoost 3. Match to Recognition Database (Facial) 4. Present to User Linux Tools Used Today ffmpeg a.k.a avconv, OpenCV, Python Scripting to Coordinate file-to-tool-to-file MPEG MPEG ffmpeg ffmpeg Raw Frames OpenCV Multicore Sam Siewert 4
Observations APIs Support Building Tools Many Tools Lead to File to File Processing Pipelines File to File is Much Slower than Buffer to Buffer However, File to File Scales with Clusters! Buffer to Buffer Scales with Clusters With MPI or Shared Memory Multi-Core Could Scripting Coordination of File to File Be Replaced with a Language? Could the Language Optimize Use of Buffers (Locality) and Parallel and Vector Processing Hardware Features? Frees Programmer to Explore Options Halide Potential Option Sam Siewert 5
Brighten, Contrast Transform Compare P = clamp[ Q*alpha + beta] Very Simple Scale Each Pixel Brightness Increase/Decrease Bias Each Pixel Contrast Increase/Decrease Make Sure Result Does Not Exceed Saturation Compare C++ OpenCV API and Halide C++ Extension Sam Siewert 6
C Code PPM Brightness/Contrast #define PIXIDX ((i*col*chan)+(j*chan)+k) #define SAT (255) void main(int argc, char *argv[]) char header[512]; unsigned char img[640*480*3], newimg[640*480*3]; int bufflen, hdrlen; unsigned row=0, col=0, chan=0, pix; int i, j, k; double alpha=1.25; unsigned char beta=25; header[0]='\0'; readppm(img, &bufflen, header, &hdrlen, &row, &col, &chan, argv[1]); for(i=0; i < row; i++) for(j=0; j < col; j++) for(k=0; k < chan; k++) newimg[pixidx] = (pix=(unsigned)((img[pixidx])*alpha)+beta) > SAT? SAT : pix; writeppm(newimg, bufflen, header, hdrlen, "brighter.ppm"); Sam Siewert 7
OpenCV C++ API Brighten/Contrast #include <iostream> #include <opencv2/highgui/highgui.hpp> #include <opencv2/core/core.hpp> using namespace cv; using namespace std; double alpha=1.0; int beta=10; /* contrast and brightness control */ int main( int argc, char** argv ) Mat image = imread( argv[1] ); // read in image file Mat new_image = Mat::zeros( image.size(), image.type() ); std::cout<<"* Enter alpha brighten factor [1.0-3.0]: ";std::cin>>alpha; std::cout<<"* Enter beta contrast increase value [0-100]: "; std::cin>>beta; // Do the operation new_image(i,j) = alpha*image(i,j) + beta for( int y = 0; y < image.rows; y++ ) for( int x = 0; x < image.cols; x++ ) for( int c = 0; c < 3; c++ ) new_image.at<vec3b>(y,x)[c] = saturate_cast<uchar>( alpha*( image.at<vec3b>(y,x)[c] ) + beta ); Loop through Rows, Columns, and Color Channels to Apply Dereference Pixel at X,Y for each Color namedwindow("original Image", 1); namedwindow("new Image", 1); imshow("original Image", image); imshow("new Image", new_image); waitkey(); return 0; Sam Siewert 8
Halide Brighten/Contrast // Adapted from Halide tutorial lesson 2. #include <Halide.h> using Halide::Image; #include "../apps/support/image_io.h" int main(int argc, char **argv) Halide::Image<uint8_t> input = load<uint8_t>(argv[1]); Halide::Func brighter; Halide::Var x, y, c; Halide::Expr value = input(x, y, c); value = Halide::cast<float>(value); value = value * 1.5f; value = Halide::min(value, 255.0f); value = Halide::cast<uint8_t>(value); brighter(x, y, c) = value; Halide::Image<uint8_t> output = brighter.realize(input.width(), input.height(), input.channels()); save(output, "brighter.png"); No loop specification, just transform function, so Halide can optimize locality, vector processing and multi-core features. return 0; Sam Siewert 9
Halide Overview http://halide-lang.org/ Decouples Pipeline Specification (Buffer-to-Buffer) and Optimization Built on LLVM and Clang Extensions to C/C++ and Embedding Extension to C++ with Use of Operator Overloading Watch Video Overview on Web Alternatives 1. APIs OpenCV, OpenNI (for C++, Python, Java) 2. Interactive VHLLs MATLAB CV Toolbox, Mathematica 3. IDL Interactive Data Language Discuss API vs. Specialize Language? Sam Siewert 10