MPEG4 Programming Guide

Transcription

1 MPEG4 Programming Guide October 2006 Version 1.0

2

3 1 Introduction Before this version the MPEG4 function is implement as USER mode application. To support full duplex function, the MPEG4 software engine has changed to put in linux kernel mode. The USER mode engine will not be maintained from now. The following is the device describtion: Major Number Minor Number MPEG4 Decoder (/dev/fdec) MPEG4 Encoder (/dev/fenc) To use it, you must insert the driver module in Embedded site. The fmcp_drv.o module must insert before fmpeg4_drv.o: #insmod fmpeg4_drv.o mp4_max_width=720 mp4_max_height=480 The parameter of mp4_max_width specifies the maximum supported width of frame, so is mp4_max_height. These value will affect the size of buffer allocation in MPEG4 buffer. PLEASE NOTE, the fmcp_drv.o module must insert before fmjpeg_drv.o. 1.1 Source Directory As the combination of ffmpeg package, the MPEG4 driver is located in ffmpeg directory. The source tree is: /ffmpeg-0.4.8/libfaraday/ftmcp100_module fmpeg4_avcodec fmpeg4_driver fmpeg4_avcodec: The MPEG4 codec source definition for ffmpeg fmpeg4_driver: The MPEG4 linux driver module that will not be released in normal case. To compiler is in ffmpeg-0.4.8: #cd ffmpeg #make The driver module fmpeg4_drv.o will be created in directory: /ffmpeg-0.4.8/libfaraday/ftmcp100_module/fmpeg4_driver/fmpeg4_drv.o The codec source definition for ffmpeg will be created in in directory: /ffmpeg-0.4.8/libfaraday/ftmcp100_module/fmpeg4_avcodec/favcodec.o The fmpeg4_drv.o will alos copy to output direcotry /tmp. Brief Spec. 1 Version 1.0 / October 2006

4 MPEG4 Programming Guide 1.2 Kernel mode MPEG4 driver Usage To use MPEG4 driver, you can refer to the directory /ffmpeg-0.4.8/libfaraday/ftmcp100_module/fmpeg4_avcodec. The source code: Makefile fmpeg4_avcodec.c fmpeg4_avcodec.h ratecontrol.c ratecontrol.h Another sample you can take for reference has no relationship with ffmpeg. /ffmpeg-0.4.8/libfaraday/ftmcp100_module/sample Makefile mpeg4_main.c fmpeg4_avcodec.h ratecontrol.c ratecontrol.h To use the kernel driver, you can modify the example by yourself. Makefile <Your main program>.c fmpeg4_avcodec.h ratecontrol.c ratecontrol.h PLEASE NOTE, the rate control program MUST include in MPEG4 kernel mode driver, that is, ratecontrol.c and ratecontrol.h. 1.3 How to set MPEG4 driver polling/irq mode on with/without RISC architecture You can change the configuration by re-write the config.mak file. And, the config file is placed in./fmjpeg_driver. In the config.mak, you can find the section that can be changed by user. If the functions or features must enable, please set their condition equal to yes. And. If the functions or features must disable, please set their condition equal to no. The FIC8120 platform is only support polling mode and non-risc architecture. The fmpeg4_drv.o is need to config liked as the following. CONFIG_FTMCP100_IRQ_MODE = no CONFIG_FTMCP100_RISC= no

5 CONFIG_FTMCP100_NO_RISC= yes The FIE8150 and FIE8160 is only not support IRQ mode on non-risc architecture. For FIE8150/FIE8160 polling mode on without RISC architecture, the fmpeg4_drv.o is need to config liked as the following. CONFIG_FTMCP100_IRQ_MODE = no CONFIG_FTMCP100_RISC= no CONFIG_FTMCP100_NO_RISC= yes For FIE8150/FIE8160 polling mode on with RISC architecture, the fmpeg4_drv.o is need to config liked as the following. CONFIG_FTMCP100_IRQ_MODE = no CONFIG_FTMCP100_RISC= yes CONFIG_FTMCP100_NO_RISC= no For FIE8150/FIE8160 IRQ mode on with RISC architecture, the fmpeg4_drv.o is need to config liked as the following. CONFIG_FTMCP100_IRQ_MODE = yes CONFIG_FTMCP100_RISC= yes CONFIG_FTMCP100_NO_RISC= no 1.4 How to set the resolution for MPEG4 driver There there resolutions for MPEG4 driver, and the default resolution is 720x576. You can choose the suitable condition to meet your case. The default resolution is liked as the following. CONFIG_2592x1944_RESOLUTION = no CONFIG_1600x1200_RESOLUTION = no CONFIG_720x576_RESOLUTION = yes 1.5 How to set the output format for MPEG4 driver There there resolutions for MPEG4 driver, and the default resolution is 720x576. You can choose the suitable condition to meet your case. If you want to enable the condition, please clear the symbol #. The default resolution is liked as the following. #CFLAGS += -DOUTPUT_FORMAT_RGB555 #CFLAGS += -DOUTPUT_FORMAT_RGB565 Brief Spec. 3 Version 1.0 / October 2006

6 MPEG4 Programming Guide #CFLAGS += -DOUTPUT_FORMAT_RGB888 #CFLAGS += -DOUTPUT_FORMAT_CbYCrY CFLAGS += -DOUTPUT_FORMAT_YUV Note: Don t forget to select a suitable configuration for fmcp_drv.o. You can change the configuration by re-write the config.mak file. The config file is place in./fmcp_common.

7 2 MPEG4 driver IOCTL definition To use MPEG4 driver, you must call the ioctl that provided by driver. 2.1 Decoder FMPEG4_IOCTL_DECODE_INIT The ioctl is to initialize MPEG4 driver. To use it, you must prepare the structure FMP4_DEC_PARAM for input parameter. FMPEG4_IOCTL_DECODE_FRAME The ioctl is to decode one MPEG4 frame. To use it, you must prepare the structure fmpeg4_parm_t for input parameter. 2.2 Encoder FMPEG4_IOCTL_ENCODE_INIT The ioctl is to initialize MPEG4 driver. To use it, you must prepare the structure FMP4_ENC_PARAM for input parameter. FMPEG4_IOCTL_SET_INTER_QUANT The ioctl is to initialize customize Quantization table. To use it, you must prepare an inter quant file for input file. FMPEG4_IOCTL_SET_INTRA_QUANT The ioctl is to initialize customize Quantization table. To use it, you m must prepare an intra quant file for input file. FMPEG4_IOCTL_ENCODE_FRAME The ioctl is to do encode one MPEG4 frame. To use it, you must prepare the structure Faraday_ENC_FRAME for input parameter. And the function call will return the output value: enc_frame.intra: The value to see if the frame is intra-frame enc_frame.length: The length of bitstream after encoded FMPEG4_IOCTL_ENCODE_INFO The ioctl is to get macro block information from MPEG4 driver. To use it, you must prepare the structure MACROBLOCK_INFO that will return back the result. Brief Spec. 5 Version 1.0 / October 2006

8 MPEG4 Programming Guide 3 The MPEG4 driver The MPEG4 driver is based on linux MISC device architecture. User application can use the provied ioctl to do MPEG4 encode or decoder. This document introduces the programming guide for MPEG4 codec. MPEG4 Initial MPEG4 Initial MPEG4 Decode Next frame MPEG4 Encode Next frame MPEG4 Close MPEG4 Close The example is comming from ffmpeg: ffmpeg-0.4.8/libfaraday/ftmcp100_module/fmpeg4_avcodec/fmpeg4_avcodec.c 3.1 MPEG4 Decode MPEG4 Decoder Initial int fmpeg4_dec_fd=0; int decoder_init(...) FMP4_DEC_PARAM tdecparam; int err; if(fmpeg4_dec_fd==0) fmpeg4_dec_fd=open(fmpeg4_decoder_dev,o_rdwr); if(fmpeg4_dec_fd<=0) return -1; tdecparam.u32api_version = 0x ; tdecparam.u32maxwidth = 640; tdecparam.u32maxheight = 480; tdecparam.u32framewidth = 640; tdecparam.u32frameheight = 480;

9 err=ioctl(fmpeg4_dec_fd,fmpeg4_ioctl_decode_init,&tdecparam); if(err<0) return -1; return 0; The major parameter for initialization is described as follows: typedef struct README FMP4_DEC_PARAM u32api_version It should be 0x To indicate the MPEG4 engine version. u32maxwidth It indicates the video image width. u32maxheight It indicates the video image height. u32framewidth It indicates the ouput frame width. The stride width may be larger than u32maxwidth. u32frameheight It indicates the ouput frame width. The stride width may be larger than u32maxheight MPEG4 Decode int decode(...) fmpeg4_parm_t int parm; err; /* give input virtial address */ parm.input_va_y=... parm.length=buf_size; /* give output virtial address */ parm.output_va_y=... parm.output_va_u=... parm.output_va_v=... err=ioctl(fmpeg4_dec_fd,fmpeg4_ioctl_decode_frame,&parm); if(err<0) return -1; return parm.length; Brief Spec. 7 Version 1.0 / October 2006

10 MPEG4 Programming Guide The major parameter for decoding is described as follows: typedef struct fmpeg4_parm_t input_va_y input_va_u input_va_v output_va_y output_va_u output_va_v got_picture Length README It indicates the Y address of input video image. It should be continued user virtual address. If not, the message Warring! The decode Y output buffer not continued! Please use mmap instead. will alarm you. The U address same usage as input_va_y The V address same usage as input_va_y It indicates the Y address of output video bitstream. It indicates the U address of output video bitstream. It indicates the V address of output video bitstream. After decoding by ioctl, this parameter got_picture returns the attribute of bitstream. To indicate if it is I-frame. It indicate the length of bitstream MPEG4 Close int decoder_end(...) if(fmpeg4_dec_fd) close(fmpeg4_dec_fd); fmpeg4_dec_fd=0; return 0; 3.2 MPEG4 Encode MPEG4 Encoder Initial RateControl ratec; int fmpeg4_enc_fd; int enc_init(...) FMP4_ENC_PARAM enc_param; int err; if(fmpeg4_enc_fd==0) fmpeg4_enc_fd=open(fmpeg4_encoder_dev,o_rdwr); if(fmpeg4_enc_fd==0) printf("fail to open %s\n",fmpeg4_encoder_dev); return -1; enc_param.u32bitrate=1024;

11 enc_param.u32framewidth=640; enc_param.u32frameheight=480; enc_param.fframerate=30; enc_param.u32initialquant=4; enc_param.u32ipinterval=15; enc_param.bshortheader=0; enc_param.benable4mv=0; enc_param.bh263quant=1; enc_param.bresyncmarker=0; enc_param.u32maxquant=31; enc_param.u32minquant=2; // ROI related encoding parameters enc_param.broienable = 0; //0:disable 1:enable enc_param.u32roix = ROIX; // must be a multiple of 16 pixels enc_param.u32roiy = ROIY; // must be a multiple of 16 pixels enc_param.u32roiwidth = enc_param.u32framewidth; // must be a multiple of 16 pixels enc_param.u32roiheight = enc_param.u32frameheight; // must be a multiple of 16 pixels RateControlInit(&ratec,enc_param.u32BitRate*1000,DELAY_FACTOR, RC_AVERAGE_PERIOD, ARG_RC_BUFFER, get_framerate(enc_param.fframerate), enc_param.u32maxquant,enc_param.u32minquant,enc_param.u32initialquant); err=ioctl(fmpeg4_enc_fd,fmpeg4_ioctl_encode_init,&enc_param); if(err<0) return -1; return 0; The major parameter for initialization is described as follows: typedef struct FMP4_ENC_PARAM u32bitrate u32framewidth u32frameheight fframerate u32initialquant bshortheader benable4mv README It indicates the bit rate value. (Kbit/s unit). It indicates the frame width. It indicates the frame height. It indicates the frame rate. It indicates the initialization quantization value. User can decides the first frame quant value. It indicates if you want to use MPEG4 short header format. It indicates if you want to use 4MV function. Please note the performance may be slower than 1MV for better motion estimation. 0: 1MV (default) 1: 4MV bh263quant bresyncmarker It selects the H.263 quantization method or MPEG4 quantization method. 0:MPEG4 (default) 1:H263 It decides if resync marker mechanism is enabled. 0:Disabled (default) 1:Enabled Brief Spec. 9 Version 1.0 / October 2006

12 MPEG4 Programming Guide u32maxquant u32minquant To set the maximum quantization value range To set the minimum quantization value range MPEG4 Encode int encode(...) int err; Faraday_ENC_FRAME enc_frame; int keyframe; /* give input Y U V address */ enc_frame.pu8yframebaseaddr =... enc_frame.pu8uframebaseaddr =... enc_frame.pu8vframebaseaddr =... enc_frame.bitstream=frame; //output User Virtual address enc_frame.quant = ratec.rtn_quant; enc_frame.intra = -1; // ROI related encoding parameters enc_param.broienable = 0; //0:disable 1:enable enc_param.u32roix = 0; // must be a multiple of 16 pixels enc_param.u32roiy = 0; // must be a multiple of 16 pixels enc_param.u32roiwidth = enc_param.u32framewidth; // must be a multiple of 16 pixels enc_param.u32roiheight = enc_param.u32frameheight; // must be a multiple of 16 pixels err=ioctl(fmpeg4_enc_fd,fmpeg4_ioctl_encode_frame,&enc_frame); if(err<0) return -1; RateControlUpdate(&ratec,enc_frame.quant, enc_frame.length, enc_frame.intra); keyframe=enc_frame.intra; //the return value in enc_frame.intra return enc_frame.length; The major parameter for encoding is described as follows: typedef struct README Faraday_ENC_FRAME pu8yframebaseaddr It indicates the Y address of YUV raw data. That is input buffer prepared to be encoding.

13 pu8uframebaseaddr pu8vframebaseaddr bitstream quant intra length broienable u32roix u32roiy u32roiwidth u32roiheight It indicates the U address of YUV raw data. That is input buffer prepared to be encoding. It indicates the V address of YUV raw data. That is input buffer prepared to be encoding. It indicates the output bitstream address to encoder driver. After encoding finished, the encoder will prepare the bitstream data in this buffer. It provides that you can assign the quant value for this frame. And you may have a rate control mechanism to decide this value. We provide a simple one in the example of ffmpeg. After encoding, it returns to indicate if it is intra-frame or inter-frame. After encoding, it returns the bitstream length. Indicate the ROI (region of interest) function if enabled Set the ROI X coordinate, it must be 16 pixel multiple Set the ROI Y coordinate, it must be 16 pixel multiple Set the region width, it must be 16 pixel multiple Set the region height, it must be 16 pixel multiple 720 (u32roix, u32roiy) 480 u32roiwidth Encode Region u32roiheight MPEG4 Close int enc_close(...) if(fmpeg4_enc_fd) close(fmpeg4_enc_fd); fmpeg4_enc_fd=0; return 0; Get Macroblock Information Brief Spec Version 1.0 / October 2006

14 MPEG4 Programming Guide int enc_get_mbinfo(macroblock_info *mbinfo) err=ioctl(fmpeg4_enc_fd,fmpeg4_ioctl_encode_info,mbinfo); if(err<0) printf("error to set FMPEG4_IOCTL_ENCODE_INFO\n"); return -1; return 0;

15 4 MPEG4 Input/Output Control When user application use the MPEG4 driver kernel mode, it must take care of input and output buffer control. That is the condiction of continues memory layout. We discuss this as follows. 4.1 MPEG4 decode To handle the decoder output parameter, the situation example lists are described as follows. Assume the resolution is 640x480. Output to LCD frame buffer Direct output the decoded data to LCD buffer for display. int lcd_fd; lcd_fd = open("/dev/fb0", O_RDWR); //open frame buffer device mmap_addr=(unsigned int)mmap(0,mmap_sz,prot_read PROT_WRITE,MAP_SHARED,lcd_fd, 0); void decode( ) fmpeg4_parm_t parm; parm.output_va_y= mmap_addr; parm.output_va_u= mmap_addr+(640*480); //assume it is 640x480 parm.output_va_v= mmap_addr+(640*480)+(320*240); Output to memory buffer unsigned int mmap_addr; int fmpeg4_dec_fd; fmpeg4_dec_fd=open(fmpeg4_encoder_dev,o_rdwr); mmap_addr=mmap(0,640*480*3/2,prot_read PROT_WRITE,MAP_SHARED,fmpeg4_enc_fd, 0); void decode( ) fmpeg4_parm_t parm; parm.output_va_y= mmap_addr; parm.output_va_u= mmap_addr+(640*480); //assume it is 640x480 parm.output_va_v= mmap_addr+(640*480)+(320*240); Brief Spec Version 1.0 / October 2006

16 MPEG4 Programming Guide If you can t make sure the output address is continues, you should use mmap instead. If the address is not continues, the warning message occurred: Warring! The decode Y output buffer not continued! Please use mmap instead MPEG4 encode Input from capture buffer Direct input from capture buffer. int capture_fd; unsigned int mmap_addr; capture_fd=open(video_device, O_RDWR); mmap(0, buf, 640*480*3/2, PROT_READ PROT_WRITE, MAP_SHARED, fd, 0); //setup the buffer ioctl(capture_fd, VIDIOCSYNC, &frame); //get capture driver parameter mmap_addr=frame. mmapaddr; //get video virtual frame buffer void encode( ) Faraday_ENC_FRAME enc_frame; enc_frame.pu8yframebaseaddr=mmap_addr; enc_frame.pu8uframebaseaddr=mmap_addr+(640*480); //assume 640x480 enc_frame.pu8vframebaseaddr=mmap_addr+(640*480)+(320x240); Input from memory buffer int fmpeg4_enc_fd; unsigned int mmap_addr; fmpeg4_enc_fd=open(fmpeg4_encoder_dev,o_rdwr); mmap_addr=mmap(0,640*480*3/2,prot_read PROT_WRITE,MAP_SHARED,fmpeg4_enc_fd, 0); void encode( ) Faraday_ENC_FRAME enc_frame; enc_frame.pu8yframebaseaddr=mmap_addr; enc_frame.pu8uframebaseaddr=mmap_addr+(640*480); //assume 640x480 enc_frame.pu8vframebaseaddr=mmap_addr+(640*480)+(320x240);

17 If you can t make sure the input address is continues, you should use mmap instead. If the address is not continues, the warning message occurred: Warring! The encode Y output buffer not continued! Please use mmap instead.. To try the mmap function in ffmpeg package, you may trace the fmpeg4_avcodec.c: //#define USE_MMAP #define USE_MMAP That will change the memory usage to MMAP. You can take it for your reference. Brief Spec Version 1.0 / October 2006

18 MPEG4 Programming Guide 5 MPEG4 Motion Dection To implement the motion dection function in FIC8120, you have two way to do it: Customer definied motion dection algorithm by MV (motion vector) Faraday s motion dection example (under ffmpeg package) 5.1 Customer definied motion dection function by MV The Faraday s MPEG4 driver provides Motion Vector information for your reference. You can use the MV information to do what you want about motion dection algorigthm. To get the motion vector. You must use: MACROBLOCK_INFO *mbinfo ioctl(fmpeg4_enc_fd,fmpeg4_ioctl_encode_info,mbinfo); Where the fmpeg4_enc_fd is the driver descriptor. And after you cann ioctl, you can get the mbinfo. That is the motion vector by macro blocks. As the following example to 320x240 frame size, you can get the mbinfo[n], where N is macro block index. mbinfo[0] mbinfo[1] mbinfo[2] mbinfo[3] mbinfo[19] mbinfo[20] mbinfo[40] mbinfo[299] 5.2 Faraday s motion dection example We provide a motion dection function in ffmpeg package with MPEG4 driver. You can refer to fmpeg4_avcodec.c source code.

19 The motion dection flow is as follows: The main program read the configuration file named m.cfg. And deside to trigger a signal to a alarm monitor (use process ID). After alarm monitor receive the signal, it use ftp or emal to notify user. You can try it in our released software package Configuration File for motion dection In ffmpeg package, you may use motion dection option. #./ffmpeg motion m.cfg And the m.cfg.8120 is the configuration for motion dection. The example to parameter of configuration for motion dection is described as follows: pid 23 md_interval 22 x0_lu 0 y0_lu 0 x0_rd 40 y0_rd 10 x1_lu 0 y1_lu 10 x1_rd 40 y1_rd 20 x2_lu 0 y2_lu 20 x2_rd 40 y2_rd 30 mv_th0 8 sad_th0 512 dev_th0 512 alarm_idx0 100 Brief Spec Version 1.0 / October 2006

20 MPEG4 Programming Guide mv_th1 8 sad_th1 512 dev_th1 512 alarm_idx1 100 mv_th2 8 sad_th2 512 dev_th2 512 alarm_idx2 100 The explain to the parameter is: pid: The process id of alarm monitor. Main program will send signal to this process id when motion occured. md_interval: The frame interval to do motion dection. 22 means one motion dection computing per 22 frames. x0_lu: The left-up X coordinate to motion dection region 0 y0_lu: The left-up Y coordinate to motion dection region 0 x0_rd: The right-down X coordinate to motion dection region 0 y0_rd:the right-down Y coordinate to motion dection region 0 x1_lu: The left-up X coordinate to motion dection region 1 y1_lu: The left-up Y coordinate to motion dection region 1 x1_rd: The right-down X coordinate to motion dection region 1 y1_rd:the right-down Y coordinate to motion dection region 1 x2_lu: The left-up X coordinate to motion dection region 2 y2_lu: The left-up Y coordinate to motion dection region 2 x2_rd: The right-down X coordinate to motion dection region 2 y2_rd:the right-down Y coordinate to motion dection region 2 mv_th0: Region 0 Motion Vector threshold (0-1922). Slow value high sensitive. sad_th0: Region 0 SAD threshold ( ). Slow value high sensitive. dev_th0:region 0 SAD threshold ( ). Slow value high sensitive. alarm_idx0 Region 0 Alarm index threshold. When computing index larger than it, the main program send alarm signal to pid. mv_th0: Region 0 Motion Vector threshold (0-1922). Slow value high sensitive. sad_th0: Region 0 SAD threshold ( ). Slow value high sensitive. dev_th0:region 0 SAD threshold ( ). Slow value high sensitive. alarm_idx0 Region 0 Alarm index threshold. When computing index larger than it, the main program send alarm signal to pid.

21 mv_th2: Region 2 Motion Vector threshold (0-1922). Slow value high sensitive. sad_th2: Region 2 SAD threshold ( ). Slow value high sensitive. dev_th2:region 2 SAD threshold ( ). Slow value high sensitive. alarm_idx2 Region 2 Alarm index threshold. When computing index larger than it, the main program send alarm signal to pid. 5.3 Motion Dection Algorithm MV algorithm It uses sum of motion vections to determine motion if larger than threshold SAD algorithm SAD means the sum absolute difference generated from motion estimation DEV algorithm DEV means N N 2 Mean Cij = / N i= 1 j= 1 Dev = N N i= 1 j= 1 C ij Mean Motion vector Motion estimation Motion Dection Algorithm Flow x^2+y^2 > mv_th && sad > sad_th? Yes Delta-dev > dev_th? Yes Index++ Check next MB Brief Spec Version 1.0 / October 2006