Media
Contents Player Camera AudioIn, AudioOut Image & ImageUtil AudioRecorder VideoRecorder AudioDecoder/Encoder, VideoDecoder/Encoder 2
Introduction The Media namespace contains classes and interfaces that let developers e asily integrate audio, video, and image processing into their applications. Key features include: Encoding and decoding of images and videos. Playing audio and video from media files stored on a device or streaming over the network. Recording audio and video. Using the camera to display a live preview and capture a still image. Using codec to encode and decode the audio/video data. 3
1 Player
Player Class Overview Playback of media (Audio and Video) Player supports Playback of audio/video in local storage & stream from the server basic controls for audio/video content (play, pause, stop, & resume) time-based seeking, volume control, looping Getting the duration of audio/video content. Multiple audio stream playback. Playback of DRM content. Supported DRM are OMA (v1.0, v2.0, v2.1), WMDRM, PlayReadyDRM. Capturing the video frame during video playback. 5
Player Class APIs Important APIs OpenBuffer () OpenFile () OpenUrl () Play () Pause () Stop () Close () SeekTo () SetLooping () SetRenderingBuffer () GetCurrentMediaStreamInfoN () CaptureVideo () 6
Player Supported Formats and Protocols Local playback of Video H.264,H.263,MPEG4,VC-1 Video Streaming H.264, H.263, MPEG4 Local playback of Audio PCM, AMR-NB, AAC, AAC+, EAAC+, MIDI, MP3, WMA Audio streaming AMR-NB, AAC, AAC+, EAAC+ Protocols RTSP with live streaming HTTP with live streaming 7
Audio Mixing Multiple Audio Playback Multiple instance playback restriction Multiple compressed audio stream with different audio codec. Multiple uncompressed audio format (ex. WAV) The maximum instances of Player is dependent on system performance. The maximum count of Player instance is 10. 8
IPlayerEventListener Used to notify the status of media player during the media playback. Callbacks for the player events are OnPlayerOpened() : when the player is opened asynchronously OnPlayerBuffering(): when the streaming data is being buffered OnPlayerEndOfClip(): when playback reaches to the end of clip OnPlayerInterrupted(): when the device is taken by a task with a higher priority OnPlayerReleased() OnPlayerErrorOccurred(): when there is an error in playback OnPlayerSeekCompleted(): when the position of the audio/video content moves asynchronously 9
IPlayerVideoEventListener To get the decoded video frame OnVideoFrameDecoded() Notifies when the video frame is decoded asynchronously 10
2 Camera
Camera Class Overview Controls the Camera Device.. To capture still images. Supports real time preview. Allows processing of Preview data. Very important feature in Camera class. Requires CAMERA privilege Features Preview Format : RGB or YCbCr Camera Sensor controls Autofocus, Flash Brightness, Contrast Exposure, ISO Level, White Balance Zoom Preview Data Controls Preview format, preview frame rate, preview resolution 12
Camera Functions Prepare device PowerOn(), PowerOff() Getter /Setter methods for camera settings Get/Set {Brightness(), Contrast(), Effect(), Flash(), Quality(), WhiteBalance()} Get/Set {CaptureFormat(), CaptureResolution()} Get/Set {PreviewFormat(), PreviewFrameRate(), PreviewResolution()} Control Capture(), StartPreview(), StopPreview(), ZoomIn(), ZoomOut() 13
ICameraEventListener Camera Listener Functions OnCameraCaptured() called when a picture has been successfully captured OnCameraPreviewed() called at each frame of preview only when camera.startpreview(.., true) OnCameraAutoFocused() Called when Camera::SetAutoFocus() API is called and completed. OnCameraErrorOccurred Called when an error occurs in the Camera. 14
Previewing Scenario Preview Format List GetSupportedPreviewFormatListN() OverlayRegion Preview on the background buffer of an overlay region Steps Set the orientation of the form to landscape Create an overlay region Make the form as the current form Get background info of the overlay region Start preview on the background 15
Taking Picture Scenario Preparation Stage Creating & implementing the listener Construct Camera instance with ICameraEventListener Construct( listener, CAMERA_PRIMARY ) PowerOn camera PowerOn() Set camera options (optional) preview format, preview resolution, ISO level, brightness, effect, contrast, exposure, white balance Attach the backgroundbuffer of the OverlayRegion to camera to start preview StartPreview() 16
Taking Picture Scenario Taking picture & runtime control Take a picture camera.capture(); an event listener will be invoked Zoom camera.zoomin(), camera.zoomout() 17
Camera Example result TestPreviewing(void) { result r = E_SUCCESS; Camera* pprimarycamera = null; Osp::Ui::Controls::OverlayRegion * poverlayregion = null; Osp::Ui::Controls::Form* pcameraform = null; Osp::Graphics::BufferInfo bufferinfo; Osp::Graphics::Rectangle rect(0, 0, 320, 240); bool modified = false; bool suitable = false; // Initialize pcameraform pprimarycamera = new Camera(); plistener = new MyCameraListener; //Code snippet continued on the next slide.. 18
Camera Example // Construct camera with camera event listener and camera type r = pprimarycamera->construct(*plistener, CAMERA_PRIMARY); //17 r = pprimarycamera->poweron(); // camera is instructed to work suitable = OverlayRegion::EvaluateBounds( OVERLAY_REGION_EVALUATION_OPTION_LESS_THAN, rect, modified ); } poverlayregion = pcameraform->getoverlayregionn(rect, OVERLAY_REGION_TYPE_PRIMARY_CAMERA); r = poverlayregion->getbackgroundbufferinfo(bufferinfo); r = pprimarycamera->startpreview(&bufferinfo, true); return r; 19
3 AudioIn, AudioOut
AudioIn Class Overview Provides low level mechanism for audio recording Records PCM data from audio input device. If you want to record audio into a container file format, use AudioRecorder class. Dynamically we can get the information about the system (no of channels supported, Sample rate supported etc) Specification Input device type: microphone Number of channels: mono, stereo Sampling depth: 8 bit, 16 bit Sample rate: 8000 Hz ~ 48000 Hz 21
AudioIn Functions Prepare Device with number of channels, sample type, rate, and depth AddBuffer() Get information GetMin/MaxBufferSize() GetOptimized{ChannelType, SampleRate, SampleType}() GetState() Control Start(), Stop(), Reset(), Unprepare() 22
Typical AudioIn Scenario State transition Construct and prepare AudioIn Add buffer to AudioIn Start If there is interruption OnAudioInInterrupted() is called; stop AudioIn automatically OnAudioInReleased() is called; start AudioIn, again If the buffer is filled OnAudioInBufferFilled() is called; store data & rewind the buffer When the user stops recording, stop AudioIn and store data in the buffer unprepare AudioIn and delete it 23
IAudioInEventListener AudioIn captures audio data from the device in asynchronous mode, an d calls IAudioInEventListener methods to pass the captured audio data Provided callbacks are OnAudioInBufferIsFilled() when AudioIn fills the input buffer and indicates the application that buffer is filled and application may use it. OnAudioInInterrupted() when any high priority tasks acquire device ex: Call OnAudioInReleased() when the interrupted tasks releases the device which had acquired. 24
AudioIn: Multiple Buffering Adding more than one buffer Automatic switch When you start recording, the 1st buffer is used When the 1st is filled, OnAudioInBufferIsFilled() is invoked At the same time, the 2nd buffer starts being used When the 2nd is filled, the 1st buffer starts being used resulting in less latency. 25
AudioIn Example void AudioInExample::AudioInRecord(void) { // Constructs an AudioIn instance paudioininstance->construct(* pmyaudioinlistener); // Prepares an AudioIn instance paudioininstance->prepare(audio_input_device_mic,audio_type_pcm_u8, AUDIO_CHANNEL_TYPE_STEREO,8000); // Prepare buffers to store PCM data pbytebuffer1 = new ByteBuffer(); pbytebuffer1->construct(max_buffer_size); pbytebuffer2 = new ByteBuffer(); pbytebuffer2->construct(max_buffer_size); pbytebuffer3 = new ByteBuffer(); pbytebuffer3->construct(max_buffer_size); //Code snippet continued on the next slide.. 26
AudioIn Example // Adds buffers paudioininstance->addbuffer( pbytebuffer1); paudioininstance->addbuffer( pbytebuffer2); paudioininstance->addbuffer( pbytebuffer3); } // Start paudioininstance->start(); 27
AudioOut Class Overview Lower level audio playback class Play PCM (uncompressed) audio data in various sample types through Audio output device Can be either Mono or stereo Unsigned 8 bit or signed 16 bit Also supports multiple buffering. Usage is very similar to AudioIn except Raw Audio Buffer is for playback SetVolume is supported 28
AudioOut Functions Prepare Prepare(), Unprepare(), WriteBuffer() Volume control Get/SetVolume() Control Start(), Stop(), Reset() Get information GetMin/MaxBufferSize() GetOptimized{ChannelType,SampleRate,SampleType}() GetState() 29
IAudioOutEventListener Callbacks provided by IAudioOutEventListener OnAudioOutBufferEndReached() Called when all the content of the buffer has been played back OnAudioOutInterrupted() Called when the output device is occupied by another task with higher priority OnAudioOutReleased() Called when the interrupting task releases the device 30
AudioOut Example void AudioOutExample::AudioOutPlay() { // Construct the AudioOut instance with a listener paudiooutinstance->construct(* pmyaudiooutlistener); // AudioOut Preparation paudiooutinstance->prepare(audio_type_pcm_u8, AUDIO_CHANNEL_TYPE_STEREO,8000); // Write 3 buffers in queue of the device side ByteBuffer* pwritebuffer = null; pwritebuffer = static_cast<bytebuffer*>( pdataarray- >GetAt( playcount++)); paudiooutinstance->writebuffer(*pwritebuffer); //Code snippet continued on the next slide.. 31
AudioOut Example } pwritebuffer = static_cast<bytebuffer*>( pdataarray->getat( playcount++)); paudiooutinstance->writebuffer(*pwritebuffer); pwritebuffer = static_cast<bytebuffer*>( pdataarray- >GetAt( playcount++)); paudiooutinstance->writebuffer(*pwritebuffer); // Start playing until the end of the array paudiooutinstance->start(); 32
4 Image and ImageUtil
Image Class Overview IMAGE privilege is required. Media::Image class provides methods to encode, decode, and convert images. Supports decoding of JPEG, GIF, PNG, BMP, WBMP, TIFF Supports Encoding to JPEG, PNG, BMP Supports Image format conversion Supports JPEG compression. Optionally, the application can automatically scale images when decoding them. The application can compress JPEG images into smaller JPEG images with barely noticeable quality loss. Images can be handled from a file in storage or a buffer in memory. 34
Image Class APIs Important APIs CompressJpeg() CompressJpegN() ConvertN() DecodeN() DecodeToBufferN() DecodeUrl() EncodeToBufferN() EncodeToFile() 35
Example: Image Decoding #define SCREEN_WIDTH 320 #define SCREEN_HEIGHT 240 result TestDecoding(Form *pform) { result r = E_SUCCESS; String path("/media/images/sample.jpg"); // Create an Image instance. Image *pimage = new Image(); r = pimage->construct(); //Decode the image Bitmap *pbitmap = pimage->decoden(path, BITMAP_PIXEL_FORMAT_RGB565, SCREEN_WIDTH, SCREEN_HEIGHT); Osp::Graphics::Rectangle rect(0,0,screen_width,screen_height); Canvas *pcanvas = pform->getcanvasn(); //Get a Canvas instance r = pcanvas->drawbitmap(rect, *pbitmap); //Draw bitmap on the canvas r = pcanvas->show(); delete pimage; delete pcanvas; delete pbitmap; //Free the memory return r;} 36
ImageUtil Class Overview(since 2.0) ImageUtil class facilitates Image pixel format conversion Image scaling (such as resizing, rotating, and flipping) Important APIs ConvertPixelFormat() Flip() Resize() Rotate() 37
5 AudioRecorder
AudioRecorder Class Overview To Record the audio data from input device to a file Input source is automatically selected by current audio path(mic, Earphone) Provides Recording controls - Record, Pause, Stop, Mute Provides API to set Recording quality - Low / Medium / High Provides setting for maximum recording time and size Provides mute during recording Provides APIs for getting supported codecs and container formats Supported Recording formats - WAV, AMR, AAC 39
IAudioRecorderEventListener Response to control OnAudioRecorderStarted(): when the recording is started OnAudioRecorderPaused(): when the recording is paused OnAudioRecorderStopped(): when the recording is stopped OnAudioRecorderCanceled(): when the recording is cancelled OnAudioRecorderClosed(): when the recording is closed Other listener functions OnAudioRecorderEndReached(): when recording reaches the maximum size or time specified OnAudioRecorderErrorOccurred(): when there is an error occurred during Audio Recording 40
AudioRecorder Example result TestAudioRecording(void) { result r = E_SUCCESS; paudiorecorder = new AudioRecorder(); // Instantiate and register the AudioRecorder event listener plistener = new MyAudioRecorderListener; r = paudiorecorder->construct(*plistener); // Create/open the file handle for storing recorded data r = paudiorecorder->createaudiofile(path, true); // Start/resume recording r = paudiorecorder->record(); if(isfailed(r)) { delete paudiorecorder; delete plistener; } return r; } 41
6 VideoRecorder
VideoRecorder Class Overview Similar to Audio Recorder Except Construct with Camera Codec: encoding methods GetSupportedCodecListN(), SetCodec() Recording resolution: dimension of video GetSupportedRecordingResolutionListN(), SetRecordingResolution() Quality RECORDING_QUALITY_LOW, MEDIUM, HIGH Mode VIDEORECORDER_MODE_VIDEO_WITH_AUDIO, VIDEO_ONLY Format VIDEORECORDING_FORMAT_MP4, 3GP 43
IVideoRecorderEventListener VideoRecorder Listener functions OnVideoRecorderCanceled(): Video Recording canceled OnVideoRecorderClosed(): VideoRecorder closed OnVideoRecorderEndReached(): Recording reaches the end of the pre-defined file size or time OnVideoRecorderErrorOccurred(): Error occurred during recording OnVideoRecorderPaused(): Recording paused OnVideoRecorderStarted(): Recording started OnVideoRecorderStopped(): Recording stopped 44
VideoRecorder Example result VideoRecorderTestForm::OnInitializing(void) { result r = E_SUCCESS; Osp::Graphics::Rectangle rect(0, 0, 320, 240); bool modified=false; //Get Overlay proper overlay region dimension bool suitable = OverlayRegion::EvaluateBounds( OVERLAY_REGION_EVALUATION_OPTION_LESS_THAN, rect, modified); poverlayregion = GetOverlayRegionN(rect, OVERLAY_REGION_TYPE_PRIMARY_CAMERA); //Code snippet continued on the next slide.. 45
VideoRecorder Example } poverlayregion->getbackgroundbufferinfo( previewbufferinfo); pcamera = new Osp::Media::Camera(); r = pcamera->construct(*this); r = pcamera->poweron(); pvideorecorder = new Osp::Media::VideoRecorder(); r = pvideorecorder->construct(*this, * pcamera); r = pcamera->startpreview(& previewbufferinfo, false); pvideorecorder->createvideofile(filename, true); pvideorecorder->record(); return r; 46
7 AudioDecoder/Encoder, VideoDecoder/Encoder
AudioDecoder Class Overview Decodes compressed audio stream into raw audio data Supported audio decoding formats CODEC_MP3 CODEC_AAC CODEC_AMR_NB. Important APIs are Decode() Probe() IsPropertySupported () GetValue() GetSupportedPropertyListN() 48
AudioDecoder Example result DecodeMP3(void) { result r = E_SUCCESS; AudioDecoder dec; ByteBuffer srcbuf, dstbuf; File srcfile; int samplerate, value; AudioChannelType channeltype; AudioSampleType sampletype; //Code snippet continued on the next slide.. 49
AudioDecoder Example } r = dec.construct(codec_mp3); r = srcbuf.construct(src_buf_size); r = srcfile.construct(l"/home/test.mp3", "rb"); r = srcfile.read(srcbuf); srcbuf.flip(); //Flip the source Buffer position to zero r = dec.probe(srcbuf, sampletype, channeltype, samplerate); r = dstbuf.construct(dst_buf_size); r = dec.decode(srcbuf, dstbuf); //dstbuf will contain the decoded data return r; 50
AudioEncoder Class Overview Used to Encode raw audio data into compressed audio stream Supported audio encoding formats CODEC_AAC CODEC_AMR_NB Important API Encode() : Encodes the audio data from the source buffer and stores the encoded data into the destination buffer 51
VideoDecoder Class Overview used to decode compressed Video stream into raw video data Supported video encoding formats CODEC_H263, CODEC_MPEG4, CODEC_H264 Important API Decode() : Decodes the video data from the source buffer and stores the decoded data into the destination buffer Probe (): Probes whether the video data can be decoded. IsPropertySupported() GetValue() GetSupportedPropertyListN() 52
VideoEncoder Class Overview Used to encode raw Video stream into compressed video data Supported video encoding formats CODEC_H263, CODEC_MPEG4, CODEC_H264. Important API Encode() IsPropertySupported() SetValue() GetSupportedPropertyListN() 53
Summary Media namespace provides functionality to easy access a variety of media types including Image, audio and video Image files can be read and written via the Image class Player class can playback media contents Using Audio/Video Recorder, you can create content files AudioIn/Out are designed for low-level recording and playback a PCM audio. You can use them for sound processing and music applications Camera class provides camera-related functionalities including real-time preview and capturing Codec framework which supports Audio/Video Encoding/Decoding. 54
Practice Example Write a program which decodes image file or image buffer to a bitmap. Write a program which flips and rotate an image Write a program which converts from one image format to another image format. Write a program which plays an audio and video file using the player class. Write a program which plays multiple audio files simultaneously. Write a program which captures a frame of the video during video playbac k and display the same. 55
Practice Example Write a program which records audio with specified quality, time and size. Write a program which uses the VideoRecorder class for recording a video with different resolutions, size, time and codec formats. Write a program which records audio data using AudioIn and plays the sa me using the AudioOut class. Write a program which encodes pcm data to aac data using AudioEncoder class. Write a program which decodes mp3 data using AudioDecoder class. Write a program which encodes yuv data to mp4 data using VideoEncoder class. Write a program which decodes mp4 data using VideoDecoder class. 56