Image Composition System Using Multiple Mobile Robot Cameras

Transcription

1 Image Composition System Using Multiple Mobile Robot Cameras We have developed a high-precision video composition system which integrates a robotic camera able to move like a real cameraman with computer graphics equipment. The system is currently being used with two robotic cameras for program production. When compositing computer graphics with the camera imagery from two different cameras, errors in position sensing for the two cameras can accumulate and result in perceptible mismatch in the images. In order to achieve a more natural composition, it was necessary to improve on the accuracy of earlier position sensing methods, which operated by sensing wheel rotation when the mobile robotic camera moved. These methods were subject to error introduced by spinning and slipping of the wheels. We have developed a calibration method that is able to make highly accurate position corrections by having the mobile robotic camera capture a single reference marker placed in the studio as a reference point for the computer graphics coordinate system. 1. Introduction Recently, virtual studio technologies involving the composition of computer graphics with camera video is increasingly being used in program production. The virtual studio offers a wide variety of possibilities for expression in video, but to do so, a camera able to sense camera position and other data (pan and tilt *1 angles, zoom magnification, etc.) is required. Generally, camera position is done by sensing the rotation direction and revolutions of the wheels used to move the camera. However, depending on the condition of the studio floor, the wheels can spin or slip if the camera stops suddenly and this can introduce significant position sensing error. This error is reflected in the positioning of computer graphics when they are composited with the photographed image, and becomes visible to the viewer. Spin or slip occurring on only some of the wheels also introduces rotation of the pedestal, shifting the camera orientation and creating problems with changing shot angles. Thus, we developed a method for correcting camera orientation by installing a gyroscope on the mobile robotic camera to sense rotations of pedestal. We also developed a method for correcting errors in position sensing by capturing a pre-marked reference point in the studio with the camera, and using camera pan, tilt and height values to compute the current position and calibrate the system. The system has enabled highly *1 Left-right camera motion is called "pan", while up-down motion is called "tilt". accurate compositing of computer graphics with camera video, even after moving the camera over long distances. In this article, we discuss the specifications of the mobile robotic camera and the newly developed calibration methods for camera position and orientation. We also describe examples of use of multiple mobile robotic cameras in a virtual studio system used for television program production. 2. Mobile robotic cameras The mobile robotic cameras are composed of (1) a camera platform with pan, tilt and lens control and (2) a robotic pedestal able to move forward, backward, left and right and able to move the platform up and down. An external view and schematic diagram are shown Figure 1, and specifications are given in Table Camera platform In order to faithfully reproduce the movement of a cameraman, from fine detail to rapid motions, AC electric motors with high torque and little backlash are used to drive the camera platform, together with rotary encoders able to detect high angular resolution. The motor angles are detected at an extremely high resolution of Focus and zoom of the lens are controllable digitally, and the lens is connected by serial communications cable to the control PC inside the mobile robotic camera. 2.2 Robotic pedestal The robotic pedestal has four wheels able to move Broadcast Technology no.38, Autumn 2009 C NHK STRL 9

2 Table 1: Robotic camera specifications (1) Camera platform Pan Tilt Operating angles Max. rotation speed Max. torque Angle detection resolution Positioning accuracy +180 to rps* 34 Nm rps 54 Nm (2) Robot pedestal Motion speed 2 to 1,000 mm/s (1) External view Position reproduction accuracy 10 mm Zoom, focus 6 Camera: HDL-40 Tally signal Video signal Lift range Lift speed Lift position reproduction accuracy Permissible load 1,170 to 1,690 mm 1 to 500 mm 0.5 mm 25 kg Tilt Weight Approx. 200 kg Lens: HJ16 x 8B Pan Dolly dimensions (WxDxH) Power mm AC 100 V Lift Pedestal control Dolly control Control PC (built-in) Max. power consumption * round per seconds 900 W Wheel steering (2) Structural diagram Wheel rotation Figure 1: Mobile robotic camera independently (2 driven, 2 following), so it is able to move in all directions while keeping the pedestal itself pointed in the same direction. We set the maximum speed at 1 m/s, based on an analysis of cameraman movements which found the maximum speed used to be 0.8 m/s. Although pedestals used by cameramen normally have three wheels, we equipped the robotic pedestal with four for additional stability. With four wheels, we were able to increase the maximum height of the camera to 2 m, and the camera is able to stop quickly without falling over, even at maximum speed. Also, considering future plans to convert the camera video and control feeds from the current wired system to wireless, we equipped the pedestal with batteries capable of two hours of continuous operation. In order to prevent collisions with people or parts of the set, four safety systems were also incorporated (Figure 2). The pedestal is equipped with motion-direction LED indicators which illuminate according to the direction of the wheels. As well, 12 infrared sensors around the outside of the pedestal cause the pedestal to stop if an obstruction is detected. Further, two emergency stop switches are placed in easy-to-see and easy-to-operate locations, and touch sensors are placed around the bottom edge of the pedestal, which cause it to stop in case of collision. 10 Broadcast Technology no.38, Autumn 2009 C NHK STRL

3 Motion direction indicator LED Emergency stop switch Infrared detection sensors Contact sensors Figure 2: Mobile robotic camera safety equipment Without correction With correction Figure 4: Mobile robotic camera position accuracy (above: image at initial position, below: image after 20 round trips) 3. Position detection for the mobile robotic camera Earlier methods, which used rotational sensors attached to the drive wheels to detect the amount and direction of wheel rotation to sense the position of the mobile robotic camera, were able to detect the current position relative to the starting position. However, with these methods, error is introduced if the drive wheels spin or slip. In particular, when operated over long periods of time, such as from the start of rehearsals through to the end of live shooting, the repeated starting and stopping motion can result in large error accumulating. The error introduced can be divided into two types, depending on how the wheels have spun or slipped. (1) When the direction and amount of spin or slip in the two drive wheels is different, rotation of the whole robot results Movement (2) When the direction and amount of spin or slip in the two drive wheels is the same, no rotation of the whole robot results but the camera orientation is shifted The shot angle on the subject changes much more for case (1), when the robot rotates, than for (2), with no rotation (Figure 3), and the amount of error increases in proportion to the distance from the subject, so it can have a major effect on the photographed image. 3.1 Correcting camera angle detection error using a gyroscope To reduce the effects of rotational errors, it was necessary to detect the amount of rotation of the mobile robotic camera due to spinning or slipping of the drive wheels. To do so, we installed a gyroscope on the mobile robotic camera and used it to detect rotation. A gyroscope is able to detect rotation angles very accurately, but the output signal exhibits a drift phenomenon, causing it to fluctuate over time. To reduce the effects of this drift, the signal is only used to detect rotation while the mobile robotic camera is in motion. We corrected the pan angle according to the rotation angle detected by the gyroscope, and reduced the effect on the photographed video. Figure 4 shows an example of images taken at a distance of 2 m after 20 round-trips, with and without correction using the gyroscope rotation output. It shows that correction eliminated almost all effect on the image. Figure 3: Mobile robotic camera rotation error Broadcast Technology no.38, Autumn 2009 C NHK STRL 11

4 Origin D Z h Y X D (1) Front view (2) Top view Figure 5: Calibration principles Origin 3.2 Calibration of camera position by photographing reference points When both of the drive wheels spin or slip with the same amount and direction, the robotic camera does not rotate and the direct effect on the photographed image is small. Error is still introduced into the detected position, however, and this appears as a mismatch when computer graphics are composited with the video. To correct the position sensing error, in addition to using the rotation and direction sensors on the drive wheels of the robotic camera, we detect the position using a fixed external reference point. Using the fact that we can detect the pan and tilt angles and height of the camera platform, we developed a method to compute the camera position by pre-marking the origin of the computer graphics coordinate system on the studio floor. Then, by photographing this reference point in the center of the photographed image, we can determine the robotic Virtual composite video DVE Video Real-time graphics rendering system DVE Video input Computer graphic operator terminal Camera image Mobile robotic camera Remote operation device Network Figure 6: System diagram 12 Broadcast Technology no.38, Autumn 2009 C NHK STRL

5 Feature camera orientation from the camera platform pan angle, and can compute the distance using the camera height and tilt angle. From these, we can accurately compute the ( X, Y, Z ) coordinates of the camera. Defining the X, Y and Z axes as shown in Figure 5, (Xc, Yc, Zc) as the position of the mobile robotic camera,,, and h, respectively, as the pan angle, tilt angle and camera height when photographing the origin, and D as the distance to the origin, we have the following relationships: Xc = - D. sin Zc = - D. cos (1) tan = h / D From these, the position of the mobile robotic camera, (Xc, Yc, Zc), is given by: Figure 7: Examples of video composition used in "This week and You" Figure 8: Example of computer graphic composition using two mobile robotic cameras (above) and studio view (below) Broadcast Technology no.38, Autumn 2009 C NHK STRL 13

6 Xc = - h. sin / tan Zc = - h. cos / tan (2) Yc = h This allows the position of the mobile robotic camera to be calibrated easily and accurately no matter where it is in the studio, by simply pointing the camera so the origin point is in the center of the field. 4. Examples of operation in live broadcasts 4.1 Operation in NHK's "This Week and You" For "This Week and You", an image composition system using mobile robotic cameras as shown in Figure 6 was used. Position data for the mobile robotic camera was detected and sent repeatedly to the computer graphic equipment, so that for shooting, all that needed to be done was to match the coordinate system of the video composition equipment with that of the mobile robotic camera. This allowed camera and computer graphic images to be matched in a relatively very short time. The cameraman decided the shots and saved them in shot memory during rehearsals, so that shots for the live broadcast could be done by simply pressing the play button. Besides video composites, regular shots such as long or group shots were also used. Figure 7 shows examples of composite shots. using mobile robotic cameras able to sense camera position and other data, and have added this to operations for live program broadcasts. We have also developed methods to easily correct error in the detected position of the mobile robotic camera, even during broadcasts. These include a method to (1) correct rotational error using a gyroscope, and (2) calibrate camera position by photographing a reference point. The methods are also applicable to regular virtual studio cameras, so calibration can now be done at any time rather than only prior to rehearsals, as was done previously. This has greatly reduced the amount of error in position sensing, allowing more natural composition of computer graphics with live video. In the future, we plan to study systems with even more mobile robotic cameras and make further improvements to usability, such as using data regarding the relative positions of the cameras to allow all camera positions to be calibrated based on calibration of a single camera position. (Kazutoshi MUTOU, Hitoshi YANAGISAWA, Takao TSUDA, Seiki INOUE) 4.2 Operation in NHK's "Today's Close-up" For "Today's Close-up" we created a virtual photography system with two each of mobile robotic cameras and computer graphics systems. Using this system, we were able to create composites of a single computer graphic with the video taken from each of the cameras, allowing us to switch effectively between the two views for the broadcast. Figure 8 shows examples of video composites used in "Today's Close-up". When increasing the number of cameras used for virtual photography, if the error in position sensing for each camera is not extremely small, they can add and result in significant mismatch when switching between views. In order to reduce this error, camera positions must be recalibrated periodically, even during shooting. However, earlier systems required the cameras to be returned to an origin in order to be calibrated, making it difficult to perform frequent calibrations due to obstruction from other cameras and equipment in the studio. The system we have developed enables calibration by simply capturing the origin point with the camera. This makes more accurate composition possible by allowing recalibration to be done at any time. 5. Conclusion We have developed an image composition system 14 Broadcast Technology no.38, Autumn 2009 C NHK STRL