By Lewis Graham Assistant Director, ASPRS Photogrammetric Applications

Transcription

1 Introduction Mobile Mapping Systems (MMS), while existing for a number of years, are now emerging as a mainstream technology for direct three dimensional data collection in corridor mapping applications. In fact, the interest level in this area is so great that, during the fall 2009 ASPRS/MAPPS conference, we decided to start a new MMS committee within the Photogrammetric Applications Division (PAD) of the ASPRS. The inaugural meeting of this committee will be held during the 2010 ASPRS Annual Conference in San Diego, California (April 26-30, 2010). As we begin our efforts to stand up this new committee, we wanted to provide an overview of Mobile Mapping technology for the membership. By Lewis Graham Assistant Director, ASPRS Photogrammetric Applications Division Why Mobile Mapping? Mobile mapping platforms allow the collection of direct 3D precision point information and conventional still and/or video imagery while traveling at highway speeds. The achievable absolute accuracy of the 3D data can be as good as 2 cm (following adjustment to control). Many potential acquirers of the technology are questioning the value of MMS as compared to conventional survey or static laser scanning. Of what value are terabytes of point cloud data when one can simply field collect specific data using conventional techniques? These naysayers are missing the real advantage of MMS moving data collection operations from the field to the back office. Not only does this paradigm shift significantly improve safety for data collectors (a major concern in highway work), it will also completely change the economics of feature collection. No longer will 10 days of good weather be needed to collect a 20-mile highway corridor by a field survey crew. Drive the mission with an MMS in 30 minutes and do all the data processing in the back office. The astute reader will also recognize the opportunity for outsourcing operations that are not possible under conventional collection techniques. This shift will rapidly change the industry. Mobile Mapping System Hardware Overview For the purposes of this paper, we consider an MMS to be a unified platform comprising one or more short range laser scanners (e.g., 100 to 200 m range), a high precision positional system and, optionally, one or more electro-optical cameras. Of course, the platform also includes the computer, storage and operational software necessary to control the mobile scanning operations. An example of the LYNX mobile mapping system from Optech, Inc. ( is depicted in Figure 1. This is a road vehicle deployed configuration suitable for highway mapping operations. Since the system is mounted on a self-contained rigid platform, it can be easily redeployed to various collection vehicles. While the information presented in this paper is applicable to the generic mobile mapping system, we will use the Optech LYNX system as our specific example. Mobile mapping systems generally comprise mobile laser scanning systems and digital ( Electro- 222 March 2010 PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING

2 Optical, EO) cameras. A laser scanner provides point data that includes range information (the distance from the scanner to the scattering object) as well as a measure of the return energy (called intensity ). The lasers emit relatively low energy pulses in the infrared spectrum and thus the pulses are both invisible and eye safe. Unlike a tripod-mounted stationary laser scanning system, the laser scanners of an MMS are generally attached to a transport vehicle (All Terrain Vehicle, truck, HiRail system and so forth). This motion of the platform results in a scanning geometry much different from a stationary tripod scanner. In fact, the geometry and indeed the data reduction processing is quite similar to the techniques used in airborne scanning systems. A rear view of a LYNX MMS is depicted in Figure 2. From left to right (as viewed from the rear), the components are: Left Laser Scanner Inertial Measurement Unit with GPS antenna Right Laser Scanner Right digital camera Not visible in the photograph is the left camera and a wheel mounted Distance Measuring Instrument. The processing computer, control console and data storage devices are situated inside the vehicle. The system generally requires a two person crew for safe operation; one to operate the vehicle and the other to operate and monitor the sensors. The laser range finder (Figure 3) is a rotating mirror system that swings the laser beam through 360. The unit can rotate at speeds from 80 to 200 revolutions per second. The laser is pulsed with frequencies up to 200 khz. The laser scanners are typically mounted at a 45 angle with respect to the vehicle track. The scanning pattern is a function of the mount angle, the forward vehicle speed, the rotational speed of the scanner and the pulse repetition rate. All of these parameters are determined during mission planning (see the green lines of Figure 4 for a typical LYNX scanning pattern). Since the lasers rotate through 360 degrees, the sides and tops of features (e.g., bridge structures, building faces, overhead wires and so forth) are imaged. The laser scanners used in the LYNX system are multiple return systems. This means that if an outbound laser pulse reflects from more than one surface where those surfaces are at different distances from the receiver, the multiple return pulses can be detected (the LYNX can detect up to 4 return pulses). This is very useful during post-processing for detecting light porous features such as vegetation. Mobile mapping systems optionally include digital cameras (still and/or video). The LYNX system discussed in this paper can be equipped with up to two cameras. The cameras are Red- Green-Blue (RGB) visible spectrum cameras with 2 Megapixel (MP) sensors. The cameras are statically mounted and are used to provide context for lidar data editing, a colorization source for color attributing the laser points or ancillary information such as traffic sign content. Since the platform of an MMS is in motion, sensors must be included in the system that can determine the precise location of the platform frame at a very high rate (typically several hundred updates per second). The common solutions in use today are generally called position and orientation (POS) systems. They consist of a Global Positioning System (GPS) receiver), an inertial measurement unit (IMU) and a Distance Measurement Instrument (DMI). The principal measurement component is the IMU. This device contains sensors that can detect rotation (gyroscopes) and acceleration (accelerometers). The computation unit of the IMU essentially runs a real time program, using motion equations to determine the location (X, Y, Z) and the orientation (pitch, yaw and roll) of the platform. The IMU needs to be reinitialized on a periodic basis with its true X, Y and Z location. The GPS is used for this purpose. Unlike airborne continued on page 224 PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING March

3 continued from page 223 systems, the mobile mapping system also includes a distance measuring instrument (DMI). The DMI is typically a device connected to one of the wheels of the vehicle. It provides linear distance information to the POS in the event of a GPS outage (e.g., travelling through an overpass, tunnel or urban canyon). Applanix ( is perhaps the world leader in commercial POS devices. The Applanix POS 420/LV is the unit used by the Optech LYNX system. The components of a POS/ LV-420 system are depicted in Figure 5. Clockwise from the upper left, the components are the Distance Measurement Instrument, the Inertial Measurement Unit, the GPS antenna and the POS computational system. Additional information regarding the Applanix POS/LV system is available at: In most MMSs (including the LYNX), the collection of positional information, image information and laser data all occur in relatively independent units. The common parameter that will be used in post-processing to synchronize all of these various data is the precise system time provided by an overall system clock. During data collection, the system is activated and deactivated either by automatic mission control software or simply by the operator turning the collection devices on and off. Data are streamed to a storage device, typically a removable hard disk. The data streamed during acquisition is in raw format and must be post-processed before any useful information can be derived. Overview of a Typical MMS Flow The typical commercial MMS acquisition and processing flow is depicted in Figure 6. Each of the processing blocks is discussed in the following paragraphs. Mission Planning, Staging The primary concerns (other than weather, of course) are computing the sensor parameters to achieve the desired data density (e.g., sensor rotational frequency, vehicle velocity, pulse repetition rate) and the GPS support infrastructure (what is the optimal acquisition time in terms of GPS constellation, do I need ancillary base stations, can I use a Continuously Operating Reference System network, can I use a Virtual Reference System, and so forth). Very, very careful GPS planning is absolutely essential to a high accuracy product. Object space considerations are also important. The MMS lidar sensors have relatively low elevations with respect to the object space. Thus, consideration must be given to foreground objects occluding more distant objects. For example, if dual railroad tracks are to be imaged by a pass down just one set of tracks, the outer side (with respect to MMS travel path) of the second set of tracks will be occluded. This may or may not be important, depending on the collection desires of the mission. Figure 7 depicts (circled in red) a self-occluding rail. The MMS travelled down the rails in the left side of the figure (shown in red). Due to the low look angle of the laser scanner, the outer sides of the rails of the adjacent tracks are shadowed from the laser pulses and hence were not collected. If GPS outages are expected in the mission (e.g., from urban canyons, overpasses, tunnels, and so forth) or if accuracies higher than that achievable from augmented GPS are required, then a supplemental control plan needs to be established. This will involve identifying lidar recognizable targets in the environment or the placement of targets where no natural targets are available. Obviously the targets can be surveyed prior to or after data acquisition. We strongly recommend establishing the accuracy envelope prior to acquiring data. We have often seen companies spend an inordinate amount of time in lidar geometric processing, tweaking the data to ever more accurate values without having an end goal. This can waste considerable production time. Drive the Mission Murphy is always in charge so acquisition crews have to be very careful to analyze how a change in 224 March 2010 PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING

4 the mission (time, weather conditions) will affect the desired capture parameters. For example, an MMS will not properly image a pavement surface that is wet. Most companies do some level of field checking of data to ensure that they have valid GPS/IMU data and that they have full ground coverage of the data takes if they are operating remotely from the data processing center. Position and Orientation System (POS) Post Processing The data reduction for POS processing is typically sensor independent. Thus, the process is the same for aerial cameras, lidars, Synthetic Aperture Radar (SAR) or whatever the sensor may be. This is an advantage for companies who deploy multiple sensors (they can do GPS/IMU data reduction using a single work group). The output of the process is a trajectory file that is a 3-dimensional track of the collection path. At each node on this track (the nodes are typically placed at 200 Hz intervals) is a data tuple containing X, Y, Z positions, Pitch, Yaw, Roll orientation angles (phi, omega, kappa in photogrammetry terminology) and support metadata such as standard deviations and so forth. This step is critical to achieving accurate data. The application used for the POS/LV system is POSPac from Applanix. The Applanix output trajectory is called a Smoothed, Best Estimated Trajectory (SBET). An ancillary file containing estimated accuracy data is also produced. Lidar Geocoding We made up this term because there is no real standard terminology for this step in the industry. It is often called postprocessing but this confuses the operation with some of the down-stream processes that follow. It involves converting the raw collection data such as time, mirror angle, sensor orientation relative to the vehicle, vehicle orientation relative to a reference system (e.g., WGS-84), and so forth into object space coordinates (most often return points). The inputs are the raw lidar data, system calibration parameters and the processed GPS/IMU trajectory (the SBET and accuracy files). This processing is always performed in proprietary software provided by the lidar sensor manufacturer. For Optech systems this software is called DashMap. The output is nearly always in the ASPRS standard format called LAS. The current version is LAS 1.2. LAS encodes the return points (there can be several return points per emitted laser pulse) into a binary file with attributes on a per-point basis. System Calibration Refinement System calibration of an MMS involves calibrating each of a number of independent sensors. The rigorous calibration schedule for each sensor is recommended by the manufacturer but must be monitored by the user for unusual events. For example, the calibration of the interior orientation (IO) of a camera may be on a recommended six- month schedule. However, if an event occurs that could change the calibration (for example, a lens replacement), then an unscheduled IO must be performed on the camera. All sensors on the platform must be calibrated relative to the sensor platform reference coordinate system (usually defined by the POS). This can be considered relative orientation. Generally, the MMS will have calibration reference files for each sensor which provide the base line calibration of the platform. Best practices dictate that calibration should be checked and, if necessary, refined with each data run (i.e., on a daily basis). In airborne systems, this process is often referred to as bore sighting. The MMS manufacturer should provide both documentation and training for platform calibration. Some software packages (e.g., TerraMatch from Terrasolid) include tools for performing platform calibration of the laser scanners simultaneously with lidar absolute geometric correction. Using a simultaneous procedure can greatly expedite processing. It is useful to note that platform calibration of an MMS is considerably different from an airborne system since most airborne systems involve calibration of a single sensor. The MMS, on the other hand, typically requires calibration of two laser scanners and two or more cameras. Maintaining calibration records is vitally important to the success of generating accurate output products. Projects invariably involve multiple day collection with their associated system calibrations. Without exact record keeping it is quite easy to apply the wrong system calibration to the data. Project Creation Following geocoding, lidar data are moved into a processing project in whatever software is being used for project management/data processing. The general idea at this stage is to bring the data into a system where it can be managed and visualized. All processing companies use a tiling scheme for data processing. An example is shown in Figure 8. This figure depicts the trajectory of a LYNX system mounted on a rail vehicle. The areas of green are tiles that have been defined in sections where lidar data are to be processed. There are two reasons for tiling data. The first is that takes of lidar data have overlap and thus the takes need to be combined in the overlap regions to ensure consistent processing. The second is that a lidar data take, in general, contains too many data points to be effectively processed as a unit (bogs down comcontinued on page 226 PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING March

5 continued from page 225 pute resources and cannot be effectively distributed to multiple editing persons for simultaneous processing). Project Data Preparation ;^\jgz -/ ANCM igv_zxidgn gzy VcY EgdXZhh^c\ I^aZh \gzzc # Tile population is the process of copying lidar points from the individual data takes to the tiles. Most processing companies will perform any necessary projection and datum transformations at this point as well. For U.S.-based commercial projects, this is typically moving the data from ITRF (or WGS) to a state plane horizontal system and an orthometric vertical system (most usually NAVD-88). A very large quantity of images can be collected contemporaneously with the lidar data. The project management system should provide functions for importing the location of the camera at the time of exposure (the camera station ) and displaying these locations as icons in the project view. A camera station, as well as the associated image, is depicted in Figure 9 below. The camera stations are depicted as orange triangles superimposed on the lidar ortho displayed in the top view. The white triangle is the selected station. Its associated image is displayed in the lower pane. The typical workflow for defining near-track assets will be to drive through the lidar data and use the associated images for object identification and attribution (for example, one of the obvious attributes for a sign will be its text). Gross QC ;^\jgz./ 8VbZgV HiVi^dch VcY V HZaZXiZY ^bv\z# This is a quick look QC to ensure that there are no gross errors in the lidar data (sensor dropout, voids in coverage due to a lack of overlap between data takes, occlusions, unacceptable noise, and so forth) that will require a reacquisition. An example of an occlusion is depicted in Figure 10. Here the lidar data have been rendered as a lidar ortho, allowing the orange background to bleed through where no lidar data exist ( voids or holidays ). Note the orange triangles in the lower right of the lidar ortho image. Here the track edge bank dropped off too steeply to be imaged by the lidar. Geometric Correction ;^\jgz &%/ DXXajh^dc kd^yh h]dlc ^c dgvc\z # 226 March 2010 In an ideal world, this step would not be required. However, the world is far from ideal (sometimes very far!). Due to calibration problems, GPS/IMU errors and unmodeled perturbations in the system, the geocoded lidar data will have errors with respect to the actual object space scatter points. The general idea of geometric correction is to use rigid elements in object space as well as surveyed control to correct the lidar data (e.g., a building edge, highway paint stripe and so on). This PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING

6 process is somewhat analogous to block bundle adjustment in photogrammetry. Commercial grade lidar systems such as LYNX have very stable internal geometry and thus this process tends to be limited by the accuracy of the GPS/IMU results. Commercial products such as Terrasolid s TerraMatch product have been enhanced to include geometric correction of MMS data and include simultaneous relative orientation correction (platform calibration) of the laser scanners. It is critically important to note that no currently available geometric correction software back propagates the absolute geometric correction into the GPS/IMU trajectory file (in the case of an Applanix system, the SBET). This means that, in general, the trajectory cannot be used as a precision reference within the project. For example, it is tempting in rail work to consider using the trajectory as the precise track centerline (assuming that the GPS antenna is offset to the centerline of the vehicle). However, in GPS compromised regions, the trajectory will drift widely from the geometrically corrected lidar data. Thus, the trajectory can only be used as the seed for initial geometric correction. Filter Assignment Once lidar data have been geometrically corrected, the major task is to assign to each point the class of the object from which that point was scattered. All points begin in a class called Unknown. Then, using various tools and techniques, the points are assigned to appropriate classes such as Ground, Rail, Wire, Water, Building, and so forth. The Class of a point is one of the attributes maintained in the LAS data structure. Due to the large number of points, automation must be used in this process. The typical flow is to assign a classification algorithm (also called a filter) in an interactive step and then actually perform the processing in an unattended process ( batch processing). Some companies use a single, general purpose algorithm for this processing regardless of terrain type. Others assign different specialized filters to different regions of the project based on object space content (agricultural, forest, light urban, heavy urban, and so on). The design of filters is probably about one third of the way through the maturity curve so there is a lot of room for improvement. Readers familiar with airborne lidar data processing will recognize this flow as analogous to that domain. However, the objects being recognized can differ from the airborne case and certainly the algorithms. as a deciduous tree overhanging the side of a building next to a road. In fact, if the goal is to classify bare earth, the algorithmic challenge is essentially the same as extracting terrain from stereo images using a correlation algorithm. MMS data present new challenges and opportunities in this area of Automatic Feature Extraction (AFE). The data densities are sufficiently high as to provide a bases for greatly improved algorithms as compared to airborne laser scanners. For example, in Figure 11 we depict the result of applying a rail extraction filter (actually a rail extraction AFE algorithm) to lidar data. Here we have colored the extracted rail hits red and superimposed the result on a lidar ortho. These will be the raw tagged data that feed the rail geometry extraction algorithms described in the previous section. Post Filter Edit As noted in the previous paragraph, the automatic filter (classification) algorithms will have misclassified data. In this step, manual editing is performed to review the results of automatic processing and manually correct the errors. As you can imagine, this is a very tedious process, again not unlike what is done with correlated stereo data. We have been encouraging the industry to break this process into two stages; a check stage and a fix stage. Lower skilled workers are used in the check stage to review the classification quality of tile and perform minor fixes. Major problems are tagged (using an annotation system) for more skilled technicians. These more skilled technicians, in turn, review the queued problems and take appropriate actions to correct (which often include cycling back through automatic filtering). Final QC In this step the tiles are reviewed (ideally by persons not involved in the previous data processing) to ensure that they meet the raw data production criteria. These criteria include classification density, percent of erroneously classified points and a final look at horizontal and vertical error. We depict, in Figure 12, a QC tool showing a profile of classified rail data (the red tracks in the upper part of the window, shown in profile view in the lower window). Filter Application This step is the actual application of the filters that were assigned in the preceding stage. Many companies have processing clusters with distributed processing, allowing this application to occur as a distributed, off-loaded task. Otherwise, they are simply run on production workstations. This step requires calendar time as opposed to person time since it is being run in batch. These algorithms do an imperfect job of classification. The quality is a function of the type of object space being processed. You can imagine algorithmically difficult situations such continued on page 228 PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING March

7 continued from page 227 Breakline/Feature Addition Derived products that require non-cultural features with resolution (note that I am saying resolution, not accuracy here!) higher than the average post spacing of the classified lidar data will require the addition of supplemental breaklines. Breaklines can be directly collected from lidar orthos (2D) or lidar point clouds (3D). GeoCue Corporation, some years ago, developed techniques to render synthetic stereo images from the lidar data itself, using either the lidar intensity values or data from existing ortho coverage to color the pixels. These synthetic stereo models can then be used on a conventional stereo extraction workstation (e.g., SOCET SET) to digitize breaklines (or any features discernable in the synthetic image, for that matter). This technique is now the most commonly used way of collecting supplemental breaklines from airborne lidar data since no auxiliary image data are required. The products that clients will demand from MMS data will greatly expand the need for heads-up digitizing. For example, we anticipate that rail geometry, while very amenable to AFE, will require some vector data addition in confused areas such as switches and frogs. Derived Product Generation Most commercial customers are after a derived product such as a gridded DEM, vectorized contour lines, road features such as top of crown, track geometry, and so forth. The generation of these products is performed using the same tools as would be used for deriving products from correlated surfaces derived from image data. As with high density correlated data, a vexing problem is the aesthetics of the created product. For example, contour lines typically exhibit jaggedness if smoothing algorithms are not applied. As you can imagine, this is an area of hot debate in the production community since making products pretty is a very labor (i.e., cost) intensive process. Mobile Mapping is adding new challenges to the data extraction process. For example, highway extraction will include soft features such as pavement crown, edge of lane, edge of shoulder, and so forth. These features will be a real challenge for automation since they are seldom cleanly viewable by a sensor (e.g., weed overgrowing the shoulder). We anticipate a big spurt in research and development efforts aimed at automating MMS feature extraction. Data Management The volumes of data collected by an MMS dwarf an airborne lidar system. Typical point densities in airborne systems are several points per square meter. The near-field data density of a LYNX is on the order of 2,500 points per square meter! Thus, the collected data can average over a gigabyte per kilometer of single pass collection. Add to this digital images being captured at up to two frames per second and you begin to realize the importance of adequate storage and a very organized data management system. It is important to note that while small projects can be processed using workstation-centric tools alone (after geocoding), this is a bit like doing production while looking through a single soda straw given enough time and money, you could do it but you wouldn t from a practical point of view. One of the frequent mistakes we have seen with companies diving into the MMS technology world is underestimating the equipment and software that will be required to successfully process the data once they have been collected. This is particularly true of companies who have traditionally collected projects using static laser scanners and are now moving to mobile systems. Our advice is to budget about $275,000 for post-processing hardware and software for managing the data from a single MMS. Summary The Mobile Mapping System market truly has come of age! This exciting new technology is offering expanded opportunities for companies to engage in new projects and develop entirely new products for customers ranging from engineering companies through state/local governments to a myriad of transportation concerns. While the technology is more than adequately mature to offer very valuable products today, many exciting avenues present for new research and development, particularly in the area of Automatic Feature Extraction. The new ASPRS MMS Committee will be engaging in all aspects of Mobile Mapping, from developing best practices for data collection and processing to creating exemplar procurement templates for clients wishing to acquire data. We invite you to join us at the inaugural meeting of the MMS committee to be held during the 2010 ASPRS Annual Conference in San Diego (April 26-30, 2010). Author Lewis Graham, GeoCue Corporation 9668 Madison Blvd., Suite 101 Madison, AL , (fax) lgraham@geocue.com 228 March 2010 PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING