CommonSENSE: Software for displaying Full Motion Video for mission-critical C4ISR working positions

Transcription

1 CommonSENSE: Software for displaying Full Motion Video for mission-critical C4ISR working positions Jeff Malacarne Barco Defense & Aerospace Abstract This paper presents an overview of Barco s CommonSENSE software library which provides GPU-accelerated video capabilities for operator positions in a networked environment. CommonSENSE performs optimized, high performance video manipulation and presentation operations and simplifies the rapid development of visual application programs involving Full Motion Video. This software is offered as an option for multiple Barco rugged computing and smart display products and establishes a common Application Programming Interface for multiple types of equipment. Following a brief discussion of networked visualization concepts, an overview of CommonSENSE benefits, architecture, and capabilities is presented. Typical applications are also summarized in terms of related working positions for Unmanned Vehicle Systems, Naval C4ISR, and Army ground vehicles. INTRODUCTION For mission-critical systems, operators and decision-makers at multiple working positions must have quick and easy access to visual information from a rapidly increasing number of sources. Command and control information, computer generated imagery, and sensor feeds all provide critical input during surveillance/recognizance and combat management decision loops. Goals for reduced crew sizes, multi-mission flexibility, and increased interoperability between multiple systems also place growing demands for Processing, Exploitation, and Dissemination (PED) of information involving Full Motion Video (FMV). Networked visualization systems provide the means to effectively distribute and present visual information to people. These systems include displays, graphics, video, and radar processing capabilities to create situation awareness and understanding from complex data. The following paragraphs provide an overview of Barco s CommonSENSE (CS) software and discuss the application and benefits for mission-critical networked visualization systems. System designers and visual application developers can utilize Barco s smart displays with CS software to easily and rapidly integrate reliable, multi-channel, full frame rate FMV into a variety of operator working positions. NETWORKED VISUALIZATION CONCEPT Networked Approach In a networked visualization architecture, all information sources and user end points interface to a common network infrastructure. Therefore, all sources can be logically connected to all destinations and the functional system configuration is defined by system management software. For an ideal system, access to all data sources could be readily available at all working positions. In reality, varying security levels, resource limitations, and deterministic performance needs may induce some level of segregation involving multiple networks. However, controlled connections between multiple sub-system networks can be achieved using gateways.

2 A networked Service-Oriented Architecture enables the deployment of resources where they are needed most. For example, centralized servers can provide powerful on-demand execution of compute intensive algorithms and data processing operations. Live sensor streams can be multicast directly to all subscribing working positions as well as to server-based image processing services for complex automated analysis. Client working positions with visualization processing and presentation capabilities can be utilized to provide the smooth user interaction with graphics applications and sensor streams. Data/Sensor Processing Services Video Radar & other sensors Link Manned working positions Collaborative Display Other Services Gateway Figure 1 Manned working positions in a networked visualization environment provide user access and interaction with a variety of sensor and computing services. Networked Visualization Clients Clients in a networked visualization environment include several types of individual working positions as well as collaborative displays for sharing information among multiple people. For mission-critical C4ISR systems involving FMV, accelerated graphics and video processing at the client can greatly enhance system interaction and performance. Utilizing visualization horsepower at the client focuses visual computing operations at the display where the output is needed. For many system configurations, this architecture provides the highest visualization performance, ensuring that time-critical operation and interactivity are maintained. Barco provides several products for implementing Networked Visualization Clients including Display Processor Modules (DPM), SmartView intelligent displays, and Vista Consoles. Advanced computing, graphics generation, and video processing capabilities are provided in or near the display to create workstation or thin-client working positions with powerful visualization performance. The latest computing elements such as Intel processors and Nvidia graphics modules are offered at various levels of ruggedization. Vista 4500 DPM-2 SmartView DPM-3 Display Processing Modules Integrated Smart Displays Consoles Figure 2 Barco s display processing equipment provides powerful visual computing capabilities for harsh environments.

3 Networked Visualization Clients for C4ISR systems must integrate multiple HD video channels and computer graphics while providing mission-critical performance and reliability. CommonSENSE software provides this capability for Barco s rugged computing and smart display products without requiring any special or non-standard hardware. CommonSENSE OVERVIEW CommonSENSE (CS) software provides GPU-accelerated FMV manipulation and presentation for Networked Visualization Clients. Supported capabilities include video windowing, integration with graphics, and image processing, as well as screen distribution and recording. CS software resides at each client position and provides an integrated and uniform mechanism for integrating FMV and screen image operations into visual application programs. Customer-developed application programs specify video processing and display formatting operations via a network-transparent C++ Application Programming Interface (API). Control using a language-independent browser-oriented interface is also supported. CS provides services for decoding camera inputs and network streams, accessing recording files, applying visual processing operations, and generating/capturing the resulting on-screen display. Standard components and interfaces are employed within the CS software to provide a variety of capabilities that are accessed via a simplified higher level API. The latest standard GPU technology provides the acceleration needed for smooth and reliable operation. CommonSENSE Capabilities Applications API Video url/http Codecs Networking H.264/ RTP File Mgmt Linux & Windows Services Standards Ethernet Hardware GPU Processing CUDA Display Output OpenGL DS/ V4L2 Visual Processing Windowing C++ Library Video Input Screen Sharing & Recording Integration with Graphics Image Processing Unmanned Vehicle Systems Naval C4ISR Ground Vehicle Situation Awareness Figure 3 CommonSENSE provides GPU-accelerated FMV visualization capabilities based on open standards for mission-critical applications. Benefits CommonSENSE offers the following benefits for integrating Full Motion Video into operational display system applications:

4 Shortens time-to-market for application development CS provides an extensible set of controls for on-screen video operations that enable rapid development of a wide variety of visualization scenarios. Complex video and imaging operations are reduced to a handful of function calls that abstract low level implementation details to a generic set of functional controls. All video data is handled by CS and only basic controls need to be specified. Provides optimized performance for video operations Customer software programs can leverage CS s off-the-shelf performance to achieve reliable, multi-channel, full frame rate video. Video decoding or processing operations are accelerated using Barco s ruggedized GPUs. The latest GPU technology is used to provide standard video decoding or programmable processing of multiple pixels in parallel. Minimizes the impact of video on other computing operations Video operations are also tuned to minimize the utilization of computing platform CPU resources. In several cases, little or no CPU is required for displaying multiple video channels. Executing video decoding or processing operations on the GPU also serves to offload the CPU and provide a more balanced use of processing resources. This leaves more computing power for application level data processing and communications as well as spare compute resources for future evolution and growth. Enables seamless hardware evolution without changes to application software CS supports the insertion of future computing and visualization technology without requiring changes to application software programs developed today. Generic video controls are utilized, enabling the seamless technology refresh as new platforms, interfaces, and components become available. Provides a common software interface (API) for multiple networked Barco products Application programs can use a common interface for controlling multiple types of Networked Visualization Clients from Barco. Systems involving different products and configurations can use a uniform mechanism to perform dynamic control operations as well as setup pre-defined screen layouts. These layouts can then be invoked and controlled at various working positions independent of the type of client equipment. Architecture CS software runs on each Networked Visualization Client to provide FMV visualization operations for that working position under the control of customer developed application programs. In general, the CS API provides controls for video operations at a similar software level as common graphics APIs such as X Windows and OpenGL. Application Programs Network Streams CommonSENSE CommonSENSE Video Video Library/Service Library/Service Graphics Library/Service -X, DirectDraw -OpenGL -CUDA/OpenCL Cameras Video Input GPU Displays Figure 4 Generalized CommonSENSE Architecture

5 Two major application models are supported: 1. Networked service for multiple application programs Much like the X Window System model (xserver/xlib), an independent CS executable (csserver) can perform all video operations as directed by multiple networked application programs. Video channels are handled end-to-end by the csserver while controls for input selection, processing operations, and on-screen formatting are specified by application software. Multiple control applications can be run locally or anywhere on the network in any combination. Control over individual video channels can be isolated or shared among multiple applications. Socket-based communication to the csserver is provided by a thin C++ library (cslibremote) which is linked with the customer application software. 2. Video library for individual application programs CS capabilities (csliblocal) can be directly linked with an application program to provide tightly integrated video and graphics operations. Video channels can be handled end-to-end by the library while controls for input selection, processing operations, and on-screen formatting are specified by application software. Additionally, directly linking with csliblocal enables high performance sharing of video textures and graphics contexts. This greatly simplifies application development for use-cases including the following: Integration with interactive annotation graphics (e.g. telestrator-style highlighting). Integration of multiple video channels into 3D space along with maps/terrain to provide a geospatial reference for the sensor field-of-view. The development of a single full screen application that integrates all on-screen graphics and video for achieving real-time performance when using a non-realtime environment (e.g. Linux or WindowsXPe). Application Programs Application Programs Application Programs cslibremote xlib, glxlib, other Control Network Streams CommonSENSE CommonSENSE Video Service Video (csserver) Library/Service Graphics Library/Service -X, DirectDraw -OpenGL -CUDA/OpenCL Cameras Video Input GPU Displays Figure 5 Running CommonSENSE as an independent executable provides network-transparent control of video processing operations from multiple application programs. Application Program Control & Video Network Streams CommonSENSE Video Library (csliblocal) -csliblocal Graphics Library/Service -X, DirectDraw -OpenGL -CUDA/OpenCL Cameras Video Input GPU Displays Figure 6 Using CommonSENSE as a video library provides direct access to video content and integration with dynamic graphics processing.

6 Both models use an identical C++ API for all functions and both can run simultaneously on a single Networked Visualization Client. When running CS as an independent executable (csserver), a browser-oriented url/http interface is also supported for language-independent control without the need for Barco libraries. C++ programming samples and a browser-based utility are provided to simplify evaluation and training. Additionally, scenarios can be pre-defined and saved, then loaded and manipulated during normal operation. Like DirectShow, gstreamer, and other available video frameworks, CS provides a means for adding video capabilities into visual application programs. Unlike these frameworks, CS offers a fully integrated capability with verified reliability and performance. These commercial software frameworks and components are utilized within CS where beneficial. CommonSENSE CAPABILITIES CommonSENSE provides application controls for video windowing, integration with graphics, image processing, and screen image operations. Video Windowing Controls for selecting video inputs and formatting onto a display screen are supported. Selecting input sources Multiple directly connected cameras and network streams with different resolutions and formats can be selected for processing and display. The types and number of inputs that can be processed simultaneously depends on the equipment configuration. Adjusting camera inputs Controls for adjusting camera video inputs are provided including brightness contrast, hue, saturation, and multiple de-interlacing modes. Positioning video on the screen Program controls for positioning and sizing the video on the screen are provided. Each video input can be displayed as a user controllable video window with standard window dressing or as a program controlled picture-in-picture overlay with no window system dressing or interactive user controls. Program control of video window stacking order is also supported to ensure critical sources remain visible on-screen. Specifying video content to be displayed - Flexible controls for mapping the input video content into the on-screen window are provided. Two methods are supported: o Select input Region-of-Interest (ROI) A rectangular region of the input video content can be selected for display in the output window. The corners of the selected region are mapped to the corners of the output window. Manipulating the output window will effectively scale the selected region to fit the output window size. The selected input region can be rotated by 90, 180, or 270 degrees to accommodate various camera orientations. Mirroring is also supported for rear-view applications. o Specify pan/zoom/rotate (TSR) - Input video can be mapped into the output window using two-dimensional translate (pan), scale (zoom), and rotation parameters. The translated center of the input video remains positioned in the center of the output window and manipulating the size of the output window will not change the specified scaling. Rotations at any angle around the center of the input video are supported for aligning video from skewed camera mounts. Mirroring is also supported for rear-view applications.

7 Picture-in-picture overlay NOTE:CS screen snapshot processing an HD H.264 video stream into two windows with <1% CPU utilization Managed window Figure 7 Multiple video windows can be displayed as overlays or managed windows, often with little or no CPU utilization Figure 8 Application programs can construct complex visualization scenarios with a few CommonSENSE function calls.

8 Integration with Graphics The purpose of FMV for C4ISR applications is to provide information, not just high quality imagery. Communicating video-related information to people is greatly enhanced by the addition of accompanying references, tags, and highlights. This is often achieved by rendering associated metadata over video or supporting interactive annotation for highlighting content of interest. Additionally, video can be mapped onto 3D terrain or integrated with synthetic vision graphics to increase understanding of the spatial orientation of the sensor s field-of-view. Mixing video and graphics in a common window on the screen involves combining and synchronizing image content from multiple sources into a singe output. Video content is constantly changing at a periodic rate. Graphics content changes at various rates, depending on the type and purpose. Generating a common output image (without special hardware or system dependencies) requires that graphics be rendered on top of each new video frame. CS supports multiple mechanisms for mixing application-generated graphics with CS-generated video: Let CS do it CS can blend user rendered graphics images, such as text strings or icons, onto CS video windows. Application programs can load, position, and enable/disable multiple images and CS will take care of rendering and blending graphics with each new frame of video. This mechanism is designed to support metadata overlays with a mix of static and dynamically changing text or other 2D graphics content. Images are generated by the application program using any off-screen rendering system available (including X Windows or DirectDraw), then passed to CS for layering over video. Updates are loaded or selected by the application program only when the overlaid information needs to change. Let the application program do it CS can provide access to video content directly to the application program in the form of OpenGL texture handles. Application programs can then apply the video textures as desired within the GPU without any need to read back the video content. This provides a high performance path without any special video processing required on the CPU. Presentation of video onto the screen must be performed by the application program within its own output window. This method is designed to support integration with complex graphics such as rendering video over 3D terrain. If desired, the application program can read back the texture content from the GPU for additional processing. Work together Application programs can synchronize their dynamic graphics rendering with the CS video pipeline using a shared OpenGL context. The application program can control the synchronization with graphics updates and can directly render OpenGL graphics over each CS video frame. CS presents the combined output to the screen within a CS video window. Xy Alt Rsvpalt Signon This one! XXXXXX ASDFDGHt JHFasdf Asdfasdf Adsfasdfasdfasdf Asdfasdf asd XXXXXX Metadata graphics over video Interactive graphics annotation 3D terrain textured with video Figure 9 Examples of video integrated with graphics - Video integration with dynamic graphics applications is simplified using CommonSENSE.

9 Image Processing CS supports GPU-accelerated imaging processing for enhancing and combining video sources. A small set of baseline capabilities and interfaces are available off-the-shelf as an example of the performance achievable for typical operations. Custom operations can be developed rapidly on a project basis to address system-specific needs. In general, any video transformation operation (with video in and video out) can be added to the CS video pipeline as a GPU program, often with little or no added latency or CPU utilization. Many high performance imaging operations are possible using CUDA/OpenCL such as contrast enhancement, noise reduction and others. For more complex image analysis capabilities, CS can share video content with 3rd party application software via OpenGL texture handles. Baseline capabilities to support image processing operations include: Image enhancement Imaging operations such as stabilization can be applied to improve video quality. Sensor or system specific enhancements can be developed on a project basis. Fusion of multiple sensors Video from different types of sensors observing the same view, such as daylight and infrared cameras, can be overlaid and fused to combine the content into one image. Capabilities for aligning multiple inputs are supported, such as lens distortion correction, along with sample blend modes including alpha blending and red/blue coloring for monochrome inputs from infrared cameras. Sensor or system specific blending operations can be developed on a project basis. Integration with imaging applications CS video can be shared with video analysis applications that provide capabilities such as intelligent motion detection, target tracking/or other operator aids for increasing image understanding. Application programs can also use CS to highlight results as graphics over video within a CS video window. Stabilization Day/night sensor fusion Application generated motion detection highlights over video Figure 10 Image processing capabilities increase the understanding of sensor inputs. Screen Operations Screen content from any position on the network can be captured and streamed for remote display or recording at other working positions. Unlike many commercial software-based remote desktop or recording implementations, CS provides the performance needed for distributing and recording screens involving dynamic 3D graphics and live sensor imagery such as FMV and radar. GPU-accelerated processing is used for screen change detection and compression. Once captured, screen content can be recorded locally or can be streamed for remote live display and recording.

10 Live remote display - Any working position screen can be streamed to any networked working position for collaboration using visual information. The entire screen or multiple regions/windows can be streamed simultaneously. Multicasting can be used to distribute contents to multiple positions simultaneously. This supports sharing an individual screen on a large area display for viewing by a group as well as interaction between individual working positions. For example, when combined with CS s graphics integration capabilities, text/graphical annotations can be shared interactively between multiple positions to visually communicate aspects of interest. Recording Recording of dynamic screen content can be done on the local working position or remotely on any supported networked equipment. CS execution on networked storage devices for recording is also supported on a project-basis. Playback - Recorded screen content can be played and displayed locally or remotely at any position. Playback streams can be multicast to view recordings at multiple positions simultaneously. Playback controls include start/stop positions, play speed, and looping. Barco has integrated CS recording and playback capabilities into the MR-300 Mission Recorder product to provide a complete screen recording solution with a higher level recording API and sample GUI. Source Working Position Full screen sharing Other Working Positions Individual window sharing Networked Screen Recording & Playback Remote Collaborative Displays Figure 11 Dynamic screen content can be shared and recorded.

11 APPLICATIONS Barco s Networked Visualization Client products running CS software can be deployed into a variety of manned positions for mission-critical systems. Related positions for unmanned vehicle systems, Naval C4ISR systems, and Army ground vehicle working are summarized in the following paragraphs. WORKING POSITIONS for UNMANNED VEHICLE SYSTEMS (UVS) Visualization is a critical component for UVS Processing, Exploitation, and Dissemination (PED) operations. Typical environments that utilize visualization clients include: Mobile Shelters vehicle mounted Ground Control Stations involving multiple manned working positions for flight and payload control with full system-level communication. For visualization, relatively powerful semi-rugged servers, workstations, and thin-clients are used, often with multiple large high-resolution displays (e.g. 30 with 2560x1600 pixels). Dismounted Terminals portable Ground Control Stations with flight and payload control, but with limited communications capability. Processing and display equipment must provide useful visualization capabilities within size, weight, and power (SWaP) constraints. Fully rugged equipment is required to withstand exposure to the elements and rough handling. Transportable Operations truck and tent-based command posts used to guide tactical operations during a mission. Semi-rugged transportable computers and displays are used, often in conjunction with larger screen collaborative viewing sub-systems. Command Centers permanent facilities providing centralized decision making. Commercial compute servers, workstations, and displays are used with significant communications connectivity. Large screen displays and walls are used for center-wide collaborative viewing. Analyst Workstations office environment working positions for detailed exploitation analysis of near-real-time or recorded video. Professional grade computing and display systems are deployed with ultra-high resolution capability for maximum viewing fidelity. Each environment and use-case involves varying needs for visualization processing and display. However, compatible video feeds and recordings may need to be viewed and shared between any of these client positions. In addition to viewing raw or processed video sources, each position can add tags or other annotation information that is communicated along with the video via metadata. This is an important capability for communicating critical information in near-realtime or enabling later searching for recorded segments of interest. Mobile Shelters Transportable Ops Centers LAN/ WAN Dismounted Terminals Command Centers Analyst Workstations Figure 12 Networked visualization positions in a generalized UVS environment.

12 WORKING POSITIONS for NAVAL C4ISR The primary function for a Naval visualization system is to deliver a visual presentation of all operational data used to describe the current tactical and strategic situation when and where it can be used most effectively. Therefore, effective presentation capabilities at all related manned positions play a critical role in mission success. Networked Visualization Client equipment at these positions implements the functions necessary for generating or accessing distributed graphics imagery as well as receiving and presenting video and other sensor sources. Typical working positions that utilize networked visualization for naval systems include: Combat Information Center (CIC) positions The CIC is the ship s central command center executing the collection, processing, display, evaluation, and rapid dissemination of tactical information and intelligence. This involves various stations for command & control, communications, surveillance/reconnaissance, weapon system control, and mission planning. Platform management stations Positions for monitoring, alarm management, and control for the ship s main engines and other on-board systems, as well as damage control functions. Bridge Positions for ship command and navigation. Auxiliary positions - Briefing Rooms A collaborative center with a large area display for communicating situational information to a group. - Officer Cabins Remote informational positions with standard PC level equipment. - Portable Terminals Portable system access points using laptops or tablets, but with limited communications capability. Each working position and use-case involves varying needs for visualization processing and display. Viewing and sharing graphics, video feeds, and recordings between positions can increase situation awareness and mission effectiveness. Figure 13 All manned positions involving computer displays are candidates for networked visualization processing.

13 WORKING POSITIONS for GROUND VEHICLES Inside armored vehicles, access to information and correct interpretation and evaluation is crucial. With an advanced situation awareness capability providing real-time, all-round perception of the local environment in mission-critical situations, decision-makers have the information they need to anticipate risks and to act accordingly. Typical vehicle and support positions that utilize networked visualization for situation awareness include: Driver Driver access to real-time graphical vehicle instrumentation and platform information is required. Additionally, video feeds for driver vision are also required when driving blind. Low latency display of day and night sensor information from multiple cameras is mandatory to drive in all possible situations Commander Display of battle management information is critical at Commander s positions to direct vehicle operations. Video feeds and access to screens from other positions is also beneficial for additional situation awareness. Gunner Remote Weapon Systems are utilized for Gunner positions with strict requirements for low latency video feeds. Auxiliary positions - Observers and other crew positions Driving and personnel compartments often carry additional personnel with needs/benefits for accessing visual information such as monitoring surveillance cameras. - Mission planning and review operations Support personnel at remote facilities can be involved in collaborative planning, training, and review of recorded missions. When utilizing smart Multi-Function Displays, viewing and sharing video feeds, sharing display content (and control of positions), recording display content for training, and mission debrief are all possible using common equipment. Core vehicle positions require low-latency access to sensor information to provide real-time interaction with steering, weapons, and decision making systems. Auxiliary working positions can tolerate some level of delay from the live sensor, but may need to integrate or fuse the vehicle data with other information sources. Commander Driver Gunner Hard-real-time network Near-real-time & recorded access Mission planning & review Other crew positions Figure 14 Networked visualization positions in a ground vehicle environment.

14 CONCLUSION Networked visualization capabilities provide significant benefits for enhancing C4ISR mission effectiveness. Barco s Network Visualization Client equipment provides the performance and reliability required for mission-critical operations. For these rugged computing and smart display platforms, CommonSENSE software simplifies the rapid development of high performance visualization applications. Systems involving Full Motion Video, integration with graphics, image processing, screen sharing, and screen recording can utilize CommonSENSE to reduce time-tomarket and get the best performance possible from standard CPU and GPU hardware.