Resetting Channels for Players, Recorders and CODECs Without Rebooting

Transcription

1 February 7, 2013 Resetting Channels for Players, Recorders and CODECs Without Rebooting Wes Whitnah, Grass Valley Software Engineering Abstract Occasionally a channel will develop a problem that persists for a long period of time or that just doesn t go away. For players this might be jittery video, drop outs or artifacts in the video output. For recorders this might be reported encoder errors or problems with the video that is recorded. For either the channel itself may not be responsive. The natural reaction to such problems is to reboot the system. However, a reboot requires the entire system to shut down including other channels that are still working. There is a better way. For many channel problems a simple reset is the only thing needed to get it working again. This document explains the basic operation of channels in the K2 Classic and K2 Summit and how to systematically reset the channel to restore normal operation. Following this systematic process a channel may become operational again without needing to reboot the entire system. In most cases the problem has a real cause that needs to be investigated and understood. If it comes back regularly be sure to report it to your customer service representative who will help determine if there are other problems that need to be resolved. Background Correct operation of a channel depends on different layers of software, hardware and FPGAs to work together. External interactions are another layer that affects channel operation. These include media storage access, networking, automation control, external applications, Dyno, STRATUS, Edius, Aurora, and others. Problems in operation may involve one or more of these layers. The procedures below address resetting different layers of the channel through normal operations. Problems contained within a layer can often be resolved by resetting one or more layers instead of resorting to rebooting the system. I/O Hardware The hardware that handles input and output of video/audio/anc/tc is set to receive and send according to specified video standards. This includes up- and down-conversion between video standards of the internal and external video. Changing input and output standards will reset this layer. Page 1 of 5

2 Encoders/Decoders/CODECs In the K2 Classic each CODEC board had discrete processors for encoding and decoding video in different compression formats (MPEG, DVCPRO). K2 Summits had discrete processors as well as FPGA-based CODECs that load the desired encoder/decoder images into the FPGA. Both K2 Classic and K2 Summits have code that runs on the real-time system (RTS) that manage the operation of the encoders and decoders. For each of these switching the video compression format of playing clips will reset decoders and switching the video compression format configuration for recording and then cueing for record will reset encoders. Software The main construction, configuration and management of channels come from Windows software. This software interacts with the real-time system (RTS) to control channel operation. Some parts of this code resets with configuration changes. This layer includes Windows drivers, services, dlls and applications. If reconfiguration does not resolve the problems at this layer it may require a reboot to recover. External The biggest factors for external interactions are with access to the media storage (especially so for SAN clients), and the amount of CPU/bandwidth used by automation control, controllers and external software. These are harder to isolate the cause as the symptoms can vary. Problems of this sort may be identified by lowering the operational channel count, number of client connections, amount of data transferred, etc. and see if the problem persists or changes. It is also useful to see if the same behavior seen with automation or external controls can be reproduced using local AppCenter operations. These types of problems usually cause resource starvation for the channel so channel resets will not resolve them. Procedures Follow these steps in order to address the main causes of channel problems with the least invasive sideeffects. These are the steps used in engineering when determining the cause of a problem. Recheck Logs and Files When the K2 has a problem it usually leaves hints to the cause in the log. The details of deciphering the log are not given here, but are provided in a separate document. For any channel operation problem check the logs and clips before assuming it is a problem with the channel itself. 1. Verify there is no file system problem. Slow access, dropped network connections, lost database connections, file system full, etc. will cause a channel to stop operating. 2. Verify all services are still working. AppService, MetadataSvr, and other services must still be operational. If these are not then problems will probably happen with all channels on the system. RTS failures will often show up as a lost connection. 3. Verify there is valid video/audio/tc. Lost video, video sync not locked, no audio detected and such errors will cause record problems that will later lead to play problems. Page 2 of 5

3 4. Verify the clips being played are good. If a clip is playing incorrectly be sure it is actually correct. Double-check play on other channels or verify it was created correctly and not from bad editing, bad import or during a corrupted record process. Restart Operation The simplest reset is to stop the operation and start it again. 1. Stop the channel (play or record) 2. Eject any clips/playlist 3. Try again When problems go away this may indicate a file system access problem or an internal picture or buffer management problem. These should be reported and investigated further. Reconfigure Reconfiguring a channel will force the software and hardware to reload and reinitialize. This is the essence of restarting without having to reboot. Players Players dynamically change configuration when clips of different video compression formats are encountered. Some play problems may be tied to the video output hardware so changing video output setting will reset this hardware. 1. Eject any clips/playlist 2. Load a clip of a different video compression format (such as MPEG, DVCPRO, AVC-I/AVC, DNxHD) 3. Eject the clip 4. Reload the original clip/playlist 5. If the problem persists eject the clip/playlist 6. Open the configuration manager (from AppCenter select System > Configuration from the 7. Go to the channel and change the video output standard 8. Close the configuration manager to accept the change 9. Open the configuration manager again 10. Change the channel video output standard back to what it was 11. Close the configuration manger to accept the change 12. Try again Recorders Recorders reset completely when they cue for record with a different video compression format (change between MPEG, DVCPRO, AVC-I, DNxHD). Page 3 of 5

4 2. Open the configuration manager (from AppCenter select System > Configuration from the 3. Go to the channel and change the video compression format 4. Close the configuration manager to accept the change 5. Record a small clip or cue for record 6. Stop recording and eject the clip 7. Open the configuration manager 8. Go to the channel and change the video compression format back to what it was 9. Close the configuration manager to accept the change 10. Try again CODECs CODECs restart completely just as Players and Recorders do, and when switching between play and record. If having troubles as a Player, follow the Players instructions. If having troubles as a Recorder, follow the Recorders instructions. If neither set of instructions work, follow these additional instructions or just do these instead of the Player/Recorder instructions. 2. If the channel was originally playing a clip, start a record. If the channel was originally recording, load a clip to play. 3. Eject any loaded clip 4. Try again Channel Type Another method of forcing a reset is to completely change the channel type. This is an option on Summits that support ChannelFlex, Video+Key/3D and SSM. Any of the above instructions will have worked for the appropriate channel types as well. Sometimes changing channel type is easier than loading clips or launching records. Pick the method that works best for you. 2. Open the configuration manager (from AppCenter select System > Configuration from the 3. Go to the channel and change the channel type (Player, Recorder, Player/Recorder, Multi-cam Recorder, V+K/3D Player, V+K/3D Recorder, SSM 2x Recorder, SSM 3x Recorder) 4. Close the configuration manager to accept the change 5. Open the configuration manager again 6. Go to the channel and change the channel type back to the way it was 7. Close the configuration manager to accept the change 8. Try again Reboot This is the last resort if all of the previous attempts to reset a channel fail. Rebooting through a shutdown or restart will force most of the channel operations back to their original initialization state. Page 4 of 5

5 In some rare cases the hardware PCIe bus may be hung, in which case a full power down may be needed. Turn the power off if a restart does not clear the problem, or if the system complains that it cannot find or talk to the real-time hardware. Replace This is the ultimate fix. If a power cycle does not clear the problem then it may be a true hardware problem. Here are a few steps to try and isolate what hardware may contribute to the problem. 1. Make sure all boards and connectors are seated and connected properly. Retry. 2. Move the CODEC board. If the problem follows the board to another slot or another machine then the CODEC board is the suspect. Replace it. 3. If the moved CODEC board works fine in another slot or machine then the problem is probably not with the CODEC board. Keep checking. 4. If a known-good CODEC board or new replacement has the same issue then there may be an underlying problem with the connectors, motherboard or something else. Time for a service call to debug the hardware. Page 5 of 5