DAB/MOT Data Carousel Support Library Linux Server Implementation

Transcription

1 DAB/MOT Data Carousel Support Library Linux Server Implementation D. Knox /1.0 28th Apr 1990 ENSIGMA Ltd Turing House Station Road Chepstow GWENT NP6 5PB

2 Ensigma Ltd. Page 2 of 31 Distribution Client O. Gardiner Ensigma Ltd Project File

3 Ensigma Ltd. Page 3 of 31 Contents 1. Introduction Change Control Release Summary Changes Intended Audience Summary Using the library Header files Linking Server Source Files and Modules OS Support Files Carousel Data Structures and Files Carousel Files Error recovery Carousel Data Structures Inter-Task communication Locking the carousel Synchronisation with the spooler Implementation Dependent Features Carousel path arguments MCSRVStartSpooler command argument Transport Interface ServerOpenDAB ServerCloseDAB ServerSendDAB... 21

4 Ensigma Ltd. Page 4 of Test and utility programs Utility Programs spstart spstats spstop vfsbuild vfsupdate Test programs srvtest srvdump srvstub Building the software File Organisation Server Customisation Compiling the libraries and utilities Compiling the test programs References... 31

5 Ensigma Ltd. Page 5 of 31 List of Tables Table 1 Server Source Files and Modules Table 2 OS Support Files Table 3 Carousel Files...14 Table 4 Files Produced by Build Process... 29

6 Ensigma Ltd. Page 6 of 31 List of Figures Figure 1 Memory Resident Carousel Description Figure 2 Directory Organisation... 27

7 Ensigma Ltd. Page 7 of Introduction The data carousel software library is defined in [1]. This document adds a little additional technical information regarding the implementation of the server library. It also details those features which are specific to the Linux implementation of the server library. 1.1 Change Control Release Summary Version Description 1.0 Initial Version Changes None. 1.2 Intended Audience This is a technical document intended for software designers who will be installing, operating and using the data carousel library in the Linux environment. We assume familiarity with Software development in the Linux environment. DAB and associated data protocols, especially packet data services and MOT. The library interface definition in [1]. 1.3 Summary Section 2 explains how to use the library to develop server application programs. Section 3 explains how the server source code is arranged into files and modules. Section 4 provides background information on the internal data and file organisation used by the server library. This information is most likely to be useful to those modifying or developing the library, but is not required for normal use.

8 Ensigma Ltd. Page 8 of 31 Section 5 explains the internal mechanisms used to ensure that the carousel spooler task and programs using the carousel library synchronise their activities properly. Section 6 discusses those aspects of the library users interface which are left undefined in [1] and which are specific to the Linux environment. This includes Section 6.3 which defines the transport functions used to deliver DAB MSC data groups to the DAB packet encoder and broadcast system. Section 7 describes a number of utility and test programs which are provided with the software distribution. Section 8 explains how to compile the software from its sources.

9 Ensigma Ltd. Page 9 of Using the library To use the library for application program development, you need to 1. Include the library header files in your source code 2. Link the library object files with the rest of your application. 2.1 Header files Header files are provided for the C programming language. Users of other languages will have to perform appropriate translations. There are three header files mcserver.h mctypes.h mcsrverrs.h The main header file Data type definitions Error code definitions The last two are included automatically by the first file so all that is required is to include the main file #include <mcserver.h> If the header files have not been installed in the standard include search path, use the -I flag of the C compiler to define the directory containing them. 2.2 Linking The server library is contained in a file called libdabmcs.a. This should be linked with your application. You can either give the full file name on the command line or, if the library is installed the linker s library search path, use -l dabmcs.

10 Ensigma Ltd. Page 10 of Server Source Files and Modules The server source code is organised into logical modules, each containing a set of procedures and data declarations related to a defined function. Each of these modules is packaged in a pair of files a program file (extension.c, containing executable code and private data) and a header file (extension.h, defining the external interface to the module). Those parts (if any) of a module interface which form part of the public library interface are declared in mcserver.h rather than in the module s header file. The server source files are listed in Table OS Support Files To date, the carousel has been implemented on both the Linux and Microsoft Windows NT operating systems. Other operating systems may be supported in future. In general, only minor changes are required for the different operating systems, and these are handled by conditional compilation. However, some operations (mainly concerned with inter-process communication mechanisms) are dramatically different in different operating systems. These major variations are packaged in a set of support files (kept in the common source subdirectory: see Figure 2) which contain alternate conditionally compiled code variants for the different supported operating systems. The OS support files are listed in Table 2.

11 Ensigma Ltd. Page 11 of 31 Server Files and Modules File(s) crc16.c,.h mcserver.h mctypes.h scorebrd.c,.h spooler.c,.h srverrh.c,.h srverrs.h srvetab.h srvrcar.c,.h srvrenc.c,.h srvrfile.c,.h srvrlock.c,.h srvrmem.c,.h srvrmod.c,.h srvrtrns.c,.h types.h Description Low level CRC16 calculations Public server library interface definition Standard data types required for public library interface Scoreboard (bitmap) manipulations. Spooler execution and control. Error handler. Error code table generated automatically from srverrs.txt Error table generated automatically from srverrs.txt Carousel manipulations Server encoding functions. Server file manipulation functions. Carousel locking functions. Dynamic memory allocation. Module manipulations. DSM-CC message transport Common data types (not part of public interface) Table 1 Server Source Files and Modules

12 Ensigma Ltd. Page 12 of 31 OS Support Files File(s) args.c,.h ftok.c,.h io.c,.h port_msg.c,.h sem.c,.h shm.c,.h Description Converts Windows command line into argc, argv for UNIX style argument processing Converts file name to identifier key string. Used in Windows in place of UNIX ftok function File existence and type tests. Used in Windows in place of UNIX stat function. Inter-process message queues. Implemented with SYSV IPC functions in Linux and custom-written equivalents in Windows. Semaphores. Implemented with SYSV IPC functions in Linux and emulated with WIN32 functions for Win/NT. Shared memory segments. Implemented with SYSV IPC functions in Linux and emulated with WIN32 functions for Win/NT. Table 2 OS Support Files

13 Ensigma Ltd. Page 13 of Carousel Data Structures and Files An open carousel is represented by a combination of memory resident data structures and disk files in the carousel directory. The memory resident structures contain a complete model of the carousel s composition and organisation (modules, groups and descriptors). Module data is not memory resident but is stored on disk, one file per module. Even empty modules have an associated disk file (zero size). When a program closes a carousel, a representation of the memory resident data structure is written out to additional disk files. These are re-loaded and the data structures are rebuilt when a program opens the carousel. In addition to the files describing the current content, organisation and modules of the carousel, the state of the carousel when it was last frozen (using MCSRVUpdateCarousel) is stored on disk as a single file. This data file contains a complete coded carousel. It contains the stream of DAB MSC data groups defining one turn of the carousel. Each data group is preceded two 4-byte signed integers. The first integer is the account code relating to the data group message. The second is the length in bytes of the following message. The spooler plays out this data file, maintaining statistics and inserting DAB continuity index counters and CRCs (if enabled) as it goes. A second data file, with the same format, contains a partial carousel representing just data which has changed since the previous version. This transition file is played out once by the spooler following a change to the carousel. 4.1 Carousel Files The files in the carousel directory are listed in Table 3. The backup directory (bck) is used for temporary backups of files during certain carousel update operations.

14 Ensigma Ltd. Page 14 of 31 Carousel Files File xxxxxxxx.cdf xxxxxxxx.ctf carousel.lock carousel.par carousel.mod modxxxxx.dat bck Content Coded carousel data file. xxxxxxxx is carousel version number. Coded transition data file. xxxxxxxx is carousel version number. Carousel lock file. Carousel description file. Module description file. Module data files (one per module, xxxxx is module id). Temporary backup directory Table 3 Carousel Files. 4.2 Error recovery Only one program may access a carousel on the server side at any time. The locking mechanism is built into the MCSRVOpenCarousel and MCSRVCloseCarousel functions. If a program exits without closing a carousel, other programs will not be able to access the carousel. Normally this should not happen, but during development or following a system error a carousel may be left locked. If this happens, the lock can be released by deleting the carousel.lock file in the carousel directory. Normally, the coded data files xxxxxxxx.cdf and xxxxxxxx.ctf are deleted automatically as they are superseded by newer carousel versions. If, for some reason, this does not happen (for example if there is a system failure during an update) lower numbered files can be deleted manually. In such cases it may be wise to regenerate the carousel to ensure consistency. 4.3 Carousel Data Structures The carousel is represented in memory by a data structure built from components which are allocated as required. This structure is illustrated in Figure 1. The lists are implemented as doubly linked circular queues with a dummy item in the parent structure forming the queue header. This arrangement allows items to be added and removed easily. For simplicity, Figure 1 does not show the double links or the circular nature of the lists.

15 Ensigma Ltd. Page 15 of 31 Carousel and module handles are, in fact, pointers to elements in this structure, although library users should not attempt to exploit this 1. Carousel Module Module Module Module Descriptor Descriptor Descriptor Figure 1 Memory Resident Carousel Description 1 We enforce this by hiding the structure definitions in the library implementation and only publishing anonymous handles (as C void * pointers).

16 Ensigma Ltd. Page 16 of Inter-Task communication Two inter-task communication and synchronisation mechanisms are used. The first provides a means of locking the carousel so that only one task can access it at any time. The second mechanism is used to protect critical regions so that the spooler and the updates to the carousel generated by library functions do not interfere with each other. 5.1 Locking the carousel A conventional UNIX style file lock is used to protect the carousel as a whole. The Linux man page for the open() system call notes a potential race condition where the lock file is mounted on an NFS file system and describes a technique to overcome it. This is what we implement. 5.2 Synchronisation with the spooler The library functions and the spooler communicate through a shared memory segment. Updates to the shared memory are protected by a semaphore. Possible race conditions surrounding the creation, initialisation and deletion of the semaphore are avoided because the spooler is started by a library call requiring that the carousel library first be opened. This ensures that spooler start-up/shutdown is protected by the global carousel locking mechanism.

17 Ensigma Ltd. Page 17 of Implementation Dependent Features 6.1 Carousel path arguments The path arguments of MCSRVCreateCarousel, MCSRVOpenCarousel and MCSRVDeleteCarousel are just UNIX directory names. The carousel is created and managed as a set of files inside the specified directory. Normally, these should not be modified or manipulated except by way of programs using the carousel server library. It is permissible, however, to delete an entire carousel simply by deleting the corresponding file tree, provided that no program is accessing the carousel (be especially careful to check for an active spooler in this case). 6.2 MCSRVStartSpooler command argument The command argument to MCSRVStartSpooler is a pointer to a character string. This argument may be NULL, in which case spooler output is directed to stdout. If it is not NULL then it must point to a null-terminated character string with two fields separated by a single colon. The first field is interpreted as a host name and the second as an IP port number. If the command argument is provided in this form, then spooler output is directed over a TCP/IP connection to the appropriate host:port target using the BBC s handshake protocol. This protocol is explained in the description of the ServerOpenDAB transport function below. The port number may be specified as decimal, octal (leading 0 ) or hexadecimal (leading 0x ) according to normal C programming convention. An example command string is dab.bbc.co.uk: Transport Interface The transport interface is defined by the following functions ServerOpenDAB() ServerSendDAB() ServerCloseDAB() These are described on the following pages.

18 Ensigma Ltd. Page 18 of 31 ServerOpenDAB Definition: DABHANDLE ServerOpenDAB(ctype, arg1, arg2...) Arguments: INT ctype Connection type VOID * arg1 Connection argument 1... VOID * argn Connection argument n Returns: DABHANDLE Handle for the connection. NULL if failure Description: This function opens a connection between the spooler and the DAB data group transport system. The handle argument returned from this call is used to direct data groups messages to this connection. The transport mechanism is specified by ctype, with the number and interpretation of additional parameters arg1 argn being dependent on the type. Currently two values of ctype are supported ctype=1. Output is directed to stdout. Each data group is preceded by an INT byte count. This facility is mainly for debugging, but may have other uses. No additional arguments are used. ctype=2. Output is directed to a TCP/IP port with the BBC s handshake protocol. In this case arg1 is a pointer to a string containing the host name and arg2 is a pointer to an integer containing the destination port number. The BBC s handshake protocol on the server side is summarised as follows. It is based on command messages with the general format

19 Ensigma Ltd. Page 19 of 31 0xF2 start byte 0xnn length MSB 0xnn length LSB 0xcc command code 0xdd data x2F stop byte The DAB packet coder requests each message from the spooler with a message having no data and command code set to 0x11 (length 1). The spooler waits for the request and then responds with a message having command code 0x12 and the data group as its data field (length is length of data group +1).

20 Ensigma Ltd. Page 20 of 31 ServerCloseDAB Definition: INT ServerCloseDAB(channel) Arguments: DABHANDLE channel Connection to be closed Returns: INT 1:success, 0: failure Description: This function closes the connection between the spooler and the message transport system identified by channel.

21 Ensigma Ltd. Page 21 of 31 ServerSendDAB Definition: INT ServerSendDAB(channel, message, INT mlen) Arguments: DABHANDLE channel Connection to be used BYTE * message pointer to message to be sent INT mlen message length (bytes) Returns: INT length of message sent or 0 if failure Description: This function sends the data group identified by the message argument to the specified channel. The number of bytes in the message is specified by the length argument.

22 Ensigma Ltd. Page 22 of Test and utility programs Several test and utility programs are provided with the carousel distribution. 7.1 Utility Programs The utility programs perform useful functions associated with the operation of a data carousel server. They do not form a fully comprehensive toolkit, but they do provide some useful functions and can be used as models for other programs spstart The spstart program starts the spooler associated with a carousel. It is a simple wrapper for the MCSRVStartSpooler function. The command line syntax is spstart [-h hostname [ -p portnumber]] [-v] path The path argument specifies the directory containing the carousel. If hostname is omitted, the spooler sends data group messages, each preceded by a 16 bit byte count, to stdout. Otherwise, the spooler uses the BBC s handshake protocol to send data group messages over a TCP/IP link to the specified host and port. If portnumber is omitted, the default port number is used. The -v flag enables verbose operation in which additional progress messages are reported spstats The spstats program reports the accounting statistics for the spooler associated with a carousel. It is a simple wrapper for the MCSRVSpoolerStatistics function. The command line syntax is spstats [-r] [-v] path The path argument specifies the directory containing the carousel. The -r flag causes the statistics to be reset to zero after they are reported. The -v flag enables verbose operation in which additional progress messages are reported spstop The spstop program stops the spooler associated with a carousel. It is a simple wrapper for the MCSRVStopSpooler function. The command line syntax is

23 Ensigma Ltd. Page 23 of 31 spstop [-v] path The path argument specifies the directory containing the carousel. The -v flag enables verbose operation in which additional progress messages are reported vfsbuild The vfsbuild program walks a directory tree to build a virtual file system in a data carousel. The command line syntax is vfsbuild [-s sourcedir] [-c carouseldir] [-v] A carousel is created in carouseldir. Then the program walks the directory tree rooted at sourcedir. For each file in the source tree a module is created in the carousel. Each carousel module is given a name descriptor consisting of the file name relative to the source directory root. In addition, files are given a type descriptor in the format of a MIME type definition based on the filename extension. Both sourcedir and carouseldir default to the current working directory. However, they must not be the same, so at least one must always be specified. The carousel directory must not be contained within the source directory tree, otherwise behaviour will be unpredictable. The -v flag enables verbose operation in which additional progress messages are reported. Note that vfsbuild creates a carousel which, it is assumed, will be transmitted with the library default data rate. This is probably not a useful assumption and the carousel s data rate should normally be changed before starting the spooler.

24 Ensigma Ltd. Page 24 of vfsupdate The vfsupdate program walks a directory tree to update a virtual file system in a data carousel. The command line syntax is vfsupdate [-s sourcedir] [-c carouseldir] [-v] This program is similar to vfsbuild, but with the following differences The carousel must already exist. It will not be created. Where a module with the same name as a file already exists in the carousel, it is updated (content replaced, version number incremented) rather than creating a new module. Modules in the carousel which do not have a corresponding file in the source directory tree are removed.

25 Ensigma Ltd. Page 25 of Test programs srvtest The srvtest program exercises normal and error paths through most of the carousel server library functions. The source code contains examples of how to do most things that the system is capable of. Note that error testing is not exhaustive. In particular, testing of disk-full and memory allocation failures is not performed srvdump The srvdump program decodes a stream of data group messages on stdin and provides a formatted display on stdout. This can be used to decode a stream of messages from a spooler. This program is of no value operationally, but is valuable to program developers for testing carousel coding. For example it has been used to test the MCSRVSpoolerUpdate function to confirm that the change to the new carousel occurs at the proper moment. The syntax is srvdump [arguments] If any optional arguments are given 2, srvdump assumes that the messages are being read from a coded carousel data file and are preceded by account code and length fields srvstub The srvstub program allows operation of the spooler with the BBC s handshake protocol over TCP/IP connections to be tested. It accepts a connection on IP port and then receives messages from a spooler. Data group messages are received and written to stdout, each preceded by a 16 bit byte count. Note that srvstub is not a full blown server. It accepts only one connection and when it is broken, the program terminates. srvstub takes no command line arguments. A common application is to pipe stdout to srvdump. For example, in a terminal window type srvstub srvdump In a second window start a spooler 2 The arguments have no particular syntax and they are not interpreted in any way. Their mere presence is enough to cause the modified behaviour

26 Ensigma Ltd. Page 26 of 31 spstart -h localhost path The first window will display a formatted dump of the spooler output.

27 Ensigma Ltd. Page 27 of Building the software The carousel software is packaged in a combined distribution containing both the server and the client software. Normally, you would build both the server and the client libraries at the same time, although you may choose to install only the server components if you wish. 8.1 File Organisation base bin lib include doc src test server client common utilities Figure 2 Directory Organisation Unpacking the distribution into an empty base directory will create the structure shown in Figure 2. The src directory contains the library sources, organised into four sub-directories, one each for the server, client and utilities and a common sub-directory containing OS dependent support functions used in both the libraries and some of the utilities. The doc directory contains system documentation. These documents are in PDF format. You need the Adobe Acrobat Reader program to read or print them 3. The test directory contains the sources of the test programs. Initially the bin, lib and include directories are empty. When the software is built, the files needed to use the library are placed there. 3 Adobe s Acrobat Reader program is available from their web site versions are available for Microsoft windows, Linux and other UNIX variants.

28 Ensigma Ltd. Page 28 of 31 The base directory, the test directory and each of the server, client and utilities directories contain Makefiles to assist with building the software in varying circumstances. Normally you need only be concerned with the top-level Makefile in the base directory. It invokes the others as required. 8.2 Server Customisation There are no compile-time customisation options for the MOT carousel server software Compiling the libraries and utilities To build the complete system, simply make the base directory your current working directory and then type make all This will place the files needed to use the library in the bin, lib and include directories. After building the software, you can recover some disk space by deleting intermediate files which were used during the build and which are no longer required. To do this simply type make clean If you want to delete all the generated files and start again from the source distribution, type make clobber The build process generates files listed in Table 4. You should move these to appropriate directories so that they are accessible to your software developers. One common convention is to place them in /usr/local/bin, /usr/local/lib and /usr/local/include. 4 There are several carousel configuration options which can be controlled at run-time using the functions documented in [1].

29 Ensigma Ltd. Page 29 of 31 Files Generated by Build Process File lib/libdabmcs.a lib/libdabmcc.a include/mcserver.h include/mcclient.h include/srverrs.h include/clierrs.h include/mctypes.h bin/ccache bin/spstart bin/spstop bin/vfsbuild bin/vfsupdate Description Server library. Client library. Server library header file. Client library header file. Server library error codes Client library error codes Library interface data types Carousel client cache manager program Server spooler start program Server spooler stop program Server virtual file system build utility program Server virtual file system update utility program bin/vfscopy Client virtual file system copy program Table 4 Files Produced by Build Process

30 Ensigma Ltd. Page 30 of Compiling the test programs To build the test programs, change to the test sub-directory and type make all You can clean up with make clean or make clobber as before. The Makefile in this directory will (re)build the libraries if necessary. However, beware that the Makefile in the test directory uses different compiler flags from the top level Makefile. After working with the test programs it is advisable to return to the base directory and rebuild the libraries using make clobber; make all to return everything to normal.

31 Ensigma Ltd. Page 31 of References [1] Ensigma Ltd. DAB/MOT Data Carousel Support Library Interface Definition