Multiwavelength Optical Networking Management and Control

Transcription

1 2038 JOURNAL OF LIGHTWAVE TECHNOLOGY, VOL. 18, NO. 12, DECEMBER 2000 Multiwavelength Optical Networking Management and Control Brian J. Wilson, Ned G. Stoffel, Jorge L. Pastor, Mike J. Post, Kevin H. Liu, Tsanchi Li, Kenneth A. Walsh, John Y. Wei, Member, IEEE, and Yukun Tsai Abstract This paper describes the network management research done by the Network Control and Management (NC&M) Task under the Multi-wavelength Optical Networking (MONET) Program. MONET is sponsored by the Defense Advanced Research Project Agency (DARPA) of the U.S. Government Department of Defense, with participation from Telcordia Technologies, AT&T, Lucent Technologies, several government agencies and regional Bell Operating Companies. MONET s vision is to develop technologies needed for a flexible, reliable, high-capacity, high-performance, cost-effective national scale optical network based on the multi-wavelength fiber-optic technology. As an important component in realizing this vision, the MONET program includes the architecture and design of a prototype network control and management system that will manage MONET s reconfigurable wavelength-division-multiplexing (WDM) all-optical network. The primary objectives of the prototype research work are to develop the architecture and framework for managing national-scale transparent reconfigurable WDM optical networks, and to demonstrate the feasibility of the NC&M prototype system in a field experiment network in Washington, DC. This prototype system will allow the program participants to conduct experiments and gain experience in the management and operations of reconfigurable optical networks. This paper describes the features and capabilities of the prototype system, addressing issues such as management architecture, information model, interoperability and algorithms of the prototype management system. We also relate some of our experience in testing, installing and using the prototype system in the MONET network. Index Terms Network management, information model, Telecommunications Management Network (TMN), wavelength-division-multiplexing (WDM), all-optical network, CORBA. I. INTRODUCTION THE emergence of broadband communications has increased the need for bulk transport of high-capacity signals and services. Multi-wavelength reconfigurable optical networks offer such a capability beyond the current transport technology, such as SONET. The Multi-wavelength Optical Networking Program (MONET) [1] sponsored by the U.S. Government s Defense Advanced Research Project Agency (DARPA,) is a consortium consisting of several government agencies (DARPA, NSA, NRL, NASA, DIA, DISA,) Telcordia Technologies, several Regional Bell Operating Companies Manuscript received June 20, 2000; revised October 6, This work was supported in part by the U.S. Government DARPA Multi-wavelength Optical Networking (MONET) Project, Contract MDA The authors are with Telcordia Technologies, Inc., Red Bank, NJ USA ( brian2@research.telcordia.com). Publisher Item Identifier S (00) (Bell Atlantic, BellSouth, Pacific Telesis, and SBC are directly involved, other regional companies are indirectly involved), Lucent Technologies, and AT&T. The aim is to address the technology, architecture, and the control and management issues for this new emerging technology. It also aims at demonstrating the viability of using transparent reconfigurable wavelength-division-multiplexing (WDM) optical network as the next generation high-performance fabric for DoD applications. One of the most important components in the MONET program is the network control and management (NC&M) research. Its objectives are: Identify the unique network control and management issues for reconfigurable WDM networks. Develop an architecture framework and the management algorithms needed for managing national-scale transparent reconfigurable WDM optical networks. Design and prototype a new generation NC&M system that incorporates these ideas to provide efficient and costeffective WDM operations. Demonstrate the feasibility of the prototype NC&M system in the Washington, DC field experiment network. As such, under the MONET Program, Telcordia developed a research prototype network control and management system aimed at addressing the WDM network management needs. An earlier version of this prototype system and its functionality was publicly demonstrated in February 1997 during the Optical Fiber Conference (OFC 97) in Dallas, TX. The correct operation and demonstration of this prototype system was an integral part in establishing the MONET project milestone reported in [2] and [3]. Preliminary results on the MONET NC&M work were previously reported in [4] [6] and [7]. For the rest of this paper, we will summarize the design of this prototype system, covering such aspects as system architecture, information model, management features and algorithms, as well as the integration and interoperability of the prototype network management system with the testbed hardware. The prototype network management system provides a rich set of configuration, connection, performance, and fault management capabilities for a reconfigurable WDM all-optical network. The salient features of this design are: Adoption of the multi-tiered Telecommunications Management Network (TMN) [8] architectural framework in the NC&M system design with well-defined interfaces and subnetwork hierarchy. Management activities and responsibilities are organized according to the TMN management functional area decomposition and management /00$ IEEE

2 WILSON et al.: MONET MANAGEMENT AND CONTROL 2039 layer separation. The result is a system that is highly modular, and one that promotes interoperability. The TMN architectural concepts and subnetwork hierarchies also allow the NC&M system to manage networks that scale up to national size. Adoption of the network-level information models specified in the ITU-T SG-4 draft recommendation [9], and from TINA-C [10]. The management information base (MIB) contains information that represents the managed network resources and their relationships. It effectively represents and captures the state of the network, and is shared by the different management applications. Adoption of the Common Object Request Broker Architecture (CORBA) framework in our NC&M specification and implementation. MONET s NC&M system is a distributed system where all the management information and computational objects are implemented as CORBA objects. Unlike existing WDM optical installations that are pre-provisioned point-to-point transmission systems, MONET s NC&M system controls a test-bed that is an optical network providing optically switched connections. The prototype management system allows a network administrator to dynamically discover information about the network, provision connections across the network, and to monitor and react to network failures and connection problems. For the most part, the work described in this paper is best described as a distributed implementation of a centralized management capability. Since the MONET NC&M prototype was designed the field of optical networking has expanded rapidly. In particular, recent efforts in the Internet Engineering Task Force (IETF) and Optical Internetworking Forum (OIF) have explored ways of distributing the management of WDM connections through the use of IP type routing protocols such as Multi-Protocol Label Switching (MPLS). These efforts are distinguished from the MONET work reported here in several other ways: MONET focused on managing a single WDM layer not the interoperability between IP and WDM layers. One aspect of MONET NC&M, Just-In-Time Signaling, does support clientbased connection management. This differs from the IETF and OIF work in terms of the signaling protocol used and MONET did not explore different routing protocols. The remainder of this paper is organized as follows: In Section II, we give an overview of the architecture of the MONET NC&M prototype system. Section III describes the network level management information model used to represent the structure and state of the MONET WDM optical network. Our presentation focuses primarily on information relevant to resource configuration and connection management. In Section IV, we describe the various network management components of our prototype system, highlighting the management features and functionality provided and the interactions among the different components. In Section V, we describe the low level control aspects of the prototype system. In Section VI, we examine the interoperability between the NC&M agent and the network element controllers focusing on the NC&M to NE interface. In Section VII, we relate some of our experience in designing, testing, installing and using the prototype system in the MONET network. Finally, in Section VIII, we conclude this paper with a summary and comment on some new research activities pursued under a MONET follow on work. II. NC&M OVERVIEW In this section, we describe the MONET network management system architecture. MONET has an overall project vision of developing the technologies needed to realize a flexible, reliable, high-capacity, high-performance, cost-effective national scale optical network based on multi-wavelength technology [1]. Toward realizing this vision, the design and implementation of a robust distributed network control and management system is an integral part of the MONET program. Design of the MONET NC&M prototype system is based on the TMN architecture. In TMN, network management functions are grouped into five management functional areas: Configuration Management Fault Management (FM) Performance Management Security Management Accounting Management Being a research effort, the primary focus of MONET s NC&M prototype system is on Configuration (Resource Configuration and Connection Management,) Fault, and Performance Management. According to the TMN architecture, in addition to these functional groupings, network management system functions are also separated into different management layers: Network Management Layer (NML) Element Management Layer (EML) Network Element Layer (NEL) In conformance to this separation, MONET s NC&M system components are all divided into NML and EML entities according to their corresponding management responsibilities. To facilitate scalability, the MONET test-bed network is organized topologically into a hierarchy of subnetworks. These subnetworks may be partitioned according to differing criteria such as geographic proximity, equipment vendor identities, or management boundaries. Fig. 1 depicts a moderate size network that is aggregated into three subnetworks. This network example will be used throughout this paper for illustration purposes. Fig. 2 illustrates the overall management system architecture that corresponds to the example network shown in Fig. 1. The management modules are designed and developed in a manner that reflects a direct correspondence to the TMN architecture. EML managers are deployed to manage the different subnetworks, such as the WDM rings and meshes in our example network, while the NML managers manage the crown WDM network a mesh of subnetworks. MONET s NC&M prototype system is fully modular and distributed. The modules in each management layer and functional area interact with each other via well-defined management interfaces. These interfaces are exported by the set of management information and computational objects defined in the module. These management information interfaces are all specified in

3 2040 JOURNAL OF LIGHTWAVE TECHNOLOGY, VOL. 18, NO. 12, DECEMBER 2000 Fig. 1. Example of WDM network. information objects export their own interfaces and data abstractions that provide the common database over which all applications interact with each other. This sharing eliminates unnecessary redundancies and enhances data consistencies. Fig. 2. MONET NC&M architecture. Common Object Request Broker Architecture (CORBA) Interface Definition Language (IDL). The management objects and applications are implemented on top of a CORBA compliant platform, and the management objects are instantiated in accordance with the deployment of the network resources and the connections they support. CORBA is the computing industry s emerging standard for distributed object computing. The open and extensible nature of the CORBA framework has attracted significant interest in telecommunications industry [11]. To summarize, this system architecture has the following characteristics: Network and System Scalability. The partitioning of a network into multiple subnetworks each with its own element management system allows the network to scale. Further it allows the use of vendor specific implementations at the subnetwork level, while maintaining standard information model and interfaces at the network level. This helps to promote management system interoperability. Common Management Information Base. The various management applications all share the same set of management information objects. These management III. INFORMATION MODEL Central to the MONET network management system is the network management information base (MIB) that represents the managed network resources and their relationships. This information base effectively represents and captures the state of the managed network, and it is shared by all network management applications. In this section, we describe the network-level information model, focusing on the information objects and the relationships that are defined to support resource configuration and connection management. A. Management Information Objects The network-level management information base used in MONET s NC&M system is developed by adopting the network-level information models developed by ITU-T SG-4 and TINA-C. MONET s WDM network-level information model defines the following objects to represent network-level resources and concepts: Layer A layer network is a transport network that Network carries a particular characteristic information. For MONET, we have a WDM layer Subnetwork Link network. A subnetwork recursively contains other subnetworks and links that are grouped together for topological or other reasons. The smallest subnetwork is a single network element. A link represents connectivity between two subnetworks. For MONET, a link represents an optical fiber transport between network elements.

4 WILSON et al.: MONET MANAGEMENT AND CONTROL 2041 Fig. 3. Entity-relationship diagram for information model. Link Termination Point (LinkTP) Link Connection (LC) Connection Termination Point (CTP) Subnetwork Connection (SNC) Trail Termination Point (TTP) Trail A LinkTP represents an end (termination) point of a link. A link connection is the resource that transfers the characteristic information over the link between two subnetworks. For MONET, a link connection represents a wavelength channel within the corresponding link. A CTP represents an end point of a subnetwork connection or link connection. A subnetwork connection is the resource that transfers characteristic information between two CTPs at the edge of the subnetwork. For the degenerate case where a subnetwork represents a network element, the SNC denotes a cross-connection in the switching fabric. The TTPs delimit a layer network. These are the access points for client layer network elements. An end-to-end trail is the resource that transfers client information between two or more TTPs. Trails support the client layer connections. These information objects are related to each other under a set of well-defined relationships using the General Relationship Model (GRM). Fig. 3 depicts the information objects and their relationships in an Entity-Relationship diagram. Specifically, the E-R diagram shows the following relationships: Partition Contain Bind Support Adapt Delimit Layer networks may be partitioned into a set of links and subnetworks, and each subnetwork may recursively be partitioned into subnetworks and links. A SNC may be partitioned into SNCs and LCs. A layer network contains several trails. A link may contain LCs corresponding to the wavelengths transferred over the fiber. Each linktp contains several CTPs depending on the number of wavelengths supported in the terminating link (fiber). A subnetwork may contain SNCs. A link is bound to two linktps. An LC is bound to two CTPs. A SNC is bound to two CTPs, and a trail is bound to two or more TTPs. Trails in a server layer network support LCs in a client layer network. It describes the relationship existing between CTPs of a client layer network and the TTPs that support them in a server layer network. A layer network is delimited by a set of TTPs whose associations may be setup and torn-down by the layer network connection management process. Similarly, a subnetwork is delimited by a set of CTPs. MONET s network management applications search and navigates these relationships in order to locate the required information, and to process various types of management requests. B. A Network and Connection MIB Example To illustrate how these information objects are used to describe a network and its connections, we consider the network and connection example depicted in Fig. 4. The example network in this figure consists of three subnetworks. Two of the subnetworks are rings each consisting of three WDM Add-Drop Multiplexers (WADMs) and two Wavelength Selective Cross-

5 2042 JOURNAL OF LIGHTWAVE TECHNOLOGY, VOL. 18, NO. 12, DECEMBER 2000 Fig. 4. Sample network and connection example. Fig. 5. Network-level resource information base for sample network. Connects (WSXCs) 1. The WSXCs are each connected to their counterparts in the other subnetworks. Fig. 5 shows the resulting MIB formed from the bottom portion of this example network. The MIB shown consists of: A WDM Layer Network consists of a CROWN subnetwork that in turn consists of the two ring subnetworks (Subnetworks 1 and 2). They are interconnected by link B. Ring subnetwork 1 consists of five network elements of which only two (WADM1 and WSXC1) are shown. They are interconnected by link A. Ring subnetwork 2 also consists of five network elements of which only two (WADM2 and WSXC2) are shown. They are interconnected by link C. Each link consists of eight link connections of which only one is shown. At each of the network element, a subset of the linktps, CTPs, and TTPs are shown. For example, at WADM1: WADM 1 has three linktps denoting three connecting fibers. In Fig. 5, we show only two of them: one denoting the incoming access link, and the other denoting the outgoing link (A). The linktp of WADM 1 associated with link (A) contains eight CTPs representing termination points for the eight wavelengths supported on the fiber. In Fig. 5, we show only a single CTP that denotes an 1 WADM and WSXC are MONET-specific terminology. They are also referred to as Optical Add-Drop Multiplexer (OADM), and Optical Cross-Connect (OXC) in other literature. end point of a particular link connection (wavelength connection). These network resource objects are organized into NML and EML layer entities, and are maintained by their corresponding NML and EML resource configuration managers. When the end-to-end connection depicted in Fig. 4 is provisioned across the network, it is represented in the MIB by a trail object, and a corresponding set of supporting subnetwork connection objects (SNCs). Fig. 6 shows the corresponding MIB view with the new information objects created as a result of provisioning this end-to-end trail: A WDM layer end-to-end trail is supported by an NML-level subnetwork connection (SNC0) over the crown subnetwork. The NML-level SNC consists of two EML-level SNCs, one over the ring 1 subnetwork (SNC1), and the other over the ring 2 subnetwork (SNC2), and a link connection over link B. The EML-level SNCs are further decomposed into link connections and NEL-level SNCs (cross-connects). The management information base is shared by all management applications that can navigate and retrieve the information needed for their processing. For example, starting from a trail object, a manager application can trace out the route of an end-to-end trail, by enumerating all the SNCs and linkconnections used to support the trail. As another example, a network

6 WILSON et al.: MONET MANAGEMENT AND CONTROL 2043 Fig. 6. Network-level resource and connection information base for sample network. administrator running a GUI can start at a link object and navigate to discover which end-to-end trails are routed over the selected link. IV. MANAGEMENT SYSTEM AND FUNCTIONALITY The MONET network management system prototype, developed by Telcordia, is a next-generation network management system designed to provide network management for the emerging reconfigurable WDM networks. It is a distributed system design based on the ITU-T TMN logical layered management architecture and implemented over a CORBA-based (Common Object Request Broker Architecture) distributed object computing platform as shown in Fig. 2. The result is a distributed management system that is highly modular. The WDM NC&M system provides a rich set of configuration, connection, performance, and fault management functionality. It provides a user friendly graphical user interface (GUI) for network/site maps, faceplates, alarms, and performance monitoring information displays, and various dialog displays for initiating configuration and connection management activities. The system s design is based on the concept of the "network as database" to reduce the data inconsistency problems and to facilitate management automation. It optimizes and simplifies the operations of the future WDM optical networks and ensures their viability and deployability. The NC&M system is divided into management modules that provide the corresponding configuration, connection, performance and fault management functionality. We next describe these modules and their management functions in the following sections. This description is not meant to be exhaustive, rather it aims to illustrate and highlight the representative features. A. Software Framework The prototype MONET NMS is implemented on a CORBAbased distributed computing platform in order to hide distribution complexity and to improve system modularity. All management information and interfaces including the NE agents, EML management modules, and the NML management modules, are all specified and implemented using the Object Management Group s (OMG s) CORBA framework. The CORBA-based implementation allows the prototype system to cope with the inherent data and system component distribution, and to ensure ease of evolution, maintenance, and integration with the service management layer. The distributed and modular design of the prototype system allows a network administrator to easily adapt to network growth. Adding new network elements or configuring new subnetworks amounts to simply deploying and configuring an associated set of management modules for the added resources. The result is a management system configuration that closely mirrors the corresponding managed network configuration. B. Configuration Management The configuration manager provides capabilities to manage the network topology, such as network connectivity and network resources, and to provide a network map. Thus, the main functional requirements of the MONET Configuration Manager are: To construct a network-level information model that represents a global view of the MONET WDM transport network (the portion of the information model depicted in Fig. 5). To describe how individual network elements are interconnected and configured to provide end-to-end connectivity. To support other management functions such as connection, fault, and performance management The information models contain definitions of information objects and their relationships, which provide transmission and switching technology-independent information specifications of network resources. In particular, they describe transport networks abstractly in terms of the network element, aggregation of network elements, topological relationship between the elements, connection end-points, and transport connections. From the viewpoint of software development, the information model defines an entity-relationship data model that represents the logical structures of the MONET Network resources. These network resources, as programmed in CORBA objects, collectively constitute the distributed object database, which is organized according to the data model.

7 2044 JOURNAL OF LIGHTWAVE TECHNOLOGY, VOL. 18, NO. 12, DECEMBER 2000 Fig. 7. Example network map for MONET. The MONET Configuration Manager is a hierarchy of two different management layers. NML (Network Management Layer) and EML (Element Management Layer) configuration managers collaboratively maintain a network-level information model of distributed CORBA objects. It also provides the capabilities to manage the individual network elements (such as circuit packs inventory and NE wavelength cross-connection) and support for remotely setting the ports and wavelengths in-service or out-of-service. A user friendly GUI is provided to allow the user to access and control the network resources. The GUI interface provides pull-down menus, easy point-and-click invocation of management functions, as shown in Fig. 2. Easy to understand icons are used to represent the network objects, such as network map, link, site, faceplate, and port. Fig. 7 is an image from the MONET NC&M System depicting the topology of the MONET network. NEs are depicted by colored icons trapezoids (ADMs), diamonds (WSXCs), and triangles (WAMPs 2 ). Links are shown by edges connecting NEs. The configuration manger is responsible for discovery of network resources from the NEs. With this capability, the NC&M system is able to discover: 2 Wavelength Amplifiers (WAMPs) are also called Optical Amplifiers (OAMPs) in other literature. The physical resources installed in each NE (circuit packs and ports). This includes the equipped and unequipped state of the resource and any existing alarm conditions. The associated logical resources of each NE (link, trail, and CTPs). This also includes state information for the resource. Any cross-connections between CTPs in the NEs including alternate CTPs for protected cross-connections and multicast cross-connections. The configuration manager reads inter-ne resources, e.g., links from a data file. This data file may be populated by another process that discovers the topology of the network by querying the NEs. This topology discovery was placed in a secondary process to insure that the NC&M system was presenting a consistent view of the network from invocation to invocation. Previous experience with automated topology discovery in other networks suffered mixed results: Frequently some NEs could not be raised causing the NC&M system to show different network maps for different management sessions. This behavior is not appropriate for the NC&M system where all expected resources should be recorded in some way in the NC&M system. The topology discovery routine may be initiated at any time

8 WILSON et al.: MONET MANAGEMENT AND CONTROL 2045 by the NC&M system. The topology discovery capability is discussed further in Section V. C. Connection Management MONET connection management covers the necessary algorithms and functionality needed to provide connection service to the network s clients. These clients may be direct end-users requesting transparent optical transport pipes, or they may be higher layer transport facilities such as a SONET section. With regard to end-to-end connection establishment and release, we distinguish between two paradigms: Management Provisioning. Provisioned connection setup is initiated by a network administrator via a management system interface. A provisioned connection setup procedure typically operates with extensive global knowledge of the network. It can compute routes, which are more optimal and may satisfy various additional routing and Quality-of-Service (QoS) constraints. End-User Signaling. Signaling connection setup is initiated by an end customer directly from a customer station via a signaling interface. Signaling connection setups are typically based on best-effort attempts in order to cope with their potentially high level of concurrency, and to deliver fast response time. Connection management research in MONET encompasses both the provisioning and the signaling paradigms. Research results in signaling setup is reported elsewhere in [12], [13]. In this paper, we focus on describing the setup and management of provisioned connections. The remainder of this section describes different capabilities of the MONET NC&M connection management capability. 1) Automatic Routing: We briefly describe the basic routing algorithm used in the MONET NC&M system. Further details of the routing algorithm may be found in [7]. The basic approach to trail creation is for the user to select the trail origin and destination (TTPs) and, optionally, the desired wavelength from the GUI. The GUI then invokes a trail creation feature in the Trail Manager. The Trail Manager then requests, also via a CORBA interface, that the Connection Manager create a connection that realizes the desired trail. The Connection Manager uses three stages to create the connections: Route selection, wavelength selection and connection completion. The route selection phase finds a path of links (and the associated spare capacity) across the network using the following procedure. First, map the trail origin and destination TTPs to the EML level subnetworks containing those termination points. Second, find a shortest path across the EML subnetworks that have some operational and idle link connections. Third, for each subnetwork traversed by the path, the algorithm then recursively determines the path across the subnetwork down to the NE level. The design of the system allows the routing algorithm to optimize other criteria (e.g., distance, utilization) or for the NML router to consider details of the EML subnetworks when computing the path. The default wavelength selection phase simply finds the first available wavelength that is spare across the network. Our research shows that for a network that does not support wavelength interchange, there is little difference between wavelength selection algorithms in terms of network throughput. The management system allows the user to specify the wavelength to use for a connection. Again, the design of the system allows the wavelength selection algorithm to be replaced. For example, algorithms that select middle wavelengths or maximize or minimize utilization of wavelengths may be inserted. The connection completion phase starts by mapping the trail terminations to the CTP associated with the selected wavelength. The NML connection manager then partitions the path from the route selection phase into EML segments and requests that the EML connection managers create EML level connections for their associated segments. This process proceeds recursively until the NE level subnetwork manager creates a cross connection in the NE corresponding to the NE level connection entry and exit points. Upon successful completion of the EML connections, the NML connection manager creates a NML level connection associating the originating and terminating CTPs. It is possible that connection completion phase may fail. This would occur if between the route selection phase and the connection completion phase, another user reserved any resources that were idle during the route selection phase. Whenever the connection completion phase fails, the connection is rolled back and any intermediate NE cross connections are released. This rollback leaves the network in a clean state for future connections. The NC&M system notifies the user that the connection failed and suggests that the user resubmit the connection request. Using the management GUI, network administrators, can query the system to find the trails traversing through a network element, link, or port by clicking that particular object of interest. Network resource utilization, such as wavelength assignments, is automatically maintained by the system. Fig. 8 shows an example connection information screen displayed by the management GUI for all connections routed through the DIA WADM. 2) Manual Routing: Since the MONET network is being used to test and analyze the performance of optical networks, the users need to set up unusual connections. An example complex connection originates at a NE, routes completely around the network and returns to the origin. The NC&M prototype uses a manual routing capability to assist creating such connections. This capability allows a user to completely specify how a connection is routed in the network. The approach taken for this feature is as follows: The user first selects the trail origin and destination TTPs as in automatic routing. The GUI then invokes a network spare feature in the Trail Manager. The spare information is returned to the GUI. The GUI displays the spare information in a map and allows the user to select the desired path of link connections across the network. The GUI then invokes a manual routing feature in the Trail Manager. The Trail Manager then directs the Connection Manager to create the specified connection.

9 2046 JOURNAL OF LIGHTWAVE TECHNOLOGY, VOL. 18, NO. 12, DECEMBER 2000 Fig. 8. Example connection information screens. This capability places no restrictions on the connection that may be made beyond what the equipment supports. For example, some equipment does not support wavelength interchange and all equipment places restrictions on what wavelengths are reachable from certain client interface ports. Perhaps the most unusual trail this capability has been used to make originated at DARPA, passed through every WADM and WSXC before returning to DARPA where it returned back around the network until returning to DARPA a second time and terminated at that location. The agencies actually made several of these trails and then spliced them together to form a trail that passed through every WADM and WSXC four times in one of their signal measurement experiments. 3) Protected Trails: The equipment in the MONET network supports a tail end protection switching capability. In this capability the terminating NE can associate a primary and alternate sink CTP with a single source (drop) CTP. 3 Whenever the signal on the primary CTP degrades below a threshold, the equipment automatically changes to the alternate sink CTP thereby affecting a protection switch. This capability is being used to evaluate interactions between client layer protection schemes and WDM layer protection schemes. Fig. 9 illustrates the MONET NE tail-end switching capability. Here a source CTP (must be associated with a drop TTP) is protected with a sink CTP and an alternate sink CTP. If the signal quality of the sink CTP degrades below a threshold and below the quality of the alternate sink CTP, the fabric automatically switches from the current sink CTP to the alternate sink CTP. The result is that the signal now flows from the alternate 3 Notice that the convention concerning source and sink described here is from the perspective of the network equipment. Thus, a sink TP is where the signal enters the NE, and the source TP is where it leaves the NE.

10 WILSON et al.: MONET MANAGEMENT AND CONTROL 2047 Fig. 9. Tail end protection switching. sink CTP to the source CTP. The previous sink CTP now becomes the alternate sink CTP. Therefore, an NMS may find the alternate sink by querying the source CTP for its current alternate CTP. Trails are modeled in the NC&M system as binding an origin TTP with one or more destination TTPs. The binding between an origin and destination TTP may be supported by a primary and protection connection. In particular, a protected trail has a single origin and destination TTP and a primary and a protection connection supporting it. The identification of the primary and protection connection is delayed until the latest moment and is determined by querying the destination CTP for its alternate CTP. This information determines the alternate NE level connection that in turn determines the protection EML and NML connections. Thus, whenever the NC&M system reports the primary connection it reports the connection that is currently supporting the trail s signal flow. The protection routing capability is supported in the NC&M system by allowing the user to request protection for a specific trail. After a trail has been made the user invokes the protect trail feature in the Trail Manager. The Trail Manager invokes the compute alternate route capability in the Connection Manager. The alternate route is a node and link disjoint path connecting the trail origin and destination. If an alternate route is available, the Connection Manager completes the connection. The Trail Manger then records this connection as a protection connection for the trail. If an alternate route is not available, then the trail protection request fails and a message is returned to the GUI. 4) Multicast Trails: The NEs in the MONET network support a multicast capability. This allows a signal from a single origin TTP to be simultaneously transmitted to several destination CTPs. The NC&M system supports this capability by allowing the user to establish multicast trails. The general approach to this feature is for the user to establish a current trail and then successively add legs to it using manual routing. The network manager may delete the connection supporting each leg of the multicast trail independently. In particular, the primary connection may be deleted while legs added to the trail remain. The trail is removed only when all legs from the multicast trail are deleted. 5) Connection and Trail Discovery: Many individuals, in several government agencies, control the test-bed network. Some of the users prefer to use the craft interface for managing the network. In particular, some users make connections directly using the craft interface. The NC&M system takes pains to maintain synchronization with the network. The philosophy of the NC&M prototype is that the network is the database the NC&M system simply reflects what is in the network and provides some convenient methods for viewing the network configuration and changing it. The network discovery features described in Section IV-B describe how the NC&M system is able to discover base data from the network. This section describes how the Connection and Trail Managers manipulate that data and create EML and NML connection and trail information from that base data. Localized cross-connect information is retrieved from the network elements using a fabric interface on the NE agent. This information is translated into the NC&M information model and NE level subnetwork connections are created to reflect these cross-connections. The EML Connection Manager then aggregates NE level subnetwork connections into EML subnetwork connections. The process is based on the realization that within the context of a subnetwork, every CTP can be considered to be an internal CTP or a boundary CTP. An internal CTP is a CTP that is contained in a linktp whose supported link is contained in the subnetwork. All other CTPs are considered boundary CTPs. With this understanding, the connection discovery algorithm proceeds as follows: 1) The EML Connection Manager retrieves all child subnetwork connections (i.e., subnetwork connections in its children subnetworks). 2) For each subnetwork connection whose destination CTP is a boundary CTP, the connection is extended as follows: 2.1. If the origin CTP is a boundary CTP, then the Connection Manager has discovered a valid, complete, EML connection across the subnetwork and the connection is recorded If the origin CTP of the connection is an internal CTP, then the Connection Manager consults the resource model to determine the link connection bound to the CTP and obtains the link connection s far end CTP. The Connection Manager then finds the child subnetwork connection whose destination CTP is the just calculated CTP. 4 This child subnetwork connection is pre-pended to the route of the EML connection being discovered and the process continues. 3) Any child subnetwork connection whose destination CTP is not a boundary CTP is ignored. These subnetwork connections will either be included as part of the route of some other EML subnetwork connection or are a dangling connection that do not support any higher level connection. After the EML Connection Managers have discovered their connections, the NML Connection Manager uses the equivalent process to discover NML subnetwork connections. 4 This child subnetwork connection is in a different child subnetwork than the current child subnetwork connection. Also, there should only be one such child subnetwork connection since the equipment supports merging connections only at drop CTPs.

11 2048 JOURNAL OF LIGHTWAVE TECHNOLOGY, VOL. 18, NO. 12, DECEMBER 2000 After the NML Connection Manager completes its discovery, the Trail Manager then determines trails for each of the discovered connections. The Trail Manager is able to resolve point-topoint trails, multicast trails, and protected trails. Only network resource model errors can prevent all NML connections from being resolved into trails. Possible errors occur when the origin and destination CTPs cannot be associated with TTPs. In such cases, the configuration manager needs to re-synchronize the management system s internal resource information database with the actual network state. All trails discovered by this process are manageable by the NC&M system. In particular, discovered trails may be deleted, protected, or rerouted. The NC&M system provides an interface that allows the user to find and delete dangling NE subnetwork connections. The user may view the fabric of a specific NE that illustrates the connections in the NE. Any connection that does not have a supported trail is a dangling connection. D. Fault Management (FM) The Fault Manager receives alarms from NEs in the network. It collects alarms and correlates them with the network topology and connection information to locate the root cause of the alarms. Once it detects and identifies the failed resources, it alerts the human network administrators through an alarm notification window. The corresponding icons on the network map also change colors to highlight the failures. The network administrator can then initiate corrective actions. Referring to the example network shown in Fig. 7, the red colored icons reflect a major or critical fault while yellow icons reflect a minor fault and green icons (or black links) reflect the absence of faults. The process consists of three parts: 1) Alarm Collection, 2) Alarm Correlation, and 3) Alarm Log. 1) Alarm Collection: The FM system attempts to obtain a list of all outstanding alarms from the NE controllers at startup time using three separate commands defined on the NC&M-to-NE interface: read shelf alarms command used to read the alarm log of currently active alarms that are associated with the shelf or its power supplies. read existing cktpck alarms: read the alarm log of all outstanding alarms associated with slots or circuit packs provisioned in the slots. read existing port alarms read the alarm log of all active alarms that are associated with ports or facilities. During normal operations, the FM system relies on autonomous alarm messages from the NE controllers to maintain its internal list of active alarms. These autonomous events are delivered via a socket connection between the NE controller and the NC&M agent. 2) Alarm Correlation and Root Cause Analysis: The FM system performs temporal correlation among incoming alarms. It uses updated topology and connection information to determine if correlated alarms share a root cause. For example, a link cut should result in a total power failure alarm at the TI IN port on the downstream end of the link. If the link was carrying optical connections, then cascading single-wavelength threshold crossing alarms should also be emitted for the downstream optical wavelength CTPs along all trails routed over the failed link. In practice, the failure of a single unidirectional link can lead to dozens of alarms, even in a small network. The FM system suppresses these secondary alarms using topology and connection information from the configuration and trail management modules. The result is a single root-cause alarm attributable to the link object. Note that this is a synthetic alarm, since the link object does not exist within the NE-level information model. In order to correlate alarms for trails which cross subnetwork boundaries, the FM system operates in a hierarchical mode with one EML FM process for each subnetwork and a single NML FM process operating at the network management level. This NML FM process models end-to-end trails and link that span multiple subnetworks. Root-cause alarms from the EML level are correlated (and suppressed, if necessary) by the NML fault correlation logic. 3) Alarm Log: Upon receipt of a new alarm, the EML FM module logs the alarm to a text file in human-readable format. For this purpose, a special object class maps enumerated values within the alarm data structure to text strings. The alarm logging can be turned off, if desired. The alarms are also entered into a machine-readable database. A text browser is available to peruse this database. Unlike the alarm log, this FM database only contains active alarms. Shortly after an alarm is cleared, the alarm record is purged from the FM database. The primary purpose of this database is to provide a persistent store for the alarms and alarm-correlation data. The FM module can be terminated and restarted, if necessary, without losing track of the state of alarm correlation processing. The fault manager receives alarms from network elements in the network. It collects alarms and correlates them with the network topology and connection information to locate the root cause of the alarms. Once it detects and identifies the failed resources, it alerts the human network administrators through an alarm notification window. The corresponding icons on the network map also change colors to highlight the failures. The network administrator can then initiate corrective actions. E. Performance Management Performance management is an area of great interest in optical networking and an area where many vendors try to distinguish their products. Performance management is a major complication in transparent optical networks since optical performance measurements do not directly relate to QoS measures used by carriers. Typical optical performance measures are limited to the following attributes: Optical power is the strength of the signal measured in dbm. Optical power may be measured at an individual wavelength or a total optical power covering all multiplexed wavelengths and may be measured at the client or transport interfaces. Optical Signal to Noise Ratio (OSNR) is the ratio of the power of the signal to the power of the background noise measured in db. OSNR is defined for individual wavelengths and may be measured at the client or transport interfaces. Wavelength registration measures where the peak optical power of a signal occurs. This may be compared with the

12 WILSON et al.: MONET MANAGEMENT AND CONTROL 2049 tuning of the optical receiver. Wavelength Registration is measured for individual wavelengths and may be measured at the client or transport interfaces. It is most relevant at the client interface, since that is where unmanaged signals from client equipment enter the network. The main complication is that carrier QoS measurements are concerned with attributes like bit error rate (BER) and errored seconds (ES). There is no simple direct relationship between the optical measurements and these QoS measurements. Vendors of SONET- or SDH-based WDM equipment can compensate for these limitations by monitoring the quality of the electrical signal at locations in the WDM network. This information may be used by a network management system to isolate network performance problems. Since the MONET network was designed to be a transparent optical network, the MONET NC&M system is limited to analyzing network performance based on the optical performance measurements. A possible extension for subsequent study is to consider coupling the client layer (e.g., ATM layer) performance measurements with knowledge of how client layer links are routed in the WDM layer in order to help isolate performance problems. The limitations of purely optical performance measurements not withstanding, the remainder of this section describes the Performance Management capabilities in the prototype system. 1) Port Optical Performance: The MONET NC&M system supports viewing performance measurements from the agent directly in the GUI. The most basic capability allows the user to pull up a network element faceplate, select a port and ask to view performance measurements for Optical Signal to Noise Ratio (OSNR), Single Wavelength Power (SWP) and Wavelength Registration (WR). These readings are taken directly from the controller on a periodic polling basis and displayed showing the current measurement. Additionally, a strip chart of the measurement s history (color-coded to show when thresholds are crossed) may be displayed. 2) Trail Performance Analysis/Debugging: The MONET NC&M system enables a network manager to analyze the performance of specific trails. Here a manager may examine individual components (e.g., cross-connects or link performance) of the trail directly through the use of the GUI. A manager may also analyze the trail s end-to-end performance. Performance management is coupled with connection management, enabling a user to view end-to-end trail performance from the Connection Details window. The user may also view the performance of a connection across a specific node or link by selecting on the Connection Route Details window. By clicking on a port, the user may query the agent for performance measurements relevant to the connection. By clicking on a network element, the user may query the agent for performance measurements where the connection enters and leaves the network element. Similarly, performance across a link is available by clicking on the link between network elements. Fig. 10 illustrates this capability. F. Functional Integration Configuration and connection management provides the necessary support to the other MONET NC&M management Fig. 10. Integrated connection and performance management. functional areas. In particular, the network and connection information databases contain essential information utilized by MONET s performance management and fault management modules. Configuration and connection management also receives feedback information from these other management modules to update its databases, and to modify its own provisioning behavior. 1) Performance Management Interactions: Performance management in MONET is responsible for monitoring and maintaining the QoS for the active established WDM connections. In particular, it is responsible for monitoring and isolating QoS related soft faults in the optical network. MONET s performance management system allows a network administrator to query the optical signal performance figures such as signal power readings and signal-to-noise ratios, along an established WDM connection. Should a QoS problem develops for a connection, a network administrator can troubleshoot the problem by stepping through the measurement points along a connection route, and examining the retrieved performance figures. This way, a network administrator can narrow and isolate the stretch of network resources that might be responsible for the perceived QoS violation. Alternatively, during the lifetime of a connection, MONET s performance management module can pro-actively manage the QoS of that connection by continuously monitoring the optical signal performance figures along its route. By establishing certain preset QoS thresholds, the performance management module can alert a network administrator when those thresholds are violated and corrective actions may then be initiated. To support these operations, the connection route information is exported and made available to the performance management module. Likewise, the optical signal performance figures collected over time may be fed back to the connection management module. By periodically updating the optical performance attributes in the corresponding network resource information objects, variations in these parameters could be taken into account in subsequent provisioning computations. 2) Fault Management Interactions: Similarly, MONET s fault management activities require support from connection management. FM receives alarms, (e.g., link cuts and equip-

13 2050 JOURNAL OF LIGHTWAVE TECHNOLOGY, VOL. 18, NO. 12, DECEMBER 2000 ment failures) from network elements. In general, many more alarms will be generated as a result of the failures, therefore alarms collected are correlated in order to determine the root-cause of the failures. In particular, alarms are correlated with the network topology and the active connections to identify their causal dependencies. The human network administrator will be alerted through the alarm notification windows, which display the alarms and their correlation results. As equipment and link failures are identified and localized by the Fault Manager, the operational states of these network resources are updated. The Trail Manager receives a notification of the failure and analyzes the network to determine the trails affected by the failure. At this point the NC&M system sends a message to the network administrator identifying the affected connections. Corrective actions such as setting up an alternative route for an affected connection may then be initiated by the network administrator. As the operational states of these resources are updated, they are reflected as changes in the corresponding management information object attributes. The connection management module notes these changes and such resources are excluded from subsequent provisioning computations. Likewise, when failures are repaired, the alarm status of these resources are cleared, such resources will again be included in subsequent provisioning computations. The Fault Manager also listens for connection creation and deletion notifications from the Connection Manager and updates its internal data structures appropriately. This capability also works for connections that are discovered by the Connection Manager. Thus, the Fault Manager always has an accurate view of the network topology and connections for use in fault correlation. V. NETWORK CONTROL Telcordia studied two agent level network control capabilities. The first control capability is in the area of signaling-based connection management called MONET Just-In-Time signaling. The second control capability supports a topology discovery capability for WDM networks. A. MONET Just-In-Time Signaling The MONET Just-In-Time (JIT) signaling capability is designed for low latency transport of data-bursts across an all-optical network. The work involved specifying the architecture for a JIT User Network Interface (UNI) signaling protocol and implementing an initial version of the protocol. This signaling protocol contained support for establishing fixed duration burst connections and variable duration circuit switched connections. The MONET JIT capability support connection setup and connection tear-down capabilities using five messages: Setup (request for service) Call-Proceeding (acknowledgment from the network to the user) Connect (acknowledgment from the destination that the connection is complete) Release (request for end-of-service) Release-Complete (acknowledgment that the connection has been deleted) The protocol is implemented in a set of signaling messages that may be encoded in a single 48-byte ATM cell. The cell is transmitted from the client host to the agent managing the NE supporting the client host. The network uses pre-defined routing tables to determine the route of the connection across the network. Details of the MONET Just-In-Time signaling capability are given in [13]. B. Topology Discovery The MONET NEs are not required to support an inter-ne signaling protocol capable of supporting network topology discovery. However, initial experience with lab networks and the test-bed network reveals a need for automatic network topology discovery. This capability allows the NC&M system to accurately reflect the topology of the WDM network with minimal work on part of the network managers. To address this issue, the NC&M system implements a network specific signaling protocol using the agents. This implementation relies on the presence of an embedded Data Communications Channel (DCC) on each transport fiber and a known mapping between ports on ATM switches in the MONET Data Communications Network and transport interfaces on the WDM elements. Fig. 11, below, shows how the MONET network carries the embedded DCC. Coming out of the ATM switch is a common 1310-nm wavelength signal. This signal passes through a transponder (TxP) that transforms the wavelength to 1510 nm. This signal is then optically multiplexed with the 8-wavelength signal from the NEs Transport Interface-OUT (TI) port. This combined 8-wavelength plus signal is carried by the fiber to the adjacent NE. There the combined signal is de-multiplexed into the standard 8-wavlength signal destined for the NE and a separate 1510 nm-wavelength signal that may be input directly into the ATM switch. Fig. 11 illustrates that by knowing the association between the ATM port and the WDM port with each NE allows one to use ATM signaling capabilities to deduce the WDM topology. The topology discovery capability in the MONET NC&M system may be summarized as follows: At install time, given ATM port WDM port mapping, create a set of PVCs on the node s ATM switch for all TI ports to the agent. Use a well-known VPI/VPI on the TI side. Agent broadcasts its NSAP and port ID (TI-out) on each outgoing PVC. Receiving agent notes the NSAP and port ID and its own input port ID in its adjacency table. Receiving agent broadcasts a response to all output PVC indicating the original NSAP, output port ID and it s own NSAP and input port ID. Agent exposes a CORBA interface to its adjacency table. A discovery client probes agents for their link configuration by discovering neighboring nodes through the adjacency table.

14 WILSON et al.: MONET MANAGEMENT AND CONTROL 2051 Fig. 11. Configuration of embedded DCN. VI. INTEROPERABILITY A crucial objective of the MONET project is interoperability. This objective is pursued in two different methods: detailed specification of the NC&M-to-NE interface; and the use of standard models and computer technology for management interfaces. The implementation and the effect of these efforts are described in this section. Telcordia, working closely with other members of the MONET consortium particularly Lucent Technologies designed and specified the MONET NC&M-to-NE interface. This interface specification includes management objects and message types (including commands and replies, notifications, alarms, alerts and autonomous event messages). The interface specification also defines state transition possibilities for slots and termination points. The following nine object types are used to specify the NC&M-to-NE interface: 1) Network Element 2) Shelf 3) Slot 4) Circuit Pack 5) Fabric 6) Optical Signal Trail Termination Point 7) Optical Wavelength Trail Termination Point 8) Optical Wavelength Connection Termination Point 9) Cross Connects The Termination Point objects have source and sink subtypes associated with them. Two types of messages are supported by the NC&M-to-NE Interface: Commands (including replies), and Autonomous Messages (including notifications, alarms, and alerts). A major source of interoperability falls under the state and state transitions for slots and CTPs. For slots, MONET uses three state variables: service state (equipped or unequipped); usage state (active or idle); and health state (good, failed, or removed). Similar state transition possibilities are defined for CTPs. The specification of allowable transitions is the result of painstaking design and analysis to allow the NE to automatically detect equipment as it is added to the element. While allowing automatic detection the NE must maintain a persistent view of the equipment so that conditions (such as circuit pack removal) may be identified as failures and appropriate alarms generated and sent to the NC&M system. Fig. 12, below, shows the slot state transition diagram used for MONET equipment. For example, the diagram shows that when a circuit pack is inserted into a slot, the slot automatically changes from the state tuple unequipped, idle, good to equipped, idle, good. It also shows that certain transitions are not allowed. For example, a user may not unequip a slot whose usage state is active. Fig. 12 also shows that an attempt to unequip a slot whose state is equipped, idle, failed, will be countered immediately by the NE (automatic detection) to state equipped, idle, good that will then change (by failure detection) back to state equipped, idle, failed. The design of the agent and its management interface enhances interoperability between different vendor hardware. The agent management interface is based on the ITU M.3100 standard specialized to WDM network technology. The network model is based on network layering concepts discussed in G.805 and network partitioning concepts discussed in G.853 [9] and TINA-C [10]. To assure seamless operation across two radically different network element designs, some managed objects support additional methods. A crucial example of this work involves reachable wavelengths for different TTPs. This capability isolates the NC&M system from differences in NE designs and capabilities. The cross connect architecture of the NEs used in MONET are given in Figs. 13 and 14. A complete discussion of these architectures is beyond the scope of this article but several points are important to observe. The Lucent element has a main fabric (divided into eight planes by wavelength) and that client interfaces (CCI add and CCI drop) are connected to the main fabric via mini-fabrics. The mini-fabrics allow the Lucent element to adapt to different client input wavelengths. That is, a client interface conforming to MONET wavelength could be added at any CCI add port. The element design does restrict a specific wavelength usage among a subset of the ports. For example, only one CCI add port on mini-fabric 1 may use wavelength but wavelength could still be used by a CCI add port on mini-fabric 2. The Tellium element (Fig. 14) has a main fabric that is divided into two segments (wavelengths to and to ). Client interface ports (NCI adds) are restricted to operating in one of the two wavelength groups). The NC&M system uses several techniques to transparently support these different fabric architectures. The first technique

15 2052 JOURNAL OF LIGHTWAVE TECHNOLOGY, VOL. 18, NO. 12, DECEMBER 2000 Fig. 12. Slot state transition diagram. Fig. 13. Lucent fabric. was to add a method to the client interface add ports (Trail Termination Points) called get useable wavelengths. This method returns a list of wavelengths that are currently reachable from the TTP. For Lucent elements, this method needs to examine what wavelengths are currently in use for the mini-fabric that the TTP is served by. For Tellium elements, the method returns the wavelength group that the TTP belongs to. The next issue is evident by carefully considering the implications of the Lucent architecture in light of the information model discussed in Section VI. The model requires that CTPs be contained in a link termination point. However, there is no obvious link TP associated with the client interface CTPs (there are linktps for transport interfaces). To resolve this discrepancy, pseudo-linktps are created for client interface CTPs. Now, since containment no longer specifies how client interface CTPs and TTPs are associated, additional methods identify the association. By adding these methods to the objects, we do not have to store this association in our data model and do not have

16 WILSON et al.: MONET MANAGEMENT AND CONTROL 2053 Fig. 14. Tellium fabric. maintain the information as new connections are added to the network either by NC&M or by the craft interface. VII. EXPERIENCES This section relates some of Telcordia s experiences in designing, implementing, and testing the system. A. CORBA Scaling The NC&M implementation is based on standard management objects where every managed object is instantiated as a CORBA object. For example, every CTP is a CORBA object. This design, with "fine-grain" objects, has implications on the scalability of the NC&M system. The first step in the analysis is to estimate the size of the information base used by the NC&M system. A rough estimate may be obtained using the following model. Let the network contain elements and links each with a capacity of wavelengths. Each NE contains add ports and drop ports. The network is partitioned into subnetworks each of whose diameter is elements. 5 Further assume that there are trails in the network. For usual networks, the most common objects are CTPs, link connections, and subnetwork connections. The number of CTPs is approximately (transport interface CTPs) plus (client interface CTPs) where depends on the architecture of the NE (for Lucent elements is 2 and for Tellium elements is ). The number of link connections is. 5 That is a typical connection passes through E elements to cross a network. The network may contain many more elements. Each NML connection is supported by EML subnetwork connections that are in turn supported by NE level subnetwork connections. If every trail was supported by one NML connection, then the total number of subnetwork connections would be. If every trail were supported by two NML connections, then the number of subnetwork connections would double (though a more efficient implementation may reduce this scaling). To give an idea on how this works out suppose a network contains Lucent elements, links, wavelengths, add and drop ports, trails and network diameter. This works out to CTPs; link connections; and subnetwork connections. The number of CTPs would be for Tellium elements the number of client interface CTPs grows as a function of the number of wavelengths. Clearly, however, the number of CTPs and SNCs dominate the size of the information base. Telcordia conducted a variety of experiments to determine the suitability of CORBA for a fine-grain implementation of a large-scale network. We did this by constructing a large number of CTP-like objects and binding with those objects. The tests were done on a lightly loaded Sun Sparc Ultra-2 system using the Orbix 2.1 CORBA implementation. Briefly the tests showed that the time to create the objects grows linearly with the number of objects until process swapping degrades the system throughput. The binding time is quite slow and seems to limit system size to a few tens of thousands of CTP objects. Different ORB vendors will likely offer different performance characteristics. But it is clear that if the number of objects were to grow rapidly (say by supporting SONET layers), Orbix 2.1 with fine grain objects is not a practical implementation. One approach to scalability is to use a coarse-grain object approach. In this approach, a manager might create one (or a few) CTP objects whose methods are parameterized by the specific CTP that the method is to be applied to. This way the system avoids having CTP objects for every possible CTP. B. NE Simulators It is critical that the NC&M system interface with vendor supplied software as early as possible. Even with a detailed specification of the MONET NC&M-to-NE interface misinterpretations occur. The only way to be sure that the systems are interoperable is work with vendor software. Examples of complexities we experienced included the treatment of linktps for transport interface equipment on some of the equipment and on how transparency is achieved in some of the equipment. C. Computing Environment Unreliability Testing against vendor software does not resolve all interoperability problems. Some problems that occur in the real network that are not easily emulated in a lab environment. Some of the more persistent problems in the MONET network are: Data communication network and processor reliability. The NC&M system is a distributed management capability realized over a large number of computers. Agents

17 2054 JOURNAL OF LIGHTWAVE TECHNOLOGY, VOL. 18, NO. 12, DECEMBER 2000 Fig. 15. MONET NC&M deployment. are installed and run on resident NC&M processors located with each NE. Those agents communicate with EML management processes running on several workstations located at government sites around the network. Those EML managers communicate with NML management processes running on workstations at the government sites. The deployment planned for MONET (see Fig. 15) shows the NC&M management stations located at NSA, NRL and DISA. Fig. 15 does not show amplifiers that are also supported by agents. The MONET network regularly loses IP layer contact with one or more network elements. Deadlocked communication between the agent and the NE controller. The NC&M system suffers a problem where the agent and the NE controller communicating over a socket deadlock. The agent socket enters a wait state pending a response from the NE controller that never arrives. The controller receives the command, acts on the command, and attempts to send the command reply, but the message never gets to the agent. This seems to occur when the agent is making connections. The only solution to the deadlock problem is to exit the agent and then contact the NE controller again. In extreme cases, the NE controller needs to be rebooted. A management problem that arises from this situation is that the NC&M system waits pending a response from the agent. When the agent is restarted, the NC&M system receives and abnormal termination and assumes that the command failed. It then rolls back other connections that had been made but leaves the connection in the deadlocked network element NC&M believes the command failed. Over time, NEs experiencing this deadlock problem accumulate isolated connections that need to be deleted individually. Fig. 16. Test network configuration with no valid trails. such a way that all transport CTPs were in-use but there were no valid trails (see Fig. 16). The vendor used special equipment to inject signals directly into the transport interface of one NE. They then routed those signals across the network using all the transport CTPs before dropping the signals at a client drop port. This allowed the vendor to test their equipment and signal transmission properties with minimal user intervention. The problem to the NC&M system is that transport CTPs are not valid TTPs therefore the connection discovery algorithm finds the NE connections and marks all the CTPs in-use. However, the algorithm fails to resolve those NE connections into EML or NML connections or trails. Initial users saw that there were no trails, assumed that the network was idle and that connections could be made. When the automatic trail creation capabilities of the NC&M system failed to make any connections several hours were spent looking for problems in the wrong part of the system. A detailed examination of the discovery process revealed that the system behaved as designed but that the design did not account for incomplete trails. Based on this experience the NC&M system implements a variety of NE and EML level connection management capabilities. These capabilities include viewing and deleting NE level subnetwork connections. In particular, dangling connections are readily visible to the user. For example, a Fabric View (Fig. 17) depicts the element fabric with the source and sink CTPs arranged on opposite sides of the fabric. Lines are drawn between CTPs where connections are present. If the connection cannot be resolved into a trail, then a dashed line is drawn so that a user immediately sees any dangling connections. The user may delete dangling connections directly from this window. These management capabilities continue to be useful for identifying and cleaning up dangling connections in the test-bed network. D. NML, EML and NE Level Management The initial implementation of the NC&M system supported viewing NML connections that support trails. The limitations of this design decision were apparent the first time we hooked up the NC&M system with a lab network. The vendor had configured a network of three NEs and two bi-directional links in E. Discovery versus Persistence The NC&M system adopts the view that the network is the database. This view is appropriate for a test-bed network where many people work on a relatively small network in a loosely cooperative environment. This view does simplify certain tasks, particularly discovery. In discovery the NC&M system does not

18 WILSON et al.: MONET MANAGEMENT AND CONTROL 2055 Fig. 18. Revised trail query implementation. Fig. 17. Fabric view window. worry about reconciling discovered data (equipment, topology and connections) with previously discovered data. This approach makes little (if any) difference for equipment and topology discovery since those objects are long-lived and are generated with the same ID from invocation to invocation. For example, CTP IDs are provided by the equipment based on their shelf, slot and wavelength. This assures that the same CTP gets the same ID each time it is discovered. However, some objects, including subnetwork connections and trails, are given arbitrary IDs by the NC&M system. This lack of persistent connection and trail IDs leads to an interesting implementation problem. The NC&M system supports the ability to display all connections that use some resource (e.g., CTP, TTP, link connection ). The original implementation stored the ID of the trail assigned to each resource. The problem arises during connection re-discovery. Since trails are given new IDs every time connection discovery occurs, the trail information needs to be updated in every object that supports a trail. The implication is that if any EML manger restarts, then trail discovery needs to occur which causes every other EML manager to update its trail reference data even though no connection in connection and trail networks has changed. This problem occurs because the system stores NML information (a trail ID) in NE and EML level objects (CTPs, TTPs, link connections, and subnetwork connections). The key to eliminating this updating is to restructure the system to remove unnecessary upward pointing data. This requires a fundamental change in how the system determines the trail being supported by a resource. The revised system design, shown in Fig. 18, replaces the trail data attribute in each object with a method to compute the trail that the resource supports. Using a set of lookup tables in each level manger makes the search faster and enhances the performance of the lookup. The approach is based on several observations: Every resource we are interested in querying for supported trail may be mapped to one or more CTPs (e.g., every TTP that is currently supporting a trail has an associated CTP). Every Connection Manager knows its Parent Connection Manager. Every CTP can readily find the Connection Manager supporting its containing subnetwork. The basic process for determining the trail supported by a resource is to: 1) Map the resource into the associated CTP using the information base. 2) The CTP queries its NE manager for the trail supported by the CTP a) The NE manager looks up the supported NE connection in its CTP to NE conn lookup table. b) The NE manager asks its parent (an EML manager) for the trail supported by the NE connection. i) The EML manager looks up the supported EML connection in its NE conn to EML conn lookup table. ii) The EML manager asks its parent (an NML manager) for the trail supported by the EML connection (1) The NML manager looks up the supported NML connection in its EML conn to NML conn lookup table. (2) The NML manager looks up the supported trail in its NML conn to trail lookup table. The NC&M system maintains the lookup tables as a performance feature. Instead of maintaining the tables, the system could search through its active connections and trails find out if it has any object supported by the queried resource. While it seems tempting to identify these objects by the CTPs or TTPs they connect, this may lead to unanticipated problems. The first problem is that the end-points alone do not uniquely identify a connection, the entire route of the connection is required to assure a match. The second problem is that even if