Large-Scale CORBA-Distributed Software Framework for NIF Controls. Robert W. Carey Lawrence Livermore National Laboratory

Transcription

1 Large-Scale CORBA-Distributed Software Framework for NIF Controls Robert W. Carey Lawrence Livermore National Laboratory November 2001

2 Key topics Layered Architecture Leveling and Dependency Management Framework Abstractions Distribution and Object-Oriented Design Patterns Distribution Complexity Persistence Layer Process Management

3 A common framework is used to build all software applications Server Integration Services Workstation Supervisory Console - System manager - Device hierarchy - Access control Database - History - Shots - Configuration Operator Controls Status Display Event Log Software Distribution Bus (exists on network) CORBA Object Request Broker 300 front-end processors interface to NIF equipment Device Control Controller Interface Driver Status Monitor Front End Processor Software objects representing control points plug in to the software distribution bus

4 ICCS Distributed Component Architecture Framework Server GUI Supervisor LCU FWL FWL FWL CORBA Logical Control Unit (LCU) objects in the Supervisors control hardware through device objects in the Front- End Processors (FEP) FWL FEP device A Framework Layer (FWL) resides in each distributed process 1ω region 3ω region POWER AMP MAIN AMPLIFIER Preamp Fiber Master oscillator

5 Key topics Layered Architecture Leveling and Dependency Management Framework Abstractions Distribution and Object-Oriented Design Patterns Distribution Complexity Persistence Layer Process Management

6 Frameworks are constructed in layers to permit retargeting ICCS Programs Client and server mainlines Supervisor applications, Front End Processors, Database servers, etc Framework Services Customized for a specific system Configuration Server, System Manager, GUI s, etc NIF Building blocks Classes that model equipment Devices, Shot phases, etc Framework Templates Layer Abstract classes for control systems Abstract devices, Configuration, Monitor, etc Support layer COTS and components Oracle DBMS, ORBexpress, etc

7 ICCS is divided into subsystems to partition activity and ensure performance Shot Director NC11 Beam Control NC14 Laser Diagnostic NC15 Injection Laser MOR NL11133 PAM NL11224 Target Diagnostic NT82 Power Conditioning NL13142 PEPC NL Shot Services NC12 Vertical Partitions Automatic Alignment NC13 Optics Inspection NC16 Wavefront Controller NL12151 Qty 24 Precision Diagnostic NL2233 Qty 2 Laser Energy NL42 Qty 6 Supervisory Subsystem Layer Master Oscillator NL11131 Qty 2 Preamp Module NL11222 Qty 48 Target Diagnostic NT321 Qty 1 Power Conditioning NL13141 Qty 4 Switch Pulser NL Qty 4 Plasma Pulser NL Qty 4 Industrial Controls NC44 Qty 1 Hartmann Image Proc. NL12455 Qty 24 Laser Power NL43 Qty 12 Front-End Processor Layer Pulse Diagnostic NL Qty 4 Alignment Controls NL41 Qty 102 Special CCD NL44 Qty 15 Process Video NC41 Qty 26 Timing NC21 Qty 14

8 Key topics Layered Architecture Leveling and Dependency Management Framework Abstractions Distribution and Object-Oriented Design Patterns Distribution Complexity Persistence Layer Process Management

9 Supervisors and FEPs are built on a common set of frameworks Generic Supervisor Main Public Supervisory Objects Private Supervisory Objects Public Factory Local System Manager Other Framework Objects Private Factory System Manager Status Monitor Message Logger Device Reservations Devi ce Configuration Machine History Alert/Event Graphical User Interface Sequence Control Language Generic Supervisor Generic Public Geneic LCU Application Configuration Shot Data Archive Shot Life Cyle Generic FEP Generic Device Generic FEP Local Local System Manager System Manager Other Framework Objects Device Factory Main Device Objects Controller Objects Controller Factory This common architecture enhances reliability and improves maintainability.

10 The frameworks provide architecture templates and utility services Framework templates define the architecture for each type of process in ICCS: Supervisory Shot Control Processes Supervisory Status and Control Processes Front End Processors Graphical User Interfaces Frameworks also provide utility services: Configuration Messaging (Events, Alerts, Logging) Status Monitoring Reservation Archiving Sequence Control

11 Application Process Architecture C Connection Objects Application Objects T C O R Object Factory System Manager Status Monitor Message Log Alerts Events Reservations Shot Archive Machine History Config/ Name Service P B Framework Agents A Heartbeat Startup/Shutdown All application processes are constructed the same They all startup and shutdown using the same protocol Content of application processes is completely data driven Service distribution is encapsulated by Framework Service APIs

12 Key topics Layered Architecture Leveling and Dependency Management Framework Abstractions Distribution and Object-Oriented Design Patterns Distribution Complexity Persistence Layer Process Management

13 O-O design patterns extended for distribution Observer (Publish/Subscribe) is used extensively by the framework Alerts, Events, Process State, Status Propagation Publishing is de-coupled for each subscriber Failure on publish cancels subscription Object Factory process content is data driven Persistence brokers serve process data via CORBA Model-View-Controller Architecture Thin GUI layer presents the appearance (View) Model defines system semantics CORBA bridges Java and Ada language environments

14 Key topics Layered Architecture Leveling and Dependency Management Framework Abstractions Distribution and Object-Oriented Design Patterns Distribution Complexity Persistence Layer Process Management

15 ICCS employs a component-based communication architecture 300 CORBA IDL classes (150 device, 50 framework and 100 supervisor) Policies define interface de-coupling mechanisms and common exception pattern Framework provides connection management Detection Notification Recovery Timed Invocation is a requirement Formal testing exposes deadlocks Goal : Fault Resilience degraded operation in the presence of server failure and recovery upon server restoration

16 Test suite characterizes detailed effects of CORBA failure modes Network Java Clients Java Servers Visibroker Ada Clients TCP/IP Ada Clients Solaris OIS TCP/IP OIS Ada Servers VxWorks Ada Servers Objects CORBA Transport OS Objects CORBA Transport OS Failures under different socket conditions: Server fails before/after initial client connection Client fails after server connection Failures during request processing: Server fails during processing Client fails during a request Client registers callback; client fails, restarts, and re-registers. Server attempts client call-back Client sends request to server. Server hangs Failure modes are characterized for both CORBA and TCP/IP

17 Process failure detection The System Manager Framework monitors heartbeats from all the processes to detect process failures The System Manager GUI informs operators of process failures System Manager Process State System Manager GUI Heartbeat Heartbeat Heartbeat ~10 CML ~10 CML ~300 CML Framework Server Supervisor FEP

18 Failure notification The connection management in each process receives process state updates of interest and maps this state to object references The connection management layer notifies client objects of server object failures 2:Get_Ref() Name Server 3:Update_Map() CML 1:Connect() Supervisor Map Object => Process 6:Update() 4:Subscribe_To_Process () CML FEP System Manager Server X 5:Heartbeat not delivered

19 Failure Recovery Connection objects manage object references and contain reconnection strategy 12:device command LCU 1:Connect() 8:Get Ref() <11:Is Initialized()> FEP device Supervisor LCU CML 3:Update Map() Connection Config_Local 9:Get Ref State() 2:Get Ref() 10:Ref State 7:Process State 4:Ref/Process Map Name Server Map Object Ref => Process 5:Subscribe() 6:Process Update () System Manager

20 Key topics Layered Architecture Leveling and Dependency Management Framework Abstractions Distribution and Object-Oriented Design Patterns Distribution Complexity Persistence Layer Process Management

21 Persistence Brokers insulate applications from DBMS Oracle8 DBMS Database ICCS Persistence Brokers ICCS File Server C O R B A Supported persistence mechanisms Oracle8i HDF XML Application Programs

22 Key topics Layered Architecture Leveling and Dependency Management Framework Abstractions Distribution and Object-Oriented Design Patterns Distribution Complexity Persistence Layer Process/Network Management

23 ICCS process/network management leverages SNMP technology Each ICCS host contains a SNMP sub-agent CPU utilization File System Statistics Memory utilization Task states HP OpenView Network Node Manager collects statistics for: ICCS Processes Network switches Network Hosts File Systems

24 ICCS Framework is positioned for scaling in 04 Successes Layering/Leveling Framework abstractions implemented Patterns and distribution Persistence Layer Iterative process gives us practice Formal testing is our savior Summary of Challenges Manage distribution complexity Systematic connection management in a fine-grained CORBA implementation TCP/IP transport configuration Configuration data management