EURO-PAR CONFERENCE. Carrier Grade Linux Platforms: Characteristics and Ongoing Efforts. EURO-PAR

Transcription

1 CONFERENCE Carrier Grade Linux Platforms: Characteristics and Ongoing Efforts 1

2 Presentation Objective In this presentation, we will discuss the characteristics of telecom platforms and present the ongoing development efforts to build Carrier Grade Linux Platforms, both by the industry and in Open Source. 2

3 The Open System Lab The Open System Lab is under the IP Network Organization at the Ericsson Research Corporate Unit. The lab is located in Montreal, Canada. 3

4 The Open System Lab People Networking Competence System Competence Linux and Open Source Competence Web Services Competence Security (Network, GPRS, GTP, Platform, Application) Universities High involvement from Graduates (Master and PhD) Professors on Sabbatical Lab Cluster technology (over 300 Processors) Security System Test Lab IPv6 Lab Router Research Lab End-to-end simulation and emulation Lab 4

5 What is Carrier Grade? Carrier Grade is a term for public network telecommunications products that require a reliability percentage up to 5 or 6 nines, or to percent. This translates into 5 minutes (5 nines) to 30 seconds (6 nines) of downtime per year. 5 nines is usually associated with Carrier Grade servers 6 nines is usually associated with Carrier Grade switches Carrier Grade Linux is a new flavor of Linux that is more robust than the garden-variety enterprise Linux. It promises to provide a standards-based, open-architecture software platform for converging telecommunications. 5

6 Faster Than You Think 10 minute Multimedia Presentation 6

7 Outline The Mobile Internet & Next Generation Networks Characteristics of Telecom Platforms Why Linux? The Server Platform (TSP) Commercial Efforts Open Source Development Lab Service Availability Forum Open Cluster Framework Open System Lab Experiences Conclusions & Challenges 7

8 The Mobile Internet 8

9 The Mobile Internet The Mobile Internet will become a part of our everyday business and personal lives: shopping, gaming, messaging, multimedia, entertainment, banking, 9

10 Always-on Revolution Always-on Capability Every device needs its own permanent IP address Today s dynamically assigned IP addresses will no longer be available The Future Millions and millions of: new Internet devices new Internet users Internet available everywhere, all of the time 10

11 Classes of Applications Personalization M-Commerce Messaging Location Services Multimedia & Entertainment 11

12 The Internet Today vs. Tomorrow The Internet Today Millions of users Web, , audio & video, The Internet Tomorrow (NGN) Billions of users and devices New technologies leading to novel applications Building a real information society that is Always-on Global village is a reality Convergence of applications and services 12

13 A networked society with always-on connectivity Different types of network infrastructures are linked through a common protocol (IP). IP New novel multimedia applications (using SIP) have different requirements to be met by the underlying IP protocol. 13

14 Next Generation Networks 14

15 Current Generation Network Architecture Services Wireless Networks Wireline Networks Data/IP Networks Cable TV Networks Multiple, single purpose networks. Bundled services are difficult. Access Transport & Switching Networks 15

16 Next Generation Network Architecture User applications Service Networks & Control & Gateways Communications control Connectivity (wireless, narrowband, broadband) Single, multipurpose backbone. Bundled services. Easy service creation. Network management. Consolidate billing. Ex: mobile + fixed + DSL Clients 16

17 Next Generation Platforms 17

18 Requirements for the New Telecom World Access independent High capacity Scalable Reliable Real-time with very low latency QoS enabled Support for multi-media services Manageable 18

19 Market Requirements (1/2) % Availability (Five nines). Global Platform with a Full Range of Access Gateways. Open API for rapid creation of new services. New Multi-Media Services for Business and Residence. Full voice/data feature transparency with existing circuit networks. PSTN voice quality and reliability. ITU Compliant; Standards Based Interfaces. 19

20 Market Requirements (2/2) ATM and IP Protocols with Inter-Working. Scalable up to Millions of Lines. Programmable Feature Server Supporting 3rd Party Development. Can Link to Any Vendor s ATM Switch or IP Router. Web Based Service Customization. 20

21 Drivers for IP Technology 21

22 Carrier Class IP Telephony and Multimedia Services Traditional Internet service is Best Effort Packets suffer from loss, delay, etc... Packet loss mainly caused by router congestion, not line transmission errors One-way packet delay is more correlated with number of hops than geographical distance. Best Effort service is enhanced by TCP through retransmission and sequencing. Real-time applications cannot be supported by TCP 22

23 IP Transport Benefits Lower cost infrastructure Faster provisioning of new features Easy integration of network elements Stimulate creation of new services Simplify harmonization of standards 23

24 Drivers of IP Technology (1/2) IP is gradually becoming a dominating transport technology thanks to recent advances in optics and routing technology and the impact that these have had on price/performance. When combined with other key technologies, such as IP-based virtual private networks (VPN), IP enables a new generation of advanced multi-service networks. The use of a common infrastructure based on a single technology simplifies network implementation and operation and helps reduce costs. 24

25 Drivers of IP Technology (2/2) There are two main arguments that drive the integration of IP technology into mobile core networks: Support for (new) IP applications to generate (new) revenues; A common transport technology to reduce costs. The entire mobile telecommunications industry is funded out of the end-user s pocket. To ensure future growth in the industry, the end-user value needs to be enhanced. Service and application offerings are the prime drivers of the entire network and terminal evolution. 25

26 Telecom Platforms Characteristics Hardware & Software Features 26

27 General Hardware Features (1/2) Telecom equipment have to deliver dependable and reliable performance during the conditions encountered under normal and abnormal circumstances (such as natural disasters). Telecom equipment have a high safety requirements and must not cause any risks or hazards to personnel, other equipment, or the physical structure where they are placed. Telecom equipment must comply with NEBS testing and design requirements to help make the equipment operate properly and safely. Network Equipment Building Systems (NEBS) are generic standard requirements to provide safety and reliability. NEBS requirements allow operators to use a single, uniform set of rules to evaluate the telecom equipment they plan to deploy fairly and impartially. Other standards are commonly used; however, they are either unidirectional or limited in scope. 27

28 General Hardware Features (2/2) Comply with standard telecom rack dimensions Provide redundant hot-swap/hot-insertion power supplies Provide hot-swap/hot-insertion disk drives Comply with -48 Volts Provide hot-swap/hot-insertion processors with multiple Ethernet ports Provide hot-swap/hot-insertion network cards Provide how-swap/hot-insertion tape drives Multiple boot options Processors should be able to boot through the network [network 1, network 2 for redundancy], Flash, floppy and CD-ROM when connected) Provide remote diagnostics support and alarms to monitor temperature 28

29 Clustered Telecom Platforms Current trends in the telecom platforms space is to move towards clustered platforms for the benefits they offer: High Availability: Isolate or reduce the impact of a failure in the node, resources, or device through redundancy and fail over techniques. Scalability: Expand the capacity of servers in terms of processors, memory, storage, or other resources to support subscribers/traffic growth Improved processing speed: High performance, fast access time and response time Efficient resources utilization: through load balancing & traffic distribution among all nodes of the cluster server Manageability: Reduce system management costs through appropriate system management facilities / middleware 29

30 Clustering in Telecom Platforms Clustering is the use of multiple loosely coupled, nothing shared nodes, to form what appears to users as a single highly available system. Application Middleware Operating System Processor Reliable & Fault-tolerant processor interconnect 30

31 Clustering in Telecom Platforms N + M redundancy of processors Mated pairs Fast inter-processor communication (TCP/IP is not fast enough) Single view of data Single view of platform (cluster) 31

32 Uptime The main operator requirements remains at 30 seconds of system interruption per year including hardware and software upgrade. Target % uptime Apply to an overall solution that involves integrated high-availability hardware, software (OS and middleware), and the application. Ensure uptime of mission critical applications, software subsystems, and hardware platforms. 32

33 High Availability (1/2) High availability (HA) is a term for the technology that enhances the uptime of computer-based communications systems by distributing functionality across multiple CPUs. In response to hardware and software failures, HA systems facilitate the rapid transfer of control (failover) from a faulty CPU, peripheral, or software component to a functional one, while preserving operations or transactions in-progress at the time of failure. 33

34 High Availability (2/2) Error Detection Damage Containment Error Recovery Fault Treatment (incl. dynamic reconfiguration) Assumption: We are dealing with systems comprising clusters of processors which share nothing. 34

35 Availability Defined Availability is best defined as: MTBF MTBF: Mean Time Between Failure MMTR: Mean Time To Repair MTBF + MTTR Example: If a system offered a MTBF of 20,000 hours with a MTTR of 2 hours, then its availability would be 99.99%, 4-nines. 35

36 Achieving High Availability A complete high availability solution that demonstrates 5- nines requires close integration of: High-availability hardware, A robust high-availability software solution, High-availability middleware, and Application software that can cause failover to redundant systems. 36

37 High Availability Levels (1/2) Source: IMEX HA Report 37

38 High Availability Levels (2/2) Source: Intel 38

39 Redundancy in HA Systems One important characteristic of HA systems is the redundancy of key subsystems. A highly available system includes: Redundant Ethernet to ensure constant networking connections Disk mirroring to ensure high levels of data reliability Redundant power supplies Example: CGL includes an enhanced kernel with hardened device drivers and fault response behaviors, such as panic handler improvements, that ensure the application logs appropriate messages and sends notifications before a kernel panic. 39

40 Other HA Features (1/2) When defining the high availability platform, some requirements are to support hot swap (reduce MTTR), remote boot, diskless operation, Concepts: hot insert (adding cards to the system not originally in place at boot time), hot remove, and identity maintenance (maintaining device identities across hot swaps and system boots). Different implementations of Linux support these features. 40

41 Other HA Features (2/2) Some other high availability challenges are: Flexible options for booting compressed and remotely hosted kernel images. Support of compressed r/w and read-only Flash file systems. Accelerated boot and daemon start times from several minutes to seconds Speeding shutdown Eliminating costly file system operations with journaling file systems. 41

42 Non-Stop Operations No single point of failure No scheduled downtime HA failover software Hot plugged components Software Configuration Control: Automatic restart of processes that originally executed on a faulty processor on the ones that are working Self healing In-service upgrade of software with no disturbance to operation 42

43 Zero Downtime Zero Downtime Operation is achieved through Data Distribution On-line Process Replication Software Fault Tolerance 43

44 Fast Recovery of Applications Maximize availability of application and services through application of specific monitoring and detection of failures. Provide automatic failover and recovery capabilities with very minimum interruptions to the users. 44

45 Failover Transparent failover capabilities: applications and users are automatically and transparently reconnected to another system. Update transactions are rolled back. Failover performance: depends on hardware configuration, instance recovery time and workload at time of failover. Multiple Failure Robustness: Should be able to survive multiple node failures and still provide protection for the mission critical applications. 45

46 Fault Tolerance (1/2) Hardware Fault Tolerance Relies on redundant processors, power supplies, Ethernet cards, disk storage,.. If an active component fails: It is isolated, A standby component becomes operational, The load is shared among available active components. Software Fault Tolerance Uses a combination of software and hardware redundancy to provide the necessary backup hardware in case of failure. The software becomes responsible for duplication, update and synchronization of information across redundant hardware components. 46

47 Fault Tolerance (2/2) Allows single site to expand to multi-site for disaster tolerant solutions. 47

48 Load Balancing Have a mechanism to balance workloads after a node failure, to minimize the performance impact on other applications running on the platform. 48

49 Concurrent Maintenance Allow scheduled maintenance to be performed on one node of a cluster while other nodes continue to provide service without noticeable degradation. 49

50 Online Configurability Allow online configurability to reduce downtime. Allow recovery to be either in active/stand or active/active modes to mitigate any idle systems. 50

51 Manageability Provide policy based management across entire spectrum of applications, software, and hardware. Single point of control for data management: Data movement Security Backup Recovery Assure availability and performance using Service Level Agreement type business approach. 51

52 Scalability Support linear growth as modular addition of HW and SW components happens. If we double the number of processors, we should expect to almost double the throughput of the system. 52

53 Database High Availability Requirements If a node or interconnect failure occurs. The surviving nodes perform recovery as: Cluster reorganization: OS cluster monitor determines cluster membership DLM Lock Rebuild: Time is proportional to the number of locks at crash Database Recovery: Cache recovery & transaction recovery 53

54 High Availability Storage HA storage is a critical and necessary part of a fault tolerant system. Hardware and software RAID support 54

55 Hardware Architecture Hardware redundancy Automatic software error recovery On-line backup Hot swap hardware replacement Adaptive hardware configuration Geographical node redundancy 55

56 Online OS and Applications Upgrade Support remote upgrade of operating systems and application software without any system interruption or disturbance. Support smooth software upgrades when old and new version of same process can coexist. Provide mechanisms to upgrade an application which is running; The system will deal with an old and new running versions of the application simultaneously. Possibility for application to arrange state transfer between old and new static process. 56

57 Capacity Control Overload protection by selectively rejecting messages when message queues become too long. Load regulation by API asking kernel if it is ok to accept traffic. 57

58 High Reliability The database used is distributed among all the processors in the system. If a processor crashes, other processor in the system takes over. The same mechanisms are also used for identifying and handling software faults. There is no risk of a software fault hanging the system: if a fault occurs, the system is automatically restarted in real time. 58

59 High Performance Must be fast enough to meet requirements Design cost/hardware cost trade-off Beware of unnecessary sub-optimization! Real-time performance is critical to many carrier grade applications. The ability to respond quickly and predictably to external events is a key feature of availability. 59

60 Software Management The software management layer provides the following services: Initial software loading Software reloading Software upgraded handling Hardware Configuration Management Loading configuration 60

61 Security Access control IPsec, etc 61

62 Key Software Issues (1/2) Provide a fast low level protocol for inter-processor communication Provide as high level support for high availability in middleware/os Cluster management / equipment management / alarm management Application start / restart / hand-over / fail-over / fail-back Replication of data (configuration data, state data) Mechanisms for software update Naming and addressing Use middleware/os mechanisms in a consistent way 62

63 Key Software Issues (2/2) The design of the software architecture of the application * from the start * should take into account: Scalability Failure Handling Error (software bug) handling Future modification Hot software upgrade Standard O&M interfaces SNMP, Corba, HTTP Common LCT components 63

64 Common Traps (1/2) 1. Design a non scaleable and non HA aware application and leave scalability and HA issues to later phases. 2. An OS and Middleware can help provide scalability and HA but they can t provide it alone! Scalability and HA must be designed into the application from the start, or the application will have to go through a major redesign later. 64

65 Common Traps (2/2) 3. Converting an existing non scaleable and non HA aware application to use a HA framework (middleware/os) requires a major redesign! 65

66 Linux for Telecom Platforms 66

67 Motivations Networks are converging for multimedia communications services. Future multimedia-type data services will require substantially greater bandwidth, requiring new architectures to reduce delivery cost. Commercial off-the-shelf (COTS) software components are a necessary part of these new architectures. Proprietary platforms, the mainstay of current architectures are more expensive to develop than open-standards based ones. Open-standards based solutions expand solution options and reduce time to market. 67

68 Vision Next generation and multimedia communication services are delivered using Linux-based open standard platforms for Carrier Grade infrastructure equipment. 68

69 Moving from Proprietary to Open Solutions NEP NEP proprietary proprietary Application Application stacks stacks and and platforms platforms From proprietary solutions to standardized, modular solutions Service Service and and Application Application Layer Layer Middleware Middleware SA SA forum forum APIs APIs Carrier-Grade Carrier-Grade OS OS AdvancedTCA AdvancedTCA (ATCA) (ATCA) 69

70 Towards Open Solutions Utilization of old infrastructures and technologies based on obsolete standards for modern requirements: Proprietary, closed system worlds Insufficient support of current and upcoming standards High administration and management costs Ongoing utilization only possible with continuously increasing costs Expired product life cycles Proprietary infrastructures did not take today's requirements into consideration How can the requirements of tomorrow be met? 70

71 Economic and Strategic Challenges Focus on generation of revenues with new data services: Networks supporting various service types are needed Verifiable ROI in legacy networks and equipment Expandable, flexible, modular solution approaches for new services Telecommunications platforms from modular HW/SW kits Reusability of these platforms Simplified, central, uniform administration Need for flexible, stable, powerful standard platforms Cost reduction, standardization, reusability 71

72 Linux and HA Linux seems a good choice for OS for HA systems because of the following motivations: Availability of source code Standard APIs and other interfaces Stable and robust platform Integrated, high performance networking Support for a broad range of processors and peripherals No runtime royalties Excellent performance in terms of throughput and real-time response Benefits: lower overall cost of deployment and faster time to market. 72

73 Linux: The Alternative to Proprietary OS (1/2) Linux was developed in networks for networks Open development (Open Source) High innovation rate Scalable for all kinds of requirements and infrastructures Independence from manufacturers and service providers New technologies and requirements are adapted in a more efficient and standardized manner than for any other operating system 73

74 Linux: The Alternative to Proprietary OS (2/2) New IP features are introduced between 6 to 18 months later on Solaris compare to Linux (or FreeBSD). Linux is available on all hardware/processors architecture (not dependent on a single hardware/processor vendor). We have access to source code in order to rapidly fix faults or add features to the kernel when required. We can contribute to the Open Source the required hooks for efficient integration of the upper-layer HA middle-ware. 74

75 Linux: General Characteristics Availability and cost (Soft) Real Time characteristics Performance & Scalability Reliability Flexibility Openness Hardware Languages Interoperability 3 rd party software Open Development Environment 75

76 Open Hardware Commercial HW Solutions Commercially Available: - cpci based Processor cards - Standard 100bTX Ethernet Switches - Off-the shelf Peripherals Ethernet Switches CD ROM Drives Tape Drive Hard Drives Future Proof Architecture: - Follow industry Price/Performance Curve (Moore s Law): Faster CPUs at Lower prices - Quick Availability of New Processors 76

77 Ericsson TSP Platform The Server Platform: A Case Scenario 77

78 Characteristics (1/2) Very high availability and robustness implemented in SW handles both HW and most SW errors. Modular HW with no single point of failure. (achieved Five 9:s, % uptime). Soft Real Time, scheduling, communication and Database Management System (DBMS) tailored for high performance. Linear Scalability by using loosely coupled processors. Zero downtime operation 78

79 Characteristics (2/2) Open hardware solution, Currently Intel/Pentium processors and easily portable to other architectures and processes Languages: C++, Java Open interfaces Interoperability: CORBA/IIOP, TCP/IP, Java RMI 3 rd party SW: Java and Linux 79

80 Processors Running Linux Application Application Application Application Application O&M&P O&M&P + #7 O&M&P + #7 O&M&P + #7 O&M&P + + #7 #7 TelORB TelORB MW TelORB MW + TelORB MW + TelORB TelORB OS + MW MW OS + ( = ( = DICOS) TelORB OS + ( = DICOS) TelORB DICOS) Prop. OS OS ( = DICOS) Std Std PC PC HW Std PC HW Std PC HW Std PC HW HW Application Application Application Application Application O&M&P O&M&P + #7 O&M&P + #7 O&M&P + #7 TelORB O&M&P + + #7 #7 TelORB Middleware TelORB Middleware Middleware Middleware TelORB Middleware Linux Linux Linux Linux Linux Std Std PC PC HW Std PC HW Std PC HW Std PC HW HW 80

81 Linear Scalability Capacity grows linearly as processors are added. New processors can be added without disturbance

82 Sample TSP Applications HLR Home Location Register AC Authentication Center SCP Service Control Point MG Mobility Gateway SCS Service Capability Server AAA Authentication Authorization and Accounting CCN Cost Control Node 82

83 Advantages Standard interfaces through CORBA, TCP/IP, Java RMI Linux in the cluster means openness for 3p SW Based on commercial processors Fault tolerance implemented in software Standard languages: C++, Java Fully scalable architecture Soft Real-time OS Includes powerful middleware: A database management system and functions for software management Fully compatible simulated environment for development on Solaris/Linux workstations 83

84 Why Open Standards? 84

85 Open Standards (1/2) Open standards are a key reason why equipment providers are moving toward Linux-based solutions Creating platforms based on open standards: Ensures interoperability with third-party software, and Makes maintenance and application development much easier Therefore, utilizing the standard Linux kernel and adhering to key Linux standards is essential 85

86 Open Standards (2/2) There are many standards-related activities in the industry to define hardware and software high availability: PICMG Group (PCI Industrial Computer Manufacturers): defining standards for high availability hardware. PICMG specifications include CompactPCI, rackmount applications and PCI/ISA for passive backplane, standard format cards. Service Availability Forum (SA Forum): focusing on APIs for hardware platform management and for application failover in the application API. Open Source Development Lab (OSDL): defining specifications for Carrier Grade Linux. 86

87 Ongoing Efforts Open Source Development Lab Software Availability Forum Open Cluster Framework 87

88 Open Source Development Lab: Towards Carrier Grade Linux 88

89 OSDL Lab Facts OSDL founded to support Linux community Industry supporting developers Vendor neutral virtual lab Non-Profit First lab opened in Beaverton, OR, USA Second facility in Yokohama, Japan Carrier Grade Linux working group formed January, 2002 Data Centre Linux working group formed August,

90 OSDL Sponsors 90

91 What OSDL is and is not OSDL is: A group of companies and individuals working together Working within the current open source processes A resource for the Linux development community OSDL is not: A Linux distributor OSDL works with the distributors OSDL benefits the community An ISV OSDL only works on open source OSDL produces no products just specs & code 91

92 CGL Working Group A forum of industry leaders to support and accelerate the development of Linux functionality for telecommunication applications MEMBER COMPANIES 92

93 Working Group Organization Steering Committee Technical Board Marketing Board Specifications Proof of Concept Validation Collateral Events Tech Marketing Chairperson must represent an OSDL member company Team members must either: Represent an OSDL member company Represent an OSDL affiliated organization directly involved with the technology Be an OSDL employee Act as an individual with no employer recognition Committee and board members must either: Represent an OSDL member company Represent an OSDL affiliated organization directly involved with the technology Be an OSDL employee 93

94 CGL Architecture Applications HA Application Interfaces Middleware Components Java CORBA Databases... HA Platform Interfaces High Availability Components Standard Interfaces (LSB, POSIX...) Hardened Device Drivers High Availability Interfaces Linux OS with Carrier Grade Enhancements Hardware Configuration and Management Interfaces Service Interfaces Co-Processor Interfaces High Availability Hardware Platforms Scope of the Carrier Grade Linux Working Group Solution-specific components to be defined by vendors 94

95 Target CGL Applications Gateways Bridges between two different technologies or administrative domains (e.g., converting Time-Domain-Multiplexed networks to IP-based). A gateway maintains a large number of connections in real-time over a large number of interfaces without losing a frame or packet. Gateways are implemented on dedicated platforms with replicated (rather than clustered) systems for redundancy. Signalling Servers Handle call control, session control, radio recourses, handles routing and maintains the status of calls over the network. 10s of thousands of connections. Management servers Customer data - configuration, local, personal preferences, handle traditional network management operations as well as service and customer management. Far less stringent response time requirements. 95

96 CGL Process Gather Requirements Publish Specifications Identify and Evaluate OSS Projects Build Proof of Concept Implementations Construct Validation Suite Establish Certification Criteria Certify Distros / Implementations 96

97 Specifications Characteristics The specification promotes: Portability Ease of programming Software availability for telecom developers looking to implement Linux in an equipment design. The specification focuses on: High availability Performance Adopting technologies that promote high availability and service availability. 97

98 Specifications Characteristics The working group hopes to develop standards that make it easier to avoid problems in coding, and thus improve the reliability of systems. These standards will ensure that companies have choices for Carrier Grade Linux but that all of them will: Meet the specification Support a rich set of high availability features Have consistent interfaces and functionality 98

99 CGL Summary - To Date CGL Working Group Started January, 2002 Latest revision of official released specs and white papers released August 2002 CCL Technical Scope White Paper CGL Requirements Definition v1.1 CGL Architecture Specification v1.1 CGL Validation Framework v1.1 99

100 CGL Specs CGL specs are divided into Categories Standards Platform Availability Serviceability Tools Performance Security Scalability Assigned a Priority Priority 1 - Current requirements Priority 2 - For inclusion in the next version of specs Priority 3 - For future inclusion (no version implied) 100

101 CGL Specs Given a Version Assignment (1.x) Core: Required feature - must be present and functional in any implementation. Configurable: Must be available, end implementation has option to disable. Not Applicable: Features not requiring code to be satisfied. Given a Version Assignment (2.x) Default Setting: specifies whether the functional implementation is expected to be functionally enabled when a CGL system is configured. On -- the function must be functionally enabled by default. Off -- the function must be functionally disabled by default but must still be available to be enabled. Toggle Control: specifies whether the functional implementation is required to support being toggled between the enabled and disabled states without rebuilding. Yes -- the function must support the capability of being toggled (moved from disabled to enabled or enabled to disabled). No -- the function is not required to support the capability of being toggled. 101

102 Standards Requirements OSDL CGL does not create standards - but it specifies those required for compliance. Requirements that reference specifications controlled outside of the OSDL CGL workgroup that are important to carrier server systems. Compliance to standards a key to adoption of Open-Standards, off the shelf components. OSDL CGL v1.1 discusses 4 broad categories of standards Linux Standard Base Compliance POSIX interfaces IETF RFCs - involving IPv6 and protocols like SCTP Service Availability Forum Compliance 102

103 Standards Requirements - Priority 1 Linux Standard Base 1.2 Compliance The goal of the LSB is to develop and promote a set of standards that will increase compatibility among Linux distributions and enable software applications to run on any compliant system. Using the version 1.1 test suite - with certain defined and limited exceptions, acknowledging the variety of carrier platforms. POSIX Interface Compliance Following the Austin Group specs, a.k.a. IEEE Std Specific compliance: Timers, Signals, Message Queuing, Semaphores, Event Logging, and Threads SNMP - including agent versions 1, 2, and 3 IPv6, IPSECv6, MIPv6 + a long list of IETF RFCs 103

104 Platform Requirements Requirements that support interactions with the hardware platforms making up carrier server systems. "Platform" capabilities are vital building blocks, innately closer to the hardware than the "availability" and "serviceability" categories. OSDL CGL does not specify platforms and architectures, rather it specifies platform capabilities. Platform capabilities are not tied to a particular vendor's implementation. The specification may suggest model implementations which are tied to platforms - but do not require them. 104

105 Platform Requirements - Priority 1 Hot swap, hot insert, hot remove, hot device identity When devices are capable of being changed on a running system, OSDL CGL will support it. Remote boot, no console operation, diskless systems, boot cycle detection Ensure that remote systems can be managed without a human presence when trouble occurs. Proprietary (binary-only) modules will be permitted by default 105

106 Availability and Serviceability Requirements Availability: Requirements that support heightened availability of carrier server systems, such as improving the robustness of SW components or by supporting recovery from failure of HW or SW. Serviceability: Requirements that support servicing and managing HW and SW on carrier server systems. This a wide-ranging set requirements that, put together, help support the availability of applications and the operating system. 106

107 Availability Requirements - Priority 1 Watchdog timer helps detect OS failures. Application heartbeat helps detect application failures. RAID Support: Mirroring will be supported. Resilient file system support: support existing and future file systems that can quickly recover when required. Disk and volume management: File systems can span physical disks and be enlarged without un-mounting or rebooting. Multiple Ethernet NIC bonding and failover: Be able to aggregate bandwidth over multiple NICs and support failover of IP addresses from NIC to NIC. Hardened driver support: OSDL CGL will encourage the development of robust drivers. 107

108 Serviceability Requirements - Priority 1 Resource monitoring OSDL CGL will introduce specifications and frameworks to monitor a system's resources and their health. Such a comprehensive standard does not yet exist. Kernel dumps OSDL CGL will support producing and storing kernel dumps. (Kernel dumps are not a standard part of Linux ). Kernel message structuring Provide an event log mechanism that permits more detailed and consistent application error logging. Platform signal handler Provide a handler so that hardware errors are logged in the above event log. Remote access to event log Provide a mechanism to access the event log remotely. Dynamic debug/probe insertion Provide a specification to permit instrumentation of live applications and the kernel. 108

109 Tools Requirements Requirements that support auxiliary capabilities not directly involved in normal execution of carrier server systems, for example debuggers used to develop modules, drivers or applications. Provide capabilities to facilitate diagnosis User-level (gdb) debugging support for threads The GNU debugger needs enhancements for threaded processes - and are underway as part of the Native Posix Thread Model implementation. Kernel Debugging Linux does not natively support a kernel debugger; but it is a requirement for debugging CGL systems. 109

110 Performance Requirements Requirements that support performance levels necessary for the environments expected to be encountered by carrier server systems. Soft real time support: Specifies a target scheduling latency of 10ms or greater. Pre-emptible kernel: The kernel will provide support for pre-emption which reduces the latency of the kernel. It allows processes to be preempted even if in kernel mode. The resulting system response is greatly increased. Raid 0 support: Striping is required for enhanced disk throughput Application (pre) loading: Applications can be pre-loaded and pinned to prevent demand paging during execution. 110

111 Other Requirements Categories Scalability: Requirements that support vertical and horizontal scaling of carrier server systems such that addition of HW resources results in acceptable increases in capacity. Clustering: Requirements that support the use of multiple carrier server systems. This is to support higher levels of service availability through redundant resources and recovery capabilities, and to provide a horizontally-scaled environment supporting increased throughput. 111

112 CGL V2.0 Status V2.0 started October, major components: Clustering Specification Security Specification General System Requirements Public drafts of each will be released early and often Security and General OS Requirements released as public drafts in April Clustering public draft released May Final release of V2.0 specifications: October

113 Ericsson Contributions to Carrier Grade Linux Asynchronous Event Mechanism (AEM) Telecom IPC (TIPC) Distributed Security Infrastructure (DSI) All released to Open Source under the GPL License. 113

114 CGL Road Map (tentative) 03/10 v2.0 Specs release 04/1 v3 Specs kick-off in NY LWE 04/5 v3 Specs feature freeze 04/7 v3 Specs release 114

115 Vanilla Linux* vs. CGL * kernel.org 115

116 Linux Kernel 116

117 Implementations Status Started - A project exists, but is not considered a current candidate to fulfill a requirement as it is too new, inactive or inattentive (to OSDL CGL directions). Experimental - One or more projects exist and are viable candidate(s) but more work is needed to fully satisfy a requirement. Production - A project fully satisfies a requirement and is ready for deployment. 117

118 Integration with Mainstream Linux Takes time Some of the enhancements will be pushed to be integrated with the kernel 2.7 Others will follow in later kernel releases Meanwhile, all enhancements will be available from sourceforge or projects web sites 118

119 Software Availability Forum 119

120 Service Availability Forum (SAF) A consortium of communications industry leaders and startups dedicated to producing standards to enable the development of carrier-grade communications systems from off-the-shelf hardware platforms and middleware. Carrier grade system provide uninterrupted user access to the services it is designed to deliver, with no loss to the continuity of those services. To meet this goal, the SAF is developing two layers of standard, carrier-grade interfaces: an application interface and a platform interface. 120

121 Open Standards Context Applications Application Interface Platform Interface HA Middleware Components -Databases -Application Servers -Communication Protocols -Directory Service Availability Middleware Carrier Grade OS DMTF OMG SAF OSDL CG Linux High Availability Hardware Platforms PICMG 121

122 SA Forum Interfaces Application Interface Platform Interface 122

123 Application Interface The Application Interface provides access to a standard set of tools for application software to use to distribute processing over multiple computing elements, and to respond to failures of those elements without loss of service delivery or continuity to any user. 123

124 Application Interface Specification Objectives Promote the rapid development of applications that deliver highly dependable voice, data and multimedia services over fixed and wireless IP networks. Target adopters Application developers Value added component supplier Platform integrators Lower costs by enabling and ensuring Portability Choice of service availability middleware vendors Choice of adopter products and services 124

125 SAF Approach By using a standard interface to manage the physical platform, developers can write the service availability middleware independently of any particular hardware. This independence allows application developers to choose the best hardware platform and the best service availability middleware to fit their needs. 125

126 Partial Scope of Application Interface Specification Areas Release 1 Scope HA Framework Availability Management Health Monitoring Error reporting HA Services Checkpointing Events Messaging Membership Synchronisation Scope for Future Releases System Management Configuration Provisioning Administration Application Services Database Java interoperability CORBA interoperability External I/O: Signaling 126

127 Conclusion Final release 1 Spec Expected publication of release 1 Spec: Q2 CY03. Released spec will be open for comments and contributions. 127

128 Open Cluster Framework 128

129 OCF An open group of industry and users of clustering services who are defining some standard APIs for clustering. Most OCF APIs are generally intended to be usable by both high-performance and high-availability clustering platforms. A working group of the Free Standards Group. Two-pronged approach (Both proceed together) Define standard cluster APIs Create component-based reference implementation 129

130 Approach API Definition Select Areas of Interest Create Sub teams Define APIs Reach agreement Publish APIs for review Refine APIs Reference Implementation Create Plumbing/Infrastructure Coordinate with API definition Define Framework components Implement components Test result Provide as Open Source 130

131 Properties of the APIs Implementation Neutral (agnostic) Royalty-Free For OSS or proprietary software Creates opportunities for interoperability Focused on Linux, but not limited to Linux 131

132 APIs areas of Interest Event services Node services Resource Services Recovery Group Services Low Level Communication Services Fencing DLM External Interfaces (GUI, CLI, SNMP, logging, etc.) 132

133 OCF Cluster Conceptual Model A cluster is a collection of nodes (computers). Failures in the cluster occur asynchronously and are observed stochastically and independently. Each cluster is divided into zero or more partitions (by communication failures, etc.). Each active node belongs to exactly one partition at a time. One of these partitions may be named the primary partition. This partition is said to have quorum. 133

134 OCF Cluster Conceptual Model The method of determining membership is defined by the implementation not by OCF standards. The method of electing a primary partition is defined by the implementation not by OCF standards. The OCF generally defines the properties an implementation must have, not how they are achieved. 134

135 OCF Current Status Active participation by IBM, SuSE, OSDL, Sun, HP, Intel, Steeleye, Oracle, BigStorage, Linux-HA, University of Delft. Effort also endorsed by Free Standards Group, Conectiva, MSC Software, OSCAR, Red Hat, SGI, Bald Guy Software, UnitedLinux Now a working group of the Free Standards Groups Preliminary Draft APIs available for Event Services, Membership, and Resource agents. 135

136 OCG Plans Draft of current APIs by 1Q Next areas: recovery, group services, fencing. Refine and add APIs => official spec. Formal review period. Complete and release reference implementation. 136

137 Carrier Grade Linux Implementations 137

138 Who is building Carrier Grade Linux? Red Hat United Linux (SuSE, SCO, Turbo Linux, Conectiva) MontaVista 138

139 Red Hat Red Hat Linux Enterprise Server 139

140 SuSE/United Linux SuSE Linux Enterprise Server 140

141 United Linux / SuSE SLES A globally available Linux operating system for industry and enterprises, based on standards. Involvement of industry and customers in the development via United Linux Technical Advisory Board. Developed by the SCO Group, Conectiva, Turbo Linux, and SuSE Linux AG. 141

142 SuSE SLES 142

143 SuSE: The CGL Platform SuSE offers global support for development of applications and services. Version 8 of the powerful, reliable, and stable industry OS. Scalable for up to 32 processors (Intel). Proven HA solutions from third-party manufacturers (Steeleye). Long-term support by leading application developers, product life cycle 5+ years 143

144 SuSE & CGL Standards Requirements: Linux Standard Base POSIX Timer Interface POSIX Signal Interface POSIX Queue Interface POSIX Semaphore Interface Event Logging POSIX IPv6 RFCs compliance IPsecv6 RFCs compliance MIPv6 RFCs compliance SNMP support POSIX threads Standards compliance Platform Requirements: Hot Insert & Remove Remote boot support Boot cycle detection Loading or proprietary modules Diskless Systems Serial console connection 144

145 SuSE & CGL Availability Requirements: Watchdog Timer Interface Application heartbeat monitor Ethernet link aggregation Ethernet link failover RAID 1 support Resilient file system support Disk and volume management Serviceability Requirements: Kernel dump targets Kernel summary dump Kernel message structuring Dynamic debug/probe insertion Platform signal handler Remote access to event log Tools Requirements: User level (gdb) debug support for threads Kernel dump analysis Kernel debugger Performance Requirements: Soft-real time performance Kernel preemption RAID 0 support Application Loading Concurrent timers scaling behavior and reporting 145

146 Evolution of Distribution SuSE release cycle is months There will be Service Packs within one version, where new features (and bug- and security fixes for the kernel) are included. They mostly back port these features (from newer kernel-versions) to their stable and current kernel version to avoid at client side the need of exchanging the systems. As a new kernel version normally affects the whole system, e.g., moving to kernel 2.6, they will include those in new versions of SLES, while an upgrade from 2.4.n --> 2.4.n+1 may be released within a service pack, but only if the risk, the effort and the need justify that. 146

147 MontaVista Linux Carrier Grade Edition 147

148 Monta Vista Linux Carrier Grade Edition (CGE) First Linux vendor to distribute an industry standardsbased COTS Carrier Grade Linux. CGE Release 3.0 based on Linux Kernel Fully complies with Open Source Development Labs Carrier Grade Linux Specification Release 1.1. MontaVista is shipping all priority 1 features of OSDL and many priority

149 MontaVista CGE Architecture HW Mgmt Fail-over Carrier Applications Soft Switch Java High Availability Services Serviceability Enhancements High Availability Enhancements Performance Enhancements Middleware & Application Services CORBA Databases MontaVista Linux Carrier Grade Kernel POSIX/Hardened Drivers/Real Time Pre-emption IA-32 Based Reference Hardware Protocol 200+ Networking & Application Packages Tools Target Tools Runtime App Patcher Field-safe App Debugger Enhanced kernel dump App Tools Kdevelop IDE gdb Gcc KDB KGDB Trace Debug Config Tools I/O Latency Target Config Lib Optimization 149

150 HA Hardware Support Key Standards Support PICMG 2.12 PICMG 2.16 ATCA (May 2003) Hardware Redundancy I/O Processing Hot Swap, Hot Insert Redundant Storage Redundant Networking CPU Redundant System Slot Remote boot across LAN/WAN 150

151 POSIX Compatible Interfaces Microsecond Timers MontaVista ported to 2.4 Accepted to 2.5 kernel POSIX event log signaling and message queues POSIX threads NGPT, migrating to NPTL 151

152 HA Hardware Support Hardened device driver support Support bonding of multiple Ethernet NICs RAID 1 support and user-level volume mgmt utilities 1 Boot cycle detection Hyper-threading Support IPMI Driver Support 152

153 Performance Features Journaling filesystem(s) O(1) real-time scheduler with CPU affinity Preemptive real-time kernel Scalability improvements kernel locks, locking primitives RAID 0 striping Application loading/locking into memory Fast boot up time Forced unmount of files systems 153

154 MontaVista Real Time Linux Real Time Pre-emptible Kernel MontaVista developed technology Added to the 2.5 kernel Provides a stable standard real-time solution Preserves Linux programming model User-level applications and Standard APIs Fixed priority scheduler Enhanced version of Order-One or (0)1 priority scheduler Added CPU affinity support 154

155 Serviceability Features Kernel resource monitoring framework. System event log with remote access capability. Produce and store kernel dumps across LAN, in-memory, disk. Multithreaded core dump. Dynamic debugging and on-line probe insertion. 155

156 Open System Lab Experience and Contributions Towards Carrier Grade Linux 156

157 Hardware Compact PCI design -48V Central Office Powered NEBS Compliant Ready 16 P3 500MHz 512 Mbytes RAM 6 Ethernet Ports / Processor 8 SCSI Disk banks (3x18GBytes) Fully Redundant and hot swap Off-the-shelf 1U Celeron/Pentium III 256/512 MB RAM 20/40 GB IDE HD Floppy/CD-ROM 2 fast Ethernet ports 2 USB ports 157

158 ARIES 2000 Focus Areas: HA Clusters as Internet Servers using TelORB and Linux 2000 HA Linux Cluster Linux Diskless Booting Linux NFS Redundancy Linux Ethernet Redundancy - Find and prototype the necessary technology to prove the feasibility of an Internet Server that has the guaranteed availability, response time and scalability using both TelORB and Linux. 158

159 ARIES 2001 Focus Areas: - Alternate Scalability Technologies - Cluster Dimensioning - Load Balancing - IPv6 - Security 2001 Enhance clustering capabilities of TelORB and Linux clusters as Mobile Internet Servers 159

160 ARIES 2002/2003 Focus Areas: - IPv6 - Asynchronous Event Mechanism (AEM) - Security on clustered servers (DSI) - Carrier Grade Linux - TIPC port for Linux Convergence of specific elements of the different platform technologies to be used in Server and Router nodes. - Focus on the clustering capabilities of different OS supporting highly available secured application in an all IP world. 160

161 Ericsson Contributions to Carrier Grade Linux Asynchronous Event Mechanism (AEM) Distributed Security Infrastructure (DSI) Telecom IPC (TIPC) All released to Open Source under the GPL License. 161

162 Asynchronous Event Mechanism (AEM) Advantages of Carrier-Grade Linux: Openness (open source software, third party software), Stability on a wide variety of architectures (UP, SMP, HP, NUMA, ) Capability to run in different environments (Large-scale distributed, embedded, real-time, ) Carrier-Grade Environments (must be supported) Media Gateway and MG Controller, Authentication controller, SS7 gateway, (IP) Application servers (transactional, streaming), IP gateway, Other cross-network gateways Imply different software requirements 162