NWD IP8800/S3640. IP8800/S3640 Software Manual. OpenFlow Feature Guide (Version 11.1 Compatible) ISSUE DATE: MAY, 2010 (FIRST EDITION)

WD-105490-001 IP8800/S3640 IP8800/S3640 Software Manual OpenFlow Feature Guide (Version 11.1 Compatible) ISSUE DATE: MA, 2010 (FIRST EDITIO)

WD-105490-001 Applicable products This manual describes OpenFlow features of the basic software for model IP8800/S3640. For the software functions, those supported by the basic software OS-F3L, OS-F3A are described. Please read the references listed below. References For the common contents with the basic software, see the following Manual; IP8800/S3600 Software Manual This manual describes software functions specific to IP8800/S3640 used in OpenFlow. This manual has the explanations on OpenFlow features, and non-openflow features/commands that are different from the basic software of IP8800/S3640. Caution when exporting The necessary procedures are to be adopted when exporting this product after first confirming the regulations of the Foreign Exchange and Foreign Trade Law, U.S. export control related regulations, etc. If any question remains, please consult with our sales department. Trademarks sflow is a registered trademark of InMon Corp. in the U.S. and other countries. Ethernet is a product name of Xerox Corp. in the U.S. Other company names and product names are trademarks or registered trademarks of their respective companies. If any question remains, please consult with our sales department. Thoroughly read and store this manual. Read and thoroughly understand safety-related explanations before using this product. Keep this manual close at hand for easy reference. ote The contents of this manual may be modified at any time for improvement without notice. Issue date Issue date: May, 2010 (First edition) WD-105490-001 Copyright Copyright (c) 2010 by EC Corporation. All rights reserved.

WD-105490-001 Introduction Intended products and software versions This manual describes OpenFlow features of the basic software for model IP8800/S3640. Please read the manual carefully, and thoroughly understand the instructions and cautions contained herein and references for the common descriptions with the basic software before operating the device. Keep the manual close at hand for easy reference when necessary. Intended users Applicable users are system managers who develop and operate network systems using OpenFlow. In addition, an understanding of the following is assumed. Basic knowledge of network system management Manual structure This manual consists of the following chapters. Chapter 1 Configuration Guide Provides the explanations of OpenFlow functions, the basic setting procedures of the configurations, and the procedures of checking the functions by using operation commands. Chapter 2 Configuration Command Reference Provides the OpenFlow configuration command entry syntax, the detailed information on parameters, and various suggestions and precautions. Chapter 3 Operation Command Reference Provides the explanations of OpenFlow features, the basic setting procedures of the configurations, and the procedures of checking the features by using operation commands. Chapter 4 Message Log Reference Provides the explanations of the log messages input by OpenFlow features. Chapter 5 Troubleshooting Provides the troubleshooting for OpenFlow features. Chapter 6 Appendix Provides the explanations for using the existing commands of IP8800/S3640 model as well. Terms in this manual The term indicates the following meaning. Legacy: Layer 2/Layer 3 switch feature Supported OS OS Version Applicable models OS-F3L 11.1.C.A and IP8800/S3640 OS-F3A later Remarks Applicable models IP8800/S3640-24T IP8800/S3640-24TW IP8800/S3640-24T2XW IP8800/S3640-24SW IP8800/S3640-24S2XW IP8800/S3640-48TW IP8800/S3640-48T2XW Change history Ver. Chapter Description 11.1.C.Ab First edition published

WD-105490-001 Contents Chapter 1 Configuration Guide... 1 1.1 OpenFlow Feature Overview...1 1.1.1 OpenFlow Overview... 1 1.1.2 Structure of Programmable Flow Switch... 2 1.1.3 OpenFlow Features of PFS... 3 1.2 Descriptions of PFS...4 1.2.1 OpenFlow Feature Overview... 5 1.2.2 OpenFlow Switch Instance Descriptions... 6 1.2.3 Search Key... 9 1.2.4 Match Conditions... 9 1.2.5 Search Order... 9 1.2.6 Action... 10 1.2.7 Statistics... 14 1.2.8 Packet In Feature... 14 1.2.9 Packet Out Feature... 15 1.3 OpenFlow Feature Action Overview...15 1.3.1 OpenFlow Protocol Support Messages... 15 1.3.2 Secure Channel Descriptions... 17 1.3.3 Flow Table Control Descriptions... 21 1.3.4 Emergency Mode Descriptions... 22 1.3.5 OpenFlow Protocol Control Port Overview... 23 1.3.6 OpenFlow Protocol Features otification Descriptions... 26 1.3.7 OpenFlow Protocol Configuration Descriptions... 26 1.3.8 Statistics Descriptions... 27 1.4 Support Specification...30 1.4.1 Accommodating Conditions... 30 1.4.2 Coexistence With Legacy Switch... 33 1.4.3 otes on Using the Existing Commands... 41 1.4.4 otes on Operation... 41 1.5 Operation Procedures...44 1.5.1 Outline... 44 1.5.2 Operational Flow... 44 1.6 Configuration of OpenFlow Features...45 1.6.1 Configuration of OpenFlow Features... 45 1.6.2 OpenFlow Switch Instance Setting... 50 1.6.3 OpenFlow Switch Instance Common Setting... 50 1.6.4 Example of RSI Mode Setting... 51 1.6.5 Parameters for RSI mode... 52 1.6.6 Example of VSI Mode Setting... 57 1.6.7 Example of VSI Mode Setting... 58 1.7 Operation of OpenFlow Features...63 1.7.1 List of Operation Commands... 63 1.7.2 Checking OpenFlow Information... 64 1.7.3 Checking Flow Table Information... 68 1.7.4 Checking Statistics of OpenFlow Protocol Messages... 70 1.7.5 Real-time Display of Send/Receive Packets Between OFC and PFS... 72 Chapter 2 Configuration Command Reference... 73 openflow-interface (<interface id list> is specified)...74 openflow-interface (<channel group number list> is specified)...76 openflow-vlan...78 emergency-mode disable...79 wildcard-hwaccel...80 l2-inband-secure-channel...81 mac-learning disable...83 outbound...84 port-modify-access...86 port-modify-trunk...87 flow-statistics...88 i

WD-105490-001 flow-statistics-mode...89 openflow...90 dpid...91 enable...92 controller...93 connect timeout...95 connect timeout retry...96 echo-request interval...97 echo-reply timeout...98 maxflow hardware...99 maxflow software... 101 mishit-action... 102 flow detection mode... 103 flow detection out mode... 104 Chapter 3 Operation Command Reference... 105 show openflow... 106 show openflow table... 119 show openflow statistics... 128 show openflow controller-session... 136 clear openflow statistics... 144 clear openflow table... 146 restart openflow... 148 dump protocols openflow... 150 show system... 152 show channel-group... 159 show channel-group statistics... 167 Chapter 4 Message Log Reference... 170 Chapter 5 Troubleshooting... 174 Chapter 6 Appendix... 175 Appendix A Availability List for Concurrent Use of the Existing Commands When OpenFlow Feature Is Enabled... 175 Appendix B Acknowledgment... 197 ii

WD-105490-001 Chapter 1 Configuration Guide 1.1 OpenFlow Feature Overview 1.1.1 OpenFlow Overview In OpenFlow architecture, OpenFlow Controller (OFC) sets the contents of a flow table on an OpenFlow Switch (OFS), and processes packets. This system identifies packets as a flow by means of combinations of 12 fields that are used by IP packets, Ethernet frames, and so on. A set of a flow identification condition and an action towards a flow is called flow entry, and the database that stores the entries is called flow table. For the details of the searchable fields, see 1.2.3 Search key. The statistics of each flow entry (the number of hit packets or octets) is held and can be checked with the OpenFlow statistics messages. An OpenFlow Switch identifies a flow by IP or Ethernet; packets are not transferred according to routing tables or FDB, but the actions can be specified per flow entry. Actions include Transmitting packets by specifying output interface. Rewriting MAC address, VLA tag, or IP address. Rewriting IEEE802.1p priorities or DSCP value in a VLA tag. ou can specify more than one output interface. ou can also transmit packets from this system to OpenFlow Controller, and have the Controller decide what action to take depending on the situation, and register flow entries to the flow table of the OpenFlow Switch. Figure 1-1 OpenFlow Overview Source MAC Address 02:00:00:10:00:01 Controller Flow Table 1 2 3 Switch Flow: Ingress Port 1 Source MAC Address 02:00:00:10:00:01 Action: Egress Port 3 Openflow can configure a very flexible network that IP or Ethernet cannot realize. Using OpenFlow features has the following advantages: (1) Virtualization of networks Even if packets from more than one private network have the same IP address structure, they can be transferred to different destinations; as OpenFlow decides transfer destination not according to the routing tables. Therefore, inside of OpenFlow network can be handled as separate networks virtually. (2) Route distribution ou can connect two OpenFlow Switches with more than one cable to create more than one link, and control which link to use on a flow basis expressly. Therefore, the routes are distributed properly to meet the needs of applications, 1

WD-105490-001 and you can use the most of the bandwidths. (3) Visualization of networks As the system holds the statistics for each flow entry, and the OpenFlow Controller can collect the information, you can check where in the network is congested on a flow basis. 1.1.2 Structure of Programmable Flow Switch Programmable Flow Switch (PFS) is the name for the entire switch system where OpenFlow Switch features are integrated onto IP8800/S3640 platform. ou can create an OpenFlow Switch instance on this system. This instance is a virtual switch that operates as an OpenFlow Switch. Physical ports and channel groups belonging to OpenFlow switch instances are called OpenFlow Interfaces. When a packet is input from an OpenFlow Interface, the system searches the flow table of the corresponding OpenFlow Switch Instance (see 1.2.2 OpenFlow Switch Instance Descriptions ). When there is a flow entry that is hit in the search, the statistics of the flow is collected and the packet is handled according to the specified action. Packets that are input from non-openflow interfaces are handled by legacy switch. Legacy switch is an existing function of IP8800/S3640, and supports L2/L3 switch functions. For the legacy switch functions that can coexist with OpenFlow, see 1.4.2 Coexistence With Legacy Switch. Figure 1-2 Structure of PFS OpenFlow PFS Flow Table OpenFlow Switch Instance Interface 1 3 RSI 2 Legacy Port MAC Address Table Legacy Switch 1 2 3 Input from OpenFlow interface. Input from other than OpenFlow interface. Input from OpenFlow interface and specify a process by legacy switch in flow table. 2

WD-105490-001 1.1.3 OpenFlow Features of PFS 1.1.3.1 OpenFlow Switch Instance This system implements RSI (Real Switch Instance) mode where OpenFlow Switch instances can be configured by specifying the physical ports or channel groups to be used as OpenFlow Interfaces, and VSI (Virtual Switch Instance) mode where VLAs are configured as OpenFlow Switch instances. RSI and VSI cannot coexist. ou can create one OpenFlow Switch instance in RSI mode, and 16 instances in VSI mode. For details, see 1.2.2 OpenFlow Switch Instance Descriptions. 1.1.3.2 OpenFlow Interface ou can specify ports or channel groups as OpenFlow interfaces in RSI mode. ou can also directly specify a port belonging to a channel group. In RSI mode, you can exclude specific VLA packets from the controlled object of OpenFlow entry in the ports and channel groups specified by the l2-inband-secure-channel command. Excluded ports and channel groups operate by legacy switch functions. In VSI mode, all the ports and channel groups included in the VLA that is specified by the openflow-vlan command operate as OpenFlow interfaces. Only the ports belonging to a channel group are not included. 1.1.3.3 Controller Connection Feature ou can specify up to four OpenFlow Controllers of the connected party per OpenFlow switch instance in this system. Connection between OpenFlow Controller and switch instance is called Secure Channel. OpenFlow messages are transmitted and received through Secure Channel. This system supports the Secure Channel connectivity check function. Once the connection between the system and the Controller is terminated, the system tries reconnecting the Controller. If it fails, the system tries connecting other Controllers. For details, see 1.3.2 Secure Channel Descriptions. 1.1.3.4 Flow Statistics Mode This system can collect the statistics of the number of received packets and octets per flow. ou can specify collecting the statistics of packets, octets, or both per system. For details, see 1.3.8 Statistics Descriptions. 1.1.3.5 Flow Table This system has three flow tables; basic flow table, extended flow table and emergency flow table. In the flow entries on the basic flow table, if you specify only the actions that are supported by the hardware, the packets that hit the flow entry are processed rapidly by the hardware. All the packets that hit the flow entry on the extended flow table are processed by the software. For details, see 1.2.2.3 Flow Table. 1.1.3.6 Port State Control Feature ou can change the settings of OpenFlow interfaces such as link up/down by using OpenFlow messages. In VSI mode, you can also set to allow each VSI to control port up/down. For details, see 1.3.5.1 Target Interface. 1.1.3.7 Emergency Mode Support This system supports the Emergency mode that is defined at OpenFlow Switch Specification. If connecting to all Controllers fails, the system overwrites the flow entries that are registered in the Emergency flow table to the basic flow table. This feature can be disabled by the emergency mode command. For details, see 1.3.4 Emergency Mode Descriptions. 1.1.3.8 Packet Control Feature at Flow Table Search Mishit If packets that are input from an OpenFlow interface do not hit any of the flow entries in the corresponding OpenFlow Switch instance, you can choose one of the actions from the following two: (1) Send the packet to OpenFlow Controller by using Packet In messages. This action is executed when controller is set by the mishit-action command. This action is defined in OpenFlow Switch Specification; the default setting of this system. 3

WD-105490-001 (2) Packets are handled by legacy switch function. This action is executed when normal is set by the mishit-action command. 1.1.3.9 MAC Address Learning Control Feature ou can specify either to execute/not to execute MAC address learning per OpenFlow Switch instance by the mac-learning command. 1.1.3.10 VSI Policing at Output Interface In VSI mode, the system supports policing feature that limits transmitting bandwidth per VSI at trunk port by using the outbound command. 1.1.3.11 Coexistence With Legacy Switch Packets that are input from non-openflow interfaces are handled by L2/L3 switch features of IP8800/S3640. 1.2 Descriptions of PFS OpenFlow Controller (OFC) and OpenFlow Switch (OFS) are integrated to operate in OpenFlow. On Programmable Flow Switch (PFS), OpenFlow Switch instance with OpenFlow Switch features operate. An OpenFlow switch instance has a flow table that is a database to decide what processing to execute. Packets that are input to the system are processed and output according to the flow entries in the flow table. OFC and OpenFlow Switch instance are connected through Secure Channel, and OFC can configure/change the contents of the flow table. Figure 1-3 Conceptual Diagram of OpenFlow OpenFlowController SecureChannel (TCP) Programmable FlowSwitch Data Flow is handled as a flow. Flow Table Flow C Flow B Flow A 4

WD-105490-001 1.2.1 OpenFlow Feature Overview Between OFC and OpenFlow Switch instances OpenFlow protocol messages (see 1.3.1 OpenFlow Protocol Support Messages ) are used to control. The following figures show a structure from establishing Secure Channel to transferring packets. For details on exchanged messages and sequences between both systems, see 1.3.1 OpenFlow Protocol Support Messages. (1) Establishing Secure Channel Figure 1-4 Secure Channel Establishing Behavior OFC SecureChannel PFS OpenFlow is enabled. Once OpenFlow features are enabled, OpenFlow Switch instance and Secure Channel are connected. (2) Flow Transfer Processing and Control from OFC After establishing Secure Channel, the following (2-1) and (2-2) are executed. (2-1) Flow Transfer Processing OFC Figure 1-5 Flow Transfer Processing OFC SecureChannel SecureChannel is disconnected. PFS PFS Input Flow Flow Table Output Flow Input Flow Flow Table Output Flow Flow Entry Flow Entry Once a packet is input to PFS, the flow table in the corresponding OpenFlow switch instance is searched. If there is a flow entry that matches the packet, the packet is processed according to the defined action. ote that even while Secure Channel is being disconnected, if a flow entry exists, the same action is to be executed. (See Figure 1-5 Flow Transfer Processing.) 5

WD-105490-001 (2-2) Control from OFC Figure 1-6 Control From OFC OFC OpenFlow Protocol Message SecureChannel PFS Flow Table Flow Entry OpenFlow Switch OFC uses OpenFlow protocol message FlowMod via Secure Channel to register, change, or delete a flow entry. It can also manage ports and configurations, or collects statistics. For details, see 1.3.1 OpenFlow Protocol Support Messages. 1.2.2 OpenFlow Switch Instance Descriptions This system can create two types of OpenFlow Switch instances; RSI (Real Switch Instance) and VSI (Virtual Switch Instance). The following figures show their conceptual diagrams. Figure 1-7 Conceptual Diagram of RSI Figure 1-8 Conceptual Diagram of VSI Controller Controller Controller Controller PFS OpenFlow Features PFS OpenFlow Switch OpenFlow Switch OpenFlow Switch (Virtual Switch Instance) Flow Table Flow Table Flow Table Flow Table Flow Table Flow Table 1.2.2.1 Real Switch Instance (RSI) RSI mode is a network model where the legacy switch is operating as the basis, and OpenFlow features are placed as an addition in order to control the network flows. OpenFlow features operate on the same network where the legacy switch is operating; OpenFlow network and the existing network are overlaid for operation. ou can specify ports or channel groups as OpenFlow interfaces in RSI mode. ou can also directly specify a port belonging to a channel group. Packets that are input from OpenFlow interfaces are handled by RSI, and the packets from non-openflow interfaces are handled by legacy switch. VLA tunneling feature cannot be used in RSI mode. 6

WD-105490-001 1.2.2.2 Virtual Switch Instance (VSI) VSI mode is a network model where the legacy switch and OpenFlow features are logically separated; because in VSI mode, network is physically shared, but the legacy switch is used in the different VLA from OpenFlow. Flow controlling of OpenFlow network is performed by a group (one system or a clustered Controller group) per independent OpenFlow network. Up to 16 VSIs can operate on PFS. Even if only one VSI is operating, the network model is different from RSI. In VSI mode, all the ports and channel groups included in the VLA that is specified by the openflow-vlan command operate as OpenFlow interfaces. Only the ports belonging to a channel group are not included. The followings are identified as OpenFlow interfaces: Combination of VLA and an ingress port Combination of VLA and channel group In VSI mode, OpenFlow interfaces are closed for a certain VLA; basically they cannot communicate with other VLAs. It is possible to communicate with other VLAs only when using L3 transferring of legacy switch features. VSI only handles the packets received at the specified VLA. Packets that are received at the VLA that is not set to any of the VSIs are handled by legacy switch. 1.2.2.3 Flow Table A flow table is configured with the flow entries that have Table 1-1 Configuration Elements of Flow Entry. Table 1-1 Configuration Elements of Flow Entry # Element Description 1 Search key Key information to search a flow 2 Action Action to be taken to flow (multiple assignments possible) 3 Statistics Statistics per flow (the number of packets/octets) 4 Flow cookie 64bit identifier held by each flow entry Once a packet is input to the system, the flow table is searched. If there is a flow entry that matches the packet, the packet is processed according to the defined action. For the actions when no flow entry is hit in the search, see 1.1.3.8 Packet Control Feature at Flow Table Search Mishit. Figure 1-9 Packet Processing of OpenFlow Flow Input Search Key Actions Flow Output, etc. Statistics Flow Cookie The following figure shows the flow table configuration of this system. There are three types of flow tables; the basic flow table that manages flow entries registered to the hardware, the extended flow table that searches on the software, and the emergency flow table that manages the entries used for emergency actions. When normal is set by the mishit-action command, the extended flow table cannot be used. When Emergency mode is disabled by the emergency-mode command, the Emergency flow table cannot be used. Figure 1-10 Configuration Diagram of Flow Table shows the flow table configuration and Flow entry operations. 7

WD-105490-001 Figure 1-10 Configuration Diagram of Flow Table PFS OFC Statistics FlowMod Emergency Flow Table OFS Switch Instance Copy Basic Flow Table r/w Hardware Flow Table FlowRemoved Extended Flow Table Flow entries in the basic flow table always have higher priorities in search than those in the extended flow table. The system sorts flow entries to the basic flow table and the extended flow table according to the priorities in search when OFC registers flow entries by FlowMod messages. The threshold of priorities in search that serves as the standard in sorting can be set by the wildcard-hwaccel command per OpenFlow switch instance. Figure 1-11 Order of Search in the Basic Flow Table and the Extended Flow Table shows the search order of flow entries, and Table 1-2 Features of the Basic Flow Table and the Extended Flow Table shows the feature comparison of the basic flow table and the extended flow table. The emergency flow table is for emergency mode operations; this table is not used for normal transfers. For Emergency mode, see 1.3.4 Emergency Mode Descriptions. Table 1-2 Features of the Basic Flow Table and the Extended Flow Table Search order Search speed Hardware Flow entry ame Transfer method Basic Flow Table First High Consumes Hardware (*1) or Software Extended flow table Later Low Do not consume Software (*1): Transfers only when actions that can be transferred by the hardware are set. Figure 1-11 Order of Search in the Basic Flow Table and the Extended Flow Table Basic Flow Table Search Sequence Exact Match priority 65535 priority 65534 priority 65533 : priority X+1 priority X priority X-1 priority X-2 : Priority 0 Extended Flow Table Flow Entry A Flow Entry B Flow Entry C : Multiple Flow Entries are entered into the same priority. Search Sequence 8

WD-105490-001 When registering a flow entry, the contents of the message is checked. If there is no problem, the entry is registered to the flow table. If there is a problem, registering of the flow entry is denied, and an error message is sent out. 1.2.3 Search Key Table 1-3 Flow Search Keys shows the detailed key information for flow search. Table 1-3 Flow Search Keys Category Physical interface Ethernet ARP IPv4 ICMP TCP/UDP Field Ingress port number Destination MAC address, Source MAC address, VLA ID, User Priority of IEEE802.1D, and Ethernet-type Destination MAC address, Source MAC address, and ARP opcode Destination MAC address, Source MAC address, Protocol, and ToS (DSCP) value ICMP Type, ICMP Code Destination port number and Source port number 1.2.4 Match Conditions OpenFlow flow entries are categorized into the following two types. Wildcard match flow entry any is specified to all or a part of the search key fields shown in Table 1-3 Flow Search Keys. Exact match flow entry any is not specified at the search key fields shown in Table 1-3 Flow Search Keys. 1.2.5 Search Order As shown in Table 1-3 Flow Search Keys, match conditions of search keys include Protocol number, Source IP address and Destination IP address. Search is executed by specifying one or more than one condition. The following figure shows a structure of deciding what process to execute when inputting packets. Figure 1-12 Order of Search in the Flow Table shows the order of searching flow entries. 9

WD-105490-001 Figure 1-12 Order of Search in the Flow Table Flow Table Flow entry specified as Exact Flow entry specified as Exact Flow entry specified as Exact Flow entry specified as Exact Flow entry specified as Exact Search Sequence Flow entry specified as Wildcard Flow entry specified as Wildcard Flow entry specified as Wildcard Flow entry specified as Wildcard Flow entry specified as Wildcard Flow entry specified as Wildcard Flow entry specified as Wildcard Wildcard match flow entry is searched from the one with the highest priority in search. Search priority has a range from 0 (minimum) to 65535 (maximum). If there is more than one Wildcard match flow entry with the same priority in search, the search is executed in the order of registration. When there is a flow entry that is hit in the search, it is handled according to the action specified in the flow entry. When there is no flow entry that is hit in the search (mishit), it is handled according to the settings of the command mishit-action. 1.2.6 Action Actions are operations to be executed to flows when a flow matches one of the search keys shown in Table 1-3 Flow Search Keys. ou can specify more than one action to a single flow. If there is no transfer actions (OUTPUT and EQUEUE), the action is considered to be discard operation. Table 1-4 Actions That Can Be Set to Flow Entries Action name Contents of action Operational packets OUTPUT Output to the specified port. All EQUEUE Specify destination port number and queue number, and outputs to the specified port with the specified priority. All SET_VLA_VID Change/Add VLA IDs. (*1) All SET_VLA_PCP Change user priority in VLA tag. (*2) Tagged packet STRIP_VLA Delete VLA tag. All SET_DL_SRC Change the source MAC address. All SET_DL_DST Change the destination MAC address. All SET_W_SRC Change the source IP address. IPv4 Packet SET_W_DST Change the destination IP address. IPv4 Packet SET_W_TOS Change IP ToS (DSCP). IPv4 Packet SET_TP_SRC Change source L4 port. TCP or UDP packet SET_TP_DST Change destination L4 port. TCP or UDP packet (*1): See (*1) VLA Tag translation (SET_VLA_VID / STRIP_VLA) (*2): Shows priority of IEEE802.1D. 10

WD-105490-001 The following section explains the actions shown in Table 1-4 Actions That Can Be Set to Flow Entries. (1) Specifying the egress port (OUTPUT) OUTPUT action can specify the ports shown in Table 1-5 Ports Available as Egress Port. (2) Specifying the egress port and priority (EQUEUE) EQUEUE action can specify the ports shown in Table 1-5 Ports Available as Egress Port as an egress port. It can also specify queue number and control priorities. Table 1-5 Ports Available as Egress Port Port ame Description H/W(*3) S/W(*4) Port o.(*1) Outputs to the port whose port number is specified. I_PORT Outputs to the port where a packet is received. ORMAL Transfers packets using legacy features. FLOOD Outputs to all ports excluding the one that received the packet and the one whose status is blocking on OpenFlow. ALL Outputs to all ports where the packet is received. COTROLLER Transfers packets to OFC via Secure Channel. LOCAL Transfers packets to PFS s own network stack. (*2) (*2) (*1): See 1.3.5.1 1.3.5.1Target Interface. (*2): Receives IPv4, IPv6, or ARP packets only. (*3): Hardware transfer. (*4): Software transfer. (3) VLA tag translation (SET_VLA_VID / STRIP_VLA) SET_VLA_VID action changes or adds VLA tags. The system considers VLA priority of untagged packets to be 3, and adds the VLA tag of the specified VLA ID. STRIP_VLA action deletes VLA tags. STRIP_VLA action can delete only one VLA tag. It is not possible to delete more than one VLA tag by specifying more than one STRIP_VLA action. The following section shows how to register a flow entry that has VLA tag translation action. In RSI mode, as shown in Figure 1-13 VLA Tag Translation Action, if the port 0/1 of the legacy switch of IP8800/S3640 is set as the access port of VLA α, and port 0/5 as the trunk port for VLA α and β, the determination is performed as shown in Table 1-6 Determining the Registration of Flow Entry by VLA Tag Translation Action (RSI). Figure 1-13 VLA Tag Translation Action This system VLA α VLA β 0/1 0/5 untagged (Access Port) tagged (Trunk Port) 11

WD-105490-001 Table 1-6 Determining the Registration of Flow Entry by VLA Tag Translation Action (RSI) Search key field Ingress port number 0/1 (Access port) 0/5 (Trunk port) OUTPUT action VLA tag translation action Determination Transmission packet 0/5 (Trunk port) 0/1 (Access port) one (*1) Untagged packet STRIP_VLA (*1) Untagged packet SET_VLA_VID α or β Tagged packet Other than - the above one (*2) Untagged packet STRIP_VLA Untagged packet SET_VLA_VID - (*1): When native VLA is set to OUTPUT action port. (*2): When native VLA is set to ingress port. The following section explains Table 1-6 Determining the Registration of Flow Entry by VLA Tag Translation Action (RSI). 1. When the ingress port number in the search key field is 0/1, and the physical port of the OUTPUT action is 0/5: When VLA tag translation action is not specified, and VLA tag translation action is STRIP_VLA, if native VLA is set to the physical port 0/5, the egress port, mismatch is not caused between the flow entry that is registered from OFC and the configuration information that is set by the legacy features of IP8800/S3640; the flow entry is registered to the flow table. If native VLA is not set, the determination result of registration is an error. In this case, registering of the flow entry is denied, and an error message is returned. When VLA tag translation action is SET_VLA_VID, and the value is either α or β, VLA tag translation action is STRIP_VLA, mismatch is not caused between the flow entry that is registered from OFC and the configuration information that is set by the legacy features of IP8800/S3640; the flow entry is registered to the flow table. When VLA tag translation action is SET_VLA_VID, and the value is either α or β, mismatch is caused between the flow entry that is registered from OFC and the configuration information that is set by the legacy features of IP8800/S3640; the determination result of registration is an error. Thus registering of the flow entry is denied, and an error message is returned. 2. When the ingress port number in the search key field is 0/5, and the physical port of the OUTPUT action is 0/1: When VLA tag translation action is not specified, if native VLA is set to the ingress port 0/5, mismatch is not caused between the flow entry that is registered from OFC and the configuration information that is set by the legacy features of IP8800/S3640; the flow entry is registered to the flow table. If native VLA is not set, the determination result of registration is an error. In this case, registering of the flow entry is denied, and an error message is returned. When VLA tag translation action is STRIP_VLA, mismatch is not caused between the flow entry that is registered from OFC and the configuration information that is set by the legacy features of IP8800/S3640; the flow entry is registered to the flow table. When VLA tag translation action is SET_VLA_VID, mismatch is caused between the flow entry that is registered from OFC and the configuration information that is set by the legacy features of IP8800/S3640; the determination result of registration is an error. Thus registering of the flow entry is denied, and an error message is returned. To transmit a tagged packet that has the same VLA tag with the received packet, or to transmit an untagged packet after receiving an untagged packet, it is possible to transmit a packet without specifying VLA tag translation actions. When receiving a FlowMod message where a condition that cannot be set in this system is specified, the system does not add/change the flow entry, and sends an error message to OFC. The following section shows how to determine whether a flow entry that has an invalid combination of conditions should be registered or not. 12

WD-105490-001 Table 1-7 Determining the Registration of Flow Entry by VLA ID of Flow Identification Condition Search key field (Ingress port number) 0/1 (Access port) 0/5 (Trunk port) Search key field (VLA ID) Determination Receivable packets any (*1) Untagged packet 0xffff (untagged) Untagged packet Other than the above - any (*1) All packets 0xffff (untagged) (*2) Untagged packet α or β (*2) Tagged packet Other than the above - (*1): When all the ingress ports and egress ports share the same VLA ID. (*2): When native VLA is set to the ingress port. It is possible to transfer packets to the VLA ID that is different from the one with the received packet if the VLA is the same with the configuration information set by the legacy features of IP8800/S3640. Table 1-8 shows the actions to take in VSI mode. Table 1-8 Determining the Registration of Flow Entry of VLA Tag Translation Action VLA mode Port VLA VLA Tunneling mode VLA port type (Output interface) Transmission packet type VLAs targeted for operations by VSI VLA actions that can be specified Access port o controlled objects o controlled objects one (Unable to specify)(*1) Trunk port Untagged packet o controlled objects one (Unable to specify)(*1) Tagged packet o controlled objects one (Unable to specify)(*1) Tunneling port Untagged packet ative VLA STRIP_VLA Tagged packet Tag VLA SET_VLA_VID Trunk port Untagged packet undefined Recommended not to use(*1) Tagged packet ative VLA STRIP_VLA Two-stacked tagged Inner VLA SET_VLA_VID packet (*1): It is possible to transmit untagged packets in VLA tunneling mode at VSI, but it can be received by other OpenFlow switch instance as there is no tag. Thus it is recommended to disable native VLA. (4) VLA priority translation (SET_VLA_PCP) SET_VLA_PCP action changes the priority value in VLA tag. When the port specified by transfer action is the access port, VLA priority is not added because the output packets have no VLA tags. (5) Source MAC address change (SET_DL_SRC) SET_DL_SRC action changes the source MAC address. (6) Destination MAC address change (SET_DL_DST) SET_DL_DST action changes the destination MAC address. (7) Source IP address change (SET_W_SRC) SET_W_SRC action changes the source IP address of IPv4 packets. (8) Destination IP address change (SET_W_DST) SET_W_DST action changes the destination IP address of IPv4 packets. 13

WD-105490-001 (9) IP ToS (DSCP) change (SET_W_TOS) SET_W_TOS action changes the ToS (DCSP) of IPv4 packets. (10) Source L4 port change (SET_TP_SRC) SET_TP_SRC action changes the source L4 port of TCP/UDP packets. (11) Destination L4 port change (SET_TP_DST) SET_TP_DST action changes the destination L4 port of TCP/UDP packets. 1.2.7 Statistics Statistics that can be collected by OFC using OpenFlow protocol messages is the statistics of flow entries, ports, and tables. Statistics of flow entry table can be displayed by operational commands of PFS. For details, see Chapter 3 Operation Command Reference. 1.2.8 Packet In Feature Packet In messages can be used to transmit packets to OFC via Secure Channel. OFC receives packets, and decides how to process them. ou can choose one of the actions from the following two when an input packet does not hit any flow entry; transmitting the packet to OpenFlow Controller by Packet In message (when mishit-action controller is set), or have the packet handled by legacy switch (when mishit-action normal is set). When mishit-action controller is enabled, and the O_PACKET_I flag is disabled (*1), a Packet In message is sent to OFC at mishit. Packet In message is also sent to OFC when the transfer action of flow entries is specified to output to Controller. This system adds an input packet of the size that is specified in OpenFlow protocol message to Packet In message, and transmits the packet to OFC. In VSI mode, packets are transmitted after removing the VLA tag that is used to identify VSI. (*1) When the exemption flag of Packet In message transmission is disabled (0). (See Table 1-16 List of Port Information for port setting flags.) Table 1-9 Configuration of Packet In Messages Field name buffer_id total_len in_port reason data Description Buffer number Total frame length Ingress port Reason for Packet In message generation Frame itself Table 1-10 Reason of Packet In Messages Field name OFPR_O_MATCH OFPR_ACTIO Description o hit in flow table search Transmission by transfer action to Controller 14

WD-105490-001 1.2.9 Packet Out Feature Packet Out message is used when OpenFlow Controller directs PFS to output packets. The packet data added to the Packet Out message is processed according to the action directed in the message. For the configuration of Packet Out message, see Table 1-11 Configuration of Packet Out Messages. Table 1-11 Configuration of Packet Out Messages Field name buffer_id in_port actions_len Actions Data Description Buffer ID Ingress port Size of action array Array of action structure Packet data When receiving a Packet Out message where OUTPUT action to TABLE is specified, search the packet by the flow table. If VSI receives a Packet Out message, add a VLA tag to identify VSI if necessary to output. (The tag is not applied at the access port.) 1.3 OpenFlow Feature Action Overview 1.3.1 OpenFlow Protocol Support Messages Table 1-12 List of Support Messages shows the list of OpenFlow Protocol messages supported by this system. For details of actions, see 1.3 OpenFlow Feature Action Overview. Table 1-12 List of Support Messages # Message ame Direction Use Support 1 Hello OFC<=>PFS Used for version negotiation 2 Error OFC<=>PFS otifies that there is an error in the message. 3 Echo Request OFC<=>PFS Echo request 4 Echo Reply OFC<=>PFS Echo reply 5 Vendor OFC<=>PFS Vendor-defined message - 6 Features Request OFC=>PFS Requests for PFS features. 7 Features Reply PFS=>OFC Replies to the requests for PFS features. 8 Get Configuration Request OFC=>PFS Requests for the OpenFlow configurations of PFS. (*1) 9 Get Configuration Reply PFS=>OFC Replies to a request for the OpenFlow configurations of PFS. (*1) 10 Set Configuration OFC=>PFS Sets OpenFlow configuration of PFS. (*1) 11 Packet In PFS=>OFC Transmits a packet to OFC. (*2) 12 Port Status PFS=>OFC otifies the change in status/settings of interface to OFC. 13 Packet Out OFC=>PFS OFC directs PFS to output packets. 14 Flow Mod OFC=>PFS OFC requests PFS to register, change, or delete a flow. 15 Flow Removed PFS=>OFC otifies statistics when deleting a flow entry or on timeout to OFC. (*3) 16 Port Mod OFC=>PFS Requests a change in interface settings. 17 Statistics Request OFC=>PFS Requests statistics. 18 Statistics Reply PFS=>OFC Replies to statistics request. 19 Barrier Request PFS=>OFC Requests OFC to ensure the order of messages. 15

WD-105490-001 # Message ame Direction Use Support 20 Barrier Reply OFC=>PFS otifies the completion of message processing in the correct order to PFS. 21 Queue Get Config Request PFS=>OFC Requests queue information. 22 Queue Get Config Reply OFC=>PFS Replies to queue information request. (*1) OpenFlow configuration is the configuration defined by OpenFlow protocols, not the configuration set to this system by the commands shown in Chapter 2 Configuration Command Reference. (*2) Transmitted only when the mishit-action controller is set, and O_PACKET_I is not set to port. (*3) Transmitted only when OFPFF_SED_FLOW_REM flag is set when registering a flow entry. : Supported : ot supported <=>: Transmits two-way messages. =>: Transmits one-way messages. 16

WD-105490-001 1.3.2 Secure Channel Descriptions Secure Channel is a control channel connected between OFC and OFS; used for exchanging OpenFlow messages. Each switch instance has an ability to connect one Secure Channel at a. Each switch instance can connect to up to four OFC IP addresses/port numbers; when connecting to OFC fails, the system tries to connect to other OFC. The following section explains the behavior of Secure Channel. 1.3.2.1 Establishing Secure Channel OFS starts connecting to OFC via Secure Channel when OpenFlow feature is enabled. Secure channel is built on a TCP session. Figure 1-14 Secure Channel Establishing Sequence shows a sequence of establishing Secure Channel. Steps (2) and later are typical processing after establishing Secure Channel, but they are not essential conditions for establishing Secure Channel. Figure 1-14 Secure Channel Establishing Sequence OFS OFC PFS Feature Enablement Establishing a TCP Session Version egotiation (Random order) Hello Hello Version egotiation (Random order) Acquiring OFS Feature Acquiring Port Status Random Order and Arbitrary Acquiring and Setting Configuration (1) OpenFlow Protocol Version egotiation Immediately after establishing a TCP session, Hello message is sent to OFC. After completing version negotiation, Secure Channel connection is established. If version negotiation detects that the version is not supported, an error message is sent out. When receiving an error message from OFC at version negotiation, TCP session is reset. Figure 1-15 Sequence for Version egotiation Failure shows a sequence where version negotiation failed. 17

WD-105490-001 Figure 1-15 Sequence for Version egotiation Failure OFS OFC Version egotiation Hello Error ot Compatible Resets TCP session (2) Checking OFS Features To Features Request message from OFC, Features Reply message is sent out. For details, see 1.3.6 OpenFlow Protocol Features otification Descriptions. (3) Port Status Change When receiving Port Mod message from OFC, the information is reflected to OFS. If it led to port status change, Port Status message is sent out. For details, see 1.3.5.2 Managing State of Port. (4) Checking configuration To Get Configuration Request from OFC, Get Configuration Reply message is sent out. When Set Configuration message is received from OFC, the configuration is reflected to OFS. For details, see 1.3.6 OpenFlow Protocol Features otification Descriptions. 1.3.2.2 Checking the Connectivity of Secure Channel After establishing Secure Channel, OFS periodically sends Echo Request messages to OFC according to Keep Alive timer. To Echo Request message from OFC, Echo Reply message is sent out. This system can set Keep Alive timer by the echo-request interval command. Figure 1-16 Secure Channel Connectivity Check Sequence OFS OFC Echo Request Echo Reply KeepAlive Timer Echo Request Echo Reply Echo Request Echo Reply After sending an Echo Request message, OFS waits for the Echo Reply message for the number of seconds set to Hold timer. ot receiving the Echo Reply message in Hold time, OFS disconnects the session. This system can set Hold timer by the echo-reply command. 18

WD-105490-001 Figure 1-17 Secure Channel at Failure Sequence OFS OFC Echo Request Echo Reply KeepAlive Timer Echo Request Hold Timer Echo Reply TCP Disconnection 1.3.2.3 Secure Channel Connection/Reconnection Trials Behavior If Secure Channel cannot establish a connection in the number of seconds set by Connect timer elapsed from the start of connection, the connection to the OFC is disconnected. When TCP three-way handshake cannot be completed, or Hello is not returned, it is considered that the connection cannot be established. When Secure Channel is disconnected, wait for the number of seconds set by Retry timer, and try reconnecting. Retry timer exponentially increases (such as 0, 1, 2, 4...) to the specified maximum number every time connection fails. When the maximum number exceeds, the maximum number is used. For example, when the maximum value of Retry timer is set to 100, 100 seconds comes after 64 seconds, and 100 seconds is used thereafter. Retry timer is reset to 0 when the connection is established normally. Figure 1-18 Secure Channel Connection Trial Sequence (Starts as connected to OFC1) OFS OFS1 OFS2 OFS3 TCP Disconnection Connect (echo timeout) TCP Disconnection (Failed to connect) TCP Disconnection (Failed to connect) TCP Disconnection connect timeout Connect connect timeout connect timeout Connect One cycle of connection trial (Failed to connect) TCP Disconnection retry timer (Failed to connect) TCP Disconnection (Failed to connect) TCP Disconnection (Failed to connect) One cycle of connection trial TCP Disconnection (Failed to connect) : : retry timer 19

WD-105490-001 Each switch instance can be set whether switching to Emergency mode is to be executed or not by using commands. When connection to all the designated OFCs is considered to be impossible, the system is switched to Emergency mode if Emergency mode is set to be enabled. The system switches to Emergency mode when Retry timer exceeds the maximum value. (See Figure 1-19 Execution Timing of Emergency Operation.) If Emergency mode is not set to be enabled, the system continues retrying connection with the Retry timer set to the maximum. Figure 1-19 Execution Timing of Emergency Operation OFS OFC1 OFC2 OFC3 retry timer (0) retry timer (1) retry timer (2) One Cycle of Connection Trial One Cycle of Connection Trial One Cycle of Connection Trial One Cycle of Connection Trial retry timer (4) : : One Cycle of Connection Trial retry timer (X) [X=max] One Cycle of Connection Trial One Cycle of Connection Trial Emergency Operation When Data path ID of a switch instance is changed, Secure Channel is disconnected. 1.3.2.4 Operation at Secure Channel Disconnection When disconnecting Secure Channel, the following operation is executed. The system tries reconnecting to OFC by the same procedures described at 1.3.2.3 Secure Channel Connection/Reconnection Trials Behavior. When Secure Channel is disconnected, OpenFlow messages are not sent out. For details, see 1.3.2.5 Message Handlings at Secure Channel Disconnection. 1.3.2.5 Message Handlings at Secure Channel Disconnection While Secure Channel is disconnected, OpenFlow messages are not sent out, but the packet forwarding is continued according to the flow entries set to flow table. If the packet matches the flow entry which executes software forwarding, it is not forwarded. 20

WD-105490-001 1.3.3 Flow Table Control Descriptions The following section explains how to control flow table. 1.3.3.1 Flow Table Control by FlowMod Messages OFC requests OFS to register, change, or delete a flow entry by using FlowMod messages. The following table shows a list of actions that can be controlled by FlowMod messages and their descriptions. Table 1-13 List of Flow Table Control Items by FlowMod Messages Command ADD MODIF MODIF_STRICT DELETE DELETE_STRICT Description Adds a flow entry. Changes all flow entries that match. Changes one flow entry that exactly matches. Deletes all flow entries that match. Deletes one flow entry that exactly matches. (*1) When the maximum number of flow entries per interface is registered, MODIF_STRICT cannot be executed. Delete the entry and register again. Adding a flow entry by FlowMod Messages Adds a flow entry by FlowMod (ADD) message. If there is already an entry that has the same search priority and the same match condition with the specified flow entry, the existing entry is overwritten. In this case, statistics is to be reset. Exact match flow entries are considered to share the same search priority. A wildcard match flow entry that has the same search priority with the specified flow entry is added to a location that comes later in search priority than the existing entry. Changing a flow entry by FlowMod Messages FlowMod (MODIF, MODIF_STRICT) message changes the action of the specified flow entry to the one specified in the massage. If there is not a flow entry that is specified in the message, execute the action of FlowMod (ADD) using the information specified in the message. Deleting a flow entry by FlowMod Messages Deletes a flow entry by FlowMod (DELETE, DELETE_STRICT) message. If there is not a specified flow entry and cannot delete one, an error message is not sent out but the error is recorded in the system. Error handling with FlowMod message If a flow entry cannot be registered to the flow table, an error message is sent to OFC. Causes of flow entry registration failure can be acquired by the error message type and code number. 21

WD-105490-001 1.3.3.2 Flow Table Control by Timeout OFS periodically checks each flow entry in the flow table, and executes timeout if necessary. The following table shows the types of timeouts and necessary processing. Table 1-14 List of Flow Table Control Items by Timeout Item Description idle_timeout Time when there is no communication hard_timeout Maximum life time (1) idle_timeout Flow entries where idle_timeout time (sec) has elapsed since the last packet hit are to be deleted. Flow entries where 0 is set to idle_timeout are not to be deleted. (2) hard_timeout Flow entries where hard_timeout time (sec) has elapsed since registered to the flow table are to be deleted. Flow entries where 0 is set to hard_timeout are not to be deleted. 1.3.4 Emergency Mode Descriptions When Secure Channel is disconnected and reconnection is tried, but connection to all the designated OFCs are considered to be impossible, the system is switched to Emergency mode if Emergency mode is set to be enabled. (See Figure 1-19 Execution Timing of Emergency Operation.) When switched to Emergency mode, the flow entries in use are disabled, and the Emergency flow entries are copied to the flow table to transfer. They are used for packet transfer as Figure 1-20 Actions When Emergency Mode Is Enabled. Figure 1-20 Actions When Emergency Mode Is Enabled Emergency Flow Table Flow Entry A Flow Entry B Flow Entry C Emergency Flow Table Flow Entry A Flow Entry B Flow Entry C Basic Flow Table Basic Flow Table Copy Flow Entry D Flow Entry E Flow Entry G Emergency Flow Entry A Flow Entry B Flow Entry H Flow Entry I Extended Flow Table Extended Flow Table *Flow Entries D to I are discarded. When switched to Emergency mode, the flow entries in the Emergency flow table are copied to the basic flow table. When Secure Channel is reconnected, the flow entries that were enabled before switching to Emergency mode are not to be automatically enabled. 22