Table of Contents 1. DUT and Test Equipments...2

Transcription

1 Table of Contents 1. DUT and Test Equipments DUT Specification Functionality test Management Interface Evaluation Firmware Upgrade and Configuration Files Update Test VLAN Test Link Aggregation Test Quality of Service Test Port counter Test and Mirror Port Test Address Lookup Test Rate Limit Test x Test Performance Test Forwarding Test Congestion Control Test Address Caching Capacity Test Address Learning Rate Test Errored Frame Filtering Test Broadcast Frame Forwarding Test Broadcast Latency Test Forward Pressure Test...43 Network Benchmarking Lab 1

2 1. DUT and Test Equipments 1.1 DUT Specification Hardware: XXX Layer 2 Switch 8 Giga ports 1 RS232 port Firmware: xxx Network Benchmarking Lab 2

3 1.2 Test Equipments Specification Spirent SmartBits: SmartBits 6000B Chassis TreaMetric XD 10/100/1000 Mbps Ethernet Modules ASTII 3.5 SmartBits SmartFlow 4.5 Smartwindow 8.5 TeraDot1x Tester 1.00 Ixia IxANVL (Automatic Network Validation Library) 6.4: Rapid Spanning Tree Protocol Validation Other tools: Web Browsers: Internet Explorer xpsp_sp2 Firefox Opera 8.5 Terminal Emulator Windows 2003 Server Windows XP MIB Browser, MIB Walker TFTP Server SNTP Server Ethernet Switch 客製化測試計劃書 Network Benchmarking Lab 3

4 2. Functionality test 2.1 Management Interface Evaluation Test Purpose: This test evaluates the easiness and correctness of the management interfaces. The CLI (Command Line Interface), Web, and SNMP (Simple Network Management Protocol) interface are evaluated, and do an error input checking for CLI and Web management interface. Test Tools: Terminal emulator Internet Explorer xpsp_sp2 Firefox Opera 8.5 SNMP Browser SNMP Walker Test Description: The test configuration of this test is shown in Figure 1. The parameters of the DUT are configured through a management interface. Then, the configured parameters are checked by another management interface to see whether they are effective. After all the parameters are configured and checked, the evaluation of the two management interfaces is done. The same procedures are repeated until all the management interfaces are evaluated. The result of a configuration should be consistent with all of the management interfaces. The CLI and Web are typed in different error syntax, and test the DUT whether it can determine. Test Configuration: Figure 1. CLI/Web/SNMP Management Interface Configuration Test observation: 1. Observe the DUT parameters can be set up on different management interfaces, and the configuration results must be the same. 2. Observe the DUT parameters can be set up on different web browsers, and the configuration results must be the same. 3. Address the problem of DUT management interface or somewhere else needs to be improved. Network Benchmarking Lab 4

5 Test Result: 1. The capability of port speed can be reached 1000Mb, but the function of speed duplex of web only can set 100Mbps Full to be maximum value. 2. Please refer to the following table. This test is passed. RFC 1213 System Group Object Name Object Access Type Result sysdescr Read-only Passed sysobjectid Read-only Passed sysuptime Read-only Passed syscontact Read-write Passed sysname Read-write Passed syslocation Read-write Passed sysservices Read-only Passed Comment RFC 1213 Interface Group Object Name Object Access Type Result ifnumber Read-only Passed ifindex Read-only Passed ifdescr Read-only Passed iftype Read-only Passed ifmtu Read-only Passed ifspeed Read-only Passed ifphysaddress Read-only Passed ifadminstatus Read-write Passed ifoperstatus Read-only Passed iflastchange Read-only Passed ifinoctets Read-only Passed ifinucastpkts Read-only Passed ifinnucastpkts Read-only Passed ifindiscards Read-only Passed ifinerrors Read-only Passed Comment Network Benchmarking Lab 5

6 ifinunknownprotos Read-only Passed ifoutoctets Read-only Passed ifoutucastpkts Read-only Passed ifoutnucastpkts Read-only Passed ifoutdiscards Read-only Passed ifouterrors Read-only Passed ifoutqlen Read-only Passed ifspecific Read-only Passed RFC 1398 Ether-Like Interface Type(dot3StatsTable) Object Name Object Access Type Result dot3statsindex Read-only Passed dot3statsalignmenterrors Read-only Passed dot3statsfcserrors Read-only Passed dot3statssinglecollisionframes Read-only Passed dot3statsmultiplecollisionframes Read-only Passed dot3statssqetesterrors Read-only Passed dot3statsdeferredtransmissions Read-only Passed dot3statslatecollisions Read-only Passed dot3statsexcessivecollisions Read-only Passed dot3statsinternalmactransmiterrors Read-only Passed dot3statscarriersenseerrors Read-only Passed dot3statsframetoolongs Read-only Passed dot3statsinternalmacreceiveerrors Read-only Passed dot3statsetherchipset Read-only Passed dot3statssymbolerrors Read-only Passed dot3statsduplexstatus Read-only Passed dot3statsratecontrolability Read-only Passed dot3statsratecontrolstatus Read-only Passed Comment Network Benchmarking Lab 6

7 RFC 1398 Ether-Like Interface Type(dot3CollTable) Object Name Object Access Type Result dot3collindex Read-only Passed dot3collcount Read-only Passed dot3collfequencies Read-only Passed Comment Network Benchmarking Lab 7

8 2.2 Firmware Upgrade and Configuration Files Update Test Test Purpose: This test checks whether the DUT can upgrade its firmware and detect possible error situations. Test Tools: TFTP Server Test Description: The configuration of this test is shown in Figure 2. The firmware/configuration file is put on a TFTP server. The DUT is configured to download the firmware/configuration file from the TFTP server. The following errors are checked while downloading the firmware/configuration file to the DUT. 1. The DUT downloads an inexistent file. 2. The network connection is broken while the DUT is downloading the file. 3. The DUT is downgraded with an old firmware. 4. The available storage space of the DUT is not enough for the file to be downloaded. 5. The DUT is upgraded with an invalid firmware file/configuration file. 6. The DUT is upgraded with an incomplete firmware file/configuration file. 7. The DUT is upgraded with a redundant firmware file/configuration file. 8. The power is shut down while the DUT is upgrading. Test Configuration: Test observation: Figure 2. Firmware Upgrade test configuration The DUT should be able to detect the error situations and display error messages. Test Result: This is a summary for the test plan 2.2 firmware upgrade and configuration files update test of XXX. Each number as shown in the following is corresponding to the error under test description. (From 1-8) Network Benchmarking Lab 8

9 1.OK 2.OK 3.Fail Two Observation : (a)boot to fail safe mode in web, console in CFE (b)web freeze (no fail save mode), console in CFE (Occasionally occurred) Error message : ##################################################################### Total memory used by CFE: 0x x ( ) Initialized Data: 0x8053B460-0x8053CEC0 (6752) BSS Area: 0x8053CEC0-0x (14032) Local Heap: 0x x ( ) Stack Area: 0x x (4096) Text (code) segment: 0x x8050D1BC (53692) Boot area (physical): 0x x Relocation Factor: I: D: Loader:elf Filesys:raw Dev:flash0.os File: Options:(null) Loading: Failed. Could not load flash0.os:: Executable is wrong-endian mii_probe: Using PHY 30 eth1: Link speed: 100BaseT FDX (switch) Device eth1: hwaddr , ipaddr , mask gateway , nameserver not set ##################################################################### Intermediate Action From fail safe mode in web, upgrade to test1 or test2 version Observation: Two Observation occurred (a) WEB back to normal when upgrade from the invalid version to either test1 or test2 version (i.e. test1 -> test1 OK. test2-> test2 OK) (b) WEB freeze when upgrade from the invalid version to either test1 or test2 version (This occurred when upgrade to different test version i.e. test1 -> test2, test2 -> test1) Network Benchmarking Lab 9

10 Error message : Error message appeared in console as shown in the following (from safe mode to either test 1 or 2) ##################################################################### Enable CP0 IRQ of EMAC1 0x3000a801 base_address = 0xb , iline = 3, ipin = 2 found robo device with 0:0143:bcd0:98:00 found 1 robo device(s). SPI unit 0: Dev 0xbcd0, Rev 0x00, Chip BCM5398_A0, Driver BCM5398_A0 PCI unit 1: Dev 0x4713, Rev 0x09, Chip BCM4713_A0, Driver BCM4713_A0 Attaching SOC unit 0... SPI device BCM5398_A0 attached as unit 0. Attaching SOC unit 1... Xxx BCM47xx 10/100 Mbps Ethernet Controller Failed to open medium BRCM-NVRAM for validation. snmp agent init! Failed to open medium BRCM-NVRAM with op=count Failed to open medium BRCM-NVRAM with op=count init_master_agent: Invalid local port (Can't assign requested address) Server Exiting with code 1 telnetd started. BCM.0> BCM.0> ##################################################################### Possible Solution: use reboot in BCM command in console in order to start WEB 4.OK 5.OK Observation: Boot to fail safe mode in web, console in CFE Intermediate Action From fail safe mode in web, upgrade to either test1 or test2 version Observation: Two Observation occurred Network Benchmarking Lab 10

11 (a) WEB back to normal when upgrade from the invalid version to either test1 or test2 version (i.e. test1 -> test1 OK. test2-> test2 OK) (b) WEB freeze when upgrade from the invalid version to either test1 or test2 version (This occurred when upgrade to different test version i.e. test1 -> test2, test2 -> test1) Error message for (b): Same as error message in 3. 6.OK Observation: Boot to fail safe mode in web, console in CFE Intermediate Action From fail safe mode in web, upgrade to either test1 or test2 version Observation: Two Observation occurred (a) WEB back to normal when upgrade from the invalid version to either test1 or test2 version (i.e. test1 -> test1 OK. test2-> test2 OK) (b) WEB freeze when upgrade from the invalid version to either test1 or test2 version (This occurred when upgrade to different test version i.e. test1 -> test2, test2 -> test1) 7.Fail (Redundant firmware = from original firmware to exactly same firmware with different file name and different firmware version string) Observation: Web freeze and console error message has appear (same as 3) Possible Solution: use reboot in BCM command in console in order to start WEB 8.OK Observation: Boot to fail safe mode in web, console in CFE Observation: Network Benchmarking Lab 11

12 Two Observations occurred with same condition (a) WEB back to normal when upgrade from the power failure to either test1 or test2 version (i.e. test1 -> test1 OK. test2-> test2 OK). (b) WEB freeze when upgrade from the power failure to either test1 or test2 version (This occurred when upgrade to different test version i.e. test1 -> test2,test2 -> test1 Network Benchmarking Lab 12

13 2.3 VLAN Test Test Purpose: This test validates the VLAN capacity that DUT declares to support. Test Tools: SmartFlow 4.5 Test Description: The configuration of this test is shown in Figure 3. The two ports of the tester (T1 and T2) are connected to the two ports of the DUT (D1 and D2) respectively. VLANs are created on the DUT, and the number of VLAN created is the capacity that the DUT claims to support. The port D1 and D2 are associated to the created VLANs; that is, the two ports are VLAN trunk ports that can accept untagged and tagged frames. The port T1 sends tagged and untagged frames to the port D1. The sent frames contain the VLAN IDs ranging from 1 to the number of the VLAN capacity the DUT claims to support. The DUT forwards the frames to the port D2. The tester checks the frames received from the port T2 and validates the VLAN capacity the DUT claims to support. Test Configuration: Figure 3 VLAN test configuration Test observation: The DUT should be able to forward untagged frames and tagged frames containing the VLAN IDs ranging from 1 to the number of the VLAN capacity the DUT claims to support. Test Result: 1. Please refer to the following table. This test is passed. Network Benchmarking Lab 13

14 VLAN ID 2~2047 Test Duration (Seconds) 30 Frame Size (Bytes) ,024 1,280 1,518 9,216 Test Frames Transmitted 44,640,976 25,337,766 13,585,939 7,047,821 3,590,438 2,884,223 2,437, ,306 Test Frames Received 44,640,976 25,337,766 13,585,939 7,047,821 3,590,438 2,884,223 2,437, ,306 Test Frames Lost Forwarding Rate at Maximum Offered Load Test Frames Lost (%) % % % % % % % % Test Frames/Second 1,488, , , , ,681 96,141 81,266 13,510 Transmitted Test Frames/Second 1,488, , , , ,681 96,141 81,266 13,510 Received Bits/Second Transmitted 999,957, ,997, ,925, ,850, ,577, ,863, ,895, ,241,658 Bits/Second Received 999,957, ,997, ,925, ,850, ,577, ,863, ,895, ,241,658 Test Frames Transmitted 44,640,976 25,337,766 13,585,939 7,047,821 3,590,438 2,884,223 2,437, ,306 Test Frames Received 44,640,976 25,337,766 13,585,939 7,047,821 3,590,438 2,884,223 2,437, ,306 Zero-loss Maximum Forwarding Rate Test Frames/Second Transmitted Test Frames/Second Received Bits/Second Transmitted 1,488, , , , ,681 96,141 81,266 13,510 1,488, , , , ,681 96,141 81,266 13, ,957, ,997, ,925, ,850, ,577, ,863, ,895, ,241,658 Bits/Second Received 999,957, ,997, ,925, ,850, ,577, ,863, ,895, ,241,658 Total Throughput (%) % % % % % % % % Network Benchmarking Lab 14

15 VLAN ID 2048~4094 Test Duration (Seconds) 30 Frame Size (Bytes) ,024 1,280 1,518 9,216 Test Frames Transmitted 44,640,976 25,337,766 13,585,939 7,047,821 3,590,438 2,884,223 2,437, ,306 Test Frames Received 44,640,976 25,337,766 13,585,939 7,047,821 3,590,438 2,884,223 2,437, ,306 Test Frames Lost Forwarding Rate at Maximum Offered Load Test Frames Lost (%) % % % % % % % % Test Frames/Second 1,488, , , , ,681 96,141 81,266 13,510 Transmitted Test Frames/Second 1,488, , , , ,681 96,141 81,266 13,510 Received Bits/Second Transmitted 999,957, ,997, ,925, ,850, ,577, ,863, ,895, ,241,658 Bits/Second Received 999,957, ,997, ,925, ,850, ,577, ,863, ,895, ,241,658 Test Frames Transmitted 44,640,976 25,337,766 13,585,939 7,047,821 3,590,438 2,884,223 2,437, ,306 Test Frames Received 44,640,976 25,337,766 13,585,939 7,047,821 3,590,438 2,884,223 2,437, ,306 Zero-loss Maximum Forwarding Rate Test Frames/Second Transmitted Test Frames/Second Received 1,488, , , , ,681 96,141 81,266 13,510 1,488, , , , ,681 96,141 81,266 13,510 Bits/Second Transmitted 999,957, ,997, ,925, ,850, ,577, ,863, ,895, ,241,658 Bits/Second Received 999,957, ,997, ,925, ,850, ,577, ,863, ,895, ,241,658 Total Throughput (%) % % % % % % % % Network Benchmarking Lab 15

16 2.4 Link Aggregation Test Test Purpose: This test validates whether the DUT supports link aggregation. The ports in the same channel group can share the traffic of the group. Test Tools: SmartFlow 4.5 Test Description: The configuration of this test is shown in Figure 4. This test needs two DUTs. The four ports of the DUT1 (D1-D4) are connected to the four ports of the DUT2 (D6-D9). The four links between the DUT1 and DUT2 are configured to form an aggregated link. The two ports of tester (T1 and T2) are connected to the DUT1 (D5) and DUT (D10). The port T1 sends traffic to the DUT1 and receives the forwarded traffic from the port T2. The amounts of the traffic the ports D6-D9 received are checked to see whether the traffic is distributed among the four links. After this, the above procedures are repeated with the scenarios of removing or recovering a link. This is for the purpose of validating the robustness of the link aggregation of the DUT. Test Configuration: Figure 4 Link aggregation test configuration Test observation: The traffic should be distributed among the links. In case of the breakdown or recovery of a link, the link aggregation still works functionally. Test Result: 1. Redundancy can be achieved when distribution criterion switch to SA+DA mode in Network Benchmarking Lab 16

17 the trunking control menu but no load balance presented. 2. This test has passed for the SA and DA mode. Network Benchmarking Lab 17

18 2.5 Quality of Service Test Test Purpose: The following classification mechanisms are tested: Class of Service The administrator may define up to eight classes of service using the bits in user priority bits in the VLAN tag. Then it is possible to utilize other QoS features to assign appropriate traffic-handling policies, including congestion management, bandwidth allocation, and delay bounds for each traffic class. Test Tools: SmartFlow 4.5 Test Description: The configuration of this test is shown in Figure 5. The five ports of the DUT (D1-D5) are connected to the five ports of the tester (T1-T5). The parameters of QoS and the rule of traffic classifications are configured on the DUT, as shown in Figure 5(b), The tester provide two different offered load from four source port (T1-T4) to the destination port (T5), 25% and 100% offered load respectively. The destination port (T5) will cause some congestion situation at destination port (T5) if there is more than 25% offered load. So, the DUT will follow the traffic classification and scheduling to provide quality of service. Two classifications are outlined in this test below: 1. No quality of service: This is no QoS mechanism enabled on the DUT. This test result will be compared with other results of the traffic classification. 2. Classification by class of service value: The traffic is classified by the user priority value of the VLAN tag. Test Configuration: (a) cable connections. (b) traffic distributions. Figure 5 Quality of service test configuration Test observation: 1. The no quality of service scheme: Network Benchmarking Lab 18

19 Measure the forwarding rate of the stream sent from the first port of the tester (T1) to the fourth port of the tester (T4), the three streams should be shared 1/3 bandwidth of T4. 2. The class of service scheme: Measure the forwarding rate of the stream sent from the first port of the tester (T1) to the fourth port of the tester (T4) whether it conforms to the traffic classification and scheduling to provide quality of service. Test Result: 1. The QoS can be achieved when the DUT has set to either Strict Mode or Weights Round Robin mode. The results are summarized as followed (for the more detail result please refer to the excel file name QoS under XXX directory): No QoS (100% Load on each port) Packet Length Tested 64 bytes 128 bytes 256 bytes 512 bytes 1024 bytes 1280 bytes 1518 bytes 9000 bytes Traffic Loading on Destination Port 400% 400% 400% 400% 400% 400% 400% 400% Pkt Forwarded Ratio of Queue % % % % % % % % Pkt Forwarded Ratio of Queue % % % % % % % % Pkt Forwarded Ratio of Queue % % % % % % % % Pkt Forwarded Ratio of Queue % % % % % % % % Strict_QoS (25% Load on each port) Packet Length Tested 64 bytes 128 bytes 256 bytes 512 bytes 1024 bytes 1280 bytes 1518 bytes 9000 bytes Traffic Loading on Destination Port 100% 100% 100% 100% 100% 100% 100% 100% Pkt Forwarded Ratio of Queue 0 (Priority 0) Pkt Forwarded Ratio of Queue 1 (Priority 1) Pkt Forwarded Ratio of Queue 2 (Priority 2) Pkt Forwarded Ratio of Queue 3 (Priority 3) % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % Network Benchmarking Lab 19

20 Strict_QoS (100% Load on each port) Packet Length Tested 64 bytes 128 bytes 256 bytes 512 bytes 1024 bytes 1280 bytes 1518 bytes 9000 bytes Traffic Loading on Destination Port 400% 400% 400% 400% 400% 400% 400% 400% Pkt Forwarded Ratio of Queue 0 (Priority 0) Pkt Forwarded Ratio of Queue 1 (Priority 1) Pkt Forwarded Ratio of Queue 2 (Priority 2) Pkt Forwarded Ratio of Queue 3 (Priority 3) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% % % % % % % % % WRR_QoS (25% Load on each port) Packet Length Tested 64 bytes 128 bytes 256 bytes 512 bytes 1024 bytes 1280 bytes 1518 bytes 9000 bytes Traffic Loading on Destination Port 100% 100% 100% 100% 100% 100% 100% 100% Pkt Forwarded Ratio of Queue 0 (Weight 1) Pkt Forwarded Ratio of Queue 1 (Weight 5) Pkt Forwarded Ratio of Queue 2 (Weight 10) Pkt Forwarded Ratio of Queue 3 (Weight 15) % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % WRR_QoS (100% Load on each port) Packet Length Tested 64 bytes 128 bytes 256 bytes 512 bytes 1024 bytes 1280 bytes 1518 bytes 9000 bytes Traffic Loading on Destination Port 100% 100% 100% 100% 100% 100% 100% 100% Pkt Forwarded Ratio of Queue 0 (Weight 1) Pkt Forwarded Ratio of Queue 1 (Weight 5) Pkt Forwarded Ratio of Queue 2 (Weight 10) Pkt Forwarded Ratio of Queue 3 (Weight 15) 3.23% 3.23% 3.23% 3.23% 3.23% 3.23% 3.48% 6.90% 16.13% 16.13% 16.13% 16.13% 16.13% 16.13% 15.85% 14.27% 32.26% 32.26% 32.26% 32.26% 32.26% 32.26% 30.06% 17.47% 48.39% 48.39% 48.39% 48.39% 48.39% 48.39% 50.62% 61.36% Network Benchmarking Lab 20

21 2.6 Port counter Test and Mirror Port Test Test Purpose: This case tests the port counter degree of accuracy and tests the mirror function whether it works. Test Tools: Smartwindow 8.5 Test Description: The configuration of this test is shown in Figure 6. The two ports of the DUT (D1-D2) are connected to the four ports of the tester (T1-T2). The parameter of mirror is configured on the DUT. The source port of the DUT is D1; destination port of the DUT is D2. The tester sends three different traffics, unicast, multicast, and broadcast from the port (T1) to the port (T2). Test Configuration: Figure 6 port counter test configuration Test Observation: To observe the web port counter degree of accuracy and mirror port data whether it is same as the DUT port (D1). Test Result: This test has passed. Network Benchmarking Lab 21

22 2.7 Address Lookup Test Test Purpose: This case tests the address lookup (ARL) function whether it works while the MAC address is added into MAC address table form web or dynamic learning form the tester. Test Tools: Smartwindow 8.5 Test Description: The configuration of this test is shown in Figure 7. The two ports of the DUT (D1-D2) are connected to the four ports of the tester (T1-T2). The tester send a frame to the port of the DUT (D1), and then check this MAC address whether it is in the MAC address table by the ARL function of the web and the BCM mode. Test Configuration: Figure 7 address lookup test configuration Test Observation: To observe the address lookup (ARL) function whether it works while the MAC address is added into MAC address table form web or dynamic learning form the tester. Test Result: This test has passed. Nevertheless, Dynamic learned Mac address can still be found(lookup) when restore XXX to factory defaults in the WEB. Network Benchmarking Lab 22

23 2.8 Rate Limit Test Test Purpose: This case tests the rate limit function whether it limits the different traffics such as broadcast, unicast, multicast, and DLF traffic. Test Tools: Smartwindow 8.5 Test Description: The configuration of this test is shown in Figure 8. The two ports of the DUT (D1-D2) are connected to the four ports of the tester (T1-T2). The parameter of rate limit is configured on the DUT. The tester sends three different traffics, unicast, multicast, and broadcast from the port (T1) to the port (T2). Test Configuration: Figure 8 rate limit test configuration Test Observation: To observe the rate limit function whether it limits the different traffics such as broadcast, unicast, multicast, and DLF traffic. Test Result: 1. Rate Limit can be achieved when switch is set to broadcast, broadcast and unknown unicast and broadcast multicast and unkown unicast. However, rate Limit does not applied to multicast when broadcast and multicast is set under storm control function menu. A summary of the result is shown in the next page. 2. Stormcontrol can only be disabled when restore factory defaults i.e. Storm control type can not switch back to disable when set to any of the four mode (Broadcast, Broadcast and Multicast, Broadcast and unknow unicast and broadcast, multicast and unknow unitcast). This bug has been fixed under the firmware version WSS: rc1 SDK: sdk-robo dev. Network Benchmarking Lab 23

24 3. The figure shown in the broadcast storm control should display bps instead of pps. This bug has been fixed under the firmware version WSS: rc1 SDK: sdk-robo dev. XXX Rate Limit summary result. Rate Limit (Kbps) Actual Received Bits/ Seconds (BC) Actual Received Bits/ Seconds (BC_MC / BC) Actual Received Bits/ Seconds (BC_MC / MC) Actual Received Bits/ Seconds (BC_UU / BC) Actual Received Bits/ Seconds (BC_UU / UU) Actual Received Bits/ Seconds (BC_MC_UU / MC) Actual Received Bits/ Seconds (BC_MC_UU / UU) Actual Received Bits/ Seconds (BC_MC_UU / BC) BC=Broadcast, MC=Multicast, UU=Unknow Unicast, /=Using mode e.g. BC+MC/BC means using broadcast under broadcast and multicast mode Network Benchmarking Lab 24

25 2.9 IGMP Snooping Test Test Purpose: This case tests the IGMP Snooping function whether it works while the multicast traffic of a group is only forwarded to ports that have members of that group. Test Tools: Smartwindow 8.5 SmartFlow 4.5 Test Description: The configuration of this test is shown in Figure 9. The two ports of the DUT (D1-D2) are connected to the two ports of the tester (T1-T2). The port 3 of the DUT (D3) is connected to the port 2 of L3 switch (S2). The port 1 of the L3 Switch (S1) is connected to the port 3 of the tester (T3). The parameter of IGMP Snooping and router port are configured on the DUT. The parameter of the IGMP and DVMRP are configured on the L3 switch. The port T1 sends the IGMP Report message to the DUT when subscriber join the IGMP group, the port T3 will send multicast traffic of a group is only forwarded to ports that have members of that group. Meanwhile, the port T2 captures the frame from the port D2 whether it receives any multicast traffic. Test Configuration: Figure 9 IGMP Snooping test configuration Test Observation: To observe the port D2 whether it receives any multicast traffic or IGMP message while the DUT turn on/off the IGMP Snooping and router port of the DUT. Test Result: 1. The IGMP Snooping test is passed for the XXX switch. Nevertheless, the frame loss is considerably higher when compares with other switches such as Cisco Catalyst 2950 or Edgecore ES4649 (Frame loss is always above 85% when compare with Catalyst or Edgecore switch which has only 0.1 or less frame loss). Furthermore, Network Benchmarking Lab 25

26 there is no frame loss if there is only one Vlan created in the L3 switch. In addition, frame loss is as higher as percent when IGMP Snooping function is switch off in the XXX switch. For more detail log, please refer to the directory name IGMP Snooping test under functionality directory. Network Benchmarking Lab 26

27 x Test Test Purpose: This case tests the basic functionally of IEEE 802.1X. Reference: IEEE 802.1X - Port Based Network Access Control. Test Tools: RADIUS server - Windows 2003 server Supplicant - Windows XP Test Description: This test uses the Windows XP as supplicants, the DUT as an authenticator and Windows 2003 server as an authentication server. The cable connections between the supplicant, DUT and Authentication Server are shown in Figure 10. The port (S1) of the supplicant is connected to the ports (D1) of the DUT. The port (D2) of the DUT is connection to the port (A1) of the Authentication Server to transmit and receive request and response messages. The parameters of the 802.1x and EAP types are configured on the DUT, supplicant, and authentication server. Test Configuration: Figure 10. IEEE 802.1X port based network access control test. Test observation: Observe the supplicants whether it can be authenticated through the DUT by the Authentication Server. Test Result: 1. This test has passed. Network Benchmarking Lab 27

28 3. Performance Test 3.1 Forwarding Test Test Purpose: To test the DUT forwarding rate, frame loss and throughput of layer 2 frames in different distributions traffic. Reference: RFC Benchmarking Terminology for LAN Switching Devices RFC Benchmarking Methodology for LAN Switching Devices Test Tools: Smartflow 4.5 Test Description: The configuration of this test is shown in Figure 12. The 8 ports of the DUT (D1-D8) are connected to the 8 ports of the tester (T1-T8). Following Figure 12 shows different types of traffic distributions. During the testing, the binary search algorithm is used to find out the zero-loss maximum forwarding rate. Moreover, the tester sends the maximum offered load traffic to the DUT, and measures the DUT to see whether it can cause frame loss. During the test, the tested frame sizes are 64, 128, 256, 512, 1024, 1280, 1518, 9000 (jumbo) bytes, and the test duration is 30 seconds for each iteration. Network Benchmarking Lab 28

29 Test Configuration: Forwarding rate testing cable connections. (a)fully Meshed traffic. (b) Many-to-many, unidirectional traffic. (c) Many-to-many, bidirectional traffic. (d) One-to-many, unidirectional traffic. (e) One-to-many, bidirectional traffic. (f) Many-to-one, unidirectional traffic. (g) Many-to-one, bidirectional traffic. T1 T8 (h) Single-pair, intra-asic unidirectional traffic. (i) Single-pair, intra-asic, bidirectional traffic. (j) Single-pair, inter-asic, unidirectional traffic. (k) Single-pair, inter-asic, bidirectional traffic. (l) Multiple-pair, intra-asic, unidirectional traffic. (m) Multiple-pair, intra-asic, bidirectional traffic. (n) Multiple-pair, inter-asic, unidirectional traffic. Figure 12 Forwarding test configuration (o) Multiple-pair, inter-asic, bidirectional traffic. Test observation: 1. Measure the DUT zero-loss maximum forwarding rate when the DUT doesn t have any frame loss. 2. Measure the DUT to see whether it can cause frame loss during the tester sends the maximum offered load traffic to the DUT. Network Benchmarking Lab 29

30 Test Result: This test has passed. However Jumbo frame lost has occurred in the following case : fully mesh traffic, many-to-many bidirectional traffic, multiple-pair intra-asic bidirectional traffic and multiple-pair inter-asic bidirectional traffic. Please refer to the excel file named RFC2889 under XXX directory for more detail log.. Network Benchmarking Lab 30

31 3.2 Congestion Control Test Test Purpose: To determine how the DUT handles congestion. Does the device implement congestion control and does congestion on one port affect an uncongested port. This procedure determines if head of line blocking and/or backpressure are presented. Reference: RFC Benchmarking Terminology for LAN Switching Devices RFC Benchmarking Methodology for LAN Switching Devices Test Tools: AST II 3.5 Test Description: The configuration of this test is shown in Figure 13(a). The four ports of the DUT (D1-D4) are connected to the four ports of the tester (T1-T4). The parameters of the congestion control function are configured on the four ports of the DUT (D1-D4). T1 and T2 are source ports, T3 is an uncongested port, and T4 is a congested port. The tester provides 50% maximum offered load from the source port (T1) to the destination port (T3), and provides 50% maximum offered load from the source port (T1) to the destination port (T4), as shown in Figure 13(b). So, T4 will cause congestion situation, but T3 will not. Test Configuration: T1 50% MOL T3 Uncongested 50% MOL T2 100% MOL T4 Congested (a) Congestion control testing cable (b) Congestion control testing traffic connections. distribution. Figure 13 Congestion control test configuration Test observation: 1. The number of test frames received on the uncongested port (T3) should be 50% of the test frames transmitted by the first source port (T1). 2. The number of test frames received on the congested port (T4) should be between 100% and 150% of the test frames transmitted by one source port. 3. If there is frame loss at the uncongested port (T3), head of line blocking is presented. Network Benchmarking Lab 31

32 4. If there is no frame loss on the congested port (T4), then backpressure is presented. 5. If the expected percentage frame loss for the congested port (T4) is 33% at 150% overload, there is no congestion control detected. Test Result: This test has passed. However, one frame lost has occurred when DUT is receiving 64 and 128 bytes traffic. Test Duration (Seconds) 30 Frame Size (Bytes) ,024 1,280 1,518 Uncongested Test Frames Expected Uncongested Test Frames Received Uncongested Test Frames Lost Head of Line Blocking Exists? Congested Test Frames Expected Congested Test Frames Received 22,321,429 12,668,919 6,793,478 3,524,436 1,795,977 1,442,308 1,219,116 22,321,428 12,668,918 6,793,478 3,524,436 1,795,977 1,442,308 1,219, YES YES NO NO NO NO NO 66,964,285 38,006,755 20,380,434 10,573,308 5,387,931 4,326,922 3,657,346 66,964,282 38,006,754 20,380,434 10,573,308 5,387,931 4,326,922 3,657,346 Congested Test Frames Lost Congested Test Frames Lost (%) % % % % % % % Congestion Control Exists? YES YES YES YES YES YES YES Network Benchmarking Lab 32

33 3.3 Address Caching Capacity Test Test Purpose: To determine the address caching capacity of a LAN switching device. Reference: RFC Benchmarking Terminology for LAN Switching Devices RFC Benchmarking Methodology for LAN Switching Devices Test Tools: AST II 3.5 Test Description: The configuration of this test is shown in Figure 14. The three ports of the DUT (D1-D3) are connected to the three ports of the tester (T1-T3). Port T1 is a learning port, port T2 is a test port, and port T3 is a monitoring port. Firstly, the source port (T2) sends a broadcast frame to the destination port (D2), and the DUT will learn the MAC address from T2. Then T1 sends the learning frames to the first port (D1). Though the source MAC addresses of learning frames are all different, the learning frames only include a fixed destination MAC address, which is the T2 s MAC address. The DUT will learn the source MAC addresses of the learning frames. Finally, the learning port (T2) will send the frames to the DUT s second port (D2), and the destination MAC addresses of the frames were learned from the port D1. Next the DUT will transfer those frames to the test port (T1) and check the learning port (T1) how many frames are received. If the amounts of the frames are more than those sent from the tester, it indicates the unnecessary frames are flood frames. If there are some flood frames on the port T3, the learning port (T1) needs to decrease the amount of learning frames. If there is no flood frame on the port T3, the learning port (T1) needs to increase the amount of learning frames. This test needs to be repeated until finding the maximum capacity of MAC address that the DUT can be learned. Test Configuration: Figure 14 Address caching capacity test configuration Test observation: To observe the maximum number of MAC addresses the DUT can learn. Network Benchmarking Lab 33

34 Test Result: The measured MAC address capacity that the DUT can cache (store) and successfully forward frames without flooding or dropping frames is Network Benchmarking Lab 34

35 3.4 Address Learning Rate Test Test Purpose: To test the maximum MAC address learning rate of the DUT. Reference: RFC Benchmarking Terminology for LAN Switching Devices RFC Benchmarking Methodology for LAN Switching Devices Test Tools: AST II 3.5 Test Description: The configuration of this test is shown in Figure 15. The three ports of the DUT (D1-D3) are connected to the three ports of the tester (T1-T3). Port T1 is a learning port, port T2 is a test port, and port T3 is a monitoring port. Firstly, the source port (T2) sends a broadcast frame to the destination port (D2), and the DUT will learn the MAC address from T2. Then T1 sends the learning frames to the first port (D1). Though the source MAC addresses of learning frames are all different, the learning frames only include a fixed destination MAC address, which is the T2 s MAC address. The DUT will learn the source MAC address of the learning frames. Finally, the learning port (T2) will send the frames to the DUT s second port (D2), and the destination MAC addresses of the frames were learned from the port D1. Next the DUT will transfer those frames to the test port (T1) and check the learning port (T1) how many frames are received. If the amounts of frames are more than those sent from the tester, it indicates the unnecessary frames are flood frames. If there are some flood frames on the port T3, the learning port (T1) needs to decrease the amount of learning frames. If there is no flood frame on the port T3, the learning port (T1) needs to increase the amount of learning frames. This test needs to be repeated until finding the maximum learning rate. Test Configuration: Figure 15 Address learning rate test configuration Test observation: 1. To observe the maximum MAC address learning rate of the DUT. 2. Whether address learning rate can be reached wire-speed. Network Benchmarking Lab 35

36 Test Result: The address learning rate can be reached 1,487,997 frames per second. Network Benchmarking Lab 36

37 3.5 Errored Frame Filtering Test Test Purpose: To determine the behavior of the DUT under errored or abnormal frame conditions. The results of the test indicate whether the DUT filters the errors or simply propagates the errored frames along to the destination. The following types of errored frames are tested. a. Oversize. b. Undersize. c. CRC errors. d. Dribble bit errors. e. Alignment errors. Reference: RFC Benchmarking Terminology for LAN Switching Devices RFC Benchmarking Methodology for LAN Switching Devices Test Tools: AST II 3.5 Test Description: The configuration of this test is shown in Figure 16. The two ports of the DUT (D1-D2) are connected to the three ports of the tester (T1-T2). The tester generates the error frames from T1 to D1, and then the number of frames received at T2 is checked. Errored frames are frames which are over-sized, under-sized, misaligned or with an errored frame check sequence. Switches, unlike IEEE 802.1d compliant bridges, do not necessarily filter all types of illegal frames. Some switches, for example, which do not store frames before forwarding them to their destination interfaces may not filter over-sized frames (jabbers) or verify the validity of the frame check sequence field. Other illegal frames are under-sized frames (runts) and misaligned frames. Oversize - The DUT may filter frames larger than 1518 bytes from being propagated through the DUT. Oversized frames transmitted to the DUT should not be forwarded. DUT supporting tagged Frames may forward frames up to and including 1522 bytes long. Undersize - The DUT must filter frames less than 64 bytes from being propagated through the DUT. Undersized frames (or collision fragments) received by the DUT must not be forwarded. CRC errors - The DUT must filter frames failing the frame check sequence validation from being propagated through the DUT. Frames with an invalid CRC transmitted to the DUT should not be forwarded. Dribble bit errors - The DUT must correct and forward frames containing dribbling bits. Frames transmitted to the DUT that do not end in an octet boundary but contain a valid frame check sequence must be accepted by the DUT and forwarded to the Network Benchmarking Lab 37

38 correct receiving port with the frame ending in an octet boundary. Alignment errors - The DUT must filter frames that fail the frame check sequence validation and do not end in an octet boundary. This is a combination of a CRC error and a dribble bit error. When both errors are occurring in the same frame, the DUT must determine the CRC error takes precedence and prevents the frame from being propagated. Test Configuration: Figure 16 Errored frames filtering test configuration Test observation: 1. When the oversize frames are transmitted to the DUT. The DUT must drop them and the frames should not be forwarded. So the T2 will not receive any frame. 2. When the undersize frames are transmitted to the DUT. The DUT must drop them and the frames should not be forwarded. So T2 will not receive any frame. 3. When the CRC error frames are transmitted to the DUT. The DUT must drop them and the frames should not be forwarded. So the T2 will not receive any frame. 4. The DUT must correct and forward frames containing dribbling bits. So the T2 will not lose any frame. 5. When alignment errors frame are transmitted to the DUT. The DUT must drop them and the frames should not be forwarded. So the T2 will not receive any frame. Test Result: All the error frame tests are passed except oversize error frame. Nevertheless, DUT can still filter frame when frame size is over 1528 (without FCS). A summary table is shown as follow: Test Duration (Seconds) 30 Errored Frame Oversize Undersize CRC Errors Dribble Bit Errors Alignment Errors Frame Size (Bytes) 1, Test Frames Transmitted 243,033 4,518,072 4,464,285 4,464,285 4,464,285 Test Frames Received 243, ,464,285 0 Test Status FAILED PASSED PASSED PASSED PASSED Network Benchmarking Lab 38

39 3.6 Broadcast Frame Forwarding Test Test Purpose: To determine the forwarding rate, frame loss and throughput of the DUT will be determined when the tester transfers broadcast frames to the DUT. Reference: RFC Benchmarking Terminology for LAN Switching Devices RFC Benchmarking Methodology for LAN Switching Devices Test Tools: AST II 3.5 Test Description: The configuration of this test is shown in Figure 17(a). The 8 ports of the DUT (D1-D8) are connected to the 8 ports of the tester (T1-T8). The parameters of the broadcast storm are disabled on the 8 ports of the DUT (D1-D8). When the port T1 of the tester transfers a broadcast frame to the 8 ports (D1-D8) of the DUT, the forwarding rate, frame loss and throughput will be measured. During the test, the tested frame sizes are 64, 128, 256, 512, 1024, 1280, 1518, 9000 bytes, and the test duration is 30 seconds for each iteration. Test Configuration: (b) Broadcast frame forwarding (a) Broadcast frame forwarding testing cable testing traffic distribution. connections. Figure 17 Broadcast frame forwarding test configuration Test observation: 1. To measure the DUT zero-loss maximum forwarding rate when the DUT transfers the broadcast traffic and the DUT doesn t have any frame loss. 2. To measure the DUT whether it can cause frame loss when the tester sends broadcast traffic of maximum offered load traffic to the DUT. Network Benchmarking Lab 39

40 Test Result: This test is passed. A summarized table of the test is given below: Frame Size (Bytes) ,024 1,280 1,518 Test Frames Transmitted Expected Test Frames Received ,048, ,857, ,027,022 81,521,736 42,293,232 21,551,724 17,307,690 14,629,386 Test Frames Received Forwarding Test Frames Lost Rate at Test Frames Lost (%) % % % % % % % Maximum Test Frames/Second Offered 1,488, , , , ,732 96,154 81,274 Transmitted Load Test Frames/Second 8,928,571 5,067,567 2,717,391 1,409, , , ,646 Received Bits/Second Transmitted 999,999, ,999, ,999, ,999, ,999, ,999, ,999,807 Bits/Second Received 5,999,999,712 5,999,999,565 5,999,999,770 5,999,999,846 5,999,999,962 5,999,999,200 5,999,998,845 Network Benchmarking Lab 40

41 3.7 Broadcast Latency Test Test Purpose: To determine the latency when the DUT forwards a broadcast frame. Reference: RFC Benchmarking Terminology for LAN Switching Devices RFC Benchmarking Methodology for LAN Switching Devices Test Tools: AST II 3.5 Test Description: The configuration of this test is shown in Figure 18(a). The 8 ports of the DUT (D1-D8) are connected to the 8 ports of the tester (T1-T8). The parameters of broadcast storm are disabled on 8 ports (D1-D8) of the DUT. When the port T1 of the tester transfers a broadcast frame to the 8 ports (D1-D8) of the DUT, the latency will be measured. During the test, the traffic frame sizes are 64, 128, 256, 512, 1024, 1280, 1518 bytes. Test Configuration: (a)broadcast frame latency testing cable connections. Test observation: (b)broadcast frame latency testing traffic distribution. Figure 18 Broadcast latency test configuration To observe the broadcast latency while the DUT transfers a broadcasting frame to all ports. Network Benchmarking Lab 41

42 Test Result: The latency for each frame is shown as follow: Frame Size (Bytes) ,024 1,280 1,518 Average Latency (us) Network Benchmarking Lab 42

43 3.8 Forward Pressure Test Test Purpose: The forward pressure test overloads a DUT port and measures the output for forward pressure. If the DUT transmits frames with an inter frame gap less than 96 bits, then the forward pressure is detected. Reference: RFC Benchmarking Terminology for LAN Switching Devices RFC Benchmarking Methodology for LAN Switching Devices Test Tools: AST II 3.5 Test Description: The configuration of this test is shown in Figure 19. The two ports of the DUT (D1-D2) are connected to the two ports of the tester (T1-T2). The tester generates the frames with an interframe gap less than 96 bits from T1 to D1, and observes the T2 whether it has any frame loss. Test Configuration: Figure 19 Forward pressure test configuration Test observation: If the port 2 of DUT (T2) had no frame loss, the forward pressure may exist on the DUT. However, if the DUT had frame loss, it indicates the DUT doesn t have forward pressure mechanism. Network Benchmarking Lab 43

44 Test Result: This case is failed; this DUT doesn t support forward pressure mechanism. Frame Size (Bytes) ,024 1,280 1,518 Test Frames Transmitted 4,518,072 2,551,020 1,363, , , , ,981 Test Frames Expected 4,518,072 2,551,020 1,363, , , , ,981 Test Frames Received 4,464,316 2,533,906 1,358, , , , ,849 Test Frames Lost 53,756 17,114 4,858 1, Forward Pressure Exits? NO NO NO NO NO NO NO Network Benchmarking Lab 44