Adding NetFPGA board into CRON Lin Xue, lxue2@tigers.lsu.edu, LANET@LSU April 2011 Change wires table, if you change the switch cable connection Note: this might not related to adding a NetFPGA, but this is the thing we faced before adding a NetFPGA. Whenever you change any cable connection on switch, you should change the wires table, for example, the table records pc4 s connection change information: Node s port Old switch port New switch port Pc4 s eth2 5 37 Pc4 s eth3 7 39 Pc4 s eth6 6 38 Pc4 s eth7 8 40 You should change the wires table: mysql> select * from wires where node_id1='pc4'; cable len type node_id1 card1 port1 node_id2 card2 port2 NULL 0 Control pc4 0 1 controlsw 0 28 NULL 0 Node pc4 2 1 datasw 1 5 NULL 0 Node pc4 3 1 datasw 1 7 NULL 0 Node pc4 6 1 datasw 1 6 NULL 0 Node pc4 7 1 datasw 1 8 5 rows in set (0.00 sec) 1
mysql> update wires set port2 = '37' where node_id1 = 'pc4' and card1 = '2'; Result: mysql> select * from wires where node_id1='pc4'; cable len type node_id1 card1 port1 node_id2 card2 port2 NULL 0 Control pc4 0 1 controlsw 0 28 NULL 0 Node pc4 2 1 datasw 1 37 NULL 0 Node pc4 3 1 datasw 1 7 NULL 0 Node pc4 6 1 datasw 1 6 NULL 0 Node pc4 7 1 datasw 1 8 5 rows in set (0.00 sec) mysql> update wires set port2 = '39' where node_id1 = 'pc4' and card1 = '3'; mysql> update wires set port2 = '38' where node_id1 = 'pc4' and card1 = '6'; 2
mysql> update wires set port2 = '40' where node_id1 = 'pc4' and card1 = '7'; Result: mysql> select * from wires where node_id1='pc4'; cable len type node_id1 card1 port1 node_id2 card2 port2 NULL 0 Control pc4 0 1 controlsw 0 28 NULL 0 Node pc4 2 1 datasw 1 37 NULL 0 Node pc4 3 1 datasw 1 39 NULL 0 Node pc4 6 1 datasw 1 38 NULL 0 Node pc4 7 1 datasw 1 40 5 rows in set (0.00 sec) Add one type for netfpga: Note: set subnode =1! mysql> insert into node_types set class='netfpga', issubnode=1, type='netfpga1g'; Result: mysql> select * from node_types where class='netfpga'; 3
+---------+-----------+-------------------+-------------------+------------+------------+----------+-----------+--------------+----- ------+--------------+----------------+-----------+------------+-----------+----------+--------------+ class type modelnetcore_osid modelnetedge_osid isvirtnode ismodelnet isjailed isdynamic isremotenode issubnode isplabdslice isplabphysnode issimnode isgeninode isfednode isswitch ishdemulator +---------+-----------+-------------------+-------------------+------------+------------+----------+-----------+--------------+----- ------+--------------+----------------+-----------+------------+-----------+----------+--------------+ netfpga netfpga1g NULL NULL 0 0 0 0 0 1 0 0 0 0 0 0 0 +---------+-----------+-------------------+-------------------+------------+------------+----------+-----------+--------------+----- ------+--------------+----------------+-----------+------------+-----------+----------+--------------+ 1 row in set (0.00 sec) Add one new netfpga node into nodes table: Note: this netfpga is not the NetFPGA host, it is the subnode netfpag! phys_nodeid should be the parent node! mysql> insert into nodes set node_id='netfpga1', type='netfpga1g', phys_nodeid= pcnetfpga1g, role='testnode'; Result: mysql> select * from nodes where node_id='netfpga1'; +----------+-----------+-------------+----------+-----------+---------------+---------------+-------------------+----------------+-- --------------+----------------+--------------------+---------------+------+--------+----------+------------+-------------+-------+- ---------+------------+--------+------------------+---------------+------------+------------+-----------------+---------+------------ -------+------------+----------------------+-----------------+--------------+----------+------+----------+------------+------------- +-------------+----------+----------+--------+-----------+------------+--------+------------+-----------------+-------------------- +-------------------+------------+---------------+---------------+-------------------------+--------------+------+ node_id type phys_nodeid role inception def_boot_osid def_boot_path def_boot_cmd_line temp_boot_osid next_boot_osid next_boot_path next_boot_cmd_line pxe_boot_path rpms deltas tarballs startupcmd startstatus ready priority bootstatus 4
status status_timestamp failureaction routertype eventstate state_timestamp op_mode op_mode_timestamp allocstate allocstate_timestamp update_accounts next_op_mode ipodhash osid ntpdrift ipport_low ipport_next ipport_high sshdport jailflag jailip sfshostid stated_tag rtabid cd_version battery_voltage battery_percentage battery_timestamp boot_errno destination_x destination_y destination_orientation reserved_pid uuid +----------+-----------+-------------+----------+-----------+---------------+---------------+-------------------+----------------+-- --------------+----------------+--------------------+---------------+------+--------+----------+------------+-------------+-------+- ---------+------------+--------+------------------+---------------+------------+------------+-----------------+---------+------------ -------+------------+----------------------+-----------------+--------------+----------+------+----------+------------+------------- +-------------+----------+----------+--------+-----------+------------+--------+------------+-----------------+-------------------- +-------------------+------------+---------------+---------------+-------------------------+--------------+------+ netfpga1 netfpga1g pcnetfpga1g testnode NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL 0-1 unknown NULL NULL fatal none NULL NULL NULL NULL NULL NULL 0 NULL NULL NULL 11000 11000 20000 11000 0 NULL NULL NULL 0 NULL NULL NULL NULL 0 NULL NULL NULL NULL +----------+-----------+-------------+----------+-----------+---------------+---------------+-------------------+----------------+-- --------------+----------------+--------------------+---------------+------+--------+----------+------------+-------------+-------+- ---------+------------+--------+------------------+---------------+------------+------------+-----------------+---------+------------ -------+------------+----------------------+-----------------+--------------+----------+------+----------+------------+------------- +-------------+----------+----------+--------+-----------+------------+--------+------------+-----------------+-------------------- +-------------------+------------+---------------+---------------+-------------------------+--------------+------+ 1 row in set (0.00 sec) Add an interface type for nf2 mysql> insert interface_types set type='nf2',max_speed='1000000',full_duplex='1',manufacturer='xilinx' ->,model='netfpga',ports='1',connector='rj45'; Result: mysql> select * from interface_types; +------+-----------+-------------+--------------+-------------------+-------+-----------+ 5
type max_speed full_duplex manufacturer model ports connector +------+-----------+-------------+--------------+-------------------+-------+-----------+ nfe 1000000 1 nvidia MCP55 1G Ethernet 1 RJ45 mxge 10000000 1 Myricom 10G-PCIE-8_L-S 1 RJ45 fxp 100000 1 Intel EtherExpress Pro 1 RJ45 nf2 1000000 1 Xilinx NetFPGA 4 RJ45 +------+-----------+-------------+--------------+-------------------+-------+-----------+ Generate UUID for each interfaces: boss# uuidgen -n 4 9baf5154-437f-11e0-b681-002128000a06 9baf5155-437f-11e0-b681-002128000a06 9baf5156-437f-11e0-b681-002128000a06 9baf5157-437f-11e0-b681-002128000a06 Add four interfaces: Note: node_id of these interfaces should be set to the subnode netfpga, not the netfpga host! mysql> insert into interfaces set node_id='netfpga1',card='1',port='1',mac='004e46324300',ip='',interface_type='nf2',iface='eth1',role='e xpt',current_speed='1000',duplex='full',uuid='9baf5154-437f-11e0-b681-002128000a06'; mysql> insert into interfaces set node_id='netfpga1',card='2',port='1',mac='004e46324301',ip='',interface_type='nf2',iface='eth2',role='e xpt',current_speed='1000',duplex='full',uuid='9baf5155-437f-11e0-b681-002128000a06'; 6
mysql> insert into interfaces set node_id='netfpga1',card='3',port='1',mac='004e46324302',ip='',interface_type='nf2',iface='eth3',role='e xpt',current_speed='1000',duplex='full',uuid='9baf5156-437f-11e0-b681-002128000a06'; mysql> insert into interfaces set node_id='netfpga1',card='4',port='1',mac='004e46324303',ip='',interface_type='nf2',iface='eth4',role='e xpt',current_speed='1000',duplex='full',uuid='9baf5157-437f-11e0-b681-002128000a06'; Result: mysql> select * from interfaces where node_id='pcnetfpga1g' or node_id='netfpga1'; +-------------+------+------+--------------+--------------+-----------+------+----------------+-------+------+---------------+----- ---+--------+----------+------+-------+--------------------------------------+ node_id card port mac IP IPaliases mask interface_type iface role current_speed duplex rtabid vnode_id whol trunk uuid +-------------+------+------+--------------+--------------+-----------+------+----------------+-------+------+---------------+----- ---+--------+----------+------+-------+--------------------------------------+ pcnetfpga1g 0 1 0002a5ebff4b 192.168.0.99 NULL NULL fxp eth0 ctrl 100000 full 0 NULL 0 0 b33051fa-3eca-11e0-a8b5-002128000a06 netfpga1 1 1 004E46324300 NULL NULL nf2 eth1 expt 0 full 0 NULL 0 0 9baf5154-437f-11e0-b681-002128000a06 netfpga1 2 1 004E46324301 NULL NULL nf2 eth2 expt 0 full 0 NULL 0 0 9baf5155-437f-11e0-b681-002128000a06 netfpga1 3 1 004E46324302 NULL NULL nf2 eth3 expt 0 full 0 NULL 0 0 9baf5156-437f-11e0-b681-002128000a06 netfpga1 4 1 004E46324303 NULL NULL nf2 eth4 expt 0 full 0 NULL 0 0 9baf5157-437f-11e0-b681-002128000a06 +-------------+------+------+--------------+--------------+-----------+------+----------------+-------+------+---------------+----- ---+--------+----------+------+-------+--------------------------------------+ 7
5 rows in set (0.00 sec) I made a mistake, we should set the iface name as nf2c not eth mysql> update interfaces set iface='nf2c0' where node_id='netfpga1' and card='1'; mysql> update interfaces set iface='nf2c1' where node_id='netfpga1' and card='2'; mysql> update interfaces set iface='nf2c2' where node_id='netfpga1' and card='3'; mysql> update interfaces set iface='nf2c3' where node_id='netfpga1' and card='4'; mysql> select * from interfaces where node_id='netfpga1'; +----------+------+------+--------------+------+-----------+------+----------------+-------+------+---------------+--------+------ --+----------+------+-------+--------------------------------------+ node_id card port mac IP IPaliases mask interface_type iface role current_speed duplex rtabid vnode_id whol trunk uuid +----------+------+------+--------------+------+-----------+------+----------------+-------+------+---------------+--------+------ --+----------+------+-------+--------------------------------------+ 8
netfpga1 1 1 004E46324300 NULL NULL nf2 nf2c0 expt 0 full 0 NULL 0 0 9baf5154-437f-11e0-b681-002128000a06 netfpga1 2 1 004E46324301 NULL NULL nf2 nf2c1 expt 0 full 0 NULL 0 0 9baf5155-437f-11e0-b681-002128000a06 netfpga1 3 1 004E46324302 NULL NULL nf2 nf2c2 expt 0 full 0 NULL 0 0 9baf5156-437f-11e0-b681-002128000a06 netfpga1 4 1 004E46324303 NULL NULL nf2 nf2c3 expt 0 full 0 NULL 0 0 9baf5157-437f-11e0-b681-002128000a06 +----------+------+------+--------------+------+-----------+------+----------------+-------+------+---------------+--------+------ --+----------+------+-------+--------------------------------------+ 4 rows in set (0.00 sec) Interface_capabilities Set interface capabilities for nf2 interface: mysql> select * from interface_capabilities; +------+--------------------+----------+ type capkey capval +------+--------------------+----------+ nfe protocols ethernet nfe ethernet_defspeed 1000000 nfe ethernet_auxspeeds 100000 mxge protocols ethernet mxge ethernet_defspeed 10000000 fxp protocols ethernet fxp ethernet_defspeed 100000 nf2 ethernet_defspeed 1000000 nf2 protocols ethernet +------+--------------------+----------+ 9
9 rows in set (0.00 sec) update interface_state mysql> insert into interface_state set node_id='netfpga1',card='4',port='1',iface='eth4',enabled='1',tagged='0',remaining_bandwidth='0'; mysql> insert into interface_state set node_id='netfpga1',card='3',port='1',iface='eth3',enabled='1',tagged='0',remaining_bandwidth='0'; mysql> insert into interface_state set node_id='netfpga1',card='2',port='1',iface='eth2',enabled='1',tagged='0',remaining_bandwidth='0'; Result: mysql> select * from interface_state; +-------------+------+------+-------+---------+--------+---------------------+ node_id card port iface enabled tagged remaining_bandwidth +-------------+------+------+-------+---------+--------+---------------------+ pc5 0 1 eth0 1 0 0 netfpga1 1 1 eth1 1 0 0 pcnetfpga1g 0 1 eth0 1 0 0 netfpga1 4 1 eth4 1 0 0 10
netfpga1 3 1 eth3 1 0 0 netfpga1 2 1 eth2 1 0 0 +-------------+------+------+-------+---------+--------+---------------------+ Make the same change as explained above: mysql> update interface_state set iface='nf2c0' where node_id='netfpga1' and card='1'; mysql> update interface_state set iface='nf2c1' where node_id='netfpga1' and card='2'; mysql> update interface_state set iface='nf2c2' where node_id='netfpga1' and card='3'; mysql> update interface_state set iface='nf2c3' where node_id='netfpga1' and card='4'; mysql> select * from interfaces where node_id='netfpga1'; +----------+------+------+--------------+------+-----------+------+----------------+-------+------+---------------+--------+------ --+----------+------+-------+--------------------------------------+ node_id card port mac IP IPaliases mask interface_type iface role current_speed duplex rtabid vnode_id whol trunk uuid +----------+------+------+--------------+------+-----------+------+----------------+-------+------+---------------+--------+------ --+----------+------+-------+--------------------------------------+ 11
netfpga1 1 1 004E46324300 NULL NULL nf2 nf2c0 expt 0 full 0 NULL 0 0 9baf5154-437f-11e0-b681-002128000a06 netfpga1 2 1 004E46324301 NULL NULL nf2 nf2c1 expt 0 full 0 NULL 0 0 9baf5155-437f-11e0-b681-002128000a06 netfpga1 3 1 004E46324302 NULL NULL nf2 nf2c2 expt 0 full 0 NULL 0 0 9baf5156-437f-11e0-b681-002128000a06 netfpga1 4 1 004E46324303 NULL NULL nf2 nf2c3 expt 0 full 0 NULL 0 0 9baf5157-437f-11e0-b681-002128000a06 +----------+------+------+--------------+------+-----------+------+----------------+-------+------+---------------+--------+------ --+----------+------+-------+--------------------------------------+ Change wires table: Note:change wires table according to the connection between NetFPGA and switch, for example, now the connection between NetFPGA and switch is like: Node s port Switch port Netfpga s port 1 5 Netfpga s port 2 6 Netfpga s port 3 7 Netfpga s port 4 8 Set node_id1 to the subnode type! mysql> insert into wires set type='node', node_id1='netfpga1',card1='1', port1='1',node_id2='datasw',card2='1',port2='5'; mysql> insert into wires set type='node', node_id1='netfpga1',card1='2', port1='1',node_id2='datasw',card2='1',port2='6'; mysql> insert into wires set type='node', node_id1='netfpga1',card1='3', port1='1',node_id2='datasw',card2='1',port2='7'; 12
mysql> insert into wires set type='node', node_id1='netfpga1',card1='4', port1='1',node_id2='datasw',card2='1',port2='8'; Result: mysql> select * from wires where node_id1='pcnetfpga1g' or node_id1= 'netfpga1' ; +-------+-----+---------+-------------+-------+-------+-----------+-------+-------+ cable len type node_id1 card1 port1 node_id2 card2 port2 +-------+-----+---------+-------------+-------+-------+-----------+-------+-------+ NULL 0 Control pcnetfpga1g 0 1 controlsw 0 37 NULL 0 Node pcnetfpga1g 1 1 datasw 1 5 NULL 0 Node pcnetfpga1g 2 1 datasw 1 6 NULL 0 Node pcnetfpga1g 3 1 datasw 1 7 NULL 0 Node pcnetfpga1g 4 1 datasw 1 8 +-------+-----+---------+-------------+-------+-------+-----------+-------+-------+ 5 rows in set (0.00 sec) node_type_features mysql> select * from node_type_features; +-------------+-----------------+--------+ type feature weight +-------------+-----------------+--------+ ciscoserver hosts-netfpga1g 0 +-------------+-----------------+--------+ 13
1 row in set (0.00 sec) The NetFPGA information from Utah site [linxue@nfhost ~]$ ifconfig eth0 Link encap:ethernet HWaddr 00:11:43:E4:3B:63 inet addr:155.98.39.28 Bcast:155.98.39.255 Mask:255.255.252.0 inet6 addr: fe80::211:43ff:fee4:3b63/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:6501 errors:0 dropped:0 overruns:0 frame:0 TX packets:1160 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:614288 (599.8 KiB) TX bytes:136893 (133.6 KiB) Base address:0xdcc0 Memory:d86e0000-d8700000 lo Link encap:local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:74 errors:0 dropped:0 overruns:0 frame:0 TX packets:74 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:10174 (9.9 KiB) TX bytes:10174 (9.9 KiB) nf2c0 Link encap:ethernet HWaddr 00:4E:46:32:43:00 inet6 addr: fe80::24e:46ff:fe32:4300/64 Scope:Link 14
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) Interrupt:28 nf2c1 Link encap:ethernet HWaddr 00:4E:46:32:43:01 inet6 addr: fe80::24e:46ff:fe32:4301/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) Interrupt:28 nf2c2 Link encap:ethernet HWaddr 00:4E:46:32:43:02 inet6 addr: fe80::24e:46ff:fe32:4302/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) Interrupt:28 15
nf2c3 Link encap:ethernet HWaddr 00:4E:46:32:43:03 inet6 addr: fe80::24e:46ff:fe32:4303/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) Interrupt:28 16