PUBLICATION. PCIExpress Hot-Plug Mechanism in Linux-based ATCA Control Systems

Transcription

1 EuCARD-PRE European Coordination for Accelerator Research and Development PUBLICATION PCIExpress Mechanism in Linux-based ATCA Control Systems Piotrowski, A (TUL) 28 October 2010 The research leading to these results has received funding from the European Commission under the FP7 Research Infrastructures project EuCARD, grant agreement no This work is part of EuCARD Work Package 4: AccNet: Accelerator Science Networks. The electronic version of this EuCARD Publication is available via the EuCARD web site < or on the CERN Document Server at the following URL : < EuCARD-PRE

2 Mechanism in Linux Operating System for FPGA-based s A. Piotrowski PCIExpress

3 Mechanism in Linux Operating System for FPGA-based s Outline 1 Mechanism 2 in Linux Operating System 3 for FPGA-based s 4 A. Piotrowski PCIExpress

4 Mechanism in Linux Operating System for FPGA-based s Mechanism /Hot-Swap solution provide methods to replace modules without turning system off, keeping operating system services running correctly after component removal and restarting or shutting down software associated to removed device. A. Piotrowski PCIExpress

5 Mechanism in Linux Operating System for FPGA-based s Hardware Support for Mechanism User Controller Driver 1 Operating System Driver 2 Driver 3 Driver 4 Service Driver Controller 1 Root Complex Controller 2 Switch A. Piotrowski PCIExpress

6 Mechanism in Linux Operating System for FPGA-based s Hardware Support for Mechanism User Controller Card Slot Power Switching and Card Reset Logic Driver 1 Operating System Driver 2 Controller 1 Driver 3 Driver 4 Root Complex Service Driver Controller 2 Switch A. Piotrowski PCIExpress

7 Mechanism in Linux Operating System for FPGA-based s Hardware Support for Mechanism User Controller Card Slot Power Switching and Card Reset Logic Power Indicator and Attention Indicator Driver 1 Operating System Driver 2 Controller 1 Driver 3 Driver 4 Root Complex Service Driver Controller 2 Switch A. Piotrowski PCIExpress

8 Mechanism in Linux Operating System for FPGA-based s Hardware Support for Mechanism User Controller Card Slot Power Switching and Card Reset Logic Power Indicator and Attention Indicator Attention Button Driver 1 Operating System Driver 2 Controller 1 Driver 3 Switch Driver 4 Root Complex Service Driver Controller 2 A. Piotrowski PCIExpress

9 Mechanism in Linux Operating System for FPGA-based s Hardware Support for Mechanism User Controller Card Slot Power Switching and Card Reset Logic Power Indicator and Attention Indicator Attention Button Card Preset Detection Pins Driver 1 Operating System Driver 2 Controller 1 Driver 3 Switch Driver 4 Root Complex Service Driver Controller 2 A. Piotrowski PCIExpress

10 Mechanism in Linux Operating System for FPGA-based s Software Support for Mechanism User Operating System Service Service Driver 1 Driver 2 Controller 1 Driver 3 Driver 4 Root Complex Driver Controller 2 Switch A. Piotrowski PCIExpress

11 Mechanism in Linux Operating System for FPGA-based s Software Support for Mechanism User Operating System Service Service Driver Driver 1 Driver 2 Controller 1 Driver 3 Driver 4 Root Complex Driver Controller 2 Switch A. Piotrowski PCIExpress

12 Mechanism in Linux Operating System for FPGA-based s Software Support for Mechanism User Operating System Service Service Driver Drivers Driver 1 Driver 2 Controller 1 Driver 3 Driver 4 Root Complex Driver Controller 2 Switch A. Piotrowski PCIExpress

13 Mechanism in Linux Operating System for FPGA-based s Software Support for Mechanism User Operating System Service Service Driver Drivers User Driver 1 Driver 2 Controller 1 Driver 3 Driver 4 Root Complex Driver Controller 2 Switch A. Piotrowski PCIExpress

14 Mechanism in Linux Operating System for FPGA-based s PCI Driver Model vs. PCI Express Driver Model Standard PCI Driver Model allows to load one driver for one device Standard PCI Express s support up to four different functions PCI Express Bus Driver was designed to support PCI Express functionalities in PCI Driver Model A. Piotrowski PCIExpress

15 Mechanism in Linux Operating System for FPGA-based s PCI Express Bus Driver Bus Driver Power Management Virtual Channel Advanced Error Reporting Root PCI Express Root Complex Root Root Up PCI Express Switch Down Down Down PME PME HP Bus Driver Bus Driver AER Bus Driver VC VC VC VC AER HP VC VC A. Piotrowski PCIExpress

16 Mechanism in Linux Operating System for FPGA-based s PCI Express Bus Driver sysfs file name service description 0000:00:01.0:pcie :00:01.0:pcie :00:1c.0:pcie :00:1c.0:pcie :00:1c.0:pcie :04:00.0:pcie :04:00.0:pcie :05:08.0:pcie :05:08.0:pcie :05:08.0:pcie :05:09.0:pcie :05:09.0:pcie :05:09.0:pcie28 PME on first Root VC on first Root PME on second Root HP on second Root VC on second Root AER on switch upstream port VC on switch upstream port AER on switch first downstream port HP on switch first downstream port VC on switch first downstream port AER on switch second downstream port HP on switch second downstream port VC on switch second downstream port PCI Express Root Complex Root Root Root Up PCI Express Switch Down Down Down PME PME Bus Driver HP Bus Driver Bus Driver AER Bus Driver VC VC VC VC AER HP VC VC A. Piotrowski PCIExpress

17 Mechanism in Linux Operating System for FPGA-based s for FPGA-based s Reconfiguration of FPGA-based PCI Express Endpoint does not emit standard event, After reprogramming, PCI Express Endpoint is in uninitialized state, Reenumeration of PCI Express Bus must be force manually by user, Fake Driver from newest version of Linux kernel is able to correctly handle reinitialization of FPGA-based device A. Piotrowski PCIExpress

18 Mechanism in Linux Operating System for FPGA-based s Fake Driver Fake Driver is able to logically remove device from the system: echo 0 > /sys/bus/pci/slots/0000:03:00.0/power Fake Driver is able to force enumeration of the device: echo 1 > /sys/bus/pci/slots/0000:00:1c.0/power A. Piotrowski PCIExpress

19 Mechanism in Linux Operating System for FPGA-based s Example? Event udev Rules Report error Report success A. Piotrowski PCIExpress

20 Mechanism in Linux Operating System for FPGA-based s Example? Event udev Rules echo 0 > /sys/bus/pci/slots/ 0000:03:00.0/power echo 1 > /sys/bus/pci/slots/ 0000:00:1c.0/power Report error Report success A. Piotrowski PCIExpress

21 Mechanism in Linux Operating System for FPGA-based s Example - udev Rules? Event udev rules udev ACTION=="add", Rules SUBSYSTEM=="pci", KERNEL=="0000:03:00.0", echo 0 > /sys/bus/pci/slots/ ATTR{vendor}=="0x10EE", ATTR{device}=="0x0008", 0000:03:00.0/power RUN+="/bin/bash /etc/develop/check_firmware 0x10EE0008" echo 1 > /sys/bus/pci/slots/ ACTION=="add", SUBSYSTEM=="pci", KERNEL=="0000:02:00.0", 0000:00:1c.0/power ATTR{vendor}=="0x10EE", ATTR{device}=="0x009", RUN+="/bin/bash /etc/develop/check_firmware 0x10EE0018" Report error Report success A. Piotrowski PCIExpress

22 Mechanism in Linux Operating System for FPGA-based s Example - device parameters parameters udevadm info --path=/sys/bus/pci/devices/0000\:03\:00.0/ --attribute-walk looking at device '/devices/pci0000:00/0000:00:1c.0/ Event 0000:03:00.0':? KERNEL=="0000:03:00.0" SUBSYSTEM=="pci" DRIVER=="" udev Rules ATTR{vendor}=="0x10ee" echo 0 > /sys/bus/pci/slots/ ATTR{device}=="0x0008" 0000:03:00.0/power ATTR{subsystem_vendor}=="0x10ee" ATTR{subsystem_device}=="0x0007" echo 1 > /sys/bus/pci/slots/ ATTR{class}=="0x050000" 0000:00:1c.0/power ATTR{irq}=="255" ATTR{local_cpus}=="ffffffff" ATTR{local_cpulist}=="0-31" ATTR{modalias}=="pci:v000010EEd sv EEsd bc05sc00i00" ATTR{enable}=="0" ATTR{broken_parity_status}=="0" Report Report ATTR{msi_bus}=="" error success A. Piotrowski PCIExpress

23 Mechanism in Linux Operating System for FPGA-based s Example - bash script./load_simple_driver udev Rules echo 0 > /sys/bus/pci/slots/ VER_R=$(./pcie-rw /dev/pcie_bar_0 r 0i 0000:03:00.0/power 1i h) if [ ${1}!= ${VER_R} ]; then logger "Wrong firmware version (${VER_R})" echo 1 > /sys/bus/pci/slots/ 0000:00:1c.0/power echo 0 > /sys/bus/pci/slots/0000:00:1c.0/power else logger "Loaded firmware version ${1}" fi Report error Event #!/bin/bash? cd /etc/develop/./unload_simple_driver cd - Report success bash script A. Piotrowski PCIExpress

24 Mechanism in Linux Operating System for FPGA-based s New version of Fake Driver is able to correctly handle reprogramming of the FPGA-based PCI Express Endpoint device, udev Manager ability to execute additional scripts during Event processing allows to increase safety of the system A. Piotrowski PCIExpress

25 Mechanism in Linux Operating System for FPGA-based s Thank You A. Piotrowski PCIExpress

26 xtca for Instrumentation special session in cooperation with EuCARD AccNet RFTech EU Project Report of RFTech presentation during Mixdes conference (June 26-27, Wroclaw, Poland) Title of presentation: PCIExpress Mechanism in Linux-based ATCA Control Systems Presenter: Adam Piotrowski (PhD student) Department of Microelectronics and Computer Science Technical University of Lodz Lodz, Poland 17 th International Conference Mixed Design of Integrated Circuits and Systems Wrocław, June 2010, Poland

27 xtca for Instrumentation special session in cooperation with EuCARD AccNet RFTech EU Project A. Piotrowski presented an overview of PCIExpress mechanism and its implementation in kernel of Linux operating system. The problem with cooperation between FPGA-based PCIExpress endpoints and was discussed. Additionally, example system that allows to connect udev device manager with user application to check correctness of loaded firmware was presented. Abstract: PCI Express architecture is a widely used communication bus designed for industrial application. Additionally, according to PICMG 3.4 specification it is a part of ATCA architecture. One of the features offered by PCI Express standard is possibility of replacing the system components without shutting down entire system. In this paper, authors present general overview of implementation in Linux operating system used in ATCA carrier board. Index Term: PCIExpress,, ATCA, Advanced Telecommunications Computing Architecture, AdvancedTCA, Linux Remarks: During the conference author was able to discuss presented solutions with people directly involved in development of ATCA-based systems with implemented PCIExpress bus. We acknowledge funding from the European Commission under the FP7 Research Infrastructures project EuCARD, grant agreement no th International Conference Mixed Design of Integrated Circuits and Systems Wrocław, June 2010, Poland