Introduction to MicroTCA

Introduction to MicroTCA Ray Larsen SLAC MicroTCA Review 1

Outline I. MicroTCA for Physics Basics II. Physics System Requirements III. COTS & Lab Developments IV. xtca Market Growth Projections V. Summary Conclusions 2

I. MicroTCA Basics 3

PICMG-ese Term PICMG ATCA Carrier Shelf RTM/µRTM AMC Micro/µTCA MCH PU, CU IPMI Shelf Mgr Wide, High xtca Definition PCI Industrial Computer Manufacturers Group, 250 corporations Advanced Telecommunications Computing Architecture large board ATCA or µtca board that supports smaller standard board Crate, ATCA (large) or µtca (small) Rear/Micro Rear Transition Module Advanced Mezzanine Card mounting on ATCA Carrier, µtca shelf Crate designed to support AMCs directly MicroTCA Carrier Hub switch module for µtca shelf Power Unit (Module), Cooling Unit (fan or fan tray) Intelligent Platform Management Interface Shelf board hosting IPMI controller (BMC, MMC controllers) High (vertical module height), Wide (front panel width) ACTA and /or MicroTCA standard platforms 4

MicroTCA Basics First a word about ATCA ATCA board, shelf is first modular computer architecture with completely serial multi-gbps backplane, dual star or mesh Board designed with standard Mezzanine card (AMC) for ease in upgrading as technology evolves (Moore s Law) ATCA managed system (IPMI), standard diagnostics detect problems at ATCA, AMC levels; take remedial action to evade machine/system interruption Target Availability for ATCA shelf (crate) is 0.99999 or greater In Telecom automated load switchover is common in systems of many identical processor blades Less practical in accelerator, but less required for subsystems with high redundancy 5

ATCA Shelf Carrier ATCA Dual Star 14-Slot Shelf ATCA Carrier w/ 4 AMCs 6

ATCA Fabric: Dual Star & Mesh 7

ATCA System Features, Extensions ATCA Board and RTM Designed for ATCA board hot swap from front Designed so all or most I/O from rear via RTM IPM system isolates problem board, calls maintenance Main board or AMC swapped while RTM remains in place IPM returns to service with no shelf-wide interruption Extensions for Physics - RTM PICMG 3.0 RTM had no management features, standard connectors, keying Physics Committee designed PICMG3.8 with IPM power management, high density fabric I/O connectors, JTAG 8

PICMG3.8 RTM Interface Standard Rear View 120 - IO Channels (3x40) IPMI, Power Connector (blue) 2 Mechanical Keys Courtesy M. Huffer et al, SLAC 9

IPMI System Basics 10

IPMI ~1990 pre-dates ATCA for use in Mainframe computers, PCs etc. Architecture, protocols, chip sets developed by Intel, HP, NEC, DELL 11

MicroTCA Shelf ATCA Carrier Board Transitions to AMC Shelf Segment of ATCA committee developed AMC shelf to lower infrastructure costs Key was 1-wide multi-tongue MCH combining star switching, shelf manager and IPMI communications MicroTCA shelves developed: 1U, 2U, 3U, horizontal and vertical card orientation; no single shelf format Double wide AMC had space for RTM but undeveloped 12

MicroTCA Infrastructure for Physics Physics TC declared 2-wide w/ RTM essential 3X larger clean analog design space than 1-wide AMC alone Rear I/O, hot-swap options Technical Committee design included: 2-wide AMC, mirror image RTM AMC extended IPMI management to RTM for power, temperature, user diagnostics, hot swap 3x30-pair fabric balanced line data connectors (same style as ATCA) Separate new connector for all RTM power and IPMI control Mechanical & E-keying, JTAG Hot swap handles, indicators, for both AMC, RTM Major Goal: Higher level of interoperability of lab-industry products by defining I/Os of generic AMCs to support multiple applications 13

AMC-RTM Mechanics-Front/Rear Panels Module sizes 14

MTCA.4 Mechanical Details 15

16

MTCA.4 Full Payload Shelf RFQ developed for Industry Infrastructure costs least when all payload slots filled Hot swap dictates all components be plug-modular including power and cooling Features: 12 payload slots mid-size front panel 2-wide modules Push-pull cooling bottom to top, front to back with redundant controllable hot swappable fans Redundant intelligent power modules of 1 KW nominal (smaller if not needed) Full function 4-tongue MCHs with Shelf Manager Dual star backplane with special layer for timing, triggering, point-to-point Designation of spare lines (extended options region) for switched point-topoint low jitter timing (~1 psec) 17

MTCA.4 AMC-RTM-Shelf Concept FAN TRAY-HOTSWAP AIR OUT FRONT END MODULE ADVANCED MEZZANINE CARD (AMC) I/ O B P C O N N I/ O B A C K PL A N E text INPUT/OUTPUT SIGNAL CONDITIONING REAR TRANSITION MODULE (RTM) USER DEFINED I/O CONNECTORS AIR IN FAN TRAY-HOT SWAP 18

12 Payload Slots 2-Wide w/ RTM Dual Star Dual MCHs Dual PM, CU 19

MTCA.4 Backplane Extensions 12 Slot MTCA.4 Backplane Point-to- Point Links 8 x M-LVDS: Trigger 20

MTCA.4 Clock, Trigger & Interlocks 21

IPMI Extension to RTM When adding RTM, AMC supplies all power, management 80W max was spit to 50W AMC, 30W RTM New connector pins sized accordingly Extension IPMI to RTM required protocol compatibility w/ ATCA ATCA committee worked on parallel effort for managed RTM, but no standard connectors as done for MTCA.4 Physics committee assured protocol compliance Following slides show implementation 22

IPMI Extensions Additional RTM control signals for MTCA.4 MP (+3.3V) PP (+12V) Power On Enable Presence MCH 1..2 PP (+12V) MP (+3.3V) PP (+12V) MP (+3.3V) Enable MMC Presence Presence AMC 1..12 µ RTM 1..12 MP (+3.3V) PP (+12V) EMMC PM 1..4 Enable Presence Backplane CU 1..2 CU Front 1..2 fans Rear Rear cooling fans 23

IPMI Extensions 2 Management extensions in MTCA.4 IPMB-L Connects the MCMC on the MCH to the MMC on the AMC Modules Radial architecture IPMB-0 Connects the MCMC on the MCH to the EMMC on the PM and CU Bused architecture I2C-bus Connects the AMC to the µrtm The µrtm is treated as managed FRU of the AMC 24

II. Physics System Requirements 25

Physics Requirements 1 Control Systems MTCA.4 optimized for controls requirements Basic Infrastructure: Powered shelf star, dual star backplane MCH, Processor, IPMI Timing Generator/Receiver Applications AMCs Generic- 3 Types: Fast multichannel high resolution low noise ADC-DAC FPGA programmable Processor Industry Pack Carrier Each leverages multiple applications with single design Strategy: Develop min. 2 suppliers each generic AMC 26

Physics Requirements 2 Applications Specific RTMs Generics can serve multiple requirements via RTMs Application specific RTMs vary in type of design skill GHz Low Level RF, high performance wide dynamic range Low speed <60 MHz 12 bit analog interlock fault detection industrial monitoring and control -- vacuum, temperature, pressure, voltage, current RTMs also can be adaptable, programmable Strategy: Build RTMs to standard generic AMC interfaces; develop COTS sources when quantities justify 27

Physics Requirements 3 AMC-RTM Examples Generic AMC Fast 16 bit ADC-DAC Typical RTMs LLRF Feedback Beam Position Monitor Cavity BPM Toroid Precision Charge FPGA Medium Performance Fast-Slow12 bit ADC Interlocks, FPGA RF Interlocks Beam Loss Monitor Beam Length Monitor 3- Industry Pack Carrier Motor Controllers Wire scanners Vacuum monitor/control Temperature monitor/control 28

III. COTS & Lab Developments 29

Shelves 3 Models 6 Payload slot development unit half-rack 6U Produced early while main production unit being designed Power and cooling non-standard, incompatible New units do have Physics backplane timing layer Schroff, ELMA 12 Payload slot production rack-mount 8U Fully redundant dual star, MCH, power & cooling units Schroff, ELMA, PT (Performance Tech) 6 Payload slot production rack-mount 2U Fully redundant dual star production unit, 3 slots w/rtms PowerBridge to Schroff specification (First prototype delivered) 30

Shelves 12 Slot Fully Redundant 40GbE PT Performance Tech 6-Slot Development Non-Redundant Schroff, Elma 12-Slot Fully Redundant Schroff, Elma 6 Slot Non redundant 3 slots w/ RTMs PowerBridge 31

MCH Carrier Manager and Data Hub Standard MCHs Support all hub switch, IPMI operations in Physics backplane except radial extended options clock lines which SLAC currently does not need Currently evaluated by SLAC are NAT, Vadatech Some firmware differences were resolved to meet MTCA.4 standards NAT currently developing 1 psec jitter switch card for radial lines for DESY Vadatech will also produce this option in future 32

MCH Units N.A.T MCH Manages 12 AMCs, 2 cooling units, 1-4 power modules Supports PCIe, Serial Rapid IO Vadatech UTC002 Currently using in LLRF System tests Fully featured, compliant, preferred front panel network I/O 33

Processors All are single wide totally compatible with ATCA carriers Processors evaluated to date are: AdLink AMC 1000 Core 2 Duo Kontron AM4020M i7 Other processors available Concurrent, Radisys, GE, etc. On order Vadatech AMC720 i7 (delayed due to silicon late delivery) 34

Timing Interim Solution: PMC Adapter Existing solution for LLRF test system and lab prototyping uses MRF (Micro Research Finland) EVR PMC adapter. Not a permanent solution as all cabling is via external coaxes and SMA or Lemo connectors; timing backplane bypassed; however works fine for test purposes New highly efficient solutions discussed in following slides MRF EVR on Vadatech PMC Adapter 35

Timing Modules DESY-Stockholm Generator-Receiver Single wide unit operational for over a year at DESY SLAC has a unit but not implemented Not compatible with SLAC s MRF (Micro-Research Finland) EVR (Event Receiver) timing network protocol Gen 2 Prototype with RTM 2 nd Generation 2-wide with RTM nearing completion Generator will be optional plug-in daughter-card Additional I/O Tx-Rx options via RTM To be produced commercially 36

Timing AMC-µRTM 1&2-Wide MTCA.4 Both single & double-wide MTCA.4 compliant Double-wide RTM allows rear expansion to multiple receivers Accesses all parallel, point-to-point serial switched lines SW Committee looking into standardizing protocols MTCA.4 Backplane New Developments in I&C Standards for Physics 37

Timing Modules 2 International Technologies (I-Tech) Receiver EVRX 3 EVR & MTCA.4 Compatible single wide in prototype stage Will also be configurable as generator To be shown at xtca Workshop June 9-10 2012 Berkeley EVRX PC Board Top View 38

I-Tech Timing EVRX EVRX 3D Models Courtesy International Technologies 39

Example Timing Solution - BPMs Current solution: 4 network cables per BPM BPMs are rackmount Pizza boxes with 4 long timing & data network cables per box, expensive, consume large rack space Cables are primary trigger, calibration triggers, clock, data MicroTCA Solution: 1 EVR cable to BPM shelf BPM RTM contains filters and calibrate circuits 4 input Ch. Single parallel bus trigger ahead of beam from timing module starts all units sampling ringing filtered input signals, processing data in FPGA, delivering to processor on demand Each BPM digitizer FPGA when ready delivers 2 consecutive timed calibrate signals via RTM After 3 cycles unit is ready for next pulse Large network cable plant eliminated; rack space minimized to 1 shelf 40

Lab-Industry 1: Fast ADC RTM & ADC-DAC Includes IPMI Extension to RF Chassis, Rear clock, trigger in & DAC out Courtesy A. Young, SLAC & Struck Co. New Developments in I&C Standards for Physics 41

Example 2: Generic FPGA & Fast-Slow 12 bit ADCs RTM for interlocks Courtesy D. Brown, SLAC & TEWS Co. New Developments in I&C Standards for Physics 42

V. xtca Market Growth Projections 43

Survey Data Limitations Survey 1 for total ATCA market commissioned by industry leader, Emerson Survey 2 looked at only blade sales which appear to be about half the total market Market segmentation shows 80%+ total market going to Telecom MicroTCA is represented partly in remaining 20% Total of $800M/yr projected for 2012 Close to entire VME market which is 80% military 44

Survey 1: ATCA Total Market Forecast Note: Stated target of PICMG for ATCA was eventual penetration of global market of 10% of $100B Note - Current ATCA market is approx same as total VME market 45

Survey 1: ATCA Adoption Potential in Adjacent Markets Markinetics also believes the broad ATCA software ecosystem and developer community must also evolve in parallel in order for ATCA to achieve greater acceptance across multiple industries, including a number of missioncritical enterprise sectors. For instance, one specific opportunity on which some ATCA suppliers have been attempting to capitalize is the widespread reallocation of military and defense-related spending toward network-centric cyber-security initiatives, with ATCA as a common, open-standard platform option. 46

Survey 1: Forecast Analysis Looking ahead, the growth trajectory for ATCA adoption certainly will be affected over the next five years, Markinetics believes, by a number of important factors, including: 1. Improving macro- and micro-economic conditions and the gradual return to an investment (rather than a cost-cutting) posture among the world s leading telecommunications services providers 2. Broader acceptance across various information-critical industries adjacent to the traditional telecom sector 3. A far more engaged software developer community focused on enabling robust ATCA-based solutions 4. Incremental advances in ATCA-compliant technology (e.g., engineering, design, reduced energy consumption, thermal management, etc.) 5. A unique capability and capacity to adequately provide a range of support services that complement the hardware product 47

Survey 2: Market Segmentation Note: Segmentation in rough agreement with Survey 1 Note: Current telecom market over 80% of total 48

VI. Summary & Conclusions 49

Summary Summaries of COTS and Lab initiatives, latter principally in RTM sections, shown in following sllides. 50

MTCA.4 Shelves 6/7 & 12 Slot Item- Shelves Mfgr Type Available 6-slot Schroff 1-Star backplane. Non redundant 7 slot Schroff 1-Star backplane. Standard PS Removable fan 6-slot ELMA 1-Star backplane. Non redundant 12-slot Schroff 2-Star redundant Managed PU, CU 12-slot ELMA 2-Star redundant Managed PU, CU 12-slot Perf Tech 2-Star redundant 6/12 for β test 6-slot Power Bridge 1 Star non redundant 2U, 3 RTM slots 12 Slot VT811 Vadatech 2-Star redundant 6/12 release 51

MTCA.4 Power Modules Item- Power Modules Mfgr Type Available Puma µblade Puma 900W Unreliable, RF Noise 6-pack mod. Schroff 300W Devmt Incompatible w/12 slot 6-pack mod. Elma 300W Devmt Incompatible w/12 slot 12-pack mod. Schroff 900W Due 8/12 6-Pack mod Power- Bridge 12 Pack mod Performan ce Tech 600W 1000W, 1200W modules Prototype delivered Beta unit 6.12 12-pack mod. Vadatech 1000W W/ AC in Due 7/12 12-pak mod. Vadatech 1200W, -DC in Due 7/12 12-pak mod Wiener 900W AC in Due 7/12 52

MCH Modules Item- Processors Mfgr Type Available UT002 Vadatech Full featured PCIe Gen3 NAT Full featured? Performance Tech PT Full featured Beta test unit 6/12? Kontron Full featured? 53

MTCA.4 Processors Item- Processors Mfgr Type Available AMC121 Performance Tech PT Intel core 2 duo Beta test unit 6/12 ASLP11 GE Intel core 2 duo AMC1000 Adlink Intel core 2 duo Unreliable AM31x Concurrent Tech Intel i7 AM930 Concurrent Tech 2-wide i7 PCIeGen3 9/12 AMC 720 Vadatech Intel i7 On Order Radisys Inteli7 54

MTCA.4 Generic AMC Modules Item- AMCs Mfgr Type Available Digitizer 8300 Struck 125MSPS 16 bit 10 CH Digitizer 720 Vadatech 125MSPS 16 bit 10 CH Q3 12 Digitizers SP Devices 1.6-7 GSPS 8-14 bits No RTM Q4 12 IP Adapter Hytek 3-IP adapter w/rtm TAMC220 TEWS 3-IP adapter w/rtm Intfce TAMC651 TEWS FPGA Processor Spartan 6 w/ RTM Intfce 55

Application RTM Adapters 2 Item- RTMs Lab/ Co. Type Mates With Available LLRF SLAC Passive DC, IPMI Extend SIS 8300 Interlock SLAC Fast/Slow 12 bit ADCs TEWS651 BPM SLAC Filter, Calibration Interface SIS8300 1 st proto in test LLRF DESY Downconverter 1.3-3.9 GHz SIS8300 Sensors Interface XFEL Avalanche photo diode pulse stretcher SIS 8300 TAMC002 TEWS 3-Industry Pack RTM TAMC651? Hytek 3-Industry Pack RTM? ADC/DAC DESY 8ch 16bit ADC, 8ch 16bit DAC DAMC2 1 st proto in test ADC DESY Coupler interlock DAMC2 1 st proto 56

Application RTM Adapters 2 Item- RTMs Lab/ Co. Type Mates With Available Digital IO DESY Machine protection system DAMC2 1 st proto in production ADC DESY Coupler interlock DAMC2 1 st proto test ADC DESY Beam loss monitors DAMC2 Development ADC DESY Toroid protection / readout DAMC2 Development UCL Clock & trigger control for exp. DAMC2 1 st proto test Interface DESY Wire scanner DAMC2 Development BPM DESY Low charge button/stripline SIS8300 1 st proto test 57

Why MicroTCA? New standards essential as technology develops, legacy systems no longer cost-effective or supportable NIM (1966), CAMAC (1976), FASTBUS (1986), VMEp (1998) Last decade revolution in analog, digital, communication technologies accelerates need for change Programmable FPGA s obsolete discrete logic components, Multilayer board design enables Gigabit backplanes >10 GHz BW Integrated SERDES communications obsolete parallel bus backplanes LVDS balanced multi Gbps backplanes minimize discrete switch blades Intelligent platform management diagnoses enterprise wide problems to board level, initiates corrective action Redundant architectures => shelf Availability of 5 nines or better (0,99999) 58

Summary of Performance Advantages Excellent analog performance Modular partition AMC-RTM allows planned upgrades FPGA, processor parts (Moore's Law) separate from analog and interface module Partition interface enables different design skills to work on complete system System based on high performance scalable state-of-art communication technology only (PCIe and Ethernet) Common interfaces and operating systems for control systems High availability architecture and remote management/ maintenance, upgrades of firmware 59

V. Summary & Conclusions MicroTCA Advantages being recognized increasingly in Europe and Asia and more labs are signing on (ESS, ESSB, GSI, Spring8) PICMG industry support is solid and growing, much stronger in US than lab support Standards collaborations under PICMG will continue to refine timing protocols, software-firmware interfaces as more users raise practical problems. MicroTCA has potential to be physics standard of choice for new projects, upgrades for next 1-2 decades 60

END OF SLIDES 61