Trigger and Data Acquisition: an ATLAS case study

Transcription

1 Trigger and Data Acquisition: an ATLAS case study Standard Diagram of ATLAS Trigger + DAQ Aim is to understand most of this diagram by the end of the lecture! 1

2 Outline Basic Trigger and DAQ concepts The LHC/ATLAS Challenge Multiple Level Triggers Dataflow and Buffering Birmingham s role: Level-1 Calorimeter Trigger 2

3 BASIC TRIGGER AND DAQ CONCEPTS 3

4 This is not a Trigger... Ceci n est pas un trigger With thanks to René Magritte 4

5 This is a trigger... Higgs Candidates enter the Gates of Heaven (ie they are saved for later analysis) Boring events are thrown into the pit of hell (ie the data is discarded it often never even leaves the detector) 5

6 This is DAQ... Detector Data consists of thousands of small fragments Need to assemble fragments of the same event and send them to disk (about 1.5 MByte per event) 6

7 Not to be confused with... 7

8 The Large Hadron Collider pp collisions, s up to 14 TeV * Bunch spacing: 25ns * Nominal luminosity: cm -2 s -1 Collisions per crossing: ~30 ** * Eventually ** Now! The trigger challenge for the General Purpose Detectors : Roughly 1 GHz known physics Large event sizes: O(Mbytes) Typically small rate of new physics channels 8

9 THE LHC/ATLAS CHALLANGE 9

10 Why is it so difficult? 1) Multiple collisions per bunch Currently up to 40 separate proton/proton collisions every time two bunches interact 10

11 Why is it so difficult? 2) New physics is rare Of course if it wasn t rare, we d have seen it already For example, in A handful of golden Higgs candidates (decaying to 4 leptons) Over 10 9 events recorded Over bunch crossings Over proton collisions All Events ATLAS Physics 11

12 Why is it so difficult? 3) Time between collisions Finding new physics in an LHC bunch collision is like finding a needle in a haystack Then you get a new haystack 25ns later 25 ns 25 ns 12

13 Why is it so difficult? 4) Data size and data rate A single event is ~1.5 MByte Full data rate is ~5 TBytes/sec Would fill a few disks Can t afford to even transmit all this from the detector Typically record ~50 TBytes per day That s >1000 HD DVDs worth of data Even at this rate, we re pushing network limitations in a large PC farm Bandwidth is a vital concept in DAQ Essentially just: how fast can you get data from one point to another 13

14 How is it possible? Some new physics processes have clear signatures Interesting particles have high mass And hence decay to give high energy products High energy electrons, photons and muons are relatively easy High energy taus are less easy to distinguish Also high energy neutrinos can be seen by energy imbalance in event (Missing Energy) Unfortunately some new physics is easily confused with boring background eg Higgs decays predominantly to b quarks producing many jets These are not so different from events where no new particle is produced Nevertheless, triggering on high-energy and multiple jets is useful 14

15 Some examples 15

16 MULTIPLE LEVEL TRIGGERS 16

17 Why have 3 (plus) levels? Ideally we would study every event carefully before accepting or rejecting One decision = single level trigger We don t because: Not enough bandwidth to get all data out Not enough bandwidth or processing time to build every event Not enough processing time to analyse every event in full detail Note: requirements of trigger are intrinsically linked to DAQ capacity So we do things in stages Each level of trigger: Requires more information and processing time to make a better decision Compensates for extra bandwidth required by next trigger level by reducing output rate 17

18 Differences in Levels Low-level triggers are faster and less precise Typically custom hardware with a fixed processing time (known as latency) All data must be stored before Level-1 trigger decision is made But only partial event information can be used High-level triggers are slower and closer to analysis quality measurements Typically standard high-spec PCs running trigger specific software Can use partial or full event information But only for those events already accepted by lower levels 18

19 Level-1 Trigger in ATLAS Has to make a decision on every bunch crossing every 25ns, ie works at 40 MHz Selects about 1 in 500 events Decision time fixed at 2.0 μs About ½ of that is used up just in signal transmission Mostly based on calorimeter and muon data At reduced resolution Detector data stored in pipeline memories for 2.5 μs The triggered data is taken out of scrolling buffer at a dead-reckoned fixed time This all happens in the detector itself 19

20 Interesting Event Happens 20

21 1 microsecond after BANG Detector Data Pipeline Buffer Trigger Data Level-1 Logic 21

22 2 microseconds after BANG Detector Data Pipeline Buffer Trigger Data Level-1 Logic Trigger Decision YES If Level-1 says yes data are copied to a different buffer Event identifier attached for building 22

23 2.5 microseconds after BANG Detector Data Pipeline Buffer Trigger Data Level-1 Logic Trigger Decision NO If Level-1 says no data fall off end of memory and are lost forever 23

24 The next stage After Level-1 decision, data is transmitted off the detector Up to now everything is synchronous From now on everything works asynchronously Requires careful formatting and labeling of fragments of data to associate to a particular event, and detector element Performed in hardware known as Readout Drivers (ROD) Data now ready for software trigger But still can t access ALL of the data Solution just look in Regions of Interest Regions identified by Level-1 which causes it to say YES Level-2 Trigger accesses full data for part of the detector 24

25 Level-2 Trigger: Operation Level-2 decision time ms Performed in parallel in a farm of PCs Variable execution time implies need for buffering We ll talk more about buffering later Level-2 requests data corresponding to RoIs Full data taken from the buffers (ROBs) 25

26 Level-2 Trigger: Decision Further event reduction by a factor of If the decision is NO, data deleted from buffers If the decision is YES, data sent to Event Builder Events now pieced together (à la IKEA) for the first time 26

27 We still can t take the rate: Level-3 (or Event Filter) Eagerly awaiting Physicists and PhD students Another software trigger Another processing farm Another event reduction (~10-20) The main difference is this one can see all the event It also can take seconds, not sub-seconds 27

28 DATAFLOW AND BUFFERING 28

29 Dataflow Dataflow is about how you can most efficiently get data from one place to another For a single link, it s the bandwidth But for a complex trigger/daq system, with multiple components, it s a lot more complex The flipside to triggering is dataflow If you can trigger precisely, you need less dataflow If you have plenty of bandwidth, you don t need to trigger Most real experiments require a delicate balance between the two And a careful analysis of bottleneck location 29

30 Bottlenecks and Buffers There are two different types of bottleneck in a DAQ system Average data flow above the link bandwidth This can only be solved by increasing link capacity Or sporadic excessive data payload Can be fixed by buffering 30

31 Buffering lessons from History Information/data flow is not so different from particle/object/traffic flow Here s an example inspired by an ATLAS networking guru Consider a Roman legion One soldier = one bit of data One legion = one packet of data Roman road = data link 31

32 Link bandwidth; one legion ~ten miles/day Scouts General Vanguard Legionaries Baggage Rearguard ~10 Miles line speed, latency and pay load all interlinked. 10 miles is only 3.5 hours march But to transfer the payload at 6 abreast takes 8 100% link occupancy. What happens when the road narrows? Trunking deployed: 6 Legionaries march abreast If the road is wide enough 450 BC 12 Tables link dimension standard Link Dimensions Widest Road leaving Rome Via Appia at its widest 12 tables standard Via Appia repaved Original Via Appia Narrowest

33 Answer: restriction requires buffering at each end A buffer is just a place to queue data that is waiting to be dealt with Different types of buffer Custom hardware Simple memory Temporary disk Implementation depends on size, speed and waiting time Average speed determined by slowest link But instantaneous data/packet flow can exceed this for a period of time Output Buffer Input Buffer 33

34 The Romans understood this Trunk Road to Gaul At either end of mountain pass: Aosta and Martigny two very pleasant old Roman towns Great St Bernard s Pass Trunk Road to Rome 34

35 The modern equivalent Mont Blanc tunnel approach road Buffer Zone To Italy 35

36 Event Building in Rome Event (Legion) Building requires a large buffer Need to wait until all fragments have arrived Unless all data from all detectors is the same size, will have to wait for the largest/slowest/last fragment Output Link Buffer Zone Input Links 36

37 Filling buffers and dead-time However, buffers come with their own problems Must choose a sensible finite size What happens when it fills? Require a Busy protocol to stop new data being formed Introduces dead-time Use inhibits and leaky-bucket algorithms to limit traffic 37

38 Is this now any clearer? 38

39 BIRMINGHAM S ROLE: THE LEVEL-1 CALORIMETER TRIGGER (L1CALO) 39

40 Triggering in ATLAS Three-stage triggering system Level-1: custom-built hardware, fixed latency target rate 75 khz Level-2: mostly software, RoI-based selection target rate 5000 Hz Event Filter: software, full detector target rate 400 Hz e/γ tau Calorimeters Calorimeter Trigger jet E T ΣE T Muon Detectors Muon Trigger μ All data buffered at bunch-crossing rate of 40 MHz for 2.5 µs Level-1 has three sub-systems: Calorimeter Trigger Muon Trigger Central Trigger (CTP) Central Trigger Processor Trigger to Front-end Buffers Level-1 Trigger Regions of Interest (RoI) to Level-2 40

41 Trigger Algorithms Cluster Processor Jet/Energy-sum Processor ECAL+HCAL e/γ or τ/hadron algorithm Central cluster > threshold Isolation requirements in surrounding rings Local ET maximum 16 thresholds possible Jet algorithm Programmable size Energy in (em+had) > threshold 8 size/threshold sets 8 Missing-ET, 8 Sum ET plus 8 Missing-ET significance thresholds 41

42 USA15 Installation 42

43 Hardware Implementation Multiple layers of FPGA processing Data reception and fanout Algorithmic processing Result merging Final stages in common CMM 43

44 Processor custom Backplane Dense, high bandwidth backplane Up to 1,150 pins per slot About 20,000 pins in all Common to CP and JEP systems Fastest signal speeds: 480 MHz differential (LVDS input) 160 MHz single ended CP system 80 MHz single ended JEP system 44

45 Installation: Analogue Cables 496 cables into 8 crates Four cables just fit front of one 9U module 45

46 Installation: Digital Cabling Up to 1400 individual LVDS signals into one crate More than 500 Gbit/s data input 46