PIC Tier-1 Report. 1st Board for the follow-up of GRID Spain activities Madrid 5/03/2008. G. Merino and M. Delfino, PIC

Transcription

1 PIC Tier-1 Report 1st Board for the follow-up of GRID Spain activities Madrid 5/03/2008 G. Merino and M. Delfino, PIC

2 New Experiment Requirements New experiment requirements for T1s on 18-Sep-07, C-RRB Fall CPU (ksi2k) ALICE ATLAS CMS LHCb Total Disk (TB) ALICE ATLAS CMS LHCb Total Tape (TB) ALICE ATLAS CMS LHCb Total

3 Experiment requirements change We can compare how the requirements have changed from Oct-2006 to Sep-2007: ATLAS disk and tape has increased slightly CMS has very big changes (less CPU and Tape and more Disk) Requirements (New-Old)/Old CPU (ksi2k) ATLAS 0% 0% 0% CMS -23% -4% -6% LHCb 0% 2% 1% Disk (TB) ATLAS 8% 6% 2% CMS 29% 14% 12% LHCb 0% 0% 0% Tape (TB) ATLAS 5% 6% 4% CMS -25% -36% -37% LHCb 0% 0% 0% 3

4 PIC Pledges updated Applying the algorithm to compute the PIC pledges from the experiment requirements that was agreed with each of the experiments and used in the MEC project proposal: CPU (ksi2k) ATLAS CMS LHCb TOTAL Disk (TB) ATLAS CMS LHCb TOTAL Tape (TB) ATLAS CMS LHCb TOTAL

5 PIC Pledges: new vs old Comparing the PIC pledges resulting from the new requirements w.r.t. those resulting from the old requirements, find that PIC will have to provide less CPU and Tape and more Disk: CPU (ksi2k) New Old Diff -9% -2% -5% Disk (TB) New Old Diff 14% 8% 5% Tape (TB) New Old Diff -17% -24% -24% Applying the same cost estimations than those in the project proposal, the impact of changing to the new pledges on the cost for results minimal (170 K cheaper, out of 5,8 M ) 5

6 PIC capacity growth plan towards 2008 MoU Pledges Will look each category in details in the next slides CPU MoU-08: 1509 ksi2k Disk MoU-08: 967TB Tape MoU-08: 953TB 6

7 CPU power calibration According to vendor CPU power specifications, by the end of 2006 PIC CPU capacity was greater than the MoU-2007 pledge (501 ksi2k) System based on HP servers, Intel Xeon Dual Core GHz Before summer 2007, the LCG MB agreed that all sites should adopt a common way to run the specint2000 benchmark: One benchmark per core, low opt gcc flags, +50% factor PIC bought the spec2k benchmark, calibrated the CPUs in Sep-07 The result was a drop of about 30% measured vs. vendor power New CPU calibration was applied from 1 st Oct 2007 onwards Total CPU power for PIC Tier-1 with new calibration ~ 480 ksi2k About 5% below the MoU-2007 pledge 7

8 CPU Capacity ramp-up A CPU purchase was initiated in Q to ramp-up towards MoU pledge and compensate for the small deficit in 2007 Based on HP blades, Intel Xeon quad core GHz Problem: Mismatch of electrical hookup prevented immediate deployment Fix will be put into racks during yearly electrical shutdown 18/03/08 An extension of the HP blade centers system is being launched that should ramp PIC capacity over MoU-2008 pledge by May 8

9 CPU Accounting PIC CPU accounting from Jan-06 until Feb-08 from WLCG Reports ksi2k*day (walltime) ene-06 mar-06 LHCb CMS ATLAS Inst. capacity (for T1) MoU Pledge WN OS migration SL3 SL4 may-06 jul-06 sep-06 nov-06 ene-07 mar-07 may-07 jul-07 sep-07 nov-07 ene-08 mar-08 SpecInt2000 normalization change Oct07 Planned HP blades deployment may-08 9

10 CPU Efficiency 120,00% cput/wallt 100,00% 80,00% 60,00% 40,00% 20,00% 0,00% ene-06 mar-06 ATLAS CMS LHCb may-06 jul-06 sep-06 nov-06 ene-07 mar-07 may-07 jul-07 sep-07 nov-07 ene-08 ATLAS: low eff in Sep-07 due to a problem of the ATLAS jobs using the WN temporary local scratch area CMS: low eff in Sep-07 due to a problem with CMS s/w affecting mostly to the CSA07 merging jobs ALL: low effs in Oct-Dec 2007 are an overall problem. FEW users sending LOTS of jobs with efficiency close to ZERO PIC is publishing user level accounting in APEL since months This should help experiment managers to spot/ban inefficient users 10

11 Disk Storage Service From Jun-06 PIC is running a SRM-disk service based on dcache The service was running stably Oct-06 Jun-07 providing 80-90TB Jun-07: service upgraded to dcache-1.7, improved stability The main issue (gridftp-doors hanging) disappeared Purchases of new disk in the 1st half of 2007: Sun X4500 servers Apr-07 Oct-07: Deployment of 14 new Sun X4500 servers Allocated capacity ramped-up to 280TB New capacity purchased during Q new Sun X4500 servers Deployed Feb-08: being used today for CCRC08 Expect to ramp-up capacity up to near 600TB 11

12 Disk Accounting 1200 LHCb CMS ATLAS Allocated capacity (for T1) +13 Sun X4500 Planned TB disk MoU Pledge (with 70% eff) +14 Sun X new Dell PowerVault SAS 0 ene-06 mar-06 may-06 jul-06 sep-06 nov-06 ene-07 mar-07 may-07 jul-07 sep-07 nov-07 ene-08 mar-08 may-08 Start of SRM-disk svc with dcache-1.6 Upgrade to dcache 1.7 Upgrade to dcache 1.8 Configuration of SRMv2.2 Ready for CCRC08 12

13 Sun X4500 disk servers Disk servers with 48 HDs each (500 or 1000 GB) We run them with Solaris-10 OS and ZFS file system Good integration with dcache (java) Good measured performance in the dcache Production service One of the X4500 servers last week receiving data and sending it to tape at rates MB/s both ways sustained 13

14 Storage LAN: Distributed resilient network architecture In order to fully exploit the throughput capacity of each server, we are connecting them using 3 bridged switches every 18 disk servers 24 Gbps available bandwidth for pool2pool replication 20 Gbps available bandwidth to the outside Originally designed for resilient Tier-0 data receptor, will now be expanded to all dcache disk servers 14

15 Disk Capacity Planning 15

16 Tape Storage Service Early 2007, seeing that the deployment of Castor2 was more difficult than originally foreseen, decided to explore an alternative solution for the MSS: dcache + Enstore Apr-07: decided to go for a dcache + Enstore solution for the MSS and drop Castor Excellent support from FNAL for deploying the new Enstore service Collaboration agreement PIC-FNAL for future developments Sep-07: SRM-tape service based on dcache + Enstore in production, ready for the start of CMS CSA07 16

17 Castor Decommission CMS: Migration done (easy one) In Q CMS confirmed that all of the data on Tape at that time could be deleted Only 500 GB were staged and kept on disk until Jan-08 for few users LHCb: Migration of about 20TB finished in Feb-2008 Very good coordination with LHCb: smooth migration Castor ready to be decommissioned for LHCb ATLAS: Everything ready to migrate ~16TB Waiting for the end of CCRC08 February tests to release some manpower on both sides By the end of March we should have completed the Castor decommission for the 3 LHC experiments 17

18 MSS roadmap mid-2006 mid-2007 ~200TB 9949b allocated. Will be unavailable during STK robot upgrade Currently ~400TB LTO3 ready to be allocated 18

19 Tape Service Metrics Enstore performance tested during CCRC08 (with 4 LTO3 drives) Writing up to near 11 TB/day Reading up to near 9 TB/day reads writes 19

20 Tape Service Metrics In January 2008 the MB agreed that Tier-1s should report some MSS metrics in a format proposed by CERN PIC was one of the first Tier-1s to report MSS metrics Now collecting this data for post-ccrc08 analysis 20

21 Tape Usage Accounting feb-08 LHCb CMS ATLAS Installed capacity MoU Pledge Castor freeze Enstore in production abr-06 jun-06 ago-06 oct-06 dic-06 feb-07 abr-07 jun-07 ago-07 oct-07 dic-07 TB tape

22 Costs tracking: CPU In December 2007, B. Panzer released an update of the CERN cost trends document PASTA-05 (Project Proposal Oct-06) PASTA-07 Euros/ksik PIC Purchases We can compare this new cost estimation (PASTA-07) w.r.t. the one used Oct-06 in the project proposal (PASTA-05, agreed by all the projects) and the actual cost of the CPU purchases executed 0 dic-04 dic-05 dic-06 dic-07 dic-08 dic-09 dic-10 dic-11 dic-12 22

23 Costs tracking: Disk Comparison of the the cost estimations and the actual cost of the Disk purchases executed Euros/TB PASTA-05 (Project Proposal Oct-06) PASTA-07 PIC Purchases dic-04 dic-05 dic-06 dic-07 dic-08 dic-09 dic-10 dic-11 dic-12 23

24 Site Reliability Site Reliability has been measured for T0 and T1s since May-06 Very useful tool in pushing the sites towards production services PIC reliability has been mostly over the average PIC is one of the 3 only sites reaching the target since July % 95% 90% Reliability 85% 80% 75% 70% 65% PIC T0/T1 average Target 60% may-06 jul-06 sep-06 nov-06 ene-07 mar-07 may-07 jul-07 sep-07 nov-07 ene-08 24

25 Wide Area Network LHC-OPN Shared 10GE lambda for the LHC-OPN and the NREN connection Unique physical path for the fiber's last mile 2 VLANs over 10 GE 2 Gbps (Best Effort) Traffic PIC Tier Gbps LHC- OPN Traffic Tier-0 PIC Traffic PIC Tier-1 25

26 Network s future at PIC The 10Gbps LHC-OPN connection In the near future the last mile unique path problem will be solved Along 2009 PIC will have a 10Gbps backup link The 2Gbps non-opn link will be upgraded to 10Gbps by Anella Científica Deployment of LAN redundancy with Spanning Tree 26

27 LHC-OPN deployment Oct-06: 10 Gbps lambda to RedIRIS POP at Barcelona May-07: Last mile to PIC deployed Jun-07: 10Gbps certification tests To be able to make use of the 10Gbps lambda to CERN we needed to move to a new IP address range (new AS) Sep-07: GridFTP doors moved to the new IP range First LHC incoming data traffic flowing through the new OPN link Nov-07: All of the storage servers moved to the new IP range First LHC outgoing data traffic flowing through the OPN link 27

28 WAN performance: import The network traffic entering PIC has been sustained at the level of 2Gbps in the last days showing good stability On Feb 25 reached a peak of about 4 Gbps 28

29 WAN performance: import CMS writing data to PIC at MB/s during several days 29

30 WAN performance: export CMS Data export from PIC to more than 20 sites at MB/s during several days 30

31 LCG Spain Network Working Group With the LHC start approaching the load on the LCG Tiers is ramping up, and this has an impact on the Academic Networks Up to now, the performance of the Spanish network for the LCG sites has been very good To ensure that this situation is maintained, it has been agreed with the Director of the Spanish NREN RedIRIS to create a Network Design and Operation Working Group Initial membership could be a representative of each Spanish LCG Tier, and a representative of RedIRIS Leave the door open for future inclusion of other relevant actors Regional Academic Networks Portuguese sites and networks 31

32 WLCG High Level Milestones: 24x7 PIC does not plan to have staff on site 24x7 Deploy critical services in a resilient way and on robust h/w Critical failures will trigger notification to a person on-call that will be able to intervene remotely or, if really needed, go to PIC Manager on Duty (weekly shift) main duties: First line incident solving: will react to critical alarms, follow recovery procedures Single contact point for support: regularly poll support mail, and redirect issues to relevant experts Calendar 2005: Phase-0 started. Shift only working hours. Oct-07: MoD procedures doc to operate Tier-1 services released Dec-07: Phase-1 of MoD on-call started MoD will work with service experts to ensure that relevant alarms are deployed and associated procedures documented Phase-2 of MoD on-call: start date yet to be set Alarms and procedures for Production Services stable 32

33 WLCG HLMs: VO-boxes SLAs Sites and experiments agreed that a new host was needed at the T1s to deploy experiment-specific agents in providing various services (mainly data management): VO-boxes Initial concern at sites for such services, since the responsibility boundary experiment vs. site was fuzzy The MB agreed that T1s should sign SLAs with the experiments defining the responsibilities and procedures to follow in the event of problems Status at PIC LHCb: Proposal presented in Oct-07, SLA signed in Jan-08 CMS: Preliminary draft proposed by mid-07, but no progress since then Now we propose CMS an SLA similar to the one signed with LHCb ATLAS: ATLAS VO-boxes are still being run at CERN, not at the (at least european) Tier1s Until the schedule for deploying the VO-box in production at PIC is clarified from ATLAS, the SLA is paused 33

34 LCG-3D service Oracle databases synchronized among T0-T1s (Oracle streams) ATLAS will run the CondDB (and possibly TAGs) on it LHCb will run the CondDB and the LFC on it Up and running at PIC since April Successfully tested: CondDB by both experiments Oracle-streams, monitoring, backup recovery procedures Last year we called for a tender for a specific (Oracle-certified) h/w platform to deploy these DDBB clusters The plan is to deploy all of the Oracle DDBB backends for any other service on the same RAC platform as the LCG-3D FTS and LFC-atlas, for the moment Final h/w for the production instances is being deployed now 28TB Netapp FiberChannel disk 9 Fujitsu Siemens blade servers (dual-dual Xeon) 2 Brocade FiberChannel switches 34

35 File Transfer Service Currently running on two servers: One head node with the agents and web service One Oracle DB backend node Robustization: Will split the head node into five hosts Web-service: 2 load-balanced nodes Agent daemons: spread across 3 hosts Oracle DB backend in a RAC 35

36 LFC File Catalogue Currently only providing this service for ATLAS. Running on two servers: One head node with the front-end daemon One MySQL DB backend node Robustization: Will split the head node into two load balanced hosts Migrate de DB backend to Oracle RAC Will deploy a duplicate of this solution for the LHCb LFC 36

37 WLCG activities involving all Tier-1s In the last year, PIC Tier-1 has been participating in the WLCG testing activities driven by the LHC experiments Main Goal: ramp-up the load on the infrastructure towards nominal values and test stability + performance of the services In Sep-07 the project decided to schedule two test periods in 1st half Seen as the last opportunity to try and test the infrastructure under realistic conditions before data taking CCRC08: Combined Computing Readiness Challenge phases: Feb-08 and May-08 In the following slides we will briefly present the most relevant results obtained by PIC, emphasizing those of the recently run CCRC08 37

38 ATLAS T0 Throughput test Oct 2007 From 16 until 22 October 2007 ATLAS did a throughput test exporting data from CERN to the 10 Tier-1s at the same time Coincidence with CMS CSA07 transfers ATLAS managed to exceed 1GB/s out from CERN CERN PIC reached 130 MB/s daily average for ATLAS, with very low error rate 38

39 ATLAS Cosmic Runs ATLAS is regularly taking cosmic muon data to test the whole detector and data acquisition chain The M4 run (23/08/ /09/2007) was the first test of exporting real data to the Tier-1s About 13 TB of data successfully transferred to PIC and then to the Tier-2s By the end of August the Enstore tape system was still not in production at PIC M4 data was first stored on disk migrated to tape few weeks later The M5 run (22/10/ /11/2007) Nominal shares for Tier-1s were imposed: ~3 TB to PIC Enstore service already in production this time the data was sent directly to tape 39

40 ATLAS CCRC Feb08 Two first weeks setting up SRM tokens and solving problems at T0 Started real activities on week-3 (17-23 Feb 2008) Data produced with load generator pushed out of CERN Started with 25% of peak rate, then increased in steps to 100% Data distributed to all 10 T1s according to MoU PIC Results of week-3: Sustained 700 MB/s for 2 days Many peaks above 1.1GB/s for several hours 6 out of the 10 Tier-1 ran smoothly PIC was one of them Throughput Errors 40

41 ATLAS CCRC08: PIC T2s distribution On the 26-Feb, the data distribution from PIC to the regional T2s was tested with successful results The data arrived with very good efficiency to most of the T2s 41

42 CMS Activity: continuous massive data transfers CMS started last year a program of continuous data transfers among all its sites Goal: Test the service stability under load conditions for long periods This has been an extremely useful tool for deploying and understanding how to operate the Data Transfer Services In the last year: PIC imported from any other CMS site: 1520 TB ~10% of the total imports to all T1s PIC exported to any other CMS site: 1400 TB ~6% of the total exports from all T1s A total of almost 3 PB import + export from PIC in the last year ~250 TB/month, ~8 TB/day 42

43 CMS CCRC Feb08 T0 T1 targets To disk: 40% of nominal rate (25MB/s for PIC) sustained for 3 days To tape: migrate the above data within 1 week with stable pattern T1 T1 targets 50% of the overall 2008 nominal outbound rate to T1s (4MB/s to each T1 for PIC) Exchange data with at least 3 T1s and at least 1 must not be in the same continent T1 T2 targets Demonstrate aggregate T1 T2 target to regional T2s only For PIC: 19,4 MB/s to T2-Spain and 3,1 MB/s to T2-Portugal T2 T1 targets Traffic only from regional T2s according to the Computing Model For PIC: 1,4 MB/s from T2-Spain and 0,5 MB/s from T2-Portugal 43

44 CMS CCRC08: T0 PIC test As presented by D. Bonacorsi on CMS computing meeting 14-Feb 44

45 CMS CCRC08: PIC T1s test As presented by D. Bonacorsi on CMS computing meeting 14-Feb 45

46 CMS CCRC08: PIC T2s test As presented by D. Bonacorsi on CMS computing meeting 14-Feb 46

47 CMS CCRC08: T2s PIC test As presented by D. Bonacorsi on CMS computing meeting 14-Feb 47

48 LHCb CCRC08 Raw data distribution to Tier-1s Share according to resource pledges from sites Test performed during 3rd week of Feb-08 (OK for 6 out of 7 T1s) Data reconstruction at Tier-0 and Tier-1s Production of rdst stored locally and data access using SRMv2.2) Report from yesterday: OK at PIC with two issues Wrong mapping of the production role to the local batch accounts Eventual no available space (dcache bug affecting several sites) Both promptly addressed and solved by PIC experts 48

49 CCRC08 Combined transfers The 4 experiments managed to export data from CERN to all the Tier-1s simultaneously for several days in February CERN total export rate exceeded 2 GB/s for several hours on Feb 22nd 49

50 CCRC08 Combined transfers Looking at the sites breakdown one can see that a) PIC is visible b) reaching nominal transfer rates from CERN 50

51 PIC support to LHCb central services DIRAC is the software infrastructure that manages the use of LHCb computing resources The Monitoring and Accounting DIRAC systems have historically been running at PIC due to the important implication of the Spanish groups (UB and USC) in the development of DIRAC Since the start of the Spanish collaboration to the DIRAC project, there has been full support from PIC to these activities, in term of providing the necessary severs to run DIRAC services Last Jan-08 PIC signed a formal agreement with LHCb to provide support for these LHCb central services, hosting the following nodes: Database server DIRAC server Web server Development server 51

52 Experiment liaisons at PIC The successful operation of a multi-experiment Tier-1 requires to have one person on-site per experiment acting as liaison Two liaisons currently at PIC: ATLAS (X.Espinal) and CMS (J.Flix) LHCb liaison was half-funded by LHCb-T2 project Funding available at the Tier-1 from the start of current project. Recruiting process is ongoing The experiment liaisons are doing a very good job and service to the collaboration: ATLAS, X. Espinal: Co-coordinator of operations shifts, member of the central MCprod and Data Processing team, member of the DDM regional team for Spain and Portugal CMS, J. Flix: Co-coordinator of the CCRC08 Tier-1 reprocessing tests, Coordinator of the CCRC08 FTS-related issues, Deputy member of the CMS CRB LHCb, A. Casajús: very useful coordination work for VObox SLA and Castor-Enstore migration The PIC Tier-1 ATLAS and CMS liaisons organize regular meetings with Spanish and Portuguese sites to coordinate T1-T2 operations 52

53 Some issues of concern Delay of the CPU ramp-up due to the electrical problem with blade centers and impact it has in our ability to stress the Storage service Confident that we will catch up after the electrical shutdown in Easter Procedure for experiments to update requirements in the C-RRBs CMS change of ~25% for 2008 received in the very last moment Would like to see the reqs./pledges update calendar described in the MoU honored (may be once the RSG will be operative) Achieve and maintain adequate reliabilities and availabilities for the complete integrated chain T0 + Spanish T1 + ATLAS/CMS/LHCb associated T2s in Spain and Portugal The current success of the WLCG is largely thanks to the EGEE underlying infrastructure it builds on EGEE-III is about to start, and will be the last one before EGI Spanish LCG sites need to push and be active in the recently created Red Nacional de e-ciencia and follow the NGI creation 53

54 Summary PIC has participated in all of the tests executed within WLCG in the last years to demonstrate readiness for LHC data taking Service reliability has been improving with time and stays above the average for Tier-1s Successful results in the WLCG readiness tests of the Tier-1 services for the three experiments An example could be the successful deployment of the new SRMv2.2 service on time for the CCRC08, while Deploying a completely new MSS backend (Enstore) Migrating the experiments data from Castor to Enstore Developing a new driver to be able to use IBM robot with Enstore Steadily ramping-up the service capacity The procurement execution is slightly delayed. Will not reach the 2008 MoU pledges by 1st April but confident that will make it by May 54

55 Backup Slides

56 Top-down cost estimate (from the Project proposal Jun-2007) Recalling the PIC new pledges: PIC T1 Capacity 2006 (*) CPU (ksi2k) Disk (TB) Tape (TB) The cost estimation can be derived from the previous slide table: Assuming equipment is decommissioned after 3 years PIC T1 Cost ( ) CPU Disk Tape Total Total cost : Funding request for h/w: (savings from prev. Project + PIC-BASE) Funding obtained ~ for h/w 56

57 Tier-1 cost estimation with new pledges C-RRB Fall-07 Applying exactly the same cost estimations than those in the project proposal, the impact of changing to the new pledges for was minimal Tier1 CPU Tier1 Disk Tier1 Tape TOTAL (euros) Total cost : To be compared to the total cost with the original pledges in the proposal:

58 Tier-1 cost estimation with new pledges C-RRB Fall-07 and new PASTA-07 intermedio Applying new cost estimations published by B. Panzer on Dec-2007: Tier1 CPU Tier1 Disk Tier1 Tape TOTAL (euros) Total cost : Good news is that cost for 2009, 2010 seems more stable 58

59 Tier-1 cost estimation with new pledges C-RRB Fall-07 and new PASTA-07 (bare) Applying new cost estimations published by B. Panzer on Dec-2007: Tier1 CPU Tier1 Disk Tier1 Tape TOTAL (euros) Total cost :

60 Tier-2 Availability Tier-2 sites availability/reliability is being officially tracked by the LCG project since October 2007 (reports produced) 110% 100% 90% Availability 80% 70% 60% 50% 40% 30% PIC ES-ATLAS-T2 ES-CMS-T2 ES-LHCb-T2 oct-07 nov-07 dic-07 ene-08 60

61 WLCG High Level Milestones The Tier-0 and Tier-1s progress is closely tracked within the MB Capacity and Reliability milestones High Level Milestones The status of HLMs is reported quarterly to the project OB 61

62 LCG 2007 Accounting Report 62

63 CPU cumulative accounting?? Gráfico de accounting cumulativo?? Hacerlo también modificando el installed nov-06 oct-07 por 30% menos a ver qué tal mal vamos con la calibración nueva 63

64 CPU Eff Summary Tier1s

65 Old Experiment Requirements Old experiment requirements, from Oct-2006: (Used in the proposal for the MEC projects ) CPU (ksi2k) required ALICE ATLAS CMS LHCb Total Disk (Tbytes) required ALICE ATLAS CMS LHCb Total Tape (Tbytes) required ALICE ATLAS CMS LHCb Total

66 How is PIC connected to non-opn sites? Traffic PIC - T2s flows through RedIRIS 2 Gbps connection with l Anella Científica (RREN) / RedIRIS (NREN) RedIRIS network is fully redundant Non-OPN traffic with PIC All traffic with PIC OPN&non-OPN 66

67 CMS export to Regional T2s Data export to each regional T2s at the level of >20MB/s daily average 67

68 ATLAS Mcprod contribution Distribution of MCprod walltime for jobs run in Jan-Feb 2008 in the ATLAS PIC cloud Main contributors: UAM, PIC, IFIC Good success rate 68

69 CMS upload to PIC from Regional T2s Upload from Regional T2s to PIC demonstrated at the level of 10MB/s daily for every site, sustained during several days 69

70 Upload/export PIC-T2s last week Upload to PIC: CIEMAT and Coimbra great quality Lisbon is having some problems IFCA medium quality Export from PIC: CIEMAT great quality Coimbra has some problems LISBON and IFCA did not restart downloads 70

71 Issue exporting CMS LoadTest data from PIC Export from PIC: From 24 til 29 Feb observed a decrease in the quality of exports from PIC to all of the sites We suspect the reason is that these export tests are using a quite small sample of test files Many sites trying to read the same files again and again dcache detects those files as very hot and, somehow is not handling the situation properly Solved on the 29th Feb by restarting a dcache cell 71

72 DDT link commissioning STAR-IFCA (presented by JFlix in CMS-PIC meeting 18-Dec-2007) SWE in DDT (use of T1-STAR channels): PIC really stable IFCA Tier-2 was able to commision all import links from all Tier-1 centers using PIC FTS (via STAR-IFCA channel, all except ASGC). RAL,FNAL PIC* INP3 We asked SWE Tier-2s beginning December to use T1-STAR channels from File Origins (to accommodate to CMS plan): IFCA change * PIC IFCA transfers routed through 622Mbps path. RedIRIS contacted. Rate drop T1-STAR channels need hard debugging. Only way to keep some Tier-1 import links commissioned is by using PIC FTS. 72

73 Total Network Traffic MRTG monitoring (total input/output to/from PIC), last week On Monday 25-Feb we exceeded 5Gbps (30 min. Average) input to PIC 73