Guillimin HPC Users Meeting November 16, 2017

Transcription

1 Guillimin HPC Users Meeting November 16, 2017 McGill University / Calcul Québec / Compute Canada Montréal, QC Canada

2 Please be kind to your fellow user meeting attendees Limit to two slices of pizza per person to start please And please recycle your pop cans. Thank you! 2

3 Outline Compute Canada News System Status Software Update Training News Special Topic Guillimin Decommissioning and Data Migration to GP4 3

4 Compute Canada News (Reminder) 2017 Resource Allocation Competitions Implementation of Awards (storage): Ongoing Progressively adjusting quotas while making space on /gs Important notes: There is some small number of migrations of allocations, either partial or full, from Guillimin to other Compute Canada systems (such as the new Cedar and Graham) Affected users should prepare their migration by configuring their Globus account and by cleaning their storage spaces: removing unnecessary data, grouping and compressing files in archives We can help answer any questions regarding migration if your allocation is on a different system 4

5 2018 Resource Allocation Competitions Fast Track (by invitation only) due: Nov. 2, 2017 RPP (platform) application due: Nov. 16, 2017 RRG (resources) application due: Nov. 16, 2017 No more allocation on Guillimin (*) Any Fast Track and RPP compute allocation on Guillimin will be moved to another cluster Groups having a storage allocation on Guillimin can request NDC-GP4 storage allocation for 2018 * More details in the special topic Compute Canada News 5

6 System Status About recent storage system issues Oct. 20: Needed to reset the IB Fabric and restart GPFS Oct. 21: Some problematic compute nodes are put offline Oct. 26: Hardware issue affecting GPFS I/O operations Oct. 27: Maintenance activities on the storage system Nov. 1: Maintenance activities on the storage system Nov. 9: Maintenance activities on the storage system We reduced the load on GPFS by pausing job scheduling and reserving full racks hours in advance We stopped GPFS on the problematic storage node; the redundant storage node got the full (reduced) load Maintenance done quickly without any GPFS issue 6

7 System Status (Reminder) Storage monitoring We get notified when a user is running jobs and has either the home or scratch space full We check both the space usage and the file count We then notify the user to cleanup the space as soon as possible and adapt job scripts accordingly Idle jobs are immediately put on hold until the user cleans up the storage space 24 hours later, if jobs are still running while the home or the scratch space is not cleaned, we will cancel the user s active jobs If GPFS accumulates too many long waiters, we will urgently cancel all user s active jobs 7

8 System Status Storage: need to cleanup /sb and /lb (*) Deadline : by the end of 2017 For groups using exclusively /sb, we will open access to a project space on /gs * More details about Guillimin Decommissioning in the following special topic 8

9 Software Update New installations on Guillimin: ADMIXTURE/1.3.0 (Genomics) DMTCP/2.5.1 (Checkpointing) Software on Compute Canada systems McGill users can now use Matlab on Cedar and Graham To use modules and non-commercial packages on Guillimin: source /software/soft.computecanada.ca.sh 9

10 Training News All upcoming events: calculquebec.eventbrite.ca Midi-conférence : Développement d'intelligence artificielle sur grappes de calcul (Laval, on site and videoconference) Suggestions for training? Please let us know! Recently completed: Oct Premiers pas sur les serveurs de calcul (Sherb.) Oct Migration de Colosse et Helios (Laval) Oct Calcul Informatique de pointe - débutants (Laval) Nov. 6,8 - Software Carpentry Python+ARC (Montreal) Nov. 7,9 - Software Carpentry Python+ARC (McGill) All materials from previous workshops are available online: wiki.calculquebec.ca/w/formations/en All user meeting presentations online at 10

11 User Feedback and Discussion Questions? Comments? We value your feedback. Contact us at: Guillimin Operational News for Users Status Pages (all CQ systems) Follow us on Twitter 11

12 Guillimin Decommissioning and Data Migration to GP4 November 16, 2017 McGill University / Calcul Québec / Compute Canada Montréal, QC Canada

13 Guillimin Decommissioning and Data Migration Outline Compute Canada Cyber Infrastructure Renewal Guillimin Decommissioning Data Migration Compute Migration Hosted CFI Infrastructure Resource Allocations in

14 Compute Canada Cyber Infrastructure Renewal Technology Refresh CFI Cyberinfrastructure Challenge 2 Program Investments totaling $125M in value System and data centre consolidation Previously: 58 systems operated at 27 locations Current: Stage 1 and 2 systems at 5 site locations Stage 1 ($75M): Arbutus (GP1), Cedar (GP2), Graham (GP3), Niagara (LP) Stage 2 ($50M): 2018 GPx Expansions (Compute & Storage) New! GP4 ($25M) General Purpose Cluster Compute - commodity processors and GPUs Storage - Capability and capacity storage including tape Experimental Systems 14

15 Compute Canada Cyber Infrastructure Renewal GP4 - General Purpose Cluster ($25M investment) Compute and Storage Will include regular x86 processors and GPUs National Data Cyberinfrastructure Large-Scale disk and tape storage with capacity management Cloud partition Timeline for availability: September 2018 Also planning for some experimental systems (TBD) 15

16 Guillimin and GP4 Guillimin Decommissioning Installation of GP4 will require that Guillimin compute and storage be decommissioned and removed Guillimin Phase 1 (2011) and Phase 2 (2013) Core integration at the level of the InfiniBand Network Guillimin Compute Phase 1 Compute - 14,400 cores Phase 2 Compute - 8,000 cores and 116 GPUs, 100 Phis Guillimin Storage Phase 1 Storage - 2 PB Phase 2 Storage - 3 PB Tape Storage PB Includes large amounts of hosted CFI nodes and storage 16

17 GP4 GP4 Timeline - Tentative and preliminary (!) 17

18 Guillimin Decommissioning Multi-step Approach Guillimin Storage Migration Guillimin Code and Application Migration Separation of Guillimin Phases 1 and 2 Separation at the level of the InfiniBand Network Approximately 2 weeks full downtime anticipated Phase 1 Compute Shutdown and Removal Phase 2 Compute Shutdown and Removal Some or all of Phase 2 Compute Nodes will be kept online and in operation so as to overlap with GP4 18

19 Guillimin Decommissioning Contributed CFI Resources in Guillimin Contributed CFI resources are part of Phase 2 and not scheduled for decommissioning until January 2019 January 2019 = the expected 5 year duration for operations of CFI funded infrastructure Includes compute, GPUs and storage The contributed Phase 2 resources will either: Be kept online and operational until January 2019 An equivalent portion of GP4 compute and/or storage resources will be assigned to the CFI project as a bridge until January 2019 More information to be communicated to award holders 19

20 Guillimin Migration Guillimin Data Migration Approximately 5 PB of GP4 disk storage will be purchased in advance so as to host all current Guillimin data (Feb - March 2018) This copy of Guillimin data will be visible on GP4 Migrations will be performed centrally by staff Aim is to reduce time required to complete migration Ensure data is copied to the new storage platform Some migrations can be performed directly by users Clean-up and remove any unused data files Migrate any of your current Guillimin files to elsewhere in Compute Canada (Cedar or Graham) All migrations efficiently managed through Globus Please start ASAP the clean-up and migrations that can be made 20

21 Guillimin Migration Guillimin Code and Application Migration GP4 not ready for production use until ~Sept 2018 Resource Allocation 2018 starts in April So no GP4 Compute for start of RAC 2018 GP4 storage is allocatable to current Guillimin groups NDC-GP4 = GP4 National Data CyberInfrastructure Compute Canada CVMFS software stack is the default across all GP platforms including Cedar, Graham and GP4 Groups can access the software stack already on Guillimin Start testing your codes based on the new stack and start copying your applications and codes to Cedar and/or Graham Default compute & storage resource amounts are larger on Cedar / Graham (2017) than Guillimin (2011/2013) 21

22 Guillimin Migration Questions regarding Guillimin compute and storage migration Contact us: Information regarding the migration will be updated and communicated to all users as the timeline and actions become more concrete Documentation for Cedar and Graham a_documentation Support inquiries: 22