CLC Genomics Workbench. Setup and User Guide

Transcription

1 CLC Genomics Workbench Setup and User Guide 1 st May 2018

2 Table of Contents Introduction... 2 Your subscription... 2 Bookings on PPMS... 2 Acknowledging the Sydney Informatics Hub... 3 Publication Incentives... 3 CLC Genomics Workbench Installation and License Server Access... 4 Downloading and installing CLC Genomics Workbench on your desktop... 4 Updating the license server address... 5 Connecting to the CLC Server (Artemis)... 6 Install the Server plugin... 6 Connect to CLC Server (Artemis)... 6 Pre-CLC on Artemis subscribers... 8 Microbial Genomics Module Plugin Installation... 9 Importing and Exporting Data Mounting classic RDS to your local desktop Downloading and installing FileZilla Transferring small data from your local desktop or classic RDS to CLC-project 11 Transferring large data or data stored on RCOS to and from CLC-project Data Analysis on CLC Server Artemis Data management How secure is my data? The import-export directory Further assistance Sydney Informatics Hub 1 May

3 Introduction These instructions enable CLC Genomics Workbench subscribers to use one of 11 Universityowned network licenses. Please note that each subscription covers one license; this means that a single user per subscribed group should access the license server at a time. This can be managed via the booking calendar. From the 30 th of April 2018, the CLC Server will be upgraded to the University s High Performance Computer (Artemis). This will provide users with access to Artemis resources and streamline connectivity with the Research Data Store (RDS). Artemis offers high computing power, higher throughput for CLC jobs and improved data security. Your subscription Thank you for purchasing a 6-month or 12-month subscription to use the CLC Genomics Workbench. During this time, please ensure that you manage your data efficiently and back important data up to the RDS regularly. At the end of your subscription, if you do not which to re-subscribe, your data will automatically be archived by the Sydney Informatics Hub. Bookings on PPMS After logging into PPMS, please book time using the calendar on Sydney Informatics Hub PPMS, under Book a system. If you intend to use the Microbial Genomics Module (MGM), please book time for both Workbench and Microbial Genomics Module (MGM) separately. **Important note regarding Workbench upgrades: When a Workbench upgrade is available, you will be prompted to upgrade the next time you open your Workbench. Please do not upgrade before confirming with the Sydney Informatics Hub at sih.info@sydney.edu.au that the latest Workbench version is compatible with the current USyd CLC License Server and CLC Server versions. Sydney Informatics Hub 1 May

4 Acknowledging the Sydney Informatics Hub The CLC Genomics Workbench is a service that is supported and provided by the Sydney Informatics Hub. Please support us by acknowledging us in your work. This is vital for the ongoing funding of the Sydney Informatics Hub. Publication Incentives The Sydney Informatics Hub offers regular incentive prizes to users of Sydney Informatics Hub facilities including the CLC Genomics Workbench. Keep an eye out on in the news section of our website for incentive announcements and be sure to enter each time you publish! Sydney Informatics Hub 1 May

5 CLC Genomics Workbench Installation and License Server Access You can use CLC Genomics Workbench as a stand-alone application on your desktop. The following instructions describe how to install the CLC Genomics Workbench and enable access to the University s CLC Genomics Workbench licenses. Downloading and installing CLC Genomics Workbench on your desktop If you do not have the latest CLC Genomics Workbench installed on your computer: 1. Select the appropriate download for your computer: Windows ch_10_1_1_64.exe Mac ch_10_1_1.dmg Linux ch_10_1_1_64.sh 2. Follow the installation setup, leave the default options checked and click accept/next as required 3. At the You need a license window: click Configure License Server Connection, Next, check Enable license server connection and select Manually specify license server. Enter the following details (please type the IP address not copy paste): Hostname/IP address: clcgenomics.lic.sydney.edu.au Port: 6200 Username: Your unikey (e.g. abcd1234) Sydney Informatics Hub 1 May

6 4. Click Next then Finish Updating the license server address If you already have CLC Genomics Workbench installed on your computer, you can update the license server details by following the instructions below. 1. For Windows, you must right click on the CLC icon and click Run as administrator. Mac start the CLC Genomics Workbench as usual 2. Go to Help -> License Manager 3. Click Upgrade Workbench License 4. At the You need a license window: click Configure License Server Connection, Next, check Enable license server connection and select Manually specify license server. Enter the following details: Hostname/IP address: clcgenomics.lic.sydney.edu.au Port: Click Finish Sydney Informatics Hub 1 May

7 Connecting to the CLC Server (Artemis) This will let you run CLC jobs on Artemis instead of your local machine. Install the Server plugin 1. For Windows, you must right click on the CLC icon and click Run as administrator. Mac start as usual 2. Go to Help -> Plugins -> Download plugins 3. Scroll down the list of plugins and select CLC Workbench Client Plugin. Download and install 4. Close the window and restart the workbench when prompted. You will now have a Server login option under the File tab Connect to CLC Server (Artemis) Each time you open the Workbench, you need to login to the Server. If you routinely run all your analyses on the Server, you may choose to check the Automatic login option on the CLC Server Login window. 1. Nominate a Researcher Dashboard (also known as DashR) project to have a directory in the CLC Genomics Server and send your project information as a PDF file to sih.info@sydney.edu.au. This PDF file can be downloaded from DashR (log in, click relevant project under Project Name and click the PDF icon. Download and save this PDF). Note: all users on this project will be granted access to your new CLC project folder Once we have processed your request, you can proceed to step Under File select CLC Server Login Sydney Informatics Hub 1 May

8 3. At User name, enter your unikey (e.g. abcd1234) followed by your password in the next field. Enter clcgenomics.hpc.sydney.edu.au into the server host field and change server port to Check save username and password and automatic login if desired 4. Once you have connected to the CLC Server (Artemis), on the left-hand panel in the navigation area, click clc_data and projects. Your new CLC-project folder where you can import files, create sub-folders and conduct analysis will be here. Sydney Informatics Hub 1 May

9 Here the CLC-project used is called CLC-SIHsandbox. Data can now be imported to your CLC-project folder and jobs run on Artemis by selecting a Grid option instead of Workbench during run or import/export operations. Pre-CLC on Artemis subscribers To older subscribers of CLC, you will notice that your data on the old CLC Genomics Server has been moved to clc data -> legacy -> abcd1234, where abcd1234 is your unikey. This data may not include recent data created on Thursday the 26 th of April. If any data was transferred, you can still export it out of the old CLC Genomics Server and import it into the Artemis CLC Genomics Server manually (see Importing and Exporting data section) until Friday the 11 th of May. You are welcome to continue your analysis in the legacy folder. However, please note that data in the legacy directory is not completely secure other users can access your data here. If you would like a folder with restricted access, please nominate a DashR project directory and we can set up a CLC-project folder for you (step 1 in Connect to CLC Server Artemis ). Please contact sih.info@sydney.edu.au if you have any issues or further questions. Sydney Informatics Hub 1 May

10 Microbial Genomics Module Plugin Installation There are two MGM licenses. Any subscribed user can use MGM (at no additional cost). Once you have installed the plugin, you will see Microbial Genomics Module under your toolbox. Use of an MGM tool uses a license, so please only use an MGM tool if you have booked a license to avoid interfering with other users booked MGM sessions. 1. For Windows, you must right click on the CLC icon and click Run as administrator. Mac start as usual 2. In the workbench, go to Help -> Plugins and resources -> Download plugins 3. Scroll down the list of plugins and select CLC Microbial Genomics Module. 4. Download (this may take some time) and install 5. Close the window and restart the workbench when prompted. You will now have MGM in your toolbox To get you started: You can create a project folder by the standard process of File -> New -> Folder, or by clicking the New icon near the top left of the workbench. To import data, similarly you can use File -> Import or use the Import icon in the taskbar. For importing sequencing data, be sure to select the relevant sequencing platform and ensure the correct check boxes are selected, such as whether the data is paired end. Analyses are found under Toolbox. For detailed information on the analysis tools available, see the documentation at: Review the numerous specific workflow tutorials available at: Sydney Informatics Hub 1 May

11 Importing and Exporting Data Mounting classic RDS to your local desktop The import-export folder, a classic RDS system, (described in importing large data or data stored on RCOS) and your own group s classic RDS systems can be mounted onto your local computer. This is useful for transferring data between different systems (e.g. RDS and Artemis), even when you are not using the CLC Genomics Workbench. If you are off-campus, you will need to connect to the University s Network by using a VPN client. Please follow the instructions provided here by ICT to map the import-export folder. If your DashR project also uses classic RDS, you may also which to map this to your local computer. For Mac OSX users The server address for your project s classic RDS is (replace FAC and ProjectName with your faculty and DashR project abbreviation) is: smb://research-data.shared.sydney.edu.au/fac/projectname The import-export server address is: smb://research-data.shared.sydney.edu.au/rds-01/prj-clc/import-export For Windows users The server address for your project s classic RDS (replace FAC and ProjectName with your faculty and DashR project abbreviation, step 4 Windows 7, step 3 Windows 8/10) is: \\research-data.shared.sydney.edu.au\fac\projectname The import-export server address is: \\research-data.shared.sydney.edu.au\rds-01\prj-clc\import-export Downloading and installing FileZilla FileZilla is a free application that will allow you to transfer files between your local desktop (including any drives mounted to your local desktop) and any other remote server, including Artemis and RCOS. For CLC Genomics, it is useful for uploading/downloading data between your local computer or mounted classic RDS ( local ) to the import-export directory ( remote, described in importing large fata of data stored on RCOS). If your DashR project uses RCOS and not classic RDS, you can mount the import-export directory instead ( local ) and connect to RCOS as a remote server. To download Filezilla, go to this website: Download the client, not the server. Install Filezilla by following the prompts. Sydney Informatics Hub 1 May

12 Transferring small data from your local desktop or classic RDS to CLC-project Please use these instructions from importing small files (<10 Gb) from your desktop or classic RDS which has been mounted onto your desktop. The next set of instructions require you to keep your computer on with a constant internet connection and connection to the University Network, which is why it is only recommended for transferring small files. For larger files, you will need to use the import-export directory which is described in Importing large data. 1. Open the CLC Genomics Workbench and make sure you are connected to the CLC Server (see Connect to CLC Server (Artemis)). 2. If you are importing data from your classic RDS, you will need to have this mounted onto your local desktop (see Mounting classic RDS to your local desktop. You don t need to do anything if you are importing from your local desktop). In the example provided here, the abbreviated DashR project name is DNMSinDogs and it uses classic RDS. We will import data in the InputFastq folder into CLC Genomics. 3. Click the import icon and the type of files you would like to import Sydney Informatics Hub 1 May

13 4. Select Workbench and click Next 5. You will then be asked to select the files to import. If you are importing from classic RDS, you will find your mounted drive under devices. Navigate to the folder containing your files. Click and hold shift to select multiple files. Click Next. Sydney Informatics Hub 1 May

14 6. Leave Save selected and click Next Sydney Informatics Hub 1 May

15 7. Now select where you want to save the data on the CLC Genomics Workbench. You should select your CLC-project folder. In this example, we will save the files in the CLC-project folder named CLC- SIHsandbox (yours will be called something different). Inside this folder, we previously created a folder called RawFastqs. We will select this folder. 8. Click Finish 9. The progress of the file transfer will appear in the Toolbox. Once complete, you should see your files in your chosen folder. Sydney Informatics Hub 1 May

16 A similar process is used to export files from your CLC-project folder to your local desktop or classic RDS that has been mounted to your local desktop. Sydney Informatics Hub 1 May

17 Transferring large data or data stored on RCOS to and from CLC-project The next set of instructions will describe how to import large data or data that is stored on RCOS into your CLC-project folder. This requires the import-export directory to be mounted onto your local computer (see mounting classic RDS to your local desktop). You will also need FileZilla (see Downloading and installing FileZilla). In the example outlined below, we will import data from a DashR project named Tracy to our CLC-project folder, here named CLC- SIHsandbox. The advantage of this process is that you can turn off your computer or CLC Genomics Workbench after you have started the transfer. The transfer will be running in the background on Artemis between the import-export directory and your CLC-project folder. This is handy for very large file transfers which may take hours to complete. 1. Ensure that the import-export directory is mounted to your local computer (see Mounting classic RDS to your local desktop). 2. Open FileZilla. We will use FileZilla to transfer between local (import-export mounted directory) and remote servers (RCOS PRJ-Tracy). First type the following into the fields at the top: Host: sftp://rcos-int.sydney.edu.au Username: abcd1234 Password: unikeypassword Replace abcd1234 with your own unikey and type in your regular unikey password into the Password field. Leave Port empty (default). Click Quickconnect. 3. The left-hand panels show your local desktop. Navigate to the import-export directory. Double click on your unikey folder to enter it. Sydney Informatics Hub 1 May

18 4. The right-hand panel shows your remote site (in this example RCOS). Navigate to the location of your files to import in your project s RCOS directory (Your RCOS directory will be called /rds/prj-projectname. In this example the ProjectName is Tracy) Note: You can navigate to a folder by typing in the path after Local site: (/Volumes/import-export/abcd1234) or Remote site: (/rds/prj-tracy/inputfastq) as above. Alternatively, you can click into folders below. You can click on the folder that is called.. to go back up to the parent folder. 5. Drag and drop the files you wish to copy (see below). You can select multiple files by clicking and holding the shift key. Your files will now be in your unikey folder in the import-export directory. 6. Open the CLC Genomics Workbench and click Import. Select the appropriate file type. Sydney Informatics Hub 1 May

19 7. Select Grid and in the drop-down menu, select one of the Artemis Data Transfer options according to how large your data is (if you are unsure, make your best guestimate). Click Next. 8. Select On the server or a place that the server has access to. Click Next Sydney Informatics Hub 1 May

20 9. You will see your unikey directory under import-export. Select the files you wish to import (click and hold shift to select multiple files). Then click the blue arrow pointing right. 10. Once the files you wish to upload are in Selected Elements, click Next Sydney Informatics Hub 1 May

21 11. Leave Save selected and click Next Sydney Informatics Hub 1 May

22 12. Select your CLC-project directory under clc_data. Here the CLC-project directory is called CLC-SIHsandbox and we have created a folder called RawFastq2. Click Finish. 13. On the bottom left corner, a message will appear to indicate that your transfer job is queued on Artemis. In this example, is the job ID. 14. [Optional] For users who are experienced with Artemis, you can check the progress of the job transfer by typing the following into the command line: qstat <job id> Sydney Informatics Hub 1 May

23 Under S (Status), the letter R indicates that the job is running. The letter Q means that the job is in queue notifications will also advise you of the status of your job. You will receive one when your job has begun. You will receive another once your job has completed. The will appear similar to the one below: Note: You will receive a third stating that there was a Post job file processing error (see below) please ignore this Your files that were imported from the import-export folder should now be in the selected sub-folder ( RawFastq2 ) in your CLC-project folder (CLC-SIHsandbox). 17. Please remove your data from the import-export directory. The import-export directory is not secure so it is important that the data is removed once your data transfer job has completed. Any data left in the import-export directory may be periodically deleted without warning. Sydney Informatics Hub 1 May

24 Sydney Informatics Hub 1 May

25 Data Analysis on CLC Server Artemis Perform a single analysis 1. Select an analytical task 2. Select Grid and an appropriate Compute node on Artemis in the drop-down menu to process your job. Select this according to the computational resources that your job requires (estimate to the best of your ability). C queues to do not provide notifications of when the job will begin and end, CE queues do send notifications. 3. Select the parameters for your task. See the CLC Manual for an explanation of the analysis tools and parameters. ndex.php?manual=introduction_clc_genomics_workbench.html Sydney Informatics Hub 1 May

26 Here all other options were left as default. Click Next to continue. Sydney Informatics Hub 1 May

27 4. Select a location within your CLC-project folder to save your results. 5. A message will indicate the CLC Server has submitted a job to the compute node and a job id has been allocated. You can also see the progress of your job in CLC Genomics on the bottom left corner. 6. You can check the status of the job in a command line terminal connected to Artemis HPC by typing qstat <job id> The status of the job will also be displayed in the CLC Genomics Workbench and ed to you. Note: Please ignore any Post job file processing error s that you receive. Sydney Informatics Hub 1 May

28 7. Your results should appear in the CLC Server folder location you specified. Sydney Informatics Hub 1 May

29 Data management The data management guidelines for data stored on the CLC server are similar to those for the Artemis HPC. The CLC server is recommended to be used for data analysis and data processing only and not for data storage. For data that is not accessed frequently, this data should be moved/exported to your Classic RDS or RCOS RDS. If you have large datasets (>1TB) that are accessed frequently and needs to be stored on the CLC Server, please let the Sydney Informatics Hub know or ). How secure is my data? Data that is in your CLC-Project folder is only accessible to the users in your DashR project. This data is not backed up. We recommend that you store valuable data in one of the RDS systems classic RDS or RCOS. You can read more about each of these RDS systems here. The import-export directory The import-export directory is to be purely used for data transfers as it is not completely secure. Once your transfer is complete, we recommend you delete your data from this directory immediately. Any data left in the import-export directory may be periodically deleted without warning. Further assistance For assistance with installation, support, or bioinformatics training please contact the Sydney Informatics Hub at or on Sydney Informatics Hub 1 May