Getting Started with TotalView on Beacon

Transcription

1 Getting Started with TotalView on Beacon 1) ssh X uname@beacon.nics.utk.edu (sets x forwarding so you can use the GUI) 2) qsub X I A UT- AACE- BEACON (sets x forwarding in the interactive job) 3) compile the offload example: a. icc openmp g o omp_pi_offload omp_pi_offload.c lm 4) NOTES ON OFFLOAD: a. The offload code is purely OpenMP, since having multiple MPI processes on the host offloading arbitrarily to the MIC is not a robust method. With some work this can be done safely, but it is out of the scope of this exercise. b. It is preferable to run from /luster/medusa/ in order that all symbols can be accessed by Totalview on host and MIC. 5) Run: module load totalview/ ) Run: totalview 7) In order to have Totalview connect to the MIC for command line debugging, create a.tvdrc file in your home directory. This lets Totalview know about our specific micssh and micscp commands by adding the following lines (for native and offload, respectively): dset TV::mic_native_server_launch_string {micssh - n %R "/bin/rm - f /User/tvdsvr*"; micscp %B/tvdsvr* %R:/User; micssh - n %R - n "/User/tvdsvr_mic - working_directory %D - callback %L - set_pw %P - verbosity %V %F"} dset TV::micnativeloadex_server_launch_string {micnativeloadex %B/tvdsvrmain_mic - d %d - a "- callback %L - set_pw %P - verbosity %V %F"} 8) In the panel under Program, enter the path to the executable.

2 9) Hit OK, then the following screen will pop up where you may set a break point. 10) Click Run, and when it asks if you d like to pause the parallel process, say yes; then, enter the remote host you are offloading to (e.g., beacon8- mic0) in order to step into the thread actions.

3 11) Finally, your answers will print out in the terminal window (should be ) and the breakpoint window will disappear. You may begin again by clicking File > NewProgram in the control window. 12) NOTE: If you must exit Totalview, use Ctrl+C from the terminal window else your x forwarding may break and you will have to start from the beginning! 13) NATIVE MODE EXAMPLE: compile native node example: a. icc mmic - openmp - g - o omp_pi_native omp_pi_native.c - lm 14) NOTES ON NATIVE MODE: You will need to have: a. Copied over the executable to the MIC card i. cp omp_pi_native $TMPDIR/mic0 ii. cp omp_pi_native $TMPDIR (so the host can see the debug symbols) iii. Set up a machine file if you want to use more than one MIC as the host 15) Compile MPI native mode example: a. mpiicc mmic - openmp - g - o omp_pi_mpi omp_pi_mpi.c lm 16) Now, execute your code like normal, except add tv right after micmpiexec. Alternately, you may use:

4 totalview - mmic - args mpiexec.hydra - bootstrap ssh - bootstrap- exec micssh - machinefile $machines - n $PROCS - wdir $TMPDIR $TMPDIR/EXE 17) Set some breakpoints by clicking on the line number in the window that comes up after hitting Enter and then Run, then click on Run again. 18) Now, you can see how many threads are executing at this point under each MPI rank. Also, you can switch to any of them by double clicking in the control window (highlighted in black in the pic below). 19) Now, click on the first breakpoint to delete it, and hit run again to continue on. Now, you should see what is in the second picture below, and you can choose which thread to view by double clicking on it in the control window.

5 20) Finally, you should get the answers printed in the terminal window with the associated rank, as seen below. 24) Running OpenMP on a remote host (the MIC):

6 Open Totalview. Click A new program (not parallel program). Select the program from the drop down menu or browse. Now, click Add Host. Enter the node number and mic as follows: beaconxxx- micx. Finally, under advanced, give the following Remote Server Launch Command: micssh - n %R "/bin/rm - f /User/tvdsvr*"; micscp %B/tvdsvr* %R:/User; micssh - n %R - n " env LD_LIBRARY_PATH=$MIC_LD_LIBRARY_PATH /User/tvdsvr_mic - working_directory %D - callback %L - set_pw %P - verbosity %V %F" Now, simply run the code and debug as if you were on the host! Congratulations! You re now officially a TotalView User! Find more info and helpful hints at This material is based upon work supported by the National Science Foundation under Grant Number Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.