MPE at SCS

Description

MPE is a software package for MPI ( Message Passing Interface ) programmers. The package provides users with a number of useful tools for their MPI programs. Current available tools under MPE are :

• A set of profiling libraries to collect information about the behaviour of MPI programs. Linking user MPI program with the libraries will generate logfile for post-mortum visualization when user program is executed.

• Various viewers for the logfiles.

• A shared-display parallel X graphics library.

• A profiling wrapper generator for MPI interface.

• Routines for sequentializing a section of code being executed in parallel.

• Debugger setup routines.

MPE is developed and supported by the Mathematics and Computer Science division of Argonne National Labratories. It was initially designed and is routinely distributed with their version of MPI, MPICH-2, but in practice can be configured to work with any of the MPI2 compliant distributions of MPI. The main website for MPE is :

http://www-unix.mcs.anl.gov/perfvis/software/MPE/index.htm

and the user manual can be downloaded in pdf format from :

ftp://ftp.mcs.anl.gov/pub/mpi/mpeman.pdf

General Use

The general usage for creating a log file of the performance of your MPI routines within a code written with MPI is simple and involves no modification to the existing code. MPE can be used to log non-MPI related routines such as various computations within the code but this will involve modifying the actual code and essentially creating logging comands with MPE routines to track this desired information. For information on how to log such additional information please consult the users manual on how to create the logging commands tailored towards non-MPI related routines. Also, there is very good instructions on how to do this with examples in the book, Using MPI2 by Gropp, Lusk, and Thakur which can be bought through MIT press at,

http://mitpress.mit.edu/catalog/item/default.asp?ttype=2&tid=3490

So whether you are just logging MPI related routines or have modified your code to log additional routines creating the log files are fairly easy and are done in the 2 steps, 1 during compilation and the 2nd during linking. So that the only significant difference from just running your MPI code is that you must compile and link the code in 2 steps with additional flags set to call forth the MPE library. After compilation and linking is finished, with these additional flags, the code may be run as usual using either mpirun or mpiexec for interactive use or through any scheduling system that is in use on the particular system you are computing on. When the code finishes executing a log file will be created with the extension .clog2.

MPE also provides a program to visualize these logfiles for you called, jumpshot. Jumpshot provides an interactive GUI that allows one to visualize the .clog2 files and manipulate the visualization interactively. The program will initially convert the .clog2 files to a .slog2 file which can be viewed and worked with directly multiple times in the future.

Cluster Specific Use

Phoenix

The MPE libraries have been built to work with the openMPI distribution on Phoenix in a special directory :

• The MPE root directory for the gcc-compiled version is /opt/openmpi/bin.

• Use the sge environment openmpi for submitting openmpi jobs.

Commands to compile and link the MPI code to create the MPE log file

In this example we compile, link, and run a fortran90 code named poisson.f90 on 4 processors interactively.

    mpif90 -I/opt/openmpi/include -c poisson.f90
    mpif90 -o poisson poisson.o -L/opt/openmpi/lib -lmpe_f2cmpi -llmpe -lmpe -lm
    mpirun -np 4 poisson
   

When this code finishes execution it will generate the file, unknown.clog2

Commands to visualize the log file

It is important to note that in order to use jumpshot you must have X11 active, i.e. on Linux you must use the flag -X or -Y when you ssh to phoenix and on a MAC you must ssh to phoenix using your Xterm with the flag -Y, and if you are a windows user then may the gods have mercy on your soul. If X11 is active then simply type,

    jumpshot unknown.clog2
   

And a prompt will open asking if you would like to convert the file to slog2 which you should click the yes button. A new prompt will open and it is here that you will be able to name the file what ever you would like but it must have the .slog2 extension. After you name it something more meaningful than unkown.slog2 then click convert and when it finishes click OK and the GUI will open up and you will be able to see a legend explaining the colors, which will be associated with the various routines that have been logged and you will have another window with a visualization of your routines seperated into the number of processors that were used during execution.

Once you have a .slog2 file saved you won't need to convert it to view it with jumpshot you can simply type:

    jumpshot poisson.slog2
   

and you will go directly to the visualization of the logged files.

Commands to be added to .cshrc file on Phoenix for openMPI version

    setenv LD_LIBRARY_PATH /opt/openmpi/lib
    setenv OPENMPIHOME /opt/openmpi/
    set path = ( $path $OPENMPIHOME/bin )
   

Commands to be added to .bashrc file on Phoenix for openMPI version

    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/openmpi/lib
    export PATH=${PATH}:/opt/openmpi/bin
   

-- SivaSakti - 08 Aug 2007