                      MPE (Multi-Processing Environment)
                      ----------------------------------

                          Version 1.2.2. July, 2001

                    Mathematics and Computer Science Division
                           Argonne National Laboratory

I.  INTRODUCTION
----------------

The Multi-Processing Environment (MPE) attempts to provide programmers with 
a complete suite of performance analysis tools for their MPI programs based
on post processing approach.  These tools include a set of profiling libraries, 
a set of utility programs, and a set of graphical tools.

The first set of tools to be used with user MPI programs is profiling libraries
which provide a collection of routines that create log files.  These log files
can be created manually by inserting MPE calls in the MPI program, or 
automatically by linking with the appropriate MPE libraries, or by combining 
the above two methods.  Currently, the MPE offers the following 3 profiling 
libraries.

   1) Tracing Library - Traces all MPI calls.  Each MPI call is preceded by a 
      line that contains the rank in MPI_COMM_WORLD of the calling process, 
      and followed by another line indicating that the call has completed.
      Most send and receive routines also indicate the values of count, tag,
      and partner (destination for sends, source for receives).  Output is to
      standard output.

   2) Animation Libraries - A simple form of real-time program animation 
      that requires X window routines.  

   3) Logging Libraries - The most useful and widely used profiling libraries 
      in MPE.  They form the basis to generate log files from user MPI programs.
      There are currently 3 different log file formats allowed in MPE.  The 
      default log file format is CLOG.  It is basically a collection of events 
      with single timestamps.  And there is ALOG which is provided for backward
      compatibility reason and it is not being developed.  And the most 
      powerful one is SLOG, stands for Scalable LOGfile format, which can be
      converted from CLOG file after CLOG file has been generated (preferred
      approach), or can be generated directly when MPI program is executed 
      (through setting the environmental variable MPE_LOG_FORMAT to SLOG).

The set of utility programs in MPE includes log format converter (e.g. 
clog2slog), logfile print (e.g. slog_print) and logfile viewer wrappers, 
logviewer, which selects the correct graphical tool to display the logfile 
based on the logfile's file extension.

Currently, MPE's graphical tools includes 3 display programs, upshot for
ALOG, jumpshot-2 for CLOG and jumpshot-3 for SLOG.  The logviewer script 
eliminates the need for user to remember the relationship of logfile formats 
to display programs.





II. CONFIGURATION
-----------------

MPE can be configured and installed as an extension to most MPI standard
compliant MPI implementations, e.g. MPICH, LAM, SGI's MPI, HP-UX's MPI 
and IBM's MPI.  It has been integrated seamlessly into MPICH distribution, 
so MPE will be installed automatically during MPICH's installation process.  



II. a)  Configuration Model
---------------------------

MPE is designed to be used as an extension to an existing MPI implementation,
so its configuration model assumes a general MPI development environment.  
Here are the some of the variables that MPE configure reads, some are read 
as environmental variables and some are read from the command line arguments 
to configure.

CC          : C compiler used to create serial executable,
              e.g. xlc for IBM MPI.
MPI_CC      : C compiler used to compile MPI program and to create parallel 
              executable,
              e.g. mpcc for IBM MPI, or mpicc for MPICH.
MPE_CFLAGS  : CFLAGS for CC and MPI_CC.

F77         : F77 compiler used to create serial executable,
              e.g. xlf for IBM MPI.
MPI_F77     : F77 compiler used to compile MPI program and to create parallel
              executable,
              e.g. mpxlf for IBM MPI, or mpif77 for MPICH.
MPE_FFLAGS  : FFLAGS for F77 and MPI_F77.

MPI_INC     : compiler's include flag (with prefix "-I") for MPI_CC/MPI_F77, 
              e.g. "-I/usr/include" for mpi.h on IRIX64.
MPI_LIBS    : compiler's library flag (with prefix "-L" for library path and
              prefix "-l" for each library name) needed by MPI_CC/MPI_F77,
              e.g. "-L/usr/lib -lmpi" for libmpi.a on IRIX64.
F2CMPI_LIBS : compiler's library flag for Fortran to C MPI wrapper library,
              e.g. "-lfmpich" when MPI_CC=mpicc & MPI_F77=mpif77 for MPICH.

Among above listed variables, CC, MPI_CC, F77 and MPI_F77 are usually set
by the corresponding environmental variables.  The rest can be set through 
command line arguments to configure.  In some MPI implementations, like 
HP-UX's, MPI_CC and MPI_F77 are reserved for use by the MPI implementation,
use the configure options to set MPI_CC and MPI_F77 instead.



II. b)  Build Options and Features
----------------------------------

MPE's configure is written using autoconf 2, and supports VPATH style install  
process.  It means the actual source directory and the building directory
can be in 2 different locations.  This allows the same source directory to be
used to build multiple versions of MPE with different options and still won't 
mess up the original source.  It is highly recommended that user should do a 
VPATH build.  Also MPE involves several different independent packages, in 
order to create a tightly integrated environment for user, it is recommended 
that user should do a "make install" to install the MPE in a separate directory
after the build is done.  The benefit is that all utility programs will be in
bin/, all libraries will be in lib/ and all graphic tools will be nicely 
organized in share/ ...

There are 2 types of configure options.

	1)  MPI implementation and User options
	2)  Generic configure flags supplied by autoconf 2

For a list of flags/switches for type 1 (not type 2) in MPE, use the script
"configure--help".

The following is not a complete list but some of the more important ones.
Generic flags:
--prefix=INSTALL_DIR            Specifies the final install directory for 
                                "make install".  All libraries, utility
                                programs, graphic programs and examples
                                are installed in a standard directory
                                structure without files created in the
                                building process.

--x-includes=X_INC              Specifies the directory where X include
                                files are located.  This is used when
                                configure has trouble in locating X in
                                user system.

--x-libraries=X_LIBS            Specifies the directory where X libraries
                                are located.  This is used when configure
                                has trouble in locating X in user system.
	
MPI implementation Options:
--with-mpicc=MPI_CC             Specify MPI C compiler to generate parallel
                                executable, e.g. mpcc for AIX.

--with-mpif77=MPI_F77           Specify MPI F77 compiler to generate parallel
                                executable, e.g. mpxlf for AIX.

--with-cflags=MPE_CFLAGS        Specify extra CFLAGS to the C and MPI_CC 
                                compilers, e.g. "-64" for IRIX64 C compiler

--with-fflags=MPE_FFLAGS        Specify extra FFLAGS to the F77 and MPI_F77
                                compilers, e.g. "-64" for IRIX64 F77 compiler

--with-mpiinc=MPI_INC           Specify compiler's include flag for MPI 
                                include directory,
                                e.g. "-I/pkgs/MPI/include" for mpi.h 

--with-mpilibs=MPI_LIBS         Specify compiler's library flag for MPI 
                                libraries,
                                e.g. "-L/pkgs/MPI/lib -lpmpich -lmpich"

--enable-f77                    Enable the compilation of routines that 
                                require a Fortran compiler.  If configuring 
                                with MPICH, the configure in the top-level 
                                MPICH directory will choose the appropriate 
                                value for you.  However, it can be overridden.
                                The default is yes, --enable-f77.

--enable-f2cmpilib              Enable the building of MPE's internal Fortran 
                                to C MPI wrapper library.  The default is yes,
                                --enable-f2cmpilib

--with-f2cmpilibs=F2CMPI_LIBS   Specify compiler's library flags for Fortran
                                to C MPI wrapper library.  Using this option 
                                will force --disable-f2cmpilib.
                                e.g. "-lfmpich" when configuring MPE for MPICH

Other User Options:
--enable-echo                   Turn on strong echoing.  The default is no,
                                --disable-echo.

--with-mpelibname=MPE_LIBNAME   Specify the MPE library name instead of the 
                                default 'mpe'.  e.g. if MPE_LIBNAME="MPE",
                                then the libraries generated will be libMPE.a,
                                liblMPE.a, libtMPE.a, libaMPE.a and
                                libMPE_f2cmpi.a.  This option is necessary
                                when configuring MPE for a existing and older
                                version of MPICH which has MPE installed.

--enable-mpe_graphics           Enable the building of MPE graphics routines.
                                If disabled, then the MPE routines that make 
                                use of X11 graphics will not be built.  This 
                                is appropriate for systems that either do not 
                                have the X11 include files or that do not 
                                support X11 graphics.  The default is 
                                enable=yes.

--enable-viewers                Enable the build of all the available log 
                                viewers.  The default is enable=yes

--with-java=JAVA_HOME           Specify the path of the top-level directory 
                                of the Java, JDK, installation.  If this 
                                option or --with-javaX is not given, configure
                                will try to locate JDK for you to build 
                                Jumpshot-2 and Jumpshot-3.  JDK 1.1.6 to 
                                JDK 1.1.8 can be used to build both Jumpshots.
                                It is recommended that user should use this 
                                option when there is only 1 version of JDK 
                                installed on the machine.

--with-java1=JAVA_HOME          Specify the path of the top-level directory
                                of the Java, JDK, installation for Jumpshot-2
                                only.  If this option or --with-java is not 
                                given, Jumpshot-2's configure will try to 
                                locate JDK for you to build Jumpshot-2.  For
                                performance reason, it is recommended to use
                                the latest Java 1, i.e. JDK-1.1.8, to build
                                Jumpshot-2.

--with-java2=JAVA_HOME          Specify the path of the top-level directory
                                of the Java, JDK, installation for Jumpshot-3
                                only.  If this option or --with-java is not 
                                given, Jumpshot-3's configure will try to 
                                locate JDK for you to build Jumpshot-3.  For
                                performance reason, it is recommended to use
                                the latest Java 2, i.e. JDK-1.3.x, to build
                                Jumpshot-3.

--with-wishloc=WISHLOC          This switch specifies the name of Tcl/Tk wish
                                executable.  If this switch is omitted, 
                                configure will attempt to locate a version.  
                                This is used only for upshot.  
                                Note: Because Tcl and Tk keep changing in 
                                incompatible ways, we will soon be dropping 
                                support for any tool that uses Tcl/Tk.

In order to achieve maximum performance benefit, it is recommended to
use latest Java 1, i.e. JDK-1.1.8, as well as latest Java 2, i.e. JDK-1.3.1
to build Jumpshots.  So instead of using --with-java=<JDK-1.1.8> option, 
--with-java1=<JDK-1.1.8> and --with=java2=<JDK-1.3.1> should be used instead.




III. INSTALLATION INSTRUCTIONS
-------------------------------  

As noted earlier, the MPE library can be installed as part of the MPICH
configure or as an extension of an existing MPI implementation.  Below are
instructions and examples for typical installation of MPE on popular MPI 
implementations. 



III. a)  Configuring as part of the MPICH configure
---------------------------------------------------

The configure in the MPICH directory will try to determine the necessary 
information and pass it to the MPE configure.  If no options are given, 
the MPE will automatically be configured by default.  However, the user 
can provide extra configuration information to MPE through MPICH 
configure with the following options:

	-mpe_opts=MPE_OPTS

where MPE_OPTS is one or more of the choices in section II.  Multiple
instances of -mpe_opts are allowed to specify different options for 
the MPE configure.



III. b) Configuring as part of an existing MPI implementation
-------------------------------------------------------------

The following are some examples for configuring MPE for an existing
MPI implementation.

For SGI MPI, e.g. denali.mcs.anl.gov, do the following for default ABI, -n32

    setenv MAKE gmake
    ${MPE_SRC_DIR}/configure --with-mpilibs=-lmpi \
                             --with-java=/usr/java-1.1.6/usr/java
    make
    make install PREFIX=${MPE_INSTALL_DIR}

for 64 bits ABI, add options --with-cflags=-64 and --with-fflags=-64 to the
configure options.


For IBM MPI, e.g. quad.mcs.anl.gov, do

    setenv MPI_CC mpcc
    setenv MPI_F77 mpxlf
    ${MPE_SRC_DIR}/configure --with-java=/homes/chan/pkgs/java/J1.1.8
    make
    make install PREFIX=${MPE_INSTALL_DIR}


For HP-UX's MPI implementation,

    ${MPE_SRC_DIR}/configure --with-mpicc=mpicc \
                             --with-mpif77=mpif77 \
                             --with-flib_path_leader="-Wl,-L"
    make
    make install PREFIX=${MPE_INSTALL_DIR}

MPE's Fortran support on HP-UX's MPI is NOT working yet.  So to get MPI 
Fortran code to generate logfile, you could use HP-UX's libfmpi.a if it
is there.  Here is the configure options.

    ${MPE_SRC_DIR}/configure --with-mpicc=mpicc \
                             --with-mpif77=mpif77 \
                             --with-flib_path_leader="-Wl,-L" \
                             --with-f2cmpilibs=-lfmpi
    make
    make install PREFIX=${MPE_INSTALL_DIR}


For prebuilt version of MPICH-1.1.2 or MPICH-1.2.0, e.g. donner, do

    setenv MPI_CC ${MPICH_INSTALL_DIR}/mpicc
    setenv MPI_F77 ${MPICH_INSTALL_DIR}/mpif77
    ${MPE_SRC_DIR}/configure  --with-f2cmpilibs=-lfmpich \
                              --with-mpelibname=newMPE \
                              --with-java1=/sandbox/jdk117_v3 \
                              --with-java2=/sandbox/jdk1.3.1
    make
    make install PREFIX=${MPE_INSTALL_DIR}

It is important to use the configure option --with-mpelibname to specify a 
different MPE library name than the default "mpe" when configuring MPE for
older MPICH.  Without this option, the linkage tests in MPE would most likely
use the old MPE libraries in the MPICH instead of the newly built MPE 
libraries in resolving the MPE symbols.  Also the option --with-f2cmpilibs
forces MPE to use the Fortran to C MPI wrapper library in previous
version of MPICH.


For LAM, do

    setenv MPI_CC ${LAM_INSTALL_DIR}/bin/mpicc
    setenv MPI_f77 ${LAM_INSTALL_DIR}/bin/mpif77
    ${MPE_SRC_DIR}/configure --with-mpilibs="-L${LAM_INSTALL_DIR}/lib -lpmpi" \
                             --with-java1=/sandbox/jdk117_v3 \
                             --with-java2=/sandbox/jdk1.3.1
    make
    make install PREFIX=${MPE_INSTALL_DIR}

Using MPE with LAM for fortran MPI program is not working until recently.  
Configure options listed above enable MPE's internal Fortran to C MPI library.
To use LAM's Fortran to C MPI library in LAM 6.3.3 or later, liblamf77mpi.a, 
do

    setenv MPI_CC ${LAM_INSTALL_DIR}/bin/mpicc
    setenv MPI_f77 ${LAM_INSTALL_DIR}/bin/mpif77
    ${MPE_SRC_DIR}/configure --with-mpilibs="-L${LAM_INSTALL_DIR}/lib -lpmpi" \
                             --with-f2cmpilibs=-llamf77mpi \
                             --with-java1=/sandbox/jdk117_v3 \
                             --with-java2=/sandbox/jdk1.3.1
    make
    make install PREFIX=${MPE_INSTALL_DIR}





IV.  EXAMPLE PROGRAMS
----------------------

As previously noted, the MPE library is composed of 3 different profiling
libraries.  During configure, the compiler's library linkage flags and 
appropriate libraries are determined.  These variables are first substituted
in the Makefile in the directory mpe/contrib/test.  The Makefile is then 
installed into directory share/examples during the final installation process.
The following is a list of these variables:

LOG_LIBS   = compiler's library flag needed to link with the logging library
TRACE_LIBS = compiler's library flag needed to link with the tracing library
ANIM_LIBS  = compiler's library flag needed to link with the animation library

The variable FLIB_PATH is the compiler's library path needed to link fortran 
MPI programs with the logging library.

During make, small test programs cpi.c and fpi.f will be linked with 
each of the above libraries.  In the output from Make, a message will be 
written regarding the success of each attempted link test.  The success 
of these linkage tests will also be included in the Make output.  If the
linkage tests are successful, then these library linkage flags can be used
for your programs as well.   

The following example programs are also included in the mpe/contrib directory:

    In mpe/contrib/mandel is a Mandelbrot program that uses the MPE 
    graphics package.  

    In mpe/contrib/mastermind is a program for solving the Mastermind
    puzzle in parallel.

These programs should work on all MPI implementations, but have not been
extensively tested.





V.  MPEINSTALL
--------------

A 'mpeinstall' script is created during configuration.  If configuring with
MPICH, then the 'mpiinstall' script will invoke the 'mpeinstall' script.  
However, 'mpeinstall' can also be used by itself.  This is only optional and 
is of use only if you wish to install the MPE library in a public place so 
that others may use it.  Final install directory will consist of an include,
lib, bin, sbin and share subdirectories.  Examples and various logfile viewers
will be installed under share.





VI. USAGE
---------

The final install directory contains the following subdirectories.

    include/ contains all the include files that user program needs to read.
    lib/     contains all the libraries that user program needs to link with.
    bin/     contains all the utility programs that user needs to use.
    sbin/    contains the MPE uninstall script to uninstall the installation.
    share/   contains user read-only data.  Besides share/examples/, user 
             usually does NOT need to know the details of other subdirectories.

In terms of usage of MPE, user usually only need to know about the files
that have been installed in include/, lib/ and bin/.  



VI. a) CUSTOMIZING LOGFILES
---------------------------

In addition to using the predefined MPE logging libraries to log all MPI 
calls, MPE logging calls can be inserted into user's MPI program to define
and log states.  These states are called User-Defined states.  States may 
be nested, allowing one to define a state describing a user routine that 
contains several MPI calls, and display both the user-defined state and 
the MPI operations contained within it.

The routine MPE_Log_get_event_number() has to be used to get unique
event numbers (this is important if you are writing a library that uses
the MPE logging routines) from the MPE system.  The routines
MPE_Describe_state() and MPE_Log_event() are then used to describe 
user-defined states.

\begin{verbatim}

int eventID_begin, eventID_end;
...
eventID_begin = MPE_Log_get_event_number();
eventID_end   = MPE_Log_get_event_number();
...
MPE_Describe_state( eventID_begin, eventID_end, "Amult", "bluegreen" );
...
MyAmult( Matrix m, Vector v )
{
    /* Log the start event along with the size of the matrix */
    MPE_Log_event( eventID_begin, m->n, (char *)0 );
    ... Amult code, including MPI calls ...
    MPE_Log_event( eventID_end, 0, (char *)0 );
}

\end{verbatim}

The logfile generated by this code will have the MPI routines within the
routine MyAmult() indicated by a containing bluegreen rectangle.

If the MPE logging library, liblmpe.a, are NOT linked with the user program, 
MPE_Init_log() and MPE_Finish_log() need to be used before and after all 
the MPE calls.   Sample programs cpilog.c and fpi.f are available in MPE 
source directory contrib/test or the installed directory share/examples to 
illustrate the use of these MPE routines.



VI. b) EXAMPLE MAKEFILE 
-----------------------

share/examples contains some very useful and simple example programs and
Makefile which illustrates the usage of MPE routines and the linkage of MPE 
libraries to generate logfiles. In most cases, users can simply copy the 
share/examples/Makefile to their home directory, and do a "make" to compile 
the suggested targets.  Users don't need to copy the .c and .f files when 
MPE has been compiled with a MAKE that has VPATH support.  The created 
executables can be launched with mpirun from the MPI implementation to 
generate sample logfiles.



VI. c) ENVIRONMENTAL VARIABLES
------------------------------

There are 2 environmental variables, TMPDIR and MPE_LOG_FORMAT, that user 
may need to set before the generation of logfiles :

MPE_LOG_FORMAT : determines the format of the logfile generated from the 
                 execution of application linked with MPE logging libraries.  
		 The allowed value for MPE_LOG_FORMAT are CLOG, SLOG and 
		 ALOG. When MPE_LOG_FORMAT is NOT set, CLOG is assumed.

TMPDIR : specifies a directory to be used as temporary storage for each
         process.  By default, when TMPDIR is NOT set, /tmp will be used.
	 When user needs to generate a very large logfile for long-running
	 MPI job, user needs to make sure that TMPDIR is big enough to
	 hold the temporary logfile which will be deleted if the
	 merged logfile can be created successfully.  In order to
	 minimize the overhead of the logging to the MPI program, it is
	 highly recommended user to use a *local* file system for TMPDIR.

	 Note : The final merged logfile will be written back to the 
	        file system where process 0 is.



VI. d) UTILITY PROGRAMS
-----------------------

In bin/, user can find several useful utility programs when manipulating 
logfiles.  These includes log format converters, e.g. clog2slog, log format
print programs, e.g. slog_print, and a script to launch display program,
logviewer.


Log Format Converters
---------------------

clog2slog : a CLOG to SLOG logfile converter.  Since the automatic generation
            of SLOG file through setting of environmental variable 
	    MPE_LOG_FORMAT to SLOG may NOT work for some non well-behaved 
	    MPI programs, using the logfile format converter can generate
	    extra diagnostic information about the condition of the logfile.
	    Also the converter allows one to adjust certain parameters of
	    the logfile, like frame size which is the segment of the logfile
	    to be displayed by Jumpshot-3's time line window.  For non
	    well behaved MPI program, one may need to increase the frame
	    size from the default 64KB to a bigger value.  For more
	    information about the converter, do "clog2slog -h"

clog2alog : a CLOG to ALOG logfile converter.  It is not being developed.
            It is provided here for backward compatibility purpose.
	    

Log Format Print Programs
-------------------------

slog_print : a stdout print program for SLOG file.  It serves to check the
             content of the logfile.  If the SLOG file is too big, it may
	     NOT be useful to use slog_print.  Also, when slog is NOT complete,
	     slog_print won't work.  So it serves as a simple test to check
	     if the SLOG file is generated completely.

clog_print : a stdout print program for CLOG file.


Display Program Selector
------------------------

logviewer : the script which involves appropriate viewer based on the file
            extension of logfile.  For instance, if the logfile is foo.slog,
	    logviewer will invoke jumpshot-3 to display the logfile.
	    Jumpshot-3 resides in share/.  For more information of
	    logviewer, do "logviewer -help" to list all available options.





VII. Using MPE in MPICH
-----------------------

MPE has been seamlessly integrated into MPICH distribution, so user may
find it easier to use MPE when using it with MPICH.  Here are the differences
of using MPE with MPICH and with other MPI implementations.



VII. a) Compilation and Linkage
-------------------------------
MPICH provides scripts to help users to compile and link C/C++ and
F77/F90 programs.  They are mpicc for C programs, mpiCC for C++ programs,
mpif77 for F77 and mpif90 for F90 programs.  In addition, these 4 scripts
allows special options to be used to link with MPE profiling libraries.
These options are :

-mpitrace   - to compile and link with tracing library.
-mpianim    - to compile and link with animation libraries.
-mpilog     - to compile and link with logging libraries.

For instance, the following command creates executable, {\tt fpilog}, which
generates logfile when it is executed.

mpif77 -mpilog -o fpilog fpilog.f 

For other MPI implementations, user needs to compile and link their application
with MPE profiling libraries explicitly as shown in the example makefile.



VII. b) Inheritance of Environmental Variables
----------------------------------------------
MPE relies on certain environmental variables (e.g. TMPDIR).  These
variables determine how MPE behaves.  It is important to make sure that
all the MPI processes receive the intended value of environmental variables.
The complication of this issue comes from the fact that MPICH contains many
different devices for different platforms, some of these devices have a
different way of passing of environmental variables to other processes.
The often used devices, like ch_p4 and ch_shmem, do not require special
attention to pass the value of the environmental variable to spawned
processes.  The spawned process inherits the value from the launching
process when the environmental variable in the launching process has been
set.  But this is NOT true for all the devices, for instance, the
ch_p4mpd device requires a special option of mpirun to be used to set
the environmental variables to all processes.

mpirun -np N cpilog -MPDENV- MPE_LOG_FORMAT=SLOG

In this example, the option -MPDENV- is needed to make sure
that all processes have their environmental variable, MPE_LOG_FORMAT,
set to SLOG.

For other MPI implementations, how environmental variables are passed
remains unchanged.  User needs to get familar with the environment and
set the environmental variables accordingly.



VII. c) Viewing Logfiles
------------------------
MPE's install directory structure is the same as MPICH's.  So all MPE's
utility programs will be located in MPICH's bin/ directory.
To view a logfile, say fpilog.slog, do

logviewer fpilog.slog

The command will select and invoke Jumpshot-3 to display the content
of SLOG file if Jumpshot-3 has been built and installed successfully.
