Sun HPC ClusterTools 5 Software User's Guide
|    
|
| Sun HPC ClusterTools 5 Software User's Guide
|
|
| C H A P T E R 8 |
|
Profiling Programs With MPPROF |
This chapter explains how to use the MPProf utilities and environment variables to extract profiling information from executing MPI programs and generate formatted reports from that data. It also explains how to convert intermediate binary files to unanalyzed ASCII files. It has four sections:
To enable MPI profiling with MPProf, set the MPI_PROFILE environment variable to 1 before starting the MPI program.
% setenv MPI_PROFILE 1 |
When MPProf is enabled, it will extract information about calls to Sun MPI routines and store the information in a set of intermediate files. A separate data file is created for each MPI process rank in which MPI calls were made.
MPProf also creates an index file describing the locations of the intermediate files. The index file name has the form:
mpprof.index.rm.jid |
where rm is the name of the resource manager and jid is the job ID. The report generator, mpprof, uses this index file to gather the profiling data and associate it with particular MPI processes.
MPProf provides the following environment variables to control various aspects of MPProf behavior:
These environment variables are explained in the following sections.
The environment variable MPI_PROFDATADIR can be used to specify a nondefault location where the intermediate files will be created for each process. This directory must exist.
If profiling is enabled and MPI_PROFDATADIR specifies a directory that does not exist, MPProf will output an error message and abort the program. If MPI_PROFDATADIR is not set and profiling is enabled, intermediate files will be created in /usr/tmp.
The environment variable MPI_PROFINDEXDIR can be used to specify a nondefault location for storing the index file. This directory must exist.
If profiling is enabled and MPI_PROFDATADIR specifies a directory that does not exist, MPProf will output an error message and abort the program. If profiling is enabled and MPI_PROFINDEXDIR is not set, the index file will be stored locally in the current directory.
The environment variable MPI_PROFINTERVAL can be used to specify a data sampling period. When this value is a number greater than 0, the intermediate files will be updated at the prescribed intervals. The data recorded with each update represent the MPI activity that occurred since the previous update.
Setting MPI_PROFINTERVAL to 0 forces updates to be made for every MPI call. Setting MPI_PROFINTERVAL to Inf (meaning infinity) causes the intermediate files to be updated only once, at MPI_Finalize time. If MPI_PROFINTERVAL is unset or is set without a value, the default value of 60 seconds will be used.t
The following example sets an interval of 5 seconds:
% setenv MPI_PROFINTERVAL 5 |
In this case, the first update of intermediate files would occur approximately 5 seconds after the MPI_Init call, with additional updates appended to the file at 5-second intervals.
If time intervals are used and an MPI program terminates before the MPI_Finalize call, any updates that were made can be used by mpprof to generate a profile of program operations up to the point of termination.
The environment variable MPI_PROFMAXFILESIZE can be used to specify, in kilobytes, the maximum amount of data a process can record in its intermediate file. The default value is 51,200 kilobytes (approximately 50 megabytes). Setting MPI_PROFMAXFILESIZE to unlimited removes any limits on the size to which the intermediate files can grow. Setting MPI_PROFMAXFILESIZE to 0 is an error and will cause MPProf to abort the program
If a write to a given intermediate file exceeds the MPI_PROFMAXFILESIZE limit, the write operation will complete, but that process will be unable to record additional profiling data. Profiling can continue for other processes that have not reached the file size limit.
This section shows how to use the mpprof command, which is used to generate reports from profiling information that is collected by the MPI library. To enable MPI profiling, you set the MPI_PROFILE environment variable to 1 before executing the mprun command.
When you use the mpprof command, you always specify an index file that points to files that contain profiling data. The naming convention for the index file is mpprof.index.rm.jid, where rm is the resource manager used and jid is the job ID assigned to the MPI program. When profiling is enabled, the MPI library creates the index file in the current directory by default.
|
Note - The mpdump command, discussed in the next section, is used to translate binary profiling data into ASCII text. |
mpprof [-h] [-V] [-r] [-S] [-g output-directory] [-p process-set] [-c columns] -s start-time] [-e end-time] [-o output-file] index-file |
where the following command options are available:
The mpprof command generates a report file based on processes listed in a specified index file. The following example generates a report file called report.txt that is based on process profile data stored in files that are specified in an index file called mpprof.index.cre.14:
% mpprof -o report.txt mpprof.index.cre.14 |
When you enter this command, a new file called report.txt is created in the current directory. If you did not use the -o report.txt command line option, the report would be sent to the standard output.
You can fine-tune profiling output by specifying a subset of processes of the job, using the -p option as shown in the following example:
% mpprof -p 0,3-5,9:12,18:27:3 -o report.txt mpprof.index.cre.14 |
The output report file is always specified before the index file. The preceding command causes these processes to be analyzed for the report: 0, 3, 4, 5, 9, 10, 11, 12, 18, 21, 24, and 27. To view the report, open the report.txt file in the current directory.
The following example runs the mpprof command to generate a report on all processes that occur starting 2 minutes (120 seconds) or later after a call to MPI_Init, which initializes the accumulation of data on processes:
% mpprof -s 120 -o report.txt mpprof.index.cre.14 |
The -s option is useful when you want to exclude startup routines from reporting because startup routines may not be critical to performance. To view the report, open the report.txt file in the current directory.
If you want to save report output to a specific directory other than the current directory, use the -g option:
% mpprof -g report.txt mpprof.index.cre.14 |
You can combine the -g reportname option with the -r option, removing the intermediate data files created during the profiling process at the same time as you store the report output:
% mpprof -r -g report.txt mpprof.index.cre.14 |
The sample report shown is the result of running the following command, which places the report in a file called report.txt:
% mpprof -p 0,3-5,9:12,18:27:3 -o report.txt mpprof.index.cre.39 |
|
Note - The file mpprof.index.cre.39 was created when this MPI command was run:
|
The sample report is shown below:
The mpdump command converts each raw (unanalyzed) intermediate file that is generated by the MPI library into a readable ASCII file. The mpdump command produces files that have the .txt extension. For example, the following command creates a series of ASCII files in the current directory (the default) based on all the processes that are listed in the index file mpprof.index.cre.14:
% mpdump mpprof.index.cre.14 |
The syntax of the mpdump command is
mpdump [-h] [-V] [-S] [-p process-set] [-o output-directory] index-file |
where the following command options are available:
The following command creates 28 ASCII text files that are based on the 28 files that were created in the previous section using the mprun command.
% mpdump mpprof.index.cre.39 |
Each of the 28 files contains profiling data about a process in the job. The contents of one of the files is shown below:
| Sun HPC ClusterTools 5 Software User's Guide
| 817-0084-10 |
|
Copyright © 2003, Sun Microsystems, Inc. All rights reserved.