5 - C H A P T E R -

C H A P T E R 5

Using Sun S3L

This chapter explains how to incorporate calls to Sun S3L routines into your message-passing program. It covers t he following topics:

Incorporating Sun S3L Function Calls Into Your Program

Referencing Sun S3L Include Files

Setting Up the Sun S3L Environment

Enabling Thread-Safe Use of Sun S3L Routines

Using the Sun S3L Safety Mechanism

Creating Sun S3L Array Handles for Dense Arrays

Creating Sun S3L Array Handles for Sparse Arrays

Freeing Sun S3L Array Handles for Dense and Sparse Arrays

Using the Sun S3L Link Switch

Accessing Online Program Examples and Man Pages

Note - Sun S3L documentation includes sample online programs that demonstrate how to call each Sun S3L routine. You are encouraged to experiment with these sample programs. See Accessing Online Program Examples and Man Pages.

Incorporating Sun S3L Function Calls Into Your Program

The basic steps required for incorporating Sun S3L function calls into an MPI program are relatively simple and can be summarized as shown in TABLE 5-1.

CODE EXAMPLE 5-1 illustrates these steps with a simple program example. They are also discussed more fully in the following subsections.


`include 's3l/s3l-f.h' include 's3l/s3l_errno-f.h'`	Place references to the appropriate Sun S3L include files in the program's header section.

`S3L_init()`	Set up a Sun S3L environment.

`S3L_thread_comm_setup()`	If the MPI program is multithreaded, use to set up support for thread-safe operation.

`S3L_set_safety()`	If in program development mode, this routine may be used to enable enhanced error checking. Alternatively, the `S3L_SAFETY` environment variable can be used for this purpose.

`S3L_declare() S3L_declare_sparse()`	Create Sun S3L array handles for any dense and/or sparse parallel arrays that will be used by Sun S3L functions.

`S3L_*(args)`	Place calls to Sun S3L functions of interest.

`S3L_free() S3L_free_sparse()`	Deallocate array handles created earlier.

`S3L_exit`	Undo the Sun S3L environment.

c c Copyright (c) 2001, by Sun Microsystems, Inc. c All rights reserved. c c This example shows how to create an S3L environment and then deallocate it when c it is no longer needed. It also shows a simple MPI program that creates a 2D c array and an S3L array handle for the array, fills the array with data, prints the c array, and deallocates the array. program main implicit none include 's3l/s3l-f.h' include 'mpif.h' integer4 nrow, ncol parameter (nrow = 8, ncol = nrow) c global array descriptor integer8 a c Initialize S3L environment. Because MPI was not already initialized c at time of s3l_init call, S3L will call mpi_init. call s3l_init(ier) call mpi_comm_rank(MPI_COMM_WORLD, me, ier) call mpi_comm_size(MPI_COMM_WORLD, np, ier) c Set array extents (lengths of array axes). ext(1) = nrow ext(2) = ncol c Specify distribution of array axes; 0 means both are block-distributed. axis_is_local(1) = 0 axis_is_local(2) = 0 c Create S3L array handle for a. call s3l_declare(a, 2, ext, S3L_float, axis_is_local, $ S3L_USE_MALLOC, ier) c Fill array a with zeros. call s3l_zero_elements(a, ier) c Print contents of a to standard out. call s3l_print_array(a, ier) c Deallocate array handle for a and exit s3l environment. call s3l_free(a, ier) call s3l_exit(ier) end

CODE EXAMPLE 5-1 S3L Program Example

c Copyright (c) 2001, by Sun Microsystems, Inc.

c All rights reserved.

c This example shows how to create an S3L environment and then

deallocate it when c it is no longer needed. It also shows a

simple MPI program that creates a 2D c array and an S3L array

handle for the array, fills the array with data, prints the c

array, and deallocates the array.

      program main

      implicit none

      include 's3l/s3l-f.h'

      include 'mpif.h'

      integer*4 nrow, ncol

      parameter (nrow = 8, ncol = nrow)

c global array descriptor

      integer*8 a

c Initialize S3L environment. Because MPI was not already initialized

c at time of s3l_init call, S3L will call mpi_init.

      call s3l_init(ier)

      call mpi_comm_rank(MPI_COMM_WORLD, me, ier)

      call mpi_comm_size(MPI_COMM_WORLD, np, ier)

c Set array extents (lengths of array axes).

      ext(1) = nrow

      ext(2) = ncol

c Specify distribution of array axes; 0 means both are block-distributed.

      axis_is_local(1) = 0

      axis_is_local(2) = 0

c Create S3L array handle for a.

      call s3l_declare(a, 2, ext, S3L_float, axis_is_local,

     $     S3L_USE_MALLOC, ier)

c Fill array a with zeros.

      call s3l_zero_elements(a, ier)

c Print contents of a to standard out.

      call s3l_print_array(a, ier)

c Deallocate array handle for a and exit s3l environment.

      call s3l_free(a, ier)

      call s3l_exit(ier)

end

Referencing Sun S3L Include Files

Place the appropriate include lines at the top of any program unit that makes a Sun S3L call. The correct include files are as follows for both C and Fortran language interfaces:

C or C++

#include <s3l/s3l-c.h> #include <s3l/s3l_errno-c.h>

F77 or F90

include 's3l/s3l-f.h' include 's3l/s3l_errno-f.h'

The first line allows the program to access the header file containing prototypes of the routines and defines the symbols and data types required by the interface. The second line includes the header file containing error codes the routines might return.

If the compiler cannot find the Sun S3L include file, verify that a path to the directory does exist. The default path is

/opt/SUNWhpc/include/

Setting Up the Sun S3L Environment

Before a message-passing program can start using Sun S3L functions, every MPI process in the program must call S3L_init. This will create a Sun S3L environment to handle calls from MPI processes. S3L_init also initializes the BLACS environment.

See Chapter 6 for detailed instructions on using S3L_init.

Enabling Thread-Safe Use of Sun S3L Routines

If your MPI program is multithreaded, you can use S3L_thread_comm_setup to create a safe environment for individual threads and sets of cooperating threads to call Sun S3L functions. S3L_thread_comm_setup creates the internal MPI communicators and data structures needed for thread-safe operation of Sun S3L routines.

Note - If S3L_thread_comm_setup is used, it must be the first Sun S3L function called after S3L_init.

See Setting Up Support for Thread-Safe Operation for additional information.

Using the Sun S3L Safety Mechanism

Sun S3L provides a user-controlled mechanism for increasing the level of error checking performed during Sun S3L operations. This safety mechanism provides, in addition to the default level, three incrementally higher levels of error checking of Sun S3L functions.

When either of the two highest error checking modes is specified, the safety mechanism also causes the MPI processes to be synchronized. This synchronization can simplify the task of isolating errors to particular processes.

The safety mechanism can be controlled either by setting the environment variable S3L_SAFETY or with a call to the S3L_set_safety function.

The safety mechanism also provides the inquiry function S3L_get_safety, which returns the current safety level.

See Controlling the Sun S3L Safety Mechanism for more information.

Creating Sun S3L Array Handles for Dense Arrays

S3L_declare and S3L_declare_detailed each creates a Sun S3L array handle that describes a dense Sun S3L parallel array. Both support calling arguments that enable the user to specify:

The array's rank (number of dimensions)

The extent of each axis

The array's data type

Which axes, if any, will be distributed locally

How memory will be allocated for the array

S3L_declare_detailed provides additional arguments, which may be used to control:

The block sizes to be used in distributing array axes

The starting address of the local subgrid

Which processes contain the start of each array axis

See Creating and Destroying Array Handles for Dense Sun S3L Arrays for detailed instructions on using S3L_declare and S3L_declare_detailed.

If the simpler S3L_declare routine is used, Sun S3L will automatically assign a process grid to the Sun S3L array. The size of the process grid will be based on values supplied by the S3L_declare arguments, with the goal of distributing the array as evenly as possible across the processes.

If S3L_declare_detailed is used instead, the programmer will have the choice of either using the default process grid assigned by Sun S3L or of specifying a custom process grid through a call to S3L_set_process_grid. This topic is discussed more fully in MPI Processes and Sun S3L Process Grids and in Creating a Custom Process Grid.

Creating Sun S3L Array Handles for Sparse Arrays

Sun S3L provides the following functions for creating Sun S3L array handles for sparse arrays:

S3L_declare_sparse

S3L_read_sparse

S3L_rand_sparse

An overview of each is provided below. All are described more fully in Chapter 12.

Note - S3L_convert_sparse, which is used to convert an array from one sparse format to another, also creates a Sun S3L array handle. But, it does so for the converted array, not for a newly created array. Since this discussion is about creating array handles for new sparse arrays, S3L_convert_sparse is not covered here. It is discussed in Converting a Sparse Matrix From One Format to Another.

Overview of `S3L_declare_sparse`

S3L_declare_sparse is similar in purpose to S3L_declare, except that it applies to sparse matrices in one of the following sparse formats:

Coordinate

Compressed Sparse Row

Compressed Sparse Column

See Supported Sparse Formats for detailed descriptions of these sparse formats.

Before calling S3L_declare_sparse, you must create three 1D arrays using S3L_declare or S3L_declare_detailed. Two of these arrays, designated row and col, hold mapping information (pointer and index values) that specify where the nonzero values are located in the sparse matrix. The third array, designated val, contains all the nonzero elements belonging to the sparse matrix.

See Declaring a Sparse Matrix for more detailed descriptions of row, col, and val.

When calling S3L_declare_sparse, the following information is passed to the function through the argument string:

The sparse format to be used

The number of rows and columns in the sparse matrix

The Sun S3L array handles for row, col, and val

See Declaring a Sparse Matrix for detailed descriptions of the arguments taken by S3L_declare_sparse.

Upon completion, S3L_declare_sparse returns a Sun S3L array handle that describes the sparse array.

Overview of `S3L_read_sparse`

S3L_read_sparse reads sparse matrix data from an ASCII file and distributes the data to all participating processes. The data read from the file includes the nonzero values of the sparse matrix, as well as mapping information that specifies the row and column layout of the nonzero values in the sparse matrix.

The data must be stored in one of the following sparse formats:

Coordinate

Compressed Sparse Row

Compressed Sparse Column

Variable Block Row

When calling S3L_read_sparse, the following information is passed to the function:

The sparse format to be used: Coordinate, Compressed Sparse Row, and so forth

The number of rows and columns in the sparse matrix

The number of nonzero elements in the sparse matrix

The data type of the sparse array: S3L_float, S3L_double, S3L_complex, or S3L_double_complex

The name of the file to be read

The format of the data to be read from the file: ascii or ASCII

Upon completion, S3L_read_sparse returns a Sun S3L array handle that represents the newly created sparse array.

See Initializing a Sparse Matrix From a File for detailed descriptions of the arguments taken by S3L_read_sparse.

Overview of `S3L_rand_sparse`

S3L_rand_sparse enables you to create a sparse matrix without either supplying the nonzero data content or specifying the exact locations of the nonzero elements. It creates a global general sparse matrix, populating it with a set of random values. The sparse format can be any one of:

Coordinate

Compressed Sparse Row

Compressed Sparse Column

Variable Block Row

When calling S3L_rand_sparse, the following information is passed to the function:

The sparse format to be used: Coordinate, Compressed Sparse Row, and so forth

The type of random pattern to be used

The number of rows and columns in the sparse matrix

The desired density level; that is, the approximate percentage of elements in the sparse matrix that will be nonzero values. The density level must be expressed as a positive number no greater than 1.0

The data type of the sparse array: S3L_float, S3L_double, S3L_complex, or S3L_double_complex

A seed value for the random number generator

If the Variable Block Row format is specified, two additional arguments are used to pass row and column indices. These specify where the blocks of nonzero values are located in the sparse matrix

Upon completion, S3L_rand_sparse returns a Sun S3L array handle that represents the newly created sparse array. See Initializing a Sparse Matrix With Random Values for detailed descriptions of the arguments taken by S3L_rand_sparse.

Freeing Sun S3L Array Handles for Dense and Sparse Arrays

Sun S3L provides separate functions for deallocating Sun S3L array handles for dense and sparse arrays. This is necessary because sparse arrays involve more complex internal data structures, which require more steps to be deallocated.

Use S3L_free to deallocate a dense Sun S3L array.

Use S3L_free_sparse to deallocate a sparse Sun S3L array.

In either case, the only argument is the Sun S3L array handle that describes the array.

Using the Sun S3L Link Switch

To link in Sun S3L at compile time, include the switch -ls3l. This automatically links in the Forte Developer 6 library as well.

Sun S3L requires the presence of the Sun Performance Library routines and its associated license file. This library is not installed with the Sun HPC ClusterTools components. Instead, it is included as part of the Forte Developer 6 compiler suite.

The following examples show the -ls3l in use with the various supported compilers:

% tmcc -dalign -o program program.c -ls3l

C++

% tmCC -dalign -o program program.cc -ls3l

F77

% tmf77 -dalign -o program program.f -ls3l

F90

% tmf90 -dalign -o program program.f90 -ls3l

Note - The -dalign option is needed because libs3l and libsunperf libraries are compiled with it.

Accessing Online Program Examples and Man Pages

Sample Code Directories

The online sample programs are located in subdirectories of the Sun S3L examples directory. Separate C and Fortran versions are provided. The generic path for these examples is:

/opt/SUNWhpc/examples/s3l/op_class[-lang_suffix]/ex_name.lang

This shows the default location of the examples directory. If it is not in /opt/SUNWhpc/, ask your system administrator for the correct path.

op_class refers to the general class of operations to which the routines of interest belong.

Note - The -lang_suffix part of the name is used for Fortran implementations only. The op_class directory name does not include -lang_suffix for examples implemented in C.

ex_name.lang is the example's file name. The lang extension is .c, or .f. For example, the following is the Fortran version of a program example that illustrates use of
s3l_outer_prod routines:

/opt/SUNWhpc/examples/s3l/dense_matrix_ops-f/outer_prod.f

The following shows the equivalent example for the C interface:

/opt/SUNWhpc/examples/s3l/dense_matrix_ops/outer_prod.c

Compiling and Running the Examples

Each example subdirectory has a makefile. Each makefile references the file
/opt/SUNWhpc/Make.simple. If you are copying the example sources and makefiles to one of your own subdirectories, you should also copy Make.simple to your subdirectory's parent directory.

Make.simple contains definitions of compilers, compiler flags, and other variables that are needed to compile and run the examples. Note that the compiler flags in this file will not provide highly optimized executables. Information on optimization flags is best obtained from the documentation for the compiler being used.

Each makefile has several targets that are meant to simplify the compilation and execution of examples. If you want to compile the source codes and create all executables in a particular example directory, use the command, make.

If you wish to run the executables, enter make run. This command will also perform any necessary compilation and linking steps, so you need not issue make before entering make run.

By default, your executables will be run on two processes. You can change this by specifying the NPROCS variable on the command line. For example, the following will start execution on four processes:

% make run NPROCS=4

Executables and object files can be deleted by make clean.

Man Pages

To read the online man page for a Sun S3L routine, enter the following:

% man routine_name