Overview of Administration Controls

The Sun HPC cluster's default configuration supports execution of MPI applications. In other words, if you have started the Sun CRE daemons on your cluster and created a default partition, as described in Chapter 2, users can begin executing MPI jobs.You may, however, want to customize the cluster's configuration to the specific requirements of your site.

This chapter provides a brief overview of the features that control a cluster's configuration and behavior. These are:

• The Sun CRE daemons - introduced here and described more fully in their respective man pages
• The RSM daemon
• The mpadmin command - introduced here and described more fully in Chapter 6, as well as in its man page
• The cluster configuration file hpc.conf - introduced here and described more fully in Chapter 7, as well as in its man page
• Authentication methods

---

The Sun CRE Daemons

Sun CRE comprises three master daemons and two nodal daemons:

• Sun CRE master daemons
• tm.rdb
• tm.mpmd
• tm.watchd

• Sun CRE nodal daemons
• tm.omd
• tm.spmd

This section present brief descriptions of the Sun CRE daemons. For complete information on the daemons, see their respective man pages.

Master Daemon tm.rdb

tm.rdb is the resource database daemon. It runs on the master node and implements the resource database used by the other parts of Sun CRE. This database represents the state of the cluster and the jobs running on it.

If you make changes to the cluster configuration, for example, if you add a node to a partition, you must restart the tm.rdb daemon to update the Sun CRE resource database to reflect the new condition.

Master Daemon tm.mpmd

tm.mpmd is the master process-management daemon. It runs on the master node and services user (client) requests made via the mprun command. It also interacts with the resource database via calls to tm.rdb and coordinates the operations of the nodal client daemons.

Master Daemon tm.watchd

tm.watchd is the cluster watcher daemon. It runs on the master node and monitors the states of cluster resources and jobs and, as necessary:

• Marks individual nodes as online or offline by periodically executing remote procedure calls (RPCs) to all the nodes.
• Clears stale resource database (rdb) locks.
• If the -Yk option has been enabled, aborts jobs that have processes on nodes determined to be down. This option is disabled by default. This option can be especially valuable on very large clusters.

Nodal Daemon tm.omd

tm.omd is the object-monitoring daemon. It runs on all the nodes in the cluster, including the master node, and continually updates the Sun CRE resource database with dynamic information concerning the nodes, most notably their load. It also initializes the database with static information about the nodes, such as their host names and network interfaces, when Sun CRE starts up.

The environment variable SUNHPC_CONFIG_DIR specifies the directory in which the Sun CRE resource database files are to be stored. The default is /var/hpc.

Nodal Daemon tm.spmd

tm.spmd is the slave process-management daemon. It runs on all the compute nodes of the cluster and, as necessary:

• Handles spawning and termination of nodal processes per requests from the tm.mpmd.
• In conjunction with mprun, handles multiplexing of stdio streams for nodal processes.
• Interacts with the resource database via calls to tm.rdb.

Spin Daemon tm.spind

tm.spind is the spin daemon. It runs on all the compute nodes of the cluster.

This daemon enables certain processes of a given MPI job on a shared-memory system to be scheduled at approximately the same time as other related processes. This co-scheduling reduces the load on the processors, thus reducing the effect that MPI jobs have on each other.

---

RSM Daemon

The hpc_rsmd daemon provides access services to Remote Shared Memory (RSM) resources and manages RSM communications paths on behalf of MPI jobs. If your cluster is not RSM-enabled, you can ignore this section.

An instance of hpc_rsmd runs on each RSM-enabled node of a cluster and:

• Provides remote shared-memory resources to MPI client processes.
• Monitors and manages the RSM paths between node pairs.
• Returns cluster configuration information to Sun CRE.

hpc_rsmd is started when the cluster is booted. After boot, the daemon can be stopped and restarted manually by means of the following script (executed by superuser). This script can also be used to clean up files and System V shared memory segments left behind when an hpc_rsmd instance exits abnormally.

# /etc/init.d/sunhpc.hpc_rsmd  [ start | stop | clean ]

---

mpadmin: Administration Interface

Sun CRE provides an interactive command interface, mpadmin, which you can use to administer your Sun HPC cluster. It can only be invoked by the superuser.

This section introduces mpadmin and shows how to use it to perform several administrative tasks:

• List the names of all nodes in the cluster
• Enable nodes
• Create and enable partitions
• Customize some aspects of cluster administration

mpadmin offers many more capabilities than are described in this section. See Chapter 6 for a more comprehensive description of mpadmin.

Introduction to mpadmin

The mpadmin command has the following syntax:

# mpadmin [-c command] [-f filename] [-h] [-q] [-s cluster_name] [-V]

When you invoke mpadmin with no options, it goes into interactive mode, displaying an mpadmin prompt. It also goes into interactive mode when invoked with the options -f, -q, or -s. In this mode, you can execute any number of mpadmin subcommands to perform operations on the cluster or on nodes or partitions.

When you invoke mpadmin with the -c, -h, or -V options, it performs the requested operation and returns to the shell level.

The mpadmin command-line options are summarized in TABLE 3-1.

TABLE 3-1 mpadmin Options

Option	Description
-c command	Execute single specified command.
-f file-name	Take input from specified file.
-h	Display help/usage text.
-q	Suppress the display of a warning message when a non-root user attempts to use restricted command mode.
-s cluster-name	Connect to the specified Sun HPC cluster.
-V	Display mpadmin version information.

Commonly Used mpadmin Options

This section describes the mpadmin options -c, -f, and -s.

-c command - Execute a Single Command

Use the -c option when you want to execute a single mpadmin command and return upon completion to the shell prompt. For example, the following use of mpadmin -c changes the location of the Sun CRE log file to /home/wmitty/cre_messages:

# mpadmin -c set logfile="/home/wmitty/cre_messages"

Most commands that are available via the interactive interface can be invoked via the -c option. See Chapter 6 for a description of the mpadmin command set and a list of which commands can be used as arguments to the -c option.

-f file-name - Take Input From a File

Use the -f option to supply input to mpadmin from the file specified by the file-name argument. The source file is expected to consist of one or more mpadmin commands, one command per line.

This option can be particularly useful in the following ways:

• It can be used following use of the mpadmin command dump, which outputs all or part of a cluster's configuration in the form of an mpadmin script. If the dump output is stored in a file, mpadmin can, at a later time, read the file via the -f option, thereby reconstructing the configuration that had been saved in the dump output file.
• The -f option can also be used to read mpadmin scripts written by the system administrator--scripts designed to simplify other cluster management tasks that involve issuing a series of mpadmin commands.

-s cluster-name - Connect to Specified Cluster

Use the -s option to connect to the cluster specified by the cluster-name argument. A cluster's name is the host name of the cluster's master node.

The mpadmin commands apply to a certain cluster, determined as follows:

• First, the cluster specified on the command line with the -s option.
• If no such option is specified, the command uses the value of the environment variable SUNHPC_CLUSTER.
• If this environment variable is not set, the command uses the cluster name supplied (usually at installation time) in /etc/hpc_system.

Understanding Objects, Attributes, and Contexts

To use mpadmin, you need to understand the concepts of object, attribute, and context as they apply to mpadmin.

Objects and Attributes

From the perspective of mpadmin, a Sun HPC cluster consists of a system of objects, which include

• The cluster itself
• Each node contained in the cluster
• Each partition (logical group of nodes) defined in the cluster

Each type of object has a set of attributes, which control various aspects of their respective objects. For example, a node's enabled attribute can be

• set to make the node available for use
• unset to prevent it from being used

Some attribute values can be operated on via mpadmin commands.

---

Note - Sun CRE sets many attributes in a cluster to default values each time it starts up. You should not change attribute values, except for the attribute modifications described here and in Chapter 6.

---

Contexts

mpadmin commands are organized into three contexts, which correspond to the three types of mpadmin objects. These contexts are summarized below and illustrated in FIGURE 3-1.

• Cluster - These commands affect cluster attributes.
• Node - These commands affect node attributes.
• Partition - These commands affect partition attributes.

 FIGURE 3-1 mpadmin Contexts

mpadmin Prompts

In interactive mode, the mpadmin prompt contains one or more fields that indicate the current context. TABLE 3-2 shows the prompt format for each of the possible mpadmin contexts.

TABLE 3-2 mpadmin Prompt Formats

Prompt Formats	Context
[cluster-name]::	Current context = Cluster
[cluster-name]Node::	Current context = Node, but not a specific node
[cluster-name]N(node-name)::	Current context = a specific node
[cluster-name]Partition::	Current context = Partition, but not a specific partition
[cluster-name]P(partition-name)::	Current context = a specific partition

Performing Sample mpadmin Tasks

To introduce the use of mpadmin, this section steps through some common tasks the administrator may want to perform. These tasks are:

• List names of nodes
• Enable nodes so that MPI jobs can run on them
• Create and enable partitions so that MPI jobs can run on them
• Customize some cluster attributes

List Names of Nodes

mpadmin provides various ways to display information about the cluster and many kinds of information that can be displayed. However, the first information you are likely to need is a list of the nodes in your cluster.

Use the list command in the Node context to display this list. In the following example, list is executed on node1 in a four-node cluster.

node1# mpadmin[node0]:: node[node0] Node:: list    node0    node1    node2    node3[node0] Node::

The mpadmin command starts up an mpadmin interactive session in the Cluster context. This is indicated by the [node0]:: prompt, which contains the cluster name, node0, and no other context information.

---

Note - A cluster's name is assigned by Sun CRE and is always the name of the cluster's master node.

---

The node command on the example's second line makes Node the current context. The list command displays a list of all the nodes in the cluster.

Once you have this list of nodes, you have the information you need to enable the nodes and to create a partition. However, before moving on to those steps, you might want to try listing information from within the cluster context or the partition context. In either case, you would follow the same general procedure as for listing nodes.

If this is a newly installed cluster and you have not already run the part_initialize script (as described in "Create a Default Partition" on page 8), the cluster contains no partitions. If, however, you did run part_initialize and have thereby created the partition all, you might want to perform the following test.

node1# mpadmin[node0]:: partition[node0] Partition:: list    all[node0] Partition::

To see what nodes are in partition all, make all the current context and execute the list command. The following example illustrates this; it begins in the Partition context (where the previous example ended).

[node0] Partition:: all[node0] P[all]:: list    node0    node1    node2    node3[node0] P[all]::

Enabling Nodes

A node must be in the enabled state before MPI jobs can run on it.

Note that enabling a partition automatically enables all its member nodes, as described in the next section.

To enable a node manually, make that node the current context and set its enabled attribute. Repeat for each node that you want to be available for running MPI jobs.

The following example illustrates this, using the same four-node cluster used in the previous examples.

node1# mpadmin[node0]:: node0[node0] N[node0]:: set enabled[node0] N[node0]:: node1[node0] N[node1]:: set enabled[node0] N[node1]:: node2[node0] N[node2]:: set enabled[node0] N[node2]:: node3[node0] N[node3]:: set enabled[node0] N[node3]::

Note the use of a shortcut to move directly from the Cluster context to the node0 context without first going to the general Node context. You can explicitly name a particular object as the target context in this way so long as the name of the object is unambiguous--that is, it is not the same as an mpadmin command.

mpadmin accepts multiple commands on the same line. The previous example could be expressed more succinctly as:

node1# mpadmin[node0]:: node0 set enabled node1 set enabled node2 set enabled node3 set enabled[node0] N[node3]::

To disable a node, use the unset command in place of the set command.

Creating and Enabling Partitions

You must create at least one partition and enable it before you can run MPI programs on your Sun HPC cluster. Even if your cluster already has the default partition all in its database, you will probably want to create other partitions with different node configurations to handle particular job requirements.

There are three essential steps involved in creating and enabling a partition:

• Use the create command to assign a name to the partition.
• Set the partition's nodes attribute to a list of the nodes you want to include in the partition.
• Set the partition's enabled attribute, which automatically enables all the partition's member nodes.

Once a partition is created and enabled, you can run serial or parallel jobs on it. A serial program runs on a single node of the partition. Parallel programs are distributed to as many nodes as Sun CRE determines appropriate for the job.

---

Note - There are no restrictions on the number or size of partitions, so long as no node is a member of more than one enabled partition.

---

Example: Creating a Two-Node Partition

The following example creates and enables a two-node partition named part0. It then lists the member nodes to verify the success of the creation.

node1# mpadmin[node0]:: partition[node0] Partition:: create part0[node0] P[part0]:: set nodes=node0 node1[node0] P[part0]:: set enabled[node0] P[part0]:: list    node0    node1[node0] P[part0]::

Example: Two Partitions Sharing a Node

The next example shows a second partition, part1, being created. One of its nodes, node1, is also a member of part1.

[node0] P[part0]:: up[node0] Partition:: create part1[node0] P[part1]:: set nodes=node1 node2 node3[node0] P[part1]:: list    node1    node2    node3[node0] P[part1]::

Because node1 is shared with part0, which is already enabled, part1 is not being enabled at this time. This illustrates the rule that a node can be a member of more than one partition, but only one of those partitions can be enabled at a time.

Note the use of the up command. The up command moves the context up one level, in this case, from the context of a particular partition (that is, from part0) to the general Partition context.

Example: Shared versus Dedicated Partitions

Sun CRE can configure a partition to allow multiple MPI jobs to be running on it concurrently. Such partitions are referred to as shared partitions. Sun CRE can also configure a partition to permit only one MPI job to run at a time. These are called dedicated partitions.

In the following example, the partition part0 is configured to be a dedicated partition and part1 is configured to allow shared use by up to four processes.

node1# mpadmin[node0]:: part0[node0] P[part0]:: set max_total_procs=1[node0] P[part0]:: part1[node0] P[part1]:: set max_total_procs=4[node0] P[part1]::

The max_total_procs attribute defines how many processes can be active on each node in the partition for which it is being set. In this example, it is set to 1 on part0, which means only one process can be running at a time. It is set to 4 on part1 to allow up to four processes to run on that partition.

Note again that the context-changing shortcut (introduced in "Enabling Nodes" on page 20) is used in the second and fourth lines of this example.

Customizing Cluster Attributes

Two cluster attributes that you may wish to modify are logfile and administrator.

Changing the logfile Attribute

The logfile attribute allows you to log Sun CRE messages in a separate file from all other system messages. For example, if you enter

[node0]:: set logfile=/home/wmitty/cre-messages

Sun CRE will output its messages to the file /home/wmitty/cre-messages. If logfile is not set, Sun CRE messages will be passed to syslog, which will store them with other system messages in /var/adm/messages.

---

Note - A full path name must be specified when setting the logfile attribute.

---

Changing the administrator Attribute

You can set the administrator attribute to specify, say, the email address of the system administrator. To do this:

[node0]:: set administrator="root@example.com"

Note the use of double quotes.

Quitting mpadmin

Use either the quit or exit command to quit an mpadmin interactive session. Either causes mpadmin to terminate and return control to the shell level.

For example:

[node0]:: quitnode1#

---

Cluster Configuration File hpc.conf

When Sun CRE starts up, it updates portions of the resource database according to the contents of a configuration file named hpc.conf. This file is organized into functional sections, which are summarized here and illustrated in TABLE 3-3.

• ShmemResource section defines certain shared memory attributes.
• MPIOptions section defines certain MPI parameters.
• CREOptions section controls the logging of system events, the handling of daemon core files, the authentication options, and placing restrictions on mprun's ability to run programs in batch mode.
• PMODULES section names and locates the protocol modules that are available for communication on the cluster.
• PM sections list and rank the protocol modules that are available for communication on the cluster.
• HPCNodes section is not used by Sun CRE. It applies only in an LSF-based runtime environment.

You can change any of these aspects of your cluster's configuration by editing the corresponding parts of the hpc.conf file. Default settings are in effect if you make no changes to the hpc.conf file as provided.

To illustrate the process of customizing the hpc.conf file, this section explains how to:

• Prepare for editing hpc.conf
• Control MPI communication attributes
• Update the Sun CRE database

---

Note - The hpc.conf file is provided with the Sun HPC ClusterTools software.

---

---

Note - You may never need to make any changes to hpc.conf. If you do wish to make changes beyond those described in this section, see Chapter 6 for a fuller description of this file.

---