| Sun Fire High-End Systems Dynamic Reconfiguration User Guide |    
|
| Sun Fire High-End Systems Dynamic Reconfiguration User Guide |
|
| C H A P T E R 1 |
|
Introduction to DR on the Sun Fire High-End System |
This chapter contains descriptions about general concepts that pertain to the dynamic reconfiguration (DR) feature on Sun Fire high-end servers.
DR on the Sun Fire high-end system enables you to perform hardware configuration changes to a live domain that is running the Solaris operating system without causing machine downtime. You can also use DR in conjunction with hot-swap to physically add boards to or remove them from the server.
You can execute DR operations from the Sun Fire high-end server system controller (SC) by using the system management services (SMS) commands: addboard(1M), moveboard(1M), deleteboard(1M), and rcfgadm(1M); or from the domain by using the cfgadm(1M) command. DR operations using SMS commands are described in Chapter 5, DR Domain Procedures.
The DR software has a command line interface through the cfgadm(1M) command, which is the configuration administration program. The DR agent also provides a remote interface to the Sun Management Center software.
The optional Sun Management Center software provides features such as domain management, as well as a graphical user interface (GUI) where you perform DR operations. If you prefer to use a graphical user interface instead of a command line interface, use the Sun Management Center software.
To use the Sun Management Center Platform software, you must attach the system controller board to a network. With a network connection, you can view both the command line interface and the graphical user interface. For instructions on how to use the Sun Management Center software, refer to the Sun Management Center User's Guide, shipped with the Sun Management Center software. For instructions on how to connect the system controller to a network connection on the system controller board, see your systems installation documentation.
Automatic DR enables an application to execute DR operations without requiring user interaction. This ability is provided by an enhanced DR framework that includes the reconfiguration coordination manager (RCM) and the system event facility, called sysevent. The RCM enables application-specific loadable modules to register callbacks. The callbacks perform preparatory tasks before a DR operation, error recovery during a DR operation, or clean-up after a DR operation. The sysevent facility enables applications to register for system events and receive notifications of those events. The automatic DR framework interfaces with the RCM and with the sysevent facility to enable applications to automatically give up resources prior to unconfiguring them and to capture new resources as they are configured into the domain.
The DR feature enables you to hot-swap system boards without bringing the server down. It is used to unconfigure the resources on a faulty system board from a domain so that the system board can be removed from the server. The repaired or replacement board can be inserted into the domain while the Solaris operating system continues to run. DR then configures the resources on the board into the domain. If you use the DR feature to add or remove a system board or component, DR always leaves the board or component in a known configuration state. See Chapter 2, DR State and Condition Models, for more information about configuration states for system board and components.
This section contains descriptions of general DR concepts that pertain to Sun Fire high-end system domains. For more information about DR concepts on the SC, refer to the System Management Services (SMS) Dynamic Reconfiguration User Guide.
For a device to be detachable, it must conform to the following items:
Some boards cannot be detached because their resources cannot be moved. For example, if a domain has only one CPU board, that CPU board cannot be detached. An I/O board is not detachable if it controls the boot drive.
If there is no alternate pathway for an I/O board, you can:
|
Note - If you are unsure whether a device is detachable, consult your Sun service representative. |
During the unconfigure operation on a system board with permanent memory (OpenBoot
PROM or kernel memory), the operating system is briefly paused, which is known as operating system quiescence. All operating system and device activity on the domain must cease during this critical phase of the operation.
Before it can achieve quiescence, the operating system must temporarily suspend all processes, CPUs, and device activities. If the operating system cannot achieve quiescence, it displays the reasons, which may include the following:
The conditions that cause processes to fail to suspend are generally temporary. Examine the reasons for any failure, and if the operating system encountered a failure to suspend a process, simply try the operation again.
When DR suspends the operating system, all of the device drivers that are attached to the operating system must also be suspended. If a driver cannot be suspended (or subsequently resumed), the DR operation fails.
A suspend-safe device does not access memory or interrupt the system while the operating system is in quiescence. A driver is suspend-safe if it supports operating system quiescence (if it can be suspended and then resumed). A suspend-safe driver also guarantees that when a suspend request is successfully completed, the device that the driver manages will not attempt to access memory, even if the device is open when the suspend request is made.
A suspend-unsafe device allows a memory access or a system interruption to occur while the operating system is in quiescence.
DR uses an unsafe driver list in the dr.conf file to prevent unsafe devices from accessing memory or interrupting the operating system during a DR operation. The dr.conf file resides in the following directory: /platform/SUNW,Sun-Fire-model_number/kernel/drv/, where model_number is the machine name, such as 15000. The unsafe driver list is a property in the dr.conf file with the following format:
DR reads this list when it prepares to suspend the operating system so that it can unconfigure a memory component. If DR finds an active driver in the unsafe driver list, it aborts the DR operation and returns an error message. The message includes the identity of the active, unsafe driver. You must manually remove the usage of the device by performing one, or more, of the following tasks.
You can retry the DR operation after you have stopped usage of the device.
|
Note - If you are unsure whether a device is suspend-safe, contact your Sun service representative. |
An attachment point is a collective term that refers to a board slot, a system board installed in the slot, and any devices connected to the board. DR can display the status of the board, the board slot, and the attachment point. The term occupant refers to the combination of a board and its attached devices.
There are two types of names for attachment points:
Where x represents the expander number for a particular board. For example, x would be 0 through 17 on the Sun Fire 15K system, and 0 through 8 on the Sun Fire 12K system.
|
Note - CPU/memory boards are installed only in slot 0. I/O boards and Max CPU boards are installed only in slot 1. |
To obtain a list of all available logical attachment points, use the cfgadm(1M) command with its -l option.
A state is the operational status of either a board slot or its occupant. A condition is the operational status of an attachment point. The cfgadm(1M) command can display nine types of states and conditions. See Chapter 2, DR State and Condition Models, for descriptions of the conditions and states for system boards and components.
There are four main types of operations related to boards: connection, configuration, unconfiguration, and disconnection. A board that is brought into a domain is first connected and then configured. A board that is removed from a domain is first unconfigured and then disconnected.
During the connect operation, the system provides power to the slot, and the operating system begins monitoring the board's temperature.
During the configure operation, the operating system assigns functional roles to the board, and loads device drivers for the board and for devices attached to it.
During the unconfigure operation, the system detaches the board logically from the operating system and takes the associated device drivers offline. Environmental monitoring continues, but devices on the board are not available for system use.
During the disconnect operation, the system stops monitoring the board and power to the slot is turned off.
To power-off a board that is in use (configured), first stop its use (unconfigure it), and then disconnect it from the domain. After a new or upgraded system board is inserted into the slot, connect the board and configure it.
The cfgadm(1M) command can connect and configure (or unconfigure and disconnect) in a single command. To connect and configure a board using a single command, see the sectionAdding a Board. To unconfigure and disconnect a board using a single command, see the sectionRemoving a Board.
If necessary, each operation (connect, configure, unconfigure, or disconnect) can be performed separately using the cfgadm(1M) command.
Hot-plug boards and modules have special connectors that supply electrical power to the board or module before the data pins make contact. Boards and devices that do not have hot-plug connectors cannot be inserted or removed while the system is running.
I/O boards and CPU/memory boards used in the Sun Fire high-end server are hot-plug devices. Some devices, such as the peripheral power supply, are not hot-plug modules and cannot be removed while the system is running.
The Sun Fire high-end server can be divided into dynamic system domains, which are comprised of logical and physical groupings of system board slots. Each domain is electrically isolated into hardware partitions, which ensures that a problem encountered in one domain cannot affect other domains.
Domain configuration is determined by the domain configuration table in the platform configuration database (PCD), which resides on the SC. The domain table controls how system board slots are logically partitioned into domains. The domain configuration represents the intended domain configuration. Thus, the configuration can include empty slots and occupied slots.
The number of slots available to a given domain is controlled by an available component list that is maintained on the system controller. (Refer to the System Management Services (SMS) Administrator Guide for more information about the available component list.) After a slot has been assigned to a domain, it becomes visible to that domain and unavailable and invisible to any other domain. Conversely, you must disconnect and unassign a slot from its domain before you can assign and connect it to another domain.
The logical domain is the set of slots that belong to the domain. The physical domain is the set of boards that are physically interconnected. A slot can be a member of a logical domain and not be part of a physical domain.
After a domain is booted, the system boards and empty slots can be assigned to (or unassigned from) a logical domain; however, they cannot become a part of the physical domain until the operating system requests it.
System boards or slots that are not assigned to a domain are available to all domains in whose available component lists they appear. These boards can be assigned to a domain by the platform administrator. Or, an available component list can be set up on the system controller to allow users with appropriate privileges to assign available boards to a domain.
You can use DR to configure or to unconfigure several types of components:
You must use caution when you add or remove I/O boards to which devices are attached. Before you can remove a board with I/O devices, all of its devices must be closed and all of its file systems must be unmounted.
If you need to remove an I/O board with attached devices from a domain temporarily and then re-add it before any other boards with I/O devices are added, reconfiguration is not necessary. In this case, device paths to the board devices remain unchanged.
Refer to the System Management Services (SMS) Dynamic Reconfiguration User Guide for special instructions for I/O devices.
Each I/O board in a domain contains an IOSRAM device. However, only one IOSRAM device, called golden IOSRAM, is used for SC-to-domain communications at a time. The golden IOSRAM contains the "tunnel" that is used for SC-to-domain communications. Because DR can remove I/O boards, it is sometimes necessary to stop using the current golden IOSRAM and make another IOSRAM device the golden IOSRAM. This process is called a "tunnel switch," and takes place whenever DR unconfigures the current golden IOSRAM.
When a domain is booted, the lowest-numbered I/O board in the domain is typically selected to be the initial golden IOSRAM.
DR supports dynamic reconfiguration of hsPCI+ I/O boards. Each hsPCI+ I/O board includes two XMITS ASICs and four hot-pluggable hsPCI slots.
Before you can delete a board, the operating system must vacate the memory on that board. Vacating a board entails flushing the contents of its non-permanent memory to swap space; and copying the contents of its permanent memory (that is, the kernel and OpenBoot PROM software) to another memory board.
To relocate permanent memory, the operating system on a domain must be temporarily quiesced. The length of the quiescence depends on the domain I/O configuration and the running workloads.
Detaching a board with permanent memory is the only time when the operating system is quiesced; therefore, you should know where permanent memory resides so that you can avoid impacting the operation of the domain significantly. To display the size of permanent memory, use the cfgadm(1M) command with its -av option. To vacate a board that has permanent memory, the operating system must find a sufficiently large block of available memory, called target memory, on which to copy the current contents of permanent memory, which is referred to as source memory.
A DR memory operation can be disallowed if the target domain does not have enough memory to hold the contents currently stored in permanent memory.
Correctable memory errors indicate that the memory on a system board (that is, one or more of its Dual Inline Memory Modules (DIMMs), or portions of the hardware interconnect) may be faulty and need replacement. When the SC detects correctable memory errors, it initiates a record-stop dump to save the diagnostic data, which can interfere with a DR operation.
When a record-stop occurs from a correctable memory error, allow the record-stop dump to complete before you initiate a DR operation.
If the faulty component causes repeated reporting of correctable memory errors, the SC performs multiple record-stop dumps. If this happens, you should temporarily disable the dump-detection mechanism on the SC; allow the current dump to finish; then initiate the DR operation. After the DR operation finishes, re-enable the dump detection.
The COD option provides additional CPU resources on COD CPU/Memory boards that you install in your Sun Fire high-end system. Although your system comes configured with a minimum number of standard (active) CPU/Memory boards, it can have a mix of both standard and COD CPU/Memory boards installed, up to a maximum 18 boards on, for example, the Sun Fire 15K server. At least one active CPU is required for each domain in the system.
You can use DR to move COD boards into and out of domains in the same way you use DR to move standard CPU/Memory boards.
You can use the CPUs on a COD board only after you purchase right-to-use (RTU) licenses for them. Each COD RTU license entitles you to receive a COD RTU license key that enables a specified number of CPUs on COD boards in a single system. Whenever you use DR to configure a COD board into a domain, make sure that enough RTU licenses are available to the target domain to enable each active CPU on the COD board. If there are not enough RTU licenses available to a target domain when you add a COD board, a status message is displayed for each CPU that cannot be enabled in the domain.
For more information about the COD option, see the System Management Services (SMS) Administrator Guide.
While the Solaris 9 4/03 operating system supports the full functionality of DR, some previous versions of the Solaris operating system did not support reconfiguration of I/O boards.
Solaris 8 2/02 software is the first release of the Solaris 8 operating system to support the full functionality of DR on domains. Requirements include appropriate patches and a new kernel update on the domain, and SMS software no earlier than SMS 1.3 on the SC.
For complete information and instructions for enabling DR on a domain that is running Solaris 8 software, visit:
http://www.sun.com/servers/highend/dr_sunfire
DR lets you disconnect, then reconnect system circuit boards without bringing the system down. You can use DR to add or remove system resources while the system continues to operate.
To illustrate reconfiguration of system resources consider the following Sun Fire 15K system configuration, as depicted in the diagram that follows.
Domain A contains system boards 0 and 2, and I/O board 2. Domain B contains system boards 1 and 3, and I/O boards 1, 3, and 4.
To assign system board 4 and I/O board 0 to Domain A, and to move I/O board 4 from Domain B to Domain A, you can use the Sun Management Center software's GUI. Or you can perform the following steps manually on the CLI in each domain as follows:
1. Enter the following configuration command on the command line in Domain B to disconnect I/O board 4 from Domain B:
2. Then, enter the following single command on the command line in Domain A, which assigns, connects, and configures system board 4 and I/O boards 0 and 4 into Domain A:
The following system configuration is the result. Only the way in which the boards are connected has changed, not the physical layout of the boards within the cabinet.
| Sun Fire High-End Systems Dynamic Reconfiguration User Guide | 817-4586-10 |
|
Copyright © 2003, Sun Microsystems, Inc. All rights reserved.