Solaris Fault Management
The Solaris Fault Management Facility is designed to be
integrated into the Service Management
Facility to provide a self-healing capability to
Solaris 10 systems.
fmd daemon is responsible for monitoring
several aspects of system health.
fmadm config command shows the current
The Fault Manager logs can be viewed with
fmdump -v and
fmdump -e -v.
fmadm faulty will list any devices flagged
fmstat shows statistics gathered by
With Solaris 10, Sun has implemented a daemon,
fmd, to track and react to fault management.
In addition to sending traditional syslog messages,
the system sends binary telemetry events to
correlation and analysis. Solaris 10 implements default
fault management operations for several pieces of hardware
in Sparc systems, including CPU, memory, and I/O bus events.
Similar capabilities are being implemented for x64 systems.
Once the problem is defined, failing components may be
offlined automatically without a system crash, or other
corrective action may be taken by
If a service dies as a result of the fault, the
Service Management Facility (SMF)
will attempt to restart it and any dependent processes.
The Fault Management Facility reports error messages in
a well-defined and explicit format. Each error code is
uniquely specified by a Universal Unique Identifier (UUID)
related to a document on the Sun web site at
Resources are uniquely identified by a Fault Managed
Resource Identifier (FMRI). Each Field Replaceable Unit
(FRU) has its own FMRI. FMRIs are associated with one
of the following conditions:
ok: Present and available for use.
unknown: Not present or not usable, perhaps because
it has been offlined or unconfigured.
degraded: Present and usable, but one or more problems
have been identified.
faulted: Present but not usable; unrecoverable
problems have been diagnosed and the resource has been
disabled to prevent damage to the system.
fmdump -V -u eventid command can be used to pull
information on the type and location of the event.
(The eventid is included in the text of the error message
provided to syslog.) The
-e option can be used to pull
error log information rather than fault log information.
Statistical information on the performance of fmd can be
viewed via the
fmstat command. In particular,
fmstat -m modulename provides information
for a given module.
fmadm command provides administrative
support for the Fault Management Facility. It allows us
to load and upload modules and view and update the resource cache.
The most useful capabilities of
fmadm are provided
through the following subcommands:
config: Display the configuration of component modules.
faulty: Display faulted resources. With the -a option,
list cached resource information. With the -i option, list
persistent cache identifier information, instead of most recent
state and UUID.
load /path/module: Load the module.
unload module: Unload module; the module name is the
same as reported by fmadm config.
rotate logfile: Schedule rotation for the specified
log file. Used with the logadm configuration file.