Next | Prev | Up | Top | Contents | Index

Using timex(1), sar(1), and par(1)

Three utilities you can use to monitor system performance are timex, sar, and par. They provide very useful information about what's happening in the system.

The operating system has a number of counters that measure internal system activity. Each time an operation is performed, an associated counter is incremented. You can monitor internal system activity by reading the values of these counters.

The timex and sar utilities monitor the value of the operating system counters, and thus sample system performance. Both utilities use sadc, the sar data collector, which collects data from the operating system counters and puts it in a file in binary format. The difference is that timex takes a sample over a single span of time, while sar takes a sample at specified time intervals. The sar program also has options which allow sampling of a specific function such as CPU usage (-u option) or paging (-p option). In addition, the utilities display the data some what differently.

The par utility has the ability to trace system call and scheduling activity. It can be used to trace the activity of a single process, a related group of processes or the system as a whole.

When would you use one utility over the other? If you are running a single application or a couple of programs, use timex. If you have a multi-user/multi-processor system, and/or are running many programs, use sar or par.

As in all performance tuning, be sure to run these utilities at the same time you are running an application or a benchmark, and be concerned only when figures are outside the acceptable limits over a period of time.

Using timex

The timex utility is a useful troubleshooting tool when you are running a single application. For example:

timex -s application
The -s option reports total system activity (not just that due to the application) that occurred during the execution interval of application. To redirect timex output to a file, (assuming you use the Bourne shell, (sh(1)) enter:

timex -s application 2> file
The same command, entered using the C shell, looks like this:

timex -s application > file

Using sar

The sar utility is a useful troubleshooting tool when you're running many programs and processes and/or have a multi-user system such as a server. You can take a sample of the operating system counters over a period of time (for a day, a few days, or a week).

Depending on your needs, you can choose the way in which you wish to examine system activity. You can monitor the system:

You can set up the system so sar will automatically collect system activity data and put it into files for you. Just use the chkconfig(1M) command to turn on sar's automatic reporting feature, which generates a sar -A listing. A crontab entry instructs the system to sample the system counters every 20 minutes during working hours and every hour at other times for the current day (data is kept for the last 7 days). To enable this feature, type:

/etc/chkconfig sar on
The data that is collected is put in /var/adm/sa in the form sann and sarnn, where nn is the date of the report (sarnn is in ASCII format). You can use the sar(1M) command to output the results of system activity.

Using sar Consecutively with a Time Interval

You can use sar to generate consecutive reports about the current state of the system. On the command line, specify a time interval and a count. For example:

sar -u 5 8
This prints information about CPU use eight times at five-second intervals.

Using sar Before and After a User-Controlled Activity

You may find it useful to take a snapshot of the system activity counters before and after running an application (or after running several applications concurrently). To take a snapshot of system activity, instruct sadc (the data collector) to dump its output into a file. Then run the application(s) either under normal system load or restricted load, and when you are ready to stop recording, take another snapshot of system activity. Then compare results to see what happened.

The following is an example of commands that will sample the system counters before and after the application:

/usr/lib/sa/sadc 1 1 file
Run the application(s) or perform any work you want to monitor, then type:

/usr/lib/sa/sadc 1 1 file 
sar -f file
If file does not exist, sadc will create it. If it does exist, sadc will append data to it.

Using sar and timex During the Execution of a Command

Often you want to examine system activity during the execution of a command or set of commands. The aforementioned method will allow you do to this. For example, to examine all system activity while running nroff(1), type:

/usr/lib/sa/sadc 1 1 sa.out 
nroff -mm file.mm > file.out 
/usr/lib/sa/sadc 1 1 sa.out 
sar -A -f sa.out
By using timex, you can do the same thing with one line of code:

timex -s nroff -mm file.mm > file.out
Note that the timex also includes the real, user, and system time spent executing the nroff request.

There are two minor differences between timex and sar. The sar program has the ability to limit its output (such as, the -u option reports only CPU activity), while timex always prints the -A listing. Also, sar works in a variety of ways, as discussed previously, but timex only works by executing a command. However, this command can be a shell file.

If you are interested in system activity during the execution of two or more commands running concurrently, put the commands into a shell file and run timex -s on the file. For example, suppose the file nroff.sh contained the following lines:

nroff -mm file1.mm > file1.out & 
nroff -mm file2.mm > file2.out & 
wait
To get a report of all system activity after both of the nroff requests (running concurrently) finish, invoke timex as follows:

timex -s nroff.sh

Using par

You can use par much as you use sar, such as

There is an extensive reference page for par(1). See that page for specifics on usage.

Use par instead of sar when you want a finer look at a suspect or problem process. Instead of simply telling you how much total time was used while your process was executing, like timex, par will break down the information so you can get a better idea of what parts of the process are consuming time. In particular, use the following command options:

-isSSdu

Check the time used by each system call and the intervening time lag.

-rQQ

Check process scheduling, to see if it should be run more or less frequently.
When tracing system calls, par prints a report showing all system calls made by the subject processes complete with arguments and return values. In this mode, par also reports all signals delivered to the subject processes. In schedule tracing mode, par prints a report showing all scheduling events taking place in the system during the measurement period. The report shows each time a process is put on the run queue, started on a processor, descheduled from a processor, including the reason that the process was descheduled. The events include timestamps. You can set up the system so par will automatically collect system activity data and put it into files for you.

The par utility works by processing the output of padc(1). This can be done in two ways: padc can be run separately and the output saved in a file to be fed to par as a separate operation or padc can be invoked by par to perform the data collection and reporting in one step.

The par utility can provide different types of reports from a given set of padc data depending on the reporting options that are specified. This is a reason why it is sometimes desirable to run the data collection as a separate step.

Wrap-up of sar, par, and timex

Now that you have learned when and how to use par, sar, and timex, you can choose one of these utilities to monitor the operating system. Then examine the output and try to determine what's causing performance degradation. Look for numbers that show large fluctuation or change over a sustained period; don't be too concerned if numbers occasionally go beyond the maximum.

The first thing to check is how the system is handling the disk I/O process. After that, check for excessive paging/swapping. Finally look at CPU use and memory allocation.

The sections immediately following assume that the system you are tuning is active (with applications/benchmark executing).

Next | Prev | Up | Top | Contents | Index