home
***
CD-ROM
|
disk
|
FTP
|
other
***
search
/
OS/2 Shareware BBS: 14 Text
/
14-Text.zip
/
wnbn2a.zip
/
WNBN2A.TXT
Wrap
Text File
|
1993-06-23
|
13KB
|
301 lines
This message is being posted in the following Compuserve
forums/sections:
OS2USER: Sections 01,15
PSPBETA: Section 16
WINNT: Section 02
CANOPUS: Sections 06,12
PURPOSE:
The purposes of this message are:
1. Promote the sharing of the information in WINBN2.TXT. WINBN2.TXT
is a file available for downloading from Compuserve Forum OS2USER
Section 01: OS/2 Public Image. WINBN2.TXT includes the first
*multitasking benchmarks* comparing the March 1993 beta releases of
OS/2 2.1 and Windows NT. The benchmarks were carried out by Timothy
Sipples, author of the OS/2 Frequently Asked Questions (FAQ) list.
THE MULTITASKING BENCHMARKS SHOW WINDOWS NT's PERFORMANCE RATINGS TO
BE ABOUT 75 PERCENT OF OS/2 2.1's PERFORMANCE RATINGS.
2. Show a method by which multitasking performance of OS/2 2.1 and
Windows NT can be benchmarked and compared. This should be of value
to both users and the trade press.
3. Share analysis and interpretations of the benchmarking results that
raise open issues for future benchmarking such as:
a. What is the relative efficiency of message passing kernel operating
systems - which are highly portable - (e.g. Windows NT; Mach 3
derivatives such as IBM Workplace OS, Taligent Pink and future
microkernel versions of OSF Unix and USL System V.4.2 Unix) when
compared to monolithic kernel operating systems - which are less
portable or non-portable - (e.g. OS/2 2.X, Microsoft Windows 4.0,
(aka Chicago) and current monolithic kernel versions of Unix)?
b. How appropriate is it to run message passing kernel operating
systems on Intel X86 microprocessors as compared to running these same
operating systems on the new generation of RISC microprocessors?
c. How does the performance of a particular message passing kernel
operating system or a monolithic kernel operating system scale as more
and more processors are added to symmetric multiprocessor (SMP)
configurations?
BENCHMARK ANALYSIS
The hardware used for these benchmarks was an IBM PS/2 Model 56
with 486SLC2-20/40 microprocessor, 12 MB RAM, 160 MB SCSI hard disk
(16msec)and integrated 16 bit VGA display adapter. The use of the VGA
display adapter should minimize biases toward either operating system
during benchmarking since both OS/2 2.1 and Windows NT should have
well tested device drivers for VGA.
The multitasking benchmark involves four components:
1. The first component of the benchmark is the singletasking
performance of each operating system as measured by WinTach, a Win16
benchmark. Higher number means better performance. Numbers are
expressed as a multiple over a baseline system.
The results were:
Component OS/2 Full Screen Win NT Full Screen Ratio
Word Processing 2.23 1.89 0.85
CAD/Draw 1.84 3.11 1.69
Spreadsheet 2.08 1.9 0.91
Paint 2.33 1.98 0.85
Overall RPM 2.12 2.22 1.05
The ratio shows the rating for Windows NT divided by the rating for
OS/2 2.1.
Analysis of component 1:
The word processing, spreadsheet and paint components have better
performance under OS/2 2.1. The CAD/Draw component has better
performance under Windows NT. Because of the weight on CAD/Draw in
determining the composite rating, Windows NT has a higher composite
rating.
Windows NT has three radio button choices for determining the balance
of resources applied to foreground task versus background task. These
radio buttons are found by selecting the sequence: Control Panel ->
System -> Tasking. The choices are:
o Best Foreground Application Response Time (referred to as "Best").
o Foreground Application More Responsive Than Background Application.
o Foreground and Background Applications Equally Responsive (referred
to as "Equal").
The benchmark document does not specify which value was selected for
the WinTach singletasking test, however it was probably "Best."
2. The second component of the benchmark is singletasking of a batch
file COUNTER.BAT. The batch file reads:
@echo on
cls
c:
cd\
echo A > counter.fil
:loop
type temp.txt > temp1.txt
xcopy temp1.txt temp.txt
del temp1.txt
echo A >> counter.fil
goto loop
The file temp.txt was constructed using the following command:
dir d:\ /s > temp.txt
The OS/2 2.1 beta code was the only software installed on the d:
drive.
On each cycle of the loop, the length of COUNTER.FIL increases by 4
bytes (one byte each for the letter A, a space, a carriage return and a
line feed).
COUNTER.FIL Length OS/2 2.1 Win NT
Full Screen DOS Sess. 236 bytes 164 bytes
Windowed DOS Session 164 bytes 164 bytes
Analysis of component 2:
OS/2 outperforms Windows NT using a Full Screen DOS Session on both
systems and the two operating systems have equal performance when using
Windowed DOS sessions.
3. The third component of the benchmark is running COUNTER.BAT in the
background and WinTach in the foreground. In this run, Windows NT
tasking is set to "Best Foreground Application Response Time," i.e.
"Best."
This is performed on both systems by the following procedure. WinTach
was loaded and made ready to go. COUNTER.BAT was started, then WinTach
was started. As soon as WinTach displayed the final screen (reporting
the benchmark figures), a switch was made to the COUNTER.BAT session
and it was immediately terminated. The author of the benchmarks
estimates that this method can produce errors in the range of plus or
minus four iterations (16 bytes).
The author of the benchmarks used this method with COUNTER.BAT in Full
Screen and Windowed DOS sessions. The results were quite close. For
the sake of brevity, I am showing only the COUNTER.BAT Full Screen DOS
session data.
Component OS/2 2.1 Full Screen Win NT Full Screen Ratio
Word Processing 1.48 1.88 1.27
CAD/Draw 1.24 3.07 2.41
Spreadsheet 1.47 1.88 1.28
Paint 1.62 1.96 1.21
Overall RPM 1.45 2.20 1.52
COUNTER.BAT Bytes 388 20 0.05
Analysis of component 3:
These benchmarks show that WinTach performance under OS/2 2.1 loads
down when multitasking. On the other hand, using the "Best" setting,
the background task gets little or no resources. In other words, in
the "Best" setting, Windows NT seems to be a singletasking operating
system.
4. The fourth and final component of the benchmark is running
COUNTER.BAT in the background and WinTach in the foreground. In this
run, Windows NT tasking is set to "Foreground and Background
Applications Equally Responsive," i.e. "Equal."
The sequence of events is exactly as described in component 3 above.
Component OS/2 2.1 Full Screen Win NT Full Screen Ratio
Word Processing 1.48 0.73 0.49
CAD/Draw 1.24 1.65 1.33
Spreadsheet 1.47 1.01 0.69
Paint 1.62 0.82 0.51
Overall RPM 1.45 1.05 0.72
COUNTER.BAT Bytes 388 308 0.79
Analysis of component 4:
This component shows "Foreground and Background Applications Equally
Responsive" (i.e. ratios of 0.72 and 0.79 compared to OS/2 2.1) just
as the Windows NT "Equal" radio button states.
HOWEVER, THE PERFORMANCE RATINGS FOR WINDOWS NT ARE ABOUT 75 PERCENT
OF THE PERFORMANCE RATINGS FOR OS/2 2.1.
1. The first interpretation from these benchmarks that I propose
for discussion is that, in this test, Windows NT has about 75 percent
of the performance of OS/2 2.1.
In other words, tradeoffs exist between benefits and costs of various
operating systems. The portability of Windows NT in comparison to
OS/2 2.1 is a benefit. The cost of this benefit is the reduced
performance of Windows NT in comparison to OS/2 2.1.
This seems to be in agreement with information that I previously
received on the Fidonet Operating System Debate Conference from
someone named Thomas McWilliams. He wrote:
"There is no way NT is going to outperform OS/2 on current Intel
hardware. NT is a message passing kernel that relies on separate
servers for almost all services. Because of the complexity of the
data structures that have to be maintained, NT has to manipulate many
more data structures in order to perform a comparable task than would
be necessary under a more traditional kernel such as OS/2."
"Because of the dearth of registers on the Intel CPU, these data
pointers must continually be re-loaded from core. This is a
tremendous overhead. This is not an NT fault per se, but rather a
fault of the hardware/software mismatch between Intel X86 architecture
and a complex message passing kernel. By the same token, OS/2--Mach
will not perform as well as the current OS/2 monolithic kernel on a
given Intel X86 machine."
2. The second interpretation from these benchmarks that I propose
for discussion is that the performance difference confirms Thomas
McWilliams' statement.
3. The third interpretation from these benchmarks that I propose for
discussion is that heavier loads most likely will show a larger
performance difference between Windows NT and OS/2 2.1. This is
simply extending Thomas McWilliams' ideas and asking "what happens as
a single processor (especially one with few registers like an X86) has
to handle more and more tasks and a wider and wider variety of tasks?"
This third interpretation suggests that careful benchmarking under
real load conditions must be done when evaluating operating systems on
various platforms.
4. The fourth interpretation from these benchmarks that I propose
for discussion is that users should expect that it is better to run
message passing kernel operating systems on RISC microprocessor
platforms than on Intel X86 platforms. Therefore users who are
considering Windows NT and other message passing kernel operating
systems will have to consider purchasing new and different hardware.
This will probably lengthen project schedules and add new forms of
project risk.
5. The fifth interpretation that I propose for discussion is that
message passing kernel operating systems on SMP hardware may not be
the "magic bullet" that the trade press has suggested.
I have asked systems engineers at both IBM and NCR about the relative
performance contribution of the processors in an N+1 processor SMP
system. They agree that the 2nd through N+1st processors achieve at
most 80 percent of the performance of the first processor. What is
the implication for the relative performance of a single processor
OS/2 2.1 system (scale performance so this counts as 1 unit) and an
SMP Intel X86 system under Windows NT? As measured in the benchmark,
the first processor of the SMP Intel X86 system under Windows NT is
0.75 units of performance (related to OS/2 2.1). Each of the 2nd
through N+1st processors contribute 0.8 * 0.75 =0.60 units of
performance. Therefore a two processor SMP Intel X86 system under
Windows NT is 0.75 + 0.6 = 1.35 units of performance. Furthermore, a
three processor SMP Intel X86 system under Windows NT is 1.35 + 0.6 =
1.95 units of performance. This analysis concludes that a three
processor SMP Intel X86 Windows NT system is required to double the
performance of a one processor OS/2 2.1 system. At the Windows NT
launch event on May 24, 1993 I asked an Intel employee how long it
takes Intel to double the performance of the X86 processor that it
sells for a given price. The reply was that Intel doubles the
performance of its chips at a given price in 18 months.
Therefore, if time is on the side of the project manager, it is
possible to wait for processor improvements and to run OS/2 2.1 or
some other monolithic kernel operating system. This will be less
expensive than buying an SMP system. This also suggests that the real
market for Windows NT SMP systems is configurations with 6 or more
processors. This would give 4 times the performance of a single
processor monolithic kernel operating system.
6. The sixth interpretation that I propose for discussion is that
users who believe that they need SMP performance should benchmark SMP
versions of monolithic kernel operating systems. I believe
that examples of such operating systems include NCR SMP Unix, Sequent
Dynix and the SMP extensions to OS/2 2.X that IBM demonstrated at
Comdex. The IBM extensions will enter beta test later this year.
According to a message from Steve Woodard of IBM, the current
expectation is that these extensions will be able to support 16
microprocessors.
Thank you for reading this message.
Best wishes.
Jonathan Handler