OS/2 Shareware BBS: 14 Text/14-Text.zip/winbn2.zip/WINBN2.TXT

Text File | 1993-06-23 | 23KB | 529 lines

Benchmarking "Round 2"
May 17, 1993

by Timothy F. Sipples
Internet: [e-mail]

Windows NT v. OS/2 2.1
(March public beta releases)

Some months ago I spent some time comparing the then current beta
releases of OS/2 2.1 and Windows NT 3.1.  Testing was conducted in a
fairly limited fashion using an older release of the Ziff-Davis WinBench
benchmarking program.

In this comparison, I have used a wide range of Windows benchmarking
programs, not just WinBench.  I have focused on Windows benchmarks to
be, if anything, more fair to Windows NT.

The hardware used for these comparisons is an IBM PS/2 Model 56 with
486SLC-20/40 CPU, 12 MB of RAM, and 160 MB SCSI hard disk drive (a
fairly good baseline system, comparable to the typical systems found on
desktops in businesses).  I did not conduct tests with 8 MB of RAM (to
be more than fair to Windows NT).  This system was equipped with the
standard IBM 16-bit onboard VGA display hardware.  A standard IBM PS/2
mouse was used, and it was not moved while benchmarks were running.

The hard disk drive was partitioned into two drives, C and D, both FAT,
and OS/2's Boot Manager.  NT was installed on Drive C (about 90 MB), and
OS/2 was installed on Drive D (about 60 MB).  Both operating systems
were installed with all default settings (unless otherwise noted below).
All benchmarking programs were installed on Drive C.

The following benchmarking programs were used:

Dhrystone for Windows (Version 1.1 by Richard Levey, Visual BASIC);
WinTach 1.0 (Texas Instruments);
WinSpeed 1.0 (Chris Hewitt);
Ziff-Davis Labs WinBench 3.11.

For the multitasking tests, a batch file, COUNTER.BAT, was used.  The
batch file reads:

@echo on
cls
c:
cd\
echo A > counter.fil
:loop
type temp.txt > temp1.txt
xcopy temp1.txt temp.txt
del temp1.txt
echo A >> counter.fil
goto loop

The file temp.txt was constructed using the following command:

dir d:\ /s > temp.txt

OS/2 supports two modes for Windows compatibility.  Full screen mode is
most closely analogous to Windows itself.  The Program Manager, and one
or more Windows applications, run in a full screen Win-OS/2 (Windows)
desktop.  Windows applications can also run "seamlessly," on the OS/2
Workplace Shell desktop.  Both modes of operation were tested.  When
running seamlessly, applications were started from the Program Manager
(to most closely approximate what Windows NT does, and to be more than
fair to NT).

OS/2 also supports both standard and enhanced mode Windows
compatibility.  Tests were conducted under the default standard mode.
Comparisons were made with enhanced mode, however, and no measurable
differences in performance were detected.

No patches were applied to either operating system.  The original,
public beta distributions were compared.  (The retail release of OS/2
2.1 was not used.)

Neither beta release could successfully complete the Windows User
Magazine benchmark suite.  OS/2 2.1 beta could not execute the WinTach
benchmark in "seamless" Win-OS/2 mode.

With WinSpeed, which provides a continuous readout of values, the value
was recorded six times, once every five seconds.  These values were then
averaged, and the average is reported below.

Note that neither company recommends using these beta releases for
performance benchmarking.  However, the results may be interesting, and
they should provide some help in benchmarking the final releases.

Here are the results.

Dhrystone for Windows
Higher number means better performance.  Linear scale.

Windows NT:        3776 CPUStones/Second
OS/2 Full Screen: 15210
OS/2 Seamless:    15037

Here OS/2 outperformed Windows NT by a factor of four.  The results are
certainly surprising, since this benchmark is written in Visual Basic (a
Microsoft product).  So surprising, in fact, that the benchmark was
rerun to confirm these results.  The numbers hold.

WinTach
Higher number means better performance.  Numbers are expressed as a
multiple over a baseline system.

   Component          OS/2  Full Screen    Win NT Full Screen

   Word Processing          2.23                  1.89
   CAD/Draw                 1.84                  3.11
   Spreadsheet              2.08                  1.9
   Paint                    2.33                  1.98

   Overall RPM              2.12                  2.22

These results are somewhat mixed.  WinTach weights the CAD/Draw category
heavily enough to give NT the slight nod in the Overall RPM figure.
However, OS/2 bested NT in all the other categories comprising the
WinTach overall figure.

Multitasking Tests with WinTach

The batch file (COUNTER.BAT) creates a file, COUNTER.FIL, which measures
the number of iterations completed.  The number of bytes in the file is
equal to the number of iterations multiplied by four (the letter A, a
space, carriage return, and line feed are recorded for each iteration).
Byte size values for COUNTER.FIL are reported below.

First COUNTER.BAT was executed by itself, without another task running.
It was run for 60 seconds.

   COUNTER.FIL Length     OS/2                 Win NT

   Full Screen DOS Sess.  236 bytes            164 bytes

   Windowed DOS Session   164 bytes            164 bytes

Thus, with COUNTER.BAT itself, OS/2 has the edge, since it bested NT in
one mode by about 40% and matched NT in the other.

Then, in the following tests, WinTach was loaded and made ready to go.
COUNTER.BAT was started, then WinTach was started.  As soon as WinTach
displayed the final screen (reporting the benchmark figures), a switch
was made to the COUNTER.BAT session and it was immediately terminated.
I estimate that this method can produce errors in the range of plus or
minus four iterations (16 bytes).  "Full Screen" and "Windowed," below,
refer to COUNTER.BAT.  (Windowed DOS sessions were maximized.)

   Component       OS/2 2.1 Full Screen   Win NT Full Screen ("Best")

   Word Processing          1.48                  1.88
   CAD/Draw                 1.24                  3.07
   Spreadsheet              1.47                  1.88
   Paint                    1.62                  1.96

   Overall RPM              1.45                  2.20

   COUNTER.FIL Bytes      388                    20

   Component       OS/2 2.1 Windowed      Win NT Windowed ("Best")

   Word Processing          1.50                  1.85
   CAD/Draw                 1.25                  3.12
   Spreadsheet              1.43                  1.86
   Paint                    1.58                  1.75

   Overall RPM              1.44                  2.15

   COUNTER.FIL Bytes      432                    16



Thus, in the default setting, NT is tuned as essentially a single
tasking operating system, which is curious given that Microsoft proposes
NT for server environments.

Both OS/2 and NT provide some control over the priority given to
foreground and background tasks.  OS/2 provides the CONFIG.SYS
parameters MAXWAIT, TIMESLICE, and PRIORITY.  NT provides a Tasking
control in the System section of its Control Panel.  This tasking
control provides three options: one providing the "best" foreground
response time (as shown above), one providing a moderate foreground
boost, and one providing an equal responsiveness to foreground and
background tasks.  These settings appear to be roughly analogous to
OS/2's PRIORITY=DYNAMIC and MAXWAIT as some high value, PRIORITY=DYNAMIC
and MAXWAIT as some moderate value, and PRIORITY=ABSOLUTE, respectively.
In practice, however, there was no measurable difference between NT's
first two Tasking settings.  The third, however, did have a measurable
impact, and those numbers are represented below (alongside the same OS/2
numbers):

   Component       OS/2 2.1 Full Screen   Win NT Full Screen ("Equal")

   Word Processing          1.48                  0.73
   CAD/Draw                 1.24                  1.65
   Spreadsheet              1.47                  1.01
   Paint                    1.62                  0.82

   Overall RPM              1.45                  1.05

   COUNTER.FIL Bytes      388                   308

   Component       OS/2 2.1 Windowed      Win NT Windowed ("Equal")

   Word Processing          1.50                  0.71
   CAD/Draw                 1.25                  1.82
   Spreadsheet              1.43                  1.02
   Paint                    1.58                  0.83

   Overall RPM              1.44                  1.10

   COUNTER.FIL Bytes      432                   308


These numbers seem to indicate that OS/2 is the better multitasker of
the two.

WinBench

Ziff-Davis Labs WinBench 3.11 (running alone) gives the following
results:

WinBench 3.11     OS/2 Full Screen      OS/2 "Seamless"   NT
BITBLT Alignment
Source Aligned
Destination
Aligned
AA 32x32          3586101 pixels/sec 2397817           2738061
AA 64x64          5270784            4492703           4713878
AA 128x128        6016760            5788669           5867332
AA 256x256        6193597            6325468           6216854
Source Aligned
Destination Not
Aligned
AN 32x32           198657             520954            628517
AN 64x64           793925             790632            922501
AN 128x128        1019222            1008052           1132575
AN 256x256        1109711            1149754           1251744
Source Not
Aligned
Destination
Aligned
NA 32x32           546133             503832            802738
NA 64x64           806287             780917           1103764
NA 128x128        1009411             998264           1251744
NA 256x256        1103764            1143825           1334551
Source Not
Aligned
Destination Not
Aligned
NN 32x32           522095             481705            595782
NN 64x64           777875             753017            857926
NN 128x128         991280             980527           1103764
NN 256x256        1088556            1127281           1230001
BITBLT OVERLAP
Horizontal
Overlap           5212353            4567712           4980364
Vertical Overlap  1180743            1148397           2278712
BITBLT ROPS
Screen to Screen
SS SRCCOPY        1808738            1756871           3092727
SS SRCPAINT        704140             699178            155191
SS SRCAND          705113             699589            158601
SS SRCINVERT       703723             699589            160364
SS SRCERASE        669936             666934            158601
SS NOTSRCCOPY      585254             892411            161560
SS NOTSRCERASE     684310             680659            156877
SS MERGECOPY       847090             835259            118301
SS MERGEPAINT      691062             377497            156877
SS PATCOPY        6439612            5035005           5790108
SS PATPAINT        632510             627634            114545
SS PATINVERT      1002273             876159           3608182
SS DSTINVERT      5286740            4607851           4758042
SS BLACKNESS     11570966            8069324           9334209
SS WHITENESS     11552512            8056478           9438669
Memory to Screen
MS SRCCOPY        2831629            2566164           1427413
MS SRCPAINT        892244             863509            643042
MS SRCAND          892411             863850            649473
MS SRCINVERT       891409             863509            643042
MS SRCERASE        853382             827816            627510
MS NOTSRCCOPY     1319378            1257007            853548
MS NOTSRCERASE     856059             829704            620762
MS MERGECOPY      1193909            1134432            147273
MS MERGEPAINT      891576             863167            643042
MS PATCOPY        6444713            5055045           5788951
MS PATPAINT        818811             790953            140124
MS PATINVERT      1006378             876804           3647832
MS DSTINVERT      5312085            4608758           4785709
MS BLACKNESS     11685140            8072542           9412648
MS WHITENESS     11659532            8040477           9436783
Memory to Memory
MM SRCCOPY        3109283            3262830            713849
MM SRCPAINT       2309796            2384477            291570
MM SRCAND         2309796            2383091            303847
MM SRCINVERT      2308005            2381400            300682
MM SRCERASE       2065594            2117427            297582
MM NOTSRCCOPY     2405455            2479656            307079
MM NOTSRCERASE    2184627            2241936            291570
MM MERGECOPY      2024603            2033780            181925
MM MERGEPAINT     2180341            2244997            297582
MM PATCOPY        3473270            3553621           6978462
MM PATPAINT       1768100            1753159            173193
MM PATINVERT      2568680            2598234            225511
MM DSTINVERT      3284043            3405168           8802378
MM BLACKNESS      4589354            4925094           8169604
MM WHITENESS      4544058            4842149           7922312
SCREEN/Memory
BITBLT (64x64
mono)
Memory to Screen
Blt               6344240            5062559           4222680
Screen to Screen
Blt                807094             808368            963009
IMAGE BLT         5116124            5286195           2852147
DRAW ARCS
Arc Major Axis X    27885              27685             71058
Arc Major Axis Y    26945              26997             63789
FILL ELLIPSES
Ellipse Major
Axis X             901457            1126338           5352360
Ellipse Major
Axis Y            1592549             941584           4911289
DRAW SINGLE
LINES
Single
Horizontal        2937451             576868           1347988
Single Vertical   1003996             826254            664390
Single Diagonal    341798             325243            421678
DRAW POLYLINES
Poly Horizontal   4923690            5015618          10969468
Poly Vertical      743174             769719            763516
Poly Diagonal      360359             374840            599669
Poly Mixed         671477             696494            997168
DRAW LINES IN
ALL DIRECTIONS
Long Lines         329359             441831            437931
DRAW WIDE LINES
Line Width 3        68754              20502            156431
Line Width 5       103125              49568            225218
Line Width 10      209436             104368            446543
Line Width 20      329179             162942            695987
POLYGON
ALTERNATE FILL    1141140             498741           3074339
POLYGON WINDING
FILL               18.9 polygons/sec    9.58              59.3
DRAW & FILL
RECTANGLES
Square
Rectangles   20923318 pixels/sec   14864968          18376880
Rounded
Rectangles   2370741               1208339           5201881
RANDOM
RECTANGLES   3058754               3331129           3653264
FILL WITH
PATTERNS
Black Pattern    27982547        28475847          29049866
12% Grey Pattern 27786828        28382375          28737501
25% Grey Pattern 27786828        28442786          28870545
37% Grey Pattern 27797337        28431783          28786375
50% Grey Pattern 27987875        28514514          29049866
62% Grey Pattern 27776327        28431783          28786375
75% Grey Pattern 27982547        28497929          29695418
87% Grey Pattern 27797337        28426285          28737501
White Pattern    27961255        28470331          29083142
Hatch Patterns   27823646        28453798          29049866
DISPLAY VARIOUS
FONTS
Display 16 pt
System            3070623         3222089           3656036
Display 21 pt
New Tms Rmn       3748763         3551673           3790435
Display 27 pt
New Tms Rmn       4596578         4369888           5130836
Display 20 pt
Arial             3549849         3301090           3793064
Display 24 pt
Arial             4513947         4261576           4609698
SCROLL VARIOUS
FONTS
Scroll 16 pt
System             170721          163487            137478
Scroll 21 pt New
Tms Rmn            184579          182071            152966
Scroll 27 pt New
Tms Rmn            273838          270682            295948
Scroll 20 pt
Arial              165588          161938            205146
Scroll 24 pt
Arial              208042          194965            197907
EXTTEXTOUT
VARIOUS FONTS
ExtTextOut 16 pt
System            5806584         4789165           3550229
ExtTextOut 21 pt
New Tms Rmn       3804645         3578639           3630130
ExtTextOut 27 pt
New Tms Rmn       4675541         4405458           4822315
ExtTextOut 20 pt
Arial             3613631         3330634           3539799
ExtTextOut 24 pt
Arial             4609851         4293893           4408150
DISPLAY TEXT -
BACKGROUND MIXED
Transparent Text  4022359         2759387           2885682
Opaque Text       5797364         4958974           3599412
GREYED TEXT        363374          332734             74050
MIXED FONTS        814001          670476            792257
STRETCH BLT
Stretch 32x32       15476           14337             13194
Stretch 48x48      105326          106790             26551
Stretch 80x80      189274          194037             60711
Stretch 96x96      224890          231093             77937
Stretch 192x192    361294          368640            187922
Stretch 32x48       75090           75956             17638
Stretch 48x96      158988          162540             42601
Stretch 48x32       83980           84768             18481
Stretch 108x96     229762          236667             85042
Stretch 21x21       29389           29444              6014
Stretch 105x105    235913          243020             87154
IMAGE DATA        3171841         2220601            831675
DIALOG BOXES      6.40 dialogs/sec     2.58              3.54
RUBBER BANDING    207367 pixels/sec  153400            153722
SCROLL TEXT IN
WINDOW
Byte Aligned
Scroll              1415 characters/sec   1392              1166
Not Aligned
Scroll              1086 characters/sec   1081              1074
ERASE WINDOW    24800036 pixels/sec   24936026          25571696
RECTANGULAR
CLIPPING
OPERATIONS
BitBlt Clipping      2252 operations/sec   1883              1136
Line Clipping        4221                  2888              2138
Text Clipping        3030                  2129               175
Apparent Disk
Tests
Sequential Read
200                 194402 bytes/sec  237381            157474
Sequential Write
200                213324             212221            239774
Random Read 200      6522               6117             19387
Random Write 200     4764               4538             25300
Sequential Read
512                405269             415137            186786
Sequential Write
512                143873             466135            283399
Random Read 512     16712               3262             76875
Random Write 512    23684              24251             92369
Sequential Read
2048               476483             489854            204546
Sequential Write
2048               401081             385026            311556
Random Read 2048    16075              18644            300339
Random Write
2048               186219             185645            374417
Sequential Read
4096                79227              72025            233767
Sequential Write
4096               451251             464863            459045
Random Read 4096   118377             120405            403719
Random Write
4096                90837              84752           1371676
OVERALL
Graphics WINMARK  4042864            2762427           4281517
Apparent Disk
WINMARK             26688              16832             95309

Note that WinBench has received some bad press lately.  (It is claimed
that it is far too easy to optimize to WinBench to inflate the results.)

The WinBench numbers resemble those of WinTach in that OS/2 seems to be
outperforming NT in more categories, but the weighting system gives NT a
slight edge in the overall index.  In this set of benchmarks I did not
attempt to run comprehensive disk performance tests.  OS/2 has two file
systems (FAT and HPFS), and NT has three (FAT, HPFS, and NTFS).  A
comprehensive test would report figures for all three file systems.
However, the WinBench disk figures are reported for completeness.

One interesting result coming out of these figures is that OS/2's
seamless Win-OS/2 mode is performing much better than in the previous
beta release.  It is no longer the case that full screen mode
outperforms "seamless" mode.  In some cases, full screen may be a bit
slower.

It is somewhat puzzling that NT performs reasonably well with the two
most popular Windows benchmarks, WinBench and WinTach, yet does not post
good results with two lesser known benchmarks, Dhrystone for Windows and
WinSpeed.  There are several possible explanations for this phenomenon.
One possibility is that NT's programmers optimize to these benchmarks.
That need not be a sinister act; these benchmarks can help determine an
operating system's weak points, and that information can be incorporated
into the development process.  It does, however, point out the need to
perform rigorous, "real world" benchmarks.  It is also generally a bad
idea to rely on a particular benchmark's index number.  It is tempting
to try and express an operating system's performance in terms of a
single index number, but the weighting used to determine that number may
have little or no resemblance to the kind of work the operating system
will be asked to perform in a particular environment.

The multitasking benchmarks are rather disturbing.  It is possible that
the default priority setting was chosen with developers in mind, since
NT is still in beta.  However, to deploy NT on a server this default
must be changed.  If an administrator runs a task at the console with
the current default setting, the entire system will grind to a virtual
halt.

All the benchmarking programs used are available via Internet anonymous
ftp from ftp.cica.indiana.edu.

T.F.S.

Addendum: WinSpeed Results

WinSpeed results are cited but not enumerated.  Here are the results
produced by that benchmark (higher number indicates greater
performance).

WinSpeed Average:   172 NT    190 OS/2 Full Screen    190 OS/2 "Seamless"
Video                70       130                      94
Disk                125       119                     114

Again, I cannot guarantee the accuracy of any disk I/O benchmarks (see
the original note in the benchmarking document), but they are reported
for completeness.