3D Testing Recommendations When trying
to judge the relative performance of different 3D boards, the use of a standard
measure or benchmark can prove very useful. There are a number of different
ways to benchmark the various 3D boards that exist on the market currently. 3D
testing may either be based on a real-world application such as a game or on a
benchmark utility such as the Microsoft Tunnel application.
1)
Application based testing The first thing to do when testing application
performance is to ensure that the same application(s) is used on all of the
different cards. Comparing the way that one application looks on one card
versus the way that a different application or version of the same application
looks on the other does not provide a meaningful measurement of each boards'
capabilities.
For an application based test to be relevant, all the
sound/joystick/resolution/quality settings should be identical when each card is
tested and the same levels and game sequence should be used in each case.
One
potential problem when testing performance using a game is that many games will
not display an actual frame rate on the screen. An accurate frame rate is by far
the best way to quantify performance, and estimating frame rates empirically can
prove to be a very difficult and inaccurate measurement. It is therefore
strongly recommended to test with a game that can display a frame rate.
2)
3D benchmark recommendations There are typically two different
bottlenecks to 3D acceleration in hardware; the maximum triangle rate and the
maximum rendering rate, measured in texels per second. The maximum triangle
rate is best tested by drawing thousands of small triangles on the screen, while
the maximum rendering rate is best tested by drawing a small number of large
triangles. Both numbers are important in understanding how an accelerator will
perform in real world applications.
Using a spinning object test such
as the Microsoft Twist test to measure the peak rendering performance of
hardware does not effectively stress the hardware capabilities, since only a
small number of pixels are being rendered at a time. One way to make such a
test more effective would be to add a simple textured backdrop to the scene such
as a sky and ground or wall, and give it some motion. This would ensure that
every part of the image is being rendered and texture mapped continually - not
just blitted.
Although calculating the maximum pixel fill performance
of a graphics card and the maximum triangle rate is important, a real-world game
test still provides a better indication of true 3D performance.
3)
Perspective correction One further consideration is to ensure that
perspective correction is being applied. In the example above, a "moving"
backdrop would be perpendicular to the viewers' eyes (like standing in front of
a wall and looking at it head on), and this type of view has no perspective to
it. Some graphics engine do not actually support true perspective correction
(which is required by all objects that are not perpendicular to the viewer's
perspective), but instead support affine texturing.
Affine texturing
produces 'warping errors' when used to map textures onto an object that requires
perspective. The more perspective is required (slant from viewer and depth in
the scene) and the larger the object is, the more apparent these warping errors
would be. Card manufacturers who use affine texturing instead of true
perspective correction attempt to minimize the warping errors produced by their
affine texturing by breaking up a large triangle into a large number of smaller
triangles and then performing the affine texturing on these smaller triangles.
While this minimizes the warping errors due to affine texturing, it results in a
significant hit in performance. Therefore, adding a backdrop which remains
perpendicular to the viewer's perspective in the test mentioned above would not
accurately show the difference in speed between hardware which performs true
perspective correction and hardware which uses affine texturing.
A
simple way to account for peak rendering performance while stressing perspective
correction capabilities would be to enclose the "plus" sign in the
Twist test inside of a large open 3D box. The camera or viewer's position would
be inside the box looking down at the rotating plus sign from the top, through
the open side. The box itself could be rotating around the axis of the camera
in either a clockwise or counterclockwise direction. The point of view would be
inside the box, so that the walls of the box cover the camera's periphery. Of
course the inside of the box could be nicely texture mapped. This way, every
pixel would need to be perspectively corrected and rendered at every frame, and
the peak pixel fill performance of the hardware would be effectively stressed.
Creating this more effective test only requires the addition of 5 faces of a box
(or 10 triangles) with very simple motion - yet this simple change adds a lot to
the significance of the test.
4) CPU Bandwidth A test
which measures how much CPU time is left for game play computing once a card has
reached its maximum frame rate can also be and interesting one. This
measurement is important because it is difficult for any test to accurately
reflect the various CPU loads and mixes of polygon numbers and sizes that will
be required from one game to another and between two frames. Measuring the
amount of CPU bandwidth available once hardware rendering becomes the bottleneck
provides an indication of how much additional information could be handled
simultaneously by the system.
In order for this test to be
representative, it should stress the hardware as much as a real world game
would. Most 3D game and multimedia titles on the market today use a limited
amount of polygons on the screen - usually less than 2000 polygons at a time,
and most of the time less than 1000. Therefore such a CPU bandwidth test should
use a sample game scene comprised of 1000 to 2000 polygons to reflect a real
world situation.
5) Specific notes on a few 3D titles. 1
- Direct 3D titles
Hellbender and Monster Truck Madness by
Microsoft Make sure you are using the rev. 1.1 of the demo versions or
1.0 of the full version of the games when you test, since previous versions may
have had hardware-related bugs. The demo versions can be downloaded at:
http://www.microsoft.com/games/
Configuration - Hardware
acceleration is enabled by going to the Options menu (ALT+O while in game),
selecting graphics options, switching the resolution to 640x480, and setting the
hardware acceleration flag.
Performance - The performance of
this title will depend on several options. Probably the most significant is
whether the cockpit is on/off. The cockpit in both Hellbender and Monster Truck
Madness is very large. This was done to make the software rendering performance
acceptable. Game performance should be compared with the cockpit off (see notes
in generalities above). Switching the cockpit off can be done by toggling
the option on the same options menu as above, or by pressing the I key for
Hellbender and the D key for Monster Truck Madness during the game.
Hyperblade
by Activision Note: There is currently a demo version on the
Activision web site. This version is DirectDraw only, and does not take
advantage of any 3D hardware. There are some betas in circulation that do take
advantage of hardware.
In these betas, you can display a frame rate by
pressing Q to bring up the menu, selecting View, and then Display Frame Rate.
Pressing Q again will return to the game. This option will be disabled in the
full release of the game, and in some later betas.
Microsoft
Organic Art Anniversary screen saver This "package" of 23
screen savers was released by Microsoft for the anniversary of Windows 95. Ten
of the screen savers use hardware acceleration, and can be distinguished by the
fact that they do not leave a "trail" behind them. The screen saver
can be found at: http://www.microsoft.com/msdownload/95saver.htm
To try
out any of the screen savers, right click on you desktop, click on Properties,
and click on the Screen Saver tab. Select the MS Organic Art as the screen
saver, and click the Settings button. From that point, in Preferences, click on
the Hardware Acceleration check box, set the resolution to 640x480x16, click on
Apply, and then click on the Playlist tab to select which ones will be played.
In order to get stats about frame rate, etc. you go to the registry
key: HKEY_CURRENT_USER/Computer Artworks/Organic Art/MS and change the value "ShowMemStats"
from 0 to 1. Now when you enable captions in the saver config, you will get
stats instead of the date and time. These will show, among other things,
frames/s, triangles/s, whether the buffers are in VRAM or System RAM, and
whether the renderer is hardware ("h/w") or software ("s/w").
Note there's a small performance hit associated with drawing the caption bar
itself.
2) Other titles Note: It is preferable to
test Direct 3D titles in order to ensure all boards are tested with the same
application, therefore providing a more accurate comparison. However, the
following are examples of direct ports that can also be used to get an
indication of the boards' performance.
MechWarrior 2 from
Activision Since this game has been ported to a wide range of 3D cards,
it could be a good test application for performance testing. Unfortunately
there is no way to print out the frame rate of MW2 on screen, but the speed at
which the 'sky' is scrolling across the top of the screen is directly related to
the frame rate of the game, therefore a good indication of the hardware's
rendering speed.
Scorched Planet from Criterion Studios Demo
versions of this game have been directly ported to 3D FX based boards as well as
the Matrox Mystique. No frame rate counter is available in this game, but by
flying around the screen at the maximum speed (by pressing the END key,) the
user can get a feeling for the frame rate at which the game is running. |