Viewperf Benchmark Development Kit

OpenGL Performance Characterization (OPC) Committee


Introduction

This document describes to independent software vendors (ISVs) and other interested individuals how to go about creating a Viewperf viewset. This document assumes that you have already read the README in the docs directory and already know how to use Viewperf. The OpenGL Performance Characterization (OPC) Committee's hope is to solicit Viewperf benchmarks from various ISVs which would be representative of the 3D rendering aspect of their application. The goal of selecting 3D graphics performance metrics with an OpenGL application perspective is ultimately to assist the end-user in making graphics platform decisions.

Designing a Viewset

A Viewset is one or more individual runs of Viewperf grouped together to represent a portion of an application. The individual runs of the test remain unique and the results for all runs are available in the metric of frames per second. There are currently five official viewsets: CDRS from Parametric Technology, Data Explorer from IBM, Design Review from Intergraph, Lightscape from Lightscape Technologies, Inc., and Advanced Visualizer from Alias/Wavefront.

Analyze the Application

Given that Viewperf Viewsets are application based, it is necessary to start the process by analyzing the application. Think about how users use the application. Ask the following questions, focusing specifically on the graphics rendering portion of the application:

1) Are there significantly different portions of the application? That is, does a user start designing in wireframe and then render using polygonal-based surfaces?

2) Does the application provide for diverse rendering options (primitives used, attributes, lighting conditions, etc.)?

3) Which visuals or pixel formats are used?

4) Does the application do texture mapping? What are the sizes of the textures? What filters are used?

5) What are the viewing conditions (is the projection parallel or perspective)?

6) Are display lists used?

7) Does the user do blending, antialiasing, or fog?

8) What window size is typically used?

9) Is there any clipping or trivial (backface or frontface) rejection, typically?

10) Other attribute and state information in addition to the above?

11) Is the application run on a local host or on a remote system over the network?

12) If there is shading, is it flat or Gouraud?

13) Is infinite or local lighting used?

14) Is the viewer local or infinite?

15) What are the color attributes: color/frame, color/primitive, or color/vertex?

By far the best way to analyze the application is to run the application while tracing on various platforms. With an OpenGL debugger or with hooks in the source code of the application, try to attain the following information:

o state information

o primitives

o attributes

o visuals

o model complexity

o data types

Which Paths Are/Will Be Critical

Using the information gathered in the analysis of the application, one now needs to determine which paths are critical. This determination can be made based either on the perception (or actual measured data is even better) as to how much time is spent in particular paths or how important are the paths. This may be shown in the following example.

Say we run a molecular modelling application which supports four different basic rendering paths:

o Dot Cloud (primary primitive is points)

o Licorice (primary primitive is lines)

o Ball & Stick (primary primitive is triangle strips)

o Solid (primary primitive is spheres)

We could then identify four paths to exercise using Viewperf. Each of the four paths would probably contain distinct rendering conditions and may require different data sets. To carry our example further, we would enable the following flags and state during each of the four runs:

1) For dot cloud: render points, unlit, no depth test.

2) For licorice: render wide lines, linewidth = 5.0, enable z-buffer, enable fog, enable linesmooth.

3) For ball & stick: render triangle strip, enable z-buffer, enable fog, enable lighting, enable backface culling.

4) For solid: same conditions as ball & stick, except also enable local viewer, local lighting and blending. Since OpenGL has no sphere entities at this time, we could use a model which had triangle strips (roughly approximating the performance of gluSphere).

Choose Datasets

There are two types of data formats for datasets in Viewperf: ASCII and binary. Each has a set of rendering modes associated with it. Given the two types of data formats for datasets and the rendering modes associated with each, one needs to choose among the ASCII Viewpoint datasets available or create mesh datasets. The dataset formats are detailed in later sections called Viewpoint ASCII Data File Format and Binary Mesh Data File Format.

It is important to select datasets of similar complexity in which users typically render while using the application. With the thousands of Viewpoint datasets available, one should be able to find one of these to suit his needs. Contact the OPC Chairman, and he will assist you in attaining datasets from Viewpoint. If you need a mesh format file, also, see the section on the Binary Mesh Data File Format. If you need assistance, contact an OPC member.

Map Application Paths to Viewperf Flags

An example may help to clarify. Let's say there is an ASCII dataset called skullM and this data set fits your needs because many users of your application use a data set of this complexity. The vertex count (14,172 in the object) and average of 3.75 vertices per primitive is typical of data sets rendered by most of your users.

The user is primarily concerned about 2 paths:

1) polylines, direct render, unlit, no z-buffer, perspective viewing, immediate mode, linewidth of 1.0, window size of 900 x 720, single buffered, linesmooth disabled, no stippled lines, flat shading, minimal clipping, and wanted the test to run for a minimum of 10 seconds, the Viewperf command would be:

viewperf -pg skullM -rm LINE -xws 900 -yws 720 -sb -f -mp 10

2) polygons, direct render, lit with 1 local light, local viewer, z-buffer enabled, perspective viewing, display list mode, window size of 900 x 720, backface culling enabled, double buffered, fog off, polysmooth off, Gouraud shading, minimal clipping, single sided lighting, and minimum period of 20 seconds, the Viewperf command would be:

viewperf -pg skullM -rm POLYGON -xws 900 -yws 720 -nl 1 -ll -lv -zb -dl -bf -mp 20

Weighting

With a Viewperf executable, and one or more datasets, you can run Viewperf and test out the Viewset you have created. Weights should be chosen for each Viewperf run. The weights should total 100%.

For example:

#Test 1: lines, direct render

viewperf -pg skullM -rm LINE -mp 10

#Test 2: same test, except use indirect render

viewperf -pg skullM -rm LINE -ir -mp 10

#Test 3: lit, z-buffered polygons, backface culling

viewperf -pg skullM -rm POLYGON -zb -nl 1 -bf -mp 10

Say we believe users of this application render datasets of this complexity 60% of the time for test 1 and 40% of the time for test 3. Test 2 is just a data point the ISV (application owner) would like to test to see what the performance of various systems would be if the application were to start also using the indirect render path, so we assign it a weight of 0%.

We would then weight Test 1 = 0.60, Test 2 = 0.0, and Test 3 = 0.40. This is just for reference now. Test 2 could also be viewed as a way to test future capabilities. The weighting is dependent on the purpose you, the ISV, have in mind for each run of Viewperf. The output of each test is in frames per second. The final composite weighting is a geometric mean calculated using the test weights and the frames per second that were rendered for each test.

As always the best way to learn is by example. Download some viewsets from the OPC web site and examine them closely for yourself.



Viewperf Data File Formats

There are two file formats currently supported for Viewperf: Viewpoint ASCII and mesh binary. An ASCII data set can be rendered as points, vectors, lines, polygons, triangles, quads or triangle fans. The binary data set provides an alternative to the ASCII format and allows quicker parse times for large objects and supports points, vectors, lines, and triangle strip rendering modes.

Viewpoint ASCII Data File Format

An ASCII data set can be rendered as points, vectors, lines, polygons, triangles, quads or triangle fans. The ASCII geometry data sets are commercially available from Viewpoint DataLabs, Orem, Utah (1-800-DATASET), from which the benchmark name was derived. The "VIEWPOINT DATASET CATALOG" 3rd Edition, from Viewpoint DataLabs, has over 1,000 datasets which are available for OPC benchmarking purposes. Contact OPC Chairman John Spitzer (or the current OPC Chairman) to attain the necessary datasets from Viewpoint DataLabs. The Viewpoint catalog can be obtained by contacting Viewpoint DataLabs International, 625 S. State Street, Orem, Utah 84058 or calling them at 1-800-DATASET (1-800-328-2738) or by fax at 801-229-3000.

This ASCII format used by Viewpoint is quite straight-forward. The three files needed are described in the following sections.

Coordinate Data

The coordinate data file (filename.coo) contains the x,y,z triplets for each vertex in the dataset in floating point format. Note that there should not be any comments or extra characters in any of the three filetypes, since the parser will not handle them at this time.

For example, one of the lighter weight datasets which Viewpoint has is an airplane with 184 vertices. The first number, an integer, is the vertex number. This index is followed by the coordinate data x, y, z. This data here comes from 117aL.coo.

1,-2.256932,-2.105091,0.071168

2,-3.641186,-3.370398,0.080528

3,-5.010109,-4.603533,0.166522

4,-5.383357,-4.953618,0.200154

5,-4.696382,-5.623616,0.203137

6,-4.233825,-5.612613,0.192264

.

.

.

183,5.094025,0.271987,0.855254

184,5.187897,0.316068,0.793475

Element Data

The element data file (filename.ele) contains the connectivity information. Lists of vertices from the coordinate file that make up elements in the dataset are given. The parser also keys off this information.

The corresponding element file lists each element (primitive) and which part of the dataset the element is part of. The word which starts each line is the part that the following vertices make up in the dataset. One primitive in the element file corresponds to one line of the form `part_name v1 v2 v3 v4 . . . vn'. There are 172 primitives (lines) in this file, where `wing2' is the first primitive (has 7 vertices) and `rudder2' is line 172, the last primitive. I deleted lines 16 to 166. The beginning and end of the file 117aL.ele remain as follows:

wing2 7 6 5 4 3 2 1

body2 13 12 11 10 7 1 9 8

rudder2 18 17 16 15 14

rudder2 20 17 18 19

rudder2 23 22 21 20 19

rudder2 24 23 19

cockpit 29 28 27 26 25

glass 33 32 31 30

glass 37 36 35 34

body1 11 12 39 38

body1 41 40 39

body1 12 41 39

body1 12 13 40 41

cockpit 44 43 42

cockpit 46 29 25 45 43 44

.

.

.

vents 64 62 63 65

vents 114 115 116 113

exhaust 9 82 8

exhaust 8 145 138

rudder1 158 159 160

rudder2 17 94 16

Vertex Normal Data

The vertex normal data file (filename.vnm) contains corresponding vertex normals for all vertices in the .coo file. The format is nx ny nz. The .vnm file has an implied vertex number. The line number is the vertex number. The 117aL.vnm file follows, with normal number 1 being the first line containing -0.014957 -0.027389 -0.999513 and the last line containing normal n (184 in this case).

-0.014957 -0.027389 -0.999513

-0.022828 -0.027856 -0.999351

-0.022828 -0.027856 -0.999351

-0.022828 -0.027856 -0.999351

.

.

.

0.481478 0.132472 0.866389

0.473063 0.161036 0.866186



Binary Mesh Data File Format

The other type of input file format is BINARY (.msh files). Take the Data Explorer (DX) viewset as an example. The binary mesh format was typical of datasets used by many Data Explorer users. To attain a mesh dataset, the developer put hooks into the application code and in the rendering loops, and output vertices and normals in ASCII format. The developer then converted the ASCII format to binary using a utility from IBM called convert.c. This utility has been included in the tar file you downloaded for your use (in the tools directory). The ASCII format for the binary files looks like:

10

2.102528 6.631618 2.476700

2.270783 6.270783 2.810535

2.190870 6.631833 2.433023

2.356914 6.271525 2.767295

2.278375 6.631922 2.390973

2.440197 6.272179 2.726543

2.366327 6.631902 2.349928

2.523854 6.272787 2.686702

2.454721 6.631777 2.309905

2.224569 6.095931 2.780533

-0.823183 0.378741 0.422995

-0.292534 0.943414 0.156187

-0.829429 0.378929 0.410440

-0.296018 0.943093 0.151487

-0.835528 0.379138 0.397678

-0.299453 0.942765 0.146703

-0.847562 0.361538 0.388495

-0.290026 0.947239 0.136465

-0.854844 0.346737 0.386025

-0.278658 0.951750 0.128538

4

2.102528 6.631618 2.476700

2.270783 6.270783 2.810535

2.190870 6.631833 2.433023

2.356914 6.271525 2.767295

-0.847562 0.361538 0.388495

-0.290026 0.947239 0.136465

-0.854844 0.346737 0.386025

-0.278658 0.951750 0.128538

The above chunk of a BINARY file displayed in ASCII shows the format. Line 1 is the number of vertices in the first primitive, 10 in this case. The next 10 lines are the x, y, and z values for vertices 1 through 10 followed by the 10 normal vectors corresponding to vertices 1 through 10. Line 22 is the vertex count for the next primitive which has 4 vertices. Lines 23-26 are coordinate x y z for vertices 1 through 4. Lines 27-30 are the corresponding normal vectors.

As with the Viewpoint ASCII format files, the Viewperf binary mesh file parser is not tolerant to comments nor unnecessary/missing data.

The following table summarizes the availability of rendering (-rendermode) options for the two input file formats:



Note that when either TRIANGLE or QUAD rendermode is selected for a Viewpoint ASCII dataset, the number of vertices per element must match the rendermode selected. That is, if -rm TRIANGLE is used, all elements must contain 3 vertices, and likewise for -rm QUAD, all elements must contain exactly 4 vertices. An error will occur and execution will stop at parse time if there are more or less than required.

The batch flag (-bt) is used to combine more than one primitive per begin/end pair for Viewpoint (ASCII) datasets. For example, if the dataset contains elements all with 3 vertices, the valid rendermodes would be POINT, VECTOR, LINE, POLYGON, TFAN, and TRIANGLE. The batch flag is valid for POINT, VECTOR, and TRIANGLE in this case. If the batch flag is set to -bt 5, Viewperf will combine 5 elements per begin/end. There would be 5 groups of 3 points in POINT mode (15 vertices), 5 groups of 3 vectors (15 segments, 30 vertices) in VECTOR mode, and 5 triangles (15 vertices) in TRIANGLE mode. The batch flag is not valid for mesh (binary) datasets. For the mesh files, the vertex count per begin/end is given in the file.


Texture File Format

Textures are stored in MTV ray-tracer format as .mtv files. This is a standard binary image file format that consists of a header giving the resolution of the image followed by interleaved red-green-blue values for each pixel. You should use your favorite image manipulation tool to convert your texture image to MTV format.

Walkthru File Format

Viewperf gives you the capability of walking through a scene. Walkthru coordinates are stored in .wlk files. The DRV viewset is an example of using walkthru files.

The format for .wlk files is ASCII lines, each line represents:

frame# eyeX eyeY eyeZ heading pitch roll

where (eyeX, eyeY, eyeZ) is the eye point (ie, viewpoint) and heading pitch roll are expressed in degrees (not radians). The viewer moves through the scene from line to line of the walkthru file. The distance between adjacent eye coordinates divided by the gap in the frame numbers yields the "speed" of the walkthrough.

End of Document


OpenGL Performance Characterization (OPC) Committee
Viewperf Benchmark Development Kit