Subject: TSQL Benchmark

From: "Rick Snodgrass" <rts>
Date: Wed, 3 Mar 1993 22:20:31 MST

To the TSQL community:

In the past six months or so, several temporal query language
benchmarks have been proposed.  Shashi Gadia has distributed a
document that combines a benchmark of two relations and 14 queries, a
discussion of TempSQL, and the benchmark queries expressed in TempSQL.
Abdullah Tansel in several technical reports has used a benchmark of
30 queries over a similar schema to evaluate the performance of his
temporal DBMS implementation. And Patrick Kalua and Edward Robertson
in their workshop submission have proposed three semantic benchmarks,
one similar to Shashi's department database and also containing 30
queries.

Developing a consensus benchmark that combines all of these efforts
would be very helpful to the community.

I propose that everyone interested in the benchmark work together to
assemble a consensus benchmark that could be presented at the workshop
in June.

To this end, I have asked Christian Jensen to coordinate this effort.
Christian did an excellent job in coordinating the initial glossary
(and its ongoing successor, which is already of similar size). The
present task will be primarily one of gathering the queries, putting
them into a logical framework, assembling the document (title: "The
TSQL Benchmark"), and in general ensuring that progress is made
towards the goal of completing the benchmark by June. Though Christian
realizes how much effort will ultimately be involved in seeing this to
fruition, he has nevertheless agreed to take on this task.

The TSQL Benchmark will differ from first generation benchmarks in
several important aspects. What we need now is a much more
comprehensive approach, starting from first principles.  This approach
will first outline informally the types of queries that can be posed
on the schema. This will allow everyone to attempt to populate the
benchmark to cover all the bases. The TSQL Benchmark will contain many
more queries (perhaps around 100) than any of the first-generation
benchmarks. The benchmark will be independent of any data model, so as
to not favor any specific language proposal. It will also not include
any queries in a specific query language; future documents can present
the benchmark in a particular language, or compare several languages
along the benchmark. The TSQL Benchmark will be in carefully crafted
English, with no other formalisms, to avoid any potential bias.

Development can continue on the first generation benchmarks, including
Shashi's.  Indeed, they have much to contribute. As insights are
generated in these benchmarks, it is hoped that they will eventually
migrate into the more comprehensive, consensual benchmark. Shashi
should certainly continue to coordinate development and extension of
his benchmark. Anyone who wishes to participate in either, or both,
is encouraged to do so.

Let me also propose a few structuring principles for the TSQL Benchmark.

1. The document will specify a semantic benchmark, with the
objective of comparing the descriptive and operational characteristics
and capabilities of the various models (Patrick and Ed originally
emphasized this distinction). While the benchmark can also be used for
performance comparisons (with more data), that will not be its primary
focus.

2. The benchmark should contain queries that appear reasonable. A
temporal query language should ideally be able to express these
conveniently and naturally.

3. The benchmark should not be considered to constitute a metric for
query language completeness. Such a metric should be based on more
formal notions. On the other hand, the benchmark should be complete in
the sense that it includes as many reasonable queries over the schema
as possible, to ensure that all aspects of query language design are
considered.

4. The benchmark is to be a consensus effort. All researchers,
especially those proposing TSQL language constructs, will be
encouraged to contribute.

5. Everyone who contributes in any significant way will be an author
of this document. Queries will be included if the authors agree that
they are reasonable.

6. The (ambitious) goal will be completion of this benchmark by the
workshop. The benchmark will be used subsequent to the workshop as new
language constructs are proposed.

Let me also propose a process to get started with the document.

The first decision should be specifying the language functionality
that will be included and, more importantly, the functionality that
will *not* be included in the first draft of the document. Since an
attempt will be made to ensure informal completeness, the scope should
be purposely quite narrow.

The next decision will be that of the schema to be adopted. Again, the
scope should be as narrow as possible.

Third, a taxonomy of queries over that schema should be developed,
attempting to be maximal to avoid holes.

After this basis has been developed and agreed upon, specification of
the actual queries, and population of the example schema with actual
data, can begin.

I ask Christian to make strawman proposals for language functionality
and schema, for discussion by the community.

This tsql email mailing list will be the medium of discussion.
Schemas/instances/queries can be proposed, discussed, and eventually
incorporated into the document. Christian will periodically make
available draft versions of the document for comment.

Given that there already exists a body of some 75 queries (admittedly,
overlapping and somewhat contradictory, but nevertheless available),
it appears that there is sufficient material to begin construction of
a comprehensive benchmark that will be of great help, initially in
comparing language proposals, and eventually in characterizing and
exemplifying temporal databases.

Sincerely,
	Rick