Subject: benchmark directions

From: "Ed Robertson" <[email-masked]>
Date: Tue, 8 Jun 1993 17:14:04 -0500 (EST)

In response to Rick's invitation, the following are our observations
on the purpose and nature of the "semantic benchmark."

Alex and Jim correctly state that 'the whole idea behind this "semantic
benchmark" is to provide some measure of what should be expressible in a
temporal query language.'  This statement has the right amount of
flexibility.  We definitely want the ability to intuitively measure but
need the "some" because we're not sure whether we'll be lucky enough to
have a real metric.  The "should" is also important; it's not "must" but
rather an indication of what's desirable.  Rick has extended their
statement with an interpretation that the measure provides a "sound
theoretical basis"; this may be too much to hope for.

Temporal databases require a different kind of benchmark, a benchmark that
deals with meaning, because the entire motivation for this area is the
specialized meanings we attach to time.  Other nascent specializations
within databases deal with new kinds and representations of information -
audio/video streams, multidimensional arrays, etc. - but temporal
information maps reasonably to integers (albeit the "June 31st" problem)
and relations (albeit fragmentation).  We deal with time specially because
we want to capture the conceptual (and hopefully implementation)
efficiencies related to the very particular semantics we use for this
area.  Thus the issues of temporal databases are related to our thought
and language.  But we are not linguists or philosophers; we have very
specific interests in the development of useful and usable products.  We
can therefore benefit by having ways of assessing the success of our
products.  The meaning of "user-friendly" is therefore not the
bells-and-whistles the purveyors of PC fluff use but rather the
naturalness of casting the queries into the particular proposed tools.

The term "semantic benchmark"* has itself caused some controversy.  That's
not surprising, since the juxtaposition of these two words was intended to
have a certain dissonance, or at least unfamiliarity.  The intention is to
cause people to stop and recognize that this is a different use of
"benchmark" than they are accustomed to.

There was some discussion at SIGMOD that what we were doing was part of
the "requirements specification" for TSQL, but we do not believe that the
final product of our effort be required to meet something derived from
these benchmarks.  First, the benchmarks are meant for evaluation and
guidance, not as an absolute criterion.  Perhaps our use of "benchmark"
harks back to the original use of the term, indicating a reference point.
Second, as noted above, we could in fact do all of the queries in regular
SQL, albeit with fragmented relations and convoluted WHERE clauses.

One difficulty with the benchmark effort as it is focused is that we have
built an apparent self-contradiction into our task.  We have claimed that
we are taking a high-level, user/meaning oriented approach but at the same
time we have developed a taxonomy which is directed toward implementation.
This in fact can be a valuable source of "creative tension" and expect
that the most interesting class of queries will be those which do not fit
into our taxonomy.  In this regard, Alex and Jim are right that we cannot
claim that the benchmark is comprehensive.  We have no basis for
justifying or even evaluating that.  Maybe "wide-ranging and inclusive"
might be better.

Maybe it's too soon to be so grandiose with our expectations.  Our TSQL
effort is likely to fall short of easy expression of all of the queries
we develop.  But we should be aware of the deficiencies as well as
the strengths.

In short, in spite of the problems, we're basically headed in the right
direction.

					Ed Robertson & Patrick Kalua

-------------------------------------------------------------------------
* For those who do not have webster installed on their system, here's what
it thinks of "bench mark" (only listed as two words) and "semantic."

bench mark n : a mark on a permanent object indicating elevation and 
   serving as a reference in topographical surveys and tidal observations
	[well at least the mention of tides gives one good hook for time]

se.man.tic \si-'mant-ik\ \-i-k*l\ \-k(*-)le-\ aj [Gk -se-mantikos 
   significant, fr. se-mainein to signify mean, fr. (Xse-ma sign, token; akin 
   to Skt dhya-ti he thinks 1: of or relating to meaning in language 2: of or 
   relating to semantics - se.man.ti.cal aj