NetNews Usenet Archive 1993 #1/NN_1993_1.iso/spool/bit/listserv/csgl/2089

Text File | 1993-01-08 | 16.1 KB | 310 lines

Comments: Gated by [e-mail]
Path: sparky!uunet!spool.mu.edu!darwin.sura.net!paladin.american.edu!auvm!MCIMAIL.COM!0004972767
Return-Path: <@VMD.CSO.UIUC.EDU:[e-mail]>
Message-ID: <11930108125911/[e-mail]>
Newsgroups: bit.listserv.csg-l
Date:         Fri, 8 Jan 1993 12:59:00 GMT
Sender:       "Control Systems Group Network (CSGnet)" <[e-mail]>
From:         Hortideas Publishing <[e-mail]>
Subject:      Buzz, buzz, buzz
Lines: 298

From Greg Williams (920108)

AVERY: The PostScript files are BIG. The best thing to do is send Bill Powers
money for postage for a hard copy. It would cost me a lot to send via e-mail,
and disks would probably cost as much as the paper itself.

CHUCK: PictureThis works ONLY with PostScript printers. It is possible to get
programs to emulate PostScript on dot-matrix printers, but I don't recommend
it. If you have FULL-TIME access to a LaserJet with a PostScript cartridge,
let me know.

>Bill Powers (930107.0830)

>But Newton DID propose a generative model. It went "Every bit of
>matter in the universe attracts every other bit of matter with a
>force proportional to the product of the masses and inversely
>proportional to the square of their separation." This was
>certainly not what was observed. The observations had to do with
>behavior of planets, the moon, and thrown and dropped objects.
>Newton proposed an underlying set of entities called "masses"
>which had the property stated in his universal law.

I think that Newton's "model" does not postulate an underlying mechanism for
gravity (in fact, folks are still working on models to do this), but only
generalizes the observations, as Skinner's functional relations among
observables describe behavior. Skinner could not say WHY a rat should EVER
become hungry; Newton could not say WHY gravitation should be at all (or why
it should not turn off at odd times, or even why the power involved is 2,
rather than something else). Both Skinner and Newton simply had faith that the
next bit of data (rat or matter) would be like previous ones they had
described correctly (in hindsight). Skinner deferred to physiologists and
evolutionary biologists to make generative models which would go beyond his
faith; Newton deferred to later physicists.

Newton did propose that an entity he called "mass" was the important feature
of matter with respect to gravitational attraction. That entity might be
construed as "underlying" the observable phenomena, but it doesn't seem to me
to provide a underlying mechanism. To me, it seems that Newton took exactly
the same position as Skinner: that functional relationships among observables
which have provided good predictions in the past are acceptable for making
future predictions without their being explained and delimited by models of
underlying mechanisms. If Newton had been asked, "But why is there such an
entity as 'mass'?", he probably would have answered that he wasn't about to
speculate on that, and not just because of timidity -- his purposes didn't
require it. Ditto for Skinner on "hunger."

Still, there is a sense in which generalized description/functional relations
can be said to be "generative": if precise predictions are "generated," even
though the basis of the "models" lies entirely at the level of the phenomena
being described, with no reference to unobservables. In PCT tracking models,
an unobservable reference level for target position is hypothesized in the
tracker, built into a supposedly "underlying generative model," and used to
predict handle movements. The behaviorist can build a mathematically similar
(even identical) "model" and claim that it makes no reference to
unobservables; he/she simply proposes a function relating observable "stimuli"
to observable "responses." He/she takes the "stimuli" as cursor positions and
velocities relative to the target position, and the "responses" as handle
velocities. Both the PCT "underlying generative model" and the behaviorist's
"functional relation" include a feedback connection from handle to cursor;
if they didn't, they wouldn't reflect the experimental set-up. And both can
generate precise predictions of handle and cursor movement (in fact, identical
predictions, if their forms and parameters are the same). Better models, in
both cases, will be judged to be ones which accurately predict both handle and
cursor positions when the tracker is given a different disturbance. Both
models are subject to limits on their predictions because of "noise" in the
subject. Such "noise" could be modeled either descriptively or with an
underlying generative model; the behaviorist would choose the former. If the
PCT modeler chose the latter, then there would be a genuine difference between
the models at the mathematical level. Otherwise, they would differ in
interpretations, only.

>>>It's not worthwhile trying to make a model fit every anomaly.

>>There goes the PCT standard of "accept nothing less
>>99.9999999... correlations."

>Curb your gazelle, sir. I have advocated requiring correlations
>of at least 0.95 before taking data very seriously. If there are
>so many anomalies in a data run that the correlation of model
>versus real behavior drops below that level, the anomalies must
>be investigated or the model must be improved. You shouldn't put
>quotations marks around things I never said.

You say I'm a (the?) CSG gadfly. I'm also the CSG archivist, and here are some
quotes from previous posts on the net to show what you (and Rick) actually
have said. At one point, you say that correlations of ".99 upward" are needed
for a "true science." And Rick says that .99+ is a "reasonable" goal. My point
is that you have not reached that goal in predicting cursor position. Does
that mean that PCT (so far) is not a "true science"? Perhaps the most
interesting line in the excerpts below concerns the "brag" that "When you do a
real PCT experiment, you get an exact match between the model and the real
behavior." Apparently, no one to date has ever done a "real PCT experiment." I
believe that my quote accurately reflects the comments below, although you are
correct that it is not an actual quote of what you said. I'll even drop the
nines after the decimal point and apologize for the exaggeration -- but, hey,
what's a few tenths between friends? When I said "there goes the PCT
standard," I was referring to your apparently selective application of your
own contention that "somebody has to aim for 0.997 or better, and keep aiming
for it no matter how slow the progress. Because only in that way are we going
to understand and not just fool ourselves into believing that we understand."
So let's quit fooling ourselves about PCT models (to date) being able to
predict cursor position well enough, OK? Correlations of less than .9 just
aren't acceptable for true science, unless your definition of true science has
changed recently.

(BEGIN INCLUDED QUOTES)

[From Bill Powers (920112.1700)]

I don't consider any correlation of less than 0.95 to be of scientific
interest, and for correlations that low, a lot of added work is implied to
reduce the span of the error bars.

[From Bill Powers (920113.1200)]

Again, I don't think that any correlation lower then the 0.90s would be
scientifically usable. And you don't get results that you could call
*measurements* until you're up around 0.95 or better.

A true science needs continuous measurement scales so that theories about
the forms of relationships can be tested. This means that correlations
have to be somewhere in the high nineties. True measurements, with normal
measurement errors, require correlations of 0.99 upward.

[From Bill Powers (920213.1300)]

One of my objections to the statistical approach to understanding
behavior is that after the first significant statistical measure is
found, the experimenter quits the investigation and publishes. If you get
a correlation of 0.8, p < 0.05, you next question should be, "Where is
all that variance coming from?" If you set your sights on 0.95, p <
0.0000001, you won't quit after the preliminary study, but will refine
the hypothesis until you get real data.

[From bill Powers (920222.1400)]

You can't base
a science on facts that have only a 0.8 or 0.9 probability of being true.
Such low-grade facts can't be put together into any kind of extended
argument that requires half a dozen facts to be true at once. You need
facts with probabilities of 0.9999 or better -- if you want to build an
intellectual structure that will hang together.

[From Bill Powers (920515.2000)]

Traditional statistical analysis is
based on very low standards of acceptance and extremely noisy data. I would
rather see less data and higher standards: say, correlations above 0.95 and
p < 0.000001. This should reduce the literature to a readable size and make
its contents worth reading.

[From Rick Marken (920624.1030)]

As I said in an earlier post, if
the relationships in your data are not .99 or greater then you should
try to fix the research until you get such relationships.

[From Rick Marken (920624.1320)]

I said (or meant to say) that the criterion for what constitutes a
scientific fact in psychology should be far stricter than it is. I think
a reasonable goal is correlations of .99+. This can be done when you are
studying control -- at least when you are studying variables that can be
quantified relatively easily.

[From Bill Powers (920625.0830)]

Suppose that you're a psychologist just starting in with HPCT. You hear a
lot of bragging: "We can get correlations of 0.997 that hold up with
predictions over a span of a year." Or "When you do a real PCT experiment,
you get an exact match between the model and the real behavior."

When you've thought up an experiment to test a model, carried it out, and
found a correlation of 0.997 between what the model does and what a real
person does, there's only one response: jubiliation. You have actually
discovered a real true fact of nature, a high-quality fact, and fact that
sticks up out of the mass of other facts like a lighthouse.

If you can get 0.997 in a simple experiment, maybe you can get the same
result with a slightly more complicated one. Yes, you can, it turns out.

Once you've set foot on this road, you can see that it leads where we want
to go. Eventually it will lead to a solid reliable understanding of all
that is possible to know about human behavior. There's no point in trying
to skip ahead and guess how it will all come out. There's no point in using
methods that produce bad data and bad guesses and lead to knowledge that
has only a minute chance of being correct. Certainly, those bigger problems
are important. Certainly we need to solve them, as soon as we can.
Certainly, we have to go on trying to cope with them using experience as a
quide where we have no understanding. But if it's a science of life we
want, somebody has to aim for 0.997 or better, and keep aiming for it no
matter how slow the progress. Because only in that way are we going to
understand and not just fool ourselves into believing that we understand.

(END INCLUDED QUOTES)

>>In principle, underlying generative models are more complete
>>than descriptions at the level of the phenomena. But in
>>practice, the former might not be able to predict better than
>>the latter, due to the complexity of the underlying mechanisms
>>and/or hair-trigger situations.

>I suppose that in some imaginary case that might be true. So far,
>however, all the generative models actually developed and tested
>under PCT have predicted a lot better than any descriptive models
>have done. I'm not going to worry about complexity of underlying
>mechanisms and hair-trigger effects until I run into them.

But you say that PCT tracking models fail to predict cursor position over time
with sufficient accuracy for true scientific work. The reason, you say, is
"noise" or "anomalies." Have you tried to make underlying generative models
for the "noise"? I think you have run into complexity and/or hair-trigger
effects and not realized it. I suspect that it will be impossible to predict
cursor position over time better by using underlying generative models of
"noise" than by using descriptive statistics to "model" the "noise."

>Fine. I'll give you a data record showing the cursor, target, and
>handle behavior point by point -- the raw data. I will generate
>the disturbance pattern at random, and scout's honor will fit the
>control model to the behavior without using any information about
>the disturbance or even knowing what disturbance pattern was
>created.

To adjust the parameters in a PCT model, you run trial models successively,
with the loop closed, using the given disturbance, and look for the best fit
to "real" handle position over time, right? To adjust the parameters in a
descriptive "model," the behaviorist would need to do the same. To be more
explicit, the fitting process requires that one USE the disturbance, even
though one needn't KNOW what it is. OK? You sent me data for handle and cursor
positions in a run, but what I really need to fit the parameters is the
ability to run the model with the feedback connection operative. I could do
that with your software, I suppose.

>>You devise an S-R model that will predict the cursor and handle
>>behavior for a new randomly-generated disturbance pattern,
>>unknown to either of us in advance, with exactly the same target
>>behavior. I will predict the new handle and cursor behavior with
>>the control-system model; you predict them with your S-R model.
>>We will then compare the results. You can use any definition of S
>>and R that you please, and any number of integrals or derivatives
>>that you can get from the data.

>>Any operations you like. But whatever you use, the S-R model must
>>be expressed as H = f(C,T).

>I will be extremely interested to see what you decide to use for
>C, without knowing what the disturbance is.

How about H = K * integral(C - T)? (K is a constant to be adjusted for best
fit to the data by running the simulation with the model in it). That's just a
first cut, of course, since it doesn't predict the cursor position well enough
for "true science."

>>Then you go through a long list of red herrings. What I meant
>>to say is that various PCT models such as proportional,
>>proportional-integral, and proportional-integral-derivative,
>>with various nonlinearities and various parameters, and various
>>descriptive "models" with various functions and parameters, ALL
>>give about equally high correlations between predicted handle
>>positions and actual handle positions, but all do NOT give
>>equally high correlations between predicted cursor positions
>>and actual cursor positions.

>How do you know they all give about equally high correlations?
>Isn't all this a conjecture?

A testable conjecture. If I make a PID descriptive "model," maybe it will fit
the cursor position better. And a bit of lag might help, too. Or maybe your
model is already "noise"-limited. If so, then adding a noise term should
improve prediction of cursor position IN A STATISTICAL SENSE, AVERAGED OVER
MANY RUNS.

>>I believe that the PCT models used to predict tracking can be
>>interpreted also as functional "models" -- the choice of
>>interpretation (underlying generative model or function
>>"model") depends only on whether one envisions a reference
>>signal for target position inside the organism (thus explaining
>>why cursor position is subtracted from target position) or one
>>simply notes that prediction is good if there is such a
>>subtraction in the function relating "input" to "output" at
>>each moment and doesn't care about explaining why.

>Sure. The canonical mathematical forms are the same whether you
>think of the signals as being real or not. If you're incurious
>about how the mathematical functions are actually implemented,
>you can just leave it there.

Then the argument is over interpretation of models, not over the models
themselves. The behaviorist will happily use your (or a better predicting)
closed-loop model (perhaps with a descriptive statistical term to model
"noise") and call it an input-output/descriptive/functional relationship
"model." And you will yell "Foul!" The sole determinant distinguishing your
interpretation from that of the behaviorist is the external stimulus vs.
internal reference signal issue. The math is the same.

As ever,

Greg