NetNews Usenet Archive 1992 #16/NN_1992_16.iso/spool/comp/ai/neuraln/2993

Internet Message Format | 1992-07-27 | 4.2 KB

Path: sparky!uunet!usc!sdd.hp.com!uakari.primate.wisc.edu!relay!afterlife!hcbarth
From: [e-mail] (Bart Bartholomew)
Newsgroups: comp.ai.neural-nets
Subject: Re: re:need for unique test sets
Message-ID: <[e-mail]>
Date: 28 Jul 92 06:02:28 GMT
References: <[e-mail]> <[e-mail]> <[e-mail]>
Organization: The Great Beyond
Lines: 74

In article <[e-mail]> [e-mail] writes:
>In article <[e-mail]>, [e-mail] (Bart Bartholomew) writes:
[ really good stuff deleted :)]
>|>     The point of having a test set (taken from the same
>|> source as the training set) is to make sure the net has found
>|> the right (or equivalent) function.
>|>     On the other hand, if the net does well on the training
>|> set but falls apart (scores badly) on the test set, you know that
>|> the net has found a nice function that describes the training
>|> set well, but *is not the function that actually generated the
>|> data*.  In that case, the net is worthless.
>
>    I guess this is pretty difficult to answer, but what
>    basis do we have for expecting that the net should
>    (or can) learn a function we have 'in mind', when there
>    are infinite trivially different ways of mapping any
>    given data set?
>    If we have no organizational basis for our learning, then
>    what inherent organizational concepts do we expect the
>    net to extract?
>
>    tom

    Quite often we believe that there is some
rational generator for our data, but we do not know
what that generator is.  We observe some apparently cause
and effect situation, encode the information, and ask the
net to see if it can produce the effect from the 'cause'.
    Sometimes it can, sometimes it can't.  We may have
made an error and the relationship does not exist.  We may
have an error in our net topology (too small, too big, no
pinch point, inappropriate pinch point, lots of things).
    Since we don't know what the generator is, we must
try to determine if the network has found the right (or
equivalent (possibly congruent)) function.  Even if the
net describes the test set perfectly, we still cannot *know
absolutely* that the net is correct until we test it with
some input and see if the output looks reasonable, or is
acceptably close by some criterion.
    At the moment, we don't have any tools that I know
of to examine a BP net and understand what's going on except
for some trivial cases.  I suspect I'd recognize an XOR net
fairly easily, but there are only four distinct configurations
for an XOR BP net.  More complicated functions are more difficult
to analyze.  Not necessarily impossible, but definately more
difficult.  At this point I expect Prof Armstrong to point
to the inordinate ease with which an ALN can be analyzed.
I remain unconvinced, however, of the universal applicability
and/or superiority of ALNs over BP in all situations.
    One advantage of using nets to hunt for a function
to describe the data is where the dimension of the problem is
too large, or where the amount of data is too small.  In either
case, there is a distinct chance that the net will find the
wrong function.  Sometimes, we get lucky.  Sometimes the net
will tell us which of the input points don't matter, or which
ones are the most informative.
    If we can find a net which adequately describes both
the training and the test set, we can sometimes use it without
knowing the function.  You don't have to be a EE to use a
television, you just have to know how to operate it.  I'm not
a mathematician, but I know how to feed an FFT and how to
interpret the output.  Similarly, nets have been used in
a variety of cases where we don't know what it's doing, but
we have confidence that it is doing the right thing.
    Like I said earlier, you pays your money and you
takes your chances.  Sometimes we don't have any acceptable
alternatives.
    Bart

--
"It's not the thing you fling, the fling's the thing." - Chris Stevens
 If there's one thing I just can't stand, it's intolerance.
*No One* is responsible for my views, I'm a committee. Please do not
infer that which I do not imply.  [e-mail]