NetNews Usenet Archive 1992 #18/NN_1992_18.iso/spool/comp/ai/neuraln/3153

Text File | 1992-08-12 | 3.0 KB | 66 lines

Newsgroups: comp.ai.neural-nets
Path: sparky!uunet!cs.utexas.edu!usc!wupost!gumby!destroyer!ubc-cs!alberta!arms
From: [e-mail] (Bill Armstrong)
Subject: Re: Characterizing learnable functions
Message-ID: <arms.713626067@spedden>
Sender: [e-mail] (News Administrator)
Nntp-Posting-Host: spedden.cs.ualberta.ca
Organization: University of Alberta, Edmonton, Canada
References: <[e-mail]> <[e-mail]> <arms.713550511@spedden> <[e-mail]>
Date: Wed, 12 Aug 1992 13:27:47 GMT
Lines: 53

[e-mail] (Stefan M. Rueger) writes:

>[e-mail] (Bill Armstrong) writes:
>>Maybe consumers of NNs should ask for and insist on Lipschitz conditions:
>>   |output(a) - output(b)| <=  const * | a - b |
>>
>>for all a, b in the domain of the function.
>>

>These Lipschitz-functions **are** continuous and thus covered  by the
>results of Cybernko, described in my original article.

>>                                             Better yet, replace the
>>const by some C(a) to take into account more a priori knowledge of
>>where you want a smoother function.
>>
>>Has anyone ever worked out this form of approximation?  At least
>>it would be more useful than continuity.

>If the C(a) have an upper bound, say K, (this is always true, if the
>function has a compact domain) then again those functions are of
>Lipschitz type and thus continuous.

I agree with you (except that you don't need a bound K, just
finiteness of C(a)).  My point was that specifying continuity of
functions in a digital world is vacuous.  Any function whatsoever on
any set of floating-point numbers (always finite) can be extended to a
continuous one on the real line (for example).  What I'm saying is
that the theory of continuity (i.e. topology) is not applicable, and
is not what users of neural networks should be asking for.  They
should say in contracts: "The network resulting from training shall
satisfy a Lipschitz condition with constant ___".

The critical thing is this: with a Lipschitz condition, you can use a
finite test set together with the Lipschitz condition to conclude that
there are no points with values outside allowable limits.  If you just
test the net, then you have no knowledge of the values produced by the
net between test points.  The net can produce "wild" values at points you
haven't tested, and hence be unsafe to use.

In short: For purposes of producing useful nets, who cares whether any
continuous function is realizable or learnable?  It just means your
theory is satisfying provided you neglect questions of representations
of real numbers and computer arithmetic.  It doesn't mean a thing in
practice.

I like your idea of a bound K.  Can you compute it for a given net?
Can you force it to be small by appropriate training procedures?
--
***************************************************
Prof. William W. Armstrong, Computing Science Dept.
University of Alberta; Edmonton, Alberta, Canada T6G 2H1
[e-mail] Tel[phone] FAX [phone]