Զѧ
ACTA AUTOMATICA SINICA
1997 23 1 Vol.23 No.1 1997



Զǰоļ㿴
ƽ
ժҪӲͬԶǰñ˷ԷģѡҪ˶ۣԵǰǰоеһЩ˿.
ؼʡ磬֪ͳƽģ.
SOME VIEWS ON THE RESEARCH OF MULTILAYER
FEEDFORWARD NEURAL NETWORKS
YAN PINGFAN
(Dept.of Automation,Tsinghua University,Beijing 100084)
AbstractThe essential function of multilayered feedforward nerwork was analysed from the point of view of various discipline.Generalization ability,model selection and the problem of limited sample size were discussed qualitatively.Some problems in the research of multilayer feedforward network are pointed out.
Key wordNeural network,multilayer perception,statistical modeling.
1
ף1ݶԵǰ˹(ANN)о״˷߶ԸָһЩҲͬ.й磬ģ2ѷһЩܺõĿ۹ڶǰ(ΪMFN)ое.MFNоܶ࣬вǺܺõģҲһЩ⣬һЩо룬ûץסؼ⣻ЩӦû˵ΪʲôANNǽϺѡʹ˸еӦһäĿ.ΪˣӦע.
1)Ӷ෽MFNõıʣ
2)MFNҪҪʲô
3)úMFNĹؼ⼰Ҫѣ
4)һЩͳᷨǶǵ⣬ӦԷ.
2MFNõ
ҪMFNòֻڶ ANN оӦĹϵͲ.
2.1ƽ
һЩ򵥺ĸȥƽһδ֪ f,ϳõ.һ׷Ԫ.׷ f ֽɲͬƵʵгɷݣԶ׼.ַֽƵķֱʼ޵(ƵгƵһ ĺ)Ƶ.ʱ()ȫԵģʱ()ֱʺܵ.Ԫڿ(ʱ) f ֳһЩСÿһСü򵥵ĺʾ(ʱ)ֱʸ߶Ƶֱʵ.
ӺƽǶȿɰMFN򵥺ɵĸϺ߿ۺϹ.ף3ݸһƽ.
¸ dάŷϿռϵɵĿռϣǳܵģһ f* һ f* 纯fn.
˲ͬṹͲͬڵ㺯.˵κһָϺӦһ MFN֮һ MFNҲдɸϺʽɰ翴ƽ㷨ͼ(Graph)ʾʽ.MFNʾĸϺһ㲻ӲԹǰ˵ּ.ڼӳ磺CMAC4ݣԴ5ݺRBFӦԪָ࣬Ϊڵ㺯磬׷ֽ.
๤⣬綯̬ϵͳģźŵ˲任ȣһһӳ6ݣģ7ݶMFNıƽչ֤ɱƽ Rn ϵĺҿɱƽ()ģ8ݸ˸һĶ()ĸɰMFNάռϵ.
2.2ع
(xiyi) i=1,,n Ѱ y  xϵ(ھ) y=f(x),ͳѧеһϷ֧عоݣ˵һǲعģͶӦһMFNһMFNҲдӦĻعģ.ģ9ڴͳƹ۵оANNѧϰʱϣ۲Ĺϵд d=g(x)+ţg(x)=Edx.ǻعģ.ģ10ݴӷݷĹ۵MFN ã֤ʱǰʻֵģԴˣģ11иϸ.
عһͣʽMFNӽͶӰѰٻع(Projction Purswit Regression,PPR)12ݣ x  y ϵɱʾΪ


¹ǣһȰѸά x ͶӰάռ uڶ u ķԺҵϱƽ yҪѧϰĲik,ƽ ҵʽֵ wkģ1314ݶPPRMFNѧϰ˱Ƚϣģ15ݴPPRĹ۵Cascade-Correlationѧϰģ16ݽPPRʵַ.
3)滯(Regularization Theory)
T.Poggioģ17аѼලѧϰƽϵ滯ǣ˵滯ɵȼһֻһMFN֮Ϊ滯.һ滯磬˺ܹһƽ磬ģ18Ƕ滯ANNȫܽ.MFNɿĳһ滯ʵ֣MFNѧϰһ119ݣ滯۵ϵԶ׼.
3MFNĻҪҪ
()ѧϰƹ()ǶMFNĻҪ(йطۣ20).ˣݴԡѧϰٶȵʱҲҪܶԣҪ.ṹ(ģ)Ȿĸӳ̶й.ĸӳ̶޷ƣʵʿɴ⣺
1)ṹΪﵽõƹҪѵ.
2)ѵѶȷṹԱ֤õƹ.
ڶʵ࣬Ѷʱθݶ֪ʶ(еĻ)ṹһBPѧϰ(Ȩֵ).ǰǽṹѧϰǲѧϰĿǰ󲿷оԲѧϰģӦоǽṹƣĿǰԴûķģ21ݶԴ˳֣̽һڽͳƽģ۵һ»ԭ.
4ͳƽģԭ
ģһֶδ֪ıƽģ͵ҪǣFlexibility()Dimensionality(㸴) Interpretability(͸).FlexibilityģͶԲͬṩ׼ȷƽ׼ȷ(ģʵɵƫ)Ϊƽƫ(Bias).DimensionalityǾȷģеĲʱĲȶԣΪƽķ(Variance)νάѡָΪѷƵҪķΧά(Ա)Ѹ(ָ).Flexibility  DimensionalityһìܣԴ˿ɼ˵£22ݣ
(xiyi)ھ y  x Ļع Eyx=õģ.Ϊ


úһ(x) E yxݵĽӽ̶Ϊ(x)ûĶƵɵ



ƫ

 MFNڼ򵥣Էӳδ֪ y=f(x) ĸԣ޷׼ȷƽ f(x)(ƫô).֮ṹڸӣҪɸѧϰδ֪̫࣬ʹѧϰȶ().
14͵ķ޵ʿWilliam Ockhamһ(ʡ)ԭ򣺡֪ʵ(һ)е߾õۡ˳ƴԭǡOccam's Razor.ģ23ݴBazes ۵Դһ˵.һ˼밮˹̹һԡAny theory must be as simple as possible,but not simpler"ľһµ.Խģ˵(ӣѵ)һµõѡ.ͨĽģ׼ܰһģһ(öȻʾΪ˹ֲʱǾ)Ƕģ͸()Լ(ÿΪʵƫԼٷ).Ϣо(AICNIC)24ݣ滯25ݣС(MDL)2627ݶ.ģзǻʾĹɾĳ̶ֳȵƽ()滯ۣعģҲԲͬʽ f(x)ƽԼԼ.
5㿴
ΪʲôANNԼúANNĹؼ⣬һЩͳǶǵᷨӦ.磺
1)ANNĹûƸӽԣԶģʽʶ𣬶ԻӦANNáӦеANNģֻԵһֱ(Ǳʵ)ԽƣģͻƵĸӳ̶Իкܴ࣬ANN һֹϵģ.е ANN ģȥһⲢעȴͳãͳеһЩ(ģѡ񡢱ѡ)Ҳûʧ.
2)ANN ϾѧϰӦֻҪӣANN ͨѧϰ⡱.ֻ ANN ѧϰнϳʷģʽʶѵۣ2829ҲͨѧϰģҲֻҪӾһѧĹɣһǿѧϰģЩѧϰѧϰĸΣǼѧϰо3032ݣһЩо߲ҪԣΪѧϰٶҪì.
3)ANN ģκηԹϵҲйϵͳģ֪ʶ.ֻ ANN бƽκηԹɵͬʱķһǺ÷Ҫļ㸴ǷԽ(Ч).ǲعģκηԹɣһ( n һ½ n ʱ eo)Ѹ֤ǲعõĲ(һЩΪǲعķʵһְ)ģ(Dimensionality)ģ33֤ǲعٶ n-r, r=(p-m)/(2p+d),pδ֪ f ƽ̶ȣm ҪƵ f ĵd ΪԿΪƽ n-(p/d)½.
 MFNûȷĹɣ˶߼(żУ)飬߼νʣάνʵĽѧϰʱάָ34.
4)ͳģʽʶͻعҪȳȡANNԶѹάMFN Ϊ˱任治˿ʼ()ѡرǵʱܵһҪʩѹԭʼάһANN35ҲʱҪר֪ʶ.
⻹ֵע⣺
1)⡡ͳģʽʶͻعҪѣ(Ҳ㹻ļԴЩ)򲻹ģͶิҲֹͳƽģеDimensionalityҲ.ģ36ݶԴ˷ANNۿɼģ3738.û֪ʶʱϢȫѵõ.ܴﵽЧ(ΪԴɿֻ).ϣѡ羡ܰйϢȡҪƥ䡱38.
2)ѡ⡡ǰָΪ֤ƽȶԣ°ģ(ά)ָɼʱ֤ƹЧǽάֻЩҪı.ѡľ巽ģʽʶͻعжУﲻ.
3)֪ʶá֪ʶпģ͵ĸԣԽкܴرٶֽϸʱӦ跨֪ʶ.Դģ39ۣŰѴͳ˹ܼ ANNһ̶ҲڴͳAI֪ʶԽANNѧϰĸ.
6
Ͽ˵ANN ķҲӶ ANN ӦоҲҪȰΪʲô ANN ȫΪδʵǲܸģֻΪ˸Чо ANN ӦṩһЩ˼.MFN һЩͳĵЧ˵ԭһЩȻڣǵоо ANN Ǻõģһԭ ANN ķչڣһЩ½϶ǿ ANN 봫ͳĲͬʹеһЩѣṹ(ģ)ѡȺ.һЩӦ(ȨԵĹ)ĴЩӦУģеδ֪(Ȩϵ).Ҳۣ40ݣӦ㹻ע.Ȼ MFN ȷԼص㣬ΪҪǣ
1)ϸӽϵͳ.Ϣıʾ봦ǷֲȨϣʵģͻкܴ룬书Ҳûдﵽǿǳⷽǰһ.
2)кܴԣͬһʽ MFNƽع顢滯ȵȣҸ.ĳҲ벻ͬʵԼҲɷϢϢһЩֲֿģʱл̬ϵͳȵ.ٽں.
3)ϵĲдṹϵĹԣӲʵ.
ʽĿʵǼƻ֪ѧ()شؼĿ
ߵλ廪ѧԶϵ100084
ο
1ųи.Եǰоļ㿴.ѧչ199426(2)181186.
2ųи.е.Զѧ199420(5)513521.
3Barron A R.Statistical properties of artificial neural networks.Proc.28th Conference on decision and control,1989,280-285.
4Albus.A new approach to manipulator control,the cerebellar model articulation controller(CMAC).J.on Dynamical Systems Measure and control,1975,220-227.
5Platt J.A resource-allocating network for function interpolation.Neural computation,1991,3:213-225.
6Barrett J F.The use of functional in the analysis of nonlinear physical system.J.Electro and control,1963,15:567-615.
7Chen Tianping et al.Approximation of continuous functions by NN with application to dynamic systems.IEEE Trans.NN,1993,4(6):910-918.
8Modha D S.et al.Multilayer functionals, in Mathamatical Approch to NN,Taylor J G ed.Amsterdam Elsevier Science Publishers,1993.235-260.
9White H.Artificial neural networks, approximation and learning theory.Cambirdge,MA.Blackwell.
10Asoh H.Nonlinear data analysis and MLP.Proc.of IJCNN-90,1990,411-415.
11Gish H.Apobabilistic approach to the understanding and training of NN classifier.Proc.ICASSP'90,1990,1361-1364.
12ݳƽ.ͶӰѰ١һµͳƷ.Ӧøͳƣ19863267276.
13Maechler M,et al.Projection pursuit learning networks for regression.Proc.Tools for AI,IEEE Press,1990,350-358.
14Hwang Jeng-Nang,et al.Regression modeling in backpropagation and projection pursuit learning.IEEE Trans.NN,1994,5(3):342-357.
15Hwang Jeng-Nang,et al.The cascade-correlation learning, A projection pursuit learning perspective.IEEE Trans.NN,,1996,7(2):278-288.
16Zhao Ying,et al.Implementing projection pursuit learning,IEEE Trans.NN, 362-392.
17Poggio T,et al.Networks for approximation and learning.Proc.IEEE,1990,78(9):1481-1496.
18Girosi F,et al.Regularization theory and neural network architectures.Neural computation,1995,7:219-296.
19Ogawa H.Neural network theory as inverse problem.J.EICE,Japan,1990,73:690-759.
20ƽ.ǰƹѧϰ.й 93 ѧļ1993꣬ϲᣬ6877.
21ƽ.ṹӦ.й 94ѧᱨ棬199410£人.
22Gemman S.Neural networks and the Bias/variance dilemma.Neural computation,1992,4:1-57.
23Garett A J.Ockham's Razor, in maximum entropy and bayes method,Grandy W T,et al.Ed.Kluwer Academic Publisher,1991.357-364.
24Nobora,et al.NIC-Determing the number of hidden unit for ANN.IEEE Trans.NN,1994,5(6):865-872.
25Wu lizhong,et al.A smoothing regularizor for feedfroward and recurrent neural networkds.IEEE Trans.NN.1996,8:461-489.
26ƽ.СǰеһЩ.ģʽʶ˹ܣ19936(2)143-148.
27Rohwer R,et al.Minimum description length,regularization and multimodel data.IEEE Trans.NN.1996,8:595-609.
28ݱ.ģʽʶ.廪ѧ磬1988.
29Sklansky J.ƽ.ģʽͿѵ.ѧ磬1987
30ź.ѧϰӦ(1)(2).ѧ1992191822.
31Anthony M,et al.Computational learning theory. Cambridge, England: Cambridge University Press,1992.
32Anthony M,et al.Computational learning theory for ANN.in Mathematical approcach to NN,J G Taylor Ed.1993,25-62.
33Stone C J.Optimal global rate of convergence for nonparametric regression.ANN.Stat,1982,10(4):1040-1053.
34Tesauro G,et al.Scaling relationships in BP learning.Complex system,1988,2:39-44.
35Mao Jianchang,et al.ANN for feature extraction and multivariate data projection.IEEE Trans.NN,1995,6(2):296-317.
36Randy S J,et al.Small sample size effects in statistical pattern recognition,recommendation for practice.IEEE Trans.PAMI-13,1991,252-264.
37ź.ѵ.Զѧ19931971-77.
38ƽ.˹ѧϰ.ѧ1995 23(4)63-67.
39Abu-Mostafa Y S.Hints.Neural computation,1995,7:639-671.
40Duin R.Superlearning and neural network magic.Pattern recognition letters,1994,15:215-217.
ƽ211.
ոڡ1995-08-07
