Զѧ
ACTA AUTOMATICA SINICA
1998 24 5  Vol.24  No.5 1998



Qѧϰ㷨BP
ڿ1)
ɡ
ժҪQѧϰWatkins1Ϣȫɷһǿѧϰ.Qѧϰ㷨BPЧϣʵ״̬δɢĵڵģѧϰ.÷ܳɹȷģ͵ƽƣҺAnderson2ݵAHC (Adaptive Heuristic Critic)ȷȣиõѧϰЧ.
ؼʡQѧϰBP磬ѧϰƣϵͳ˹.
LEARNING TO CONTROL AN INVERTED PENDULUM USING
Q-LEARNING AND NEURAL NETWORKS
JIANG GUOFEIWU CANGPU
(Department of Automatic Control, Beijing Institute of Technology, Beijing 100081)
AbstractQ-learning is a reinforcement learning method to solve Markovian decision problems with incomplete information. This paper presents a novel method to control an inverted pendulum with unquantized states by using Q-learning and neural networks. Simulation results are included to show that the new method can not only balance the determined or stochastic inverted pendulums successfully but also lead to a better effect of learning when compared with Anderson's AHC method.
Key wordsQ-Learning, BP neural network, learning control, inverted pendulum, Gaussian noise.
1
ڸַϵͳУһʮֵ͵.ǿѧϰʵֵڵƽƣѾȡ˲ٳɹ.1983Bartoˣ3磬AHC(Adaptive Heuristic Critic)ѧϰ㷨ʵ״̬ɢĵڿ.1989꣬Anderson2ݽһ˫AHCʵ״̬δɢĵڵƽ.Peng4ͨ״̬ɢΪ162LookupʾQֵķʵ˻Qѧϰ㷨ĵڵƽ.ȻЩ״̬УɢЩLookupʾķQѧϰ㷨ͳ涯̬滮һ״̬Ŀռ临⣬νά.֮һòĽṹʾQֵͽ׶ʽ.ͨѵBPƽQֵBPķʵ˻Qѧϰ㷨״̬δɢȷڵģѧϰ.ĵĿõڿ֤ʵQѧϰϵķȥʵĳЩ״̬ϵͳģͿƵĿ.
2Qѧϰ
ڽQѧϰǰȼϾģͣÿʱ䲽k=1,2,,۲Ϲ̵״̬ΪxkѡakյʱrkʹϵͳתƵһ״̬ykתƸΪPxkyk(ak).ƵĿѰһſƲԣʹδÿʱ䲽񱨳ۿۺ͵󣬼󻯣0Ʀ<1Ϊۿ.
һԦУQֵΪ
(1)

֮Qֵ״̬xִпaԦеıۿۺ͵.QѧϰĿľתƸʺ񱨳δ֪ŲԵQֵ.ΪQ*(x,a)Q*(x,a),x,a.Ц*ʾŲ.
Qѧϰʵ£ÿʱ䲽k۲쵱ǰ״̬xkѡִпakٹ۲״̬ykܼʱrkȻʽQk-1ֵ
(2)
ЦkΪѧϰӣ
Watkins1֤ѧϰУkһ£һ(x,a)Ԫõʽ(2)εkʱQk(x,a)Ը1Q*(x,a).
3ϵͳ
ϵͳָͼ1ʾķϵͳ.Сɵ޶Ĺƶ.Сϵĵһ˱С.һ˿СڵĴֱƽת.ƵĿͨƶСƶʹƽⲢС͹ײ.


ͼ1ϵͳ
һ£ϵͳĸ״̬
xСڹϵλãȣƫ봹ֱĽǶȣ
С˶ٶȣڵĽٶ.
ϵͳ˶
(3)
(4)
g=9.8m/s2ٶȣmc=1.0kgСm=0.1kgl=0.5mڵһ볤ȣc=0.000 5С͹Ħϵp=0.000 002ںСĦϵFt=10.0NʱtСĵ.СΪ4.8.ͨEulerֵƣ²ַ浹ϵͳ
x(t+1)=x(t)+(t),(5)
(t+1)=(t)+(t),(6)
(t+1)=(t)+(t),(7)
(t+1)=(t)+(t).(8)
ʱ䲽һΪ0.02.Ȼϸĵϵͳһȷϵͳ.Ϊ˵QѧϰķͬϵͳģͿƣȷԵģһźһģͣڷ·淽(6).
(t+1)=(t)+(t)+(,2),(9)
Ц(,2)Ϊ˹.
4QѧϰBPĵڿ
ʵֵڿƵķͬǿѧϰУΨһܴӻõķǵƫ봹ֱĽǶȳ12Сڡ2.4״͹ײʱһʧź.˱Ķ弴ʱrtΪ

ڿִһϵоߺŵõӳٵʧźţͷʱ⣬ȷЩӦöʧܸ.ʵQѧϰ㷨ڸʱ䲽QֵĸµнʧźŽзQֵȷӦߵ.赹ƽʧʱļʱΪ(rt=-1)˶ӦQֵСľ߾͸пܵµϵͳƽʧ.ͬʱʵ״̬δɢĵڿʱ漰״̬ռܴʱQֵķ(Ҳнͷṹ).
ĻQѧϰBP״̬δɢڿϵͳĽṹͼ2ʾ.Ϊͼж״̬X=(x,,,)T.BPΪ״̬X;aΪQ(X,a,W)WΪȨ.ϵͳ£ÿʱ䲽k۲⵹ڵĵǰ״̬ΪXkBPʵQ(Xk,a,Wk)ֵĳ̽ѡǰak.Ȼ۲⵹ڵĺ״̬Yk+1Ƿʧź(ȷʱrk).ϵͳٸ(10)ʽ¶Ԫ(Xk,ak)QֵȻźe=Q(Xk,ak)-Q(Xk,ak,Wk)BPȨWkΪWk+1ʹBPʵƽºQ(Xk,ak)Ȼת״̬Yk+1ϵĹ.δ״̬ռɢϵͳBPķδѵ״̬-߶ԪQֵ.⣬BPȨѧϰУһ״̬;ӦQֵбƽܻӰ״̬һӦQֵͼ2ʾBPʵϿBP(ÿһ)õõѧϰЧ
.(10)


ͼ2QѧϰBP״̬δɢڿϵͳĽṹͼ
5漰
ϽʵʷУBP.ÿ㣬㣬һ.BPʵ˱׼ʹֲڣ-1.01.0֮.Qѧϰ㷨ѧϰӦ=0.2ۿӦ=0.95.ģ͵ķУ˹(,2)ľֵ=0׼=0.1(һ(t)ֵ(-1.51.5)֮).ڵڿϵͳппֻ(ƻ)˱ֱѡӦQֵϴĿΪǰ.ӺBPʼȨΪֵͬȷԺģ͸ʮ飬ÿ鵱ڵ̽(ʧܴ)100 00λһ̽ƽⲽ500 000ʱֹڵѧϰ¿ʼһ.ڷУһܱ̽500 000ΪѾܳɹƵƽ.״̬ɢĵڿУÿƽʧܺ󣬵ڵĳʼ״̬һX=0λ.ڷУΪ˱֤ڵõڸֳϵĿƾ飬ÿƽʧܺ󣬽ʼ״̬λΪһΧڵֵ.ƽʮĽõQѧϰBP״̬δɢڿƵĽͼ3ʾ.


ͼ3ʵ״̬δɢڿƵѧϰ
ͬ£QѧϰAHC(ѧϰ)ʵ״̬δɢȷԵģͿƵĽбȽϣֻQѧϰBPķѧϰЧãÿƽ1 000ʧܺͿԳɹƵƽ.Anderson2ݵAHCԼҪ6 000.ƷܿƵƽ⣬ÿֻ̽мٲ.QѧϰBPķͬʵģ͵ƽƣѧϰѶȣÿƽҪ2 600ʧܺܿƵƽ.
Ҫ˵ǣܻQѧϰBPķȡýϺÿЧ߸עַ֤ʵĳЩ״̬ϵͳģͿƵĿ.ʵϣڿֻһ.
1)ȻѧصĿ.
ߵλ(ѧԶϵ100081)
ο
1Watkins C J C H. Learning from delayed rewardsPh.D.Dissertation.UK:King's College,1989
2Anderson C W. Learning to control an inverted pendulum using nerual networks. IEEE Control System Magazine, 1989, 9(3):3137
3Barto A G, Sutton R S, Anderson C W. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Trans. on SMC, 1983, 13(5):834846
4Peng J. Efficient dynamic programming-based learning for controlPh.D.thesis.USA:Northeastern University,1993
ոڡ1997-01-22
