ѧ
Journal of Software
2000 Vol.11 No.1 P.54-61



ڱ۵Ķ̬ʶ
Զ¡vԡ
ժҪһֻڱ۵Ķ̬ʶ.ͼ˶ıײģͺ³ع,һֻ˶ָͼ˶Ʒ.ͼ˶,ֱ۱仯ģͷֱΪƵı,СŻ㷨Ʋοģ,ûģķ༼ʶ.120Ĵʵ,ֶ̬ʶʶʸߡС㷨ȶԺõŵ.
ؼʡӾ,˻ӿ,ʶ,ͼ˶ģ,³ع,̬滮ƥ.
ͼšTP391
Appearance-Based Dynamic Hand Gesture Recognition from Image Sequences with Complex Background
ZHU Yuan-xinXU Guang-youHUANG Yu
(Department of Computer Science and TechnologyTsinghua UniversityBeijing100084)
AbstractIn this paper, the authors present an appearance-based approach to dynamic hand gesture recognition. A motion-based segmentation scheme for image motion estimation is proposed using variable-order parameterized models of image motion and robust regression. Based on image motion parameters, two different appearance change models of hand gestures are created. Template-Based classification technique is then employed to perform hand gesture recognition in which reference templates are created with a mini-max type of optimization. A series of experiments on 120 image sequences show that high recognition rate, low computation load, and high stability can be achieved with the proposed methods.
Key wordsComputer vision, human computer interface, hand gesture recognition, image motion model, robust regression, dynamic programming matching.
ʶΪһоȵѾмʮʷ,ֱ,ڼӾʶоԱĹעȤ.ʶоҪĿľǰּȻֱ۵Ľʽ˻ӿ(human computer interface),ʵָΪϰߵ˻ӿڣ1.,ʶ𻹿ʵάơٳСӻҽѧо.ʶĽڱʶ𡢴̬ʶʱࡢӾͼƴӺͻݵƵоֱƹ.
еʶ𷽷ԷΪࣺά/ֱ۽ģķͻڱ۽ģķ2.ƽģ,õķǸָֻάģͣ3,,صļʹ෽ǳ,ʵӦҲңԶ.ڱ۵ķҪоֱͼı۱仯ʶ,ص㲻ֱֻ۵ľֹάṹ,˶ͼı۱仯.,Ѿ˿ʼоڱ۵ʶ46.ף4һֵ(ͨһάռͼάʱͼõ)ɵΪƵı.Ȼ,жûߵûƲùȥʱģʽ׼ȷʱ,ֱʶ𷽷ͻʧЧ,ǵķȱͨ.ף5ͨȡͼ˶߽硢ٱ߽˶ʶ,ûиʶ,ҲûƵ״Ϣ.
һֻڱ۵ʶ.,ͼ˶ıײģͺ³ع,һֻ˶ָ֡ͼ˶Ʒ.,֡ͼ˶,Ĵֲͬı۱仯ģ͡ʱձʾͼ˶ʾ,ֱʶ,ʵԶߵܽбȽϷ.Ȼ,ָһΰ˶״ɫϢͳһʶ;.,Ʋʵ˶12ƽʶʵϵͳ,ʶʳ90.
1 ˶ָ֡ͼ˶
1.1ͼ˶ģ
ڻӾĹʶϵͳ,ûʱһϵͳ,ǲؿ˶ƹдڵάڵ.ͼ˶ģͶһͼ˶ȷ貢ͨһͽ׶ʽʾ˶7.õĲͼ˶ģƽģ͡ģͺƽģ.,ʽ(1)ľͼ˶ƽģ.
(1)
aiǳ(һ˶),u(x)صx=(x,y)֡λ,u(x,y)v(x,y)ֱˮƽʹֱ.Ϊ˱д,Ƕ
(2)
T,APֱʾͼ˶ƽơԼƽģ͵Ĳ.ͼ˶,ai(i0,1,...,7)иԵ,ЩǵϷֱʾƽ˶(u)ֱ˶(v)ͬԵ(e)α(d),Լƹ۲췽ת(r)ƫת(y)(p).ʽ(3)ʾ.
u=a0,v=a3,e=a1+a5,d=a1-a5,r=-a2+a4,y=a6,p=a7.(3)
1.2³ع
ͼ˶ģ֮,ǲ³عȥعģͲ8.³ͳѧҪĿǻعõϾݵģ,ͬʱعġڵ㡱(ģ͵ݵ)͡㡱(ģͳϴݵ).
Ƿصļ,ģͲ,ôеȺ㶨ͿɱΪ
(4)
,IȺ,tʾʱ.ʽ(4)ұ̩չ,ȥһε,õʽ(5).
(5)
,I=Ix,IyTͼݶ;Ix,IyItֱͼȹڿռάʱƫ.ΪƲ,ҪĳСʽ(6)еĿ꺯.
(6)
Цһ߶Ȳ.ΪּȲƽҲǸ,С˷,ôƳ˶Ȼǲ׼ȷ,ĳһ.,ǲGeman-McClure,ʽ(7)ʾ,ʽвв
7
ض϶κ,Geman-McClureڵ֮ṩ˸ƽĹ.
ôԵĳɳ㷨ʹʽ(6)Ŀ꺯ﵽС.˵,Ŀ꺯Eaiĵ,ôΪСE,n1ĵ·̾
(8)
0أ2ǳɳڲ,ڵn1an1iĹƽС֡.0أ2ʱ,֤÷,ʶԦصľȡֵ.TaiEĶƫϽ,
(9)
ڵʹ,ÿһε,ݦn1095nС߶Ȳҵֵ.Ľ,ʼʱڵݵ㶼ڵ,ŵĽпʼ,Ӱ𽥱С.
1.3ֱʲ
Ϊ׼ȷع֡˶,ֱʲԣ7.ȹ˹,Ȼһռֱʿʼ,ʼ˶0Ϊ,Ƴ˶0.ѵõ˶00Ͷ䵽һֱʼ͵óһֱʼĳʼ˶1,ݦ1Ѹütʱ̵ͼt1ʱ̵ͼ׼,Ȼ׼ͼt1ʱ̵ͼƸü˶1.ظ,ֱƳ߷ֱʼ˶.
1.4˶ָ֡˶
֪,߶Ȳ³عзҪ.ף8ָ,ѡңCΪҵĳʼֵһ͹Ż.ض³,Cһ,,Geman-McClureʱ,C=.в.Ȼ,ھʵʱ,ǲûô,Ҳûи,ָֻڦһڵı仯,õĽ൱ȶ.ǿ,ɴʹǷCΪҵĳʼֵ,ʼвֲ޷Ԥ,˵óĳʼܺ⣻,ѡCΪҵĳʼֵҪĵ,ĸʱ(ϴ,ͼʱ).ĸһķȷҵĳʼȡֵ.
Ļ˶ָͼ˶Ʒ,ÿһһ³ع.
1Ŀǰͼ˶()Ӹӱָ.Ǽ豳ֹֻ(ʶϵͳ,üھǳ),ȻѡͼΪ,ѡƽ˶ģͽе1³ع.ΪԶȷҵĳʼȡֵиЧʵĵ,ѡCTΪҵĳʼֵ,,Cͬ,TǸף9ֵѡ㷨֡ͼĲͼĻҶֱͼһֵ,ǰѲͼĻҶȷΪֵ.,ԴһʼͰѾ˶()Ϊ㴦,Чʸ,.Ϊݵ㶼Ǳص,ԹƳ˶(˶ǳС)Ǳ˶.ݻع˶ԼҵֵԲвз(),ͿԵõ˶Ĵֲڷָ(лһЩ).
2Ŀʵ׼ȷ֡˶,ҵõ˶ľϸָ.ʹöάģͻƽģͼƽ˶,˶ȱ仯ȺСʱ,ֽǺ.ǰѵ1õĴֲڷָΪµķ,ѡ÷ƽģͽе2³ع.ڷС,Ҿݵ㶼˶,ǰѦΪҵĳʼֵ(ӵͬ,в).Ȼݵ㶼˶,Իع֡˶.Ȼ,ͨв(ڵ)ͿԵõ˶ľϸָ.
2 Ƶı
ʶϵͳ̶ܴܺȡĳȡ빹.֡˶,Ĺֲͬı۱仯ģ(ʱձʾͼ˶ʾ),ֱΪƵıʶ.
2.1Ƶʱձʾ
һƵͼ,gTI1,I2,...,IT},Tʾеʱ䳤,Itt1,2,...,T)ĵt֡ͼ.ѵt-1֡͵t֮֡ͼ˶(ƽ)ΪɵĲΪPt,ôӦƵʱձʾPTͿԱģΪ
10
2.2Ƶͼ˶ʾ
ǰ,ͼ˶ģ͵Ĳaii0,1,2,...,7)ȷ,ǿԶһȷ˶֡ͼ˶.I(t-1)I(t)֮˶Ϊmt,mtݵĶΪ
(11)
Ϊͬûʱڵʲ,Ϊͼ(gT)һ۱仯ģͣͼ˶ʾ.ʽ(12)ͼ˶ʾĶ.
12
Mеͼ˶ʾ,ǰ(ÿ)֡˶ķȨۼӺһԺõǼȨӡ.ʾڻ㣻U,V,E,D,RPֱʾͼ˶ʾˮƽλơֱλơͬ͡Ρƹ۲췽תƫתԼȷ.
3 ʶ
3.1̬滮ƥODPM
ͬûʱڵʲʱձʾʱԲ,ЩԲʶеһҪ.ʶʵ,κԱ任ӱ˵ܺܺõش߶ȸӵķԲ.ʶõŶ̬滮ƥ(optimal dynamic programming matching,ODPM)10ʱձʾ֮С.̬滮ƥǾзʱһЧģʽƥ㷨.ʹĳָԵķԹʱϵĲƽģ,ͨһģʽʱʹ֮һģʽﵽ̶ȵص(ʱĲвС),Ӷʱձʾģʽ֮ʱ.ģʽ֮СĲв֮ʱһľ(ĳΪODPM).ö̬滮ԸЧʵʵС.
3.2ƵĲοģ
ѵLֹ,ÿJ.ѵеÿ,ǷֱʱձʾPTl,jݺͼ˶ʾMl,j,l(l1,2,...,L)ţj(j=1,2,...,J)ţTʾlƵĵjʱձʾĲĸ.ƵĲοģǴÿƵJʱձʾJͼ˶ʾ,ҳܷӦƱֲһʱձʾһͼ˶ʾֱΪƵʱձʾģͼ˶ʾģ.
ѵlƵʱձʾģΪl,ôʱձʾģתΪѡȡʹlڸѵĳű׼.Ƶʱձʾ֮ODPM,СŻ㷨11Ϊÿƴһʱձʾģ.
һͼ˶ʾӳ˹ᴩ(Ƶ)ͼеĸͼ˶,,ǰѵlƵͼ˶ʾģ嶨ΪƵJͼ˶ʾƽֵl,
(13)
3.3ģʶ㷨
ˣlݺͣl(l=1,2,...,L),ģ༼ʶ.ȡƵʱձʾģΪʶʱĲοģ,ô,һδ֪ʱ,ϵͳȼʱձʾ,Ȼʱձʾиʱձʾģ֮ODPM,ҳС.С벻ĳԤõʱ,ƾǸʱձʾģӦͬһ,ʶ.
ѡȡƵͼ˶ʾģΪʶʱĲοģ,ôʶ̻,ֻҪδ֪Ƶͼ˶ʾ,ʹŷϿռODPM.ʵʱ,ѵÿһ,Ǽȼʱձʾģ,Ҳͼ˶ʾģ,ҶԷֱģΪοģõʶбȽϷ.
4 ˶ƾ
ͼ1֡˶еһʵ,ͼ1(a)(b)ֱһϡƵͼĵ2֡͵3֡ͼ.ѡƽģͶͼе1³عõ˶ǣ0.06184,0,0,0.1446,0,0,0,0.ͼ1(c)еĻɫصǸݵ1λعڵ,ֹĺ΢СŶıԼĳЩ.еĺɫصǸݵ1λع,ȻһЩ˶ϴ,ǵļϾ͹˶Ĵֲڷָ.ĴֲڷָΪµķ,ѡƽģͽе2³عó˶ǣ0.3644,0.004,0.0066,2.2835,0.0012,0.0086,0,0.ݵ2λعڵ͹˶ľϸָ,ͼ1(d)ʾ,ͼ1(e)ʾ,һЩ.ͼ1(f)ǰͼ1(a)ͼ1(b)֮ĲͼֵԺõĽͼ.Ƚͼ1(d)(f),ѿ,ͼ1(f)кܶ,ͷ㡢ֱ,ͼ1(d)û.ͼ1,Ļ³ع˶Ʒ׼ȷع֡ͼ˶,ͬʱõ˶ľϸָ.



(a)(b)(c)



(d)(e)(f)
Fig.1An instance of the motion estimation between frames
ͼ1֡˶Ƶһʵ
ͼ2֡˶Ƶһʵ.ͼ2(a)(b)ֱһҡƵͼĵ2֡͵3֡ͼ.1λعĲֵΪ0.1217,0,0,0.0042,0,0,0,0,2λعĲֵΪ2.7344,0.0008,0.01120,0.1343,0.0037,0.0160,0,0.0003.ͼ2(c)(f)ʾĺֱͼ1жӦͼĺͬ.



(a)(b)(c)



(d)(e)(f)
Fig.2Another instance of the motion estimation between frames
ͼ2֡˶Ƶһʵ
5 ʶ
һʵάĹ,ӶʵָȻΪϰߵ˻ӿ.,ǵʵϵͳѡõ12Ʒֱǡϡ(MU)¡(MD) (ML)ҡ(MR)ǰ(MF)󡱡ƫת(YL)ƫת(YR)˳ʱת(CC)ʱת(CCC)¸(PD)͡(PU).
˴ʵ֤۱ʱձʾģ͡ͼ˶ʾģԼѵʶ㷨Ч.10λʵǰ12,ǵõ120ͼ(ÿ10)ѵ.ÿаһ,ʱԼ1s(Ϊ10Hz).ͼ256ĻҶͼ,С160120.PII(266MHz)PC,֡˶ƵƽʱԼ450s,ʶһƵʱԼ4s.
ͼ3֡˶Ĺ켣ʽ˴4ͼмʱձʾ.ÿͼа1,4аƷֱMU,MD,MLMR.a0,a1,a2,a3,a4a5Ĺ켣(a6,a7ֵС,ʺ)ǵ㻮,䲻֮ͬ,ͬ켣ĵ㻮ÿݵõķŲͬ,ֱǡ*󡱡͡.ͼ4Կ,ʱձʾǸ߶ȿֵ,˵ʱձʾɢԴ.ͼ(a),(b),(c)(d)ֱӦMU,MD,MLMRƵĳʱձʾ.


(a)(b)


(c)(d)
Fig.3Spatial-Temperal representation of hand gesture described by 
trajectory of motion parameters between frames
ͼ3֡˶Ĺ켣ʽʱձʾ
ѡȡʱձʾΪƱʶ,ѵʵֵʶʼ1.2ͼ˶ʾΪƱʱѵϵõʶ.Ƚϱ1ͱ2,ѿ,ͼ˶ʾΪƱʱ,ʶʸ.֮,ΪƱʱʶٶȸ.3ͼ˶ʾΪƱʱڲԼ(ÿ10)ϵõʶ.
Table 1Recognition rates achieved on training set with spatio-temporal
representations as features of hand gestures
1ʱձʾΪƱʱѵϵõʶ

Hand gesturesMUMDMLMRMFMNYLYRCCCCCPDPUAverage
Recognition rates()9090901008080100809090809088.3

,ʶ,ƽ.
Table 2Recognition rates achieved on training set with overall image
motion representations as features of hand gestures
2ͼ˶ʾΪƱѵϵõʶ

Hand gesturesMUMDMLMRMFMNYLYRCCCCCPDPUAverage
Recognition rates()901001001009080901009010010010095

,ʶ,ƽ.
Table 3Recognition rates achieved on testing set with oveall image
motion representations as features of hand gestures
3ͼ˶ʾΪƱڲԼϵõʶ

Hand gesturesMUMDMLMRMFMNYLYRCCCCCPDPUAverage
Recognition rates()1001001001009080809090100708090

,ʶ,ƽ.
6 ۺչ
ͼ˶ıײģͺ³ع,һֻ˶ָԵ֡˶Ʒ.ڷصܶ(Լ),ҪƵĲֺ(8),Բͼ˶ģͶԷͼ˶ṩ˳Լ,ֵȶԸ.ѡײģ(ƽơƽ)ʹüС,Чʸ.,³ع˶ʱ,λ˶߽Ĵ㡱,ǶԽӰ챻.
֡˶,Ĵʱձʾͼ˶ʾֱ۱仯ģΪƵı,ֱʶ.ʵ,ģΪƵıõʶʶ88,ʹúʱʶʸ,ʶٶȸ.
ȻΪ˶ġƷ,ǵõ˶ľϸָ,ǽԷָ˶״ɫԼз,Ա㴴ı۱仯ģ,ӶһƵʶ.ҪӦ˻ӿ,оԶԶָ.
оõ863߿ƼĿ(No.863-306-03-01).
Զ£廪ѧѧ뼼ϵ100084
v廪ѧѧ뼼ϵ100084
ԡ廪ѧѧ뼼ϵ100084
E-mail: yuanxinzhu@263.net
ο
1Zhu Yuan-xin, Xu Guang-you, Yu Zhi-he. Vision-Based hand gesture interpretation: a survey. In: Wu Quan-yuan, Zhang Yue-liang eds. Proceedings of the 3rd National Conference on Artificial Interface and Artificial Application. Beijing: Publishing House of Electronics Industry, 1997. 279284
(ףԶ,v,־.ڼӾƽͷо.ȪԴ,Ծ.ܽӿӦ½չ2ȫܽӿӦѧļ.ӹҵ,1997.279284)
2Pavlovic V, Sharma R, Huang T S. Visual interpretation of hand gestures for human-computer interaction: a review. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1997,19(7):677695
3Heap T, Hogg D. Towards 3D hand tracking using a deformable model. In: Irfan Essa ed. Proceedings of the 2nd International Conference on Automatic Face and Gesture Recognition. Los Alamitors, CA: IEEE Computer Society Press, 1996. 1401452
4Darrell T J, Essa I A, Pentland A P. Task-Specific gesture analysis in real-time using interpolated views. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1996,16(12):12361242
5Quek F. Eyes in the interface. Image and Vision Computing, 1995,13(6):511525
6Zhu Yuan-xin, Huang Yu, Xu Guang-you et al. Motion-Based segmentation scheme to feature extraction of hand gestures. In: Zhou J, Jain A K, Zhang Tian-xu et al eds. Proceedings of SPIE, Vol 3545. Washington, DC: SPIE, 1998.228231
7Bergen J R, Keith P, Hanna J et al. Hierarchical model-based motion estimation. In: Sandini G ed. Proceedings of the 2nd European Conference on Computer Vision. Berlin: Springer-Verlag, 1992.237252
8Black M J, Anandan P. The robust estimation of multiple motions: parametric and piecewise-smooth flow fields. Computer Vision and Image Understanding, 1996,63(1):75104
9Otsu N. A threshold selection method from gray-level histogram. IEEE Transactions on Systems, Man and Cybernetics, 1979,9(1):6266
10Sakoe H, Chiba S. Dynamic programming optimization for spoken word interpretation. IEEE Transactions on Acoustics, Speech, and Signal Processing, 1978,26(1):4349
11Rabiner L R. On creating reference templates for speaker independent interpretation of isolated words. IEEE Transactions on Acoustics, Speech, and Signal Processing, 1978,26(1):3441
1998-11-10ո
1999-02-01޸
