ѧ
JOURNAL OF SOFTWARE
2000Vol.11No.4P.473-480




ͳƵĺԱעķĽ
κŷ⽡
ժҪӴԸʾʻʾĽṹֵ仯ȷ,ĿǰõĻͳƵĺԱעѵϹģעȷ֮ڵķԹϵ˷.Ϊ˳ѵϿ,߱עȷ,ôص﷨Ժͼǿδ֪ʵĴԸĽ,˱ע.ղԺͿŲԵȷʷֱﵽ96.5%96%.
ؼʡԱע,nԪ﷨,,﷨.
ͼšTP18
Analysis and Improvement of Statistics-Based Chinese Part-of-Speech Tagging
WEI OuWU JianSUN Yu-fang
Institute of SoftwareThe Chinese Academy of SciencesBeijing100080
AbstractIn this paper, a popular statisticsbased training and tagging method for Chinese texts is studied, and the nonlinear relation between training set and tagging accuracy is analyzed from the aspects of the structure and numerical value of the matrix of transition probabilities and the matrix of symbol probabilities. In order to make use of training corpus sufficiently and get the higher tagging accuracy, the training and tagging method is improved from two aspects: using other grammatical attributes of words, and strengthening the processing of unknown words. With the improved method, open test and close test showed that the overall accuracies are about 96.5% and 96% respectively.
Key wordsPart-of-Speech tagging, n-gram, corpus, grammatical attribute.
 Աעþͨȡʵķ,ĵﾳϵ,дʵ﷨,ʹһʼмִ,ضĳֻʵһ.ԱעӦжһҪʵ,ȻԴҲһܻĿ,ڴԶעо۾Ҫ.,żķչ,ϿĲ,ͳƵȻԴ𽥳ΪĿǰѧеһоȵ.ڻͳƵĺԱע,ڵоԱҲ˺ܶ̽1.
ȸĿǰõƵѵȡԪ﷨ģ͵Ĳ,Viterbi㷨ԺԱעʵ,ȻӴԸʾʻʾĽṹֵ仯ȷѵϹģעȷ֮ڵķԹϵ˷.Ϊ˳ѵ,߱עȷ,Ĵôص﷨Ժͼǿδ֪ʵĴԸĽ,˱ע.ղԺͿŲԵȷʷֱﵽ96.5%96%.
1Աעͳѧģ
1.1n-Ԫ(n-gram)﷨ģ
WǴʻ㼯,TǴԱǼ,һĴʴS=S1,S2,...,St,...,SM,κһStW.ҪһĲ,SпܵıǴҳʺϸضʴһCS=C1C2...Ct...CM,CtT.
P(CS|S)ΪڸʴSǴCSĺ.ݱҶ˹ʽ,

P(CS)ǱǴCS,P(S|CS)ڱǴCS֪´ʴS,P(S)ǴʴSķ.ԱעþҪҵıǴCS,ʹ
Ϊ˼ٲռĹģ,ԼStĳֻĴCt,ǰt-1޹.,ԼֲϢCtĳ㹻,ΪCtĳֻŵtʵǰĺٵ(n-1)(n>1)ʵĴ.ģͳΪnԪ﷨ģ.ʵһn-1׵ɷ,ȡn=2,ʱõľǶԪ﷨ģ(bi-gram),ڶԪ﷨ģ,

P(Ct|Ct-1)ֻڴʵĴй,ǳΪԸʲP(St|Ct),ʱ,ǳƴΪʻʲ.
ڴԱǼT,ʻ㼯W,TйNT,WйNWʻ,ôеĴԸʲһNTNTĶάANTNT,һԪaij(1i,jNT)ʾӴԱTiTjתƸP(Tj|Ti)еĴʻʲһNTNWĶάANTNT,һԪbjk(1jNT,1kNW)ʾڳִԱTjʱʻWkĸP(Wk|Tj).
1.2Ļȡͱע
ѾעõĺϿ,ԲñΪƵѵ(relative frequencyRF)ķȡԸʺʹʻʲ2,

N(Ti,Tj)ѵдԱTjTiֵĴ,N(Ti)ǱTiֵĴ,N(Wk,Tj)ѵдʻWkĴԱΪTjĴ,N(Tj)ǱTjֵĴ.
Ϊ˽ѵϡ,ԲóԼвƽ3,ԸʲпܲΪ0,ѵϲΪ0Ĳֵ,һСĳֵ.ANTNT,A=min(1/Nr,c/Nrsw)BNTNW,B=min(1/NW,c/Nrsw)Nrswѵϵܴ,c=0.1,ЩֵƳɱЩѵֻһε¼ĸʴԼС10ֵ.
ΪһĴʴS=S1S2...St...SMѰCSĹҲǴԱעĹ.ʵʵıעϵͳ,һѡS1SMΪΨһĴʻ֮ĴΪעλ.ڶÿʣStNtܵĴ,ˣS1SMпܵı·γһͼ.
ͨԱ·ͼĽṹͶԪ﷨ģ͵صķ,Ŀǰһûڶ̬滮Viterbi㷨űǴѡ,˼ǰѽΪѽ.֪S1ĴA2,ӦıΪn1,SMĴΪTSM,ӦıΪnM,fait,j(1tM,1jNT)ʾS1TS1StĴԱΪTj·ĸȨֵ,pesait,j(1tM,1jNT)¼·St-1ѡĴԱֵ,C1,Mݱѡѱ·ֵ.ڶԪ﷨ģ͵Viterbi㷨£
(1) ʼ,j1NT,
(a) jn1,fai1,j=1,fai1,j=0;
(b) pesai1,j=0;
(2) t2(M-1)ת(3);
(3) k1NT,:
(a) 
(b) 
(4)
(5) t(M-2)2,·ϵĴԱ,Ct=pesait,Ct+1ݣ.
2Աעʵ
ѡȡԱǼʱ,ף4жԴķΪ,˰26,82,25㡢,ܹ107ǵĴʱǼ.ʹõѵ廪ѧϿеһ,аá¡šѧ5,Լ30ʴ.ЩѾֹעӹ.ݲͬ,ѡȡ,4ʵĿŲ,Լ20ʡ15ʡ10ʡ7ʡ5ʺ3ʵѵղ.ף4մΪõĴԱǼ,ͬԹ,˷ʴʵ,Լ5.ʹ÷ʴʵҪŵ㣺(1) ݴʵԶѵδֵĴĴͨƽӦĴʻֵ,Ӷ뵥ѵ϶Ĵʻʾ,ϵͳıעܣ(2) ף4ѡʹ淶,,Խδ֪ʵĳƵ.
,ʵһĿǰõĻƵѵViterbi㷨ĴԶעģʽRF-Basic,ӲͬģѵϳԺԱעʵ,õķղԺͿŲԵĽ1.
Table 1Tagging results using different training sets
1ͬѵģµıע

Training set
(10 thousand)Tagging accuracy
of open testTagging accuracy
of close test (%)Tagging accuracy of unknown
words on open test (%)Tagging accuracy of unknown
words on close test (%)
393.796.542.358.0
594.596.243.652.5
794.696.047.552.0
1094.996.150.453.6
1595.195.952.250.7
2095.295.853.452.2

ѵϹģ(),ڿŲԱעȷ,۷ղԱעȷ,
ܿŲδ֪ʵıעȷ,ݷղδ֪ʵıעȷ.
,شӿŲԵĽ,һѵעȷʵĹϵ.ͼ1Կ,ѵϵĹģȷʵ߲Թϵ,ܵ˵,ѵϵĹģԽ,õĸʲԽӽʵ,עȷҲ.,ѵϹģﵽһ̶Ⱥ,עȷʵӷԽԽС,ϵͳܸҲԽԽ,ﵽ״̬.


Fig.1The relation between training test and tagging errors
ͼ1ѵϹģעʵĹϵ
2.1ѵԸʾANTNTĹϵ
첻ͬѵеֱͬǶԳֵ,2.
nԪ﷨ģ,ѵϵĴСӦʹƽÿֱܵͬǶѵٳ10.ҲΪ,ѵϵĴСӦʹƽÿѵʵʳֵͬԪѵлõۼƴΪ10.ʵ,ΪѵϵĴСֱͬǶ֮Ĺϵɱעѡѵݵɵȶ෽ؾ,ƬشһǶǲǡ,Ӧݾж.ǵϵͳ,ӱ2пԿ,ѵϵ,ֱͬǳֵĸ𽥱,ƽÿ1,³ֱֵͬǶԵĸʹ127/ٵ40/.,Ϲģ,½ڼС,Ҳ˵,ֱͬǶ𽥴ﵽȶ.
Table 2The number of same tagging pair in different training corpora
2ͬѵдֱͬǶԵĸ

Training
corpora set
(10 thousand)Numbers of
part-of-speech
same agging paiNew adding numbers of same
tagging pair per 10,000 words
between contiguous corporaThe average appearance number of
new adding same tagging pair
between contiguous corpora
31 514
51 7681273.11
71 9871102.40
102 141512.31
152 403521.45
202 063401.21

ѵϹģ(),ڳֵĴֱͬǶԵĸ,ѵ֮ƽÿֱͬǶԸ,ѵֱ֮ͬǶƽÿֵĴ.
ЩӵֱͬǶֵĴѵϹģĹϵ.ӱ2пԿ,35ӵ254Ƕ,ֵܴΪ791,ƽÿһǶԳ3.11;1525ӵ200Ƕ,ܹ242,ƽÿǶԳ1.21.Ҳ˵,ѵϹģ,ӵֱͬǶԲڸ𽥼,ѵгֵĴҲ𽥼,ռıԽԽС.,ǿԵóۣѵϹģ,ֱͬǶԵȶ,ԸʾĽṹȶ.
һANTNTֵϵı仯,A1,A2,

ʾA1A2ľ仯.

ѵϴ320ӦĴԸʾı仯3.
Table 3The variety of part-of-speech probability matrix in different training corpora
3ͬѵ´Ըʾı仯

ANTNTA2A1
A3WA5W0.66
A5WA7W0.66
A7WA10W0.53 
A10WA15W0.32
A15WA20W0.23

ѵϴ35ı仯Ϊ66%,1520ı仯Ϊ23%,仯½65%֮.Ҳ˵,ԸʾֵҲȶ.
ɴ˿Կ,ѵϹģСʱ,Ըʾӽṹֵ仯Ƚϴ,ѵϴﵽһģ,Ըʾȶ,ѵϵӰԽԽС. 
2.2ѵʻʾBNTNTĹϵ
ѵϵӶԴʻʾBNTNTӰ.
ѵϵ,гֵĴ.ӵĴ,һǴʵѰ֪,һδ֪.Ƿֱǽͳ,4.δ֪(UNKNOWNWORD),Ϊһʽд.ڴʵѰĴ,Կ,ֱͬǶԵĳ,ѵϴﵽһģ,ӵĴķԽԽС,ѵеĴĳ𽥴ﵽ״̬.
Table 4The appearance number of words in different training corpora
4ͬѵдĳָ

Training corpora
set (10 thousand)The numbers of 
known wordsNew adding numbers of known word pair per
10,000 words between contiguous corporaThe appearance number of 
unknown words
34 245732
55 0271 3911 306
76 7228481 961
107 5722832 778
159 1743203 874
2010 3342325 298

ѵϹģ(),֪ʳֵĸ,ѵ֮ƽÿӵ֪ʸ,δ֪ʳֵĴ.
ǰʹõĺ仯Ħֵһʻʾֵ仯,5.ʹԸʾһ,ѵϵԴʻʾֵ仯ӰԽԽС.֪ʻʾʵɣһǼʵĴʻ,һǷǼʵĴʻ.Viterbi㷨пԿ,ǼʵĴʻʶڱע岻.ǽһʵĴʻʵı仯,ӱ5пԿ,ʵ,ѵϹģ,ʻʾмʵĴʻʲֵı仯С,ӽӽȶ״̬.
Table 5The variety of word probability matrix in different training corpora
5ͬѵ´ʻʾı仯

B1B2B1Probability part of pluralistic part-of-speech words in B
B3WB5W0.390.30
B5WB7W0.36 0.26
B7WB10W0.300.22 
B10WB15W0.190.08
B15WB20W0.150.07

٦ãBмʴʻʲ).
ѵԸʾʹʻʾķ֪,ѵϹģĳһΧʱ,ѵõڵͳƸģнϴı仯,ϵͳıעȷҲнϴ.ѵϵĹģﵽһ̶Ⱥ,ģ͵ľԽԽӽ,仯ķҲԽԽС,ģȶ.,ϵͳıעȷʵ߾ͷǳ,ʵ½ԽԽС.ӶʹѵϵĹģСעȷʵֳ֮Թϵ.
3ѵעĸĽ
ķԿ,ѵϵĹģﵽһ̶Ⱥ,ͨѵϹģĴС߱עȷǲʵ.,оѵϵĹģ¾ܵ߱עȷ,һ.ⷽһЩ,ԭѵעһЩĽ,ͼеѵƱע.
3.1ݴԶйصĴԼȴ
֪,໮ֵǴʵ﷨,Ǵʵ﷨ֻһʵ﷨Եһ,һʶһǲ.ݴ໮ִȻࡢ,Ϣܶȴ,ͬһĸĴ﷨Իв,ֻдӴ﷨ԵĶǶȳ,ܱȽȫʶһ.ͼôй﷨ԴĴԵж.ף4,Դﰴմ˷,һÿ﷨Ըϸ,ʹȷĴ﷨ϢΪ.
һֲע,vgo(ʲ)+n()롰vgn(ʴʱ)+n().ڡʱڵ쵼ɲҪѧϰѧ롰Ϊһ,ǽѧ仰,һǰĴʵĴԶvgn,Ĵʶ.,ڵ1,εǡѧ,vgo,ڵ2,ѧǡı,ĴӦΪvgn.,ԭıע,϶ͻһע.
ף4,ʴһʴֱܷܶ(ġ)ɶнṹġǰ.,ѧڿֱܶεʴ,ѧڲֱܶεʴ.ǻע⵽,ף4ݵĶʿ,ҲһʴǷֱԹɶнṹġ.һǡǡ.,,ڵ1,ΪvgoΡѧĸֵӦôڵ2ΪvgoΡѧĸ.ҲڱһЩжʴԵʱɶ.ڶʵļǺбһ,߱עȷʻкô.
,ף4ṩ,ǰԭеࣺ(κ,nf)һ(,nr)ǰٷֳ-ǰ(nfqdk)-ǰ(nfqdf)һ-ǰ(nrqdk)һ-ǰ(nrqdf)4С,Ѷʲ(vgo)ٰԷֳɶʲ-(vgohmk)Ͷʲ-(vgohmf)С.Viterbi㷨,faitj=ʱ,µĴ
aij,Tivgo,Tjnfnr,ôжǷSt-ivgohmk,Stnfqdknrqdk,aij滻aij,һ,ڷŴvgohmknfqdkvgohmknrqdk֮ϵĸʹϵ,ӵֵͨʵıȽѡȡ.
Ҫע,ַ,ĳЩġ+ʡĽṹ,ܻήǵıעȷ.磬ڡѧϰļ,ȿǡѧϰΪĶΡļɶнṹ,ҲǡļΪѧϰıɡṹ.ڡѧϰС,ļҲСǰ,,ѧϰпܱעvgo.,ʵ,ô﷨ԸдԱע,ǿԱȽЧ߱עȷʵ.ԺĹУҲⷽķ,ʹַ.
3.2Ľδ֪صĴԸʵĹ
ڱϵͳ,ݶϿͳ,һ23%Ĵδ֪.ӱ1пԿ,ڿŲ¶δ֪ʵıעȷʴԼΪ50%.רҵԽǿʵı,δ֪ʵĳʿܻ,ΪӰϵͳעȷʵһҪ,,ȡЧֶζδ֪ʽбע.ǵǼǿδ֪صĴԸʵĹ.
ڴʴS=S1S2...St...SM,Stδ֪,ڼSt-1(St,Tj)·ĸȨֵʱ,ԭǣ

,Stδ֪ʱ,P(T(UW)=Tj|Tt-(UW))(UWʾδ֪UNKOWNWORD)滻ԭʽеaij=P(Tj|Ti)ٽм,P(T(UW)=Tj|Tt-(UW))ǱTiֵδ֪ʵĴԱΪTjĸֵ.ڽδ֪ʿһļ,ĴתƸʴһĴתƸֵӦ,Ը׼ȷطӳͬԶδ֪Ӱ,˶Ԥδ֪ʵĴҲЧ.
ڼP(T(UW)=Tj|Tt-(UW))ʱ,ǵϡӰ,òֵƷ3ݽвƽ,P(T(UW)=Tj|Tt-(UW))ļ£
P(T(UW)=Tj|Tt-(UW))=1P(T(UW)=Tj|Tt-(UW))+2P(T(UW)=Tj).


N(Tt-(UW),T(UW)=Tj)ѵбTtֵδ֪ʵĴԱTjĴ,N(Tt-(UW))Ttųδ֪ʵĴ,N(T(UW)=Tj)δ֪ʵıΪTjĴ,N(UW)ѵδ֪ʵĸ.
ǴԭеѵעеĸĽ.ǵĿľҪеѵ,Ľͳѵͱע,ӸϸĽǶʹͳƲ׼ȷطӳڵĴ﷨ܵĸʷֲ.Ľķûжԭѵͱע㷨ʱռ临ӶȲϴӰ,ͬأԭѵͲϽʵõĽ6.
Table 6Part-of-Speech tagging result after improving
6ĽĴԱעʵ

Training corpora
set (10 thousand)Tagging accuracy
of open test (%)Tagging accuracy
of close test (%)Tagging accuracy of unknown
words on open test (%) Tagging accuracy of unknown
words on close test (%)
395.097.258.268.9
595.697.055.661.3
795.896.863.162.1
1096.096.866.765.2
1596.096.664.461.1
2096.196.565.565.3

ѵϹģ(),ڿŲԱעȷ,۷ղԱעȷ,
ܿŲδ֪ʵıעȷ,ݷղδ֪ʵıעȷ.
ӱ6пԿ,ԭķ,ĽڿŲ,ע½20%,δ֪ʵĴ½18%;ڷղ,ע½25%,δ֪ʵĴ½22%.3ʵѵõĿŲԵȷʼﵽԭ10ʵЧ,Ч˱עȷ.
4
ĴĿǰõĻͳƵĺԱע,ԲͬѵϹģµʵ,ӴԸʾʻʾĽṹֵ仯ȷѵϹģעȷ֮ڵķԹϵ˷ԸĽ,õһǿĴģʽRF-Enhanced,ЧԶעȷ.
Ϲе,Ϊ,ںԱע,ۺô֪ԸԱע.Ϣ˵һάռ,֮ͬ໥Ӱ.ڽдԱעʱ,ֻǴԡ֮Ĺϵǲȫ,(﷨)Ҳжϴ.
,ͨԱעͳƷ,һЩ״Իߴ,ǵ﷨ܵص,ӦؼǿйصĸϢͳ,⴦.ԶһЩܽ,ƶɹ,ڱע.ʵ,Ϊ,ڹıעͳƵķ֮䲢ì,ʵǶһЩĸֵȽϴܽ.ڻͳƵķ,ͳģͱеȱӰ,ѵõĸʲһܹЧطӳԱһЩȷԵ﷨ص,ùȴֲⷽȱ.
ԺĹ,ǽЩо,ϣܳѧ֪ʶ,ͳƷ,ʹעܻøһ.
κŷйѧԺо100080
⽡йѧԺо100080
񷼣йѧԺо100080
ο
1Zhuo Qiang. Chinese corpus tagging using rule techniques and statistics techniques. Journal of Chinese Information Processing, 1995,9(2):110
(ǿ.ͳϵĺע.Ϣѧ,1995,9(2):110)
2Bernard Merialdo. Tagging English text with a probabilistic model. Computational Linguistics, 1994,20(2):155171
3Zhuo Qiang. Corpus-Based and statistics-oriented natural language processing techniques. Computer Science, 1995,22(4):3640
(ǿ.ϿͳѧȻԴ.ѧ,1995,22(4):3640)
4Yu Shiwen, Zhu Xuefeng, Wang Hui et al. The Detail of Modern Chinese Syntax Information Dictionary. Beijing: Tsinghua University Press, 1998
(ʿ,ѧ,ݵ.ִ﷨Ϣʵ.廪ѧ,1998)


