о뷢չ
JOURNAL OF COMPUTER RESEARCH AND DEVELOPMENT
1999 36 5 Vol.36 No.5 1999



֧ŻݵƵ
ׯԽͦƺס¡Thomas S.Huang
ժҪƵĴͳ֮һȴƵժȡıϢ(⡢ؼֵȵ)ȻһؼּϻشûĲѯ.ԶժȡıϢĹδԶƵժȡϢҪ˹ɣʵӦ֤ǲʵ.һַһεļˣõͲƵݣɫ״˶ȵȣĿڿ˷˹ժȡؼ漰.˻ToCƵṹƵĻȡϢȻWordNetһӴʻϵͳṩ.÷ӦWebMARSƵϢϵͳУнϵͳļܴԸ.
ؼʡƵĻͷ
ͼšTP391TP18
USING SEMANTIC ASSOCIATION TO SUPPORT 
CONTENT-BASED VIDEO QUERIES
ZHUANG Yue-Ting, PAN Yun-He,
(Department of Computer Science,Zhejiang UniversityHangzhou310027)
Rui Yong, and Thomas S.Huang
(University of Illinois at Urbana-Champaign)
AbstractThe traditional way of video retrieval is to annotate the video, get textual information such as caption, keyword, and so on, and then answer users' query based on this information. Since automatic annotation is still impossible, this approach is rather limited. Another approach is to utilize the low-level video content, such as color, texture, shape, and motion feature, the purpose of which is to overcome the difficulties encountered in human annotation. It is understood that user's query should be based on both the textual information and its visual contents. The ToC-based semantic representation is put forward. The basic keyword set is obtained from closed-caption. Combined with WordNetan electronic lexical dictionary, a prototype system-WebMARS VIR is implemented.
Key wordsvideo retrieval, closed-caption, shot, scene
1
ĿǰԽԽƵڸWebվ֮УInternetڳΪһ޴Ƶֿ.ЧؼƵΪݿԼϢоĹؼ.
ͳƵ˹شƵժȡıϢ(ͨ֡ز)Ȼı͵Ĺؼֽвѯ.һЩϵͳժȡƵаĶԼ֮Ĺϵ12.ף2УϵͳֹشƵժȡϸıϢȻϵͳܹǳӵĲѯ.硰ҳгGene KellyGiner RogersϵȻWebϴڵƵ֮޴ֹժȡϢǲʵ֮.
һ;ȫݣʹòѯȫͼ/ƵݣɫֵȵȣĿǰֹȷʱִԵ.ⷽоδ3.
ΪûʺʹõĲѯӦǼ漰ؼ漰ƵͲݵ.һַͨƵĻ(closed-caption)ժȡؼ.н߻ݵƵݿ.
2ϵͳṹ
ƵϢ֮޴ȫֹ޷.еоּʹܹɼԶɵĲ.ͼ1ʾΪWeb-MARSƵϢϵͳ(VIR)жƵݵȡṹ.


ͼ1WebMARS VIRмɵƵȡģ
 MARSmultimedia analysis and retrieval systemUIUCϵͳ4ݣչWebʹãнܵɵƵϵͳһ.
Ƶݣ羵ͷ߽硢ؼ֡ȱȡ֮Զؼ뵽ݿ֮.ЩĻƵ(ıؼ)ҲͬʱȡԶؽݿ֮. 
Ƶıǹؼ.½нһΪƵݱ(ToC)5ݵĲλṹڸýṹа.ͼ2ʾΪƵͲϵĲѯģ.


ͼ2WebMARS VIRеĲѯģ
ΣһûĲѯΪ
ҳͼͬද(action)ľͷ?(ͬʱͼ).
ϵͳݿƥ䵥ʡactionVDB֮УӦƵͷͷأ򣬴WordNetҳͬ(Synset)ȻһVDBвѯ.ڸ֮Уactionȿָmilitary actionҲָhuman activityWordNetԣÿһʶͬһϵĳһ㣬ǿͨ׷ݡaction㣬;ĸڵݿȽϣ.
3ƵıƵݱ(ToC)
ΪܶƵҪҵһЧķƵ.зΪͨһַǽṹģͷ6. 
һ⣺һһ1000ҳݵ.Ķ֮ǰȷݱ(table of content,ToC)ҵһ»һӭҪ.ͷһض(ѯ)ҵһһؼ֣ҳӶҵӦ½.֮ToCȫ飬߽м().ǰڶͷضʱǳãͨToCʹϢҪض;.ڶضϢʱر.ڰ˽ȫʱͬҪȻĿǰƵԣȱٵȴToCԼ.ҪоToC֧ŶƵķ.
ƵɲεرΪ5ṹֱΪƵ(video)(scene)(group)ͷ(shot)Լؼ֡ϵ.ͼ3ʾΪλƵ.гΪһʱڽӵľͷļϣΪ㾵ͷ㳡֮м㣬֮佨.ʱڽľͷӾƵľͷɲͬΪϵĳ.


ͼ3λƵ
Ϊ֧ŲǻݵǻݵļҪϵToCṹ.ͼ4ʾΪƵToCĶӦϵ.


ͼ4֧Լ˫عܵƵ
Ϊܴ͵ĻƵıǾͱȽ͵ΪToCṹҪ֧.Ϊ֧żǴƵȡʵݣ
¼(event)
(object)
ص(site)
ؼ(keyword)
ͬʵ壬ӦToCĲβһ(ͼ4).ڹؼֶԣToCеĲӦؼֿĳֳģָھͷؼ֡бʶ, ¼飬ص¼ĵط.硰ڹ԰ɢһ¼԰ǡص㡱 ˡǶ󣬸ù԰ǡڡ꽨ġȵΪؼֱȡ.
ͬʵ壬жϵáӼȨֵ(link weight)(01֮)̻ĳһToCеʵʵ֮ӵǿ̶.ĳһƵAĵһͷ롰LWΪ0.9Ǹþͷ֮еһҪݣȨֵToC֮Сǰ(back and forth)ת.
4Ƶȡ
ԶЧػȡƵа?ҪһַǴƵĻлȡıϢ.
4.1ƵĻ(closed-caption)
ƵиĻϢΪʧȥҲܹͬۿ.Ϊ֧ŻƵݵļвõķǽĻϢȡ.
ƵеĻڱ׼NTSCƵϢĵһĵ21мһĸݽϵӽĿȥ.ЩĻıĿйصһЩ.£ƵĿенݵֱӱ.ʱÿһ֡лַ߰ĸַ.֡еһЩַһ𽫻һʻһ.ַھıԣɫ͡ԼĻеλãĻĽ.
ǵϵͳУʹBroadway(һֻPCӰ)ƵֻʹSunBelt˾Ļ׽TextGrabberڻȡĻͬʱƵй֡λͬĻؼֻؼӽӦϵͼ5ʾ.


ͼ5ƵĻȡͼ
ĻıͬƵĿݣ¿ɹɳ漸ࣺ
(1) ͵Ļ.ĻȫǶԽĿз龰.DiscoveryƵŵĶĿĻϵǻڷģ߾õĶӦϵ.
(2) Ի͵ĻĻǻ֮ĶԻĻ뻭龰ϵ.ڶ˵ĶԻ漰ȴĳһս.ûѯĽǶȿûԸ¼(硰ABڽ̸)ҲԸݶԻݷֱвѯ磺һӰĳЩʵĶԻʹûԽƬֻѯҪƵƬ.
ͼ6ʾΪ֮֡ĶӦ.


ͼ6֡ĶӦ
ǴӰա(Independence Day)ժȡһĻ

It is so fuzzy.

oh, no.
Good morning, Lucas
You see these? I got a whole god damn crop full of these
If your father's not in the air in 20 minutes 
whispers
all right, go ahead. Put it on.
General, You might want to watch this.
TV: Ladies and gentlemen

Ͽš ݡ֮еĻӦ(sound)壬Ҳɿһ¼.ĶӦ֮1.
1

ı ʼ֡ ֡
fuzzy 10 17
horn honks 18 33
lucas 50 57
whisper 420 430


4.2ĻıĽ
ĳһͷӦĻıȡͨһؼȡAZ TaggerٶıϢн.ͷ͹ؼ֮Ȩֵǰlw=tfidfģtfidfֱƵ(term frequency)ԼĵƵ(inverse document frequency)һһĵгֵƵԽߣlwĹԽһڶĵг֣䵹(idf)ӳ˶lwĹ.
 AZTaggerɣǴѧŵ˹ѧĸȡ.4.3WordNet
WordNet˹ٴѧGeorge Miller˿ĵӴʵϵͳ.WordNetʲǰӽĸ(ΪSynset)֯.WordNetеÿһʶһɸ(sense)ÿһ嶼ͬͬʼ.ɲͬӹϵɲͬʼ.IS-AɵʼΪhypernyms/hyponyms,MEMBER-OFϵholonyms,PART-OFmeronyms2. 
2WordNetϵ

ϵ ʼ
IS-A hypernyms/hyponyms
MEMBER-OF holonyms
PART-OF meronyms

WordNetṩ˸֮IS-Aϵ(ͼ7)֮Ҫ.


ͼ7WordNetIS-A
磺ĻгֵĳһؼǡvehicleûѯвõǡcarͨIS-AԼ롰carΪӽĸǡvehicle.
AIоУǱ֮ϵЧߣڵǣǳʱWordNetһֳɵĴ͵.
ڵ֮ľʽм㣬tqr,tdbֱΪûѯԼݿеĵʽ㣬붨Ϊ

TΪ滯ĿĶԤֵķΧΪ01ݣTܵͣôƵĸʶΪԶ()TܸߣʱҪһЩ.
5ʵ
ʵԭToC֯ṹWordNetĻıȡӦWebMARSƵϢϵͳ(VIR)֮.Informix Universal Server for Unix(V9.12)Ƶ洢DBMS.ƵݿԴWeb(crawler)õURLָWebվ.ĿǰΪ˲VIRѾԤ20ƵƬ.
ƵݿеƵToCĽṹ洢ڿͻ˵WebӦóûĲѯ͵httpd ˣCGIд.CGIWordNetԼInformixݿͨ. ͼ8ûĲѯΪҵгLadyGentlemanƵͷ.ƵݿУĻ洢ıϢǡwoman͡manWordNet鵽man͡gentlemanWoman͡Ladyϵ£ 
Gentlemanmanmaleperson,
Ladywomanfemaleperson
صĽͼ8ʾ.


ͼ8һ
6ᡡ
Уо˻ToCƵṹƵĻȡĹؼּ.WordNet߼ܣʵʾһ֧ŻݵƵЧ.
õȻѧ(Ŀ 69803009).
߼飺ׯԽͦУ19656ʿڣоΪCADýϢ˹.ƺףУ194611ڣйԺԺʿҪоΪͼѧCAD˼ά.£У19703ʿоҪоΪýϢ.Thomas S.HuangУ19356ʿڣҪоΪӾ.
ߵλׯԽͦƺס㽭ѧѧϵݡ310027
ǡªThomas S.Huangŵ˹ѧ
ο
1Chang S,Jungert E.Pictorial data management based upon the theory of symbolic projections.Journal of Visual Languages and Computations,1991,10(3):195215
2Adali S,Candan K S,Chen S -S et al.Advanced video information system: Data structures and query processing.Multimedia Systems,1996,4(4):172186
3Gupta A,Jain R.Visual information retrieval.Communications of the ACM,1997,40(5):3042
4Huang T S,Mehrotra S,Ramchandran K.Multimedia analysis and retrieval system (MARS)project.In:Proc of 33rd Annual Clinic on Library Application of Data ProcessingDigital Image Access and Retrieval.San Jose,CA,1996.260265
5Rui Y,Huang T S,Mehrotra S.Exploring video structures beyond the shots.In:Proc of IEEE Conf Multimedia Computing and Systems.Austin,Texas,1998.237240
6Rubin B,Davenport G.Structured content modeling for cinematic information.ACM SIGCHI Bulletin,1989,21(2):7879
7Salton G,McGill M J.Introduction to Modern Information Retrieval.New York:McGraw-Hill Book Company,1983
ԭյڣ1998-07-03
޸ĸյڣ1998-11-30
