ѧ
Journal of Software
2000 Vol.11 No.1 P.73-78



WEBĵйṹϢĳȡ
ԥ塡ݹ־Ÿ
ժҪΪ˶WEBϲġ̬ϢݿķʽɺͲѯ,ö󽻻ģ(object exchange model,OEM)WEBϢģ.Ϊ˽ҳиֱʾΪӦOEM,˰ṹϢĳȡ㷨,Խ.÷ԳȡṹͰṹϢ,еĳȡͨԸǿ.
ؼʡʽ,ݳȡʽ,󽻻ģ.
ͼšTP311
Extracting Semi-Structured Information from the WEB
HUANG Yu-qingQI Guang-zhiZHANG Fu-yan
(Multimedia Computer InstituteNanjing UniversityNanjing210093)
AbstractIn order to integrate and query irregular and dynamic information on WEB in a database-like fashion, the authors use object exchange model (OEM) to construct information model of WEB in this paper. To express each component of pages as an OEM object, the authors design an algorithm which extracts semi-structured data from HTML pages, and the testing results are given. This method can extract structured and semi-structured data. It has better applicability than other existing methods.
Key wordsHeuristics rule, data extracting format, object exchange model.
WWWǿľ޴Ϣ,дõϢ,Ϣ洢Ϊ̬HTMLҳ,Ҫͨ鿴.ȻǿеڶЧϢĲѯ,ǲѯǻصHTMLҳ,ҪֱӲѯҳϾȷϢʮ.
ݿĹ۵㿴,WWWϵĴԴṹϢ,ĳָʽʾ.AAAIԴ,ṩ˷ֶεĻ,ߡƵϢ.AAAI-1997ҳͼ1ʾ,AAAI-97 Technical Papers,Invited TalksȵǸпʶı,ڻҳṹĲѯǳ.,ѯFind Kentaro Toyama's Papers,Ӧ÷ҳKentaro Toyama.ǿͨȡҳϵĽṹϢ,װṩĲѯ.,ϢĿ޴,ԴƵ,ִԴĸʽ䶯,WEBĵṹϢֹȡǲʵʵ.ף1ݲʽ,ոĴСƵҳϵĲνṹ.÷ûбĴСĲ޷ȡ,ܴбͱ,ʽҲ,ҪûֹУϵͳĽṹ.ף2ݲûҳļԲνṹгȡ,ļҪûȡ̵ľͱдȡ,ֻĳҳ.ף3WWWԴĳȡ,ҪǳƷҳ,ҪϢͽкǿļ,ֻܳȡϵ͵,÷ڸһ͵ҳ.

Fig.1Fragments of AAAI conference homepage
ͼ1AAAIҳƬ
ǰֳȡڵǣ(1) ܽݶȡ,(2) ͬҳҪдͬĳȡ.Ϊ˽ûҪݶȡ,ҹڸҳʽĳȡ㷨,HTMLıиƬϵĳȡ㷨,㷨ûָݳȡʽΪ,ʽгȡ.ĳȡ㷨ڲİṹ,ԸҳеĽṹݺͰṹݶͳһ,ͨǿ.
1ṹϢĳȡ
ṹϢȡ̾ǽһ鳬ӵHTMLҳתΪǶ׵ݶ,ȡݵҳַ͸ݳȡʽΪ.HTMLҳĸʽı,ݳȡʽ뱻.Ϊݳȡʽһ򵥵ıļ,κα༭ֱ޸.WWWϵҪϢǰṹģ4,ͳݿейĺ;̬Ľṹ,˹̹ѧOEMģͷǳṹ,˳ȡOEMʽ.OEMһģʽģ,رʺϱʾϵİṹ.OEMбʾһͼ,ڶΨһĸ,ǶӶ.OEM1ǡ1ͺ1ֵ.ñ˴洢ɲеֵ.洢OEMеֵԭԵģstring,ҲһOEM.
1.1ʶα
ıʾĶοʼıǳHTMLбʾΪֻߴߴַбԼ.ڱ1гĶαʽʶһҳĸα,αʽʽһµĵʻΪαⱻϵͳ.ΪαҳһεĿʼ,αʶʱ,вͬĶξͱʶ.
Table 1A list of section expression which identify section titles in pages
1ʶҳ϶αĶαʽб
Section expressionDescriptionExamples
h1-6#/h1-6 Title font sectionܡh2Invited Papers/h2
b#/bPhysical bold-faced sectionݡbInnovation in Database Management:
Computer Science ... /b
strong#strongLogical bold-faced sectionޡstrongAreastrong
fontsize=1-7#/fontSection with font sizeߡfont size=7good weather/font
ulli#/ulList section without orderࡴullianame=Jacobs97 href=..
/../indices/ a-tree/j/.../a:../ul
olli#/olList section with orderᡴolliTodayliTomorrow/ol
table#tr#th#td#/tableTable section⡴tabletrthFood/thtrtdA
/tdtdB/td/table
ٶαʽ,,,ܱ,ֵĶ,߼ֵĶ,
СĶ,б,б,.
1.2ȷҳνṹ
һǻȡҳиκ֮ĳιϵ.Ϊȷ֮Ĳ,Ҫʽ(1) ӶαһεıС(2) ŵĶγһεӶΣ(3) бΡбκͱиԪػΪֵ.ɴ,ݸαʽ֮ͬĶα,Ϊα⹹½ڵ,ݱСžʹ½ڵΪһεӽڵ,ͬһбθڵ㻥Ϊֵܽڵ.ڵ,ֵĴҳ϶ӦεĴͬ.
ǵȫʽʶһҳṹʱ,ѡûͼһµĶα,Ϊαʽʽܸе.ϵͳûͨ޸ݳȡʽļϵͳʧĶλɾϵͳѡĶ,ʽݳȡʽ,Զ׼ȷ֮õõƽ.
1.3ݳȡʽ
ݳȡʽ,ǶԸȤҳеıǵ.ضıǱʾǶĵʻһ,һĳ־ζһ¶εĿʼ,,ʶʶһҳеĶ.һҳԷֽΪͬĶ,һְӶ,ҪʶڵǶ׽ṹ.,AAAIҳϵAAAI-97 Technical Papers,Invited Talks.AAAI-97 Technical PapersӶAgents,Automated Reasoning.
һݳȡʽNʽΪı,α,Դַ.,ıʾԴзȡҪıΪ´OEMֵαΪOEMıǣԴȡֵΪһҳַǰȡʽгֵĶα,ʾڳȡıԴǸҳHTMLıǵǰֵ.1дһ,дΪԴġжΪαġȡֵɶı޶OEMӶ.ûһҳиȤͨһΧ,,һҳݳȡʽĵ1ͨ˵˸ҳòֵĿʼַǡݵOEMıǺ͸ҳϵַ.˵AAAI-97ҳдOEMݳȡʽ.
: 1h2#DBLP:,root,get(http://SunSite.Informatik.RWTH-Aachen.DE/.../AAAI-97.html)
2/hx#hx,hx#/hx,root
3li#li,Section,hx#/hx
1ӸURLԴļлȡڱǡh2DBLP֮ı,root-var.޶ҪȡķΧ,ɸڵ.
2URLԴļе1γ֡h1-6֮ıȡ,hx/hx֮ıΪµĶα,/hxhx֮ı,ӵǰλõһֵ֮ܶıΪöαֵ,xʾ16֮ȡֵ.,ʽ,Сɸҳĳνṹ,AAAIҳ˵,ڡh2h3͡h43͵,ڸڵ3ӽڵ.
32Ķαֵгȡli/li֮ıΪSection-varֵ,츴ӶSection,ΪѾӽڵ.
ͼ1ʾҳHTMLıƬͼ2ʾ.ݸʽнжαĴ͸ֵ,ʽ,ȷ֮Ĳνṹ,ҳ6ṹ,rooth2#/h2h3#/h3h4#/h4Sectionauthor,Title.ݱװһOEM,ͼ3ʾ.

Fig.2HTML fragments of AAAI conference homepage
ͼ2AAAIҳHTMLıƬ

Fig.3OEM objects extracted from AAAI homepage
ͼ3AAAIҳȡOEM
1.4ݳȡʽʽ㷨
룺ݳȡʽSECTXT1,SECVAR1,SOURCE1,...,SECTXTM,SECVARM,SOURCEM
SECTXTi,SECVARi,SOURCEiֱiеĶı,α,Դ
ҳOEMOEMdata
OEMdata EXTRACTOR(SECTXT1,SECVAR1,SOURCE1,...,SECTXTM,SECVARM,SOURCEM)
SOURCE1ַȡHTMLı
ȡSECTXT1ҷָ֮ıΪֵ
ΪSECVAR1ĸR0
SECVAR1ƺֵValue¼Ѵ
While(ݳȡʽδ)
Ѵе1¼Ϊǰ
While(ڵǰдںSOURCEmһµSECVARmӦֵValue)
ValueΪԴ
SECTXTmСǡ
SECTXTmԴлȡıΪΪSECVARmĸֵ
SECVARmΪַ,ӶǶSECVARm
SECVARmValueֵлȡ
ɶΪΪSECVARm
ЩӵSECVARmֵValue¼Ѵ

SECTXTmСǵ,ʽȷνṹ
Դлȡгߴ,ӴС¼б
б׼¼Ϊǰ¼
While(ǰ¼Ϊ)
յǰǴԴлȡıΪֵ
SECVARmԴлȡ
ɶΪΪSECVARm
ЩӵSECVARmֵValue¼Ѵ
ValueΪԴб
б¼һ¼Ϊǰ¼


SECTXTmǵ,ʽȷνṹ
Դлȡгߴ,С¼б
б׼¼Ϊǰ¼
While(ǰ¼Ϊ)
յǰǴԴлȡıΪֵ
SECVARmԴлȡ
ɶΪΪSECVARm
ЩӵSECVARmֵValue¼Ѵ
ValueΪԴб
б¼һ¼Ϊǰ¼


Ѵһ¼Ϊǰ

ȡһݸʽ

Return OEMdata
óȡ㷨ʽݳȡʽ,ȡʽıSECTXTmгСǵϡǵʱ,ʽȷνṹ.ڳСǵ,ҳԴߴ,ҴӴС,ڸߴɸ,ڵ,ϴɵĶֵΪСɶʱԴ.ڳǵ,ҳԴ,ҴС,ڸɸ,ڵ,СɵĶֵΪϴɶʱԴ.
2عıȽ
ʹóȡHTMLҳʾΪOEM,ԼĳȡʽĳȡЧ2.ṹƵԴȡʽͬ,,VLDB顢SIGMOD顢PODSAAAI.,ڽṹƵԴ,,ϺͨѧоĿй걨͡ձȡʽĲܴ.ǿǽԴսṹԷ,ͬһԴʹһֱ׼ȡʽ,й걨͡ձʹһֱ׼ĳȡʽ,ֻҪһ׼ȡʽ޸,ͿڸԴĳȡʽ.,ȡʱ䲻ļĴСй,ҺҳĲνṹй,ҳεӶ.ף1ݵʽ,ûбСĲ޷ȡ,ܴбͱ,ڱ2иҳڵĶ󲿷޷ȡ.ĺף2ݵ,ĽȡķHTMLҳʽֿ,ûֻҪHTMLĸʽ,ĳȡͳһĳȡɣף2Ҫûҳʽͳȡ,ⲻûĸ,ȡͨ.ף3ֻܳȡ͵,޷ԾиǶײεĶгȡ,ڱ2еĸ޷سȡ.,ĳȡ㷨ԽûҪȫȡ,ڸҳʽ,ͨǿ.
Table 2Time and cost for extracting various WEB sources
2ȡͬWEBĵԴҪʱʹ 
WEB sourcesExtracting time(ms)Extracting format linesDeepest level of OEMFile size (K)
VLDB Conferences7657528.6
SIGMOD Conferences5907435.7
PODS Conferences1727413.5
AAAI Conferences19826660.4
Research of SHJD2506316.7
Info. Center of CAS1254512.4
China Youth Daily325313.0
People's Daily626320.0
WEBԴ,ڳȡʱ,۳ȡʽ,OEM,ļС,VLDB,SIGMOD,PODS,AAAI,ϺͨѧоĿ,йѧԺ鱨,й걨,ձ. 
3Ĺ
ʶһҳĲνṹʱ,ϵͳܲʶҳϵĶα,ҪϵͳмûѧϰͱǵĽṹ.,ڳȡʽⲿ,Դļıʱ,Ҫļи,ȡҪѧϰ,Զ²²HTMLṹ.
ԥ壨ϾѧýоϾ210093
ݹ־ϾѧýоϾ210093
ŸףϾѧýоϾ210093
ο
1Ashish A, Knoblock C. Wrapper generation for semi-structured Internet sources. SIGMOD Record, 1997,26(4):815
2Hammar J, Garcia-Molina H, Cho J et al. Extracting semi-structured information from the Web. SIGMOD Record, 1997,26(2):1825
3Kushmerick N, Weld D S, Doorenbos R. Wrapper induction for information extraction. In: American Association for Artificial Intelligence ed. International Joint Conference on Artificial Intelligence (IJCAI). San Francisco, CA: Morgan Kaufmann Publishers, Inc., 1997. 729735
4Chawathe S, Garcia-Molina H, Hammer J et al. The TSIMMIS project: integration of heterogeneous information sources. In: Ashish G et al eds. Proceedings of the 10th Anniversary Meeting of the Information Processing Society of Japan. San Francisco, CA: Morgan Kaufmann Publishers, Inc., 1994. 718
1998-11-17ո
1999-02-12޸
