ѧ
JOURNAL OF SOFTWARE
1999 19 3  Vol.19 No.3 1999



Myrinetûռ侫Э
ס֣γ
ժҪͨϵͳӰ칤վȺϵͳܵҪ.ڷͱȽ3ֳõָ֮ϲЭĴӰ칤վȺϵͳܵҪƿ.640MbpsMyrinetӵ8̨Sun SPARCվɵĻȺϵͳʵһûĸܵľͨЭ顪RCP(reduced communication protocol).ͨЭ๦ܡݿֱӲӲȷ,ﵽӳ١Ч.RCPĻ·ӳʱTCP/IPСöࣨ200s vs 1 540s,ôҲߵöࣨ178Mbps vs 34Mbps,ЭĴʴﵽ80.5%,ΪϲӦóPVMṩһ򵥡׵Ľӿ.
ؼʡվȺ,ͨЭ,д,PVM.
ͼšTP393
A Reduced User-level Communication Protocol for Myrinet
DONG Chun-lei ZHENG Wei-min
(Department of Computer Science and Technology Tsinghua University Beijing 100084)
Abstract Communication subsystem is a major factor that affects the overall performance of a workstation cluster system. In this paper, the authors analyzed and compared the performance of three popular networks, and pointed out that the high level protocol processing is the bottleneck of communication in workstation cluster system. A user level high performance protocol, called RCP(reduced communication protocol), had been built on a cluster of 8 Sun SPARC workstations connected with 640Mbps Myrinet. Low overhead and high efficiency of the protocol are achieved by simplifying the protocol's redundant functions, reducing the times of data copying, and operating directly on hardware buffer. The round trip latency of RCP is much lower than TCP/IP(200s vs 1 540s), the application available bandwidth is much improved(178Mbps vs 34 Mbps) and the bandwidth usage of Myrinet on RCP reaches to 80.5%. A very simple interface to application or PVM is provided as well.
Key wordsWorkstation cluster, reduced communication protocol, parallel processing, PVM.
ͨϵͳǲйվȺϵͳҪɲ.ʵϵͳϢݹ,ܵĺûӰ첢мٱȺЧʵҪ֮һ1,2.ʵһֿٵϢͨŻ߹վȺϵͳܵЧ3,4.
(1) ͨϵͳ״
 緢չѸ
ڰмڵĶӦöԸ,ƶ缼ķٷչ.Ŀǰ˶͵ĸ,Fast Ethernet,ATM,Myrinet,ЩٶǴͳEthernet10.ڸ,ʹӰͨϵͳܵƿѴӹȥӲתƵͨ,Ҳ˵,Ըϵͳ,·ĴӳѾС,ӰͨϵͳߵȴͨЭĴ.ͨЭĴ,谭˸ܵĴ.Myrinet·Ϊ640Mbps,TCP/IPЭõĴֻ42Mbps.ɼ,ϲͨЭĿʹĸܵòֵķ.
 ֵȴ,ӳٸĽС
νܵ,ֵָ.봫ͳEthernet,ķֵ˺̶ܴȵ,10Mbps100Mbps,640Mbps,1Gbps.ĴӳǴݰĴ.ϵͳͨܵһҪӳ,ϵͳļ,ֱӰϵͳĲЧ.Щķֵ,ӦóĿɼҲнϴ,ʵϵͳĲз,ӳٵĸƲ.1Ƕ10Mbps Ethernet640Mbps 
1 EthernetMyrinetӳʱ

TCPӳ٣sUDPӳ٣s
Ethernet1 4381 146
Myrinet1 5061 370

MyrinetһֽݰӳʱĲԽSparc 20վϣ.
ӱ1п,MyrinetͨӳٷͨEthernet.ɼ,ڴСݰʱܲ,Сݰϵͳܵĺû,ԲмкܴӰ,ΪûƵĲ㷨ҪСݽ,Ҳм㻷,PVM,ʱҲ跢ͺܶСĿϢϵͳļ໤.Ӧö,СݰĴӳҲ𵽾ص.,UC BerkeleyͨNFSݰĸٷ,95%ݰĴС192ֽ,ݰƽС382ֽڣ5.,ССݰӳʱϵͳܵҪ֮һ.
 ͨЭ
缼ӿ,ûжԴͳͨЭнϴĸĽ.ʹATMȫͬEthernetĸ,Ҳ뷽跨ʹTCP/IP6Э,һҪԭڲƷļݼƹ,ΪTCP/IPЭ鼸ռͨŵĸ.ԲлȺϵͳ˵,ԲҪ,ˣԶԴͳͨЭơʵϴ޸,˷ͳЭıײ,ʹԽԵԳչ.
(2) Ӱͨϵͳܵ
ǰķ˵,Ƹܵƿϵ,ĽǶȷӰͨϵͳܵҪ.ͨſҪЭ鴦ͲϵͳĿ.ϵͳṩЭʵֻ,ҲЭĳЩʵַ,Э鱾ĴʽҲӰͨӳٵ.ؽнΪϸķ.
 ͳTCP/IPЭظʵֵĹܺܶ
TCP/IPЭʡ߲ʹݰƵ,һεṹ.Эζ,ʹöݵĲͿ,ӳʱ.,ڶЭʵ,㻹ظʵ˺ܶͬĹ,:(a) ·㵽㶼Ҫв;(b) ·㵽Ӧò㶼Ҫӽͷ;(c) 㵽Ӧò㶼ҪЭĴ;(d) ·㵽Ӧò㶼Ҫ;(e) 㵽Ӧò㶼ҪװͶĻ.
ЩĹȻȷݵ޲,·ʴ10-410-9,ദݼʱύӦó.ɼ,εЭṹͨƿҪԭ֮һ,ϲĳЩ,ɾĴ,һͨЭ,ͨܵҪ.
 Э鸴ӵĻӳ
Э鴦ܶ๦,ơơطơӵƵ,Щܵʵֶ뻺.
ݵķװ,ɿһԴ,Դ޵,ĹҪ.,ͨĻƶȽϸ.,Berkeley UNIXһֽmbufsĽṹЭݰл,mbufs㷨ܸ,ܴ.DECstation 5000,ԵֽTCPϢ,mbufsҪ100s,512ֽݰҪ300s5,ɼ,ӳҲܴ.μЭ鸴ӵĻҲоҪ.
 ϵͳ⿪ɺ
ϵͳṩϵͳúԭЭʵֵĵײ֧.Эʵ漰л/ҳ桢I/O豸жӦȲϵͳ,ʱЩܱЭ鱾Ĵ.,Sun 3/60ϵͳ,TCP/IPһݰЭ鴦ʱΪ100s,ϵͳĶ⿪ȴߴ240s,ͨܶԲϵͳһ̶ϵ.Ŀǰ,һֹ㷺õ˼ǲ޸UNIXϵͳ,ûռʵһû̬Э,ʹЭܹ·ϵͳӰ,ֱӶӲ豸в.ͿɼСⲿֵĿ,޶ȵͨϵͳ.
Ϊ˼СͨЭ鴦,ʵһMyrinetͨЭRCPreduced communication protocol,˼ͳTCP/IPЭ,ԵײͨŽӿΪ,ЭĲκͲҪĻ,ﵽٿ򻯻ͽЭ鴦Ŀ.
1 ͨЭRCPĽṹ
ͨЭRCPһMyrinetAPIϵûռЭ,ΪӦóṩ򡢿ɿ˫.һûռͨһֱͨӿԶ̵ûռ,ҪϵͳĶλԤ,RCPĽṹͼ1ʾ.

ͼ1һЭ뾫ͨЭRCPĽṹ
RCPĿҪСĻ·ӳ,·Ĵ:.MyrinetAPI,ԳMyrinetӲ,ΪӦóṩӲֱӿ.
RCPƵһĿҪʵһ򵥡׵û̽ӿ.ĿǰվȺϵͳϵĲг򿪷PVM,RCPֱͨPVMĺӿʵ,,ӦóֻҪٵĸĶͿھЭִPVMг.RCPЭPVM,ûӿ,ҼPVMһݿ,ʡPVMڲڴҪ,ΪӦóṩڴ,Ӷ˷̴,Ч.
2 RCPʵ
RCPĻ˼û֮ṩһ˳򡢿ɿͨ.һƽЭ,ͷͽշȫԵ,ִз,ִн,ĳһͬ,PVMеʽͽ.
2.1 RCPа
RCPͨͼ2ʾ.

ͼ2RCPͨ
Myrinet8 432Bytes,,RCPĴݳֻܿɷڴռ.ݵĴֱûṩԭռн,Ƚϵͳ,ּݿ.
2.2 ˳ԡɿ
ΪȡýϸߵЧ,ͬʱڲԵĽ,RCPֻ԰ͰͷϢм,ݲּУ,ΪMyrinetӿڿм,ܱ֤ȷ͵İȵĿĵ.ݵ˳һӵȫּά,ڷͺͽȫԳƵ.ɿһ򵥵ĻЭѡش֤.ڴСԸӲĴС縺ɵضı.RCPĻЭ,ͼ3(a)ʾ,ɷ˫Լ,շ˫ͬ,ȷڴС,Ȼͷһڵ,շȷϺ,ٽһڵĴ.г,Ӧָĸ,һط,ڵĳߴԽԽС,ɴ,ͼ3(b)ʾ.

ͼ3򵥵ĻЭ
2.3 ӽ
RCPһԵȵЭ,ӹþ̬ʽ,ÿһͨ˽̶Ҫάһ,ӲҪԷϢ,ֻڱһ¼,ΪԺķͼṩҪϢ.
2.4 ʼ
ʵʵӦó,֪֮ܲ׼ȷʱϵ,ִ֪еԭ,ִеԭ.RCPԹ.ִеԭʱ,շͷһͬϢ:READY,ȴݣͷݸϢȷڴС.ִеԭ,ΪһǷʹݣݳȣ1,ͷշһͬϢ,Ȼ뷢ͽ׶ΣһǷСݣݳȣ1,ͬϢһ𷢳,շһȷ.ԼССݰĴӳ.,ûĳָʱյӦ,ͷΪʧ,·,ͼ4ʾ.

ͼ4RCPPVMӦóĹϵ
3 RCP벢пĽӿ
Ϊ˾Ӧó޸,ǶԲг򿪷PVM7ݵĲͨ̽д,ֱԭPVMpvm_psend()pvm_precv()һµͨ(ͼ5ʾ),ӶӦóĽӿ.PVMӦóҪʽRCPķͽպ,ֻÿͨŽ̵ĿʼһRCPЭʼ,ͨͨ.RCPάһŴPVMʶͨͨתͨϢ.

ͼ5RCPPVMӦóĹϵ
,RCPΪ˺pvm_precv()Ӧ,ṩһPVMĶಥpvm_pmcast(),ֱRCPûݹ㲥ָ.
4 RCPܷ
ͻ-÷¡ѧJose C. Brustoloni810MbpsEthernetϸĽTCP/IPЭʵ,ʹӳټС750s.UtahѧMark Swanson9һֻڷͷЭ,ӳٺͬ.RCP廪ѧѧ뼼ϵĲйվȺϵͳ8̨Sun Sparcstation 20/50+640Mbps Myrinet) Ͻ˲.RCPĻ·ӳΪ200s,ԭTCP/IPЭ1/7;Ӧóɼ178Mbps,TCP/IP4,ͼ6ʾ.

ͼ6MyrinetTCPRCPĴԱ
5  
ĶԹվȺϵͳͨܽ˲Ժͷ.,MyrinetAPIƲʵһûռľͨЭRCP,ݿͻ,Ӳ,еͻ·ӳ١ߴʺͷı̽ӿڵŵ.ʹRCPͨŵPVMӦó.
оõ863߿ƼĿ.߶,1964,ʿ,ҪоΪд,Ⱥϵͳ,Э.֣γ,1946,ڣʿʦ,ҪоΪд.
ͨѶϵ:ף 100084,廪ѧѧ뼼ϵ
ߵλ廪ѧѧ뼼ϵ100084
E-mail: zwm-dcs@tsinghua.edu.cn
ο
1David Culler et al. LogP: toward a realistic model of parallel computation. In: Proceedings of the Principles and Practice of Parallel Processing. http:now.cs.berkeleg.edu/paper/logp.ps, 1993. 112
2Thekkath A, Nguyen T D, Moy E et al. Implementing network protocols at user level. In: Proceedings of SIGCOMM '94. Sept. 1994. 1423
3Felderman, DeSchon A, Cohen D et al. ATOMIC: a high-speed local communication architecture. Journal of High Speed Networks, 1994,3(1):130
4Dong Chun-lei, Zheng Wei-min, Wang Ding-xing et al. A scalable parallel workstation cluster system. In: Proceedings of APDC'97. Los Alamitos: IEEE Computer Society Press, 1997. 307313
5Thekkath C, Levy H. Limits to low-latency communications on high-speed networks. ACM Transactions on Computer Systems, May 1993,11(2):179203
6Postel B. Transmission Control Protocol. RFC 792, Sept. 1981
7Beguelin, Dongarra J, Geist A et al. PVM: experiences, current status and future direction. In: Proceedings of the Supercomputing'93. Los Alamitos: IEEE Computer Society Press, 1993. 765766
8Jose C Brustoloni, Brian N Bershad. Simple protocol processing for high-bandwidth low-latency networking. Technical Report, CMU-CS-93-132
9Mark Swanson, Leigh Stoller. Low latency workstation cluster communications using sender-based protocols. Technical Report, UTAH-CS, 1994
1998-03-17ո壩
