о뷢չ
JOURNAL OF COMPUTER RESEARCH
AND DEVELOPMENT
1999 36 12 Vol.36 No.12 1999



չһӳļϵͳơʵּ
ªף־ΰų
ժҪⳬǵ͵ĻȺϵͳCOSMOSΪƿĿչĵһӳļϵͳ.ҪCOSMOSԭϵͳơʵּ.ص˫Ⱥʽ桢ֲʽԪݹ̴洢ȹؼI/O׼ԭļϵͳۣԽ˸ԭϵͳڱ֤ϵͳһӳĻϣ߱õĿչ.
ؼʡһӳļϵͳ,չ,˫Ⱥʽ,̷,ֲʽԪݹ
ͼšTP302
DESIGN,IMPLEMENTATION AND EVALUATION OF
A SCALABLE SINGLE-IMAGE FILE SYSTEM
WANG Jian-Yong*, ZHU Ming-Fa, XU Zhi-Wei, ZHU Ning-Ning, and ZHANG Chi
(Department of Computer Science and Technology, Peking University, Beijing 100871)
(National Research Center for Intelligent Computing Systems, Beijing 100080)
(Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100080)
AbstractThe Dawning super-server is a typical cluster system, for which a COSMOS file system has been developed. Mainly presented in this paper are the design, implementation, and evaluation of the prototype COSMOS file system. Here highlit are some key technologies including dual-granularity cooperative caching, distributed meta-data management, and network disk stripping. The performance of the prototype file system is also evaluated using the standard I/O benchmarks. The initial performance measurements show that our approach is a promising one in providing scalable single-image file system.
Key wordssingle-image file system, scalability, dual-granularity cooperative caching, network disk stripping, distributed meta-data management
1
ȻеķֲʽļϵͳNFS1ݵѱǹ㷺ܣΪ·ЧĹݵֶΣԶԶӦĿǰ㻷չҪ.깤վȺϵͳѸͷչʹ˽ϴߣӦõI/Oûϵͳĵһӳ󡢿չԺͿԵȹ˸ߵҪͬʱȺϵͳҲṩ˸õԴܵ硢ֲڸڵϵĸõĴ̼ڴ桢ٶȸĴȣΪʵָܵļϵͳṩ˿.ⳬǵ͵ĻȺϵͳҪĿҪṩܣߴʱӣĻͨŻ⣬ҪһҪṩһӳܣ絥һڵ㡢һƵ㡢һļϵͳȵ.ΪⳬƿļϵͳCOSMOSǰԭϵͳ(¼ΪS2FS,ascalable single-image file system)ԵһӳļϵͳĿչԼоҪS2FSԭϵͳơʵּ.
2ϵͳſ
ȻS2FSƻⳬõļȴձԣͨϵĻȺļϵͳ.ϸĵһϵͳӳõI/OܺͿչԼ֤ӦóȫƼS2FSҪĿ.ΪˣS2FSѡ˹ļϵͳΪʵֵһϵͳӳķúʽ漰зֲʽĴ洢ͿƼļϵͳI/Oܼչԣļϵͳ֤UNIXļϵͳȫƼ.
ͼ1S2FSϵͳṹ.ɺزû.S2FSĺĲļϵͳһʵֵģҪǽ߼ļϵͳI/O󣬲һĸʽתû.S2FSû岿֣3ûDaemon̹ɣֱΪͻ(client)(manager)ʹ洢(storager)ЭͬɺĲתI/O.д洢AIXJFSʵ˾ݴ洢ͻݼԪݵĻ棬𻺴һԵάԪݵĴ洢.Ӧó򷢳S2FSصļϵͳã߼ļϵͳļϵͳI/O󱻴S2FSıؿͻI/OɱؿͻĻõ㣬תɹ֪ͨӦĿͻ洢ɾļѽתؿͻȻ󾭹ļϵͳ߼ļϵͳѽظӦó.

ͼ1S2FSϵͳṹ
3ؼ
㵥һϵͳӳͶƼԵȱ߽£ΪS2FSļϵͳI/Oܼչԣ˻Ŀ¼ЧʹЭ飬˫ȻһЭ2.Ϊ˱ⵥһƿ⣬ݴ洢ԪݹֿĲԣʵ˷ֲʽݴ洢Ԫݹ.ȻݴѧУxFSϵͳѾserverlessķǸⳬصѡ˲ͬʵֻ.ǶS2FSĳЩؼصԽ.
3.1˫Ⱥʽ
ԽԽķֲʽϵͳøܵ罫ԶڴΪһ͵ڴṹΣΪͨߴʱӵԶڴķҪȱش̿ö.Զ̻ṹ(remote caching architecture)3ǿ˸洢Σڴ桢Զڴʹ̣֮ϵĲ.ϵͳн㻥ȡԵıڴԳԶڴ棬ֶԳʽṹȻͬڴͳĿͻ/ģͣΪϵͳһͻıڴа˱Էһֽ֮λȵ(peer-to-peer)ṹЩף4ݳ֮Ϊʽ(cooperative caching).
ʹûģ͵ļϵͳУһҪǾȣͨҲΪļ/дݴ䵥λ.Ҫļ塢ͨŴʱӡӦñѡͨŴϸ߶ʱӽϴʱȲӦ̫Сͨſ̫෴ļϸӦóֳ϶дصʱȲӦ̫󣬷Ƶġٹ¼ĳ.
NFSSpriteȷֲʽļϵͳڴļʱѻݽЧ֤xFSṩϸĺʽһԣڿһͨƻάһ.NFSSpriteϵͳʹõĴȻ𡮼ٹ⣬ƵĲд(concurrent write-sharing)ǵܽἱ½.Ϊ˽⣬Spriteõķǣдʱص棨cache disablingNFSɴûṩϸһԱ֤.ȻڴȷļϵͳxFSϸһάһԽķغĳЩҪĿ.
ΪS2FS˫ȻΪ˿˷ȺϸȷиԵȱ.ͻһļʱûͻͻûвдÿͻһļصͻԳͻʽ򿪸ļʱ֪ͨͻոļĻص.ͻʧһļĻصôʹƻƲڿһļ(鼶Ļ״̬תͼ2ʾ)д𡮼ٹĿԽС.

(cache)״̬ת
12456ЧתЧ󣬻滻
3712д813ƽ1116д
910ļflushϵͳsync1415ת.
ͼ2ͻ˻״̬ͼ
Ƶ˫ЭļݿάһԵģS2FSͼļһлдʱʹÿһġĿ¼ЧʹЭ.ǸдĶ壺һļͬʱڶϱ򿪣һдʽ򿪵ģôļͳΪڡд״̬.ڿͻԹʽдݣϸȣݿ鼶һάһԻĹأЩǣS2FSʵ˫ȻһЭ.һͻһļҲڡд״̬ʱÿͻһļص(callback)ŵͻԳͻʽ򿪸ļʱ֪ͨÿͻøÿͻļص.ͻյһļصʱжϳʱĳЩͻԡдʽ򿪸ļӴڿ鼶ʹûƵķάһ.
S2FSֻصͶ/дص¼дص.һͻһļΪ޸һļһļ磬ļȡģʽO_WRONLYO_RDWRûСдʱÿͻһдصһͻO_RDONLYĴȡģʽһļûСдÿͻ͵õһص.ͼ3̻˿ͻļ5ص״̬Լǵת.һͻһļȴûеõκλصζŲĹдڽ֮.һͻӵдصƶϳĿǰΨһ˸ļĿͻ.

״̬ת(ġдζļ򿪵ĴȡģʽO_WRONLYO_RDWRζļĴȡģʽO_RDONLY)
1ļд򿪣26򿪣371011дرգ48رգ591221д򿪣1315ͻдرͬļ14ͻд()ͬһļ16ͻдͬһļ1719򿪣д򿪣رգдرգ1820򿪣ر.
ͼ3S2FSͻ˵ļ״̬תͼ
ʵЭĹ.Ƿ˫ЭڵЭ¼ôܹάһԶķأЩ£ܹͿͻ֮ͨ۸ΪҪǣܹṩĳЩаʾhintϢS2FSЩʾϢһ˵ĸԼͻ/翪.
3.2̷洢
ͳķֲʽļϵͳõһļӦпͻI/O󣬰ݼԪݵĴ洢͹һԵάȹ.ֵһϵṹеһôƺʵֵļ򵥣ȻԼϵͳĿչԺI/Oܣҵһγɵһ.Ϊ˿˷ͳļϵͳһȱ㣬S2FSʱ޼ʽȻϵͳдڶS2FSʵļ洢λ͸ԣûϵͳֻһĵһļ.xFSǽݵĴ洢ԪݵĹֿʵ֣רŵıΪ洢ģʵֲַʽĴ̴洢.ͬxFSͬǣS2FSĴ洢ֻ洢ļĿ¼ļݣԪ(ļϵͳĳڵ)ɹ洢.⣬S2FSǻAIXϵͳʵֵģAIXļϵͳJFSѶԪݽ־ʽṩ첽I/OܣS2FSRAID1洢ģͣڴСд⣬ûʵ־ʽ洢ܣһҲxFSͬ.
Ϊߴ洢ϵͳܣͱһͻԸߵĴ̴ȡļļͬʱܹЧشСļд.ϵͳӦݷֲڶĶ߲ж.ΪʵϵͳĿչĿ꣬ϵͳӦܹЧֳ֧ɰǧ̣ϵͳӦܹؿƴ̷飨diskstrippingĲж.̷P.ChenΪʵִģĴģ5ݣxFSļϵͳãչΪ洢4ݣҲΪS2FSϵͳʵ˴洢鹦ܣ֮Ϊ̷飬xFS洢ΪӣΪxFSϵͳĴ洢߼϶Ӧ1̣S2FSϵͳĴ洢ɸϵͳͬʱҶ̣ǽд洢ĻλǴ̶Ǵ洢Ҳǳ֮Ϊ̷ԭ.
ʵִ洢Ĺؼݽṹ̷ӳͼ4(a)ʾ˴Ӵ洢ʶһϵ̵ת.ͼ4(b)S2FSϵͳкͼ4(a)Ӧ̵߼ϵ.ȻΪS2FSĴ洢ģѡ˴ܵRAID1ṹͼ4ԿĿǰİ汾ûʵ־ܣͼ4ʾṹĻùܲ.
ĳͻյļĳ㷨תѡһɸù𴴽͹ļ.յļ󣬹תround-robinΪļѡһ洢źһ߼ʼ̺ţ洢ž˸ļӦĴ洢̣洢ʹ̶ԣ߼ʼ̺žļݴһ̿ʼ.ȻӦĴ洢ʹļɴ洢ļעS2FSϵͳļڴ洢JFSļʽڵ).S2FSϵͳڴϵλɴ洢ź߼ʼ̺.ͻ/дļʱͻȼ/дݵ߼ţݸļĴ洢ź߼ʼ̺ڵ߼̺ţȻ߼̺ڵĴ洢Ͷ/д󣬴洢ĸԼļϵƫִJFSϵͳɴ˴ζ/д.

ͼ4̷ӳϵ
3.3ֲʽԪݹ
ΪʵֵһϵͳӳS2FSһؼԭǱ֤λ͸ԣûùںʽ̷еλãȷϵͳݵλǷǳҪ.S2FSϵͳУɹɵģֲͨʽԪݹṩչġֲʽλ÷.⣬һάʽһ.
S2FSϵͳеÿֻάϵͳһļӼλϢ.ʹЩλϢܹѿͻI/OתȷĻ洢λãЭͻͬݵĴȡ.ϵͳͨʹùӳ(ͼ5)ʵֲַʽԪݹӳһʶָʾһһļϵͳ.ϵͳȰļΨһڵΪ룬ͨĳɢкõһ߼ţԸ߼Ϊ±ԴӹӳҳļӦĹӳṩһֳܣʵֹλö.

ͼ5ӳ
ZebraǽذݺͿ·ֿʵֵһļϵͳϵͳֻһ6.xFSΪ˱Zebraϵͳеһƿ⣬ϵͳԪݷֲ˶ϣϵͳĿչԣ4.ʵxFSϵͳ֮ǰģзֲʽڴϵͳѾֲַʽԣйػһԵԪݷֲڶ.S2FSԪݹxFSǳƵļߵľʵԴڲٲ.S2FSĴ洢ϵͳǻAIXļϵͳJFSҴ洢ĵλΪ̶Ǵ洢S2FSԪݵݺ֯xFSнϴinodeü¼ļÿݵĴλãá洢ʶʼ߼̺š.⣬S2FSxFSĻһЭ滻㷨ͬйػԪݼڽϴ.S2FSĹxFSĹһǲͬģǰ߳˶Ԫݽй⣬ԪݵĴ洢.


4ϵͳ
ǽS2FSNFSͬʵ黷нܱȽϣԼS2FSǷﵽǵĿ.ǵʵ黷6̨PowerPCվɣϷֱAIX4.1AIX4.2֮10Mbit/s̫.6У1̨ƵΪ100MHzΪ133MHz.ǰ1̨ƵΪ133MHzĻΪNFSķNFSĿͻҲѡʣƵΪ133MHzĻ.S2FSϵͳĹʹ洢ֲ6.
ͨ׼Ļ׼S2FSԭϵͳĶ/дļͿչ.ԿչԵĲǻAndrew׼ģڻ-÷¡ѧרڴӦóһļϵͳܵı׼Գ.5׶ΣֱӦڴĿ¼ļĿ¼״̬Ϣɨļļ.ļ/дĲԣǲBonnie׼.BonnieTimBrayдġרڲļϵͳ˳/дĳһϵеĲԣĵַдддȻдأĵַԼĶ.Bonnieнʱӡ׶ƽÿֽܹԼCPU.
4.1/д
ȲS2FSϵͳĶ/дNFS˶Ա.ǵʵУ/дļĴСΪ2Mֽڣÿζ/дݿСΪ8Kֽ.ѡһ̨133MHzΪNFSķҲS2FSԭϵͳĹʹ洢.ͬʱѡһ̨133MHzĽΪNFSS2FSĿͻˣBonnie׼.нͼ6ͼ7.

ͼ6Bonnieд

ͼ7BonnieĶ
ͼ6ԿS2FSдǴݵдʤNFSܶ.ͼ7˵S2FSַNFS飨8KֽڣܲNFSͬһ־ҪԭǣǵĲԻ10bpsַ̫ܳS2FSϵͳĺʽ桢ֲʽ洢ȼƣS2FSԭϵͳδдŻ.ڶS2FSдŻøܵS2FSϵͳݵܹﵽ򳬹NFS.
4.2I/Oչ
ڱʵУǷֱ6洢4ͻĿı仯Χ{1234}.ͼ8ʾS2FSNFSϵͳУͻ1ӵ4ʱƽÿAndrewʱ.ԿNFSS2FSϵͳAndrewʱƽΪƽ.ȻNFSͻҪԶԶS2FSҪS2FSĲͻDaemon֮ͨſɵģͻӵ4ʱS2FSѳNFS.˵S2FSϵͳĿչҪNFSͬʱҲ֤S2FSϵͳ޼ʽܹЧؿ˷ͳļϵͳĵһƿ.

ͼ8AndrewĿչԲ
5ᡡ
ͳķֲʽļϵͳڲ˼ʽķ׳ΪƿûгֿǼ㼼ķչƣֲʽϵͳI/Oƿ.Ĳúʽ桢зֲʽݴ洢Ԫݹȴʩڱ֤ϵͳһӳͶƼԵĻϣʺڻȺķֲʽļϵͳĿչԽо.ӦöI/OֹģI/OȡҲڲϷ仯㼼ϳֳµķչƣһжΪδ͵ķֲʽļϵͳ˸ս.
õҡƻĿ(Ŀ863-306-ZD01).
߼飺,У19695ʿ,ҪоΪֲʽϵͳ.
ף,У19455оԱ,ʿʦ,ҪоΪϵṹ
˹.
־ΰ,У19569оԱ,ʿʦ,ҪоΪܼϵ
ͳ.
ųڣУ19755˶ʿҪоΪֲʽϵͳ.
ߵλ£ѧѧ뼼ϵ100871
ף־ΰųڣܼо뿪ġ100080
йѧԺ㼼о100080
ο
1Sandberg R. The Sun network file system: Design, implementation and experience. In: Proceedings of USENIX Summer Conference, California: University of California Press, 1987. 300313
2Wang Jianyong, Zhu Mingfa, Xu Zhiwei. Cooperative cache management in S2FS. In: Proc of the 6th International Conference on Parallel and Distributed Processing Techniques and Applications. Las Vegas, 1999
3Leff A, Wolf J L, Yu P S. Replication algorithms in a remote caching architecture. IEEE Transactions on Parallel and Distributed Systems, 1993, 4(11): 11851204
4Anderson T E et al. Serverless network file systems. ACM Transactions on Computer Systems, 1996, 14(1): 4179
5Chen P, Lee E, Gibson G, Katz R et al. RAID: High-performance reliable secondary storage. ACM Computing Surveys, 1994, 26(2): 145188
6Hartman J H, Ousterhout J K. The zebra striped network file system. ACM Transactions on Computer Systems, 1995, 13(3): 274310
ԭյڣ1999-05-18޸ĸյڣ1999-07-20.
