基于Ceph對(duì)象存儲(chǔ)集群的負(fù)載均衡設(shè)計(jì)與實(shí)現(xiàn)①

楊飛,朱志祥,梁小江(西安郵電大學(xué),西安 7006)(陜西省信息化工程研究院,西安 7006)

?

基于Ceph對(duì)象存儲(chǔ)集群的負(fù)載均衡設(shè)計(jì)與實(shí)現(xiàn)①

楊飛1,朱志祥2,梁小江2
1(西安郵電大學(xué),西安 710061)
2(陜西省信息化工程研究院,西安 710061)

摘 要:海量并行數(shù)據(jù)訪問(wèn)ceph對(duì)象存儲(chǔ)集群時(shí),會(huì)出現(xiàn)訪問(wèn)數(shù)據(jù)錯(cuò)誤率增加和訪問(wèn)速率降低的問(wèn)題.首先設(shè)計(jì)和部署ceph對(duì)象存儲(chǔ)集群,根據(jù)用戶請(qǐng)求數(shù)設(shè)計(jì)對(duì)象網(wǎng)關(guān)節(jié)點(diǎn),實(shí)現(xiàn)用戶數(shù)據(jù)的交互功能,然后在服務(wù)節(jié)點(diǎn)安裝和部署haproxy,實(shí)現(xiàn)服務(wù)節(jié)點(diǎn)的負(fù)載均衡功能,能夠降低服務(wù)節(jié)點(diǎn)的壓力,最后設(shè)計(jì)和實(shí)現(xiàn)四種基于ceph對(duì)象網(wǎng)關(guān)節(jié)點(diǎn)的存儲(chǔ)集群.通過(guò)大量對(duì)比測(cè)試,本設(shè)計(jì)方案的數(shù)據(jù)訪問(wèn)錯(cuò)誤率降低0.96%,用戶數(shù)據(jù)的訪問(wèn)速率提升74.04%.

關(guān)鍵詞:Ceph; 對(duì)象網(wǎng)關(guān); 負(fù)載均衡; 錯(cuò)誤率; 訪問(wèn)速率

1　引言

隨著云計(jì)算與大數(shù)據(jù)的不斷發(fā)展,對(duì)于網(wǎng)絡(luò)數(shù)據(jù)的存儲(chǔ)和處理能力提出新的要求,當(dāng)前的數(shù)據(jù)存儲(chǔ)系統(tǒng)已無(wú)法滿足海量增長(zhǎng)的網(wǎng)絡(luò)數(shù)據(jù)[1].Ceph分布式文件系統(tǒng)集群能夠根據(jù)海量數(shù)據(jù)的增長(zhǎng)而進(jìn)行集群擴(kuò)展,有效降低服務(wù)器并行訪問(wèn)的壓力,提高數(shù)據(jù)訪問(wèn)的速率,降低數(shù)據(jù)訪問(wèn)的錯(cuò)誤率[2].

本文以ceph分布式文件系統(tǒng)為研究對(duì)象,首先設(shè)計(jì)部署ceph存儲(chǔ)集群[3],結(jié)合mysql與keystone實(shí)現(xiàn)ceph對(duì)象網(wǎng)關(guān)節(jié)點(diǎn)的統(tǒng)一認(rèn)證.然后設(shè)計(jì)ceph對(duì)象存儲(chǔ)集群的對(duì)外網(wǎng)關(guān)功能,實(shí)現(xiàn)服務(wù)器與ceph對(duì)象存儲(chǔ)集群的數(shù)據(jù)交互,能夠根據(jù)訪問(wèn)服務(wù)器用戶量的增長(zhǎng),而擴(kuò)展ceph對(duì)象存儲(chǔ)集群的對(duì)外網(wǎng)關(guān)節(jié)點(diǎn),實(shí)現(xiàn)ceph對(duì)象存儲(chǔ)集群的多區(qū)域數(shù)據(jù)管理,通過(guò)負(fù)載均衡降低服務(wù)器并行數(shù)據(jù)的壓力,從而提升服務(wù)器并行數(shù)據(jù)訪問(wèn)速率,同時(shí)能夠降低用戶訪問(wèn)數(shù)據(jù)的錯(cuò)誤率[4].

2　整體設(shè)計(jì)框架

2.1設(shè)計(jì)部署

本設(shè)計(jì)方案中包括11個(gè)節(jié)點(diǎn)服務(wù)器: east、west、south、north為ceph對(duì)象存儲(chǔ)集群的對(duì)外網(wǎng)關(guān)節(jié)點(diǎn),通過(guò)haproxy負(fù)載均衡節(jié)點(diǎn)對(duì)east、west、south、north四個(gè)對(duì)象網(wǎng)關(guān)節(jié)點(diǎn)進(jìn)行負(fù)載均衡,node1、node2、node3、node4是ceph對(duì)象存儲(chǔ)集群的存儲(chǔ)和元數(shù)據(jù)節(jié)點(diǎn),而deploy節(jié)點(diǎn)為ceph集群部署節(jié)點(diǎn),DNS節(jié)點(diǎn)為ceph對(duì)象存儲(chǔ)集群的統(tǒng)一認(rèn)證中心,haproxy節(jié)點(diǎn)為負(fù)載均衡節(jié)點(diǎn),采用輪詢的方式減輕用戶訪問(wèn)對(duì)象網(wǎng)關(guān)節(jié)點(diǎn)的壓力.表1為設(shè)計(jì)部署說(shuō)明表.

表1　設(shè)計(jì)部署說(shuō)明表

2.2設(shè)計(jì)框架

并行用戶數(shù)據(jù)通過(guò)ceph對(duì)象存儲(chǔ)集群的haproxy節(jié)點(diǎn)將海量并行數(shù)據(jù)輪詢分發(fā)到east、west、south、north四個(gè)ceph對(duì)象網(wǎng)關(guān)節(jié)點(diǎn),實(shí)現(xiàn)并行數(shù)據(jù)的負(fù)載均衡[5].在四個(gè)對(duì)象網(wǎng)關(guān)節(jié)點(diǎn)中設(shè)計(jì)和部署mysql和keystone實(shí)現(xiàn)ceph對(duì)象存儲(chǔ)集群的統(tǒng)一認(rèn)證[6].用戶可以對(duì)ceph對(duì)象存儲(chǔ)集群進(jìn)行數(shù)據(jù)操作和管理.圖1為整體設(shè)計(jì)框架.

圖1　整體設(shè)計(jì)框架

在east、west、south、north對(duì)象網(wǎng)關(guān)節(jié)點(diǎn)通過(guò)統(tǒng)一認(rèn)證后,在haproxy節(jié)點(diǎn)通過(guò)負(fù)載均衡降低服務(wù)器并行數(shù)據(jù)的壓力,從而有效地提升服務(wù)器并行數(shù)據(jù)訪問(wèn)速率,同時(shí)能夠降低用戶訪問(wèn)數(shù)據(jù)的錯(cuò)誤率.

3　統(tǒng)一認(rèn)證和負(fù)載均衡

3.1統(tǒng)一認(rèn)證

首先在DNS、east、west、south、north節(jié)點(diǎn)上面安裝配置mysql.修改mysql配置文件,允許客戶端的訪問(wèn)mysql數(shù)據(jù)庫(kù),然后在mysql中創(chuàng)建keystone數(shù)據(jù)庫(kù),安裝配置keystone,對(duì)keystone進(jìn)行初始化,并重啟mysql和keystone的服務(wù).

修改每個(gè)對(duì)象網(wǎng)關(guān)節(jié)點(diǎn)的ceph.conf配置文件,添加keystorgw_keystone_url = 10.10.10.61:5000 .這樣對(duì)象網(wǎng)關(guān)節(jié)點(diǎn)可以使用DNS節(jié)點(diǎn)產(chǎn)生的統(tǒng)一token,實(shí)現(xiàn)了所有對(duì)象網(wǎng)關(guān)節(jié)點(diǎn)的統(tǒng)一認(rèn)證.

首先在haproxy節(jié)點(diǎn)啟動(dòng)負(fù)載均衡的服務(wù),然后在DNS節(jié)點(diǎn)產(chǎn)生所有對(duì)象網(wǎng)關(guān)節(jié)點(diǎn)的統(tǒng)一認(rèn)證的token.

在DNS節(jié)點(diǎn)輸入命令:DNScurl -d '{"auth": {"tenantName": "admin","passwordCredentials":{"uS ername": "admin","password":"hastexo"}}}' –H "Con tent -type:application/json" http://10.10.10.61:5000/v2 .0/tokens | Python –mjson.tool

在east、west、south、north對(duì)象網(wǎng)關(guān)節(jié)點(diǎn)使用命令進(jìn)行統(tǒng)一認(rèn)證:

3.2負(fù)載均衡

在haproxy節(jié)點(diǎn)設(shè)置和部署ceph對(duì)象存儲(chǔ)集群的負(fù)載均衡服務(wù).配置和修改haproxy.cfg文件:

負(fù)載均衡節(jié)點(diǎn)的IP:10.10.10.71,負(fù)載均衡服務(wù)端口為8888.監(jiān)控界面的端口為8899,所有對(duì)象網(wǎng)關(guān)節(jié)點(diǎn)的操作都會(huì)通過(guò)10.10.10.71:8888進(jìn)行均衡均衡.圖2為負(fù)載均衡登陸界面[7].

圖2　負(fù)載均衡登錄界面

在ceph對(duì)象存儲(chǔ)集群中,設(shè)計(jì)和部署四種不同的ceph對(duì)象存儲(chǔ)架構(gòu)圖.在對(duì)象網(wǎng)關(guān)節(jié)點(diǎn)設(shè)計(jì)四種不同的負(fù)載均衡模式,分別為表2單個(gè)對(duì)象網(wǎng)關(guān)、表3兩個(gè)對(duì)象網(wǎng)關(guān)、表4三個(gè)對(duì)象網(wǎng)關(guān)、表5四個(gè)對(duì)象網(wǎng)關(guān).

表2　單個(gè)對(duì)象網(wǎng)關(guān)

表3　兩個(gè)對(duì)象網(wǎng)關(guān)

表4　三個(gè)對(duì)象網(wǎng)關(guān)

表5　四個(gè)對(duì)象網(wǎng)關(guān)

在haproxy負(fù)載均衡節(jié)點(diǎn)上配置haproxy.cfg,然后啟動(dòng)haproxy的服務(wù),在負(fù)載均衡監(jiān)控界面可以對(duì)不同設(shè)置的對(duì)象網(wǎng)關(guān)節(jié)點(diǎn)進(jìn)行監(jiān)控.

4　應(yīng)用與測(cè)試

4.1測(cè)試數(shù)據(jù)

本方案使用webbench并行測(cè)試軟件對(duì)整個(gè)ceph對(duì)象存儲(chǔ)集群進(jìn)行壓力測(cè)試[8],測(cè)試時(shí)間為120s,通過(guò)大量對(duì)比測(cè)試,能夠得出海量并行數(shù)據(jù)訪問(wèn)服務(wù)器的失敗率和數(shù)據(jù)速率.表6為單個(gè)對(duì)象網(wǎng)關(guān)測(cè)試數(shù)據(jù)、表7為兩個(gè)對(duì)象網(wǎng)關(guān)測(cè)試數(shù)據(jù)、表8為三個(gè)對(duì)象網(wǎng)關(guān)測(cè)試數(shù)據(jù)、表9為四個(gè)對(duì)象網(wǎng)關(guān)測(cè)試數(shù)據(jù).

表7　兩個(gè)對(duì)象網(wǎng)關(guān)測(cè)試數(shù)據(jù)

表8　三個(gè)對(duì)象網(wǎng)關(guān)測(cè)試數(shù)據(jù)

表9　四個(gè)對(duì)象網(wǎng)關(guān)測(cè)試數(shù)據(jù)

4.2分析結(jié)果

對(duì)測(cè)試的數(shù)據(jù)結(jié)果進(jìn)行分析和計(jì)算,從而清晰的得出負(fù)載均衡對(duì)整個(gè)ceph對(duì)象存儲(chǔ)集群的性能優(yōu)化情況,包括并行訪問(wèn)數(shù)據(jù)的速率和訪問(wèn)數(shù)據(jù)的錯(cuò)誤率.圖3為數(shù)據(jù)訪問(wèn)速率測(cè)試對(duì)比圖,圖4為數(shù)據(jù)錯(cuò)誤率測(cè)試對(duì)比圖.

圖3　數(shù)據(jù)訪問(wèn)速率測(cè)試對(duì)比圖

圖4　數(shù)據(jù)錯(cuò)誤率測(cè)試對(duì)比圖

5　總結(jié)

設(shè)計(jì)和部署ceph存儲(chǔ)集群,本設(shè)計(jì)方案能夠根據(jù)用戶的訪問(wèn)請(qǐng)求數(shù),動(dòng)態(tài)地設(shè)計(jì)和實(shí)現(xiàn)基于ceph對(duì)象存儲(chǔ)集群的對(duì)象網(wǎng)關(guān)節(jié)點(diǎn),從而不斷適應(yīng)網(wǎng)絡(luò)數(shù)據(jù)的快速增長(zhǎng).同時(shí)在不同的對(duì)象網(wǎng)關(guān)節(jié)點(diǎn)采用統(tǒng)一的認(rèn)證中心,確保用戶數(shù)據(jù)的安全性和封閉性.

通過(guò)負(fù)載均衡和統(tǒng)一認(rèn)證測(cè)試,本設(shè)計(jì)能夠有效的降低海量并行數(shù)據(jù)訪問(wèn)服務(wù)器的壓力,同時(shí)保證用戶數(shù)據(jù)的安全性.并行訪問(wèn)速率提升74.04%,訪問(wèn)數(shù)據(jù)的錯(cuò)誤率降低0.96%.

參考文獻(xiàn)

1吳廣君,王樹(shù)鵬,陳明,等.海量結(jié)構(gòu)化數(shù)據(jù)存儲(chǔ)檢索系統(tǒng).計(jì)算機(jī)研究與發(fā)展,2012,49.

2馮幼樂(lè),朱六璋.CEPH動(dòng)態(tài)元數(shù)據(jù)管理方法分析與改進(jìn).電子技術(shù),2010,47(9).

3李翔,李青山,魏彬.Ceph分布式文件系統(tǒng)的究及性能測(cè)試.2014(5):1–15

4Weil SA,Brandt SA,Miller EL,et al.Ceph: A scalable,high-performance distributed file system.Proc.of the 7th Symposium on Operating Systems Design and Implementation (OSDI).2006.307–320.

5鄒仁明,彭雋,李軍.OpenStack開(kāi)源云平臺(tái)高可用架構(gòu)的設(shè)計(jì)與實(shí)現(xiàn).中國(guó)計(jì)算機(jī)用戶協(xié)會(huì)網(wǎng)絡(luò)應(yīng)用分會(huì)2014年第十八屆網(wǎng)絡(luò)新技術(shù)與應(yīng)用年會(huì)2014.

6Tang B,Sandhu R.Extending openStack access control with domain trust.Network and System Security.Springer International Publishing,2014: 54–69.

7Liu K.To achieve load-balance of elective system with haproxy.Computer Knowledge & Technology,2011.

8Yan CR,Shen JY,Peng QK,et al.A throughput-driven scheduling algorithm of differentiated service for web cluster.Wuhan University Journal of Natural Sciences,2006,11(1): 88–92.

Design and Implementation of Load Balancing Based on Ceph Object Storage Cluster

YANG Fei1,ZHU Zhi-Xiang2,LIANG Xiao-Jiang2
1(Xi’an University of Posts and Telecommunications,Xi’an 710061,China)2(Shaanxi Information Engineering Research Institute,Xi’an 710061,China)

Abstract:When mass parallel data access the ceph object storage cluster,the problems of increasing error rate and low access rate are existed.At the first,the authors design and deploy the ceph object storage cluster,and then design object gateway nodes to achieve interactive features of user data according to user requests.Second,I install and deploy haproxy on four service nodes,which can achieve the function of load balancing and reduce the pressure of four service nodes.Finally,the authors design and implementation of four different kinds of storage clusters based on ceph object gateway nodes.Through a lot of comparison tests,this design can not only reduce the error rate of 0.96 percent,but also enhance access rate of 74.04 percent.

Key words:ceph; object gateway; load balance; error rate; access rate

收稿時(shí)間:①2015-08-14;收到修改稿時(shí)間:2015-09-17