RHEL6.2平臺(tái)利用KVM虛擬機(jī)搭建高可用負(fù)載均衡WEB集群(keepalived+lvs+iscsi+gfs)

asia_ji

RHEL6.2平臺(tái)利用KVM虛擬機(jī)搭建高可用負(fù)載均衡WEB集群(keepalived+lvs+iscsi+gfs)

最近學(xué)習(xí)KVM虛擬化及負(fù)載均衡集群技術(shù)，參考了很多帖子，在學(xué)習(xí)的過(guò)程中順便將練手的過(guò)程記錄如下，希望能給大家些許幫助！
keepalived+lvs+iscsi+gfs搭建高可用負(fù)載均衡web集群，主要參考了這個(gè)帖子：http://www.linuxidc.com/Linux/2011-12/48409.htm
也不知是不是原作者，謹(jǐn)此感謝！

一、環(huán)境搭建及虛擬機(jī)部署：

1、虛擬機(jī)宿主：
主機(jī)名                        IP地址
WEB                                172.16.9.1

RHEL6.2基本服務(wù)器和虛擬化



KVM虛擬機(jī)采用橋接方式安裝

網(wǎng)卡橋接：
[root@Web:/root]$cp /etc/sysconfig/network-scripts/ifcfg-eth0 /etc/sysconfig/network-scripts/ifcfg-br0

修改為如下即可：
[root@Web:/root]$more /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE="eth0"
NM_CONTROLLED="yes"
ONBOOT=yes
HWADDR=5C:F3:FC:94:88:EC
TYPE=Ethernet
BRIDGE=br0
BOOTPROTO=none
PREFIX=27
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="System eth0"
注：eth0不要設(shè)置IP

[root@Web:/root]$more /etc/sysconfig/network-scripts/ifcfg-br0
DEVICE="br0"
ONBOOT=yes
TYPE=Bridge
BOOTPROTO=none
IPADDR=172.16.9.2
GATEWAY=172.16.9.1
DNS1=219.149.6.99



2、虛擬機(jī)安裝：

安裝、克隆總計(jì)8臺(tái)虛擬機(jī)
虛擬機(jī)                                        主機(jī)名                                IP
LVS-MASTER                                LVS-MASTER                        172.16.9.10
LVS-BACKUP                                LVS-BACKUP                        172.16.9.11
Mysql-A                                        Mysql-A                                172.16.9.21
Mysql-B                                        Mysql-B                                172.16.9.22
Web-A                                        Web-A                                172.16.9.31
Web-B                                        Web-B                                172.16.9.32
Web-C                                        Web-C                                172.16.9.33
Firesystem                                Firesystem                        172.16.9.41

VIP IP                172.16.9.254

root@Web:[/root]~$virt-install \
--name Web-A \
--ram 2048 \
--vcpus=64 \                ##虛擬機(jī)CPU個(gè)數(shù)
--disk path=/var/lib/libvirt/images/Web-A.img,size=11 \
--network bridge=br0 \
--os-variant=rhel6 \
--cdrom /var/lib/libvirt/rhel-server-6.2-x86_64-dvd.iso \
--vnclisten=172.16.9.1 \
--vncport=6901 \
--vnc

安裝過(guò)程選RHEL6.2基本服務(wù)器即可

3、克隆虛擬機(jī)：

暫停虛擬機(jī)以便克隆
root@Web:[/root]~$virsh suspend Web-A
開(kāi)始克隆
root@Web:[/root]~$virt-clone --connect qemu:///system --original=Web-A --name=Web-B --file=/var/lib/libvirt/images/Web-B.img
root@Web:[/root]~$virt-clone --connect qemu:///system --original=Web-A --name=Web-C --file=/var/lib/libvirt/images/Web-C.img
root@Web:[/root]~$virt-clone --connect qemu:///system --original=Web-A --name=Mysql-A--file=/var/lib/libvirt/images/Mysql-A.img
root@Web:[/root]~$virt-clone --connect qemu:///system --original=Web-A --name=Mysql-B --file=/var/lib/libvirt/images/Mysql-B.img
root@Web:[/root]~$virt-clone --connect qemu:///system --original=Web-A --name=Firesystem --file=/var/lib/libvirt/images/Firesystem.img
root@Web:[/root]~$virt-clone --connect qemu:///system --original=Web-A --name=LVS-MASTER --file=/var/lib/libvirt/images/LVS-MASTER.img
root@Web:[/root]~$virt-clone --connect qemu:///system --original=Web-A --name=LVS-BACKUP --file=/var/lib/libvirt/images/LVS-BACKUP.img

恢復(fù)虛擬機(jī)
root@Web:[/root]~$virsh resume Web-A


如需強(qiáng)制關(guān)閉虛擬機(jī)，如下：
root@Web:[/etc/libvirt/qemu]~$virsh destroy  Web-A


虛擬機(jī)克隆完畢后，需修改vnc端口
root@Web:[/root]~$vi /etc/libvirt/qemu/Web-B.xml

<graphics type='vnc' port='6902' autoport='no' listen='172.16.9.32'>
      <listen type='address' address='172.16.9.32'/>
    </graphics>

所有克隆虛擬機(jī)的VNC端口修改完畢后，從啟libvirtd服務(wù)

從啟libvirtd服務(wù)
root@Web:[/root]~$service libvirtd restart
Stopping libvirtd daemon: [  OK  ]
Starting libvirtd daemon: 15:06:04.222: 51157: info : libvirt version: 0.9.4, package: 23.el6_2.4 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2012-01-06-02:50:25, x86-007.build.bos.redhat.com)
15:06:04.222: 51157: warning : virGetHostname:1957 : getaddrinfo failed for 'Web': Name or service not known
[  OK  ]

出現(xiàn)此警告warning : virGetHostname:1957 : getaddrinfo failed for 'Web': Name or service not known
解決方法：
root@Web:[/root]~$vi /etc/hosts
添加
172.16.9.2                Web


root@Web:[/root]~$service libvirtd restart
Stopping libvirtd daemon: [  OK  ]
Starting libvirtd daemon: [  OK  ]



啟動(dòng)全部虛擬機(jī)
root@Web:[/root]~$virsh start XXX

root@Web:[/root]~$virsh list
Id Name                 State
----------------------------------
  1 LVS-MASTER               running
  2 LVS-BACKUP               running
  3 Filesystem                      running
  4 Web-A                                running
  5 Web-A                           running
  6 Web-C                           running
  7 Mysql-A                          running
  8 Mysql-B                         running

虛擬機(jī)啟動(dòng)后網(wǎng)卡無(wú)法正常工作，解決方法如下：

kvm虛擬機(jī)克隆后網(wǎng)卡無(wú)法啟動(dòng)解決辦法：
1 修改/etc/udev/rules.d/70-persistent-net.rules文件，刪除第一個(gè)網(wǎng)卡記錄，并將第二個(gè)的NAME="eth1"改為NAME="eth0"
2 如果在/etc/sysconfig/network-scripts/ifcfg-eth0中有配置網(wǎng)卡信息的話，如：#HWADDR="00:0C:29:C8:1A:92"，將其注釋。


順便修改主機(jī)名和ip地址
修改主機(jī)名：
root@Web-A:[/root]~$vi /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=LVS-BACKUP
GATEWAY=172.16.9.1

更新系統(tǒng)時(shí)間

yum -y install  ntp
ntpdate time.nist.gov
echo "00 01 * * * /usr/sbin/ntpdate time.nist.gov"  > /etc/crontab


重啟系統(tǒng)


全部虛擬機(jī)準(zhǔn)備好后，開(kāi)始搭建高可用負(fù)載均衡WEB集群！



二、keepalived+lvs搭建高可用負(fù)載均衡WEB集群

rhel6.2系統(tǒng)盤(pán)里有ipvsadm-1.25-9.el6.x86_64.rpm
keepalived-1.2.2.tar.gz需要下載http://www.keepalived.org



1、LVS-MASTER及LVS-BACKUP安裝步驟：

登錄LVS-MASTER
root@LVS-MASTER:[/root]~$rpm -ivh ipvsadm-1.25-9.el6.x86_64.rpm
root@LVS-MASTER:[/root]~$yum -y install kernel-devel
root@LVS-BACKUP:[/root]~$yum -y groupinstall "Development tools"
root@LVS-MASTER:[/root]~$tar xzvf keepalived-1.2.2.tar.gz
root@LVS-MASTER:[/root]~$cd keepalived-1.2.2
root@LVS-MASTER:[/root/keepalived-1.2.2]~$./configure --prefix=/usr/local/keepalived --with-kernel-dir=/usr/src/kernels/2.6.32-220.el6.x86_64/

出錯(cuò)：
configure: error:
  !!! OpenSSL is not properly installed on your system. !!!
  !!! Can not include OpenSSL headers files.            !!!
解決方法：
root@LVS-MASTER:[/root/keepalived-1.2.2]~$yum -y install openssl-devel

出錯(cuò)：
checking for poptGetContext in -lpopt... no
configure: error: Popt libraries is required
解決方法：
root@LVS-MASTER:[/root/keepalived-1.2.2]~$yum -y install popt-devel

再次運(yùn)行
root@LVS-MASTER:[/root/keepalived-1.2.2]~$./configure -- prefix=/usr/local/keepalived --with-kernel-dir=/usr/src/kernels/2.6.32-220.el6.x86_64/
完畢后出現(xiàn)如下信息：
############################################
Keepalived configuration
------------------------
Keepalived version       : 1.2.2
Compiler                 : gcc
Compiler flags           : -g -O2
Extra Lib                : -lpopt -lssl -lcrypto
Use IPVS Framework       : Yes
IPVS sync daemon support : Yes
IPVS use libnl           : No
Use VRRP Framework       : Yes
Use Debug flags          : No
############################################



root@LVS-MASTER:[/root/keepalived-1.2.2]~$make
root@LVS-MASTER:[/root/keepalived-1.2.2]~$make install
############################################
make -C keepalived install
make[1]: Entering directory `/root/keepalived-1.2.2/keepalived'
install -d /usr/local/keepalived/sbin
install -m 700 ../bin/keepalived /usr/local/keepalived/sbin/
install -d /usr/local/keepalived/etc/rc.d/init.d
install -m 755 etc/init.d/keepalived.init /usr/local/keepalived/etc/rc.d/init.d/keepalived
install -d /usr/local/keepalived/etc/sysconfig
install -m 755 etc/init.d/keepalived.sysconfig /usr/local/keepalived/etc/sysconfig/keepalived
install -d /usr/local/keepalived/etc/keepalived/samples
install -m 644 etc/keepalived/keepalived.conf /usr/local/keepalived/etc/keepalived/
install -m 644 ../doc/samples/* /usr/local/keepalived/etc/keepalived/samples/
install -d /usr/local/keepalived/share/man/man5
install -d /usr/local/keepalived/share/man/man8
install -m 644 ../doc/man/man5/keepalived.conf.5 /usr/local/keepalived/share/man/man5
install -m 644 ../doc/man/man8/keepalived.8 /usr/local/keepalived/share/man/man8
make[1]: Leaving directory `/root/keepalived-1.2.2/keepalived'
make -C genhash install
make[1]: Entering directory `/root/keepalived-1.2.2/genhash'
install -d /usr/local/keepalived/bin
install -m 755 ../bin/genhash /usr/local/keepalived/bin/
install -d /usr/local/keepalived/share/man/man1
install -m 644 ../doc/man/man1/genhash.1 /usr/local/keepalived/share/man/man1
make[1]: Leaving directory `/root/keepalived-1.2.2/genhash'
############################################
安裝文件路徑


2、LVS-MASTER及LVS-BACKUP主機(jī)配置keepalived

(1)、LVS-MASTER

root@LVS-MASTER:[/root]~$mkdir /etc/keepalived
root@LVS-MASTER:[/root]~$ln -s /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
root@LVS-MASTER:[/root]~$ln -s /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
root@LVS-MASTER:[/root]~$ln -s /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
root@LVS-MASTER:[/root]~$ln -s /usr/local/keepalived/bin/* /bin/
root@LVS-MASTER:[/root]~$ln -s /usr/local/keepalived/sbin/* /sbin/
root@LVS-BACKUP:[/root]~$modprobe ip_vs
root@LVS-MASTER:[/root]~$vi /etc/rc.local
############################################
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local
modprobe ip_vs
############################################


LVS-MASTER主機(jī)keepalived.conf文件設(shè)置：
root@LVS-MASTER:[/root]~$vi /etc/keepalived/keepalived.conf
############################################
! Configuration File for keepalived

global_defs {
   notification_email {
   root@localhost         
   }
   notification_email_from root@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        172.16.9.254
    }
}

virtual_server 172.16.9.254 80 {
    delay_loop 6
    lb_algo rr
    lb_kind DR
#    nat_mask 255.255.255.0
    persistence_timeout 50
    protocol TCP

    real_server 172.16.9.31 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }
    real_server 172.16.9.32 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }
    real_server 172.16.9.33 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }
}
############################################

啟動(dòng)keepalived
root@LVS-MASTER:[/root]~$service keepalived restart
停止 keepalived：[失敗]
正在啟動(dòng) keepalived：[確定]

root@LVS-MASTER:[/root]~$chkconfig --level 2345 keepalived on


(2)、LVS-BACKUP

LVS-BACKUP主機(jī)keepalived.conf文件設(shè)置：
root@LVS-BACKUP:[/root]~$vi /etc/keepalived/keepalived.conf
############################################
root@LVS-BACKUP:[/root]~$more /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
   root@localhost         
   }
   notification_email_from root@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 50
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        172.16.9.254
    }
}

virtual_server 172.16.9.254 80 {
    delay_loop 6
    lb_algo rr
    lb_kind DR
#    nat_mask 255.255.255.0
    persistence_timeout 50
    protocol TCP

    real_server 172.16.9.31 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }
    real_server 172.16.9.32 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }
    real_server 172.16.9.33 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }
}
############################################

啟動(dòng)keepalived
root@LVS-BACKUP:[/root]~$service keepalived restart
停止 keepalived：[失敗]
正在啟動(dòng) keepalived：[確定]

root@LVS-BACKUP:[/root]~$chkconfig --level 2345 keepalived on



3、REAL_SERVER（WEB-A、WEB-B、WEB-C）做如下設(shè)置：

分別在 REAL_SERVER（WEB-A、WEB-B、WEB-C）上執(zhí)行以下命令:

[root@Test-Web-B ~]# yum groupremove "MySQL Database client" "MySQL Database server"                ##原來(lái)安裝了mysql，卸載掉


安裝web服務(wù)及php
[root@Web-A ~]# yum -y groupinstall "Web Server" "PHP Support"

[root@Web-A ~]# vi /etc/sysctl.conf
################################################
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2
################################################

[root@Web-A ~]# sysctl -p

[root@Web-A ~]# ifconfig eth0:0 172.16.9.254 netmask 255.255.255.255 up
[root@Web-A ~]# route add -host 172.16.9.254 dev eth0:0
[root@Web-A ~]# echo `hostname` > /var/www/html/index.html
[root@Web-C ~]# vi /etc/httpd/conf/httpd.conf
修改
ServerName
[root@Web-A ~]# service httpd start




4、測(cè)試：

(1)
把所有三個(gè)WEB服務(wù)全停掉
[root@Test-Web-A ~]service httpd stop
[root@Test-Web-B ~]service httpd stop
[root@Test-Web-C ~]service httpd stop

訪問(wèn)VIP
root@Web:[/root]~$lynx 172.16.9.254

Alert!: Unable to connect to remote host.


Looking up 172.16.9.254 first
Looking up 172.16.9.254
Making HTTP connection to 172.16.9.254
Alert!: Unable to connect to remote host.

lynx: Can't access startfile http://172.16.9.254/

(2)
開(kāi)啟Test-Web-A服務(wù)：
[root@Test-Web-A ~]service httpd start
root@Web:[/root]~$lynx 172.16.9.254
顯示頁(yè)面如下：
Test-Web-A

(3)
停掉Test-Web-A服務(wù)，啟動(dòng)Test-Web-B服務(wù)：
[root@Test-Web-A ~]service httpd stop
[root@Test-Web-B ~]service httpd start


root@Web:[/root]~$lynx 172.16.9.254
刷新幾次后，顯示頁(yè)面如下：
Test-Web-B

(4)
停掉Test-Web-B服務(wù)，啟動(dòng)Test-Web-C服務(wù)：
[root@Test-Web-B ~]service httpd stop
[root@Test-Web-C ~]service httpd start


root@Web:[/root]~$lynx 172.16.9.254
刷新幾次后，顯示頁(yè)面如下：
Test-Web-C

(5)
把所有三個(gè)WEB服務(wù)全啟動(dòng)
[root@Web-A ~]service httpd start
[root@Web-B ~]service httpd start
[root@Web-C ~]service httpd start

分別用不同ip的主機(jī)訪問(wèn)會(huì)發(fā)現(xiàn)顯示不同的頁(yè)面



今天試驗(yàn)到此結(jié)束，接下來(lái)是mysql雙主高可用服務(wù)器的搭建及iscsi+gfs文件系統(tǒng)的搭建。

明天繼續(xù)…………

niao5929

RHEL6.2的鏡像難找。呵呵�；仡^在DEBIAN6。0。4上實(shí)驗(yàn)一下。呵呵

chenzhj

你的這個(gè)只是一個(gè)實(shí)驗(yàn)環(huán)境中。

Mysql2009

非常不錯(cuò)的文章，幫頂一下了！

zjpixyniannian

頂~~~~~~學(xué)習(xí)一下。。。下周我也實(shí)踐一把。。

zhengwei_zw

非常給力哈。支持！

whenmind

mark

hellogzd

問(wèn)題是這樣的，我一共用了三臺(tái)服務(wù)器，其中一臺(tái)是Direct Server，其余兩臺(tái)是Real server。按照網(wǎng)上介紹的方法，進(jìn)行配置。結(jié)果最后訪問(wèn)http://虛擬IP 總是出現(xiàn)網(wǎng)頁(yè)無(wú)法訪問(wèn)的情況，不能轉(zhuǎn)到實(shí)際的服務(wù)器上。而通過(guò)ipvsadm命令卻可以看到http訪問(wèn)已經(jīng)平均分發(fā)到兩臺(tái)服務(wù)器上了，即inActConn數(shù)隨著訪問(wèn)頁(yè)面的增多，也增多。
請(qǐng)各位大俠給支個(gè)招。