阿里云-云小站(无限量代金券发放中)
【腾讯云】云服务器、云数据库、COS、CDN、短信等热卖云产品特惠抢购

Keepalived实现HAproxy高可用详解

155次阅读
没有评论

共计 13652 个字符,预计需要花费 35 分钟才能阅读完成。

一,keepalived 介绍

keepalived 是一个可以实现某些资源高可用的开源软件,其主要的组件包括 core,check,vrrp,libipfwc,libipvs,这里说下各个组件的功能。

core:keepalived 的核心组件,负责主进程的启动和维护以及加载解析配置文件等。

check:负责 healthchecker, 负责各种健康检查方式,和对应的配置解析以及 LVS 的配置解析。

vrrp:vrrpd 的子进程。

libipfwc:结合 iptables 的 ipchains 库来使用。

libipvs:结合 LVS 使用。

keepalived 启动后会生成 3 个进程,master 主进程,VRRP 子进程,healthchecker 子进程。

VRRP 协议是实现 keepalived 高可用的一个基础,下面说一下 VRRP 的实现原理:

VRRP 虚拟路由 (VRRP router),VRRP 是一个“选举”协议,它能够动态地将一个虚拟路由器的责任指定至同一个 VRRP 组中的其它路由器上,VRRP 的优势:

冗余:可以使用多个路由器设备作为 LAN 客户端的默认网关,大大降低了默认网关成为单点故障的可能性;

负载共享:允许来自 LAN 客户端的流量由多个路由器设备所共享;

多 VRRP 组:在一个路由器物理接口上可配置多达 255 个 VRRP 组;

多 IP 地址:基于接口别名在同一个物理接口上配置多个 IP 地址,从而支持在同一个物理接口上接入多个子网;

抢占:在 master 故障时允许优先级更高的 backup 成为 master;

通告协议:使用 IANA 所指定的组播地址 224.0.0.18 进行 VRRP 通告;

VRRP 追踪:基于接口状态来改变其 VRRP 优先级来确定最佳的 VRRP 路由器成为 master;

二,实验环境:

192.168.30.116 OS:CentOS 6.4 x86_64  master.luojianlong.com

192.168.30.117 OS:Centos 6.4 x86_64  backup.luojianlong.com

keepalived 版本:keepalived-1.2.7

首先,分别在 2 台服务器上,安装 keepalived,haproxy,由于系统版本是 Centos 6.4,这 2 个软件已经被整合在内部了,所以使用 yum 来安装

[root@master ~]# yum -y install keepalived haproxy

[root@backup ~]# yum -y install keepalived haproxy

接下来,先配置 master 与 backup 服务器的优先级,virtual_router_id(同一个实例 2 台服务器必须相同),编辑配置文件:

[root@master ~]# cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak

[root@backup ~]# cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak

[root@master ~]# vi /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {

  notification_email {

    acassen@firewall.loc

    failover@firewall.loc

    sysadmin@firewall.loc

  }

  notification_email_from Alexandre.Cassen@firewall.loc

  smtp_server 192.168.200.1

  smtp_connect_timeout 30

  router_id LVS_DEVEL

}

vrrp_instance VI_1 {

    state MASTER

    interface eth0

    virtual_router_id 51

    priority 100

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1111

    }

    virtual_ipaddress {

        192.168.30.230

    }

}

[root@backup ~]# vi /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {

  notification_email {

    acassen@firewall.loc

    failover@firewall.loc

    sysadmin@firewall.loc

  }

  notification_email_from Alexandre.Cassen@firewall.loc

  smtp_server 192.168.200.1

  smtp_connect_timeout 30

  router_id LVS_DEVEL

}

vrrp_instance VI_1 {

    state BACKUP

    interface eth0

    virtual_router_id 51

    priority 99

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1111

    }

    virtual_ipaddress {

        192.168.30.230

    }

}

 

 

global_defs:全局配置标识破;

notification_email:表示告警时发送的邮件地址;

notification_email_from:表示发送邮件的源地址;

smtp_server:发送邮件的 smtp 服务器地址;

router_id:机器标识;

vrrp_instance:定义一个 vrrp 实例;

state:state 指定 instance(Initial) 的初始状态,就是说在配置好后,这台服务器的初始状态就是这里指定的,但这里指定的不算,还是得要通过竞选通过优先级来确定,里如果这里设置为 master,但如若他的优先级不及另外一台,那么这台在发送通告时,会发送自己的优先级,另外一台发现优先级不如自己的高,那么他会就回抢占为 master;

interface:实例绑定的网卡,因为在配置虚拟 IP 的时候必须是在已有的网卡上添加的;

 

virtual router id:这里设置 VRID,这里非常重要,相同的 VRID 为一个组,他将决定多播的 MAC 地址;

 

priority 100:设置本节点的优先级,优先级高的为 master;

 

advert int:检查间隔,默认为 1 秒;

 

virtual ipaddress:这里设置的就是 VIP,也就是虚拟 IP 地址,他随着 state 的变化而增加删除,当 state 为 master 的时候就添加,当 state 为 backup 的时候删除,这里主要是有优先级来决定的,和 state 设置的值没有多大关系,这里可以设置多个 IP 地址;

 

authentication:这里设置认证;

 

auth type:认证方式,可以是 PASS 或 AH 两种认证方式;

 

auth pass:认证密码;

启动俩台服务器的 keepalived

[root@master ~]# service keepalived start

Starting keepalived:                                      [OK]

[root@backup ~]# service keepalived start

Starting keepalived:                                      [OK]

[root@master ~]# tail -f /var/log/messages

Jan 10 11:40:56 localhost Keepalived_healthcheckers[19368]: Using LinkWatch kernel netlink reflector…

Jan 10 11:40:56 localhost Keepalived_vrrp[19369]: Opening file ‘/etc/keepalived/keepalived.conf’.

Jan 10 11:40:56 localhost Keepalived_vrrp[19369]: Configuration is using : 63019 Bytes

Jan 10 11:40:56 localhost Keepalived_vrrp[19369]: Using LinkWatch kernel netlink reflector…

Jan 10 11:40:56 localhost Keepalived_vrrp[19369]: VRRP sockpool: [ifindex(2), proto(112), fd(10,11)]

Jan 10 11:40:57 localhost Keepalived_vrrp[19369]: VRRP_Instance(VI_1) Transition to MASTER STATE

Jan 10 11:40:58 localhost Keepalived_vrrp[19369]: VRRP_Instance(VI_1) Entering MASTER STATE

Jan 10 11:40:58 localhost Keepalived_vrrp[19369]: VRRP_Instance(VI_1) setting protocol VIPs.

Jan 10 11:40:58 localhost Keepalived_healthcheckers[19368]: Netlink reflector reports IP 192.168.30.230 added

Jan 10 11:40:58 localhost Keepalived_vrrp[19369]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.30.230

Jan 10 11:41:03 localhost Keepalived_vrrp[19369]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.30.230

Jan 10 11:41:14 localhost Keepalived_vrrp[19369]: VRRP_Instance(VI_1) Received lower prio advert, forcing new election

Jan 10 11:41:14 localhost Keepalived_vrrp[19369]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.30.230

[root@master ~]# ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

    inet6 ::1/128 scope host

      valid_lft forever preferred_lft forever

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:0c:29:f3:fc:ba brd ff:ff:ff:ff:ff:ff

    inet 192.168.30.116/24 brd 192.168.30.255 scope global eth0

    inet 192.168.30.230/32 scope global eth0

    inet6 fe80::20c:29ff:fef3:fcba/64 scope link

      valid_lft forever preferred_lft forever

发现刚才定义的 virtual ipaddress 在 master 服务器上,因为优先级较高

停止 master 服务器的 keepalived 服务器,看 IP 会不会转移到 backup

[root@master ~]# service keepalived stop

Stopping keepalived:                                      [OK]

[root@backup ~]# tail -f /var/log/messages

Jan 10 12:12:46 localhost Keepalived_vrrp[18581]: Opening file ‘/etc/keepalived/keepalived.conf’.

Jan 10 12:12:46 localhost Keepalived_vrrp[18581]: Configuration is using : 63017 Bytes

Jan 10 12:12:46 localhost Keepalived_vrrp[18581]: Using LinkWatch kernel netlink reflector…

Jan 10 12:12:46 localhost Keepalived_healthcheckers[18580]: Opening file ‘/etc/keepalived/keepalived.conf’.

Jan 10 12:12:46 localhost Keepalived_healthcheckers[18580]: Configuration is using : 7324 Bytes

Jan 10 12:12:46 localhost Keepalived_healthcheckers[18580]: Using LinkWatch kernel netlink reflector…

Jan 10 12:12:46 localhost Keepalived_vrrp[18581]: VRRP sockpool: [ifindex(2), proto(112), fd(10,11)]

Jan 10 12:12:47 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Transition to MASTER STATE

Jan 10 12:12:47 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Received higher prio advert

Jan 10 12:12:47 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Entering BACKUP STATE

Jan 10 12:16:27 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Transition to MASTER STATE

Jan 10 12:16:28 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Entering MASTER STATE

Jan 10 12:16:28 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) setting protocol VIPs.

Jan 10 12:16:28 localhost Keepalived_healthcheckers[18580]: Netlink reflector reports IP 192.168.30.230 added

Jan 10 12:16:28 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.30.230

Jan 10 12:16:33 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.30.230

[root@backup ~]# ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

    inet6 ::1/128 scope host

      valid_lft forever preferred_lft forever

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:0c:29:5b:50:f9 brd ff:ff:ff:ff:ff:ff

    inet 192.168.30.117/24 brd 192.168.30.255 scope global eth0

    inet 192.168.30.230/32 scope global eth0

    inet6 fe80::20c:29ff:fe5b:50f9/64 scope link

      valid_lft forever preferred_lft forever

发现 IP 已经转移到 backup 服务器

下面重新启动 master 的 keepalived

[root@master ~]# service keepalived start

Starting keepalived:                                      [OK]

[root@backup ~]# tail -f /var/log/messages

Jan 10 12:12:46 localhost Keepalived_vrrp[18581]: VRRP sockpool: [ifindex(2), proto(112), fd(10,11)]

Jan 10 12:12:47 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Transition to MASTER STATE

Jan 10 12:12:47 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Received higher prio advert

Jan 10 12:12:47 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Entering BACKUP STATE

Jan 10 12:16:27 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Transition to MASTER STATE

Jan 10 12:16:28 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Entering MASTER STATE

Jan 10 12:16:28 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) setting protocol VIPs.

Jan 10 12:16:28 localhost Keepalived_healthcheckers[18580]: Netlink reflector reports IP 192.168.30.230 added

Jan 10 12:16:28 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.30.230

Jan 10 12:16:33 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.30.230

Jan 10 12:18:20 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Received higher prio advert

Jan 10 12:18:20 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Entering BACKUP STATE

Jan 10 12:18:20 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) removing protocol VIPs.

Jan 10 12:18:20 localhost Keepalived_healthcheckers[18580]: Netlink reflector reports IP 192.168.30.230 removed

[root@master ~]# ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

    inet6 ::1/128 scope host

      valid_lft forever preferred_lft forever

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:0c:29:f3:fc:ba brd ff:ff:ff:ff:ff:ff

    inet 192.168.30.116/24 brd 192.168.30.255 scope global eth0

    inet 192.168.30.230/32 scope global eth0

    inet6 fe80::20c:29ff:fef3:fcba/64 scope link

      valid_lft forever preferred_lft forever

发现 IP 已经重新转移到 master 服务器

现在编写 haproxy 状态检测脚本,来实现 haproxy 的健康检测:

[root@master ~]# cat haproxy_pid.sh

#!/bin/bash

while :

do

haproxypid=`ps -C haproxy –no-header | wc -l`

if [$haproxypid -eq 0];then

  service haproxy start

  sleep 5

  haproxypid=`ps -C haproxy –no-header | wc -l`

  echo $haproxypid

    if [$haproxypid -eq 0];then

  /etc/init.d/keepalived stop

    fi

fi

sleep 5

done

# 启动 backup 的 haproxy

[root@backup ~]# service haproxy start

Starting haproxy:                                          [OK]

模拟故障,先让 httpd 进程开启,修改 haproxy 监听端口为 80,使得 haproxy 进程无法启动,看资源会不会转移到 backup 服务器

[root@master ~]# vi /etc/haproxy/haproxy.cfg

frontend  main *:5000 改为 frontend  main *:80

# 启动 httpd 进程

[root@master ~]# scp -pr /etc/haproxy/haproxy.cfg root@192.168.30.117:/etc/haproxy/

[root@master ~]# service httpd start

Starting httpd: httpd: apr_sockaddr_info_get() failed for master.luojianlong.com

httpd: Could not reliably determine the server’s fully qualified domain name, using 127.0.0.1 for ServerName

                                                          [OK]

[root@master ~]# netstat -antpl | grep :80

tcp        0      0 :::80                      :::*                        LISTEN      19965/httpd

# 运行状态检测脚本

[root@master ~]# nohup /root/haproxy_pid.sh &

[root@master ~]# scp -pr haproxy_pid.sh root@192.168.30.117:/root/

[root@bakcup ~]# nohup /root/haproxy_pid.sh &

[root@master ~]# tail -f /var/log/messages

Jan 10 12:02:29 localhost Keepalived_vrrp[19849]: VRRP sockpool: [ifindex(2), proto(112), fd(10,11)]

Jan 10 12:02:29 localhost Keepalived_vrrp[19849]: VRRP_Instance(VI_1) Transition to MASTER STATE

Jan 10 12:02:30 localhost Keepalived_vrrp[19849]: VRRP_Instance(VI_1) Entering MASTER STATE

Jan 10 12:02:30 localhost Keepalived_vrrp[19849]: VRRP_Instance(VI_1) setting protocol VIPs.

Jan 10 12:02:30 localhost Keepalived_vrrp[19849]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.30.230

Jan 10 12:02:30 localhost Keepalived_healthcheckers[19848]: Netlink reflector reports IP 192.168.30.230 added

Jan 10 12:02:35 localhost Keepalived_vrrp[19849]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.30.230

Jan 10 12:14:49 localhost Keepalived[19847]: Stopping Keepalived v1.2.7 (02/21,2013)

Jan 10 12:14:49 localhost Keepalived_vrrp[19849]: VRRP_Instance(VI_1) sending 0 priority

Jan 10 12:14:49 localhost Keepalived_vrrp[19849]: VRRP_Instance(VI_1) removing protocol VIPs.

[root@master ~]# ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

    inet6 ::1/128 scope host

      valid_lft forever preferred_lft forever

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:0c:29:f3:fc:ba brd ff:ff:ff:ff:ff:ff

    inet 192.168.30.116/24 brd 192.168.30.255 scope global eth0

    inet6 fe80::20c:29ff:fef3:fcba/64 scope link

      valid_lft forever preferred_lft forever

[root@backup ~]# ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

    inet6 ::1/128 scope host

      valid_lft forever preferred_lft forever

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:0c:29:5b:50:f9 brd ff:ff:ff:ff:ff:ff

    inet 192.168.30.117/24 brd 192.168.30.255 scope global eth0

    inet 192.168.30.230/32 scope global eth0

    inet6 fe80::20c:29ff:fe5b:50f9/64 scope link

      valid_lft forever preferred_lft forever

[root@backup ~]# ps aux | grep haproxy

haproxy  19054  0.0  0.0  18688  1280 ?        Ss  12:47  0:00 /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid

root    19097  0.0  0.0 103248  828 pts/0    S+  12:58  0:00 grep haproxy

发现 IP 已经转移到 backup 服务器,实现了 keepalived 对于 haproxy 故障的高可用。

keepalived 常见的启动报错:

5913 May 16 15:26:04 localhost Keepalived_vrrp: ip address associated with VRID not present in received packet : 192.168.57.75

5914 May 16 15:26:04 localhost Keepalived_vrrp: one or more VIP associated with VRID mismatch actual MASTER advert

5915 May 16 15:26:04 localhost Keepalived_vrrp: bogus VRRP packet received on eth0 !!!

5916 May 16 15:26:04 localhost Keepalived_vrrp: VRRP_Instance(VI_1) ignoring received advertisment…

5917 May 16 15:26:05 localhost Keepalived_vrrp: ip address associated with VRID not present in received packet : 192.168.57.75

5918 May 16 15:26:05 localhost Keepalived_vrrp: one or more VIP associated with VRID mismatch actual MASTER advert

5919 May 16 15:26:05 localhost Keepalived_vrrp: bogus VRRP packet received on eth0 !!!

5920 May 16 15:26:05 localhost Keepalived_vrrp: VRRP_Instance(VI_1) ignoring received advertisment.

解决方法:

在同一网段内 virtual_router_id 值不能相同,如果相同会在 messages 中收到 VRRP 错误包,所以需要更改 virual_router_id。

HAproxy 的详细介绍 :请点这里
HAproxy 的下载地址 :请点这里

推荐阅读:

Haproxy+Keepalived 搭建 Weblogic 高可用负载均衡集群 http://www.linuxidc.com/Linux/2013-09/89732.htm

Keepalived+HAProxy 配置高可用负载均衡 http://www.linuxidc.com/Linux/2012-03/56748.htm

CentOS 6.3 下 Haproxy+Keepalived+Apache 配置笔记 http://www.linuxidc.com/Linux/2013-06/85598.htm

Haproxy + KeepAlived 实现 WEB 群集 on CentOS 6 http://www.linuxidc.com/Linux/2012-03/55672.htm

Haproxy+Keepalived 构建高可用负载均衡 http://www.linuxidc.com/Linux/2012-03/55880.htm

正文完
星哥说事-微信公众号
post-qrcode
 
星锅
版权声明:本站原创文章,由 星锅 2022-01-20发表,共计13652字。
转载说明:除特殊说明外本站文章皆由CC-4.0协议发布,转载请注明出处。
【腾讯云】推广者专属福利,新客户无门槛领取总价值高达2860元代金券,每种代金券限量500张,先到先得。
阿里云-最新活动爆款每日限量供应
评论(没有评论)
验证码
【腾讯云】云服务器、云数据库、COS、CDN、短信等云产品特惠热卖中