阿里云-云小站(无限量代金券发放中)
【腾讯云】云服务器、云数据库、COS、CDN、短信等热卖云产品特惠抢购

基于Keepalived+Nginx部署高可用7层负载均衡方案

118次阅读
没有评论

共计 6711 个字符,预计需要花费 17 分钟才能阅读完成。

高可用是个老生常谈的问题了,开源的高可用软件已经做的相当成熟了,之前也在 debian 下做过 lvs+heartbeat 的 4 层 LB,一直很稳定(可惜流量不大啊委屈),现在由于业务的需要,做一个基于 keepalived+nginx 的高可用 7 层负载均衡。

拓扑结构也比较简单,就不画拓扑图了:2 个节点上分别安装配置 keepalived 和 nginx,配置 nginx 反向代理后端的 real server

比较关键的几个点:

1、为避免同一个局域网中有多个 keepalived 组中的多播相互响应,采用单播通信

2、状态切换的过程触发邮件通知、短信通知、web 通知、log 记录,便于通过各种途径了解主备工作状态

3、nginx 的检测脚本采用了轻量级的方式:”killall -0 nginx”,还可以使用 pidof nginx 的方式或者调用其他自定义检测脚本的方式

4、特别要注意优先级的大小及检测到异常时权重的变化

5、了解免费 ARP 的工作机制

6、了解 VRRP 协议的适用范围:局域网,第一跳网关冗余

7、单个 vrrp 实例工作在主备模式,为最大程度的利用 2 个节点的资源,可以做多个 vrrp 实例,实现高可用和负载均衡

为便于软件包的管理,采用 CentOS 自带的 keepalived,nginx1.8.0 采用 nginx 官方源,整体安装也比较简单。

#yum install keepalived nginx -y

设置关键服务的开机启动

#chkconfig nginx on
#chkconfig keepalived on

查看 keepalived 包安装了那些文件(文档一定好好看):

#rpm -ql keepalived

/etc/keepalived
/etc/keepalived/keepalived.conf
/etc/rc.d/init.d/keepalived
/etc/sysconfig/keepalived
/usr/bin/genhash
/usr/libexec/keepalived
/usr/sbin/keepalived
/usr/share/doc/keepalived-1.2.13
/usr/share/doc/keepalived-1.2.13/AUTHOR

………

keepalived 的主配置文件,2 台机器的配置文件略有区别,具体请看配置文件中注释

[root@proxy101 ~]# more /etc/keepalived/keepalived.conf
####Configuration File for keepalived
#### 内部 API 网关 keepalived HA 配置
#### laijingli 20151213

global_defs {
  notification_email {
    xxxx@qq.com
  }
  notification_email_from yn_alert@xxx.me
  smtp_server 127.0.0.1
  smtp_connect_timeout 30
  router_id proxy101  ## proxy101 on master101 , proxy102 on backup102
}

###simple check with killall -0 which is less expensive than pidof to verify that nginx is running
vrrp_script chk_nginx {
        script “killall -0 nginx”
        interval 1
        weight 2
 fall 2
 rise 1
}

vrrp_instance YN_API_GATEWAY {
    state MASTER  ## MASTER on master101 ,  BACKUP on backup102
    interface em1
    virtual_router_id 101
    priority 200  ## 200 on master101 ,  199 on backup102
    advert_int 1
    ### 采用单播通信,避免同一个局域网中多个 keepalived 组之间的相互影响
    unicast_src_ip 192.168.0.101  ## 本机 ip
    unicast_peer {
 192.168.0.102  ## 对端 ip
    }
    authentication {
        auth_type PASS
        auth_pass testpass
    }
    virtual_ipaddress {
        192.168.0.105  ## VIP
    }
    ### 如果只有一块网卡的话监控网络接口就没有必要了
    #track_interface {
    # em1
    #}
    track_script {
        chk_nginx
    }
    ### 状态切换是发送邮件通知,本机记录 log,后期会触发短信通知
    notify_master /usr/local/bin/keepalived_notify.sh notify_master
    notify_backup /usr/local/bin/keepalived_notify.sh notify_backup
    notify_fault  /usr/local/bin/keepalived_notify.sh notify_fault
    notify        /usr/local/bin/keepalived_notify.sh notify
    smtp_alert
}

VRRP 实例组节点状态切换触发邮件通知、短信通知、log 记录的脚本
# more /usr/local/bin/keepalived_notify.sh
#!/bin/bash
###keepalived notify script for record ha state transtion to log files

### 将将状态转换过程记录到 log,便于排错
logfile=/var/log/keepalived.notify.log
echo ————— >> $logfile
echo `date` [`hostname`] keepalived HA role state transition:  $1 $2 $3 $4 $5 $6 >> $logfile

### 将状态转换记录到 nginx 的文件,便于通过 web 查看 ha 状态(一定注意不要开放到公网)
echo `date` `hostname` $1 $2 $3 $4 $5 $6 > /usr/share/nginx/html/index.html

nginx 的部分配置文件,仅供参考

# more /etc/nginx/nginx.conf

### 运维管理用途: 用于区别 vip 跑在那台服务器上
 server {
      listen      80;
      server_name  localhost;

      location / {
         root  /usr/share/nginx/html;
         index  index.html index.htm;
      }

  ## nginx monitor use only
  ###add by lai monitor nginx status
                location /server-status {
                        stub_status on;
   allow 127.0.0.1;
   allow 192.168.0.0/24;
                }
        }

通过启停 keepalived 和 nginx 来模拟故障,测试 vrrp 实例的状态切换过程(即 VIP 的漂移):

/etc/init.d/keepalived start/stop

/etc/init.d/nginx stop start/stop

通过启停 em1 内网网卡、em2 公网网卡来模拟故障,测试 vrrp 实例的状态切换过程(即 VIP 的漂移)

ifdown em1

ifdown em2

查看 VIP 跑在那台服务器上:

# ip addr show|grep 192.168
    inet 192.168.0.101/24 brd 192.168.0.255 scope global em1
    inet 192.168.0.105/32 scope global em1

测试:

[root@test28 ~]$ curl 192.168.0.101
Mon Dec 14 16:27:10 CST 2015 proxy101 INSTANCE YN_API_GATEWAY MASTER 202
[root@test28 ~]$ curl 192.168.0.102
Mon Dec 14 16:34:40 CST 2015 proxy102 INSTANCE YN_API_GATEWAY BACKUP 199
[root@test28 ~]$ curl 192.168.0.105
Mon Dec 14 16:27:10 CST 2015 proxy101 INSTANCE YN_API_GATEWAY MASTER 202

# tail /var/log/keepalived.notify.log
Mon Dec 14 16:25:13 CST 2015 [proxy101] keepalived HA role state transition:
Mon Dec 14 16:25:13 CST 2015 [proxy101] keepalived HA role state transition: INSTANCE YN_API_GATEWAY MASTER 202
—————
—————
Mon Dec 14 16:26:34 CST 2015 [proxy101] keepalived HA role state transition:
Mon Dec 14 16:26:34 CST 2015 [proxy101] keepalived HA role state transition: INSTANCE YN_API_GATEWAY BACKUP 200
—————
—————
Mon Dec 14 16:27:10 CST 2015 [proxy101] keepalived HA role state transition:
Mon Dec 14 16:27:10 CST 2015 [proxy101] keepalived HA role state transition: INSTANCE YN_API_GATEWAY MASTER 202

# tail /var/log/messages
Dec 14 16:27:08 localhost Keepalived_vrrp[74308]: VRRP_Instance(YN_API_GATEWAY) forcing a new MASTER election
Dec 14 16:27:08 localhost Keepalived_vrrp[74308]: VRRP_Instance(YN_API_GATEWAY) forcing a new MASTER election
Dec 14 16:27:09 localhost Keepalived_vrrp[74308]: VRRP_Instance(YN_API_GATEWAY) Transition to MASTER STATE
Dec 14 16:27:10 localhost Keepalived_vrrp[74308]: VRRP_Instance(YN_API_GATEWAY) Entering MASTER STATE
Dec 14 16:27:10 localhost Keepalived_vrrp[74308]: VRRP_Instance(YN_API_GATEWAY) setting protocol VIPs.
Dec 14 16:27:10 localhost Keepalived_vrrp[74308]: VRRP_Instance(YN_API_GATEWAY) Sending gratuitous ARPs on em1 for 192.168.0.105
Dec 14 16:27:10 localhost Keepalived_healthcheckers[74307]: Netlink reflector reports IP 192.168.0.105 added
Dec 14 16:27:10 localhost Keepalived_vrrp[74308]: Remote SMTP server [127.0.0.1]:25 connected.
Dec 14 16:27:10 localhost Keepalived_vrrp[74308]: SMTP alert successfully sent.
Dec 14 16:27:15 localhost Keepalived_vrrp[74308]: VRRP_Instance(YN_API_GATEWAY) Sending gratuitous ARPs on em1 for 192.168.0.105

主备状态转换通知邮件:

基于 Keepalived+Nginx 部署高可用 7 层负载均衡方案

通过抓包查看来了解详细的工作过程:

# tcpdump -ni em1 vrrp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em1, link-type EN10MB (Ethernet), capture size 65535 bytes
17:36:47.098225 IP 192.168.0.101 > 192.168.0.102: VRRPv2, Advertisement, vrid 101, prio 202, authtype simple, intvl 1s, length 20
17:36:47.388540 IP 192.168.0.22 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 120, authtype simple, intvl 1s, length 20
17:36:48.099409 IP 192.168.0.101 > 192.168.0.102: VRRPv2, Advertisement, vrid 101, prio 202, authtype simple, intvl 1s, length 20
17:36:48.389504 IP 192.168.0.22 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 120, authtype simple, intvl 1s, length 20
17:36:49.100544 IP 192.168.0.101 > 192.168.0.102: VRRPv2, Advertisement, vrid 101, prio 202, authtype simple, intvl 1s, length 20
17:36:49.390487 IP 192.168.0.22 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 120, authtype simple, intvl 1s, length 20
17:36:50.101713 IP 192.168.0.101 > 192.168.0.102: VRRPv2, Advertisement, vrid 101, prio 202, authtype simple, intvl 1s, length 20
17:36:50.391453 IP 192.168.0.22 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 120, authtype simple, intvl 1s, length 20

《Keepalived 权威指南》下载见 http://www.linuxidc.com/Linux/2012-05/60951.htm

Nginx+Keepalived 实现站点高可用  http://www.linuxidc.com/Linux/2016-12/137883.htm

Nginx+Keepalived 实现站点高可用 (负载均衡)  http://www.linuxidc.com/Linux/2016-12/138221.htm

构建高可用集群 Keepalived+Haproxy 负载均衡 http://www.linuxidc.com/Linux/2016-12/138917.htm

CentOS6.5 下 Keepalived 高可用服务单实例配置 http://www.linuxidc.com/Linux/2016-12/138110.htm

Keepalived+Nginx 实现高可用(HA)http://www.linuxidc.com/Linux/2017-02/140422.htm

本文永久更新链接地址 :http://www.linuxidc.com/Linux/2017-02/141084.htm

正文完
星哥说事-微信公众号
post-qrcode
 0
星锅
版权声明:本站原创文章,由 星锅 于2022-01-21发表,共计6711字。
转载说明:除特殊说明外本站文章皆由CC-4.0协议发布,转载请注明出处。
【腾讯云】推广者专属福利,新客户无门槛领取总价值高达2860元代金券,每种代金券限量500张,先到先得。
阿里云-最新活动爆款每日限量供应
评论(没有评论)
验证码
【腾讯云】云服务器、云数据库、COS、CDN、短信等云产品特惠热卖中