Configure HAProxy and Keepalived for Load Balancing and Reverse Proxy on Red Hat/Scientific/CentOS Linux 5/6

HAProxy is an open source load balancer/reverse proxy that can provide high availability for your network services. While generally used for web services, it can also be used to provide more reliability for services such as SMTP and terminal services. In addition we can combine it with the Keepalived package to allow high availability/failover for the HAProxy server itself. HAProxy plus Keepalived can provide a good solution for high availability at a very low cost in comparison to proprietary hardware based load balancers.

In this example I will configure 2 HAProxy/Keepalived servers (lb1/lb2) that will direct traffic to 2 Apache web servers (web1/web2). I will not detail the set up of the web servers. Here is a list of the server and IP address configuration scheme:

lb1  192.168.20.11
lb2  192.168.20.12
web1  192.168.20.21
web2  192.168.20.22

Configure on both lb1/lb2

First we need to activate the Extra Packages for Enterprise Linux (EPEL) repository, which should host packages of the HAProxy and Keepalived software. Install EPEL repo info into YUM:

# rpm -ivh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-5.noarch.rpm

Now we will install the HAproxy and Keepalived software:

# yum -y install haproxy keepalived

Configure Keepalived on lb1/lb2

Move the existing config because we will basically be starting from scratch.

# mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak

Edit the Keepalived config file and make it look something like below:

# nano /etc/keepalived/keepalived.conf

vrrp_script chk_haproxy {           # Requires keepalived-1.1.13
script "killall -0 haproxy"     # cheaper than pidof
interval 2                      # check every 2 seconds
weight 2                        # add 2 points of prio if OK
}
vrrp_instance VI_1 {
interface eth0
state MASTER
virtual_router_id 51
priority 101                    # 101 on master, 100 on backup
virtual_ipaddress {
192.168.20.20
}
track_script {
chk_haproxy
}
}

Make sure and change the priority under the vrrp_instance to 100 on the backup HAProxy server “lb2”. Other than that parameter the rest of the keepalived.conf file should be identical on both lb1 and lb2. Also if you want to add additional virtual IP addresses for additional services/servers in your network simply add an IP address under the virtual_ipaddress directive.

Now we need to configure the system to allow HAProxy to access shared virtual IP addresses. First make a backup of the sysctl.conf file:

# cp /etc/sysctl.conf /etc/sysctl.conf.bak

Now add this line to the end of the /etc/sysctl.conf file:

net.ipv4.ip_nonlocal_bind = 1

Now execute sysctl to apply the new parameter:

# sysctl -p

Extra configuration of iptables is required for keepalived, in particular we must enable support for multicast broadcast packets. If this doesn’t work you can always disable the iptables firewall for testing purposes. Of course this is not recommended in production environments!

Enable the multicast packets:

# iptables -I INPUT -d 224.0.0.0/8 -j ACCEPT

You may need to add this rule for the VRRP IP protocol:

# iptables -I INPUT -p 112 -j ACCEPT

In addition insert a rule that will correspond with the traffic that you are load balancing, in my case HTTP:

# iptables -I INPUT -p tcp --dport 80 -j ACCEPT

Finally save the iptables config so it will be restored after restarting and start Keepalived:

# service iptables save
# service keepalived start

Check lb1

Now let’s check to see if Keepalived is listening on the virtual IP address that we specified:

# ip addr sh eth0

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
link/ether 00:50:56:87:00:0f brd ff:ff:ff:ff:ff:ff
inet 192.168.20.11/24 brd 192.168.20.255 scope global eth0
inet 192.168.20.20/32 scope global eth0
inet6 fe80::250:56ff:fe87:f/64 scope link
valid_lft forever preferred_lft forever

Note that the virtual IP address is bound to network interface eth0.

Check lb2

# ip addr sh eth0

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
link/ether 00:50:56:87:00:22 brd ff:ff:ff:ff:ff:ff
inet 192.168.20.12/24 brd 192.168.20.255 scope global eth0
inet6 fe80::250:56ff:fe87:22/64 scope link
valid_lft forever preferred_lft forever

Note that the virtual IP address is not bound on the backup HAProxy server lb2. You can test failover by disconnecting lb1 from the network or shutting down the keepalived service on lb1.

Configure HAProxy on lb1/lb2

First make a backup of the HAProxy config file for good measure:

# cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.bak

Basically you will want to keep the global and default configuration sections intact. Remove or remark out the frontend and backend sections below them. The frontend/backend configuration is a more modern way to implement proxying, however I will perform the more traditional configuration and specify my options in one configuration section for each specific virtual IP address that I will proxy.

# nano /etc/haproxy/haproxy.cfg

listen webfarm 192.168.20.20:80
mode http
balance source
cookie JSESSIONID prefix
option httpchk HEAD /check.txt HTTP/1.0
option httpclose
option forwardfor
server web1 192.168.20.21:80 cookie A check
server web2 192.168.20.22:80 cookie B check

You can read about the different configuration options available on the HAProxy website. There is a plethora of them! Basically the “balance source” option will have each client connect to the same web server unless it fails. A variable will be inserted in a session cookie that will specify which server to direct a client to. Also HAProxy will check if a web server is available with “option httpchk” by inspecting a file on the web server root named check.txt.  You will want to create this file on each web server. The “option httpclose” and “option forwardfor” are needed to allow the web server to log the actual IP address source for each client request, otherwise the HAProxy server IP will be logged because it is proxying the connection.

Enable HAProxy and Keepalived to start automatically on system boot and start HAProxy:

# chkconfig haproxy on
# chkconfig keepalived on
# service haproxy start

Configure logging on web1/web2

This part is optional but there are some configuration changes needed on the web servers in order to have proper logging. In this example my web1/web2 servers are Apache and Red Hat/CentOS based.

# nano /etc/httpd/conf/httpd.conf

First since the HAProxy servers for available on the web server by requesting the check.txt file, we will want to disable logging of requests for this file. Otherwise all access to this file will be logged.

Remark out any lines that begin with “CustomLog”. Then at the end of the file add these lines which will prevent logging access to check.txt:

SetEnvIf Request_URI "^/check\.txt$" dontlog
CustomLog logs/access_log combined env=!dontlog

Additionally you will want to specify that Apache log the client IP address forwarded by HAProxy, not the IP of the HAProxy server. Remark out the existing LogFormat directive similar to the one commented below and substitute the next line with the “X-Forwarded-For” parameter. Please note that the second line is wrapped, it should be only a single line below the commented line.

#LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

Hopefully that should do it. You can test by shutting down or disconnecting the network on either of your web servers.

For planned maintenance you can delete the check.txt file on the web server that you will be working on. With this configuration new client requests will go to the server that passes the check but existing client sessions should be maintained on the existing server with the check file removed. In this way you can have a scheme where you wait for client connections to be drained completely on the server with the check file removed, then you can perform maintenance. That way downtime is minimized and the user experience with sessions is maintained.

Good luck with HAProxy!

References

http://haproxy.1wt.eu/download/1.2/doc/architecture.txt

http://www.howtoforge.com/setting-up-a-high-availability-load-balancer-with-haproxy-keepalived-on-debian-lenny