Uprading HAProxy 1.4 to 1.5 for native SSL support without downtime
By default Ubuntu 14.04 comes with HAProxy 1.4. It works fine until you need to support SSL for your web application because HAProxy 1.4 doesn’t support SSL natively. This was my case when I had to install SSL for a website which was served behind a load balancer using HAProxy 1.4 on a Ubuntu 14.04 box. At this point I had to choose between keeping using HAProxy 1.4 with some other programs to support SSL or upgrading HAProxy to 1.5 (current dev branch). I prefer the idea of using native support things so I decided to make an upgrade. But on a production system this would be more complex because we don’t want our customers to suffer from server downtime. Something was confused at first but some minutes later I came up with a solution.
The site which needs SSL support was served by two web servers. The traffic to these two servers was not much because it was a brand new website. A server alone could handle all traffic efficiently at near midnight. So making change at this time will be more appropriate and safer. My strategy was to use `iptables` to forward web traffic to a web server behind the load balancing server (forwarding to two servers is fine, but this was not neccessary). Then I’d remove HAProxy 1.4, compile HAProxy 1.5 (with SSL support), test it and if everything is OK, I’d remove iptables rules. Here’s how (The below commands were performed on the load balancing server):
1. Forward web traffic to a web server
First, enable IP Forwarding:
echo "1" > /proc/sys/net/ipv4/ip_forward
iptables -P FORWARD ACCEPT
Then forward web traffic:
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j DNAT --to-destination 10.30.37.73:80
iptables -t nat -A POSTROUTING -j MASQUERADE
At this point all web requests come to the load balancing server will be served by the web server 10.30.37.73 directly, so we can safely remove HAProxy 1.4:
apt-get remove haproxy
(Before removing HAProxy, you may want to backup the old config file: `cp /etc/haproxy/haproxy.cfg ~/haproxy.cfg.bak`)
2. Compile HAProxy 1.5
We need to install the prerequisites before we can compile new HAProxy:
apt-get build-dep -y haproxy
apt-get install -y libssl-dev
Get the source code and begin compiling it:
cd /usr/local/src/
wget http://haproxy.1wt.eu/download/1.5/src/devel/haproxy-1.5-dev24.tar.gz
tar xvf haproxy-1.5-dev24.tar.gz
cd haproxy-1.5-dev24/
make TARGET=linux2628 CPU=native USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1
make install
We reuse the old init script of HAProxy 1.4 (it was not removed):
vim /etc/init.d/haproxy
Change HAPROXY variable to new value:
HAPROXY=/usr/local/sbin/haproxy
HAProxy still can not start yet because the /var/lib/haproxy folder was removed. We create it again:
mkdir /var/lib/haproxy
chown -R haproxy:haproxy /var/lib/haproxy
Then start HAProxy:
/etc/init.d/haproxy start
Check if it is running:
/etc/init.d/haproxy status
Now HAProxy 1.5 is already running but serve nothing yet. We’ll add our configuration so that it can support SSL:
vim /etc/haproxy/haproxy.cfg
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
maxconn 5000
user haproxy
group haproxy
daemon
defaults
log global
mode http
option httplog
option dontlognull
contimeout 5000
clitimeout 50000
srvtimeout 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
frontend web_frontend
bind :443 ssl crt /root/ssl/ssl-bundle.pem
bind :80
default_backend web_backend
backend web_backend
mode http
balance roundrobin
option httpclose
option http-server-close
option forwardfor
server web-1 10.30.37.73:80 check
server web-2 10.30.37.74:80 check
Then restart HAProxy:
/etc/init.d/haproxy restart
Check if HAProxy is running on port 80 and 443:
netstat -napt | grep 80
netstat -napt | grep 443
3. Test and deploy
Before we remove the iptables rules (which forward web traffic to a web server), we need to make sure that HAProxy is doing it job exactly. Either `curl` or `wget` is useful in this situation:
curl -H "Host: www.example.com" http://localhost
curl -k -H "Host: www.example.com" https://localhost
The output of the above command should be what you’re familiar to. Then we’re ready to remove iptables rules so that HAProxy would do what it’s supposed to do:
iptables -t nat -F
That’s it. Now open your browser to check your site again. It should be working and support SSL already (Keep calm and check everything again if it isn’t :))