“He wears a Fanny Pack and he’s into mycology”

Actual Zero-Downtime with HAProxy

The internet is rife with promises of 100% availability when using HAProxy for load balancing. THEY ARE LIES!

When you instruct HAProxy to reload it’s configuration, the following occurs:

  1. A new process is created that signals the old process to unbind it’s ports
  2. The new process binds to the ports
  3. Existing connections are finished on the old process, or passed on to the new process

Seems pretty straight forward but there’s a small window of time between steps 1 and 2 where neither process is bound to the configured ports and requests are rejected. For most this seems to be a negligible window of error, but if you’re serving enough traffic and your requests are important to you it can be a deal breaker.

Some TCP

When a client makes a request over TCP/HTTP a series of handshakes occurs before the connection is established. The details of this handshake are for another post. All you need to know is that the first packet comes from the requestor and it’s typically referred to as a SYN (synchronize) packet. Once the requestor sends this packet, it waits for a response and retries at a decaying interval until it receives a response or times out.

So how can we use this to make sure we don’t lose requests while reloading HAProxy configuration?

Disclaimer: This next part describes a way to avoid dropped connections that some might consider dangerous. Use at your own risk ;)

Hacking TCP

Turns out, we can use iptables to trick clients into retrying their connections while the process is down!

iptables -I INPUT -p tcp -m multiport —dports 80,443 —syn -j DROP && sleep 0.5 && \
/etc/init.d/haproxy reload;
iptables -D INPUT -p -tcp -m multiport —dports 80,443 —syn -j DROP

So what’s happening here. First a rule is added to the beginning of the iptables INPUT chain that drops all SYN tcp packets on ports 80 and 443. After waiting a half a second for the changes to take affect, we reload HAProxy config and then remove the SYN packet rule from the firewall. Notice the semicolon after the HAProxy reload command that ensures the iptables rule will be removed even if the reload fails.

Running this command to restart your HAProxy process will help to guarantee reloading load balancer configuration won’t hurt your sites availability. Happy hacking! ☺

This post originally appeared on Medium.

#engineering