Introduction to High Availability in Modern Infrastructure

In today's digital economy, application downtime directly translates to financial loss and damaged brand reputation. To mitigate this risk, enterprise systems must be engineered with redundancy at every layer. One of the most critical components to secure is the entry point of your network infrastructure: the load balancer. If a single load balancer goes down, your entire application becomes inaccessible, regardless of how many backend application servers you have running.

This comprehensive technical guide demonstrates how to architect a highly available (HA) load balancing tier using HAProxy for traffic distribution and Keepalived for infrastructure health monitoring and failover management. We will implement this solution using a Floating IP (Failover IP) across two Virtual Private Servers (VPS) hosted on Hetzner Cloud. By the end of this tutorial, you will have a resilient infrastructure capable of automatically routing traffic away from a failed node within seconds.

Understanding the Architectural Components

Before diving into the configuration files, it is vital to understand how the components interact within the Hetzner Cloud ecosystem:

HAProxy (High Availability Proxy): An industry-standard, open-source software load balancer and proxying solution known for its exceptional performance, reliability, and low memory footprint. It will distribute incoming HTTP/HTTPS traffic to your backend application nodes.
Keepalived: A routing software based on the Virtual Router Redundancy Protocol (VRRP). It constantly monitors the health of the HAProxy instances. If the primary node fails, Keepalived automatically shifts the Virtual IP (VIP) to the secondary node.
Hetzner Floating IP: A public IP address provided by Hetzner that can be dynamically reassigned to any VPS within the same location via the Hetzner Cloud API. This acts as our Virtual IP (VIP).

System Architecture Note: In a standard on-premise environment, Keepalived broadcasts ARP packets to shift a local IP. However, in cloud environments like Hetzner, cloud firewalls often block VRRP broadcasts between separate virtual machines. Therefore, we must couple Keepalived with Hetzner's API to explicitly rebind the Floating IP during a failover event.

Prerequisites and Environment Setup

To follow along with this guide, ensure you have the following resources provisioned on Hetzner Cloud:

Two VPS Instances: Running Ubuntu 22.04 LTS or Ubuntu 24.04 LTS, located in the same datacenter region (e.g., FSN1 or NBG1). We will refer to them as lb01 (Primary) and lb02 (Secondary).
One Floating IP: Created via the Hetzner Cloud Console and assigned initially to lb01.
Hetzner Cloud API Token: A read/write API token generated from your Hetzner Project Security settings. This token allows Keepalived to reassign the Floating IP programmatically.

For the purpose of this guide, we will use the following placeholder IP addresses. Replace them with your actual Hetzner infrastructure values:

lb01 (Primary): 192.168.1.10
lb02 (Secondary): 192.168.1.11
Floating IP (VIP): 203.0.113.50

Step 1: Installing and Configuring HAProxy

First, we must install and configure HAProxy on both lb01 and lb02. The configurations should ideally be identical to ensure consistent behavior regardless of which node handles the traffic.

Execute the following commands on both servers to update the package repository and install HAProxy:

sudo apt update
sudo apt install haproxy -y

Once installed, backup the default configuration file and create a new one:

sudo mv /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.bak
sudo nano /etc/haproxy/haproxy.cfg

Paste the following robust configuration block into the file. This setup configures basic HTTP load balancing and enables a secure statistics dashboard:

global
    log /dev/log local0
    log /dev/log local1 notice
    chroot /var/lib/haproxy
    user haproxy
    group haproxy
    daemon

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    timeout connect 5000
    timeout client  50000
    timeout server  50000

frontend http_front
    bind *:80
    stats uri /haproxy?stats
    default_backend web_servers

backend web_servers
    balance roundrobin
    option httpchk GET /health
    default-server inter 3s fall 3 rise 2
    server web01 10.0.0.10:80 check
    server web02 10.0.0.11:80 check

Save and close the file. Start and enable the HAProxy service to run on system boot:

sudo systemctl enable haproxy
sudo systemctl start haproxy

Step 2: Preparing the Hetzner API Failover Script

Because native VRRP broadcasting does not dynamically alter cloud provider routing tables, we need a script that Keepalived can trigger whenever a node transitions to the MASTER state. This script leverages curl and the Hetzner Cloud API to pull the Floating IP to the current active host.

First, install curl and jq (a command-line JSON processor) on both servers:

sudo apt install curl jq -y

Next, create the notification script at /usr/local/bin/assign-floating-ip.sh:

sudo nano /usr/local/bin/assign-floating-ip.sh

Populate the script with the following logic, replacing the placeholder variables with your actual Hetzner API Token and Floating IP ID:

#!/bin/bash

TOKEN="YOUR_HETZNER_API_TOKEN"
FLOATING_IP_ID="YOUR_FLOATING_IP_ID"

# Get the Server ID of the local Hetzner VPS
SERVER_ID=$(curl -s -H "Authorization: Bearer $TOKEN" [https://api.hetzner.cloud/v1/servers](https://api.hetzner.cloud/v1/servers) | jq -r '.servers[] | select(.name=="'$(hostname)'") | .id')

if [ -z "$SERVER_ID" ]; then
    echo "Error: Could not determine Server ID for $(hostname)" >> /var/log/keepalived-failover.log
    exit 1
fi

# Change routing of the Floating IP via Hetzner API
RESPONSE=$(curl -s -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"server\": $SERVER_ID}" \
  "[https://api.hetzner.cloud/v1/floating_ips/$FLOATING_IP_ID/actions/assign](https://api.hetzner.cloud/v1/floating_ips/$FLOATING_IP_ID/actions/assign)")

echo "$(date): Assigned Floating IP to $SERVER_ID ($(hostname)). Response: $RESPONSE" >> /var/log/keepalived-failover.log

Make the script executable so Keepalived can run it seamlessly:

sudo chmod +x /usr/local/bin/assign-floating-ip.sh

Step 3: Installing and Configuring Keepalived

With our API integration ready, we can proceed to install Keepalived on both servers:

sudo apt install keepalived -y

Keepalived uses a primary configuration file located at /etc/keepalived/keepalived.conf. Unlike HAProxy, this configuration must differ slightly between the primary and secondary nodes to establish a proper priority hierarchy.

Configuring Node 1 (lb01 - Master)

Create the file on lb01:

sudo nano /etc/keepalived/keepalived.conf

Add the following configuration block. Ensure you replace eth0 with your system's actual primary network interface name (verifiable via ip a):

vrrp_script check_haproxy {
    script "killall -0 haproxy"
    interval 2
    weight 2
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 101
    advert_int 1
    
    authentication {
        auth_type PASS
        auth_pass SecureClusterPassword123
    }
    
    track_script {
        check_haproxy
    }
    
    notify_master "/usr/local/bin/assign-floating-ip.sh"
}

Configuring Node 2 (lb02 - Backup)

Create the file on lb02:

sudo nano /etc/keepalived/keepalived.conf

Add the backup configuration block. Note the changes in state (BACKUP) and priority (100):

vrrp_script check_haproxy {
    script "killall -0 haproxy"
    interval 2
    weight 2
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    
    authentication {
        auth_type PASS
        auth_pass SecureClusterPassword123
    }
    
    track_script {
        check_haproxy
    }
    
    notify_master "/usr/local/bin/assign-floating-ip.sh"
}

Enable and start the Keepalived service on both instances:

sudo systemctl enable keepalived
sudo systemctl start keepalived

Step 4: Linux Kernel Optimization for Floating IPs

By default, the Linux kernel prevents applications from binding to an IP address that does not explicitly exist on the local network interface. Because our Hetzner Floating IP is routed dynamically via the API and might not always appear in ip addr show right away, HAProxy may fail to start if it tries to listen on the Floating IP directly.

To fix this, we must enable non-local binding on both load balancers. Open the system control configuration file:

sudo nano /etc/sysctl.conf

Append the following line to the end of the file:

net.ipv4.ip_nonlocal_bind=1

Apply the changes immediately without rebooting the server:

sudo sysctl -p

Testing the High Availability Failover Mechanism

With all configurations actively running, it is time to perform a simulation test to confirm our failover mechanism works as intended.

Check Initial State: Log into the Hetzner Cloud Console. Verify that the Floating IP is currently pointing to lb01. Check the log file on lb01 via tail -f /var/log/keepalived-failover.log to ensure the script ran successfully.
Simulate a Failure: Stop the HAProxy service on lb01 to simulate an application crash:
```
sudo systemctl stop haproxy
```
or explicitly force Keepalived to drop state by stopping it:
```
sudo systemctl stop keepalived
```
Observe the Failover: Monitor the syslog or our custom log file on lb02. Within 1 to 3 seconds, Keepalived on lb02 will detect that lb01 is offline, transition itself to MASTER state, and trigger the script.
Verify Hetzner Routing: Refresh your Hetzner Cloud Console. You will see the Floating IP instantly reassign its target to lb02. Traffic continues to flow seamlessly to your backend application servers without interruption.

Conclusion and Best Practices

You have successfully implemented a resilient, enterprise-grade high availability load-balancing cluster using Keepalived and HAProxy on Hetzner Cloud. By utilizing Hetzner's API alongside VRRP logic, you effectively worked around traditional cloud networking limitations to achieve automated infrastructure self-healing.

As you move this architecture into production, consider adopting these essential operational practices:

Automate Configuration Syncs: Use configuration management tools like Ansible, Puppet, or SaltStack to ensure that any change made to haproxy.cfg on the primary node is perfectly mirrored to the backup node.
Implement Rigid Monitoring: Set up automated alerts via Prometheus, Grafana, or Datadog to notify your engineering team whenever a failover occurs, allowing you to investigate the root cause of the primary node's failure.
Secure the API Token: Restrict filesystem permissions on your API script. Run chmod 700 /usr/local/bin/assign-floating-ip.sh so that non-root users cannot read your sensitive Hetzner API credentials.

Building a High-Availability Load Balancer: Configuring Keepalived and HAProxy with Failover IP on Hetzner Cloud