Introduction: The Growing Threat of Distributed Botnets

In the modern cloud computing landscape, Virtual Private Servers (VPS) are constant targets for malicious actors. Among the most pervasive threats are brute-force attacks, where automated networks of compromised devices—known as botnets—attempt to gain unauthorized access via protocols like SSH, FTP, or custom API endpoints. Unlike traditional single-IP attacks, modern botnets distribute their requests across thousands of distinct, international IP addresses, rotating them rapidly to evade basic rate-limiting thresholds.

Relying solely on standard firewall rules or legacy utilities like iptables can severely degrade system performance when dealing with massive blocks of malicious IPs. Every incoming packet must be evaluated against the firewall's ruleset sequentially. When that ruleset expands to include thousands of blacklisted IP ranges, CPU utilization spikes, leading to increased latency and potential denial of service for legitimate users. To mitigate this, system administrators must implement a highly optimized, scalable solution: nftables combined with IPSet structures.

The Architecture: Why nftables and IPSet?

For years, iptables was the standard framework for packet filtering in Linux. However, its architecture was not designed for modern multi-core processors and massive, dynamic blacklists. Enter nftables, the successor that replaces the old infrastructure with a more efficient virtual machine-based execution design. It provides a cleaner syntax, reduces code duplication, and processes rules significantly faster.

When dealing with vast botnet infrastructures (often spanning entire country-specific subnets or vast ASN blocks), hardcoding individual IP addresses into native firewall rules is highly inefficient. This is where IPSet (IP sets) or native nftables sets become invaluable. A set allows you to store multiple IP addresses, networks, or even port numbers framework-native, enabling instantaneous O(1) lookups. Instead of checking a packet against 10,000 separate rules, the firewall performs a single hash-table lookup against the set, drastically reducing CPU overhead.

Prerequisites and Environment Setup

Before proceeding with the optimization process, ensure your environment meets the following baseline requirements:

A Linux VPS running a modern distribution (e.g., Debian 12, Ubuntu 24.04, or RHEL 9) with root or sudo privileges.
A functional installation of nftables.
Access to updated IP intelligence feeds or threat intelligence lists containing known botnet IP ranges.

To begin, update your package repository and install the necessary utilities using your package manager:

sudo apt update && sudo apt install nftables ipset curl -y

Ensure that the nftables service is enabled to persist across system reboots:

sudo systemctl enable nftables
sudo systemctl start nftables

Step-by-Step Configuration: Implementing the Firewall

Step 1: Defining the nftables Base Structure

Unlike its predecessor, nftables does not come with pre-configured tables like filter or nat. We must define our own tables and chains to establish an organized structure. Let us create a standard input filtering table optimized for high throughput.

Open or create your primary configuration file, typically located at /etc/nftables.conf, and establish the base layout:

Note: It is critical to always allow established and related traffic first to prevent disconnecting your current SSH session.

table inet filter {
    set botnet_blacklist {
        type ipv4_addr
        flags interval
    }

    chain input {
        type filter hook input priority 0; policy accept;

        # Allow established and related connections
        ct state established,related accept

        # Drop loopback interface traffic alternatives if necessary, but explicitly allow local loopback
        iifname "lo" accept

        # Drop traffic originating from the botnet blacklist set
        ip saddr @botnet_blacklist drop

        # Protect SSH from brute-force attempts
        tcp dport 22 ct state new meter ssh_meter { ip saddr timeout 1m limit rate over 5/minute } drop
        tcp dport 22 accept
    }
}

Step 2: Leveraging High-Performance Sets

In the configuration above, we defined a native nftables set named botnet_blacklist. The flags interval parameter is crucial; it allows us to add entire subnets (such as CIDR blocks like 192.0.2.0/24) rather than just single IP addresses. By routing all traffic matching this set directly to a drop target early in the chain, we save immense processing power.

Step 3: Integrating IPSet Aggregations and Automation

To defend against international botnets, we must populate our firewall sets with reliable, real-time threat intelligence data. Manually gathering these IPs is impossible, so we automate the ingestion process using a script that fetches known malicious subnets from reputable open-source projects (such as FireHOL, Spamhaus, or Blocklist.de).

Create an automation script named /usr/local/bin/update_blacklist.sh to seamlessly refresh the sets:

#!/bin/bash
# URL of a reputable aggregated botnet IP list
URL="[https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level1.netset](https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level1.netset)"
TMP_FILE="/tmp/botnet_list.txt"

# Fetch the latest list
curl -s "$URL" -o "$TMP_FILE"

if [ -s "$TMP_FILE" ]; then
    # Flush the existing elements in the nftables set
    nft flush set inet filter botnet_blacklist

    # Parse and efficiently add IPs to the set
    echo "Processing IP ranges into nftables..."
    grep -E '^[0-9]' "$TMP_FILE" | while read -r line; do
        nft add element inet filter botnet_blacklist { "$line" }
    done
    echo "Firewall blacklist successfully updated."
else
    echo "Error: Failed to retrieve threat intelligence data."
fi

rm -f "$TMP_FILE"

Make the script executable and run it for an initial population:

sudo chmod +x /usr/local/bin/update_blacklist.sh
sudo /usr/local/bin/update_blacklist.sh

Monitoring, Testing, and Performance Evaluation

Once implemented, you must monitor the firewall to ensure it behaves as expected without disrupting legitimate commercial traffic. You can inspect the current rules and the number of active entries in your sets with the following command:

sudo nft list ruleset

To verify the efficiency and monitor packet drops in real-time, use the summary or system logging mechanisms. If an IP from the blacklist attempts to connect, the packet counter associated with the drop rule will increment immediately without causing a measurable increase in system CPU usage.

To ensure this protection remains up to date against dynamic botnet infrastructure, automate the script execution using a system cron job. Edit the crontab:

sudo crontab -e

Add the following line to automatically refresh the botnet IP blocks daily at midnight:

0 0 * * * /usr/local/bin/update_blacklist.sh > /dev/null 2>&1

Conclusion

Protecting a VPS from global brute-force botnets requires a strategy that balances strict security posture with high computational efficiency. By moving away from legacy legacy frameworks and adopting nftables combined with dynamic set lookups, you create a defensive layer capable of dropping malicious packets at wire-speed. This infrastructure keeps your system highly responsive, protects critical resources, and ensures operational continuity against even the most persistent distributed brute-force campaigns.

Optimizing nftables and IPSet on VPS: Defending Against Global Botnet Brute-Force Attacks