In this post, I will be sharing my recent experience of setting up a load-balanced internet connection and my learnings from it.
TL; DR: Load balanced internet connection is not the smartest idea; you might be better off with a failover based setup.
Back here in India, no ISP is super reliable. However, the internet has become such an essential part of our daily lives that it sucks when it’s down. I’ve had two internet connections at my home office for years now. I used to switch between them manually when one went down. It worked fine, but we developers hate it when we have to do something manually, so I decided to automate it.
My first thought was to set up a failover connection. I have two fiber connections at home, one with 200mbps bandwidth and the other with 100mbps. I started looking for a router that could set up a failover connection. All I needed was two WAN ports, 1 LAN port, and an OS that allows me to keep pinging both my WAN connections, and if one goes down, switch the LAN port to use the other WAN connection.
I initially planned on customizing a Raspberry Pi to add a couple more NIC (another fun project, yeah!). However, after doing a little more research, I came across Ubiquiti’s Edgerouter X. It did all I wanted and was cheaper than my estimated costs for the jacked Raspberry Pi. The only problem was finding one in India. There are no retail stores that sell Ubiquiti’s products here, so I had to get in touch with one of the official importer and master distributor of Ubiquiti to get my hands on it. I finally got it and couldn’t have been happier with it.
When I was looking at Edgerouter’s features, I noticed that it supports load balancing that diverts different requests to different networks based on the load. This essentially meant, when I opened a website, the CSS could load from my first ISP and Javascript from the other. When I download something in multi-threaded mode, I should be able to download at full 300mbps speed (some threads will use 200mbps connection, others will use 100mbps).
It sounded great in theory, so I decided to try it out. To my surprise, it worked as expected! I was able to get 300mbps speed up and down! I decided to continue using it, and it seemed to work perfectly. When one internet connection went down, the router automatically routes all the traffic to the other network, and when the other network came back, it automatically started using both again. I thought I had found the perfect solution until I decided to do some internet banking.
One of the (less useful) security features of my bank is that they automatically log you out if your login cookie was generated via one IP but then used by another IP. This meant, Since I had two IPs sending simultaneous requests to my bank with the same login cookie, I was immediately being logged out. Albeit I love crypto, I can’t live without my bank just yet, so I had to fix it. One way would’ve been to disable one WAN when I have to use the bank, but what did I say about developers hating to do manual work?
I basically needed to make sure that requests to my bank were sent via only one IP. I found another neat feature in Edgerouter X. It allowed me to set sticky routes. i.e., I could manually add routes and define if the connection is being made to this IP then always use that WAN connection. I tried, and it worked for my bank website. I was happy thinking all my problems are finally solved now!
My happiness lasted for only one more week, though. I faced a similar problem again, but this time it wasn’t my bank. It was cPanel logging me out again and again. At this point, I realized that there are still a lot of websites/applications that expect you to use a single IP address at all times, and I will keep facing problems if I load balanced like this. So, I decided to switch up things. I went back to my original idea of failover based internet. It was super easy to set up in Edgerouter. Now, I use my 200mbps connection by default for everything, but if it ever goes down, my router automatically switches me to the 100mbps connection. No manual work needed!
I then thought of load balancing between devices. I.e., make some devices use the slower connection always but if, the slower goes down, automatically switch to the higher speed one. This theoretically meant that I could saturate my whole 300mbps speed without having issues with multiple IPs. My computer will always use one specific internet connection. However, there’s a problem with this approach. I will be dooming some of my devices to always use the slower connection. In real life, most of the internet requests happen in bursts. Unless you are downloading something, you will rarely ever saturate a 200 mbps connection. This meant, as long as my usage didn’t cross the point at which I am constantly saturating 200mbps connection, I can keep using my faster connection on all devices without any noticeable performance drop. This is what I am doing now, and it has been working perfectly for me for the past month.
In the end, it seems like my first basic idea of a simple failover was the best one. It was a good learning experience setting it all up, though.