I have been working from home for nearly a decade and have always been concerned about my ability to get things done. To satisfy that, I have run two internet connections.
Years ago, I had two DSL lines from AT&T, predating U-Verse, and there was no bonding, which worked well for us because we were early cord-cutters and had been streaming most services since about 2009. I had no load-balancing capability. I would have devices that received DHCP configuration point to one gateway. Then I would manually configure other devices to point to the other gateway. Later, U-Verse was available to me; essentially, two bonded DSL connections were great because one of the lines would go offline with some regularity.
Upon moving, we had Comcast Xfinity and Century Link DSL available. I had a pretty bad experience with Comcast from about 2000 through 2015 and didn’t want to get that service, but Century Link was going to take months to get set up, and they charged a flat fee ($45/mo) for whatever speed I could get, up to 10 Mbps); I only was able to get 4 Mbps. I reluctantly set up Comcast, and it was unreliable. Anything that could buffer or cache was OK, but anything that required real-time communications was horrible, like Skype for Business or WebEx meetings. So, I eventually had Century Link set up and my work machines manually configured to use that link.
About this time, a coworker recommended the Ubiquiti EdgeRouter Lite. I could load balance my two internet connections and establish VPN connections to public cloud networks for testing. We then had a local fiber provider available, and I ditched Century Link. I always intended to end Comcast if the fiber proved reliable, but many years later, I still have both.
EdgeRouter specifications (https://dl.ubnt.com/datasheets/edgemax/EdgeRouter_Lite_DS.pdf):
- Dual-core 500 Mhz CPU
- 512 MB DDR2 RAM
- 12W power draw
- 3x GbE interfaces
Many might suggest that load balancing is silly. Still, I had a large household for many years, so there was always someone else home, and more often than not, there was some variety of streaming. So, while an individual connection is not load-balanced, multiple connections are distributed, which works great.
Then, the pandemic happened, and everyone was home all the time. A household that had ballooned to 8 people, and everyone needed connectivity for work, school, and sanity. The situation put a strain on the EdgeRouter, regularly becoming CPU bound. In addition, the providers bumped up our speeds so that we had gigabit throughput on each link, and we could not take advantage of it. The issues started easing as the pandemic became less omnipresent.
However, a couple of weeks ago, something screamed out to me to replace that EdgeRouter, leading to the search for new hardware. I ultimately landed on a solution using OPNsense per the recommendation of a German friend. I was now searching for hardware because I had no intention of running it as a virtual machine.
Requirements:
- Supports OPNsense
- Reasonable load balancing of 2+ WAN connections
- Allow bridging of LAN interfaces
- Not CPU bound
- 2.5 GbE interfaces
Specifications (https://www.amazon.com/gp/product/B09PHGWPMB):
- Intel Celeron J4125 (quad core 2 Ghz CPU)
- 8GB DDR4 RAM
- 10W power draw
- 4x 2.5 GbE interfaces
My concern at this point is the compatibility of the NICs. I came across this post from 2021 using the same hardware with great success: http://www.bluemind.org/2249-2/
I then realized that the barebones option is nearly $100 cheaper, and the RAM and storage acquired separately about half the price difference. The only issue was that I would be waiting some time to get an mSATA drive delivered (a week) when Amazon was ready to get everything to my house by the next day. I went ahead and ordered the mSATA drive and bought a standard SATA SSD to try it out, and I could always use an additional SATA drive.
Other parts:
- RAM: SK Hynix 8GB DDR4: https://www.amazon.com/gp/product/B092RLJGVJ
- Storage: Kingston SSD KC600 256GB mSATA: https://www.amazon.com/gp/product/B092RLJGVJ
- Temporary storage: WD easystore 240GB SSD: https://www.bestbuy.com/site/wd-easystore-240gb-internal-ssd-sata/6411188.p
I downloaded OPNsense and made a bootable thumb drive on macOS:
sudo dd if=OPNsense-23.1-OpenSSL-vga-amd64.img of=/dev/rdisk4 bs=1m
syncI assembled the system with the RAM and temporary storage. I booted off of the thumb drive and began the installation. It was straightforward; I signed in with the `installer` user to do an install and let it do all the defaults. It chose ETH0 for the WAN and ETH1 for the LAN. I booted up and got online immediately, and updated the system.
The first step was to follow this guide to set up load balancing between the two WAN links: https://docs.opnsense.org/manual/how-tos/multiwan.html.
Most of the guide focuses on failover. While that is great, I am never big on failover; if I am paying for the resources regardless, I want to use them all the time. I don’t want to be in a situation where I haven’t used one link for a while, only to learn that it isn’t working when the other has just failed. If I am using both all the time, I can rest assured that when one fails, the other is more likely to be working. The only difference is that when configuring the two links following the guide, each link should have the tier within the Gateway Group.
An issue that I had been experiencing was frequent reauthentication requests to certain services on the EdgeRouter, so I wanted to enable Sticky Connections so that connections would remain on the same link. The configuration worked, but then I began to experience issues after about 10 minutes where nothing seemed to be working. I found that for the best experience, I should:
- Enable: System > General “Gateway Switching”
- Disable: Firewall > Settings > Advanced “Use shared forwarding between packet filter, traffic shaper, and captive portal”
That resolved the load-balancing issues for me. Next, I wanted to take the two remaining interfaces and bridge them so that they would behave like a two-port switch. I followed this guide: https://docs.opnsense.org/manual/how-tos/lan_bridge.html
Everything was working fine. I connected one interface to my wireless access point (WAP) and the other to the LAN; however, I couldn’t reach anything on the wireless network, including the WAP, from my wired system. So, I connected one of my PCs to that interface, and the same issue occurred. However, I could see that the MAC address was in the ARP cache running `arp -a`; this was true in both directions. I read somewhere that the BSD stack was still sending traffic between the bridged interfaces through the firewall, so I created a firewall rule on the LAN bridged interface to all traffic between “LAN net” and “LAN net.” I could then ping and RDP to the PC. I plugged the WAP back in, and I could reach its interface.
The last step was to turn on hardware acceleration settings one at a time to see if things were working well, which all panned out without any issues.
Speed tests were all pushing 1 Gbps which I expected since the interfaces on my devices are only GbE.
When the mSATA drive arrived a few days later, I swapped it in and removed the SATA SSD and the cables. There are no fans in the system, so having the cables seemed like one thing more than necessary to obstruct some airflow. I backed up the configuration but instead decided to do everything from scratch. What took me several hours of tweaking to resolve the initial issues with load balancing and bridging, I was able to complete them in about 15 minutes. However, the only change I addressed was during the initial setup; I chose which interfaces did what:
- ETH0 -> WAN0
- ETH1 -> WAN1
- ETH2 -> LAN (not a type)
- ETH3 -> LAN2
I purposely named ETH2 as LAN because the bridge would assume that name, and I would later call the interface LAN0 once I set up bridging. That took care of any discomfort with the interfaces not being used in a sensible order for me.
I upgraded my cable modem the next day, which was great to have something connected up with 2.5 GbE finally. My internet service was upgraded from 1000 Mbps to 1200 Mbps, saving me $22/month. I still have some additional work to do. While the principal WAP is connected to one interface, the satellite WAP is wired into my switch. So, I will get a new 2.5 GbE switch to replace my 5x GbE switch. Then the GbE switch will move over to the other interface, with both WAPs connected, which will require getting under the floor to run another ethernet cable.
Dustin Dortch 