Virtualizing Multi-WAN pFSense on ESXi 5.5

So in an effort to both over-complicate my hose network setup and introduce a platform to test alternative Firewall distros over this past week end I stood up an ESXi 5.5. box.  It's running on a Dell Optiplex 790:

Core I5-2400 3.1ghz - 8gb ram - 128gb generic SSD - 3 x Intel nics, 1 x Broadcom nic.


I have 3 x 3mb ATT 'Uverse' circuits, I had to opt for ATT Business in order to get those.  Which is not a bad thing.  I signed no contract so I can leave whenever I want.  And I pay no more than a residential customer.

Anyway  here's a cool diagram of my new found setup.

The nics I wound up using were all pcie.. the single Broadcom, a single Intel then a dual headed Intel.

I gave the pF vm 2gb of ram, and a 9gb virtual hard disk.. it's load is light and even those resources are at least twice what any pF appliance will have.  Now the ESXi install is actually running of a 4gb class 10 micro SD card in a micro SD to USB adapter.  The generic 128gb SSD is only used for the Datastore.

Multi-WAN with ATT IPDSl and PFsense.

So in the move I lost my ultra fast Charter 60mb cable I went on the prowl.  Since the new house is in a more rural setting I was forced to go with ATT IPDSL.  Luckily though I was able to sign up with ATT Business so I can get more than just 1 3mb pipe.  In the end I wound up with 3 x 3mb DSL lines, I can add a last one making 4 if the need ever arises.

Anyway since I wanted to stick with using PF as a firewall I added a second PCI gigabit nic, and created 3 WAN interfaces.

Now many people have had issue with this in the past, especially when using the Motorola NVG510 DSL modems.  These models do not have a true bridge mode.. only what Moto calls an 'IP Passthrough' which in theory should be Bridge mode and apparently sometimes, for some folks it does not work correctly.

So anyway I setup IP Passthrough mode on all three of my modems, picking DHCPS-Fixed and I specified the MAC address of teh nic that particular modem was uplinked to.  While inside I also disabled WIFI since I provide my own with an 802.11N wifi ap with a MUCH better range than these crappers. I also disabled all other Firewall features.. since PF will do a much better job anyway I don;t want my traffic being looked at twice.

Then within PF I configured all three interfaces was WAN, WAN1 and WAN2.. then I gave each a unique public DNS server as a Monitor IP.  If the interface loses any pings to that host it will consider the link down.  Two of my WAN interfaces use Google's public DNS ip's.. because it'll be a cold day in hell when those ever go down. Then after that I rebooted each modem.. once up the modems are supposed to give the PF interface a 192.168 address for approx 3 minutes.. then it should pass it's own external IP to PF.  Now what happened in my case was that my Primary WAN circuit did that.  The last two did not.. PF kept using a 192.168 address but the were passing data correctly so I didn't argue about it.

Anyway once PF could see all three gateways (or Monitor IP's) up I then created a Gateway group named 'LoadBalance', then created a firewall superseding the existing LAN Net out rules specifying the gateway group.

That's basically it, I also posted about it on the PF Forums as well
https://forum.pfsense.org/index.php?topic=87639.0

PfSense nic compatibility issue

So lately I've had a weird issue with my Pf box at home, the WAN throughput chart scales from 0 to 55+Mb/s constantly when the kids are up, Rokus in use.. Ipad, etc.  However Ookla tests always hit a hard limit of 21mb on the download.

Nothing I do exceeds this limit, aside from bypassing Pf.  I questioned my hardware at first.. a Dell Optiplex 320, 1.6ghz P 2140, 2gb DDR2 ram, 80gb hdd, and a Broadcom 5709 dual gigabit PCIe nic.

Cpu utilization, and ram usage never spike.. hell it's hard to get them above 50%.  I was running avahi, pfblocker and a couple reporting packages which I have since removed.
My desktop is Windows 8.1 Enterprise, 8gb DDR3 2100, Core i7 3770 and a wireless N card.  Home network consists of a Netgear gigabit switch, and a Netgear 802.11N WAP.

Now obviously I know.. testing broadband throughput requires no other traffic on the lan segment.  Last thursday night was when I tried swapping my hardware out for a Lenovo SFF, Core2Duo 2.8ghz, 6gb DDR3 box with the same nics but guess what.. Pf would not recognize any card plugged into the PCIe slot.  My guess, damned things BIOS was locked to only being able to use a video card in that slow.. nothing else.  And my nic, the dual broadcom, was a PCIe x4...pc only had the x16 and a pair of x1's.
As a test, I grabbed a x1 broadcom nic from work.. didn't recognize it either.  I've read where Pf can sometimes have issues with Broadcom and Realtek nics, but this fresh 2.1.4 install would see and use the onboard Realtek.. but not the Broadcom.

Ok it's been a few days and I've gotten my PF config all working and working quite well.  I never was able to get the Lenovo box to work correctly because of those broadcom nics.  And I have not been able to determine why.. even the PF community couldn't answer it.  However my overall throughput issue was caused by a bad patch cable.  Yeah.. I know.

The last thing most of us think about is cabling... I have supported networks from the routing to the physical layer for over 14 years and I can count the number of bad cables (ones that had no physical damage) on one and half hands, seriously.

So now I'm able to push my 60mb download to the max and I have been doing so regularly ;)