Setting up a fast wirespeed gigE firewall/router w/ QoS/VPN

[Bingie 99]

Hi all,

Finally getting around to my next project.  I want a really fast firewall, only need 2 gigabit ethernet ports (but more is always better) but is fast enough to handle wirespeeds, even enabling encryption like a vpn.  Prefer runs something industry standard like pfsense, openwrt, routeros etc.

Anyone running anything like this?  I'm looking for ideas.  OS doesn't need to be pre-installed, might just get a fast little box and install OS on it, if it's supported.

Thanks

 

Edited February 25, 2022 by Bingie

[Bingie 99]

I think I found a nice 4-port mini-pc designed to be used as a firewall/router, and supports all of the firewall distro's like pfsense etc.

https://www.amazon.com/Fanless-Gigabit-Firewall-Appliance-Computer/dp/B09J4J73YW/ref=sr_1_6?crid=3BJ0KIBWHV9SH&keywords=netgate&qid=1643510850&sprefix=netgate%2Caps%2C158&sr=8-6&th=1

A little pricey at $248 but sounds perfect for this project.

[Bingie 99]

For anyone reading this, I tried both pfsense and OPNsense on the above hardware, and even though they install and run fine, I didn't care for either.  They are both based on BSD instead of linux, the bootup sequence is stupid (you have to wait an additional minute while it errors out looking for something), and the web gui is not very intuitive at all.

I then installed VyOS on the mini-pc, and it runs beautifully.  VyOS was built from the ground up on Debian linux, and it has a command line configuration tool just like Cisco IOS routers, yet you also have access to Debian linux itself.  The router commands have been integrated into the Debian bash shell.  It's brilliant.  It's been around for many years, so works very well and is incredibly robust.  Oh, and it's free open source.  w00t!

https://vyos.io/

I'm really happy with it so far.  The hardware is overpowered for a home firewall/router.  CPU utilization barely tops 1% when doing gigabit wire speed routing tests.  I could easily install a VM server on it, but I'm keeping the vyos install baremetal, so in the future, if there are any problems, I don't have to figure out whether it's vyos or the VM server software.

I'll post more here, when I finish getting things configured.  It's already up and running, and I'm using it as my home firewall connection to the Internet.  I have routing, NAT and firewall setup.  I'm still working on the wlan and openvpn settings.  When I'm done, I'll post a default configuration file here, so you can see how easy it is to configure.

[Bingie 99]

Today I setup the wireless LAN interface and tested it.  It's 802.11b/g/n and 2.4G but not the 5G.  2.4G might be okay to stream one tv up to 1080p.

They do make this server with different hardware configurations.  There is also a 2-port GigE switch with 5Ghz wireless, if 2 LAN ports is enough  That box would probably be fine for most home firewalls.  I wanted the above 4-port GigE box though, so I could add a DMZ and LAB ethernet zones.  I already have wireless access points, so I probably won't even use the built-in 2.4Ghz wireless.

 

[Bingie 99]

I setup openvpn on the router, to connect my entire home network to NordVPN.  It works flawlessly.

I have a 200Mb/s Internet link, and this little box is so fast, when doing a speedtest.net over the openvpn tunnel, it maxes my Internet link, even with everything encapsulated, and cpu hits about 30%.  I'm tempted to upgrade my internet pipe to something faster, just to test this box LOL.

speedtest over non-encapsulation barely pushes the cpu over 1%.  This box is plenty fast.  Overkill really, but that's better than underpowered.

 

[Bingie 99]

Out of curiosity, I did a double nested VPN test today (vpn within vpn), and NordVPN has a datacenter in Ukraine, in Kyiv.  I was pleased to see my full bandwidth 200Mb/s was sustained to Kyiv.  The first vpn encryption was on my pc, but the router had to re-encrypt the encrypted packets on the firewall.  The cpu on the firewall went up to 35%, which for a 2-hop openvpn, I think is pretty impressive.  I'm really liking vyos and this little firewall box.

[rbjtech 4257]

Does vyos have a well supported IPS ? (snort etc?)

That is the feature that usually slows down edge devices.

The mini-pc you have is a nice balance of hardware - I still use a 5 year old Asus N3150 mini-ITX which has an integrated 4 Core CPU (Celeron) but only @ 1.6Ghz.

tbh- mine has never hit more than 50% peak cpu (with IPS) so your should have plenty of grunt for future needs. 

 

[Bingie 99]

No IDS/IPS, not officially.  You're not the first to ask for that feature.

But... when you login to a vyos router, you are at the Debian linux command prompt.  If you want, you can enable debian repositories, and install anything you want.  Some people have done that.  It's not recommended obviously, because you then risk breaking vyos features.  I won't do it.

I picked the above hardware because the chips support gigabit wirespeed in hardware.  I can route at gigabit wirespeed and the cpu barely gets over 1%, which leaves the cpu available for things like vpn encryption.  You can still set firewall rules both in vyos and in debian linux iptabes, like blocking an address that makes too many failed access attempts.

I keep a half dozen ssh sessions open, each one running iftop for each interface, including the wan and the openvpn tunnel, so I can see in real time who is talking to who.  Makes it super easy checking for things like dns leaks.  The only traffic leaving my router goes to a nord vpn server.  I can see everything is fine, which is good enough for me.  Don't think I need intrusion detection anymore, since I'm not leaving any external ports open, not even ping.  I drop everything.

I'm in love with vyos.  Every day, I smile as i check everything.  It's incredibly stable, no reboots after making lots of changes.  It's been around for years, so it's seasoned.  If you like debian linux, and command line routers, you'll love it.  So much easier IMHO than pfsense or OPNsense, which lock you into a GUI that isn't very intuitive, and the shell access on those is quite limited, even if you do know BSD.

 

[Carlo 4330]

On 2/28/2022 at 10:02 AM, Bingie said:

I setup openvpn on the router, to connect my entire home network to NordVPN.  It works flawlessly.

I have a 200Mb/s Internet link, and this little box is so fast, when doing a speedtest.net over the openvpn tunnel, it maxes my Internet link, even with everything encapsulated, and cpu hits about 30%.  I'm tempted to upgrade my internet pipe to something faster, just to test this box LOL.

 

On 2/28/2022 at 8:34 PM, Bingie said:

Out of curiosity, I did a double nested VPN test today (vpn within vpn), and NordVPN has a datacenter in Ukraine, in Kyiv.  I was pleased to see my full bandwidth 200Mb/s was sustained to Kyiv. 

 

On 3/6/2022 at 10:49 AM, rbjtech said:

Does vyos have a well supported IPS ? (snort etc?)

That is the feature that usually slows down edge devices.

Something to keep in mind.  You really don't want any software running on your firewall.  You want that to be a brick wall and very, very simple to use so you always fully know how it's configured.  The only time you should need to touch it is to close or open a port, hopefully assigning it a vlan and a destination address.  You don't want to run dhcp, dns, vpns, ids, filter or any thing else on it.  To many people use high end software "appliances" this way and you're just asking for trouble.

Use the KISS formula.  Your ISP provided firewall is probably plenty and all that's needed for a "FireWall".  Run your "Swiss-Knife" super-de-duper Security Appliance inside your firewall so you have an extra layer protecting you when you make a boo-boo not understanding the outcome of the setting you changed.  It's really easy to fubar setups when your doing routing, vlan switching, IDS, IPS, DHCP, DNS filtering, Geo location, proxy, etc  systems.

Far too much software running that was "put together" but not written together so the left hand doesn't know what the right hand is doing and that's one way exploits happen. You're far better off setting up your VPN, DHCP, DNS, QOS on an interior machine real or virtual. Then do the same with Pi-Hole with some additional filtering on it. Wrap that up in a VLAN so there is one way in and one way out which passes by your IDS on one side and IPS on the other side. I say this because they work drastically different.  IDS is a monitoring system, while IPS is a control system. IDS doesn’t alter the network packets in any way, whereas IPS prevents the packet from delivery based on the contents of the packet, much like how a firewall prevents traffic by IP address. Forget the UTM (Unified Threat Management) way of having everything running together as it's a terrible idea.  If something is going on in the network and that one box gets overloaded you won't be able to use the tools to understand what's going on or what the cause is.  Breaking things down so that one box does one thing make it very clear at a high level where the problem is.  Setup a logging server where all apps dump logs and spend the time to setup monitoring against the logs.

That sounds like a lot of machines but it's really not.  They don't need to be physical and can be ran virtual or a combination of both.  Hell a couple of Raspberry Pis can do a lot of this like Pi-Hole, DHCP, DNS,  and maybe VPN and QOS depending on network speeds.  The nice thing about VM is that you can snapshot them and have a couple of versions to roll back a bad upgrade or to deploy a second copy if needed.  If somethings up/strange you could fire up a 2nd or 3rd IDS placed inline with specific things to look for.

So nothing wrong with pfSense, OPNsense or Untangled and you could run all 3 in small VMs using each for what it's best at but do yourself a favor and don't try to use it as the brick wall. Instead let it compliment your oh-so-simple firewall that only serves the purpose of opening ports and redirecting them and maybe tracking bandwidth but that's it.

The three just mentioned are UTMs that can do a bit of routing type work but honestly not that great. 
Vyos is the flip side in that it's a router pure and simple that trying to do a bit of firewall/UTM but isn't that great. It is fast however as I've had it running doing 40Gb using 4 10Gb NICs without breaking a sweat.  It excels at policy based routing, traffic shaping, VLANS to any kind of VPN. It's at home routing between physical and virtual networks and VLANs while being able to analyzing traffic for anomalies and collecting statistics.  I like to think of it as "Traffic Management"

Also what many people do wrong and blow it.  You have your file server or NAS with all you documents and media.  You've got this awesome security in place at the door keeping people out.  What happens when you, the kids or wife open the wrong email or walks a virus or malware in the door on the notebook used at work?  Does it have a clear path to encrypt or destroy all your files?

What/how do you combat that?  Have anything sitting in front of your servers (real or virtual)? Anything monitoring what's going on inside the box? Have snapshots of data/files to easily reverse anything thing done? Have any "honey traps" in place that give you immediate feedback when touched in any way?  For example nothing touches or scans a few parts of the system as it's a trap. Just pulling a directory  or viewing the files is a notification. Actually touching one can quickly shut the machine down.

Monitoring inside the network is just as important if not more so then standing at the door.  Often times the most harm is done because someone held the door open for them to enter.

[Bingie 99]

@cayars no more coffee for you

That's a lot of great info, and I agree with all of it.  KISS is the way to go.  That's why i'm running bare metal, and I've already stripped off a lot of features, trimming it down to minimal.

I'm back to pfsense for now, trying to figure out if I have a dns leak.  pfsense may be much slower than vyos UGH but it does have great real time monitoring tools, and much more robust dns options, which sadly, are almost non-existent on vyos.  Like you suggested, I've already stripped out a lot of the "features" clutter, keeping it simple.  Easier to troubleshoot.

I'll hopefully go back to vyos for the speed, but it's lack of management and dns options might force me to pay for untangle instead.  Those bsd guys over at pfsense need to wise up, and switch to debian linux, just like TrueNAS did with CORE.

It's late, I've had a rough day, but will re-read your post in the morning after some coffee Thanks for all the great feedback, a perfect addition to this thread which I hope others might find helpful.

 

[Bingie 99]

Thanks @cayars that's a lot of great info!

I've heard of pi-hole before, but never tried it.  I just read up on it, and sounds like a great idea.  Maybe it can stop the popup ads on my new roku tv's too HAH been meaning to track those down.  I have a spare mini-pc that will run pi-hole just fine.  Another project LOL! 

I finished my testing, going back to vyos on this nice hardware box.  I miss the speeds, and easy to configure file.

I'll have to ponder the IDS/IPS system.  I no longer have any ports open to the Internet, but as you mentioned, internal threats such as opening a nasty link or pdf file is much more common these days.  Maybe that will be my next project, after I switch this router back to vyos, and install pi-hole on the spare mini-pc to handle dns and dhcp.

 

[Carlo 4330]

Pi-Hole is pretty easy to setup and allows for easy testing as the only change to your network is the DNS setting.  So you can test on one machine until your satisfied.  Then you can modify your router or DHCP server to use your new Pi-Hole machine for DNS.

It does some good stuff like encrypted DNS lookups which is what a lot of people goof on using VPNs since their ISP gets a lookup request for everywhere they go.
In Pi-Hole you'll change this to a faster DNS forwarding server that's also secure.  You can set filtering and I don't mean just "adult content" but notoriously bad domain names/ips.
That last one alone is worth installing for.  The add blocking is nice but of course doesn't unblock everything as it only works for ads using specific domains.

You might want to consider using the mini-pc to run docker.  I can give you a list of cool docker images worth having.  One is a security scan that you turn on, run, then can turn it back off if you want. Docker is useful for having a few network virus/malware checkers as well so you can run them on demand.  Even a very week PC is good in this way as you can turn on just the docker instance you want to use at that point in time (while leaving Pi-Hole running).

I personally like a couple network status/notification programs running all the time as well.  A quick glance at the dashboard after setup can show you everything from WAN bandwidth in  use to memory free on all your PC to disk access and latency times and makes drilling down to find issues quite easy as you have a history view to look at.  But you want to talk about a "time suck" you can play for days and days so you have to learn "good enough".

On 2/25/2022 at 12:08 PM, Bingie said:

For anyone reading this, I tried both pfsense and OPNsense on the above hardware, and even though they install and run fine, I didn't care for either.  They are both based on BSD instead of linux, the bootup sequence is stupid (you have to wait an additional minute while it errors out looking for something), and the web gui is not very intuitive at all.

I know you didn't care for pfSense or OPNSense but I bet they have a lot more to offer you then vyos does.  Do you really need advanced routing (more than pfsense) on your LAN?
pfSense for example can do Pi-Hole functionality and a lot of security filtering that would be hard to duplicate easily in vyos.

If you think pfSense isn't intuitive you should try learning to use any of the Mikrotik layer 3 router/switches. You'll be wanting the good old days of pfSense back.  LOL
I actually really like Mikrotik equipment and personally don't find them hard at all but most people find them overwhelming with so many features available.  People get hung up on things like the software showing all kinds of wifi, BGP options when your little "switch" doesn't have those features.  It's setup to run a campus or enterprise, so you can control the interaction between devices based on trunk ports, vlans or anything really.  Once that fact clicks it starts to make sense and you can turn off menu/features you don't have available.

But it's super cool to be able to hit one management interface and make adjustments to my adult son & daughters network in their houses which are connected via dedicated VPN to my system and part of my "campus". I can do things like dynamically change their routers "default" dns servers or update routes so packets leaving their house take a different path including the ability to use a relay or VPN server if it has better round trip times. I've got some scripting that does this automatically whenever my son is using his Xbox so he gets fantastic ping times to the servers he's using.  Things like this would be next to impossible with most software but is actually pretty easy with MikroTik.

[Bingie 99]

2 hours ago, cayars said:

I know you didn't care for pfSense or OPNSense but I bet they have a lot more to offer you then vyos does.  Do you really need advanced routing (more than pfsense) on your LAN?

Hi @cayars thanks, great info.  A few things:

Unfortunately, pfsense performance was terrible, compared to vyos.  pfsense takes several minutes to boot, as opposed to seconds, it takes over a minute to turn up openvpn tunnel, as opposed to seconds, etc.  The worst thing is the throughput.  I have a 200mb uplink to my ISP.  With vyos connecting my LAN to nordvpn via openvpn, I can do a speedtest and hit max upload speed with minimal cpu use, and no problems at all.  Not so with pfsense.  The same speedtest over the pfsense openvpn to nordvpn struggled to get up to about half of max speed (about 100mb), the cpu struggled, and there was a good 30% packet loss.  That's a joke.  They should be embarrassed.  BSD just doesn't do well on multi-core pc's, not like linux anyways.  I do miss the rich pfsense web configurator? tool though, it is really nice, once you get used to it.  vyos has nothing, unless you count snmp (not using) or ssh in, which I keep several shells open to run iftop on the wan and vpn tunnel, just to see what's going on.

Hey, I have a docker question for you.  The mini-pc has a valid win10 home license on it.  Is docker for windoze any good?  Or should I put debian server on it to get docker that way.  I don't want to put virtual box on windows, just to run dockers, that would annoy me, too wasteful.

I would be willing to put a bare metal hypervisor on it, although the only one I'm familiar with so far is Qubes OS, which I absolutely love on a laptop.  Big fun, once you used to it, make a new vm in seconds, and link it thru a vpn, so cool.  I still have a few linux vps servers out there, just to play around with.  I've been meaning to try the XenServer hypervisor, or maybe the Oracle VM server.  I've looked at both, but just haven't had a real need for one yet.  Maybe now I do.  With a type 1 hypervisor, I can run pi-hole in one vm, then whatever else I want too.  I like this idea.  I think I'll re-read some reviews of those.  Qubes uses Xen, so I'll start with the XenServer.

I've looked at Mikrotik, but when I read the RouterOS reviews, it struck me as too primitive.  Those boxes are dirt cheap though, no denying that.

 

[Bingie 99]

Figures, XenServer discontinued, the free version anyways, they want you to buy Citrix now.

Proxmox is based on Debian, that one looks good.  The Oracle VM Server is focused on redhat/centos and opensuse, and I ran all of those for many years, long enough to know I like Debian better.  No more rpm's for this kid.

 

[Bingie 99]

Okay, here's the latest:

 

1.  ROUTER MINI-PC:  Running VyOS router software built on Debian linux (on bare metal)

   * provides routing services for home network only (DNS/DHCP moved to a LAN server)

   * connects to NordVPN using OpenVPN tunnel, all Internet traffic isolated using NAT to VPN only, no leaks, if tunnel goes down, all Internet access stops

   * basic firewall enabled on router, all external ports closed including ping

 

2.  SERVICES MINI-PC:  Running Proxmox type-1 hypervisor (on bare metal)

   * provides virtual machines (VM) and linux containers (LXC)

   * running Pi-Hole in a linux container (debian), provides network ad blocking, DNS and DHCP services to home network

 

Things to do list:

1.  add a linux container on proxmox, to run influxDB and grafana, to remote monitor all my computers/servers/devices, starting with vyos router

2.  add some kind of network security app on Proxmox, to monitor all computers/servers/devices for suspicious activity, and alert me immediately

 

I'm working on the influxDB/grafana today.  For monitoring VyOS routers, it seems the most commonly used, so I'll start there.  I also see posts from people recommending other monitoring tools such as prometheus, zabbix, etc.  I may or may not try those, will cross that bridge when I come to it.

I haven't even started looking at intrusion detection tools yet.

 

[Bingie 99]

Followed this guide, for installing Pi-Hole on Proxmox in a LXC:

https://www.datahoards.com/installing-pi-hole-inside-a-proxmox-lxc-container/

 

[Bingie 99]

LOL Pi-Hole blocks all the ads on all of our Roku's now, even the new Roku tv's.  That alone is worth donating to pihole.

I see pihole can get donations every time you buy from Amazon, which we do, so I think I'll set that up.

 

[Bingie 99]

Okay, Proxmox on the spare mini-pc wasn't such a good idea, it's a waste of limited resources, and doesn't do docker, just lxc and vm's.

Installed debian stable on the spare pc, I'll just use docker instead.  Learning docker now, I have put this off for way too long

I guess I should post the vyos router config, it's done, and I'm thrilled with it.  I was waiting to add the snmp stuff to it before posting, but that will be a while before I have an NMS properly setup in dockers.

 

[Bingie 99]

Docker sucks.  Easier to just install the apps directly using deb packages, less hassle, less configuration.  Takes much less resources too.

[Bingie 99]

UPDATE:  It's been a while since I posted here, figure I'd give an update.

I built a custom Debian firewall a month ago (seems like months), it runs great.  It's been done for a while now, still evaluating different management packages.  Runs twice as fast as pfSense, and easier to configure and troubleshoot.

I wrote an extensive documentation project for it, put it up on github, but it's still flagged as private.  If anyone wants in, let me know, I'll add you to it.

It's a complete home router setup, includes: 

* debian stable only (bare metal, debian repositories only, no 3rd party)

* dnsmasq (dns & dhcp)

* openvpn client (to vpn service)

* hostapd (wireless access point)

* iptables (routing and firewall config, may add nftables option)

* snmp (for remote management)

* wireguard (testing still needs work, wireguard currently has severe security flaws, no longer recommended by experts, I'll add it anyways)

I'm still testing various nms packages, haven't found one I like yet.  Installing another one today, we'll see how it goes.

The management is the last piece.  I've already tried a bunch of options, stuff on the router vs. a separate management pc.  When I'm happy with management, I'll update all the other doc's, then finally release the project.  It's written in markdown (md) language.  Should be a hit, it's a gazillion times better than any other how-to doc's I could find on the Internet.

Cheers

[Bingie 99]

This looks promising.  A really nice GUI tool running directly on the router, updates in real time.

Now I need to build my own custom dashboard, just the way I like it.