MetalLB on Kubernetes with pfSense
This is an addendum to the MetalLB recipe, explaining how to configure MetalLB to perform BGP peering with a pfSense firewall.
Complete the MetalLB installation, including the process of identifying ASNs for both your pfSense firewall and your MetalLB configuration.
Install the FRR package in pfsense, under System -> Package Manager -> Available Packages
Configure FRR Global/Zebra
Under Services -> FRR Global/Zebra, enable FRR, set your router ID (this will be your router's peer IP in MetalLB config), and set a master password (because apparently you have to, even though we don't use it):
Configure FRR BGP
Under Services -> FRR BGP, globally enable BGP, and set your local AS and router ID:
Configure FRR BGP Advanced
Use the tabs at the top of the FRR configuration to navigate to "Advanced"...
... and scroll down to eBGP. Check the checkbox titled "Disable eBGP Require Policy:
Isn't disabling a policy check a Bad Idea™?
If you're an ISP, sure. If you're only using eBGP to share routes between MetalLB and pfsense, then applying policy is an unnecessary complication.1
Configure BGP neighbors
It's useful to bundle our configurations within a "peer group" (a collection of settings which applies to all neighbors who are members of that group), so start off by creating a neighbor with the name of "metallb" (this will become a peer-group). Set the remote AS (because you have to), and leave the rest of the settings as default.
Why bother with a peer group?
If we're not changing any settings, why are we bothering with a peer group?
We may later want to change settings which affect all the peers, such as prefix lists, route-maps, etc. We're doing this now for the benefit of our future selves 💪
Now add each node running MetalLB, as a BGP neighbor. Pick the peer-group you created above, and configure each neighbor's ASN:
Once you've added your neighbors, you should be able to use the FRR tab navigation (it's weird, I know!) to get to Status / BGP, and identify your neighbors, and all the routes learned from them. In the screenshot below, you'll note that most routes are learned from all the neighbors - that'll be service backed by a daemonset, running on all nodes. The
192.168.32.3/32 route, however, is only received from
192.168.33.22, meaning only one node is running the pods backing this service, so only those pods are advertising the route to pfSense:
If you're not receiving any routes from MetalLB, or if the neighbors aren't in an established state, here are a few suggestions for troubleshooting:
- Confirm on PFSense that the BGP connections (TCP port 179) are not being blocked by the firewall
- Examine the metallb speaker logs in the cluster, by running
kubectl logs -n metallb-system -l app.kubernetes.io/name=metallb
- SSH to the pfsense, start a shell and launch the FFR shell by running
vtysh. Now you're in a cisco-like console where commands like
show ip bgp sumand
show ip bgp neighbors <neighbor ip> received-routeswill show you interesting debugging things.
Chef's notes 📓
If you decide to deploy some policy with route-maps, prefix-lists, etc, it's all found under Services -> FRR Global/Zebra 🦓 ↩
Tip your waiter (sponsor) 👏
Did you receive excellent service? Want to compliment the chef? (..and support development of current and future recipes!) Sponsor me on Github / Ko-Fi / Patreon, or see the contribute page for more (free or paid) ways to say thank you! 👏
Employ your chef (engage) 🤝
Is this too much of a geeky PITA? Do you just want results, stat? I do this for a living - I'm a full-time Kubernetes contractor, providing consulting and engineering expertise to businesses needing short-term, short-notice support in the cloud-native space, including AWS/Azure/GKE, Kubernetes, CI/CD and automation.
Learn more about working with me here.
Flirt with waiter (subscribe) 💌
Want to know now when this recipe gets updated, or when future recipes are added? Subscribe to the RSS feed, or leave your email address below, and we'll keep you updated.