MetalLB on Kubernetes with pfSense

This is an addendum to the MetalLB recipe, explaining how to configure MetalLB to perform BGP peering with a pfSense firewall.

Preparation

Complete the MetalLB installation, including the process of identifying ASNs for both your pfSense firewall and your MetalLB configuration.

Install the FRR package in pfsense, under System -> Package Manager -> Available Packages

Configure FRR Global/Zebra

Under Services -> FRR Global/Zebra, enable FRR, set your router ID (this will be your router's peer IP in MetalLB config), and set a master password (because apparently you have to, even though we don't use it):

Configure FRR BGP

Under Services -> FRR BGP, globally enable BGP, and set your local AS and router ID:

Configure FRR BGP Advanced

Use the tabs at the top of the FRR configuration to navigate to "Advanced"...

... and scroll down to eBGP. Check the checkbox titled "Disable eBGP Require Policy:

Configure BGP neighbors

Peer Group

It's useful to bundle our configurations within a "peer group" (a collection of settings which applies to all neighbors who are members of that group), so start off by creating a neighbor with the name of "metallb" (this will become a peer-group). Set the remote AS (because you have to), and leave the rest of the settings as default.

Individual Neighbors

Now add each node running MetalLB, as a BGP neighbor. Pick the peer-group you created above, and configure each neighbor's ASN:

Serving

Once you've added your neighbors, you should be able to use the FRR tab navigation (it's weird, I know!) to get to Status / BGP, and identify your neighbors, and all the routes learned from them. In the screenshot below, you'll note that most routes are learned from all the neighbors - that'll be service backed by a daemonset, running on all nodes. The 192.168.32.3/32 route, however, is only received from 192.168.33.22, meaning only one node is running the pods backing this service, so only those pods are advertising the route to pfSense:

Troubleshooting

If you're not receiving any routes from MetalLB, or if the neighbors aren't in an established state, here are a few suggestions for troubleshooting:

1. Confirm on PFSense that the BGP connections (TCP port 179) are not being blocked by the firewall
2. Examine the metallb speaker logs in the cluster, by running kubectl logs -n metallb-system -l app.kubernetes.io/name=metallb
3. SSH to the pfsense, start a shell and launch the FFR shell by running vtysh. Now you're in a cisco-like console where commands like show ip bgp sum and show ip bgp neighbors <neighbor ip> received-routes will show you interesting debugging things.

Chef's notes 📓

Tip your waiter (sponsor) 👏

Did you receive excellent service? Want to compliment the chef? (..and support development of current and future recipes!) Sponsor me on Github / Ko-Fi / Patreon, or see the contribute page for more (free or paid) ways to say thank you! 👏

Employ your chef (engage) 🤝

Is this too much of a geeky PITA? Do you just want results, stat? I do this for a living - I'm a full-time Kubernetes contractor, providing consulting and engineering expertise to businesses needing short-term, short-notice support in the cloud-native space, including AWS/Azure/GKE, Kubernetes, CI/CD and automation.

Learn more about working with me here.

Flirt with waiter (subscribe) 💌

Want to know now when this recipe gets updated, or when future recipes are added? Subscribe to the RSS feed, or leave your email address below, and we'll keep you updated.

Your comments? 💬