All articles

Three healthy components, one broken MASQUERADE

A debug story: NodePorts stopped answering after a Netbird upgrade. The cause was a Netfilter mark shared by three components.

· 2 min read

After upgrading Netbird to v0.30+, NodePort access stopped working on our OVH OKS clusters. Nothing was crashing. Pods were healthy, nodes were Ready, the Netbird agent was connected. Yet packets to a NodePort went nowhere.

First rule: what changed?

The only change was the Netbird upgrade. That narrows the search but does not explain anything yet: Netbird alone does not break NodePorts. Something was interacting with it.

Follow the packet

When every component looks fine, stop reading logs and look at the packet. nftables can trace it through every hook and chain:

nft add table inet debug
nft add chain inet debug trace '{ type filter hook prerouting priority -301; }'
nft add rule inet debug trace tcp dport 30080 meta nftrace set 1
nft monitor trace

(30080 is just an example NodePort. Priority -301 runs before the raw table, so you see the packet from the start.)

The trace shows which rules touch the packet, in which order, and which mark it carries at each step. That is how the problem stopped being “NodePorts are broken” and became “this packet is not getting masqueraded”.

How kube-proxy decides to masquerade

kube-proxy does not masquerade directly. It sets a bit in the packet mark (KUBE-MARK-MASQ) and a later chain (KUBE-POSTROUTING) masquerades whatever carries that bit. That is a contract: this bit means “masquerade me”.

Marks are a shared 32-bit space. Netbird and the OKS networking stack also use mark bits for their own purposes. After the upgrade, their bits overlapped with the one kube-proxy relies on, so the mark was being set or altered by more than one component, and MASQUERADE no longer matched when it should.

The fix, in two layers

  1. Workaround: a rule change validated on a live cluster, then industrialised as a DaemonSet that applies it on every node, so a node restart or a new node does not bring the bug back.
  2. Root cause: a documented issue upstream with Netbird, because a workaround you maintain forever is a debt.

What I take from it

  • Nobody owns a bug that lives between three components. Someone has to write it down. I also documented it in our internal wiki.
  • Trace first, theorise second. The trace gave a fact to argue about.
  • A shared namespace (marks, ports, table names, routing tables) is where “healthy” components collide.
Email me All articles