Policy based routing with Tailscale

Introduction

I have been maintaining a private network of Linux hosts. That network has servers in different locations connected to VPN. I found Tailscale as a natural fit for a VPN, because mesh network between servers is most convenient topology for my servers.

One host, however, is a hypervisor hosting several Linux containers. That host runs both Tailscale and bridge interface to connect containers to the hypervisor and the rest of the private network. Tailscale supports this topology, thus, configuring the network was still straightforward. Both containers and servers were happily using a configured exit node.

One day, I decided that I need to make certain containers not to have access to Tailscale exit node chosen at the hypervisor. This wasn’t supported directly by any Tailscale UI, so I ran into roadblocks quickly. Then, I started to consider another options for the hypervisor.

The only way to manage the hypervisor is from within VPN — therefore, Tailscale interface should be installed on the hypervisor. This is where I considered a policy–based routing.

By using a knowledge of how routing in Tailscale and Linux works, I was able to implement my own routing alongside Tailscale’s. The implementation is simple and maintainable enough that I’m proud to share it with visitors.

Want to learn how I successfully implemented this approach? Want to know how to do a similar thing on your own server? Then this article is for you.

For this article, Tailscale’s version is:

1.98.5
  tailscale commit: AlpineLinux
  long version: 1.98.5-AlpineLinux
  go version: go1.26.3

Routing policy to be implemented

• Hypervisor needs VPN connection, but doesn’t particularly require using an exit node.
• All containers need all their packets to be routed to exit node by default.
• Some containers, however, instead require to be routed to host LAN only.

The difficulty

When I set:

tailscale set --exit-node "${your_exit_node}"

Tailscale didn’t ask me what packets should be routed to exit node. It just routed them all. And I discovered this is a built-in, not configurable by the CLI, behaviour.

The only option I found to affect routing with exit node is:

tailscale set --exit-node-allow-lan-access=true

It’s actually good. Leaving LAN routes was what I wanted. But still, all packets heading to the Internet have no choice other than getting to Tailscale exit node.

This essentially gave me a flow that is contrary to my idea:

To work around this, I decided to learn what actually happens behind all of that Tailscale UI.

How Linux routing works

Linux’s IP routing decisions can be affected by three prominent components:

Routing table

Given the destination address, matches it to the output interface and next hop. On modern Linux systems, routing tables are maintained by iproute2, particularly the ip route command.

Routing policy database (RPDB)

Maintains a sequence of rules matching each IP packet against certain routing tables. Packet can be matched by its source IP, interface name, port, mark (also packet mark, firewall mark, fwmark), etc. Also maintained by iproute2, particularly the ip rule command.

Netfilter

A set of tables, chains and rules that filter and process packets. Or firewall, to make it simple. Aside from filtering certain packets, Netfilter can also affect Linux’s routing decisions — the main mechanism for that is setting a packet mark — a 32-bit integer number assigned to packet. Every packet has one Netfilter mark, and by default its value is set to 0. Usually maintained on modern Linux systems by Nftables.

Given this information, I can clarify my PBR idea with how it actually works on Linux, and also study how Tailscale implements its routing policies.

How Tailscale routing works

While tailscaled is running, it maintains a set of RPDB rules and routing tables. First, let’s see what these rules are:

ip rule list

0:	from all lookup local
5210:	from all fwmark 0x80000/0xff0000 lookup main
5230:	from all fwmark 0x80000/0xff0000 lookup default
5250:	from all fwmark 0x80000/0xff0000 unreachable
5270:	from all lookup 52
32766:	from all lookup main
32767:	from all lookup default

Tables main and default are standard on Linux. Tailscale maintains rules 5210, 5230, 5250 and 5270:

• Rule 5270 makes every packet look up its destination at table 52.
• Other rules make a packet avoid doing that if packet mark is equal to 0x80000 within mask 0xff0000.

Mask acts as a bitwise operation:

fwmark & 0xff0000 == 0x80000

Also, unlike IP mask, fwmark’s masks are not necessarily a prefix, or a sequence of 1s, then 0s.

Table 52 is managed by tailscaled. Here is an example of it:

ip route list table 52

default dev tailscale0 
throw 10.36.1.0/30 
throw 10.36.2.0/24 
100.100.1.1 dev tailscale0 
100.100.1.3 dev tailscale0 
100.100.1.4 dev tailscale0 
100.100.1.5 dev tailscale0 
100.100.100.100 dev tailscale0 
throw 127.0.0.0/8 
throw 192.168.1.0/24 

In this table:

• A default route is set, because Tailscale exit node is in use.

• If the destination is LAN subnet, lookup is terminated as if the route wasn’t found, and thus, packet proceeds to next rule.

  This is only true if you set the following Tailscale option:

  tailscale set --exit-node-allow-lan-access=true

  If not, table 52 also intercepts LAN destinations.

• Each peer (addresses in subnet 100.100.1.0/24 in my case) is defined by one route.

The solution

Now, knowing all the details, let’s actually start modifying some packet marks to implement our own routing policy alongside Tailscale.

First of all, if you run this while tailscaled is running:

nft list ruleset

Then you may notice that tailscaled has already beat you to managing Netfilter itself. By default, it creates and manages its own tables. You can either study them and implement your rules alongside them, or decide that you don’t need them — perfectly valid option, peer connectivity in Tailscale won’t die from it.

I, for instance, do not like when services interfere with my server’s firewall. So let’s get that ugly shit out of here!

tailscale set --netfilter-mode=off

Now the Netfilter is entirely my responsibility. Just the way I like it.

Tweaking the routing policy database

When I started designing my routing policy, I saw 2 choices:

• Make the default behaviour not affected by Tailscale in any way.
• Make the default behaviour use Tailscale.

I find first option more pure and idealistically correct. However, implementing it would make me clash with Tailscale’s behaviour more than I perhaps want to. Tailscale already chose to incorporate iself in system’s RPDB — to override that, I must do the same thing. I will have to mint my fwmark and mask, then write the same amount of RPDB rules.

I decided to avoid that hassle and instead chose to integrate with Tailscale itself. I don’t have to create a new routing table. I can create less RPDB rules. Overall, I’ll get an easier to maintain system.

Because Tailscale already has a policy to bypass its own routing, I can use it as well. However, I’m not really a fan of 2 radical choices that Tailscale gives me:

• Route both to exit node and peers.
• Do not route either to exit node or peers.

This is where I add my own rule: all packets by default routed to Tailscale peers.

To do that, I need to somehow lookup table 52 while ignoring the default route. And for that goal, ip rule actually got a perfect option — suppress_prefixlength. suppress_prefixlength 0 ignores the default route (which is 0.0.0.0/0), while still looking up longer prefixes. That’s just what I need. All that’s left is to just put that rule before Tailscale’s rules — i.e. set a priority lower than 5210.

ip rule add pref 5190 fwmark 0x8000/0x8000 lookup 52 suppress_prefixlength 0

With fwmark 0x8000/0x8000, I chose to encode my rule into a single-bit flag stored in packet mark. I find this approach most convenient when adding new mark policies to existing system.

And now, the routing policy algorithm becomes:

Marking packets with Netfilter

And now I can finally start to make certain hosts not to use Tailscale’s routing policy and instead use mine. This part is actually fairly routine and requires basic Nftables skills to implement. All I have to do is to set packet mark to value 0x88000/0xff8000 via Nftables.

This chain will exclude hosts on the LAN.

 1table inet tailscale-pbr {
 2	set no_exit_node {
 3		type ipv4_addr
 4		elements = {
 5			10.36.2.5,
 6			10.36.2.2,
 7		}
 8	}
 9
10	chain prerouting {
11		type filter hook prerouting priority mangle
12		ip saddr == @no_exit_node meta mark & 0xff8000 != 0x88000 \
13			meta mark set meta mark & 0xff007fff | 0x88000
14	}
15}

You may wonder why not just do this?

meta mark set 0x88000

By adding !=, & and |, I’m treating this rule not as a sole owner of packet’s fwmark, but only its bits needed for my policy.

I have also decided to store affected hosts’ IP addresses in a named set no_exit_node. This approach allows to modify the set without modifying the rule:

nft add element inet tailscale-pbr no_exit_node { 10.36.2.6 }
nft delete element inet tailscale-pbr no_exit_node { 10.36.2.6 }

This is convenient when the rule logic itself does not require to be changed. Set operations are also atomic, which is an especially useful property for PBR. Imagine you want a host to have LAN gateway permanently. That you won’t tolerate a single packet being sent to the Tailscale exit node. In that case, having non-atomic update would create a brief moment where host can send packets bypassing the PBR.

And last but not least, this chain will exclude the Tailscale host itself.

1table inet tailscale-pbr {
2	chain output {
3		type route hook output priority mangle
4		meta mark & 0xff8000 != 0x88000 \
5			meta mark set meta mark & 0xff007fff | 0x88000
6	}
7}

Testing this setup

For local host, this:

traceroute -m 1 nftables.org

Outputs:

traceroute to nftables.org (92.243.20.29), 30 hops max, 60 byte packets
 1  192.168.1.1 (192.168.1.1)  1.029 ms  0.952 ms  0.891 ms

192.168.1.1 is local ISP’s gateway.

Thus, local host is routed correctly.

For LAN hosts, this:

ip route get $(dig +short A nftables.org)
traceroute -m 2 nftables.org

Can give 2 possible routing results, depending on which host I run:

92.243.20.29 via 10.36.2.1 dev eth0 src 10.36.2.2 uid 0 
    cache 
traceroute to nftables.org (92.243.20.29), 30 hops max, 60 byte packets
 1  10.36.2.1 (10.36.2.1)  0.134 ms  0.048 ms  0.044 ms
 2  192.168.1.1 (192.168.1.1)  0.362 ms  0.323 ms  0.303 ms

Here, source address of a host is present in a no_exit_node Netfilter set, and thus, doesn’t use Tailscale exit node.

92.243.20.29 via 10.36.2.1 dev eth0 src 10.36.2.6 uid 0 
    cache 
traceroute to nftables.org (92.243.20.29), 2 hops max, 60 byte packets
 1  10.36.2.1 (10.36.2.1)  0.144 ms  0.053 ms  0.047 ms
 2  100.100.1.4 (100.100.1.4)  40.261 ms  40.900 ms  40.998 ms

Here, source address of a host is not present in a set, and thus, uses Tailscale exit node under 100.100.1.4 address.

Conclusion

With this solution, Tailscale continues to manage its VPN connection as usual, but now a Linux host decides who gets to enter the exit node. All of that was made with a few RPDB rules and Netfilter chains. While Tailscale doesn’t support PBR directly, it still leaves its own implementation simple enough for other system components to build around.