Bridged nodes sharing local IP to be used as gateway for LXC

[Mike Machuidel]

Hello everyone,

I am running multiple nodes each having multiple LXC containers. On
those nodes I am using Tinc to share the subnet 10.20.0.0. The LXC
containers may migrate between nodes and should keep the same
configuration.
On all nodes traffic from the external interfaces (connected to the
internet) is forwarded to the bridge interfaces and masqueraded. The
LXC containers are using 10.20.0.1 as their default gateway which
should stay the same independent of the node they are running on.

How to do the following while running Tinc in switch mode,

on Node A:

  ip link set dev eth0 up
  ip addr add 192.168.200.30/24 dev eth0
  ip route add default via 192.168.200.1 dev eth0    # Gateway
connected to the internet

  brctl addbr br0
  brctl setfd br0 0
  ip link set dev br0 up
  ip addr add 10.20.0.1/16 dev br0   # using same IP on node as node B
so the LXC containers can use same gateway IP
  iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
  ip route add 10.20.0.0/16 dev br0

  sysctl -w net.ipv4.ip_forward=1

on Node B:

  ip link set dev eth0 up
  ip addr add 192.168.160.10/24 dev eth0
  ip route add default via 192.168.160.1 dev eth0    # Gateway
connected to the internet

  brctl addbr br0
  brctl setfd br0 0
  ip link set dev br0 up
  ip addr add 10.20.0.1/16 dev br0   # using same IP on node as node A
so the LXC containers can use same gateway IP
  iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
  ip route add 10.20.0.0/16 dev br0

  sysctl -w net.ipv4.ip_forward=1

meaning that both nodes will share the same IP which the LXC container
can use as their default gateway?

To be honest I already tried this configuration. It ran well for about
2 days till I noticed that sometimes LXC containers on Node B were
using the default gateway on Node A and the other way around.

Is it possible to force the 10.20.0.1 on the local interface to be
used instead of the traffic being bridged over to the other side? Like
setting some metrics?

Best regards,

Mike