Help needed with Tinc Setup on remote hosts and lots of ISPs / Failover Problems between ISPs

[Raimund Sacherer]

Hello List, 

This is our setup which we are trying in a couple of our remote offices:


+---------------------------------------+                         +-------------------------------------------------------------+
|                                       |                         |                                                             |
|   +----------------+                  |                         |   +---------+                                               |
|   |                |    +---------+   |                         |   |         |                                               |
|   |  +---------+   |    |         |   |                         |   |  ISP-A  +--------------+                                |
|   |  |         |   <----+  ISP-D  |   |                         |   |         |              |                                |
|   |  |  FW-01  |   |    |         |   |                         |   +---------+              |                                |
|   |  | Tinc A  |   |    +---------+   |                         |                            |                                |
|   |  +---------+   |                  |                         |   +---------+           +--v------+                         |
|   |                |                  |                         |   |         |           |         |          +----------+   |
|   |                |                  |                         |   |  ISP-B  +----------->    FW   +---------->          |   |
|   |  +---------+   |                  |                         |   |         |           | Cluster |          | Tinc (VM)|   |
|   |  |         |   |    +---------+   |                         |   +---------+           |         |          |          |   |
|   |  |  FW-02  |   |    |         |   |                         |                         +---^-----+          +----------+   |
|   |  | Tinc B  |   <----+  ISP-E  |   |                         |   +---------+               |                               |
|   |  +---------+   |    |         |   |                         |   |         |               |                               |
|   |                |    +---------+   |                         |   |  ISP-C  +---------------+                               |
|   |  FW Cluster    |                  |                         |   |         |                                               |
|   |                |                  |                         |   +---------+                                               |
|   +----------------+                  |                         |                                                             |
|                                       |                         |                                                             |
|            Remote Office X            |                         |                        Head Quarter                         |
|                                       |                         |                                                             |
+---------------------------------------+                         +-------------------------------------------------------------+
                                                                                             


FW-01 and FW-02 are Master/Slave firewalls (pfSense with Carp failover). We have currently 3 "remote offices" connected which have all basically the same setup, one office has more internet lines. 

Currently I have it configured this way:

"Remote Office 1", Tinc A: ConnectTo ISP-A and ConnectTo ISP-B over ISP-D, Tinc B: the same, ISP-E not used
"Remote Office 2", Tinc A: ConnectTo ISP-A and ConnectTo ISP-B over ISP-D, Tinc B: the same, ISP-E not used
"Remote Office 3", still not set up, here I have 5 Internet lines (as described below) and I am not sure how to set it up correctly.
 

Can you recommend me the correct or best-practice way to connect all those offices over their different connections to maintain a stable VPN system in which an outage or degradation of one ISP does not effect the network overall? Remote offices do not talk much between each other, the main communication is between remote office and head quarter, but that might change in the future. 




A) How to connect correctly Tinc A and Tinc B with Tinc VM?
Would it be a good design to interconnect like that: 

Tinc A via ISP-D with ConnectTo ISP-A, ISP-B, ISP-C
Tinc B via ISP-E with ConnectTo ISP-A, ISP-B, ISP-C
Tinc A having a higher priority than Tinc B


B) What if I have a remote office where I have (because of reliability problems) more Internet Connectivity?
In Remote Office 3 I have 3 ADSL, 2 2Mbit SDSL and 1 FTTH connection, if I have the two tinc daemons on two firewalls, how would I interconnect those? I would like at least the FTTH, one SDSL and one ADSL line to be in the VPN setup. But I only have 2 tincd's on the two firewalls.


C) I had a problem with ISP-A having a high packet loss in the direction from ISP-A to ISP-D. ISP-D to ISP-A was fine. Tinc did not switch from sending the VPN over ISP-A to ISP-B and I had communication problems until ISP-A had corrected their packet loss problem. 

Any Idea what went wrong? I tried to eliminate the ISP-A from the tinc config files in "Remote Office 1", restarted the tincd but it kept showing up and kept being elected as VPN Path. I later found out I had ISP-A also configured in "Remote Office 2" (which I thought I had not). 


I want to thank the developers and supporters of Tinc and I think it is a great piece of software and I am looking forward to expand it to be our only VPN Solution in the future as we are currently opening about 2-3 remote offices around the globe every year. 


Best Regards
R.