tinc stopped working after restart

[Åsmund Rabbe]

Hi.

I'm in desperate need of some good advice.

I have a tinc network with 16 nodes. It's a star topology where all nodes
are connecting to the one node (Node1) that have a static IP.

Node 1 accepts incomming connections
Node 2 through 16 connects to Node1

One of the nodes (Node5) stopped working a while ago (2 - 3 weeks or so),
other than that everything was working fine. Today I decided to find the
problem with this one node (Node5) not connecting.
In the log of my star node (Node1) I found:

connection from AAA.AAA.AAA.AAA port AAAAA
connection closed by Node5 (AAA.AAA.AAA.AAA port AAAAA)
closing connection with Node5 (AAA.AAA.AAA.AAA port AAAAA)

this was repeating over and over. I've had some similar problems before and
it startet working again after a "tinc -n vpn restart".
So I tried "tinc -n vpn restart". Then hell brakes loose...

None of my nodes reconnected, well after a while one of them did (Node4).
All of the nodes but two ar located at very remote locations (three to six
hours away)

First thing I did was "tinc -n vpn" -> "log 5" a lot of error messages I
didnt understand
Started with checking my clock. It was 15 min out of sync. But it had no
effect to sync it.
Tried another restart and a reboot of the server, but no effect.

I've been trying to keep all nodes at same version (1.1Pre10) but not all
of them are.
Node1 is Pre10
Node 14,15 and 16 is at 1.1Pre11

IP1 through IP15 is substituted IP addresses

logentrys "log 5" in Node1 (1.1pre10 )(star node)
regarding Node15 (1.1pre11):
Connection from IP15 port 57815
Sending ID to <unknown> (IP15 port 57815): 0 Node1 17.3
Sending 11 bytes of metadata to <unknown> (IP15 port 57815)
Got ID from <unknown> (IP15 port 57815): 0 Node15 17.1
o2i_ECPublicKey failed: error:10067066:elliptic curve
routines:ec_GFp_simple_oct2point:invalid encoding
Sending METAKEY to Node15 (IP15 port 57815): 1 94 64 0 0 548E70.....
Sending 525 bytes of metadata to Node15 (IP15 port 57815)
Got METAKEY from Node15 (IP15 port 57815): 1 94 64 0 0 8254CB121.....
Sending CHALLENGE to Node15 (IP15 port 57815): 2 C25C898C33.....
Sending 515 bytes of metadata to Node15 (IP15 port 57815)
Got CHALLENGE from Node15 (IP15 port 57815): 2 E70D6E51C4.....
Sending CHAL_REPLY to Node15 (IP15 port 57815): 3 8E1B60823B9......
Sending 43 bytes of metadata to Node15 (IP15 port 57815)
Got CHAL_REPLY from Node15 (IP15 port 57815): 3 1061E9F77.......
Sending ACK to Node15 (IP15 port 57815): 4 CAAjAosXiIur.......
Sending 93 bytes of metadata to Node15 (IP15 port 57815)
Connection closed by Node15 (IP15 port 57815)
Closing connection with Node15 (IP15 port 57815)

logentrys "log 5" in Node1 (1.1pre10)(star node)
regarding Node2 (1.1pre10)
Connection from IP2 port 57870
Sending ID to <unknown> (IP2 port 57870): 0 Node1 17.3
Sending 11 bytes of metadata to <unknown> (IP2 port 57870)
Got ID from <unknown> (IP2 port 57870): 0 Node2 17.1
Peer Node2 (IP2 port 57870) tries to roll back protocol version to 17.1
Error while processing ID from Node2 (IP2 port 57870)
Closing connection with Node2 (IP2 port 57870)

logentrys "log 5" in Node2 (1.1pre10)
regarding Node1 (1.1pre10):
Trying to connect to Node1 (IP1 port 655)
Connected to Node1 (IP1 port 655)
Unable to read ECDSA public key: error:0906D06C:PEM
routines:PEM_read_bio:no start line
Parsing ECDSA public key file `/etc/tinc/vpn/hosts/Node1' failed.
Sending ID to Node1 (IP1 port 655): 0 Node2 17.1
Sending 14 bytes of metadata to Node1 (IP1 port 655)
Got ID from Node1 (IP1 port 655): 0 Node1 17.3
Sending METAKEY to Node1 (IP1 port 655): 1 94 64 0 0 A087953A.........
Sending 525 bytes of metadata to Node1 (IP1 port 655)
Connection closed by Node1 (IP1 port 655)
Closing connection with Node1 (IP1 port 655)
Could not set up a meta connection to Node1
Trying to re-establish outgoing connection in 475 seconds


Keep in mind everything worked flawlessly(almost) before the restart of
Node1.
Node2 and 3 have the same version 1.1Pre10 I've tried to restart Node3 but
it had no effect. Messages are the same as in Node2

Node4, the only one that reconnected have 1.1pre10 protocol 17.3
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.tinc-vpn.org/pipermail/tinc/attachments/20150518/64c654d7/attachment.html>