Message: Unhandled exception while attempting agent update check.

Joseph.Saladino_7499

Hello,

Just recently, we have been having agents fall into a "Checking..." state, which has caused issues with deployments. When checking the logs for the agents from the servers page, there are no errors listed. Similarly, we do not see anything listed in the event viewer on the machines themselves.

We did, however, notice an influx of the below error on the admin page:

System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.IO.Stream.ReadByte()
at Inedo.Agents.ServerConnection.<SendHandshakeAsync>d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Inedo.Agents.InedoAgentClientBase.<ConnectAsync>d__24.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Inedo.BuildMaster.Windows.ServiceApplication.ServerCheckerRunner.<CheckAgentAsync>d__10.MoveNext()

All of the affected servers are running v46 of the InedoAgent, and are Windows 2008 or 2012R2. We are running v6.1.27 of BuildMaster

Any help would be appreciated.

Thanks!

atripp

Hi @Joseph-Saladino_7499 ,

The message, "An existing connection was forcibly closed by the remote host" basically just means that the remote server abruptly hung-up the connection.

Based on the fact that there's no errors happening on the remote servers (agents), then I wonder if it's related to firewalls blocking a connection?

It could also be related to the agent refusing to connect because the AES encryption key is not correct. How are the agents configured (with / without encryption key?)?

Hope this at least points in the direction.

Thanks,
Alana

Joseph.Saladino_7499

@atripp Hello! Thanks for the reply!

We have the firewalls on the servers in question turned off, so I wouldn't think that would be the issue.

We are using AES encryption for the agent connections. They seem to work fine for quite a while, then suddenly fall into the "Checking..." state after a random period of time. There doesn't seem to be any rhyme or reason as to why they start failing to connect. This started happening about a month ago, but we cannot figure out what might be causing the connection to drop.

atripp

Hi @Joseph-Saladino_7499 ,

Hmm, that's really weird; the next thing I would try is to just restart all the machines (including the BuildMaster server). That's easy to do and the issue might go away. It could be a software error in the Windows networking stack

It's possible that there's some intermediate network device that's interfering. Router, QoS device, etc. That's harder to diagnose, and would require some tracing ... starting by making sure that the RST packet is actually being received on the BuildMaster server (that's usually what indicates the error message), and then figuring out what device is sending it, and why.

Our code doesn't work at the packet-level, but if the remote server (Inedo agent) is where the RST packets are coming from, it likely has to do with some obscure Windows setting... maybe even firmware on the network card. But at least something to look at.

Cheers,
Alana