TFS repository triggers, exhausting available socket connections

[Douglas]

On a daily basis, we receive these errors, it takes about 24 hours to build up enough socket connections to exhaust them. They are all from the build server to our tfs server.

Posted is an example of the action error:

Initializing Action...
RemoteAction Initialization Complete:
Temp Directory: C:\BuildMaster_AGTTMP_A34_S0\TMP
Source Directory: C:\BuildMaster_AGTTMP_A34_S0\WRK
Target Directory: [redacted]
Server: [redacted] (Id: 40)
Initialization Complete. Executing...
Executing Get Latest Command
An unhandled exception occurred while executing this action: System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress) at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Int32 timeout, Exception& exception) --- End of inner exception stack trace --- at Inedo.BuildMaster.Extensibility.Actions.RemoteCommandExecution.RemoteActionExecuterBase.remoteAction_RemoteCommand(Object sender, RemoteCommandEventArgs e) at Inedo.BuildMaster.Extensibility.Actions.RemoteActionBase.ExecuteRemoteCommand(String name, String[] args) at Inedo.BuildMaster.Extensibility.Actions.SourceControl.GetLatestAction.Execute() at Inedo.BuildMaster.Extensibility.Actions.ActionBase.ExecuteOnce()

An example of the buildmaster error log entry:

Unhandled Exception: Microsoft.TeamFoundation.TeamFoundationServiceUnavailableException: Team Foundation services are not available from server http://sdvtfs1:8080/.
Technical information (for administrator):
Unable to connect to the remote server

[Karl]

We'll have to look through the source code of the TFS extension to verify that all connections are properly closed. We can send you a beta of the extension once we determine the root cause.

[Douglas]

Thank you. We were unsure if that was the case or not.

[Ryan]

Our BuildMaster error log is still receiving these errors on a daily basis. I don't want this error to occur during a deploy again.

Have you been able narrow down what is causing the issue?

[Karl]

Unfortunately, we have not been able to repro this issue at this point. The only similar scenario we've found in other cases involving the TFSDK is: http://youtrack.jetbrains.com/issue/TW-13080

We do now have a new extension for TFS2012, it should be backwards compatible with 2010 (but it uses the new 2012 SDK). If you have BuildMaster 3.2.4 or later installed, you could try connecting through that to see if it's really a bug in the TFS SDK since we are not able to reproduce this.

If there is any other information you could provide, specifically: the BM version, whether the provider connects via a remote server ([along with its type of agent IIS or hosted), the OSes of both the BM server and the agent server, the authentication mechanism of the TFS provider, an indication of how many build schedules are using the provider, and anything else you could think of, we can try a more accurate reproduction.

[John]

I was finally able to reproduce this on our test server. I added several automatic builds and ran the service locally for a few hours. I could see via netstat that the TCP connections to our TFS test server were not going away (which I imagine after a day or so would eventually exhaust the available socket connections).

I've tried a few possible methods through the 2010 SDK and I am having some trouble figuring out how to close the connections, as the typical disposal process doesn't seem to be fixing this.

I will continue to investigate this issue and once I figure out a fix I will notify this thread and publish it to the extension gallery.

[Ryan]

That's great news John! (I had a 3 failed builds last night because of this issue)please let me know if you have any questions or if there is any additional information I can provide.