Execution hangs during deployment

We upgraded to BM version 4.8.3 (Build 9) on Saturday 9/12/15 and have seen some issues with deployments hanging after the upgrade. Some of our deployments appear to be hanging for 15 minutes or more before deployment steps are executed. This problem is not consistent so we are trying to nail down the cause of the issue. We think it might be related to when the BM server tries to talk with agents on servers that have been shutdown ... but that is just a working theory. It might also be the agent upgrades after the server upgrade ... perhaps there is a problem with the new BM client version? In any case deployments which typically would complete in less than 30 seconds are now running as long as 17 minutes or more. We saw there is a new release but it doesn't appear to contain a fix for the type of issues we are seeing. We would appreciate any info or guidance you can provide on this problem. Are there troubleshooting steps we can follow to find out what is causing the delay.

Product: BuildMaster
Version: 4.8.3

There have not been any updates that would have impacted this recently, or known issues like this. Unfortunately, there are a lot of things that can cause this, but one of the most common ones is one or more agents with a bad network connection.

Because agent communication is pooled, and I/O is a bottleneck, often time one server is waiting for another to finish, sometime it can be difficult to tell which specific agents are causing the problem.

The easiest way to find problem agents is this...

Stop the BuildMaster service from the Web UI, then Start it
You will see a bunch of "AgentUpdaterChecks" running simultaneously
These should take seconds at most to complete
Try to identify ones that are taking too long
Once you've found agents that never seem to complete the check, stop the service, then disable those servers (Edit Server > Active)
Go back to the beginning

Once you isolate the servers, it should be relatively easy to find them out.