What is the correct process for deploying to multiple servers when one or more servers may not be active?

Guest

We have many servers in our cloud environment and have decided to enable rolling deployments to ensure the least amount of downtime for our users. What is the best process for deploying to large numbers of servers where one of more of them may not be powered up when the deployment runs?

I see that you can set resume on failure and also log errors are warnings, but it seems like buildmaster takes a long time get past the operations if the servers are not active.

Also, we tried using server groups, but even with resume on next turned on, if one of the servers in the group was not active, the process failed.

Any ideas?

Product: BuildMaster
Version: 4.1.0

Guest

Server groups would be the way to go for this sort of deployment. After some discussion, we decided on the following behavior: inactive servers should not be included as part of the server group when it comes to deployment. I think that would solve the issues you're having, and also seems to make more sense in general. The only issue I see is if a server gets disabled, and then it doesn't get deployed to when it should have, but I feel logging is enough to handle that case.

Let us know whether this makes sense or not.

Guest

I agree that inactive servers should be ignored and a warning issued. I am not even sure that in our case it will help since we are using the following scenario to achieve rolling deployment:

Stop IIS on server #1 (or copy app_offline.html to root) - this takes site out of rotation from our load balancer

First

Deploy files to server #1

Start IIS on server #1

Then

Stop IIS on server #2

Deploy files to server #2

Start IIS on server #2

This way there is always a server online to handle client requests.

What I was looking for in BuildMaster for this purpose was a way to either:

In the action group (right now I have the stop/deploy/start in separate group for each server), I should be able to check a box saying "Only perform this action group if server XXXX (or server group) is active"

Another option that will be much better once we move past 8 servers when creating separate groups for each one is too cumbersome would be to be able to apply a group of actions to a server group, where all the actions in the action group are applied to each server in order.

For example, if I apply my current action scenario to a server group using BM:

Stop IIS on server - oops.. now all servers are unavailable! :(
Deploy files to server
Start IIS on server

There are some cases where you might want it to apply to all for each action, but since everything else in BM is sequential, applying actions to groups should be as well (or at least have the option to be).

So to recap, Ideally need these features:

Do not execute action group if server inactive
Apply action group to server group sequentially (server 1 - action 123, server 2 - action 123, etc)

Thanks for the feedback, we will be purchasing enterprise version shortly.

Guest

We will at some point be reworking how executions are run to support this first-class. In the meantime, you can separate server groups into finer groups (e.g. ServerGroupA1, ServerGroupA2) then share the action groups within the same environment's plan using server variables such that they perform all the actions against ServerGroupA1, then once completed on that set of servers, they are performed against ServerGroupA2. This would give you the typical "take down half of the servers" rolling deployment scenario. If you adding extra server groups, further cut them down, and added more shared groups, it would let you take down a third, quarter, etc. at a time.

Guest

That is what I was thinking to do. Unfortunately at this time it doesn't properly work if one of the servers is not active, the entire action fails. I can set all the actions to treat errors as warnings, but this is less than ideal and it still waits forever to try to access the servers for each step. (I could set timeout but not even sure what that should be for a deploy job - if it had connection timeout, then it would work.)

We will be looking for a update then to address some of these features. Really enjoy using the product so far.

I need to upgrade to the latest version, but at least in 4.1.0, if I click on a existing server group, I get a 500 error.

Guest

We have added this fix to our issue tracker as BM-1428. Keep an eye out on the release notes for it (not sure if it will be 4.1.8 or in 4.2.X when it first comes out).