Load Balance BuildMaster Instance

Hello BuildMaster ,

I have some issue on how we can load balance buildmaster instance ( the web server , the windows service , sql server ) , in order to have a buildmaster very scalable(very fast ) and high available ( in case of desaster use second server )

What I want to do to have a configuration Active - Active of 2 or more windows servers with buildmaster via F5 load balancer or any load balancer technology .

Should I consider to replicate some specific folder or configuration apart form the SQL Server Databe ( build master database ) ?

Thanks in advance

Andry

Product: BuildMaster
Version: 6.1.3

Hi Andry,

The BuildMaster web application is already very fast, and load-balancing is totally unnecessary. There are occasionally some problems that cause slowness (bugs, configurations), but load balancing will not fix them. Actually, even if you were to scale your usage of BuildMaster to 10,000 users, you would find performance just isn't going to be a problem.

We will eventually add support for this in BuildMaster and Otter, but only because some organizations require all applications to be "load balanced', regardless of the maximum peak load.

So let's talk about high availability, but more specifically disaster recovery. This is a complicated topic.

BuildMaster and Otter are Orchestration Tools (i.e. Orchestrators), and the idea of implementing "Automatic Rollover of Multi-master Replication for Orchestrators" is really not possible. I'll explain more below, but what you first need to ask is "how fast do I need my DR to act for BuildMaster/Otter".

For most organizations that heavily rely on our tools, that answer is "within two hours". Others, it's "within two days". We would really need to understand your requirements, because we have lots of BuildMaster and Otter customers with various offsite/geo DR configurations, and it is passes their semi-regular DR testing.

These are all “custom” configurations, and in retrospect. Eventually ,I want to write some documentation or blog posts about these different configurations. If you can share more details about your DR scenarios, then we can help describe and build this documentation.

Automatic Rollover of Multi-master Replication for Orchestrators

This is a very specific, “unsolvable” problem and is important to understand. Let me explain.

First, multi-master replication is advantageous because it acts as a sort of de-centralized ecosystem, while allowing for a single source of truth.

This is a theoretical problem that prevents multi-master replication from being used in server orchestration tools. We can have multi-master replication in ProGet, because it’s not a server orchestrator.

Think of a master as a totally separate, independent instance that can do multi-site replication over WAN (i.e. not a local cluster with shared storage)
You can only have one active master at a time
- Otherwise all masters will try control/orchestrate the same set servers
- This creates a totally unpredictable server environment
Thus you need multi-master communication, to “elect” a single, active master from all available masters
- When the active master is no longer available, the standby masters hold a new election and select a new active
- Active masters must always check with available masters to make sure there is no other active masters before acting
A major point of failure is when multi-master communication failures
- If two masters can’t communicate with eachother, they both will assume the active role
- Then you’re back at the unpredictable environment problem

There are two solutions we can think of to this problem:

Having a “master of masters” controller, but that is also susceptible to outages and a single point of failure
Using human intelligence to select a master, which works only as fast as a human can act

Given the risks, and the fact that humans will already be “working around the clock” in this problem space, the latter seems preferred. And in this case, if humans are available, they can also take more intelligent action than “switching a master instance”.. which means multi-master replication may not be so beneficial for orchestrators

Hi Alex ,

Thanks for your reply , it is giving me some clear idea now .

My goal is to have a buildmaster fully available in case of distaster , the I can rely on second server , with the less human intervention and downtime

First point

So , I will implement the following :

I will have 2 Master Servers , working with the same database . One will work permanently and the second Idle in case of disaster ( to start automatically or manually ) .
Then my questions are :

How should I configure both BuildMaster instance in order to have the same exact configuration on both server , regarding the version , the extensions and the temporary files ?

To proceed in that way I have some ideas :

Put in NAS / Shared Server these buildmaster "mandatory" paths , to change in /administration/all-settings page , on both buildmaster instance :

Extensions.ExtensionsPath ( Extensions)
System.ServiceTempPath ( BuildMaster Servcie Temp path )
System.WebTempPath ( Build Master Web app Path )
Artifacts.BasePath (Artifacts path )

In That way , the 2 Buildmaster Servers will have the same DATABASE and WORKING PATH at any time

Is my technical approach correct or I miss something ?

Second point

Is possible , to "dockerize" , the BuildMaster Instance ( docker container , windows nano server ) ? and would you recommend it ?

I know that is very easy for web application but the windows service is a challenge or not possible because Buildmaster should work on a physical windows filesystem for the "vital path" ( artifact , extensions , temp path )

I'm asking these questions in order the better architecture possible in order to take a fully adavantage of BuildMaster .

Best Regards ,

Andry

apxltd

Hi, sorry on the slow reply!

I think you've got a good undersatnding of the situation, but a couple of comments:

do not share the Temporary files (ServiceTempPath, WebTempPath); these should be kept on the same server as the BuildMaster web/service app. These are only used during the runtime of those applications.
on the back up server, make sure both the Web Application and Service are set to DISABLED, or configured with a bad username/password such that they cannot be started easily; having two identical BuildMaster instances pointing to the same database and the same agents will cause that multi-master problem you dont' want to deal with :)
we have had some customers put our products in a Windows Container, but largely the support on the Microsoft side isn't so great, and it's more trouble than it's worth; we are moving towards DotNetCore so we can have BuildMaster run on Linux (and linux containers)

Does that make sense? Let me know how it goes.

And by the way, I'd love to document this better... would you be interested in helping me with this, especially once you have it running in your set up? I think it would really help the community :)

Hi Alex ,

Thank you for your feedback . I understand better now .

Of Course , I will share with you and the community , the way I'm doing when they are correctly setup , tested and approved .

Best Regards ,

Andry