BuildMaster Configuration File Deployment

paul_6112

Hi Support,
When using the 'Deploy Configuration File' operation, if I have a configuration file with multiple Templates (using Key Value Pairs) it appears its not possible to select which Template is used only the Instance ?

Would this be something you could add please?

My use case is I have a config file that has additional settings depending on the server role, so I can use an 'If' block to deploy from the correct template.

The workaround seems to be using multiple configuration files and using the rename file; but this seems to defeat the purpose of multiple templates, given it is the same configuration file ?

Thanks,
Paul

For example test.config has two Instances with two Templates :

Only the Instance however can be selected on the deployment :

stevedennis

Hi @paul-reeves_6112 ,

This is by design; Configuration File Templates were intended to simplify maintenance of Configuration File Instances by combining common things into the template. Not saying it's the "right" design, but that's the use case.

However, we can definitely consider changing the behavior, to allow you to specify the default Template or a different Template when deploying.

Could I trouble you to share the configuration files (sensitive data redacted of course), so we can see the use-case better? We really want to document the configuration files better in the coming months, and having examples like this will help us tremendously.

We also want to make sure it's the best way to solve the problem. There is also the option of using those ASP/PHP-like OtterScript Snippets in Configuration Files, too. Maybe that's better to put that in your template? I don't know,...

Lots of options, and we want to make sure we document how/when to choose which ones.

Thanks,
Steve

paul_6112

Hi @stevedennis ,
Sorry, but I don't think I will be able to share the configuration files; but let me see if I can provide some examples to see if you can guide me on this.
I've also now had a play with Text Templates and there seems to be positives and negatives.

I run a distributed architecture application made up of many components across multiple servers.
We already combine the configuration to a suitable template as the various component applications by making use of the appSettings File attribute to point to a common file so for example myapp.exe.config and web.config will both use :

<appSettings file="C:\ProgramData\application.config">

This common application.config file is what I'm trying to template for the version control feature.
The predominant reason being if a new setting is added, it simply means I can update the template to push the new setting out.

application.config might look something like this as an example :

<?xml version="1.0" encoding="utf-8"?>
<appSettings>
<add key="Database" value="Server=1.1.1.1;User Id=user;Password=password;Initial Catalog=mydb;" />
<add key="InstanceId" value="1" />
<add key="Trigger" value="manual" />
</appSettings>

As I have a Test and Production environment which will have different settings in this common appplication.config file, I can use the Key/Value Pair template to cover this requirement.

<?xml version="1.0" encoding="utf-8"?>
<appSettings>
<add key="Database" value="Server=1.1.1.1;User Id=user;Password=password;Initial Catalog=$Database;" />
<add key="InstanceId" value="$InstanceId" />
<add key="Trigger" value="$Trigger" />
</appSettings>

Testing Key/Value Pair

Database = Test
InstanceID = $InstanceID
Trigger = Auto

Production Key/Value Pair

$Database = Prod
InstanceID = $InstanceID
$Trigger = Manual

InstanceId is bit more tricky, this is used if I have more than once instance of the same component installed in the same system; based on the Variable order of precedence I have been testing using a default Global Configuration Variable of

$InstanceID = 1

Then on specific servers introducing the Variable as required i.e.

$InstanceID = 2

So far I believe this fits in with Configuration File Templates, and seems to work as I can deploy the right Configuration File Instance to the right environment.
Based on my use (abuse?) of the variables, it seems I cannot manually deploy a configuration file however.

The reason for this request on the forum, is that a single Configuration File supports adding multiple templates within the single configuration file; but you cannot select which template is deployed from the Configuration File.

My particular use case to have multiple templates for the same configuration file is whilst application.config is common depending on the particular deployment and particular server role there can be a number of additional settings; i.e. conceptually the configuration file is deployed where
Instance = Environment
Template = Role
But Templates with Key/Value pairs lets me keep all this together in one interface.

Therefore whilst my 'default' template might be as first shown, I might have a second template version

<?xml version="1.0" encoding="utf-8"?>
<appSettings>
<add key="Database" value="Server=1.1.1.1;User Id=user;Password=password;Initial Catalog=$Database;" />
<add key="InstanceId" value="$InstanceId" />
<add key="Trigger" value="$Trigger" />
<add key="WebKey" value="$WebKey" />
</appSettings>

The reason this is separate is not all deployments will require the WebKey , and only the web server roles actually require this key.

Therefore based on a If block, I could select Deploy the configuration file version I want by selecting both the environment and template.

I believe I can accomplish something similar using Text Template

<?xml version="1.0" encoding="utf-8"?>
<appSettings>
<add key="Database" value="Server=1.1.1.1;User Id=user;Password=password;Initial Catalog=$Database;" />
<add key="InstanceId" value="$InstanceId" />
<add key="Trigger" value="$Trigger" />

<% if $RoleName == Web { %>
<add key="WebKey" value="$WebKey" />
<% } %>

</appSettings>

However I would need to introduce another level of complexity that WebKey would be different between Testing and Production.

Text Templates also come with the disadvantage that they don't support versioning, and I would need to specify all the Variables between Test and Production environments within each environment.

Typing this up I think has actually helped solidify some thoughts:-

Overall I think Configuration Files is a better fit, and whilst some improvements would be good I like the Key/Value pairs.
Versioning probably doesn't matter too much as any new settings won't be understood by the old software versions if a downgrade was required.
Equally deploying the WebKey to all server roles also probably doesn't actually cause an impact.
Hopefully you can see a use case in being able to select a template within a configuration file; I actually have two different potential use cases

Moving the configuration file to the global level and the template within the configuration file allows different application configurations but the configuration file is shared.
The original intent of this request was to allow a specific template within the configuration file to be deployed, thus I could deploy a Specific Instance and Specific Template of the configuration file to specific environments and roles giving a relationship in the configuration file where Instance = Environment and Template = Role.

I feel like I'm potentially heading to a precarious configuration with Variables assigned in several different places, global, environment, roles, and application?

Thanks,
Paul

apxltd

Thanks for the very detailed write-up @paul-reeves_6112!

Just some background -- back in the v5 days, we introduced Text Templates and intended it to just totally replace the Configuration Files feature. It seemed simpler/better at the time... but I was wrong, that was a mistake... and well, now we have two features that kind of overlap, and are quite confusing at times.

Aside from simplicity (in many use-cases), the main benefit to Configuration Files is restricting view/editing of different instances. There are other benefits as well, but that's what most users really like about them -- especially since it's a lot simpler than using secret storage for values.

Attaching versions to releases is also nice, as it provides better visibility in changes. The main benefit in this all is that developers can "see everything except the actual production values" (like they can see prod was changed, just not what was changed), and then help operations (who can see everything) debug/diagnose/etc.

As @stevedennis mentioned, this is definitely something I want to improve throughout v7, from content and enhancements. So i'm very open to your (seemingly) advanced use case :)

[1] Selecting Templates on Deploy

This is should be pretty easy to do, but let's just "pin" that to make sure I can makes sure it would be the right solution. I wonder if even using a Text Template or just a file on disk do the transform? That way, the template could be stored in source control, and the values in BuildMaster.

[2] $InstanceId per Server

The $InstanceId approach most definitely should work, and actually -- exactly how we envisioned the cascading variables to work, when integrated into configuration files :)

But but then I saw this: it seems I cannot manually deploy a configuration file however

This sounds like a bug to me... when you manually deploy a configuration file, you have to pick a Server... and that Server should be in context when doing variable resolution. I haven't tested it myself, but that might be what's missing here? That could be a regression, or an untested corner case, or something.

[3] Conditional $WebKey per environment

I should probably know this offhand... but are you saying the OtterScript blocks don't work inside of Configuration Files? In theory you should be able to do it, but that might not have been something we brought over. It should be easy to implement, and we'd want to make it opt-in.

But if that's the only difference between the two Templates (i.e. a conditional $WebKey), then the OtterScript block you cameup seems like a good idea. There's also $ListIndexOf(@ServersInRole(Web), $ServerName) >= 0 to test the server's roles, though a simpler function would be nicer now that I think about it...

[4] Variables assigned in several different places, global, environment, roles, and application

Generally speaking, "convention-driven" tends to help minimize configuration (like using role names, application names, etc., when possible), but otherwise... the variables are a good fit for this. All told, this is pretty common, and it can be a lot easier to maintain as well.

You can see all variables in all scopes on the ADmin > variables page, and it helps make it pretty clear how things are configured and to sort of audit.

Conclusion

Overall I think Configuration Files is a better fit, and whilst some improvements would be good I like the Key/Value pairs.

Agreed, they seem like a better fit for your use case - especially since I assume the $Database is a secret.

Versioning probably doesn't matter too much as any new settings won't be understood by the old software versions if a downgrade was required.

Agreed; the main benefit on top of using different/old versions is "seeing which version was deployed with which relesae" in a very easy way. But keep in mind, "rollbacks" are where old versions might be nice to have tied to the release.

Equally deploying the WebKey to all server roles also probably doesn't actually cause an impact.

Also agreed; but... philosophically we don't want users to have to change their release processes just to use BuildMaster, so I want to make sure at least we support this case ;)

Hopefully you can see a use case in being able to select a template within a configuration file; I actually have two different potential use cases

For sure -- and even more than that, transforming with Text Templates or files.

All told -- thanks for the dialog. I'd love to hear your feedback on these other ideas., and we'll try fix some of these edge case bugs ASAP .

Alex