Posts made by apxltd

Hi @philippe-camelio_3885 ,

We'll investigate/fix this via OT-507 in an upcoming Otter release. I suspect it's related to the SDK changes that came out of BuildMaster 2023; we do not test Git with Otter, so it's not surprising it doesn't work.

I really like Otter, but with each major version too many functions change or broken, it's frustrating

Yeah same here; unfortunately we're struggling with "product market fit" so the product is going through changes and

The first versions (v1/v2) were designed as an improvement on the "desired state" concepts from Puppet, Chef, DCS, etc. But the whole "Infrastructure as Code" market never really "took" on Windows. And obviously "no one" uses IaC on Linux anymore thanks to Docker.

In the next versions (v3), we repositioned Otter to be "PowerShell job/script runner" and "compliance as code". This is how most customers used the product, so it seemed like there was a market there. That broke a lot of DSC stuff that hardly anyone used (sadly!)

Very open to ideas on where to take Otter; I think security/compliance monitoring is maybe the right direction, but if that's the case, we need to figure out how to get a lot more pre-built code/scripts in Otter.

To make this work smoothly, a webhook for SCA events would really be immensely helpful. Is something like that already on the 2024 SCA roadmap?

We do have a webhook notifier for "non-compliant packages found in build" planned, so perhaps this would be on the list!

When a SBOM scan is uploaded, no issues are created initially even though the UI suggests that analysis was done already. One has to run analysis a second time with the issue checkbox set for issues to be populated.

I just published some preview documentation, but the concept/model is slightly changed here:

When builds in certain stages are analyzed, an "Issue" for each noncompliant or inconclusive package will be created. These are intended to allow a human review and override noncompliant packages.

Basically, the idea is that nearly every build will be created through a CI process and ignored until it needs to be later tested. And that happens later in the release pipeline, after the build is promoted to a testing stage.

Our new guidance will be run pgutil builds create (basically new name for pgscan inspect) at build time, eactly like it's done now. And the later, when you deploy to a testing environment or otherwise are ready for testing, run pgutil builds promote. At that point, the issues are created.

We were thinking to have "Unresolved Issues" present on the project overview page, and it'd be really messy if it's mostly just CI builds.

Hope taht helps explain the thought process.

@jw thanks for additional insight!

Unfortunately we simply won't have the opportunity to explore this until well past ProGet 2024, and only after we've gotten sufficient feedback from other early adopters on other gaps. I think there are other important things we need to consider as well, and handlign this is so much more complicated to handle this than it may seem, especially at scale and with how our ProGet is configured in the field.

There are also other mechanisms like policy exceptions built-in that could easily handle System.* and runtime.* packages, as I suspect the only thing you would worry about those are vulnerabilities.

As an alternative, I would if you could just write a tool/script to:

1. query for inconclusive builds
2. download inconclusive/missing package builds through a feed
3. trigger a reanalysis of the build

That's not optimal, but that is one thousand times easier than getting something liket his working in PRoGet.

Hi @jw ,

Although the released version will be able check for vulnerabilities without needing the package metadata, reading server properties (deprecation/unlisting), checking if it's latest patch version, doing license detection, etc. require having the package metadata.

However, the package metadata should already be in ProGet by the time you upload the sbom. When doing package restores from ProGet, the packages will be cached automatically. If that's not happening for you, make sure to clear your nuget package caches.

Ultimately we designed the SCA feature is designed to be used in conjunction with ProGet as a proxy to the public repositories. It's not a "stand-alone" tool, so it won't work well if packages aren't in ProGet.

The reason is, if the package metadata isn't in ProGet, it has to be searched for on a remote server. In your sample (one build, two packages), you're right.. it's just a few seconds to search that data on nuget.org. But in production, users have 1000's of active builds each with 1000's of packages... and that *currently * takes about an hour to run an analysis.

Adding 100k's of network requests to connectors to constantly query nuget.org/npmjs.org for server metadata would add hours to that time, triggers api rate limits, and causes lots of performance headaches. Plus, this "leaks" a lot of data about package usage, which is an added security concern. This is a major issue with tools like DependencyTrack - they're basically impossible to scale like ProGet.

Thanks,
Alex

Hi @jw ,

First, the reason you're getting "Package not in feed" (which would also happen in the ProGet 2023 feature as an Issue) is because that Sqlite package has not been cached or pulled to ProGet. However, if you just click Download (and thus cache) the package, then it would be in the feed, and this would go away.

When you browse a remote package in the UI, ProGet is querying nuget.org and displaying whatever their API shows. This query/data is not cached or retained otherwise - which is why it's missing when doing an analysis.

In ProGet 2024, "missing packages" wont be issues per se. Instead, an analysis will be "Inconclusive" -- and this means that there's not enough information to complete the analysis. If your policies don't check license rules (or there's an exception for license checking of Microsoft.* packages), then we wouldn't need the local package to analyze it - and this would be considered compliant.

However, this functionality doesn't work yet. That's just how it will work.

Alex

Hi @andy222 ,

The Git issue should be resolved; that was related to some authentication issues with newer versions of GitHub Enterprise. It broke a lot of tools across the board, apparently. Anyway, I see it just got fixed today. Check out BuildMaster 2023.10-rc.5 if you can

How about pushing a NuGet package instead? That's a much more common scenario (folks migrating from Octopus), and it's one we're going to add some good first-class support for in BuildMaster 2024.

Thanks,
Alex

Hi @andy222,

Very sorry for the frustrations here - it's frustrating for all of us too (me especially) when it doesn't work

Just to give some context here --- we made a huge investment in BuildMaster 2022/23, and one of the areas was a major improvement for how we integrate with ProGet. The feed::FeedName and directory::AssetDirName convention is brand new, and it's an improvement/simplification on the "Secure Resource" convention that you discovered. However, both conventions -- as well as directly specifying those values on the operation -- are still supported. There's a lot (too much?) flexibility.

Unfortunately this particular scenario / use case (downloading assets from ProGet) was simply not one that we focused on:

• Uploading assets is uncommon, but still happen
• Downloading assets is rare... and not something I've seen done in BuildMaster much

NuGet packages are not uncommon when coming from TeamCity/Octopus model. Regardless, we didn't focus on this use case. We felt most new users are seeking Git, .NET, Maven, Build, CI Import, and Docker, so that's where most of our effort was focused..

That said.... I would love to make your scenario / use case something really easy to set-up, so hopefully that will give you the confidence to continue!

In this case, it's a "trivial problem" where something just didn't get "wired" up the right way. I can clearly see from the code that the ApiUrl parameter is not being wired-up as it should be.

We'll get this fixed ASAP, just an extension fix. We just need to set up the scenario and make sure it works on our end first.

Cheers,
Alex

@emer-connelly_2117 very cool, thanks for sharing!!

Hi @sebastian-lieschke_5424 ,

Thanks for the request; this is something that is close to requesting a new feed type, so I'll use that rubric to decide.

As we wrote in that link, new feeds can be very time-consuming to research, develop, document, maintain, etc. Like with all software, even estimating the cost is costly - so we can't really even begin the initial research until there's sufficient demand or market opportunity to justify the possible investment.

To be honest, I don't see there being much demand or any market opportunity for this. Time will tell, and maybe someone will comment on this in the future. But for now this seems really niche.

That being said - I took a quick look at the document you linked, and I don't see API docs (i.e. those missing endpoints you mentioned). Maybe it's something as simple as a basic JSON document. Maybe it's an absurdly complex and undocumented API.

However, if you can figure how the API works, and it turns out to be something like a simple JSON/XML index file.... and you can prototype/fake that using a static file inside of a ProGet Asset Directory... then we can likely implement that quite easily.

I know that's how RPM and Helm Chart feeds got started long ago :)

Alex

Hi @sebastian ,

This will all get a pretty big overhaul in ProGet 2024. I'll share the details in the coming weeks, but here is a sneak peak:

This is what it would look like when viewing the MyFeed licensing rules:

The "Scope" refers to the name of a policy, and you can create shared policies, so this would mean shared sets of licensing rules. You can also bulk-edit license rules on a policy:

I think the new features will change your workflows a bit... maybe you'll use "Warn"? Or perhaps maybe you won't block Non-compliant packages? So for now, I'd wait and see :)

Alex

@valeon @miles-waller_2091 @olivier @It-purchasing_9924 @entro_4370

Took a bit, but CRAN (R) feeds have arrived

https://blog.inedo.com/inedo/introducing-cran-feeds-in-proget/

@mrbill @entro_4370 CRAN (R) feeds have arrived

https://blog.inedo.com/inedo/introducing-cran-feeds-in-proget/

@dima-tinte_1260 @rob-leadbeater_2457 @sdohle_3924

Debian (Apt) Connectors are here! Check out this blog article to learn more:
https://blog.inedo.com/inedo/new-debian-feeds/

@shfunke_1795 @jrottmann_6111 @sdohle_3924 @bahues_9728 @appplat_4310

Thanks for insight into this! I'm happy to report that starting in ProGet 2023.22, you can create Alpine (APK) feeds with connectors :)

@paul_6112 said in Do you plan to upgrade JQuery in a future ProGet release?:

This was picked up by nessus on BuildMaster v7

Lol wow - that's ridiculous

As I mentioned before, it's a forked library thus not vulnerable. So I suppose you can continue reporting it as a "false positive" to whoever seems to care, and perhaps we'll also just edit the version number out to appease that the security tool

Thanks @carl-westman_8110 , I appreciate the feedback!!

As someone coming from Azure Artifacts, I'd love to get your impression on our draft ProGet vs Azure Artifacts page - we're slowly starting to try to articulate the high-level differences and benefits to ProGet. But I swear marketing copy about the software is harder to write than the software itself

We also have BuildMaster vs. Azure DevOps comparison page too, though it's quite a bit more involved.

Hi @carl-westman_8110 ,

Thanks for the feature request! I'm afraid this one's a bit too niche to implement as described, and this use case isn't something we'd want to support for Universal Packages.

However, Asset Directories are a good fit for this, and one of the use cases is a Static CDN. So that means you could use it for web assets like docs if you wanted.

You'll still need to publish the docs like you would to the other webserver. And of course you could hyperlink to the document root from the universal package description as well.

Cheers,

Alex

Thanks for the feedback @philippe-camelio_3885 !

The Applications page is "ancient", and was originally designed to show "what build is in what server/environment". That was super-useful at the time, and I suppose still is, depending on the use case.

But with multiple pipelines per application (like you have now) this view isn't so useful. I'm definitely open to redesigning / rethinking some of these dashboard/aggregate pages.

This is something we can think about for v2024 (since v2023 is just a couple weeks away ). I've put a note onto our roadmap planning board, and may jump back or email you directly for some feedbakc/insight

@shayde @sebastian really appreciate the help, we'll get this incorporated ASAP !!

Hi @jchitel_9895 ,

Thanks for the additional info! We "moved" your new topic back to this one, since we link these on this page in the docs and want to keep everything in one place: https://docs.inedo.com/docs/proget-feeds-other-types

Keep in mind that feed types are a significant initial and ongoing investment (it's a product in a product), and at first glance, Homebrew doesn't seem to make any commercial sense.

First and foremost, there doesn't seem to be a market here. Homebrew itself isn't commercialized. They tried a Kickstarted from 2013, but it seems to remain a hobby-type project. Compare that to Chocolatey (which may be a little older, and also did a kickstarter I think) -- they now have a decent size fulltime staff now.

But secondly (and on a technical level), there isn't a "Homebrew Repository" or "Homebrew Server" - as you mentioned, it's Git-based - which means all it's doing is cloning Git repositories, and probably using tags and specific repo layouts to determine packages.

Cheers,
Alex

Thanks for clarifying @philippe-camelio_3885. I see the issue in the code now. I think it's been this way for quite a while

On the View Page, there seems to be some special handling for credentials;

return "ssh-rsa " + Convert.ToBase64String(cred.PublicKey);

I'm just going to delete the "ssh-rsa " bit, since apparently that can be incorrect. That's an SDK change actually, so it'll take a bit to be reflected in the products.

On the View Secret Fields page, we're just coercing the value to a string:

return InedoLib.UTF8Encoding.GetString(bytes);

That doesn't seem right either, but I'll just leave that as is. I know we redid that page in BuildMaster and eventually will bring to Otter.

There probably should really be a special page altogether for this type of credential, instead of using the generic "Edit credentials" page. Not a big priority but perhaps some day :)

@hwittenborn awesome! I'm not sure if you saw it yet, but we have a new API called the Common Package API; I hope to fold in Promotion, Repackaging, and Deployment under this so it can be consistent

FYI; we replied to a critical ticket that duplicated this, but in case anyone is reading this...

We have a SQL patch available, but aren't totally sure if it will help. We can send it.

We are really struggling to reproduce this in any consistent matter; we believe the issue might be related to sql server's plan caching. we're seeing inappropriate execution plans being used , and then suddenly changing, for the same input.

The patch simplifies the queries we're using, so hopefully it will result in a more consistent query plan. Our next idea , is to try forcing some join hints.

We're also asking for a backup of the database? That will help us analyze your data. We're thinking maybe you have some unique distributions (like tons of versions for one package, just a few for another).

Curious ... why not just enable non-interactive login on the servers? Seems like both ought to be enabled in most cases, right?

Hi @sebastian ,

Thanks for the feedback; that's my understanding of the scores. The main value is time savings in manual assessment. I wish it was easier to get in, and it's too bad our datasets don't have it. We'll need to review this in the coming months, since it involves a lot more work on our end to refactor things.

As for Visual Studio, one of our engineers here just prototyped that, and I think we'll do it. That's much simpler and easy to sneak in a maintenance release :)

In the past, it wasn't possible in Visual Studio before (they used a different API that was nuget.org-only), and it wasn't feasible in ProGet due to the indexing. But now it's much easier.

I think this will be one of those "freemium preview" features. Not that it impacts you, but we're thinking that free users will get a link to a page in ProGet that's like "vulnerabilities are paid feature, here are the benefits, etc. To see which vulns the linked package has, navigate to feed, click here, etc. Paid users can assess, block, etc."

Paid users would just link to the package or vulnerability or something. Just our idea figured I'd share ;)

Cheers,
Alex

Hi @sebastian,

As @gdivis mentioned, it will be a bit of an undertaking to fill-in the missing data (in particular the ID/Scores) using the NVD datasets. We are also exploring aggregating some additional datasets as well, in particular ones for system packages (Debian, RPM), so we can incorporate container image scanning directly in ProGet.

It seems our userbase (and the community at large) is slowly starting to "grok" vulnerabilities, but it's quite a ways off.

We've got some other work ahead of this, so I'm going to say perhaps late June / early July is when we can consider picking up on this again. Looking forward to any feedback you have in the mean time :)

Thanks,
Alex

I'm curious to know too if anyone else is interested in this.

FYI: we were intentionally "lazy" when we came across these, because the expressions got pretty complex (in the specs) and it seemed more suitable for a human to determine what WITH and this and that meant.

ProGet already supports multiple licenses per packages (they are treated as an OR), but we thought it might be unintuitive to only support OR, so we just left it as is.

Thanks for the additional thoughts @sebastian!

I agree... it's not totally infeasible from a technical standpoint, but it's still pretty tricky. Just to comment on a technical thing, FYI...

Now, to be able to serve the package to the user, Proget has to download it first, right?

Actually, ProGet "streams" content from connectors. This means that, when a user requests a package from ProGet (and that package is on a remote connector), ProGet will then request a package from the connector. As the file is being downloaded, ProGet will send the same data back to the user and optionally write that data to disk. If we didn't do this, ProGet would be basically unusually slow.

A ZIP archive (what package files use) use a tail index, which means you have to read it backwards from End of File. So it's not possible to read an embedded file unless we've downloaded the entire package.

There are a few other "gotchas" we'd need to consider, even for cached/local packages. For example, we can't open/seek the package file just to know the license and if the package should be blocked - especially when it comes to cloud storage (for the same reason - tail indexing). So, we would obviously need to store package license file info in the database too... but then we'd need a way to deal with existing packages on disk that don't yet have that info.

We may also want to add some sort of heuristic analysis of license text, even if it's simple as a basic distance check. Personally I think that's a bad idea to rely on... but other products do, and the reality is most users would just skim a license anyway.

This all becomes a lot easier after v2023 with centralized data and a package analyzer that can background scan all these, but still not trivial. And then there's the real hard part... the UI and documentation

We definitely don't want to hack something in like packageid:// and package::// -- those have been a total pain and plus, I hate the design

Anyway -- just wanted to give more technical insight into why ProGet behaves like this, and why I'm hesitant to jump on the "reading license file" approach without adding somethign that's a lot more valuable than what we have now.

Hi @sebastian

Thanks for the idea -- yes, I think it's an interesting approach!

We explored it a while ago, and this was where we ended up...

1. It's even more confusing to use than packageid://, so we'd need to find a better UI solution

2. We'd want to store the full license text as well, so it'd be easy to confirm the contents

3. This is all a nontrivial engineering effort

4. We're not sure how many packages this would impact and how much value / time savings it would represent

5. None of this would even work for remote packages, which is by far what most users find confusing and have issues with

6. It would probably require less engineering effort to scan/query all packages on NuGet and make a "database" of package licenses using a little human intelligence

7. It would require even less effort to just ask package authors to specify license codes, and then eventually the problem will go away on its own probably

And then we gave up because there were more priority things to address ;)

Cheers,
Alex

@sebastian said in Package license definition:

That being said, having a new entity "package owner" or "publisher" or something like that and being able to filter for that entity could be a cool new feature. This could also be used in the SBOM reporting feature (like: 40 packages come from Microsoft, 20 from vendor A, 7 from vendor B).

We considered doing "something" with this metadata ages ago, but found that a lot of packages (including npm, etc.) have multiple owners/authors. On the top page of nuget.org, just 3/20 seem to have a single author

In general, the human-driven "package approval workflow" seems to be the best bet. Maybe p[ainful at first, but "not too bad" in the long-run

@sebastian @pmsensi

Thanks for added insights! This doesn't seem as simple as I had hoped...

The current solution in ProGet now (i.e. packageid:// and package:// urls that can be associated with license codes) feels hacky, but seems we don't have many options.

Well, one thing that might work.... submitting a pull request to those packages to add the MIT license code to their project file? It's probably just an oversight on their part...

@sebastian said in Package license definition:

What's driving me nuts at the moment is that Microsoft seems to be using embedded license info files for quite a number of their packages. Assigning licenses to those is going to take a while...

Oh really? They even have their own accepted SPDX code... whyyy Microsoft

In the past, we thought of adding a kind of wildcard URL for licenses, like a "package://Microsoft.*" => "MSPL" would basically associate all packages with that prefix that don't otherwise have a SPDX code, or an explicit license.

Wonder if that would help here?

Hey @mrbill and all,

We've since had a handful of "mentions" for it on the presales side, but it was more like a "wishlist" than anything else. There's probably some opportunity however - so I'm leaning towards re-evaluating for v2023.

Big thing will be partners who can show us how to use R -- I added this to our internal list for the v2023 review :)

Cheers,

Alex

@lm excellent, thank you for the help!

OK - we'll wire these up via PG-2223, which may come in the next or following maintenance release. In theory it's trivial and low risk.

Legacy / HTTP-only Format (still works):
<WebServer Enabled="true/false" Urls="web server listen URLs" />

New / HTTPS-inclused Format:

<WebServer Enabled="true/false">
  <Endpoint 		  
      Url="{http|https}://host.name:1000/">
      CertFile=".pfx, .pem, .crt"
      KeyFile=".key"
      Password=""
      Subject=""
      Store=""
      Location=""
      AllowInvalid=""
     />
  </Endpoint>
</WebServer>

This lets us document an "Endpoint" using simple table of key/value pairs, as well as simplify error messages and the examples that people will just copy/paste:

• Plain HTTP
• PFX-file based
• .crt+pem file based
• Windows cert store integration, only works on Windows.

Oh --- and good point about self-signed; if anything we should use that ACME library to get a LetsEncrypt cert or something. At least in theory. A few other products seem to be doing that now as well. No idea how well that would work - but for evaluation purposes I think it's pretty important. Especially if you don't know how to configure HTTPS otherwise.

Hi @lm ,

Thanks for the feedback - and you are correct; it is ASP.NET 6.

Starting with the configuration file is a great idea. The Inedo Hub provides a UI to that anyways.

The documentation for our configuration file is here:
https://docs.inedo.com/docs/installation-configuration-files

Obviously it's pretty easy to just wire up values (and we'd be happy to do that), but my only hesitation is documentation and cross-platform compatibility.

Brainstorming a bit but... from the UI, we will just support one URL I think. I don't know why you'd want multiple...

The docs you linked from Microsoft are confusing, and it's not really clear what fields to use or why. I guess, in general, you'd select a file (which InedoHub would generate?) or the cert store (that's Windows only, right?).

And I guess what, when you generate a cert yourself, it's a pfx or something? I always forget the differences....

How would you suggest we modify that element to support the different options they have?

  <WebServer Enabled="true/false" Urls="web server listen URLs" />

What would you want to put in, or suppose other (advanced) users would want to put in?

Cheers,
Alex

Hi @tkolb_7784

I wanted to provide an official answer to this:

Are there plans for (easy) HTTPS support in the internal webserver? Applying a self hosted certificate and deploying it to all build servers and developer machines seems over the top for me right now.

Yes. Not exactly sure how yet, but we would like the integrated webserver to support this as easily as possible.

Thanks for the updates/info!

Hopefully some other folks will find this and express some interest -- I'll update if I hear anything from other users, or there's any marketing interest.

By the way, this is different than Android Package Manager APKs. Someone asked about that once because they were curious if ProGet could be like a private app store for mobile apps.

Hi @shfunke_1795 , @jrottmann_6111

Thanks, I just added this to our documentation page!

I didn't look too deeply, but I found some initial documentation:
https://wiki.alpinelinux.org/wiki/Package_management

It seems this is "like Yum/RPM but for Alpine Linux"? None of us here use Alpine Linux, so there's a pretty big learning curve to get started. Any help here would be appreciated, and definitely move this along :)

Is this related to "APK" that Android uses?

Is the "API" mostly like basic file downloads, based on an index file? Are you able to "hack" or do a PoC using ProGet Asset Directories?

@jblaine_9526 Thanks for clarifying!

I think SAML integration is one of those "write-once" things, where we completely forget about how it works every time

I'm not so familiar with SAML behind the scenes... do you know how "SAML group claims" work? For example...

• Is it something that comes back in the XML response, or does it require a separate request?
• What do the "group claims" look like? Like a list of human-readable group names?

And them most importantly... what should ProGet do with such claims upon receipt? Treat the user as if they're in the group (kind of like LDAP groups), and allow permissions to be assigned against that group (like LDAp, but without searching)?

Hi @jeff-miles_5073 ,

Thanks for the first inquiry; I just updated the documentation! These must be "relatively new", and I know we've had a few customers using universal packages for this.

We have some Terraform integrations on our roadmap for 2023, so this definitely something we'll look into on our own as well.

Cheers,
Alex

We've been getting all sorts of outage/clear notices from our main datacenter since yesterday, so something's definitely up. Maybe a switch or something.

I flipped to our DR instance of ProGet until we get it figured out :)

Conda feeds are now available!

https://docs.inedo.com/docs/proget-feeds-python-conda

cc/ @stewart-ridgway_3626 @ben-hegarty_7893

I'm afraid it didn't make the roadmap for Q4. There's no documented API that we saw, and reverse engineering and then supporting the inevitable quirks takes a lot of effort -- and there still doesn't seem to be a market opportunity for us here I'm afraid (no search volume, etc.)...

This SO Question seems to indicate that using Git repositories is a popular way to reference packages...

https://stackoverflow.com/questions/54143695/how-to-use-my-dart-packages-private-and-not-show-on-pub-dart-lang#54143758

As I mentioned, our usage of this library is minimal, and we do not use it in a manner that would impact product security (i.e. "passing HTML from untrusted sources - even after sanitizing it - to one of jQuery's DOM manipulation methods (i.e. .html(), .append(), and others). So please consider this a "false positive".

The only benefit to our customers/users is to appease security teams that don't really understand how security or vulnerabilities work

However, that's actually a benefit in and of itself -- can you let me know what security scanner you use? What security processes are being used such that a security team is even looking at third-party vendor tools like ours? How do they learn or understand "false positives", etc? Understanding that would help us.

What I don't want to do is play the game of "introduce new bugs/glitches by constantly upgrading javascript libraries just to appease clueless security teams" -- so learning how they scan would really help me decide.

Thanks!

hey @harald-somnes-hanssen_2204,

I'd like to add something like this to ProGet v6. How has progress been w/ your end users? Do they ask about this?

Anyway, here's my rough idea...

I don't know if I like those words, "Feed Usage Instructions", but we have "Usage Instructions" on the Package overview page, so there's that.

Clicking on that would provide instructions similar to npm's, or the Visual Studio by default.

HOWEVER, I'm really thinking that you should be able to customize that page! Maybe you have specific technical instructions you want to add, etc.

LMK your thoughts, curious to know what you think, especially after it's been a while.

@scroak_6473 ooh I forgot to mention, but this feature was released a little while ago. It's a preview feature, and you can enable it by going to [User Icon] > Personal API Keys > Enable.

LMK your thoughts!

Thanks for the very detailed write-up @paul-reeves_6112!

Just some background -- back in the v5 days, we introduced Text Templates and intended it to just totally replace the Configuration Files feature. It seemed simpler/better at the time... but I was wrong, that was a mistake... and well, now we have two features that kind of overlap, and are quite confusing at times.

Aside from simplicity (in many use-cases), the main benefit to Configuration Files is restricting view/editing of different instances. There are other benefits as well, but that's what most users really like about them -- especially since it's a lot simpler than using secret storage for values.

Attaching versions to releases is also nice, as it provides better visibility in changes. The main benefit in this all is that developers can "see everything except the actual production values" (like they can see prod was changed, just not what was changed), and then help operations (who can see everything) debug/diagnose/etc.

As @stevedennis mentioned, this is definitely something I want to improve throughout v7, from content and enhancements. So i'm very open to your (seemingly) advanced use case :)

[1] Selecting Templates on Deploy

This is should be pretty easy to do, but let's just "pin" that to make sure I can makes sure it would be the right solution. I wonder if even using a Text Template or just a file on disk do the transform? That way, the template could be stored in source control, and the values in BuildMaster.

[2] $InstanceId per Server

The $InstanceId approach most definitely should work, and actually -- exactly how we envisioned the cascading variables to work, when integrated into configuration files :)

But but then I saw this: it seems I cannot manually deploy a configuration file however

This sounds like a bug to me... when you manually deploy a configuration file, you have to pick a Server... and that Server should be in context when doing variable resolution. I haven't tested it myself, but that might be what's missing here? That could be a regression, or an untested corner case, or something.

[3] Conditional $WebKey per environment

I should probably know this offhand... but are you saying the OtterScript blocks don't work inside of Configuration Files? In theory you should be able to do it, but that might not have been something we brought over. It should be easy to implement, and we'd want to make it opt-in.

But if that's the only difference between the two Templates (i.e. a conditional $WebKey), then the OtterScript block you cameup seems like a good idea. There's also $ListIndexOf(@ServersInRole(Web), $ServerName) >= 0 to test the server's roles, though a simpler function would be nicer now that I think about it...

[4] Variables assigned in several different places, global, environment, roles, and application

Generally speaking, "convention-driven" tends to help minimize configuration (like using role names, application names, etc., when possible), but otherwise... the variables are a good fit for this. All told, this is pretty common, and it can be a lot easier to maintain as well.

You can see all variables in all scopes on the ADmin > variables page, and it helps make it pretty clear how things are configured and to sort of audit.

Conclusion

Overall I think Configuration Files is a better fit, and whilst some improvements would be good I like the Key/Value pairs.

Agreed, they seem like a better fit for your use case - especially since I assume the $Database is a secret.

Versioning probably doesn't matter too much as any new settings won't be understood by the old software versions if a downgrade was required.

Agreed; the main benefit on top of using different/old versions is "seeing which version was deployed with which relesae" in a very easy way. But keep in mind, "rollbacks" are where old versions might be nice to have tied to the release.

Equally deploying the WebKey to all server roles also probably doesn't actually cause an impact.

Also agreed; but... philosophically we don't want users to have to change their release processes just to use BuildMaster, so I want to make sure at least we support this case ;)

Hopefully you can see a use case in being able to select a template within a configuration file; I actually have two different potential use cases

For sure -- and even more than that, transforming with Text Templates or files.

All told -- thanks for the dialog. I'd love to hear your feedback on these other ideas., and we'll try fix some of these edge case bugs ASAP .

Alex

@sylvain-martel_3976 thanks for the additional information; I'm warning up to this idea, but there's a one main area of concern

Are you sure WinGet can help you achieve "Infrastructure as Code"? I'm becoming more and more skeptical of this use case.

The main reason is that there are very few server applications on Microsoft's public WinGet repository. Even their own products like SQL Server aren't there. It's mostly desktop tools like Adobe Reader, Visual Studio, etc. This tells me Microsoft doesn't see WinGet helping with IaC, either.

This is largely the same concern I had with Chocolatey, in particular organizations trying to use Chocolatey for IaC. Their public repository is mostly desktop tools, and using Chocolatey to manage server-based applications is really difficult.

This has really nothing to do with Chocolatey or WinGet, but more that server-based products tend to require infrastructure configuration (user accounts for services, certificates for websites, IP address bindings, firewall changes, database connection strings) -- where as a tool like Adobe Reader simply does not. And infrastructure changes like this are rarely automated as part of the installer.

I definitely get the use case of WinGet (or Chocolatey) as means to distribute desktop applications, and if that were the case - there's a lot more features we need to consider (like proxying Microsoft's public gallery, etc.).... but at the moment I'm just not seeing it for IaC / Server applications.

Perhaps, try doing a proof of concept on your end? I guess you just need to set up a Git Repository, and then, there's your basic WinGet repo?

Ultimately using WinGet in an IaC scenario sounds like a bit more work (and more moving pieces) than just running a PowerShell script to download and run a handful of installers... but I don't know.

I'd love to see what you put together, or more specifics as you learn them yourself.

Thanks for the detailed explanation @sylvain-martel_3976 - this is quite helpful for me! A few more questions if you don't mind :)

All. First-party and 3rd party. 90% 3rd party though.

Can you give an estimate of the number of Winget "packages" you expect to have in your feed? Like 10? 100? 1000?

What would the process of adding a third-party application to this feed? Will you be recreating the WinGet metadata (package?) from Microsoft's official feed?

Are most of these third-party applications in Microsoft's feed, or are they like from vendors or whatnot?

I'm trying to get a feeling for the concept of "connectors" in WinGet, and if it even makes sense.

Well, we use a file server to host the packages.

Is this just like an ordinary file server, where you use UNC paths? We are have plans to do some major improvements on ProGet's Asset Directory, including enabling replication, cloud storage, and (hopefully) versioning.

So to summarize your usecase...

• WinGet is a means to manage (mostly) third-party applications that you keep on a private file server, and wish to install on other servers through automation
• WinGet is preferred to a regular file server because it includes some metadata like requirements on what to install
• WinGet is preferred to Chocolatey for security and cost reasons

What I'd like to do is build a usecase for how organizations can use WinGet in combination with ProGet (in particular the asset directories) to add value.

Whoops! We often test ProGet using our own ProGet instance, and it looks like someone (ahem.. no need to mention names) disabled Semantic Versioning for Containers on our instance for testing purposes.

It's now back on, which means you should be able to use latest again, or even 5 or 5.2 if you wanted.