Posts made by apxltd

Thanks for the additional thoughts @sebastian!

I agree... it's not totally infeasible from a technical standpoint, but it's still pretty tricky. Just to comment on a technical thing, FYI...

Now, to be able to serve the package to the user, Proget has to download it first, right?

Actually, ProGet "streams" content from connectors. This means that, when a user requests a package from ProGet (and that package is on a remote connector), ProGet will then request a package from the connector. As the file is being downloaded, ProGet will send the same data back to the user and optionally write that data to disk. If we didn't do this, ProGet would be basically unusually slow.

A ZIP archive (what package files use) use a tail index, which means you have to read it backwards from End of File. So it's not possible to read an embedded file unless we've downloaded the entire package.

There are a few other "gotchas" we'd need to consider, even for cached/local packages. For example, we can't open/seek the package file just to know the license and if the package should be blocked - especially when it comes to cloud storage (for the same reason - tail indexing). So, we would obviously need to store package license file info in the database too... but then we'd need a way to deal with existing packages on disk that don't yet have that info.

We may also want to add some sort of heuristic analysis of license text, even if it's simple as a basic distance check. Personally I think that's a bad idea to rely on... but other products do, and the reality is most users would just skim a license anyway.

This all becomes a lot easier after v2023 with centralized data and a package analyzer that can background scan all these, but still not trivial. And then there's the real hard part... the UI and documentation

We definitely don't want to hack something in like packageid:// and package::// -- those have been a total pain and plus, I hate the design

Anyway -- just wanted to give more technical insight into why ProGet behaves like this, and why I'm hesitant to jump on the "reading license file" approach without adding somethign that's a lot more valuable than what we have now.

Hi @sebastian

Thanks for the idea -- yes, I think it's an interesting approach!

We explored it a while ago, and this was where we ended up...

1. It's even more confusing to use than packageid://, so we'd need to find a better UI solution

2. We'd want to store the full license text as well, so it'd be easy to confirm the contents

3. This is all a nontrivial engineering effort

4. We're not sure how many packages this would impact and how much value / time savings it would represent

5. None of this would even work for remote packages, which is by far what most users find confusing and have issues with

6. It would probably require less engineering effort to scan/query all packages on NuGet and make a "database" of package licenses using a little human intelligence

7. It would require even less effort to just ask package authors to specify license codes, and then eventually the problem will go away on its own probably

And then we gave up because there were more priority things to address ;)

Cheers,
Alex

@sebastian said in Package license definition:

That being said, having a new entity "package owner" or "publisher" or something like that and being able to filter for that entity could be a cool new feature. This could also be used in the SBOM reporting feature (like: 40 packages come from Microsoft, 20 from vendor A, 7 from vendor B).

We considered doing "something" with this metadata ages ago, but found that a lot of packages (including npm, etc.) have multiple owners/authors. On the top page of nuget.org, just 3/20 seem to have a single author

In general, the human-driven "package approval workflow" seems to be the best bet. Maybe p[ainful at first, but "not too bad" in the long-run

@sebastian @pmsensi

Thanks for added insights! This doesn't seem as simple as I had hoped...

The current solution in ProGet now (i.e. packageid:// and package:// urls that can be associated with license codes) feels hacky, but seems we don't have many options.

Well, one thing that might work.... submitting a pull request to those packages to add the MIT license code to their project file? It's probably just an oversight on their part...

@sebastian said in Package license definition:

What's driving me nuts at the moment is that Microsoft seems to be using embedded license info files for quite a number of their packages. Assigning licenses to those is going to take a while...

Oh really? They even have their own accepted SPDX code... whyyy Microsoft

In the past, we thought of adding a kind of wildcard URL for licenses, like a "package://Microsoft.*" => "MSPL" would basically associate all packages with that prefix that don't otherwise have a SPDX code, or an explicit license.

Wonder if that would help here?

Hey @mrbill and all,

We've since had a handful of "mentions" for it on the presales side, but it was more like a "wishlist" than anything else. There's probably some opportunity however - so I'm leaning towards re-evaluating for v2023.

Big thing will be partners who can show us how to use R -- I added this to our internal list for the v2023 review :)

Cheers,

Alex

@lm excellent, thank you for the help!

OK - we'll wire these up via PG-2223, which may come in the next or following maintenance release. In theory it's trivial and low risk.

Legacy / HTTP-only Format (still works):
<WebServer Enabled="true/false" Urls="web server listen URLs" />

New / HTTPS-inclused Format:

<WebServer Enabled="true/false">
  <Endpoint 		  
      Url="{http|https}://host.name:1000/">
      CertFile=".pfx, .pem, .crt"
      KeyFile=".key"
      Password=""
      Subject=""
      Store=""
      Location=""
      AllowInvalid=""
     />
  </Endpoint>
</WebServer>

This lets us document an "Endpoint" using simple table of key/value pairs, as well as simplify error messages and the examples that people will just copy/paste:

• Plain HTTP
• PFX-file based
• .crt+pem file based
• Windows cert store integration, only works on Windows.

Oh --- and good point about self-signed; if anything we should use that ACME library to get a LetsEncrypt cert or something. At least in theory. A few other products seem to be doing that now as well. No idea how well that would work - but for evaluation purposes I think it's pretty important. Especially if you don't know how to configure HTTPS otherwise.

Hi @lm ,

Thanks for the feedback - and you are correct; it is ASP.NET 6.

Starting with the configuration file is a great idea. The Inedo Hub provides a UI to that anyways.

The documentation for our configuration file is here:
https://docs.inedo.com/docs/installation-configuration-files

Obviously it's pretty easy to just wire up values (and we'd be happy to do that), but my only hesitation is documentation and cross-platform compatibility.

Brainstorming a bit but... from the UI, we will just support one URL I think. I don't know why you'd want multiple...

The docs you linked from Microsoft are confusing, and it's not really clear what fields to use or why. I guess, in general, you'd select a file (which InedoHub would generate?) or the cert store (that's Windows only, right?).

And I guess what, when you generate a cert yourself, it's a pfx or something? I always forget the differences....

How would you suggest we modify that element to support the different options they have?

  <WebServer Enabled="true/false" Urls="web server listen URLs" />

What would you want to put in, or suppose other (advanced) users would want to put in?

Cheers,
Alex

Hi @tkolb_7784

I wanted to provide an official answer to this:

Are there plans for (easy) HTTPS support in the internal webserver? Applying a self hosted certificate and deploying it to all build servers and developer machines seems over the top for me right now.

Yes. Not exactly sure how yet, but we would like the integrated webserver to support this as easily as possible.

Thanks for the updates/info!

Hopefully some other folks will find this and express some interest -- I'll update if I hear anything from other users, or there's any marketing interest.

By the way, this is different than Android Package Manager APKs. Someone asked about that once because they were curious if ProGet could be like a private app store for mobile apps.

Hi @shfunke_1795 , @jrottmann_6111

Thanks, I just added this to our documentation page!

I didn't look too deeply, but I found some initial documentation:
https://wiki.alpinelinux.org/wiki/Package_management

It seems this is "like Yum/RPM but for Alpine Linux"? None of us here use Alpine Linux, so there's a pretty big learning curve to get started. Any help here would be appreciated, and definitely move this along :)

Is this related to "APK" that Android uses?

Is the "API" mostly like basic file downloads, based on an index file? Are you able to "hack" or do a PoC using ProGet Asset Directories?

@jblaine_9526 Thanks for clarifying!

I think SAML integration is one of those "write-once" things, where we completely forget about how it works every time

I'm not so familiar with SAML behind the scenes... do you know how "SAML group claims" work? For example...

• Is it something that comes back in the XML response, or does it require a separate request?
• What do the "group claims" look like? Like a list of human-readable group names?

And them most importantly... what should ProGet do with such claims upon receipt? Treat the user as if they're in the group (kind of like LDAP groups), and allow permissions to be assigned against that group (like LDAp, but without searching)?

Hi @jeff-miles_5073 ,

Thanks for the first inquiry; I just updated the documentation! These must be "relatively new", and I know we've had a few customers using universal packages for this.

We have some Terraform integrations on our roadmap for 2023, so this definitely something we'll look into on our own as well.

Cheers,
Alex

We've been getting all sorts of outage/clear notices from our main datacenter since yesterday, so something's definitely up. Maybe a switch or something.

I flipped to our DR instance of ProGet until we get it figured out :)

Conda feeds are now available!

https://docs.inedo.com/docs/proget-feeds-python-conda

cc/ @stewart-ridgway_3626 @ben-hegarty_7893

I'm afraid it didn't make the roadmap for Q4. There's no documented API that we saw, and reverse engineering and then supporting the inevitable quirks takes a lot of effort -- and there still doesn't seem to be a market opportunity for us here I'm afraid (no search volume, etc.)...

This SO Question seems to indicate that using Git repositories is a popular way to reference packages...

https://stackoverflow.com/questions/54143695/how-to-use-my-dart-packages-private-and-not-show-on-pub-dart-lang#54143758

As I mentioned, our usage of this library is minimal, and we do not use it in a manner that would impact product security (i.e. "passing HTML from untrusted sources - even after sanitizing it - to one of jQuery's DOM manipulation methods (i.e. .html(), .append(), and others). So please consider this a "false positive".

The only benefit to our customers/users is to appease security teams that don't really understand how security or vulnerabilities work

However, that's actually a benefit in and of itself -- can you let me know what security scanner you use? What security processes are being used such that a security team is even looking at third-party vendor tools like ours? How do they learn or understand "false positives", etc? Understanding that would help us.

What I don't want to do is play the game of "introduce new bugs/glitches by constantly upgrading javascript libraries just to appease clueless security teams" -- so learning how they scan would really help me decide.

Thanks!

hey @harald-somnes-hanssen_2204,

I'd like to add something like this to ProGet v6. How has progress been w/ your end users? Do they ask about this?

Anyway, here's my rough idea...

I don't know if I like those words, "Feed Usage Instructions", but we have "Usage Instructions" on the Package overview page, so there's that.

Clicking on that would provide instructions similar to npm's, or the Visual Studio by default.

HOWEVER, I'm really thinking that you should be able to customize that page! Maybe you have specific technical instructions you want to add, etc.

LMK your thoughts, curious to know what you think, especially after it's been a while.

@scroak_6473 ooh I forgot to mention, but this feature was released a little while ago. It's a preview feature, and you can enable it by going to [User Icon] > Personal API Keys > Enable.

LMK your thoughts!

Thanks for the very detailed write-up @paul-reeves_6112!

Just some background -- back in the v5 days, we introduced Text Templates and intended it to just totally replace the Configuration Files feature. It seemed simpler/better at the time... but I was wrong, that was a mistake... and well, now we have two features that kind of overlap, and are quite confusing at times.

Aside from simplicity (in many use-cases), the main benefit to Configuration Files is restricting view/editing of different instances. There are other benefits as well, but that's what most users really like about them -- especially since it's a lot simpler than using secret storage for values.

Attaching versions to releases is also nice, as it provides better visibility in changes. The main benefit in this all is that developers can "see everything except the actual production values" (like they can see prod was changed, just not what was changed), and then help operations (who can see everything) debug/diagnose/etc.

As @stevedennis mentioned, this is definitely something I want to improve throughout v7, from content and enhancements. So i'm very open to your (seemingly) advanced use case :)

[1] Selecting Templates on Deploy

This is should be pretty easy to do, but let's just "pin" that to make sure I can makes sure it would be the right solution. I wonder if even using a Text Template or just a file on disk do the transform? That way, the template could be stored in source control, and the values in BuildMaster.

[2] $InstanceId per Server

The $InstanceId approach most definitely should work, and actually -- exactly how we envisioned the cascading variables to work, when integrated into configuration files :)

But but then I saw this: it seems I cannot manually deploy a configuration file however

This sounds like a bug to me... when you manually deploy a configuration file, you have to pick a Server... and that Server should be in context when doing variable resolution. I haven't tested it myself, but that might be what's missing here? That could be a regression, or an untested corner case, or something.

[3] Conditional $WebKey per environment

I should probably know this offhand... but are you saying the OtterScript blocks don't work inside of Configuration Files? In theory you should be able to do it, but that might not have been something we brought over. It should be easy to implement, and we'd want to make it opt-in.

But if that's the only difference between the two Templates (i.e. a conditional $WebKey), then the OtterScript block you cameup seems like a good idea. There's also $ListIndexOf(@ServersInRole(Web), $ServerName) >= 0 to test the server's roles, though a simpler function would be nicer now that I think about it...

[4] Variables assigned in several different places, global, environment, roles, and application

Generally speaking, "convention-driven" tends to help minimize configuration (like using role names, application names, etc., when possible), but otherwise... the variables are a good fit for this. All told, this is pretty common, and it can be a lot easier to maintain as well.

You can see all variables in all scopes on the ADmin > variables page, and it helps make it pretty clear how things are configured and to sort of audit.

Conclusion

Overall I think Configuration Files is a better fit, and whilst some improvements would be good I like the Key/Value pairs.

Agreed, they seem like a better fit for your use case - especially since I assume the $Database is a secret.

Versioning probably doesn't matter too much as any new settings won't be understood by the old software versions if a downgrade was required.

Agreed; the main benefit on top of using different/old versions is "seeing which version was deployed with which relesae" in a very easy way. But keep in mind, "rollbacks" are where old versions might be nice to have tied to the release.

Equally deploying the WebKey to all server roles also probably doesn't actually cause an impact.

Also agreed; but... philosophically we don't want users to have to change their release processes just to use BuildMaster, so I want to make sure at least we support this case ;)

Hopefully you can see a use case in being able to select a template within a configuration file; I actually have two different potential use cases

For sure -- and even more than that, transforming with Text Templates or files.

All told -- thanks for the dialog. I'd love to hear your feedback on these other ideas., and we'll try fix some of these edge case bugs ASAP .

Alex

@sylvain-martel_3976 thanks for the additional information; I'm warning up to this idea, but there's a one main area of concern

Are you sure WinGet can help you achieve "Infrastructure as Code"? I'm becoming more and more skeptical of this use case.

The main reason is that there are very few server applications on Microsoft's public WinGet repository. Even their own products like SQL Server aren't there. It's mostly desktop tools like Adobe Reader, Visual Studio, etc. This tells me Microsoft doesn't see WinGet helping with IaC, either.

This is largely the same concern I had with Chocolatey, in particular organizations trying to use Chocolatey for IaC. Their public repository is mostly desktop tools, and using Chocolatey to manage server-based applications is really difficult.

This has really nothing to do with Chocolatey or WinGet, but more that server-based products tend to require infrastructure configuration (user accounts for services, certificates for websites, IP address bindings, firewall changes, database connection strings) -- where as a tool like Adobe Reader simply does not. And infrastructure changes like this are rarely automated as part of the installer.

I definitely get the use case of WinGet (or Chocolatey) as means to distribute desktop applications, and if that were the case - there's a lot more features we need to consider (like proxying Microsoft's public gallery, etc.).... but at the moment I'm just not seeing it for IaC / Server applications.

Perhaps, try doing a proof of concept on your end? I guess you just need to set up a Git Repository, and then, there's your basic WinGet repo?

Ultimately using WinGet in an IaC scenario sounds like a bit more work (and more moving pieces) than just running a PowerShell script to download and run a handful of installers... but I don't know.

I'd love to see what you put together, or more specifics as you learn them yourself.

Thanks for the detailed explanation @sylvain-martel_3976 - this is quite helpful for me! A few more questions if you don't mind :)

All. First-party and 3rd party. 90% 3rd party though.

Can you give an estimate of the number of Winget "packages" you expect to have in your feed? Like 10? 100? 1000?

What would the process of adding a third-party application to this feed? Will you be recreating the WinGet metadata (package?) from Microsoft's official feed?

Are most of these third-party applications in Microsoft's feed, or are they like from vendors or whatnot?

I'm trying to get a feeling for the concept of "connectors" in WinGet, and if it even makes sense.

Well, we use a file server to host the packages.

Is this just like an ordinary file server, where you use UNC paths? We are have plans to do some major improvements on ProGet's Asset Directory, including enabling replication, cloud storage, and (hopefully) versioning.

So to summarize your usecase...

• WinGet is a means to manage (mostly) third-party applications that you keep on a private file server, and wish to install on other servers through automation
• WinGet is preferred to a regular file server because it includes some metadata like requirements on what to install
• WinGet is preferred to Chocolatey for security and cost reasons

What I'd like to do is build a usecase for how organizations can use WinGet in combination with ProGet (in particular the asset directories) to add value.

Whoops! We often test ProGet using our own ProGet instance, and it looks like someone (ahem.. no need to mention names) disabled Semantic Versioning for Containers on our instance for testing purposes.

It's now back on, which means you should be able to use latest again, or even 5 or 5.2 if you wanted.

Thanks @sylvain-martel_3976 ! That's really helpful to know.

From a positioning standpoint, I think that WinGet is trying to replace the Windows Store while offering a more developer-friendly experience to create your own apps on Windows. But it's hard to say -- it's only 1.0 as you said, and the positioning has changed a few times already.

Our plan is to use Winget to deploy apps on servers using ansible playbook.

Can you tell us more about this? Lots of questions, hope you don't mind us trying to understand a bit more...

What apps? As in, applications your company created (first-party apps), or third-party ones (like Firefox or SQL Server, etc).

Where to host contents? WinGet doesn't seem to host the installers or package contents, so where will you host these files? Will you keep them on the internet (third-party apps), or have your own custom source (like a CDN or ProGet Asset Directory)

[First-party / custom apps] Do you already make installers for your apps? If not, how will you automate installation?

[First-party / custom apps], Have you explored ProGet's Universal Packages instead? I think these would be better/easier, especially for server applications. We've seen a lot of people use these over Chocolatey.

[Third-party apps] Why use a private feed? Is it for security/vulnerability? Quality checking? Etc.

@atripp @philippe-camelio_3885 whoops!

This error should be fixed already, and I just published 1.10.2 - please try that!

Thanks Simon! Much appreciated.

This will officially be our first "v6 preview feature" --- figuring out the general direction of v6 has been on my list, so thanks for helping out on that ;D

I can't imagine it will change between now and v6, but this allows us to work on documentation, etc. in the meantime. No ETA yet (I'll let the team take it from here as PG-1945), but probably in near upcoming maintence releases.

@scroak_6473 ah yes for sure -- actually it really could only be a user-level function with this proposal. Basically, this is just a page that allows non-admins to manage a subset of API keys (i.e. ones with only "Feed API" and with their name in the impersonate field) -- we proposing to change how API work, etc.

Let's make sure to definitely clarify this in the UI & documentation. I think you have a lot more experience working with "regular" ProGet users (i.e. non-admins) than we do, so I'm hoping you give some advice on how to best communicate how this all works to regular users, as to means to reduce the administrative work you do.

User Drop Down (from your screenshot)

• Add "Personal API Keys" link above "Change Password"
• Show link only to Authenticated users (i.e. Anonymous can't see it)
• Link navigates to /my/api-keys

View Package Page

• should we also link to Personal API Keys? Where to mention this?

Personal API Keys (new page)

• Show InfoBox based on user type (see below)
• Show table with three columns: Api Key, Description, and delete button ()
• Clicking on API Key opens a modal that allows editing the description
• "Create API Key" Button works as above (only description allowed)

Messages For Admins
This page shows only API keys with "Feed API" permissions that impersonate your username ({username}), and is intended for non-administrators to manage their own API keys. Because you have Admin_ConfigureProGet permission, you can [manage all API Keys in ProGet].

Messages For Non-Admins
Something about <user>:<password>? This page allows you to manage API Keys that ...?

cc/ @harald-somnes-hanssen_2204 I remember you had some other "regular user" suggestions as well (but unfortunately they didn't make the 5.3 release)

Hey @philippe-camelio_3885 ,

Ah, looks like this is a bug in the Scripting extension, and will happen if you don't specify the .AHCURRENTVALUE comment block.

I just patched it real quick (please install Scripting v1.10.2-CI.1), and now at a absolute minimum this would work:

<# 
.DESCRIPTION
Create AD Group 

.AHEXECMODE
$ExecMode
#>

Given your script, here are the other values I recommend...

.AHCONFIGTYPE
ADGroup
.AHCONFIGKEY
$Name

There's something a little funny about the collected Value after remediating drift, but taht's something we'll look at later. For now we need to get these docs published; they're draft for now.

Anyway please give this a shot and let us know how it goes

@Stephen-Schaff

Very glad to hear! It makes absolutely no sense makes that Managed Pipeline Mode would have any impact on how these libraries behave... but there you have it. These libraries are ancient, and even Microsoft has no idea how they work anymore.

Apparently Microsoft is working on shipping brand new libraries (no COM interop magic) in .NET6, which means by .NET8 they just might be usable, and then perhaps in another decade they won't require mysterious changes

@Stephen-Schaff FYI; looks like this got added as a result of your feedback. Easy change, mostly to prevent the "locked out" symptom when switching.

Let us know what you find out about test system, etc.

Thanks @Stephen-Schaff, we will definitely figure out how to get this working.

The /debug/integrated-auth is just a URL you can type in, like https://<yourinstance /debug/integrated-auth. It provides some details about the current configuration, like this:

https://proget.inedo.com/debug/integrated-auth

However, it will only help us identify potential problems once you've changed the configuration.

--

Since downtime is a concern, let's start by getting a testing environment. Can you install ProGet on a similar server, and then just use the Trial license key? You can request one at my.inedo.com.

If this is a "permanent server" where you wanted a testing/training place for all the various functionality, we would ask for a license... but this usage is fine, most especially given all the headaches.

Once you have that, testing should be a lot easier.

If we can reproduce the problem on the test server, then that's good news - because we can try some basic config changes.

The first thing that comes to mind is set LoadUserProfile=true on IIS AppPool. That "works wonders" due to the bizarre behaviors of some of Microsoft's libraries. The second is trying a different username. We can just keep going, all the way down to running the webserver outside of IIS.

We can also just keep iterating on code, finding places to add new debug information, etc.

At this point, even if you could test before switching, it wouldn't help... because it would still not work. Making it easier to switch without needing to reset password, is a problem for a different day... for now we need to figure out how to add diagnostic information.

If it's as simple as LoadUserProfile=true, then and lesson learned -- I guess we'll try to detect and warn, or atleast be trained on support to know how to respond.

And if we can't reproduce the problem on test, then we'll just keep coming up with ways to figure it out quickly while minimizing downtime. For example, if we absolutely had to, we could get a second ProGet instance on one server, just to try this out, and not impact the first.

Anyways from here --- let's see what we can do on the test server!

@Stephen-Schaff I'm sorry that you're having configuration challenges; active directory is very complicated, especially on old servers, complex domains, etc.

Another thing to try is visiting /debug/integrated-auth in your instance, but that only uses the current directory provider (so you need to switch first), and it may not provide any new information.

I tried using your Active Directory debug tool and it the settings work perfectly there. The same settings fail in ProGet.

The software/code is identical, so if this is happening then it means something is different in your production environment. The most likely case is that the user account running ProGet has different permissions or is otherwise unable to query the directory in the same way. You could also try setting LoadUserProfile=true, that seems to help with Microsoft's libraries for unknown reasons.

We are happy to add product enhancements to make this easier to debug, and identify what's wrong. At this point, it's a big guessing game as to what's different about your environment, because it works for everyone else.

As much as I really like ProGet (both price and features), I am going to have to spend some time today to go shopping for a product that can integrate with my Active Directory system. :(

I can probably convince my Operations Team to give it one more go if you have any suggestions.

We're doing our best to help you and try to solve this bizarre and complicated problem that only seems to impact you. I know you're frustrated, and we are too, because we don't like seeing our software not work in some environments. We could just as easily blame you or whoever configured your environment, or perhaps Microsoft, who wrote the buggy libraries we use.

But you're a smart guy, and you must know that this abrasive communication style causes only unnecessary stress to the people you interact with. You are certainly intelligent to know it's not appropriate or productive, either.

So, please be more considerate. Or if this is your preferred vendor management approach and communication style, then do "go shopping" and do find another vendor.

@Stephen-Schaff said in Feature Request: Limit Container Images to those that are inherited from another feed:

It is possible that you are underestimating the value that enterprises place on governance of their processes.

It's really valuable, but from a marketing/sales perspective, it's hard to "compete" with the likes of ServiceNow for governance improvement. Containers are already too much in the weeds for the folks with governance problems/pain points... and they can "just throw a guy they trust to manage the details, like Stephen"

@Stephen-Schaff said in Feature Request: Limit Container Images to those that are inherited from another feed:

Multistage Dockerfiles to not pass any part of the Dockerfile that was used from a previous stage on to the later stages (aside from any files copied).

That makes sense, but you could still validate that containers "inherit" from a known base image. In ProGet, you can navigate across base images.

@Stephen-Schaff thanks again for all the thoughts. This is making a lot of sense.

To summarize the Multistage-approach, "each build is done in its own sandbox so that the tools used to build the software are kept in lock-step with the software", or something like that.

Anyways, I really dig it, and I like where your thoughts are on this! This is the sort of "engineering/process rigor" I always envisioned to help companies build with our tools.

Actually, I'm already thinking of how we can do all of this with not just ProGet (probably using a function I spec'd out ages ago... happy to share that idea with you at some point, it'd totally work I think), but also integrate some great self-service workflows into BuildMaster for this sort of thing.

Now all that said.... I'm an engineer by trade and by training, and maybe ten years ago I would jump on this idea, and code it all myself over a weekend

My company starts to balk after you get over a few thousand dollars unless the value is really big.

As well they should! I do the same

The reason I ask the "how much more would you pay" question is to get a "gut feeling" for demonstrating/articulating value. It sounds like this gap is pretty big right now, and that's something I'd like to work on.

This is more a marketing exercise, and not something I'd normally do in a venue like this, but the next InedoCon is a ways off, and you seem to have a great grasp on these topics already!

Now just to be clear, we don't intend on price hikes, especially for a feature like this, but one of our biggest challenges (thanks to an engineering CEO ) is demonstrating value to buyers.

So working on this, some quick back-of-the-napkin math shows that "30% of $2k/year" is $600, or at most 6 developer hours of internal value (whether that means saved or gained). That's not very valuable.

What I'd like to demonstrate is the business case for why this feature is easily worth $10,000/year (and probably even $100k/year) to companies with 50-100+ developers. That would be very valuable.

This is a really difficult exercise...

---

It's actually pretty easy easy to do demonstrate this kind of value with the high-availability feature alone. When 100 developers lose an hour of productivity due to a failure of ProGet, that's $10,000 lost. If your ProGet Basic instance goes caput, you'll lose a LOT more than that.

Of course, the problem with the high-availability feature, however, is the skills and mindset required to monitor/maintain high-availability things. If you don't have the infrastructure team on call that knows how to monitor, how to respond, how to support developers, etc., then it's not a feature you can use.

---

The value of this feature idea isn't so clear, at least in terms of Time (delivering quicker, increasing productivity), Money (lowering labor costs, increasing profits), or Risk reduction.

But even if we figure those out, the major problem I see with this feature is the skill/complexity in using it.

With the high-availability feature, you need to have a high-availability organization, but a lot of organizations already have that, so it fits right in.

The knowledge/skills gaps seem much larger, and the Problems an organization would need to have are complex. A buyer might think, "So an 'unapproved container' made its way to production, despite passing all the acceptance tests... is it really a problem?"

Anyways just brainstorming! But this is just part of the process.

There's not, but how about doing a "manual sync" by just copy/pasting the infrastructure JSON? Then switch once you're ready?

Thanks @Stephen-Schaff

Great idea, I added a product change issue (PG-1906) that will block enabling windows auth within ProGet unless the user can be found. This should greatly reduce the problems.

@Stephen-Schaff thanks for taking a look!

In addition to container development patterns, I'm also researching the so-called software supply-chain attacks... and I think that the inherent complexity of Docker and containerized development expands the attack surface exponentially.

The first is that the Dockerfile is usually thought of as the source of truth for the container build. Any configuration that is setup in the auto build should look at the final container image layer in the Dockerfile (so you get the real one if it is a multistage file). It could allow the user to select from any history of that container image as the "base" image. But just selecting any base image could put the selection at odds with the Dockerfile.

Can I pick your brain a bit more on this? What do you mean "source of truth for the container build"?

Is this Dockerfile typically stored in source control, alongside the application source code? In a world of cookie-cutter microservices and microapps, it seems these Dockerfiles should be nothing more than the absolute basics on top of a base specific image?

Separately, but related... I just read up on multi-stage builds and the Builder pattern vs. Multi-stage builds in Docker, but I'm struggling to see how this all adds up to real-world use inside of organizations?

Is this really a pattern that is emerging? Giving development teams not only control over their own base image, but over the tools to compile/build their code?

This just seems to add another layer to what ought to be a very simple process, and more opportunities for supply-chain attacks.

BuildMaster seems to have a "we will build your Dockerfile for you" feature. Which is fine until you need some complicated stuff in the Dockerfile that is not supported as a feature.

Definitely something I want our content to address; I think the "build for you" is good for these cookie-cutter microservices/microapps, which seem to be a better way to go than pushing all the Docker complexity to development teams.

I think that when this becomes a "Must Have" for me, I will see if I can meet the need with a web hook.

Well you definitely get the value in it -- but convincing others the importance of this sort of features is a real challenge, and that's where the content I mentioned comes in.

Question: how much more would you pay (or encourage your organization to pay) for ProGet if this feature were to be there? 20% 100%? 1000%? Why is it worth that extra?

I ask because that's an important way for us to start thinking about "value", which is what the check-signers want... and ultimately how we need to think about which features and content to invest in developing.

Hi @Stephen-Schaff_8186 , thanks for the feature request!

I like it. This is a good idea, and sounds like a way for us to enforce the "base image" pattern on the ProGet-side that we want to promote with our products. This is how we do it on the BuildMaster-side: https://inedo.com/buildmaster/containerized-development-in-buildmaster

Would you mind taking a look at it, and sharing your opinion on the approach? This is a solution we developed from the "outside looking in", since we simply don't have firsthand experience with this problem.

We are focusing a lot of our efforts on "why to" content, as I call it. This is even more important than the "how to" approach (which, that above article demonstrates), because it speaks to a problem many folks won't even realize they have. "this is just the way docker is" is something I hear in the field, all the time. My take is -- if we don't have a lot of detailed "why to" content about the feature, then it may as well not exist.

To give you an idea of what I'm talking about, our team just finished writing a pretty comprehensive and easy-to-understand guide on .NET5 Migration; the articles are here -- https://blog.inedo.com/tag/net-5 (fill out the form on any of those posts to get a copy).

We've got some other guides in the pipeline, and I want to start on containerized development as well. So the more feedback from advanced folks in the field, like yourself, who think of these ideas, the better!

Cheers.

Hey @nicolas-morissette_6285,

Thanks for clarifying!

So it sounds like this is a non-critical integration point. In other words, you're not building a workflow that relies on the two to be integrally connected, it's more a "convenience" such that one team get another team's developer library packages more easily.

I'm more nervous about enabling / encouraging the first use case; given the velocity and unpredictability of ADO's changes, there's a high change it will "just break" some day -- and as you've seen from their support, it's hard to get "them" to care. We most definitely care, which is why I'm cautious about putting us in a position to be responsible for customers' infrastructure breaking on account of a third-party API suddenly changing.

We most definitely work with a lot of teams at Microsoft, but their internal org hasn't really changed in the many years since I first saw this...

... so a lot of times, it's partnering with customers to push different groups at Microsoft to work together to serve what should be their common interest (i.e. a paying customer). For example, getting AzureDevOps Packages and NuGet team to collaborate on API things.

After the change in guard at VSO/ADO(i.e. after Ed Blankenship), we really don't have any meaningful contacts or pull w/ that group. But we definitely have contacts and work closely with other teams (like NuGet team, etc).

All that said, we'll also work-around this quirk, and I've put in a change (PG-1862) to accommodate this. But just

Hi @nicolas-morissette_6285 ,

Can you help me understand the use case here of ProGet and Azure DevOps Packages? Basically I'm a bit hesitant to invest in continued work-arounds for bugs/quirks in ADO Packages.

The ADO Packages team has made it very clear they aren't interested in spec compliance (whether NuGet or otherwise), and that they won't support interoperability with any other tool aside from Azure DevOps. That makes sense because they want you to only and entirely use Azure DevOps.

We've already work around one quirk, and I have little confidence that fixing this quirk will actually solve the problem (i.e. enable the use case you're trying to build) and that it will stay fixed. In a few weeks/months, when they ship some new functionality, your hybrid-ADO Packages usecase may break again.

For example, when ADO-Packages decided to block "private upstream feeds" (I guess that's what it's called?), it disabled the use-case of having ADO-Packages pull from ProGet servers, or vendor's feeds, or anywhere other than NuGet.org or ADO-Packages feeds. Fortunately it's easy enough to work-around in ADO.

Bottom line, using ADO Packages this way may prove to be very fragile, and you may be better served by not using ADO Packages altogether. A lot of other users have made that decision, and ADO works fine (better IMO) without using their Packages feature... and it's highly likely it will continue to indefinitely.

Alex

Hard to say, but there were some bugs with NuGet.org, where some packages would not show up when queried through the v2 API, but would come with v3. I wonder if it's related to you using a v3 API connector, but a v2 API call to ProGet? It should work.

You can append #v2 to a nuget.org connector's API to force it to use the v2 api instead of the v3.

There's no limit set by ProGet to the size of a feed, and we have customers with terrabytes of packages, so it should be okay.

Hello;

Kaspersky is totally wrong. This is a 100% false positive. Can you share this thread with them?

LoupeViewer.exe is a tool that, among other things, inspects MSIL of live processes for debugging purposes, and it's embedded in our installer. It seems their (or whatever provider they use) heuristics algorithm saw an ".exe that embeds an .exe that uses MSIL inspection hooks like this", and flagged it as this signature.

It shouldn't take more than a few seconds to verify that LoupeViewer.exe is not a trojan, and it'd be easy to inspect the disassembly.

Microsoft and several other providers have already removed whitelisted this in response to other customers submitting this.

Our installer is safe. Rest assured that if the package is signed by Inedo, you can be absolutely certain that no "viruses," malware, Trojans, or other malicious code were injected into our installers.

Please see this article for more information: https://inedo.com/support/kb/1113/false-positive

Hi Stijn,

The PowerShell client is indicating that there's an error downloading that file, and that when it tries to open the package (which is a zip file), it's corrupt.

I don't know what the error is, but if you follow that URL in your browser, you should receive a .nupkg file. If you can open it up as a zip file, and it opens ok, then the error might have to do with a proxy server, or something, that's corrupting the fileon the way from ProGet to PowerShell?

Hello;

We are aware of the vulnerabilities.

Our usage of this library is minimal, and we do not use it in a manner that would impact product security (i.e. "passing HTML from untrusted sources - even after sanitizing it - to one of jQuery's DOM manipulation methods (i.e. .html(), .append(), and others). So please consider this a "false positive".

Upgrading the library to 3.5.0 would add no value to ProGet users, but it would provide significant risk of UI regressions and risk of introducing unknown security vulnerabilities, that haven't yet been reported.

Aaand, published :)

Appreciate verifying & helping us finding this bug.

Just a quick in-progress preview of what we've got coming; feedback welcome!

Hi Arno,

Are you actually using Conan?

I've talked with a lot of users over the years about C/C++ development, and the emergence of Conan, and after a careful evaluation, not a single user opted to not use it due to the complexity it adds. Some instead switched to NuGet (which supports C++), and others just use UPack / Universal Packages.

So far as I can tell (I'm not a C/C++ developer), the big challenge is that Conan "wants to be" a cross-platform open source hub (like the NuGet.org of C/C++), but C/C++ isn't cross-platform, and there's no real standardization in how you configure C/C++ project, so the Conan platform has to jump through a lot of hoops to make it work.

From my understanding, you need to make "recipes" that are like a kind of installation script, and these are written in Python (??), and executed using Bash. So, on Windows, that many more steps to follow.

Another problem I heard was development environments vs target environments. Like how you can code on Windows, but your target is embedded something, so it adds a whole new layer of complexity. I also remember something about, building from source vs libraries, and how it was a lot of work to get it to work the way you want.

Our customers had zero interest in taking open source packages from the Conan's public repository; it was only for their own, internal-use packages.

Again... this is all second-hand for me, so, if you have more insight please add it.

@olivier we're mostly still looking for insight/answers into the questions @gdivis had asked, so if you can help on this we could at least move the ball forward

@gdivis said in Support for R and CRAN:

1. ProGet typically needs to understand versioning rules for packages (to show which is the latest, to do retention policies, etc). From what I've gathered, CRAN uses 2 or more integers separated by dots. Does anyone know if that's required universally, or if it's just a convention?
2. CRAN doesn't appear to have a server-side repository API, instead just exposing a list of packages using HTML. Is my understanding correct here?
3. Submitting a package to CRAN looks like it's only done manually using a form here. Is that right, or is there some other automated mechanism that is used?

Thanks for the additional info... I definitely appreciate the desire for consistency in package management.

we have to manually copy down the packages

Why is this? Is there a sort of internet restriction policy in your build environment?

What do you mean by "drop in value"... is it like, "we already have ProGet, and it would save X amount of time to not have to create/manage a separate, private Conda repository"?