Posts made by apxltd

@sebastien-gamby_3349 check with the VSCode team.

They first need to implement support for private repository/galleries, and then we'd need to see some level of stability before considering it.

Hey @splatteredbits ,

We've only had a single request for a private Chef Supermarket over all these years, and I'm almost certain that was from you

My general impression of the market/demand is that Chef has become a "legacy solution" (compared to Docker, K8, etc.), and Progress has no plans to increase its relevance. They appear to be operating in "maintenance mode", and it's kind of like an SVN in a Git world.

There's still some activity on recent packages, but it looks like it's just minor version bumps. I could read this wrong... but I don't think it makes sense for us to invest in a declining platform without a lot of user demand.

Best,
Alex

Hi @splatteredbits ,

I don't think this would be technically possible, at least from my initial review.

Swift has neither "packages" nor "repositories" - the package index is nothing more than a packages.json file that points to other repositories on GitHub.

The client clones those repositories and presumably checks-out a tag to get a specific version. Likewise, to "publish" a package to the Swift Package Index, you submit a GitHub issue.

So I guess, if you wanted a private Swift repository, you'd just need to make a Git repository and follow those conventions?

Cheers,
Alex

@Jonathan-Engstrom they have some hacky something-or-other, but I'd gauge it's at least a decade or two away from being ready for prime time. At least, given their track record on their PowerShellGet/PowerShellGallery v3 launch plans (pushed to 2027.... maybe??)

There's just no appetite for WinGet in the enterprise space, especially since Chocolatey does everything better and has a significantly more reliable / responsive team behind it.

On our end, it doesn't make to try jumping into such an immature ecosystem, especially one run by Microsoft, who's leadership is currently focused on making their BingAI Copilot relevant and selling more Azure web hosting

Windows barely fits into their strategy, let a lone WinGet. But we'll see... maybe the WinGet team can fight against the current enough and increase adoption.

At this point, WinGet It's not a bet I'm willing to take though

@Jonathan-Engstrom suggest asking WinGet team to build support and a use case for private repositories. Currently there is none and thus, it's impossible to use with ProGet.

Until then... WinGet is basically just the Windows Store, except you it run from the Commandline and has a ton of shady, unvetted packages from internet randos

Thanks for the details @mikes_4196

If you're for any kind of control or oversight into your packages (i.e. typical business use case) as opposed to having everyone just rawdog a bunch of shady installers from the internet, I'd definitely recommend looking into Chocolaty again and talk to their support team.

I can't speak to the issues, but that's what everyone is using... and "no one" is using WinGet outside of hobbyists and "shadow IT" devs. There's a good reason for that.

What we would like to have available is download caching.

This just isn't technically feasible due to their terrible design. Keep in mind that the WinGet "repository" is just a giant Git repository with a bunch of .yaml files that point to installer URLs on random urls on the internet.

https://github.com/microsoft/winget-pkgs/tree/master/manifests

From at technical standpoint, WinGet has no real specifications, no ability to script non-MSI installers, no APIs (unless you count git clone as an API), and worst of all no package files.

Open to having my mind changed here, but WinGet remains untenable for any serious organizational usage and has no future until Microsoft decides to build a private repository use case.

Cheers,

Alex

@dimas thanks for that!

We're currently studying/monitoring the performance for public mirroring, and one "concerning" thing I recently saw was a sqllite3 index file explode in file size. We don't know how that happened, but suspect it might have to do with some "aggressive" testing / request cancellation.

Let us know if you spot anything:

https://forums.inedo.com/topic/5410/debian-feed-mirror-performance

Hey @dimas ,

Ah, that makes sense. Looking at the manifest files, they have the same Identity@Id (which is what ProGet uses for the Package Name) but a different Identity@TargetPlatform (which must be new-ish):

<Identity Language="en-US" Id="python" Version="2025.2.0" Publisher="ms-python" TargetPlatform="win32-x64"/>
<Identity Language="en-US" Id="python" Version="2025.2.0" Publisher="ms-python" TargetPlatform="linux-x64"/>

We could probably figure something out in ProGet, but it's not trivial when it comes time to changing how we identify Package Names.

Can you try editing the manfiest file inside the vsix archive, and seeing if that works for your use case?

So basically:

<Identity Language="en-US" Id="python-win32-x64" Version="2025.2.0" Publisher="ms-python" TargetPlatform="win32-x64"/>
<Identity Language="en-US" Id="python-linux-x64" Version="2025.2.0" Publisher="ms-python" TargetPlatform="linux-x64"/>

Not quite sure if that would work, but figure it's worth a try.

Thanks,
Alex

@michal-roszak_0767 sorry, this is definitely sub-optimal and needs work

Basically, when you have API Key Hashing available, there's no way for the page to filter the incoming requests. So, all of the items are shown. It's not really a trivial to fix and requires us to rethink a bit how we are storing/logging this data. We can explore revisiting as part of ProGet 2026.

@dimas said in Incomplete proget debian connector local index file for ubuntu noble-backports dist:

why does the proget connector omit empty repository components altogether, instead of making them empty like virtually every other Debian repo?

Most other Debian repos use or fork Dpkg to generate the folder structure based on files on disk.

ProGet doesn't. Instead, we parse/index all packages in the repository in a local sqllite3 database. That database has a "Packages" table, and we use that to generate feed indexes (Release, InRelease). It's possible to change, and involves modifying the sqllite3 databases to add a Components table, and then changing the way we generate all the indexes.

It's not too hard but it's not trivial and probably isn't worth it unless we get more feedback on the matter.

Thanks,
Alex

Hi @dimas ,

It looks like VSCode still doesn't support private galleries. I know users have been requesting it for over eight years now.... so maybe it'll come soon! But until then, we can't implement it in ProGet.

I'm not sure what you mean about the python extension... but I do know that a visx file uses some kind of GUID to uniquely identify an extension. If the different "versions" of python visx use that same ID/Version, then I suppose it would get overwritten.

Alex

@steviecoaster the Native API should provide access to all of those procs FYI

And not just the name of course, but it looks like a great augmentation/tool for sure!

@steviecoaster said in Pagootle: pgutil, but PowerShell:

InedoOps

I always wanted to read that as Inedo Oops , so great new choice

@steviecoaster love it!!

Hi @e-rotteveel_1850 ,

Just as an FYI, this was much more complex that it seemed, and we ended up creating a whole new feature to address this called Feed Integrity Checks.

This will shipped in friday's maintenance release, and in theory should spot and allow you to resolve the issues:https://docs.inedo.com/docs/proget/feeds/feed-overview/proget-feed-storage#feed-integrity-checks

Alex

Hey @dan-brown_0128,

And of course, the other issue with using SHA Digests is that they are unsortable/unordered, and there's no way to know if you're on an "newer" or "older" version. Git inherently has this same problem -- and users who try to name packages after commit hashes quickly learn how poorly that works :)

Looking around, it does look like some registries actually support immutable tags (Ex ECR, Harbor). Some tags make sense to be excluded from immutability - like latest since those are dynamic by intention.

ProGet's tags are effectively immutable -- in that you need the overwrite permission. But we also have the Semantic Container Versioning, which takes it a step further and enforces tags. Unfortunately, a lot of users don't use this feature or follow our guidance - and they end up with all the predictable headaches.

As an engineer, I'm obviously offended in principle by these bad practices, but more practically... the consequences of this kind of misuse/abuse is Tool Blame. And that means we look bad and don't get a renewal.

So, my philosophy/approach as a product vendor is "provide a good solution", not necessary "enough rope to hang yourself". As I mentioned before, OCI Repositories are poorly-designed solution to a misunderstood problem -- and my read is that it's going to be a passing fad.

Cheers,

Alex

Hey @davidroberts63,

Sure - let me elaborate on that.

A key tenet of software versioning is that a "version" represents a single, static/fixed revision of software - like there is only one "MyProduct 4.2.8", and no matter where/when you download that version, it's always the same thing.

OCI/Docker repositories do not have versions. It's just not a concept in the model, period. Instead, container images are "tagged," and it is up to the publisher to determine which container images have which tags and when those tags change.

This is all by design, and there's nothing inherently "wrong" with updating "MyProduct:4.2.8" until it's the right "version" of the container. If you want a "specific version of a version" you need to use the that utterly useless digest code. That's how these repositories are designed and used.

Compare this to package repositories, which are write-once by design and deletes are exceptional. This holds to the "a version is a static/fixed revision" principle.

ProGet does have the Semantic Versioning for Containers but it's not very popular, because it doesn't follow industry norm and developers want more control and customization over this.

Replacing/overwriting tags is the norm in a lot of container-based workflows, which causes all the problems you would expect.

All this is to say, I just can't see how OCI registries for generic artifacts are anything more than a poorly-designed solution in search of a misunderstood problem.

Cheers,
Alex

Hi @dan-brown_0128,

Thanks for continuing the discussion.

From that quote, I get the sense that the container feeds were implemented by trying to force the process into what was implemented for traditional file-based packages rather than treating container images as a different style of artifact. This would explain why it seems OCI isn't a direct drop in fit -- because other traditional package consumers (pypi, nuget, maven, etc) do not behave the same as OCI.

Docker and Asset feeds are implemented considerably different than the other feeds, and even have a different UI/pages because of that. But the model of "feeds" and "proxying" (i.e. what people want to do with ProGet) is just fundamentally incompatible.

The OCI API is effectively identical to Docker API, but there's no sensible way to display/visualize "generic" artifacts. They have no name, just useless digest hashes. It'd be like using Git without comments, dates, or files names.

On the shock that for container images that you have to provide the URL to access the image -- that's not all that different than traditional software packages. Just on those traditional package tools you're specifying that URL Prefix as part of the client's configuration.

That's a pretty big deal IMO, and why this approach is dead in the water when dependencies are involved. It's a huge headache with Helm already - URL Prefixs need to change.

It's bad enough every single one of your build/deployment scripts need to be updated if you want to change the "URL Prefix" of a "source" (switching products, domain name changes, etc.) -- but these URL Prefixes are embedded in manifest/metadata files stored in the registry.

Of course, you can't "edit" those files stored in the registry, since the digest/hash would change, so you have to republish everything... which would cause anything that references those digests/hash to break.

If you've "transcended versioning" and always deploy the latest commit of everything always, then none of this really matters - but versioning is important for a lot of users still. Tags are not suitable for versioning.

Last one -- The URL thing. Take a look at some other common registries, including the public GitHub container registry. For those, the URL is always ghcr.io but they then prefix the image with the User/Org (eg: https://github.com/blakeblackshear/frigate/pkgs/container/frigate/394110795?tag=0.15.1). Truthfully, Proget could do a similar approach: proget.corp/MyOciOrDockerFeed/image/path/here:1.2.3

This is what ProGet has to do now -- we "hijack" the first part of the container namespace for the feed name, and GitHub does that same thing. But you're basically stuck forever with that host/feed name.

Hey Michal,

Check out latest pgutil (2.1.7) - I just made a quick change and now Asset will work.

Also, so will Maven2.

Best,
Alex

hi @jacob-kemme_1710 , thanks for all of the details.

I'm pretty sure the Docker feeds already support OCI containers. They are basically identical from an API standpoint, and we've made a few tweaks here and there to accept different media types or parse a field in a manifest differently.

I know for sure that vnd.oci.image.manifest.v1+json is already supported.. and if there's something missing, it should be trivial to add. I'd post that as a separate feature request, since it's likely a basic tweak.

As for Helm... that's a different story. I don't think it's going to work, and it doesn't seem like a good fit for ProGet. Nor is a "generic OCI registry", which is why I'm hesitant to even consider the feature.

The main issue I have is that an OCI registry is tied to a hostname, not to a URL. This is not what users expect or want with ProGet -- we have feeds. Users want to proxy public content, promotion content across feeds, etc. None of this is possible in an OCI registry.

We got Docker working as a feed by "hijacking" the repository namespace to contain a feed name. Helm charts don't have namespaces, so this is a no go.

We got Terraform feeds working by requiring some stupid prefix on the module name. There's just no other way to do it.

I don't think that would be smart to do with Helm. And obviously we couldn't do that for "generic OCI" content since it's just meaningless blobs/binaries.

I'm open to learning more... but my initial take is that if users don't find values in "feeds" (well-defined content types, permissions, segmentation, replication, etc), and just want a "generic place to shove random content stuff" then ProGet isn't the tool?

Alex

@dan-brown_0128 thanks for clarifying

2500 is a lot more than I'm thinking... I was envisioning a handful of frequently-updated / highly-trusted packages (like AWSSDK, ). I think the UI would struggle a bit.

Thinking about this further... a blanket "all future versions are OK" policy seems like it will cause more problems than benefits. That's a lot of decision-making burden to shift to developers, and they're just going to hit "update" the moment they're prompted.

Best case scenario, that update delivers Zero Business Value... but there's a chance it'll introduce a regression, etc. I'd really need to study the data before recommending a practice like this (let adding a supported usecase for it) I'd be open to some sort of "automatic approval" feature, but I'd want to see some other factors in there, other than just "name matches X".

As it stands, "package approval/promotion" isn't very widely used as it's solving a problem most organizations don't believe they have. Instead they drop $300K+ for Clownstrike or another security tool and believe it just "does everything for them". At that price, how could it not!

So I think it's best to pin this for a bit. It's still a pretty "frontier" discussion/feature, and I don't want to "invent" something that's not going to get a lot of widespread use.

Hi @dan-brown_0128, @scampbell_8969,

We were reviewing this as part of our ProGet 2025+ roadmap, but I'm not sure if it makes sense to do. To summarize...

"Once a package has been approved (at a name level), all subsequent versions are OK assuming there's no vulnerabilities or license issues. However, the current promotion model requires that we promote each and every version. By automating version promotion, it would allow developers access to newer versions of packages sooner, making access easier and devs will be more likely to upgrade."

What about just using a connector with package filters in this case? For example:

1. Create feed nuget-unapproved, nuget-approved
2. Restrict promotion from nuget-unapproved -> nuget-approved
3. Create connector NuGet.org-all and associate with nuget-unapproved
4. Create connector NuGet.org-approved and associate with nuget-approved
5. Edit connector filters on NuGet.org-approved to block everything except packages you want

This would be possible today and would avoid adding a complex feature

Thanks,
Alex

Thanks @steviecoaster! All that sounds great, especially since it won't require changing anything on our end :)

Much appreciateD!!

@steviecoaster great thanks! That sounds good to me, I think having a new install option would be great.

Aside from supporting the package (which I'm not really worried about since you built it!!), the main concern I would have is keeping up with versions. We have frequent ProGet releases and pgutil is basically on demand.

I vaguely remember there was some kind of auto-packaging thing? Or maybe I'm dreaming that?

We could add something to our deployment plan that "does something" as well.

Hi @steviecoaster ,

That's awesome! I just added steviecoaster as a maintainer of the package, and it says pending approval.

Quasi-related, but I delisted all versions of romp and upack since we no longer publish/support those tools, but they are still showing up: https://community.chocolatey.org/profiles/inedo

Not sure if it takes time to reindex or whatnot?

Cheers,
Alex

Hi @mikes_4196 ,

Per the discussion above... can you tell me a bit about "we" (i.e. the size/profile of your organization) and describe how you are using winget in your organization? And why did you use WinGet instead of Chocolatey?

Per the discussion above, it does not look like a tool/ecosystem that is usable at scale. I get that it's built in to Windows, but I haven't seen a case study demonstrating a real deployment -- even within teams at Microsoft.

The "central repository" is just a giant mess of files in a GitHub repository and can't reasonably be "proxied" via a connector like other feeds. They apparently have "private repositories", but there's no documentation/guidance on how to create/use private packages for end users, so I don't think many are doing this.

But open to learning more.

Cheers,
Alex

@forbzie22_0253,

From an API standpoint, I believe only npm packages support server-side tags. Rubygems might, but in any case we strongly advise against using them: https://blog.inedo.com/npm/smarter-npm-versioning-with-semver/

Server-side tagging is not likely something we will support in the future. Deprecation and Listing are really only there because many of the client APIs support them.

I'd need to see a strong, specific use case. If it's related to quality (e.g. dev, staging, prod), then it's a "hard no" because our solution to quality is prerelease version numbering/tagging with repackaging.

I do know that Sonatype Nexus has always supported tagging, but their repository is more of a "fileserver with server-side metadata" and ProGet is package-based. The only documented use case they have for tags has been quality, and our solution/approach is far superior.

Cheers,
Alex

Hi @kichikawa_2913 @jeff-miles_5073 @martin-helgesen_8100 ,

Good news! We've got a Terraform Feed working on version 2024.20-rc.4 and it seems ready for release:

I'd love to get a second set of eyes on our approach and the docs; this was a really interesting protocol/API to work with because there are no "Terraform Packages" - basically everything is just a pointer to a GitHub repository.

So, ProGet just packages it in a universal package and the Terraform CLI seems to be happy. What's a little unfortunate is that the hostname/feed need to be in the package, but that's also how Docker works. So I guess it can't be helped.

Thanks,
Alex

Hi @zs-dahe ,

The Package Promotion API only be checks for permission against toFeed - you don't need any other permission on the toFeed or fromFeed.

As far as more granular permissions, perhaps setting up a Personal Key for like a builds user would do the trick? That'll let you reuse the API Key and set up very granular permissions.

We decided not to duplicate that granular permission setting in the API Keys because it's already confusing enough

Thanks,
Alex

Thanks for clarifying @matt-lowe!

some IT managers are very much stuck on the 'Microsoft' rut and are unable to see past it, no matter the cost and time it adds to most IT management
...
TBH the only way I managed to get ProGet installed was for managing PowerShell scripts for a small project and have managed to keep it ticking over with other options.

I've definitely heard that... personally I think this is where vendor/sales can really come in and make a difference.

Sometimes it's a lot harder to sell things internally... it's not so much about "giving a sales pitch" but more that decision makers can have a different relationship with sales execs.

It's one thing to blow off a vendor demo or delay a project... but team morale can get shot if you do that internally, so it's safer to stick to status quo. And of course, sales execs can challenge the status quo in a much more direct way... trying that as an employee might be uncomfortable or feel insubordinate.

Anyway this isn't related to WinGet, but we might be able to help with the bigger picture -- might be worth spending 15-20 minutes just talking through it and seeing if there's any paths to go from there. We're seeing a lot of movement on the WinOps/ProGet/Chocolatey in big orgs, so clearly they're getting something over DIY solutions or WinGet.

Feel free to shoot a note at the contact form, and we'll take it from there!

Hi @matt-lowe

You're the first person to ask about it in three years. I kinda forget it's a thing :)

So since then, feeds are a lot easier to build.. but I also don't want to implement the next Bower feed

I don't remember much interest in WinGet from our WinOps userbase... and I only occasionally see mention of it on Twitter𝕏. I think Chocolatey has a lot more traction and I get the impression that it's more capable, stable, and mature. And it's very well supported.

What's your impression? Have you been using it? What's better about WinGet over Chocolatey?

I think this somewhat recent Reddit comment captures the general sentiments about Microsoft projects like WinGet:

AFAIK, winget was/is a Microsoft led attempt to offer something instead of everyone using chocolately. Sadly though, it's not a replacement.
 
That is, Microsoft has an "answer", but not necessarily commitment behind winget. So, unknown how well packages sitting behind winget will be maintained. And historically, it has been under scrutiny. Is it good now? Not sure anyone can ever say for sure.
 
Now, there will be those that say "fixed", today's winget isn't like the absolute mess it was early on. And that might be true. But how would we (the customer) know? Again, biggest problem with Microsoft is the "not knowing". Or the "deception" of making a project look "better" and then realizing it was all just a ruse.
 
Up to you. Today's weather report, winget is "good". But can't really say anything about tomorrow. Oh, and thank you Microsoft for making yet another ambiguous thing (not handling its delivery, deployment or long term outlook well).

The fact that their package repository is just stupidly large GitHub repo maintained with pull requests is concerning. The discussion with @sylvain-martel_3976 above - I think they were just considering WinGet, and didn't use it yet. Not sure what ever happened with that.

Anyway let me know your thoughts!

Alex

@davidroberts63 check out 2024.14... added via PG-2783 :)

Also, on the builds page, I'd recommend having a sort and/or filter ability for the Stage.

Oh yes, I could see that helping... actually i'm not sure if it's even listed on the Build page. How many active builds do you have BTW?

@carl-westman_8110 very happy to hear it, thanks for the kind words :)

@carl-westman_8110 it's definitely a nifty and convenient function, but a poor feature from a product standpoint.

Users don't expect functionality to bundle content like this, so they won't use it unless we push it -- and to do that, we need to articulate benefits, show how it's a better solution than what they're doing now, and convince them to take the time to switch.

That's obviously a lot of work, on top of all the code to get it working.

I know a handful of users have found some value in virtual packages (and we use it internally), but being reminded about them now -- with about a decade more of experience behind me -- it feels like the wrong thing to focus on at the time, with limited resources.

That said, it doesn't even come close my top 10 worst product choices... so probably better we built that, than something else

Alex

Hi @carl-westman_8110 ,

It's a Virtual Package; they're kind of a niche solution that, in retrospect, we probably shouldn't have created them.

Alex

Thanks for clarifying @sebastian

So I'm not exactly thrilled by this UI, but maybe this is fine.

What do you think?

This is a kind of "quick and dirty" page that would show up if you clicked on that GPL-2.0 license and the "# projects" number.

Here's one for the packages as well:

Hi @stuart-houston_1512 ,

If you're using almost entirely libraries that are available on PyPI.org and Anacaond.org, that makes the most sense. Users probably won't notice much of a difference.

Once libraries start being pulled through ProGet, you can start setting up policies/compliance rules to restrict packages. Or at least get warned about them.

At some point you can set up a package approval workflow, but I usually don't recommend that from "day one" - it's a bit too restrictive for end users, who are used to any package, any time.

If it's easy for you to identify first-party packages (maybe they are prefixed with MyCorp or something), then you can bring those in with bulk import. If no one uses (downloads) them after while, you can delete them with retention policies.

Cheers,
Alex

Hi @stuart-houston_1512 ,

I would go with the second option, i.e. migrating packages from your internal repository to ProGet. It should be relatively easy to do this with a bulk file-system import.

The concern I would have with trying to configure ProGet up as a "proxy" to an old, internal conda repository you configured is that ProGet doesn't really operate as a "proxy" (i.e. blindly forwarding requests), but instead aggregates results from multiple sources using an API.

The Conda API isn't very well documented, doesn't provide much metadata about packages, and an old internal server will most certainly have bugs/quirks that ProGet would never be aware of. So your end users will end up with a buggy experience.

Connecting to the official Anaconda repositories is fine, and if there are any issues/bugs (like they change the API or something), we can easily reproduce and fix it.

Alex

@sebastian I like that idea!

That information is readily linked in the database, so it's just a matter of figuring out how to get ProGet to display it.

Did you guys see the "Noncompliant packages" report (i.e. /sca/compliance-report)?

This is by far the easiest pattern: a non-sortable list of the top 100/500/1000 items.

That means your "show packages with Apache-2.0 licenses" wouldn't show everything, but I can't imagine you'd want to do that anyway. I'm thinking, you'd want to see the 7 packages with Artistic-2.0 instead.

I'd also like to ditch the "License Usage Issues" infobox, or at least replace it with something useful. It made sense with the ProGet 2023 license rules, but with policies we cannot easily query why a package/build is noncompliant.

Good news everyone, ProGet 2024.11 now has support for pub (Dart/Flutter) feeds! I'm going to lock this topic now, but if you run into any issues please start a new topic

cc/ @aaron-bryson_0311 @fabian-droege_8705 @proget-markus-koban_7308 @jensen-chappell_6168 @harald-somnes-hanssen_2204 @bvandehey_9055

Hi everyone,

Quick sneak peak of the work-in-progress:

Cheers,
Alex

Turns out the errors I encountered were entirely related to the "new" SQL Server driver that ProGet 2024 was using. Although this was in beta for well over a year, it was "only" in production for a few months by the time we incorporated it into ProGet. This driver issue impacts anyone who uses .NET on Linux.

As I mention, I think it's pretty clear that Microsoft has effectively abandoned SQL Server. It's one thing to release something this low-quality and untested... and another to have it linger with issues like this in production for months. We will be moving to Postgres, which do not have these endemic problems.

In the meantime, I have instituted the "one year rule" for Microsoft products - meaning, unless it's been shipped in their general release channel for a year, we will not even consider using it.

@sebastian thanks for sharing!

The issue that are describing sounds like more like the "ASP.NET Firehose" problem, and less like this obnoxious driver problem. They're both really obnoxious problems

The firehose problem hit basically anyone who went from .NET Framework to .NET Core, and had a high-traffic application. Basically, Microsoft gutted all of the "request throttling" that was in ASP.NET, which took into consideration a lot of factors, from system resources to processor type to cores, etc., and then queued requests appropriately --- now it's just left for the "user" to handle.

It was on the .NET8 roadmap to get fixed, but I guess they gave us a broken sql driver instead.

Anyway... great to hear the request throttle did the trick. I'm thinking we should just ship with a default of 100 and leave it at that. Maybe "wrap" some of the performance errors and advise the user to adjust the limit down or something.

I wasn't quite ready to write a blog post on this, but I figured I'd journal my thoughts on the forums to start - in the event that anyone is experience Linux performance issues with SQL Server.

Background: Low Linux Servers for ProGet Cloud Trial Edition

I've been "playing around" with low-spec Linux VMs (1-2 vCPU, 2-4 GB) for the purposes of creating "cloud trials" of ProGet. Eventually, I'd like to offer new users the ability to fill out a form on my.inedo.com that would automatically provision a server for them to evaluate.... but since we're paying for the resources, I want to keep our costs low.

We would never recommend such a low spec for production usage, but its fine for testing. That is, until a powerful developer machine hits it with an onslaught of requests during a NuGet or npm restore. This means hundreds of requests per second that need to be proxied to public repositories if the packages aren't cached.

To simulate this, I wrote a basic C# console program that just downloaded 1000's of packages in parallel from NuGet/npm feeds on ProGet with no caching enabled. This means that each request will trigger database queries and outbound requests to nuget.org. It's a DoS program, basically.

My goal was to find out what combination of settings would allow ProGet to not totally crap out during evaluation usage, and maybe discover a way to warn users that ProGet is under heavy load.

Unexpected Problem: Broken Microsoft SQL Server Driver

WAS: Unexpected Problem: Linux SQL Server Warmup Required

Turns out the errors I encountered were entirely related to the "new" SQL Server driver that ProGet 2024 was using. Although this was in beta for well over a year, it was "only" in production for a few months by the time we incorporated it into ProGet. This driver issue impacts anyone who uses .NET on Linux.

As I mention below, I think it's pretty clear that Microsoft has effectively abandoned SQL Server. It's one thing to release something this low-quality and untested... and another to have it linger with issues like this in production for months. We will be moving to Postgres, which do not have these endemic problems.

In the meantime, I have instituted the "one year rule" for Microsoft products - meaning, unless it's been shipped in their general release channel for a year, we will not even consider using it.

I noticed that if I ran my DoS program shortly after the container started, I would see a flood of messages like this: Microsoft.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 35 - An internal exception was caught)

That was not at all expected.

After researching this a bit, it looks like this is a well-known, multi-year bug in the Microsoft's SQL Server driver for Linux. It's documented in their GitHub repository, and it plagues even Microsoft's own business-line products. There's no ETA and virtually no response form Microsoft on the matter but it's endemic in their driver.

Long story short, low-spec Linux instance of ProGet needs to "warmup". There is some kind of bug in the client that is triggered when you try to open a bunch of connections at the same time. But once the connections are open, it seems to be fine.

In the short term, I don't know to address it. As long as the connections are opened "slowly enough" it's fine.... maybe? But no idea how to control that.

Expected Problem: Too Many Requests

A single, underspec'd machine can only handle so much load, and I wanted to find out what those limits were, and what errors I would get in "new user evaluation" scenario.

On Linux, with my test scenario, this manifests with basically the same, "error 35". I believe it's socket exhaustion, but who knows? In most production cases, we see SQL/network timeout - this is because the database is usually filled with a ton of data, and is taking a bit longer to respond. In my test scenario, it wasn't.

Non-solution: Slow the REquests

When I added a client-slide throttle - or even spaced out issuing the requests by 10ms - there were virtually no errors. If only NuGet and npm clients behaved that way...

Solution : Concurrent Request Limit

Under Admin > HTTP/S & Certificate Settings > Web Server Configuration, you can set a value called "Concurrent Request Limit". That made all the difference after warming up.

25 Requests worked like a charm. No matter what I threw at the machine, it could handle it. The web browsing experience was terrible, which makes me think we may want to create a separate throttle for web vs API in a case like this.

250 Requests caused moderate errors. Performance was better while browsing the web, but I'd get an occasional 500 error.

I wish I could give better advice, but "playing with this value" seems to be the best way to solve these problems in production. For our evaluation machines, I think 25ish is fine.

Another Solution : Nginx Request Limiting

I wanted to compare ProGet's built-in limiter with Nginx's rate limited when used as a reverse proxy. I played with it a bunch, and found that the same "25" setting basically eliminated errors on my micomachine.

Here's my Nginx config file:

limit_req_zone global zone=global_limit:10m rate=25r/s;

server {
    listen 80;
    server_name localhost;
    location / {
        limit_req zone=global_limit burst=500 nodelay;
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Interestingly enough, it also made the web browsing experience seem much better. I'll be honest, I have no idea how, but it has something to do with that nodelay keyword, and some algorithm they use.

My conclusion is that Nginx does a better job of appearing to not be as slow, but like I mentioned earlier, having a separate queue for Web vs API requests would probably make our tiny Linux box run a lot better.

Addressing Production Linux Performance Issues

Ultimately, I think "playing" with request limiting and being aware of the "warm up" is important. A newly-spun up container may exhibit

Long-term Solution: Moving away from SQL Server

This pains me to admit as a lifelong Microsoft devotee, but it's time for us to move to Postgres. This quote from Brent Ozar, one of the world's most prominent Microsoft SQL Server consultants, sums the scenario up nicely:

“[Microsoft SQL Server] shouldn’t be used for most new applications you build today. If you have an existing app built on it, you’re kinda stuck, but if you’re building a new application, you should use Postgres instead.”

My opinion.... Microsoft has all-but given up on SQL Server; this issue in particular has not only been open for two years, but it impacts their own products -- and instead of fixing it, their own product teams simply are moving to other databases

Many years back, we had a Postgres version of ProGet. We eventually dropped it because it's too much of a pain to maintain two sets of database code, and we never built the tools to migrate from one to the other.

That's something we'll have to do when going back to Postgres -- build some sort of database migrator. We'll also need to support both versions for a bit, and eventually say goodbye to SQL Server. No timeline on this, but just something I've been thinking about lately.

Anyway - just wanted to journal my thoughts, and the forums would be a nice place to post them - maybe I'll turn this to a blog or newsletter later :)

Alex

Hi @sebastian,

Thanks for sharing your thoughts on this! Few things to point out...

[1] The "Missing Package Problem" is not as bad in ProGet 2024, mostly because it will only apply when there's a license rule. In ProGet 2023, a "missing package" would happen even for vulnerabilities.

[2] We're working on a new feature/module (tentatively called "Remote Metadata") that is intended to routinely cache/update meatdata from public repos like nuget.org, npmjs.org, etc. This feature enables two usecases:

• Routinely update "server-side metadata" like Deprecated, Unlisted on cached packages
• Fetch metadata for packages not in ProGet during build analysis

It works somewhat independently, and basically it'll just show up on the Admin tabs as like "Remote Metadata" and you can configure providers, urls, etc.

I hope to have a prototype in a couple weeks and will post some details on a new forum posts. As an FYI this is something we will limit in the Free/Basic editions and have full-featured in the ProGet Enterprise product.

[3] "Package In ProGet" could be a policy rule to add after RMetadata feature, though it's probably not a big deal if ProGet can detect licenses thanks to RMetadata

Best,

Alex

@Alex_9348 found them!

https://s3.amazonaws.com/cdn.inedo.com/release/rules.pdf

Enjoy :)

@fabian-droege_8705 thanks for the insight! I know basically nothing about the platform, so good to hear from someone familiar w/ the ecosystem.

We'll take another look in the coming weeks, and I'll post an update once we have a better idea of roadmap. Just skimming the thread here, it seems that there is a package/archive format, an API, and documentation for private repositories - so that's a huge positive

@proget-markus-koban_7308 over the past few years, a lot of the "chatter" I've seen on social/news over the past couple years seems to have trended negatively towards Dart/Flutter as a platform/technology - and I haven't seen growth. It seems pretty niche still?

And then just last week or so, I saw headlines about Google laying off the Dart/Flutter team.

What do you think of the future of the platform?

Maybe I'm getting old and 14px seemed to small

In addition, we don't display a lot of textual information so the larger font size seemed to fill out the whitespace nicer, particularly in tables (which there are a lot).

I was never really all that happy with how this filled out whitespace (from ProGet 2023):

I'm not thrilled about ProGet 2024, but it felt like an improvement... and most importantly, "something different" than past few years:

That said, for ProGet 2025 I'd love to do a much more notable style refresh (logos? etc?), perhaps even some navigation tweaks. So open to ideas there

@philippe-camelio_3885 thank you, I appreciate that! No plans to give up -- it's a passion of mine, and I feel someday we'll figure out a better product marketing fit :)