Hi @carl-westman_8110 ,

Under Admin > Diagnostic Center, you should see some error-level messages logged. Can you share those? That will help us identify what the actual error messages are.

Under Admin > Scheduled Jobs, what do you see scheduled at midnight? Are you aware of other external scheduled jobs, for example backup or nightly builds?

One thing to be aware of... users with similar symptoms have discovered they were "rate limited" by Azure SQL. This doesn't lead to resource graph spikes or "419 errors" (like an API rate limit), but query throttling; some queries simply take 100x longer. And this eventually leads to random errors.

I think 10DTU is relatively small, so this could be happening? Anyway the error messages will show.

This may be a good opportunity to migrate to PostgreSQL, which will not have any kind of limits.

Thanks,
Steve

Hi @Ashley ,

Amazing, thanks for this investigation! This is very helpful.

You've also captured the exact issue we have with a lot of these APIs; no real specs (or worse... wrong specs), which involves a reverse engineering effort. This quote is wonderful:

In summary, due to the lack of an official JSON schema and potential changes between npm versions, it's crucial to treat the npm audit --json output structure as subject to change. Relying solely on the npm CLI source code for the most accurate and up-to-date information is recommended, along with implementing robust version checks and testing in your development processes.

Wouldn't it be nice if we could build our products like that! Anyway, I was curious and I asked internally, and the original engineer is almost certain that npmjs used to have the CVE number there, which we also used to emit until changing to PGVD number in ProGet 2023. But that is consistent with their position of "we don't do specs, just read the latest commit to figure it out.

But it doesn't really matter, that's what we have now...

To ensure consistency with the public npm registry, would it be possible to change ProGet to use a number for the id field on the audit response?

If you discovered this in ProGet 2025, we'd just change it in ProGet 2026 and not worry about it. But we need to be super-careful making API changes in a maintenance release. We can't quite get away with Microsoft/GitHub levels of quality :)

Seemingly harmless changes lead to broken builds, which are really painful to debug; for example, we recently added upload-time to PyPI API. That caused older versions of the pip client to break due to a bug in how they parsed dates.

There are probably too many client/versions to answer the question "what is this field even used for and what are the consequences of changing it?"

Perhaps an easier route is to just get pnmp to change to a string, like the rest of them? Or, just ignore it and "let it be"?

Over time, if you follow our recommend best practices, these audits will show fewer and fewer vulnerabilities. We plan to continuously refine the PVRS algorithm to better combat both "vulnslop" (i.e. AI-discovered and AI-generated vulnerabilities that cannot be exploited in any real-world scenario nor would cause any real-world harm if so) and misreporting.

Thanks,
Steve

Hi @Ashley ,

Thanks for the detailed information. I haven't set up an npm/pnpm environment to test this, but I wanted to share a few thoughts, and see if you can test something.

I doubt the encoding is an issue.

Idea 1: Different API Endpoints

There are two vulnerability APIs, and I think one is kind of deprecated? Anyway ProGet has never implemented that one.

If you haven't already, I'd make sure pnpm is calling the same endpoint.

Idea 2: Null Severity

In the example you provided, severity is only set on one of the three items. If this is the problem, then it would mean that pnpm is erroring when processing the resultset.

This will be trivial to verify: can you (temporarily) override the assessment of PGV-262413Y and PGV-262413X to be Contain? Then, run it again. You should see a "critical" serverity in the results then.

That being said, this is a bug. These vulnerabilities should be suppressed and not show up in the results at all; we'll fix that via PG-3317.

Idea 3: Nonnumeric Id

The other possibility; ProGet is using a string identifier, but npm is using an integer identifier. I suspect that npm used to do CVEs that field, but who knows. None of this is documented.

I'm not really sure how how test this without some kind of proxy/MITM interception, or reverse-engineering their source code.

Maybe this is a GitHub issue on the pnpm repository?

Thanks,
Steve

Hi @svc-4x9p2a_6341 ,

This is known issue and will be addressed via PG-3309 in the next maintenance release. You can just ignore the messages for the time being, it only happens like once a week or so.... I wouldn't modify the database either.

Thanks,
Steve

Hi @mhelp_5176,

Connecting to the embedded database requires local access to the ProGet server. Once an attacker has that, then they already have full access to the database, as it's stored as files on disk... basically it's the equivalent of storing the safe key inside of the safe :)

As such. it doesn't make sense for us to add complexity to the product to support changing a password that's already secured.

Thanks,
Steve

Hello,

Thanks for reporting this; this behavior is by design.

ProGet uses authentication tickets (i.e. encrypted cookies). If the username on a valid ticket cannot be located in the user directory, then a "user not found" error will occur, as it did here.

This errant behavior is generally desired for troubleshooting cases where users are intermittently not available from a user directory. Otherwise it's very difficult to troubleshoot, since it will just appear as a random log-out.

Thanks,
Steve

@sigurd-hansen_7559 wow very cool! We didn't anticipate anyone would customize these templates beyond adding a variable or two, so it it didn't seem to make a lot of sense to invest in a proper editor (like we have in the other products)

Anyway, it'd be interesting to see what kind of templates you develop.

Hi @alkhleif_2585 ,

There isn't much demand for this kind of feature (I think we've only seen one or two requests for this over very many years) so it's not on our roadmap.

And it's unlikely we will add it to the roadmap considering that it would be quite costly to build, support, and maintain. Especially considering that we'd need to support all of the major "vaults" out there as well.

It doesn't seem to add much value to ProGet as these types of read-only credentials can be very long living (several years) without any security risk, and take just a few minutes to rotate when needed.

That being said, you could probably write a "sync script" pretty easily that continuously updates the connector credentials by connecting to the vault and then updating it via the API.

Cheers,
Steve

Hi @vdubrovskyi_1854 ,

Without knowing which specific package you're referring to, it's hard to give a more specific example.

Historically, Packagist was effectively a database of "GitHub Repository pointers" and Composer was effectively just a wrapper around the Git client. They can both still operate in that mode, and needs to for older, non-standard packages. I suspect that's the type of package you're referring to here.

ProGet's Composer feeds work with "packages" (i.e. not the GitHub pointers), which account for nearly all of the modern packages on Packagist.org. For the small number of packages that don't follow the standard (mostly older, legacy packages), you'll need to download them, properly package them, and then upload them to follow the standard.

Unfortunately there's no technically feasible/sensible way to solve this problem - since it would require ProGet to serve as a GitHub pointer database, which just doesn't make sense.

Cheers,
Steve

Hi @Ashley ,

It should have been; I just double-checked and see a commit for "Add full PEP 700 conformance for PyPI feeds" that appears to add the fields discussed here and some more.

Thanks,
Steve

@Nils-Nilsson FYI quick update, but it does look like LDAP searching is having issues in the UI.. so perhaps it's some kind of UI//library upgrade regression. Anyway we'll update once we have a fix.

Hi @Nils-Nilsson,

Aside from library/dependency upgrades, I'm not aware of any changes to AD/LDAP that would yield this change. So it's definitely possible a dependency update caused that, we'll keep our eyes out... but things did pass our ordinary LDAP/AD testing, so any other info would be really helpful ... like if it's easy to do a new testing instance so you can upgrade/downgrade to find if that's definitely the issue.

As for permissions/tasks, there were definitely no changes here. The tasks/permissions data were not touched as part of the upgrade, nor was the data/schema changed; the renaming of "Feed Groups" to "Feed & Project Groups" is entirely cosmetic.

Behind the scenes (in the database), it's still FeedGroup_Id. The only schema changes were related to creating new tables for vulnerability data and adding a column to projects table to support grouping. There were no data updates.

Thanks,
Steve

Hi @ybaskar-temp_3339 ,

Thanks for confirming that.

I think it'd be worth exploring how to change those migration/environment constraints. While it's unfortunate to "lose history" and need to navigate to a separate instance, there's a moderate amount of risk in things simply not working the way it did before.... we're talking a decade of evolution, including platform, architectural, and even usage changes.

That said, for an in-place upgrade, here's the approach that I would use.

1. Use the Legacy (Traditional) installer to get to BuildMaster 6.1.
2. Verify things work as expected on BuildMaster 6.1, agents updated, etc
3. Upgrade to BuildMaster 6.2, again using the traditional installer
4. Verify things work as expected in same manner
5. Uninstall
6. Ensure all components (iis site, service, program files.... except database, artifacts) have been removed
7. Install BuildMaster 2025 as per normal, pointing to existing database

Note that there's a decent change that you'll run into database or binding errors after #7, so make sure to run buildmaster.service.exe run from the commandline to better see messages and errors if things don't start up.

Cheers,
Steve

@steviecoaster sure thing!

https://github.com/inedo/inedo-docs/tree/master/Content/proget/feeds/chocolatey#.md

There's a "hidden in plain sight" GitHub link under the heading of each article, mostly for us :)

Figured a PR/example would be easier than back-forth on forums. Thanks!

Hi @pg_user_8607 ,

Unfortunately, the only information we receive from the underlying library is LdapReferralException, so there's nothing else we can log. My understand is that the server is limiting the information and you ought to see more information from the LDAP/AD server's query logs (they are like HTTP access logs). That's the absolute best place to look.

In our experience, a referral typically means the domain name is incorrect (e.g. user@domain.com instead of user@domain.local), but it could be any of the things you mentioned to. Unfortunately, LDAP/AD configuration can be a pain in rare cases (which it sounds like you are), and there's just no way around that.

As for monitoring, here is what we recommend:

• periodic monitoring of the /health endpoint (every 5 minutes)
• (optional) HTTP access logs (retain for 7 days)

There's nothing required beyond that. Those "container logs" (i.e. proget console output) that you see are primarily intended for us (product engineers) to troubleshoot problems and there's not much value in trying to use/storing them.

For a tool like ProGet, trying to do extract/monitor detailed metrics is counter productive and leads to information overload. Many "errors" are not problems are a total waste of everyone's time to troubleshoot.

For auditing, ProGet maintains internal audit logs (you can query them from the database if you really want to "export" them), or you can use webhooks if you want to publish events. But again, we don't think that's productive; they just become a "secondary log" that no one looks because it's harder to query than ProGet database.

For authentication-related information, a combination of HTTP log monitoring (403 errors) and LDAP/AD server is the best thing to check.

Cheers,
Steve

Hi @ybaskar-temp_3339,

Upgrading from BuildMaster 5.7 to 2025 has never been supported; you'll need to first upgrade to BuildMaster 6.1, and then you can upgrade to BuildMaster 2025. Please see:
https://docs.inedo.com/docs/buildmaster-upgrading

It's certainly possible there was a bug/oversight in newer versions of the Inedo Hub that didn't "block" the upgrade, but it should never have been possible. Also I do believe we removed really old versions from the Inedo Hub, so you may need to download the offline installer to get to BuildMaster 6.1.

Another approach to consider is to just "start from scratch" with BuildMaster 2025, and maintain BuildMaster 5.7 as a separate instance... moving to read-only over time. If a lot of your applications are templatized anyway, you may find the built-in Build Scripts work with minimal configuration. And as long as you're on the latest version of Inedo Agent (on the servers), they should be able to both communicate fine. This approach is the lowest risk by far.

Cheers,
Steve

@steviecoaster sounds good, feel free to submit a PR with suggestions if you'd like anything to add. For us, we usually just point to connectors documentation, though the topic doesn't really come up with this feed

Hi @steviecoaster ,

We haven't had any users ask for help on this; it's pretty straight-forward in ProGet, as the UI guides users to do this and even pre-populates the default URLs. But if you think there's some nonobvious things or issues you've seen, then a technical note would be great (like we have on Connectors for Debian Feeds).

I should note... we did not create a dedicated HOWTO guide for this use case (unlike, for example, HOWTO: Proxy Packages from NuGet.org in Visual Studio or CLI), mostly because we don't generally recommend it. Instead, we recommend internalizing the packages.

What do you think about editing this section to clarify that?

https://docs.inedo.com/docs/proget/feeds/chocolatey#internalizing-chocolatey-packages

Thanks,
Steve

@dwilson_7533 that sounds about right! In modern versions of BuildMaster, it's possible to change Pipelines ("new" name for Workflows) a little easier, or even create builds without them.

Speaking of, I'd recommend giving it a shot, we've got some pretty cool Git integration these days :)

Hi @caterina,

Well, it's Microsoft, so their teams and guidance is all over the place... and that article is "AI-assisted" (labeled as such), so it's probably even more inconsistent than Microsoft's normal docs :)

But, I definitely agree on the convenience of WIA over tokens. In any case, we don't have a plan to discontinue it from our products - but it is a "risky" feature, both in terms of making ProGet less secure and regressions (from Windows updates, or client/ProGet upgrades). So that's why we encourage users to move away.

Anyway.... as for the issue, can you try adding ?bypassIntegrated=false to the URL? We are linking directly to the NPM API, which is bypassed. I'd like to see if that solves it.

Thanks,
Steve