Hi @apxltd,

Wow, that was fast... I noticed that this was apparently already implemented in 2023.5 (PG-2359). Just wanted to let you know that I tested this with 2023.6 and it works like a charm

@kichikawa_2913 Not sure if this is related to your problem, but we had some strange problems after upgrading to ProGet 6.0.8 / 6.0.9 as well, so maybe sharing the following info might help you:

We first encountered problems after upgrading to ProGet 6.0.8: Users could access feeds, but could not download any packages. After upgrading to version 6.0.9 this was resolved.

We then had a problem where access became very slow until users could not access any ressources on our ProGet server anymore (even those with "Anonymous" access rights!). We deactivated the "6.1 Preview Features" and restarted everything (including the IIS). This resolved the issue. We also updated the InedoCore extension, but I don' t think this was related to the problem.

I think the 6.1 features are still kind of buggy. We try to avoid activating them for now.

Hi again

Two more things I noted while looking into this: It seems that the current implementation (reading the dependencies property) doesn't consider transitive dependencies at all. Take the package chalk as a example. I have one package that has a requirement of the form "chalk": "^2.0.0", which is resolved to the following dependency

    "chalk": {
      "version": "2.4.2",
      "resolved": "https://registry.npmjs.org/chalk/-/chalk-2.4.2.tgz",
      "integrity": "sha512-Mti+f9lpJNcwF4tWV8/OrTTtF1gZi+f8FqlyAdouralcFWFQWF2+NgCHShjkCb+IFBLq9buZwE1xckQU4peSuQ==",
      "requires": {
        "ansi-styles": "^3.2.1",
        "escape-string-regexp": "^1.0.5",
        "supports-color": "^5.3.0"
      }

However, I have some more packages with requirements like "chalk": "^4.1.0", which are resolved to sub-dependencies like this:

        "chalk": {
          "version": "4.1.2",
          "resolved": "https://registry.npmjs.org/chalk/-/chalk-4.1.2.tgz",
          "integrity": "sha512-oKnbhFyRIXpUuez8iBMmyEa4nbj4IOQyuhc/wy9kY7/WVPcwIO9VA668Pu8RkO7+0G76SLROeyw9CpQ061i4mA==",
          "dev": true,
          "requires": {
            "ansi-styles": "^4.1.0",
            "supports-color": "^7.1.0"
          }

Those sub-dependencies are not scanned at all, if I'm reading the current implementation of NpmDependencyScanner correctly. This results to only version 2.4.2 being reported into our SCA reports. Version 4.1.2 is not listed.

The packagesproperty of the new file versions 2 and 3 seems to list all packages in a flat list, putting information about relations between packages on the top level. This leads to entries like "node_modules/chalk" (resolving the version 2.4.2), "node_modules/critters/node_modules/chalk" (resolving the version 4.1.2), "node_modules/eslint/node_modules/chalk" (4.1.2 as well), ...

So if we extract the correct name of the package and add to corresponding version to it, our SCA reports should now (correctly?) report both versions of the chalk package.

The second thing I noticed here: we will get multiple entries of the same package/version combination. This probably won't bother ProGet, as I guess those entries will be considered identical, but I guess it wouldn't hurt to put a Distinct()in somewhere on the client side. Like, maybe changing the line

            return new[] { new ScannedProject(projectName, ReadDependencies(doc)) };

to

            return new[] { new ScannedProject(projectName, ReadDependencies(doc).Distinct()) };

or maybe changing to code of the ScannedProject constructor from

            this.Dependencies = dependencies.ToList();

to

            this.Dependencies = dependencies.Distinct().ToList();

Cheers,
Sebastian

@shayde
I can't speak for @atripp and the folks at Inedo, but I can tell you that we have submitted PRs for pgscan via GitHub in the past and they have all been accepted. Caterina and I actually already discussed implementing this, because it doesn't look too complicated, but we probably won't get a chance to look into it closer for a few more days. So if you want to go ahead, I think this would be much appreciated

Hi @apxltd

first of all, sorry for the badly cropped screenshot. In this specific scenario, I'm interested in the projects using libraries that are under GPL-2.0 or PolyForm-Noncommercial-1.0.0 license (but they were too far down on the page to fit into the screenshot). So what I'm really looking for is that one project that uses a PolyForm-license and those seven projects that use GPL-2.0 licenses (some of those are internal projects, which would be OK, but I want to make sure no other projects use packages with those licenses).

The /sca/compliance-report isn't really useful for us at the moment, because I can only see a handful of entries, none of which mention one of the two licenses above. I assume that this is due to the "max. 1,000 builds" limit in the package analyzer. That's why the "License Usages Overview" is so helpful (and would be even more helpful if it had a way to quickly navigate to the affected projects/builds/packages ).

@apxltd

I thing it looks pretty good, to be honest. It's simple enough and has all the information I would be looking for.

Only minor adjustment maybe for people who have a lot of projects/build affected by a certain license would be to make the list sortable by "Project" or "Package", so it would be easier to scroll through larger lists.

@atripp said in Problem with PGV-22381O7 (tree-kill1.2.2 incorrectly flagged vulnerable):

Anyway, ISL is managed by a different team, so I'll submit an internal request to review. It doesn't seem so urgent, just inconvenient/incorrect and easy to workaround in ProGet. But let me know if I misread that.

You are correct; it's not an urgent problem, because I can just choose a different assessment for that entry (we have a special assessment "Manually Unblocked" for cases like that). The other two entries are correct (i.e. they display the affected version as < 1.2.2 or <= 1.2.1).

As for "Withdrawn" vulnerabilities... we're open to ideas for what to change in ProGet 2025. There used to just be a handful, but there are a lot more now. Our original was to just delete them from ProGet, but instead we just showed the icon. Maybe we should delete them.

My guess is that PGV-22381O7 wasn't updated after it was withdrawn, so when version 1.2.2 of the package was released (and GHSA-mxq6-vrrr-ppmg updated "affected versions" to <= 1.2.1; again, I'm just guessing that that's what happened), the PGV entry didn't really reflect that.

It would expect one of the following two things to happen with withdrawn vulnerabilities:

1. The team at Inedo Security Labs should update withdrawn packages, especially when they flag "all" versions of a package as vulnerable.

2. There should be a special treatment for withdrawn vulnerabilities within ProGet. Maybe not deleting them (because I'm pretty sure there will be cases where I will be looking at a package and think "I swear this one had a vulnerability, but now I can't find it?" ), but maybe auto-assess a special status to it.

Hi there,

I'm not sure if this is a ProGet issue, a problem with the ProGet Vulnerabilities Database or something completely different, so I'm just going to post my observation.

The npm package tree-kill has been flagged as vulnerable in all versions - including the latest version 1.2.2 - due to PGV-22381O7:

There are several problems with this.

First of all, as you can see on the second screenshot, that vulnerability has been withdrawn, which raises the first question: How should ProGet handle withdrawn vulnerabilities?

However, the main problem here is that PGV-22381O7 claims that "all" versions of tree-kill are affected:

That statement is incorrect. In fact, tree-kill fixed the problem with version 1.2.2.

PGV-22381O7 seems to be based on https://github.com/advisories/GHSA-mxq6-vrrr-ppmg, which is a duplicate of https://github.com/advisories/GHSA-884p-74jh-xrg2. While the first one only claims that versions <= 1.2.1 are affected (but does not display any information on patched versions), the second one clearly states that version 1.2.2has been patched:

One more thought on this: Is it generally a good idea to flag "all version" of a package as vulnerable? Wouldn't that affect any future version of that package as well, regardless of whether it has been patched or not? Or is the idea that those entries will be updated as soon as there is a patched version of the package?

@apxltd

I thing it looks pretty good, to be honest. It's simple enough and has all the information I would be looking for.

Only minor adjustment maybe for people who have a lot of projects/build affected by a certain license would be to make the list sortable by "Project" or "Package", so it would be easier to scroll through larger lists.

Hi @apxltd

first of all, sorry for the badly cropped screenshot. In this specific scenario, I'm interested in the projects using libraries that are under GPL-2.0 or PolyForm-Noncommercial-1.0.0 license (but they were too far down on the page to fit into the screenshot). So what I'm really looking for is that one project that uses a PolyForm-license and those seven projects that use GPL-2.0 licenses (some of those are internal projects, which would be OK, but I want to make sure no other projects use packages with those licenses).

The /sca/compliance-report isn't really useful for us at the moment, because I can only see a handful of entries, none of which mention one of the two licenses above. I assume that this is due to the "max. 1,000 builds" limit in the package analyzer. That's why the "License Usages Overview" is so helpful (and would be even more helpful if it had a way to quickly navigate to the affected projects/builds/packages ).

@stevedennis said in License Usage Overview - Non-compliant Licenses in Use:

That said, the Licenses Overview page predates Policies, and I don't think the "License Usage Issues" makes a lot of sense anymore. The old model (block/allow) was much simpler with a basic Allow/Block rule. However, Policies are quite a bit more complicated.

We're very open to ideas on what to do in its place, or if you have any suggestions on what could be improved in general in the SCA UI. It's very easy for us to "see" what you're talking about, since we have the backup now :)

If you don't mind me hijacking this thread: The Licenses Overview page already shows the number of affected packages, builds and projects per license:

Wouldn't it be nice to be able to actually get a list of those packages, builds and projects? For example, each of those numbers could be a link and clicking on it would open a popup or another page with the complete list. Basically like the "Usage & Statistics" page for packages.

We currently have our own little tool which does exactly that (reading the data directly from the ProGet database), but it would be nice to have this build-in in ProGet.

@apxltd said in Linux Performance & SQL Server Issues:

Expected Problem: Too Many Requests

A single, underspec'd machine can only handle so much load, and I wanted to find out what those limits were, and what errors I would get in "new user evaluation" scenario.

On Linux, with my test scenario, this manifests with basically the same, "error 35". I believe it's socket exhaustion, but who knows? In most production cases, we see SQL/network timeout - this is because the database is usually filled with a ton of data, and is taking a bit longer to respond. In my test scenario, it wasn't.

Non-solution: Slow the REquests

When I added a client-slide throttle - or even spaced out issuing the requests by 10ms - there were virtually no errors. If only NuGet and npm clients behaved that way...

Solution : Concurrent Request Limit

Under Admin > HTTP/S & Certificate Settings > Web Server Configuration, you can set a value called "Concurrent Request Limit". That made all the difference after warming up.

25 Requests worked like a charm. No matter what I threw at the machine, it could handle it. The web browsing experience was terrible, which makes me think we may want to create a separate throttle for web vs API in a case like this.

250 Requests caused moderate errors. Performance was better while browsing the web, but I'd get an occasional 500 error.

I wish I could give better advice, but "playing with this value" seems to be the best way to solve these problems in production. For our evaluation machines, I think 25ish is fine.

This doesn't seem to be Linux-specific. We've run into performance issue with Angular projects (restoring several hundred npm packages all at once when building from scratch) in the past on our Windows server. The problem seemed to be the pool size of the SQL client.

Our solution was to set the Concurrent Request Limit to 100, which seems to be the default pool size for the MS SQL client in .NET.

Before setting the request limit, npm install was 4x slower than connecting to the original npm registry.

After setting the request limit, connecting to our ProGet server was even faster than connecting to the original npm registry.

This is on a virtual 2 Core, 8 GB RAM Windows Server 2016 hosting ProGet 2023 and the corresponding MS SQL Server 2016 Express Edition, if anyone is curious.

Hi everyone,

I stumbled upon this thread because we have been faced with a similar problem in the past (missing packages due to projects not yet configured to use ProGet as a proxy, especially for npm packages), and I was wondering whether this whole process could be simplified with a specific webhook notifier for missing packages (or at least for inconclusive packages). There already is an event type called "Noncompliant Package Discovered", so it would seem rather straight forward to add another event type called "Inconclusive Package Discovered" or "Package Not Found During Analysis".

Such a webhook could be used for two things:

• Trigger a download of the missing package from ProGet, resulting in the package being cached there (basically the solution that has been discussed in this thread, except that it's now done on demand) and the SCA analysis being able to handle the package correctly the next time it runs.
• Notify project owners that their project apparently is not configured to use ProGet as a proxy for all package types or that there is some other kind of problem.

Or maybe this is already possible with one of the existing webhook events and I'm just reading the documentation wrong?

@stevedennis
Any chance that this will get fixed in ProGet 2023? We can't upgrade to 2024 at the moment due to the "1000 active builds" limitation.

Hi @stevedennis

We updated to 2023.35, and while OR expressions now seem to get recognized, I think the interpretation of those combinations is now incorrect. In earlier versions of ProGet, when a package had an allowed and a blocked license (combined with "OR"), the package itself was not blocked, which was OK because at least one of the licenses you could choose from was compliant. Now those packages seem to get blocked, which seems incorrect.

Let's take a look at the npm package jszip 3.10.1 for example. Its package.json entry is "license": "(MIT OR GPL-3.0-or-later)". This somehow translates to "(MIT, GPL-3.0-or-later)" on the package's Metadata page:

The individual licenses (MIT and GPL-3) are correctly identified. What's incorrect, though, is that the package is being blocked due to it using the non-compliant GPL-3.0-or-later license:

While that license is indeed configured as non-compliant (and any package solely under that license would correctly be blocked), the MIT package is configured as compliant:

Thus, that package should not be blocked from being downloaded. However, not only is the package blocked, it also shows up as as issue in our SCA reports:

What's even more weird is that on the package's general page (the non version-specific under "URL/feeds/FEEDNAME/jszip/versions") it says that the MIT license is actually the one that is non-compliant, which is definitely incorrect.

Note that this problem is not specific to jszip or those specific licenses. Another example would be the npm package node-forge, which is dual licensed under BSD-3-Clause (compliant) and GPL-2.0 (non-compliant). The corresponding license expression is (BSD-3-Clause OR GPL-2.0), but the package is blocked, its usage shows up as an issue in our reports and BSD-3-Clause is incorrectly shown as non-compliant on the package's page:

Hi there,

detection of license expressions containing more than one license (e.g. (MIT OR Apache-2.0)) was introduced to ProGet last year and it seemed to work fine. However, I'm currently looking at a few npm packages with SPDX tags like (AFL-2.1 OR BSD-3-Clause), (Apache-2.0 OR MPL-1.1) or (BSD-3-Clause OR GPL-2.0) and none of them seem to be detected.

Here is a list of npm packages and their license expressions, none of which seem to get detected correctly:

Here is a screenshot for one of those packages within ProGet:

Strangely enough, for Nuget packages it seems to work fine:

Any ideas? We are currently on ProGet 2023.34. I'm pretty sure this worked in an earlier version. I mean, looking back at the other thread, I apparently tested this in ProGet 2023.1 and it worked fine. So something must have happened between 2023.1 and 2023.34.