Hi @apxltd,
Wow, that was fast... I noticed that this was apparently already implemented in 2023.5 (PG-2359). Just wanted to let you know that I tested this with 2023.6 and it works like a charm
Hi @apxltd,
Wow, that was fast... I noticed that this was apparently already implemented in 2023.5 (PG-2359). Just wanted to let you know that I tested this with 2023.6 and it works like a charm
@kichikawa_2913 Not sure if this is related to your problem, but we had some strange problems after upgrading to ProGet 6.0.8 / 6.0.9 as well, so maybe sharing the following info might help you:
We first encountered problems after upgrading to ProGet 6.0.8: Users could access feeds, but could not download any packages. After upgrading to version 6.0.9 this was resolved.
We then had a problem where access became very slow until users could not access any ressources on our ProGet server anymore (even those with "Anonymous" access rights!). We deactivated the "6.1 Preview Features" and restarted everything (including the IIS). This resolved the issue. We also updated the InedoCore extension, but I don' t think this was related to the problem.
I think the 6.1 features are still kind of buggy. We try to avoid activating them for now.
Hi again
Two more things I noted while looking into this: It seems that the current implementation (reading the dependencies
property) doesn't consider transitive dependencies at all. Take the package chalk
as a example. I have one package that has a requirement of the form "chalk": "^2.0.0"
, which is resolved to the following dependency
"chalk": {
"version": "2.4.2",
"resolved": "https://registry.npmjs.org/chalk/-/chalk-2.4.2.tgz",
"integrity": "sha512-Mti+f9lpJNcwF4tWV8/OrTTtF1gZi+f8FqlyAdouralcFWFQWF2+NgCHShjkCb+IFBLq9buZwE1xckQU4peSuQ==",
"requires": {
"ansi-styles": "^3.2.1",
"escape-string-regexp": "^1.0.5",
"supports-color": "^5.3.0"
}
However, I have some more packages with requirements like "chalk": "^4.1.0"
, which are resolved to sub-dependencies like this:
"chalk": {
"version": "4.1.2",
"resolved": "https://registry.npmjs.org/chalk/-/chalk-4.1.2.tgz",
"integrity": "sha512-oKnbhFyRIXpUuez8iBMmyEa4nbj4IOQyuhc/wy9kY7/WVPcwIO9VA668Pu8RkO7+0G76SLROeyw9CpQ061i4mA==",
"dev": true,
"requires": {
"ansi-styles": "^4.1.0",
"supports-color": "^7.1.0"
}
Those sub-dependencies are not scanned at all, if I'm reading the current implementation of NpmDependencyScanner
correctly. This results to only version 2.4.2 being reported into our SCA reports. Version 4.1.2 is not listed.
The packages
property of the new file versions 2 and 3 seems to list all packages in a flat list, putting information about relations between packages on the top level. This leads to entries like "node_modules/chalk"
(resolving the version 2.4.2), "node_modules/critters/node_modules/chalk"
(resolving the version 4.1.2), "node_modules/eslint/node_modules/chalk"
(4.1.2 as well), ...
So if we extract the correct name of the package and add to corresponding version to it, our SCA reports should now (correctly?) report both versions of the chalk
package.
The second thing I noticed here: we will get multiple entries of the same package/version combination. This probably won't bother ProGet, as I guess those entries will be considered identical, but I guess it wouldn't hurt to put a Distinct()
in somewhere on the client side. Like, maybe changing the line
return new[] { new ScannedProject(projectName, ReadDependencies(doc)) };
to
return new[] { new ScannedProject(projectName, ReadDependencies(doc).Distinct()) };
or maybe changing to code of the ScannedProject
constructor from
this.Dependencies = dependencies.ToList();
to
this.Dependencies = dependencies.Distinct().ToList();
Cheers,
Sebastian
@shayde
I can't speak for @atripp and the folks at Inedo, but I can tell you that we have submitted PRs for pgscan via GitHub in the past and they have all been accepted. Caterina and I actually already discussed implementing this, because it doesn't look too complicated, but we probably won't get a chance to look into it closer for a few more days. So if you want to go ahead, I think this would be much appreciated
@rhessinger said in Proget: Documentation of 2024 Projects Preview feature:
With all that said, you can customize these build stages by navigating to Reporting & SCA -> Projects and then hover over the multi-button in the upper right corner and select "Build Stages".
Thanks @rhessinger, this is what I was looking for! Looking at settings page really helps understanding what they are doing.
I have one follow-up question: There is an option called Set status of promoted build to "Released" at the stages' settings page, but it does not seem to be set on any of the four predefined stages. What's the effect of that option?
Hi there,
I was wondering if there is some documentation of the 2024 preview feature "Projects" within the SCA reports available. I found this new entry in the documentation, but I doesn't really answer my questions (yet). At the moment, the article basically just says that stages exist, but doesn't really go into details.
Here are some of the things I was wondering about:
What's the actual effect of a version being in one of the predefined stages? Am I correct in assuming that only active builds and versions in the "Production" stage are being analyzed? What's the effect of a version being in "Test" or "Integration"? What's the number of versions/builds per stage? We usually have several versions in production (i.e. customers use them in production) at the same time, but it seems that promoting a build to "Production" will move older builds to "Archive" (allowing only 2 build in Production). Is there a way to prevent this or increase the number of builds per stage? The article also mentioned that it would be possible to define our own stages, but I couldn't find that option yet (using ProGet 2023.31).
Hi @rhessinger
Thanks, but I don't think we will need a pre-release. The remaining problem is basically a convenience issue (getting from the report to the package's info page with one click). Everything else seems to work as expected with the latest pgscan version.
@rhessinger @atripp
I don't think the remaining problem is a pgscan issue. The latest version of pgscan (v1.5.12 including commits eb61208, bc663c4 and 28e60ad) should handle the @
within scope names correctly. Here is a little sample from a generated SBOM file:
<component type="library">
<group>@angular</group>
<name>common</name>
<version>17.0.9</version>
<purl>pkg:npm/%40angular/common@17.0.9</purl>
</component>
<component type="library">
<group>@angular</group>
<name>compiler</name>
<version>17.0.9</version>
<purl>pkg:npm/%40angular/compiler@17.0.9</purl>
</component>
<component type="library">
<group>@angular</group>
<name>core</name>
<version>17.0.9</version>
<purl>pkg:npm/%40angular/core@17.0.9</purl>
</component>
The problem seems to be on the ProGet side. The %40
is correctly displayed as @
in the UI, and packages seem to be identified correctly as well.
The only remaining problem is that the /packages/from-purl
API endpoint seems to have problems with two @
characters. Btw, both of them are apparently encoded as %40
within the URL. Here is an example URL: http://proget-host/packages/from-purl?pUrl=pkg%3Anpm%2F%40angular%2Fcore%4017.0.9
. The endpoint seems to convert both %40
parts back to the @
character, which results in the error message "pkg:npm/@angular/core@17.0.9 is an invalid purl".
Without knowing too much about the code within ProGet it's hard to speculate how to solve this, but my guess is that either the endpoint needs to be able to handle two @
characters within a purl, or the purl itself needs to be encoded differently when passed as a parameter with another URL. When I change the example URL of my previous example in a way that the %
of the first %40
encoding is encoded as well (i.e. replacing %
with %25
, I get the following URL, which actually works: http://proget-host/packages/from-purl?pUrl=pkg%3Anpm%2F%2540angular%2Fcore%4017.0.9
(%40angular
has been replaced with %2540angular
).
Cheers,
Sebastian
Hi Alex,
thanks for the feedback and the sneak peak at Proget 2024. Just a little background on how this came up: A developer actually came to me the other day and asked "Hey, can I use packages with license XYZ?", and my initial reply was "Well, just check what our Proget server says about that license" (we also have a written policy on how to handle the most common open source licenses, but it's not as frequently updated as our Proget license rules), but then I realized "Oh wait, you probably can't see whether there is an explicit rule about that license, because you don't have access to the license rule page..."
I'm looking forward to those new features in Proget 2024, and I'm pretty sure they might change some of our workflows. If after testing Proget 2024 I still feel like there is a need for developers to see details about rules, even though they can't edit them, I will update this post.
Have a nice weekend!
Proget visually indicates whether a license is blocked/allowed (or more precisely: whether packages with that licenses can be downloaded) and also whether this is due to an explicit rule or just by default. However, this is only done at the "License Types & Rules" view (Reporting & Software Composition Analysis (SCA) > License Usage > License Types & Rules):
The "License Usage" view just states whether used licenses are allowed or blocked, but it does not indicate whether a license is blocked by an explicit rule or by default:
Same goes for SCA reports or for package information pages: they indicate very prominently whether that packages can be downloaded based on its license, but doesn't reveal whether this is due to an explicit rule or just by default.
The feature request is for Proget to indicate whether there is an explicit rule for a license in all of the views mentioned above.
Why is this needed?
Developers regularly have to add new packages during development of new products, which means that every now and then they will stumble upon a license that hasn't been used so far. Proget has exactly two ways of handling "new" licenses: block or allow.
If we set Proget to block all licenses that have not been explicitly allowed, a developer who is faced with a blocked download currently has no way of knowing whether this is because someone has already made the decision that that specific license cannot be used, or if it's just because nobody has made a decision about that license so far.
If the information whether or not an explicit rule exists for that license was indicated on a view that developers have access to (regular developers do not have access to the aforementioned "License Types & Rules" view), they could go to to whoever is in charge of making that decision and ask them to please review license XYZ and set up an explicit rule for it in Proget.
Hi everyone,
just created a PR to improve support for both the old and the new file format.
File format 1 & 2 ('dependencies'): Added code to also read sub-dependencies.
File Format 2 & 3 ('packages'): Added code to deal with the new way of handling package alias. Apparently, there is an optional 'name' property. Couldn't find anything about that in the format's specification, so this is purely based on observation of our test projects.
The code looks good to me
Hi again
Two more things I noted while looking into this: It seems that the current implementation (reading the dependencies
property) doesn't consider transitive dependencies at all. Take the package chalk
as a example. I have one package that has a requirement of the form "chalk": "^2.0.0"
, which is resolved to the following dependency
"chalk": {
"version": "2.4.2",
"resolved": "https://registry.npmjs.org/chalk/-/chalk-2.4.2.tgz",
"integrity": "sha512-Mti+f9lpJNcwF4tWV8/OrTTtF1gZi+f8FqlyAdouralcFWFQWF2+NgCHShjkCb+IFBLq9buZwE1xckQU4peSuQ==",
"requires": {
"ansi-styles": "^3.2.1",
"escape-string-regexp": "^1.0.5",
"supports-color": "^5.3.0"
}
However, I have some more packages with requirements like "chalk": "^4.1.0"
, which are resolved to sub-dependencies like this:
"chalk": {
"version": "4.1.2",
"resolved": "https://registry.npmjs.org/chalk/-/chalk-4.1.2.tgz",
"integrity": "sha512-oKnbhFyRIXpUuez8iBMmyEa4nbj4IOQyuhc/wy9kY7/WVPcwIO9VA668Pu8RkO7+0G76SLROeyw9CpQ061i4mA==",
"dev": true,
"requires": {
"ansi-styles": "^4.1.0",
"supports-color": "^7.1.0"
}
Those sub-dependencies are not scanned at all, if I'm reading the current implementation of NpmDependencyScanner
correctly. This results to only version 2.4.2 being reported into our SCA reports. Version 4.1.2 is not listed.
The packages
property of the new file versions 2 and 3 seems to list all packages in a flat list, putting information about relations between packages on the top level. This leads to entries like "node_modules/chalk"
(resolving the version 2.4.2), "node_modules/critters/node_modules/chalk"
(resolving the version 4.1.2), "node_modules/eslint/node_modules/chalk"
(4.1.2 as well), ...
So if we extract the correct name of the package and add to corresponding version to it, our SCA reports should now (correctly?) report both versions of the chalk
package.
The second thing I noticed here: we will get multiple entries of the same package/version combination. This probably won't bother ProGet, as I guess those entries will be considered identical, but I guess it wouldn't hurt to put a Distinct()
in somewhere on the client side. Like, maybe changing the line
return new[] { new ScannedProject(projectName, ReadDependencies(doc)) };
to
return new[] { new ScannedProject(projectName, ReadDependencies(doc).Distinct()) };
or maybe changing to code of the ScannedProject
constructor from
this.Dependencies = dependencies.ToList();
to
this.Dependencies = dependencies.Distinct().ToList();
Cheers,
Sebastian
@shayde
I added two comments to your PR. I think transitive dependencies need to be handled differently, otherwise you'll get concatenate package names like @angular-devkit/build-angular/jsonc-parser 3.2.0
, @angular-devkit/core/jsonc-parser 3.2.0
, @angular-devkit/schematics/jsonc-parser 3.2.0
, [...] when it should probably really just be jsonc-parser 3.2.0
.