Request for Creation of API for Package Auditing Before Dependency Restoration

[fabrice.mejean]

Hello @stevedennis ,

You raise a good point. Indeed, we should consider adding the 'feed' parameter, as the concept of obsolescence is tied to it. This begs the question of whether it should remain this way or if it should be managed more globally, similar to how vulnerabilities are handled. However, that would go beyond a simple API addition.

On the other hand, I am absolutely fine with making multiple calls to achieve my goals, as long as ProGet can handle the load.

Additionally, I would like to ask about the use of 'pgutil packages audit' and whether it indicates compliant or non-compliant. Does this status utilize all the rules (license, vulnerability, and others), and is there a plan to provide insights into the reasons behind the result? If it takes into account all the rules, then the 'other rules' are dependent on the feed, making the 'feed' parameter essential.

Thank you for your thoughts on this matter. Looking forward to your feedback!

Best regards,

Fabrice MEJEAN

[stevedennis]

Hi @fabrice-mejean,

This begs the question of whether [obsolescence] should remain this way or if it should be managed more globally, similar to how vulnerabilities are handled. However, that would go beyond a simple API addition.

I could see this making sense for OSS packages hosted at their official source (e.g. NuGet.org, npmjs, etc) -- but you're right, much more than an API change and we should limit scope for now :)

Additionally, I would like to ask about the use of 'pgutil packages audit' and whether it indicates compliant or non-compliant. Does this status utilize all the rules (license, vulnerability, and others), and is there a plan to provide insights into the reasons behind the result? If it takes into account all the rules, then the 'other rules' are dependent on the feed, making the 'feed' parameter essential.

Good point -- the feed parameter would be important if you wanted feed-scoped policies. Otherwise, only the global policy could apply... which is limiting.

As for "reasons behind the result", there is a short text available in the compliance system that is displayed in the UI and error messages for non-compliant results. So I think we could bring that into the pgutil output like this:

$> pgutil packages metadata --feed=myNugetFeed --package=Junk.Package --version=1.2.3

Junk.Packge-1.2.3 (Deprecated, Unlisted)
 Compliance      : Warn (Deprecated; Unknown License)
 License         : RSGPL
 Vulnerabilities : None

So to summarize my understanding... here's what I'm thinking.

1. The pgutil packages metadata command will provide package metadata (i.e. from the manifest file), server metadata (listed, unlistd), vulnerabilities, and compliance -- similar to what the ProGet UI shows
2. It will work like the other packages commands, which require a feed parameter
3. We will not do the pgutil packages audit command afterall, since we don't have an easy way to work on a "set of packages" with regards to feeds

And of course, the new API call would return the data in a structured manner. We can share that as we get closer to implementation.

How's this sound?

Thanks,
Steve

[stevedennis]

In retrospect, I don't think pgutil would display the output like that exactly, but you get the idea. It'd basically be a bunch of label/text pairs -- basically not that different from what's in the ProGet UI

[pmsensi]

@stevedennis

Re: Get package license with ProGetClient

the pgutils packages metadata + API end point to be used in Inedo.Proget lib is what I need :)

Do you have a timeline for the implementation ?

Best Regards,
Pedro

[fabrice.mejean]

Hi @stevedennis,

I agree with your proposal; it should allow me to meet my current needs.

Regarding obsolescence management, unfortunately, the majority of open-source components do not manage obsolescence, let alone end-of-life dates. That is why we mark all packages older than 3 years as obsolete. This allows us to verify that the open-source project is still alive, and that in case of a vulnerability, we are still able to carry out a corrective version upgrade. I will certainly create another post about this soon, but it gives you something to think about.

I would appreciate a YouTrack reference to track this feature.

Thank you again.

Fabrice MÉJEAN

[atripp]

Hi @fabrice-mejean @pmsensi ,

We've got this spec'd out and on the roadmap now as PG-3126! It'll come through a maintenance release, along with pgutil security commands for configuring users, groups, and tasks.

The target is 2025.13, which is planned for October 24. I don't know if we'll hit that target, but that's what we're aiming for.

Please check out the specs on PG-3126; I think it captures what you're looking for, which is basically an expanded metadata object that includes compliance data, detected licenses, and vulnerabilities.

Thanks,
Alana

[pmsensi]

@atripp Looks good!

For my clarification, the property DependencyGroups, will be a list of all the (main) dependencies of the package ?

Best Regards,
Pedro

[atripp]

Hi @pmsensi,

Correct -- it'll be whatever data is on the "Dependencies" tab in ProGet, which is basically whatever is in the manifest file (.nuspec, etc).

Thanks,
Alana

[pmsensi]

Hello @atripp

Sorry to open an old topic.

While checking the content of the result of GetPackageMetadataAsync (PackageMetadata), I don't see any reference to the Dependencies.

The public APi has changed ? Or there is a new way to get the dependencies information ?

Best Regards,
Pedro

[atripp]

Hi @pmsensi,

I don't believe that was ever in the PackageMetadata object; that information is stored in ecosystem-specific formats inside of the manifest file (e.g. .nuspec) in the package and it's not something we can easily generalize. The UI shows a "simplified" version but it's not always correct or suitable for metadata/api results.

You'd have to parse those metadata files to get that information.

hope that helps,

Alana

[pmsensi]

Hello @atripp ,

oh ok, I thought that will be available, reading your last reply from last year.

Anyway, that information is not even stored in database ?, so means, I need to use the Nuget api to get that information ?

Best regards,
Pedro