ProGet SCA 2024 Preview Feedback - Package detection still hit or miss

jw

I'm curious if you looked at any of the audit endpoints/commands in pgutil yet? That's kind of the direction we're thinking it will make sense to go - basically pgutil packages audit --package=myGroup/myPackage --version=1.2.3

I haven't yet, but thanks for the pointer, I will have a look.

I don't know what the HTTP Endpoint is offhand, but that does make sense to add something to PackageInfo, since we have it in the database already pretty easily. We could display a complianceStatus (Compliant, Warn, Noncompliant, Inconclusive, Error) and a complianceDetail string - that's what we have in the database. I think properties are easier to work with than objects... what do you think?

From what I understood from the docs, PackageInfo is part of ReleaseInfo (now probably called BuildInfo after the release=>build rename?) and used in the /api/sca/builds?project= endpoint.

Adding complianceStatus (Compliant, Warn, Noncompliant, Inconclusive, Error) to PackageInfo already makes a lot of sense, though it needs be very precisely defined what each status means, especially "Inconclusive" and "Error".

Right now, I'm specifically caring about the state when a package could not be fully scanned because it was not found in cache, not sure if "Inconclusive" means exactly that or if there are other triggers for that state.

As for the detail string, you are probably referring to what is currently shown in the tooltip when hovering warn on the /projects2/packages?buildId=5 page?

Generally, I believe the type of compliance violation is an important piece of information and should be stored as atomic values. At the moment it appears to be a concatenated string of violations?

On the API I would like to see something like an array of enum strings (like my code example above) or a dedicated object within PackageInfo, something like ComplianceViolationInfo with boolean properties for each violation type would also be fine.

In the UI it could look something like this:

This would be much more user-friendly than having to hover each warning and read a long string, or alternatively having to sift through all the generated issues.

As for Download Package behavior -- we do intend to get the Common Packages API to work with connectors. That involves a lot of refactoring that just didn't make it in ProGet 2024 (only PyPi and Apk were refactored).

Glad to hear that this is already on the roadmap

Cheers

stevedennis

Hi @jw ,

We added a compliance property via PG-2658 in the next maintenance release.

It basically shows what's in the database (which is also what the page in the UI does):

writer.WritePropertyName("compliance");
writer.WriteStartObject();
writer.WriteString("result", Domains.PackageAnalysisResults.GetName(package.Result_Code));
if (package.Detail_Text is not null)
    writer.WriteString("detail", package.Detail_Text);
if (package.Analysis_Date.HasValue)
    writer.WriteString("date", package.Analysis_Date.Value);
writer.WriteEndObject();

I think you can rely on result=Inconclusive meaning the package isn't in ProGet. That's all we use the status for now, but in the future it might be used for something else. A result=Error means that our code crashed and you shouldn't ever see that.

We'll definitely considering doing something other than a single result string down the line, but for now this was the easiest :)

Thanks,
Steve

jw

Thanks for adding the properties to 2024.3.

During testing I encountered unexpected behavior regarding the values:

To trigger the "Inconclusive" state I used the "Delete Cached Package" menu option on the package site and then re-ran the analysis on a build.

My expectation was that the package would be reported as inconclusive, since it is no longer available in cache and thus can't be fully scanned.

Instead I get the following results:

        {
            "purl": "pkg:nuget/AutoMapper@10.1.1",
            "licenses": [
                "MIT"
            ],
            "compliance": {
                "result": "Warn",
                "detail": " because of Package Status is Unlisted, Package Status is Deprecated, No license detected.",
                "date": "2024-05-13T09:55:04.467Z"
            },
            "vulnerabilities": []
        },

The warnings in the detail string do not make sense, since the package is neither unlisted, nor deprecated and the license is also correctly detected as MIT.

stevedennis

Hi @jw ,

I haven't investigated this yet, but I assume that the results are the same in the UI? That's all just pulling data from the database, so I would presume so.

Could you find the relavent parts of the analysis logs? That helps us debug much easier.

Thanks,
Steve

jw

I haven't investigated this yet, but I assume that the results are the same in the UI? That's all just pulling data from the database, so I would presume so.

Yes, the UI shows the same.

Could you find the relavent parts of the analysis logs? That helps us debug much easier.

It was actually not that easy to find the AutoMapper package in the logs, since the name does not appear anywhere. I made a custom SBOM with just the AutoMapper package and this is what the log looks like:

Analyzing compliance...
Beginning license rule analysis...
Default rules: undectableLicense=Warn, unspecifiedLicense=Compliant
The package is not cached or local to any feed; without package metadata, license detection is limited.
No licenses detected on package; applying undectableLicense rule (Warn)
License rule analysis complete.
The package is not cached or local to any feed; cannot determine if Deprecated.
No policies define a deprecation rule, so default Warn will be used.
The package is not cached or local to any feed; cannot determine if Unlisted.
No policies define an unlisted rule, so default Warn will be used.
Package is Warn because of Package Status is Unlisted, Package Status is Deprecated, No license detected.

atripp

@jw thanks for the detailed research, this definitely is wrong. We should log the package (I think that was there at one point), but also this should be inconclusive.

We'll get this fixed via PG-2676 in an upcoming maintenance release, hopefully this Friday's or the following :)

stevedennis

Hi @jw ,

FYI - We just wanted to clarify what "inconclusive" meant - this was a "late" change on our end, and we realized the documentation wasn't very clear. Here is how we describe it now:

Inconclusive Analysis

A build package (and thus a build as a whole) can be have an "inconclusive" compliance status. This will occur when two conditions are met:

A rule would cause the build package to be Noncompliant, such as Undetected Licenses = Noncompliant or Deprecated = Noncompliant

The package is not cached or otherwise pulled to ProGet, which means ProGet doesn't have enough information about the package to perform an analysis because the package is

You can resolve this by pulling or downloading (i.e. caching) the package in a feed in ProGet, or not defining rules that require server-based metadata. For example, vulnerability-based rules can be checked without the package, but deprecation or license detection cannot.

The analysis message is incorrect however, it should be "Package is Warn because of Package Status is unknown, No license detected."

Thanks,
Steve

jw

Inconclusive Analysis

A build package (and thus a build as a whole) can be have an "inconclusive" compliance status. This will occur when two conditions are met:

A rule would cause the build package to be Noncompliant, such as Undetected Licenses = Noncompliant or Deprecated = Noncompliant

The package is not cached or otherwise pulled to ProGet, which means ProGet doesn't have enough information about the package to perform an analysis because the package is

Not sure I fully understand condition 1. Even if not such rule is configured, compliance could not be established because licensing information is not available, due to the package being not cached. Am I missing something?

The analysis message is incorrect however, it should be "Package is Warn because of Package Status is unknown, No license detected."

Should the string really say "Warn" when the package state is Inconclusive? Aren't Warn (Policies failing, Vulnerabilities detected, etc.) and Inconclusive (Package not found) two mutually exclusive states..?

atripp

Hi @jw ,

You are technically correct. In retrospect, it's not a perfect design.

However, this behavior is based on most user perception/desire. "Noncompliant / Inconclusive / Error" are all considered "Red" conditions (i.e. bad/blocked, etc.), and to the user they are basically the same thing. "Warn" is a "Yellow" condition (i.e. caution).

A package that would only become "Yellow" if a rule is violated shouldn't turn "Red" if that rule can't be evaluated. So that's why it's Warn.

This addresses one of the major complaints about the "missing packages" problem in ProGet 2023, and the "Red" packages in the ProGet 2024 preview feature.

Thanks,
Alana

sebastian...

Hi everyone,

I stumbled upon this thread because we have been faced with a similar problem in the past (missing packages due to projects not yet configured to use ProGet as a proxy, especially for npm packages), and I was wondering whether this whole process could be simplified with a specific webhook notifier for missing packages (or at least for inconclusive packages). There already is an event type called "Noncompliant Package Discovered", so it would seem rather straight forward to add another event type called "Inconclusive Package Discovered" or "Package Not Found During Analysis".

Such a webhook could be used for two things:

Trigger a download of the missing package from ProGet, resulting in the package being cached there (basically the solution that has been discussed in this thread, except that it's now done on demand) and the SCA analysis being able to handle the package correctly the next time it runs.
Notify project owners that their project apparently is not configured to use ProGet as a proxy for all package types or that there is some other kind of problem.

Or maybe this is already possible with one of the existing webhook events and I'm just reading the documentation wrong?

apxltd

Hi @sebastian,

Thanks for sharing your thoughts on this! Few things to point out...

[1] The "Missing Package Problem" is not as bad in ProGet 2024, mostly because it will only apply when there's a license rule. In ProGet 2023, a "missing package" would happen even for vulnerabilities.

[2] We're working on a new feature/module (tentatively called "Remote Metadata") that is intended to routinely cache/update meatdata from public repos like nuget.org, npmjs.org, etc. This feature enables two usecases:

Routinely update "server-side metadata" like Deprecated, Unlisted on cached packages
Fetch metadata for packages not in ProGet during build analysis

It works somewhat independently, and basically it'll just show up on the Admin tabs as like "Remote Metadata" and you can configure providers, urls, etc.

I hope to have a prototype in a couple weeks and will post some details on a new forum posts. As an FYI this is something we will limit in the Free/Basic editions and have full-featured in the ProGet Enterprise product.

[3] "Package In ProGet" could be a policy rule to add after RMetadata feature, though it's probably not a big deal if ProGet can detect licenses thanks to RMetadata

Best,

Alex