ProGet SCA 2024 Preview Feedback - Package detection still hit or miss

apxltd

To make this work smoothly, a webhook for SCA events would really be immensely helpful. Is something like that already on the 2024 SCA roadmap?

We do have a webhook notifier for "non-compliant packages found in build" planned, so perhaps this would be on the list!

When a SBOM scan is uploaded, no issues are created initially even though the UI suggests that analysis was done already. One has to run analysis a second time with the issue checkbox set for issues to be populated.

I just published some preview documentation, but the concept/model is slightly changed here:

When builds in certain stages are analyzed, an "Issue" for each noncompliant or inconclusive package will be created. These are intended to allow a human review and override noncompliant packages.

Basically, the idea is that nearly every build will be created through a CI process and ignored until it needs to be later tested. And that happens later in the release pipeline, after the build is promoted to a testing stage.

Our new guidance will be run pgutil builds create (basically new name for pgscan inspect) at build time, eactly like it's done now. And the later, when you deploy to a testing environment or otherwise are ready for testing, run pgutil builds promote. At that point, the issues are created.

We were thinking to have "Unresolved Issues" present on the project overview page, and it'd be really messy if it's mostly just CI builds.

Hope taht helps explain the thought process.

jw

I managed to implement the workaround for the uncached packages.

Right now I am doing the following:

API call to $"api/sca/builds?project={projectName}&version={version}"
- Parse the build Id out of the ViewBuildUrl property
API call to native API $"api/json/Projects_GetBuildInfo?ProjectBuild_Id={projectBuildId}"
- Cross reference ProjectBuildPackagesFeeds with ProjectBuildPackagesExtended to find out which packages could not be mapped to feeds
Call to the download button URL $"nuget/{feedName}/package/{packageName}/{packageVersion}" for each package that could not be mapped to any feed
API call to ($"api/sca/analyze-build?project={projectName}&version={version}" to update the build

While it does work, I'm not fully happy with the implementation and would like to ask for some improvements to the regular API.

Compliance information in PackageInfo

Right now the PackageInfo object does not contain any information about compliance violations. Would it be possible to extend it with the warnings that are shown in the compliance column of the /projects2/packages page?

Ideally, it would be some sort of enum, that has atomic values for all the known violations and can be filtered easily. This would help to to avoid the call to the native API in step 2, which as far as I understand you don't recommend using anyways.

{
  "purl": "pkg:myGroup/myPackage@1.2.3",
  "vulnerabilities": [],
  "licenses": ["MIT", "Apache-2.0"],
  "compilanceWarnings": [
     "PackageNotFound",
     "NoLicenseDetected",
     "Deprecated"
  ]
}

What would also be nice to have is atomic values for the package name and version, so one doesn't have to parse it out of the purl.

Download Package API behavior
Right now the /api/packages/MyNugetFeed/download?purl=pkg:nuget/MyNugetPackage@1.0.0 API returns 404 when trying to download a package that is not cached yet.

As a workaround I am using the URL of the download button used in the UI, but I would prefer to use a proper API endpoint that has more chances to be stable in the future.

I think it would be good if the download API could be changed to also trigger package downloading and caching from connectors, so basically the same behavior as the endpoint behind the download button.

atripp

Hi @jw ,

Thanks for the update, that sounds like a decent work-around for the time being. It will likely be a while before we can develop something more generalized.

I'm curious if you looked at any of the audit endpoints/commands in pgutil yet? That's kind of the direction we're thinking it will make sense to go - basically pgutil packages audit --package=myGroup/myPackage --version=1.2.3

I don't know what the HTTP Endpoint is offhand, but that does make sense to add something to PackageInfo, since we have it in the database already pretty easily. We could display a complianceStatus (Compliant, Warn, Noncompliant, Inconclusive, Error) and a complianceDetail string - that's what we have in the database. I think properties are easier to work with than objects... what do you think?

As for Download Package behavior -- we do intend to get the Common Packages API to work with connectors. That involves a lot of refactoring that just didn't make it in ProGet 2024 (only PyPi and Apk were refactored).

Cheers,
Alana

jw

I'm curious if you looked at any of the audit endpoints/commands in pgutil yet? That's kind of the direction we're thinking it will make sense to go - basically pgutil packages audit --package=myGroup/myPackage --version=1.2.3

I haven't yet, but thanks for the pointer, I will have a look.

I don't know what the HTTP Endpoint is offhand, but that does make sense to add something to PackageInfo, since we have it in the database already pretty easily. We could display a complianceStatus (Compliant, Warn, Noncompliant, Inconclusive, Error) and a complianceDetail string - that's what we have in the database. I think properties are easier to work with than objects... what do you think?

From what I understood from the docs, PackageInfo is part of ReleaseInfo (now probably called BuildInfo after the release=>build rename?) and used in the /api/sca/builds?project= endpoint.

Adding complianceStatus (Compliant, Warn, Noncompliant, Inconclusive, Error) to PackageInfo already makes a lot of sense, though it needs be very precisely defined what each status means, especially "Inconclusive" and "Error".

Right now, I'm specifically caring about the state when a package could not be fully scanned because it was not found in cache, not sure if "Inconclusive" means exactly that or if there are other triggers for that state.

As for the detail string, you are probably referring to what is currently shown in the tooltip when hovering warn on the /projects2/packages?buildId=5 page?

Generally, I believe the type of compliance violation is an important piece of information and should be stored as atomic values. At the moment it appears to be a concatenated string of violations?

On the API I would like to see something like an array of enum strings (like my code example above) or a dedicated object within PackageInfo, something like ComplianceViolationInfo with boolean properties for each violation type would also be fine.

In the UI it could look something like this:

This would be much more user-friendly than having to hover each warning and read a long string, or alternatively having to sift through all the generated issues.

As for Download Package behavior -- we do intend to get the Common Packages API to work with connectors. That involves a lot of refactoring that just didn't make it in ProGet 2024 (only PyPi and Apk were refactored).

Glad to hear that this is already on the roadmap

Cheers

stevedennis

Hi @jw ,

We added a compliance property via PG-2658 in the next maintenance release.

It basically shows what's in the database (which is also what the page in the UI does):

writer.WritePropertyName("compliance");
writer.WriteStartObject();
writer.WriteString("result", Domains.PackageAnalysisResults.GetName(package.Result_Code));
if (package.Detail_Text is not null)
    writer.WriteString("detail", package.Detail_Text);
if (package.Analysis_Date.HasValue)
    writer.WriteString("date", package.Analysis_Date.Value);
writer.WriteEndObject();

I think you can rely on result=Inconclusive meaning the package isn't in ProGet. That's all we use the status for now, but in the future it might be used for something else. A result=Error means that our code crashed and you shouldn't ever see that.

We'll definitely considering doing something other than a single result string down the line, but for now this was the easiest :)

Thanks,
Steve

jw

Thanks for adding the properties to 2024.3.

During testing I encountered unexpected behavior regarding the values:

To trigger the "Inconclusive" state I used the "Delete Cached Package" menu option on the package site and then re-ran the analysis on a build.

My expectation was that the package would be reported as inconclusive, since it is no longer available in cache and thus can't be fully scanned.

Instead I get the following results:

        {
            "purl": "pkg:nuget/AutoMapper@10.1.1",
            "licenses": [
                "MIT"
            ],
            "compliance": {
                "result": "Warn",
                "detail": " because of Package Status is Unlisted, Package Status is Deprecated, No license detected.",
                "date": "2024-05-13T09:55:04.467Z"
            },
            "vulnerabilities": []
        },

The warnings in the detail string do not make sense, since the package is neither unlisted, nor deprecated and the license is also correctly detected as MIT.

stevedennis

Hi @jw ,

I haven't investigated this yet, but I assume that the results are the same in the UI? That's all just pulling data from the database, so I would presume so.

Could you find the relavent parts of the analysis logs? That helps us debug much easier.

Thanks,
Steve

jw

I haven't investigated this yet, but I assume that the results are the same in the UI? That's all just pulling data from the database, so I would presume so.

Yes, the UI shows the same.

Could you find the relavent parts of the analysis logs? That helps us debug much easier.

It was actually not that easy to find the AutoMapper package in the logs, since the name does not appear anywhere. I made a custom SBOM with just the AutoMapper package and this is what the log looks like:

Analyzing compliance...
Beginning license rule analysis...
Default rules: undectableLicense=Warn, unspecifiedLicense=Compliant
The package is not cached or local to any feed; without package metadata, license detection is limited.
No licenses detected on package; applying undectableLicense rule (Warn)
License rule analysis complete.
The package is not cached or local to any feed; cannot determine if Deprecated.
No policies define a deprecation rule, so default Warn will be used.
The package is not cached or local to any feed; cannot determine if Unlisted.
No policies define an unlisted rule, so default Warn will be used.
Package is Warn because of Package Status is Unlisted, Package Status is Deprecated, No license detected.

atripp

@jw thanks for the detailed research, this definitely is wrong. We should log the package (I think that was there at one point), but also this should be inconclusive.

We'll get this fixed via PG-2676 in an upcoming maintenance release, hopefully this Friday's or the following :)

stevedennis

Hi @jw ,

FYI - We just wanted to clarify what "inconclusive" meant - this was a "late" change on our end, and we realized the documentation wasn't very clear. Here is how we describe it now:

Inconclusive Analysis

A build package (and thus a build as a whole) can be have an "inconclusive" compliance status. This will occur when two conditions are met:

A rule would cause the build package to be Noncompliant, such as Undetected Licenses = Noncompliant or Deprecated = Noncompliant

The package is not cached or otherwise pulled to ProGet, which means ProGet doesn't have enough information about the package to perform an analysis because the package is

You can resolve this by pulling or downloading (i.e. caching) the package in a feed in ProGet, or not defining rules that require server-based metadata. For example, vulnerability-based rules can be checked without the package, but deprecation or license detection cannot.

The analysis message is incorrect however, it should be "Package is Warn because of Package Status is unknown, No license detected."

Thanks,
Steve

jw

Inconclusive Analysis

A build package (and thus a build as a whole) can be have an "inconclusive" compliance status. This will occur when two conditions are met:

A rule would cause the build package to be Noncompliant, such as Undetected Licenses = Noncompliant or Deprecated = Noncompliant

The package is not cached or otherwise pulled to ProGet, which means ProGet doesn't have enough information about the package to perform an analysis because the package is

Not sure I fully understand condition 1. Even if not such rule is configured, compliance could not be established because licensing information is not available, due to the package being not cached. Am I missing something?

The analysis message is incorrect however, it should be "Package is Warn because of Package Status is unknown, No license detected."

Should the string really say "Warn" when the package state is Inconclusive? Aren't Warn (Policies failing, Vulnerabilities detected, etc.) and Inconclusive (Package not found) two mutually exclusive states..?

atripp

Hi @jw ,

You are technically correct. In retrospect, it's not a perfect design.

However, this behavior is based on most user perception/desire. "Noncompliant / Inconclusive / Error" are all considered "Red" conditions (i.e. bad/blocked, etc.), and to the user they are basically the same thing. "Warn" is a "Yellow" condition (i.e. caution).

A package that would only become "Yellow" if a rule is violated shouldn't turn "Red" if that rule can't be evaluated. So that's why it's Warn.

This addresses one of the major complaints about the "missing packages" problem in ProGet 2023, and the "Red" packages in the ProGet 2024 preview feature.

Thanks,
Alana

sebastian...

Hi everyone,

I stumbled upon this thread because we have been faced with a similar problem in the past (missing packages due to projects not yet configured to use ProGet as a proxy, especially for npm packages), and I was wondering whether this whole process could be simplified with a specific webhook notifier for missing packages (or at least for inconclusive packages). There already is an event type called "Noncompliant Package Discovered", so it would seem rather straight forward to add another event type called "Inconclusive Package Discovered" or "Package Not Found During Analysis".

Such a webhook could be used for two things:

Trigger a download of the missing package from ProGet, resulting in the package being cached there (basically the solution that has been discussed in this thread, except that it's now done on demand) and the SCA analysis being able to handle the package correctly the next time it runs.
Notify project owners that their project apparently is not configured to use ProGet as a proxy for all package types or that there is some other kind of problem.

Or maybe this is already possible with one of the existing webhook events and I'm just reading the documentation wrong?

apxltd

Hi @sebastian,

Thanks for sharing your thoughts on this! Few things to point out...

[1] The "Missing Package Problem" is not as bad in ProGet 2024, mostly because it will only apply when there's a license rule. In ProGet 2023, a "missing package" would happen even for vulnerabilities.

[2] We're working on a new feature/module (tentatively called "Remote Metadata") that is intended to routinely cache/update meatdata from public repos like nuget.org, npmjs.org, etc. This feature enables two usecases:

Routinely update "server-side metadata" like Deprecated, Unlisted on cached packages
Fetch metadata for packages not in ProGet during build analysis

It works somewhat independently, and basically it'll just show up on the Admin tabs as like "Remote Metadata" and you can configure providers, urls, etc.

I hope to have a prototype in a couple weeks and will post some details on a new forum posts. As an FYI this is something we will limit in the Free/Basic editions and have full-featured in the ProGet Enterprise product.

[3] "Package In ProGet" could be a policy rule to add after RMetadata feature, though it's probably not a big deal if ProGet can detect licenses thanks to RMetadata

Best,

Alex