ProGet SCA - License URLs are modified when saved to DB

jw

When saving a license URL to the license database in ProGet, both the protocol scheme and the "www" subdomain gets dropped.

As a consequence it is impossible to get the original URL, e.g. when reading the license definition back out via API. It is impossible to tell if the scheme was http or https or if www. was present in the original URL or not.

While URLs might currently be considered deprecated in NuGet (actually just the field in the nuspec spec is deprecated, but there is no replacement yet), in context of SBOM it continues to be a valid way of specifying license information. CycloneDX Spec

Is there any way I can configure ProGet to not manipulate URLs when they are saved to the database?

Example:

/api/json/Licenses_GetAllLicenseData response:

{
    "License_Id": 635,
    "License_Url": "testing.com/EULA"
}

atripp

Hi @jw ,

This is not possible; ProGet only stores this portion of the URL and uses that URL fragment for license detection. This is important, because then users won't have to specify every variation of a license URL that packages will present.

Thanks,
Alana

jw

Hello @atripp

Is this something that could be added?

We could really use this functionality to deal with proprietary licenses that do not have OSS SPDX identifiers.

Currently we add a custom license to the ProGet license database, then map either via license URL or via PUrls and then we modify the incomplete SBOM to to add a proper license name and URL to the component.

The only issue right now is that we cannot get the original unmodified license URL back out of ProGet.

Maybe a field could be added to the General tab that optionally holds an unmodified license URL for a license?

{
    "External_Id": "Custom-DevExpress-EULA",
    "License_Id": 628,
    "Title_Text": "DevExpress EULA"
    "Official_License_Url": "https://www.devexpress.com/Support/EULAs"
},

atripp

Hi @jw ,

It's unlikely we will want to add/change this; what you're describing isn't a supported use case, and adding a "second URL" type field that would be empty on nearly every license would be confusing.

Our general recommendation for dealing with non-OSS licenses has been to create a code like DEVEXPRESS or ASPOSE, and treat them as all the others.

If there's more user demand for the particular use case we'll definitely reconsider. For now the licenses are confusing enough :)

Thanks,
Alana

jw

It's unlikely we will want to add/change this; what you're describing isn't a supported use case, and adding a "second URL" type field that would be empty on nearly every license would be confusing.

Makes sense.

Though, I don't quite understand why the URLs are modified in the first place. Ignoring protocols and subdomains during comparisons is something that could be done in code. A small hint in the UI that there is no need to add all URL permutations should do the trick. This would cover both uses cases and no information is lost.

Our general recommendation for dealing with non-OSS licenses has been to create a code like DEVEXPRESS or ASPOSE, and treat them as all the others.

Not sure what you mean by "code". The CylconeDX Spec says for licence Ids only valid SPDX license IDs must be used same goes for SPDX License Expressions, they list the valid values in the spec.

That is why for proprietary licenses we chose "Option 2" of specifying a name + url instead.

If there's more user demand for the particular use case we'll definitely reconsider. For now the licenses are confusing enough :)

Alright, let's see if there are more users with similar use cases. Right now it looks like we jumped on the SBOM/SCA bandwagon a bit too early. ;)
I'll try to find a workaround.

jw

Two related issues came up (Version 2024.9):

There is currently a mixture between URLs with www. and without prefix in the database. Saving a license entry causes existing URLs to be modified in the database. So even if one just wants to add another URL to the list, existing entries are modified.
Storing license URLs without modification and applying additional logic, like ignoring www. subdomains, during comparison still seems like the sensible solution here.
LGPL licenses (and possibly others) cannot be saved at all, due to an URL collision.

dean-houston

Hi @jw ,

[1] is somewhat expected behavior, as some older versions of ProGet allowed non-normalized URLs to be added; newer versions should not allow this. We do not plan to "clean-up" the data at this time, but if you're "brave" you could could like do it with some API/DB calls

[2] This is probably some quirk related to older data, but you can probably just cut the items to clipboard, save the license, the edit again, and it should work-around the quirk

FYI, the URLs that we have in our database for LGPL-3.0-only are: as follows

gnu.org/licenses/lgpl+gpl-3.0.txt
gnu.org/licenses/lgpl-3.0-standalone.html
opensource.org/licenses/LGPL-3.0

I'm guessing you have the www from an older version

-- Dean

jw

[1] is somewhat expected behavior, as some older versions of ProGet allowed non-normalized URLs to be added; newer versions should not allow this. We do not plan to "clean-up" the data at this time, but if you're "brave" you could could like do it with some API/DB calls

I'm actually hoping that the normalizing the URLs design decision would be revisited at some point. Modifying the URLs has a number of unwanted and unneeded side effects as I described further above.

[2] This is probably some quirk related to older data, but you can probably just cut the items to clipboard, save the license, the edit again, and it should work-around the quirk

I did some more digging:

A number of SPDX identifiers got deprecated and replaced, see bottom of the page here https://spdx.org/licenses/

New ProGet installations only ship the new identifiers, this means there is no conflict, but mapping packages to those deprecated license identifiers, like LGPL-3.0 will probably not work, unless there already is special code for it..?

Existing ProGet installations still have deprecated license identifiers like LGPL-3.0 in their databases and the URLs of those are now in conflict with those of the new identifiers like LGPL-3.0-only.

In my case I could solve it by deleting the deprecated licenses and adding their SPDX identifiers to the new licenses.

That said, I could only delete the deprecated licenses after no packages were referencing them anymore. In my case I was lucky since only a few packages were affected, other customers might have a harder time with that.

Not fully sure that is the proper solution though, so feedback is welcomed.

Related: https://docs.debricked.com/product/license-risk-management/proxying-non-standard-license-identifiers

dean-houston

Hi @jw,

There's no problem deleting the deprecated licenses and adding their SPDX identifiers to the new licenses. When you delete a license, it will remove the association with packages.

However, the compliance analysis scheduled job will reassociate them. This runs nightly, or you can manually run it. Also, if you visit the package page or download the package, it should associate with the new license.

We'll definitely reconsider the design/approach if there's more demand, but we need to really make sure there's a value to the user -- there's a relatively high cost to change things and then there's a chance of regression/bugs, which is really frustrating to users.

-- Dean

jw

There's no problem deleting the deprecated licenses and adding their SPDX identifiers to the new licenses. When you delete a license, it will remove the association with packages.

Trying that results in the following exception:

(500) Server Error
547`16`0`Licenses_DeleteLicense`12`The DELETE statement conflicted with the REFERENCE constraint "FK__LicensePackageNameIds__Licenses". The conflict occurred in database "ProGet", table "dbo.LicensePackageNameIds", column 'License_Id'.

atripp

Hi @jw ,

This will be fixed via PG-2739 in the next maintenance release (scheduled for Friday).

If you want to fix right away, you can download the .sql file that I attached to the linked issue above and run it against the database. Then the delete will work. Upgrading on top of the patch is fine, but if you downgrade then the patch code is overwrriten.

Cheers,
Alana

jw

Thanks for the quick fix.

We didn't have that many packages assigned to the licenses in question, so the workaround of disconnecting all packages from a license and then deleting that license did the trick.