Package license definition

[sebastian...]

@pmsensi said in Package license definition:

1222 packages are a lot! For now, we do not have many packages, but we will have them for sure :)

Yeah, we have been using ProGet for a while now... :-)

I'm pretty sure some are false positives, though. It seems that when a package has been downloaded, but there is no usage of it in any product (happens for infrastructure packages like angular/core, angular/cli or some Visual Studio extensions), there is no license entry in the database even though the package has a perfectly fine SPDX tag.

What's driving me nuts at the moment is that Microsoft seems to be using embedded license info files for quite a number of their packages. Assigning licenses to those is going to take a while...

[apxltd]

@sebastian said in Package license definition:

What's driving me nuts at the moment is that Microsoft seems to be using embedded license info files for quite a number of their packages. Assigning licenses to those is going to take a while...

Oh really? They even have their own accepted SPDX code... whyyy Microsoft

In the past, we thought of adding a kind of wildcard URL for licenses, like a "package://Microsoft.*" => "MSPL" would basically associate all packages with that prefix that don't otherwise have a SPDX code, or an explicit license.

Wonder if that would help here?

[pmsensi]

@apxltd

I don't know if using the package name will be enough... There are a lot of packages from Microsoft, that don't start with Microsoft.

Would be nice to be able to filter the licenses based on other metadata, for example, owners.

At the moment, in our company, we use TrustedSigners to allow/block the installation of some packages from external sources, as you can see below. So, maybe being able to assign licenses by owner will be a big win for us here, I don't know @sebastian thoughts about this approach.

  <trustedSigners>
    <repository name="nuget.org" serviceIndex="https://api.nuget.org/v3/index.json">
        <certificate fingerprint="0e5f38f57dc1bcc806d8494f4f90fbcedd988b46760709cbeec6f4219aa6157d" hashAlgorithm="SHA256" allowUntrustedRoot="false" />
        <certificate fingerprint="5a2901d6ada3d18260b9c6dfe2133c95d74b9eef6ae0e5dc334c8454d1477df4" hashAlgorithm="SHA256" allowUntrustedRoot="false" />
        <owners>Microsoft;dotnetfoundation;aspnet;Microsoft Corporation;confluent</owners>
      </repository>
  </trustedSigners>

[sebastian...]

@apxltd Using URLs for packages would be a nice feature, especially as package URLs contain version numbers. Getting rid of those might already be helpful, because some package come in a rather large number of versions. Of course, even applying a wildcard to just the version of a package might lead to wrong results, because in theory a package could change it's licese from one version to the next, but that is probably not a very realistic scenario.

However, things can become messy when different wildcard URLs could be applied to the same package. Unfortunately, not all Microsoft packages use MSPL. Some use MIT, some use the proprietary licenses... It's a real mess. So it probably wouldn't be as easy as making just one rule for microsoft.*. But still, using wildcards could make things a bit easier.

@pmsensi We haven't used TrustedInstaller yet. The approach is interesting, and yes: it would probably make sense to have a central service like a ProGet server check package owners, but it would be a completely different approach to a very different problem. I don't think applying licenses to package owners makes a lot of sense, because - as written above - package owners like Microsoft can apply different licenses to different packages.

That being said, having a new entity "package owner" or "publisher" or something like that and being able to filter for that entity could be a cool new feature. This could also be used in the SBOM reporting feature (like: 40 packages come from Microsoft, 20 from vendor A, 7 from vendor B).

[apxltd]

@sebastian @pmsensi

Thanks for added insights! This doesn't seem as simple as I had hoped...

The current solution in ProGet now (i.e. packageid:// and package:// urls that can be associated with license codes) feels hacky, but seems we don't have many options.

Well, one thing that might work.... submitting a pull request to those packages to add the MIT license code to their project file? It's probably just an oversight on their part...

[apxltd]

@sebastian said in Package license definition:

That being said, having a new entity "package owner" or "publisher" or something like that and being able to filter for that entity could be a cool new feature. This could also be used in the SBOM reporting feature (like: 40 packages come from Microsoft, 20 from vendor A, 7 from vendor B).

We considered doing "something" with this metadata ages ago, but found that a lot of packages (including npm, etc.) have multiple owners/authors. On the top page of nuget.org, just 3/20 seem to have a single author

In general, the human-driven "package approval workflow" seems to be the best bet. Maybe p[ainful at first, but "not too bad" in the long-run

[sebastian...]

@apxltd said in Package license definition:

In the past, we thought of adding a kind of wildcard URL for licenses, like a "package://Microsoft.*" => "MSPL" would basically associate all packages with that prefix that don't otherwise have a SPDX code, or an explicit license.

Wonder if that would help here?

A colleague of mine actually had an interesting idea today: How about calculating hash values (like SHA-1) for embedded license files and assigning licenses to those hash values? That way one would only have to assign licenses to each license text once (if the license text is identical across different packages or package versions).

The workflow would be similar to assigning licenses to actual packages / versions, but instead of adding an pseudo URL like "packages://SomeVendor.SomePackage/1.2.0" we could do something like "hash://0xA1B2C3", where 0xA1B2C3 would be the hash value of the content of the license file. All other packages / versions with the exact same license text would automatically be mapped to the same license.

Of course, some license texts include the name of the product or a copyright note, so we would still get multiple entries for the same license. But it should be significantly less than adding one entry for every package / version.

What do you guys think about this idea?

[apxltd]

Hi @sebastian

Thanks for the idea -- yes, I think it's an interesting approach!

We explored it a while ago, and this was where we ended up...

1. It's even more confusing to use than packageid://, so we'd need to find a better UI solution

2. We'd want to store the full license text as well, so it'd be easy to confirm the contents

3. This is all a nontrivial engineering effort

4. We're not sure how many packages this would impact and how much value / time savings it would represent

5. None of this would even work for remote packages, which is by far what most users find confusing and have issues with

6. It would probably require less engineering effort to scan/query all packages on NuGet and make a "database" of package licenses using a little human intelligence

7. It would require even less effort to just ask package authors to specify license codes, and then eventually the problem will go away on its own probably

And then we gave up because there were more priority things to address ;)

Cheers,
Alex

[sebastian...]

HI @apxltd,

1. It's even more confusing to use than packageid://, so we'd need to find a better UI solution

Would it be more confusing to have two or three hash values instead of dozens, maybe hundreds of packageid:// entries for a given license? People are getting used to using hash values (e.g. with GIT commits). One would have to be able to view the original license text, of course (see next point)...

2. We'd want to store the full license text as well, so it'd be easy to confirm the contents

Agreed. Otherwise, there is no way of confirming that the hash value was assigned to the correct license. But I don't think that would be infeasible. One could either store the content of the license file in a dedicated table in the database, or in file. And we only have to store it once per hash value. A single license file takes what, 1 maybe 2 KB? Let's say we will have a dozen, maybe a hundred different license files at the end (unlikely, probably more in the lower double digits area). That would take less than 1 MB in total.

3. This is all a nontrivial engineering effort

Agreed, but it's not too complex either. There is obviously already code in Proget which detects that there is an embedded license file and that can display the content of that file. I'd say you are probably almost halfway there :-)

4. We're not sure how many packages this would impact and how much value / time savings it would represent

Here is an example: Consider the Google.Apis.* packages (https://www.nuget.org/packages?q=google.apis). They all have the exact same Apache 2.0 license file embedded, so we would have to assign just a single hash value instead of dozens / hundreds of individual packages.
Another point is updates of packages. At the moment we would have to assign a license to every new version of a package. I think this feature could be a huge time saver.

5. None of this would even work for remote packages, which is by far what most users find confusing and have issues with

Yes, for this to work, Proget would have to download the given package and read to content of its license file. However, there are two major parts of Proget where license become relevant, and I believe the it's not an unrealistic scenario that Proget has downloaded the package in question in both scenarios:

1. Blocking packages. We want to prevent users from downloading packages with specific licenses. Let's assume a user wants to download X with version 1.2.3, and that package has an embedded license file. Now, to be able to serve the package to the user, Proget has to download it first, right? Either it has done so already and the package is cached, or Proget has to download it on demand. In both cases Proget has a chance to read the license file and compute a has value for its content, if it hasn't already done that before.
2. Reporting. Most packages that are analyzed by the SCA feature should have been downloaded in the past via Proget.

6. It would probably require less engineering effort to scan/query all packages on NuGet and make a "database" of package licenses using a little human intelligence

I think downloading all packages (including all versions of each package) and analyzing them would take a lot of effort and resources.

7. It would require even less effort to just ask package authors to specify license codes, and then eventually the problem will go away on its own probably

Will try this, starting with the Google folks...

[sebastian...]

@sebastian said in Package license definition:

7. It would require even less effort to just ask package authors to specify license codes, and then eventually the problem will go away on its own probably

Will try this, starting with the Google folks...

One more thought on the last point: What about packages that do not use a supported open source license? Think of Oracle.ManagedDataAccess. There simply is not SPDX tag that matches their proprietary license

That package is actually a good example for a scenario where hash-based license assignments would make a lot of sense. They update their license every now and then. Last update was from version 21.7.0 to 21.8.0. A hash-based license check would have caught that and labeled the new license as "unknown". As a result, version 21.8.0 would have been blocked until someone reviews the new license and manually approves it.

[apxltd]

Thanks for the additional thoughts @sebastian!

I agree... it's not totally infeasible from a technical standpoint, but it's still pretty tricky. Just to comment on a technical thing, FYI...

Now, to be able to serve the package to the user, Proget has to download it first, right?

Actually, ProGet "streams" content from connectors. This means that, when a user requests a package from ProGet (and that package is on a remote connector), ProGet will then request a package from the connector. As the file is being downloaded, ProGet will send the same data back to the user and optionally write that data to disk. If we didn't do this, ProGet would be basically unusually slow.

A ZIP archive (what package files use) use a tail index, which means you have to read it backwards from End of File. So it's not possible to read an embedded file unless we've downloaded the entire package.

There are a few other "gotchas" we'd need to consider, even for cached/local packages. For example, we can't open/seek the package file just to know the license and if the package should be blocked - especially when it comes to cloud storage (for the same reason - tail indexing). So, we would obviously need to store package license file info in the database too... but then we'd need a way to deal with existing packages on disk that don't yet have that info.

We may also want to add some sort of heuristic analysis of license text, even if it's simple as a basic distance check. Personally I think that's a bad idea to rely on... but other products do, and the reality is most users would just skim a license anyway.

This all becomes a lot easier after v2023 with centralized data and a package analyzer that can background scan all these, but still not trivial. And then there's the real hard part... the UI and documentation

We definitely don't want to hack something in like packageid:// and package::// -- those have been a total pain and plus, I hate the design

Anyway -- just wanted to give more technical insight into why ProGet behaves like this, and why I'm hesitant to jump on the "reading license file" approach without adding somethign that's a lot more valuable than what we have now.

[sebastian...]

Hi @apxltd, thanks for the insights!

I half expected this to be way more complicated than I had hoped for, but one can dream...