Navigation

    Inedo Community Forums

    Forums

    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    1. Home
    2. sebastian...
    3. Posts
    S
    • Profile
    • Following
    • Followers
    • Topics
    • Posts
    • Best
    • Groups

    Posts made by sebastian...

    • RE: Package license definition

      @apxltd said in Package license definition:

      In the past, we thought of adding a kind of wildcard URL for licenses, like a "package://Microsoft.*" => "MSPL" would basically associate all packages with that prefix that don't otherwise have a SPDX code, or an explicit license.

      Wonder if that would help here?

      A colleague of mine actually had an interesting idea today: How about calculating hash values (like SHA-1) for embedded license files and assigning licenses to those hash values? That way one would only have to assign licenses to each license text once (if the license text is identical across different packages or package versions).

      The workflow would be similar to assigning licenses to actual packages / versions, but instead of adding an pseudo URL like "packages://SomeVendor.SomePackage/1.2.0" we could do something like "hash://0xA1B2C3", where 0xA1B2C3 would be the hash value of the content of the license file. All other packages / versions with the exact same license text would automatically be mapped to the same license.

      Of course, some license texts include the name of the product or a copyright note, so we would still get multiple entries for the same license. But it should be significantly less than adding one entry for every package / version.

      What do you guys think about this idea?

      posted in Support
      S
      sebastian...
    • RE: Package license definition

      @apxltd Using URLs for packages would be a nice feature, especially as package URLs contain version numbers. Getting rid of those might already be helpful, because some package come in a rather large number of versions. Of course, even applying a wildcard to just the version of a package might lead to wrong results, because in theory a package could change it's licese from one version to the next, but that is probably not a very realistic scenario.

      However, things can become messy when different wildcard URLs could be applied to the same package. Unfortunately, not all Microsoft packages use MSPL. Some use MIT, some use the proprietary licenses... It's a real mess. So it probably wouldn't be as easy as making just one rule for microsoft.*. But still, using wildcards could make things a bit easier.

      @pmsensi We haven't used TrustedInstaller yet. The approach is interesting, and yes: it would probably make sense to have a central service like a ProGet server check package owners, but it would be a completely different approach to a very different problem. I don't think applying licenses to package owners makes a lot of sense, because - as written above - package owners like Microsoft can apply different licenses to different packages.

      That being said, having a new entity "package owner" or "publisher" or something like that and being able to filter for that entity could be a cool new feature. This could also be used in the SBOM reporting feature (like: 40 packages come from Microsoft, 20 from vendor A, 7 from vendor B).

      posted in Support
      S
      sebastian...
    • RE: Package license definition

      @pmsensi said in Package license definition:

      1222 packages are a lot! For now, we do not have many packages, but we will have them for sure :)

      Yeah, we have been using ProGet for a while now... :-)

      I'm pretty sure some are false positives, though. It seems that when a package has been downloaded, but there is no usage of it in any product (happens for infrastructure packages like angular/core, angular/cli or some Visual Studio extensions), there is no license entry in the database even though the package has a perfectly fine SPDX tag.

      What's driving me nuts at the moment is that Microsoft seems to be using embedded license info files for quite a number of their packages. Assigning licenses to those is going to take a while...

      posted in Support
      S
      sebastian...
    • RE: ProGet NuGet upload user tracking

      Have you tried setting the log level for that API key to "Everything"? And do you see log entries for that key when uploading NuGet packages?

      posted in Support
      S
      sebastian...
    • RE: ProGet NuGet upload user tracking

      Have you tried disabling anonymous access?

      posted in Support
      S
      sebastian...
    • RE: Package license definition

      Hi @pmsensi,

      we are facing a similar challenge. Basically, what I am interested in is a list of packages that we are currently using in our products that don't have a known license assigned to them (or: Proget was unable to identify the license, e.g. because the license info just states "see licenense.txt" or something like that). I think that you are looking for something similar and are not asking to analyze all packages that exist on Nuget or npm, because that would probably would not be feasible (as @atripp already mentioned in her reply).

      We are using the new Reporting & SCA feature extensively, and we are getting some very nice license statistics out of it. However, the one thing that is currently missing is the number or list of packages with an unknown license. I hope that this will be added in a future release. In the mean time, looking at the database can be a workaround to at least get a list of the package name.

      I think starting with 2022, ProGet started storing license infos of known packages in the database. I am assuming that that info is stored for packages that have been downloaded via ProGet (i.e. "cached") or maybe also for packages that have been reported to ProGet via pgscan (@atripp please feel free to add some info to this).

      One simple approach to get the license info of all "known" packages, would be this:

      select 
      	pi.Package_Name, l.External_Id, l.Title_Text
      from 
      	PackageIds pi
      left join 
      	PackageLicenses pl on pl.Package_Id = pi.Package_Id
      left join
      	Licenses l on pl.License_Id = l.License_Id
      group by pi.Package_Name, l.External_Id, l.Title_Text
      order by 1;
      

      Note that this select groups by package names and ignores versions. You could now simply look for packages without a known license like this:

      select 
      	pi.Package_Name, l.External_Id, l.Title_Text
      from 
      	PackageIds pi
      left join 
      	PackageLicenses pl on pl.Package_Id = pi.Package_Id
      left join
      	Licenses l on pl.License_Id = l.License_Id
      where l.External_Id is null
      group by pi.Package_Name, l.External_Id, l.Title_Text
      order by 1;
      

      However, if you look at the result of first query, you might notice that there might be multiple entries for some packages. That might be because a package has changed its license or just its license info (maybe switching from a "see license.txt" to an actual SPDX tag), or maybe because there was a bug in previous versions of ProGet that has been fixed in 2202.18 (https://inedo.myjetbrains.com/youtrack/issue/PG-2263). So you might get some false positives with this approach.

      To get a list of packages without any entry of a known license, we have to eliminate the ones with multiple entries. There might be a better and more readable way to get this done, but the query that worked for us is this one:

      select 
      	distinct pi.Package_Name
      from 
      	PackageIds pi
      left join 
      	PackageLicenses pl on pl.Package_Id = pi.Package_Id
      left join
      	Licenses l on pl.License_Id = l.License_Id
      where l.External_Id is null
      and pi.Package_Name in 
      (select Package_Name
      from 
      (
      select 
      a.Package_Name
      from 
      (
      select 
      	pi.Package_Name, l.External_Id, l.Title_Text
      from 
      	PackageIds pi
      left join 
      	PackageLicenses pl on pl.Package_Id = pi.Package_Id
      left join
      	Licenses l on pl.License_Id = l.License_Id
      group by pi.Package_Name, l.External_Id, l.Title_Text
      
      ) a
      group by a.Package_Name
      having count(*) = 1
      ) b);
      

      BTW: in case you are curious: that select gives us a list of 1222 packages.

      posted in Support
      S
      sebastian...
    • RE: Vulnerabilities: finding affected consumers

      Hi @stevedennis,

      having a PackageLicense table would be great for two reasons:
      a) It would give as the ability to do some reporting on used licenses.
      b) There are lots of packages that use embedded license files. ProGet already has a feature where we can manually assign the corresponding license to a specific version of a package, but as far as I can tell, this is done by generating a pseudo URL for each package and assigning that URL to the corresponding license. While this does work fine as long as it is done only for a small number of packages, I'm not so sure how it affect usability (and maybe performance) when this is done on a larger scale. Having a separate table that connects packages (or package versions) to specific licenses might be a cleaner way to store and process that information.

      I'd have to check whether we could give you a dump of our database, but we are only just starting to use ProGet, so there isn't really too much data there yet. The fact that we are new to this is actually the reason we are so interested in reports about licenses and vulnerabilities: We are setting up workflows, etc. and at some point we want to completely block downloads of vulnerable packages or packages that use restrictive/unknown licenses, but before we activate hard filtering rules we want to get an overview on how this would affect products currently in development (so we can adapt our rules or define exceptions). At the moment we are generating data by integrating pgscan to our build pipelines and gathering download statistics for packages (and hope that we will be able to analyze that data in a useful way), so feel free to reach out in a month or so. Maybe we will have enough meaningful data to be of use for you guys by then.

      Cheers,
      Sebastian

      posted in Support
      S
      sebastian...
    • RE: Vulnerabilities: finding affected consumers

      Hi @stevedennis,

      we did a quick test and it turned out that connecting vulnerabilities to consumers of affected packages is actually pretty straight forward.

      The following statement gives us all consumers:

      SELECT 
      	v.Vulnerability_Id,
      	d.Dependent_Package_Name, 
      	MIN(d.Dependent_Version_Text) AS Min_Dependent_Package_Version, 
      	MAX(d.Dependent_Version_Text) AS Max_Dependent_Package_Version,  
      	d.Package_Name, 
      	d.Version_Text, 
      	v.Score, 
      	v.Title_Text 
      FROM 
      	PackageDependents d 
      JOIN 
      	Feeds f 
      	ON d.Feed_Id = f.Feed_Id
      JOIN 
      	(SELECT * FROM Vulnerabilities CROSS APPLY STRING_SPLIT(Package_Versions, ',') ) v
      	ON d.Package_Name = v.Package_Name 
      	AND d.Version_Text = LTRIM(v.value)
      	AND f.FeedType_Name = v.FeedType_Name
      GROUP BY
      	v.Vulnerability_Id,
      	d.Dependent_Package_Name, 
      	d.Package_Name, 
      	d.Version_Text, 
      	v.Score, 
      	v.Title_Text
      ;
      

      Another statement shows all downloads of affected packages (useful in case not all products already provide consuming data to ProGet):

      SELECT
      	v.Vulnerability_Id,
      	d.Package_Id, 
      	d.Version_Text, 
      	v.Score, 
      	v.Title_Text,
      	d.User_Name,  
      	MAX(d.Download_Date) AS Last_Downloaded,
      	COUNT(*) AS Downloads_Count
      FROM 
      	PackageDownloads d 
      JOIN 
      	Feeds f 
      	ON d.Feed_Name = f.Feed_Name
      JOIN 
      	(SELECT * FROM Vulnerabilities CROSS APPLY STRING_SPLIT(Package_Versions, ',') ) v
      	ON d.Package_Id = v.Package_Name 
      	AND d.Version_Text = LTRIM(v.value)
      	AND f.FeedType_Name = v.FeedType_Name
      GROUP BY 
      	v.Vulnerability_Id,
      	d.Package_Id, 
      	d.Version_Text,
      	d.User_Name,
      	v.Score, 
      	v.Title_Text
      ;
      

      Note that in both cases we assume that Package_Versions is either a single version or a comma separated list of versions (this was the case for all the examples we have at the moment).

      [Coming back to my original question, that is actually something you guys could integrate into the Vulnerabilities page of ProGet in a slightly modified way: Basically, group both statements by Vulnerability_Id to get the download count and consumers count and display those numbers in separate columns for a quick overview. When a user clicks on one of those numbers, you can display a detailed list for that specific vulnerability.]

      One thing we could not find yet was information about the licensing of each package. That is an information which is displayed within ProGet on every package's overview page, but it is apparently not stored within the database (or maybe we just didn't find it)? This is something we would like to be able to link to consumers as well (which product uses which set of licenses?). Is there a simple way to get to that information?

      Cheers,
      Sebastian

      posted in Support
      S
      sebastian...
    • RE: Vulnerabilities: finding affected consumers

      Hi Steve,
      we did consider the SQL approach, but didn't go with it initially, since (according to the documentation) it is not really a recommended way to interact with ProGet.
      But if you say that you might consider supporting this scenario in the future, we can definitely look into it. Digging into the relational schema shouldn't be a problem. I will do some testing and update this thread when we have some results.

      Cheers,
      Sebastian

      posted in Support
      S
      sebastian...
    • RE: Vulnerabilities: finding affected consumers

      Hi Steve,
      thanks for your reply. One thing that would make things a bit easier would be if there would be a link from each entry in the "Vulnerabilities" list to the corresponding package's page.

      Example:
      Let's say our ProGet server was named "myprogetserver.intranet" and our Nuget proxy feed (i.e. the feed that is connected to the api.nuget.org) was named "Nuget".
      [CVE-2019-1302] defines a vulnerability for Microsoft.AspNetCore 2.2.0, so it would be great to have a link to myprogetserver.intranet/feeds/NuGet/Microsoft.AspNetCore/2.2.0/ or myprogetserver.intranet/feeds/NuGet/Microsoft.AspNetCore/2.2.0/vulnerabilities.
      [CVE-2021-26701] defines a vulnerability for several versions of System.Text.Encodings.Web (4.0.1, 5.0.0, 4.7.1, 4.7.0, 4.6.0, 4.5.0, 4.0.0 and 4.4.0), so each displayed version should be a link to the corresponding page (myprogetserver.intranet/feeds/NuGet/System.Text.Encodings.Web/4.0.0/, myprogetserver.intranet/feeds/NuGet/System.Text.Encodings.Web/4.0.1/, ...).

      I understand that this is a non-trivial request, because packages could occur in several different feeds and information of packages that have not been downloaded are not cached, but maybe one solution to this would be to assign default feeds per package type for the Vulnerabilities view? Of course the perfect solution would be to have an additional column in that view that would list all affected consumers of those packages.

      Since you mentioned generating our own report: We are actually considering building our own tool to analyze & display packages and products and the dependencies between them. What would help as a lot for this would be a kind of generic API for all feeds and package types. What we would need is this:

      • list of all feeds
      • per feed: feed type and list of all packages in that feed
      • per package: list of all versions of that package
      • per package version: license information, dependencies and vulnerabilities

      Basically, it would be very similar to the API for Universal feeds, just for all package types (or at least Nuget and npm).

      Any thoughts on the points above?

      Cheers,
      Sebastian

      posted in Support
      S
      sebastian...
    • Vulnerabilities: finding affected consumers

      Hi there,

      I was wondering: what would be the best/recommended way to identify consuming packages/products of vulnerable packages?

      We currently use OSS Index as a vulnerability source with automatic assessments. For products currently in development, this works fine (vulnerable packages are blocked). However, what happens, when a vulnerability is discovered after a product is released? We use pgscan to add consumer infos to packages for this purpose, but there does not seem to be a very convenient way to get from the list of vulnerabilities to the affected consumers.

      We check the list of vulnerabilities on a regular basis and look for newly discovered vulnerabilities. However, there is no link from the vulnerability to the corresponding packages, never mind a list of consumers. That is why our current workflow is currently a bit cumbersome: we take the list of vulnerabilities, then open a new browser tab to search the corresponding feeds, copy & paste the names of the packages into feed's search, open the package's info package, search for the affected versions of the package and then navigate to "Usage & Statistics" to find the affected consumers. This does not seem to be very efficient. Surely, there must be a better way to do this...?

      posted in Support
      S
      sebastian...
    • RE: ProGet Retention Rules: option to keep package statistics

      Thanks for clarifying this! I did a test run with a package that has not been downloaded before and it seems you are correct: The total downloads per version (on the "Overview" and "All Versions" tabs) are reset to zero, but the statistics under "Usage & Statistics" seem to be unaffected.

      posted in Support
      S
      sebastian...
    • ProGet Retention Rules: option to keep package statistics

      It seems that when cached packages are deleted by a retention rule, not just the cached package content is removed from disk, but all statistics are reset as well (we have observed this on our npm proxy feed, but it might affect all feed types). Can anyone say if this affects "Consumers" infos as well?

      It would be nice to have an option to delete only the cached package content (i.e. what you get when you click "Download") but keep all the metadata/statistics (i.e. how often was that package version downloaded, what other packages/products is it consumed by, ...). Does such an option exist and we just missed it?

      posted in Support
      S
      sebastian...
    • RE: Permissions only work when set for specific user, not a group (LDAP)

      @kichikawa_2913 Not sure if this is related to your problem, but we had some strange problems after upgrading to ProGet 6.0.8 / 6.0.9 as well, so maybe sharing the following info might help you:

      We first encountered problems after upgrading to ProGet 6.0.8: Users could access feeds, but could not download any packages. After upgrading to version 6.0.9 this was resolved.

      We then had a problem where access became very slow until users could not access any ressources on our ProGet server anymore (even those with "Anonymous" access rights!). We deactivated the "6.1 Preview Features" and restarted everything (including the IIS). This resolved the issue. We also updated the InedoCore extension, but I don' t think this was related to the problem.

      I think the 6.1 features are still kind of buggy. We try to avoid activating them for now.

      posted in Support
      S
      sebastian...
    • 1
    • 2
    • 2 / 2