Posts made by dan.brown_0128

The reason for blocking application packages like this, that have vulnerable dependencies, comes down to security governance. While yes, it can be inconvenient if a package that has worked before is now blocked, that does prevent us from introducing known vulnerabilities into our environment. In the event of an emergency deployment (ex prod rollback, etc), we could apply a temporary exemption to allow the package to still deploy -- after doing a risk assessment.

To use the log4j example again -- if we have an application that was built with a vulnerable version of log4j, nobody would want that package to get deployed again (while also remediating it anywhere that it was already deployed). From what I can tell the most effective way would be to block the download from ProGet - if we can leverage it's automatic blocking functions.

Adding the audit into the deployment process is definitely one way to partly add this security layer but it'd require active implementation for all of our deployments, and opens more opportunity for teams to skip or work around it. Basically its better than nothing yes, but it's not the most effective security enforcement measure.

For #3 (which unfortunately is probably the biggest potential win for us), I might have an idea...

Currently when editing the Project's settings, you can specify Feeds - which I'm understanding are the feed(s) that provide the packages outlined in the SBOM - ie the dependency source from the build).

What if at that level it was broken out to input/output feeds? The artifact MyCorp.Package, which resulted in the SBOM, was stored in the "internal-apps" feed, and it was built using dependencies in the "approved-nuget-feed." Now, this would assume that the project name and resultant package name are the same, but that shouldn't be too unreasonable.

Again at a high level, what we're wanting is to be able to block our custom built packages if its dependencies (identified by SBOM) become vulnerable (or are otherwise deemed noncompliant). Since these packages will never be known by any OSS index or the ProGet Security, it wouldn't be flagged by traditional vulnerability means, and that's why the SCA/SBOM is useful.

In re-evaluating the SCA Functions of Proget 2024, we've come up with a few functionalities that would be big benefits. Posting it publicly in case others have similar thoughts

1. Be able to add a link to our CI build and CD deployment on a build's release (for Inedo, this was mentioned in EDO-10735)
   This could probably be as simple as adding two optional fields in the "Create Build" modal and/or in the build promotion process. Of course this would also want/need to be implemented into the pguitl CLI.

2. Dependent Package's 'Usage in Projects' should show latest version in addition to prior versions (EDO-10736)
   It'd be helpful to quickly see all versions of an SCA project that reference a specific version of a dependent package. Right now it's labeled for the Latest version, although on our instance it appears to not correctly show the latest.

3. Link between SCA Build and the artifact
   Bringing back up this topic from last year. Having a way to link an SCA Project/build back to the artifact that corresponds to the full BOM would be great, especially if it shows the vulnerabilities of the package discovered from the BOM analysis. This would let us block downloading of packages we've built that may now contain vulnerabilities that have come up over time.
   I wrote up a more narritive version of what I'm thinking:

• Imagine we have MyCorp.Package v1.0.0 which is a nuget stored in the internal-apps feed
• It has dependencies of Ext.Dep.1.1.1 and Ext.MS.Dep2.2.2 which are in the approved-nuget feed
• We upload the SBOM
• If you go to Ext.Dep or Ext.MS.Dep in approved-nuget feed you will see that each is used in the MyCorp.Package project (makes sense, those are dependencies of the package).
• But if you go to MyCorp.Package in internal-apps, you don't see it used in a project, since its not a dependency, but rather is the project itself. Wouldn't it make sense for this to link? That way if your package has a vulnerable dep (and the project shows vulnerabilities), you can see that at the top level (and maybe block downloading the applications package)t at the top level (and maybe block downloading the applications package)

I've noticed that other Artifact management systems are offering products for IaC Scanning -- looking for vulnerabilities, known misconfigurations, best practices, etc. This could be a feature that would be a nice supplement to ProGet.

Most of what I've seen has the IaC scanning centered around Terraform templates. I'm using ARM/Bicep since we're completely Azure based for cloud presence.

I'm not 100% sure how the other products are doing IaC scanning -- it may be along the lines of scanning a Terraform Private Registry, ensuring that modules there are up to standards. With Bicep/ARM, Azure provides Template Specs as a manner of storing and cataloging private modules that can be imported into templates. I don't think its prudent to reinvent Template Specs into ProGet, but it would be nice to get the best practice and quality checks to be made against ARM/Bicep templates stored inside of artifacts in ProGet.

In our instance, the templates are typically stored inside of a nuget package, although a universal package could be used instead.

Are there any others looking into IaC scanning, particularly in an Azure + ARM/Bicep organization?

I wonder if its an issue if the old JSONPath was pulled locally first, then jsonpath was pulled. Right now, we just have a workaround by disabling npm audit support in ProGet, which allows our builds to run, but that disables a legitimate feature because of this one edge case.

My guess is that some DB cleanup would need to happen to purge JSONPath and jsonpath from all memory?

Thanks for the details. We did delete jsonpath and JSONPath from our local feed, but once ProGet caches the jsonpath package locally, it starts to display as JSONPath on the ProGet site. Is there a way to force the site to recognize it as jsonpath instead since v1.1.1 is the only one locally?

The vulnerability has already been assessed as "Not Applicable" and that's where the "info" comes up, as shown by your code snippet. The problem is that NPM does not have info as a classification and cannot parse it

Part of the issue here is that the results from ProGet do not match that of the public registry. Yes, it is an edgecase where there are 2 packages with the same name but different capitalization, but its also a fairly common package.

npm audit is a step that is run by default as part of npm install and other standard functions. This is to advise of package updates, end of support, and vulnerabilities by priority. Simply replacing NPM with pgutil is not a small feat, especially with standard common tooling already in use across the field.

The fact that our builds fail because of the mismatch is problematic, and shows that two unique packages are being mixed by ProGet. The audit caught it, and prevented the two from being crossed

We ran into an issue when a NPM package has a vulnerability, but an exception has been made to allow the package download. The example we have was static-eval v 2.1.1

We have an exception on the PGV-2133354 vulnerability since it has been withdrawn.

npm audit and npm install fail because the severity level of the vulnerability with exception is info which is not a supported level by NPM. the output from npm is:

npm ERR! undefined is not iterable (cannot read property Symbol(Symbol.iterator))

NPM Audit Severity Levels
And you can see in the NPM Audit code that it does not expect info as a severity level

Output from ProGet npm/v1/security/advisories/bulk endpoint:

Right now we are NOT able to build with this package without disabling vulnerability detection, which is not permitted by corporate policy. How can we correct the output from ProGet so that npm will successfully build

we recently had users having issues if their NPM project depended on the jsonpath package. Our NPM feed uses a connector to the public NPM registry, and we had pulled version 1.1.1 to our local proget. Apparently when this was done, the package was saved to ProGet as JSONPath.

This resulted in the packument data pull in npm audit to fail since the metadata JSON (at https://progetserver/npm/feedname/jsonpath) had the name JSONPath but the package name is all lower case.

Removing the package and all versions from ProGet and clearing it from disk does appear to work, but once we pull it back into ProGet, it will fail.

NPM is very clear in their documentation that package names should always be all lowercase because of case sensitivity differences between OSes.

How can we fix ProGet so that we can pull the jsonpath package locally but keep the proper naming convention?

I was finally able to try this with a mounted network drive (using SysInternals), and that works

Quick update -- Just installed 2023.30 in our test environment. At first I was still not able to download packages with + in the version (despite the index being correct). After clearing the feed and re-promoting packages it does seem to work now!

Oh and also to note: there is no network traffic (captured by our networking team) or from IIS logs when attempting to apt install unixodbc-common from ProGet (pulled to feed). Maybe apt is thinking that filename is invalid and skips it

I'm starting to think this is related to package versions with a +. We can keep going down the road of unixodbc-common_2.3.11-2+deb12u1_all.deb for example.

If I disable the cache and do not pull packages to proget, I can download/install just fine.

If I promote the package to another feed, I cannot install it. BUT if I try a package without a + in the version/filename in that second feed, it does install or at least resolve (ex: bcal)

There is one big difference in the index that might be the key... The original/upstream Debian index does not urlencode the filename but it looks like ProGet is

debian upstream: Filename: pool/main/u/unixodbc/unixodbc-common_2.3.11-2+deb12u1_all.deb
Proget: Filename: pool/bookworm/main/all/unixodbc-common/unixodbc-common_2.3.11-2%2Bdeb12u1_all.deb

Dockerfile used:

FROM privatedockerrepo/python:3.8-slim
RUN apt update 
RUN apt install -y wget gnupg2 apt-transport-https apt-rdepends
RUN  echo 'deb https://user:pass@proget.local/debian/import-debian bookworm main contrib non-free' | \
    tee /etc/apt/sources.list.d/proget-approved-deb.list
RUN wget -qO - https://user:pass@proget.local/debian/import-debian/keys/import-debian.asc | apt-key add -
RUN mv /etc/apt/sources.list.d/debian.sources /etc/apt/sources.list.d/debian.sources.old
RUN apt update && apt install unixodbc-common
CMD ["bash"]

Output from first run:

PS> docker build .
[+] Building 42.5s (15/15) FINISHED                                                                                                                                                                                                                               docker:default 
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                        
 => => transferring dockerfile: 1.17kB                                                                                                                                                                                                                                      
 => [internal] load metadata for privatedockerrepo/python:3.8-slim                                                                                                                                                                                                 
 => [auth] python:pull token for privatedockerrepo                                                                                                                                                                                                                
 => [internal] load .dockerignore                                                                                                                                                                                                                                           
 => => transferring context: 2B                                                                                                                                                                                                                                             
 => [internal] load build context                                                                                                                                                                                                                                           
 => => transferring context: 71B                                                                                                                                                                                                                                            
 => [1/7] FROM privatedockerrepo/python:3.8-slim                                                                                                                                          
 => [2/7] RUN apt update                                                                                                                                                                                                                                             
 => [3/7] RUN apt install -y wget gnupg2 apt-transport-https apt-rdepends                                                                                                                                                                                            
 => [4/7] RUN  echo 'deb https://user:pass@proget.local/debian/import-debian bookworm main contrib non-free' |     tee /etc/apt/sources.list.d  0.0s 
 => [5/7] RUN wget -qO - https://user:pass@proget.local/debian/import-debian/keys/import-debian.asc | apt-key add -                                                                                           
 => [6/7] RUN mv /etc/apt/sources.list.d/debian.sources /etc/apt/sources.list.d/debian.sources.old                                                                                                                                                                   
 => [7/7] RUN apt update && apt install unixodbc-common                                                                                                                                                                                                                    
 => exporting to image                                                                                                                                                                                                                                                      
 => => exporting layers                                                                                                                                                                                                                                                     
 => => writing image sha256:...                                                                                                                                                                                 

Output from second run:

PS> docker build .
[+] Building 42.5s (15/15) FINISHED                                                                                                                                                                                                                               docker:default 
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                        
 => => transferring dockerfile: 1.17kB                                                                                                                                                                                                                                      
 => [internal] load metadata for privatedockerrepo/python:3.8-slim                                                                                                                                                                                                 
 => [auth] python:pull token for privatedockerrepo                                                                                                                                                                                                                 
 => [internal] load .dockerignore                                                                                                                                                                                                                                           
 => => transferring context: 2B                                                                                                                                                                                                                                             
 => [internal] load build context                                                                                                                                                                                                                                           
 => => transferring context: 71B                                                                                                                                                                                                                                            
 => [1/7] FROM privatedockerrepo/python:3.8-slim                                                                                                                                          
 => [2/7] RUN apt update                                                                                                                                                                                                                                             
 => [3/7] RUN apt install -y wget gnupg2 apt-transport-https apt-rdepends                                                                                                                                                                                            
 => [4/7] RUN  echo 'deb https://user:pass@proget.local/debian/import-debian bookworm main contrib non-free' |     tee /etc/apt/sources.list.d  0.0s 
 => [5/7] RUN wget -qO - https://user:pass@proget.local/debian/import-debian/keys/import-debian.asc | apt-key add -                                                                                           
 => [6/7] RUN mv /etc/apt/sources.list.d/debian.sources /etc/apt/sources.list.d/debian.sources.old                                                                                                                                                                   
 => ERROR [7/7] RUN apt update && apt install unixodbc-common                                                                                                                                                                                                              
------
 > [7/7] RUN apt update && apt install unixodbc-common:
0.413
0.413 WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
0.413
29.70 Get:1 https://proget.local/debian/import-debian bookworm InRelease [39.1 kB]
30.13 Get:2 https://proget.local/debian/import-debian bookworm/non-free all Packages [53.7 kB]
30.50 Get:3 https://proget.local/debian/import-debian bookworm/contrib all Packages [27.7 kB]
30.74 Get:4 https://proget.local/debian/import-debian bookworm/contrib amd64 Packages [66.1 kB]
31.02 Get:5 https://proget.local/debian/import-debian bookworm/non-free amd64 Packages [125 kB]
31.27 Get:6 https://proget.local/debian/import-debian bookworm/main amd64 Packages [10.6 MB]
36.09 Get:7 https://proget.local/debian/import-debian bookworm/main all Packages [5719 kB]
38.67 Fetched 16.7 MB in 38s (436 kB/s)
38.67 Reading package lists...
38.99 Building dependency tree...
39.08 Reading state information...
39.09 21 packages can be upgraded. Run 'apt list --upgradable' to see them.
39.09 W: https://user:pass@proget.local/debian/import-debian/dists/bookworm/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
39.10
39.10 WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
39.10
39.10 Reading package lists...
39.42 Building dependency tree...
39.51 Reading state information...
39.52 E: Unable to locate package unixodbc-common
------
Dockerfile:16
--------------------
  14 |     RUN wget -qO - https://user:pass@proget.local/debian/import-debian/keys/import-debian.asc | apt-key add -
  15 |     RUN mv /etc/apt/sources.list.d/debian.sources /etc/apt/sources.list.d/debian.sources.old
  16 | >>> RUN apt update && apt install unixodbc-common
  17 |
  18 |     CMD ["bash"]
--------------------
ERROR: failed to solve: process "/bin/sh -c apt update && apt install unixodbc-common" did not complete successfully: exit code: 100 =>                                                                                                                                                                                                                                                       

This is an oddly specific issue I'm finding. Background info - fully standard, no customizations

• ProGet is running on Windows Server
• Proget v 2023.29 (Build 17)
• New Debian connector set up with bookworm
• debian feed called import-deb linked to above connector.
• Debian bookworm container setup with private ProGet Debian apt source + key, public Debian feed removed

When I build the container for the first time (works on a VM or any other Debian instance), it will successfully download and install the package from Proget (for discussion, unixodbc-common. This is good!

If I do another build or attempt to install the same package on another computer, apt cannot locate the package. This is bad! I can, however, still see and download the package from the Proget UI and even via curl on the container/vm, ruling out network issues.

If I then delete that package from Proget, and attempt to install via apt -- it works again, but only once. Once the package has been pulled from the Public Debian repo and saved into ProGet, it seems apt cannot find it again.

That's what I suspected. We'll do some more testing on our end about UNC/mounted drives and see what comes up. Are there any debug logs I can get from the ProGet service that might give more info?

@stevedennis - If we go the route of storing the sqlite DB onto a local disk on each node of a cluster, are there any risks that would come of that? Either packages being out of sync, or slowdowns?

We're still researching the network path on our end

@stevedennis -- we were able to rule out antivirus and similar. Windows Defender shows no threats or actions, and nothing on our file servers.

We were able to have success by switching the Storage.DebianPackagesLibrary path to a local path (E:\ProGetPackages\.debian vs \\server.corp\qa\progetpackages\.debian).

With the local path, the connector passes heathcheck and I see remote packages in the feed. Unfortunately this will not work well for our production use since we're using a HA setup.

I did try copying the connector folder (and sqlite file) back to the network share and changed the storage setting back to the network share. When I ran the heathcheck, the sqlite file was deleted. Looks to me that there's something happening specific to the Debian2 connector and storage paths using a network location (\\server\path notation). Is this something you can validate too?

I checked on the file share, and the sqllite database is not found. ProGet definitely can write to the directory, and does for all other feeds.

Just to confirm -- even if ProGet is using SqlServer for all other database operations, does the Debian feed connector require SqlLite? Our corporate policy currently prohibits the use of sqllite and that could very well be why we don't see the file on disk

@gdivis -- We are running ProGet on Windows

Upping the timeout doesn't seem to make a difference. The "unable to open database file" message is logged almost immediately whenever the connector healthcheck runs.

When viewing a new Deb feed with the connector added, it only shows the message "An unexpected error occurred while listing packages: unable to open database file. Additional information has been logged in the diagnostics center." and doesn't clear (24 hours).

Here's the stack trace that shows in the messages. Not sure what Sqlite DB it's trying to access. The main database is SqlServer

Details:code = CantOpen (14), message = System.Data.SQLite.SQLiteException (0x800007FF): unable to open database file
at System.Data.SQLite.SQLite3.Open(String strFilename, String vfsName, SQLiteConnectionFlags connectionFlags, SQLiteOpenFlagsEnum openFlags, Int32 maxPoolSize, Boolean usePool)
at System.Data.SQLite.SQLiteConnection.Open()
at Inedo.ProGet.Feeds.Debian2.Debian2ConnectorIndex.Open(String fileName, Debian2Connector connector)
at Inedo.ProGet.Feeds.Debian2.Debian2Connector.GetIndex()
at Inedo.ProGet.Feeds.Debian2.Debian2Connector.GetAndUpdateIndexAsync()
at Inedo.ProGet.Feeds.Debian2.Debian2Connector.ListPackagesAsync(Nullable`1 maxCount)+MoveNext()
at Inedo.ProGet.Feeds.Debian2.Debian2Connector.ListPackagesAsync(Nullable`1 maxCount)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
at Inedo.ProGet.Feeds.Debian2.Debian2Feed.GetConnectorPackagesAsync(Func`2 getPackages)+MoveNext()
at Inedo.ProGet.Feeds.Debian2.Debian2Feed.GetConnectorPackagesAsync(Func`2 getPackages)+MoveNext()
at Inedo.ProGet.Feeds.Debian2.Debian2Feed.GetConnectorPackagesAsync(Func`2 getPackages)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
at System.Linq.AsyncEnumerable.UnionAsyncIterator`1.MoveNextCore() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/Union.cs:line 131
at System.Linq.AsyncIteratorBase`1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 70
at System.Linq.AsyncIteratorBase`1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 75
at System.Linq.Internal.Lookup`2.CreateAsync(IAsyncEnumerable`1 source, Func`2 keySelector, IEqualityComparer`1 comparer, CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/Lookup.cs:line 105
at System.Linq.Internal.Lookup`2.CreateAsync(IAsyncEnumerable`1 source, Func`2 keySelector, IEqualityComparer`1 comparer, CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/Lookup.cs:line 105
at System.Linq.AsyncEnumerable.GroupedAsyncEnumerable`2.MoveNextCore() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/GroupBy.cs:line 1094
at System.Linq.AsyncIteratorBase`1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 70
at System.Linq.AsyncIteratorBase`1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 75
at Inedo.ProGet.Feeds.Debian2.Debian2Feed.GetLatestVersionOfEach(IAsyncEnumerable`1 packages)+MoveNext()
at Inedo.ProGet.Feeds.Debian2.Debian2Feed.GetLatestVersionOfEach(IAsyncEnumerable`1 packages)+MoveNext()
at Inedo.ProGet.Feeds.Debian2.Debian2Feed.GetLatestVersionOfEach(IAsyncEnumerable`1 packages)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
at System.Linq.AsyncEnumerable.SelectEnumerableAsyncIterator`2.MoveNextCore() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/Select.cs:line 221
at System.Linq.AsyncIteratorBase`1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 70
at System.Linq.AsyncIteratorBase`1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 75
at System.Linq.AsyncEnumerable.<ToListAsync>g__Core|424_0[TSource](IAsyncEnumerable`1 source, CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/ToList.cs:line 36
at System.Linq.AsyncEnumerable.<ToListAsync>g__Core|424_0[TSource](IAsyncEnumerable`1 source, CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/ToList.cs:line 36
at Inedo.ProGet.Feeds.Debian2.Debian2Feed.SearchPackagesAsync(String query, Int32 maxCount, Boolean includePrerelease)
at Inedo.ProGet.WebApplication.Pages.Packages.ListPackagesPage.PackageList.InitializeAsyncInternal()
at Inedo.ProGet.WebApplication.Pages.Packages.ListPackagesPage.PackageList.InitializeAsync()

Trying to set up the new Debian feed and connector on ProGet 2023.29

I can create the feed and connector, but the healthcheck fails with:

Connector debian-buster health check reported error: unable to open database file

Connector Settings:
Type Debian2
Name debian-buster
Url http://ftp.debian.org/debian/

I have tried restarting the Service and the ISS site and app pool, but still get the same connector error. Any ideas?

Does PGVC watch/monitor vulnerabilities that could be in Debian packages, or is it only on Nuget/Python/Java/etc?

We are looking to switch from using OSS Feeds to PGVC, but are running into challenges on whitelisting locations/services on the firewall to access the repo.

Is there a listing of URLs/IPs and ports that our ProGet instance will need access to for PGVC to work?

@rhessinger Thanks for the clarification. Just to be sure - there's no linkage between the SCA Project back to the Nuget package / feed, right? The two modules are basically independent of each other

@atripp I've set up a testing app just to experiment with. It currently only exists in one feed

In this example, v2013.7.13.4 has issues associated from SCA (one vulnerability from 3 CVE and one missing package)

When I go to that version within the feed, I see no known vulnerabilities. Note we don't have any OSS Index associated with this feed since it's our internally built tools - closed source.

My thought is that the vulnerabilities listed on the Project + Version/Release level should propagate down to the package in the feed as well. Is that a fair understanding of how it should work?

We have started evaluating the SCA / SBOM function within ProGet. When using pgscan to build/load the SBOM, I see the release and project appearing in the "Reporting & SCA" section -- with a nice breakdown of all imports and associated vulnerability risk.

As it stands right now, the two locations show conflicting data:

• the SCA/Project page says that there are vulnerabilities

but

• The Package page shows that there are none (persumably since our internally built packages aren't found on any OSS listing)

Is there any way that these findings can be propagated to the actual Package/artifact within our internal feed?