Is it always failing with the follow?

Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..

Yes, the error is exactly this one every time.

Looking over your Prod dates... did you happen to upgrade after 2024.03.14? Or maybe that's when you enabled the preview feature?

The production instance was coming from 2023.29, the .30 was skipped and the .31 applied only on 22.03.

My test instance is a bit tricky, since I did not track when I was updating something or enabling preview features. Inedo Hub says my last installation was on 17.03.2024, so that must have been the upgrade to .31, I was slightly off in my screenshot.

Inedo Hub log does not seem to reveal which version was installed in the logs. Can I find that somewhere else?

Could you try running sp_updatestats and cleaning fragmentation on those tables usig SQL Management Studio? Clearly there's something we're missing here...

Just ran the command on my test instance. According to the output nothing was updated. Ran another VulnerabilityDownloader job to double check, but it failed again.

I will now try to patch my test instance (SQL Server 2016) to the latest version and see if that helps in any way.

I have found that my test instance also suffers from the same problem.

Both instances and their SQL-Servers are running on Windows and were installed with Inedo Hub. The only modification that was done to SQL Server is making the instance remotely accessible via TCP.

Hardware should not be an issue, both VMs have 4 (or 8) cores, 8GB RAM and SSD disk space assigned to them.

What is interesting that the error is not actually time related. On my local test instance, that has a more powerful CPU, the same error occurs, but already after 6-7 minutes.

From the execution logs it seems like it is unrelated to a particular version or upgrades and rather some issue that appears over time.

The screenshots below show all available execution logs for the VulnerabilityDownloader. I marked the date when ProGet was updated to to a particular version left in the tables.
Production instance:

Microsoft SQL Server 2022 (RTM-GDR) (KB5032968) - 16.0.1110.1 (X64) 
	Nov  9 2023 22:31:58 
	Copyright (C) 2022 Microsoft Corporation
	Express Edition (64-bit) on Windows Server 2022 Standard 10.0 <X64> (Build 20348: ) (Hypervisor)

Test instance:

Microsoft SQL Server 2016 (SP1) (KB3182545) - 13.0.4001.0 (X64) 
	Oct 28 2016 18:17:30 
	Copyright (c) Microsoft Corporation
	Express Edition (64-bit) on Windows 10 Pro 6.3 <X64> (Build 19045: ) (Hypervisor)

ProGet 2023.31 (Build 5)

Executing the VulnerabilityDownloader job always leads to the error below:

Updating local PGVD index...
Downloading latest vulnerability definitions from Inedo Security Labs...
Download complete. Unpacking database...
Processed 1000 entries.
Processed 2000 entries.
Processed 3000 entries.

Processed 37000 entries.
Processed 38000 entries.
Processed 39000 entries.
Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
Update complete.

Few observations:

• The whole import runs around 12mins before it dies
• One thread of the SQL Server is sitting at 100% CPU during import
• The import progress slows down significantly after the 25000 entries mark

Initially we thought our anti virus is slowing down things too much, but even after excluding everything ProGet-related this error still appears.

Another somewhat related question:

When a SBOM scan is uploaded, no issues are created initially even though the UI suggests that analysis was done already. One has to run analysis a second time with the issue checkbox set for issues to be populated.

Is this intentional or what is the idea behind that?

I was expecting to get a full analysis after uploading either via API or UI.

I can understand that this is a lot of effort and really appreciate that this request is not discarded right away.

The workaround you proposed is something that I have already looked into myself.

To make this work smoothly, a webhook for SCA events would really be immensely helpful. Is something like that already on the 2024 SCA roadmap?

However, the package metadata should already be in ProGet by the time you upload the sbom. When doing package restores from ProGet, the packages will be cached automatically. If that's not happening for you, make sure to clear your nuget package caches.

Not all packages will always be acquired via the remote restore mechanism, as I eluded to here.
They still should be analyzed by SCA and not show up on SCA as "inconclusive".

For example, I just cannot get the System.Security.Cryptography.Primitives populated in the ProGet cache, via regular dotnet restore. The package is always taken from the local dotnet SDK installation folder.

There are also other cases like these framework packages. We have a number of people working via VPN and for performance reasons they are access nuget.org directly, via Microsoft's CDN, which gives them much better performance. Their restore operations will not trigger cache population in ProGet.

Bottom line, there are plenty of scenarios why a package might not be readily available in the ProGet package cache.

Ultimately we designed the SCA feature is designed to be used in conjunction with ProGet as a proxy to the public repositories. It's not a "stand-alone" tool, so it won't work well if packages aren't in ProGet.

The reason is, if the package metadata isn't in ProGet, it has to be searched for on a remote server. In your sample (one build, two packages), you're right.. it's just a few seconds to search that data on nuget.org. But in production, users have 1000's of active builds each with 1000's of packages... and that *currently * takes about an hour to run an analysis.

That is how every other SCA systems works, that does not have a built-in package server.

Right now we are using DepTrack behind ProGet as a caching proxy. If DepTrack wants to analyze any package it will just pull the information from ProGet, if that package is already cached, great. If that package is not cached ProGet will download it and it will be cached from here on out, zero manual intervention required.

This very same scenario will not work in ProGet without manual intervention by either adding packages on some exclusion list or downloading them manually to populate the cache.

This limitation will always put ProGet SCA in a disadvantage when being compared to other systems.

Adding 100k's of network requests to connectors to constantly query nuget.org/npmjs.org for server metadata would add hours to that time, triggers api rate limits, and causes lots of performance headaches. Plus, this "leaks" a lot of data about package usage, which is an added security concern. This is a major issue with tools like DependencyTrack - they're basically impossible to scale like ProGet.

I agree, this is not how things should be set up. If someone decides to run a setup like this, they are not doing a good job.

ProGet should always be in the middle as a caching proxy and the cache download should only be there to fill the gaps for packages that for whatever reason are not available yet.

I was pleased to see that the license files for OSS licenses are now part of the ProGet database and can be accesses via API, so thank you for that.

There might be a small problem with ill-formed files. I have only verified this for the Apache-2.0 license, but others seem to be affected as well.

Viewing the files from the ProGet UI, there seem to be line breaks missing:

A comparison between what can be downloaded from the ProGet /api/json/Licenses_GetAllLicenseData API vs. what is stored in the SPDX GitHub repository:

https://raw.githubusercontent.com/spdx/license-list-data/main/text/Apache-2.0.txt

Hi @apxltd

Thanks for the response. I will try to not reiterate the cache issue too much, as I have raised it here before
https://forums.inedo.com/topic/3953/proget-sca-missing-package-because-of-nuget-proxy-cache-miss/6?_=1710244266850 and also in my mail about ProGet SCA Feedback to you.

We were really hoping this would be improved as part of the SCA 2024 changes. Both current solutions, either manually downloading missing packages or maintaining exclusion lists seems like a workaround to something that could be fully automated.

I expect this to be one of the first questions my colleagues will ask me, once I introduce them to the SCA features. ProGet should try to do its absolute best to create the most complete analysis possible, even if that means it takes a few extra seconds. All the manual effort that should go into it, should be focused on solving the actual issues, like assessing and fixing vulnerabilities.

Package detection is still a lot hit or miss like in the previous version.

Here are two packages pulled in via nuget.org ProGet proxy, one is detected perfectly fine the other one raised an issue as "Package not in feed". When navigating to both of them they are found and reported with green label (no vulnerabilities or license violations).

What is also a bit confusing is the "inconclusive" label. To me this would mean the state of the package could not be determined, but on the issues package it gives a proper "Package not in feed" explanation. I would expect to see this state here too.

Three closely related issues:

1. When trying to bulk delete a project from /projects, I get a popup with the following error:

(500) Server Error
The DELETE statement conflicted with the REFERENCE constraint "FK__ProjectIssues__Projects". The conflict occurred in database "ProGet", table "dbo.ProjectIssues", column 'Project_Id'. The statement has been terminated.

2. The button Disable ProGet 2024 Projects (Delete All Builds) does not work
   When clicking the button the page reloads, but nothing changes. This might be related to the error above.
   **Edit
   Just found out it actually works, the builds are removed only the projects stay there, as the button text actually suggests.

3. I did not find any other menu option to delete a project or a build anywhere in the UI
   The only thing I found is the project bulk delete option which suffers from the issue above.
   Am I missing something?

@rhessinger

Thanks for the update, looking forward to the next release.

When trying to upload a SBOM JSON file create with recent tooling to ProGet 2023.30 (Build 16) the following error is returned:

(500) Server Error
Unsupported specification version: 1.5

Are there any plans to support this spec version?

@stevedennis

Famous last developer words: "Changing this one flag can't possibly break anything"

Thank you for considering

Cheers

We are also interested in README rendering.

I've been looking to find where the project URL is displayed, but were unable to find it. Maybe this could be added aswell?
https://learn.microsoft.com/en-us/nuget/reference/nuspec#projecturl

Also related is the releaseNotes field, that would be similar to README rendering
https://www.nuget.org/packages/Microsoft.Extensions.Configuration/9.0.0-preview.1.24080.9#releasenotes-body-tab
https://learn.microsoft.com/en-us/nuget/reference/nuspec#releasenotes

Essentially we are looking for a way to make changelog information of a given package easily accessible to developers.

I'm looking to enable HTTP to HTTPS redirection in ProGet.

Is this something that could be added?

https://learn.microsoft.com/en-us/aspnet/core/security/enforcing-ssl?view=aspnetcore-6.0&tabs=visual-studio%2Clinux-ubuntu#usehttpsredirection

An option to opt-into this behavior would be very much preferred to having to install another webserver/proxy just to handle this redirection.

Hey @atripp,

Thank you for that snippet, very insightful.

Oddly enough that issue is still on the .NET 8 Planning milestone, even though .NET 8 was already released this November. Ah.. one shall not question Microsoft's planning... ;)

On a server with multiple IPs, I tried to bind ProGet only to one of those IPs with Port 80 and 443 while using the integrated webserver and bind another webserver to the same ports of a 2nd IP.

<WebServer Enabled="true" Urls="https://192.168.0.1:443;http://192.168.0.1:80" Subject="mycert" Store="My" Location="LocalMachine" AllowInvalid="True" />

I observed that for some reason ProGet is using the Kestrel server when the binding a port on all IPs (https://*:443 ) but switches to HTTP.sys when binding a specific IP like I tried with the config above. This can be observed with tools like netstat or TcpView. With Kestrel the ports are held by the application exe itself, while with HTTP.sys hosting the ports are occupied by the System process.

The problem with HTTP.sys-based hosting seems to be, that even though the ports are only bound on one IP, they are also blocked from being used by another process on another IP. Once the ProGet service starts, both ports show up in the port exclusion list netsh int ipv4 show excludedportrange protocol=tcp.

As far as I know there usually is no reason to use HTTP.sys unless specific features are required (see here). Kestrel should also be capable of binding to concrete IP:Port combinations.
Also with Kestrel all these netsh http add urlacl settings don't seem to be required anymore. I made a small test app and I could bind my process to a port below 1024 without urlacls just fine when using Kestrel and running the process as Network Service.

I'd be interested to learn why ProGet is switching between Kestrel and HTTP.sys for these two IP binding scenarios. Also any pointers how to reuse the ports on another IP would be very welcome.

Thank you for the pointers.

I think I finally got it working, though I must admit I'm still not a 100% sure what combination in what order actually led to success.

I'm already in contact with @apxltd about your planned SCA changes. I will try to write up what tripped me as part of that feedback.

Hi @stevedennis

I have this feed feature enabled:

Or are you referring to the SCA setting "Vulnerability Download Blocking Configuration"?

Hi @atripp

I tried what you suggested. Here is the output from the PackageAnalyzer job:

DEBUG: 2023-10-17 11:53:19Z - Fetching list of licenses...
DEBUG: 2023-10-17 11:53:19Z - Found 327 licenses.
DEBUG: 2023-10-17 11:53:19Z - Analyzing 1 feeds...
INFO : 2023-10-17 11:53:19Z - Beginning analysis of nuget-proxy feed...
DEBUG: 2023-10-17 11:53:19Z - Fetching list of known NuGet vulnerabilities...
DEBUG: 2023-10-17 11:53:19Z - Found 0 vulnerabilities.
INFO : 2023-10-17 11:53:19Z - Recorded data for 1 packages in nuget-proxy feed.
DEBUG: 2023-10-17 11:53:19Z - Analyzing 1 active releases...
DEBUG: 2023-10-17 11:53:19Z - Analyzing TestProject 1.0.0...

Sadly, no matter what I do (delete and reupload, overwrite, clicking on "analyze" on the Release), the vulnerability is never shown.

Is there anything else I could try?