[PROGET] Migration from 5.3.38 to 2023.22 - problems with npm packages

[philippe.camelio_3885]

Hi,

We are looking to migrate our Proget from 5.3.38 to 2023.22.
We have a NPM feed (proxy to https://registry.npmjs.org) and we have a terrible loss of performance using the new Proget - especially during the first compilation. We never saw this behaviour with Proget 5.3

Furthermore, the compilation fails with an error as some packages are not available (FileNotFoundException) while the package have been cached on Proget 2023.22.
Removing a package from the cache solved the problem
But a lot of packages have problems

The same compilation is OK when using the old Proget 5.3.38

Any thoughts ?

New proget is on Docker

Best Regards
PhilippeC.

[atripp]

Hi @philippe-camelio_3885 ,

First, note that ProGet 2023 requires a data migration, and if that failed then you will likely have a lot of other issues: https://docs.inedo.com/docs/proget-upgrade-2023

I don't have enough info with the errors reported to know where the problems are, but I'll share some general information.

ProGet 2022+ also uses a new platform (.NET6 vs .NET4); in general .NET6 performs better/faster, but .NET4 seems to handle "underpowered" hardware better. You can mimic the behavior of .NET4 by setting the Web.ConcurrentRequestLimit to between 200 to 500.

The "package not found" is a separate error, but may be related to a migration. It's hard to say, but it's easy to troubleshoot. You should be able to navigate to the package (string-width 4.2.3) and download it from the UI. If you get a file not found error, then investigate where that file is.

Cheers,
Alana

[philippe.camelio_3885]

Hi @atripp

I made a test with empty npm feed.
The proget is the docker version.

[philippe.camelio_3885]

@philippe-camelio_3885

I will try a lower value of Web.ConcurrentRequestLimit as you siggest.

About hardware, proget vms are on the same model/config of esx server. So they should used the same amount of cpu.
I will see if I can make more tests.

Thanks for the reply

[philippe.camelio_3885]

Hi @atripp
I made some tests.

I increased RAM up to 32 Go with 8 vCPU (3.30 GHz)
VM located on SSD Storage

Test case:

• delete the existing npm Feed
• create a npm feed (with default parameters)
• run a npm ci from Buildmaster. I am using one of our application (npm registry set to proget)
• Assign Web.ConcurrentRequestLimit
  a) set value to 0
  b) set value to 300

My observations:

1. Server crashed with less than 32 Go of RAM.

2. With Web.ConcurrentRequestLimit to 0,
   compilation is in error due to missing packages.
   Ram peak up to 28 Go

3. With Web.ConcurrentRequestLimit to 300
   Compilation OK
   less ram consuming

Figure 1: Ram consuming for Web.ConcurrentRequestLimit

To be complete:
Proget is running as docker (docker-compose)
On the same VM, there is 2 other containers

• Buildmaster
• Otter

[dean-houston]

Hi @philippe-camelio_3885 ,

You mention the server is crashing? That's really strange, and we've never quite seen anything like this before. We routinely test on much less powerful hardware in nearly identical scenarios that you are doing now.

SQL Server / .NET will use as much ram as it can, so the usage isn't surprising or really concerning.

Otherwise we don't really any info info on the errors --- You'll need to track down error messages from the ProGet side of things; perhaps the Docker container error logs, the SQL Server error logs, etc. Maybe try putting the npm client (i.e. BuildMaster) on a different server, just to help isolate any potential issues.

We've seen a lot of weird things happen -- every now and then, there's a problem with the network controller/hardware/driver on the server, and it's having a hard time processing the 1000's of simultaneous "loopback" connections that npm is making from the BuildMaster container to the ProGet container, and the "outbound" connections that npm would be making to npmjs.org.

But until you find out what those errors are on the ProGet side, it's impossible to guess.

[philippe.camelio_3885]

Hi @dean-houston
The server crashed because lack of ram.
With 32 Go, no more problem

I think I have to update my 3 docker-compose to add RAM limitations for each SQL Server and Proget/Buildmaster/Otter to contain them.
Until now, I had small applications (mainly apache/mariadb) and it is the first time I have SQL Express and dotNet apps on docker.

About the logs, I did not have time to send you something relevant and helpful. I will see what I can get.

And you are right, until now I solved symptoms (adding RAM) and not the origin of the problem (unknown)

Thanks for your time s
Best regards
Philippe

[philippe.camelio_3885]

@dean-houston
About the missing files (FileNotFoundException), I made a mistake in my docker-compose file, so the packages volume wasn't persistent.
Noob error , sorry about that.

Best regards
PhilippeC.