InedoHub Installer Offline

paul_6112

Hi Support,
I'm noticing some strange behaviour with the newer Hub versions if you could assist please?

I use the Hub on an internet connected machine to create an offline installer.
I then copy this to a Windows Server 2019 Core server, and run the Hub to upgrade my version fully offline.

I've been working my way through the various v7.0.0 builds recently with the associated InedoHub.exe versions, and I'm seeing some of them them get 'stuck'

For example :

The install appears to have actually finished however, the services started back up and the version is showing as upgraded.

However the hub is sat doing something ?

Process Monitor shows it creating 100's of Threads..

Thanks,
Paul

atripp

Hi @paul-reeves_6112 , sorry on the slow reply, we talked about this in our engineering meeting, but then I forgot to reply to it.

We've seen this come up a few times, but it's not limited to offline installations. We are thinking the problem is on the Windows Service Control Manager side of things (it's not sending the right signal), but we can't seem to pin it down. Our general plan is to add a timeout or something, and hope that it addresses the problem.

Not sure what's going on with the threads though; it might be related to the in-process database we use (Sqllite), doing some background things? I don't know...

Can you let us know if you it keeps happening?

paul_6112

Hi @atripp,
If there is any additional logging I can provide or you can increase debug log level in any places to help narrow this down, happy to take a look.

This could be coincidence as I expect the operations don't change between builds - but perhaps there is some subtle difference here and this gives you anything to think about.

Upgrade from v7.0.6 to v7.0.7 I saw the problem on two servers (hence this forum post)
Upgrade from v7.0.7 to v7.0.8 however the two servers both upgraded without issue. (Although there is a large memory balloon after the upgrade service starting ?)

Thanks,
Paul

atripp

hi @paul_6112 - just curiosu if this continues to happen? We had another report about this, but are still struggling to find out why / and a resolution. We're going to try to take another stab at figuring this out again.

paul_6112

Hi @atripp,
Sorry for slow response got a bit buried in work..

I've just down 8 upgrades for you as a test...and 1 got stuck, but it was not consistent between the two servers.

Test Server

Version 7.0.8 (Build 1) - Start
Version 7.0.9 (Build 2) - Hub Installer completed okay, initial CPU spike after install. Took several minutes for Webpage to be available.
Version 7.0.10 (Build 5) - Hub Installer completed okay, initial CPU spike after install.
Version 7.0.11 (Build 2) - Hub Installer completed okay, initial CPU spike after install.
Version 7.0.12 (Build 4) - Hub Installer completed okay, initial CPU spike after install.

Production Server

Version 7.0.8 (Build 1) - Start
Version 7.0.9 (Build 2) - Hub Installer completed okay, initial CPU spike after install. Took several minutes for Webpage to be available.
Version 7.0.10 (Build 5) - Hub Installer completed okay, initial CPU spike after install.
Version 7.0.11 (Build 2) - Hub Installer got stuck, initial CPU spike after install.
Version 7.0.12 (Build 4) - Hub Installer completed okay, initial CPU spike after install.

This is the same as the original post where it gets stuck :

Thanks,
Paul

gdivis

Hi @paul_6112 - The screenshot shows Inedo Hub v1.3.6. Was all of the testing done on this version? We did finally get a fix for this in Inedo Hub v1.3.7 (see DH-63), but did not confirm 100% that this was the underlying issue as we were not able to reliably reproduce it.

Thanks for passing along this test data!

-Greg

paul_6112

Hi @gdivis,
Each version used its own hub version that it downloaded so no..

My process is to use InedoHub on an internet connection workstation to create an offline installer.
These are then run on the servers, so its a self contained Hub and Installers, therefore bit puzzled on the versioning here...

v7.0.9 upgrade used hub v1.3.1
v7.0.10 upgrade used hub v1.3.7
v.7.0.11 upgrade used hub v1.3.6
v7.0.12 upgrade used hub v1.3.7

I'll update this on the next installs that are done on v1.3.7 +

Thanks,
Paul

paul_6112

Hi @gdivis ,

I've upgraded both servers from v7.0.12 to v7.013 using Hub v1.3.7.

Neither installer got stuck; but there was the initial CPU and memory spike after install.
The slight concern is it took over 10 minutes for the web page to become available.

Thanks,
Paul

Dan_Woolf

Hi @paul_6112,

What are the hardware (or virtual hardware) specs of your server?

Thanks,
Dan

paul_6112

Hi @Dan_Woolf,
Virtual 2vCPU and 4GB Ram.
SQL Express 2019 is also installed.

The two build master services are currently now idling at ~160mb ram and ~60mb ram.
After the upgrade startup, it easily hits over 1gb ram; however during that 10 minutes there was no obvious CPU contention.

I've got 45 Indeo Agent Servers on Test and 76 Inedo Agent Servers on Production so after startup its got to confirm the ready status; but limited 'manual' application jobs which I'm experimenting pushing software updates so its not got loads to catch up on...

Thanks,
Paul

gdivis

Hi @paul_6112 ,

That actually makes sense. We have an issue in our internal backlog to resolve high memory usage when a large number of agents are scanned just after an upgrade. I've published it as a publicly visible issue (BM-3758), and tentatively scheduled it for the next release.

-Greg

paul_6112

Hi @gdivis,
I've just installed v7.0.14 and the behaviour seems improved.

On the original thread track;
Using hub v1.3.7 the upgrade from v7.0.13 to v7.0.14 went through without getting stuck.

After upgrade, both Test and Production web pages were able to load in less than 2 minutes.
There is CPU spiking; but seems more spread out and the max memory is now two thirds of what it was; however within 6 minutes of startup its back around the 160MB - 350MB (it was idling at 160MB before upgrade) which is a definite improvement.

Thanks,
Paul