Otter and High Memory Usage

[ForgotMyUsername]

Otter - PowerShell Remnants and High Memory Usage

I raised this issue a while back but the Forum and I agreed that we thought it was something else and I was happy for the issue to be closed. Now I am using Otter in full flight mode, I'm seeing the problem re-occurring and wanted to re-open it for further diagnosis.

When using Otter and launching other processes from it (which are command line, silent), Otter is still keeping the powershell.exe process open even after a successful configuration run. Overtime this means that memory usage creeps up, eventually making the box unusable. The processes can be shut down by either force closing otter_agent.exe which cascades the close down or by using taskkill.exe /im:powershell.exe /f. However this is manual intervention which makes widespread use of it difficult - the whole advantage of Otter is not logging on to remote terminals!

I have tried the following to rectify this:

1. Ensuring servers are only in one role.
2. Restarting the service via the Web GUI (which restarts the Execution Dispatcher role).
3. Removing any Otter Ensure scripts from the run.
4. Restarting the service on the box.

I only noticed this when my monitoring window started showing excessive memory usage and sure enough Otter's PowerShell has hogged it all. From a programming perspective I can understand (although not agree with) if a configuration run fails that PowerShell might stick around for that individual process (if the Otter Agent can't tell it's hung or waiting for an issue). However, on a successful run there seems no explanation for this - can the last action of the Otter agent on a successful run not be to shut down any owned PowerShell scripts? Furthermore, I am unsure as to why if there are four PS scripts in the Otter script, all four stay around even when only one has the EXE launch in it.

I cannot prove that it is the EXE causing this problem but can confirm that:

1. The other server roles which do not have the EXE launches are NOT affected.
2. The EXEs are all using Start-Process with a -Wait. However, they all do successfully complete.

For information I am using the following EXES:

1. Certutil - to import certificates (in enterprise mode)
2. Openssl - to generate certificates.

I did try the workaround that was suggested - to wrap the PowerShell in an Otterscript with a timeout but that didn't work either.

Sorry for the long post!

[atripp]

Hi @ForgotMyUsername,

Under-the-hood, Otter uses an in-process PowerShell Host (i.e. a runspace) to execute scripts, and does not invoke PowerShell.exe:
https://github.com/Inedo/inedox-scripting/blob/master/Scripting/InedoExtension/PowerShell/PowerShellScriptRunner.cs#L181

And as you can see in the code, Otter is properly creating/disposing of those runspaces.

I'm not sure where the powershell.exe processes are coming from, but it sounds like you have some problematic scripts. These will cause a lot of memory to be consumed and never released. In addition, perhaps they are calling other scripts using powershell.exe? Or something? It's really hard to guess...

In any case, Otter already has a mechanism called Isolation (with isolated) that can be used to help solve problematic scripts like this.

When something is run as isolated, then a new Otter.Agent.exe process is created to handle the unit of work (such as a script execution). When the work is done, that process is terminated; if it doesn't terminate, then it is killed within 5 seconds.

Cheers,
Alana

[ForgotMyUsername]

@atripp my apologies. I figured this out. It turns out that I was returning objects during the configure which completely screwed it up. That isolate is a life saver. I'm on execution 118000 and the errors have gone. What did we do before Otter. Thank you to you all.

[atripp]

@ForgotMyUsername great news, thanks for letting us know :)