Inedo Agent with Otter Bug - Legacy Powershell Instances

[OtterFanboy]

I have found an issue with the Inedo Agent when running an Otter configuration remediation or configuration check whereby legacy Powershell.exe and Otter.Agent.exe processes are not cleaned up if an execution is cancelled half way through. Where a configuration has a substantial number of steps, a large amount of RAM can be taken up by Powershell instances which are hung which could result in system instability. The fix to this is to close the Otter.Agent process, close individual PowerShell instances, or restart the main Otter service. Restarting the Agent service does not fix the issue. On sites with a large number of servers, this could cause administrative headache if a rogue script is deployed.

This error is particularly prevalent when creating PowerShell scripts that require user input (by mistake) - for example calling a Start-Process which should launch a silent application but for whatever reason shows a GUI. At this point the Powershell process called is hung, the Agent run configuration shows being stuck on that script, and the cancel method for scripts is unusable (it says "if the script can be cancelled it will" - but these never can).

However the error also exists when cancelling a run of multiple Powershell scripts as part of a configuration half way through. In this case, the run is cancelled successfully but the legacy processes are retained on the Agent machine. The fix is as above.

Potential ways of fixing this:

1. Providing an option for a maximum run time for a configuration after which the agent shutdowns all the processes it has started up on that run, thereby eliminating ghost runs.
2. Providing timeouts for script runs of individual items in the desired configuration thereby eliminating rogue processes.
3. Force closing any processes created during an execution run when an execution cancellation is requested rather after a specified time rather than waiting for the graceful ending of the current step (which in the case of waiting user input) would never happen.

Additionally (very tiny) under "Administration - System Status" Infrastructure Sync is typo'd to say Infrastrucure Sync.

[stevedennis]

Hi @OtterFanboy ,

Under the hood, Otter runs PowerShell scripts through a custom PowerShell host that runs within the agent's process. While this generally leads to performance improvements, it can lead to memory leaks or worse if PowerShell scripts are created improperly. It sounds like this is the case here -- especially if your scripts are prompting for input, not terminating, etc.

This is where Process Isolation can come in. Instead of using the agent's process to run the PowerShell script, you can use OtterScript's with isolation feature. This creates a new process that runs the script.

Here is an example of an OtterScript plan that runs the same PowerShell script in different processes, even though it is the same script and runs on the same agent:

for server targetsv
{
    with isolation
    {
        PsExec >>Write-Host "Process ID: $pid">>;
    }
    with isolation
    {
        PsExec >>Write-Host "Process ID: $pid">>;
    }
}

Note that you can also specify timeouts with with timeout=1000 as well.

Cheers,
Steve

[OtterFanboy]

@stevedennis This is amazing, what a great feature.