[OTTER] Create new server thru API hangs

[philippe.camelio_3885]

Hi,
When I create a list of hosts using API, otter hangs and restart.
If I restart the script, it is running fine.

I have seen in the logs, that a status check start on the first call.

---

Error logs report:

Microsoft.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
---> System.ComponentModel.Win32Exception (258): Unknown error 258
at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__209_0(Task`1 result)
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Inedo.Data.DatabaseContext.DbResult.CreateAsync(DbCommand command, DatabaseContext context, DateTimeOffset startTime, Stopwatch stopwatch)
at Inedo.Data.DatabaseContext.ExecuteInternalAsync(String storedProcName, GenericDbParameter[] parameters)
at Inedo.Data.SqlServerDatabaseContext.ExecuteInternalAsync(String storedProcName, GenericDbParameter[] parameters)
at Inedo.Data.DatabaseContext.ExecuteTableAsync[TRow](String storedProcName, GenericDbParameter[] parameters)
at Inedo.Otter.Service.TaskRunners.ExecutionDispatcherRunner.GetExecutionsToExecuteOrCancelAsync() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E423767\Src\src\Otter.Service\TaskRunners\ExecutionDispatcherRunner.cs:line 107
at Inedo.Otter.Service.TaskRunners.ExecutionDispatcherRunner.RunAsync(CancellationToken cancellationToken, Object arguments) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E423767\Src\src\Otter.Service\TaskRunners\ExecutionDispatcherRunner.cs:line 33
at Inedo.TaskRunner.RunInternalAsync(Object args)

But I can see any error at SQL level.

I am running Otter 2023.4 / SQL Server on docker

Any advice ?

Best regards
PhilippeC.

[philippe.camelio_3885]

I moved the otter stack (otter / sql express) on a larger server (72 cores/256 Go ram), and obviously it works well.
As the first server wasn't very small - at least for our environment (8vcpus, 32 Go RAM), is it possible you make some tuning at the .net level or the database level ?

I never had performance problem from older versions but there were running on Windows.

Best regards

PhilippeC.

[philippe.camelio_3885]

Update

Even on a large server, the problem occurs, but the server recovers faster

Using this exemple, Otter hangs after server1 registration.
and display this error message

If I execute again the same command, everything is OK

function Register-OT(){
Param(
   [Parameter(Mandatory=$false)]
   [string]$API = "https://otter.local/api", 
 
   [Parameter(Mandatory=$false)]
   [string]$ApiKey= "apikey", 
 
   [Parameter(Mandatory=$false)]
   [string]$Environment= "someenv", 
 
   [Parameter(Mandatory=$false)]
   [string]$Role= "os-l-base", 

   [Parameter(Mandatory=$false)]
   [ValidateSet('none','manuallyRemediate')]
   [string]$drift="none",
 
   [Parameter(Mandatory=$false)]
   [string]$CredentialsName="some-credentiel",
 
   [Parameter(Mandatory=$false)]
   [string]$TempPath="/tmp/otter",

   [Parameter(Mandatory=$false)]
   [string]$Port="22",
 
   [Parameter(Mandatory=$false)]
   [string]$FQDN = "somehost.local"
) 

$Serveur=$FQDN.Split(".")[0]
 
$URI=$API+"/infrastructure/servers/list"
$aListServer=Invoke-RestMethod -Method Post -Uri $URI -Headers @{'X-ApiKey' = $ApiKey}
 
if ($aListServer.Name -contains $Serveur){
      write-host "Le serveur $Serveur existe deja"
 }else{
    $URI=$API+"/infrastructure/servers/create/"+$Serveur
    $body = @{
        name="$Serveur";
        hostname="$FQDN";
        serverType="ssh";
        active=$true;
        drift=$drift;
        port="$port";
        roles=@($Role);
        environments=@($Environment);
        CredentialsName=$CredentialsName;
        TempPath=$TempPath
    }
}
#
# Appel de l'API
#
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

Invoke-RestMethod -Method Post -ContentType 'application/json;' -Uri $URI -Body (ConvertTo-Json $body) -Headers @{'X-ApiKey' = $ApiKey}
}
Register-OT -FQDN "server1.local" 
Register-OT -FQDN "server2.local" 
Register-OT -FQDN "server3.local" 
Register-OT -FQDN "server4.local" 

[atripp]

Hi @philippe-camelio_3885 ,

We haven't run into any of these issues in our testing, but this error is a "generic database timeout", which is implying that "something" is going on with the database. That's the only information we have, "something".

To troubleshoot this, you'll need to use some of SQL Server's performance/activity monitoring tools to spot what's going on at the time. It could be anything from outdated statistics to a missing index to a bad query. Or who knows.

Best,
Alana