Imported 1058 servers, can no longer pull up list of servers

Jonathan.Engstrom

So after adding about 200 servers in our environment already, I added the other 1000 or so but adding them to their own separate environments and roles that are empty, as well as having basic configurations. They are doing ReportOnly, default raft, and using powershell. Now the screen pulls up an IIS error page when I try to list the servers out. My question is how do I get these removed via import/export or API? I wish I would of saved a copy of the config before I imported it. This is another good reason why it would be nice to have powershell access and not rely on an web/db combo to work with it. Server resource utilization is still pretty low. Help would be appreciated, thanks!

Server Error in '/' Application.
The wait operation timed out
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.ComponentModel.Win32Exception: The wait operation timed out

Source Error:

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.

Stack Trace:

[Win32Exception (0x80004005): The wait operation timed out]

[SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.]
System.Data.SqlClient.<>c.<ExecuteDbDataReaderAsync>b__180_0(Task1 result) +1219418 System.Threading.Tasks.ContinuationResultTaskFromResultTask2.InnerInvoke() +101
System.Threading.Tasks.Task.Execute() +70
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
Inedo.Data.<CreateAsync>d__10.MoveNext() +343
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
Inedo.Data.<ExecuteInternalAsync>d__33.MoveNext() +1237
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
Inedo.Data.<ExecuteInternalAsync>d__7.MoveNext() +364
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
Inedo.Data.<ExecuteTableAsync>d__32`1.MoveNext() +373
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
Inedo.Otter.WebApplication.Pages.<CreateChildControlsAsync>d__2.MoveNext() +377
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
Inedo.Otter.WebApplication.Pages.<InitializeAsync>d__1.MoveNext() +350
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
Inedo.Web.PageFree.<ProcessRequestAsync>d__46.MoveNext() +279
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
System.Web.TaskAsyncHelper.EndTask(IAsyncResult ar) +64
System.Web.CallHandlerExecutionStep.InvokeEndHandler(IAsyncResult ar) +212
System.Web.CallHandlerExecutionStep.OnAsyncHandlerCompletion(IAsyncResult ar) +166

Version Information: Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.7.3163.0

Product: Otter
Version: 2.1.2

So to give an update for anyone else who may have this issue, I tried using the API to delete the servers out, as I still had a list of all of the machines I added. The API was unresponsive and timed out, so that was not an option. It would not go to the main servers page, showing only the above error, which doesn't shed any insight to me as to what is going on. The only thing I would use was the import servers in administrator, which thankfully adds them all cumulatively, the newest ones are on the bottom. I was able to delete the servers after a few minutes, and get my Otter Instance to work again.

While I should of taken a backup of my import servers page, that list really should be on the raft and saved in the code repository. I also had a day old sql backup I could of used as well, but that is dramatic.

I am also perplexed at why this happened in the first place. I have been noticing that sometimes when I import servers, I see in the manage service output log after I import them, they keep reporting "hostname not found" yet when I copy the server name, delete that server, and recreate it using the create server button, and pasting the same server name, it then starts working. I don't know if Otter is not reading the json correctly, or Visual Studio Code is not formatting Json correctly, or maybe it's my fault entirely.

Also, is there any reason why in the Infrastructure API, I can't list out just one server? List is expressly verboten per the documentation, why?

Having consistent API structures would also be very helpful!

Unfortunately it's hard to say what happened on the original import; can you reproduce it consistently? If so, you can send us the infrastructure file as a ticket, and we can try to reproduce it.

The "host not found" message sounds like it's coming from the networking stack; maybe something in the import isn't correct? It's not a JSON formatting error, because then you would see a kind of parsing problem right away. Can you compare a manually-entered server's configuration JSON to your import? Maybe you'll spot the difference. It could be something simple.

The API list command is intended to provide a list of all servers. We would need to add a new API endpoint to provide info about a single server; but it's kind of easy/fast to get that same info from the list, I think.

Jonathan.Engstrom

As I had stated before, I have copied the text of the server name itself from a failed imported server, and created one using the exact same name pasted from the clipboard, and then it would start working. I have not seen any formatting issues, and it appears to be correct. It may be incorrect though, it is a possibility. I have a large project I have been trying to get to work in Otter, and unfortunately this importing issue, and the fact there is no way to work with servers, environments, or roles by addressing them on a command line language like PowerShell. My options are:

-Trying to change the servers by import/export server data to change them. Since my imports have not been rock solid, and have caused me many issues as of late, I have concerns of late with that method.

-Try to access the API to manipulate that.(?) Yeah, since I can't call individual servers, I am working with a glob of data and alot of "noise/clutter". Maybe someone can share with me a good way to do this in PowerShell?

-Add 1000+ servers into Otter using the web interface. Well......, there's that. PowerShell Module please? :D

It would help if there was more verbosity to the logging and/or give me more indications of what the issues are. I am not able to glean anything from the "Hostname not found" error when I can clearly find it without issue. Again, lack of data makes troubleshooting more difficult.

As far as my configuration, it is almost 10,000 lines long, and I have a long way to go yet. I am not sure what is easy or quick about picking 10 servers or so out of that output and changing things.

Just as an FYI, we're continuing our investigation and exploration of this under ticket
EDO-5623

UPDATE: we've addressed a lot of these in the current version of Otter.