1. Home
  2. Categories
  3. Support
  4. Buildmaster Version 7.0.9 (Build 2) keeps suffering database timeouts

Welcome to the Inedo Forums! Check out the Forums Guide for help getting started.

If you are experiencing any issues with the forum software, please visit the Contact Form on our website and let us know!

Buildmaster Version 7.0.9 (Build 2) keeps suffering database timeouts

  • A

    I know you hate these words (we all do), but after the upgrade, navigating from Applications to Releases, times out on a server that only I am accessing. 6e838449-8a69-4079-943b-8cdf313d1a38-image.png

    The problem is; the SQL queries are timing out. The server as an AWS MSSQL server instance.
    It isn't reporting any issues and as the Buildmaster web site does connect, it isn't that the server is unreachable.

    I can connect to the database from the Buildmaster hosted server via Management Studio, using the same credentials and can execute simple queries on the database that do not execute slowly: -

    SELECT  [Application_Id] ,[Release_Number] ,[ReleaseStatus_Name] ,[Notes_Text]
	,[CreatedBy_User_Name] ,[CreatedOn_Date] ,[ModifiedBy_User_Name]
	,[ModifiedOn_Date] ,[CancelledBy_User_Name] ,[CancelledReason_Text]
	,[CancelledOn_Date] ,[Release_Sequence] ,[Release_Id] ,[Furthest_Build_Number]
	,[ReleaseTemplate_Name] ,[Pipeline_Name]
FROM [BuildMaster].[dbo].[Releases]

    164b34ba-c3ab-4d83-a90d-45acfe7ec7a6-image.png

    The logged message: -

    Message: An error occurred in the web application: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.

    The redacted database connection string, as set in the Inedo hub configuration section, is below and hasn't been changed recently: -

    Data Source=servername.someuid.region.rds.amazonaws.com;Initial Catalog=BuildMaster;User ID=database-user;Password=someRandomString;Network Library=dbmssocn

    Below is the rest of the log entry, including the stack trace: -

    Details: URL: http://buildmaster.domain.extension/releases

    Referrer: http://buildmaster.domain.extension/all-applications

    User: some-user-name

    User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36

    Stack Trace:-

    at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlDataReader.TrySetMetaData(_SqlMetaDataSet metaData, Boolean moreInfo)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
at Inedo.Data.DatabaseContext.ExecuteInternal(String storedProcName, GenericDbParameter[] parameters)
at Inedo.Data.SqlServerDatabaseContext.ExecuteInternal(String storedProcName, GenericDbParameter[] parameters)
at Inedo.Data.StrongDataReader.<Read>d__9`1.MoveNext()
at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
at Inedo.BuildMaster.Web.WebApplication.Controls.ReleaseList.<.ctor>b__1_0()
at System.Lazy`1.CreateValue()
at System.Lazy`1.LazyInitValue()
at Inedo.BuildMaster.Web.WebApplication.Controls.ReleaseList.<InitializeAsync>d__33.MoveNext()
--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Inedo.Web.Controls.SimplePageControl.<InitializeAsync>d__18.MoveNext()
--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Inedo.Web.Controls.SimplePageControl.<InitializeAsync>d__18.MoveNext()
--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Inedo.Web.PageFree.SimplePageBase.<InitializeAsync>d__59.MoveNext()
--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Inedo.BuildMaster.Web.WebApplication.Pages.BuildMasterSimplePage.<InitializeAsync>d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Inedo.Web.PageFree.SimplePageBase.<ExecutePageLifeCycleAsync>d__57.MoveNext()
--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Inedo.Web.PageFree.SimplePageBase.<ProcessRequestAsync>d__56.MoveNext()
--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.TaskAsyncHelper.EndTask(IAsyncResult ar)
at Inedo.Web.Handlers.AsyncHandlerWrapper.EndProcessRequest(IAsyncResult result)
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
    0
  • stevedennis
    inedo-engineer

    Hi @antony-booth_1029,

    What version did you upgrade from? That could help trace code changes.

    Does this happen only on one page (i.e. /releases)? Then it's probably related to a bad query/unexpected data, but you only have 161 releaes according to your query. Not so much.... easy way to test that is by adding querystring params: /releases?ApplicationId=2&Status=Active for example.

    If it's easy to see queryes on your RDS server, then we can see what query might be bad.

    Does this happen only intermittently/randomly? If it's only you, then the problem probably isn't database/server load. And even with a ton of people, that's really rare. On very old instances with lots of retention jobs and years/gigs of data, doing an index cleanup is necessary, but I don't think that's the case here. The simplest/fastest thing I can think to do is reboot the BuildMaster server, and hope it goes away (maybe it's a weird underlying network stack thing).

    Does this happen all the time (like nothing at all works on the website). Then it's probably network related?

    Thanks,
    Steve

    0
  • A

    Thanks for the reply.

    I really can't remember the previous version of Buildmaster, although it was 6.something
    After some time, I think it may be network related, as I had tried stopping Buildmaster service and web site, with no effect, rebooting the server with no effect, checking CPU and memory, with no issues.

    Having recently checked the server again, the pages are more responsive, returning results before the 30 second timeout, without pre-filtering the result set with the query parameters you suggested.

    Both the database (AWS RDS) and the server (AWS EC2) are hosted in the same VPC, in the same region, so shouldn't be having bandwidth or other network issues, however as the performance has changed with no changes to the Buildmaster environment, I conclude the issue is/was with AWS itself. Either the VM hosting or internal networking.

    Thanks for the suggestions.

    0 atripp 1 Reply
  • atripp
    inedo-engineer

    @antony-booth_1029 thanks for the update!

    Anecdotally, I've heard of similar issues with AWS RDS. I understand that, behind the scenes, it's some large/shared SQL Server database installation, so I wonder if other traffic impacts it? We've certainly confirmed that was the case in some organizations with their own large SQL Server cluster.

    You may find using SQL Server Express + Self Backup is okay enough, too.

    0
Log in to reply
 
Inedo Website Home • Support Home • Code of Conduct • Forums Guide • Documentation