Inedo Community Forums Forums
    • Recent
    • Tags
    • Popular
    • Login

    Welcome to the Inedo Forums! Check out the Forums Guide for help getting started.

    If you are experiencing any issues with the forum software, please visit the Contact Form on our website and let us know!

    Buildmaster Version 7.0.9 (Build 2) keeps suffering database timeouts

    Scheduled Pinned Locked Moved Support
    4 Posts 3 Posters 13 Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A Offline
      antony.booth_1029
      last edited by

      I know you hate these words (we all do), but after the upgrade, navigating from Applications to Releases, times out on a server that only I am accessing. 6e838449-8a69-4079-943b-8cdf313d1a38-image.png

      The problem is; the SQL queries are timing out. The server as an AWS MSSQL server instance.
      It isn't reporting any issues and as the Buildmaster web site does connect, it isn't that the server is unreachable.

      I can connect to the database from the Buildmaster hosted server via Management Studio, using the same credentials and can execute simple queries on the database that do not execute slowly: -

      SELECT  [Application_Id] ,[Release_Number] ,[ReleaseStatus_Name] ,[Notes_Text]
	,[CreatedBy_User_Name] ,[CreatedOn_Date] ,[ModifiedBy_User_Name]
	,[ModifiedOn_Date] ,[CancelledBy_User_Name] ,[CancelledReason_Text]
	,[CancelledOn_Date] ,[Release_Sequence] ,[Release_Id] ,[Furthest_Build_Number]
	,[ReleaseTemplate_Name] ,[Pipeline_Name]
FROM [BuildMaster].[dbo].[Releases]

      164b34ba-c3ab-4d83-a90d-45acfe7ec7a6-image.png

      The logged message: -

      Message: An error occurred in the web application: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.

      The redacted database connection string, as set in the Inedo hub configuration section, is below and hasn't been changed recently: -

      Data Source=servername.someuid.region.rds.amazonaws.com;Initial Catalog=BuildMaster;User ID=database-user;Password=someRandomString;Network Library=dbmssocn

      Below is the rest of the log entry, including the stack trace: -

      Details: URL: http://buildmaster.domain.extension/releases

      Referrer: http://buildmaster.domain.extension/all-applications

      User: some-user-name

      User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36

      Stack Trace:-

      at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlDataReader.TrySetMetaData(_SqlMetaDataSet metaData, Boolean moreInfo)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
at Inedo.Data.DatabaseContext.ExecuteInternal(String storedProcName, GenericDbParameter[] parameters)
at Inedo.Data.SqlServerDatabaseContext.ExecuteInternal(String storedProcName, GenericDbParameter[] parameters)
at Inedo.Data.StrongDataReader.<Read>d__9`1.MoveNext()
at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
at Inedo.BuildMaster.Web.WebApplication.Controls.ReleaseList.<.ctor>b__1_0()
at System.Lazy`1.CreateValue()
at System.Lazy`1.LazyInitValue()
at Inedo.BuildMaster.Web.WebApplication.Controls.ReleaseList.<InitializeAsync>d__33.MoveNext()
--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Inedo.Web.Controls.SimplePageControl.<InitializeAsync>d__18.MoveNext()
--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Inedo.Web.Controls.SimplePageControl.<InitializeAsync>d__18.MoveNext()
--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Inedo.Web.PageFree.SimplePageBase.<InitializeAsync>d__59.MoveNext()
--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Inedo.BuildMaster.Web.WebApplication.Pages.BuildMasterSimplePage.<InitializeAsync>d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Inedo.Web.PageFree.SimplePageBase.<ExecutePageLifeCycleAsync>d__57.MoveNext()
--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Inedo.Web.PageFree.SimplePageBase.<ProcessRequestAsync>d__56.MoveNext()
--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.TaskAsyncHelper.EndTask(IAsyncResult ar)
at Inedo.Web.Handlers.AsyncHandlerWrapper.EndProcessRequest(IAsyncResult result)
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
      1 Reply Last reply Reply Quote 0
      • stevedennisS Offline
        stevedennis inedo-engineer
        last edited by

        Hi @antony-booth_1029,

        What version did you upgrade from? That could help trace code changes.

        Does this happen only on one page (i.e. /releases)? Then it's probably related to a bad query/unexpected data, but you only have 161 releaes according to your query. Not so much.... easy way to test that is by adding querystring params: /releases?ApplicationId=2&Status=Active for example.

        If it's easy to see queryes on your RDS server, then we can see what query might be bad.

        Does this happen only intermittently/randomly? If it's only you, then the problem probably isn't database/server load. And even with a ton of people, that's really rare. On very old instances with lots of retention jobs and years/gigs of data, doing an index cleanup is necessary, but I don't think that's the case here. The simplest/fastest thing I can think to do is reboot the BuildMaster server, and hope it goes away (maybe it's a weird underlying network stack thing).

        Does this happen all the time (like nothing at all works on the website). Then it's probably network related?

        Thanks,
        Steve

        1 Reply Last reply Reply Quote 0
        • A Offline
          antony.booth_1029
          last edited by

          Thanks for the reply.

          I really can't remember the previous version of Buildmaster, although it was 6.something
          After some time, I think it may be network related, as I had tried stopping Buildmaster service and web site, with no effect, rebooting the server with no effect, checking CPU and memory, with no issues.

          Having recently checked the server again, the pages are more responsive, returning results before the 30 second timeout, without pre-filtering the result set with the query parameters you suggested.

          Both the database (AWS RDS) and the server (AWS EC2) are hosted in the same VPC, in the same region, so shouldn't be having bandwidth or other network issues, however as the performance has changed with no changes to the Buildmaster environment, I conclude the issue is/was with AWS itself. Either the VM hosting or internal networking.

          Thanks for the suggestions.

          atrippA 1 Reply Last reply Reply Quote 0
          • atrippA Offline
            atripp inedo-engineer @antony.booth_1029
            last edited by

            @antony-booth_1029 thanks for the update!

            Anecdotally, I've heard of similar issues with AWS RDS. I understand that, behind the scenes, it's some large/shared SQL Server database installation, so I wonder if other traffic impacts it? We've certainly confirmed that was the case in some organizations with their own large SQL Server cluster.

            You may find using SQL Server Express + Self Backup is okay enough, too.

            1 Reply Last reply Reply Quote 0

            Hello! It looks like you're interested in this conversation, but you don't have an account yet.

            Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.

            With your input, this post could be even better 💗

            Register Login
            • 1 / 1
            • First post
              Last post
            Inedo Website Home • Support Home • Code of Conduct • Forums Guide • Documentation