@atripp We had the same error when upgrading each node on a high availability install. Does the upgrade routine delete and recreate the stored procedures for each node upgrade? It felt like there was no check of a version for each node upgrade.

@atripp said in Maven Andriod Endpoint URL:

https://dl.google.com/android/maven2

We have a problem with this on our old server. We have the packages on the feed. We have imported them to the new ProGet Server. The connector is setup correctly.

It is healthy

It is connected to the feed.

When we request the package, we are after we get the following.

We have attempted to import the old servers' packages directory as a temporary fix, but we get the following error for all the imports from the other package directory.

Added Maven Package com.android.tools.build/gradle 1.5.0.
Could not add file "\proget-smb.wtg.zone\Proget\F12\android.arch.core\common\1.1.0\common-1.1.0-sources.jar". There was not matching pom file.
Could not add file "\proget-smb.wtg.zone\Proget\F12\android.arch.core\common\1.1.0\common-1.1.0.jar". There was not matching pom file.

When we look at packages that were successfully imported, we get the following error:

With a corresponding record in the diagnostic log.

Level:Error
Category:Web
Message:An error occurred in the web application: Object reference not set to an instance of an object.
Details:URL: http://proget.wtg.zone/feeds/Maven-Android/com.android.tools.build/com.android.tools.build.gradle/1.5.0
Referrer: https://proget.wtg.zone/packages
User: SW
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0
Stack trace: at Inedo.ProGet.Feeds.Maven.MavenFeed.Inedo.ProGet.Feeds.IBrowsablePackageFeed.ListPackageVersionsAsync(BrowsablePackageId id)
at Inedo.ProGet.WebApplication.Pages.Packages.PackagePageBase.CreateChildControlsAsync()
at Inedo.ProGet.WebApplication.Pages.ProGetSimplePage.InitializeAsync()
at Inedo.Web.PageFree.SimplePageBase.ExecutePageLifeCycleAsync()
at Inedo.Web.PageFree.SimplePageBase.ProcessRequestAsync(AhHttpContext context)
at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context)

Hi Everyone,

We have an issue with our ProGet server running 24.0.6. We have added some new caching feeds and getting the following error in the logs consistently a few times a day.

Level:Warning
Category:Connector
Message:Unable to fetch connector data; using cached copy.
Details:System.Net.WebException: The operation has timed out.
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location ---
at System.Net.WebRequest.GetResponseAsync()
at Inedo.ProGet.Feeds.Connector.GetDirectResponseAsync(String url)
at Inedo.ProGet.Feeds.Connector.GetCachedResponseAsync(String url)

We are unable to tell from the logs which connector is timing out. We want to check the network and firewall and ensure no errors for the public URL. Is there a way to determine this?

Regards Scott

I set our metadata caching to 10000 queries over 24 hours. I will reduce to the default first and monitor.

Thanks Dean

Hi Dean,

We have narrowed the problem down to a particular query causing SQL CPU Blocking. Connectors_GetCachedResponse. This is causing significant timeouts when our server is under load.

ALTER PROCEDURE [dbo].[Connectors_GetCachedResponse]
(
  @Connector_Id INT,
  @Request_Hash BINARY(32)
)
AS BEGIN

SET NOCOUNT ON

BEGIN TRY
  BEGIN TRANSACTION

UPDATE [ConnectorResponses]
   SET [LastUsed_Date] = GETUTCDATE(),
   [Request_Count] = [Request_Count] + 1
   WHERE [Connector_Id] = @Connector_Id
   AND [Request_Hash] = @Request_Hash

SELECT *
   FROM [ConnectorResponses]
   WHERE [Connector_Id] = @Connector_Id
   AND [Request_Hash] = @Request_Hash

COMMIT
END TRY BEGIN CATCH
  IF XACT_STATE()<>0 ROLLBACK
  EXEC [HandleError]
END CATCH

END

Hi ProGet Support,

We have an issue where our High-Availability ProGet infrastructure uses over 1000 SQL connections and reaches the max pool size. Some investigation on the SQL server using the activity monitor has shown the following queries running.

CREATE PROCEDURE [PgvdVulnerabilities_GetVulnerabilitiesForPackage]
(
	@PackageName_Id INT
)
AS
BEGIN
 
    SELECT * INTO #PgvdPackageNames FROM [PgvdPackageNames_Extended] WHERE [PackageName_Id] = @PackageName_Id
    SELECT * FROM [PgvdVulnerabilities_Extended] 
     WHERE [Pgvd_Id] IN (SELECT [Pgvd_Id] FROM #PgvdPackageNames)
 
    SELECT * FROM #PgvdPackageNames
 
    SELECT * FROM [PgvdAssessments_Extended]
     WHERE [Pgvd_Id] IN (SELECT [Pgvd_Id] FROM #PgvdPackageNames)
 
END

CREATE PROCEDURE [Feeds_GetExtendedConfiguration]
(
	@Feed_Id INT
)
AS
BEGIN
 
	SET NOCOUNT ON
 
	SELECT *
	  FROM [FeedPackageAccessRules]
	 WHERE [Feed_Id] = @Feed_Id
	 ORDER BY [Sequence_Number]
 
	SELECT C.*
	  FROM [FeedConnectors] FC
	       INNER JOIN [Connectors] C
		           ON C.[Connector_Id] = FC.[Connector_Id]
	 WHERE FC.[Feed_Id] = @Feed_Id
	 ORDER BY FC.[Sequence_Number]
 
	SELECT *
	  FROM [FeedPackageFilters]
	 WHERE [Feed_Id] = @Feed_Id
	 ORDER BY [Sequence_Number]
 
END

This is causing an object lock

Our CD/CI process keeps requesting packages, and within 5 minutes, all the have 1000's of connections, and we get the following errors in ProGet when the lock releases.

An error occurred processing a GET request to http://proget.XXXXX/npm/Registry/cache-base: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
Details:System.InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
at Microsoft.Data.Common.ADP.ExceptionWithStackTrace(Exception e)
--- End of stack trace from previous location ---
at Inedo.Data.SqlServerDatabaseContext.CreateConnectionAsync()
at Inedo.Data.DatabaseContext.ExecuteInternalAsync(String storedProcName, GenericDbParameter[] parameters)
at Inedo.Data.SqlServerDatabaseContext.ExecuteInternalAsync(String storedProcName, GenericDbParameter[] parameters)
at Inedo.Data.DatabaseContext.ExecuteTableAsync[TRow](String storedProcName, GenericDbParameter[] parameters)
at Inedo.ProGet.WebApplication.FeedEndpoints.Npm.NpmPackageMetadataHandler.TryProcessRequestAsync(AhHttpContext context, WebApiContext apiContext, NpmFeed feed, String relativeUrl)
at Inedo.ProGet.WebApplication.FeedEndpoints.Npm.NpmHandler.ProcessRequestAsync(AhHttpContext context, WebApiContext apiContext, NpmFeed feed, String relativeUrl)
at Inedo.ProGet.WebApplication.FeedEndpoints.FeedEndpointHandler.FeedRequestHandler.ProcessRequestAsync(AhHttpContext context)

We are running version 23.0.31.

Regards Scott

@rhessinger Wonderful, thank you very much and credit to Pawel Ostrowski who also did a lot of investigation on this

Hi Alana,

What we did find to get our small applications working was the following post. This mentioned that when opening the file the path should start with four backslashes. "\\"

https://stackoverflow.com/questions/41330660/sqlite-database-file-cant-be-opened-when-placed-in-network-folder

https://stackoverflow.com/questions/17303076/sqlite-cannot-open-network-file-programmatically-even-though-worked-before

This worked for our small applications and allowed the SQL lite package to work over the SMB share.

Regards Scott

Hi Alana,

We have written many simple programs using the same SQL Lite package as above. We cannot find a scenario where SQL Lite works over an SMB share, and we have been able to find content that says not to use it over the network. We were able to confirm that the create database option works fine. It is the opening that fails every time. What happens with the OpenOrCreateDatabase is it rolls back the file created when the open fails, so you end up with nothing.

https://www.sqlite.org/useovernet.html

https://stackoverflow.com/questions/788517/sqlite-over-a-network-share

We are using the IIS setup. Has this been tested with IIS and SMB share for Conda? I assume the index information for the package repository is what is going into the SQL Lite DB? Would it not be faster to store this information locally on each node? Does the SQL Lite DB need to be shared?

Is there an option to set the Conda/Debian package storage on the network share and the SQL Lite DB locally for each node?

Regards Scott

Hi Alana,

We wrote a program to use the SQL Lite library and ran it from the ProGet Node.

This is from the ProGet node server to our ProGet SMB share, and there are no issues. It is just the ProGet application that is having a problem.

Hi Support,

We have an issue creating a Conda feed to an SMB share. The SQL Lite file cannot be created there; we are running a High Availability structure with no issues with NuGet, Npm, Python, and Maven feeds. With the Conda feed, we get

Is there a particular port that ProGet requires when communicating with SMB share for SQL Lite? We can get the feed working with local storage on one of the nodes but not with the shared storage. When we put the SQL lite file there manually and refresh the feed, it is deleted. It is like clearing the file and then attempting to write it there using a different process than the other feeds.

Regards Scott

Thank you very much for this, Alana. Very much appreciated.