Posts made by brett.polivka

Digging more into this, it appears that "pip search" is basically unsupported at this point, even by pypi.org. You can disregard this issue.

My understand is that /simple is the endpoint for package retrieval and /pypi is for searching. Regardless, neither works.

This happens both with empty and non-empty feeds.

When attempting to search a feed using "pip search", I receive the following error:

An error occurred processing a POST request to http://<redacted>/pypi/maglabs/pypi: Root element is missing.

System.Xml.XmlException: Root element is missing.
at System.Xml.XmlTextReaderImpl.Throw(Exception e)
at System.Xml.XmlTextReaderImpl.ThrowWithoutLineInfo(String res)
at System.Xml.XmlTextReaderImpl.ParseDocumentContent()
at System.Xml.Linq.XDocument.Load(XmlReader reader, LoadOptions options)
at System.Xml.Linq.XDocument.Load(Stream stream, LoadOptions options)
at Inedo.ProGet.WebApplication.FeedEndpoints.Pypi.XmlRpcHandler.ProcessRequestAsync(AhHttpContext context, WebApiContext apiContext, PypiFeed feed, String relativeUrl) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E359832\Src\ProGet.WebApplication\FeedEndpoints\Pypi\XmlRpcHandler.cs:line 18
at Inedo.ProGet.WebApplication.FeedEndpoints.Pypi.PypiHandler.ProcessRequestAsync(AhHttpContext context, WebApiContext apiContext, PypiFeed feed, String relativeUrl) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E359832\Src\ProGet.WebApplication\FeedEndpoints\Pypi\PypiHandler.cs:line 35
at Inedo.ProGet.WebApplication.FeedEndpoints.FeedEndpointHandler.FeedRequestHandler.ProcessRequestAsync(AhHttpContext context) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E359832\Src\ProGet.WebApplication\FeedEndpoints\FeedEndpointHandler.cs:line 188

There's nothing in the diagnostic logs beyond "Object reference not set to an instance of an object."

As a workaround I set the account and containers to public but limited access via a private endpoint. Not ideal, but better than nothing.

Now I'm able to push the docker layers, but after they are pushed, I receive a 500 Internal Server Error from the server. The logs show:

A 500 error occurred in docker: Object reference not set to an instance of an object.
A 500 error occurred in docker: Object reference not set to an instance of an object.
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
Request finished HTTP/1.1 PUT http://<redacted>/v2/docker/busybox/manifests/latest application/vnd.docker.distribution.manifest.v2+json 527 - 500 109 application/json 21769.4992ms

I think I know the problem.

My guess is that you're performing the copy operation specifying the normal URI of the source blob, which is the public URI. Instead, you need to use the Shared Access Signature (SAS) URI. This can be obtained via the GenerateSasUri method on the BlobClient.

I created a brand new storage account, copied over the connection string, and still receive the same errors.

Looking at the http traffic, it's able to upload the blob without issue but is failing when it's trying to copy the uploaded blob to it's final destination.

Setting the account and containers to public is not an option.

Hello,

Upon upgrading to 2022 I can no longer push docker images when using Azure Blob storage.

At first I received these errors:

A 500 error occurred in docker: Public access is not permitted on this storage account.
RequestId:6d0b5687-601e-0062-2695-9c1193000000
Time:2022-07-21T00:04:04.4475288Z
Status: 409 (Public access is not permitted on this storage account.)
ErrorCode: CannotVerifyCopySource
Content:
<?xml version="1.0" encoding="utf-8"?><Error><Code>CannotVerifyCopySource</Code><Message>Public access is not permitted on this storage account.
RequestId:6d0b5687-601e-0062-2695-9c1193000000
Time:2022-07-21T00:04:04.4475288Z</Message></Error>
Headers:
Server: Windows-Azure-Blob/1.0,Microsoft-HTTPAPI/2.0
x-ms-request-id: 6d0b5687-601e-0062-2695-9c1193000000
x-ms-client-request-id: f079cd5b-6216-4469-a1c8-45c59f7ff124
x-ms-version: 2021-06-08
x-ms-error-code: CannotVerifyCopySource
Date: Thu, 21 Jul 2022 00:04:04 GMT
Content-Length: 246
Content-Type: application/xml

While I shouldn't need to enable public access, for testing purposes I did. After doing so, I started receiving this error:

A 500 error occurred in docker: Server failed to authenticate the request. Please refer to the information in the www-authenticate header.
RequestId:f3d12d81-001e-0006-219f-9ce00b000000
Time:2022-07-21T01:13:15.7594171Z
Status: 401 (Server failed to authenticate the request. Please refer to the information in the www-authenticate header.)
ErrorCode: CannotVerifyCopySource
Content:
<?xml version="1.0" encoding="utf-8"?><Error><Code>CannotVerifyCopySource</Code><Message>Server failed to authenticate the request. Please refer to the information in the www-authenticate header.
RequestId:f3d12d81-001e-0006-219f-9ce00b000000
Time:2022-07-21T01:13:15.7594171Z</Message></Error>
Headers:
Server: Windows-Azure-Blob/1.0,Microsoft-HTTPAPI/2.0
x-ms-request-id: f3d12d81-001e-0006-219f-9ce00b000000
x-ms-client-request-id: 0e5467a9-7e39-42ec-93da-53b037e6ad7f
x-ms-version: 2021-06-08
x-ms-error-code: CannotVerifyCopySource
Date: Thu, 21 Jul 2022 01:13:15 GMT
Content-Length: 297
Content-Type: application/xml

The connection string I entered contains the access key so there should be no reason it can't authenticate. This also worked fine before I upgraded.

As it is I can no longer push docker images, so any assistance you could provide would be greatly appreciated.

Thank you,
Brett

@Dan_Woolf Excellent. Thank you for your assistance with this!

@Dan_Woolf Adding library/ to the image name also doesn't work.

Hi @Dan_Woolf,

It looks like it works if there is a prefix on the repository name, but doesn't work if there isn't. I've tried setting the "Do not use library/ prefix for repositories without namespaces (this may be required for some container registries like Microsoft Container Registry)" option on the connector but it makes no difference.

Thanks,
Brett

Hi Dan,

• I'm running ProGet 6.0.6
• It only includes Docker images
• It does not include Helm packages

Thanks,
Brett

Hello,

Does anyone have a docker connector successfully working against azurecr.io? (Azure Container Registry)

No matter what I try, I just get a "manifest unknown" error back from ProGet. Even if I intentionally change the connector username or password to something that should fail, I still just get the same error. There is nothing useful in the logs, so I'm at a loss.

Thanks

I'm not sure exactly when it started failing.

I know the PAT is good, as I can use it outside of ProGet without issue.

I don't doubt that the changes may be on the GitHub side of things. Are you able to connect to ghcr in your environment?

I'm happy to help however I can. Is there a way to get more verbose logging? I can also run any test releases you want to publish.

I can't pull or search images either. No errors in the Diagnostic center.

Hello,

I have a docker connector to ghcr.io (the github container registry) which authenticates with a personal access token. This used to work just fine, but recently stopped. I can authenticate directly to ghcr.io from the command line with the same username and password, but I can now no longer connect and the ConnectorHealthCheckRunner reports a 403 error.

Connector ghcr health check reported error: The remote server returned an error: (403) Forbidden.

Nothing on my end has changed, it just stopped working.

Any ideas?

Thanks,
Brett Polivka

If it's not already on the roadmap, it would be wonderful if support for Rust Cargo packages could be added.

The language is quickly gaining in popularity and I would love to be able to use ProGet to host private packages.

Thanks. I get that things get deprecated and libraries need to be upgraded, but you guys really need to do a better job of regression testing your changes. I seem to be tripping over these regressions on a pretty regular basis lately. As it is now, I'm dead in the water until this next fix gets published.

Feels like every release fixes one Azure Blob problem and creates another.

Getting the following error when attempting to upload a python package:

Message: An error occurred processing a POST request to http://xxxx.com/pypi/python/legacy: Specified method is not supported.

Details: System.NotSupportedException: Specified method is not supported.
at Microsoft.Azure.Storage.Blob.BlobWriteStreamBase.get_Length()
at Inedo.ProGet.Feeds.Pypi.PypiFeed.InstallPackageInternalAsync(String fileName, Stream packageStream, InstallPackageOptions options, Boolean promoted) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp_E119635\Src\ProGetCoreEx\Feeds\Pypi\PypiFeed.cs:line 593
at Inedo.ProGet.WebApplication.FeedEndpoints.Pypi.LegacyUploadHandler.ProcessRequestAsync(HttpContext context, PypiFeed feed, String relativeUrl) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp_E119635\Src\ProGet.WebApplication\FeedEndpoints\Pypi\LegacyUploadHandler.cs:line 35
at Inedo.ProGet.WebApplication.FeedEndpoints.Pypi.PypiHandler.ProcessRequestAsync(HttpContext context, PypiFeed feed, String relativeUrl) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp_E119635\Src\ProGet.WebApplication\FeedEndpoints\Pypi\PypiHandler.cs:line 37
at Inedo.ProGet.WebApplication.FeedEndpoints.FeedEndpointHandler.FeedRequestHandler.ProcessRequestAsync(HttpContext context) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp_E119635\Src\ProGet.WebApplication\FeedEndpoints\FeedEndpointHandler.cs:line 191

Looks good!

I had to create a volume mount for /var/proget/extensions to get the extension upgrade to persist, but it seems to fix the problem.

Thanks!

Any advice on how to do that when using the docker version?

Proget version: 5.3.23
Azure extension version: 1.11.0

I'm attempting to push some Windows-based images, which contain some very large, multi-GB layers. Theses pushes never succeed, as I keep getting the following errors in the server logs:

A 500 error occurred in docker: The request body is too large and exceeds the maximum permissible limit.

Where "docker" is the name of my feed.

I'm using Azure blob store as the storage backend, and from what I can tell, this is where the error is actually coming from. My guess is that the issue stems from the 100MB block size limit imposed by Azure for block blobs.

https://docs.microsoft.com/en-us/rest/api/storageservices/understanding-block-blobs--append-blobs--and-page-blobs

Thanks to @atripp as well. Appreciate your patience dealing with this (and me).

Hi Rich,

This seems to have done the trick! I've been running constant restores against it for an hour with no timeouts.

Thanks!!

I'm still seeing the same problem and the stack traces. Can you confirm that the 5.3.23-ci.4 release is the right version, because it still contains the parallel query.

If I can venture a guess as to what's happening, I think that the ThreadPool is being completely saturated.

Kestrel will use the ThreadPool to service a request. That request is then using a parallel query, which will also use the ThreadPool. At some point, every thread in the pool is blocked, waiting for a task to complete, but the task can't complete because it needs to be scheduled onto the ThreadPool, but every thread in the pool is blocked.

I'd be curious to see what happens if the ParallelEnumerable query is replaced with a standard Enumerable query.

One more piece of of info... I was able run the same test on Windows (I extracted the apps from the Docker image and copied them to a Windows machine), and it exhibits exactly the same behavior.

Glad to help.

Yes, almost every thread shows that exact same stack trace.

If there's any additional testing or information I can provide, please let me know.

More info...

I was able to attach a debugger to the DotNetCore version, and there are approximately 140 threads, all waiting on the same ManualResetEvent. Here's the stack trace:

System.Private.CoreLib.dll!System.Threading.ManualResetEventSlim.Wait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.SpinThenBlockingWait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.InternalRunSynchronously(System.Threading.Tasks.TaskScheduler scheduler, bool waitForCompletion) C#
System.Linq.Parallel.dll!System.Linq.Parallel.SpoolingTask.SpoolStopAndGo<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo, int>(System.Linq.Parallel.QueryTaskGroupState groupState, System.Linq.Parallel.PartitionedStream<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo, int> partitions, System.Linq.Parallel.SynchronousChannel<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>[] channels, System.Threading.Tasks.TaskScheduler taskScheduler) C#
System.Linq.Parallel.dll!System.Linq.Parallel.DefaultMergeHelper<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo, int>.System.Linq.Parallel.IMergeHelper<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>.Execute() C#
System.Linq.Parallel.dll!System.Linq.Parallel.MergeExecutor<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>.Execute() C#
System.Linq.Parallel.dll!System.Linq.Parallel.MergeExecutor<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>.Execute<int>(System.Linq.Parallel.PartitionedStream<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo, int> partitions, bool ignoreOutput, System.Linq.ParallelMergeOptions options, System.Threading.Tasks.TaskScheduler taskScheduler, bool isOrdered, System.Linq.Parallel.CancellationState cancellationState, int queryId) C#
System.Linq.Parallel.dll!System.Linq.Parallel.PartitionedStreamMerger<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>.Receive<int>(System.Linq.Parallel.PartitionedStream<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo, int> partitionedStream) C#
System.Linq.Parallel.dll!System.Linq.Parallel.FirstQueryOperator<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>.WrapHelper<int>(System.Linq.Parallel.PartitionedStream<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo, int> inputStream, System.Linq.Parallel.IPartitionedStreamRecipient<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo> recipient, System.Linq.Parallel.QuerySettings settings) C#
System.Linq.Parallel.dll!System.Linq.Parallel.FirstQueryOperator<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>.WrapPartitionedStream<int>(System.Linq.Parallel.PartitionedStream<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo, int> inputStream, System.Linq.Parallel.IPartitionedStreamRecipient<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo> recipient, bool preferStriping, System.Linq.Parallel.QuerySettings settings) C#
System.Linq.Parallel.dll!System.Linq.Parallel.UnaryQueryOperator<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo, Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>.UnaryQueryOperatorResults.ChildResultsRecipient.Receive<int>(System.Linq.Parallel.PartitionedStream<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo, int> inputStream) C#
System.Linq.Parallel.dll!System.Linq.Parallel.ListQueryResults<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>.GivePartitionedStream(System.Linq.Parallel.IPartitionedStreamRecipient<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo> recipient) C#
System.Linq.Parallel.dll!System.Linq.Parallel.UnaryQueryOperator<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo, Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>.UnaryQueryOperatorResults.GivePartitionedStream(System.Linq.Parallel.IPartitionedStreamRecipient<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo> recipient) C#
System.Linq.Parallel.dll!System.Linq.Parallel.QueryOperator<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>.GetOpenedEnumerator(System.Linq.ParallelMergeOptions? mergeOptions, bool suppressOrder, bool forEffect, System.Linq.Parallel.QuerySettings querySettings) C#
System.Linq.Parallel.dll!System.Linq.Parallel.QueryOpeningEnumerator<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>.OpenQuery() C#
System.Linq.Parallel.dll!System.Linq.Parallel.QueryOpeningEnumerator<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>.MoveNext() C#
System.Linq.Parallel.dll!System.Linq.ParallelEnumerable.GetOneWithPossibleDefault<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>(System.Linq.Parallel.QueryOperator<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo> queryOp, bool throwIfTwo, bool defaultIfEmpty) C#
System.Linq.Parallel.dll!System.Linq.ParallelEnumerable.FirstOrDefault<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>(System.Linq.ParallelQuery<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo> source, System.Func<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo, bool> predicate) C#
InedoLib.dll!Inedo.Web.PageFree.PageFreeHandler.GetHttpHandler(Inedo.Web.Handlers.GetHandlerParams args) C#
InedoLib.dll!Inedo.Web.Handlers.DynamicHttpHandling.GetHandler(System.Web.HttpContext context, string requestType, string url, string pathTranslated) C#
InedoLib.dll!Inedo.Web.AhWebMiddleware.InvokeAsync(Microsoft.AspNetCore.Http.HttpContext context) C#
Microsoft.AspNetCore.HostFiltering.dll!Microsoft.AspNetCore.HostFiltering.HostFilteringMiddleware.Invoke(Microsoft.AspNetCore.Http.HttpContext context) C#
Microsoft.AspNetCore.Hosting.dll!Microsoft.AspNetCore.Hosting.HostingApplication.ProcessRequestAsync(Microsoft.AspNetCore.Hosting.HostingApplication.Context context) C#
Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests<Microsoft.AspNetCore.Hosting.HostingApplication.Context>(Microsoft.AspNetCore.Hosting.Server.IHttpApplication<Microsoft.AspNetCore.Hosting.HostingApplication.Context> application) C#
Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequestsAsync<Microsoft.AspNetCore.Hosting.HostingApplication.Context>(Microsoft.AspNetCore.Hosting.Server.IHttpApplication<Microsoft.AspNetCore.Hosting.HostingApplication.Context> application) C#
Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnection.ProcessRequestsAsync<Microsoft.AspNetCore.Hosting.HostingApplication.Context>(Microsoft.AspNetCore.Hosting.Server.IHttpApplication<Microsoft.AspNetCore.Hosting.HostingApplication.Context> httpApplication) C#
Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnectionMiddleware<Microsoft.AspNetCore.Hosting.HostingApplication.Context>.OnConnectionAsync(Microsoft.AspNetCore.Connections.ConnectionContext connectionContext) C#
Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TransportManager.BindAsync.AnonymousMethod__0(Microsoft.AspNetCore.Connections.ConnectionContext c) C#
Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection<Microsoft.AspNetCore.Connections.ConnectionContext>.ExecuteAsync() C#
Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection<System.__Canon>.System.Threading.IThreadPoolWorkItem.Execute() C#
System.Private.CoreLib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() C#
[Native to Managed Transition]

While I wouldn't think it would be enough to cause the type of performance issues I'm seeing, I've noticed that the DotNetCore version logs significantly more data to the console than the mono version.

Is there a way to configure the console log level?

As a test I reverted to the 5.3.19 Mono version, and it is extremely fast and completes the entire restore within seconds.

I then flipped back to the 5.3.19 .NET Core version, and it is so slow that the restore will not complete.

Nothing else changed between these two runs other than Mono vs. .NET Core.

I just upgraded. Made no difference.

My CI builder and Proget are running in the same kubernetes cluster in Azure, so there's no chance that network saturation is an issue.

The database is also in Azure and isn't near capacity, and there's no way that Microsoft's network between my cluster and the DB is being saturated by 100 queries.

As I said before, 100 simultaneous queries should be easy for any server to handle. That's nowhere near enough to saturate a network stack.

Is there any way to turn on more detailed debugging logs to see what the server is actually doing?

I'm sorry, but moving to Enterprise is an unacceptable solution. If the standard version can't handle restoring a single project in isolation, why does it exist?

Any reasonable server should be able to handle 100 simultaneous requests in under the timeout period, which is currently over 1 minute. Even if every request required a database query, which is already a suboptimal solution, even the slowest database should be able to handle 100 queries in under a minute. The database I'm using is not even close to saturated, so the problem lies elsewhere.

I have no connectors to nuget.org. nuget restore will query all available sources for all packages, so I can't prevent it from hitting proget for all packages.

--no-cache or not won't matter, because this is part of a CI build process that builds from a clean environment, so it won't have a cache.

I'm only restoring a single project, so --disable-parallel also won't matter.

Hello,

I am running ProGet 5.3.21 in Docker against an Azure Sql Database. During a NuGet package restore, I continually see timeouts, to the point that I'm not able to successfully complete a restore. The majority of the requests to ProGet are for packages which are not held locally, but are instead provided from nuget.org (but are requested from ProGet regardless).

I've increased the max pool size to 400, but I never see the connection count go over around 120 connections.

The database is not shared, and there is minimal load on any of the systems.

Anyone have any ideas?

Thanks