Azure Blob errors with upgrade to 2022

brett.polivka

Hello,

Upon upgrading to 2022 I can no longer push docker images when using Azure Blob storage.

At first I received these errors:

A 500 error occurred in docker: Public access is not permitted on this storage account.
RequestId:6d0b5687-601e-0062-2695-9c1193000000
Time:2022-07-21T00:04:04.4475288Z
Status: 409 (Public access is not permitted on this storage account.)
ErrorCode: CannotVerifyCopySource
Content:
<?xml version="1.0" encoding="utf-8"?><Error><Code>CannotVerifyCopySource</Code><Message>Public access is not permitted on this storage account.
RequestId:6d0b5687-601e-0062-2695-9c1193000000
Time:2022-07-21T00:04:04.4475288Z</Message></Error>
Headers:
Server: Windows-Azure-Blob/1.0,Microsoft-HTTPAPI/2.0
x-ms-request-id: 6d0b5687-601e-0062-2695-9c1193000000
x-ms-client-request-id: f079cd5b-6216-4469-a1c8-45c59f7ff124
x-ms-version: 2021-06-08
x-ms-error-code: CannotVerifyCopySource
Date: Thu, 21 Jul 2022 00:04:04 GMT
Content-Length: 246
Content-Type: application/xml

While I shouldn't need to enable public access, for testing purposes I did. After doing so, I started receiving this error:

A 500 error occurred in docker: Server failed to authenticate the request. Please refer to the information in the www-authenticate header.
RequestId:f3d12d81-001e-0006-219f-9ce00b000000
Time:2022-07-21T01:13:15.7594171Z
Status: 401 (Server failed to authenticate the request. Please refer to the information in the www-authenticate header.)
ErrorCode: CannotVerifyCopySource
Content:
<?xml version="1.0" encoding="utf-8"?><Error><Code>CannotVerifyCopySource</Code><Message>Server failed to authenticate the request. Please refer to the information in the www-authenticate header.
RequestId:f3d12d81-001e-0006-219f-9ce00b000000
Time:2022-07-21T01:13:15.7594171Z</Message></Error>
Headers:
Server: Windows-Azure-Blob/1.0,Microsoft-HTTPAPI/2.0
x-ms-request-id: f3d12d81-001e-0006-219f-9ce00b000000
x-ms-client-request-id: 0e5467a9-7e39-42ec-93da-53b037e6ad7f
x-ms-version: 2021-06-08
x-ms-error-code: CannotVerifyCopySource
Date: Thu, 21 Jul 2022 01:13:15 GMT
Content-Length: 297
Content-Type: application/xml

The connection string I entered contains the access key so there should be no reason it can't authenticate. This also worked fine before I upgraded.

As it is I can no longer push docker images, so any assistance you could provide would be greatly appreciated.

Thank you,
Brett

atripp

Hi @brett-polivka ,

In ProGet 2022, we upgraded to a completely new version of the Azure SDK, since the old version was long since deprecated. So, it's not unexpected to see issues... though we didn't see these in our testing :)

I'll do my absolute best to help! Our code doesn't really deal with authentication or access rules; that's all handled by the Azure SDK, which uses that connection string.

I searched "CannotVerifyCopySource", and a lot of SDK users (including tools like AzCopy) repot this issue when switching to the new SDK/API. I don't know Azure well enough to undersatnd, but some said that it had something to do with "storage account firewall".

Beyond that, I'm not totally sure how to troubleshoot, but the first thing that comes to mind is the connection string. I'm not sure what to look for, but I know those can get quite complex... and maybe you're using a option that behaves differently in the old vs new sdk? Maybe you need to specify something in the connection string now?

Here's some docs on connection strings:
https://docs.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string

The second thing I would try is to create a totally new Azure Blob Container, using the default settings, and a very basic connection string. That should work (that's what we do), and then try to compare/contrast the differences between the containers.

Please let me know what you find!

Cheers,
Alana

brett.polivka

I created a brand new storage account, copied over the connection string, and still receive the same errors.

Looking at the http traffic, it's able to upload the blob without issue but is failing when it's trying to copy the uploaded blob to it's final destination.

Setting the account and containers to public is not an option.

brett.polivka

I think I know the problem.

My guess is that you're performing the copy operation specifying the normal URI of the source blob, which is the public URI. Instead, you need to use the Shared Access Signature (SAS) URI. This can be obtained via the GenerateSasUri method on the BlobClient.

brett.polivka

As a workaround I set the account and containers to public but limited access via a private endpoint. Not ideal, but better than nothing.

Now I'm able to push the docker layers, but after they are pushed, I receive a 500 Internal Server Error from the server. The logs show:

A 500 error occurred in docker: Object reference not set to an instance of an object.
A 500 error occurred in docker: Object reference not set to an instance of an object.
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
Request finished HTTP/1.1 PUT http://<redacted>/v2/docker/busybox/manifests/latest application/vnd.docker.distribution.manifest.v2+json 527 - 500 109 application/json 21769.4992ms

atripp

Thanks much @brett-polivka !

That's a great tip! That just might be it, and I can see the code here just uses Uri, as opposed to a Sas URI:
https://github.com/Inedo/inedox-azure/blob/master/Azure/InedoExtension/FileSystems/AzureFileSystem.cs#L91

For that error, can you see what ProGet logged in Admin > Diagnostic Center?

brett.polivka

There's nothing in the diagnostic logs beyond "Object reference not set to an instance of an object."

Dan_Woolf

Hi @brett-polivka,

We are currently looking into this issue and we should have an update for you soon.

Thanks,
Dan

gdivis

Hi @brett-polivka,

We've published v2.0.1 of the Azure extension for ProGet, which should resolve this issue. You can install it from the Admin->Extensions page.

Let us know if you are still having the issue after updating the extension.

Thank you!
-Greg