ProGet - Error Pulling from Docker Repository with Multiple Servers

[james.woods_8996]

I'm testing an installation of ProGet on Openshift Kubernetes using Linux pods. The database is SQL Server. We have no shared disk storage provisioned as we want to exclusively use S3 for artifact storage. Testing a Docker feed with an AWS S3 bucket with only one pod in our cluster worked fine. Here are the commands and the logs for that:

Command Line
PS C:\Users\james.woods> podman pull proget-au.wtg.zone/cargowise-cloud-docker/hello:latest
Trying to pull proget-au.wtg.zone/cargowise-cloud-docker/hello:latest...
Getting image source signatures
Copying blob sha256:ce980a8f5545faa3125a489aad32c00d6cf13d80a302308c3963b524085657af
Copying config sha256:5dd467fce50b56951185da365b5feee75409968cbab5767b9b59e325fb2ecbc0
Writing manifest to image destination
5dd467fce50b56951185da365b5feee75409968cbab5767b9b59e325fb2ecbc0

Logs
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Request starting HTTP/1.1 GET http://proget-au.wtg.zone/v2/ - - -
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
Request finished HTTP/1.1 GET http://proget-au.wtg.zone/v2/ - 401 145 application/json 0.2699ms
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Request starting HTTP/1.1 GET http://proget-au.wtg.zone/v2/_auth?account=james.woods&scope=repository%3Acargowise-cloud-docker%2Fhello%3Apull&service=proget-au.wtg.zone - - -
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
Request finished HTTP/1.1 GET http://proget-au.wtg.zone/v2/_auth?account=james.woods&scope=repository%3Acargowise-cloud-docker%2Fhello%3Apull&service=proget-au.wtg.zone - 200 369 application/json 15.4765ms
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Request starting HTTP/1.1 GET http://proget-au.wtg.zone/v2/cargowise-cloud-docker/hello/manifests/latest - - -
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
Request finished HTTP/1.1 GET http://proget-au.wtg.zone/v2/cargowise-cloud-docker/hello/manifests/latest - 200 427 application/vnd.docker.distribution.manifest.v2+json 12.2290ms
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Request starting HTTP/1.1 GET http://proget-au.wtg.zone/v2/cargowise-cloud-docker/hello/blobs/sha256:5dd467fce50b56951185da365b5feee75409968cbab5767b9b59e325fb2ecbc0 - - -
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
Request finished HTTP/1.1 GET http://proget-au.wtg.zone/v2/cargowise-cloud-docker/hello/blobs/sha256:5dd467fce50b56951185da365b5feee75409968cbab5767b9b59e325fb2ecbc0 - 200 3319 application/octet-stream 99.8158ms
Running Feed Replication...
Feed Replication completed.

However as soon as we increase the number of pods to more than one the pull no longer works. Here are the commands and logs for that case:

Command Line
PS C:\Users\james.woods> podman pull proget-au.wtg.zone/cargowise-cloud-docker/hello:latest
Trying to pull proget-au.wtg.zone/cargowise-cloud-docker/hello:latest...
Error: initializing source docker://proget-au.wtg.zone/cargowise-cloud-docker/hello:latest: reading manifest latest in proget-au.wtg.zone/cargowise-cloud-docker/hello: received unexpected HTTP status: 500 Internal Server Error

Logs
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Request starting HTTP/1.1 GET http://proget-au.wtg.zone/v2/cargowise-cloud-docker/hello/manifests/latest - - -
A 500 error occurred in (unknown feed): The key {ba4cc1b5-2f99-49ac-bf9d-9f7d7f696431} was not found in the key ring. For more information go to https://aka.ms/aspnet/dataprotectionwarning
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
Request finished HTTP/1.1 GET http://proget-au.wtg.zone/v2/cargowise-cloud-docker/hello/manifests/latest - 500 204 application/json 0.7512ms

Is there some shared state that each server needs to be able to access?

[atripp]

Hi @james-woods_8996,

It sounds like there's a problem decrypting authentication tickets. This is almost certainly due to not having the same encryption key on both servers.

Check out https://docs.inedo.com/docs/installation/linux/docker-guide#supported-environment-variables to learn more, but it's an environment variable like this:

-e PROGET_ENCRYPTION_KEY='37D27A670394F7D82CE57F1F07D69747'

It's a 32-char hex string. You can just generate a GUID and delete the "-" to have one.

Thanks,
Alana

[james.woods_8996]

Thanks for that, I'd put an alphanumeric rather than a hex key in two of my sites. Once I changed them to hex then all started working.