Welcome to the Inedo Forums! Check out the Forums Guide for help getting started.
If you are experiencing any issues with the forum software, please visit the Contact Form on our website and let us know!
Docker Repo null userName error
-
- Create a Docker Swarm using the new “Docker Swarm Mode”, built in since Docker 1.12. Swarm should have at least two nodes: 1 manager and one worker.
- Upload a Docker image to ProGet.
- SSH into the manager node and login to ProGet using:
docker login <proget host> - SSH into the manager node of the Swarm and try to create a service using the following command, replacing portions that are wrapped in <>:
docker service create –name <some service name> --replicas 1 --constraint 'node.hostname== <worker host name>' --with-registry-auth <image name>:<version> - Check the status of the service via the command line using:
docker service ps <service name>
Services that have failed or been rejected will give an indication as to why. In this case, the message reads: Unable to find image… - Check the ProGet logs, and you will see an error message stating:
An error occurred in the web application: Value cannot be null. Parameter name: username - Remove the test service using:
docker service rm <service name> - SSH into the worker node and pull the container image using:
docker pull <image name>:<version> - SSH into the manager node and try the create again, this time omitting the –with-registry-auth parameter.
- The service should be up and running correctly.
The details of the proget error are:
URL: https://repoman.kaneva.com/v1/repositories/kava-docker/library/config-service/images
Referrer: (not set)
User: (unknown)
User Agent: docker/1.13.0 go/go1.7.3 git-commit/49bf474 kernel/3.10.0-514.6.1.el7.x86_64 os/linux arch/amd64
Stack trace: at Inedo.ProGet.Extensibility.UserDirectories.BuiltInDirectory.TryGetAndValidateUser(String userName, String password)
at Inedo.ProGet.WebApplication.ProGetHttpModule.ValidateUser(HttpApplication app, String userName, String password)
at Inedo.Web.InedoHttpModule.AuthenticateRequestAsync(HttpApplication app)
at Inedo.Web.InedoHttpModule.ProcessBegin(Object sender, EventArgs e, AsyncCallback cb, Object extraData)
at System.Web.HttpApplication.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)Product: ProGet
Version: 4.6.7
-
I can't reproduce the problem you're having. Here's a log of what I tried with ProGet 4.7.4, Docker Machine 0.9.0, and Docker 1.13.1:
PS C:\> docker-machine create --driver virtualbox --engine-insecure-registry 192.168.1.100:83 swarm-manager Running pre-create checks... Creating machine... (swarm-manager) Copying C:\Users\Owner\.docker\machine\cache\boot2docker.iso to C:\Users\Owner\.docker\machine\machines\swarm-manager\boot2docker.iso... (swarm-manager) Creating VirtualBox VM... (swarm-manager) Creating SSH key... (swarm-manager) Starting the VM... (swarm-manager) Check network to re-create if needed... (swarm-manager) Windows might ask for the permission to create a network adapter. Sometimes, such confirmation window is minimized in the taskbar. (swarm-manager) Found a new host-only adapter: "VirtualBox Host-Only Ethernet Adapter #2" (swarm-manager) Windows might ask for the permission to configure a network adapter. Sometimes, such confirmation window is minimized in the taskbar. (swarm-manager) Windows might ask for the permission to configure a dhcp server. Sometimes, such confirmation window is minimized in the taskbar. (swarm-manager) Waiting for an IP... Waiting for machine to be running, this may take a few minutes... Detecting operating system of created instance... Waiting for SSH to be available... Detecting the provisioner... Provisioning with boot2docker... Copying certs to the local machine directory... Copying certs to the remote machine... Setting Docker configuration on the remote daemon... Checking connection to Docker... Docker is up and running! To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: C:\ProgramData\choco latey\lib\docker-machine\bin\docker-machine.exe env swarm-manager PS C:\> docker-machine create --driver virtualbox --engine-insecure-registry 192.168.1.100:83 swarm-worker Running pre-create checks... Creating machine... (swarm-worker) Copying C:\Users\Owner\.docker\machine\cache\boot2docker.iso to C:\Users\Owner\.docker\machine\machines\swarm-worker\boot2docker.iso... (swarm-worker) Creating VirtualBox VM... (swarm-worker) Creating SSH key... (swarm-worker) Starting the VM... (swarm-worker) Check network to re-create if needed... (swarm-worker) Waiting for an IP... Waiting for machine to be running, this may take a few minutes... Detecting operating system of created instance... Waiting for SSH to be available... Detecting the provisioner... Provisioning with boot2docker... Copying certs to the local machine directory... Copying certs to the remote machine... Setting Docker configuration on the remote daemon... Checking connection to Docker... Docker is up and running! To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: C:\ProgramData\chocolatey\lib\docker-machine\bin\docker-machine.exe env swarm-worker PS C:\> docker-machine env swarm-manager | iex PS C:\> docker swarm init --advertise-addr 192.168.99.100 time="2017-03-03T11:28:40-06:00" level=info msg="Unable to use system certificate pool: crypto/x509: system root pool is not available on Windows" Swarm initialized: current node (kb681q841gbm18jno2ktf157u) is now a manager. . To add a worker to this swarm, run the following command: . docker swarm join \ --token SWMTKN-1-2q02a4egqsnybt7mjb7108zipjwvony48c8iad2kk6dobzt4k8-6ozavuh9gr7vpk1brracn5h6o \ 192.168.99.100:2377 . To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions. . PS C:\> docker-machine env swarm-worker | iex PS C:\> docker swarm join --token SWMTKN-1-2q02a4egqsnybt7mjb7108zipjwvony48c8iad2kk6dobzt4k8-6ozavuh9gr7vpk1brracn5h6o 192.168.99.100:2377 time="2017-03-03T11:29:38-06:00" level=info msg="Unable to use system certificate pool: crypto/x509: system root pool is not available on Windows" This node joined a swarm as a worker. PS C:\> docker-machine env swarm-manager | iex PS C:\> docker pull nginx time="2017-03-03T11:31:42-06:00" level=info msg="Unable to use system certificate pool: crypto/x509: system root pool is not available on Windows" Using default tag: latest latest: Pulling from library/nginx 693502eb7dfb: Pull complete 6decb850d2bc: Pull complete c3e19f087ed6: Pull complete Digest: sha256:52a189e49c0c797cfc5cbfe578c68c225d160fb13a42954144b29af3fe4fe335 Status: Downloaded newer image for nginx:latest PS C:\> docker tag nginx 192.168.1.100:83/dockertest/nginx time="2017-03-03T11:33:11-06:00" level=info msg="Unable to use system certificate pool: crypto/x509: system root pool is not available on Windows" PS C:\> docker login 192.168.1.100:83 time="2017-03-03T11:33:20-06:00" level=info msg="Unable to use system certificate pool: crypto/x509: system root pool is not available on Windows" Username: Admin Password: Login Succeeded PS C:\> docker push 192.168.1.100:83/dockertest/nginx time="2017-03-03T11:33:40-06:00" level=info msg="Unable to use system certificate pool: crypto/x509: system root pool is not available on Windows" The push refers to a repository [192.168.1.100:83/dockertest/nginx] e3a86ffb1c45: Pushed cf0d2468e726: Pushed d17d48b2382a: Pushed latest: digest: sha256:32e3cf2703f4e8f99426884247f9e0d4d5acdde4914fe7df70f74c76ba6216fe size: 6300 PS C:\> docker service create --name q-6011 --replicas 1 --constraint 'node.hostname==swarm-worker' --with-registry-auth 192.168.1.100:83/dockertest/nginx:latest time="2017-03-03T11:35:29-06:00" level=info msg="Unable to use system certificate pool: crypto/x509: system root pool is not available on Windows" unable to pin image 192.168.1.100:83/dockertest/nginx:latest to digest: missing or empty Content-Length header 5x9ctqpt9d2duv8gblzpws4ed PS C:\> docker service ps q-6011 time="2017-03-03T11:36:19-06:00" level=info msg="Unable to use system certificate pool: crypto/x509: system root pool is not available on Windows" ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS lcsz1x2ndgda q-6011.1 192.168.1.100:83/dockertest/nginx:latest swarm-worker Running Running 58 seconds ago
-
Hey Ben,
Looking at your response to Ryan's question, it appears that you actually built the image on the worker node. When you did the "docker service create", it might have worked because the image was already present on the worker node. I'm seeing the same issue as Ryan, and that's the only way we've been able to work around it so far, by manually pulling the image on the worker before we do the call to create or update the service.
Looking deeper into the error messages, it appears that the Docker image digest on ProGet doesn't match what you get when you either push or pull the image. For example:
- On Product: ProGet Version: 4.6.7
- Running Docker 1.13.1
- Run "docker pull node:alpine"
- Run "docker tag node:alpine my-proget-repo:443/node-test"
- Run "docker push my-proget-repo:443/node-test"
- Take note of the SHA256 digest for the image.
- On ProGet, examine the SHA256 digest of the image by clicking on the "latest" tag in the repo. Note that it doesn't match the digest from the previous step.
- Delete your local copy of the image "docker rmi node:alpine && docker rmi my-proget-repo:443/node-test"
- Try to pull the image from ProGet using the digest specified in ProGet: "docker pull my-proget-repo:443/node-test:latest@sha256....". This should fail.
- Pull the image without specifying the digest. Note that the digest does not match what's specified in ProGet.
That's what we're actually seeing when we do a "docker service create". The manager node sends along the image digest from ProGet, and the worker node isn't able to verify it.
Any idea what could be causing this difference?
-
Ok, I've found the problem: Docker image manifests have a really complicated way to compute their digest:
- Parse the manifest as JSON, keeping key order.
- Remove the "signatures" key of the root object.
- Re-encode the new root object as JSON, with 3(!) space indents, LF newlines, and no newline at the end.
- Compute a SHA256 hashsum.
The problem was that on Windows, ProGet re-encoded the new root object with CRLF newlines.
This is fixed for the next version. (4.7.7)
