Baby steps! I still see no entries in the diagnostics center regarding vulnerabilities.

Thanks,
Scott

Hi Rich,

Still is failing, different reason:

{"Event":"could not download layer: expected 2XX","Level":"warning","Location":"driver.go:136","Time":"2021-09-20 19:01:26.736879","status code":404}
{"Event":"failed to extract data from path","Level":"error","Location":"worker.go:122","Time":"2021-09-20 19:01:26.737319","error":"could not find layer","layer":"sha256:f033c4f65cdbf0bfa21d5543e56c0c41645eca4d893494bb4f0661b0f19ccc79","path":"https://<proget_server>/api/docker-blobs/download/sha256%3Af033c4f65cdbf0bfa21d5543e56c0c41645eca4d893494bb4f0661b0f19ccc79"}
{"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2021-09-20 19:01:26.737385","elapsed time":166008805,"method":"POST","remote addr":"<proget_web_node>:59778","request uri":"/v1/layers","status":"400"}

Thanks,
Scott

Hi Dan,

I have updated Clair to 2.1.7. wget appears to ping ProGet successfully (curl as well), however, the vulnerability downloader fails to authenticate:

Inside Clair container:

curl -I https://<proget_server>
HTTP/2 200
cache-control: private
content-length: 23531
content-type: text/html; charset=UTF-8
server: Microsoft-IIS/10.0
x-aspnet-version: 4.0.30319
x-proget-version: 5.3.36.4
x-powered-by: ASP.NET
date: Mon, 20 Sep 2021 17:42:08 GMT

wget --spider https://<proget_server>
Connecting to <proget_server> (<proget_cluster_IP>:443)
remote file exists
/tmp # read escape sequence

On build machine

docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3a8a5c58b8cc quay.io/coreos/clair:v2.1.7 "/usr/bin/dumb-init …" 11 minutes ago Up 6 minutes zealous_jemison
e794f6f91d08 postgres:9.6 "docker-entrypoint.s…" 10 days ago Up 10 days 0.0.0.0:5432->5432/tcp, :::5432->5432/tcp charming_gould
[root@optjenkinscent09 ~]# docker attach 3a8a5c58b8cc
{"Event":"could not download layer","Level":"warning","Location":"driver.go:130","Time":"2021-09-20 17:43:52.834425","error":"Get https://<proget_server>/api/docker-blobs/download/sha256%3A80369df487363e56aea88d4d41b61f1607fc2ec198e9327cfde36a5346c71bf2?API_Key=0E-6E-66-D2-5B-CE-CB-78-1A-C0-62-1B-E9-50-72-59-54-50-90-41-43-AB-F7-55-B5: x509: certificate signed by unknown authority"}
{"Event":"failed to extract data from path","Level":"error","Location":"worker.go:122","Time":"2021-09-20 17:43:52.834515","error":"could not find layer","layer":"sha256:80369df487363e56aea88d4d41b61f1607fc2ec198e9327cfde36a5346c71bf2","path":"https://<proget_server>/api/docker-blobs/download/sha256%3A80369df487363e56aea88d4d41b61f1607fc2ec198e9327cfde36a5346c71bf2"}
{"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2021-09-20 17:43:52.834572","elapsed time":91078190,"method":"POST","remote addr":"<proget_web_node>:59389","request uri":"/v1/layers","status":"400"}

Maybe there is a process in the Clair container that can't detect our trusted CA cert.

Hope this helps.

Thanks,
Scott

Hi,

Adding a little more info. I installed curl in the clair container and it comes back with success when wget does not.

curl -I https://<proget_server>
HTTP/2 200
cache-control: private
content-length: 22803
content-type: text/html; charset=UTF-8
server: Microsoft-IIS/10.0
x-aspnet-version: 4.0.30319
x-proget-version: 5.3.36.4
x-powered-by: ASP.NET
date: Mon, 20 Sep 2021 15:10:11 GMT

Thanks,
Scott

Hi Dan,

The Diagnostic Center does not contain any warnings or errors regarding vulnerabilities.

Thanks,
Scott

No problem Rich and thanks for looking into this. I think we will both learn something in the end. This is not a high priority at the moment, fyi.

Thanks,
Scott

Hi Rich,

Unfortunately, I am not having any success. I have installed our trusted cert in the clair container and wget still fails ssl verification. I see the cert in /etc/ssl/certs ca-cert-USWILCA01v.pem -> /usr/local/share/ca-certificates/USWILCA01v.crt This trusted cert is installed on a mixture of Win\Linux environments (vms, local, and docker images) without issue.

Any other advice? Is this a futile attempt over https? :)

Thanks,
Scott

Hi Rich,

The Web.BaseUrl can be pinged from the Clair image. Our ProGet cluster is 443, but the exposed ports of the Clair container 6060 and 6061 (health check). Our Clair API URL where Clair is running is http://<Clair_Host_IP>:6060. Since our cluster is 443, do we have to install our trusted cert inside the Clair container? When I run wget from inside the Clair container (which is running apline linux):

wget <Web.BaseUrl>
Connecting to <Web.BaseUrl> (<Web.BaseUrl_IP>:443)
ssl_client: <Web.BaseUrl>: certificate verification failed: unable to get local issuer certificate
wget: error getting response: Connection reset by peer

API Authorization Header is null
We do allow anonymous pulls from ProGet

Thanks,
Scott

Hi Rich,

I guess clair had an issue pulling all layers since I do not have a scan log to provide. Is there a way to verify a valid connection from the clair server other than getting a valid ping to the ProGet cluster? Or vice versa?

Thanks,
Scott

I also want to mention that this is in a NLB high availability environment which contains 3 web nodes.

Hi,

Yes. ProGet can be pinged from the clair server and Web.BaseUrl is set correctly.

Thanks,
Scott

Hi Rich,

What is your current version of ProGet: 5.3.36
What is your current version of the Clair extension: 1.9.0
What version of Clair are you running: 2.1.2
Is this something new that has recently started happening? Yes. Just setup the integration last week
Does it give you that warning for all layers or just some specific layers? There are ~69K WARN entries in the job output so I am thinking all layers. But without more log info from ProGet, I can't be 100% sure
Would it be possible to share any sort of example image from a third-party registry that has this issue? Sorry, no.

Hi,

We have setup Clair integration with ProGet and connections appears ok. However, the only log entries from running the VulnerabilityDownloader job are WARN: BadRequest for layer sha256:<Layer_ID>. What can I do to diagnose these warnings?

Thanks,
Scott

Just for reference, I was able to get the correct digest of a tag and delete it with this PowerShell script...

$image = "some_image"
$some_image_tag = "some_tag"

get digest

$curlReturn = curl -I HEAD https://<proget_repo>/v2/<container>/library/$image/manifests/$some_image_tag
foreach ($line in $curlReturn)
{
if ($line -match "Docker-Content-Digest: (.+)")
{
$digest = $Matches[1]
Write-Host "$image digest is $digest"
}
}

delete tag digest

$URI = "https://<proget_repo>/v2/<container>/library/$image/manifests/$digest"
Write-Host "Deleting tag $some_image_tag from image $image"
Invoke-WebRequest -Uri $URI -Method Delete -Headers @{"X-ApiKey"="<proget_api_key>"; }

Thanks for the reply!

By the way, do you have preliminary release notes for 5.3 that we can look at?

Thanks,
Scott

Hi,

I am trying to obtain the correct digest of a particular docker tag and have seen two methods, with different outputs (see below), to accomplish this.  Which is correct?  From your documentation, I need to get the digest associated with 'Docker-Content-Digest', which is option 2.  However, documentation doesn't explicitly mention using curl (unless it's assumed?).

1. place 'https://<proget_repo>/v2/<container>/library/<image_name>/manifests/<tag>' in a web browser. Alternatively, use Powershell's Invoke-WebRequest

2. type 'curl -v -H Accept: "application/vnd.docker.distribution.manifest.v2+json" -X HEAD https://<proget_repo>/v2/<container>/library/<image_name>/manifests/<tag>' on the command line

Option 1 output:
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 4536,
"digest": "sha256:d8622010e224b5c7dc541c0e8843b993fd5c5afe422ed1309aac207bd06885a7"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 22524609,
"digest": "sha256:804555ee037604c40de144f9f8da0d826d38db82f15d74cded32790fe279a8f6"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 17692725,
"digest": "sha256:970251047358aea56ba6db6975b14ff12470b75de0c2477f4445240ddd727fd4"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 2978257,
"digest": "sha256:f3d4c41a4fd13f35c0b46f19a4e27845f4695163cc7174d908ff84836bbc2f5a"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 62145592,
"digest": "sha256:bd391c46585f9f8d84992bbaa9087189148c1601968eaaf097d5b3ed60840e5e"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 126,
"digest": "sha256:1674b9537e7545c5665f935cab34e05e09971a88dab2b86fdf00e516089708f5"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 25187509,
"digest": "sha256:51ecaa038642c6c35b4be45cf823e06d5f3f50f8f3f5e9cfb52c6b5151a57dc5"
}
]
}

Option 2 output:

Warning: Setting custom HTTP method to HEAD with -X/--request may not work the
Warning: way you want. Consider using -I/--head instead.

• Could not resolve host: application
• Closing connection 0
  curl: (6) Could not resolve host: application
  Warning: Setting custom HTTP method to HEAD with -X/--request may not work the
  Warning: way you want. Consider using -I/--head instead.
• Trying 10.2.0.87...
• TCP_NODELAY set
• Connected to <proget_repo> port 443 (#1)
• schannel: SSL/TLS connection with <proget_repo> port 443 (step 1/3)
• schannel: checking server certificate revocation
• schannel: sending initial handshake data: sending 190 bytes...
• schannel: sent initial handshake data: sent 190 bytes
• schannel: SSL/TLS connection with <proget_repo> port 443 (step 2/3)
• schannel: encrypted data got 1842
• schannel: encrypted data buffer: offset 1842 length 4096
• schannel: sending next handshake data: sending 158 bytes...
• schannel: SSL/TLS connection with <proget_repo> port 443 (step 2/3)
• schannel: encrypted data got 51
• schannel: encrypted data buffer: offset 51 length 4096
• schannel: SSL/TLS handshake complete
• schannel: SSL/TLS connection with <proget_repo> port 443 (step 3/3)
• schannel: stored credential handle in session cache

HEAD /v2/<container>/library/<image>/manifests/<tag> HTTP/1.1
Host: <proget_repo>
User-Agent: curl/7.55.1

• schannel: client wants to read 102400 bytes
• schannel: encdata_buffer resized 103424
• schannel: encrypted data buffer: offset 0 length 103424
• schannel: encrypted data got 483
• schannel: encrypted data buffer: offset 483 length 103424
• schannel: decrypted data length: 454
• schannel: decrypted data added: 454
• schannel: decrypted data cached: offset 454 length 102400
• schannel: encrypted data buffer: offset 0 length 103424
• schannel: decrypted data buffer: offset 454 length 102400
• schannel: schannel_recv cleanup
• schannel: decrypted data returned 454
• schannel: decrypted data buffer: offset 0 length 102400
  < HTTP/1.1 200 OK
  < Cache-Control: public, max-age=300
  < Content-Length: 1583
  < Content-Type: application/vnd.docker.distribution.manifest.v2+json
  < Vary: Authorization
  < Server: Microsoft-IIS/10.0
  < X-AspNet-Version: 4.0.30319
  < X-ProGet-Version: 5.2.23.7
  < Docker-Distribution-API-Version: registry/2.0
  < Docker-Content-Digest: sha256:219db0d935303e3d454231d6146ee56d658c8399f10de95648cdd91379139ce5
  < X-Powered-By: ASP.NET
  < Date: Tue, 11 Feb 2020 19:25:03 GMT
  <
• schannel: client wants to read 1583 bytes
• schannel: encrypted data buffer: offset 0 length 103424
• schannel: Curl_read_plain returned CURLE_RECV_ERROR
• schannel: encrypted data buffer: offset 0 length 103424
• schannel: encrypted data buffer: offset 0 length 103424
• schannel: decrypted data buffer: offset 0 length 102400
• schannel: schannel_recv cleanup
• Closing connection 1
• schannel: shutting down SSL/TLS connection with <proget_repo> port 443
• Send failure: Connection was reset
• schannel: failed to send close msg: Failed sending data to the peer (bytes written: -1)
• schannel: clear security context handle

Do I run option 1: 'DELETE /v2/<container>/library/<image>/manifestsd8622010e224b5c7dc541c0e8843b993fd5c5afe422ed1309aac207bd06885a7'

or option 2 (Docker-Content-Digest)

'DELETE /v2/<container>/library/<image>/manifests/219db0d935303e3d454231d6146ee56d658c8399f10de95648cdd91379139ce5'

Thanks,
Scott

Thanks for the update. I am looking forward to 5.3! :)

Hi and thank you for the reply! From what I have read, no Delete tag API exists. At least not yet. It appears that it may be in a future docker release.

https://github.com/docker/distribution/pull/2169

What I am looking for is what is outlined on this page:

https://serverfault.com/questions/872332/easy-way-to-remove-a-tag-from-a-private-docker-registry/872336

ideally: DELETE http://<docker_repo>/v2/<image>/<tag>

But in the meantime, it would be great if we could script how ProGet accomplished this in the UI. Although, ProGet's terminology is delete package\image to delete. Maybe it can be delete image\tag to delete? Also, assuming if you delete the last tag, the entire image is deleted as well.

Please correct me if I am wrong :)

Scott

Hi,

Can anyone from Inedo chime in?

Thanks,
Scott