Clair integration with ProGet results in 'BadRequest for layer sha256' warnings in VulnerabilityDownloader job

[scusson_9923]

Hi,

We have setup Clair integration with ProGet and connections appears ok. However, the only log entries from running the VulnerabilityDownloader job are WARN: BadRequest for layer sha256:<Layer_ID>. What can I do to diagnose these warnings?

Thanks,
Scott

[rhessinger]

Hi @scusson_9923,

I'm sorry to bombard you with some questions, but I think this will be a good way to start.

• What is your current version of ProGet?
• What is your current version of the Clair extension?
• What version of Clair are you running?
• Is this something new that has recently started happening?
• Does it give you that warning for all layers or just some specific layers?
• Would it be possible to share any sort of example image from a third-party registry that has this issue?

If you can answer those for me, that should give us a good start to resolve this issue.

Thanks,
Rich

[scusson_9923]

Hi Rich,

What is your current version of ProGet: 5.3.36
What is your current version of the Clair extension: 1.9.0
What version of Clair are you running: 2.1.2
Is this something new that has recently started happening? Yes. Just setup the integration last week
Does it give you that warning for all layers or just some specific layers? There are ~69K WARN entries in the job output so I am thinking all layers. But without more log info from ProGet, I can't be 100% sure
Would it be possible to share any sort of example image from a third-party registry that has this issue? Sorry, no.

[rhessinger]

Hi @scusson_9923,

Thanks for answering those. Let me do some checks in our lab and check back. The integration will always yield some warnings because not everything comes back in the expected layer format (configuration layers is a good example of this), but Clair randomly makes changes to their providers and their API and I want to make sure the test cases still work as expected.

Thanks.
Rich

[rhessinger]

Hi @scusson_9923,

Two other things I forgot to ask.

Is your ProGet server accessible from your clair server? How the integration works, is ProGet sends a list of images to scan to Clair, which Clair then downloads the image layers from ProGet and scans them. Then ProGet calls back to clair to get the results.

Do you have your Web.BaseUrl set in the Advanced Settings? Because this scan runs from the ProGet Service, the Web.BaseUrl needs to be set so we know what URL to send to Clair to download the image layer.

Thanks,
Rich

[scusson_9923]

Hi,

Yes. ProGet can be pinged from the clair server and Web.BaseUrl is set correctly.

Thanks,
Scott

[scusson_9923]

I also want to mention that this is in a NLB high availability environment which contains 3 web nodes.

[rhessinger]

Hi @scusson_9923,

Looks like I misspoke earlier, the Clair integration will never return warnings unless an error happened while pulling an image. Would you be able to send a copy of the Vulnerability Scan logs to support@inedo.com with a subject of [QA-664] Clair Logs, so I can review the logs?

Thanks,
Rich

[scusson_9923]

Hi Rich,

I guess clair had an issue pulling all layers since I do not have a scan log to provide. Is there a way to verify a valid connection from the clair server other than getting a valid ping to the ProGet cluster? Or vice versa?

Thanks,
Scott

[rhessinger]

Hi @scusson_9923,

When you ping the server from the Clair image, are you pinging the server or the value that is in Web.BaseUrl? You should be using the value within Web.BaseUrl since that is the connection it is using. Also, are you using a port other than 443 for your ProGet cluster for the Clair connection?

Also, on your Clair configuration, do you have anything set for API Authorization Header? If so, could you try to remove that and see if that fixes your issue?

Lastly, does your Docker registry allow anonymous to pull your images? If not, could you temporarily allow anonymous access to that registry and give it a try? This will allow us to see if it is an issue with our automatic key creation logic for Docker images.

That would be the starting point I think to troubleshoot this. If that doesn't resolve the issue, then the next step would be to do some custom PowerShell calls to do a direct test with Clair.

I'm sorry for all the back and forth with this, but there definitely seems to be something blocking the connection, so now we just need to see which portion of the system is blocking it.

Thanks,
Rich

[scusson_9923]

Hi Rich,

The Web.BaseUrl can be pinged from the Clair image. Our ProGet cluster is 443, but the exposed ports of the Clair container 6060 and 6061 (health check). Our Clair API URL where Clair is running is http://<Clair_Host_IP>:6060. Since our cluster is 443, do we have to install our trusted cert inside the Clair container? When I run wget from inside the Clair container (which is running apline linux):

wget <Web.BaseUrl>
Connecting to <Web.BaseUrl> (<Web.BaseUrl_IP>:443)
ssl_client: <Web.BaseUrl>: certificate verification failed: unable to get local issuer certificate
wget: error getting response: Connection reset by peer

API Authorization Header is null
We do allow anonymous pulls from ProGet

Thanks,
Scott

[rhessinger]

Hi @scusson_9923,

That is definitely the issue. It looks like the best way is to add your self-signed cert to the ca and add a docker mount to that (-v /path/to/quay/cert/ca.crt:/etc/pki/ca-trust/source/anchors/ca.crt). You may be able to do it with the Clair config also, but I could not find anything easily for that.

Thanks,
Rich

[scusson_9923]

Hi Rich,

Unfortunately, I am not having any success. I have installed our trusted cert in the clair container and wget still fails ssl verification. I see the cert in /etc/ssl/certs ca-cert-USWILCA01v.pem -> /usr/local/share/ca-certificates/USWILCA01v.crt This trusted cert is installed on a mixture of Win\Linux environments (vms, local, and docker images) without issue.

Any other advice? Is this a futile attempt over https? :)

Thanks,
Scott

[rhessinger]

Hi @scusson_9923,

Please give me a little bit of time to work through this. If I have learned anything about Docker, it is that certificates are handled differently on every image. I need to do some digging to find out what is needed to make this work. I don't think HTTPS is a lost cause, we just need to figure out how Clair needs to handle these certs.

Thanks,
Rich

[scusson_9923]

No problem Rich and thanks for looking into this. I think we will both learn something in the end. This is not a high priority at the moment, fyi.

Thanks,
Scott

[Dan_Woolf]

Hi @scusson_9923 ,

Can you please confirm that you do not see any errors in the diagnostic center after running a vulnerability scan? I just want to confirm that this is an SSL problem and not an underlying ProGet issue.

Thanks,
Dan

[scusson_9923]

Hi Dan,

The Diagnostic Center does not contain any warnings or errors regarding vulnerabilities.

Thanks,
Scott

[scusson_9923]

Hi,

Adding a little more info. I installed curl in the clair container and it comes back with success when wget does not.

curl -I https://<proget_server>
HTTP/2 200
cache-control: private
content-length: 22803
content-type: text/html; charset=UTF-8
server: Microsoft-IIS/10.0
x-aspnet-version: 4.0.30319
x-proget-version: 5.3.36.4
x-powered-by: ASP.NET
date: Mon, 20 Sep 2021 15:10:11 GMT

Thanks,
Scott

[Dan_Woolf]

Hi @scusson_9923 ,

Thanks for sending the updated info. Would you be able to attach to output of the Clair container (docker attach <RUNNING_CLAIR_CONTAINER_NAME>) and run the vulnerability downloader from ProGet? Clair will output the exact HTTP error that is happening when connecting to ProGet. That will help to confirm if it is an SSL issue or not.

One other thing to try would be to upgrade Clair to 2.1.7. I know they fixed an issue relating to a RHEL feature they use, that happened to be affecting my test environment.

Sorry for the delay on our end. Clair has released a new version and API which makes finding the older documentation a bit harder. We have not tested it with the latest version of Clair (4.2.2), but I would expect their to be some API issues. We are waiting for Clair's development to stabilize a bit more before we implement the new API. Last time we tested it, Clair's v4 API had so many bugs that it prevented us from supporting it.

Thanks,
Dan

[scusson_9923]

Hi Dan,

I have updated Clair to 2.1.7. wget appears to ping ProGet successfully (curl as well), however, the vulnerability downloader fails to authenticate:

Inside Clair container:

curl -I https://<proget_server>
HTTP/2 200
cache-control: private
content-length: 23531
content-type: text/html; charset=UTF-8
server: Microsoft-IIS/10.0
x-aspnet-version: 4.0.30319
x-proget-version: 5.3.36.4
x-powered-by: ASP.NET
date: Mon, 20 Sep 2021 17:42:08 GMT

wget --spider https://<proget_server>
Connecting to <proget_server> (<proget_cluster_IP>:443)
remote file exists
/tmp # read escape sequence

On build machine

docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3a8a5c58b8cc quay.io/coreos/clair:v2.1.7 "/usr/bin/dumb-init …" 11 minutes ago Up 6 minutes zealous_jemison
e794f6f91d08 postgres:9.6 "docker-entrypoint.s…" 10 days ago Up 10 days 0.0.0.0:5432->5432/tcp, :::5432->5432/tcp charming_gould
[root@optjenkinscent09 ~]# docker attach 3a8a5c58b8cc
{"Event":"could not download layer","Level":"warning","Location":"driver.go:130","Time":"2021-09-20 17:43:52.834425","error":"Get https://<proget_server>/api/docker-blobs/download/sha256%3A80369df487363e56aea88d4d41b61f1607fc2ec198e9327cfde36a5346c71bf2?API_Key=0E-6E-66-D2-5B-CE-CB-78-1A-C0-62-1B-E9-50-72-59-54-50-90-41-43-AB-F7-55-B5: x509: certificate signed by unknown authority"}
{"Event":"failed to extract data from path","Level":"error","Location":"worker.go:122","Time":"2021-09-20 17:43:52.834515","error":"could not find layer","layer":"sha256:80369df487363e56aea88d4d41b61f1607fc2ec198e9327cfde36a5346c71bf2","path":"https://<proget_server>/api/docker-blobs/download/sha256%3A80369df487363e56aea88d4d41b61f1607fc2ec198e9327cfde36a5346c71bf2"}
{"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2021-09-20 17:43:52.834572","elapsed time":91078190,"method":"POST","remote addr":"<proget_web_node>:59389","request uri":"/v1/layers","status":"400"}

Maybe there is a process in the Clair container that can't detect our trusted CA cert.

Hope this helps.

Thanks,
Scott

[rhessinger]

Hi @scusson_9923,

I was just researching this a bit and it looks like they may have added a toggle to disable SSL checks in Clair when downloading docker layers. Can you try adding -insecure-tls to your docker run statement for Clair?

Thanks,
Rich

[scusson_9923]

Hi Rich,

Still is failing, different reason:

{"Event":"could not download layer: expected 2XX","Level":"warning","Location":"driver.go:136","Time":"2021-09-20 19:01:26.736879","status code":404}
{"Event":"failed to extract data from path","Level":"error","Location":"worker.go:122","Time":"2021-09-20 19:01:26.737319","error":"could not find layer","layer":"sha256:f033c4f65cdbf0bfa21d5543e56c0c41645eca4d893494bb4f0661b0f19ccc79","path":"https://<proget_server>/api/docker-blobs/download/sha256%3Af033c4f65cdbf0bfa21d5543e56c0c41645eca4d893494bb4f0661b0f19ccc79"}
{"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2021-09-20 19:01:26.737385","elapsed time":166008805,"method":"POST","remote addr":"<proget_web_node>:59778","request uri":"/v1/layers","status":"400"}

Thanks,
Scott

[rhessinger]

Hi @scusson_9923,

Progress! It looks like we are past the SSL issue now. Can you check the diagnostics center in ProGet and see if there are any errors in there now?

Thanks,
Rich

[scusson_9923]

Baby steps! I still see no entries in the diagnostics center regarding vulnerabilities.

Thanks,
Scott

[rhessinger]

Hi @scusson_9923,

What happens if you try to wget https://<proget_server>/api/docker-blobs/download/sha256%3Af033c4f65cdbf0bfa21d5543e56c0c41645eca4d893494bb4f0661b0f19ccc79 from the Clair container? Does that also return a 404 error? Just to confirm, all the requests in the Vulnerability log are warnings still correct?

Thanks,
Rich

[scusson_9923]

Hi,

I get 'wget: server returned error: HTTP/1.1 403 Forbidden'

When I try curl with a ProGet user\password, I get 'An API key is required to download Docker blobs.'

I saw the API key in the previous logs (container without -insecure-tls), but not now.

Thanks,
Scott

[rhessinger]

Hi @scusson_9923,

Could you please generate a temporary API Key and try wget with https://<proget_server>/api/docker-blobs/download/sha256%3Af033c4f65cdbf0bfa21d5543e56c0c41645eca4d893494bb4f0661b0f19ccc79?API_Key=<API_KEY> from the container? Can you also try that from your browser (it should try to download the file)?

It is throwing me off that you are getting a 404 for all the layer download requests. It sounds like either ProGet cannot find the layer, which should show an error in the log, or that Clair is calling to the wrong server for the download the layer.

I apologize for all the back and forth with this. This is the first time we have experienced this with Clair and I'm still trying to determine which component has the issue.

We are currently running Clair on our ProGet.inedo.com and it doesn't seem to have this issue. I'm also not able to recreate this locally, which makes this a bit more difficult.

Thanks,
Rich

[scusson_9923]

Hi Rich,

I created an API key with full access and still get wget: server returned error: HTTP/1.1 403 Forbidden.

When I place the same link in my local browser, I get:
The specified API key does not permit access to the Docker Blob Reader API.

In the Access Key Log:
/api/docker-blobs/download/sha256:f033c4f65cdbf0bfa21d5543e56c0c41645eca4d893494bb4f0661b0f19ccc79?API_Key=<apikey>
Response code: 403

As a test, I used wget to download a file from an Assets directory inside the Clair container successfully.

Thanks,
Scott

[rhessinger]

This looks to be an issue when common blob storage is not enabled for a Docker registry. I have created a ticket, PG-2009, to track this fix. It is expected to be released in ProGet 5.3.39 which is scheduled to release on October 8th, 2021. I will post back here if anything changes.

There was also an issue, PG-2008, that was fixed 5.3.38 that would sometimes return a 500 error to Calair when trying to download the layer. PG-2008 seems to only affect ProGet running on Linux.

Thanks,
Rich

[furqan.ahmad_4378]

@rhessinger Did you get a chance to apply update on Proget, Does that fix that issue? I'm also getting the same error: {"Event":"could not download layer: expected 2XX","Level":"warning","Location":"driver.go:136","Time":"2021-12-14 02:33:01.975151","status code":404}
{"Event":"failed to extract data from path","Level":"error","Location":"worker.go:122","Time":"2021-12-14 02:33:01.975206","error":"could not find layer","layer" docker-blobs.

Looking forward to hearing from you.

Thanks,
Furqan

[rhessinger]

Hi @furqan-ahmad_4378,

ProGet 5.3.39 and later include the fixes for these issues. We also improved the performance of the list repositories page in ProGet 6.0.3+. What version of ProGet are you currently running? Also, is this happening for every layer?

Thanks,
Rich

[furqan.ahmad_4378]

@rhessinger Thank you for sharing this information, We are on 5.3.27 at the moment. Yup its on every docker layer :(

[furqan.ahmad_4378]

Hi @rhessinger May I ask you which Clair docker Image you are on?

[rhessinger]

Hi @furqan-ahmad_4378,

So 5.3.27 definitely has the bug. The bug affected Docker Registries not using shared blob storage. Beginning in ProGet 5.3, all new Docker Registries will use common blob by default, but any existing registries that were upgraded do not. This caused the vulnerability scanner to not find the layers during the scan when looking at images in a registry not using common blob storage.

I'm currently testing with the quay.io/coreos/clair:v2.1.7 image. We have not implemented Clair's newest API yet because it does not seem to be stable as of yet. They are still making too many changes too frequently.

Hope this helps!

Thanks,
Rich

[furqan.ahmad_4378]

@rhessinger Thank you, Much appreciated. :)