Clair integration with ProGet results in 'BadRequest for layer sha256' warnings in VulnerabilityDownloader job

rhessinger

I was just researching this a bit and it looks like they may have added a toggle to disable SSL checks in Clair when downloading docker layers. Can you try adding -insecure-tls to your docker run statement for Clair?

Thanks,
Rich

scusson_9923

Hi Rich,

Still is failing, different reason:

{"Event":"could not download layer: expected 2XX","Level":"warning","Location":"driver.go:136","Time":"2021-09-20 19:01:26.736879","status code":404}
{"Event":"failed to extract data from path","Level":"error","Location":"worker.go:122","Time":"2021-09-20 19:01:26.737319","error":"could not find layer","layer":"sha256:f033c4f65cdbf0bfa21d5543e56c0c41645eca4d893494bb4f0661b0f19ccc79","path":"https://<proget_server>/api/docker-blobs/download/sha256%3Af033c4f65cdbf0bfa21d5543e56c0c41645eca4d893494bb4f0661b0f19ccc79"}
{"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2021-09-20 19:01:26.737385","elapsed time":166008805,"method":"POST","remote addr":"<proget_web_node>:59778","request uri":"/v1/layers","status":"400"}

Thanks,
Scott

rhessinger

Hi @scusson_9923,

Progress! It looks like we are past the SSL issue now. Can you check the diagnostics center in ProGet and see if there are any errors in there now?

Thanks,
Rich

scusson_9923

Baby steps! I still see no entries in the diagnostics center regarding vulnerabilities.

Thanks,
Scott

rhessinger

Hi @scusson_9923,

What happens if you try to wget https://<proget_server>/api/docker-blobs/download/sha256%3Af033c4f65cdbf0bfa21d5543e56c0c41645eca4d893494bb4f0661b0f19ccc79 from the Clair container? Does that also return a 404 error? Just to confirm, all the requests in the Vulnerability log are warnings still correct?

Thanks,
Rich

scusson_9923

Hi,

I get 'wget: server returned error: HTTP/1.1 403 Forbidden'

When I try curl with a ProGet user\password, I get 'An API key is required to download Docker blobs.'

I saw the API key in the previous logs (container without -insecure-tls), but not now.

Thanks,
Scott

rhessinger

Hi @scusson_9923,

Could you please generate a temporary API Key and try wget with https://<proget_server>/api/docker-blobs/download/sha256%3Af033c4f65cdbf0bfa21d5543e56c0c41645eca4d893494bb4f0661b0f19ccc79?API_Key=<API_KEY> from the container? Can you also try that from your browser (it should try to download the file)?

It is throwing me off that you are getting a 404 for all the layer download requests. It sounds like either ProGet cannot find the layer, which should show an error in the log, or that Clair is calling to the wrong server for the download the layer.

I apologize for all the back and forth with this. This is the first time we have experienced this with Clair and I'm still trying to determine which component has the issue.

We are currently running Clair on our ProGet.inedo.com and it doesn't seem to have this issue. I'm also not able to recreate this locally, which makes this a bit more difficult.

Thanks,
Rich

scusson_9923

Hi Rich,

I created an API key with full access and still get wget: server returned error: HTTP/1.1 403 Forbidden.

When I place the same link in my local browser, I get:
The specified API key does not permit access to the Docker Blob Reader API.

In the Access Key Log:
/api/docker-blobs/download/sha256:f033c4f65cdbf0bfa21d5543e56c0c41645eca4d893494bb4f0661b0f19ccc79?API_Key=<apikey>
Response code: 403

As a test, I used wget to download a file from an Assets directory inside the Clair container successfully.

Thanks,
Scott

rhessinger

This looks to be an issue when common blob storage is not enabled for a Docker registry. I have created a ticket, PG-2009, to track this fix. It is expected to be released in ProGet 5.3.39 which is scheduled to release on October 8th, 2021. I will post back here if anything changes.

There was also an issue, PG-2008, that was fixed 5.3.38 that would sometimes return a 500 error to Calair when trying to download the layer. PG-2008 seems to only affect ProGet running on Linux.

Thanks,
Rich

furqan.ahmad_4378

@rhessinger Did you get a chance to apply update on Proget, Does that fix that issue? I'm also getting the same error: {"Event":"could not download layer: expected 2XX","Level":"warning","Location":"driver.go:136","Time":"2021-12-14 02:33:01.975151","status code":404}
{"Event":"failed to extract data from path","Level":"error","Location":"worker.go:122","Time":"2021-12-14 02:33:01.975206","error":"could not find layer","layer" docker-blobs.

Looking forward to hearing from you.

Thanks,
Furqan

rhessinger

Hi @furqan-ahmad_4378,

ProGet 5.3.39 and later include the fixes for these issues. We also improved the performance of the list repositories page in ProGet 6.0.3+. What version of ProGet are you currently running? Also, is this happening for every layer?

Thanks,
Rich

furqan.ahmad_4378

@rhessinger Thank you for sharing this information, We are on 5.3.27 at the moment. Yup its on every docker layer :(

furqan.ahmad_4378

Hi @rhessinger May I ask you which Clair docker Image you are on?

rhessinger

Hi @furqan-ahmad_4378,

So 5.3.27 definitely has the bug. The bug affected Docker Registries not using shared blob storage. Beginning in ProGet 5.3, all new Docker Registries will use common blob by default, but any existing registries that were upgraded do not. This caused the vulnerability scanner to not find the layers during the scan when looking at images in a registry not using common blob storage.

I'm currently testing with the quay.io/coreos/clair:v2.1.7 image. We have not implemented Clair's newest API yet because it does not seem to be stable as of yet. They are still making too many changes too frequently.

Hope this helps!

Thanks,
Rich

furqan.ahmad_4378

@rhessinger Thank you, Much appreciated. :)