kubernetes scanner not showing results

cronventis

So I set up the kubernetes scanner to be able to see the usage of each image in a feed.
Although the feed is set up to use the scanner, and the scanner seems to work, there is no usage for an image, even though I'm sure there is.

So.. Has this feature been used by anyone yet? And is it working and I'm just doing something wrong?

Dan_Woolf

Hi @cronventis,

Can you please tell me what version of ProGet you are using and what version of the Kubernetes extension you have installed? I can confirm that we have users that have it working successfully. When you say the scanner seems to work, does that mean that some images are detected and some are not?

Thanks,
Dan

cronventis

I'm on the latest ProGet (Version 6.0.5 (Build 9)) and I'm using the latest Kubernetes extension (1.10.1).

It seems to work because of the logs:

Scanning Kubernetes API at https://xxx.xx
Beginning package/container scan using Kubernetes API at https://xxx.xx...
Package/Container Scanner completed.
Scan using Kubernetes API at https://xxx.xx complete; recording data...

Dan_Woolf

Hi @cronventis,

Do you see any records in the ContainerUsage database table in the ProGet database. When the scanner runs, it will dump records into that table and then the UI will display the records from there.

Thanks,
Dan

cronventis

Yes, there are 495 entries in that table...

rhessinger

Hi @cronventis,

Thanks for checking that for us. That does mean the Container Scanner is working. Did you happen to configure the container usage on your Docker Registry in ProGet? You can check this by navigating to Docker Registry -> Manage Feed -> Detection & Blocking and then you should see your container scanner under Container Usage Scanning.

If you have configured your feed to use that, the next step would be to look at what the Kubernetes scanner identified for your images and how ProGet has these images stored. If you look at your ContainerUsage table, you will need to look at two columns; Container_Name and Image_Id. The Container_Name is your repository's name and the Image_Id is the Digest for your image. Can you confirm that you see the container names you are expecting in that table?

Thanks,
Rich

cronventis

@rhessinger I think so. One example:

maybe its related to the library/ part?

rhessinger

Hi @cronventis,

Thank you for checking that for me. I dug into this further and it looks like we only actually use the Image_Id to find the image within a feed. We expect Kubernetes to return the Config Digest and that is what is stored in the Image_Id column. I'm going to work on creating this scenario and verify what is being pulled from Kubernetes. With the holidays coming up, I will not be able to look into this until next week. You should hear back from us by Tuesday or Wednesday of next week.

Thanks,
Rich

rhessinger

Hi @cronventis,

I set up an example in our test environment here and everything is working as expected. I did verify that the Image_Id that is in the database should match the Image Digest in the Image Data table as well.

Is this image a local image or a cached image? Currently, usage will only show for a local image.

If it is a local image, I think the next steps will be to run a few different SQL queries to identify why the usage is not matching to the image.

Thanks,
Rich

cronventis

Its a local image.. build and pushed to ProGet by our pipeline..
Let me know how I can support you with the queries.

dean-houston

@cronventis would you be able to get us the following?

a backup of your database
a list of your namespaces (/api/v1/pods)

If so, we can provide a secure link to let you upload them to us. Note the namespace list might be multi-paged, which means you may need to use the continue argument; https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.19/#list-all-namespaces-pod-v1-core

This would be the easiest for us to sift through, and you wouldn't have to run queries, etc.

If not, we'll think of something else :)

cronventis

@dean-houston If possible I would prefer not to do that... There might be some sensible data that should not be shared... I know it makes your task a little harder, but I could offer a remote session or something similar...

rhessinger

Hi @cronventis,

Would you be able to send over your Docker tables from your ProGet database to us? They won't contain any of the actual layers of the images, just the digests and names so we can see where the disconnect is happening.

The easiest way to do this would be to :

Create a new database called pgexport
Running the following SQL to dump the docker tables into that database:

  SELECT * INTO [pgexport].[DockerBlobs] FROM [DockerBlobs]
  SELECT * INTO [pgexport].[DockerImageLayers] FROM [DockerImageLayers]
  SELECT * INTO [pgexport].[DockerImages] FROM [DockerImages]
  SELECT * INTO [pgexport].[DockerRepositoryTags] FROM [DockerRepositoryTags]
  SELECT * INTO [pgexport].[ContainerUsage] FROM [ContainerUsage]

then backup pgexport and send it to us

If the backup is pretty small, you can just email it to support@inedo.com with the subject of [QA-729] Database Export or I can create a one drive share link you can upload to.

Would you be able to do that?

Thanks,
Rich

cronventis

@rhessinger E-Mail is on the way.. :)

rhessinger

Hi @cronventis,

Thanks for sending this over to us! I can confirm we have received it and we are currently looking into this. I'll let you know when we have more information.

Thanks,
Rich

rhessinger

Hi @cronventis,

I took a look through your tables and there is definitely something a bit odd going on with your Kubernetes output. Could you tell me what version of Kubernetes you are running?

Would it be possible to see the output from your Kubernetes API? Again, this is something you can send us via suppor@inedo.com with the subject of [QA-729] Kubernetes API. To get the image list, you can simply run this PowerShell against your Kubernetes API:

$uri = [System.Uri]'http://localhost:8080/api/v1/pods?limit=999'
$response = Invoke-RestMethod -Method GET -Uri $uri.ToString()
$response | ConvertTo-JSON | Out-File "C:\temp\response.json"

Just change HTTP://localhost:8080 to your Kubernetes API host and port.

Thanks,
Rich

cronventis

@rhessinger said in kubernetes scanner not showing results:

[QA-729] Kubernetes API

E-Mail is on its way again.. :)

*it probably should be suppor(t)@

rhessinger

Hi @cronventis,

Thanks! I got the email and I'm taking a look now.

Thanks,
Rich

rhessinger

Hi @cronventis,

I just wanted to let you know that I'm still researching this. Is it possible to tell me what version of Kubernetes and Rancher is installed?

Thanks,
Rich

cronventis

@rhessinger
Rancher: v2.6.2
Kubernetes: v1.19.9

Dan_Woolf

Hi @cronventis,

Just an update: We are still looking into the issue. We are currently having trouble recreating the same API output as yours. In our tests, the imageId returned for the container status contains a different digest. We are digging further into the Rancher documentation, specifically around their Kubernetes configuration.

If you have any tips on some specific Rancher/Kubernetes configurations we should look for, please let us know.

Thanks,
Dan

cronventis

@Dan_Woolf Sorry to say that, but I really dont know how to help you further. I sent you another json (could be identical, or only have small changes) but was created in a different way.. Maybe this is more useful?

atripp

Hi @cronventis, just wanted to let you know that this is complicated and it's not something we cannot quickly debug/diagnose.

Based on our analysis, the data being returned from your Kubernetes API is different than our instance, and the instances we've seen in the field. Our instance's API is returning the configuration digest, but it looks like your instance is returning the manifest digest.

Which one is correct? Why is your instance doing that? Why is ours doing this? It's a mess

Code-wise, it would be a trivial fix in ProGet to make. Basically we just change this...

var data = await new DB.Context(false).ContainerUsage_GetUsageAsync(Feed_Id: this.FeedId, Image_Id: this.Image.ContainerConfigBlob_Digest);

... to this...

var data = await new DB.Context(false).ContainerUsage_GetUsageAsync(Feed_Id: this.FeedId, Image_Id: this.Image.Image_Digest);

... except that would break our instance and the others that return configuration digests.

We're tempted to "munge" the data results (basically just concatenate both database resultsets), but it would be really nice to know (1) which is correct and (2) why one instance does one thing.

Anyways that's our latest thought. Do you have any insight into this? This is just so bizarre.

Well, we'll kepe thinking about it on our end as we have time. Just wanted to give you a sitrep.

Cheers,
Alana

cronventis

@atripp Uh.. Ok.. Yes this sounds like an interesting topic ;)

I can give you one more insight.. This is our first k8s setup, and we have chosen to run our own cluster, locally. So we don't use aws, azure or google... Basically this menas: we could have done anything wrong when setting up the cluster.. Missing something, adding something that does not belong there, and so on...

For me personally, the information in ProGet is currently nothing I HAVE to have, I actually just wonted to test this and see what is possible ;)

In maybe 6-12 months we will probably do a migration to azure k8s, maybe then it will work.. But I think for now, I have wasted enough of your time... Thank you for taking such a detailed look into this issue!

Dan_Woolf

Hi @cronventis,

Thanks for the details. Let us know when you shift over to Azure k8s if that fixes your issue or not. I would love to figure out what configuration made it change that value, but I can definitely understand holding off until you make your final shift.

Thanks,
Dan

cronventis

@Dan_Woolf I may have another Idea.... We are running on containerd ... not on dockerd...
Maybe this makes a difference: https://github.com/kubernetes/kubernetes/issues/95968

atripp

Hi @cronventis ,

Great find - that seems to explain what we're seeing: containerd reports on containers differently than dockerd. So, we'll just search for container images based on configurationblob_digest OR image_digest

This change was trivial, and will be in the next maintenance release (or available as a prerelease upon request) as PG-2081 - scheduled release date is Feb 11.

Cheers,
Alana

cronventis

@atripp Wunderfull... Glad I could help.. Looking forward to test when it arrives :)