kubernetes scanner not showing results

rhessinger

I set up an example in our test environment here and everything is working as expected. I did verify that the Image_Id that is in the database should match the Image Digest in the Image Data table as well.

Is this image a local image or a cached image? Currently, usage will only show for a local image.

If it is a local image, I think the next steps will be to run a few different SQL queries to identify why the usage is not matching to the image.

Thanks,
Rich

cronventis

Its a local image.. build and pushed to ProGet by our pipeline..
Let me know how I can support you with the queries.

dean-houston

@cronventis would you be able to get us the following?

a backup of your database
a list of your namespaces (/api/v1/pods)

If so, we can provide a secure link to let you upload them to us. Note the namespace list might be multi-paged, which means you may need to use the continue argument; https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.19/#list-all-namespaces-pod-v1-core

This would be the easiest for us to sift through, and you wouldn't have to run queries, etc.

If not, we'll think of something else :)

cronventis

@dean-houston If possible I would prefer not to do that... There might be some sensible data that should not be shared... I know it makes your task a little harder, but I could offer a remote session or something similar...

rhessinger

Hi @cronventis,

Would you be able to send over your Docker tables from your ProGet database to us? They won't contain any of the actual layers of the images, just the digests and names so we can see where the disconnect is happening.

The easiest way to do this would be to :

Create a new database called pgexport
Running the following SQL to dump the docker tables into that database:

  SELECT * INTO [pgexport].[DockerBlobs] FROM [DockerBlobs]
  SELECT * INTO [pgexport].[DockerImageLayers] FROM [DockerImageLayers]
  SELECT * INTO [pgexport].[DockerImages] FROM [DockerImages]
  SELECT * INTO [pgexport].[DockerRepositoryTags] FROM [DockerRepositoryTags]
  SELECT * INTO [pgexport].[ContainerUsage] FROM [ContainerUsage]

then backup pgexport and send it to us

If the backup is pretty small, you can just email it to support@inedo.com with the subject of [QA-729] Database Export or I can create a one drive share link you can upload to.

Would you be able to do that?

Thanks,
Rich

cronventis

@rhessinger E-Mail is on the way.. :)

rhessinger

Hi @cronventis,

Thanks for sending this over to us! I can confirm we have received it and we are currently looking into this. I'll let you know when we have more information.

Thanks,
Rich

rhessinger

Hi @cronventis,

I took a look through your tables and there is definitely something a bit odd going on with your Kubernetes output. Could you tell me what version of Kubernetes you are running?

Would it be possible to see the output from your Kubernetes API? Again, this is something you can send us via suppor@inedo.com with the subject of [QA-729] Kubernetes API. To get the image list, you can simply run this PowerShell against your Kubernetes API:

$uri = [System.Uri]'http://localhost:8080/api/v1/pods?limit=999'
$response = Invoke-RestMethod -Method GET -Uri $uri.ToString()
$response | ConvertTo-JSON | Out-File "C:\temp\response.json"

Just change HTTP://localhost:8080 to your Kubernetes API host and port.

Thanks,
Rich

cronventis

@rhessinger said in kubernetes scanner not showing results:

[QA-729] Kubernetes API

E-Mail is on its way again.. :)

*it probably should be suppor(t)@

rhessinger

Hi @cronventis,

Thanks! I got the email and I'm taking a look now.

Thanks,
Rich

rhessinger

Hi @cronventis,

I just wanted to let you know that I'm still researching this. Is it possible to tell me what version of Kubernetes and Rancher is installed?

Thanks,
Rich

cronventis

@rhessinger
Rancher: v2.6.2
Kubernetes: v1.19.9

Dan_Woolf

Hi @cronventis,

Just an update: We are still looking into the issue. We are currently having trouble recreating the same API output as yours. In our tests, the imageId returned for the container status contains a different digest. We are digging further into the Rancher documentation, specifically around their Kubernetes configuration.

If you have any tips on some specific Rancher/Kubernetes configurations we should look for, please let us know.

Thanks,
Dan

cronventis

@Dan_Woolf Sorry to say that, but I really dont know how to help you further. I sent you another json (could be identical, or only have small changes) but was created in a different way.. Maybe this is more useful?

atripp

Hi @cronventis, just wanted to let you know that this is complicated and it's not something we cannot quickly debug/diagnose.

Based on our analysis, the data being returned from your Kubernetes API is different than our instance, and the instances we've seen in the field. Our instance's API is returning the configuration digest, but it looks like your instance is returning the manifest digest.

Which one is correct? Why is your instance doing that? Why is ours doing this? It's a mess

Code-wise, it would be a trivial fix in ProGet to make. Basically we just change this...

var data = await new DB.Context(false).ContainerUsage_GetUsageAsync(Feed_Id: this.FeedId, Image_Id: this.Image.ContainerConfigBlob_Digest);

... to this...

var data = await new DB.Context(false).ContainerUsage_GetUsageAsync(Feed_Id: this.FeedId, Image_Id: this.Image.Image_Digest);

... except that would break our instance and the others that return configuration digests.

We're tempted to "munge" the data results (basically just concatenate both database resultsets), but it would be really nice to know (1) which is correct and (2) why one instance does one thing.

Anyways that's our latest thought. Do you have any insight into this? This is just so bizarre.

Well, we'll kepe thinking about it on our end as we have time. Just wanted to give you a sitrep.

Cheers,
Alana

cronventis

@atripp Uh.. Ok.. Yes this sounds like an interesting topic ;)

I can give you one more insight.. This is our first k8s setup, and we have chosen to run our own cluster, locally. So we don't use aws, azure or google... Basically this menas: we could have done anything wrong when setting up the cluster.. Missing something, adding something that does not belong there, and so on...

For me personally, the information in ProGet is currently nothing I HAVE to have, I actually just wonted to test this and see what is possible ;)

In maybe 6-12 months we will probably do a migration to azure k8s, maybe then it will work.. But I think for now, I have wasted enough of your time... Thank you for taking such a detailed look into this issue!

Dan_Woolf

Hi @cronventis,

Thanks for the details. Let us know when you shift over to Azure k8s if that fixes your issue or not. I would love to figure out what configuration made it change that value, but I can definitely understand holding off until you make your final shift.

Thanks,
Dan

cronventis

@Dan_Woolf I may have another Idea.... We are running on containerd ... not on dockerd...
Maybe this makes a difference: https://github.com/kubernetes/kubernetes/issues/95968

atripp

Hi @cronventis ,

Great find - that seems to explain what we're seeing: containerd reports on containers differently than dockerd. So, we'll just search for container images based on configurationblob_digest OR image_digest

This change was trivial, and will be in the next maintenance release (or available as a prerelease upon request) as PG-2081 - scheduled release date is Feb 11.

Cheers,
Alana

cronventis

@atripp Wunderfull... Glad I could help.. Looking forward to test when it arrives :)