Retention rules not removing old docker images

[nicholas.boltralik_3634]

We have a feed with about 100 GB of docker images. New versions of the images are pushed up weekly with the same tag and it seems like the old images are not being deleted. Pulling by tag gets the latest image but pulling by UUID allows us to get an image that should have been removed by the retention rules.

The retention rules keep images that are tagged or that that have been downloaded in the last 14 days. We've tried reducing that down to 5 days and dry-running, but it did not remove a significant number of images.

We just upgraded to the latest version which has the PG-2477 fix but the issue still exists.

Are there any ways to get it to delete these old docker images?

Retention rules for reference

[atripp]

Hi @nicholas-boltralik_3634 ,

Do you have anything else defined on the other tabs? That could have an impact if so...

Otherwise, best way to troubleshoot this would be to look at the retention logs. That should give an idea of what's happening in the policy. And then if you can find a specific example of an untagged image, that would definitely help to track it down...

Cheers,
Alana

[nicholas.boltralik_3634]

@atripp the other tabs are set to the most permissive settings.

I see packages being deleted in the logs and verified that they are no longer available by UUID, I assume that's due to the bugfix I mentioned earlier. But there's still lots of old images being retained.

In the database I see they exist in the Docker image and blob tables but don't have any identifying info in the tags table so I'm not sure how to tell where they came from or how to get them

[atripp]

@nicholas-boltralik_3634 thanks for sharing the other details; nothing is jumping out as off to me in the code (and i can't reproduce this with a trivial case), so please give us some time to do a little more investigation - it's most certain we won't get this in the upcoming maintenance release (this Friday), but we have one on Nov 17 that hopefully we'll be able to figure this out in

[nicholas.boltralik_3634]

Thanks @atripp
I'm not sure it could be reproduced in the current release now that the bug that caused them to accumulate is fixed. But is there a way I could tell proget to delete them? I'd like to avoid needing to delete both the physical files and DB entries and risk breaking something

[atripp]

Hi @nicholas-boltralik_3634 ,

One thing I'm thinking is that this is a UI-bug, in that an option is set on the rule that cannot be edited in the UI; can you try deleting that particular rule, then re-adding it?

Otherwise There's no easy way to delete these;

• You could tag them, and then browse in the UI, and delete.
• Or you could do a DELETE request using the API, but that's not really that easy to do due to how Docker API auth works.

But we should be able to fix this, just need to figure what's causing it.

FYI -- The bug fix you mentioned earlier ( PG-2477) had to do with tag matching. The specific case was, if you had the rule "Delete Images (not requested for 10 days, matching 0.0.0-*) from the feed" set-up, then an image with BOTH 0.5.0-abc4.1.1-posse7.4.2.2 and 0.0.0-issue-PC3-1507 would not be deleted. I don't think it's related.

Thanks,
Alana

[nicholas.boltralik_3634]

@atripp Ah so maybe we are encountering a new bug.

I removed and re-added the rule but it didn't change anything. I'll look into tagging them and deleting through the API.

[gdivis]

Have you had any success deleting these after retagging? We haven't been able to reproduce this so far.