Proget: retention policy for branches in a package

MaxCascone

Hi! We are trying to set up a coherent retention policy for our development feeds. We are currently using a process that designed with a lot of input from Inedo, so i hope you might have some insight on this detail of managing it.

For a given repo/package, we push all ci builds from Jenkins to a <feedname>_dev feed. Only packages that are cleared for production deploy are repackaged into a <feedname> feed, which is considered the production feed.

So we end up with a ton of branch builds littering the _dev feeds, most of which are never needed. I've been unable to come up with a generic retention policy for the dev feeds that prunes the unneeded builds, but always leaves at least one build from each branch available, until it can fade out after say 30 days.

What I am considering is adding a step in the publish process to use the Retention API to create a new retention policy for every new branch published to ProGet. That way we can set up a specific policy for each new branch.

The problem is obviously that we will have to clean up the retention policies at some point.

An example might help:

package 1
- versions
  1. branch1-v1
  2. branch1-v2
  3. branch2-v1
  4. branch2-v2

If I want to keep only the latest 5 versions, it will not guarantee that branch2 will keep its latest version available if branch1 keeps publishing builds.

Without the api approach, I can't see a way to mitigate this using the existing tools available: deleting/keeping versions or names that match/don't match a wildcard string. Maybe I am just not seeing it!

Thanks in advance!

Also relates to https://forums.inedo.com/post/12249

MaxCascone

Here's another example. If I have two rules:

all except the most recent 3 versions matching *mybranch* from the feed
all except the most recent 3 versions matching *mybranch2* from the feed

This is achieved by creating two rules, each set up with 'delete all except the latest 3 versions' and 'delete packages with matching versions' with *mybranch* and *mybranch2*

Which will the behavior be?

It will remove everything except the last 3 versions of both mybranch and mybranch2, OR
it will not remove anything except the last 3 versions of both mybranch and mybranch2

I want the behavior of #2. My plan is upon publishing from Jenkins, it will also set up a rule, if it does not exist already, for the branch name it is pushing to only keep 3 versions of that branch. I don't want any given rule to affect packages with any other branch names other than what is in that rule. So if I have a rule to delete branch1, I don't ever want it to delete anything from branch2.

After reading over this 18942 times, I believe the mechanism requires all criteria of each rule to pass before executing the delete - so i think the behavior will be what I want, that of #2 above, and only packages with versions matching mybranch or mybranch2 will be deleted (except the most recent 3 of each).

Please confirm! Thanks!

To avoid this kind of uncertainty, maybe the UI text for each rule could be clarified to something like, "Package versions matching *mybranch2*, except the most recent 3 of that package, will be deleted". I think i am reading all except the most recent 3 versions matching mybranch from the feed to mean, "everything else will be deleted except the most recent 3 versions matching mybranch". Which I don't think is right, but it's just not clear enough to be sure.

Thanks!!

atripp

Hi @mcascone!

To the "simple" question about the retention behavior, each retention rule starts by building a list of all packages. It loops over every package, and removes items from the list based on criteria you select. The packages not removed from the list are deleted. This ultimately has the effect of having everything be an "AND" in a single rule. This means *mybranch* and *mybranch2* can be reduced to*mybranch2*.

The rules run one after another. So the second rule would start with a new list, and elimate items based on what you checked off.

To the more complex question... why not just let the "dev" packages packages get messy? You can use a time- and/or usage-based rule. That might simplify things a lot. You can enable differential storage on Windows, which will reduce real space consumtion by like 90% or more.

Or maybe use a different feed? Just throwing ideas out :)

Cheers,
Alana

MaxCascone

This means mybranch and mybranch2 can be reduced to mybranch2.

Sorry but wouldn't this be the reverse: *mybranch* will match *mybranch2*?

MaxCascone

And also to clarify again:
all except the most recent 3 versions matching mybranch from the feed

means:
in this feed, delete matches of 'mybranch', except the latest 3 versions of those matches, which would only impact the versions matching mybranch and leave all other non-matches untouched;

and not:
in this feed, delete everything except the latest three versions of 'mybranch', which would leave ONLY the latest three versions of mybranch, and delete eveything else, which would obviously be a major problem.

Is that correct? I mean, it has to be, but I just want to be sure.

atripp

@mcascone said in Proget: retention policy for branches in a package:

This means mybranch and mybranch2 can be reduced to mybranch2.

Sorry but wouldn't this be the reverse: *mybranch* will match *mybranch2*?

What I mean to say.... because it's an AND conditional, the *mybranch* is effectively ignored. Everything that matches *mybranch2* will also match *mybranch*, but the opposite isn't true. E.g. mybranch1 won't match both conditions.

@mcascone said in Proget: retention policy for branches in a package:

in this feed, delete matches of 'mybranch', except the latest 3 versions of those matches, which would only impact the versions matching mybranch and leave all other non-matches untouched;

Correct. And do note that you can set retention policies to run in dry mode, where nothing is deleted, to verify it's the behavior you want.