ProGet bug - Regression: RPM feed repodata broken since 25.0.8

stevedennis

Hi @inedo_1308 ,

This is likely a regression due to PG-3074 -- we'll take a look and report back!

Thanks for letting us know.

-- Dean

gdivis

We've now fixed this and several other related RPM issues as PG-3094, scheduled for release this Friday in ProGet 2025.9. Thanks again for the bug report!

felfert

@gdivis Thanks for the update

-Fritz

felfert

@gdivis I just updated to 25.0.9 (aka 2025.9 Build 16), but unfortunately it still behaves the same.

I digged a little deeper and found out, that the files below repodata/ are not updated at all anymore. I tried the following things:

using the reindex link in the feed's "Storage and retention management page"

INFO : 2025-08-30 17:43:00Z - Performing reindex for Rpm feed 13: cloudtools-rpm
INFO : 2025-08-30 17:43:00Z - Deactivating feed.
DEBUG: 2025-08-30 17:43:06Z - Checking for orphaned FeedPackages...
DEBUG: 2025-08-30 17:43:06Z - No orphaned FeedPackages found.
DEBUG: 2025-08-30 17:43:06Z - Recalculating latest versions of packages...
DEBUG: 2025-08-30 17:43:06Z - Health check complete; recording results...
INFO : 2025-08-30 17:43:06Z - Re-activating feed.

Deleted the latest package of the feed an d then re-uploaded it.

Here ist a listing of the feed's storage (mapped here to /var/data/proget/packages/.rpm/F13):

ls -ltrR
.:
total 8
drwxr-xr-x. 2 root root 4096 Aug 30 19:41 packages
drwxr-xr-x. 2 root root 4096 Aug 30 19:44 repodata

./packages:
total 309796
-rw-r--r--. 1 root root 12490071 Jun 29  2024 cloudtools-2.5.233-1.noarch.rpm
-rw-r--r--. 1 root root 12490704 Jun 29  2024 cloudtools-2.5.234-1.noarch.rpm
-rw-r--r--. 1 root root 12495077 Jun 29  2024 cloudtools-2.5.236-1.noarch.rpm
-rw-r--r--. 1 root root 12494704 Jun 29  2024 cloudtools-2.5.235-1.noarch.rpm
-rw-r--r--. 1 root root 12495445 Jun 29  2024 cloudtools-2.5.237-1.noarch.rpm
-rw-r--r--. 1 root root 12495891 Jun 29  2024 cloudtools-2.5.240-1.noarch.rpm
-rw-r--r--. 1 root root 12691058 Jun 29  2024 cloudtools-2.5.241-1.noarch.rpm
-rw-r--r--. 1 root root 12691053 Jun 29  2024 cloudtools-2.5.242-1.noarch.rpm
-rw-r--r--. 1 root root 12691035 Jun 29  2024 cloudtools-2.5.243-1.noarch.rpm
-rw-r--r--. 1 root root 12684216 Jul  5  2024 cloudtools-2.5.244-1.noarch.rpm
-rw-r--r--. 1 root root 12684455 Jan 19  2025 cloudtools-2.5.246-1.noarch.rpm
-rw-r--r--. 1 root root 12678409 Apr 12 10:42 cloudtools-2.5.249-1.noarch.rpm
-rw-r--r--. 1 root root 12678450 May 20 14:30 cloudtools-2.5.250-1.noarch.rpm
-rw-r--r--. 1 root root 12678452 May 20 14:48 cloudtools-2.5.251-1.noarch.rpm
-rw-r--r--. 1 root root 12678743 May 27 05:56 cloudtools-2.5.252-1.noarch.rpm
-rw-r--r--. 1 root root 12678824 May 27 10:01 cloudtools-2.5.253-1.noarch.rpm
-rw-r--r--. 1 root root 12678806 May 29 08:45 cloudtools-2.5.254-1.noarch.rpm
-rw-r--r--. 1 root root 12871348 Jun  1 22:59 cloudtools-2.6.255-1.noarch.rpm
-rw-r--r--. 1 root root 12896455 Jun  2 08:36 cloudtools-2.6.256-1.noarch.rpm
-rw-r--r--. 1 root root 12896459 Jun  2 11:39 cloudtools-2.6.257-1.noarch.rpm
-rw-r--r--. 1 root root 12896469 Jun  2 11:41 cloudtools-2.6.258-1.noarch.rpm
-rw-r--r--. 1 root root 12785737 Jun  2 15:22 cloudtools-2.6.262-1.noarch.rpm
-rw-r--r--. 1 root root 12785736 Jun  6 09:55 cloudtools-2.6.263-1.noarch.rpm
-rw-r--r--. 1 root root 12786981 Jun 11 17:24 cloudtools-2.6.265-1.noarch.rpm
-rw-r--r--. 1 root root 12787141 Aug 30 19:41 cloudtools-2.6.267-1.noarch.rpm

./repodata:
total 12
-rw-r--r--. 1 root root 1468 Aug 20 09:02 2b01ecf5699ac41767bfacd995e22b274cfdb81920751989325f34678e402232-filelists.xml.gz
-rw-r--r--. 1 root root 2846 Aug 20 09:02 dd5a0e1508733bde65393ae882018efb96c8af3433913ce07be20e08fad96fae-primary.xml.gz
-rw-r--r--. 1 root root 1354 Aug 20 09:02 280d74eede038f8fede238c851204ad378e0373e473374541891b0937dbc99e9-other.xml.gz

As you can see in this listing, the package cloudtools-2.6.267-1.noarch.rpm has a timestamp from today (because I just uploaded it), but the *.xml.gz files have not been touched since Aug 20.
And it is those files which contain the actual index and all the info necessary for downloading/installing a package.

If you want these files, I can provide them, but unfortunately this forum has no facility to attach files. (At least I can't find some)

Later:
I just noticed 2 errors in the Diagnostic Center from the time I deleted the cloudtools-2.6.267-1.noarch.rpm:

An error occurred in the web application: cloudtools 2.6.267-1 not found.

URL: http://proget.graudatastorage.intern/feeds/cloudtools-rpm/cloudtools/2.6.267-1
Referrer: http://proget.graudatastorage.intern/feeds/cloudtools-rpm
User: felfert
User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:141.0) Gecko/20100101 Firefox/141.0
Stack trace:    at Inedo.ProGet.WebApplication.Pages.Packages.PackagePageBase.CreateChildControlsAsync() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E589193\Src\src\ProGet\WebApplication\Pages\Packages\PackagePageBase.cs:line 85
   at Inedo.ProGet.WebApplication.Pages.ProGetSimplePage.InitializeAsync() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E589193\Src\src\ProGet\WebApplication\Pages\ProGetSimplePage.cs:line 69
   at Inedo.Web.PageFree.SimplePageBase.ExecutePageLifeCycleAsync()
   at Inedo.Web.PageFree.SimplePageBase.ProcessRequestAsync(AhHttpContext context)
   at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context)

::Web Error on 08/30/2025 19:39:13::

An error occurred in the web application: cloudtools 2.6.267-1 not found.

URL: http://proget.graudatastorage.intern/feeds/cloudtools-rpm/cloudtools/2.6.267-1
Referrer: http://proget.graudatastorage.intern/feeds/cloudtools-rpm
User: felfert
User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:141.0) Gecko/20100101 Firefox/141.0
Stack trace:    at Inedo.ProGet.WebApplication.Pages.Packages.PackagePageBase.CreateChildControlsAsync() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E589193\Src\src\ProGet\WebApplication\Pages\Packages\PackagePageBase.cs:line 85
   at Inedo.ProGet.WebApplication.Pages.ProGetSimplePage.InitializeAsync() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E589193\Src\src\ProGet\WebApplication\Pages\ProGetSimplePage.cs:line 69
   at Inedo.Web.PageFree.SimplePageBase.ExecutePageLifeCycleAsync()
   at Inedo.Web.PageFree.SimplePageBase.ProcessRequestAsync(AhHttpContext context)
   at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context)

::Web Error on 08/30/2025 19:39:00::

Thanks
-Fritz

felfert

@gdivis, @stevedennis

Just a humble reminder - PG-3094 did not fix the problem in 25.0.9 for me.

Thanks
-Fritz

gdivis

Hi @felfert,

Apologies for the delay, I was out of the office recently. We tested this one quite a bit before release and didn't see any cases of the index not getting updated. Of course, I believe you that it is happening, but it's not something I've been able to reproduce. One thing to note is that those repodata files in storage only get updated on-demand - so something would have to actually make a request for the index to get ProGet to rebuild it (this is something that we are planning to change in a separate issue, but that is how it works currently).

Of course, we'd be happy to do some testing with your packages specifically if you'd like to upload some - just open a ticket at https://my.inedo.com/tickets/new which does allow file attachments. Ignore the header that says it's for paid users only; we'll associate it with this forum post.

-Greg

felfert

Hi, thanks for the response..

@gdivis said in ProGet bug - Regression: RPM feed repodata broken since 25.0.8:

One thing to note is that those repodata files in storage only get updated on-demand - so something would have to actually make a request for the index to get ProGet to rebuild it

For clarification, can you elaborate a little bit on that? I mean either deleting or uploading a package would count as "something", or does the content of the repodata files get completely held in memory by proget even if some update happens?

Anyway, I will setup a proget sandbox instance and see if I can reproduce it there. Will report back and/or open a ticket when I got more details.

felfert

Ok, while I was testing on the sandbox (without being able to reproduce the error), the repodata files on my "production" proget got regenerated and since that happened, everything is working again. So this just leaves the question:

Out of curiosity:
How does one force the regeneration of those repodata files?

Obviously not one of these (tried those before)

Delete a package from the feed
Upload that package again.
Delete an older package from the feed (not uploading again)
Clicking on "re-index" in "Storage and Retention" of the feed

Cheers
-Fritz

gdivis

Any change to feed that could affect a package is supposed to trigger this, and that does include deleting a package, adding a package, or a detected change in any repodata files from any associated connectors. What happens is any operation like that sets an index dirty flag for the feed, and if a request is made for it after that, it is rebuilt. There is also a 30 second delay before this can get triggered, so if you:

upload/delete a package
wait 30s
get repodata/repomd.xml

It should show an updated timestamp and revision.

We are currently moving to more of a job-based system for these index rebuilds, which also will provide better visibility into their creation and allow for manually triggering them. We're starting this process with Debian feeds since those exhibit the most problems with the current system, but plan to extend it to rpm and apk as well.

Anyway, if you're not seeing it update even after the 30s delay then there must be some other issue at work, so we'll need to dig into that.

felfert

@gdivis That was the missing information (wait 30sec AND fetch repodata) - it works as described.

To summarize what lead me to incorrectly assume that it was not fixed:

My initial expectation was, it would be fixed by merely updating proget
After that did not show progress, I found the repodata files in the feed storage path having an old timestamp
Then I deleted and re-uploaded the latest rpm and noticed no timestamps change
I never tried to actually fetch repodata, because I saw the old timestamps

Thanks again for your clarification
-Fritz