Debian Connectors Performance Issue

geraldizo_0690

Hi,

some Debian feeds, especially for kali-rolling, are very slow after a day since the connector was renewed. I have to repeatedly reset or renew the connector in order to be able to work with it.

If not, an “apt update” takes several minutes or it is going to be stucked up. The index.sqlite files grow to up to 50 GB. In addition, index.sqilte3-shm and -wal also grow to almost the same size. And CPU consumption increases proportionally to these problems. And if the connectors are not renewed, the sqlite database consumes all the available storage space in the partition where the volumes for packages are located.

Additional Information:

Upstream link that we are testing and using:
http://mirror.netcologne.de/kali/
http://http.kali.org/kali

ProGet version: 2025.20(Build 23) with external database (MSSQL), both in containers

Connector configuration: Index poll frequency is already set to triple digits in minutes.

Further Information I've already researched:

Please investigate this behavior. In addition to RPM repos, we also use a lot of Debian repos.
If the cause is known, can you implement additional clean measures in addition to “Index poll frequency”?

Can the problem perhaps be avoided by using a proxy?

gdivis

Hi @geraldizo_0690,

We've reproduced this and have a fix in internal testing as PG-3225. An incorrect index is causing the local connector database index to grow much larger than it should, and this effect is compounded in frequently updated repos like Kali.

We are currently testing the fix internally and will have it included in Friday's release of ProGet 2025.22. If you'd like to test the fix yourself soon, I can make a prerelease image available to you.

Thanks!
-Greg

geraldizo_0690

Thank you for your response. We also have a Proget test environment where we can test this. We would like to test it.

Here is another example of this behavior. These are only Debian connectors:

root@xxxxxxxxxxxxxx:/usr/share/ProGet/LocalStorage/Connectors# du -chs * | grep G
2.4G    C104
2.5G    C105
2.5G    C106
62G     C117
13G     C266
9.2G    C268
15G     C281
3.1G    C63
119G    total

gdivis

You can test the fix from this prerelease image: proget.inedo.com/productimages/inedo/proget:25.0.22-ci.4

In this version, it will automatically recreate each of those databases the next time it needs an update, and after that you should no longer see this unbounded growth. Let us know if you're still seeing a problem!

geraldizo_0690

@gdivis Thank you very much. We will do the test. Stay tuned for my feedback.

geraldizo_0690

proget:25.0.22-ci.4 seems to work.

The CPU consumption stays steady while ProGet is not under heavy load. The Debian connectors does not grow infitely. So for the third day this is the output:

root@xxxxxxxxxxxxxx:/usr/share/ProGet/LocalStorage/Connectors# du -chs * | grep G
2.6G    total

I'll get back to you on this in a week.
Thank you for the support.