Hi @jolaka9284_9458,

Thanks for providing all these details. I was able to pinpoint the reason for the slow down and why it specifically happens on some crates and not others. This is related to some code that we have to determine when to point cargo to pull dependencies from ProGet vs crates.io. Specifically this comes from cargo's API specs for dependencies:

registry — cargo metadata uses a value of null to indicate that the dependency comes from crates.io. The index uses a value of null to indicate that the dependency comes from the same registry as the index. When creating an index entry, a registry other than crates.io should translate a value of null to be https://github.com/rust-lang/crates.io-index and translate a URL that matches the current index to be null.

As you can see, the value specified in the metadata is different than the value the index needs to return. In ProGet, we will return null if the package exists in the feed (including connectors) and https://github.com/rust-lang/crates.io-index if it does not. This is to support the case when ProGet is not used as a mirror and instead for only local crates. Unfortunately the use ProGet as a mirror option is stored only in the client config and is not sent to ProGet.

This is the reason why crates with a lot of dependencies take longer to generate the index than ones that don't and why you'll occasionally get timeouts, but after the retry it works. We have some caching on this to help with performance, but it's not a forever cache.

I'm going to work on some potential improvements for this and will let you know when I have a solution ready. Unfortunately, the only workaround we have for this currently is to use a package approval workflow (like our npm Package Approval blog article).

Thanks,
Rich

Hi @gravufo,

Would you be able to rerun the database scripts on your database? You will just need to run the Run inedosql to update the database step of our manual install guide. Can you see if this fixes your issue?

Thanks,
Rich

Hi @viceice,

That error is safe to ignore. It is currently a known bug and we are looking to fix that in an upcoming version of ProGet. The ticket tracking the fix for the log message is PG-1841.

Long story short, the ProGet service correctly detected that the product wasn't activated, and then logged that message. But it was doing it every time it accessed license information, which is on every connector health check, replication run, etc.
Activation happens automatically as soon as someone visits the Web application, and re-activation is required after upgrading certain versions.

Thanks,
Rich

Hi @mcascone ,

We have fixed the issue and the published date will now update when the package is overwritten. This will be released in ProGet 5.3.9 which is due out this Friday Augst 14, 2020.

Thanks,
Rich

Hi @viceice,

Thanks for the clarification on your environment! I see what is going on now. I have created a ticket, PG-1809, to track the fix for this. We expect this to be released in ProGet 5.3.11 which we are expecting to be released in September 11, 2020. Basically in that instance, we are not respecting the values within the X-Forwarded-* headers. I'll let you know if anything changes on the timeline.

Thanks,
Rich

Hi @markus4830,

I'm definitely sorry about this. The change was made to help to aide in improvements to other areas of the system related to NuGet. Unfortunately, it looks like it affected the NuGet API. Expect a more permanent solution in the near future.

Thanks,
Rich

Hi @nuno-guerreiro-rosa_9280,

We have finally been able to recreate this issue in our sandbox. We are currently looking into a fix, but we expect to have one in the next version of ProGet, 5.3.10. This looks to be an issue with the mono framework. We use mono runtime in our Docker images for ProGet. We are also going to be releasing a .Net Core based technical preview of ProGet Docker in version 5.3.10. This will be in addition to our standard mono based version. Our internal testing is going very well and it looks to have removed a lot of the gotchas that mono has.

Thanks,
Rich

Hi @harald-somnes-hanssen_2204,

The fix is scheduled for release in ProGet 5.3.16 which is due out on Friday. I'll reply back if anything changes.

Thanks,
Rich

Hi @tkolb_7784,

If you are hosting on a windows machine, the easiest solution right now is to migrate your server to use IIS and then add an SSL binding to your site. If you do not want to purchase a new certificate and the self-signed certificate is too much work, you can use Let's Encrypt and configure it via winacme.

If you do not want to use IIS, then you will need to use a reverse proxy to handle SSL connections. Any reverse proxy can be used and a pretty simple one to configure is stunnel. Most reverse proxies can also be used with Let's Encrypt.

If you are hosting via Linux (Docker), then you will need to use a reverse proxy to handle SSL connections. We have a documentation page for different Linux-based reverse proxies including an example for setting up NGINX. These reverse proxies also support Let's Encrypt also.

Please let me know if you have any questions.

Thanks,
Rich

Hi @gravufo,

Great! Glad to hear it! Please post back if you find anything else.

I also recommend that you switch to the Inedo Hub in the future. We are in the process of deprecating our traditional installer. The Inedo Hub has the ability to update an installation previously installed with the traditional installer and the Inedo Hub now supports offline installations as well, if you need that functionality.

Thanks,
Rich

Hi @caspacokku_2900,

I apologize for the confusion. I meant that running through the test example you provided (looks like that comment has since been deleted) and this new one, I was unable to recreate both the PostgreSQL errors and the "The given key 'version' was not present in the dictionary." error. I'm thinking the dictionary error is related to either the PostgreSQL errors or a hiccup in the network connection where the response was not fully returned, but had a successful response code causing the request to still be cached. What is also confusing is that the cache should refresh the request after 30 minutes. This means if the caching is causing the issue, it should resolve itself within 30 minutes. Also, I see you have set the metadata caching to 100, which cargo hits pretty quickly. I typically set that to 1000 requests in a crates.io connector because cargo is a very chatty client.

The main reason I was asking for a reproduction case is that I wanted to rule out an error in parsing the cargo metadata. With this latest test case you provided, I can verify that everything appears to be parsing correctly. I ran this build 30 times back to back with your settings and connector caching enabled (both metadata and crate caching). In between each build, I cleared the local registry cache and ran cargo clean. I was not able to reproduce these errors or cause issues with the connector cache. This leads me to believe that there is an environmental factor that is causing this issue. Can you please check with your IT team to see if there is anything they are seeing (external HTTP request manipulation, network packets dropping, RAM correction errors, etc..) on your server?

Thanks,
Rich

Hi @parthu-reddy,

That error looks to be an error that happens when the web server is shutting down and a message is stuck in the queue. We plan to fix this in the next maintenance release, PG-3214. It should be fin to ignore. Were you seeing something specific happening in ProGet when this error was happening?

Thanks,
Rich

Hi @caspacokku_2900,

Thanks for sending an example over. I did test this, but I was unable to recreate the postgres errors you were seeing. If you are using ProGet as a mirror to crates.io, one thing that may be helpful is to modify the Dependency Resolution on the Feed Properties page. Standard resolution has to check in ProGet to determine if the dependency exists or not to determine when to tall the cargo client to use ProGet vs crates.io. That will help with the retries and getting rate limited on crates.io. These are things to just help reduce load from Cargo on the ProGet server.

Does this happen every time you pull packages using cargo?

Thanks,
Rich

Hi @parthu-reddy,

Sorry about that, I got hung up on the request logging, let me dig into this a bit more. Are you using PostgreSQL or SQL Server?

Thanks,
Rich

Hi @parthu-reddy,

This was an oversight on our part. The UI just updates a value in the ProGet.confg file. If you update the ProGet.config file on the other two servers and add EnableRequestLogging="true" RequestLoggingDirectory="C:\ProGetLogs" to the WebServer node, that will enable HTTP request logging on those two servers as well.

I'm going to update the docs to include how to enable this when using a Load Balanced/HA configuration. I have also added a ticket, PG-3211, to update the UI to give guidance on this configuration.

Thanks,
Rich

Hi @caspacokku_2900,

Would you be able to send us over a Cargo.toml that can recreate the errors you are seeing? Although this all points to server overload, that will at least allow me to attempt to recreate the errors you are seeing.

Thanks,
Rich

Hi @matthias-schmitz_2037,

We actually were made aware of this by another customer this week (PG-3201) and we will be releasing a fix this Friday in ProGet 2025.19.

Thanks,
Rich

Hi @jolaka9284_9459,

We have pushed some improvements in ProGet 2025.18. These improvements include:

• Improved connector metadata caching (may require an increase in the number of cached requests)
• Improved index generation code
• A new Feed management setting for Dependency Resolution
  • Standard Resolution, check in ProGet if that dependency exists
  • Force all dependencies through ProGet
  • Always use this feed unless Registry is specified

In your case, since you are using ProGet as the mirror, I would suggest forcing all dependencies through ProGet, which can be configured on Feed -> Manage Feed.

Thanks,
Rich

Hi @kquinn_2909

That is why you are not seeing the builds. That build types API is returning the Id as WebProjectsReplicatorBuildOnChange, where the builds API is returning WebProjects_Replicator_BuildOnChange as the Id. Are you using any sort of shared build type?

Thanks,
Rich

Hi @koksime-yap_5909,

No problem! Happy to help!

Thanks,
Rich