ProGet slow fetching cargo packages

jolaka9284_9458

Hello.

We have currently two package mirrors on the same server running, trying to replace one, and evaluating ProGet as that's the one we wanna switch to.

When doing a cargo build with these dependencies:

[dependencies]
anyhow = "*"
bollard = "0.19"
blake3 = "^1.5"
clap =  { version = "=4.5.32", features = ["wrap_help"] }
macaddr = { version = "1.0", features = ["serde_std"] }
optional_struct = { version = "0.5" }
regex-lite = { version = "0.1.6" }
tokio = { version = "1.37.0", features = ["rt-multi-thread", "macros", "net", "sync", "io-util", "time", "fs"] }
serde = { version = "*", features = ["derive"] }
serde_json = "1.0"
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
tracing-journald = { version = "0.3"}
nix = "*"

We get a very slow fetching of packages. To compare: we counted seconds for a build to complete for each, you and artifactory. Both with cache cleared
Proget : 88s
Artifactory : 15s

We are not familiar with ProGet so I wish for some help regarding why these numbers are so far apart, we are on the trial license currently.

rhessinger

Hi @jolaka9284_9458,

Can you please answer the following so I can get an idea of your environment setup?

What version of ProGet you are running?
Is your config.toml configured to proxy all cargo requests through ProGet (overriding [source.crates-io])?
Do you have connector package caching and/or metadata caching enabled?
Are you using SQL Server or PostgreSQL (the embedded database, InedoDB, etc..)?

I'm not able to test against Artifactory very easily, but testing against crates.io. I'm not seeing a huge difference in speed (~20 seconds in your test case). In my testing, the time varies greatly based on the number of versions a package has and speeds up significantly once the indexes have been cached locally.

My theory is that Artifactory caches these indexes on disk on the server or is not returning the complete dataset (hosted vs local), compared to ProGet where we create the package index each time it's requested and leave the local index caching to cargo. When ProGet generates it's package index, it reaches out to any connectors on the feed and then merges them with the local package versions stored in ProGet.

There are pros and cons to each solution and are only really noticeable when the local cargo cache is cleared. When cargo has a local cache, it will send a If-Modified-Since header in its index request, allowing that index to be skipped if nothing has changed recently.

Is it also possible to get a list of your HTTP requests and timings made during these? I don't need the response body, but the URL Path (host and feed names can be anonymized), response time taken, and response code is all I really need. In ProGet 2025, you can enable HTTP Request Logging to obtain these.

Thanks,
Rich

jolaka9284_9458

Hello. Excuse me, I failed to provide basic stuff.

What version of ProGet you are running?

It's 2025.17 (Build 12).

Is your config.toml configured to proxy all cargo requests through ProGet (overriding [source.crates-io])?

Yes

[source.crates-io]
replace-with = "cds_registry"

[registries.cds_registry]
index = "sparse+http://serverhost.example.com:8624/cargo/public-cargo/"
#Seems to be needed even though we allow anonymous access
token = ""

[registry]
default = "cds_registry"
global-credential-providers = ["cargo:token"]

[net]
#network retries
retry = 10
#tells cargo to use git when fetching from the registry instead of the cargo default libgit2
git-fetch-with-cli = true

git-fetch-with-cli = false made no difference.

Do you have connector package caching and/or metadata caching enabled?

I have Connector Crate Caching enabled, with a cache size of 38 MB. No retention policy so it will be there forever.
Metadata caching was not enabled (Enabling this will not make a difference)

Are you using SQL Server or PostgreSQL (the embedded database, InedoDB, etc..)?

We installed using docker so i'm guessing InedoDB? Checking proget-database under /home/ I can see postgresql

Is it also possible to get a list of your HTTP requests and timings made during these? I don't need the response body, but the URL Path (host and feed names can be anonymized), response time taken, and response code is all I really need. In ProGet 2025, you can enable HTTP Request Logging to obtain these.

You can download the file here. https://send.vis.ee/download/cc7739c4f7f93c3b/#qJl3zTC6C3G7OFea5fWaDg
Expires after 10 downloads or 3days