1. Home
  2. Categories
  3. Support
  4. ProGet slow fetching cargo packages

Welcome to the Inedo Forums! Check out the Forums Guide for help getting started.

If you are experiencing any issues with the forum software, please visit the Contact Form on our website and let us know!

ProGet slow fetching cargo packages

  • J

    Hello.

    We have currently two package mirrors on the same server running, trying to replace one, and evaluating ProGet as that's the one we wanna switch to.

    When doing a cargo build with these dependencies:

    [dependencies]
anyhow = "*"
bollard = "0.19"
blake3 = "^1.5"
clap =  { version = "=4.5.32", features = ["wrap_help"] }
macaddr = { version = "1.0", features = ["serde_std"] }
optional_struct = { version = "0.5" }
regex-lite = { version = "0.1.6" }
tokio = { version = "1.37.0", features = ["rt-multi-thread", "macros", "net", "sync", "io-util", "time", "fs"] }
serde = { version = "*", features = ["derive"] }
serde_json = "1.0"
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
tracing-journald = { version = "0.3"}
nix = "*"

    We get a very slow fetching of packages. To compare: we counted seconds for a build to complete for each, you and artifactory. Both with cache cleared
    Proget : 88s
    Artifactory : 15s

    We are not familiar with ProGet so I wish for some help regarding why these numbers are so far apart, we are on the trial license currently.

    0
  • rhessinger
    inedo-engineer

    Hi @jolaka9284_9458,

    Can you please answer the following so I can get an idea of your environment setup?

    • What version of ProGet you are running?
    • Is your config.toml configured to proxy all cargo requests through ProGet (overriding [source.crates-io])?
    • Do you have connector package caching and/or metadata caching enabled?
    • Are you using SQL Server or PostgreSQL (the embedded database, InedoDB, etc..)?

    I'm not able to test against Artifactory very easily, but testing against crates.io. I'm not seeing a huge difference in speed (~20 seconds in your test case). In my testing, the time varies greatly based on the number of versions a package has and speeds up significantly once the indexes have been cached locally.

    My theory is that Artifactory caches these indexes on disk on the server or is not returning the complete dataset (hosted vs local), compared to ProGet where we create the package index each time it's requested and leave the local index caching to cargo. When ProGet generates it's package index, it reaches out to any connectors on the feed and then merges them with the local package versions stored in ProGet.

    There are pros and cons to each solution and are only really noticeable when the local cargo cache is cleared. When cargo has a local cache, it will send a If-Modified-Since header in its index request, allowing that index to be skipped if nothing has changed recently.

    Is it also possible to get a list of your HTTP requests and timings made during these? I don't need the response body, but the URL Path (host and feed names can be anonymized), response time taken, and response code is all I really need. In ProGet 2025, you can enable HTTP Request Logging to obtain these.

    Thanks,
    Rich

    0
  • J

    Hello. Excuse me, I failed to provide basic stuff.

    What version of ProGet you are running?

    • It's 2025.17 (Build 12).

    Is your config.toml configured to proxy all cargo requests through ProGet (overriding [source.crates-io])?

    • Yes
    [source.crates-io]
replace-with = "cds_registry"

[registries.cds_registry]
index = "sparse+http://serverhost.example.com:8624/cargo/public-cargo/"
#Seems to be needed even though we allow anonymous access
token = ""

[registry]
default = "cds_registry"
global-credential-providers = ["cargo:token"]

[net]
#network retries
retry = 10
#tells cargo to use git when fetching from the registry instead of the cargo default libgit2
git-fetch-with-cli = true

    git-fetch-with-cli = false made no difference.

    I also used the default config.toml config and it was the same thing:

    [registries]
proget = {index = "sparse+http://serverhost.example.com:8624/cargo/public-cargo/" }

[registry]
default = "proget"

[source]
[source.proget]
registry = "sparse+http://serverhost.example.com:8624/cargo/public-cargo/"
[source.crates-io]
replace-with = "proget"

    Do you have connector package caching and/or metadata caching enabled?

    • I have Connector Crate Caching enabled, with a cache size of 38 MB. No retention policy
      (However if I check what the total size we downloaded during cargo build it's 856 MB)
    • Metadata caching was not enabled (Enabling this did not make a difference)

    Are you using SQL Server or PostgreSQL (the embedded database, InedoDB, etc..)?

    • We installed using docker so i'm guessing InedoDB? Checking proget-database under /home/ I can see postgresql

    Is it also possible to get a list of your HTTP requests and timings made during these? I don't need the response body, but the URL Path (host and feed names can be anonymized), response time taken, and response code is all I really need. In ProGet 2025, you can enable HTTP Request Logging to obtain these.

    .
    .
    .
    .
    .

    We also get alot of timeouts after having a clean slate.
    rm -rf ~/.cargo/registry/cache/*
    rm -rf ~/.cargo/registry/index/*
    cd nytest
    cargo clean
    cargo build

    superuser@debian:~/nytest$ cargo build
    Updating `cds_registry` index
warning: spurious network error (10 tries remaining): [28] Timeout was reached (Operation timed out after 30002 milliseconds with 0 bytes received)
warning: spurious network error (10 tries remaining): [28] Timeout was reached (Operation timed out after 30002 milliseconds with 0 bytes received)
warning: spurious network error (10 tries remaining): [28] Timeout was reached (Operation timed out after 30002 milliseconds with 0 bytes received)
warning: spurious network error (10 tries remaining): [28] Timeout was reached (Operation timed out after 30002 milliseconds with 0 bytes received)
warning: spurious network error (10 tries remaining): [28] Timeout was reached (Operation timed out after 30002 milliseconds with 0 bytes received)
warning: spurious network error (10 tries remaining): [28] Timeout was reached (Operation timed out after 30001 milliseconds with 0 bytes received)
warning: spurious network error (10 tries remaining): [28] Timeout was reached (Operation timed out after 30001 milliseconds with 0 bytes received)
warning: spurious network error (10 tries remaining): [28] Timeout was reached (Operation timed out after 30001 milliseconds with 0 bytes received)
warning: spurious network error (10 tries remaining): [28] Timeout was reached (Operation timed out after 30001 milliseconds with 0 bytes received)
warning: spurious network error (10 tries remaining): [28] Timeout was reached (Operation timed out after 30001 milliseconds with 0 bytes received)
warning: spurious network error (10 tries remaining): [28] Timeout was reached (Operation timed out after 30001 milliseconds with 0 bytes received)
warning: spurious network error (10 tries remaining): [28] Timeout was reached (Operation timed out after 30001 milliseconds with 0 bytes received)
warning: spurious network error (10 tries remaining): [28] Timeout was reached (Operation timed out after 30001 milliseconds with 0 bytes received)
warning: spurious network error (10 tries remaining): [28] Timeout was reached (Operation timed out after 30001 milliseconds with 0 bytes received)
warning: spurious network error (10 tries remaining): [28] Timeout was reached (Operation timed out after 30001 milliseconds with 0 bytes received)
warning: spurious network error (10 tries remaining): [28] Timeout was reached (Operation timed out after 30001 milliseconds with 0 bytes received)
warning: spurious network error (10 tries remaining): [28] Timeout was reached (Operation timed out after 30001 milliseconds with 0 bytes received)
warning: spurious network error (10 tries remaining): [28] Timeout was reached (Operation timed out after 30001 milliseconds with 0 bytes received)
warning: spurious network error (10 tries remaining): [28] Timeout was reached (Operation timed out after 30001 milliseconds with 0 bytes received)
warning: spurious network error (10 tries remaining): [28] Timeout was reached (Operation timed out after 30001 milliseconds with 0 bytes received)
warning: spurious network error (10 tries remaining): [28] Timeout was reached (Operation timed out after 30001 milliseconds with 0 bytes received)
warning: spurious network error (10 tries remaining): [28] Timeout was reached (Operation timed out after 30001 milliseconds with 0 bytes received)
warning: spurious network error (10 tries remaining): [28] Timeout was reached (Operation timed out after 30001 milliseconds with 0 bytes received)
warning: spurious network error (10 tries remaining): [28] Timeout was reached (Operation timed out after 30001 milliseconds with 0 bytes received)
warning: spurious network error (10 tries remaining): [28] Timeout was reached (Operation timed out after 30001 milliseconds with 0 bytes received)
warning: spurious network error (10 tries remaining): [28] Timeout was reached (Operation timed out after 30001 milliseconds with 0 bytes received)
warning: spurious network error (10 tries remaining): [28] Timeout was reached (Operation timed out after 30001 milliseconds with 0 bytes received)
warning: spurious network error (10 tries remaining): [28] Timeout was reached (Operation timed out after 30001 milliseconds with 0 bytes received)
warning: spurious network error (10 tries remaining): [28] Timeout was reached (Operation timed out after 30001 milliseconds with 0 bytes received)
warning: spurious network error (10 tries remaining): [28] Timeout was reached (Operation timed out after 30001 milliseconds with 0 bytes received)
warning: spurious network error (10 tries remaining): [28] Timeout was reached (Operation timed out after 30001 milliseconds with 0 bytes received)
  Downloaded aho-corasick v1.1.4 (registry `cds_registry`)
  Downloaded anstyle v1.0.13 (registry `cds_registry`)
  Downloaded anstream v0.6.21 (registry `cds_registry`)
  Downloaded anstyle-query v1.1.5 (registry `cds_registry`)
  Downloaded anstyle-parse v0.2.7 (registry `cds_registry`)
  Downloaded anyhow v1.0.100 (registry `cds_registry`)
  Downloaded arrayref v0.3.9 (registry `cds_registry`)
  Downloaded arrayvec v0.7.6 (registry `cds_registry`)
  Downloaded atomic-waker v1.1.2 (registry `cds_registry`)
  Downloaded autocfg v1.5.0 (registry `cds_registry`)
  Downloaded base64 v0.22.1 (registry `cds_registry`)
...
``
    0
  • rhessinger
    inedo-engineer

    Hi @jolaka9284_9458,

    Thanks for providing all these details. I was able to pinpoint the reason for the slow down and why it specifically happens on some crates and not others. This is related to some code that we have to determine when to point cargo to pull dependencies from ProGet vs crates.io. Specifically this comes from cargo's API specs for dependencies:

    registry — cargo metadata uses a value of null to indicate that the dependency comes from crates.io. The index uses a value of null to indicate that the dependency comes from the same registry as the index. When creating an index entry, a registry other than crates.io should translate a value of null to be https://github.com/rust-lang/crates.io-index and translate a URL that matches the current index to be null.

    As you can see, the value specified in the metadata is different than the value the index needs to return. In ProGet, we will return null if the package exists in the feed (including connectors) and https://github.com/rust-lang/crates.io-index if it does not. This is to support the case when ProGet is not used as a mirror and instead for only local crates. Unfortunately the use ProGet as a mirror option is stored only in the client config and is not sent to ProGet.

    This is the reason why crates with a lot of dependencies take longer to generate the index than ones that don't and why you'll occasionally get timeouts, but after the retry it works. We have some caching on this to help with performance, but it's not a forever cache.

    I'm going to work on some potential improvements for this and will let you know when I have a solution ready. Unfortunately, the only workaround we have for this currently is to use a package approval workflow (like our npm Package Approval blog article).

    Thanks,
    Rich

    0
Log in to reply
 
Inedo Website Home • Support Home • Code of Conduct • Forums Guide • Documentation