An error duing cargo build

caspacokku_2900

Oh. that's concerning. We do have more connectors like npm, docker and python and all those are fine. It's only cargo that we have had problems with, that generate tons of errors.

Our system is baseline debian 13. Using docker to run ProGet.

Connection String: 
Web UI Version: 2025.18 (Build 13)
Database Schema Version: 25.0.18.13
SQL Server Version: PostgreSQL 17.7 (Debian 17.7-3.pgdg12+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14+deb12u1) 12.2.0, 64-bit
db_owner: True
Database Name: proget
Database Collation: C (UTF8)
Database Recovery Model: Default (1)
Database Read Committed Snapshot: True
Database Auto Update Stats: True

atripp

Hi @caspacokku_2900 ,

Yeah it's pretty weird. The errors are all over the place - not like in a specific database query or anything like that. This also isn't a sign of server-overload that we've seen.

It's as if internal network connectivity is somehow breaking within a container? Or there's "something else" wrong with the internal PostgreSQL server? These are all "deep system" level errors in basically operating-system level code (drivers, etc).

We've seen this with another user with really weird errors, but no idea on how to reproduce it. Maybe it's an "error reporting an error".

As for the feed... there's nothing special about cargo vs other feeds from an API/usage standpoint. If anything, npm hammers the server a lot harder with its 1000+ package restores. And this has nothing to do with connectors.

Unfortunately we don't have a lot to go on:

how about increasing hardware
can you try a different physical server
could it somehow be the underlying operating system?
any patterns as to when this is happening (lots of traffic etc)

Any clues or consistency would help.

Thanks,
Alana

caspacokku_2900

@atripp Hi.

So my team has clarified a bit. We get these Exception while reading from stream and Internal Npgsql bug: unexpected value 0 of enum BackendMessageCode when the Connector Crate Caching is disabled, because if it's ON, we get this error, so to workaround this issue they disabled the caching:

    Updating `proget` index
warning: spurious network error (10 tries remaining): [28] Timeout was reached (Operation too slow. Less than 10 bytes/sec transferred the last 30 seconds)
warning: spurious network error (10 tries remaining): failed to get successful HTTP response from `https://proget.example.com/cargo/public-cargo/aw/s-/aws-lc-sys` (172.16.1.3), got 500
body:
{"errors":[{"detail":"The given key \u0027version\u0027 was not present in the dictionary.\r\nSystem.Collections.Generic.KeyNotFoundException: The given key \u0027version\u0027 was not present in the dictionary.\n   at Inedo.ProGet.Feeds.Cargo.CargoMetadata.\u003C.ctor\u003Eg__getDependency|3_4(KeyValuePair\u00602 dependency) in C:\\Users\\builds\\AppData\\Local\\Temp\\InedoAgent\\BuildMaster\\192.168.44.60\\Temp\\_E627894\\Src\\src\\ProGet\\Feeds\\Cargo\\CargoMetadata.cs:line 152\n   at System.Linq.Enumer…
warning: spurious network error (9 tries remaining): failed to get successful HTTP response from `https://proget.example.com/cargo/public-cargo/aw/s-/aws-lc-sys` (172.16.1.3), got 500
body:
{"errors":[{"detail":"The given key \u0027version\u0027 was not present in the dictionary.\r\nSystem.Collections.Generic.KeyNotFoundException: The given key \u0027version\u0027 was not present in the dictionary.\n   at Inedo.ProGet.Feeds.Cargo.CargoMetadata.\u003C.ctor\u003Eg__getDependency|3_4(KeyValuePair\u00602 dependency) in C:\\Users\\builds\\AppData\\Local\\Temp\\InedoAgent\\BuildMaster\\192.168.44.60\\Temp\\_E627894\\Src\\src\\ProGet\\Feeds\\Cargo\\CargoMetadata.cs:line 152\n   at System.Linq.Enumer…
warning: spurious network error (8 tries remaining): failed to get successful HTTP response from `https://proget.example.com/cargo/public-cargo/aw/s-/aws-lc-sys` (172.16.1.3), got 500
body:
{"errors":[{"detail":"The given key \u0027version\u0027 was not present in the dictionary.\r\nSystem.Collections.Generic.KeyNotFoundException: The given key \u0027version\u0027 was not present in the dictionary.\n   at Inedo.ProGet.Feeds.Cargo.CargoMetadata.\u003C.ctor\u003Eg__getDependency|3_4(KeyValuePair\u00602 dependency) in C:\\Users\\builds\\AppData\\Local\\Temp\\InedoAgent\\BuildMaster\\192.168.44.60\\Temp\\_E627894\\Src\\src\\ProGet\\Feeds\\Cargo\\CargoMetadata.cs:line 152\n   at System.Linq.Enumer…
warning: spurious network error (7 tries remaining): failed to get successful HTTP response from `https://proget.example.com/cargo/public-cargo/aw/s-/aws-lc-sys` (172.16.1.3), got 500
body:
{"errors":[{"detail":"The given key \u0027version\u0027 was not present in the dictionary.\r\nSystem.Collections.Generic.KeyNotFoundException: The given key \u0027version\u0027 was not present in the dictionary.\n   at Inedo.ProGet.Feeds.Cargo.CargoMetadata.\u003C.ctor\u003Eg__getDependency|3_4(KeyValuePair\u00602 dependency) in C:\\Users\\builds\\AppData\\Local\\Temp\\InedoAgent\\BuildMaster\\192.168.44.60\\Temp\\_E627894\\Src\\src\\ProGet\\Feeds\\Cargo\\CargoMetadata.cs:line 152\n   at System.Linq.Enumer…
warning: spurious network error (6 tries remaining): failed to get successful HTTP response from `https://proget.example.com/cargo/public-cargo/aw/s-/aws-lc-sys` (172.16.1.3), got 500
body:
{"errors":[{"detail":"The given key \u0027version\u0027 was not present in the dictionary.\r\nSystem.Collections.Generic.KeyNotFoundException: The given key \u0027version\u0027 was not present in the dictionary.\n   at Inedo.ProGet.Feeds.Cargo.CargoMetadata.\u003C.ctor\u003Eg__getDependency|3_4(KeyValuePair\u00602 dependency) in C:\\Users\\builds\\AppData\\Local\\Temp\\InedoAgent\\BuildMaster\\192.168.44.60\\Temp\\_E627894\\Src\\src\\ProGet\\Feeds\\Cargo\\CargoMetadata.cs:line 152\n   at System.Linq.Enumer…
warning: spurious network error (5 tries remaining): failed to get successful HTTP response from `https://proget.example.com/cargo/public-cargo/aw/s-/aws-lc-sys` (172.16.1.3), got 500
body:
{"errors":[{"detail":"The given key \u0027version\u0027 was not present in the dictionary.\r\nSystem.Collections.Generic.KeyNotFoundException: The given key \u0027version\u0027 was not present in the dictionary.\n   at Inedo.ProGet.Feeds.Cargo.CargoMetadata.\u003C.ctor\u003Eg__getDependency|3_4(KeyValuePair\u00602 dependency) in C:\\Users\\builds\\AppData\\Local\\Temp\\InedoAgent\\BuildMaster\\192.168.44.60\\Temp\\_E627894\\Src\\src\\ProGet\\Feeds\\Cargo\\CargoMetadata.cs:line 152\n   at System.Linq.Enumer…
warning: spurious network error (4 tries remaining): failed to get successful HTTP response from `https://proget.example.com/cargo/public-cargo/aw/s-/aws-lc-sys` (172.16.1.3), got 500
body:
{"errors":[{"detail":"The given key \u0027version\u0027 was not present in the dictionary.\r\nSystem.Collections.Generic.KeyNotFoundException: The given key \u0027version\u0027 was not present in the dictionary.\n   at Inedo.ProGet.Feeds.Cargo.CargoMetadata.\u003C.ctor\u003Eg__getDependency|3_4(KeyValuePair\u00602 dependency) in C:\\Users\\builds\\AppData\\Local\\Temp\\InedoAgent\\BuildMaster\\192.168.44.60\\Temp\\_E627894\\Src\\src\\ProGet\\Feeds\\Cargo\\CargoMetadata.cs:line 152\n   at System.Linq.Enumer…
warning: spurious network error (3 tries remaining): failed to get successful HTTP response from `https://proget.example.com/cargo/public-cargo/aw/s-/aws-lc-sys` (172.16.1.3), got 500
body:
{"errors":[{"detail":"The given key \u0027version\u0027 was not present in the dictionary.\r\nSystem.Collections.Generic.KeyNotFoundException: The given key \u0027version\u0027 was not present in the dictionary.\n   at Inedo.ProGet.Feeds.Cargo.CargoMetadata.\u003C.ctor\u003Eg__getDependency|3_4(KeyValuePair\u00602 dependency) in C:\\Users\\builds\\AppData\\Local\\Temp\\InedoAgent\\BuildMaster\\192.168.44.60\\Temp\\_E627894\\Src\\src\\ProGet\\Feeds\\Cargo\\CargoMetadata.cs:line 152\n   at System.Linq.Enumer…
warning: spurious network error (2 tries remaining): failed to get successful HTTP response from `https://proget.example.com/cargo/public-cargo/aw/s-/aws-lc-sys` (172.16.1.3), got 500
body:
{"errors":[{"detail":"The given key \u0027version\u0027 was not present in the dictionary.\r\nSystem.Collections.Generic.KeyNotFoundException: The given key \u0027version\u0027 was not present in the dictionary.\n   at Inedo.ProGet.Feeds.Cargo.CargoMetadata.\u003C.ctor\u003Eg__getDependency|3_4(KeyValuePair\u00602 dependency) in C:\\Users\\builds\\AppData\\Local\\Temp\\InedoAgent\\BuildMaster\\192.168.44.60\\Temp\\_E627894\\Src\\src\\ProGet\\Feeds\\Cargo\\CargoMetadata.cs:line 152\n   at System.Linq.Enumer…
warning: spurious network error (1 try remaining): failed to get successful HTTP response from `https://proget.example.com/cargo/public-cargo/aw/s-/aws-lc-sys` (172.16.1.3), got 500
body:
{"errors":[{"detail":"The given key \u0027version\u0027 was not present in the dictionary.\r\nSystem.Collections.Generic.KeyNotFoundException: The given key \u0027version\u0027 was not present in the dictionary.\n   at Inedo.ProGet.Feeds.Cargo.CargoMetadata.\u003C.ctor\u003Eg__getDependency|3_4(KeyValuePair\u00602 dependency) in C:\\Users\\builds\\AppData\\Local\\Temp\\InedoAgent\\BuildMaster\\192.168.44.60\\Temp\\_E627894\\Src\\src\\ProGet\\Feeds\\Cargo\\CargoMetadata.cs:line 152\n   at System.Linq.Enumer…
error: failed to get `aws-lc-sys` as a dependency of package `aws-lc-rs v1.14.1`
    ... which satisfies dependency `aws-lc-rs = "^1.14"` (locked to 1.14.1) of package `rustls v0.23.35`
    ... which satisfies dependency `rustls = "^0.23.27"` (locked to 0.23.35) of package `tokio-rustls v0.26.4`
    ... which satisfies dependency `tokio-rustls = "^0.26.0"` (locked to 0.26.4) of package `rumqttc v0.25.0`
    ... which satisfies dependency `rumqttc = "^0.25"` (locked to 0.25.0) of package `mycrate v0.0.0 (/mycrate)`
Caused by:
  failed to query replaced source registry `crates-io`
Caused by:
  download of aw/s-/aws-lc-sys failed
Caused by:
  failed to get successful HTTP response from `https://proget.example.com/cargo/public-cargo/aw/s-/aws-lc-sys` (172.16.1.3), got 500
  body:
  {"errors":[{"detail":"The given key \u0027version\u0027 was not present in the dictionary.\r\nSystem.Collections.Generic.KeyNotFoundException: The given key \u0027version\u0027 was not present in the dictionary.\n   at Inedo.ProGet.Feeds.Cargo.CargoMetadata.\u003C.ctor\u003Eg__getDependency|3_4(KeyValuePair\u00602 dependency) in C:\\Users\\builds\\AppData\\Local\\Temp\\InedoAgent\\BuildMaster\\192.168.44.60\\Temp\\_E627894\\Src\\src\\ProGet\\Feeds\\Cargo\\CargoMetadata.cs:line 152\n   at System.Linq.Enumer…

An error occurred processing a GET request to http://proget.example.com/cargo/public-cargo/aw/s-/aws-lc-sys: The given key 'version' was not present in the dictionary.

System.Collections.Generic.KeyNotFoundException: The given key 'version' was not present in the dictionary.
   at Inedo.ProGet.Feeds.Cargo.CargoMetadata.<.ctor>g__getDependency|3_4(KeyValuePair`2 dependency) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E627894\Src\src\ProGet\Feeds\Cargo\CargoMetadata.cs:line 152
   at System.Linq.Enumerable.SelectManySingleSelectorIterator`2.MoveNext()
   at Inedo.ProGet.WebApplication.FeedEndpoints.Cargo.CargoIndexHandler.ProcessRequestAsync(AhHttpContext context, WebApiContext apiContext, CargoFeed feed, String relativeUrl) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E627894\Src\src\ProGet\WebApplication\FeedEndpoints\Cargo\CargoIndexHandler.cs:line 106
   at Inedo.ProGet.WebApplication.FeedEndpoints.Cargo.CargoIndexHandler.ProcessRequestAsync(AhHttpContext context, WebApiContext apiContext, CargoFeed feed, String relativeUrl) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E627894\Src\src\ProGet\WebApplication\FeedEndpoints\Cargo\CargoIndexHandler.cs:line 89
   at Inedo.ProGet.WebApplication.FeedEndpoints.Cargo.CargoFeedHandler.ProcessRequestAsync(AhHttpContext context, WebApiContext apiContext, CargoFeed feed, String relativeUrl) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E627894\Src\src\ProGet\WebApplication\FeedEndpoints\Cargo\CargoFeedHandler.cs:line 107
   at Inedo.ProGet.WebApplication.FeedEndpoints.CanonicalFeedHandler`1.Inedo.Web.IHttpAsyncHandler.ProcessRequestAsync(AhHttpContext context) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E627894\Src\src\ProGet\WebApplication\FeedEndpoints\CanonicalFeedHandler.cs:line 85

::Web Error on 16/01/2026 13:02:14::

So the team has disabled the Connector Crate Caching entirely. But if the Caching is ON, we need to clear the cache for the aws-lc-sys package to download. We are not entirely sure if these other Exception while reading from stream and Internal Npgsql bug: unexpected value 0 of enum BackendMessageCode are appearing only during when the Connector Crate Caching is disabled or not.

So if we could get this error fixed, so that our Connector Crate Caching is ON again, we could probably see if that's related to the Exceptions and Internal Npgsql bug

rhessinger

Hi @caspacokku_2900,

Would you be able to send us over a Cargo.toml that can recreate the errors you are seeing? Although this all points to server overload, that will at least allow me to attempt to recreate the errors you are seeing.

Thanks,
Rich

caspacokku_2900

I have put Concurrent request limit to = 70 because this has solved a bottleneck performance issue btw.

[package]
name = "nytest"
version = "0.1.0"
edition = "2024"

[dependencies]
anyhow = "*"
bollard = "0.19"
blake3 = "^1.5"
clap =  { version = "=4.5.32", features = ["wrap_help"] }
macaddr = { version = "1.0", features = ["serde_std"] }
optional_struct = { version = "0.5" }
regex-lite = { version = "0.1.6" }
tokio = { version = "1.37.0", features = ["rt-multi-thread", "macros", "net", "sync", "io-util", "time", "fs"] }
serde = { version = "*", features = ["derive"] }
serde_json = "1.0"
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
tracing-journald = { version = "0.3"}
nix = "*"
scopeguard = "1.2"
aws-lc-sys = "0.35"

Make sure to enable Connector Crate Caching. If the cache is empty make sure to fill it up by doing a cargo build then clean the cache on the machine

cargo build
rm -f ../.cargo/registry/cache/*
rm -f ../.cargo/registry/index/*
cargo clean

rhessinger

Hi @caspacokku_2900,

Thanks for sending an example over. I did test this, but I was unable to recreate the postgres errors you were seeing. If you are using ProGet as a mirror to crates.io, one thing that may be helpful is to modify the Dependency Resolution on the Feed Properties page. Standard resolution has to check in ProGet to determine if the dependency exists or not to determine when to tall the cargo client to use ProGet vs crates.io. That will help with the retries and getting rate limited on crates.io. These are things to just help reduce load from Cargo on the ProGet server.

Does this happen every time you pull packages using cargo?

Thanks,
Rich

caspacokku_2900

@rhessinger Sorry, I do not follow currently. Our postgres errors isn't a priority for us. I think you were the one to mention some corruption which we will have to deal with later. The "The given key 'version' was not present in the dictionary." error which I displayed here is a problem because we cannot have the Connector Crate Caching enabled, thus building very slowly. If we start the cache, it stops working after a while and doesn't work again until we turn off the cache.

We have also Concurrent request limit set to 70

Step to follow:

Create a feed called public-cargo to crates.io
Set Dependecy Resolution -> "always use this feed when the registry is unspecified"
Enable Connector Crate Caching
Enable Metadata Caching 100 queries for 30 min
Use Cargo.toml as displayed below:

[package]
name = "rusttest"
version = "0.0.0"
edition = "2021"

[dependencies]
anyhow = "1"
assert_matches = "1.5"
async-trait = "0.1"
bincode = { version = "2", features = ["serde"] }
bindgen = "0.72"
blake3 = "1.5"
bollard = "0.20"
bytes = "1"
cidr = "0.3"
clap = "4.5"
criterion = "0.8"
csv = "1.3"
cursive = { version = "0.21.1", default-features = false, features = ["crossterm-backend"] }
data-url = "0.3.1"
futures = "0.3.31"
futures-util = "0.3.31"
http = "1.2.0"
http-body-util = "0.1.2"
http-serde = "2"
humantime = "2.1"
hyper = "1"
hyper-rustls = "0.27.5"
hyper-util = "0.1"
itertools = "0.14"
jsonschema = { version = "0.40", default-features = false }
libc = "0.2.169"
memmap2 = "0.9.5"
mockall = "0.14"
nix = "0.31"
opcua = { version = "0.12", features = ["client", "console-logging", "vendored-openssl"] }
pgp = "0.18"
proptest = "1.4"
prost = "0.14"
prost-types = "0.14"
rand = "0.9"
regex-lite = "0.1.6"
rumqttc = "0.25"
russh = "0.54"
russh-sftp = "2.1"
rustls = { version = "0.23", default-features = true }
rustls-pemfile = { version = "2.2.0" }
serde = "1"
serde_bytes = "0.11.15"
serde_json = "1"
serde_with = "3.9"
serial_test = "3.1"
stopwatch2 = "2"
strum = "0.27"
tar = "0.4"
tempfile = "3.10"
thiserror = "2"
time = "0.3"
tokio = "1.47"
tokio-rustls = { version = "0.26", default-features = false }
tokio-stream = "0.1.17"
tokio-util = "0.7.13"
tonic = "0.14"
tonic-prost = "0.14"
tonic-prost-build = "0.14"
tracing = "0.1.41"
tracing-journald = "0.3"
tracing-subscriber = "0.3"
wasmtime = "41"

Edit your .cargo/config.toml:

[source.crates-io]
replace-with = "proget"

[registries.proget]
index = "sparse+https://proget.example.com/cargo/public-cargo/"
# Seems to be needed even though we allow anonymous access
token = ""

[registry]
default = "proget"
global-credential-providers = ["cargo:token"]

[net]
# network retries
retry = 10
# tells cargo to use git when fetching from the registry instead of the cargo default libgit2
git-fetch-with-cli = true

rhessinger

Hi @caspacokku_2900,

I apologize for the confusion. I meant that running through the test example you provided (looks like that comment has since been deleted) and this new one, I was unable to recreate both the PostgreSQL errors and the "The given key 'version' was not present in the dictionary." error. I'm thinking the dictionary error is related to either the PostgreSQL errors or a hiccup in the network connection where the response was not fully returned, but had a successful response code causing the request to still be cached. What is also confusing is that the cache should refresh the request after 30 minutes. This means if the caching is causing the issue, it should resolve itself within 30 minutes. Also, I see you have set the metadata caching to 100, which cargo hits pretty quickly. I typically set that to 1000 requests in a crates.io connector because cargo is a very chatty client.

The main reason I was asking for a reproduction case is that I wanted to rule out an error in parsing the cargo metadata. With this latest test case you provided, I can verify that everything appears to be parsing correctly. I ran this build 30 times back to back with your settings and connector caching enabled (both metadata and crate caching). In between each build, I cleared the local registry cache and ran cargo clean. I was not able to reproduce these errors or cause issues with the connector cache. This leads me to believe that there is an environmental factor that is causing this issue. Can you please check with your IT team to see if there is anything they are seeing (external HTTP request manipulation, network packets dropping, RAM correction errors, etc..) on your server?

Thanks,
Rich

caspacokku_2900

Hello.
It seems like I've found all the solutions to all of my problems. Thanks

Error #1:
warning: spurious network error (10 tries remaining): [28] Timeout was reached (Operation too slow. Less than 10 bytes/sec transferred the last 30 seconds)
Solution #1:
Thanks to you. I increased Cargo's metadata caching from 100 to 2000 for 1 hr. This may have solved this error.

Error #2:
The given key 'version' was not present in the dictionary
Solution #2:
This might aswell be a series of random errors sprouting because of a (as you said) response was not fully returned or maybe because we don't let enough requests through and they get queued, sometimes some packets don't arrive on time (even with increased timeout) or get malformed and therefore create wrong requests. This solution is aligned with 502 Bad Gateway aswell as the Exceptions and unexpected value. I increased the Concurrent request limit to 100 instead of 70. This way the cargo build did not wait downloading. We have alot more users so this got increased to 300.

Error #3:
ProGet returning 502 Bad Gateway
Solution #3:
Too many requests, ProGet web hanged. I turned on the Concurrent request limit.

Error #4:
Exception while reading from stream and Internal Npgsql bug: unexpected value 0 of enum BackendMessageCode
Solution #4:
This has disappeared completely after I increased the Concurrent request limit. I increased the Concurrent request limit to 100 instead of 70. This way the cargo build did not wait downloading. We have alot more users so this got increased to 300.

rhessinger

Hi @caspacokku_2900,

Thanks for posting this! We are happy to hear you up and running! We are currently reviewing a default value for the concurrent request limit in ProGet. As more and more package managers push towards a larger dependency tree, this limit is becoming a requirement for some feeds (cargo, npm, and NuGet being among the highest).

Thanks,
Rich