Posts made by rhessinger

Hi @michael-day_7391,

Thanks for sending this over! We'll get this resolved within the next couple of maintenance releases of ProGet.

Thanks,
Rich

Hi @michael-day_7391,

What version of Windows Server are you running?

When generating that command, ProGet will call Environment.UserName to determine the username the service is running as. I'm guessing this is returning the computer account instead of NETWORK SERVICE. I have crated a ticket, PG-3219, to review it, but I would like to test it on the same version of Windows Server as you.

Thanks,
Rich

Hi @michael-day_7391,

The docs say to use NETWORK SERVICE because that is the default account ProGet users when it's installed. You need to use the username of the account that the ProGet service is running as. If your ProGet service is running as MYDOMAIN\PROGET-SERVER$", then your netsh command should use that. If it is using NETWORK SERVICE, then you should be use NETWORK SERVICE. If you want to switch to a different username, then you need to first remove the urlacl by running the following, then re-add it with the correct name.

netsh http delete urlacl url==https://*:443/

The event viewer error Windows event viewer shows event id 1000, can be ignored. That is a default message that shows because we configure the URL binding in code.

The netsh http add sslcert command should only be used when you are using a haostname binding, since you are unsign *:443, you should not need this.

While you are working through this issue, I would change UseHttpsRedirection="True" to UseHttpsRedirection="False" in your config so you can still access the site over HTTP until you get this figured out. The ERR_CONNECTION_CLOSED message typically indicates an issue with the certificate binding. The most common issue that can cause this is when the permissions are incorrect on your certificate.

1. Open the Certificate Manager using Window's MMC
2. Navigate to Certificates (Local Computer) -> YOUR_STORE (ex: Personal) -> Certificates
3. Right click on the certificate and select All Tasks -> Manage Private Keys
4. And give read access to your service account

See more in our Troubleshooting guide.

Can you verify the permissions on the certificate?

Thanks,
Rich

Hi @caspacokku_2900,

I apologize for the confusion. I meant that running through the test example you provided (looks like that comment has since been deleted) and this new one, I was unable to recreate both the PostgreSQL errors and the "The given key 'version' was not present in the dictionary." error. I'm thinking the dictionary error is related to either the PostgreSQL errors or a hiccup in the network connection where the response was not fully returned, but had a successful response code causing the request to still be cached. What is also confusing is that the cache should refresh the request after 30 minutes. This means if the caching is causing the issue, it should resolve itself within 30 minutes. Also, I see you have set the metadata caching to 100, which cargo hits pretty quickly. I typically set that to 1000 requests in a crates.io connector because cargo is a very chatty client.

The main reason I was asking for a reproduction case is that I wanted to rule out an error in parsing the cargo metadata. With this latest test case you provided, I can verify that everything appears to be parsing correctly. I ran this build 30 times back to back with your settings and connector caching enabled (both metadata and crate caching). In between each build, I cleared the local registry cache and ran cargo clean. I was not able to reproduce these errors or cause issues with the connector cache. This leads me to believe that there is an environmental factor that is causing this issue. Can you please check with your IT team to see if there is anything they are seeing (external HTTP request manipulation, network packets dropping, RAM correction errors, etc..) on your server?

Thanks,
Rich

Hi @parthu-reddy,

That error looks to be an error that happens when the web server is shutting down and a message is stuck in the queue. We plan to fix this in the next maintenance release, PG-3214. It should be fin to ignore. Were you seeing something specific happening in ProGet when this error was happening?

Thanks,
Rich

Hi @caspacokku_2900,

Thanks for sending an example over. I did test this, but I was unable to recreate the postgres errors you were seeing. If you are using ProGet as a mirror to crates.io, one thing that may be helpful is to modify the Dependency Resolution on the Feed Properties page. Standard resolution has to check in ProGet to determine if the dependency exists or not to determine when to tall the cargo client to use ProGet vs crates.io. That will help with the retries and getting rate limited on crates.io. These are things to just help reduce load from Cargo on the ProGet server.

Does this happen every time you pull packages using cargo?

Thanks,
Rich

Hi @parthu-reddy,

Sorry about that, I got hung up on the request logging, let me dig into this a bit more. Are you using PostgreSQL or SQL Server?

Thanks,
Rich

Hi @parthu-reddy,

This was an oversight on our part. The UI just updates a value in the ProGet.confg file. If you update the ProGet.config file on the other two servers and add EnableRequestLogging="true" RequestLoggingDirectory="C:\ProGetLogs" to the WebServer node, that will enable HTTP request logging on those two servers as well.

I'm going to update the docs to include how to enable this when using a Load Balanced/HA configuration. I have also added a ticket, PG-3211, to update the UI to give guidance on this configuration.

Thanks,
Rich

Hi @caspacokku_2900,

Would you be able to send us over a Cargo.toml that can recreate the errors you are seeing? Although this all points to server overload, that will at least allow me to attempt to recreate the errors you are seeing.

Thanks,
Rich

Hi @matthias-schmitz_2037,

We actually were made aware of this by another customer this week (PG-3201) and we will be releasing a fix this Friday in ProGet 2025.19.

Thanks,
Rich

Hi @jolaka9284_9459,

We have pushed some improvements in ProGet 2025.18. These improvements include:

• Improved connector metadata caching (may require an increase in the number of cached requests)
• Improved index generation code
• A new Feed management setting for Dependency Resolution
  • Standard Resolution, check in ProGet if that dependency exists
  • Force all dependencies through ProGet
  • Always use this feed unless Registry is specified

In your case, since you are using ProGet as the mirror, I would suggest forcing all dependencies through ProGet, which can be configured on Feed -> Manage Feed.

Thanks,
Rich

Hi @kquinn_2909

That is why you are not seeing the builds. That build types API is returning the Id as WebProjectsReplicatorBuildOnChange, where the builds API is returning WebProjects_Replicator_BuildOnChange as the Id. Are you using any sort of shared build type?

Thanks,
Rich

Hi @koksime-yap_5909,

No problem! Happy to help!

Thanks,
Rich

Hi @kquinn_2909,

I'm working to recreate this issue, but I'm struggling to recreate it. Would you be able to send us the results of the following API?

app/rest/projects/WebProjects_Replicator?fields=buildTypes(buildType)

The thought I currently have is that the Build Type list is returning something different that is filtering out those builds (https://github.com/Inedo/inedox-teamcity/blob/7d447a5f4f3c3e38c98012c65e9db40afec224b6/TeamCity/InedoExtension/TeamCityClient.cs#L84)

Thanks,
Rich

Hi @koksime-yap_5909,

After looking into this further, although we can fix the import to handle the images without a media type, it will still be broken when you attempt to pull those images using the Docker client. This is because ProGet will only send the manifest as it is stored in ProGet, where Nexus will manipulate it to match the accept headers of the client. Since it is only returning the manifest as it is stored, the media type will be null which will cause the Docker client to fail with a missing or empty Content-Type header.

Since we cannot assume what the mediaType should be, I think it would be better to resolve those prior to importing into ProGet. The nice thing is you can use the import logs to identify what images are missing the media type. You can run the import multiple times and it will only import the missing images and layers.

Thanks,
Rich

Hi @koksime-yap_5909,

I apologize for the delay. This is next on my list to review and I will have an update for you on Monday.

Thanks,
Rich

Hi @udi-moshe_0021,

Would you be able to provide the Installation logs from a failed install? You can get them by rerunning the Offline installer and then clicking on the Logs tab.

The other ting I would try doing is try running ProGet from the command line and then sharing the output with us. That may tells us any startup issues the application has. To do that, open a command prompt, navigate to the install directory (cd "C:\Program Files\ProGet\Service"), and then run proget.exe run. Could you share the output from that?

Thanks,
Rich

Hi @henderkes,

No problem! Both these issues are expected to be fixed in this Friday's release of ProGet 2025.18.

Thanks,
Rich

Hi @henderkes,

So we have identified the issue. When it comes to your packages, there where two issues that were preventing them from being updated and installed.

1. The packages you provided to me are unsigned. According to the APK documentation, that is not supported. In reality, that is a case that they support and have supported for some time. So we have a ticket, PG-3192, to correct the checksum generation for unsigned packages.
2. When a checksum is not generated, ProGet was inserting the hash of the package instead, which would cause the index to fail.
   • In older versions of APK tools V2, this would fail the install of that package, but show you the all packages in the index.
   • In newer versions of APK Tools V2, it will stop processing once a package with a bad checksum is shown. Which is why it looked like only one package existed in the feed. Installing these packages will most likely fail with a checksum error.
   • In APK Tools v3, it errors on apk update and these packages cannot be installed at all.

We are going to be adding the checksum verification to the Feed integrity checks and a way to regenerate the checksums in ticket PG-3193. That should resolve all these issues for you. That also explains why my test cases worked fine, because most public index packages are signed packages.

Thanks,
Rich

Hi @udi-moshe_0021,

Our Windows test servers are a vanilla install of Server Standard (we test against a few different versions) with nothing but the defaults configured and the latest Microsoft updates applied. It is possible that specifying a different user could cause it. Just make sure that the user you specified has read, write, and modify permissions to the Config File Directory (typically C:\ProgramData\Inedo). Upon first start of the ProGet Service, it will create and setup the embedded database and update the ProGet.config file. Based on error, that looks to be where it is failing. I wonder if you are missing the modify permission on that directory and it is failing to update the ProGet.config file.

Thanks,
Rich

Thanks,
Rich

Hi @henderkes,

Sorry for the delay on this. Right now, there looks to be a couple issues I'm working through with this. I should have an update for you on Friday. I believe I have identified the issues, but I'm trying to confirm some things based on your APK packages. Just to confirm, the packages you shared are APK Tools V2 packages correct?

Thanks,
Rich

Hi @koksime-yap_5909,

Thanks for sharing the steps. I'm going to attempt to reproduce this and I'll let you know what i find!

Thanks,
Rich

Hi @udi-moshe_0021,

I just ran some tests with both 2025.16 and 2025.17 using the embedded database on a clean install of Windows Server 2022 and everything installed and ran as expected. The only other thing I could think of could be a permissions issue. What account did you have ProGet service running as?

Thanks,
Rich

Hi @koksime-yap_5909,

Ah ok. It make sense that we would not be able to pull those based on the accept headers. How many images do you have without a media type? Are these mostly old images or are there new ones missing this as well? Do you know how to recreate an image in Nexus without a media type?

Thanks,
Rich

Hi @henderkes,

Thanks for sending those packages over. I was able to recreate some issues with those packages and we are currently still looking for the cause. I'll send over an update once I have a bit more information.

Thanks,
Rich

Hi @koksime-yap_5909,

There are certain types of OCI images that are not supported in ProGet (covered in this forums post), but the application/vnd.oci.image.manifest.v1+json image type manifest is supported. I'm guessing this is a quirk with Artifactory. What I have learned through implementing this importer is that Artifactory is very dependent on the order of accept headers. It is possible that the order is not quite right. Our order for the importers are:

private static HttpRequestMessage CreateDockerHttpRequest(HttpMethod method, string url)
{
    var request = new HttpRequestMessage(method, url);
    // Order matters here, especially with Artifactory
    request.Headers.Accept.Add(new("application/vnd.docker.distribution.manifest.v2+json"));
    request.Headers.Accept.Add(new("application/vnd.docker.distribution.manifest.list.v2+json"));
    request.Headers.Accept.Add(new("application/vnd.docker.distribution.manifest.v1+prettyjws"));
    request.Headers.Accept.Add(new("application/json"));
    request.Headers.Accept.Add(new("application/vnd.oci.image.manifest.v1+json"));
    request.Headers.Accept.Add(new("application/vnd.oci.image.index.v1+json"));

    return request;
}

Is it possible to share an example Docker file that we could build an image that has this import image? That way I can verify it mixed with standard Docker images.

Thanks,
Rich

Hi @henderkes,

I did try to download that script to extract the repo you were referring to, but Alpine does not have bash installed, only ash, so I get an error when attempting to run it. Could you please provide me a direct link? To be honest, this all screams a configuration error on Alpine or an issue with the APK directly.

In my test, I created a feed in ProGet with a connector created pointing to "https://repos.zend.com/zendphp/apk_alpine320/x86_64/". I then added my feed to alpine, apk update, and then apk add php85zend. Everything worked as expected.

Thanks,
Rich

Hi @henderkes,

Can you provide where you are getting your test packages from? I have a feeling it has to do with them more than the index. I setup the environment you specified with alpine packages I pulled from Zend PHP and had no issues. I was able to update the index and install multiple packages.

Thanks,
Rich

Hi @henderkes,

Thank you for sending all this over! We are going to dig into this a bit further and will let you know what we find.

Thanks,
Rich

Hi @henderkes,

Based on the Alpine's Package Keeper documentation, although the apk-tools are at version 3, the index and packages are still v2. From what I can tell, there is no timeline on when the v2 Index format will be dropped and the use of the v3 index is an opt-in feature. Currently ProGet only supports the v2 index and package format. I have added apk V3 support to be reviewed for ProGet 2026.

Could you please provide a little more details around what operations are giving you the errors and the log output that shows the error?

Thanks,
Rich

Hi @jolaka9284_9458,

Thanks for providing all these details. I was able to pinpoint the reason for the slow down and why it specifically happens on some crates and not others. This is related to some code that we have to determine when to point cargo to pull dependencies from ProGet vs crates.io. Specifically this comes from cargo's API specs for dependencies:

registry — cargo metadata uses a value of null to indicate that the dependency comes from crates.io. The index uses a value of null to indicate that the dependency comes from the same registry as the index. When creating an index entry, a registry other than crates.io should translate a value of null to be https://github.com/rust-lang/crates.io-index and translate a URL that matches the current index to be null.

As you can see, the value specified in the metadata is different than the value the index needs to return. In ProGet, we will return null if the package exists in the feed (including connectors) and https://github.com/rust-lang/crates.io-index if it does not. This is to support the case when ProGet is not used as a mirror and instead for only local crates. Unfortunately the use ProGet as a mirror option is stored only in the client config and is not sent to ProGet.

This is the reason why crates with a lot of dependencies take longer to generate the index than ones that don't and why you'll occasionally get timeouts, but after the retry it works. We have some caching on this to help with performance, but it's not a forever cache.

I'm going to work on some potential improvements for this and will let you know when I have a solution ready. Unfortunately, the only workaround we have for this currently is to use a package approval workflow (like our npm Package Approval blog article).

Thanks,
Rich

Hi @jolaka9284_9458,

Can you please answer the following so I can get an idea of your environment setup?

• What version of ProGet you are running?
• Is your config.toml configured to proxy all cargo requests through ProGet (overriding [source.crates-io])?
• Do you have connector package caching and/or metadata caching enabled?
• Are you using SQL Server or PostgreSQL (the embedded database, InedoDB, etc..)?

I'm not able to test against Artifactory very easily, but testing against crates.io. I'm not seeing a huge difference in speed (~20 seconds in your test case). In my testing, the time varies greatly based on the number of versions a package has and speeds up significantly once the indexes have been cached locally.

My theory is that Artifactory caches these indexes on disk on the server or is not returning the complete dataset (hosted vs local), compared to ProGet where we create the package index each time it's requested and leave the local index caching to cargo. When ProGet generates it's package index, it reaches out to any connectors on the feed and then merges them with the local package versions stored in ProGet.

There are pros and cons to each solution and are only really noticeable when the local cargo cache is cleared. When cargo has a local cache, it will send a If-Modified-Since header in its index request, allowing that index to be skipped if nothing has changed recently.

Is it also possible to get a list of your HTTP requests and timings made during these? I don't need the response body, but the URL Path (host and feed names can be anonymized), response time taken, and response code is all I really need. In ProGet 2025, you can enable HTTP Request Logging to obtain these.

Thanks,
Rich

Hi @coboj59760_1341,

I just pushed a pre-release that includes the fix. Please install ProGet 25.0.17-ci.12 and that should resolve this issue. Please let us know if that resolves your issue.

Thanks,
Rich

Hi @coboj59760_1341,

Thank you very much for all the information on how to recreate the error. In Cargo 1.84.0 (released in January 2025), they changed the default value for default_features from false to true. In the case of the examples you provided, these cargo manifests did specify a value for default_features causing it to change to false once the package is cached in ProGet.

I have created a ticket, PG-3182, to track the fix. This will be released in next weeks release of ProGet. I can also provide a pre-release version of ProGet if you would like. Just let me know!

Thanks,
Rich

Hi @koksime-yap_5909,

Sounds good. The fix will be released on Friday December 19th.

Thanks,
rich

Hi @koksime-yap_5909,

Thank you very much for all the information. I was able to recreate your issue and I'm working on a fix right now. I have created a ticket, PG-3178, to track the fix and should have it resolved shortly. The next release is Friday of next week, but I can provide you with a pre-release to try as soon as the fix is ready if you would like. Please let me know.

Thanks,
Rich

Hi @koksime-yap_5909,

Thanks for clarifying that for me. In my testing, the name property always started with repository/<<REPOSITTORY_NAME>>/.... I'm going to dig into this further and should have an update for you tomorrow. One last question, do you have any rules in your reverse-proxy that would be stripping that out of components API?

Thanks,
Rich

Hi @koksime-yap_5909,

Thanks for sending this over! It will take me a but to dig through this. Just a couple of quick follow up questions:

1. What is psuedo-group? Is that a repository/registry name?
2. What is subgrouping/groups? It may be helpful to see a screenshot of this example inside of Nexus.

Thanks,
Rich

Hi @koksime-yap_5909,

Thanks for following up! Is it possible to share the contents of the JSON your API generates? You can do that by logging into Nexus and entering the following URL in your browser: https://«NEXUS_HOST_AND_PORT»/service/rest/v1/components?repository=«REPOSITORY_NAME»

Please not the Repository Name is your Docker Registry name.

The code does some stuff to strip off the registry name so only the namespace and image name are left. I'm guessing that something is being returned slightly differently. Can you also tell us what version of Nexus you are running? It might help so I can do more testing with that version.

Thanks,
Rich

Hi @koksime-yap_5909,

The separate proxy server port is what is causing the import issue with this. After looking at this further, the main issue is that specified httpPort is overriding the port, even when an https:// URL is being used. I created a ticket, PG-3155, to fix that and add the ability to override the URL used for the Docker API. This fix will be released in ProGet 2025.15. If you are interested, I can get you a pre-release build with the fix in it either later today or tomorrow. Just let me know!

Thanks,
Rich

Hi @felfert,

Thanks for sending that. Since there are no errors right at the beginning, there was no errors in the upgrade process. Most likely restarting the container will resolve the schema issue. I have created a ticket, PG-3154, to add some more output logging in the schema update code to better understand the source of this issue going forward. Please let us know if restarting the container after updating to the 2025.14 does not resolve the schema version mismatch.

Thanks,
Rich

Hi @felfert,

The database schema is updated when the container starts. In the cases where you get a schema mismatch, typically just restarting the container will resolve the issue. This seems to happen every now and again, only in Docker, but we have been unable to reproduce it on our end to fix it. Next time you attempt to update to the new image and see this issue, could you provide us with the output from the start of your container?

Thanks,
Rich

Hi @koksime-yap_5909,

ProGet uses a combination of the Nexus REST API and the Docker API to pull images from Nexus. Just like packages, only local images will be pulled (as in only hosted Nexus Docker registries are supported). The basic process is:

1. Get a list of images using the Nexus components API.
2. Get the Docker API endpoint using the Nexus registries API (this is what you would use in the Docker client)
3. Pull the image (manifest, layers, etc...) using the Nexus Docker API

I'm guessing the error is occurring building the URL to Docker API endpoint. When you say that you are using an external proxy, did you configure the subdomain and port in Nexus to be the subdomain and port of your proxy? Would you also be able to send us the JSON that is returned for the following GET request in the Nexus API?

https://«NEXUS_HOST_AND_PORT»/service/rest/v1/repositories/docker/hosted/«NEXUS_DOCKER_REGISTRY_NAME»

If you can send me the value for the url property and the object for docker property, I can tell you what URL ProGet using to connect to the Docker API. Feel free to obfuscate the host name, subdomain (if configured), and registry name. The JSON will look something like this:

{
  "name": "internal",
  "format": "docker",
  "type": "hosted",
  "url": "http://localhost:8081/repository/docker-example",
  
...

  "docker": {
    "v1Enabled": false,
    "forceBasicAuth": true,
    "httpPort": 8082,
    "httpsPort": 8083,
    "subdomain": "docker-a",
    "pathEnabled": true
  }
}

Thanks,
Rich

Hi @lukas-herzog_4227,

Please try to restart your ProGet container. That will trigger another check for the database upgrade and should fix the issue. If restarting your container does not fix the issue, please let us know.

Thanks,
Rich

Hi @aristo_4359,

It looks like this should be possible with most of the services. I created a ticket, PG-3117, to investigate and add the feature. We will let you know if we find anything that will prevent us from implementing this.

Thanks,
Rich

Hi @aristo_4359,

Thank you for sending that over. I have recreated the issue and created PG-3116 to track the fix. We should have this resolved within the next two maintenance releases of ProGet.

Thanks,
Rich

Hi @aristo_4359,

Is this all rpm packages or just specific versions of a package? Can you share package names and version on the ones that will not delete?

@aristo_4359 said in RPM Bulk Edit Delete does not work:

Also a suggestion, when user delete package the page ideally stay on same page, instead redirecting to main feed on deleting single package and redirecting to first page when deleting using bulk edit

Thanks for the suggestion. The reason we redirect to the feed page after deleting a single package is because we don't know if the package will still exist (has other versions and/or is a remote package) in that feed. Although it is not trivial to check before we redirect, it could be added. I am going to add this to the list to discuss for ProGet 2026.

Thanks,
Rich

Hi @m-ruf_4197,

Sure thing! I have added ticket PG-3115 to track the feature. It should be released within the next couple of maintenance releases of ProGet.

Thanks,
Rich

Hi @caterina,

Good catch! We'll get that fix merged in.

Thanks,
Rich

Hi @windowsteam_8276,

Thanks for bringing this to our attention. We just pushed Otter 2025.1 that includes a fix, Ot-522, for that issue. Please upgrade to that version and let us know if that resolves the issue. Thanks!

Thanks,
Rich