Posts made by rhessinger

Hi @sigurd-hansen_7559,

This is a known issue, PG-3246, and will be fixed on Friday in ProGet 2025.25.

Thanks,
Rich

Hi @sigurd-hansen_7559,

We can add back support to not do the recursive search. If you would like to use a search scope, we would recommend using the OpenLDAP/Generic LDAP user directory and specifying your own LDAP queries. With that said, I think that is overkill for this situation and adding a way to disable recursive groups will be the easiest. I will get that added on the April 3rd release of ProGet 2025.25.

Thanks,
Rich

Hi @sigurd-hansen_7559,

Let me start with a little background in V5 vs V4. With V5, our goal was to only support Active Directory because we have since created an OpenLDAP/Generic LDAP specific user directory. This allowed us to strip out the extra LDAP querying we had to do in V4 to support both. This significantly reduces the amount of LDAP queries to look up a user. It also allowed us to lean into specific Active Directory LDAP features that we could not do when supporting both.

One of those features we can now use is the built in LDAP query for a user's groups and a group's members. When we look up groups we use the following LDAP query:

(&(objectCategory=group)(member:1.2.840.113556.1.4.1941:=«USER_DISTINGUISHED_NAME»))

This means that we will return all groups (including nested groups) just like how Active Directory would on Windows without having to make a circular walk of the groups tree in a single LDAP request.

Could you answer a few questions for me?

1. Is your ProGet instance running on Windows or Docker?
2. When you say significant group memberships, approximately how many are there? (~10, ~20, ~50, etc...)
3. Is your domain part of a multi-directory forest?
4. In your V4 user directory, can you change the recursive group search to "Recursive Search (Active Directory Only)" and see if you see the same slow downs?

Thanks,
Rich

Hi @gbeckett,

Glad to hear it is working for you! Please let us know if you run into any other issues.

Thanks,
Rich

Hi @gbeckett,

I just pushed out a pre-release version of ProGet 2025.24-rc.3 and can be installed via InedoHub. Please see our our guide on installing pre-release verions for more information. Please let us know if you have nay problems.

Thanks,
Rich

Hi @phil-sutherland,

Thanks for sending this over to us. This looks to be a regression in the PostgreSQL migration. I have created a ticket, PG-3241, to fix the issue. This will be fixed in the next maintenance release of ProGet 2025.24, due out on March 20, 2026.

Thanks,
Rich

Hi @gbeckett,

It looks like there was a merge issue and part of this fix was undone. I have created a new ticket, PG-3240 to fix it. If you would like, I can create a pre-release of ProGet 2025.24 with the fix that you can install now. If not, the fix will be released at the end of next week in ProGet 2025.24.

Thanks,
Rich

Hi @Valentijn,

We had a hiccup in one of our edge nodes that was preventing this version from showing up. If you check now, it should be available.

Thanks,
Rich

Hi @phil-sutherland_3118,

There is nothing off the top of my head that would fix this. The next step would be to see your LDAP queries and potentially some example data. It would probably be best to wait until you get a license if you do not feel comfortable sharing that data on the forums. If you care to look, you can see our code for both our AD User Directories and our OpenLDAP/Generic LDAP user directories on our GitHub repo: https://github.com/inedo/inedox-inedocore

Thanks,
Rich

Hi @sirko_6724,

Please let us know what you figure out. When we built our OpenLDAP user directory, we built it based on OpenLDAP's best practices for groups and this is the first time we have seen an issue with groups missing the member attribute. I know you are adverse to making changes to your OpenLDAP server, but another option would be to populate the users on the groups using a Dynamic List. That would allow you to add the user's dn to the group without having to change all the records.

Thanks,
Rich

Hi @phil-sutherland_3118,

Just to confirm, are you using an Active Directory domain or OpenLDAP doimain in your environment? Your initial post makes it sound like you are using Active Directory and not OpenLDAP. If you are connecting to Active Directory, you can use the following two LDAP settings to get groups and members:

• For List User's Groups: (&(objectCategory=group)(member:1.2.840.113556.1.4.1941:=%s)) and for get a groups
• For List Group's Members: (&(objectCategory=user)(memberOf:1.2.840.113556.1.4.1941:=%s))

These will only work when using our OpenLDAP/GenericLDAP user directory and connecting to an Active Directory domain, OpenLDAP domains do not implement these special queries.

That other forums post is a OpenLDAP domain that is using custom attributes for groups.

Thanks,
Rich

Hi @phil-sutherland_3118,

That is expected behavior. ProGet will always strip off the domain suffix from the username. Users in the past have created multiple user directories (one for each suffix) and then using the username@suffix (ex: user.name@company.group) as the login.

An alternative approach is to use the OpenLDAP/GenericLDAP user directory instead. It requires you to enter the AD-based LDAP attributes and queries, but it will not strip off the domain suffix.

Thanks,
Rich

Hi @caspacokku_2900,

Thanks for posting this! We are happy to hear you up and running! We are currently reviewing a default value for the concurrent request limit in ProGet. As more and more package managers push towards a larger dependency tree, this limit is becoming a requirement for some feeds (cargo, npm, and NuGet being among the highest).

Thanks,
Rich

Hi @michael-day_7391,

Thanks for sending this over! We'll get this resolved within the next couple of maintenance releases of ProGet.

Thanks,
Rich

Hi @michael-day_7391,

What version of Windows Server are you running?

When generating that command, ProGet will call Environment.UserName to determine the username the service is running as. I'm guessing this is returning the computer account instead of NETWORK SERVICE. I have crated a ticket, PG-3219, to review it, but I would like to test it on the same version of Windows Server as you.

Thanks,
Rich

Hi @michael-day_7391,

The docs say to use NETWORK SERVICE because that is the default account ProGet users when it's installed. You need to use the username of the account that the ProGet service is running as. If your ProGet service is running as MYDOMAIN\PROGET-SERVER$", then your netsh command should use that. If it is using NETWORK SERVICE, then you should be use NETWORK SERVICE. If you want to switch to a different username, then you need to first remove the urlacl by running the following, then re-add it with the correct name.

netsh http delete urlacl url==https://*:443/

The event viewer error Windows event viewer shows event id 1000, can be ignored. That is a default message that shows because we configure the URL binding in code.

The netsh http add sslcert command should only be used when you are using a haostname binding, since you are unsign *:443, you should not need this.

While you are working through this issue, I would change UseHttpsRedirection="True" to UseHttpsRedirection="False" in your config so you can still access the site over HTTP until you get this figured out. The ERR_CONNECTION_CLOSED message typically indicates an issue with the certificate binding. The most common issue that can cause this is when the permissions are incorrect on your certificate.

1. Open the Certificate Manager using Window's MMC
2. Navigate to Certificates (Local Computer) -> YOUR_STORE (ex: Personal) -> Certificates
3. Right click on the certificate and select All Tasks -> Manage Private Keys
4. And give read access to your service account

See more in our Troubleshooting guide.

Can you verify the permissions on the certificate?

Thanks,
Rich

Hi @caspacokku_2900,

I apologize for the confusion. I meant that running through the test example you provided (looks like that comment has since been deleted) and this new one, I was unable to recreate both the PostgreSQL errors and the "The given key 'version' was not present in the dictionary." error. I'm thinking the dictionary error is related to either the PostgreSQL errors or a hiccup in the network connection where the response was not fully returned, but had a successful response code causing the request to still be cached. What is also confusing is that the cache should refresh the request after 30 minutes. This means if the caching is causing the issue, it should resolve itself within 30 minutes. Also, I see you have set the metadata caching to 100, which cargo hits pretty quickly. I typically set that to 1000 requests in a crates.io connector because cargo is a very chatty client.

The main reason I was asking for a reproduction case is that I wanted to rule out an error in parsing the cargo metadata. With this latest test case you provided, I can verify that everything appears to be parsing correctly. I ran this build 30 times back to back with your settings and connector caching enabled (both metadata and crate caching). In between each build, I cleared the local registry cache and ran cargo clean. I was not able to reproduce these errors or cause issues with the connector cache. This leads me to believe that there is an environmental factor that is causing this issue. Can you please check with your IT team to see if there is anything they are seeing (external HTTP request manipulation, network packets dropping, RAM correction errors, etc..) on your server?

Thanks,
Rich

Hi @parthu-reddy,

That error looks to be an error that happens when the web server is shutting down and a message is stuck in the queue. We plan to fix this in the next maintenance release, PG-3214. It should be fin to ignore. Were you seeing something specific happening in ProGet when this error was happening?

Thanks,
Rich

Hi @caspacokku_2900,

Thanks for sending an example over. I did test this, but I was unable to recreate the postgres errors you were seeing. If you are using ProGet as a mirror to crates.io, one thing that may be helpful is to modify the Dependency Resolution on the Feed Properties page. Standard resolution has to check in ProGet to determine if the dependency exists or not to determine when to tall the cargo client to use ProGet vs crates.io. That will help with the retries and getting rate limited on crates.io. These are things to just help reduce load from Cargo on the ProGet server.

Does this happen every time you pull packages using cargo?

Thanks,
Rich

Hi @parthu-reddy,

Sorry about that, I got hung up on the request logging, let me dig into this a bit more. Are you using PostgreSQL or SQL Server?

Thanks,
Rich

Hi @parthu-reddy,

This was an oversight on our part. The UI just updates a value in the ProGet.confg file. If you update the ProGet.config file on the other two servers and add EnableRequestLogging="true" RequestLoggingDirectory="C:\ProGetLogs" to the WebServer node, that will enable HTTP request logging on those two servers as well.

I'm going to update the docs to include how to enable this when using a Load Balanced/HA configuration. I have also added a ticket, PG-3211, to update the UI to give guidance on this configuration.

Thanks,
Rich

Hi @caspacokku_2900,

Would you be able to send us over a Cargo.toml that can recreate the errors you are seeing? Although this all points to server overload, that will at least allow me to attempt to recreate the errors you are seeing.

Thanks,
Rich

Hi @matthias-schmitz_2037,

We actually were made aware of this by another customer this week (PG-3201) and we will be releasing a fix this Friday in ProGet 2025.19.

Thanks,
Rich

Hi @jolaka9284_9459,

We have pushed some improvements in ProGet 2025.18. These improvements include:

• Improved connector metadata caching (may require an increase in the number of cached requests)
• Improved index generation code
• A new Feed management setting for Dependency Resolution
  • Standard Resolution, check in ProGet if that dependency exists
  • Force all dependencies through ProGet
  • Always use this feed unless Registry is specified

In your case, since you are using ProGet as the mirror, I would suggest forcing all dependencies through ProGet, which can be configured on Feed -> Manage Feed.

Thanks,
Rich

Hi @kquinn_2909

That is why you are not seeing the builds. That build types API is returning the Id as WebProjectsReplicatorBuildOnChange, where the builds API is returning WebProjects_Replicator_BuildOnChange as the Id. Are you using any sort of shared build type?

Thanks,
Rich

Hi @koksime-yap_5909,

No problem! Happy to help!

Thanks,
Rich

Hi @kquinn_2909,

I'm working to recreate this issue, but I'm struggling to recreate it. Would you be able to send us the results of the following API?

app/rest/projects/WebProjects_Replicator?fields=buildTypes(buildType)

The thought I currently have is that the Build Type list is returning something different that is filtering out those builds (https://github.com/Inedo/inedox-teamcity/blob/7d447a5f4f3c3e38c98012c65e9db40afec224b6/TeamCity/InedoExtension/TeamCityClient.cs#L84)

Thanks,
Rich

Hi @koksime-yap_5909,

After looking into this further, although we can fix the import to handle the images without a media type, it will still be broken when you attempt to pull those images using the Docker client. This is because ProGet will only send the manifest as it is stored in ProGet, where Nexus will manipulate it to match the accept headers of the client. Since it is only returning the manifest as it is stored, the media type will be null which will cause the Docker client to fail with a missing or empty Content-Type header.

Since we cannot assume what the mediaType should be, I think it would be better to resolve those prior to importing into ProGet. The nice thing is you can use the import logs to identify what images are missing the media type. You can run the import multiple times and it will only import the missing images and layers.

Thanks,
Rich

Hi @koksime-yap_5909,

I apologize for the delay. This is next on my list to review and I will have an update for you on Monday.

Thanks,
Rich

Hi @udi-moshe_0021,

Would you be able to provide the Installation logs from a failed install? You can get them by rerunning the Offline installer and then clicking on the Logs tab.

The other ting I would try doing is try running ProGet from the command line and then sharing the output with us. That may tells us any startup issues the application has. To do that, open a command prompt, navigate to the install directory (cd "C:\Program Files\ProGet\Service"), and then run proget.exe run. Could you share the output from that?

Thanks,
Rich

Hi @henderkes,

No problem! Both these issues are expected to be fixed in this Friday's release of ProGet 2025.18.

Thanks,
Rich

Hi @henderkes,

So we have identified the issue. When it comes to your packages, there where two issues that were preventing them from being updated and installed.

1. The packages you provided to me are unsigned. According to the APK documentation, that is not supported. In reality, that is a case that they support and have supported for some time. So we have a ticket, PG-3192, to correct the checksum generation for unsigned packages.
2. When a checksum is not generated, ProGet was inserting the hash of the package instead, which would cause the index to fail.
   • In older versions of APK tools V2, this would fail the install of that package, but show you the all packages in the index.
   • In newer versions of APK Tools V2, it will stop processing once a package with a bad checksum is shown. Which is why it looked like only one package existed in the feed. Installing these packages will most likely fail with a checksum error.
   • In APK Tools v3, it errors on apk update and these packages cannot be installed at all.

We are going to be adding the checksum verification to the Feed integrity checks and a way to regenerate the checksums in ticket PG-3193. That should resolve all these issues for you. That also explains why my test cases worked fine, because most public index packages are signed packages.

Thanks,
Rich

Hi @udi-moshe_0021,

Our Windows test servers are a vanilla install of Server Standard (we test against a few different versions) with nothing but the defaults configured and the latest Microsoft updates applied. It is possible that specifying a different user could cause it. Just make sure that the user you specified has read, write, and modify permissions to the Config File Directory (typically C:\ProgramData\Inedo). Upon first start of the ProGet Service, it will create and setup the embedded database and update the ProGet.config file. Based on error, that looks to be where it is failing. I wonder if you are missing the modify permission on that directory and it is failing to update the ProGet.config file.

Thanks,
Rich

Thanks,
Rich

Hi @henderkes,

Sorry for the delay on this. Right now, there looks to be a couple issues I'm working through with this. I should have an update for you on Friday. I believe I have identified the issues, but I'm trying to confirm some things based on your APK packages. Just to confirm, the packages you shared are APK Tools V2 packages correct?

Thanks,
Rich

Hi @koksime-yap_5909,

Thanks for sharing the steps. I'm going to attempt to reproduce this and I'll let you know what i find!

Thanks,
Rich

Hi @udi-moshe_0021,

I just ran some tests with both 2025.16 and 2025.17 using the embedded database on a clean install of Windows Server 2022 and everything installed and ran as expected. The only other thing I could think of could be a permissions issue. What account did you have ProGet service running as?

Thanks,
Rich

Hi @koksime-yap_5909,

Ah ok. It make sense that we would not be able to pull those based on the accept headers. How many images do you have without a media type? Are these mostly old images or are there new ones missing this as well? Do you know how to recreate an image in Nexus without a media type?

Thanks,
Rich

Hi @henderkes,

Thanks for sending those packages over. I was able to recreate some issues with those packages and we are currently still looking for the cause. I'll send over an update once I have a bit more information.

Thanks,
Rich

Hi @koksime-yap_5909,

There are certain types of OCI images that are not supported in ProGet (covered in this forums post), but the application/vnd.oci.image.manifest.v1+json image type manifest is supported. I'm guessing this is a quirk with Artifactory. What I have learned through implementing this importer is that Artifactory is very dependent on the order of accept headers. It is possible that the order is not quite right. Our order for the importers are:

private static HttpRequestMessage CreateDockerHttpRequest(HttpMethod method, string url)
{
    var request = new HttpRequestMessage(method, url);
    // Order matters here, especially with Artifactory
    request.Headers.Accept.Add(new("application/vnd.docker.distribution.manifest.v2+json"));
    request.Headers.Accept.Add(new("application/vnd.docker.distribution.manifest.list.v2+json"));
    request.Headers.Accept.Add(new("application/vnd.docker.distribution.manifest.v1+prettyjws"));
    request.Headers.Accept.Add(new("application/json"));
    request.Headers.Accept.Add(new("application/vnd.oci.image.manifest.v1+json"));
    request.Headers.Accept.Add(new("application/vnd.oci.image.index.v1+json"));

    return request;
}

Is it possible to share an example Docker file that we could build an image that has this import image? That way I can verify it mixed with standard Docker images.

Thanks,
Rich

Hi @henderkes,

I did try to download that script to extract the repo you were referring to, but Alpine does not have bash installed, only ash, so I get an error when attempting to run it. Could you please provide me a direct link? To be honest, this all screams a configuration error on Alpine or an issue with the APK directly.

In my test, I created a feed in ProGet with a connector created pointing to "https://repos.zend.com/zendphp/apk_alpine320/x86_64/". I then added my feed to alpine, apk update, and then apk add php85zend. Everything worked as expected.

Thanks,
Rich

Hi @henderkes,

Can you provide where you are getting your test packages from? I have a feeling it has to do with them more than the index. I setup the environment you specified with alpine packages I pulled from Zend PHP and had no issues. I was able to update the index and install multiple packages.

Thanks,
Rich

Hi @henderkes,

Thank you for sending all this over! We are going to dig into this a bit further and will let you know what we find.

Thanks,
Rich

Hi @henderkes,

Based on the Alpine's Package Keeper documentation, although the apk-tools are at version 3, the index and packages are still v2. From what I can tell, there is no timeline on when the v2 Index format will be dropped and the use of the v3 index is an opt-in feature. Currently ProGet only supports the v2 index and package format. I have added apk V3 support to be reviewed for ProGet 2026.

Could you please provide a little more details around what operations are giving you the errors and the log output that shows the error?

Thanks,
Rich

Hi @jolaka9284_9458,

Thanks for providing all these details. I was able to pinpoint the reason for the slow down and why it specifically happens on some crates and not others. This is related to some code that we have to determine when to point cargo to pull dependencies from ProGet vs crates.io. Specifically this comes from cargo's API specs for dependencies:

registry — cargo metadata uses a value of null to indicate that the dependency comes from crates.io. The index uses a value of null to indicate that the dependency comes from the same registry as the index. When creating an index entry, a registry other than crates.io should translate a value of null to be https://github.com/rust-lang/crates.io-index and translate a URL that matches the current index to be null.

As you can see, the value specified in the metadata is different than the value the index needs to return. In ProGet, we will return null if the package exists in the feed (including connectors) and https://github.com/rust-lang/crates.io-index if it does not. This is to support the case when ProGet is not used as a mirror and instead for only local crates. Unfortunately the use ProGet as a mirror option is stored only in the client config and is not sent to ProGet.

This is the reason why crates with a lot of dependencies take longer to generate the index than ones that don't and why you'll occasionally get timeouts, but after the retry it works. We have some caching on this to help with performance, but it's not a forever cache.

I'm going to work on some potential improvements for this and will let you know when I have a solution ready. Unfortunately, the only workaround we have for this currently is to use a package approval workflow (like our npm Package Approval blog article).

Thanks,
Rich

Hi @jolaka9284_9458,

Can you please answer the following so I can get an idea of your environment setup?

• What version of ProGet you are running?
• Is your config.toml configured to proxy all cargo requests through ProGet (overriding [source.crates-io])?
• Do you have connector package caching and/or metadata caching enabled?
• Are you using SQL Server or PostgreSQL (the embedded database, InedoDB, etc..)?

I'm not able to test against Artifactory very easily, but testing against crates.io. I'm not seeing a huge difference in speed (~20 seconds in your test case). In my testing, the time varies greatly based on the number of versions a package has and speeds up significantly once the indexes have been cached locally.

My theory is that Artifactory caches these indexes on disk on the server or is not returning the complete dataset (hosted vs local), compared to ProGet where we create the package index each time it's requested and leave the local index caching to cargo. When ProGet generates it's package index, it reaches out to any connectors on the feed and then merges them with the local package versions stored in ProGet.

There are pros and cons to each solution and are only really noticeable when the local cargo cache is cleared. When cargo has a local cache, it will send a If-Modified-Since header in its index request, allowing that index to be skipped if nothing has changed recently.

Is it also possible to get a list of your HTTP requests and timings made during these? I don't need the response body, but the URL Path (host and feed names can be anonymized), response time taken, and response code is all I really need. In ProGet 2025, you can enable HTTP Request Logging to obtain these.

Thanks,
Rich

Hi @coboj59760_1341,

I just pushed a pre-release that includes the fix. Please install ProGet 25.0.17-ci.12 and that should resolve this issue. Please let us know if that resolves your issue.

Thanks,
Rich

Hi @coboj59760_1341,

Thank you very much for all the information on how to recreate the error. In Cargo 1.84.0 (released in January 2025), they changed the default value for default_features from false to true. In the case of the examples you provided, these cargo manifests did specify a value for default_features causing it to change to false once the package is cached in ProGet.

I have created a ticket, PG-3182, to track the fix. This will be released in next weeks release of ProGet. I can also provide a pre-release version of ProGet if you would like. Just let me know!

Thanks,
Rich

Hi @koksime-yap_5909,

Sounds good. The fix will be released on Friday December 19th.

Thanks,
rich

Hi @koksime-yap_5909,

Thank you very much for all the information. I was able to recreate your issue and I'm working on a fix right now. I have created a ticket, PG-3178, to track the fix and should have it resolved shortly. The next release is Friday of next week, but I can provide you with a pre-release to try as soon as the fix is ready if you would like. Please let me know.

Thanks,
Rich

Hi @koksime-yap_5909,

Thanks for clarifying that for me. In my testing, the name property always started with repository/<<REPOSITTORY_NAME>>/.... I'm going to dig into this further and should have an update for you tomorrow. One last question, do you have any rules in your reverse-proxy that would be stripping that out of components API?

Thanks,
Rich