ProGet Slow with Active Directory (New) directory

[ryan.dick_9361]

Hello,

I have been testing with a trial version of ProGet 5.2.29 (we have a licensed version as well, trying to test on a test server before touching the production server). The testing is to try and switch us from using the LDAP directory to instead use the Active Directory (New) directory. We are doing this to try and have users from a different (but trusted) domain to have the ability to authenticate with the ProGet server.

The problem is on the test server, when I switch to Active Directory (New) the performance of the ProGet website becomes incredibly slow. It can take 5-8 minutes just for the page to initially load. This does not happen with the LDAP directory. I've searched around the forums but most of the related topics are quite old and I believe should be fixed.

I was curious if there are any troubleshooting tips that anyone has to assist with this situation?

I will add that we have quite a few trusted domains, and many of these aren't normally accessible on the network. I'm not sure if this is part of the problem. I am however using the "Only specified trusted domains" option and specifically specifying the two domains we are needing to use with ProGet.

Thanks,
Ryan

[atripp]

Hi Ryan,

We'd definitely appreciate any help in getting the forum posts updated, so if you remember which ones you visited, we can try to update them. It'd be nice if we can add a section like "slowness" to this document.. https://docs.inedo.com/docs/various/ldap/troubleshooting

Just to be clear, what do you mean by "Only specified trusted domains"? Can you send a screenshot of your configs?

Can you disable Integrated Windows Authentication from IIS?

About how big is your active directory installation? domains / users / etc?

Also as an FYI, here is the code we use (SDK 1.5 is used in ProGet 5.2, hence the 1.5 version instead of master): https://github.com/Inedo/inedox-inedocore/blob/1.5/InedoCore/InedoExtension/UserDirectories/ADUserDirectory.cs

Thanks.
Alana

[ryan.dick_9361]

The old posts I was referring to may have been more for the LDAP configuration instead of Active Directory (New) from looking at it again. And there may be less of them than I thought (I stepped away from testing on this for a few weeks, searches were done before then). The posts all seemed to suggest there was a bug that was fixed in a later version. For instance, https://forums.inedo.com/topic/817/the-proget-ldap-integration-is-still-very-slow-from-package-manager?_=1591190309380

The "Only specified trusted domains" is under the Edit Active Directory (New), where it asks for "Search Mode." I should have typed it "Specific List" instead of “specified trusted domains.” It seems it doesn't matter if it is "current domain only" or if it is "Specific List" to the two domains we are wanting to be able to authenticate with.

When testing under "current domain only" the settings are blank / default for the other settings (domains to search, domain controller host, NetBIOS name mapping). I can send a screenshot but would rather do that through email. If I'm trying with "specific list" I have to add the two domains to the specific list and to the NETBIOS name mapping for it to work.

It seems from testing yesterday and today that it may be related to the "Search recursively" option. I had tried toggling this on / off before but it seems you may need to reboot (or maybe just restart the ProGet service) before that takes effect. Turning it off fixes the performance issues after a reboot.

I ran a PowerShell script on our AD and it returned 2915 users and 2987 groups. Each user could potentially be a member of quite a large number of groups. We are a manufacturing company and each "job" has 1 distribution group, and 2 security groups, and it could be several years before each job is closed out and the users removed from the group.

I personally have 57 groups, but when searching recursively with PowerShell it balloons to 1792. The PowerShell query takes about 30 seconds to process my group membership recursively. With "search recursively" turned off in ProGet, the ProGet website takes 5-30 seconds to load. When I turn "search recursively" back on, it goes to 9 min 39 seconds for me to load the page. Second time was 10 mins 30 seconds.

I have a lot more groups than most users, though. I just ran the PowerShell script on one of the other users that is also testing, and he is a member of 258 groups when searching recursively, and 243 when searching non-recursively. He was saying it loaded in less than a minute with the "search recursively" on, but this was when there wasn't many defined permissions under Users and Tasks for the "Active Directory (New)" directory. I edited the database to move all the permissions that were working under LDAP directory 2 to directory 4 for Active Directory (New) and now he is back up to 8 mins for loading the page.

So it seems to be both the "search recursively" slows it down for users with a large number of groups, but it also slows down for users without as many groups, if a lot of permissions are defined in "Users and Tasks." When this is occurring we have 50 user-level and 148 group-level permissions. However those same permissions are fine under the LDAP directory, and also fine under LDAP on our licensed production 4.8.7 ProGet server.

I installed Wireshark on the ProGet server. I'm not an expert at interpreting Wireshark logs, but I can tell with the "search recursively" on it is slamming the domain controller with traffic.

I sent the code you had provided to another developer who is more familiar with C# to see if there is any testing that we can do with it. Still waiting back on that.

I've also thought of maybe trying to use two LDAP directories, one for each domain, and using the Hybrid directory to combine them. I'm not sure if this will work like I expect or not. Right now I can't get the 2nd LDAP to work, though. If I do the "Test Searching" on it, instead of finding the users it returns "Unspecified Error." I may not have the correct settings, though.

I did just do another test where I switched back to LDAP, and configured permissions using a group that had no direct members, and instead only nested groups. The permissions worked, so this seems to confirm in LDAP mode it is able to search the group’s recursively fine.

[atripp]

Thanks for the very detailed analysis.

The NetBIOS mapping is only used for Windows Integrated Authentication, and to convert names like SOMTHNG\username to username@something.local.

Unfortunately, as you noticed, the "Search recursively" can be painfully slow in some configurations. We're not sure why, it doesn't seem to necessarily be related to the number of groups, or the nesting levels.

The SearchGroupsRecursively option (see code) does not attempt to extract all groups, it simply crawls up the group tree; this is the only way that we could find to get users' nested groups, and we even asked to Microsoft who confirmed this was the way to do it.

HOWEVER, when I say "slow", I mean on the order of "30 seconds", like your PowerShell script. 9 minutes is crazy! Even 5 seconds to load a page, without that option, seems very slow, so there might be some underlying networking/configuration issues?

In any case we can make a few optimizations to recursive searching, but you shouldn't bee seing 5-second page load times in any case.

[simon.dixey_1544]

@atripp said in ProGet Slow with Active Directory (New) directory:

se we can make a few optimizations to recursive searching, but you shouldn't bee seing 5-second page load times in

We also get incredible slow performance with "Search Recursively" enabled, our AD directory is quite large (tens of thousands of users), and having read this post, we have improved performance now as a result of disabling "search recursively". Ours was taking around 30 seconds to load a page before, it was pretty painful!

I've developed an alternative to AD which suits our requirements and that is to use Atlassian Crowd - I'd be happy to share the source for that when it's finalised.

(I also have a support ticket to log with AD regarding "A referral was returned from the server", but I'll post that through the proper support channels.)