Catastrophic failure when switching User Directory

[apxltd]

Thanks @Stephen-Schaff

Great idea, I added a product change issue (PG-1906) that will block enabling windows auth within ProGet unless the user can be found. This should greatly reduce the problems.

[Stephen.Schaff]

<sigh>

We tried again today. I set the NETBIOS mapping and it still failed....

I am not sure what to do now.

We are stuck. I can't update user permissions because of a bug in the Legacy LDAP setup I use. And I can't switch to the newer Active Directory LDAP setting because of... I don't know why. And after I make a change, I can't change back (I have to restore the server and database from a back up to get it working again).

I tried using your Active Directory debug tool and it the settings work perfectly there. The same settings fail in ProGet.

As much as I really like ProGet (both price and features), I am going to have to spend some time today to go shopping for a product that can integrate with my Active Directory system. :(

I can probably convince my Operations Team to give it one more go if you have any suggestions.

[apxltd]

@Stephen-Schaff I'm sorry that you're having configuration challenges; active directory is very complicated, especially on old servers, complex domains, etc.

Another thing to try is visiting /debug/integrated-auth in your instance, but that only uses the current directory provider (so you need to switch first), and it may not provide any new information.

I tried using your Active Directory debug tool and it the settings work perfectly there. The same settings fail in ProGet.

The software/code is identical, so if this is happening then it means something is different in your production environment. The most likely case is that the user account running ProGet has different permissions or is otherwise unable to query the directory in the same way. You could also try setting LoadUserProfile=true, that seems to help with Microsoft's libraries for unknown reasons.

We are happy to add product enhancements to make this easier to debug, and identify what's wrong. At this point, it's a big guessing game as to what's different about your environment, because it works for everyone else.

As much as I really like ProGet (both price and features), I am going to have to spend some time today to go shopping for a product that can integrate with my Active Directory system. :(

I can probably convince my Operations Team to give it one more go if you have any suggestions.

We're doing our best to help you and try to solve this bizarre and complicated problem that only seems to impact you. I know you're frustrated, and we are too, because we don't like seeing our software not work in some environments. We could just as easily blame you or whoever configured your environment, or perhaps Microsoft, who wrote the buggy libraries we use.

But you're a smart guy, and you must know that this abrasive communication style causes only unnecessary stress to the people you interact with. You are certainly intelligent to know it's not appropriate or productive, either.

So, please be more considerate. Or if this is your preferred vendor management approach and communication style, then do "go shopping" and do find another vendor.

[Stephen.Schaff]

Another thing to try is visiting /debug/integrated-auth in your instance

I am not sure what this is, but I am willing to try out anything. Do you have any instructions on how to do this?

The software/code is identical, so if this is happening then it means something is different in your production environment.

I only have one environment for this. It is all production. We have one Active Directory setup at my company. I used the same one for ProGet and for the tool.

Though it may be important to note, the "Test Tool" has the same functionality as the test tool embedded into ProGet. That test tool works fine (both before and after the change of the User Directory). It is ProGet itself that cannot find the user. The Test Tool can always find the user (both the one integrated into ProGet and the one that is run from Visual Studio.)

this abrasive communication style causes only unnecessary stress to the people you interact with

I apologize that this came across as abrasive. Basically I am trying to communicate that I am suck. I first reported my problem over a month ago. My project is dependent on being able to setup permissions. And I cannot find a way to do that. And the debug tools all show that everything is working great. Up till when I make the switch. Then the user cannot be found.

I cannot test this out in a non-production environment because we have not purchased a second license for testing. And the free version does not include the very thing I need to test (AD Integration). So I have very short downtime windows to try to troubleshoot this.

Again, I am sorry my communication was abrasive. I was trying to say that I can't leave it like this. I have to have working permissions. I don't want that to be with another product. But permissions are required.

[apxltd]

Thanks @Stephen-Schaff, we will definitely figure out how to get this working.

The /debug/integrated-auth is just a URL you can type in, like https://<yourinstance /debug/integrated-auth. It provides some details about the current configuration, like this:

https://proget.inedo.com/debug/integrated-auth

However, it will only help us identify potential problems once you've changed the configuration.

--

Since downtime is a concern, let's start by getting a testing environment. Can you install ProGet on a similar server, and then just use the Trial license key? You can request one at my.inedo.com.

If this is a "permanent server" where you wanted a testing/training place for all the various functionality, we would ask for a license... but this usage is fine, most especially given all the headaches.

Once you have that, testing should be a lot easier.

If we can reproduce the problem on the test server, then that's good news - because we can try some basic config changes.

The first thing that comes to mind is set LoadUserProfile=true on IIS AppPool. That "works wonders" due to the bizarre behaviors of some of Microsoft's libraries. The second is trying a different username. We can just keep going, all the way down to running the webserver outside of IIS.

We can also just keep iterating on code, finding places to add new debug information, etc.

At this point, even if you could test before switching, it wouldn't help... because it would still not work. Making it easier to switch without needing to reset password, is a problem for a different day... for now we need to figure out how to add diagnostic information.

If it's as simple as LoadUserProfile=true, then and lesson learned -- I guess we'll try to detect and warn, or atleast be trained on support to know how to respond.

And if we can't reproduce the problem on test, then we'll just keep coming up with ways to figure it out quickly while minimizing downtime. For example, if we absolutely had to, we could get a second ProGet instance on one server, just to try this out, and not impact the first.

Anyways from here --- let's see what we can do on the test server!

[apxltd]

@Stephen-Schaff FYI; looks like this got added as a result of your feedback. Easy change, mostly to prevent the "locked out" symptom when switching.

Let us know what you find out about test system, etc.

[Stephen.Schaff]

@apxltd

We tried to swap User Directories again. (Since we did it on Friday morning, 5.3.25 had not dropped yet. So we only upgraded to 5.3.24, not 5.3.25 that had the feature you showed above. I hope to try again on Friday to try that feature out.)

We first upgraded to 5.3.24. Then we set LoadUserProfile=true. Then we swapped the User Directory. Then we rebooted the server.

It once again did not find the user.

We went to the debug/integrated-auth URL and this is what it showed:

IntegratedAuthEnabled:   False
LOGON_USER:              MY-NETBIOS\1234
HttpContext.User.Identity.Name: MY-NETBIOS\1234
HttpContext.User.Identity.IsAuthenticated:      True
WebUserContext.UserInfo.Name:   Anonymous
---------
Current User Directory:
Queries the current domain, global catalog for trusted domains, or a specific li
---------
Domain:         MY-NETBIOS
Id:             1234
---------
LOGON_USER parsed as: 1234@mydomain.net
Username not found.
Additional messages:
 - Trying to a Users search for principal "1234@mydomain.net"
 - Search string is "(&(|(objectCategory=user)(objectCategory=msDS-GroupManagedServiceAccount))(sAMAccountName=1234))"...
 - Domain alias "mydomain.net" will be used.
 - Searching domain mydomain.net...
 - Trying to a Users search for principal "MY-NETBIOS\1234"...
 - Search string is "(&(|(objectCategory=user)(objectCategory=msDS-GroupManagedServiceAccount))(sAMAccountName=MY-NETBIOS\5C1234))"...
 - No domain specified, searching through aliases.
 - Searching domain mydomain.net...
 - Principal not found.

NOTE: I have replaced the text for my NetBios with MY-NETBIOS and my domain with mydomain.net and the user name with 1234 to de-identify this data.

I plugged the first query into a powershell command and it worked perfectly:

C:\src> Get-ADUser -Credential My-ProGetADAccount -LDAPFilter '(&(|(objectCategory=user)(objectCategory=msDS-GroupManagedServiceAccount))(sAMAccountName=1234))' | Format-Table Name,SamAccountName -A

Name           SamAccountName
----           --------------
Doe, John      1234

(Again, I replaced anything identifying when pasting here.)

The first query it tried should have worked. I am confused why it did not. I am hopeful you have some advice on what else we could try.

[atripp]

@Stephen-Schaff ah that's too bad, quite frustrating! Any luck setting up test server, so it doesn't' inconvenience production?

I don't why it didn't work, but it's not what I would have expected.

We had been assuming this was failing in the TryParseLoginUserName method, which is where that NETBIOS mapping occurs. It seems to be working fine, and is surprising to see.

Instead, it seems to be failing in TryGetUserAsync, which calls the TryGetPrincipal method. The TryGetUser method is called in a bunch of places (and when it returns null for an authenticated user, you'll get that can't find user error), but it's also on the "Configure User Directory Page", when you hit the "test get user" button.

You showed that you tested the connectivity using "test search", but there's good reason one query (get) would work, but not the other (search).

That doesn't make a lot of sense to me. I'm thinking, another test from that page is in order.

Here's the (messy) code for /debug/integrated-auth.

                WriteLine($"Id:\t\t{domain.Id}");
                {
                    var messages = new List<string>();

                    WriteLine("---------");
                    var ad = WebUserContext.CurrentUserDirectory;

                    ad.MessageLogged +=
                        (s, e) => messages.Add(e.Message);

                    var parsedLogonUser = ad.TryParseLogonUser(context.Request.ServerVariables["LOGON_USER"]);
                    if (parsedLogonUser == null)
                        WriteLine("Could not parse LOGON_USER.");
                    else
                        WriteLine("LOGON_USER parsed as: " + parsedLogonUser.Name);

                    var user = await ad.TryGetUserAsync(context.Request.ServerVariables["LOGON_USER"]);
                    if (user == null)
                        WriteLine("Username not found.");
                    else
                        WriteLine($"Username:\t\t{user.Name}");
                    WriteLine("Additional messages:");
                    foreach (var m in messages)
                        WriteLine(" - " + m);
                }

Here's the (messy) code for the "Test" button next to "Test get user" on that page:

        var btnTestGetUser = new PostBackButtonLink("Test", () =>
        {
            var log = new StringBuilder();
            try
            {
                instance = instance ?? (UserDirectory)Activator.CreateInstance(this.Type);
                editor.WriteToInstance(instance);

                instance.MessageLogged += (s, e) => log.AppendLine($"[{e.Level}] {e.Message}");

                var principal = instance.TryGetUser(txtTestUser.Value);
                if (principal == null)
                {
                    divSearchResults.Controls.Add(InfoBox.Warning(new P("User ", new Element("code", txtTestUser.Value), " not found.")));
                    return;
                }
                else
                {
                    divSearchResults.Controls.Add(InfoBox.Success(
                        new P("User ", new Element("code", txtTestUser.Value), " found: "),
                        new Ul(
                            new Li("Name: ", principal.Name ?? ""),
                            new Li("EmailAddress: ", principal.EmailAddress ?? ""),
                            new Li("DisplayName: ", principal.DisplayName ?? "")
                        )
                    ));

                    if (!string.IsNullOrEmpty(txtTestUserGroup.Value))
                    {
                        if (principal.IsMemberOfGroup(txtTestUserGroup.Value))
                            divSearchResults.Controls.Add(InfoBox.Success(new P("Member of ", new Element("code", txtTestUserGroup.Value))));
                        else
                            divSearchResults.Controls.Add(InfoBox.Warning(new P("Is not member of ", new Element("code", txtTestUserGroup.Value))));
                    }
                }
            }
            catch (Exception ex)
            {
                divSearchResults.Controls.Add(InfoBox.Error(new P($"Error: {ex.Message}")));
            }
            if (log.Length > 0)
                divSearchResults.Controls.Add(new Element("textarea", log.ToString()) { Style = "width:500px; height:50px;" });
            divSearchResults.Visible = true;
        });

Lots of code, but I wanted to share both of these, so we're looking at exactly the same thing, if you need it.

** Can you try testing "get user" again (not "search user") using that page? You will most certainly see the exact same set of error messages. **

If this is the case, then the problem is most definitely related to credentials/permissions, and really doesn't seem to be related to NETBIOS alias, after all.

Next steps.

1. Confirm that you're getting same error from the "Get user" test but that "Search user" works
2. Remove the NETBIOS alias mapping, make sure results are identical (get doesn't work, search does)
3. Enter wrong credentials, like a bad username or bad password; ensure that "Search user" fails
4. Correct the credentials, and makes sure that "search user" works again
5. Try your own, personal credentials to see if it makes a difference
6. Disable LDAPS (if it was enabled) and try again
7. Create a ProGet test server, see if you can replicate behavior
8. Open a ticket with Microsoft with replication results (the tool works, the servers don't, etc)

I hate that last step... but there's no reason on earth why this same, basic query that's run by the same C# code using the same credentials would work in one environment (desktop app on one server) but not another (web app)

[Stephen.Schaff]

Wahoooo!

We were able to successfully swap user directories this morning!

They key was changing the Managed Pipeline Mode setting for ProGet's App Pool in IIS. While set to Integrated (what we had been using) it would not work. Once I set it to Classic, I was able to finagle it to work. I had to fiddle with the Integrated Login setting in ProGet and the Anonymous Authentication setting in IIS, but it worked out!

As a side note, the debug/integrated-auth page still says Username not found and Principal not found, (it is the same as what I pasted in above when it was not working). Just thought I would let you know in case you are interested.

I am very excited to get moving forward on my project and stick with ProGet! Thank you for your hard work helping me get this working!

Stephen

[apxltd]

@Stephen-Schaff

Very glad to hear! It makes absolutely no sense makes that Managed Pipeline Mode would have any impact on how these libraries behave... but there you have it. These libraries are ancient, and even Microsoft has no idea how they work anymore.

Apparently Microsoft is working on shipping brand new libraries (no COM interop magic) in .NET6, which means by .NET8 they just might be usable, and then perhaps in another decade they won't require mysterious changes