API Key "impersonate user" doesn't work when impersonating an LDAP user

scroak_6473

Hey Guys,

I've just configured LDAP/AD authentication on ProGet (.net5) V5.3.17 and also removed anonymous user access.

Therefore I am trying to setup docker image feed access from our docker hosts using API keys.

I've setup the api key like this:

Using the "Impersonate user" username of an LDAP account that has administrator permissions set for it in the "Users & Tasks" form:

My plan is to reduce the amount of access this user has later.

But when I try to do a "docker login" using "api" as the username and the API key as the password I get this error:

If I remove the "impersonate user" field from the key and leave it blank I can login without issue.

Thanks

Simon

atripp

Hi Simon, I'm guessing there's a 500 error being thrown at the same time? Or, perhaps, it's a permission error? In any case, can you try to get a feeling for the underlying error message? That will help debug it.

scroak_6473

Hi @atripp

I was able to use "Boomerang" chrome plugin to communicate with the proget V2 endpoint directly and found the following:

URL: https://proget.xxxxx.com/v2/_auth
Basic HTTP auth username: api
Basic HTTP auth password: <api key here>

When impersonate user is set within proget like the screenshots above I get the following returned:

<!DOCTYPE html>
<html>
    <head>
        <title></title>
        <script type="text/javascript" src="/resources/InedoLib/AH/AH.js?950.1.0.3"></script>
        <link type="text/css" rel="stylesheet" href="/resources/InedoLib/styles/common.css?950.1.0.3" />
        <link type="text/css" rel="stylesheet" href="/resources/InedoLib/styles/normalize.css?950.1.0.3" />
        <link type="text/css" rel="stylesheet" href="/resources/InedoLib/styles/fonts.css?950.1.0.3" />
        <link type="text/css" rel="stylesheet" href="/resources/InedoLib/styles/icons.css?950.1.0.3" />
        <link type="text/css" rel="stylesheet" href="/resources/InedoLib/styles/controls.css?950.1.0.3" />
        <link type="text/css" rel="stylesheet" href="/resources/InedoLib/styles/nonmodal.css?950.1.0.3" />
        <link type="text/css" rel="stylesheet" href="/resources/styles/proget.css" />
        <script type="text/javascript" src="/resources/InedoLib/jquery-1.11.3.min.js?950.1.0.3"></script>
        <link type="text/css" rel="stylesheet" href="/resources/InedoLib/jquery-ui-1.11.4/jquery-ui.min.css?950.1.0.3" />
        <link type="text/css" rel="stylesheet" href="/resources/InedoLib/jquery-ui-1.11.4/jquery-ui.structure.min.css?950.1.0.3" />
        <script type="text/javascript" src="/resources/InedoLib/jquery-ui-1.11.4/jquery-ui.min.js?950.1.0.3"></script>
        <link type="text/css" rel="stylesheet" href="/resources/InedoLib/jquery-ui-1.11.4/jquery-ui.theme.min.css?950.1.0.3" />
        <link type="text/css" rel="stylesheet" href="/resources/InedoLib/jquery-ui-1.11.4/jquery-ui.hacks.css?950.1.0.3" />
        <script type="text/javascript" src="/resources/InedoLib/inedojq/inedojq_frameddialog.js?950.1.0.3"></script>
        <script type="text/javascript" src="/Resources/Scripts/urls.js"></script>
        <link type="text/css" rel="stylesheet" href="/resources/styles/v5.css" />
        <script type="text/javascript">$(function(){ AhValidation.InitializeForms(); });</script>
        <link rel="icon" type="image/x-icon" href="/Resources/Images/favicon.ico" />
        <script type="text/javascript">(function () {
    $.ajaxPrefilter(function (options) {
        if (!options.beforeSend) {
            options.beforeSend = function (xhr) {
                xhr.setRequestHeader('AHAntiCsrfToken', 'CfDJ8OKwK+rfXXJAmA1eylO2/kEZZj+tsETzFOJDnEEQg/dchD9TrmaI/hm7UD2VjfFoyUg6gxFMhRrMr+oxVgijmjx4jbnkpwC6KFMOaKPHN1WCi1i4ke1NpNWqWq+IizvnnACFp6gFbopXWz+qkZnwbnBkFlgi1dIyWUsJFM51Ve6M');
            }
        }
    });
})();</script>
        <script type="text/javascript">$(function(){ AhValidation.InitializeForms(); });</script>
    </head>
    <body>
        <div id="notification-bar-wrapper"></div>
        <script type="text/javascript">$(function(){var fn = function(){$('#notification-bar-wrapper').load('/0x44/ProGet.WebApplication/Inedo.ProGet.WebApplication.Controls.Layout.NotificationBar/GetNotifications #notifications', [], function(){ setTimeout(fn, 10000);});}; fn();});</script>
        <div class="content-container" id="navigation-bar">
            <div class="content">
                <a href="/">
                    <img class="logo" alt="" src="/resources/images/layout/logo.svg" />
                </a>
                <div class="user-controls">
                    <a class="dropdown settings" href="/administration"></a>
                    <a class="dropdown user" href="/log-in?ReturnUrl=%2Ferrors%2Fuser-not-found%3FdirectoryName%3DQueries%2520the%2520current%2520domain%252C%2520global%2520catalog%2520for%2520trusted%2520domains%252C%2520or%2520a%2520specific%2520li%26userName%3Dapi"></a>
                    <div id="user-navigation-container">
                        <ul id="user-navigation">
                            <li>
                                <a href="/change-password" onclick="$.inedojq_frameddialog(this.href,{resizable: true,refreshOnClose: false,width:645,height:480});return false;">Change Password</a>
                            </li>
                            <li>
                                <a target="_blank" href="http://inedo.com/support/documentation/proget?utm_source=proget&amp;utm_medium=product&amp;utm_campaign=proget5">Documentation</a>
                            </li>
                            <li>
                                <a class="logoff" href="/log-out">Log Off</a>
                            </li>
                        </ul>
                    </div>
                </div>
                <ul class="navigation">
                    <li>
                        <a href="/feeds">Feeds</a>
                    </li>
                    <li>
                        <a href="/packages">Packages</a>
                    </li>
                    <li>
                        <a href="/assets">Assets</a>
                    </li>
                    <li>
                        <a href="/containers">Containers</a>
                    </li>
                    <li>
                        <a href="/licenses/rules">Licenses</a>
                    </li>
                    <li>
                        <a href="/vulnerabilities">Vulnerabilities</a>
                    </li>
                </ul>
            </div>
        </div>
        <form method="post" enctype="application/x-www-form-urlencoded">
            <input type="hidden" name="AHAntiCsrfToken" value="CfDJ8OKwK+rfXXJAmA1eylO2/kGF1WuSIIUz6SQ9hvTJv4Ylv+pIINp26Yo1xC38Xp8nhLb3SGM9ljRE3t1Gsu4MGGMg1SzS+6qxIW+1AvpBMnL+sf2vPdo23WRQ8Sqw5gnKTjm1cgz5M1V27HPvXGdkHQJ+noL4LGLcfjan53rd9pde" />
            <noscript>
                <div class="content-container">
                    <div class="content">
                        <div class="info-box error">
                            <h2>Please Enable JavaScript!</h2>
                            <p>ProGet requires the use of JavaScript to function properly.  Please enable JavaScript in your browser for this site.</p>
                        </div>
                    </div>
                </div>
            </noscript>
            <div id="banner-wrapper">
                <div class="content-container banner">
                    <div class="content">
                        <div class="banner-section">
                            <h1>User Not Found</h1>
                        </div>
                    </div>
                </div>
            </div>
            <div class="content-container">
                <div class="content">
                    <div class="info-box error">
                        <p>There was an error attempting to load the user &#39;api&#39; from the user directory &#39;Queries the current domain, global catalog for trusted domains, or a specific li&#39;. This can be caused by an invalid user name or if cookies are not cleared after switching directory providers.</p>
                        <div class="action-button-container">
                            <a class="button solid red" name="ah0~ah3~ah0~ah1~ah1" onclick="AhValidation.TriggerPostBack(this, true, &#39;click&#39;);">Clear Authentication Cookies</a>
                        </div>
                    </div>
                    <p>If the user exists and clearing cookies does not resolve the problem, a ProGet system administrator can re-enable the built-in directory and Admin account by logging into the ProGet server (i.e. 8a15be05340a) and executing the following command:</p>
                    <code class="console">(proget-installation-directory)\Service&gt; .\ProGet.Service.exe resetadminpassword</code>
                    <p>Once the user directory is reset, the ProGet web application must be restarted which can be done by clicking here: 
                        <a href="/0x44/ProGet.WebApplication/Inedo.ProGet.WebApplication.Pages.Errors.UserNotFoundErrorPage/RestartWeb">Restart Web Application</a>
                    </p>
                    <p>It is also recommended that all users clear their cookies before attempting to visit ProGet again.</p>
                </div>
            </div>
        </form>
        <div class="content-container" id="footer">
            <div class="content">
                <p class="current-user">&nbsp;&bull; 
                    <span title="ID: api">Anonymous User</span>
                </p>
                <p class="version-number">Version 
                    <span id="ProGet Basic Edition_Version">5.3.17 (Build 19)</span>
                </p>
                <p>ProGet Basic Edition © 2020 Inedo, LLC</p>
            </div>
        </div>
    </body>
</html>

When I remove the "impersonate user" and leave blank I get the following:

{
    "token": "09F0C9F0E2B02BEADF5D7240980D5ECA53B6FE41B2F80BC9DD5C15C55FB002E1D726526020369C86B6DF863B8248439B32682488A51F85A76DFCFD7CF8C3340C1D5570B14195A10B34AD79D7899C2428B22A7D276E74F97197F2114D9A075DE9AF24939D79A3348F40935B995E193E07E038AB67143E94FE0DC936B49AD3417A90D5AF50DBE6831BA8F8FCD2330B1244FA71F551",
    "expires_in": 3600,
    "issued_at": "2020-12-08T18:27:33.6719276Z"
}

So not a 500 error unfortunately, but there does appear to be an error message in the HTML above.

Thanks

Simon

rhessinger

Hi Simon,

Unfortunately hitting the URL in the browser will not show the underlying error because the Docker client is a relatively chatty client. It will hit a number of different API endpoints to perform the login. You would need to use a tool like Wireshark or Fiddler to proxy requests to the server to see which URL has the problem.

Are you using nginx as a reverse-proxy to ProGet? Is it possible to bypass it to try to use docker login directly on the ProGet server to rule out any header manipulation issues?

Thanks,
Rich

scroak_6473

Hi @rhessinger

I've been able to capture some wireshark traces of this happening. Is there anyway you can email me? So that I can send you the traces? I'd rather not post them on this forum?

Thanks

Simon

atripp

Hi Simon, you can send to support at inedo dot com. Please include [QA-473] in the subject, so we can find it easily :)

scroak_6473

Hi @atripp

I have sent the email, thanks.

Simon

atripp

Thanks @scroak_6473; I found the email, and can see a lot of information from what you sent.

I can clearly see the identical api challenge/response, and the different behaviors from ProGet.

Unfortunately, I'm not able to reproduce the scenario on this end, using our own instance and a domain impersonated account. But I think that's because this issue may have already been fixed with PG-1859; would you be able to upgrade to 5.3.22 to confirm?

scroak_6473

Hi @atripp,

Unfortunately the error still persists in v5.3.22.

Simon

atripp

@scroak_6473 is it the exact same message? Basically, the "api" user not found in the directory?

scroak_6473

Hi @atripp

I actually get a regular 500 error now when I attempt to access using a browser (no fancy Proget 500 error)

I've been able to get some additional logs from our container. I've emailed them to you

Thanks

Simon

atripp

Hi Simon,

That is strange, it's basically your browser "hiding" the underlying error. Sometimes that happens if the response body is too short.... which could happen if the server got in some really bizare state.

I could find the logs you sent, but they were very random and they also don't make sense; it's random ASP.NET errors, but we can't see the full situation. In general the 500 errors should be logged in ProGet > Admin; this will provide a stack trace as to what errors are happening.

If you can't get to the admin page, then something is really wrong with the server. I would try restrating your container.

Alana

scroak_6473

Hi @atripp,

There are no logs under the admin page that relate to this error that I could see, additionally when I get this 500 error just for this page the rest of the proget server is fine no 500 errors.

Is there any way we can do a call or screenshare for you to look at this issue? It's preventing me from using ProGet for some additional integrations.

During the call I have access to splunk logging and DynaTrace diagnostic tools I can show you.

Thanks

Simon

Stephen.Schaff

I looked into this a bit with one of my impersonations that was working fine.

When I do a docker login with a username of "api" and the api key, it works just fine. When I looked at how mine was setup compared to yours, I noticed that you have your user listed "email style". (CMSProxy@YourDomain.com). Mine just has a username (no @domain.com).

Curious, I added my domain to mine to see what would happen and when I did I got the following error:

error parsing HTTP 403 response body: invalid character '<' looking for beginning of value

This seemed similar to what you got, though I also got a lot of HTML that you did not get.

Still, it might not hurt to try using the actual username (as defined in your LDAP) instead of the email address. It is very likely just CMSProxy. (If you have already tried this, then I apologize for adding noise to this conversation.)

scroak_6473

Hey @Stephen-Schaff

Thanks for the suggestion. I have tried removing the @yourdomain.com piece from the impersonate field and i'm still having the same issue.

Curious what version of proget are you running? are you running it in a container?

Thanks

Simon

atripp

@scroak_6473 we could definitely try a screen share, but in a case like this (where we have no idea what's wrong), it's mostly digging in the code and trying to think of things to try to get a clue for more information. Currently, I'm at a loss... because the error you have shouldn't be happening, but it clearly is.

So now, I had a new idea. I would like to eliminate Docker from equation, as it handles the
"api" username slightly differently than everywhere else. Plus you can do this all in your browser.

Can you try to visit a restricted (i.e. not anonymous view) NuGet endpoint using the "api" user name, and a password?

For example, it should look like /nuget/NuGetLibraries/v3/index.json, and then your browser should prompt for a Username/Password.

Depending on the result of this, we will explore different code paths, and then might need to add some more debugging codes.

Best,
Alana

scroak_6473

Hi @atripp,

A nuget feed is actually the endpoint i've been testing on since I upgraded to v5.3.22 and sent you the logs via email, i've been able to extract the same logs in an easier to read format, do this help?

You can see my original GET request to "http://proget.xxxxxxxx.com/nuget/nuget-inntopia/v3/index.json" in the second log line below
Then the exception "'Inedo.ProGet.Web.Security.UserNotFoundException'"
And finally the 500 error being thrown in the last log message

proget.inedo.com/productimages/inedo/proget:5.3.22@sha256:3c2d2dcac563ec335ebdfb2bd4b9c0b08311b941eed415eb6cad0e49877ac39c/Proget_Server.1.pq13ondxps0li5kutof6qmug7/c53e518d3efb [40m[32minfo[39m[22m[49m: Microsoft.AspNetCore.Hosting.Diagnostics[1]
proget.inedo.com/productimages/inedo/proget:5.3.22@sha256:3c2d2dcac563ec335ebdfb2bd4b9c0b08311b941eed415eb6cad0e49877ac39c/Proget_Server.1.pq13ondxps0li5kutof6qmug7/c53e518d3efb Request starting HTTP/1.1 GET http://proget.xxxxxxxx.com/nuget/nuget-inntopia/v3/index.json - -
proget.inedo.com/productimages/inedo/proget:5.3.22@sha256:3c2d2dcac563ec335ebdfb2bd4b9c0b08311b941eed415eb6cad0e49877ac39c/Proget_Server.1.pq13ondxps0li5kutof6qmug7/c53e518d3efb [41m[30mfail[39m[22m[49m: Microsoft.AspNetCore.Server.Kestrel[13]
proget.inedo.com/productimages/inedo/proget:5.3.22@sha256:3c2d2dcac563ec335ebdfb2bd4b9c0b08311b941eed415eb6cad0e49877ac39c/Proget_Server.1.pq13ondxps0li5kutof6qmug7/c53e518d3efb Connection id "0HM6CBB8CS3D5", Request id "0HM6CBB8CS3D5:00000036": An unhandled exception was thrown by the application.
proget.inedo.com/productimages/inedo/proget:5.3.22@sha256:3c2d2dcac563ec335ebdfb2bd4b9c0b08311b941eed415eb6cad0e49877ac39c/Proget_Server.1.pq13ondxps0li5kutof6qmug7/c53e518d3efb Inedo.ProGet.Web.Security.UserNotFoundException: Exception of type 'Inedo.ProGet.Web.Security.UserNotFoundException' was thrown.
proget.inedo.com/productimages/inedo/proget:5.3.22@sha256:3c2d2dcac563ec335ebdfb2bd4b9c0b08311b941eed415eb6cad0e49877ac39c/Proget_Server.1.pq13ondxps0li5kutof6qmug7/c53e518d3efb at Inedo.ProGet.WebApplication.ProGetHttpModule.AuthorizeRequestAsync(HttpApplication app) in C:\InedoAgent\BuildMasterTemp\192.168.44.60\Temp\_E106466\Src\ProGet.WebApplication\ProGetHttpModule.cs:line 332
proget.inedo.com/productimages/inedo/proget:5.3.22@sha256:3c2d2dcac563ec335ebdfb2bd4b9c0b08311b941eed415eb6cad0e49877ac39c/Proget_Server.1.pq13ondxps0li5kutof6qmug7/c53e518d3efb at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context)
proget.inedo.com/productimages/inedo/proget:5.3.22@sha256:3c2d2dcac563ec335ebdfb2bd4b9c0b08311b941eed415eb6cad0e49877ac39c/Proget_Server.1.pq13ondxps0li5kutof6qmug7/c53e518d3efb at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context)
proget.inedo.com/productimages/inedo/proget:5.3.22@sha256:3c2d2dcac563ec335ebdfb2bd4b9c0b08311b941eed415eb6cad0e49877ac39c/Proget_Server.1.pq13ondxps0li5kutof6qmug7/c53e518d3efb at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context)
proget.inedo.com/productimages/inedo/proget:5.3.22@sha256:3c2d2dcac563ec335ebdfb2bd4b9c0b08311b941eed415eb6cad0e49877ac39c/Proget_Server.1.pq13ondxps0li5kutof6qmug7/c53e518d3efb at Dynatrace.OneAgent.Introspection.Shared.NewAspNetCoreTracingMiddlewareBase`1.Invoke(Context context)
proget.inedo.com/productimages/inedo/proget:5.3.22@sha256:3c2d2dcac563ec335ebdfb2bd4b9c0b08311b941eed415eb6cad0e49877ac39c/Proget_Server.1.pq13ondxps0li5kutof6qmug7/c53e518d3efb at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
proget.inedo.com/productimages/inedo/proget:5.3.22@sha256:3c2d2dcac563ec335ebdfb2bd4b9c0b08311b941eed415eb6cad0e49877ac39c/Proget_Server.1.pq13ondxps0li5kutof6qmug7/c53e518d3efb [40m[32minfo[39m[22m[49m: Microsoft.AspNetCore.Hosting.Diagnostics[2]
proget.inedo.com/productimages/inedo/proget:5.3.22@sha256:3c2d2dcac563ec335ebdfb2bd4b9c0b08311b941eed415eb6cad0e49877ac39c/Proget_Server.1.pq13ondxps0li5kutof6qmug7/c53e518d3efb Request finished HTTP/1.1 GET http://proget.xxxxxxx.com/nuget/nuget-inntopia/v3/index.json - - - 500 0 - 30.1883ms

Also Dynatrace is also showing this, does that help?

Again I can show you Dynatrace in more detail if we can setup a screenshare.

atripp

@scroak_6473 this is very helpful actually, thank you!

There might be a relation to the LdapReferalException here, so we're going to do some more research and try to suggest what to try next. It might involve some new code (and potentially an upgrade).

Please stay tuned...

atripp

@scroak_6473 sorry but this one is just really tough and we didn't get the chance to dig in deeper, we're still trying to find time as a team to figure it out

scroak_6473

Thanks for the update @atripp

I await you're fix for this one. Let me know if I can be of anymore assistance.

Thanks

Simon

atripp

@scroak_6473 thanks again for all the stack traces and help

Good news, we finally were able to track this down. It seems to impact .NET5 only (i.e. ProGet for Linux), and involves using impersonation with a somewhat "faulty" LDAP setup (like yours, with those expected referral errors).

There's an easy fix, and it involves a small library change on our end. We will get it in the next maintence release of ProGet, but I don't have the issue numbers off hand yet.

atripp

@scroak_6473 FYI, this has been fixed in our library (ILIB-115) and we plan to upgrade to that library version via PG-1902 in the next maintence release -- so this should be fixed in 5.3.24, scheduled for the Friday the 5th

If you'd like a pre-release let us know, and we can let you know when it's merged in so you can use it sooner.

scroak_6473

Awesome thanks @atripp

I'll wait for the 5.3.24 release. I'll let you know how testing goes!

Thanks

Simon

scroak_6473

Hi @atripp

Just to let you know that I just upgraded version 5.3.24 and can confirm that this issue is now resolved!

Thanks so much for all your help!

Thanks

Simon