API Key "impersonate user" doesn't work when impersonating an LDAP user

[rhessinger]

Hi Simon,

Unfortunately hitting the URL in the browser will not show the underlying error because the Docker client is a relatively chatty client. It will hit a number of different API endpoints to perform the login. You would need to use a tool like Wireshark or Fiddler to proxy requests to the server to see which URL has the problem.

Are you using nginx as a reverse-proxy to ProGet? Is it possible to bypass it to try to use docker login directly on the ProGet server to rule out any header manipulation issues?

Thanks,
Rich

[scroak_6473]

Hi @rhessinger

I've been able to capture some wireshark traces of this happening. Is there anyway you can email me? So that I can send you the traces? I'd rather not post them on this forum?

Thanks

Simon

[atripp]

Hi Simon, you can send to support at inedo dot com. Please include [QA-473] in the subject, so we can find it easily :)

[scroak_6473]

Hi @atripp

I have sent the email, thanks.

Simon

[atripp]

Thanks @scroak_6473; I found the email, and can see a lot of information from what you sent.

I can clearly see the identical api challenge/response, and the different behaviors from ProGet.

Unfortunately, I'm not able to reproduce the scenario on this end, using our own instance and a domain impersonated account. But I think that's because this issue may have already been fixed with PG-1859; would you be able to upgrade to 5.3.22 to confirm?

[scroak_6473]

Hi @atripp,

Unfortunately the error still persists in v5.3.22.

Simon

[atripp]

@scroak_6473 is it the exact same message? Basically, the "api" user not found in the directory?

[scroak_6473]

Hi @atripp

I actually get a regular 500 error now when I attempt to access using a browser (no fancy Proget 500 error)

I've been able to get some additional logs from our container. I've emailed them to you

Thanks

Simon

[atripp]

Hi Simon,

That is strange, it's basically your browser "hiding" the underlying error. Sometimes that happens if the response body is too short.... which could happen if the server got in some really bizare state.

I could find the logs you sent, but they were very random and they also don't make sense; it's random ASP.NET errors, but we can't see the full situation. In general the 500 errors should be logged in ProGet > Admin; this will provide a stack trace as to what errors are happening.

If you can't get to the admin page, then something is really wrong with the server. I would try restrating your container.

Alana

[scroak_6473]

Hi @atripp,

There are no logs under the admin page that relate to this error that I could see, additionally when I get this 500 error just for this page the rest of the proget server is fine no 500 errors.

Is there any way we can do a call or screenshare for you to look at this issue? It's preventing me from using ProGet for some additional integrations.

During the call I have access to splunk logging and DynaTrace diagnostic tools I can show you.

Thanks

Simon

[Stephen.Schaff]

I looked into this a bit with one of my impersonations that was working fine.

When I do a docker login with a username of "api" and the api key, it works just fine. When I looked at how mine was setup compared to yours, I noticed that you have your user listed "email style". (CMSProxy@YourDomain.com). Mine just has a username (no @domain.com).

Curious, I added my domain to mine to see what would happen and when I did I got the following error:

error parsing HTTP 403 response body: invalid character '<' looking for beginning of value

This seemed similar to what you got, though I also got a lot of HTML that you did not get.

Still, it might not hurt to try using the actual username (as defined in your LDAP) instead of the email address. It is very likely just CMSProxy. (If you have already tried this, then I apologize for adding noise to this conversation.)

[scroak_6473]

Hey @Stephen-Schaff

Thanks for the suggestion. I have tried removing the @yourdomain.com piece from the impersonate field and i'm still having the same issue.

Curious what version of proget are you running? are you running it in a container?

Thanks

Simon

[atripp]

@scroak_6473 we could definitely try a screen share, but in a case like this (where we have no idea what's wrong), it's mostly digging in the code and trying to think of things to try to get a clue for more information. Currently, I'm at a loss... because the error you have shouldn't be happening, but it clearly is.

So now, I had a new idea. I would like to eliminate Docker from equation, as it handles the
"api" username slightly differently than everywhere else. Plus you can do this all in your browser.

Can you try to visit a restricted (i.e. not anonymous view) NuGet endpoint using the "api" user name, and a password?

For example, it should look like /nuget/NuGetLibraries/v3/index.json, and then your browser should prompt for a Username/Password.

Depending on the result of this, we will explore different code paths, and then might need to add some more debugging codes.

Best,
Alana

[scroak_6473]

Hi @atripp,

A nuget feed is actually the endpoint i've been testing on since I upgraded to v5.3.22 and sent you the logs via email, i've been able to extract the same logs in an easier to read format, do this help?

• You can see my original GET request to "http://proget.xxxxxxxx.com/nuget/nuget-inntopia/v3/index.json" in the second log line below

• Then the exception "'Inedo.ProGet.Web.Security.UserNotFoundException'"

• And finally the 500 error being thrown in the last log message

proget.inedo.com/productimages/inedo/proget:5.3.22@sha256:3c2d2dcac563ec335ebdfb2bd4b9c0b08311b941eed415eb6cad0e49877ac39c/Proget_Server.1.pq13ondxps0li5kutof6qmug7/c53e518d3efb info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
proget.inedo.com/productimages/inedo/proget:5.3.22@sha256:3c2d2dcac563ec335ebdfb2bd4b9c0b08311b941eed415eb6cad0e49877ac39c/Proget_Server.1.pq13ondxps0li5kutof6qmug7/c53e518d3efb Request starting HTTP/1.1 GET http://proget.xxxxxxxx.com/nuget/nuget-inntopia/v3/index.json - -
proget.inedo.com/productimages/inedo/proget:5.3.22@sha256:3c2d2dcac563ec335ebdfb2bd4b9c0b08311b941eed415eb6cad0e49877ac39c/Proget_Server.1.pq13ondxps0li5kutof6qmug7/c53e518d3efb fail: Microsoft.AspNetCore.Server.Kestrel[13]
proget.inedo.com/productimages/inedo/proget:5.3.22@sha256:3c2d2dcac563ec335ebdfb2bd4b9c0b08311b941eed415eb6cad0e49877ac39c/Proget_Server.1.pq13ondxps0li5kutof6qmug7/c53e518d3efb Connection id "0HM6CBB8CS3D5", Request id "0HM6CBB8CS3D5:00000036": An unhandled exception was thrown by the application.
proget.inedo.com/productimages/inedo/proget:5.3.22@sha256:3c2d2dcac563ec335ebdfb2bd4b9c0b08311b941eed415eb6cad0e49877ac39c/Proget_Server.1.pq13ondxps0li5kutof6qmug7/c53e518d3efb Inedo.ProGet.Web.Security.UserNotFoundException: Exception of type 'Inedo.ProGet.Web.Security.UserNotFoundException' was thrown.
proget.inedo.com/productimages/inedo/proget:5.3.22@sha256:3c2d2dcac563ec335ebdfb2bd4b9c0b08311b941eed415eb6cad0e49877ac39c/Proget_Server.1.pq13ondxps0li5kutof6qmug7/c53e518d3efb at Inedo.ProGet.WebApplication.ProGetHttpModule.AuthorizeRequestAsync(HttpApplication app) in C:\InedoAgent\BuildMasterTemp\192.168.44.60\Temp\_E106466\Src\ProGet.WebApplication\ProGetHttpModule.cs:line 332
proget.inedo.com/productimages/inedo/proget:5.3.22@sha256:3c2d2dcac563ec335ebdfb2bd4b9c0b08311b941eed415eb6cad0e49877ac39c/Proget_Server.1.pq13ondxps0li5kutof6qmug7/c53e518d3efb at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context)
proget.inedo.com/productimages/inedo/proget:5.3.22@sha256:3c2d2dcac563ec335ebdfb2bd4b9c0b08311b941eed415eb6cad0e49877ac39c/Proget_Server.1.pq13ondxps0li5kutof6qmug7/c53e518d3efb at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context)
proget.inedo.com/productimages/inedo/proget:5.3.22@sha256:3c2d2dcac563ec335ebdfb2bd4b9c0b08311b941eed415eb6cad0e49877ac39c/Proget_Server.1.pq13ondxps0li5kutof6qmug7/c53e518d3efb at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context)
proget.inedo.com/productimages/inedo/proget:5.3.22@sha256:3c2d2dcac563ec335ebdfb2bd4b9c0b08311b941eed415eb6cad0e49877ac39c/Proget_Server.1.pq13ondxps0li5kutof6qmug7/c53e518d3efb at Dynatrace.OneAgent.Introspection.Shared.NewAspNetCoreTracingMiddlewareBase`1.Invoke(Context context)
proget.inedo.com/productimages/inedo/proget:5.3.22@sha256:3c2d2dcac563ec335ebdfb2bd4b9c0b08311b941eed415eb6cad0e49877ac39c/Proget_Server.1.pq13ondxps0li5kutof6qmug7/c53e518d3efb at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
proget.inedo.com/productimages/inedo/proget:5.3.22@sha256:3c2d2dcac563ec335ebdfb2bd4b9c0b08311b941eed415eb6cad0e49877ac39c/Proget_Server.1.pq13ondxps0li5kutof6qmug7/c53e518d3efb info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
proget.inedo.com/productimages/inedo/proget:5.3.22@sha256:3c2d2dcac563ec335ebdfb2bd4b9c0b08311b941eed415eb6cad0e49877ac39c/Proget_Server.1.pq13ondxps0li5kutof6qmug7/c53e518d3efb Request finished HTTP/1.1 GET http://proget.xxxxxxx.com/nuget/nuget-inntopia/v3/index.json - - - 500 0 - 30.1883ms

Also Dynatrace is also showing this, does that help?

Again I can show you Dynatrace in more detail if we can setup a screenshare.

[atripp]

@scroak_6473 this is very helpful actually, thank you!

There might be a relation to the LdapReferalException here, so we're going to do some more research and try to suggest what to try next. It might involve some new code (and potentially an upgrade).

Please stay tuned...

[atripp]

@scroak_6473 sorry but this one is just really tough and we didn't get the chance to dig in deeper, we're still trying to find time as a team to figure it out

[scroak_6473]

Thanks for the update @atripp

I await you're fix for this one. Let me know if I can be of anymore assistance.

Thanks

Simon

[atripp]

@scroak_6473 thanks again for all the stack traces and help

Good news, we finally were able to track this down. It seems to impact .NET5 only (i.e. ProGet for Linux), and involves using impersonation with a somewhat "faulty" LDAP setup (like yours, with those expected referral errors).

There's an easy fix, and it involves a small library change on our end. We will get it in the next maintence release of ProGet, but I don't have the issue numbers off hand yet.

[atripp]

@scroak_6473 FYI, this has been fixed in our library (ILIB-115) and we plan to upgrade to that library version via PG-1902 in the next maintence release -- so this should be fixed in 5.3.24, scheduled for the Friday the 5th

If you'd like a pre-release let us know, and we can let you know when it's merged in so you can use it sooner.

[scroak_6473]

Awesome thanks @atripp

I'll wait for the 5.3.24 release. I'll let you know how testing goes!

Thanks

Simon