NPM Connector to Azure DevOps

[greig.maybery_5810]

Re: ProGet Connector to VSTS / Azure DevOps
Hi,
According to the case (PG-1409) this was fixed in 5.1.16
I am using a server on 5.2.24, and am having issues connecting to an Azure Devops npm feed with a connector.
I can successfully make a nuget connector from the same Proget server to the same ADO organisation, and can make an NPM connection to registry.npmjs.org.

I tried 3 username variants:

1. my email which is used to login to Microsoft, like in the Nuget connector (EMAIL)
2. Random name, like in the npmrc (RNAME)
3. Empty name as described in the referenced case (ENAME)

I have tried 2 password variants:

1. Personal Access Token generated in ADO, as used in the nuget connector (PAT)
2. Encrypted PAT, as used in the npmrc (EPAT)

Both email based runs (EMAIL-*) returned a 400 error:

Error processing JSON: System.Net.WebException: The remote server returned an error: (400) Bad Request.
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Inedo.ProGet.Feeds.Npm.NpmConnector.<RequestBearerAuthTokenAsync>d__16.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Inedo.LazyAsync`1.<GetValueAsync>d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Inedo.ProGet.Feeds.Npm.NpmConnector.<CreateWebRequestAsync>d__13.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Inedo.ProGet.Feeds.Npm.NpmConnector.<DownloadStreamAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Inedo.ProGet.ScheduledTasks.Npm.FullNpmConnectorIndexScheduledTask.<IndexConnectorAsync>d__1.MoveNext()

Both empty username (ENAME-*), and the one with a random name and PAT (RNAME-PAT) returned a 404 error:

Error processing JSON: System.Net.WebException: The remote server returned an error: (404) Not Found.
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Inedo.ProGet.Feeds.Npm.NpmConnector.<DownloadStreamAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Inedo.ProGet.ScheduledTasks.Npm.FullNpmConnectorIndexScheduledTask.<IndexConnectorAsync>d__1.MoveNext()

The random name with encrypted PAT (RNAME-EPAT) returned a 401:

Error processing JSON: System.Net.WebException: The remote server returned an error: (401) Unauthorized.
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Inedo.ProGet.Feeds.Npm.NpmConnector.<DownloadStreamAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Inedo.ProGet.ScheduledTasks.Npm.FullNpmConnectorIndexScheduledTask.<IndexConnectorAsync>d__1.MoveNext()

This looked like it was working, but there was an issue with the PAT. I issued another PAT with full API access from and Organisation Admin and tried again, with the same results.

I have run out of ideas, any assistance would be appreciated.

[atripp]

Hello;

I'm not really familiar with how Azure DevOps implements npm authentication. but it seems there's definitely a problem. The endpoint is not returning JSON, and it's giving a 400 error; my guess is that it's an empty response, so who knows what the error message is. I wonder, if the URL is wrong? Are there AzureDevOps logs you can inspect?

Otherwise it's hard to say where the problem is with Azure DevOps because the behavior changes all the time. In the past, I've seen 400 errors come and go, and sometimes it's their way of telling you that you've configured a token wrong. Or, they have a bug implementing NPM's api (this is common), so maybe try a different request? Both errors are happening while trying to index the connector. Try searching, etc.

In any case, the easiest thing to do would be to replicate the way ProGet does authentication. Here's how we handle it in ProGet; first, the logic to determine if a bearer token should be used...

    protected override async Task<HttpWebRequest> CreateWebRequestAsync(string url)
    {
        var request = await base.CreateWebRequestAsync(url).ConfigureAwait(false);

        if (this.Password != null && (string.IsNullOrEmpty(this.UserName) || this.UserName.Contains('@')))
        {
            var bearerToken = await this.BearerAuthToken.ValueAsync.ConfigureAwait(false);
            if (bearerToken != null)
            {
                request.Headers.Set(HttpRequestHeader.Authorization, "Bearer " + bearerToken);
            }
        }

        request.Accept = "application/json";

        return request;
    }

And the bearer-token acquisition logic....

    private async Task<string> RequestBearerAuthTokenAsync()
    {
        if (string.IsNullOrEmpty(this.UserName))
        {
            return AH.Unprotect(this.Password);
        }

        var request = await base.CreateWebRequestAsync(this.ResolveUrl("-/user/org.couchdb.user:" + Uri.EscapeDataString(this.UserName))).ConfigureAwait(false);
        request.Accept = "application/json";
        request.ContentType = "application/json";
        request.Method = "PUT";

        using (var requestStream = await request.GetRequestStreamAsync().ConfigureAwait(false))
        using (var writer = new StreamWriter(requestStream, InedoLib.UTF8Encoding))
        using (var jsonWriter = new JsonTextWriter(writer))
        {
            jsonWriter.WriteStartObject();
            jsonWriter.WritePropertyName("_id");
            jsonWriter.WriteValue("org.couchdb.user:" + this.UserName);
            jsonWriter.WritePropertyName("name");
            jsonWriter.WriteValue(this.UserName);
            jsonWriter.WritePropertyName("password");
            jsonWriter.WriteValue(AH.Unprotect(this.Password));
            jsonWriter.WritePropertyName("email");
            jsonWriter.WriteValue(this.UserName);
            jsonWriter.WritePropertyName("type");
            jsonWriter.WriteValue("user");
            jsonWriter.WritePropertyName("roles");
            jsonWriter.WriteStartArray();
            jsonWriter.WriteEndArray();
            jsonWriter.WritePropertyName("date");
            jsonWriter.WriteValue(DateTimeOffset.UtcNow);
            jsonWriter.WriteEndObject();
        }

        using (var response = await request.GetResponseAsync().ConfigureAwait(false))
        using (var responseStream = response.GetResponseStream())
        using (var reader = new StreamReader(responseStream, InedoLib.UTF8Encoding))
        using (var jsonReader = new JsonTextReader(reader))
        {
            var result = JObject.Load(jsonReader);
            return (string)result.Property("token")?.Value;
        }
    }

It doesn't look like the error is happening when requesting a token, just when doing the request. hope this is a good start at least!

[greig.maybery_5810]

@atripp

Hi,
Could you point me to your documentation detailing exactly how to connect to an Azure DevOps NPM feed using a connector and personal access tokens. This will save me from testing all of the permutations.

[atripp]

Unfortunately we don't have any documentation specifically for Azure DevOps NPM feeds; they change fairly often for us to to keep track. We did try/test it at one point, a while back, but our code for this feature hasn't substantially changed since then.

It's supposed to be as simple as an empty username (ENAME) and a token as your password (PAT). That's used to request a Bearer token from the NPM api, and send that back in a header.

I looked at their docs, and it says "username (can be anything except empty), PAT, and email". Not sure why they require username. Do they look you up by email? Weird.

Anyways, that's strange. So I guess, I'd also try EMAIL + PAT. That should
also work.

[greig.maybery_5810]

@atripp
Hi,
Thanks for the verification, this was actually one of my assumed cases covered under the results EMAIL-*, it returns a 400 error, full error message is above.
I have implemented a manual work around that unfortunately adds to the cognitive load on our development teams. While a resolution is not urgent, it would be appreciated.

[atripp]

So, if I understand correctly, using Email for username and PAT for password causes a problem. I'm guessing, this is happening during a Bearer token request.

Would you be able to run ProGet through a proxy server, so you can see the requests that are being made, and reproduce those requests? Alternatively, could you use the code I provided to request a Bearer token?

It's possible that you have an intermediate proxy server that's generating that 400 error, or it's a bug on Microsoft's end. The Microsoft documentation could also be wrong. But let's try to see if we can work-around or get information to Microsoft to fix it.