Nuspec file with an empty group in dependencies breaks Proget

nicolas.morissette_6285

Hey there here's our use case:

We created a connector to an AzureDevops NuGet feed from our ProGet feed
3 packages on 4 can be fetched without any issue from our Proget feed
1 package fails when we try to install it from our Proget feed, but works fine directly from the Azure DevOps feed

An error occurred while retrieving package metadata for '**PackageName_Version**' from source '**our Proget Feed url**'.
  id

The difference with this package is that the .nuspec file contains a group which specifies targetFramework=".NETStandard2.0" under the dependencies section but this group is empty (no dependency specified under the group)
NuspecIssue.png
We figured out that maybe ProGet assumes there will be at least 1 dependency under the group and looks for the id of the 1st one but finds none and crashes giving us this message
Verbose logs from nuget.exe tell us the problematic endpoint is:
https://feed/nuget/nuget/v3/registrations-gz/PackageName/index.json
We noticed that all the versions from the problematic package have this form:

catalogEntry: {
  authors: "***",
  dependencyGroups: [
    {
      targetFramework: "netstandard2.0",
      dependencies: [
        {
          id: "",
          range: ""
        }
      ]
    }
  ],
  description: "Package Description",
  id: "**PackageName**",
  published: "**Date**",
  version: "**Date**"
},

The problematic part appear to be the empty id & range in the dependencies array, which should not the be there.

We looked at the NuSpec specifications and it seems normal to have a dependency group event if there`s none.

Let us know if it's really a ProGet bug or any more information regarding this error. Thanks a lot!!

rhessinger

Hi @nicolas-morissette_6285,

This is most certainly interesting to me. This does not seem to be an issue with ProGet from what I can see. I created a simple example based on Newtonsoft.json. I created nuspec file that looks like this:

<?xml version="1.0" encoding="utf-8"?>
<package xmlns="http://schemas.microsoft.com/packaging/2013/05/nuspec.xsd">
  <metadata minClientVersion="2.12">
    <id>Newtonsoft.Json</id>
    <version>13.0.0</version>
    <title>Json.NET</title>
    <authors>James Newton-King</authors>
    <owners>James Newton-King</owners>
    <requireLicenseAcceptance>false</requireLicenseAcceptance>
    <license type="expression">MIT</license>
    <licenseUrl>https://licenses.nuget.org/MIT</licenseUrl>
    <icon>packageIcon.png</icon>
    <projectUrl>https://www.newtonsoft.com/json</projectUrl>
    <description>Json.NET is a popular high-performance JSON framework for .NET</description>
    <copyright>Copyright © James Newton-King 2008</copyright>
    <tags>json</tags>
    <repository type="git" url="https://github.com/JamesNK/Newtonsoft.Json" commit="7c3d7f8da7e35dde8fa74188b0decff70f8f10e3" />
    <dependencies>
      <group targetFramework=".NETStandard2.0" />
    </dependencies>
  </metadata>
</package>

I navigated to the URL you specified aboved and got this:

{
  "count": 1,
  "items": [
    {
      "count": 1,
      "items": [
        {
          "@id": "http://proget.localhost/nuget/NoConnPublic/v3/registrations-gz/newtonsoft.json/13.0.0.json",
          "@type": "Package",
          "catalogEntry": {
            "@id": "http://proget.localhost/nuget/NoConnPublic/v3/catalog/newtonsoft.json/13.0.0.json",
            "@type": "PackageDetails",
            "authors": "James Newton-King",
            "dependencyGroups": [
              {
                "targetFramework": ".NETStandard2.0",
                "dependencies": []
              }
            ],
            "description": "Json.NET is a popular high-performance JSON framework for .NET",
            "id": "Newtonsoft.Json",
            "licenseUrl": "https://licenses.nuget.org/MIT",
            "licenseExpression": "MIT",
            "minClientVersion": "2.12",
            "projectUrl": "https://www.newtonsoft.com/json",
            "published": "2020-11-06T20:09:19.47Z",
            "tags": "json",
            "title": "Json.NET",
            "version": "13.0.0"
          },
          "packageContent": "http://proget.localhost/nuget/NoConnPublic/v3/flatcontainer/Newtonsoft.Json/13.0.0/Newtonsoft.Json.13.0.0.nupkg",
          "registration": "http://proget.localhost/nuget/NoConnPublic/v3/registrations-gz/newtonsoft.json/index.json"
        }
      ],
      "parent": "http://proget.localhost/nuget/NoConnPublic/v3/registrations-gz/newtonsoft.json/index.json",
      "lower": "13.0.0",
      "upper": "13.0.0"
    }
  ]
}

My guess is this is most likely an issue with the AzureDevOps registry returning the dependencyGroups incorrectly through the connector. The fact that id and range are specified but do not have a value is definitely what is breaking ProGet because if it is specified we expect it to exist. Are you able to get that registration JSON directly from Azure DevOps' registry and see if you see that empty id and range in there?

Thanks,
Rich

nicolas.morissette_6285

Hey Rich,
Here's an example directly from the feed (I used a viewer but you can see the values). It doesn't look like it has empty values

rhessinger

Hi @nicolas-morissette_6285,

Could you try uploading you .nupkg directly into ProGet and see if you get any error?

Thanks,
Rich

nicolas.morissette_6285

Hey Rich,
It works when I download it manually from the Azure DevOps feed and upload it manually in our ProGet NuGet feed. Then I can install the package without any issue.

rhessinger

Hi @nicolas-morissette_6285,

I have dug through our connector code and this looks to be an issue with the Azure DevOps' implementation of the NuGet v3 API, specificially their RegistrationBaseUrl. If you notice in my example above, the dependencies property is an empty array. This is what we are expecting. Since Azure DevOps is actually populatating it with an object, we attempt to parse the dependency, and then the id property is empty which is causing our code to fail. I would submit this as an issue to Azure DevOps. I don't think this is an issue with your NuGet package considering it has the nuspec properly defined and works fine when uploading to ProGet.

Thanks,
Rich

nicolas.morissette_6285

Hey Rich,
Thanks a lot for your support.
I contacted Microsoft as suggested but they sent me back to your team as predicted.
Here is their answer :

rhessinger

Hi @nicolas-morissette_6285,

I apologize for the back and forth with them. I just created a ticket, PG-1851, to add a special case for handling this kind of package from Azure DevOps Registry. It is set to release in ProGet 5.3.17 which is due out November 20, 2020.

Thanks,
Rich

nicolas.morissette_6285

Hey Rich,
No need to apologize I appreciate your quick feedback and action.
We will make sure to update when this version comes out!

Thanks again,

Nicolas Morissette

rhessinger

Hi @nicolas-morissette_6285,

No problem. I'll let you know if anything is delayed in that release!

Thanks,
Rich

nicolas.morissette_6285

Hey Rich,
My bad I posted it on the wrong feed.
We still have the issue after updating to 5.3.18 this morning:

Do you know if that should have been fixed? According to the change log it should been fixed:
PG-1851 - FIX: NuGet v3 dependencies with empty ID will throw null exception

rhessinger

Hi @nicolas-morissette_6285,

No problem. Thanks for moving the comment over here! I will delete it from the other topic.

I can confirm the fix I made was released and is currently in the code. Do you see any errors in the Diagnostic Center in ProGet?

Thanks,
Rich

nicolas.morissette_6285

Hey Rich,
We have some errors (404 not found..) but nothing related to that. I tried to pull the NuGet wtih the nuget install and nothing showed up in the Diagnostic Center log messages.

I didn't kill and recreate the connector to teh DevOps feed. Do I need to do that? I only deleted the NuGet and all its versions from our feed to force the cache.

rhessinger

Hi @nicolas-morissette_6285,

You should not need to do anything to the connector. Can you try and open the package in the ProGet UI? Does that package/version load? Also, can you download it within the ProGet UI?

Thanks,
Rich

nicolas.morissette_6285

Hey Rich,
I can see the package and the versions available in the ProGet UI:

The ones that have never been installed manually (with the antenna icon) from the UI using the download button in the UI will not work trying to download it with the NuGet install command.

The one that already been downloaded (so cached) will work just fine with the NuGet install command.

Since the feed is used mainly by TFS for the build pipeline we can't rely on people doing download the new NuGet manually every time.

Is there something I did wrong setuping the connector?
Thanks again for the incredible support!

rhessinger

Hi @nicolas-morissette_6285,

Thanks for the updated information. This is most likely not a configuration issue with the connector. The only thing that could be affected by the connector setup is if you are using a v2 API NuGet feed instead of a v3 on from Azure DevOps, but I believe you already confirmed above that it was the v3 API from Azure DevOps.

Let me look into this a bit further. It looks that I must have changed the code that the UI uses, but not what the nuget.exe uses. I'm not sure how that is possible, but let me take another look.

Thanks,
Rich

nicolas.morissette_6285

Hey Rich,
Just to clarify something. My connector to Azure DevOps uses the V2 version as the V3 would simply not work.
This is the current configuration:

I created another one with the same token using the V3 Url. Here's the result in the Administration overview when I do a connector health check:

rhessinger

Hi @nicolas-morissette_6285,

Would you be able to post the JSON from your v3 API? It looks like AzureDevOps is missing the URL for the SearchQueryService. If you navigate to https://proget.inedo.com/nuget/NuGetLibraries/v3/index.json You'll see the very first service API URL is for SearchQueryService. I'm wondering if Azure DevOps is implementing a weird version of it, if at all. If you don't feel comfortable posting it here, you can email it to support@inedo.com, let me know, and I can pull it from there.

I'm sorry for the confusion. Since you originally posted the values from the registrations-gz, I made the assumption you were using the v3 API. The fix I added was for the V3 API JSON. I'll have to see what I can do with the V2 version because it parsed a little differently than the v3 version.

Thanks,
Rich

nicolas.morissette_6285

Hey Rich,
No problem it's my fault for omitting this detail. I assumed that ProGet was just not compatible with V3 because of the error message we were getting.

I sent the Json for this particular DevOps via email (subject : Nuspec file with an empty group in dependencies breaks Proget)

Thanks again for the fast feedback

rhessinger

Hi @nicolas-morissette_6285,

Thanks for sending that over. This confirmed my thoughts. It looks like they only include the SearchQueryService/3.0.0-beta, not the SearchQueryService, which is why the v3 connector does not work. I'm going to talk this over with the internal team and see how we want to approach it. ProGet currently does not implement the 3.0.0-beta API endpoint because it is still likely to change, which increases the likelihood of it breaking in ProGet.

I'll discuss this internally and figure out the best way to proceed.

Thanks for sending this over!

Thanks,
Rich