Nuspec file with an empty group in dependencies breaks Proget

nicolas.morissette_6285

Hey there here's our use case:

We created a connector to an AzureDevops NuGet feed from our ProGet feed
3 packages on 4 can be fetched without any issue from our Proget feed
1 package fails when we try to install it from our Proget feed, but works fine directly from the Azure DevOps feed

An error occurred while retrieving package metadata for '**PackageName_Version**' from source '**our Proget Feed url**'.
  id

The difference with this package is that the .nuspec file contains a group which specifies targetFramework=".NETStandard2.0" under the dependencies section but this group is empty (no dependency specified under the group)
NuspecIssue.png
We figured out that maybe ProGet assumes there will be at least 1 dependency under the group and looks for the id of the 1st one but finds none and crashes giving us this message
Verbose logs from nuget.exe tell us the problematic endpoint is:
https://feed/nuget/nuget/v3/registrations-gz/PackageName/index.json
We noticed that all the versions from the problematic package have this form:

catalogEntry: {
  authors: "***",
  dependencyGroups: [
    {
      targetFramework: "netstandard2.0",
      dependencies: [
        {
          id: "",
          range: ""
        }
      ]
    }
  ],
  description: "Package Description",
  id: "**PackageName**",
  published: "**Date**",
  version: "**Date**"
},

The problematic part appear to be the empty id & range in the dependencies array, which should not the be there.

We looked at the NuSpec specifications and it seems normal to have a dependency group event if there`s none.

Let us know if it's really a ProGet bug or any more information regarding this error. Thanks a lot!!

rhessinger

Hi @nicolas-morissette_6285,

This is most certainly interesting to me. This does not seem to be an issue with ProGet from what I can see. I created a simple example based on Newtonsoft.json. I created nuspec file that looks like this:

<?xml version="1.0" encoding="utf-8"?>
<package xmlns="http://schemas.microsoft.com/packaging/2013/05/nuspec.xsd">
  <metadata minClientVersion="2.12">
    <id>Newtonsoft.Json</id>
    <version>13.0.0</version>
    <title>Json.NET</title>
    <authors>James Newton-King</authors>
    <owners>James Newton-King</owners>
    <requireLicenseAcceptance>false</requireLicenseAcceptance>
    <license type="expression">MIT</license>
    <licenseUrl>https://licenses.nuget.org/MIT</licenseUrl>
    <icon>packageIcon.png</icon>
    <projectUrl>https://www.newtonsoft.com/json</projectUrl>
    <description>Json.NET is a popular high-performance JSON framework for .NET</description>
    <copyright>Copyright © James Newton-King 2008</copyright>
    <tags>json</tags>
    <repository type="git" url="https://github.com/JamesNK/Newtonsoft.Json" commit="7c3d7f8da7e35dde8fa74188b0decff70f8f10e3" />
    <dependencies>
      <group targetFramework=".NETStandard2.0" />
    </dependencies>
  </metadata>
</package>

I navigated to the URL you specified aboved and got this:

{
  "count": 1,
  "items": [
    {
      "count": 1,
      "items": [
        {
          "@id": "http://proget.localhost/nuget/NoConnPublic/v3/registrations-gz/newtonsoft.json/13.0.0.json",
          "@type": "Package",
          "catalogEntry": {
            "@id": "http://proget.localhost/nuget/NoConnPublic/v3/catalog/newtonsoft.json/13.0.0.json",
            "@type": "PackageDetails",
            "authors": "James Newton-King",
            "dependencyGroups": [
              {
                "targetFramework": ".NETStandard2.0",
                "dependencies": []
              }
            ],
            "description": "Json.NET is a popular high-performance JSON framework for .NET",
            "id": "Newtonsoft.Json",
            "licenseUrl": "https://licenses.nuget.org/MIT",
            "licenseExpression": "MIT",
            "minClientVersion": "2.12",
            "projectUrl": "https://www.newtonsoft.com/json",
            "published": "2020-11-06T20:09:19.47Z",
            "tags": "json",
            "title": "Json.NET",
            "version": "13.0.0"
          },
          "packageContent": "http://proget.localhost/nuget/NoConnPublic/v3/flatcontainer/Newtonsoft.Json/13.0.0/Newtonsoft.Json.13.0.0.nupkg",
          "registration": "http://proget.localhost/nuget/NoConnPublic/v3/registrations-gz/newtonsoft.json/index.json"
        }
      ],
      "parent": "http://proget.localhost/nuget/NoConnPublic/v3/registrations-gz/newtonsoft.json/index.json",
      "lower": "13.0.0",
      "upper": "13.0.0"
    }
  ]
}

My guess is this is most likely an issue with the AzureDevOps registry returning the dependencyGroups incorrectly through the connector. The fact that id and range are specified but do not have a value is definitely what is breaking ProGet because if it is specified we expect it to exist. Are you able to get that registration JSON directly from Azure DevOps' registry and see if you see that empty id and range in there?

Thanks,
Rich

nicolas.morissette_6285

Hey Rich,
Here's an example directly from the feed (I used a viewer but you can see the values). It doesn't look like it has empty values

rhessinger

Hi @nicolas-morissette_6285,

Could you try uploading you .nupkg directly into ProGet and see if you get any error?

Thanks,
Rich

nicolas.morissette_6285

Hey Rich,
It works when I download it manually from the Azure DevOps feed and upload it manually in our ProGet NuGet feed. Then I can install the package without any issue.

rhessinger

Hi @nicolas-morissette_6285,

I have dug through our connector code and this looks to be an issue with the Azure DevOps' implementation of the NuGet v3 API, specificially their RegistrationBaseUrl. If you notice in my example above, the dependencies property is an empty array. This is what we are expecting. Since Azure DevOps is actually populatating it with an object, we attempt to parse the dependency, and then the id property is empty which is causing our code to fail. I would submit this as an issue to Azure DevOps. I don't think this is an issue with your NuGet package considering it has the nuspec properly defined and works fine when uploading to ProGet.

Thanks,
Rich

nicolas.morissette_6285

Hey Rich,
Thanks a lot for your support.
I contacted Microsoft as suggested but they sent me back to your team as predicted.
Here is their answer :

rhessinger

Hi @nicolas-morissette_6285,

I apologize for the back and forth with them. I just created a ticket, PG-1851, to add a special case for handling this kind of package from Azure DevOps Registry. It is set to release in ProGet 5.3.17 which is due out November 20, 2020.

Thanks,
Rich

nicolas.morissette_6285

Hey Rich,
No need to apologize I appreciate your quick feedback and action.
We will make sure to update when this version comes out!

Thanks again,

Nicolas Morissette

rhessinger

Hi @nicolas-morissette_6285,

No problem. I'll let you know if anything is delayed in that release!

Thanks,
Rich

nicolas.morissette_6285

Hey Rich,
My bad I posted it on the wrong feed.
We still have the issue after updating to 5.3.18 this morning:

Do you know if that should have been fixed? According to the change log it should been fixed:
PG-1851 - FIX: NuGet v3 dependencies with empty ID will throw null exception

rhessinger

Hi @nicolas-morissette_6285,

No problem. Thanks for moving the comment over here! I will delete it from the other topic.

I can confirm the fix I made was released and is currently in the code. Do you see any errors in the Diagnostic Center in ProGet?

Thanks,
Rich

nicolas.morissette_6285

Hey Rich,
We have some errors (404 not found..) but nothing related to that. I tried to pull the NuGet wtih the nuget install and nothing showed up in the Diagnostic Center log messages.

I didn't kill and recreate the connector to teh DevOps feed. Do I need to do that? I only deleted the NuGet and all its versions from our feed to force the cache.

rhessinger

Hi @nicolas-morissette_6285,

You should not need to do anything to the connector. Can you try and open the package in the ProGet UI? Does that package/version load? Also, can you download it within the ProGet UI?

Thanks,
Rich

nicolas.morissette_6285

Hey Rich,
I can see the package and the versions available in the ProGet UI:

The ones that have never been installed manually (with the antenna icon) from the UI using the download button in the UI will not work trying to download it with the NuGet install command.

The one that already been downloaded (so cached) will work just fine with the NuGet install command.

Since the feed is used mainly by TFS for the build pipeline we can't rely on people doing download the new NuGet manually every time.

Is there something I did wrong setuping the connector?
Thanks again for the incredible support!

rhessinger

Hi @nicolas-morissette_6285,

Thanks for the updated information. This is most likely not a configuration issue with the connector. The only thing that could be affected by the connector setup is if you are using a v2 API NuGet feed instead of a v3 on from Azure DevOps, but I believe you already confirmed above that it was the v3 API from Azure DevOps.

Let me look into this a bit further. It looks that I must have changed the code that the UI uses, but not what the nuget.exe uses. I'm not sure how that is possible, but let me take another look.

Thanks,
Rich

nicolas.morissette_6285

Hey Rich,
Just to clarify something. My connector to Azure DevOps uses the V2 version as the V3 would simply not work.
This is the current configuration:

I created another one with the same token using the V3 Url. Here's the result in the Administration overview when I do a connector health check:

rhessinger

Hi @nicolas-morissette_6285,

Would you be able to post the JSON from your v3 API? It looks like AzureDevOps is missing the URL for the SearchQueryService. If you navigate to https://proget.inedo.com/nuget/NuGetLibraries/v3/index.json You'll see the very first service API URL is for SearchQueryService. I'm wondering if Azure DevOps is implementing a weird version of it, if at all. If you don't feel comfortable posting it here, you can email it to support@inedo.com, let me know, and I can pull it from there.

I'm sorry for the confusion. Since you originally posted the values from the registrations-gz, I made the assumption you were using the v3 API. The fix I added was for the V3 API JSON. I'll have to see what I can do with the V2 version because it parsed a little differently than the v3 version.

Thanks,
Rich

nicolas.morissette_6285

Hey Rich,
No problem it's my fault for omitting this detail. I assumed that ProGet was just not compatible with V3 because of the error message we were getting.

I sent the Json for this particular DevOps via email (subject : Nuspec file with an empty group in dependencies breaks Proget)

Thanks again for the fast feedback

rhessinger

Hi @nicolas-morissette_6285,

Thanks for sending that over. This confirmed my thoughts. It looks like they only include the SearchQueryService/3.0.0-beta, not the SearchQueryService, which is why the v3 connector does not work. I'm going to talk this over with the internal team and see how we want to approach it. ProGet currently does not implement the 3.0.0-beta API endpoint because it is still likely to change, which increases the likelihood of it breaking in ProGet.

I'll discuss this internally and figure out the best way to proceed.

Thanks for sending this over!

Thanks,
Rich

apxltd

Hi @nicolas-morissette_6285 ,

Can you help me understand the use case here of ProGet and Azure DevOps Packages? Basically I'm a bit hesitant to invest in continued work-arounds for bugs/quirks in ADO Packages.

The ADO Packages team has made it very clear they aren't interested in spec compliance (whether NuGet or otherwise), and that they won't support interoperability with any other tool aside from Azure DevOps. That makes sense because they want you to only and entirely use Azure DevOps.

We've already work around one quirk, and I have little confidence that fixing this quirk will actually solve the problem (i.e. enable the use case you're trying to build) and that it will stay fixed. In a few weeks/months, when they ship some new functionality, your hybrid-ADO Packages usecase may break again.

For example, when ADO-Packages decided to block "private upstream feeds" (I guess that's what it's called?), it disabled the use-case of having ADO-Packages pull from ProGet servers, or vendor's feeds, or anywhere other than NuGet.org or ADO-Packages feeds. Fortunately it's easy enough to work-around in ADO.

Bottom line, using ADO Packages this way may prove to be very fragile, and you may be better served by not using ADO Packages altogether. A lot of other users have made that decision, and ADO works fine (better IMO) without using their Packages feature... and it's highly likely it will continue to indefinitely.

Alex

nicolas.morissette_6285

This post is deleted!

nicolas.morissette_6285

Hey @apxltd,
I didn't expect it to reach all the way to you but it is a honor that you take care of the issues.

Here is our use case:
Most of the dev teams in our organization use DevOps on premise and push the NuGet to our Proget NuGet feed directly since we manage most build definitions.
Some of them need or want the new features that are available on the cloud version of DevOps so they move all their work or part of it online and my team doesn't manage what they do in their online collection. This is why some packages are hosted on their cloud DevOps feed. It's no problem until other teams need to use their packages and that's what happened here with this very specific case of an empty dependencies section.

To begin with I totally agree with you that the ADO Packages team are really not playing nice and fair by not being compliant with spec to make sure we only use their products. As you might saw with the previous comments I tried to speak to Microsoft about that too and they basically told me to deal with it as they won't change anything...

I'm now surprised to get the same kind of answer coming from your company. We are probably not the only customer using DevOps in the cloud and running into these issues but I might be wrong. Off course as you said I could do like some others did. I could try to contact all the devs and try to configure their DevOps pipelines so they push directly to our private Proget NuGet feed but since it's an independent online platform outside our infrastructure it's easier said than done as we are a 1000+ employees company.

I will see what would be possible to do on our side as well with your recommendations. I understand what you say and that it would request a constant effort from your team to keep this use case alive and working every time Microsoft updates their platform but if that's not a heavy load on your side I think it would be a good investment since Microsoft is a major player and they keep pushing people to go on the cloud version. On a more positive note we do enjoy Proget a lot and I am glad to get feedback from your team.

apxltd

Hey @nicolas-morissette_6285,

Thanks for clarifying!

So it sounds like this is a non-critical integration point. In other words, you're not building a workflow that relies on the two to be integrally connected, it's more a "convenience" such that one team get another team's developer library packages more easily.

I'm more nervous about enabling / encouraging the first use case; given the velocity and unpredictability of ADO's changes, there's a high change it will "just break" some day -- and as you've seen from their support, it's hard to get "them" to care. We most definitely care, which is why I'm cautious about putting us in a position to be responsible for customers' infrastructure breaking on account of a third-party API suddenly changing.

We most definitely work with a lot of teams at Microsoft, but their internal org hasn't really changed in the many years since I first saw this...

... so a lot of times, it's partnering with customers to push different groups at Microsoft to work together to serve what should be their common interest (i.e. a paying customer). For example, getting AzureDevOps Packages and NuGet team to collaborate on API things.

After the change in guard at VSO/ADO(i.e. after Ed Blankenship), we really don't have any meaningful contacts or pull w/ that group. But we definitely have contacts and work closely with other teams (like NuGet team, etc).

All that said, we'll also work-around this quirk, and I've put in a change (PG-1862) to accommodate this. But just

nicolas.morissette_6285

Hey @apxltd ,

Yeah having to deal with Microsoft on many different issues ourselves I totally get what you mean with this picture. I also totally understand your point and that makes sense! I'm glad the NuGet teams also push for the changes when they can.

For us since everything is standardized and automated using the NuGet Restore task in our DevOps build pipelines yes it's really more convenient for us to point to our Proget Feed (thanks again for your product) and gather everything from there. We can for sure find workarounds also on our side but we had to try and contact your team as your company has been way more helpful to us than any support from Microsoft.

Thanks a lot for yours and Rich`s precious time spent helping us through this issue.