Different query results nuget feed

[pmsensi]

Hello,

I have a feed connected to nuget.org and to an internal feed.

When I do a query to this feed for a specific package, some information is missing compared to the result if I do the same query to nuget.org.

Query to internal feed result (https://internal/nuget/Development/v3/search?q=microsoft.playwright.nunit&take=1):

Query to nuget.org feed result (https://azuresearch-usnc.nuget.org/query?q=microsoft.playwright.nunit&take=1)

From what I see, the result of the internal query only returns some information about the local packages, but other important information from metadata is missing, for example, the owners.

Can you please advise what I need to do on this feed/query to be able to return the same information as present on nuget.org, since the feed is connected to there?

Thank you.

[dean-houston]

Hi @pmsensi ,

Unfortunately, with multiple connectors, "quirky" search results are to be expected, especially with broad searches.

The technical reason for this is quite complex, but has to do with how the NuGet search API is designed (unbounded, paged results with pages of indeterminate size), the fact we need to support both v2 and v3 APIs, and the need to aggregate multiple (often conflicting) search resultsets into a single resultset in a performant manner.

What are you trying to accomplish with the search api? It's mostly intended for visual studio.

If you already know the package name (microsoft.playwright.nunit), you should use the registrations api (e.g. microsoft.playwright.nunit/index.json).

As far as missing fields, I can't answer why those aren't there - it could be relatively new to the search api, or omitted for performance reasons.

Cheers,
Dean

[pmsensi]

Hi @dean-houston ,

thank you for your quick response.

The missing values from the search API are not new, the owners exist since v2 at least.
I searched for a package that I don't have any local package, then the search works as expected (all versions and owners are there). Then I deleted the cached package (microsoft.playwright.nunit) and the search result contains now all versions and owners.

So, I would say that for me is a clear relation between the cache and the missing information.

I understand that can be hard, and can cause performance problems, but the version information is present when I access the package page from ProGet, so, that information should be available over search API and not only the local packages, in my opinion.

Of course, a possible solution is to disable the cache from that feed/connector. So then, no local packages will be present. Of course, with this change, I will not be able to use the full power/features of ProGet :\

Regarding the registration API, the amount of data present is huge! A lot of not necessary information and the owners entry is not there too :)

I would like to point out that, the owners were just an example of the missing properties, there are more missing. But the important ones for me are the versions.

I'm using the search API in our internal tool to verify if a new version is available, if exists on the feed, etc.
I encounter this problem, when I was setting our internal tool to only use ProGet feeds (mixed connectors), so that way, we can block internet access (nuget.org) to the server where that tool is running, and only ProGet server will have access to nuget.org.

Do you see any other way to get all the versions ?

Thank you for your help.

[NanciCalo]

Hey @pmsensi ,

Just to jump in here... it sounds like there are two issues.

I think the first issue is that properties like owners, verified, etc. are not returned for local packages.

Those properties are not "package metadata" properties (i.e. stored inside the package), but they're "server metadata" properties (i.e. stored outside the package, on the server). Other server metadata properties are downloadCount and listed.

ProGet does not support all of the server metadata properties that nuget.org does; but even if it did, the properties would not be copied from a connector when a package is added to the feed. This is why downloadCount turns to 0 when cache a package.

However, server metadata properties are "passed through" remote packages.

The "package metadata" properties are contained in the .nuspec manifest file:

<package xmlns="http://schemas.microsoft.com/packaging/2013/05/nuspec.xsd">
  <metadata>
    <id>Microsoft.Playwright.NUnit</id>
    <version>1.35.0</version>
    <title>Microsoft.Playwright.NUnit</title>
    <authors>Microsoft</authors>
    <license type="expression">MIT</license>
    <licenseUrl>https://licenses.nuget.org/MIT</licenseUrl>
    <icon>icon.png</icon>
    <projectUrl>https://github.com/microsoft/playwright-dotnet</projectUrl>
    <description>Playwright enables reliable end-to-end testing for modern web apps. This package brings in additional helpers and fixtures to enable using it within NUnit.</description>
    <copyright>© Microsoft Corporation. All rights reserved.</copyright>
    <tags>headless,chrome,firefox,webkit,playwright</tags>
    <repository type="git" url="https://github.com/microsoft/playwright-dotnet.git" commit="ca7b02ac910e573666f48d502bfabb1c17639e68"/>
    <dependencies>
      <group targetFramework=".NETStandard2.0">
        <dependency id="Microsoft.Playwright.TestAdapter" version="1.35.0" exclude="Build,Analyzers"/>
        <dependency id="Microsoft.Playwright" version="1.35.0" exclude="Build,Analyzers"/>
        <dependency id="Microsoft.NET.Test.Sdk" version="16.11.0" exclude="Build,Analyzers"/>
        <dependency id="NUnit" version="3.13.2" exclude="Build,Analyzers"/>
        <dependency id="NUnit3TestAdapter" version="4.0.0" exclude="Build,Analyzers"/>
      </group>
    </dependencies>
  </metadata>
</package>

The second issue, I think, is that the versions property is not aggregated. This isn't really feasible/possible to do as a result of the "paged searching" that Dean mentioned.

For example, if you search search a feed for q=foo,take=2, then ProGet will work as follows:

1. Search local packages for foo
2. Return up to 2 search results; if not enough results, then ...
3. Forward search to first connector
4. Return search results; if not enough, then...
5. Forward search to second connector
6. Etc...

To get the behavior that you'd like, ProGet would need to:

1. Search local packages for foo
2. For each search result, query each connector for a list of all package versions (registrations)
3. Aggregate those results into the version property
4. Return up to 2 search results; if not enough result then...

So this is why it's not really possible to get search API to behave like you'd expect.

Hope that helps,
Nanci

[pmsensi]

Hello @NanciCalo ,

thank you for the clarification.

So, as I wrote, the only way to have the expected behavior is to disable the cache from the feed/connector right ? Then the search mechanism will not search for local packages.

Best Regards

[stevedennis]

If you're looking for nuget.org-specific metadata, I recommend querying nuget.org directly; of course if you need to work-around internet access issues, you could configure a special feed/connector with no caching.

But if you're looking for latest version of a package, the registration API is your best choice. That's what Visual Studio (NuGet client) does for every package and dependency, every time a restore happens.