Aggregating 2 Feeds in ProGet Free Edition

atripp

@kc_2466 there are some 500 errors, so please check Admin > Diagnostic Center to see what those are about. Those should each be logged.

kc_2466

For the PyPi feed:

An error occurred processing a GET request to http://servername/pypi/pypi-aggregate/download/package_name/0.0.10/package_name-0.0.10-py3-none-any.whl: Specified method is not supported.

System.NotSupportedException: Specified method is not supported.
   at System.Net.Http.HttpBaseStream.get_Length()
   at Inedo.IO.DisposingStream.get_Length()
   at Inedo.Web.AhHttpResponse.TransmitStreamAsync(Stream stream, String fileName, String contentType, Nullable`1 contentLength)
   at Inedo.ProGet.WebApplication.FeedEndpoints.Pypi.PackageDownloadHandler.ProcessRequestAsync(AhHttpContext context, WebApiContext apiContext, PypiFeed feed, String relativeUrl)
   at Inedo.ProGet.WebApplication.FeedEndpoints.Pypi.PypiHandler.ProcessRequestAsync(AhHttpContext context, WebApiContext apiContext, PypiFeed feed, String relativeUrl) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E520165\Src\ProGet.WebApplication\FeedEndpoints\Pypi\PypiHandler.cs:line 34
   at Inedo.ProGet.WebApplication.FeedEndpoints.FeedEndpointHandler.FeedRequestHandler.ProcessRequestAsync(AhHttpContext context) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E520165\Src\ProGet.WebApplication\FeedEndpoints\FeedEndpointHandler.cs:line 191

::Web Error on 01/24/2025 08:48:40::

And for the NPM Feed:

An error occurred processing a GET request to http://servername/npm/npm-aggregate/@company_name/package_name/-/package_name-1.0.28.tgz: Specified method is not supported.

System.NotSupportedException: Specified method is not supported.
   at System.Net.Http.HttpBaseStream.get_Length()
   at Inedo.IO.DisposingStream.get_Length()
   at Inedo.Web.AhHttpResponse.TransmitStreamAsync(Stream stream, String fileName, String contentType, Nullable`1 contentLength)
   at Inedo.ProGet.WebApplication.FeedEndpoints.Npm.NpmPackageDownloadHandler.TryProcessRequestAsync(AhHttpContext context, WebApiContext apiContext, NpmFeed feed, String relativeUrl)
   at Inedo.ProGet.WebApplication.FeedEndpoints.Npm.NpmHandler.ProcessRequestAsync(AhHttpContext context, WebApiContext apiContext, NpmFeed feed, String relativeUrl) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E520165\Src\ProGet.WebApplication\FeedEndpoints\Npm\NpmHandler.cs:line 111
   at Inedo.ProGet.WebApplication.FeedEndpoints.FeedEndpointHandler.FeedRequestHandler.ProcessRequestAsync(AhHttpContext context) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E520165\Src\ProGet.WebApplication\FeedEndpoints\FeedEndpointHandler.cs:line 191

::Web Error on 01/24/2025 09:20:33::

In both cases if I change the feed name in the URL to pypi-internal or npn-internal as applicable, then those files are downloaded correctly.

It's also worth noting that in the web user interface neither the pypi-aggregate nor the npm-aggregate feeds show any packages on the feed's page.

The URLs that the connectors are configured to are of the form:
http://servername/pypi/pypi-internal/simple
and
http://servername/npm/npm-internal/

gdivis

@kc_2466 Thanks for reporting these issues! They turned out to be trivial to fix, so we are able to include them as PG-2879 and PG-2880 in today's 2024.25 release.

kc_2466

@gdivis - Brilliant, I've just switched over to 2024.25, and they both are now working perfectly.
PyPi still isn't showing any packages in on the webpage for the feed - but that's not essential.
Thanks for the impressive swift resolution.

kc_2466

ProGet Version 2024.25
I've found exactly the same problem exists with Rust Crates... You can access all the packages correctly in the cargo-external feed.

However if you create a feed (cargo-aggregate) to aggregate cargo-external and cargo-internal then you can see the packages listed on the feed page for cargo-aggregate, but if you try to click through to any of the packages, or you try to download from this feed via cargo, then you get a 500 error.
(Note: caching is switched off on the aggregate feed)

dean-houston

Hi @kc_2466 , we'll get this one fixed via PG-2885 in the next maintenance release!

gdivis

Hi @kc_2466,

I can't reproduce this in ProGet 2024.25. Is it possible you are using a prerelease version? It should report ProGet 2024.25 Build 11 in the page footer.

I'm confirming because this was fixed across all feeds as a relatively late commit before the release.

Thanks!
-Greg

kc_2466

Hi @gdivis

Ah yes, i was on 2024.0.25 build 10 (interesting to note that 24.0.25-ci10 shows up on https://proget.inedo.com/containers/repositories/ProductImages/inedo/proget/overview in the 'recent images' section of the overview tab, but 24.0.25-ci11 doesn't (but it is listed in 'all tags')

I've now switched over to 2024.0.25-ci11 but still have the same problem.

An error occurred processing a GET request to http://<servername>/cargo/cargo-aggregate/bi/tf/bitflags: The remote server returned an error: (404) Not Found.

System.Net.WebException: The remote server returned an error: (404) Not Found.
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location ---
   at System.Net.WebRequest.GetResponseAsync()
   at Inedo.ProGet.GlobalExtensions.GetResponseAsync(HttpWebRequest request, CancellationToken cancellationToken) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E523661\Src\ProGetCoreEx\GlobalExtensions.cs:line 36
   at Inedo.ProGet.Feeds.Connector.GetDirectResponseAsync(String url, CancellationToken cancellationToken) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E523661\Src\ProGetCoreEx\Feeds\Connector.cs:line 525
   at Inedo.ProGet.Feeds.Cargo.CargoConnector.GetPackagesAsync(PackageNameId name, String version, CancellationToken cancellationToken)+MoveNext() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E523661\Src\ProGetCoreEx\Feeds\Cargo\CargoConnector.cs:line 96
   at Inedo.ProGet.Feeds.Cargo.CargoConnector.GetPackagesAsync(PackageNameId name, String version, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at Inedo.ProGet.Feeds.StandardCanonicalPackageFeed`5.AggregateWithConnectors(IEnumerable`1 localPackages, Func`3 getConnectorPackages, CancellationToken cancellationToken)+MoveNext() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E523661\Src\ProGetCoreEx\Feeds\StandardCanonicalPackageFeed.cs:line 395
   at Inedo.ProGet.Feeds.StandardCanonicalPackageFeed`5.AggregateWithConnectors(IEnumerable`1 localPackages, Func`3 getConnectorPackages, CancellationToken cancellationToken)+MoveNext() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E523661\Src\ProGetCoreEx\Feeds\StandardCanonicalPackageFeed.cs:line 395
   at Inedo.ProGet.Feeds.StandardCanonicalPackageFeed`5.AggregateWithConnectors(IEnumerable`1 localPackages, Func`3 getConnectorPackages, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at System.Collections.Generic.AsyncEnumerableHelpers.ToArrayWithLength[T](IAsyncEnumerable`1 source, CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncEnumerableHelpers.cs:line 49
   at System.Collections.Generic.AsyncEnumerableHelpers.ToArrayWithLength[T](IAsyncEnumerable`1 source, CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncEnumerableHelpers.cs:line 41
   at System.Linq.OrderedAsyncEnumerable`1.ToListAsync(CancellationToken cancellationToken)
   at Inedo.ProGet.WebApplication.FeedEndpoints.Cargo.CargoIndexHandler.ProcessRequestAsync(AhHttpContext context, WebApiContext apiContext, CargoFeed feed, String relativeUrl) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E523661\Src\ProGet.WebApplication\FeedEndpoints\Cargo\CargoIndexHandler.cs:line 68
   at Inedo.ProGet.WebApplication.FeedEndpoints.Cargo.CargoFeedHandler.ProcessRequestAsync(AhHttpContext context, WebApiContext apiContext, CargoFeed feed, String relativeUrl) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E523661\Src\ProGet.WebApplication\FeedEndpoints\Cargo\CargoFeedHandler.cs:line 107
   at Inedo.ProGet.WebApplication.FeedEndpoints.CanonicalFeedHandler`1.Inedo.Web.IHttpAsyncHandler.ProcessRequestAsync(AhHttpContext context) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E523661\Src\ProGet.WebApplication\FeedEndpoints\CanonicalFeedHandler.cs:line 85

::Web Error on 02/07/2025 10:07:23::

kc_2466

To be fair, i think i might have a bigger problem - i may have corrupted my database somehow.
If I go to the Feeds, and then click 'connectors', i get I get a 500 error "Invalid Feed type".

An error occurred in the web application: Invalid feed type.

URL: http://<servername>/connectors
Referrer: http://<servername>/feeds
User: <username>
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36
Stack trace:    at Inedo.ProGet.Feeds.Connector.GetConnector(Connectors dbConnector) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E523495\Src\ProGetCoreEx\Feeds\Connector.cs:line 234
   at Inedo.ProGet.WebApplication.Pages.Connectors.ListConnectorsPage.CreateChildControls() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E523495\Src\ProGet.WebApplication\Pages\Connectors\ListConnectorsPage.cs:line 38
   at Inedo.ProGet.WebApplication.Pages.ProGetSimplePage.InitializeAsync() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E523495\Src\ProGet.WebApplication\Pages\ProGetSimplePage.cs:line 56
   at Inedo.Web.PageFree.SimplePageBase.ExecutePageLifeCycleAsync()
   at Inedo.Web.PageFree.SimplePageBase.ProcessRequestAsync(AhHttpContext context)
   at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context)

::Web Error on 02/07/2025 15:52:30::

I've tried deleting all the Cargo feeds to try and fix the issue, and then re-created them, but that's not helped.

dean-houston

@kc_2466 the "invalid feed type" will come up if you have a connector or feed that was created in a newer version of ProGet that wasn't available in an older version of ProGet... and you downgraded to the older version.

It looks like it's a connector, based on the URL. Easiest way to fix is to just upgrade, delete, downgrade

gdivis

Looks like I had misunderstood the exact problem. One related issue was already fixed in 2024.25, but the aggregation issue with one connector having a package and another not having it was not. I have confirmed that both will be fixed in today's release of 2024.26, along with that Invalid Feed Type error.

Sorry for the confusion!

kc_2466

I've now found a cleaner way of adding this entry to the host file...

If using docker run, add the argument:

--add-host=<host_server_name>=127.0.0.1

if using a docker compose file, add an entry:

extra_hosts:
    - "<host_server_name>: 127.0.0.1"