Aggregating 2 Feeds in ProGet Free Edition

kc_2466

I'm currently evaluating ProGet using the Free Edition.

I've setup a feed 'npm-internal' with no connectors to act as storage for internal packages
I've also setup a feed 'npm-external' to proxy and cache registry.npmjs.org

I would like to be able to aggregate these into 1 feed, which the documentation suggests is possible in the free edition.
You are allowed to connect to 'other feeds in the same instance'

I appreciate it's possible to achieve this by just using one feed, but then it might become difficult to distinguish which were internal packages should you want to migrate to a different system in the future.

So i've tried to create a feed to combine to the two, but unsuccessfully:

Create a new NPM feed -> Connect to a ProGet Feed -> Connect to a Feed on this server -> choose 'npm-internal', give it a name etc. If you go to the feed page for this new feed, it says an error occurred while querying connectors for this feed (Connection refused). If you go to the connectors settings for this feed, you can add the npm-external connector, but if you go back to the feed page then you now get 2 connection refused errors.
Version 24.24 (Build 1) - running inside a docker container on Linux

atripp

Hi @kc_2466 ,

Self-connectors (i.e. a connector to another feed in the same instance) use the HTTP API, so the server (container) needs to talk to itself. The "connection refused" means that there's some kind of network configuration problem here.

You may need to try using a different host name/port/etc to allow for "self-communication" like this.

Cheers,
Alana

kc_2466

Thanks for the explanation - i think i've managed to resolve the connection refused issue, but that hasn't resolved the problem, and I now also have a error reporting license violations.

I had originally mapped port 80 on the docker container to port 8080, as my docker was running rootless, and it wouldn't bind to ports below 1024. However confusingly when tring to connect to the host machine on port 8080 from inside the container it resulted in connection refused... but port 80 seemed to be accepting connections (even though there was no service running on port 80 on the host).

So switching back to port 80 seemed like the easiest solution - which has made the errors reporting connection refused disappear, however I now have a red banner reporting license violations, and the aggregate feed still doesn't work. (maybe the violation detection blocks the feeds in question?).

Does the license violation work when using docker containers? In that does it understand that the container running ProGet isn't the same machine as the machine apparently providing the feeds (which is the docker host)

Alternatively have I caused this problem by changing the port number which makes it look like a different instance, and there is still a reference to port 8080 somewhere. (i've updated Advanced Settings -> Web.BaseURL)

kc_2466

I've found other forum posts that suggests unsetting Web.BaseURL or creating an entry in /etc/hosts which maps the machine name to 127.0.0.1 - but neither of these seem to help.

atripp

Hi @kc_2466

License violations are recorded from requests that are not local. You can clear recorded violations by clicking "change" on the license key and then save. No need to actually change it.

This is the logic used to determine if a request is local:

public bool IsLocal
{
    get
    {
        var connection = this.NativeRequest.HttpContext.Connection;
        if (connection.RemoteIpAddress != null)
        {
            if (connection.LocalIpAddress != null)
                return connection.RemoteIpAddress.Equals(connection.LocalIpAddress);
            else
                return IPAddress.IsLoopback(connection.RemoteIpAddress);
        }

        if (connection.RemoteIpAddress == null && connection.LocalIpAddress == null)
            return true;

        return false;
    }
}

So if you are continuing to see violations, it means that you need to make sure that the local/inbound IPs are the same or the inbound request is loopback (127.0.0.1).

This may require some configuration in your ngnix container.

Cheers,
Alana

kc_2466

Many thanks, i've now resolved the violation issue. It was very helpful to be able to clear the violation list to be able to see what was going on. Will the violation banner disappear on it's own in due course?

The solution was to add an entry in /etc/hosts in the docker container - I hadn't got it quite right previously. To do this:

docker exec -it proget bash
printf "127.0.0.1 <host_server_name>\n" >> /etc/hosts

atripp

@kc_2466 great news, thanks for letting us know.

If you don't see any violations recorded, then the banner should go away soon. It's cached.

kc_2466

I've now tested aggregating 2 feeds for NuGet, Conan, NPM and PyPi with mixed results:

NuGet aggregate feed works absolutely perfectly.

Conan aggregate feed essentially works perfectly - however when you fetch a new Conan package from the aggregate feed, it doesn't show up in the package list in the web-ui of the "conan-external" feed which is set to cache. (note the aggregate feed is not set to cache). However it looks like any dependencies of that package are showing up as fetched.

PyPi aggregate feed doesn't work at all (although both the internal and external feeds are working separately):

> pip install yolk3k
Looking in indexes: http://<servername>/pypi/pypi-aggregate/simple
Collecting yolk3k
ERROR: Could not install packages due to an OSError: HTTPConnectionPool(host='<servername>', port=80): Max retries exceeded with url: /pypi/pypi-aggregate/download/yolk3k/0.9/yolk3k-0.9.tar.gz (Caused by ResponseError('too many 500 error responses'))

NPM aggregate feed doesn't work at all either (although both the internal and external feeds are working separately):

> npm i zlib
(very long pause)
npm error code E500
npm error 500 Internal Server Error - GET http://<servername>/npm/npm-aggregate/zlib/-/zlib-1.0.5.tgz - NotSupportedException
npm error A complete log of this run can be found in: /home/???/.npm/_logs/2025-01-23T14_41_20_729Z-debug-0.log

atripp

@kc_2466 there are some 500 errors, so please check Admin > Diagnostic Center to see what those are about. Those should each be logged.

kc_2466

For the PyPi feed:

An error occurred processing a GET request to http://servername/pypi/pypi-aggregate/download/package_name/0.0.10/package_name-0.0.10-py3-none-any.whl: Specified method is not supported.

System.NotSupportedException: Specified method is not supported.
   at System.Net.Http.HttpBaseStream.get_Length()
   at Inedo.IO.DisposingStream.get_Length()
   at Inedo.Web.AhHttpResponse.TransmitStreamAsync(Stream stream, String fileName, String contentType, Nullable`1 contentLength)
   at Inedo.ProGet.WebApplication.FeedEndpoints.Pypi.PackageDownloadHandler.ProcessRequestAsync(AhHttpContext context, WebApiContext apiContext, PypiFeed feed, String relativeUrl)
   at Inedo.ProGet.WebApplication.FeedEndpoints.Pypi.PypiHandler.ProcessRequestAsync(AhHttpContext context, WebApiContext apiContext, PypiFeed feed, String relativeUrl) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E520165\Src\ProGet.WebApplication\FeedEndpoints\Pypi\PypiHandler.cs:line 34
   at Inedo.ProGet.WebApplication.FeedEndpoints.FeedEndpointHandler.FeedRequestHandler.ProcessRequestAsync(AhHttpContext context) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E520165\Src\ProGet.WebApplication\FeedEndpoints\FeedEndpointHandler.cs:line 191

::Web Error on 01/24/2025 08:48:40::

And for the NPM Feed:

An error occurred processing a GET request to http://servername/npm/npm-aggregate/@company_name/package_name/-/package_name-1.0.28.tgz: Specified method is not supported.

System.NotSupportedException: Specified method is not supported.
   at System.Net.Http.HttpBaseStream.get_Length()
   at Inedo.IO.DisposingStream.get_Length()
   at Inedo.Web.AhHttpResponse.TransmitStreamAsync(Stream stream, String fileName, String contentType, Nullable`1 contentLength)
   at Inedo.ProGet.WebApplication.FeedEndpoints.Npm.NpmPackageDownloadHandler.TryProcessRequestAsync(AhHttpContext context, WebApiContext apiContext, NpmFeed feed, String relativeUrl)
   at Inedo.ProGet.WebApplication.FeedEndpoints.Npm.NpmHandler.ProcessRequestAsync(AhHttpContext context, WebApiContext apiContext, NpmFeed feed, String relativeUrl) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E520165\Src\ProGet.WebApplication\FeedEndpoints\Npm\NpmHandler.cs:line 111
   at Inedo.ProGet.WebApplication.FeedEndpoints.FeedEndpointHandler.FeedRequestHandler.ProcessRequestAsync(AhHttpContext context) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E520165\Src\ProGet.WebApplication\FeedEndpoints\FeedEndpointHandler.cs:line 191

::Web Error on 01/24/2025 09:20:33::

In both cases if I change the feed name in the URL to pypi-internal or npn-internal as applicable, then those files are downloaded correctly.

It's also worth noting that in the web user interface neither the pypi-aggregate nor the npm-aggregate feeds show any packages on the feed's page.

The URLs that the connectors are configured to are of the form:
http://servername/pypi/pypi-internal/simple
and
http://servername/npm/npm-internal/

gdivis

@kc_2466 Thanks for reporting these issues! They turned out to be trivial to fix, so we are able to include them as PG-2879 and PG-2880 in today's 2024.25 release.

kc_2466

@gdivis - Brilliant, I've just switched over to 2024.25, and they both are now working perfectly.
PyPi still isn't showing any packages in on the webpage for the feed - but that's not essential.
Thanks for the impressive swift resolution.

kc_2466

ProGet Version 2024.25
I've found exactly the same problem exists with Rust Crates... You can access all the packages correctly in the cargo-external feed.

However if you create a feed (cargo-aggregate) to aggregate cargo-external and cargo-internal then you can see the packages listed on the feed page for cargo-aggregate, but if you try to click through to any of the packages, or you try to download from this feed via cargo, then you get a 500 error.
(Note: caching is switched off on the aggregate feed)

dean-houston

Hi @kc_2466 , we'll get this one fixed via PG-2885 in the next maintenance release!

gdivis

Hi @kc_2466,

I can't reproduce this in ProGet 2024.25. Is it possible you are using a prerelease version? It should report ProGet 2024.25 Build 11 in the page footer.

I'm confirming because this was fixed across all feeds as a relatively late commit before the release.

Thanks!
-Greg

kc_2466

Hi @gdivis

Ah yes, i was on 2024.0.25 build 10 (interesting to note that 24.0.25-ci10 shows up on https://proget.inedo.com/containers/repositories/ProductImages/inedo/proget/overview in the 'recent images' section of the overview tab, but 24.0.25-ci11 doesn't (but it is listed in 'all tags')

I've now switched over to 2024.0.25-ci11 but still have the same problem.

An error occurred processing a GET request to http://<servername>/cargo/cargo-aggregate/bi/tf/bitflags: The remote server returned an error: (404) Not Found.

System.Net.WebException: The remote server returned an error: (404) Not Found.
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location ---
   at System.Net.WebRequest.GetResponseAsync()
   at Inedo.ProGet.GlobalExtensions.GetResponseAsync(HttpWebRequest request, CancellationToken cancellationToken) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E523661\Src\ProGetCoreEx\GlobalExtensions.cs:line 36
   at Inedo.ProGet.Feeds.Connector.GetDirectResponseAsync(String url, CancellationToken cancellationToken) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E523661\Src\ProGetCoreEx\Feeds\Connector.cs:line 525
   at Inedo.ProGet.Feeds.Cargo.CargoConnector.GetPackagesAsync(PackageNameId name, String version, CancellationToken cancellationToken)+MoveNext() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E523661\Src\ProGetCoreEx\Feeds\Cargo\CargoConnector.cs:line 96
   at Inedo.ProGet.Feeds.Cargo.CargoConnector.GetPackagesAsync(PackageNameId name, String version, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at Inedo.ProGet.Feeds.StandardCanonicalPackageFeed`5.AggregateWithConnectors(IEnumerable`1 localPackages, Func`3 getConnectorPackages, CancellationToken cancellationToken)+MoveNext() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E523661\Src\ProGetCoreEx\Feeds\StandardCanonicalPackageFeed.cs:line 395
   at Inedo.ProGet.Feeds.StandardCanonicalPackageFeed`5.AggregateWithConnectors(IEnumerable`1 localPackages, Func`3 getConnectorPackages, CancellationToken cancellationToken)+MoveNext() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E523661\Src\ProGetCoreEx\Feeds\StandardCanonicalPackageFeed.cs:line 395
   at Inedo.ProGet.Feeds.StandardCanonicalPackageFeed`5.AggregateWithConnectors(IEnumerable`1 localPackages, Func`3 getConnectorPackages, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at System.Collections.Generic.AsyncEnumerableHelpers.ToArrayWithLength[T](IAsyncEnumerable`1 source, CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncEnumerableHelpers.cs:line 49
   at System.Collections.Generic.AsyncEnumerableHelpers.ToArrayWithLength[T](IAsyncEnumerable`1 source, CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncEnumerableHelpers.cs:line 41
   at System.Linq.OrderedAsyncEnumerable`1.ToListAsync(CancellationToken cancellationToken)
   at Inedo.ProGet.WebApplication.FeedEndpoints.Cargo.CargoIndexHandler.ProcessRequestAsync(AhHttpContext context, WebApiContext apiContext, CargoFeed feed, String relativeUrl) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E523661\Src\ProGet.WebApplication\FeedEndpoints\Cargo\CargoIndexHandler.cs:line 68
   at Inedo.ProGet.WebApplication.FeedEndpoints.Cargo.CargoFeedHandler.ProcessRequestAsync(AhHttpContext context, WebApiContext apiContext, CargoFeed feed, String relativeUrl) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E523661\Src\ProGet.WebApplication\FeedEndpoints\Cargo\CargoFeedHandler.cs:line 107
   at Inedo.ProGet.WebApplication.FeedEndpoints.CanonicalFeedHandler`1.Inedo.Web.IHttpAsyncHandler.ProcessRequestAsync(AhHttpContext context) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E523661\Src\ProGet.WebApplication\FeedEndpoints\CanonicalFeedHandler.cs:line 85

::Web Error on 02/07/2025 10:07:23::

kc_2466

To be fair, i think i might have a bigger problem - i may have corrupted my database somehow.
If I go to the Feeds, and then click 'connectors', i get I get a 500 error "Invalid Feed type".

An error occurred in the web application: Invalid feed type.

URL: http://<servername>/connectors
Referrer: http://<servername>/feeds
User: <username>
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36
Stack trace:    at Inedo.ProGet.Feeds.Connector.GetConnector(Connectors dbConnector) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E523495\Src\ProGetCoreEx\Feeds\Connector.cs:line 234
   at Inedo.ProGet.WebApplication.Pages.Connectors.ListConnectorsPage.CreateChildControls() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E523495\Src\ProGet.WebApplication\Pages\Connectors\ListConnectorsPage.cs:line 38
   at Inedo.ProGet.WebApplication.Pages.ProGetSimplePage.InitializeAsync() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E523495\Src\ProGet.WebApplication\Pages\ProGetSimplePage.cs:line 56
   at Inedo.Web.PageFree.SimplePageBase.ExecutePageLifeCycleAsync()
   at Inedo.Web.PageFree.SimplePageBase.ProcessRequestAsync(AhHttpContext context)
   at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context)

::Web Error on 02/07/2025 15:52:30::

I've tried deleting all the Cargo feeds to try and fix the issue, and then re-created them, but that's not helped.

dean-houston

@kc_2466 the "invalid feed type" will come up if you have a connector or feed that was created in a newer version of ProGet that wasn't available in an older version of ProGet... and you downgraded to the older version.

It looks like it's a connector, based on the URL. Easiest way to fix is to just upgrade, delete, downgrade

gdivis

Looks like I had misunderstood the exact problem. One related issue was already fixed in 2024.25, but the aggregation issue with one connector having a package and another not having it was not. I have confirmed that both will be fixed in today's release of 2024.26, along with that Invalid Feed Type error.

Sorry for the confusion!