Aggregating 2 Feeds in ProGet Free Edition

[kc_2466]

I'm currently evaluating ProGet using the Free Edition.

I've setup a feed 'npm-internal' with no connectors to act as storage for internal packages
I've also setup a feed 'npm-external' to proxy and cache registry.npmjs.org

I would like to be able to aggregate these into 1 feed, which the documentation suggests is possible in the free edition.
You are allowed to connect to 'other feeds in the same instance'

I appreciate it's possible to achieve this by just using one feed, but then it might become difficult to distinguish which were internal packages should you want to migrate to a different system in the future.

So i've tried to create a feed to combine to the two, but unsuccessfully:

• Create a new NPM feed -> Connect to a ProGet Feed -> Connect to a Feed on this server -> choose 'npm-internal', give it a name etc. If you go to the feed page for this new feed, it says an error occurred while querying connectors for this feed (Connection refused). If you go to the connectors settings for this feed, you can add the npm-external connector, but if you go back to the feed page then you now get 2 connection refused errors.

• Version 24.24 (Build 1) - running inside a docker container on Linux

[atripp]

Hi @kc_2466 ,

Self-connectors (i.e. a connector to another feed in the same instance) use the HTTP API, so the server (container) needs to talk to itself. The "connection refused" means that there's some kind of network configuration problem here.

You may need to try using a different host name/port/etc to allow for "self-communication" like this.

Cheers,
Alana

[kc_2466]

Thanks for the explanation - i think i've managed to resolve the connection refused issue, but that hasn't resolved the problem, and I now also have a error reporting license violations.

I had originally mapped port 80 on the docker container to port 8080, as my docker was running rootless, and it wouldn't bind to ports below 1024. However confusingly when tring to connect to the host machine on port 8080 from inside the container it resulted in connection refused... but port 80 seemed to be accepting connections (even though there was no service running on port 80 on the host).

So switching back to port 80 seemed like the easiest solution - which has made the errors reporting connection refused disappear, however I now have a red banner reporting license violations, and the aggregate feed still doesn't work. (maybe the violation detection blocks the feeds in question?).

Does the license violation work when using docker containers? In that does it understand that the container running ProGet isn't the same machine as the machine apparently providing the feeds (which is the docker host)

Alternatively have I caused this problem by changing the port number which makes it look like a different instance, and there is still a reference to port 8080 somewhere. (i've updated Advanced Settings -> Web.BaseURL)

[kc_2466]

I've found other forum posts that suggests unsetting Web.BaseURL or creating an entry in /etc/hosts which maps the machine name to 127.0.0.1 - but neither of these seem to help.

[atripp]

Hi @kc_2466

License violations are recorded from requests that are not local. You can clear recorded violations by clicking "change" on the license key and then save. No need to actually change it.

This is the logic used to determine if a request is local:

public bool IsLocal
{
    get
    {
        var connection = this.NativeRequest.HttpContext.Connection;
        if (connection.RemoteIpAddress != null)
        {
            if (connection.LocalIpAddress != null)
                return connection.RemoteIpAddress.Equals(connection.LocalIpAddress);
            else
                return IPAddress.IsLoopback(connection.RemoteIpAddress);
        }

        if (connection.RemoteIpAddress == null && connection.LocalIpAddress == null)
            return true;

        return false;
    }
}

So if you are continuing to see violations, it means that you need to make sure that the local/inbound IPs are the same or the inbound request is loopback (127.0.0.1).

This may require some configuration in your ngnix container.

Cheers,
Alana

[kc_2466]

Many thanks, i've now resolved the violation issue. It was very helpful to be able to clear the violation list to be able to see what was going on. Will the violation banner disappear on it's own in due course?

The solution was to add an entry in /etc/hosts in the docker container - I hadn't got it quite right previously. To do this:

docker exec -it proget bash
printf "127.0.0.1 <host_server_name>\n" >> /etc/hosts

[atripp]

@kc_2466 great news, thanks for letting us know.

If you don't see any violations recorded, then the banner should go away soon. It's cached.

[kc_2466]

I've now tested aggregating 2 feeds for NuGet, Conan, NPM and PyPi with mixed results:

NuGet aggregate feed works absolutely perfectly.

Conan aggregate feed essentially works perfectly - however when you fetch a new Conan package from the aggregate feed, it doesn't show up in the package list in the web-ui of the "conan-external" feed which is set to cache. (note the aggregate feed is not set to cache). However it looks like any dependencies of that package are showing up as fetched.

PyPi aggregate feed doesn't work at all (although both the internal and external feeds are working separately):

> pip install yolk3k
Looking in indexes: http://<servername>/pypi/pypi-aggregate/simple
Collecting yolk3k
ERROR: Could not install packages due to an OSError: HTTPConnectionPool(host='<servername>', port=80): Max retries exceeded with url: /pypi/pypi-aggregate/download/yolk3k/0.9/yolk3k-0.9.tar.gz (Caused by ResponseError('too many 500 error responses'))

NPM aggregate feed doesn't work at all either (although both the internal and external feeds are working separately):

> npm i zlib
(very long pause)
npm error code E500
npm error 500 Internal Server Error - GET http://<servername>/npm/npm-aggregate/zlib/-/zlib-1.0.5.tgz - NotSupportedException
npm error A complete log of this run can be found in: /home/???/.npm/_logs/2025-01-23T14_41_20_729Z-debug-0.log

[atripp]

@kc_2466 there are some 500 errors, so please check Admin > Diagnostic Center to see what those are about. Those should each be logged.

[kc_2466]

For the PyPi feed:

An error occurred processing a GET request to http://servername/pypi/pypi-aggregate/download/package_name/0.0.10/package_name-0.0.10-py3-none-any.whl: Specified method is not supported.

System.NotSupportedException: Specified method is not supported.
   at System.Net.Http.HttpBaseStream.get_Length()
   at Inedo.IO.DisposingStream.get_Length()
   at Inedo.Web.AhHttpResponse.TransmitStreamAsync(Stream stream, String fileName, String contentType, Nullable`1 contentLength)
   at Inedo.ProGet.WebApplication.FeedEndpoints.Pypi.PackageDownloadHandler.ProcessRequestAsync(AhHttpContext context, WebApiContext apiContext, PypiFeed feed, String relativeUrl)
   at Inedo.ProGet.WebApplication.FeedEndpoints.Pypi.PypiHandler.ProcessRequestAsync(AhHttpContext context, WebApiContext apiContext, PypiFeed feed, String relativeUrl) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E520165\Src\ProGet.WebApplication\FeedEndpoints\Pypi\PypiHandler.cs:line 34
   at Inedo.ProGet.WebApplication.FeedEndpoints.FeedEndpointHandler.FeedRequestHandler.ProcessRequestAsync(AhHttpContext context) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E520165\Src\ProGet.WebApplication\FeedEndpoints\FeedEndpointHandler.cs:line 191

::Web Error on 01/24/2025 08:48:40::

And for the NPM Feed:

An error occurred processing a GET request to http://servername/npm/npm-aggregate/@company_name/package_name/-/package_name-1.0.28.tgz: Specified method is not supported.

System.NotSupportedException: Specified method is not supported.
   at System.Net.Http.HttpBaseStream.get_Length()
   at Inedo.IO.DisposingStream.get_Length()
   at Inedo.Web.AhHttpResponse.TransmitStreamAsync(Stream stream, String fileName, String contentType, Nullable`1 contentLength)
   at Inedo.ProGet.WebApplication.FeedEndpoints.Npm.NpmPackageDownloadHandler.TryProcessRequestAsync(AhHttpContext context, WebApiContext apiContext, NpmFeed feed, String relativeUrl)
   at Inedo.ProGet.WebApplication.FeedEndpoints.Npm.NpmHandler.ProcessRequestAsync(AhHttpContext context, WebApiContext apiContext, NpmFeed feed, String relativeUrl) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E520165\Src\ProGet.WebApplication\FeedEndpoints\Npm\NpmHandler.cs:line 111
   at Inedo.ProGet.WebApplication.FeedEndpoints.FeedEndpointHandler.FeedRequestHandler.ProcessRequestAsync(AhHttpContext context) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E520165\Src\ProGet.WebApplication\FeedEndpoints\FeedEndpointHandler.cs:line 191

::Web Error on 01/24/2025 09:20:33::

In both cases if I change the feed name in the URL to pypi-internal or npn-internal as applicable, then those files are downloaded correctly.

It's also worth noting that in the web user interface neither the pypi-aggregate nor the npm-aggregate feeds show any packages on the feed's page.

The URLs that the connectors are configured to are of the form:
http://servername/pypi/pypi-internal/simple
and
http://servername/npm/npm-internal/

[gdivis]

@kc_2466 Thanks for reporting these issues! They turned out to be trivial to fix, so we are able to include them as PG-2879 and PG-2880 in today's 2024.25 release.

[kc_2466]

@gdivis - Brilliant, I've just switched over to 2024.25, and they both are now working perfectly.
PyPi still isn't showing any packages in on the webpage for the feed - but that's not essential.
Thanks for the impressive swift resolution.

[kc_2466]

ProGet Version 2024.25
I've found exactly the same problem exists with Rust Crates... You can access all the packages correctly in the cargo-external feed.

However if you create a feed (cargo-aggregate) to aggregate cargo-external and cargo-internal then you can see the packages listed on the feed page for cargo-aggregate, but if you try to click through to any of the packages, or you try to download from this feed via cargo, then you get a 500 error.
(Note: caching is switched off on the aggregate feed)