Inedo Community Forums Forums
    • Recent
    • Tags
    • Popular
    • Login

    Welcome to the Inedo Forums! Check out the Forums Guide for help getting started.

    If you are experiencing any issues with the forum software, please visit the Contact Form on our website and let us know!

    Aggregating 2 Feeds in ProGet Free Edition

    Scheduled Pinned Locked Moved Support
    20 Posts 4 Posters 79 Views 1 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • K Offline
      kc_2466
      last edited by

      Many thanks, i've now resolved the violation issue. It was very helpful to be able to clear the violation list to be able to see what was going on. Will the violation banner disappear on it's own in due course?

      The solution was to add an entry in /etc/hosts in the docker container - I hadn't got it quite right previously. To do this:

      docker exec -it proget bash
printf "127.0.0.1 <host_server_name>\n" >> /etc/hosts
      atrippA K 2 Replies Last reply Reply Quote 0
      • atrippA Offline
        atripp inedo-engineer @kc_2466
        last edited by

        @kc_2466 great news, thanks for letting us know.

        If you don't see any violations recorded, then the banner should go away soon. It's cached.

        1 Reply Last reply Reply Quote 0
        • K Offline
          kc_2466
          last edited by

          I've now tested aggregating 2 feeds for NuGet, Conan, NPM and PyPi with mixed results:

          NuGet aggregate feed works absolutely perfectly.

          Conan aggregate feed essentially works perfectly - however when you fetch a new Conan package from the aggregate feed, it doesn't show up in the package list in the web-ui of the "conan-external" feed which is set to cache. (note the aggregate feed is not set to cache). However it looks like any dependencies of that package are showing up as fetched.

          PyPi aggregate feed doesn't work at all (although both the internal and external feeds are working separately):

          > pip install yolk3k
Looking in indexes: http://<servername>/pypi/pypi-aggregate/simple
Collecting yolk3k
ERROR: Could not install packages due to an OSError: HTTPConnectionPool(host='<servername>', port=80): Max retries exceeded with url: /pypi/pypi-aggregate/download/yolk3k/0.9/yolk3k-0.9.tar.gz (Caused by ResponseError('too many 500 error responses'))

          NPM aggregate feed doesn't work at all either (although both the internal and external feeds are working separately):

          > npm i zlib
(very long pause)
npm error code E500
npm error 500 Internal Server Error - GET http://<servername>/npm/npm-aggregate/zlib/-/zlib-1.0.5.tgz - NotSupportedException
npm error A complete log of this run can be found in: /home/???/.npm/_logs/2025-01-23T14_41_20_729Z-debug-0.log
          atrippA 1 Reply Last reply Reply Quote 0
          • atrippA Offline
            atripp inedo-engineer @kc_2466
            last edited by

            @kc_2466 there are some 500 errors, so please check Admin > Diagnostic Center to see what those are about. Those should each be logged.

            K 1 Reply Last reply Reply Quote 0
            • K Offline
              kc_2466 @atripp
              last edited by

              For the PyPi feed:

              An error occurred processing a GET request to http://servername/pypi/pypi-aggregate/download/package_name/0.0.10/package_name-0.0.10-py3-none-any.whl: Specified method is not supported.

System.NotSupportedException: Specified method is not supported.
   at System.Net.Http.HttpBaseStream.get_Length()
   at Inedo.IO.DisposingStream.get_Length()
   at Inedo.Web.AhHttpResponse.TransmitStreamAsync(Stream stream, String fileName, String contentType, Nullable`1 contentLength)
   at Inedo.ProGet.WebApplication.FeedEndpoints.Pypi.PackageDownloadHandler.ProcessRequestAsync(AhHttpContext context, WebApiContext apiContext, PypiFeed feed, String relativeUrl)
   at Inedo.ProGet.WebApplication.FeedEndpoints.Pypi.PypiHandler.ProcessRequestAsync(AhHttpContext context, WebApiContext apiContext, PypiFeed feed, String relativeUrl) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E520165\Src\ProGet.WebApplication\FeedEndpoints\Pypi\PypiHandler.cs:line 34
   at Inedo.ProGet.WebApplication.FeedEndpoints.FeedEndpointHandler.FeedRequestHandler.ProcessRequestAsync(AhHttpContext context) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E520165\Src\ProGet.WebApplication\FeedEndpoints\FeedEndpointHandler.cs:line 191

::Web Error on 01/24/2025 08:48:40::

              And for the NPM Feed:

              An error occurred processing a GET request to http://servername/npm/npm-aggregate/@company_name/package_name/-/package_name-1.0.28.tgz: Specified method is not supported.

System.NotSupportedException: Specified method is not supported.
   at System.Net.Http.HttpBaseStream.get_Length()
   at Inedo.IO.DisposingStream.get_Length()
   at Inedo.Web.AhHttpResponse.TransmitStreamAsync(Stream stream, String fileName, String contentType, Nullable`1 contentLength)
   at Inedo.ProGet.WebApplication.FeedEndpoints.Npm.NpmPackageDownloadHandler.TryProcessRequestAsync(AhHttpContext context, WebApiContext apiContext, NpmFeed feed, String relativeUrl)
   at Inedo.ProGet.WebApplication.FeedEndpoints.Npm.NpmHandler.ProcessRequestAsync(AhHttpContext context, WebApiContext apiContext, NpmFeed feed, String relativeUrl) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E520165\Src\ProGet.WebApplication\FeedEndpoints\Npm\NpmHandler.cs:line 111
   at Inedo.ProGet.WebApplication.FeedEndpoints.FeedEndpointHandler.FeedRequestHandler.ProcessRequestAsync(AhHttpContext context) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E520165\Src\ProGet.WebApplication\FeedEndpoints\FeedEndpointHandler.cs:line 191

::Web Error on 01/24/2025 09:20:33::

              In both cases if I change the feed name in the URL to pypi-internal or npn-internal as applicable, then those files are downloaded correctly.

              It's also worth noting that in the web user interface neither the pypi-aggregate nor the npm-aggregate feeds show any packages on the feed's page.

              The URLs that the connectors are configured to are of the form:
              http://servername/pypi/pypi-internal/simple
              and
              http://servername/npm/npm-internal/

              1 Reply Last reply Reply Quote 0
              • gdivisG Offline
                gdivis inedo-engineer
                last edited by

                @kc_2466 Thanks for reporting these issues! They turned out to be trivial to fix, so we are able to include them as PG-2879 and PG-2880 in today's 2024.25 release.

                1 Reply Last reply Reply Quote 0
                • K Offline
                  kc_2466
                  last edited by

                  @gdivis - Brilliant, I've just switched over to 2024.25, and they both are now working perfectly.
                  PyPi still isn't showing any packages in on the webpage for the feed - but that's not essential.
                  Thanks for the impressive swift resolution.

                  1 Reply Last reply Reply Quote 0
                  • K Offline
                    kc_2466
                    last edited by kc_2466

                    ProGet Version 2024.25
                    I've found exactly the same problem exists with Rust Crates... You can access all the packages correctly in the cargo-external feed.

                    However if you create a feed (cargo-aggregate) to aggregate cargo-external and cargo-internal then you can see the packages listed on the feed page for cargo-aggregate, but if you try to click through to any of the packages, or you try to download from this feed via cargo, then you get a 500 error.
                    (Note: caching is switched off on the aggregate feed)

                    dean-houstonD 1 Reply Last reply Reply Quote 0
                    • dean-houstonD Offline
                      dean-houston inedo-engineer @kc_2466
                      last edited by

                      Hi @kc_2466 , we'll get this one fixed via PG-2885 in the next maintenance release!

                      1 Reply Last reply Reply Quote 0
                      • gdivisG Offline
                        gdivis inedo-engineer
                        last edited by

                        Hi @kc_2466,

                        I can't reproduce this in ProGet 2024.25. Is it possible you are using a prerelease version? It should report ProGet 2024.25 Build 11 in the page footer.

                        I'm confirming because this was fixed across all feeds as a relatively late commit before the release.

                        Thanks!
                        -Greg

                        1 Reply Last reply Reply Quote 0
                        • K Offline
                          kc_2466
                          last edited by

                          Hi @gdivis

                          Ah yes, i was on 2024.0.25 build 10 (interesting to note that 24.0.25-ci10 shows up on https://proget.inedo.com/containers/repositories/ProductImages/inedo/proget/overview in the 'recent images' section of the overview tab, but 24.0.25-ci11 doesn't (but it is listed in 'all tags')

                          I've now switched over to 2024.0.25-ci11 but still have the same problem.

                          An error occurred processing a GET request to http://<servername>/cargo/cargo-aggregate/bi/tf/bitflags: The remote server returned an error: (404) Not Found.

System.Net.WebException: The remote server returned an error: (404) Not Found.
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location ---
   at System.Net.WebRequest.GetResponseAsync()
   at Inedo.ProGet.GlobalExtensions.GetResponseAsync(HttpWebRequest request, CancellationToken cancellationToken) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E523661\Src\ProGetCoreEx\GlobalExtensions.cs:line 36
   at Inedo.ProGet.Feeds.Connector.GetDirectResponseAsync(String url, CancellationToken cancellationToken) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E523661\Src\ProGetCoreEx\Feeds\Connector.cs:line 525
   at Inedo.ProGet.Feeds.Cargo.CargoConnector.GetPackagesAsync(PackageNameId name, String version, CancellationToken cancellationToken)+MoveNext() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E523661\Src\ProGetCoreEx\Feeds\Cargo\CargoConnector.cs:line 96
   at Inedo.ProGet.Feeds.Cargo.CargoConnector.GetPackagesAsync(PackageNameId name, String version, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at Inedo.ProGet.Feeds.StandardCanonicalPackageFeed`5.AggregateWithConnectors(IEnumerable`1 localPackages, Func`3 getConnectorPackages, CancellationToken cancellationToken)+MoveNext() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E523661\Src\ProGetCoreEx\Feeds\StandardCanonicalPackageFeed.cs:line 395
   at Inedo.ProGet.Feeds.StandardCanonicalPackageFeed`5.AggregateWithConnectors(IEnumerable`1 localPackages, Func`3 getConnectorPackages, CancellationToken cancellationToken)+MoveNext() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E523661\Src\ProGetCoreEx\Feeds\StandardCanonicalPackageFeed.cs:line 395
   at Inedo.ProGet.Feeds.StandardCanonicalPackageFeed`5.AggregateWithConnectors(IEnumerable`1 localPackages, Func`3 getConnectorPackages, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at System.Collections.Generic.AsyncEnumerableHelpers.ToArrayWithLength[T](IAsyncEnumerable`1 source, CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncEnumerableHelpers.cs:line 49
   at System.Collections.Generic.AsyncEnumerableHelpers.ToArrayWithLength[T](IAsyncEnumerable`1 source, CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncEnumerableHelpers.cs:line 41
   at System.Linq.OrderedAsyncEnumerable`1.ToListAsync(CancellationToken cancellationToken)
   at Inedo.ProGet.WebApplication.FeedEndpoints.Cargo.CargoIndexHandler.ProcessRequestAsync(AhHttpContext context, WebApiContext apiContext, CargoFeed feed, String relativeUrl) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E523661\Src\ProGet.WebApplication\FeedEndpoints\Cargo\CargoIndexHandler.cs:line 68
   at Inedo.ProGet.WebApplication.FeedEndpoints.Cargo.CargoFeedHandler.ProcessRequestAsync(AhHttpContext context, WebApiContext apiContext, CargoFeed feed, String relativeUrl) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E523661\Src\ProGet.WebApplication\FeedEndpoints\Cargo\CargoFeedHandler.cs:line 107
   at Inedo.ProGet.WebApplication.FeedEndpoints.CanonicalFeedHandler`1.Inedo.Web.IHttpAsyncHandler.ProcessRequestAsync(AhHttpContext context) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E523661\Src\ProGet.WebApplication\FeedEndpoints\CanonicalFeedHandler.cs:line 85

::Web Error on 02/07/2025 10:07:23::
                          1 Reply Last reply Reply Quote 0
                          • K Offline
                            kc_2466
                            last edited by

                            To be fair, i think i might have a bigger problem - i may have corrupted my database somehow.
                            If I go to the Feeds, and then click 'connectors', i get I get a 500 error "Invalid Feed type".

                            An error occurred in the web application: Invalid feed type.

URL: http://<servername>/connectors
Referrer: http://<servername>/feeds
User: <username>
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36
Stack trace:    at Inedo.ProGet.Feeds.Connector.GetConnector(Connectors dbConnector) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E523495\Src\ProGetCoreEx\Feeds\Connector.cs:line 234
   at Inedo.ProGet.WebApplication.Pages.Connectors.ListConnectorsPage.CreateChildControls() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E523495\Src\ProGet.WebApplication\Pages\Connectors\ListConnectorsPage.cs:line 38
   at Inedo.ProGet.WebApplication.Pages.ProGetSimplePage.InitializeAsync() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E523495\Src\ProGet.WebApplication\Pages\ProGetSimplePage.cs:line 56
   at Inedo.Web.PageFree.SimplePageBase.ExecutePageLifeCycleAsync()
   at Inedo.Web.PageFree.SimplePageBase.ProcessRequestAsync(AhHttpContext context)
   at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context)

::Web Error on 02/07/2025 15:52:30::

                            I've tried deleting all the Cargo feeds to try and fix the issue, and then re-created them, but that's not helped.

                            dean-houstonD 1 Reply Last reply Reply Quote 0
                            • dean-houstonD Offline
                              dean-houston inedo-engineer @kc_2466
                              last edited by

                              @kc_2466 the "invalid feed type" will come up if you have a connector or feed that was created in a newer version of ProGet that wasn't available in an older version of ProGet... and you downgraded to the older version.

                              It looks like it's a connector, based on the URL. Easiest way to fix is to just upgrade, delete, downgrade

                              1 Reply Last reply Reply Quote 0
                              • gdivisG Offline
                                gdivis inedo-engineer
                                last edited by

                                Looks like I had misunderstood the exact problem. One related issue was already fixed in 2024.25, but the aggregation issue with one connector having a package and another not having it was not. I have confirmed that both will be fixed in today's release of 2024.26, along with that Invalid Feed Type error.

                                Sorry for the confusion!

                                1 Reply Last reply Reply Quote 0
                                • K Offline
                                  kc_2466 @kc_2466
                                  last edited by

                                  I've now found a cleaner way of adding this entry to the host file...

                                  If using docker run, add the argument:

                                  --add-host=<host_server_name>=127.0.0.1

                                  if using a docker compose file, add an entry:

                                  extra_hosts:
    - "<host_server_name>: 127.0.0.1"
                                  1 Reply Last reply Reply Quote 0

                                  Hello! It looks like you're interested in this conversation, but you don't have an account yet.

                                  Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.

                                  With your input, this post could be even better 💗

                                  Register Login
                                  • 1 / 1
                                  • First post
                                    Last post
                                  Inedo Website Home • Support Home • Code of Conduct • Forums Guide • Documentation