Hello, we haven't seen this before, so it's a bit strange to diagnose.
Can you share exactly what version of ProGet you're using, as well as some screenshots showing the step-by-step? There are some subtle ways different things are displayed, and that might clue us in where to look next.
cheers,
Alana
@nuno-guerreiro-rosa_9280 we have definitely tested similar configurations, and of course our customers have such usage all the time; there haven't been problems like this, and moving to SQL Server has significantly improved performance across the user base (I'm afraid Postgres is not supported)
Do note when you have connectors configured in ProGet, then almost each request to ProGet will often yield other network requests to those connectors. When NuGet builds a dependency tree with 100+ packages, it makes a tremendous amount of requests, often asking "what's the latest version of this package", and the like.
But anyways, it still should be okay. At this point, I'd recommend you to just try setting up a basic virtual machine at like, AWS LightSail or something, and see what you can reproduce.
I see, thanks! So, this is to help a user set-up Visual Studio for the first time, who hasn't done it then?
Well, some feeds already have a "configuration help" button; NuGet does not, and maybe it's not so obvious 
We added this tip in ProGet 5.3, but it only shows up when you're configuring a feed, and is intended to "train" the ProGet administrator to where the API Endpoint is (that's why it gives instructions on how to look for it).
This TODO is really obvious, and you can't miss it.
So maybe we can some of the language from the "TODO" tip (like the NPM one), and the make a more obvious "configuration help" button?
@vadim-k_6062 good to know that fixed it! We will add a message to the InedoHub then to help users if they come up with this -- https://inedo.myjetbrains.com/youtrack/issue/DH-42
As I mentioned before, we haven't seen these sorts of issues, but NuGet is a very "chatty" protocol, so a lot of requests are to be expected.
If you can provide some kind of guidance on how to reproduce things, we can certainly consider trying to reproduce it --- but right off the bat, unless you made a mistypo about "1GB of NuGet packages" in a restore that's a kind of red-flag to me.
The "chatty" protocol was never designed for that sort of traffic (tens of megabytes in a restore, maybe), so you'll need to do some networking tweaks (QoS?) to make it so the network stack doesn't get overloaded (which is what sounds like is happening).
Hello;
This error indicates a problem communicating with the domain controller. There are a handful of reasons this error can happen, here are a few:
Thus, if you wait and just reboot the server, it just might go away.; otherwise, it might invovle inspecting some of the traffic between the servers. Even if you use our exact code, LDAP just returns the same error, unfortunately.
Let us know what you find / try!
Hello;
Unfortunately, we've had only one other user report this issue, and we didn't hear how they solved it.
Basically, this is failing very early on during the installation process, during the "package extraction" process.
This would most likely be caused by only one of two things:
It might also be related to temporary file locking, so try rebooting to see if it helps.
Otherwise, check what could be preventing those package files to be extracted; it's typically the quarantine, so check the log files for that.
Please let us know what you find!
Interesting, that sounds like it might be helpful. Normally we'd just add it, but in this case... for users with integrated windows auth... maybe not so helpful. or if they use API keys, vs passwords, etc.?
In our 5.3 planning documents, I saw some kind of feature called "custom usage text" or something. It didn't make it, in part b/c no one requested it and the only use case we could think of was Universal Packages, where using upack doesn't make sense a lot of times.
Anyways, it seemed tough to market/explain, but the idea was you would be able to edit/add the usage text on the feeds. I guess, this is the first time I heard of a suggestion outside of universal packages, so maybe we can revive the feature idea :)
Got any other usecases you can think of? That'd help us go along way with adding the feature.
@viceice said in docker login failed via https reverse proxy:
So my assumption would be, that proget will respect the X-Forward* headers as already doing in other feeds.
You're right, when writing the Basic realm="<url>" header, ProGet does not look at the X-Forward* headers. I'm not sure if we should? Would that be a security problem? It kind of seems like it might? Or maybe not? I didn't find anything about the topic discussed on the 'net after a quick search...
How about just configuring your reverse proxy to just rewrite that realm in the auth header to go from http to https? I suspect, thatw ould help.
If you can do that, and then we can update the docs on how to do it, other Free users would be very much appreciative :)
And eventually we will fix this in the software, via PG-1849 - should be a simple fix, cheers!
Hi @viceice
The http://proget-test.kriese.eu/ being returned as the Realm because it's set as the BaseUrl.
If you remove that, then it will work; however, it would cause your instance of ProGet Free to report licensing errors due to your reverse proxy configuration.
Ultimately this is a reverse proxy issue, and we might be able to have the Relm respect the X-Forward* headers.... however, I'm not sure if that's appropriate / okay to do?
It might be a security problem? Can you find any documentation or discussion on this topic?
The code change is easy, but we want to confirm it's okay to make before considering it further.
Cheers,
Alana
I don't know a ton about Maven, but other users have reported that they've used a process like this for a disk-based repository.
There will be errors, particularly if you have invalid POM files or your directory structure doesn't match the required MAVEN convention, so inspect those case-by-case to determine if it matters (like an bad artifact from 5 years ago can probably be ignored).
That's just what I heard from customers, so if you have more details on how you do it, we'd love to hear! THanks much
Hello;
That was a mistake/typo in the docs, which i've since corrected;
https://github.com/Inedo/inedo-docs/commit/cd7091e8eaf37939949d0681f137a78d579acbc6
The correct url is https://«proget-server»/«feed-name»/«packageName»/«versionNumber[optional]»
But note, that's only for NuGet package. You can easily find the download url for any package from the UI, by looking at the Download button on the package page.
Cheers,
Alana
Hello, please review License Key Activation Docs, especially the note at top talking about Automatic License Activation Not Working in older versions.
Also... please upgrade. ProGet 5.3 is great!! 
Hello; this is definitely quite strange.
We haven't had any problems in our test labs, using significantly less-powerful hosts and significantly more traffic. Other users aren't reporting this problem on any platform, so I'm inclined to say it's configuration-related, but what configuration?
How is ProGet configured? If it's just a single feed and a single connector to that feed, then it's not your Proget configuration.
In any case, I'm certain it's not the database itself, but it's related to the network stack related. SQL Server uses network connections, and the "connection timeout" happens when the network stack gets overwhelmed. This can happen when a TON of connections are open, but not closed.
One thing we've seen is that certain network-based reporting tools (monitoring/logging) end up causing problems. They try to send a error over the network that the stack is overloaded, which then gets queued up, and continues to overload the stack. Eventually it calms down.
We've also seen bad hardware cause this. One time, it was even a bad wireless access point in. No idea how that happened, but something to do with routing and packets.
So perhaps try a new server, like make one at LightSail , totally fresh. If you can find a way to reproduce it, then it'll be good, because we can at least investigate it further then.
This isn't currently supported, but something we can consider adding if you're open to using our Feature Request Process.
One thing that would go a long way in helping us understand the usecase (and way to explain it to users) is suggesting where precisely we should document this, and how.
https://docs.inedo.com/docs/proget/reference/api/asset-directories-api (source)
There isn't a verification process per se; there is a re-index process that will delete orphaned packages, scan for symbols, rebuild the hash codes, etc. You can access it from Manage Feed > Storage & Retention page.
This might help, but you could also do a drop-path import as well.
No problem, ask away :)
You could. Not sure what the use case would be...
But, the URLs for NuGet package versions are quite predictable, and you can discover them from the Download button in the UI. For example, the download URL for InedoLib v950.0.7 just looks like https://proget.inedo.com/nuget/NuGetLibraries/package/InedoLib/950.0.7
The NuGet API does not support Bearer authentication. You can specify an API key when publishing packages (X-NUGET-APIKEY header), and it might work when downloading packages? Haven't tested, and no one asked before. But you could also specify api:<apikey> as the basic auth credentials as well, so I guess that's really easy too.
@atripp said in Creating PowerShell repository, protecting pull/download by API key:
this credential can be the name/password of a user inside of ProGet
Or a user that's configured in your Active Directory, assuming you have enabled that integration.
Hello, for sure!
It's pretty easy; just don't give the Anonymous user any access to your feeds, and then authentication will always be required, either when browsing the ProGet application or using the API (such as Install-Module).
When you use the Register-PSRepository command, you can the Credential option to specify a credential.
This credential can be the name/password of a user inside of ProGet (let's say, Admin:Admin), or it can be username of api with a password of an api key you've configured (so, api:my-secret-key).
Hello,
We aren't getting any other reports of this problem, and it's very unusual to get a 500 error that isn't logged. So it's really hard to guess what the problem could be.
Can you enable the "Feed Error Logging" to increase the logging? https://docs.inedo.com/docs/proget/installation/diagnostic-center
Alternatively, you set-up a brand new ProGet instance, then create a Docker feed, and try pushing it?
Thanks.
Alana
How much space are you seeing ManualExecutions taking? About how many rows? It should be auto-pruning those.
In any case you should be able to just DELETE [Executions] WHERE [Execution_Id] = ??, and it will cascade the deletes as needed, to the other tables (ScopedExecutionLogEntries being the biggest). The manual execution data is not so useful in the mid-term, it's mostly helpful or short-term debugging.
Hello;
That's a lot of data; most of the data will be taken up with the execution logs. You can use the sp_spaceused to verify it, checking the ScopedExecutionLogEntries table.
Usually you would want to use retention policies, but if SQL Server is blocking you from using the database, then you'll have to manually trim the logs, at least to get the software running again.
Let me share some SQL Code from that might help you do this.
First, find builds executions you don't want anymore; the code in the RetentionPolicies_GetBuildsWithLogsToPurge stored procedure might help, here is a way to find some builds you don't want.
SELECT B.*
FROM [Builds_Extended] B
INNER JOIN [Applications] A
ON A.[Application_Id] = B.[Application_Id]
WHERE A.[ApplicationGroup_Id] = @ApplicationGroup_Id
AND (@Pipeline_Name IS NULL OR B.[Pipeline_Name] = @Pipeline_Name)
AND (@DeployedReleasesOnly_Indicator = 'N' OR [ReleaseStatus_Name] = 'Deployed')
AND (@RejectedOnly_Indicator = 'N' OR [BuildStatus_Name] = 'Rejected')
AND (@AlwaysRetainAfter_Date IS NULL OR B.[CreatedOn_Date] < @AlwaysRetainAfter_Date)
AND EXISTS(
SELECT TOP 1 SEL.[Scope_Sequence]
FROM [PipelineStageTargetExecutions_Extended] BEX
INNER JOIN [ScopedExecutionLogs] SEL
ON SEL.[Execution_Id] = BEX.[Execution_Id]
WHERE B.[Application_Id] = BEX.[Application_Id]
AND B.[Release_Number] = BEX.[Release_Number]
AND B.[Build_Number] = BEX.[Build_Number]
)
ORDER BY [CreatedOn_Date]
Then, you can do EXEC Builds_PurgeExecutionLogs @Build_Id = ??, where ?? is the ID of the build.
Thanks for the very detailed test case, we will be reviewing it quite soon and get a fix ASAP!
Hello, this has finally been scheduled for 5.3.13, which is shipping tomorrow. It addresses only this specific test case, so please let us know if you're spotting other errors.
@nuno-guerreiro-rosa_9280 said in cannot login on authenticated feeds after upgrade:
@atripp Hello atripp. Just to be clear, this occurs not only authenticating with docker but also with private NuGet authenticated feeds. I have just switched to progetmono and both private docker and NuGet feeds are authenticating correctly so the issue is related to the .NET Core release.
If you can share some more information about the authentication in .NET Core, that would be appreciated. To date we can't reproduce any authentication problems, with either Docker or NuGet.
@nuno-guerreiro-rosa_9280 said in cannot login on authenticated feeds after upgrade:
error occurred processing a GET request to http://myproget.blabla.com/nuget/feed-nuget/v3/flatcontainer/microsoft.extensions.dependencyinjection/index.json: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
The timeout error can happen even with a Single client, when the server is underpowered. Essentially, the client is flooding the server with a ton of network connections. Increasing the server power should help.
Hello; can you try switching to the ProGetMono container to see if it makes any difference?
If it does, can you share details of how your NuGet and Docker feeds are authenticated? And provide the specific errors you're seeing?
We check for local requests using HttpRequest.IsLocal, which basically just looks for 127.0.0.1. If it's not local, then a license violation is recorded.
If the server doesn't resolve proget.xxxx.com as 127.0.0.1, then configuring BaseUrl will be a problem, especially with NuGet. The NuGet API requires absolute urls (issue #1), and many API responses are just URLS that the client (in this case, the connector) must follow to get the "real answer" (issue #2). So this will trigger license violations.
If you need to specify a BaseUrl (you really shouldn't have to), then you'll need add a /etc/hosts entry for proget.xxxx.com 127.0.0.1
For this, you could just disable the connector, and use a Promoted-package workflow, where you manually promote approved packages from one feed (Unapproved) to another feed (Approved).
When the request is very simple, like "give me this exact version of this exact package", then it's not forwarded. But typically the request is, "what's the latest version of this package". Of course, that must be forwarded and aggregated against all connectors.
Why the client makes such a request in some cases is a mystery, but that's why they rewrote everything from scratch into a new, v3 api.
Not at present, but it's definitely on our roadmap, but there's not a ton of demand for it so we haven't prioritized it.
The primary usecase seems to be having a Otter or BuildMaster server in the cloud that in-house servers connect to. Is that what you were thinking?
ProGet will use the v3 (JSON-LD) API when possible, but not all v2 (ODATA) queries can be "translated" to the v3 (JSON-LD) API, and thus they need to be forwarded to the v2 (ODATA) end-point.
ODATA is a general-purpose querying API (like SQL but for the web), so clients do a lot with it (sorting, etc.). The v3 (JSON-LD) API is much more limited. So I recommend you to update the clients.
Ah ha, thanks! I updated the docs to include CommonCachePath instructions.
Hello; can you share some more information about how you've arrived at this error?
What steps you performed, and specific error message? Thanks!
Hello;
I'm not able to reproduce this on the latest ProGet, and I'm thinking the problem may have been the package file was on disk from the previous version.
Here's what I did:
/nuget/nuuget-public/package/ProxyKit/2.3.3+build.0ProxyKit.2.3.3.nupkgCheers,
Alana
Hello; this error seems to be indicating that your FTP server is providing dates in an unexpected / non-standardized format.
https://github.com/Inedo/inedox-ftp/blob/master/FTP/InedoExtension/FtpWebRequestExtensions.cs#L70
It should be easy to fix the extension, if we can see the data...
Could you provide us with the dates that you're seeing?
Thanks.
Alana
Hello;
I haven't tested this on ProGet v5.0.10 (note: it's 2.5years old), but I can see that there's a +build.0 in the version. There's been a number of changes since that version of ProGet to better address the quirks surrounding build metadata. So please upgrade to the latest ProGet.
Best,
Alana
Hello;
Can you share the errors you're seeing in the Admin > Diagnostic Center? Those will be logged w/ the 500 errors.
Thank!
@nicholas-boltralik_3634 very sorry on the slow response, we were internally discussing and tracking this, but I guess I didn't update you here.
We will target to make a software change in PG-1814, and hopefully get this working in the next or following maintenance release.
Thanks for updating and letting us know the problem was related to a connection string!
@jyip_5228 FYI, as part of ProGet 5.3.10 release, we shipped the ProGetCore container image as well.
You can follow the normal steps in the Linux and Docker Installation Guide to install/upgrade, but just use progetcore for the container instead of proget.
Aside from support for the Lucene-based Maven feed indexing (in progress), it seems to be feature complete. And of course, if there are problems, you can switch back to proget:5.3.10 or downgrade as needed (no database schema changes).
For example, docker pull proget.inedo.com/productimages/inedo/progetcore:5.3.10
@nuno-guerreiro-rosa_9280 just to let you know, as part of ProGet 5.3.10 release, we shipped the ProGetCore container image as well.
You can follow the normal steps in the Linux and Docker Installation Guide to install/upgrade, but just use progetcore for the container instead of proget.
Aside from support for the Lucene-based Maven feed indexing (in progress), it seems to be feature complete. And of course, if there are problems, you can switch back to proget:5.3.10 or downgrade as needed (no database schema changes).
For example, docker pull proget.inedo.com/productimages/inedo/progetcore:5.3.10
hi @scroak_6473 , just to let you know, as part of ProGet 5.3.10 release, we shipped the ProGetCore container image.
You can follow the normal steps in the Linux and Docker Installation Guide to install/upgrade, but just use progetcore for the container instead of proget.
Aside from support for the Lucene-based Maven feed indexing (in progress), it seems to be feature complete. And of course, if there are problems, you can switch back to proget:5.3.10 or downgrade as needed (no database schema changes).
For example, docker pull proget.inedo.com/productimages/inedo/progetcore:5.3.10
Getting LDAP/LDAPS to work on Linux was a whole different problem to solve; the three major libraries (DotNetCore, Mono, Novell) all had separate and strange bugs. We'll be blogging about this, but for now, it might be a step in the right direction for addressing the problems you're seeing, at the very least.
Hello, lots of questions to address, but since you prefer to do a regular online upgrade, let's just suggestion that :)
The [config] button in Inedo Hub lets you configure a package source; this shoudl be https://proget.inedo.com/upack/Products. If you whitelist proget.inedo.com then it should be fine.
@ludovic_2596 this sounds familiar, from an early version of 5.3, but I can't find a report for it. It seems to work fine in the latest version for me.. can you try the upgrade?
The NuGet API does require absolute URLS to be returned, hence this behavior. And since the IP address changes (we check for local requests using HttpRequest.IsLocal, which basically just looks for 127.0.0.1), the license violation triggers.
Instead of using the Web.BaseUrl property, you could configure your proxy to use the X-Forwarded-* headers. I attached the code for how the URL is constructed in ProGet, and how these headers work.
var baseUrl = ProGetConfig.Web.BaseUrl;
if (string.IsNullOrEmpty(baseUrl))
{
var request = HttpContextThatWorksOnLinux.Current?.Request;
if (request != null)
{
var requestUrl = request.Url;
var forwardedHost = request.Headers["X-Forwarded-Host"];
int? forwardedPort = AH.ParseInt(request.Headers["X-Forwarded-Port"]);
var forwardedProtocol = request.Headers["X-Forwarded-Proto"];
var host = AH.CoalesceString(forwardedHost, requestUrl.Host);
var protocol = AH.CoalesceString(forwardedProtocol, requestUrl.Scheme);
int port = forwardedPort ?? requestUrl.Port;
var uri = new UriBuilder(protocol, host, port);
if (uri.Uri.IsDefaultPort)
uri.Port = -1;
baseUrl = uri.ToString();
if (!baseUrl.EndsWith("/"))
baseUrl += "/";
}
else
{
baseUrl = "/";
}
}
else
{
baseUrl = baseUrl.TrimEnd('/') + "/";
}hi @philippe-camelio_3885 , this will be fixed in OT-380, in the upcmoing Otter (2.2.23), which is planned to release on this Friday.
@scroak_6473 got it, thanks! Okay, I've requested to engineering about adding it to the upcoming ProGetCore container, and we'll update this with more details as we have i
hi @Adam1, I looked into this further, and this is by design, but it should really be clarified in the UI a little bit better (I updated the docs).
A server can exist in multiple environments, but it's not recommended.
Basically, when a server is in multiple environments, then there can be no single environment in context. This means that the variable function $EnvironmentName will return empty, and variables cannot be resolved against those environments.
This is unlike a role (which is set when executing a configuration plan, or explicitly set with for role X). So, in this case, I recommend you to use multiple roles.
hi @Adam1; thank you for the detailed reproduction instructions.
You are correct, this is indeed a bug, and it will be fixed in the next maintenance release (scheduled for Friday) as OT-381. If you'd like a pre-release, we can easily share one to you as well!
Hmmm, I'm not sure if it's possible... I don't know if PowerShell even allows multiple versions of the same module/resource to be installed on the same server?