There isn't a verification process per se; there is a re-index process that will delete orphaned packages, scan for symbols, rebuild the hash codes, etc. You can access it from Manage Feed > Storage & Retention page.
This might help, but you could also do a drop-path import as well.
No problem, ask away :)
You could. Not sure what the use case would be...
But, the URLs for NuGet package versions are quite predictable, and you can discover them from the Download button in the UI. For example, the download URL for InedoLib v950.0.7 just looks like https://proget.inedo.com/nuget/NuGetLibraries/package/InedoLib/950.0.7
The NuGet API does not support Bearer authentication. You can specify an API key when publishing packages (X-NUGET-APIKEY header), and it might work when downloading packages? Haven't tested, and no one asked before. But you could also specify api:<apikey> as the basic auth credentials as well, so I guess that's really easy too.
@atripp said in Creating PowerShell repository, protecting pull/download by API key:
this credential can be the name/password of a user inside of ProGet
Or a user that's configured in your Active Directory, assuming you have enabled that integration.
Hello, for sure!
It's pretty easy; just don't give the Anonymous user any access to your feeds, and then authentication will always be required, either when browsing the ProGet application or using the API (such as Install-Module).
When you use the Register-PSRepository command, you can the Credential option to specify a credential.
This credential can be the name/password of a user inside of ProGet (let's say, Admin:Admin), or it can be username of api with a password of an api key you've configured (so, api:my-secret-key).
Hello,
We aren't getting any other reports of this problem, and it's very unusual to get a 500 error that isn't logged. So it's really hard to guess what the problem could be.
Can you enable the "Feed Error Logging" to increase the logging? https://docs.inedo.com/docs/proget/installation/diagnostic-center
Alternatively, you set-up a brand new ProGet instance, then create a Docker feed, and try pushing it?
Thanks.
Alana
How much space are you seeing ManualExecutions taking? About how many rows? It should be auto-pruning those.
In any case you should be able to just DELETE [Executions] WHERE [Execution_Id] = ??, and it will cascade the deletes as needed, to the other tables (ScopedExecutionLogEntries being the biggest). The manual execution data is not so useful in the mid-term, it's mostly helpful or short-term debugging.
Hello;
That's a lot of data; most of the data will be taken up with the execution logs. You can use the sp_spaceused to verify it, checking the ScopedExecutionLogEntries table.
Usually you would want to use retention policies, but if SQL Server is blocking you from using the database, then you'll have to manually trim the logs, at least to get the software running again.
Let me share some SQL Code from that might help you do this.
First, find builds executions you don't want anymore; the code in the RetentionPolicies_GetBuildsWithLogsToPurge stored procedure might help, here is a way to find some builds you don't want.
SELECT B.*
FROM [Builds_Extended] B
INNER JOIN [Applications] A
ON A.[Application_Id] = B.[Application_Id]
WHERE A.[ApplicationGroup_Id] = @ApplicationGroup_Id
AND (@Pipeline_Name IS NULL OR B.[Pipeline_Name] = @Pipeline_Name)
AND (@DeployedReleasesOnly_Indicator = 'N' OR [ReleaseStatus_Name] = 'Deployed')
AND (@RejectedOnly_Indicator = 'N' OR [BuildStatus_Name] = 'Rejected')
AND (@AlwaysRetainAfter_Date IS NULL OR B.[CreatedOn_Date] < @AlwaysRetainAfter_Date)
AND EXISTS(
SELECT TOP 1 SEL.[Scope_Sequence]
FROM [PipelineStageTargetExecutions_Extended] BEX
INNER JOIN [ScopedExecutionLogs] SEL
ON SEL.[Execution_Id] = BEX.[Execution_Id]
WHERE B.[Application_Id] = BEX.[Application_Id]
AND B.[Release_Number] = BEX.[Release_Number]
AND B.[Build_Number] = BEX.[Build_Number]
)
ORDER BY [CreatedOn_Date]
Then, you can do EXEC Builds_PurgeExecutionLogs @Build_Id = ??, where ?? is the ID of the build.
Thanks for the very detailed test case, we will be reviewing it quite soon and get a fix ASAP!
Hello, this has finally been scheduled for 5.3.13, which is shipping tomorrow. It addresses only this specific test case, so please let us know if you're spotting other errors.
@nuno-guerreiro-rosa_9280 said in cannot login on authenticated feeds after upgrade:
@atripp Hello atripp. Just to be clear, this occurs not only authenticating with docker but also with private NuGet authenticated feeds. I have just switched to progetmono and both private docker and NuGet feeds are authenticating correctly so the issue is related to the .NET Core release.
If you can share some more information about the authentication in .NET Core, that would be appreciated. To date we can't reproduce any authentication problems, with either Docker or NuGet.
@nuno-guerreiro-rosa_9280 said in cannot login on authenticated feeds after upgrade:
error occurred processing a GET request to http://myproget.blabla.com/nuget/feed-nuget/v3/flatcontainer/microsoft.extensions.dependencyinjection/index.json: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
The timeout error can happen even with a Single client, when the server is underpowered. Essentially, the client is flooding the server with a ton of network connections. Increasing the server power should help.
Hello; can you try switching to the ProGetMono container to see if it makes any difference?
If it does, can you share details of how your NuGet and Docker feeds are authenticated? And provide the specific errors you're seeing?
We check for local requests using HttpRequest.IsLocal, which basically just looks for 127.0.0.1. If it's not local, then a license violation is recorded.
If the server doesn't resolve proget.xxxx.com as 127.0.0.1, then configuring BaseUrl will be a problem, especially with NuGet. The NuGet API requires absolute urls (issue #1), and many API responses are just URLS that the client (in this case, the connector) must follow to get the "real answer" (issue #2). So this will trigger license violations.
If you need to specify a BaseUrl (you really shouldn't have to), then you'll need add a /etc/hosts entry for proget.xxxx.com 127.0.0.1
For this, you could just disable the connector, and use a Promoted-package workflow, where you manually promote approved packages from one feed (Unapproved) to another feed (Approved).
When the request is very simple, like "give me this exact version of this exact package", then it's not forwarded. But typically the request is, "what's the latest version of this package". Of course, that must be forwarded and aggregated against all connectors.
Why the client makes such a request in some cases is a mystery, but that's why they rewrote everything from scratch into a new, v3 api.
Not at present, but it's definitely on our roadmap, but there's not a ton of demand for it so we haven't prioritized it.
The primary usecase seems to be having a Otter or BuildMaster server in the cloud that in-house servers connect to. Is that what you were thinking?
ProGet will use the v3 (JSON-LD) API when possible, but not all v2 (ODATA) queries can be "translated" to the v3 (JSON-LD) API, and thus they need to be forwarded to the v2 (ODATA) end-point.
ODATA is a general-purpose querying API (like SQL but for the web), so clients do a lot with it (sorting, etc.). The v3 (JSON-LD) API is much more limited. So I recommend you to update the clients.
Ah ha, thanks! I updated the docs to include CommonCachePath instructions.
Hello; can you share some more information about how you've arrived at this error?
What steps you performed, and specific error message? Thanks!
Hello;
I'm not able to reproduce this on the latest ProGet, and I'm thinking the problem may have been the package file was on disk from the previous version.
Here's what I did:
/nuget/nuuget-public/package/ProxyKit/2.3.3+build.0ProxyKit.2.3.3.nupkgCheers,
Alana
Hello; this error seems to be indicating that your FTP server is providing dates in an unexpected / non-standardized format.
https://github.com/Inedo/inedox-ftp/blob/master/FTP/InedoExtension/FtpWebRequestExtensions.cs#L70
It should be easy to fix the extension, if we can see the data...
Could you provide us with the dates that you're seeing?
Thanks.
Alana
Hello;
I haven't tested this on ProGet v5.0.10 (note: it's 2.5years old), but I can see that there's a +build.0 in the version. There's been a number of changes since that version of ProGet to better address the quirks surrounding build metadata. So please upgrade to the latest ProGet.
Best,
Alana
Hello;
Can you share the errors you're seeing in the Admin > Diagnostic Center? Those will be logged w/ the 500 errors.
Thank!
@nicholas-boltralik_3634 very sorry on the slow response, we were internally discussing and tracking this, but I guess I didn't update you here.
We will target to make a software change in PG-1814, and hopefully get this working in the next or following maintenance release.
Thanks for updating and letting us know the problem was related to a connection string!
@jyip_5228 FYI, as part of ProGet 5.3.10 release, we shipped the ProGetCore container image as well.
You can follow the normal steps in the Linux and Docker Installation Guide to install/upgrade, but just use progetcore for the container instead of proget.
Aside from support for the Lucene-based Maven feed indexing (in progress), it seems to be feature complete. And of course, if there are problems, you can switch back to proget:5.3.10 or downgrade as needed (no database schema changes).
For example, docker pull proget.inedo.com/productimages/inedo/progetcore:5.3.10
@nuno-guerreiro-rosa_9280 just to let you know, as part of ProGet 5.3.10 release, we shipped the ProGetCore container image as well.
You can follow the normal steps in the Linux and Docker Installation Guide to install/upgrade, but just use progetcore for the container instead of proget.
Aside from support for the Lucene-based Maven feed indexing (in progress), it seems to be feature complete. And of course, if there are problems, you can switch back to proget:5.3.10 or downgrade as needed (no database schema changes).
For example, docker pull proget.inedo.com/productimages/inedo/progetcore:5.3.10
hi @scroak_6473 , just to let you know, as part of ProGet 5.3.10 release, we shipped the ProGetCore container image.
You can follow the normal steps in the Linux and Docker Installation Guide to install/upgrade, but just use progetcore for the container instead of proget.
Aside from support for the Lucene-based Maven feed indexing (in progress), it seems to be feature complete. And of course, if there are problems, you can switch back to proget:5.3.10 or downgrade as needed (no database schema changes).
For example, docker pull proget.inedo.com/productimages/inedo/progetcore:5.3.10
Getting LDAP/LDAPS to work on Linux was a whole different problem to solve; the three major libraries (DotNetCore, Mono, Novell) all had separate and strange bugs. We'll be blogging about this, but for now, it might be a step in the right direction for addressing the problems you're seeing, at the very least.
Hello, lots of questions to address, but since you prefer to do a regular online upgrade, let's just suggestion that :)
The [config] button in Inedo Hub lets you configure a package source; this shoudl be https://proget.inedo.com/upack/Products. If you whitelist proget.inedo.com then it should be fine.
@ludovic_2596 this sounds familiar, from an early version of 5.3, but I can't find a report for it. It seems to work fine in the latest version for me.. can you try the upgrade?
The NuGet API does require absolute URLS to be returned, hence this behavior. And since the IP address changes (we check for local requests using HttpRequest.IsLocal, which basically just looks for 127.0.0.1), the license violation triggers.
Instead of using the Web.BaseUrl property, you could configure your proxy to use the X-Forwarded-* headers. I attached the code for how the URL is constructed in ProGet, and how these headers work.
var baseUrl = ProGetConfig.Web.BaseUrl;
if (string.IsNullOrEmpty(baseUrl))
{
var request = HttpContextThatWorksOnLinux.Current?.Request;
if (request != null)
{
var requestUrl = request.Url;
var forwardedHost = request.Headers["X-Forwarded-Host"];
int? forwardedPort = AH.ParseInt(request.Headers["X-Forwarded-Port"]);
var forwardedProtocol = request.Headers["X-Forwarded-Proto"];
var host = AH.CoalesceString(forwardedHost, requestUrl.Host);
var protocol = AH.CoalesceString(forwardedProtocol, requestUrl.Scheme);
int port = forwardedPort ?? requestUrl.Port;
var uri = new UriBuilder(protocol, host, port);
if (uri.Uri.IsDefaultPort)
uri.Port = -1;
baseUrl = uri.ToString();
if (!baseUrl.EndsWith("/"))
baseUrl += "/";
}
else
{
baseUrl = "/";
}
}
else
{
baseUrl = baseUrl.TrimEnd('/') + "/";
}hi @philippe-camelio_3885 , this will be fixed in OT-380, in the upcmoing Otter (2.2.23), which is planned to release on this Friday.
@scroak_6473 got it, thanks! Okay, I've requested to engineering about adding it to the upcoming ProGetCore container, and we'll update this with more details as we have i
hi @Adam1, I looked into this further, and this is by design, but it should really be clarified in the UI a little bit better (I updated the docs).
A server can exist in multiple environments, but it's not recommended.
Basically, when a server is in multiple environments, then there can be no single environment in context. This means that the variable function $EnvironmentName will return empty, and variables cannot be resolved against those environments.
This is unlike a role (which is set when executing a configuration plan, or explicitly set with for role X). So, in this case, I recommend you to use multiple roles.
hi @Adam1; thank you for the detailed reproduction instructions.
You are correct, this is indeed a bug, and it will be fixed in the next maintenance release (scheduled for Friday) as OT-381. If you'd like a pre-release, we can easily share one to you as well!
Hmmm, I'm not sure if it's possible... I don't know if PowerShell even allows multiple versions of the same module/resource to be installed on the same server?
@nicholas-boltralik_3634 I investigated this further, and the behavior seems to have existed since 5.2 and earlier.
I'm not totally sure how works Maven, but I'm almost certain you're not supposed to create an artifact with a version that has -SNAPSHOT in it. Instead, the -SNAPSHOT seems to be intended for use inside of a <dependency> only.
if your project depends on a software component that is under active development, you can depend on a snapshot release, and Maven will periodically attempt to download the latest snapshot from a repository when you run a build. Similarly, if the next release of your system is going to have a version “1.8,” your project would have a “1.8-SNAPSHOT” version until it was formally released.
For example , the following dependency would always download the latest 1.8 development JAR of spring:
<dependency> <groupId>org.springframework</groupId> <artifactId>spring</artifactId> <version>1.8-SNAPSHOT”</version> </dependency>
My understanding of how the server logic works, if you request a version with -SNAPSHOT in it, then the latest version is returned. In other words, -SNAPSHOT is not a real version, it's just something the server uses t send back the latest version. And since there is no latest version, you get an error.
It seems to be a problem to allow a version called -SNAPSHOT to be uploaded, because then it's ambiguous. When you ask for -SNAPSHOT do you want the version named that, or the latest version, etc.
To be honest this is all confusing to me... I'm wondering, why do you create a -SNAPSHOT version?
Hello; good news, we can now fix this as part of SDK-64, which will ship in ProGetCore, a new container that will contain the .NET5 version of ProGet that's designed to replace the mono stack.
We're planning to ship a public technology preview by the end of next week (as part of 5.3.11), and the ProGetCore container will eventually replace ProGet container (which will become ProGetMono). This will happen well before November, but we don't know when.
PG-1804 will fix the caching problem you identified, however.
Hello; what product is this for... Otter?
Hi Nik,
I can confirm that I got the same error message when I created a snapshot POM file, and tried to download it:
<project>
<modelVersion>4.0.0</modelVersion>
<groupId>com.mycompany.app</groupId>
<artifactId>my-app</artifactId>
<version>2-SNAPSHOT</version>
</project>
However, this works:
<project>
<modelVersion>4.0.0</modelVersion>
<groupId>com.mycompany.app</groupId>
<artifactId>my-app</artifactId>
<version>1</version>
</project>
However, I found the code where this was happening, and it seems to have always been working this way? I don't really get how SNAPSHOT works, but there seems to be some special handling with SNAPSHOT that I need to research some more. I'll follow-up once I learn more
@jyip_5228 said in Nuget package not found in proget, but searchable in UI:
This issue appears to still exist occasionally, is there a way to increase logging to try and troubleshoot this?
Errors on the ProGet side are logged under Admin > Diagnostics. However, the error message is coming from the nuget.exe side, so we have no idea why it can't find the package. Clearly, it's available on ProGet... and if you can see the package in the UI, then will should show up in the API as well. No one else is reporting this either, which leads me to think it's outside of the ProGet software.
We don't have a mechanism in side of ProGet to capture all incoming/outgoing traffic, but could you run the cli tool through a proxy, and capture the results?
Regarding the caching, we were able to reproduce that -- it as PG-1804, which will cause remote packages to sometimes cache, even if you have the caching disabled
hi Marcus,
Since you're hooking ProGet to npmjs.org, then the search results of npmjs.org are returned.
This is the usual behavior of npmjs.org, and you can see how it works by going to the webste: https://www.npmjs.com/search?q=%40angula
I can't speak for how or why they designed things they way they did, but I can say that, in npm, the @-symbol denotes a package scope, and it's only sort-of-kind-of part of the package's name. It's messy, but suffice it to say, if a search string contains only a scope name, then npmjs.org seems to return all packages within that scope.
Otherwise, the search algorithm reverts to its default behavior, which I guess is searching names, descriptions, those sorts of things. Note that searching angular doesn't return packages in the @angular scope, either: https://www.npmjs.com/search?q=angular
Ah got it, thanks!
Seems like an easy line to add (and probably doesn't increase container size much), but since it's only used during intiial setup, is it easy to run apt-get install iputils-ping first, or does that require a special kind of network access? Or is this common in other product-based containers?
@scroak_6473 oh, I guess we already rewrote this for our .NET5 plans.
In the coming weeks, we'll shipping a new container, ProGetCore that is build on .NETCore (soon: .NET5), and won't use any of Mono. So, then, we'll have a chance of easily adding this!
Definitely, it will be in the next release.
We don't consider it a serious security vulnerability; unprivileged users can create new feeds, but they can't view or use them. Obviously it has potential for "vandalism" (we see already some test feeds created on our public instance, for example 
), so we will take care of it right away.
@scroak_6473 oh I see, the usual ping! Thanks.
How/when is this helpful? Is it something that is called from outside the container? My image is, "I need to diagnose my network configuration, so I SSH into my Docker container, but ping isn't there?"
Thanks @scroak_6473; that method would probably work if we were using the protocol-level libraries (i.e. System.DirectoryServices.Protocols) to connect, but we're working at one-level above that (i.e. DirectorySearcher).
These libraries use a protocol called ADSI, which basically a wrapper around LDAP, but with more security (either via SSL or something else, I forgot) and some Microsoft extensions. But ADSl can also connect to any LDAP server.
Hello;
Thanks for the bug report! I've logged this as PG-1801, and it will get shipped in the next maintenance release.
Cheers,
Alana
This is was addressed in PG-1795, which was released in 5.3.9 just recently!
hi Simon,
Unfortunately the code solution I mentioned isn't feasible. Basically, to use that method, it would require us to rewrite our library to work at the "protocol level" (TCP/IP) instead of the "directory level" (users/groups/objects); so it's not so trivial...
Could you install the certificate to the Docker container to see if it works?
In this case, please ensure that the ProGet service is running and has permission to read and delete from that folder (Admin > Service). You can manually trigger the drop path monitor from that page, and see what he's doing behind the scenes.
Great!! I've logged this as PG-1798, and it's planned for 5.3.10 (Aug 28), but may get delayed depending on other priories.