Posts made by atripp

Unfortunately, when symbol serving doesn't work, it can be a pain to diagnose...

Can you "start from scratch", documenting your steps along the way, so that I can try to reproduce exactly what you're doing?

Start by making a very simple, hello world sort of library (maybe one class, with some basic code you can easily step/through and debug).

After that, then create some brand new feed (it sounds like you want two feeds? a symbols and a package feed?), then configure the new feed in Visual Studio. Then follow the other steps, like seeing if you can find the symbols in Visual Studios, etc.

If you can share the exact steps you did, and the package you upload, then I can reproduce the error you're seeing by following steps using the package.

Hello; please restart the ProGet service. This will cancel all executions upon restart.

The legacy pdb format bundles sources in the NuGet package (in which case, it comes from ProGet), whereas pdb file uses source link.

So it depends on the format you make the pdb. It sounds like it's the new format, however.

Sorry, wrong software - please see our friends at otter.ai for transcription. We do server automation.

Cheers

In fact, we're working on this project now :)

It's going to be available in Otter 3.0, which will be released in the coming weeks. And even better, you can run Otter 3.0 on Linux/containers :)

NuGet/VS will aggressively cache packages, so if you're not seeing the "Symbol Load Information" it probably means you have an old, cached package or DLL. It's hard to get-around this, so it's best to really just create new package versions so the caching isn't happen

Symbol serving can be pretty tricky to get working, so make sure to follow the troubleshooting steps in the documentation.

You'll see SymbolLoad information for your DLL, which should look something like this:

If you're not seeing a call to your ProGet server, then Visual STudio isn't configured as needed.

If ProGet is returning a 404, then the symbol isn't indexed. You you see the status of symbols that ProGet found on the "Symbols" tab of your package.

@philippe-camelio_3885 FYI, we will add a checkbox for auto-purging soon, BM-3655

Hi @sbolisetty_3792 , just to let you know, this will be fixed in the next maintence release of BuildMaster (BM-3654). Basically, if a Server has a Single environment, then that environment will be used for credential resolution.

HOWEVER, note that this won't work for your dev-cap server because it's in two environments. So in that case, you couldn't use an environment-specific credential.

Hello, the configuration plans can do a ton of great things, but they're a bit confusing -- and a big thing we want to be improving in the next year, with both software and documentation changes.

But I'll explain a couple things you might already know, for the sake of helping someone who might read this in future. Using your first script (without the execution policy):

• The OtterScript is executed twice in a row; first in "Collect" mode then "Ensure" mode if it
• Ensure-File always executes in Collect (and it records whether the file exists), and it may execute in the "Ensure" run (where it would create/overwrite) if it reported drift
• Start-Service never executes in Collect, but may run in "Ensure" mode... but may executes if another operation in the block reported drift (i.e. if Ensure-File reported drift)
• Post-Http never executes in Collect mode, and it never executes or Ensure modes, because it's the only statement in a block

The with executionPolicy=always policy changes this, and it's the intended use of this execution directive. But... it's an editor bug, so we'll fix it.

So... all that said... I don't think I'd recommend doing an error handling in a Configuration plan like this; it feels more appropriate for an Orchestration plan that you run for a purpose, to like provision or set-up a server.

Otter will perform a routine configuration scan at least every hour, so there's a good chance this will just end up sending the same error message over and over:

1. Drift is a detected
2. Configuration FAILS to change
3. Error notice is sent

This isn't all that helpful, and is more of an indication of an outage more than anything else. And this isn't a great way to detect an outage. Instead, you can check the status of the server; if there is a failure during a Configuration execution, the server status becomes Error, and it can then be investigated about the details.

Hi there, I'm afraid the documentation is incorrect

There is no such way to do that in the current version, so for the time being all users will have full access.

However, the next version of Otter is in the works, and it's going to be great; not only will it run as as a container on Linux, but we're making a lot of UI improvements so that you can do things like run PowerShell / Shell directly as a job.

We'll be bringing over things from BuildMaster as well, like the combined script page, Job Template Variables - and also the pseduo-users like Anonymous, Authenticated, and Everyone.

You will then be able to add/restrict users just like BuildMaster, though all users will still be full admins.

Hello, we haven't seen this before, so it's a bit strange to diagnose.

Can you share exactly what version of ProGet you're using, as well as some screenshots showing the step-by-step? There are some subtle ways different things are displayed, and that might clue us in where to look next.

cheers,
Alana

@nuno-guerreiro-rosa_9280 we have definitely tested similar configurations, and of course our customers have such usage all the time; there haven't been problems like this, and moving to SQL Server has significantly improved performance across the user base (I'm afraid Postgres is not supported)

Do note when you have connectors configured in ProGet, then almost each request to ProGet will often yield other network requests to those connectors. When NuGet builds a dependency tree with 100+ packages, it makes a tremendous amount of requests, often asking "what's the latest version of this package", and the like.

But anyways, it still should be okay. At this point, I'd recommend you to just try setting up a basic virtual machine at like, AWS LightSail or something, and see what you can reproduce.

I see, thanks! So, this is to help a user set-up Visual Studio for the first time, who hasn't done it then?

Well, some feeds already have a "configuration help" button; NuGet does not, and maybe it's not so obvious

We added this tip in ProGet 5.3, but it only shows up when you're configuring a feed, and is intended to "train" the ProGet administrator to where the API Endpoint is (that's why it gives instructions on how to look for it).

This TODO is really obvious, and you can't miss it.

So maybe we can some of the language from the "TODO" tip (like the NPM one), and the make a more obvious "configuration help" button?

@vadim-k_6062 good to know that fixed it! We will add a message to the InedoHub then to help users if they come up with this -- https://inedo.myjetbrains.com/youtrack/issue/DH-42

As I mentioned before, we haven't seen these sorts of issues, but NuGet is a very "chatty" protocol, so a lot of requests are to be expected.

If you can provide some kind of guidance on how to reproduce things, we can certainly consider trying to reproduce it --- but right off the bat, unless you made a mistypo about "1GB of NuGet packages" in a restore that's a kind of red-flag to me.

The "chatty" protocol was never designed for that sort of traffic (tens of megabytes in a restore, maybe), so you'll need to do some networking tweaks (QoS?) to make it so the network stack doesn't get overloaded (which is what sounds like is happening).

Hello;

This error indicates a problem communicating with the domain controller. There are a handful of reasons this error can happen, here are a few:

• To many Domain/LDAP queries, were there any new applications deployed that may be overly chatty with AD?
• A domain controller is offline, but still in the DNS or a change to the IP address. Have you deprecated or changed the IP address on any of your domain controllers recently?
• Overall network communication errors.
• Server requests are sent to a proxy prior to connecting to the server.
• Incorrect certificate

Thus, if you wait and just reboot the server, it just might go away.; otherwise, it might invovle inspecting some of the traffic between the servers. Even if you use our exact code, LDAP just returns the same error, unfortunately.

Let us know what you find / try!

Hello;

Unfortunately, we've had only one other user report this issue, and we didn't hear how they solved it.

Basically, this is failing very early on during the installation process, during the "package extraction" process.

This would most likely be caused by only one of two things:

1. disk is full; the packages are extracted to a temporary directory, so all drives should have at least 1GB just to be totally safe
2. anti-virus is quarantining recently written files to disk

It might also be related to temporary file locking, so try rebooting to see if it helps.

Otherwise, check what could be preventing those package files to be extracted; it's typically the quarantine, so check the log files for that.

Please let us know what you find!

Interesting, that sounds like it might be helpful. Normally we'd just add it, but in this case... for users with integrated windows auth... maybe not so helpful. or if they use API keys, vs passwords, etc.?

In our 5.3 planning documents, I saw some kind of feature called "custom usage text" or something. It didn't make it, in part b/c no one requested it and the only use case we could think of was Universal Packages, where using upack doesn't make sense a lot of times.

Anyways, it seemed tough to market/explain, but the idea was you would be able to edit/add the usage text on the feeds. I guess, this is the first time I heard of a suggestion outside of universal packages, so maybe we can revive the feature idea :)

Got any other usecases you can think of? That'd help us go along way with adding the feature.

@viceice said in docker login failed via https reverse proxy:

So my assumption would be, that proget will respect the X-Forward* headers as already doing in other feeds.

You're right, when writing the Basic realm="<url>" header, ProGet does not look at the X-Forward* headers. I'm not sure if we should? Would that be a security problem? It kind of seems like it might? Or maybe not? I didn't find anything about the topic discussed on the 'net after a quick search...

How about just configuring your reverse proxy to just rewrite that realm in the auth header to go from http to https? I suspect, thatw ould help.

If you can do that, and then we can update the docs on how to do it, other Free users would be very much appreciative :)

And eventually we will fix this in the software, via PG-1849 - should be a simple fix, cheers!

Hi @viceice

The http://proget-test.kriese.eu/ being returned as the Realm because it's set as the BaseUrl.

If you remove that, then it will work; however, it would cause your instance of ProGet Free to report licensing errors due to your reverse proxy configuration.

Ultimately this is a reverse proxy issue, and we might be able to have the Relm respect the X-Forward* headers.... however, I'm not sure if that's appropriate / okay to do?

It might be a security problem? Can you find any documentation or discussion on this topic?

The code change is easy, but we want to confirm it's okay to make before considering it further.

Cheers,
Alana

I don't know a ton about Maven, but other users have reported that they've used a process like this for a disk-based repository.

1. Traverse all directories and upload all POM files with a path relative to the root
2. Traverse all directories again, and upload all non-POM and non-checksum files (like .md5)

There will be errors, particularly if you have invalid POM files or your directory structure doesn't match the required MAVEN convention, so inspect those case-by-case to determine if it matters (like an bad artifact from 5 years ago can probably be ignored).

That's just what I heard from customers, so if you have more details on how you do it, we'd love to hear! THanks much

Hello;

That was a mistake/typo in the docs, which i've since corrected;

https://github.com/Inedo/inedo-docs/commit/cd7091e8eaf37939949d0681f137a78d579acbc6

The correct url is https://«proget-server»/«feed-name»/«packageName»/«versionNumber[optional]»

But note, that's only for NuGet package. You can easily find the download url for any package from the UI, by looking at the Download button on the package page.
Cheers,
Alana

Hello, please review License Key Activation Docs, especially the note at top talking about Automatic License Activation Not Working in older versions.

Also... please upgrade. ProGet 5.3 is great!!

Hello; this is definitely quite strange.

We haven't had any problems in our test labs, using significantly less-powerful hosts and significantly more traffic. Other users aren't reporting this problem on any platform, so I'm inclined to say it's configuration-related, but what configuration?

How is ProGet configured? If it's just a single feed and a single connector to that feed, then it's not your Proget configuration.

In any case, I'm certain it's not the database itself, but it's related to the network stack related. SQL Server uses network connections, and the "connection timeout" happens when the network stack gets overwhelmed. This can happen when a TON of connections are open, but not closed.

One thing we've seen is that certain network-based reporting tools (monitoring/logging) end up causing problems. They try to send a error over the network that the stack is overloaded, which then gets queued up, and continues to overload the stack. Eventually it calms down.

We've also seen bad hardware cause this. One time, it was even a bad wireless access point in. No idea how that happened, but something to do with routing and packets.

So perhaps try a new server, like make one at LightSail , totally fresh. If you can find a way to reproduce it, then it'll be good, because we can at least investigate it further then.

This isn't currently supported, but something we can consider adding if you're open to using our Feature Request Process.

One thing that would go a long way in helping us understand the usecase (and way to explain it to users) is suggesting where precisely we should document this, and how.

https://docs.inedo.com/docs/proget/reference/api/asset-directories-api (source)

There isn't a verification process per se; there is a re-index process that will delete orphaned packages, scan for symbols, rebuild the hash codes, etc. You can access it from Manage Feed > Storage & Retention page.

This might help, but you could also do a drop-path import as well.

No problem, ask away :)

You could. Not sure what the use case would be...

But, the URLs for NuGet package versions are quite predictable, and you can discover them from the Download button in the UI. For example, the download URL for InedoLib v950.0.7 just looks like https://proget.inedo.com/nuget/NuGetLibraries/package/InedoLib/950.0.7

The NuGet API does not support Bearer authentication. You can specify an API key when publishing packages (X-NUGET-APIKEY header), and it might work when downloading packages? Haven't tested, and no one asked before. But you could also specify api:<apikey> as the basic auth credentials as well, so I guess that's really easy too.

@atripp said in Creating PowerShell repository, protecting pull/download by API key:

this credential can be the name/password of a user inside of ProGet

Or a user that's configured in your Active Directory, assuming you have enabled that integration.

Hello, for sure!

It's pretty easy; just don't give the Anonymous user any access to your feeds, and then authentication will always be required, either when browsing the ProGet application or using the API (such as Install-Module).

When you use the Register-PSRepository command, you can the Credential option to specify a credential.

This credential can be the name/password of a user inside of ProGet (let's say, Admin:Admin), or it can be username of api with a password of an api key you've configured (so, api:my-secret-key).

Hello,

We aren't getting any other reports of this problem, and it's very unusual to get a 500 error that isn't logged. So it's really hard to guess what the problem could be.

Can you enable the "Feed Error Logging" to increase the logging? https://docs.inedo.com/docs/proget/installation/diagnostic-center

Alternatively, you set-up a brand new ProGet instance, then create a Docker feed, and try pushing it?

Thanks.
Alana

How much space are you seeing ManualExecutions taking? About how many rows? It should be auto-pruning those.

In any case you should be able to just DELETE [Executions] WHERE [Execution_Id] = ??, and it will cascade the deletes as needed, to the other tables (ScopedExecutionLogEntries being the biggest). The manual execution data is not so useful in the mid-term, it's mostly helpful or short-term debugging.

Hello;

That's a lot of data; most of the data will be taken up with the execution logs. You can use the sp_spaceused to verify it, checking the ScopedExecutionLogEntries table.

Usually you would want to use retention policies, but if SQL Server is blocking you from using the database, then you'll have to manually trim the logs, at least to get the software running again.

Let me share some SQL Code from that might help you do this.

First, find builds executions you don't want anymore; the code in the RetentionPolicies_GetBuildsWithLogsToPurge stored procedure might help, here is a way to find some builds you don't want.

	SELECT B.*
	  FROM [Builds_Extended] B
	       INNER JOIN [Applications] A
		           ON A.[Application_Id] = B.[Application_Id]
	 WHERE A.[ApplicationGroup_Id] = @ApplicationGroup_Id
       AND (@Pipeline_Name IS NULL OR B.[Pipeline_Name] = @Pipeline_Name)
	   AND (@DeployedReleasesOnly_Indicator = 'N' OR [ReleaseStatus_Name] = 'Deployed')
	   AND (@RejectedOnly_Indicator = 'N' OR [BuildStatus_Name] = 'Rejected')
	   AND (@AlwaysRetainAfter_Date IS NULL OR B.[CreatedOn_Date] < @AlwaysRetainAfter_Date)
       AND EXISTS(
	        SELECT TOP 1 SEL.[Scope_Sequence]
			  FROM [PipelineStageTargetExecutions_Extended] BEX
		INNER JOIN [ScopedExecutionLogs] SEL
		        ON SEL.[Execution_Id] = BEX.[Execution_Id]
                         WHERE B.[Application_Id] = BEX.[Application_Id]
                           AND B.[Release_Number] = BEX.[Release_Number]
                           AND B.[Build_Number] = BEX.[Build_Number]
        )
	 ORDER BY [CreatedOn_Date]

Then, you can do EXEC Builds_PurgeExecutionLogs @Build_Id = ??, where ?? is the ID of the build.

Thanks for the very detailed test case, we will be reviewing it quite soon and get a fix ASAP!

Hello, this has finally been scheduled for 5.3.13, which is shipping tomorrow. It addresses only this specific test case, so please let us know if you're spotting other errors.

@nuno-guerreiro-rosa_9280 said in cannot login on authenticated feeds after upgrade:

@atripp Hello atripp. Just to be clear, this occurs not only authenticating with docker but also with private NuGet authenticated feeds. I have just switched to progetmono and both private docker and NuGet feeds are authenticating correctly so the issue is related to the .NET Core release.

If you can share some more information about the authentication in .NET Core, that would be appreciated. To date we can't reproduce any authentication problems, with either Docker or NuGet.

@nuno-guerreiro-rosa_9280 said in cannot login on authenticated feeds after upgrade:

error occurred processing a GET request to http://myproget.blabla.com/nuget/feed-nuget/v3/flatcontainer/microsoft.extensions.dependencyinjection/index.json: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.

The timeout error can happen even with a Single client, when the server is underpowered. Essentially, the client is flooding the server with a ton of network connections. Increasing the server power should help.

Hello; can you try switching to the ProGetMono container to see if it makes any difference?

If it does, can you share details of how your NuGet and Docker feeds are authenticated? And provide the specific errors you're seeing?

Hi @mike-vanoorschot_1180

We check for local requests using HttpRequest.IsLocal, which basically just looks for 127.0.0.1. If it's not local, then a license violation is recorded.

If the server doesn't resolve proget.xxxx.com as 127.0.0.1, then configuring BaseUrl will be a problem, especially with NuGet. The NuGet API requires absolute urls (issue #1), and many API responses are just URLS that the client (in this case, the connector) must follow to get the "real answer" (issue #2). So this will trigger license violations.

If you need to specify a BaseUrl (you really shouldn't have to), then you'll need add a /etc/hosts entry for proget.xxxx.com 127.0.0.1

For this, you could just disable the connector, and use a Promoted-package workflow, where you manually promote approved packages from one feed (Unapproved) to another feed (Approved).

When the request is very simple, like "give me this exact version of this exact package", then it's not forwarded. But typically the request is, "what's the latest version of this package". Of course, that must be forwarded and aggregated against all connectors.

Why the client makes such a request in some cases is a mystery, but that's why they rewrote everything from scratch into a new, v3 api.

Not at present, but it's definitely on our roadmap, but there's not a ton of demand for it so we haven't prioritized it.

The primary usecase seems to be having a Otter or BuildMaster server in the cloud that in-house servers connect to. Is that what you were thinking?

ProGet will use the v3 (JSON-LD) API when possible, but not all v2 (ODATA) queries can be "translated" to the v3 (JSON-LD) API, and thus they need to be forwarded to the v2 (ODATA) end-point.

ODATA is a general-purpose querying API (like SQL but for the web), so clients do a lot with it (sorting, etc.). The v3 (JSON-LD) API is much more limited. So I recommend you to update the clients.

Ah ha, thanks! I updated the docs to include CommonCachePath instructions.

Hello; can you share some more information about how you've arrived at this error?

What steps you performed, and specific error message? Thanks!

Hello;

I'm not able to reproduce this on the latest ProGet, and I'm thinking the problem may have been the package file was on disk from the previous version.

Here's what I did:

1. Create a new Feed with a connector to NuGet NuGet
2. Download package using the URL /nuget/nuuget-public/package/ProxyKit/2.3.3+build.0
3. File on disk is ProxyKit.2.3.3.nupkg

Cheers,
Alana

Hello; this error seems to be indicating that your FTP server is providing dates in an unexpected / non-standardized format.

https://github.com/Inedo/inedox-ftp/blob/master/FTP/InedoExtension/FtpWebRequestExtensions.cs#L70

It should be easy to fix the extension, if we can see the data...

Could you provide us with the dates that you're seeing?

Thanks.
Alana

Hello;

I haven't tested this on ProGet v5.0.10 (note: it's 2.5years old), but I can see that there's a +build.0 in the version. There's been a number of changes since that version of ProGet to better address the quirks surrounding build metadata. So please upgrade to the latest ProGet.

Best,
Alana

Hello;

Can you share the errors you're seeing in the Admin > Diagnostic Center? Those will be logged w/ the 500 errors.

Thank!

@nicholas-boltralik_3634 very sorry on the slow response, we were internally discussing and tracking this, but I guess I didn't update you here.

We will target to make a software change in PG-1814, and hopefully get this working in the next or following maintenance release.

Thanks for updating and letting us know the problem was related to a connection string!