Posts made by rhessinger

Hi @jyip_5228,

Are you still running the default max connections property? Or have you increased it? Also, how many concurrent builds are you running at a time?

Please send us an update on how the .Net Core version is working out for you.

Thanks,
Rich

Hi @jyip_5228,

I just verified and the fix is still in place. We have seen connection issues in the past with the Mono framework due to how connection sharing is implemented in the mono framework. How often does this issue happen? If you restart the ProGet image, how long before you see the connection pool error?

Just to let you know, as part of ProGet 5.3.10 release, we shipped the ProGetCore container image as well.

You can follow the normal steps in the Linux and Docker Installation Guide to install/upgrade, but just use progetcore for the container instead of proget.

Aside from support for the Lucene-based Maven feed indexing (in progress), it seems to be feature complete. And of course, if there are problems, you can switch back to proget:5.3.10 or downgrade as needed (no database schema changes).

For example, docker pull proget.inedo.com/productimages/inedo/progetcore:5.3.10

Thanks,
Rich

Hi @nuno-guerreiro-rosa_9280,

I just wanted to let you know that this was released in 5.3.10. Please let us know if there are still any issues with this.

Thanks,
Rich

Hi @philippe-camelio_3885,

For an Ensure-PSModule command, here is what I'm thinking.

Ensure-PSModule {
   ModuleName: ...
   Version: ...
   MinVersion:...
   MaxVersion:...
   Force: false
   AllowClobber: false
   Repository: ... 
   Credential: ...(I expect this to be a ResourceCredential)
   Scope:...
   Properties: ... (any other additional properties to pass to Install-Module)
}

Where it just ensures that version (or within the min/max) is installed.

Am I understanding your request correctly?

Thanks,
Rich

Hi @viceice,

Thanks for the clarification on your environment! I see what is going on now. I have created a ticket, PG-1809, to track the fix for this. We expect this to be released in ProGet 5.3.11 which we are expecting to be released in September 11, 2020. Basically in that instance, we are not respecting the values within the X-Forwarded-* headers. I'll let you know if anything changes on the timeline.

Thanks,
Rich

Hi @viceice,

Can you please confirm your connector URL contains the https? How is ProGet setup? Is it installed on windows? Are you using IIS? It might be easier to add an SSL binding in IIS for this as well.

Thanks,
Rich

Hi @viceice,

Is your proxy passing X-Forwarded-Proto also? That should be passing https. You could try hard coding that in the proxy.

Thanks,
Rich

Hi @scroak_6473,

I just wanted to send over a few screenshots so you can see what is releasing tomorrow:

When you click on the layer digest on the Vulnerabilities, Repository Vulnerabilities, and Image Vulnerabilities pages, it now will show this modal dialog:

The List Repositories page now looks like this:

The All Tags for a Repository now looks like this:

The All Images for a Repository looks like this:

Thanks,
Rich

Hi @philippe-camelio_3885,

I wanted to let you know that this feature has been fixed and will be released in BuildMaster 6.2.17 which is due out tomorrow! I'll reply back if there are any changes to this schedule!

Thanks,
Rich

Hi @ludovic_2596,

Can you please give me your entire ProGet version? Also, what feed type are you having trouble deleteing packages from?

Thanks,
Rich

Hi @viceice,

You should be able to use http://localhost or http://<PROGET_IP_ADDRESS> in your connector without issue. Please give that a try and let me know if you run into any issues.

Thanks,
Rich

Hi @viceice,

ProGet free edition only supports self-connectors when they connect directly to ProGet. If you bypass the proxy, that should resolve the license violations for you.

Thanks,
Rich

Hi @jyip_5228,

That may be your issue. NuGet is currently on version 5.7. Could you please try upgrading your NuGet.exe and see if that fixes the issue?

Thanks,
Rich

Hi @nuno-guerreiro-rosa_9280,

We have finally been able to recreate this issue in our sandbox. We are currently looking into a fix, but we expect to have one in the next version of ProGet, 5.3.10. This looks to be an issue with the mono framework. We use mono runtime in our Docker images for ProGet. We are also going to be releasing a .Net Core based technical preview of ProGet Docker in version 5.3.10. This will be in addition to our standard mono based version. Our internal testing is going very well and it looks to have removed a lot of the gotchas that mono has.

Thanks,
Rich

Hi @jyip_5228,

For the v2/v3 thing. If the connector is using v3, there is potential to fall back to v2 if you have issues connecting over teh v3 API. That may be why we are seeing the v2 error.

To follow up on Alana's comment. You may need to use a tool like Wireshark, Fiddler, or Burp to catch the request and response traffic between nuget.exe and ProGet. That would be the only way to see the full response from ProGet and determine if it does attempt v3 before the v2 request.

One last question, what version of the nuget.exe are you using? Is it the latest version?

Thanks,
Rich

Hi @nuno-guerreiro-rosa_9280,

This is the first time we have heard any issues with this. We also have not been able to recreate this issue internally. Is there anything unique in your ProGet setup?

Thanks,
Rich

Hi @gravufo,

Thanks for following up. Glad to hear it is working!

Thanks,
Rich

Hi @jyip_5228,

Looking at your last connector error stack trace, it looks like your connector is using the NuGet v2 URL. Would you be able to send a screenshot of your NuGet connector configuration? NuGet has added limiting to their V2 API and I'm wondering if that is causing your issue.

Thanks,
Rich

Hi @gravufo,

After you enabled the Close Database Connections Early have you tried restarting your web server? You shouldn't have to restart it, but I have seen some issues when there were too many connections open prior to enabling the setting. Coudl you run the following SQL next time you have the issue?

SELECT DB_NAME(dbid) as DBName, COUNT(dbid) as NumberOfConnections, loginame as LoginName
FROM sys.sysprocesses
WHERE dbid > 0 AND DB_NAME(dbid) = 'ProGet'
GROUP BY dbid, loginame 

Thanks,
Rich

Hi @john-b_3261 ,

Can you please verify that your ProGet windows service is running and that the account the service is running under has access to those folders? Our nightly cleanup scheduled jobs typically clean up these folders./

Thanks,
Rich

Hi @jyip_5228,

Does this only happen with remote packages? Or do you see this with packages in ProGet as well? When this happens, if you run nuget locals all -clear on the same account that your build agent is running under, will it work then? I have seen some similar issues like this in the past, but typically they are caused by an issue with local caching in NuGet.

Thanks,
Rich

Hi @Joseph-Saladino_7499,

What version of BuildMaster are you currently running? In BuildMaster 6.2 there is a history on pipeline changes. You would view it by clicking on Deploy -> Deployment Pipelines and then clicking on the last modified date for the pipeline you would like to see history for. Here is a screenshot of the UI:

Hope this helps!

Thanks,
Rich

Hi @dusbn

For the drop path to work. The packages do need to be at the root of the drop path directory. Do you have the packages within subfolders?

Thanks,
Rich

Hi @nicholas-boltralik_3634,

What version of ProGet are you running?

Thanks,
Rich

Hi @dusbn,

I think you are looking to use the Drop Path. This would allow you have ProGet monitor a folder and automatically upload a package when it exists in that folder.

Alternatively you could use a PowerShell script to iterate through each nupkg and use nuget push to push each package.

Thanks,
Rich

Hi @scroak_6473,

You most likely have to setup a Resource Credential. On the Manage User Directories page, you should see a button for AD Credentials. Create a credential that has domain access. From there, in your domains to search, enter a value of {domain},{Secure Resource Name} ex: exampldomain.com,DomainAdmin. Can you give that a try and see if it connects?

Thanks,
Rich

Hi @mcascone ,

We have fixed the issue and the published date will now update when the package is overwritten. This will be released in ProGet 5.3.9 which is due out this Friday Augst 14, 2020.

Thanks,
Rich

Hi @markus4830,

I'm definitely sorry about this. The change was made to help to aide in improvements to other areas of the system related to NuGet. Unfortunately, it looks like it affected the NuGet API. Expect a more permanent solution in the near future.

Thanks,
Rich

Hi @thomas-ibel_2956,

I am not aware of any other changes that would have affected that specifically. There were a couple more changes to other areas of the system. It is always possible that there was an update to a Debian package that mono depends on that helped to improve connection handling in Docker. But I cannot confirm that was the case. The only thing I can confirm is that the top two layers in the image have a slight difference to them when comparing 5.3.8 and 5.3.8-ci.20 .

Thanks,
Rich

Hi @Jonathan-Engstrom ,

When you click your name in the upper right corner, go to Edit Profile. On the left you should see a button that says Change Email. Do you see a button there?

Thanks,
Rich

Hi @scroak_6473,

I created a branch and a blank page on GitHub for Docker Swarm. You should be able to add your notes and examples there. Please let me know if you have any issues!

Thanks,
Rich

Hi @thomas-ibel_2956,

Glad to hear it! Please let me know if this continues or if you start having issues again.

Thanks,
Rich

Hi All,

We have just released ProGet 5.3.8. Would you please upgrade and test with the Close Database Connections Early setting enabled?

Thanks,
Rich

Hi @jyip_5228 ,

We just released ProGet 5.3.8, which includes the Close Database Connections Early option in the advanced settings. Could you please upgrade and give this a try?

Thanks,
Rich

Hi @pluskal_4199,

Thanks for the information.

The FeedCleanup job is automatically scheduled and you can see the date and time by navigating to Administration -> Scheduled Jobs. Would you be able to tell me if the time that was automatically configured matches when you are seeing the timeout?

Our Docker image currently is running ProGet using mono which comes with its own subset of issues. I'm seeing the SNI_Error is a pretty common error with mono and we have seen mono has more trouble releasing connections than when running within the .NET Framework. The most annoying part of mono, is they don't always give us the most accurate and descriptive error messages. This could just be a similar exception to the max connection pool but based on DB transactions. We are currently in the process of migrating to .NET 5 and we will hopefully have an updated Docker image over the next couple of months. If you are interested, please keep an eye out for more information to come.

I would like to see some of these results on Windows-based installations also to help guide our solution.

Thanks,
Rich

Hi @pluskal_4199 ,

Are these builds running at the same time as the Feed Cleanup job for your NuGet feed by chance? Also, are you seeng the same error The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.?

Thanks,
Rich

Hi @jyip_5228,

Just an update! We will be shipping a potential fix in PG-1783, which adds a new checkbox in advanced settings (unchecked by default):

Close Database Connections Early
EXPERIMENTAL (for debugging/testing only) - As of ProGet 5.3, database connections are left open during the lifecycle of a NuGet API request as a means to reduce overhead; however, this may be causing ProGet to run out of available connections to SQL Server. Set this value true to open/close database connections as needed on NuGet feeds.

We'll update when this is shipped --- but if we can get some folks to verify that this works better (we can't repro, at all ), then we will likely make it the default. Hopefully this will do it. Seems better than raising connection pool limits

If so, then the savings in connection open/close overhead don't seem to make it particularly worthwhile. This "keep open" technique made a ton of differences elsewhere in our software, but since a NuGet API requests may yield a ton of other network requests (via connectors) and block, the pool may be getting drained too quickly...

just a theory, as I mentioned. Anyway hope this helps, stay tuned!!

Thanks!

Hi @pluskal_4199 ,

That is great to hear! How often were you seeing the error prior? Also what OS Version, SQL Server Edition, and SQL Server Version are you running?

Thanks,
Rich

Hi @christian_panten,

Are there any errors showing in the Diagnostics center?

Thanks,
Rich

Hi @christian_panten,

Can you please send over the error you are seeing when running nuget push? As long as your -Source is the NuGet API URL, then you shouldn't need to do anything special.

Thanks,
Rich

Hi @christian_panten_6482,

You would just push the snupkg the same way, but only push the snupkg. For example:

nuget push -source sources-feed MyPackage.1.2.0.snupkg

I also wanted to point out this from our documentation as well:

To summarize, push only the symbol package to your ProGet feed if you intend to use the symbol server. To prevent symbols from being downloaded with the NuGet package, see the Strip symbol files from packages downloaded from this feed option on the Manage Feed page under Symbol Server settings.

Thanks,
Rich

Hello @jyip_5228,

I'm having trouble recreating this locally. If my memory servers me right, you are currently running ProGet in Docker, is that correct? I will attempt to spin-up the Docker-based version of ProGet locally to attempt to recreate this issue. I do wonder if this is related to your database connection pool errors. Do you see any errors in the Diagnostics Center under Administration?

Thanks,
Rich

Hi @christian_panten_6482,

Since you are using SourceLink, you should be able to use the Symbol Server built into ProGet as long as you have ProGet running on Windows. For information on setting up a symbol server and pushing packages to it, please review our NuGet Symbol and Source Server documentation.

Hope this helps!

Thanks,
Rich

Hi @jyip_5228,

Thanks for the update. I'm currently implementing a potential fix to the connection stuff. I'll let you know when I have something ready.

Thanks,
Rich

Hi @jyip_5228,

Have you seen this issue with any other packages? Or is this just MethodTimer.Fody. Also, do you have multiple versions of MethodTimer.Fody installed on this machine?

Thanks,
Rich

Hi @jyip_5228,

Can you try adding the -NoCache parameter?

ex: C:\>nuget install MethodTimer.Fody -Version 3.1.2 -Source https://xxx/nuget/ks-nuget/v3/index.json -NoCache

Also, could you try clearing the local NuGet file caches by running nuget locals all -clear?

Can you please let me know if the install command works after either of those options.

Thanks,
Rich

Hi @jyip_5228,

Thanks for giving me the updated counts. Glad this workaround is working for you. We are still working on the issue internally. Do you happen to notice any other errors in the logs right before the timeout happens?

Thanks,
Rich

Hi @jyip_5228,

The default setting for SQL Server 2017 is unlimited. As long as you didnt set one on your SQL Server, then you should be good.

Thanks,
Rich

Hello @mcascone,

Thanks for sending this over to me. So there looks to be an issue with the universal feed not updating the published date when overwriting a package. I happened to test 3 other feed types and it worked fine. I have created an internal ticket to fix this and I will update you as soon as it is scheduled for release.

Thanks,
Rich

Hi @jyip_5228,

The last exception you are seeing is currently a known issue that we are currently working through. We have noticed this tends to affect the docker based ProGet installations the most. In most cases, increasing the max connection pool limit in the SQL connection string seems to fix the issue. Depending on your SQL Server setup, you may need to increase the allowed connections on SQL Server as well. Can you please try to increase the max connection pool in your SQL connection string? Also, can you please tell us what version and edition of SQL Server you are currently running?

Thanks,
Rich