Posts made by rhessinger

Hi @antony-booth_1029 ,

I have fixed this issue and it will be released in BuildMaster 6.2.15 which is due out on July 17. I'll post back here if anything changes.

Thanks,
Rich

Hi @antony-booth_1029 ,

It looks like this is just an issue with the editor. If you set the value to include the = sign and save it. When you run everything is fine and the value includes the = sign. When you open to edit it though, the = is stripped off, and then when you save it, it removes = from the config file. I am currently working to correct the issue.

Thanks,
Rich

Hi @gravufo,

The Inedo Hub lets you select which version you would like to install from a dropdown. You can also customize the feed that you would like to look for those packages. If that does not cover what you are looking for, technically you could generate offline installers for each version you would like to keep.

Thanks,
Rich

Hi @gravufo,

Great! Glad to hear it! Please post back if you find anything else.

I also recommend that you switch to the Inedo Hub in the future. We are in the process of deprecating our traditional installer. The Inedo Hub has the ability to update an installation previously installed with the traditional installer and the Inedo Hub now supports offline installations as well, if you need that functionality.

Thanks,
Rich

Hi @gravufo,

Would you be able to rerun the database scripts on your database? You will just need to run the Run inedosql to update the database step of our manual install guide. Can you see if this fixes your issue?

Thanks,
Rich

Hi @gravufo ,

Did you get any errors during the upgrade process? Also, could you download inedosql and check for errors from the inedosql GitHub page?

Thanks,
Rich

Hi @scroak_6473,

So I finally figured it out. Our ProGet docker image actually runs mono to run ProGet. Apparently this is an issue with the new way we load extensions in mono. I have put a fix in the next version of ProGet (5.3.6), but if you would like to leverage it now, you will just need to change a config value. Just navigate to Administration -> Advanced Settings and locate the value Extensions.UseNewExtensionLoader and uncheck it and save. Then you will need to restart your docker image, i.e. sudo docker restart proget. Then the vulnerability source should start working, or at least show you any real issues you have.

Thanks,
Rich

Hi @scroak_6473.

I have been able to recreate the error and I can confirm that this is only an issue with running ProGet in docker. I'm currently working to fix this issue.

Thanks,
Rich

Hi @eduardbabayan_1578 ,

What version of ProGet are you running?

Thanks,
Rich

Hi @antony-booth_1029 ,

Looks like we had an issue in how the javascript resource is loading for this modal. I have fixed this and it will be released in the next version of BuildMaster 6.2.14 which is due out at the end of this week.

Thanks,
Rich

Hi @scroak_6473,

We are still working on the issue. It looks to be only an issue with the ProGet docker image, but I don't have a lead on why yet. If you have the ability to install this on windows, I can confirm it is working there. I know you are running a free trial of ProGet. If you need to extend the trial because of this issue, please contact us through https://inedo.com/contact.

Thanks,
Rich

Hi @scroak_6473 ,

I working on reproducing this locally, but I'm running into some issues with my local Docker installation. I'll reply once I run some local tests. The majority of my testing for Clair has been done using the windows IIS or integrated web service installs.

Thanks,
Rich

Hi @luminiscence_6236 ,

I'm glad to hear you figured out the issue. I'm going to add an internal ticket to add better error handling around docker connectors.

Thanks,
Rich

Hi @scroak_6473,

I was hoping recreating the source would fix it. Based on your error, it looks like the error happens before it even attempts to call Clair. If it made at least one call to Clair, your output would look similar to this:

Can you provide a screenshot of your extensions page? I would like to verify the extension versions. Also, could you screenshot your Manage Feed -> Detection & Blocking tab?

Thanks,
Rich

Hi @luminiscence_6236 ,

If you click on the Containers link at the top. Does it error when you click on a local tag? Also do you have any connectors on your docker registry?

Thanks,
Rich

Hi @scroak_6473 ,

Everything looks to be correct. Can you try removing your vulnerability source from your Docker feed, then delete the vulnerability source then re-add it again and reassociate it to your feed? The only thing I can assume from that error stack trace is it is failing to parse the Configuration for your vulnerability source.

Thanks,
Rich

Hi @scroak_6473 ,

Do you have any other vulnerability sources setup? Or just clair?

Thanks,
Rich

Hi @scroak_6473,

I think i see your issue. Can you please change your URL to http://proget.qa.xxxx.xxxx (or https:// if you are using SSL). The Base URL requires the http:// or https:// in the value. This might be causing your issue.

Thanks,
Rich

Hi @luminiscence_6236 ,

Do you see any other errors in the diagnostic center? Does the page load when you click the Containers link at the top of the page? If that is the one that is erroring, are you able to navigate to your registry by first going to the Feeds then clicking on your registry there?

Thanks,
Rich

Hi @scroak_6473,

Could you please verify what version of ProGet you are currently running? Is it 5.2.3 or 5.3.2? I also think that there is a bug with Clair in 5.3.2, would you be able to upgrade to 5.3.4 and test it again? You can find the 5.3.4 Docker image on our public ProGet: https://proget.inedo.com/containers/repositories/tags/overview?feedId=22&repositoryName=inedo%2Fproget&tag=5.3.4.

Can you also tell me which Clair image you have installed on your Docker host?

Also, you will need to make sure that the Clair image has access to ProGet. The flow of Clair is a bit weird. First ProGet will make a request to Clair to start the process. Clair will then make a request to ProGet (with a temporary auth key) to download each image layer to scan. Then ProGet will make one final call to Clair to download the found vulnerability data.

Once, you send me this information we can figure out where to go from there. I do have a way to manually test the calls to Clair leveraging a PowerShell script, but the information gathering to run it is a bit tedious.

Thanks,
Rich

Hi @luminiscence_6236 ,

Could you please provide the entire stack trace for the error? You should be able to retrieve it from the Diagnostics Center on the Administration page.

Thanks,
Rich

Hi @patchlings,

No problem! We are still investigating some workarounds, but we don't have anything specific yet.

We don't currently have a timeline for the .NET 5 switch. We have attempted a .NET Core transition in early 2019, but we ran into too many instability issues and scrapped it. We will wait for .NET 5 to be officially released (Microsoft says November 2020) before we attempt the switch, but it will all come down to the stability of ProGet on .NET 5.

Thanks,
Rich

Hi @patchlings ,

That is the heart of the Mono bug that I referenced. Currently Mono has a memory leak around HTTP requests which will cause our connectors to hang when the memory usage is too high. They have reduced the leaking substantially starting in Mono 5.16, but there is still a leak. If you search around on this error, you will find similar questions with almost identical errors still in 2020. Unfortunately, we are in the hands of the Mono team until we switch off Mono and migrate to .NET 5.

Thanks,
Rich

Hi @patchlings,

We believe this error is due to a regression/bug in the Mono, which is related to change to their HttpWebRequest stack. In the future, after Microsoft releases it, we plan to move to .NET 5 for a more stable platform.

The best resolution, for now, is to move to Windows/IIS; it's the most stable. Is this possible?

If not... we don't know when Mono will fix the bug, or if it's possible for us to work-around, but we think it will get resolved in the future. Would you be able to make ProGet restart once per day? I think this would help with the memory leak.

Thanks,
Rich

Hi @patchlings,

Thanks for all the data. I really appreciate your help with this! I'm currently reviewing our code to see if I can find a correlation with this.

Thanks,
Rich

Hi @patchlings,

Thanks! I'm not surprised that debug exaccerbates the issue, but it's definitely good to know.

Thanks,
Rich

Hi @patchlings,

Thank you for checking this for me. So the good thing is that this is definitely not a threading issue. Can you give me a stub status when it's working during normal use and one after it starts behaving poorly?

Thanks,
Rich

Hi @patchlings ,

Thanks for all this information. Would you be able to get me an open thread count for when everything is running fine and when things are not? You should be able to find it by running sudo ps huH p <PID_OF_MONO_PROCESS> | wc -l. Also, are you running these commands inside the docker image? Or on the host?

Thanks,
Rich

Hi @patchlings,

Thank you very much for the information. I'm researching the issue more on our side. I may send over some more questions your way while I'm working on this.

Thanks,
Rich

Hi @patchlings,

Do these connector errors happen on all your feeds? Or just NuGet?

Thanks,
Rich

Hi @philippe-camelio_3885,

Technically you can use VSCode, but you will still have to at least install the MS Build Tools for Visual Studio 2019. Then you will have to run the msbuild.exe to compile your code. Here is Microsoft's guide for how to work with C# in VSCode. Hope this helps!

Thanks,
Rich

Hi @patchlings ,

I was doing some Docker research around this issue. Something I found Docker users complaining about was an issue with the total number of available connections on the host. It seems that the more containers running on a host (especially with database containers on it), the connection count can get spread a bit thin under load. Have you tried increasing the max number of connections? Here is a StackOverflow about increasing them Increasing the maximum number of TCP/IP connections in Linux.

Also, how many connectors do you have on your NuGet feed?

Thanks,
Rich

Hi @philippe-camelio_3885,

Do you know how to create your own extension? If not, check out creating an extension using the SDK.

To create an $InedoProduct and $InedoProductVersion variables, you can create a couple of new classes extending Inedo.Extensibility.ScalarVariableFunction. Then simply implement the ExecuteScalar method and have it return Inedo.SDK.ProductName or Inedo.SDK.ProductVersion. You can see an example of this with the ServerNameVariableFunction in the Inedox-InedoCore extension on GitHub. If you can submit a pull request, I'll be happy to test it and get it published right away!

Thanks,
Rich

Hi @patchlings,

Have you enabled the nginx HTTP Stub Status module on your nginx proxy? I would be interested to see the number of open and waiting connections. Although this article is old, nginx: See Active connections / Connections Per Seconds seems to show a pretty easy configuration to see this status. I would be curious to see these counts when the problem exists. Would you be able to enable this and post the counts when everything is working and when it is not?

Thanks,
Rich

Hi @scroak_6473,

That's great to hear that the impersonation has fixed the issue. We have fixed this issue to not require impersonation in ProGet 5.3.2. I will let you know if anything changes!

Thanks,
Rich

Hi @scroak_6473 ,

Yes, I have included this fix in 5.3 as well. It will be released as part of ProGet 5.3.1.

Thanks,
Rich

Hi @scroak_6473 ,

Thanks for bringing this up to us! I have fixed this issue and it will be released in ProGet 5.2.31.

Thanks,
Rich

Hi @aleksandarsh_4585 ,

I wanted to put the final solution in this thread for what solved the error. It looks like the issue was around which extensions were installed. After upgrading to 6.1, you needed to enable the legacy features in Administration > Advanced Settings and install the Legacy extensions in Administration > Extensions. We also had to uninstall and extensions that had errors loading including Amazon, Jira, and Jenkins and only leave the legacy versions installed.

Thanks,
Rich

Hi @aleksandarsh_4585 ,

I'm currently looking into if you can back up from a version older than 6.1.

I did want to verify that you have the latest extensions installed in your 6.1 instance. I would also recommend restarting IIS (or the independent web server if you are using that) and the BuildMaster service. Would you be able to take screenshot of your extensions page? The error you sent out is typically an error caused by a missing extension. In these cases, it will also cause your build pipeline page to error on load. That could explain why you can't load your application when you click on it.

You might be able to see what extension is failing by going to /pipelines/edit-json?pipelineId=464. Could you also try that?

Thanks,
Rich

Hi @aleksandarsh_4585 ,

Would you be able to send over a screenshot of your extensions page?

I think the best way to migrate this application is to install a fresh install of BuidMaster 6.2 and use option 3 from the release notes which is to export your application and reimport it into 6.2.

Export a single application from BuildMaster at a time into a universal package, and import that package into your new BuildMaster 6.2 instance. This will give you a good opportunity to discover any issues and to gradually migrate your applications to the new instance as you are able.

Generally, this approach will work best if you have a large number of applications, and some of them rely on legacy features that have been removed in BuildMaster 6.2. This way, you can bring over applications as the legacy dependencies are removed to take advantage of new features and capabilities, while still retaining the original instance for risk mitigation.

You can view the documentation for backing up and restoring applications here.

Thanks,
Rich

Hi @aleksandarsh_4585,

What version of BuildMaster did you upgrade to (i.e., 6.2.10)? It looks like it did not run the scripts for 6.2 on your database. I just wanted to confirm that the Manual Install zip you downloaded matches the version of BuildMaster that you installed.

Also, could you run the following query? I would like to see if any script errors happened during the SQL upgrade.

select top 10 * from __BuildMaster__DbSchemaChanges2 where Success_Indicator = 'N' order by Executed_Date DESC

Thanks,
Rich

Hi @aleksandarsh_4585,

It sounds like either the database or the application didn't actually update. Would you be able to make a direct SQL query against the ProGet database that is in the config? I would like you to run two different queries:

select * from Applications_Extended

and

select top 10 executed_date, Script_Guid, Script_Name from __InedoDb_DbSchemaChanges order by Executed_Date DESC

Thanks,
Rich

Hi @jonnpear_8217 ,

I have just updated the documentation for switching to IIS. Please take a look and let me know if you have any questions.

Thanks,
Rich

Hi @huomm23_2930 ,

I'm glad to hear that worked for you! Thanks for giving me an update.

Thanks,
Rich

Hi @huomm23_2930,

In the Basic and Enterprise versions of ProGet, you can simply use the publish role on the users you don't want to have the overwrite permissions on, but in the free version, you can restrict the Administrator role from being able to overwrite your packages. To do this, you will need to perform the following steps:

1. Navigate to Administration > Security > Tasks tab
2. Click the Customize Tasks button
3. Click Administrator in the list
4. Uncheck Overwrite package in the Feeds section
5. Click Save

That should prevent your packages from being overwritten.

Please let me know if this works out for you!

Thanks,
Rich

Hi @Adam,

I have pushed an updated extension that includes this fix. The fix is included in the Windows extension 1.0.18+ and 1.7.1+. You should see an update when you check in Administration > Extensions. Please let me know if you have nay issues!

Thanks,
Rich

Hi @aclauss,

If you go to the feeds page in ProGet, in the upper left corner you should see a toggle button that has two options: Feeds and Connectors. Please click the Connectors button and you should see a list of your connectors. From there you can click Create Connector to create a new connector.

Once you create the connector, you can tie it to your feed by going back to the manage feed page, click connectors, and now that connector dropdown you see before should contain the connector you recently created.

Hope that helps!

Thanks,
Rich

Hi @Jonathan-Engstrom ,

Have you tried wrapping your PSEnsure inside of a with block to force everything to run as a specific Resource Credential?

Example using a resource credential named AdAdminResourceCredentials:

with credentials = AdAdminResourceCredentials
{
	PSEnsure
	(
	    Key: RSHStuff,
	    Value: True,
	    Collect: >>

	        $ErrorActionPreference = 'SilentlyContinue'
	        $TargetOU = 'Users'
	        $Value = (Get-ADComputer -Identity $env:COMPUTERNAME).DistinguishedName -match $TargetOU
	        $CurrentValue = (Get-ADComputer -Identity $env:COMPUTERNAME).DistinguishedName
	        ((($Value -eq $true) -and (($env:COMPUTERNAME) -match 'MyComputerName')) -or ($env:COMPUTERNAME -notmatch 'MyComputerName')) 
	        >>,
	    Configure: >>
	                
	        $samAccountName = $ServerName
	        $newOU = [adsi]"LDAP://$TargetOU"
	        $comp= ([adsisearcher]"samaccountname=$($ServerName)$").FindOne()
	        Write-Host $comp.GetDirectoryEntry()
	                                
	                                >>,
	    Debug: true,
	    Verbose: true
	);
}

Please note that my PowerShell in collect and configure is not 1 to 1 with yours, I would replace my PowerShell script with yours in those two properties.

I also was talking with one of the solutions architects on a similar Otter execution question. He suggested that sometimes it is easier to figure out PowerShell issues by running PSExec under the local system account and attempting to run your PowerShell commands that way. It will sometimes show you more detailed errors than Otter. I would try running PSExec -s powershell, which will open up a PowerShell console as the local system account, and then typing in the lines of PowerShell you have had trouble with.

Thanks,
Rich

Hi @Jonathan-Engstrom ,

What version of PowerShell do you have installed on your SQL server? I'm running 5.1.18362.628.

Thanks,
Rich

Hi @Jonathan-Engstrom ,

I have noticed the $ServerName uses the name of my server configured in Otter, not the actual computer name. Have you tried running the remediation using $env.computername instead of $ServerName? Or can you verify the Server Name in Otter matches the computers name?

Thanks,
Rich