Posts made by rhessinger

Hi @patchlings,

No problem! We are still investigating some workarounds, but we don't have anything specific yet.

We don't currently have a timeline for the .NET 5 switch. We have attempted a .NET Core transition in early 2019, but we ran into too many instability issues and scrapped it. We will wait for .NET 5 to be officially released (Microsoft says November 2020) before we attempt the switch, but it will all come down to the stability of ProGet on .NET 5.

Thanks,
Rich

Hi @patchlings ,

That is the heart of the Mono bug that I referenced. Currently Mono has a memory leak around HTTP requests which will cause our connectors to hang when the memory usage is too high. They have reduced the leaking substantially starting in Mono 5.16, but there is still a leak. If you search around on this error, you will find similar questions with almost identical errors still in 2020. Unfortunately, we are in the hands of the Mono team until we switch off Mono and migrate to .NET 5.

Thanks,
Rich

Hi @patchlings,

We believe this error is due to a regression/bug in the Mono, which is related to change to their HttpWebRequest stack. In the future, after Microsoft releases it, we plan to move to .NET 5 for a more stable platform.

The best resolution, for now, is to move to Windows/IIS; it's the most stable. Is this possible?

If not... we don't know when Mono will fix the bug, or if it's possible for us to work-around, but we think it will get resolved in the future. Would you be able to make ProGet restart once per day? I think this would help with the memory leak.

Thanks,
Rich

Hi @patchlings,

Thanks for all the data. I really appreciate your help with this! I'm currently reviewing our code to see if I can find a correlation with this.

Thanks,
Rich

Hi @patchlings,

Thanks! I'm not surprised that debug exaccerbates the issue, but it's definitely good to know.

Thanks,
Rich

Hi @patchlings,

Thank you for checking this for me. So the good thing is that this is definitely not a threading issue. Can you give me a stub status when it's working during normal use and one after it starts behaving poorly?

Thanks,
Rich

Hi @patchlings ,

Thanks for all this information. Would you be able to get me an open thread count for when everything is running fine and when things are not? You should be able to find it by running sudo ps huH p <PID_OF_MONO_PROCESS> | wc -l. Also, are you running these commands inside the docker image? Or on the host?

Thanks,
Rich

Hi @patchlings,

Thank you very much for the information. I'm researching the issue more on our side. I may send over some more questions your way while I'm working on this.

Thanks,
Rich

Hi @patchlings,

Do these connector errors happen on all your feeds? Or just NuGet?

Thanks,
Rich

Hi @philippe-camelio_3885,

Technically you can use VSCode, but you will still have to at least install the MS Build Tools for Visual Studio 2019. Then you will have to run the msbuild.exe to compile your code. Here is Microsoft's guide for how to work with C# in VSCode. Hope this helps!

Thanks,
Rich

Hi @patchlings ,

I was doing some Docker research around this issue. Something I found Docker users complaining about was an issue with the total number of available connections on the host. It seems that the more containers running on a host (especially with database containers on it), the connection count can get spread a bit thin under load. Have you tried increasing the max number of connections? Here is a StackOverflow about increasing them Increasing the maximum number of TCP/IP connections in Linux.

Also, how many connectors do you have on your NuGet feed?

Thanks,
Rich

Hi @philippe-camelio_3885,

Do you know how to create your own extension? If not, check out creating an extension using the SDK.

To create an $InedoProduct and $InedoProductVersion variables, you can create a couple of new classes extending Inedo.Extensibility.ScalarVariableFunction. Then simply implement the ExecuteScalar method and have it return Inedo.SDK.ProductName or Inedo.SDK.ProductVersion. You can see an example of this with the ServerNameVariableFunction in the Inedox-InedoCore extension on GitHub. If you can submit a pull request, I'll be happy to test it and get it published right away!

Thanks,
Rich

Hi @patchlings,

Have you enabled the nginx HTTP Stub Status module on your nginx proxy? I would be interested to see the number of open and waiting connections. Although this article is old, nginx: See Active connections / Connections Per Seconds seems to show a pretty easy configuration to see this status. I would be curious to see these counts when the problem exists. Would you be able to enable this and post the counts when everything is working and when it is not?

Thanks,
Rich

Hi @scroak_6473,

That's great to hear that the impersonation has fixed the issue. We have fixed this issue to not require impersonation in ProGet 5.3.2. I will let you know if anything changes!

Thanks,
Rich

Hi @scroak_6473 ,

Yes, I have included this fix in 5.3 as well. It will be released as part of ProGet 5.3.1.

Thanks,
Rich

Hi @scroak_6473 ,

Thanks for bringing this up to us! I have fixed this issue and it will be released in ProGet 5.2.31.

Thanks,
Rich

Hi @aleksandarsh_4585 ,

I wanted to put the final solution in this thread for what solved the error. It looks like the issue was around which extensions were installed. After upgrading to 6.1, you needed to enable the legacy features in Administration > Advanced Settings and install the Legacy extensions in Administration > Extensions. We also had to uninstall and extensions that had errors loading including Amazon, Jira, and Jenkins and only leave the legacy versions installed.

Thanks,
Rich

Hi @aleksandarsh_4585 ,

I'm currently looking into if you can back up from a version older than 6.1.

I did want to verify that you have the latest extensions installed in your 6.1 instance. I would also recommend restarting IIS (or the independent web server if you are using that) and the BuildMaster service. Would you be able to take screenshot of your extensions page? The error you sent out is typically an error caused by a missing extension. In these cases, it will also cause your build pipeline page to error on load. That could explain why you can't load your application when you click on it.

You might be able to see what extension is failing by going to /pipelines/edit-json?pipelineId=464. Could you also try that?

Thanks,
Rich

Hi @aleksandarsh_4585 ,

Would you be able to send over a screenshot of your extensions page?

I think the best way to migrate this application is to install a fresh install of BuidMaster 6.2 and use option 3 from the release notes which is to export your application and reimport it into 6.2.

Export a single application from BuildMaster at a time into a universal package, and import that package into your new BuildMaster 6.2 instance. This will give you a good opportunity to discover any issues and to gradually migrate your applications to the new instance as you are able.

Generally, this approach will work best if you have a large number of applications, and some of them rely on legacy features that have been removed in BuildMaster 6.2. This way, you can bring over applications as the legacy dependencies are removed to take advantage of new features and capabilities, while still retaining the original instance for risk mitigation.

You can view the documentation for backing up and restoring applications here.

Thanks,
Rich

Hi @aleksandarsh_4585,

What version of BuildMaster did you upgrade to (i.e., 6.2.10)? It looks like it did not run the scripts for 6.2 on your database. I just wanted to confirm that the Manual Install zip you downloaded matches the version of BuildMaster that you installed.

Also, could you run the following query? I would like to see if any script errors happened during the SQL upgrade.

select top 10 * from __BuildMaster__DbSchemaChanges2 where Success_Indicator = 'N' order by Executed_Date DESC

Thanks,
Rich

Hi @aleksandarsh_4585,

It sounds like either the database or the application didn't actually update. Would you be able to make a direct SQL query against the ProGet database that is in the config? I would like you to run two different queries:

select * from Applications_Extended

and

select top 10 executed_date, Script_Guid, Script_Name from __InedoDb_DbSchemaChanges order by Executed_Date DESC

Thanks,
Rich

Hi @jonnpear_8217 ,

I have just updated the documentation for switching to IIS. Please take a look and let me know if you have any questions.

Thanks,
Rich

Hi @huomm23_2930 ,

I'm glad to hear that worked for you! Thanks for giving me an update.

Thanks,
Rich

Hi @huomm23_2930,

In the Basic and Enterprise versions of ProGet, you can simply use the publish role on the users you don't want to have the overwrite permissions on, but in the free version, you can restrict the Administrator role from being able to overwrite your packages. To do this, you will need to perform the following steps:

1. Navigate to Administration > Security > Tasks tab
2. Click the Customize Tasks button
3. Click Administrator in the list
4. Uncheck Overwrite package in the Feeds section
5. Click Save

That should prevent your packages from being overwritten.

Please let me know if this works out for you!

Thanks,
Rich

Hi @Adam,

I have pushed an updated extension that includes this fix. The fix is included in the Windows extension 1.0.18+ and 1.7.1+. You should see an update when you check in Administration > Extensions. Please let me know if you have nay issues!

Thanks,
Rich

Hi @aclauss,

If you go to the feeds page in ProGet, in the upper left corner you should see a toggle button that has two options: Feeds and Connectors. Please click the Connectors button and you should see a list of your connectors. From there you can click Create Connector to create a new connector.

Once you create the connector, you can tie it to your feed by going back to the manage feed page, click connectors, and now that connector dropdown you see before should contain the connector you recently created.

Hope that helps!

Thanks,
Rich

Hi @Jonathan-Engstrom ,

Have you tried wrapping your PSEnsure inside of a with block to force everything to run as a specific Resource Credential?

Example using a resource credential named AdAdminResourceCredentials:

with credentials = AdAdminResourceCredentials
{
	PSEnsure
	(
	    Key: RSHStuff,
	    Value: True,
	    Collect: >>

	        $ErrorActionPreference = 'SilentlyContinue'
	        $TargetOU = 'Users'
	        $Value = (Get-ADComputer -Identity $env:COMPUTERNAME).DistinguishedName -match $TargetOU
	        $CurrentValue = (Get-ADComputer -Identity $env:COMPUTERNAME).DistinguishedName
	        ((($Value -eq $true) -and (($env:COMPUTERNAME) -match 'MyComputerName')) -or ($env:COMPUTERNAME -notmatch 'MyComputerName')) 
	        >>,
	    Configure: >>
	                
	        $samAccountName = $ServerName
	        $newOU = [adsi]"LDAP://$TargetOU"
	        $comp= ([adsisearcher]"samaccountname=$($ServerName)$").FindOne()
	        Write-Host $comp.GetDirectoryEntry()
	                                
	                                >>,
	    Debug: true,
	    Verbose: true
	);
}

Please note that my PowerShell in collect and configure is not 1 to 1 with yours, I would replace my PowerShell script with yours in those two properties.

I also was talking with one of the solutions architects on a similar Otter execution question. He suggested that sometimes it is easier to figure out PowerShell issues by running PSExec under the local system account and attempting to run your PowerShell commands that way. It will sometimes show you more detailed errors than Otter. I would try running PSExec -s powershell, which will open up a PowerShell console as the local system account, and then typing in the lines of PowerShell you have had trouble with.

Thanks,
Rich

Hi @Jonathan-Engstrom ,

What version of PowerShell do you have installed on your SQL server? I'm running 5.1.18362.628.

Thanks,
Rich

Hi @Jonathan-Engstrom ,

I have noticed the $ServerName uses the name of my server configured in Otter, not the actual computer name. Have you tried running the remediation using $env.computername instead of $ServerName? Or can you verify the Server Name in Otter matches the computers name?

Thanks,
Rich

Hi @Jonathan-Engstrom ,

The $TargetOU was intentionally malformed to force a configuration drift. In the Configure property, it did not matter because I did not actually run the MoveTo operation. That was the only thing I didn't actually do. But according to your error:

Exception calling "FindAll" with "0" argument(s): "An operations error occurred.
"
You cannot call a method on a null-valued expression.`.

The only line that could have caused that error was ([adsisearcher]"samaccountname=$($ServerName)$").FindOne(). I actually decompiled the MoveTo method from Microsoft to verify that it did not call a FindAll or FindOne method. I did verify that Microsoft calls FindAll from within FindOne by decompiling FindOne.

If you are trying to verify which line it actually threw the error on, you could use Write-Host in your PowerShell between each line and enable Debug and Verbose on your PSEnsure operation. The execution log will then show you the Write-Host outputs.

Currently, my best guess is that the machine and user combination that you are remotely attempting to run this on either does not have access to query/update the domain or does not have adsisearcher library installed on that machine for that user.

Thanks,
Rich

Hi @Jonathan-Engstrom ,

I'm stumped on this one. I took your Otter script and put it in my local version of otter. I swapped sql for my computer name and didn’t run the actual .MoveTo operation and everything runs perfectly fine in my instance.

Here is my OtterScript:

PSEnsure
(
    Key: RSHStuff,
    Value: True,
    Collect: >>

        $ErrorActionPreference = 'SilentlyContinue'
        $TargetOU = 'Users'
        $Value = (Get-ADComputer -Identity $env:COMPUTERNAME).DistinguishedName -match $TargetOU
        $CurrentValue = (Get-ADComputer -Identity $env:COMPUTERNAME).DistinguishedName
        ((($Value -eq $true) -and (($env:COMPUTERNAME) -match 'MyComputerName')) -or ($env:COMPUTERNAME -notmatch 'MyComputerName')) 
        >>,
    Configure: >>
                
        $samAccountName = $ServerName
        $newOU = [adsi]"LDAP://$TargetOU"
        $comp= ([adsisearcher]"samaccountname=$($ServerName)$").FindOne()
        Write-Host $comp.GetDirectoryEntry()
                                
                                >>,
    Debug: true,
    Verbose: true
);

Thanks,
Rich

Hi @Jonathan-Engstrom

I'm sorry, I should have clarified. I have a local install of Otter and I put that PowerShell inside a PSEnsure operation in my configuration for my server. It ran without issue for me, outside of the small tweaks I had to make to the syntax that I noted earlier. Would you be able to post your Otter Script operation for the PSEnsure block? Also, when you are testing the PowerShell directly, are you testing it using the same account that the otter service (or agent) is running as?

Thanks,
Rich

Hi @PhilippeC

You could create a PowerShell script in BuildMaster and call that from your Otter Script Plan. Here is an example:

PowerShell Script

param ([string] $Path, [ref] $ArtifactFiles)

$files = @()
foreach($name in (get-childitem $Path | where {$_.extension -eq ".zip"}).Name) {
    $files += $name
}
$ArtifactFiles = $files

Otter Script Plan:

Deploy-Artifact Files
(
    To: target
);

PSCall AppName::GetZipFiles
(
    Path: target,
    ArtifactFils => @files
);

foreach $file in @files
{
    # Do something with $file
}

Hope this helps!

Thanks,
Rich

Hi @Jonathan-Engstrom,

I entered your PowerShell locally on my computer this way

$newOU =  [adsi]"LDAP://$TargetOU"
$comp= ([adsisearcher]"samaccountname=$($ServerName)$").FindOne()
$comp.GetDirectoryEntry().MoveTo($newOU)

And that seems to work for me. The main thing I did have to change was adding in parentheses after FindOne and GetDirectoryEntry and I had to change $test to $comp. Is it possible that your PowerShell has the same issues?

Thanks,
Rich

Hi @Jonathan-Engstrom,

Can you try switching up your PowerShell a bit to this:

$samAccountName = $ServerName
$newOU =  [adsi]"LDAP://$TargetOU"
$test = ([adsisearcher]"samaccountname=$($samAccountName)$").FindOne()
$comp.GetDirectoryEntry.MoveTo($newOU)

Thanks,
Rich

Hi @Jonathan-Engstrom ,

Sorry for the delay. I'm working on recreating the error locally. I'll follow up with you once I have more information.

Thanks,
Rich

Hi @Jonathan-Engstrom,

In the log I see:

@Jonathan-Engstrom said in $PSCredential- round two:

Exception calling "FindAll" with "0" argument(s): "An operations error occurred.
"
You cannot call a method on a null-valued expression.

Can you please include the OtterScript PowerShell command you are calling? Also, do you see any errors in the Diagnostic Center found at http://{YOUR_OTTER_SERVER}/administration?

Thanks,
Rich

Hi @Jonathan-Engstrom,

Could you please reply with the log output from the failed PowerShell operation?

Thanks,
Rich

Hi @Jonathan-Engstrom,

You will need to create a Resource Credential named MyCredential and then you can use that within your psensure PowerShell like this:

Move-ADObject -Identity (Get-ADComputer -Identity $env:COMPUTERNAME).ObjectGuid -Server $pdc -TargetPath $TargetOU -Credential $PSCredential(MyCredential)

By using a Resource Credential in Otter, the username and password will be securely stored in the Otter database. Please let me know if this does not work for you.

Thanks,
Rich

Hi @Jonathan-Engstrom ,

I'm sorry about this, but it looks like our documentation was out of date. We just updated our documentation and this should help to answer your question. $PSCredential now can be called in two different ways; with a username and password or by passing the resource credential name.

In order to use a resource credential within your PowerShell, you would just need to pass your resource credential name. For example:

# convert a resource credential named MyCredential to a PSCredential object
PSCall MyPowerShellScript
(
    Credentials: $PSCredential(MyCredential)
);

Please let me know if this doesn't work for you.

Thanks,
Rich

Hi @mhull_0872,

We just released the latest version of ProGet 5.2.26 on 2/28/2020 which includes the new Package Deployment API. You can learn more about the package deployment tracking feature here. We also have documentation on how to use the API endpoint here.

Thanks,
Rich