Posts made by atripp

Hi @Jon ,

Unfortunately it's "not that simple", and this will require a bit of troubleshooting to figure out what the issue is, precisely. There are a lot of areas where this could occur, and we can't add a generalized "job killer" until we understand what the issue is.

You'll have to dig in behind the scenes (Admin > Executions) and identify exactly where things are freezing. The last log message will indicate that. Try to find as many examples as possible.

Keep in mind that Otter is not not really "connected" to a server, and a server does not (and cannot) "call home". Instead, Operations (i.e. OtterScript) opens a connection to a server, sends a command, then disconnects. All network errors we've ever seen in this process will yield a crash.

However, unless explicitly specified, a command will not timeout. So this means if you run a PowerShell script that basically just says "sleep indefinitely", then the Execution will never complete. Obviously no one would write that script, but some PowerShell scripts have a consequence of that. No built in Operation should ever cause that to happen, which is why we need to know precisely where this is happening.

Best,
Alana

Hi @dario-wigger_0347,

The configuration looks okay, and is basically what we have documented. So I think it ought to work...

From here, I would start by monitoring the HTTP traffic between Maven and ProGet. You can use a tool like Fiddler Classic on Windows or Wireshark. What you should see is:

• Maven issues a request, and ProGet returns a 401 with WWW-Authenticate: Basic header
• Maven reissues the request, sending a Authorization: Basic XXXXXXX

The XXXXX will be Base64 encoded api:your-api-key. If you don't see that, then something else is wrong. I would post it and maybe we can help.

You can also try downloading maven artifacts from an "Incognito" window. Your browser should also prompt you for a username/password, and then you can enter api and your api key.

Thanks,
Alana

Hi @PhilipWhite ,

Thanks for confirming that; after researching this, it seems that mounting /tmp with noexec is "old-school security measure" to prevent issues that have since been fixed, but is not an uncommon practice today. So we will make a code change to have Use Agent Temp folder instead of the system temporary directory for extension files extraction - this will be implemented via BM-3927 , hopefully in the next maintenance release (Tomorrow).

As for the Centos7 server, the underlying error message would be buried in the sshd logs. My guess is something with noexec, but who knows?

The Linux integration in BuildMaster v4 and BuildMaster 2022+ are a bit different:

• BuildMaster v4 just issues SSH commands and transfers files
• BuildMaster 2022+ transfers an agent executable, then uses SSH to instruct the agent to run the commands

So this added complexity seems to be causing some issues, but we'll figure it out!

FYI: eventually we intend to publish a Linux-version of the Inedo Agent, primarily to make it easier to configure than SSH.

Best,
Alana

Hi @phwhite_9282 ,

We're a bit baffled by this one, and think the issue is related to "some kind of configuration" on your RHEL server and the /tmp folder. Likely, security related. For testing purposes, can you make the BuildMaster user account (i.e. what you SSH in as) a root user?

Here's some more context... to help troubleshoot this further.

The first error was clearly related to some "strange" SSHD rejection (problem initializing a process), but changing the agent's temp path fixed it. We don't know what that error is specifically... but behind the scenes, BuildMaster starts by transferring executable files to the Agent's Temp directory (which clearly succeeded) and then executing those files (which failed).

The second error is an error loading a file from disk ('/tmp/Inedo/ExtensionCache/fafcab6c8d528766939b379f9e8f3dc9ba44db15/package/runtimes/linux-x64/native/libgit2-a2bde63.so). This usually means the file isn't on disk -- do you see that file on disk? The agent would have unpacked it there.

Our working theory is that some kind of security configuration is quarantining things that are written to /tmp and preventing execution. Which is unusual, because that's a common place where basically everything uses as a temp space.

Thanks,
Alana

Hi @phwhite_9282,

This is a low-level error, and could mean a number of different things on the SSH side.

Can you let us know:

1. What version of BuildMaster are you using?
2. What is the operating system of the linux server?

Thanks,
Alana

Hi @hanna-doeberl_5189,

You should be able the operating system on your ProGet server without impacting the installed applications (like ProGet); we're not aware of any issues/problems arising with ProGet specifically.

As for upgrading ProGet itself, check out the docs on that:
https://docs.inedo.com/docs/proget-upgrade-guide

Cheers,
Alana

Hi @richard-allen_8963 ,

I'm afraid I'm really not sure how different versions the NuGet client tools behave in different scenarios... just really how ProGet works in implementing the Vulnerabilities information on the NuGet API.

This API is used by Visual Studio, and in our testing, they show up as we expected -- so if you're not seeing the desired behavior, that's where you'd want to look. Using something like Fiddler, you can see what's being called.

I do know there is an older, NuGet.org-only API that the tools may be calling. But we only implement the one I mentioned above.

Best,
Alana

Hi @Justinvolved ,

That looks like an error coming from PowerShell; you can see what Powershell commands are being emitted executed, so I'm guessing it's some kind of issue with the source?

I'd try to start there, and see if you figure out the underlyinig Powershell error. Here is the code that's generating the scrpt

https://github.com/Inedo/inedox-scripting/blob/master/Scripting/InedoExtension/Configurations/PsModule/PsModuleConfiguration.cs#L224

Cheers,
Alana

Hi @richard-allen_8963 ,

In ProGet Free Edition, if vulnerabilities are enabled on the feed, a package will a vulnerability will always appear as critical, and the advisory URL will direct you to /vulnerable-nuget-package-info on your instance of ProGet.

In paid editions, you will get more relevant information and you can assess these vulnerabilities to control them from being displayed as critical.

I'm not entirely sure how to suggest to debug, but in the API, you should see advisoryUrl
listed on thePackageDetails` resource, along with deprecation as well, so I would try looking at the API for a sample package, and making sure you can see both deprecation and advisory info

Thanks,
Alana

Hi @caterina ,

Sadly I wasn't able to fix w/ that last release (I didn't have a test case to try, so kind of a blind fix), and was going to give it proper shot this weekend,.... but it looks like you guys submitted a pull request? Did that releaae solve the issue?
https://github.com/Inedo/pgscan/pull/46

Thanks so much,

Alana

Hi @caterina ,

We recommend migrating to the standard format (as they call it now) , which is .snupkg.

The .symbols.nupkg format is considered legacy, and the "embedded" format that we used to recommend was effectively a varation of the old format.

Cheers,
Alana

Hhi @caterina ,

What does the configuration of your Sybmol Server look like, on the Manage Feed page?

Thanks,
Alana

Hi @richard-allen_8963 ,

ProGet 2023 will return vulnerabilities when using the audit command and Visual Studio. Here is more information on our general approach:
https://blog.inedo.com/nuget/vulnerabilities/

If you're not seeing them, make sure you're using the NuGet v3 API endpoint. That's the one that ends in index.json -- the v2 API does not support vulnerabilities.

There is a restriction on vulnerability inforamtion in the Free Version, but they are returned in the API.

Thanks,
Alana

Hi @philippe-camelio_3885,

We didn't change the operation that I can tell, but I researched/tested it using the CmdLets that Otter invokes: Register-PSRepository and Unregister-PSRepository .

Here's what I discovered:

• Unregister-PSRepository -Name PSGallery is fine
• Register-PSRepository -Name PSGallery -Source ... is not allowed at all
• Register-PSRepository -Default -Source ... is also not allowed
• Register-PSRepository -Default is apparently the only allowed option

You can see the Unregister/Register approach that we use in PsRepositoryConfiguration.cs#L187. So that's why you get this error....

How about doing this as a work-around?

{
    Ensure-PsRepository
    (
        Exists: false,
        Name: PSGallery
    );
    PSExec Register-PSRepository -Default;
}

That should have the same effect of restoring to the default values...

Thanks,
Alana

Cheers,
Alana

Hi @philippe-camelio_3885 ,

I just tried this, and noticed that the connector is timing out. This is the query that ProGet makes when you type that in, and it takes 13-15 seconds when I click the link:
https://www.powershellgallery.com/api/v2/Search()?$top=25&$skip=0&searchTerm='VMWARE.PowerCLI'&targetFramework=''&includePrerelease=true&semVerLevel=2.0.0

That's longer than the default (10s) time out, so I guess this result is expected. If you increase the timeot to 20 seconds, i guess it should work.

Cheers,
Alana

Thanks @philippe-camelio_3885; can you walk me through the scenario of what you're doing? The "Error" status doesn't seem quite right....

Otherwise, the "status" on a server can be a complicated thing in Otter sometimes, since it represents a few things.

Under Admin > Service, you can manually trigger some of the Task Runners that are responsible for changing status. Can you identify which one needs to be run in order to get Otter to report the desired status.

Once we understand what you're doing, and what task isn't being run by the service on time, we can try to fix

Hi @philippe-camelio_3885,

We're aware of this issue, and it's related to some bugs/quirks with the search results from the PowerShell Gallery API (v2). It's not really feasible to work-around, and unfortunately Microsoft will not fix the issue because it should be fixed in PowerShell Gallery API (v3).... which is due any year now. Its ~3 years late.

The results are suboptimal in both ProGet 5.3 and ProGet 2023, but ProGet 2023 is worse. This has to due with some performance optimizations we made with not paging as many results.

Fortunately this only impacts the search api, which is mostly limited to the ProGet UI. As long as you know the exact package name, you can naviage and install it fine.

Cheers,
Alana

Hi @philippe-camelio_3885 , what was the previous version? Otter 2022?

Hi @caterina ,

We just published pgscan 1.5.10, and I believe that it will solve this issue. Can you give it a shot?

Thanks,
Alana

Hi @Justinvolved

It's been a while, but I believe we investigated this a little while back, but discovered it wasn't possible/feasible to do with the API we were using. That would have been in the .NET4+ days, not .NET6+, so it may have changed.

I don't remember

Here's the code we use to bind the Otter configuration to Microsoft.Web.Administration libraries: https://github.com/Inedo/inedox-windows/blob/master/Windows/InedoExtension/Configurations/IIS/IisSiteConfiguration.cs

It might be trivial now, and we'd be happy to accept a pull request if you're able to figure/try it out!

Cheers,
Alana

Hi @Justinvolved ,

Application Pools seem to run under a special user account (not NETWORK SERVICE, but like IIS\AppPoolName), so you need to give that user access.

In any case, make sure to restart the app pool/IIS, otherwise the permissions may not cascade? It can be a bit tricky unfortunately.

Cheers,
Alana

Hi @caterina ,

Thanks for sending this; looking over the SBOM, it looks like the scope is incorrectly encoded in the package.

• Correct: <purl>pkg:npm/%40ampproject/remapping@2.2.1</purl>
• Incorrect: <purl>pkg:npm/@ampproject/remapping@2.2.1</purl>

Are you using pgscan to generate this? I think so, and in that case it seems to be a bug. We may wish to have ProGet also accept it, but I'm not sure.

FYI, here is a minimal SBOM that demonstrates the issue:

<?xml version="1.0" encoding="utf-8"?>
<bom serialNumber="urn:uuid:6109c18a7f7c403b9f9f11ff73c8fe34" version="1" xmlns="http://cyclonedx.org/schema/bom/1.2">
  <metadata>
    <timestamp>2024-01-17T08:23:58.4621303Z</timestamp>
    <component type="application">
      <name>Test</name>
      <version>1.0.0</version>
    </component>
  </metadata>
  <components>
    <component type="library">
      <name>@ampproject/remapping</name>
      <version>2.2.1</version>
      <purl>pkg:npm/@ampproject/remapping@2.2.1</purl>
    </component>
  </components>
</bom>

When npm/@ is replaced with npm/%40, the file is imported as expected.

Cheers,
Alana

Hi @Srinidhi-Patwari_0272 ,

In your command, you're specifying D:\ProGet as the target directory. The configuration files are written to %PROGRAMDATA%\Inedo\SharedConfig\.

Cheers,
Alana

Hi @andy222 ,

Based on the error/stack trace, it looks like you're getting a HTML page as a response instead of the expected JSON. I don't know what url is being queried, but it's clearly incorrect.

Looking at the OtterSCript, you shouldn't be specifying an ApiKey and EndpointUrl when also specifying a Source, since BuildMaster will automatically discover those from the source. I suspect this is where the issue is.

Try this instead:

ProGet::Download-Asset temp/TestEnvVariables_Win.zip
(
    Source: directory::Assets,
    To: C:\temp
);

Hi @mike_5084 ,

Thanks for sharing this! Few things I wanted to share...

1. Retention Rules are definitely the way to do house keeping; it's much simpler and is something that just works in the background

2. We're definitely aware of this particular deletion issue, but it's tricky to reproduce (even using your scenario)... that said, we do have a patch for FeedPackageVersions_DeletePackageVersion available, but we're waiting for some other users to test it before releasing in the mainline. It sounds like you already deleted the packages, so it's not a big impact.

3. We introduced the Common Packages in ProGet 2023, and we will likely introduce a "Rate limit" of some kind in ProGet 2024 Free edition, since retention policies are one of the features that help "sell" ProGet to managers

Cheers,
Adam

Thanks @caterina , confirming receipt! We will review this in the next week or so, please stay tuned.

@philippe-camelio_3885 that's a lot of namespaces! Sure, great idea, I logged it to BM-3922

I'm not sure if we can sort repository names however; those come in a "paged" UI, but we might :) At least you can search for those by typing.

Hi @andy222 ,

The workflow you describe -- TeamCity (CI) + BuildMaster (CD) -- is definitely supported.

However, artifacts are "pulled" (imported) from a TeamCity build. There are two patterns:

1. BuildMaster "monitors" TeamCity projects for new builds and imports builds that meet certain conditions you define; see TeamCity Project Monitors
2. TeamCity kicks-off an import in a BuildMaster application using the API; see Automatically Importing to BuildMaster after a CI Build

That being said, you can also create a TeamCity (CI) + ProGet (Package Repository) + BuildMaster (CD) workflow. This scenario is the closest to the "Octopus Deploy" workflow (i.e. TeamCity pushes a NuGet package), and a number of users will use this as a kind of "bridge" to migrate away from Octopus.

Cheers,
Alana

Hi @caterina ,

Can you share an SBOM that I can use to reproduce this? Doesn't need to be the whole one, just one that contains an example of a scoped package that you're not seeing.

Cheers,
Alana

@philippe-camelio_3885 thanks for the bug report!

we'll get fixed via OT-506 in an upcoming maintenance release :)

Hi @sbaeurle ,

I'm glad to issue is mostly resolved now!

In general, ProGet issues 500 errors when there are problems like database connections, overload, etc. We have not seen a 401 issued in a case like that. That's not to say it didn't happen to you, but that's not how ProGet will behave.

401/403 are only issued on API endpoints calls; if there's an error during authentication (like if you added an intentional bug to a procedure that checks the api key validity), then a 500 should be issued.

Cheers,
Alana

Hi @Justinvolved ,

This is a symptom that the web application lacks permission to control the service. There is an ancient tool called subinacl.exe that can fix this. We also have an ancient article based on that tool, but don't have anything updated:
https://inedo.com/support/kb/1090/granting-service-control-privileges

I did a quick search and found this content piece that also explains it
https://www.winhelponline.com/blog/view-edit-service-permissions-windows/?expand_article=1

Let us know what you find, we'd love to get that article updated :)

Cheers,
Alana

Hi @manager_7279 ,

This is likely because of a lack of database permissions; you'll need to grant permissions to the BuildMaster database in the local SQL Server database. We don't have a step-by-step guide for that, but it involves giving that user BuildMasterUser_Role permission using something like SQL Server Management Studio.

As an alternative, you can install the Inedo agent on the BuildMaster server (defaults to Local System) and then connect to that instead of localhost.

Cheers,
Alana

Hi @manager_7279 ,

Stopping/Starting IIS application pools requires administrative access. Looking at your stack trace, it seems like you're trying to stop an application pool on localhost (i.e. the server that BuildMaster is running on)?

That's not a common requirement; instead you'd stop/start an application on a different server.

In any case, the BuildMaster service runs under NETWORK SERVICE, and will not have permission to modify IIS application pools on localhost. So if you plan to do this (i.e. stop/start IIS on localhost), then you will need to make the BuildMaster server run under LOCAL SYSTEM instead of NETWORK SERVICE.

This will likely also require you to update database permissions, to allow LOCAL SYSTEM to connect to the database as well.

Alana

Sounds like a plan :)

Let us know if we can help , or if you run into any issues!

Hi @sbaeurle ,

That's definitely strange; there's really nothing in ProGet that would cause intermittent 403s.

If you aren't doing so already, I would suggest switching to using API Keys in ProGet instead of LDAP. Make sure they are not "Personal Keys" but Feed keys. That will eliminate any potential issue with LDAP queries returning inconsistent results.

Otherwise, you'd almost need to capture those bad requests and see how they're different. ProGet does not have a means of capturing HTTP traffic, so you'd have to use something like Fiddler or Wireshark.

Cheers,
Alana

@scampbell_8969 excellent, please stay tuned, we'll get back within a couple weeks

Hi @sergio-gonzalez_0157 ,

The easiest solution to this is to set the parent folder's as you'd desire. For example, if you're deploying everything to c:\Websites\<app_name> then just set c:\Websites to be [COMPUTER NAME]\Users .

You can also write a PowerShell script to change permissions, but that's more complex and I recommend just using the parent-folder

Thanks,
Alana

Hi @scampbell_8969 ,

The screenshot is not yet available, it's code that we're testing.

In the meantime, you need to navigate to the package in a ProGetfeed, then assign the license. It will add a special url like package:// that is used to associate the package with the license.

Can you email the files to support at inedo dot com, with the subject [QA-1368], then we can ffind it? Just let us know when you email the files

Hi @scampbell_8969,

It's hard to say without knowing which specific packages you're referring to, but there are several known issues with ProGet 2023's license detection. It is something we're currently redoing in ProGet 2024. We plan to get this new detection logic in ProGet 2023, at least as a preview feature, in the coming weeks

Here's a screenshot of working code:

I assume these are all publicly available packages. Can you share the SBOM files for your Releases? This will be extremely valuable for us to test with.

It's kinda hard to find packages with embedded licenses to be honest.

Thanks,
Alana

Hi @philippe-camelio_3885 ,

I think I understand your question; but just to make sure...

You have two servers with the inedo-runner role, and you're using the Acquire/Release operations like this:

Acquire-Server
(
    Role: inedo-runner,
    AcquiredServer => $AcquiredServer
);

###### code that uses $AcquiredServer some how #####

Release-Server
(
    Role: inedo-runner,
    Server: $AcquiredServer
);

However, you'd like the same server to be acquired more than once, so that you could run two or three simultaneous operations on inedo-runner servers, instead of just one?

I'm afraid that's not possible, and isn't something we plan to add to the roadmap any time soon. One option is to create a second role inedo-runner2, and then use that to Aquire servers half the time

Cheers,
Alana

Hi @mike_5084 ,

Note that running ProGet as a server cluster will require a ProGet Enterprise license. So I think you may want to use Azure Container Services? That's usually what we see most ProGet Basic users use on Azure (aside from a VM).

Under the hood, Azure Kubernetes/Container Services uses Docker. We support ProGet running on Docker, and have some instructions which you've certainly seen:
https://docs.inedo.com/docs/installation-linux-docker-guide

However, we don't support Azure directly, as we're not familiar enough with Azure to advise what Azure-specific commands are required, how to troubleshot Azure-errors, what networking configuration is required on Azure, etc.

Contrast this with Windows, where we can typically advise on how to do nearly all Windows admin/troubleshooting things, from managing the Windows certificate store to changing service configuration using sc.exe, etc.

That being said, if you can "translate" our documentation to Azure, and are comfortable setting it all up, then you should be fine.

Alana

Hi @jw,

According to the comments in our code...

   // kestrel will not do port sharing in .NET6 (despite what docs imply), so use http.sys
   // Revisit in .NET8 - https://github.com/dotnet/aspnetcore/issues/39640
   foreach (var u in urls)
   {
       var m = urlRegex.Match(u);
       if (m.Success && m.Groups[1].Value is not "*" and not "localhost")
       {
           useHttpSys = true;
           break;
       }
   }

This was something that was considered for .NET7, but clearly that never happened :)

The general suggestion is to use a reverse proxy to reuse the ports.

Cheers,
Alana

Hi @stanislav-halchuk_7538 ,

Thanks for reporting this; we'll get this fixed the next maintenance release (shipping this friday) via PG-2544

Alana

Hi @chowarth_6088 ,

Can you give some more details about the errors that you're receiving, and how to reproduce this?

Thanks,
Alana

Hi @Justinvolved ,

The biggest consideration for using "separate applications" is versioning and independent deployment. For example, our extensions are all separate applications, but nearly all of them are bundled with product release. For example, see the the the %ExtensionVersions variable on a recent build of BuildMaster.

The Git::Checkout-Code will default to $Commit and $Repository variable functions for the BranchOrCommit and From arguments; those variable functions contain the values that you selected when creating a build. So you can just specify different values as needed.

See: https://docs.inedo.com/docs/buildmaster-git-source-control#build-scripts-operations

Likewise, the Deploy-Artifact operation defaults to $ApplicationName, $BuildNumber, Default, etc, so you can specify values as needed.

But like all things in software, no matter what approach you use, it will probably end up being "wrong" and you'll want to redo it later. So just plan on doing a "v2" down the line :)

Alana

Hi @sbaeurle ,

That is strange; ProGet will issue a 403 when the request is unauthorized, meaning that authentication (name/password) was correct, but the user/key doesn't have permission. That isn't really an error that would happen intermittently on the ProGet side.

It's possible that ProGet isn't issuing a 403 error. Is there some kind of authentication in front of ProGet, on the azure side? ProGet does not log these requests, so your best bet may be to monitor traffic and see if you identify what is issuing the 403.

Best,
Alana

Hi @devadmins_4004,

I haven't tried to reproduce this yet, but I looked the code briefly and am not sure why it wouldn't work. Is this ProGet 2023?

What is the specific response? Response code + body if possible.

I wonder if you could try to use the username of api and the password of your api key?

Thanks,
Alana

Hi @cshipley_6136 ,

We recommend using the /health monitoring endpoint. As you can see, it makes a connection to the database by calling that stored procedure.

We do not recommend calling other URLs; and there is no reason another page would work. Calling other pages will impact performance and likely yield incorrect results.

You're right, it might have to do with a "connection pool" problem -- but that's all managed at the driver level. You can control connection pool settings by changing connection string values, but to be honest we have no idea what the impact will be.

Your best bet is to open a case with Microsoft, who has the expertise to know how to troubleshoot these intermittent "error 40's". They may have specialized tools or "secret debugging commands" you can use.

This is all happening at the operating-system or driver-level, and while we're doing our absolute best to help you troubleshoot, but we have never seen this happen before and our main resource is searching google. And this is not an easy error to search for,..as you've probably noticed.

Please keep us in the loop.

Best,
Alana

Thanks @cshipley_6136 ; they're both the same "error 40" - which is a hostname/dns issue.

I doubt its a bug in the SQL Server driver (but it could be)... the most likely culprit is some kind of networking/configuration issues.

I would try to identify what's happening around that timestamp, network changes, etc. This is where our expertise in troubleshooting is not so strong. It's a really easy error to reproduce (just enter a wrong host name in the connection string, disable dns, unplug a network cable, etc), but tracing the underlying cause requires lots of poking around on the network.