Posts made by atripp

Hi @norm-ross_1437 ,

I'm not sure what version of ProGet you're using, but this sounds familiar and I'm almost certain we fixed this specific case a while ago. So I would try to upgrade.

I don't realy recommend playing around with data in the tables....

Best,
Alana

Hi @v-makkenze_6348 ,

I'm not sure what version you upgraded from, but there hasn't really been many changes that would have caused this I think.

This is a general database timeout error, which could be happening under heavy load (so trying again should work), or it could be a result of needing to update statistics, etc.

Thanks,
Alana

@lucas-almeida_8120 in that case, I would use a import path - https://docs.inedo.com/docs/proget-bulk-import-with-droppath

Hi @scott-wright_8356 ,

Thanks for sharing that; I think I see where the issue is.

This is recent regression... can you run the 1.FeedPackageVersions_ExtendedWithMaven.sql script that's attached to PG-2604? I think that will fix the issue.

Thanks,
Alana

Hi @scott-wright_8356 ,

That's strange; I'm not sure how that's possible. There's clearly some bad/unexpected data in the maven tables, and I'm not sure how that happened. Maybe an error/bad API call? I don't know.

Can you help us identify this by running EXEC Packages_GetPackages or SELECT * FROM FeedPackageVersions_ExtendedWithMaven, and seeing if you can spot any rows with with a null Package_Name`?

Specifically, I'm looking for something that would cause this below function to crash. The only thing I can think of is a null Package_Name.

        public static string BuildUrl(FeedId feedId, string group, string name, string version)
        {
            var builder = new StringBuilder();
            builder.Append("/feeds/");
            builder.Append(Uri.EscapeDataString(feedId.Name));
            builder.Append('/');
            if (!string.IsNullOrWhiteSpace(group))
            {
                builder.Append(Uri.EscapeUriString(group));
                builder.Append('/');
            }

            builder.Append(Uri.EscapeDataString(name));

            if (!string.IsNullOrEmpty(version))
            {
                builder.Append('/');
                builder.Append(Uri.EscapeDataString(version));
            }

            return builder.ToString();
        }

Hi @forbzie22_0253 ,

We are aiming to release ProGet 2024 in early April; here are the upgrade notes in progress, which will give some idea of what's coming: https://docs.inedo.com/docs/proget-upgrade-2024

Best,
Alana

Hi @lucas-almeida_8120 ,

This error is indicating that your database not being upgraded along with the rest of your installation. Or is otherwise corrupt. I can't imagine how that could happen, so unfortunately you'll have to do some digging on this.

Here is where the connection string for the ProGet database is stored:
https://docs.inedo.com/docs/installation-configuration-files

If you review the installation logs in the Inedo Hub, you will see what database/connectionstring is being used.

Good luck, and let us know what you find!

Alana

Hi @scott-wright_8356,

This is usually due to a networking error; it may also be resolved by restarting. It's also just something you can ignore. The service messanger isn't very important in ProGet.

Best,
Alana

Hi @scott-wright_8356,

When installing ProGet with the Inedo Hub, you can specify the SQL Server and Database Name. If it's an existing ProGet database, then it will be upgraded/downgraded as needed. I'm don't know what the errors were, but typically the installing users has a lack of db_owner permissions.

To upgrade ProGet in an HA set-up, you can just use the Inedo Hub to upgrade each node. Usually it takes 1-2 minutes per node, so you can do it manually pretty easily. There shouldn't be any errors.

Best,
Alana

Hi @scott-wright_8356 ,

This error means that "something" that is taking an exclusive lock handle on files within your SMBShare. Unfortunately, the operating system does not provide information about what that "something" is, but it's typically an anti-virus, search-indexing, or back-up tool.

ProGet uses shared read handles, but obviously needs a lock handle when writing. There are a few race conditions where this could occur, but they are extremely rare and require multiple requests on different servers in a cluster that attempt to add the same package at the same time.

However those are very rare, so I suspect "something else" is locking those files. Usually it's an anti-virus tool - since it "looks like" a zip file, it's being opened? You'd need to use a tool to monitor openarting system handles, like sysinternals procmon.

Best,
Alana

Hi @forbzie22_0253 ,

We do not currently have an API Key API, but this is something we are planning for ProGet 2024.

Today, you'd need to use the Native API. Since you're automating an installation, it's easeist to just use SQL to call those API/Stored Procs, since you'll also want to modify other properties like license key, etc.

Best,
Alana

Hi @proget-markus-koban_1462,

Thanks for clarifying; the "User directory" is outdated and I removed that from the documentation. In old versions of ProGet, you'd need to set up a separate directory. Now they are just added as Built-in users.

The Trial version should allow you to configure SAML. It sounds like the issue is that Keycloak is not redirecting to ProGet. I really wish I knew how to help troubleshoot... but after logging in, Keycloak is supposed to POST to /saml-acs-callback in the ProGet instance. I wonder if there's n additional thing that's not configured on Keycloak?

Best,
Alana

Hi @hashim-abu-gellban_3562 ,

This is just a user-friendly name for a URL within the Azure service, and isn't sent to ProGet.

You use practically anything I'm sure... we suggest proget-saml but anything is fine.

Alana

Hi @redbaron2_9872 ,

This error is unrelated to removing packages.

Based on the stack trace, it's coming from the Package Policies Preview Feature that was introduced in ProGet 2023.30. Not sure exactly how that's possible, but that code should not be running unless the feature was enabled under Admin > Policies.

Please let us know if that was the source of the error.

Best,
Alana

Hello,

I'm not familiar with Keycloak to give you a direct answer, but ProGet can integrate with LDAP and/or SAML protocols. Both of those will require configuration within ProGet, under Admin > Manage Security.

https://docs.inedo.com/docs/various-saml-overview
https://docs.inedo.com/docs/various-ldap-v4-advanced

Best,
Alana

Hi @v-makkenze_6348 ,

Can you try it a few times? Hopefully it will work eventually. It typically takes only 5-10 seconds, but it looks like that procedure is using the buggy SQL MERGE statement, which has issues on a variety of SQL Server versions...

We will rewrite that procedure in a future release.

Cheers,
Alana

Hi @moez-ul-hasan_3818 ,

ProGet only supports SAML-based authentication; as you noticed, we recommend and document using SAML for Azure. So I would just recommend creating a SAML-based Enterprise App for that.

OpenID is a totally different protocol, and our products do not currently have support for that.

Cheers,
Alana

Hi @forbzie22_0253 ,

Not really... we have tried it from time to time, and like your own testing, it seems to work okay?

In general, both the PowerShell Gallery and the client tools (PSResourceGet, PSGet) are "quirky", and we do our best to document this behavior for ProGet users. We link to several open GitHub issues in that documentation, and you've spotted several of other "quirks" logged in GitHub as well.

The issue isn't really about ProGet, but the fact that PowerShell tooling is designed to work exclusively with PowerShellGallery.com. Everything else is a a secondary use case, which is why you'll see a lot of issues logged with non-PowerShellGallery.com usage (ProGet, ADO, etc).

PowerShelllGallery.com is "based on" the NuGet Server API, but it's not documented how or where it diverges. Sometimes the PowerShell team will fix the "divergences", other times they won't. If we can work-around the "divergence" then we will, otherwise we just call it a "quirk" and document it.

Hope that helps,
Alana

Hi @mlorenzschleipen_5169 ,

Thank you for the reproduction information; I was able to reproduce this and we will fix via PG-2578.

It's a bit close to our maintenance release window (we ship later today), so it's scheduled for PRoGet 2023.30 (May 1). Let us know if you are interested ina pre-release, and we can get it as soon as available.

Thanks,
Alana

@forbzie22_0253 performance should be about the same, correct

Hi @forbzie22_0253 ,

SQL Server Express is the default database that we ship with ProGet, and is fine for many small/Basic installations. Once you go into a High-availability and Load-balancing configuration, higher editions of SQL Server are strongly recommended.

Thanks,
Alana

Hi @dan-brown_0128 ,

The latest version includes a migration tool, so I would recommend using that.

The URL has been changing throughout ProGet 2023, but you will be safe whitelisting cdn.inedo.com and security.inedo.com. This is only required for downloading updates, as it's an offline database that ships with ProGet.

Cheers,
Alana

Hi @forbzie22_0253 ,

ProGet Free edition may be used in commercial use. There is no limit on the number of instances, but a ProGet Free instance may not connect/network to another ProGet instance -- they need to be stand-alone instances.

ProGet Free instances can be connected to NuGet.org, just not other instances of ProGet.

Best,
Alana

Hi @scott-wright_8356 ,

Based on the stack trace, it looks like the timeout is occurring while running the Dashboards_GetLargestPackages procedure. We've tested that with feeds with 100K to 1M packages with no issues.

Suince you were digging into other SQL performance issues, can you try running that in SQL Server? Specifically Dashboards_GetLargestPackages 1, 10

Thanks,
Alana

Hi @scott-wright_8356,

We haven't seen any other issues with this procedure in particular, but it's something we can consider to update.

Have you considered that doing the commit with UPDLOCK and SERIALIZABLE could be causing issues with AlwaysOn and sycnchronsist commits going to the secondary?

One thing we faced with ProGet 2023's new database model was handling different bugs in different version of SQL Server's analysis engine. Without this pessimistic lock, some versions of SQL Server 2019 will deadlock while updating totally unrelated indexed view.

That said, you shouldn't need to use synchronous commits with ProGet in AlwaysOn. That's going to slow things down a lot to begin with, and that level of data-integrity is so important in ProGet.

Thanks,
Alana

Hi @yogeshshines_9136 ,

This should be resolved by downloading the latest Inedo Hub from https://my.inedo.com/downloads

Thanks,
Alana

Hi @Justinvolved ,

I've never seen that error and it makes no sense. It's a random Windows COM error. It's not BuildMaster-specific, it's happening when invoking the MWA libraries to save IIS configuration. This would happen if you performed the identical action from with IIS Manager as well, wrote a PowerShell script, etc.

I searched "a specified logon session does not exist. it may have already been terminated" and the advice is all over the place. It has something to do with permissions, I guess? Adding "IIS" adds more specific results, so maybe that will help.

So my advice from here is to just search and try random things that people said work. Feel free to share what you found !

Best,
Alana

Hi @sbaeurle,

I'm afraid we'll be sticking to following SemVer2 for the foreseeable future. That's a very well-documented standard that is machine readable and predicable. Microsoft's versioning scheme is anything but that.

That said, Microsoft's container tagging isn't at all consistent, as you can see: https://hub.docker.com/_/microsoft-windows

The operating system versioning is even more bizarre, and Microsoft abandoned any sensible Major/Minor scheme in the late 1990's. For example, Windows 7 was 6.1, Windows 8 was 6.2, Windows 10 is 10.0, and Windows 11 is.... 10.0. S Except when it's aliased as 23H2 something. Servers are.... no one really knows.

Then add to that patching, which may or may not impact the version number.

To clear up this confusion, we recommend you use SemVer to make versions of your internal base images, based on some Microsoft build.

This is the mapping scheme we recommend:

• Major is the product number (19 for Server 2019, 22 for 2022)
• Minor is the "service pack" version (0 for first version, 1 for 1903, etc)
• Patch is your internal revision number of that image (when you install patches, etc)

Another "tip" is overloading digits. For example, if you see a case where you will want to "revise your patches", then just increment your patch version by 10. So 0, 10, 20, 30, 40. Then you can "patch" 10 by going to 11, 12, 13.

Hope that helps

Cheers,
Alana

Hi @PhilipWhite ,

Thanks for clarifying!

Long story short, you probably want to just use $PipelineStageName. In your case, it would be which would be "Test" (based on the screenshot that says Test Stage).

Overall, it's a little confusing but Pipeline Stages and Environments are orthogonal concepts:

• Environments are used to group servers and scope permissions
• Stages define a phase in the deployment/release process (i.e. a pipeline)

A Deployment Target (e.g. "Deploy to XXXXX") is part of a stage, and may be associated with an Environment. Just click "edit", then check "Environment-specific permissions", then select an environment.

That would put an environment in context (and thus $EnvironmentName would return what you expect), but more practically it would:

• throw an error if any of the servers used in the execution are not a part of that environment
• not allow someone to press the "deploy" or "force" button to that stage , unless they have permissions specific to that environment

It's a little confusing at first, which is why we "kind of" hide environments from the user.

This is most definitely a common point of confusion, so we are really open to feedback if you can think of how to improve documentation/user experience!!

Alana

Thanks @dongjie789_8066; of course we can definitely consider creating a different API, but you are first non-Edge user to ask about it :)

It's best to handle that through your My Inedo account with your work email, so we can work with your team/company about it

Hi @PhilipWhite ,

Can you clarify what you mean by empty?

Like, if you were to do Log-Information Hello $ApplicationName!;, it would just log Hello!? Or are you seeing it somewhere else?

There is an $ApplicationName "variable function" that will return the name of the current application in context. There is almost always an application in context, unless you're doing something like a system-level scheduled job or something.

However, variable functions have a low precedence, so if you were to do something like set $ApplicationName = whatever;, it would create a runtime variable called ApplicationName, and then $ApplicationName would resolve to that.

You can explicitly invoke a function by doing $ApplicationName(). So, Log-Information Hello $ApplicationName()!; should always work.

As for nothing showing up on that application page.. that's weird --- but something to dig through separately for sure.

Cheers,
Alana

Hi @PhilipWhite ,

You're correct, this is a regression in the validation code...

if (!value.All(c => (char.IsLetterOrDigit(c) && char.IsLower(c)) || c == '.' || c == '_' || c == '-' || c == '/'))
    return new ValidationResults(false, $"Docker Repository names may only contain lowercase letters, digits, periods, underscores, slashes, or dashes.");

I guess we recently added char.IsLower because UpperCase characters caused all sorts of problems.... easy fix, and easy work around.

1. Just enter blah in the Repository name field, Click Save
2. Click "view all"
3. Click on the Docker repository in that view, edit

We'll get this fixed for teh next maintenance release via BM-3932

Cheers,
Alana

Added to our PRoGet 2024 roadmap to ivnestigate/explore - it might be easier this time!

ProGet 2023 introduced some big changes to replication, and since it's something that's infrequently configured, we did not create an API for it yet. We have some undocumented/special APIs that are used by ProGet Edge Edition, which relies heavily on replication configuration... but we want to work w/ users on this.

I couldn't find a ProGet Enterprise license associated with your email address, but if you can submit a ticket with your license key, I can make sure that someone reaches out to discuss this with you.

Hi @PhilipWhite

The Docker:: operations are intended to work with a Docker Repository Connection that you've configured for the application.

The Repository parameter refers to the name under Application > Settings > Connections, and the error message is saying you don't have a connection named mydockerserver/foo configured.

When it comes to Docker, our general guidance is to use the namespace part of a repository name, so instead of proget.corp.local/myDockerFeed/corp/myapp you would juse use corp/myapp or myapp.

Hope that helps,
Alana

Hi @philippe-camelio_3885 ,

Thanks for the bug report; this was a regression in the generated OtterScript, where it specified the wrong parameter name. Anyway, fix it via BM-3929, which will ship in the next maintenance release. Of course let us know if you'dl ike a pre-release version w/ that change, and I'm happy to ship it :)

Thanks,
Alana

Hi @philippe-camelio_3885,

Can you share the relevant OtterScript and execution log portion?

Behind the scenes, ProGet::Scan will use the package-lock.json file of your project. It's based on the pgscan tool (https://github.com/Inedo/pgscan) as an FYI.

Cheers,
Alana

Hi @scott-wright_8356 ,

Based on your other post, your server is under very heavy load, and another symptom is that clearing those logs can timeout. I believe this was improved in ProGet 2023, but deleting can be relatively slow when a ton of log messages have accumulated.

Just run TRUNCATE LogMessages against the database.

Best,
Alana

Hi @scott-wright_8356 ,

This is a common symptom of "server overload", and based on the usage pattern this is not surprising. It's not a question of hardware, but the fact that the NuGet client is effectively doing a "Denial of Service" on your ProGet server by issuing hundreds of simultaneous requests.

Keep in mind that every one of these requests must be forwarded to nuget.org (and perhaps other connectors?), so the traffic/waits multiply very quickly - and the symptom are timeouts like this on unrelated resources (like SQL Server). The FindPackagesById() query is notoriously slow on nuget.org, so there is a lot of waiting.

Setting up a ProGet Server Cluster that uses load balancing will be the best solution to handle these huge spikes in traffic There are a few other ways to squeeze more performance out of an instance.

1. Enable Metadata Caching on the connector, so that the queries don't need to be forwarded every time
2. Switch to NuGet v3 API endpoint; you are using the v2 endpoint, which is resource intensive already and doesn't use queries like FindPAckagesById()
3. Throttle requests; ProGet 2023 has a built in request limiter (advanced settings > Web.ConcurrentRequestLimit), and ProGet 2022 has a similar function only for NuGet feeds

Hope that helps,
Alana

Hi @philippe-camelio_3885,

This is the result of a long-standing WONTFIX bug within PowerShell Remoting. Considering that DSC is effectively dead, we don't see any point in try to work-around the bug. You can see more details/discussion here (Otter role issue after upgrading to 22.0.2).

In general, we advise all users to move away from PowerShell DSC. It was basically killed by Microsoft, and its corpse was resurrected to be used in Azure Automanage. From the docs:

DSC 2.0 is supported for use with Azure Automanage's machine configuration feature. Other scenarios, such as directly calling DSC Resources with Invoke-DscResource, may be functional but aren't the primary intended use of this version.

Since last year, there was a quasi-community effort started to bring back DSC as "DSC 3.0", but it's in early alpha and has no official support. Given the pace of similar PowerShell developments (like the v3 of PowerShell Gallery), I would expect 3-5 years for a somewhat stable version.

Best,
Alana

Hi @philippe-camelio_3885,

Both are working fine in our test lab.

I only looked at DSC, but according to the CollectDscModulesJob (which is what is doing the collection/parsing for the CollectDscModulesOperation), it's calling Get-DscResource and extracting ModuleName and Version.

I can't imagine why Version would be null/empty, but that seems to be the case??

Thanks,
Alana

Thanks for the report and work-around @Jon , we'll investigate this as part of the Otter 2024 roadmap.

@Jon I answered this in another thread, but

The error for "/api/releases" should be logged under Admin > Diagnostic Center; it's likely related to an unexpected/missing data in the application; you can narrow it down by specifying application id or something

If you can find what the error is, we can work to id/fix it!

Hi @Jon,

The Release and Build API Documentation is outdated and needs a lot of work. We are aware of the low-quality documentation, and this is on our list to rewrite.

To address your issues.

[1] The error for "/api/releases" should be logged under Admin > Diagnostic Center; it's likely related to an unexpected/missing data in the application; you can narrow it down by specifying application id or something

[2] You'll need to specify application/json as the content type when posting JSON documents; otherwise the request will be read as application/x-www-form-urlencoded values or querystring parameters.

[3] The pipeline name is incorrect, it should simply be Release (for an application pipeline) or global::MyGlobalPipeline for global pipelines.

Thanks,
Alana

Hi @Jon ,

Thanks for the note; this behavior is intended, and was decided after many years of supporting products in the field. The service is configured to automatically restart after a crash, and a database error will crash the service.

This is easiest to manage, since most database connection errors (even permissions-based ones) are are temporary in nature, so this behavior means the problem will be automatically resolved.

Best,
Alana

Hi @Jon ,

Unfortunately it's "not that simple", and this will require a bit of troubleshooting to figure out what the issue is, precisely. There are a lot of areas where this could occur, and we can't add a generalized "job killer" until we understand what the issue is.

You'll have to dig in behind the scenes (Admin > Executions) and identify exactly where things are freezing. The last log message will indicate that. Try to find as many examples as possible.

Keep in mind that Otter is not not really "connected" to a server, and a server does not (and cannot) "call home". Instead, Operations (i.e. OtterScript) opens a connection to a server, sends a command, then disconnects. All network errors we've ever seen in this process will yield a crash.

However, unless explicitly specified, a command will not timeout. So this means if you run a PowerShell script that basically just says "sleep indefinitely", then the Execution will never complete. Obviously no one would write that script, but some PowerShell scripts have a consequence of that. No built in Operation should ever cause that to happen, which is why we need to know precisely where this is happening.

Best,
Alana

Hi @dario-wigger_0347,

The configuration looks okay, and is basically what we have documented. So I think it ought to work...

From here, I would start by monitoring the HTTP traffic between Maven and ProGet. You can use a tool like Fiddler Classic on Windows or Wireshark. What you should see is:

• Maven issues a request, and ProGet returns a 401 with WWW-Authenticate: Basic header
• Maven reissues the request, sending a Authorization: Basic XXXXXXX

The XXXXX will be Base64 encoded api:your-api-key. If you don't see that, then something else is wrong. I would post it and maybe we can help.

You can also try downloading maven artifacts from an "Incognito" window. Your browser should also prompt you for a username/password, and then you can enter api and your api key.

Thanks,
Alana

Hi @PhilipWhite ,

Thanks for confirming that; after researching this, it seems that mounting /tmp with noexec is "old-school security measure" to prevent issues that have since been fixed, but is not an uncommon practice today. So we will make a code change to have Use Agent Temp folder instead of the system temporary directory for extension files extraction - this will be implemented via BM-3927 , hopefully in the next maintenance release (Tomorrow).

As for the Centos7 server, the underlying error message would be buried in the sshd logs. My guess is something with noexec, but who knows?

The Linux integration in BuildMaster v4 and BuildMaster 2022+ are a bit different:

• BuildMaster v4 just issues SSH commands and transfers files
• BuildMaster 2022+ transfers an agent executable, then uses SSH to instruct the agent to run the commands

So this added complexity seems to be causing some issues, but we'll figure it out!

FYI: eventually we intend to publish a Linux-version of the Inedo Agent, primarily to make it easier to configure than SSH.

Best,
Alana

Hi @phwhite_9282 ,

We're a bit baffled by this one, and think the issue is related to "some kind of configuration" on your RHEL server and the /tmp folder. Likely, security related. For testing purposes, can you make the BuildMaster user account (i.e. what you SSH in as) a root user?

Here's some more context... to help troubleshoot this further.

The first error was clearly related to some "strange" SSHD rejection (problem initializing a process), but changing the agent's temp path fixed it. We don't know what that error is specifically... but behind the scenes, BuildMaster starts by transferring executable files to the Agent's Temp directory (which clearly succeeded) and then executing those files (which failed).

The second error is an error loading a file from disk ('/tmp/Inedo/ExtensionCache/fafcab6c8d528766939b379f9e8f3dc9ba44db15/package/runtimes/linux-x64/native/libgit2-a2bde63.so). This usually means the file isn't on disk -- do you see that file on disk? The agent would have unpacked it there.

Our working theory is that some kind of security configuration is quarantining things that are written to /tmp and preventing execution. Which is unusual, because that's a common place where basically everything uses as a temp space.

Thanks,
Alana