Posts made by dean-houston

Hi @steviecoaster,

The current package import tool uses the NuGet API. It's not really easy to use, and I'm afraid our API access code isn't really "portable" -- it's tightly integrated into Connectors, which are tightly integrated into Feeds, etc.

Here's a guide on how to query all published packages from a NuGet feed:
https://learn.microsoft.com/en-us/nuget/guides/api/query-for-all-published-packages

That said, next week we will be releasing a brand-new package importer that will connect to Sona Nexus, Artifactory, AzureDevOps, ProGet, GitHub, and GitLab. These use the provider-specific APIs and work much better than what we have now.

Functionality it's the same, but now your credentials are stored in ProGet. You can also run it multiple times, and it will only import new packages. This is useful for the cases where you are transitioning usage.

-- Dean

Hi @michal-roszak_0767 ,

Thanks for letting us know!

Unfortunately the Feed Wizard seems to have some quirky behavior when configuring certain combinations of options, as you've noticed. We are actually in the process of rewriting the new feed wizard to be a bit more simpler (esapecially behind-the-scenes), hopefully in the next couple weeks it'll be in a new maintenance release.

In the meantime, if you encounter these errors... I would just create the connectors on the MAnage Feed page. Which it sounds like you've done :)

-- Dean

Hi @daniel-pardo_5658 ,

The UI-based package editor is intended for small packages, up to 50 MB or so. It looks like there is a platform-enforced limit of 2GB. For now, you will need to download, edit, re-upload.

That said, I switched the stream that we're using to something that can accommodate larger packages, but I did not test it so I really don't know if it will actually work on these packages. It'll be in the upcoming maintenance release via PG-2964

-- Dean

Hi @scusson_9923 ,

Sorry, I misunderstood; I thought you were doing PSExec.

In this case you are just executing the pwsh process, so you need to figure out how to have that process return an exit code.

I don't know if that's the same as powershell.exe, but a AI told me "To return an error code from a PowerShell script, the exit statement followed by the desired error code should be used."

So I guess exit -1 or something like that?

-- Dean

Hi @scusson_9923 ,

Exec (or Execute-Process) runs an operating system process, so that's where the return code comes in.

If you're doing something like a PSCall, then you can create output parameters/variables in the script, and then test those values.

-- Dean

Hi mwatt_5816,

BuildMaster does support "release-less" builds, though you may need to enable it under the application's Settings > Configure Build & Release Features > Set Release Usage to optional. That will allow you to create a build that's not associated with a release.

It's also possible to do "ad-hoc" builds (i.e. builds with no pipeline), but we don't make it easy to do in the UI because it's almost always a mistake (once you already have pipelines configured). So in your case, I think you should create a secondary pipeline for this purpose.

-- Dean

Hi @procha_8465 ,

I'm afraid we'll need a bit more information here to help you. There are a lot of changes between ProGet 5.2 and ProGet 2024 and between older/newer versions of the npm client.

If you can put together a reproduction case, ideally on a new instance of ProGet 2024, that'll help us determine what you're trying to do and how to help.

-- Dean

Hi @f-medini_8369,

Here is our documentation on how to use Prometheus:
https://docs.inedo.com/docs/installation/logging/installation-prometheus

-- Dean

Hi @simon-marriott_1201,

If you haven't seen it already, I'd check out How Files and Packages Work in ProGet for Artifactory Users.

Long story short is, you should consider a more modern approach than the Maven-based file/folder that Artifactory uses. Many have found a lot of success with Universal Feeds & Packages.

We don't maintain a TeamCity plugin, but it's really easy to create, publish, deploy packages using the pgutil command line (i.e. pgutil upack create); see HOWTO: Create Universal Packages

Hope that helps!

-- Dean

Hi @uwer_4638 ,

The underlying issue is that you're making a "NuGet v2 API" request to your ProGet feed, which ProGet is then forwarding to connectors, and DevExpress does not support NuGet API V2.

So, you'll need to track down whatever is making that request (perhaps you're using an old endpoint URL), or simply just disable the V2 API on your feed. This will cause an error on the client, and should show you pretty quickly what's making that outdated call.

-- Dean

@layfield_8963 great news, that was a really strange error. The UI upload uses some kind of chunking and file appending, so it sounds like that was it.

Hi @scusson_9923,

In this case, you'll likely want to select 5 as the type.

For reference, here are the valid types:

    //
    // Summary:
    //     Specifies the type of a raft item.
    //
    // Remarks:
    //     All types except BinaryFile and TextFile are "regulated" and only allow well-known
    //     files; for example,
    public enum RaftItemType
    {
        //
        // Summary:
        //     A role configuration plan.
        RoleConfigurationScript = 1,
        //
        // Summary:
        //     [Uninclused] A Script with .otter syntax is prefered
        OrchestrationPlan = 2,
        //
        // Summary:
        //     [Uninclused] A Script with .otter syntax is prefered
        Module = 3,
        //
        // Summary:
        //     A script.
        Script = 4,
        //
        // Summary:
        //     An unclassified binary file.
        //
        // Remarks:
        //     BinaryFiles cannot be edited in a text editor, compared, etc; they are always
        //     treated as raw content
        BinaryFile = 5,
        //
        // Summary:
        //     A deployment plan.
        DeploymentScript = 6,
        //
        // Summary:
        //     An unclassified text file.
        //
        // Remarks:
        //     TextFiles can be edited in UI , may have lines replaced on deploy, and can be
        //     used as templates
        TextFile = 7,
        //
        // Summary:
        //     A pipeline.
        Pipeline = 8,
        //
        // Summary:
        //     [Uninclused] Feature is deprecated
        ReleaseTemplate = 9,
        //
        // Summary:
        //     A job template.
        JobTemplate = 10,
        //
        // Summary:
        //     Files used with build tools like Dockerfile.
        BuildFile = 11
    }

I'm not sure if TextFile (7) will work; in Otter it was intended to be used as a text template, which means lines in it are replacement. You may need to play around and see what works.

-- Dean

Hi @scusson_9923 ,

What is the file you are uploading? What happens when you upload through the UI?

Can you share the PowerShell snippet you're using?

What are you specifying for RaftItemType_Code?

-- Dean

@kc_2466 the "invalid feed type" will come up if you have a connector or feed that was created in a newer version of ProGet that wasn't available in an older version of ProGet... and you downgraded to the older version.

It looks like it's a connector, based on the URL. Easiest way to fix is to just upgrade, delete, downgrade

@kc_2466 thanks for the heads up, we'll target reviewing/fixing this for the following maintenance release (i.e. 2024.27 / Feb 21) via PG-2893

Hi @jimbobmcgee ,

Thanks for all the details; we plan to review/investigate this via OT-518 in an upcoming maintenance release, likely in the next few two-week cycles.

-- Dean

Hi @caterina ,

Looking over the code, I can see that; we will also fix that in the next maintenance release. The notifier should not be dispatched when there are 0 issues.

-- Dean

@cooperje_6513 that error means that the Windows service account user does not have access to the SQL Server database; you'll want to grant NT AUTHORITY\NETWORK SERVICE access

You can do this with SQL Server Management Studio, or a scritp like this should work:

  CREATE LOGIN [NT AUTHORITY\NETWORK SERVICE] FROM WINDOWS WITH DEFAULT_DATABASE=[ProGet]
  CREATE USER [NT AUTHORITY\NETWORK SERVICE] FOR LOGIN [NT AUTHORITY\NETWORK SERVICE]
  ALTER USER [NT AUTHORITY\NETWORK SERVICE] WITH DEFAULT_SCHEMA=[dbo]
  ALTER ROLE [ProGetUser_Role] ADD MEMBER [NT AUTHORITY\NETWORK SERVICE]

Hi @caterina ,

I was able to find the issues; the correct value should be this:

$ToJson(%(
    issues: @BuildIssues,
    buildNumber: $BuildNumber,
    releaseNumber: $BuildReleaseNumber,
    projectName: $BuildProjectName
))

I've updated the documentation and also ProGet (via PG-2890), which will be in teh next maintenance release. But if you use the above template it should work.

-- Dean

@jimbobmcgee fantastic, we'll review/merge soon! thanks much :)

Hi @scusson_9923 ,

This seems to be an issue related to release vs debug builds (works fine locally, but not when deplyoed to server), and we'll investigate and fix via OT-517 in an upcoming maintenance release (2024.4) - not sure on the exact schedule, but we're targeting the next couple weeks

-- Dean

@jimbobmcgee thanks; we'll definitely investigate this later, but it will likely not be for a few months until we can do some "heads down" time with this stuff

Honestly I don't remember how any of this works, so I could be wrong and you need to do something else. It's clearly not something we document.

Our primary use case is more like this, uploading basic scripts:
https://docs.inedo.com/docs/otter/scripting-in-otter/otter-scripting-powershell

@jimbobmcgee thanks for reposting this here as well!

PSEval is definitely not meant for scripts like that, due to how the parsing works... but as you noticed, the $PSEval($ps) should work. We probably won't change this.

PSExec (i.e. Execute-Powershell) can capture variables, but not output streams. So something like this:

set $hello = world;
$PSExec >>
  $hello = 'dears';
>>;
Log-Information Hello $hello;

Similar to my comments on the PSEval thread, this is another one of those rabbitholes that can break stuff, since the existing behavior seems to work for some users. So we're super-cautious about it.

It's likely we won't change these behaviors as they are "good enough" for the intended usecase of Otter.

@jimbobmcgee that's a nice idea; we'd definitely be open to a pull request on those FYI

Based on other list/map functions I think it'd be relatively straight-forward and an easy pattern to follow:
https://github.com/Inedo/inedox-inedocore/blob/master/InedoCore/InedoExtension/VariableFunctions/Lists/ListRemoveVariableFunction.cs

Just not something we can focus on now though

@jimbobmcgee thanks for reposting this here as well

Working with PowerShell output variables is a very long-standing challenge, in particular because PowerShell has very inconsistent returns based on Windows, Windows Core, LinuxCore. We made some fixes not too long ago, but it's still not perfect and was a ton of effort that ended up breaking some user scripts.

And as you probably saw poking around the execution engine code, a variable prefix ($, @, %) is more of a convenience/convention, and the prefix isn't really available in any useful context. I'm almost certain you can do stuff like $MyVar = @(1,2,3) for example. This is very likely not something we will want to change.

Keep in mind that OtterScript was never designed as a general-purpose scripting language, but as a light-weight orchestration script to run other scripts. So these limitations happen.

I will make a note of this on our long-term roadmap, but it's likely we won't take action on it due to sensitivity of all this and not wanting to break existing scripts.

@jimbobmcgee thanks for reposting this here

This is a long-standing behavior of Otter/OtterScript and it's most likely not a trivial fix and would involve updating the parser/execution engine (after remembering how it all works) - so not something we'll do in a maintenance release for a community/free user, as I'm sure you'll understand

However now that it's here, I will link it to our internal roadmap planning and consdieration for Otter 2025.

Hi @james-woods_8996,

Each instance of ProGet needs its own file storage (S3 bucket, disk, etc). You definitely do not want to use the same storage across instances - that'd cause a major issue.

Some users have been tempted to use a combination of Database Replication + Disk Replication with third-party "external" replication tools, and learned the hard way that it's an absolute disaster once deployed. So don't try that :)

Basically the "external" (non ProGet) replication is way too slow to handle the type of traffic ProGet receives, and the files/database replication cycles are never in sync.

-- Dean

Hi @cooperje_6513 ,

It sounds like you had done some manually/IIS configuration, or perhaps an error occurred at some point. In any case, I would manually remove all components (service, IIS, etc), and you can delete any registered installation in the c:\ProgramData\upack folder. Just keep your installed package files (typically c:\ProgramData\ProGet).

Then, just install fresh, pointing to the same database. Use the Integratred Web SErver, not IIS. That's what we recommend now.

NOTE that the WebApp folder is no longer used.

Hi @kc_2466 , we'll get this one fixed via PG-2885 in the next maintenance release!

@jimbobmcgee thanks for the detailed analysis! This is on my list, but haven't had time to properly reproduce and test this.... but as you noted, the code fix looks so easy.

I just made a quick change (OT-513) and prerelease (2024.2-rc.1). Can you give it a shot?

https://docs.inedo.com/docs/installation/windows/howto-install-prerelease-product-versions

Hi @ghollosy_9163,

I'm afraid this will not be an easy problem to troubleshoot; there were some major changes in ProGet 5.3 with regards to Docker feeds, so it's possible that there was a data migration problem during the upgrade process.

Ultimately it looks like your container configuration files aren't there. I don't know why, but I'll try to point you to where/how they are stored.

Docker blobs are stored on disk, and you'll see files named by their hash. Based your screenshot, it'd be a file like bf3c18259..... These files are also indexed in the database, in the DockerBlobs table.

If the files aren't anywhere on disk, then check backups. It's possible they got accidently deleted, and you can just add them back.

If the files are missing on disk, then they are probably gone for ever. If they are not in the database, then perhaps something happened during the migration? I really don't know. It's possible to insert them in the database, but we don't have an automatic way to do this - basically you'd need to write a script.

-- Dean

hi @jaehyung-shin_8059 ,

If you click on the latest tag, do you see any information about the packages in the container?

Container images are scanned on upload if they are small or queued to be scanned in the background if they are large. There are sometimes errors scanning containers, and if so, these errors will be logged... typically under Admin > Executions.

-- Dean

Hi @jaehyung-shin_8059 ,

From what you described, it sounds like you'd like to have a single tag (like :v3) resolve to different architectures?

This is called a multi-platform image, and you'll need to study/follow the Docker guidance on how to create one: https://docs.docker.com/build/building/multi-platform/

Long story short, this is not something that you can do by simply pushing two images to the same tag like you're doing; it's quite a bit more complex, and you have to build the images in a very specific manner.

But once you do that, ProGet does indeed support them (they are called "fat manifests") and you will see a note/warning when viewing an image for one.

-- Dean

@parthu-reddy yes, but we do recommend migrating...

There are some known bugs/quirks with Maven (Classic) feeds and we will not fix them.

We will likely not migrate them to PostgresSQL, which means you won't be able to migrate to PostgreSQL in ProGet 2025+.

So it's likely that ProGet 2027 will not support them.

Hi @parthu-reddy ,

Huh... that's interesting... but not surprising. The Maven (New) feeds assume POM files are valid XML, but I guess Marvin supports invalid XML for POM files

I fixed this via PG-2859 which add Add Support for Invalid POM files on Maven (New) Feeds. You can access it via 2024.23-rc.1 on the prerelease feed (InedoHub), or the 24.0.23-rc.1 Docker image.

Hopefully this will work-around these migration issues.

-- Dean

Hi @parthu-reddy ,

This error means that the previously listed pom file (ant-1.6.5.pom) is somehow corrupted. It cannot be loaded as XML due to the error.

I assume it's this artifact:
https://repo1.maven.org/maven2/ant/ant/1.6.5/

That one is fine at the source. I would just remove it from your old/classic feed, since I assume it's just something from maven central. And it can then be downloaded again.

-- Dean

Hi @parthu-reddy,

For a major upgrade, we recommend to schedule a maintenance window and then stop traffic from the load balancer. Then, upgrade the servers.

For a maintenance release, it's fine to just upgrade them one at a time. It only take as few minutes to do that. There may be a few errors with the database/code being out of sync, but it likely will not cause any problems.

-- Dean

@parthu-reddy this is a somewhat expected error, and it's something we'll improve in a future version of ProGet, but in all a cases it will cause some kind of problem.

The problem is that you added a new feed/connector (in this case, a Maven (New)) that is unknown to previous versions of ProGet. So when you try to load that unknown type in older versions of ProGet, you get this error.

If you deleted the connector before downgrading, it would have worked. So, I would upgrade, delete, then you can downgrade again.

-- Dean

@sneh-patel_0294 I don't think so, its only displayed on that page...

Hi @sneh-patel_0294 ,

We've never seen that before nor had anyone else report it. The only thing I could guess is network communication on port 33237. I guess I would try restarting the web server? I would try it on different servers?

It's just really weird behavior, but that's the onnly thing I could guess.

You can access a dialog to disable the Automatic Failover, which would also disable the network communication via /administration/cluster/configure

Changing the license key would also disable it. These are the only ways I can think to test if it's "something" blocking/delaying/interfering the 33237 communication. We've seen some firewalls do that.

-- Dean

Hi @Yoeri-rousseaux_8527,

Unfortunately the PowerShell gallery is pretty buggy/slow/glitchy these days. We don't think it's actively maintained by Microsoft anymore, as their attention is shifted to "next gen" (PSGet 3.0).

Connectors to the gallery don't work as well as they should, and you may find using an approval workflow (i.e. pulling them proactively, promoting them to other feeds, etc) is the most reliable. Many queries to PowerShellGallery.org (especially a FindPackagesById()) will simply timeout, which will cause ProGet to log an error (Admin > Diagnostic Center) and return a 404 in the API.

There are also versioning and API quirks that you should be aware of. They apply to some packages (mostly older, but some newer, sometimes). It's not easy for us to work-around these without a major development effort, and most of our users have learned how to deal with these annoyances.

However, this should all work in PSGet3.0. Now they are about 5 years behind on the release... BUT with Copilot's help, perhaps 2025 just might be the year they get it fixed!

-- Dean

Hi @sneh-patel_0294 ,

That page mostly displays some items stuff from the database (specifically ClusterNodes_GetNodes) and attempts to do some network communication on port 33237; there's likely some kind of firewall/trap that's preventing communication on that port, and it will timeout after awhile.

You should be able to visit that page from any browser, as opposed to localhost. That page is mostly just useful in making sure the load-balancer is configured correctly.

-- Dean

Great news, thanks for the update!

Hi @sneh-patel_0294 ,

That error message is coming from the operating system; it doesn't necessarily mean a permissions issue.

Does it happen every time for every package, consistently?

If that's the case, then it's certainly some kind of permission configuration. The user running the ProGet Web Service (or IIS App pool) may not have the appropriate permissions to the folder.... or it could be something related to network access? I don't really know.

The operating system is opaque with the error message, and you might have to use a tool like procmon to see exactly what's going on. That will show you what programs/processes request file handles.

If this is sporadic, then it means the file is locked. It's possible for ProGet to lock the file, but it's unlikely and would require basically two processes trying to write to the same file at the same time. We've only seen that with misconfigured build servers that publish same build twice.

More likely the file locking is coming from like backup, index scanning, or malware that's masquerading as "security software". Procmon will also advise this, if you can catch it.

-- Dean

Hi @forbzie22_0253 ,

Tags are a field in the nuspec file, which is embedded within the NuGet package:
https://learn.microsoft.com/en-us/nuget/reference/nuspec#tags

As such, we do not recommend using tags in NuGet, because they are "permanent" and you can't "untag" packages. It makes it hard to think of a tagging system that will be useful for the long-term: https://blog.inedo.com/nuget/best-practices-internal-nuget-packages/

-- Dean

Hi @arkady-karasin_6391 ,

A 403 error means Not Authorized; since those are publicly-available repositories, my guess is that you have a Proxy or Content Filter that's blocking that URL, and the ProGet Server is getting a 403 response from that intermediate server.

-- Dean

@joel-shuman_8427 thanks for the heads up!

I just updated it
https://github.com/Inedo/inedo-docs/blob/master/CONTRIBUTING.md

HI @parthu-reddy ,

Thanks for the feedback; I update the docs to mention that the feeds/permissiosn would also need to be updated as well.

We hope to handle an in-place migration in a future version, but didn't want to delay shipping the feed. Please let us know if you have any issues/feedback.

Note we are adding several improvements to the Maven (New) feeds in ProGet 2024.15:

• PG-2798 Add Direct Download (Artifact Import) Support for Maven (New) Feeds
• PG-2797 Add OSS Metadata Caching Support to Maven (New) Feeds
• PG-2796 Improve MavenIndex Download Visibility for New Feed Connectors
• PG-2792 Feed Management API Returns Empty Type on Maven2 and new feeds
• PG-2794 Add Simulated Directory Browsing to Maven (New) Feeds

-- Dean

Hi @parthu-reddy ,

I'm afraid we don't have enough information to help with this; it appears to be an error the tool you're using (packet), and there's not enough information in the screenshot to see what error packet is encountering. It just says "Packaged failed with could not download..."

We aren't familiar enough with packet to know how to follow their Stack Trace.

If you can't find a clear error message, I would use an HTTP Proxy tool like Fiddler Classic to inspect the traffic that packet is making, and see if you can spot an issue there.

I would also try downgrading packet , as it's very possible there's a regression in the tool.

-- Dean

Hi @kichikawa_2913,

We see multiple connectors pretty often, and it rarely presents a problem.

The main downside comes in the overhead of aggregation; for some queries like "list all package versions", each connector will need to be queried and have the results aggregated. So it could cause performance issues for for very high-traffic feeds - at least that's what we see on the support side of things.

However, if you plan on using a package-approval workflow, then it won't be a problem, as your approved-npm feed wouldn't have any connectors.

Hope that gives some insight,

Dean