Posts made by atripp

Hi @borisov_1556

You say that ProGet doesn't modify files after uploading, only during it. But can ProGet strip and therefore modify packages after uploading in case of triggering re-indexing in a "symbol server" settings?

When you enable any of the "strip" options, ProGet will modify the file during the download process, when it's requested by a client (web browers, nuget.exe, etc). The zip-file is basically rewritten on-the-fly when sending it to the client requesting it. This modified file is not saved to disk or persisted inside of ProGet.

The symbol serving reindexing is unrelated.

do you mean that not only the content of a file, but also the compression type of every file in the .nupkg will be preserved as-is, and ProGet handles that explicitly?

Correct. ProGet effectively just "deletes" the "stripped" files entries from the archive file while it's being streamed. The existing file entries are not modified.

A file entry in a zip archive contains the compression type. So to "change" an entry's compression, you would need to "delete" it from the archive, and then "add" it back to the archive.

So, the point is... something else must be modifying your zip file. ProGet is not. You can verify this by:

1. download file from NuGet.org, check the hash of the file
2. uplaod file to ProGet, check the hash of the file in the package store
3. download file from ProGet, check the hash of the file on disk

MAybe nuget.exe is doing a modifcation? It's really hard to guess, but i would continue the investigation by finding what could be modifying the file.

Cheers,
Alana

Hi @arozanski_1087

The method name is NpmFeeds_SetTagVersion so I think the error you're seeing is about case sensitivity in the method name.

Hi @borisov_1556 ,

Thanks for the detailed information.

At first, ProGet does not modify NuGet package files after you upload it.

If you enabled any of the "Strip" options, then ProGet will stream a modified archive file during download; this will have a different hash and be a different file. However, this wouldn't alter how existing items are stored in the archive file.... those would be streamed as-is.

ProGet also has a "repackage" feature, which will create a new package altogether. But I doubt you're using this, and you shouldn't use this with public / third-party packages, it's used for CI/CD pipelines.

I'm not sure what's going on, but I would investigate how you're uploading the files? Maybe you're re-signing them? But otherwise, if you just download a package file from nuget.org, then upllad it to proget, then download it again... it's going to be identical file.

Let us know what you find!

Cheers,
Alana

Hi @arozanski_1087

Thanks for sharing that!

Based on the snippet you sent, my guess is that this was originally for a npm package, and someone tried to make it work for a NuGet package? But that would have never worked, b/c NuGet packages can't be tagged... so his is really just a broken / never worked code snippet.

To tag npm packages, I recommend you to just use the client:
https://docs.npmjs.com/cli/v8/commands/npm-dist-tag

If you want to use the native API, then note the parameters are also wrong. NpmFeeds_SetTagVersion uses these parameters

• Feed_Id (Int32)
• Package_Name (String)
• Scope_Name (String)
• Tag_Name (String)
• Tag_Version (String)

Hi @arozanski_1087,

Do you mean that you are currently tagging npm packages via the API (NpmFeeds_SetTagVersion), but that you wish to tag NuGet packages?

I ask, because there is no such method NugetFeeds_SetTagVersion, at least not in ProGet v6 or other versions that I'm aware of.

It's also not possible to tag NuGet packages -- the reason is, the package tags are stored in the manifest file, and you would need to reupload the package file. npm tags, however, are stored as metadata on the server (not in the package manifest file).

The method NpmFeeds_SetTagVersion is part of the ProGet Native API, and you can find what methods are available in /reference/api inside of your version of ProGet.

Cheers,
Alana

Just jumping in here, but I think you need the .exe

docker exec -it proget exec /usr/local/proget/service/ProGet.Service.exe resetadminpassword

It's not needed on Windows to run an executable, but I think it's required for Linux.

Hi @karsten-meichsner_9039 ,

Oh wonderful!

If you're able/interested to try it out, then we should easily be able to add a checkbox in the UI for it and send you a patch/prerelease to try.

That's how many of the other options got added

Alana

Hi @james-traxler_1560,

I'm not really sure I totally understand the issue, or how to resolve it... but I'll explain how a GET manifest request to your URL would work.

Basically, ProGet returns whatever manifest file you uploaded, or whatever was downloaded via the connector. We do not generate the manifest files, or pay attention to the Accept request header.

I understand that application/vnd.docker.distribution.manifest.list.v2+json is a "fat manifest", which is basically a manifest that points to other manifests on different architectures. So you don't know exactly what image you'll get until you install it on the machine.

So if you want to inspect layers, you would first need to decide which manifest to use, I guess

Cheers,
Alana

Hi @james-traxler_1560 ,

Thanks for testing that

It will be released in v6.0.16, which is planned to ship on Friday June 10th.

Cheers,
Alana

Hi @marc-ledent_9164 ,

This isn't something that our LDAP/AD integration supports on Linux at this time. It's possible using the Windows-version, which has an LDAP integration that uses different libraries (that only work on Windows).

We do plan on rewriting the LDAP/AD integration with a different library, as to allow this level of customization, but it's not something we can do right away. It'll be later this year.

Cheers,
Alana

@ehsan-bahrami91_9979 great!

In that case, can you use 6.0.16-rc.1 by following these instructions on Windows:
https://docs.inedo.com/docs/desktophub-overview#prerelease-product-versions

Or if it's on Linux, just install the image with that tag.

After installing that version, it should do the trick. Please let me know :)

@ehsan-bahrami91_9979 thanks for confirming that

There was a regression/issue with recording this information, but it will be resolved via PG-2143 in 6.0.16 (shipping next Friday). If you'd like me to prepare a patch/prerelease, let me know!

Hi @karsten-meichsner_9039 ,

We'll definitely keep that in mind; we currently use the AWS SDK, which means we don't work at the HTTP-level and can't easily control the requests.

Basically our code does this:

new CopyObjectRequest
{
    SourceBucket = this.BucketName,
    SourceKey = this.BuildPath(sourceName),
    DestinationBucket = this.BucketName,
    DestinationKey = this.BuildPath(targetName),
    CannedACL = this.CannedACL,
    ServerSideEncryptionMethod = this.EncryptionMethod,
    StorageClass = this.StorageClass
},

If you know of any way to configure the SDK to send a different request, we'd be happy to try that out!

Cheers,
Alana

Hi @ehsan-bahrami91_9979 ,

Can you tell me what version of ProGet you're using? It should be in the bottom-right corner.

Thanks,
Alana

Hi @karsten-meichsner_9039,

I'm glad you could solve the issue! I'm not very familiar with Ceph, but it sounds like it works with the AWS API, which we use to talk to AWS. I added a brief notes to the docs about Ceph/RGW usage, but if you have suggestion to improve, please let me know :)

As for the "Use server-side encryption" option, the checkbox sets the option on the API Request (e.g. CopyObjectRequest) to use AWS Server-side Encryption. ProGet does not encrypt the data.

Cheers,
Alana

Hi @chris-f_7319 ,

If you follow the instructions to use the reset tool, it should reset and solve the problem:
https://docs.inedo.com/docs/various-ldap-troubleshooting

(proget-installation-directory)\Service> .\ProGet.Service.exe resetadminpassword

Cheers,
Alana

Hi @karsten-meichsner_9039 ,

This error is happening at the SSL-level, which is managed by the operating system (platform). There's nothing you can configure in ProGet to resolve this, so you'll need to make some changes to the operating system.

Can you tell me, are you using ProGet for Windows, or on Linux (Docker)? In either case, you'll need to trust the root certificate, but the ways to do this are a bit different.

In Windows, the easiest way is to install the certificate using the UI (Certificates-Current User > Trusted Root Certification Authorities into Certificates (Local Computer) > Trusted Root Certification Authorities). You can also verify that the installation was successful by trying to navigate to a URL in your S3 bucket.

In Linux/Docker, it's a bit trickier; in general you'll want to copy the .crt into /usr/local/share/ca-certificates and then run /usr/sbin/update-ca-certificates. There's a few ways to do this, but one common way is to build a docker image on top of our image. You can also SSH into the running container and handle this as well.

Cheers,
Alana

Hi @jramon-ortega_2700,

Thanks so much for verifying all of this. I talked to the team about this, and wanted to share an update, and further the conversation. We'd love to get your opinion.

Is PowerShell DSC dying/dead?

This is very clearly a "pretty big bug" that Microsoft hasn't fixed after very many years, which is surprising to see. It doesn't seem to be reported or discussed much.

At this point, we're really unclear on the future of PowerShell DSC. From our research, it never got widespread usage in the community because of the complexity of Pull servers, MOF, etc. Ultimately, most Windows admins just chose to use regular PowerShell scripts to set-up servers because it's simpler to understand and use.

Microsoft seems to have demoted it to a "community" project as well. It looks like they have some effort they plan to invest in it (based on a 2021 blog post), but there hasn't been recent updates or activity.

It seems that the PowerShell team is really focused on PowerShell Core now, and DSC isn't so much a fit? Maybe they will fix the remoting bug in Core? But who knows

PowerShell DSC Alternatives?

There are several alternatives:

• Otter's native operations, like Ensure-RegistryKey and Transfer-Files; these are more performant than the DSC resources and have a UI that make them easier to use
• PowerShell Scripts (using AHEXECMODE, etc)
• Executing powershell scripts using powershell.exe; this isn't great, because we can't easily parse output or read variables, etc.

How can Inedo help/invest in automation?

Ultimately, you (and our users) want to use automation to solve problems, and PowerShell DSC is just one tool to get there. It's convenient because there's many resources.

We can invest in working-around this bug, but it's not trivial. In the same time, we could improve a lot of Otter operations and write/document a lot of PowerShell scripts.

from a marketing/user standpoint, we don't know how many people are using PowerShell DSC , and if they would even find Otter to add value.

So getting your ideas/opinoins would be appreciated :)

Cheers,
Alana

Hi @james-traxler_1560,

Thanks for all of the info!

We are using ProGet Version 5.3.28 (Build 16), and calling the API directly.

Ah, that explains it. We really only test this with the Docker client, which wouldn't trigger this issue.

Usage scenario: removing or renaming tags for a manifest. As there is no specific route for this use case, we are DELETE-ing the manifest through the v2 docker API and re-adding (with PUT) the new/remaining tags.

Is this a Docker API limitation? We were thinking of adding a API for tag management for our own CI/CD platform (BuildMaster), but just worked-around it, and no one ever asked.

we are concurrently executing 6 PUT requests:

This is probably where the issue is. Looking at the database code, there is an opportunity for this to happen between a DELETE and INSERT statement.

It's pretty easy to fix, can you give it a shot? If it works in your test, then I'll commit the change. But note this will be for v6.0.15, so unless you upgrade to that or later, then you won't have the patched code.

https://inedo.myjetbrains.com/youtrack/issue/PG-2140

If you download the file attached to the above link, you can run it against your SQL Server database.

Thanks,
Alana

Hi @Joseph-Saladino_7499 ,

Hmm, that's really weird; the next thing I would try is to just restart all the machines (including the BuildMaster server). That's easy to do and the issue might go away. It could be a software error in the Windows networking stack

It's possible that there's some intermediate network device that's interfering. Router, QoS device, etc. That's harder to diagnose, and would require some tracing ... starting by making sure that the RST packet is actually being received on the BuildMaster server (that's usually what indicates the error message), and then figuring out what device is sending it, and why.

Our code doesn't work at the packet-level, but if the remote server (Inedo agent) is where the RST packets are coming from, it likely has to do with some obscure Windows setting... maybe even firmware on the network card. But at least something to look at.

Cheers,
Alana

Hi @james-traxler_1560 ,

We can help with this...

In general, the error makes sense if you're trying to re-add an image that's already there... but can you describe how you're doing that?

I.e. are you using the Docker client to push an image, or using the API directly? execute the PUT statement?

What is the version of ProGet that you're using?

Can you give a full stack trace (not sure if there's an error)?

Thanks,
Alana

@jramon-ortega_2700 thanks so much for testing this, and verifying the issue is occurring outside of Otter. This is what we were afraid of, but it shows where the problem is.

This is definitely a bug/issue with PowerShell, and likely the way that the DSCR_Application module is designed? We don't really know.

I doubt Microsoft will fix this... but it doesn't hurt to ask. You have a very good reproduction case now.

It might be possible for us to work-around this, but unfortunately it's not a trivial matter. We'd need to find an alternative to "local remoting" to handle process communication.

What would be good to know, how widespread is this?

Is it only this DSCR_Application resource? Maybe the creator of DSCR_Application can work-around the issue?

Using Chocolatey Packages, I guess, could be a good replacement? Maybe there's a way to do it with your own Ensure-based PowerShell script instead of a DSC resource?

We like the ability to use DSC resources, but if it's only one or two, then maybe it's not worth the effort to work-around?

Hi @Joseph-Saladino_7499 ,

The message, "An existing connection was forcibly closed by the remote host" basically just means that the remote server abruptly hung-up the connection.

Based on the fact that there's no errors happening on the remote servers (agents), then I wonder if it's related to firewalls blocking a connection?

It could also be related to the agent refusing to connect because the AES encryption key is not correct. How are the agents configured (with / without encryption key?)?

Hope this at least points in the direction.

Thanks,
Alana

Hi @marc-ledent_9164,

There's really just one, and that's the artifacts (BuildMaster) or packages (ProGet).

To configure server level shared storage, you'll want to configure:

• For BuildMaster, the Artifacts.BasePath value in Administration > Advanced Settings.
• For ProGet, the Storage.PackagesRootPath value in Administration > Advanced Settings.

Do note that ProGet also allows configuration of storage at a feed level, which you can find on the Manage Feed page for each feed. Click change on the right side of the Disk Path property.

This guide might be helpful:

https://docs.inedo.com/docs/installation-high-availability-load-balancing

Hi @jramon-ortega_2700,

I'm not a PowerShell expert myself, but I looked at the code for the operation.

We use the Invoke-DscResource CmdLet to run the configuration...

https://github.com/Inedo/inedox-scripting/blob/master/Scripting/InedoExtension/Operations/PowerShell/Dsc.cs#L140

Not sure if that helps, just wanted to point you in the rightd irection

Cheers,
Alana

Hi @jramon-ortega_2700 ,

Another strange error! I chatted with the engineers here, and can summarize this a bit.

In Otter 2022, we went to .NET6, which has those annoying bugs with modules in a "hosted powershell" environment. So, we switched to using something called "local remoting" - which basically allows us to communicate with the PowerShell process using PowerShell's remoting functionality. So basically, PowerShell is now executing using "PSRemoting" - so it kind of makes sense that a remoting error is happening.

After researching the "serialized xml is nested to deeply" message, this seems to be a powershell bug -- it's an error displaying an error message over remoting, maybe. It happens with PSRemoting and DSC it seems, but who knows

Are you able to try to run this via PSRemoting, to see if you can reproduce the error? Any errors logged in the event log.

A few posts mentioned that this has something to do with security?

If you search "serialized xml is nested too deeply" you should find a handful of things to try... mind giving those a shot, and see if it makes any difference?

Thanks,
Alana

Hi @Rohan-Patel_7666

I think that this issue is related to TLS configuration; basically your operating system doesn't support the TLS configuration required by websites like nuget.org.

Here is a registry work-around that you can try:
https://inedo.com/support/kb/1161/tls-v12-configuration-and-connection-errors

This will force TLS 1.2.

Cheers,
Alana

@jramon-ortega_2700 thanks for the bug-report; this seems to be related another library upgrade, but we'll get it fixed soon - please stay tuned :)

@kichikawa_2913 thanks for letting me know; well, if you see it come again, please do inform us. It may be a caching issue on Linux/containers, but hard to say

@kichikawa_2913 This is really really weird.

Can you try creating a credential again? Perhaps restart the container? I've never seen anything like this... clearly the extension has loaded, since you can view all the components ‍️

Which of the credentials did you create by the way?

Hi @kichikawa_2913,

Do you have multiple web nodes? I'm thinking that, perhaps, one of the web nodes didn't load the extension?

Hw about if you lick on TeamCity in that list? It should enumerate all of the components in the extension.

You were able to create it... so we know it loaded at some point...

Alana

@kichikawa_2913 we don't have a typical set-up per se, really

A lot of set-ups are very simple, since the team is new to CI/CD, and they just have the BuildMaster server do everything.

Others have dozens of build agents configured in a pool to service various teams, and use lots of shared configuration.

Most are somewhere in the middle -- since they started simple, and gradually explored over time. That's generally what i'd start with -- tackle one pattern/team, then move on to the next and improve over time.

Hi @kichikawa_2913 ,

Given that you're already familiar and comfortable with using build agents, then we'd recommend a similar set-up with BuildMaster. To do this, you'd provision/configure your build server(s) and then have BuildMaster connect to them, generally using the Inedo Agent.

One common pattern is to create a role, such as build-server, and then target it in your pipeline stage, like this:

This way, it will act as a server pool. You can also do this in OtterScript as well, for some very fine-grained control; it's considered a resource pool.

In any case, whatever server you target is where the code will retreived from source, built, and tested.

hope this helps!

Hi @gurdip-sira_1271 ,

We don't think anything changed with regards to Git Rafts & Otter v2022, and don't have any issues on our end with them.

Is this a new instance with them or something?

Cheers,
Alana

Hi @jramon-ortega_2700 ,

This should be fixed now! If you go to Admin > Extensions, can you download Scripting 2.0.1?

Thanks,
Alana

Hi @jramon-ortega_2700 ,

We just discovered this issue with Otter v2022; it has to do with a bug in the hosted PowerShell runtime, .NET6, and loading certain modules

It doesn't look like Microsoft will fix it unfortunately, but we're currently scrambling to work-around the issue and get it fixed ASAP.

Cheers,
Alana

Hi @zakaria-smahi_2127,

That badge looks like something generated by a CI-server, and I think it's showing the latest version or tag or something related to a release branch? I don't have much context.

It's not something that ProGet (a package manager) would generate, since all of the packages in ProGet are considered "published".

Our CI/CD platform, BuildMaster, can generate these badges:
https://docs.inedo.com/docs/buildmaster-ci-cd-continuous-integration-server-badges

Cheers,
Alana

@gurdip-sira_1271 it's possible there's a bug in the visual mode with that, especially w/ rafts...

Here is how you you would call scripts from another raft:
PSCall RaftName::Set-WebServer;

When you don't specify a raft name, then Default is used as the raft.

Hi @gurdip-sira_1271 ,

If your script has no parameters, then there's nothing to really show on the Visual Editor. However, you should be able to save the statement.

Behind the scenes, the visual editor should just make a PSCall Set-WebServer; in text mode. What does the text mode look like?

Cheers,
Alana

Hi @gurdip-sira_1271,

I guess the bit I am confused about is if a restart is needed mid script. Would the agent on the server resume the script post reboot?

You would need to perform the restart in OtterScript.

for server REMOTEMACHINE
{
   PSCall MyScript1.ps1;

   Restart-Server;

   PSCall MyScript2.ps1;
}

For script assets, how would I reference from my Powershell script? Or should I use OtterScript and then have the step above and then my script?

In this case, you'll want to use OTterScript to deploy the script assets first.

To add, I get this error when making a job template:

That's a regression, and we'll fix it pretty soon. It has to do with Job Template folders in Git Rafts. We're shipping Otter v2022 today actually :)

In addition, when I drag a ps1 script in an otterscript orchestration, the popup is blank and i can't add the script.

Does your script have any parameters? If so, that is to be expected. Its should save if you hit the save button however. Note that in Otter v2022, shipping very soon, the script editor is totally rewritten and should be fixed.

Cheers,
Alana

Hi @gurdip-sira_1271

There is a Restart-Server OtterScript operation that you can use. You can specify a timeout/timer.

You'll want to store that chocoPackages.txt file as a script asset, and then deploy it to the server you're running the scripts on:

Get-Asset chocoPackages.txt
(
    Type: File
);

However, you may want to consider this approach for your chocopackges:

https://docs.inedo.com/docs/otter-howto-ensure-chocolatey-packages

Hi @gurdip-sira_1271 ,

I want to share this guidebook that we wrote;
https://inedo.com/support/resources/ebooks/windows-first-iac-cca

Basically it discusses and walks you through how to CI/CD with your "infrastructure as code" using Otter and BuildMaster (our CI/CD tool). The screenshots are outdated, but after a lot of research, our conclusion is that the "pipeline" approach is likely the best.

Cheers,
Alana

Hi @chris_5235,

Unfortunately I'm not really sure what's happening, but I suspect it has to do with an HTTPS redirect?? I would look there next.

As you mentioned in the OP, the package can be download from the endpoint, so long as you provide the credentials. So, I guess, PowerShell isn't sending credentials after a redirect, perhaps?

One thing that you might want to do, is use a tool like Fiddler to capture the request/responses. That will let you verify that the issue is indeed PowerShell not passing credentials to ProGet. You can look at the authentication header.

Things to look at:

• make sure ProGet is sending an https download link for the package file in the FindPackagesById() request
• X-Forwarded-Proto is set to be https
• If you must, set BaseUrl under advanced settings

Cheers,
Alana

Hi @mcascone ,

I think that was a feature that got lost in 5.3, with the new UI.

Easy enough to bring back, I think, so we'll add it via PG-2134

Cheers,
Alana

Hi @gurdip-sira_1271 ,

Your Git repository must be structured in a particular way to work as a Git Raft; but it's best to let Otter manage all of that. Here is a sample: https://github.com/Inedo/OtterRaftTest

The repository you use for a Git-based raft shouldn't be used for anything else - just raft storage.

Cheers,
Alan

Hi @gurdip-sira_1271 ,

Great, sounds like its working :)

I think the issue is that you didn't have the folder-structure in place. Scripts should go in the /scripts folder basically.

Cheers,
Alana

@nmarkkula_7220 make sure the v3 API is enabled on your feed; the endpoint will look like https://MY.FQDN/nuget/MYNUGETFEED/index.json

visual studios detects the API once it queries it, so if you changet the URL then it should work

@nmarkkula_7220 I'm not totally sure, but I'm pretty sure Visual Studio used to pay attention to that flag. Perhaps it's a regression or something in Visual Studio?

But it looks like that's on the v2 API, and Microsoft (and we) aren't really maintaining that anymore. I would try using the v3 endpoint instead.

@nmarkkula_7220 this can be a confusing issue

Listed/Unlisted is a serverside metadata, like the download count. When you cache or pull a package from nuget.org, then ProGet won't have severside metadata. This is why download counts start at zero once you upload/pull a package to ProGet.

You will have to unlist the package in Proget too.