Hi @jramon-ortega_2700 ,
We just discovered this issue with Otter v2022; it has to do with a bug in the hosted PowerShell runtime, .NET6, and loading certain modules 
It doesn't look like Microsoft will fix it unfortunately, but we're currently scrambling to work-around the issue and get it fixed ASAP.
Cheers,
Alana
That badge looks like something generated by a CI-server, and I think it's showing the latest version or tag or something related to a release branch? I don't have much context.
It's not something that ProGet (a package manager) would generate, since all of the packages in ProGet are considered "published".
Our CI/CD platform, BuildMaster, can generate these badges:
https://docs.inedo.com/docs/buildmaster-ci-cd-continuous-integration-server-badges
Cheers,
Alana
@gurdip-sira_1271 it's possible there's a bug in the visual mode with that, especially w/ rafts...
Here is how you you would call scripts from another raft:
PSCall RaftName::Set-WebServer;
When you don't specify a raft name, then Default is used as the raft.
Hi @gurdip-sira_1271 ,
If your script has no parameters, then there's nothing to really show on the Visual Editor. However, you should be able to save the statement.
Behind the scenes, the visual editor should just make a PSCall Set-WebServer; in text mode. What does the text mode look like?
Cheers,
Alana
I guess the bit I am confused about is if a restart is needed mid script. Would the agent on the server resume the script post reboot?
You would need to perform the restart in OtterScript.
for server REMOTEMACHINE
{
PSCall MyScript1.ps1;
Restart-Server;
PSCall MyScript2.ps1;
}
For script assets, how would I reference from my Powershell script? Or should I use OtterScript and then have the step above and then my script?
In this case, you'll want to use OTterScript to deploy the script assets first.
To add, I get this error when making a job template:
That's a regression, and we'll fix it pretty soon. It has to do with Job Template folders in Git Rafts. We're shipping Otter v2022 today actually :)
In addition, when I drag a ps1 script in an otterscript orchestration, the popup is blank and i can't add the script.
Does your script have any parameters? If so, that is to be expected. Its should save if you hit the save button however. Note that in Otter v2022, shipping very soon, the script editor is totally rewritten and should be fixed.
Cheers,
Alana
There is a Restart-Server OtterScript operation that you can use. You can specify a timeout/timer.
You'll want to store that chocoPackages.txt file as a script asset, and then deploy it to the server you're running the scripts on:
Get-Asset chocoPackages.txt
(
Type: File
);
However, you may want to consider this approach for your chocopackges:
https://docs.inedo.com/docs/otter-howto-ensure-chocolatey-packages
Hi @gurdip-sira_1271 ,
I want to share this guidebook that we wrote;
https://inedo.com/support/resources/ebooks/windows-first-iac-cca
Basically it discusses and walks you through how to CI/CD with your "infrastructure as code" using Otter and BuildMaster (our CI/CD tool). The screenshots are outdated, but after a lot of research, our conclusion is that the "pipeline" approach is likely the best.
Cheers,
Alana
Hi @chris_5235,
Unfortunately I'm not really sure what's happening, but I suspect it has to do with an HTTPS redirect?? I would look there next.
As you mentioned in the OP, the package can be download from the endpoint, so long as you provide the credentials. So, I guess, PowerShell isn't sending credentials after a redirect, perhaps?
One thing that you might want to do, is use a tool like Fiddler to capture the request/responses. That will let you verify that the issue is indeed PowerShell not passing credentials to ProGet. You can look at the authentication header.
Things to look at:
https download link for the package file in the FindPackagesById() requestX-Forwarded-Proto is set to be httpsBaseUrl under advanced settingsCheers,
Alana
Hi @mcascone ,
I think that was a feature that got lost in 5.3, with the new UI.
Easy enough to bring back, I think, so we'll add it via PG-2134
Cheers,
Alana
Hi @gurdip-sira_1271 ,
Your Git repository must be structured in a particular way to work as a Git Raft; but it's best to let Otter manage all of that. Here is a sample: https://github.com/Inedo/OtterRaftTest
The repository you use for a Git-based raft shouldn't be used for anything else - just raft storage.
Cheers,
Alan
Hi @gurdip-sira_1271 ,
Great, sounds like its working :)
I think the issue is that you didn't have the folder-structure in place. Scripts should go in the /scripts folder basically.
Cheers,
Alana
@nmarkkula_7220 make sure the v3 API is enabled on your feed; the endpoint will look like https://MY.FQDN/nuget/MYNUGETFEED/index.json
visual studios detects the API once it queries it, so if you changet the URL then it should work
@nmarkkula_7220 I'm not totally sure, but I'm pretty sure Visual Studio used to pay attention to that flag. Perhaps it's a regression or something in Visual Studio?
But it looks like that's on the v2 API, and Microsoft (and we) aren't really maintaining that anymore. I would try using the v3 endpoint instead.
@nmarkkula_7220 this can be a confusing issue
Listed/Unlisted is a serverside metadata, like the download count. When you cache or pull a package from nuget.org, then ProGet won't have severside metadata. This is why download counts start at zero once you upload/pull a package to ProGet.
You will have to unlist the package in Proget too.
Hi @luke_1024 ,
Unfortunately this just doesn't look like something that's going to be easy/trivial to fix. Maybe it's a bug in the compression library we're using, or maybe cargo-deb is doing something unusual with the compression format that we're not expecting, or isn't documented in the normal debian commands.
It looks like you recreate the package with dpkg-deb then it works.
We will upgrade the libraries in the next major version, and maybe it will go away - but for now it doesn't seem worthwhile fixing, To date, just a single user (you 
) has reported this, and if we hear from customers, we'll consider it too.
In the meantime, the workaround of repackaging should do the trick.
Hi @mcascone,
To do this, you'd want to download the latest version of the package, edit it in a zip file, then reupload it. A little tedios but fastest way I think
Cheers,
Alana
Thanks for letting us know, @hwittenborn
We must not have the http --> redirect setup in our DR server , we'll try to get that soon
The sort-order for feeds is not configurable the ProGet UI, but some tools (like Visual Studio) do allow for sorting.
Otherwise, we haven't had any other requests for improving the ProGet UI for Chocolatey Feeds (especially not from customers / paid users that we're aware of), but if we hear more requests we'll definitely consider improvements like this.
Hi @alin-kovacs_4228 ,
We don't maintain documentation for that version I'm afraid; the Native API is available, and you can find it in the software with /reference/api
However, your best bet would be to work with our team to figure out what you need to do and get some help migrating to a newer version or something :)
Cheers,
Alana
The high-availability / cluster configuration can be a little tricky... but glad that changing it worked.
The message "More than half of the servers are in an error state, consider restarting the BuildMaster service." must be a cached error message? We try to detect if there's a major problem with the service / agents, and then trigger that message.
Can you try to restart the BuildMaster service on each of the nodes, and see if it goes away?
Cheers,
Alana
Hi @rmusick_7875 ,
No problem, happy to help.
[1] I'm afraid it's not really possible to sort the packages; they're displayed in the order that's returned from the remote feed (i.e. chocolatey.org), and sorting isn't possible when requesting results. Typically it's done by popularity, but sometimes they come back with recently-updated. I'm afraid it's not predictable. Internally, ProGet sorts a feed by recently updated.
[2] We used to do this, but it creates a lot of confusion because many package titles look like they are a package ID. It was especially confusing for packages with a title like Initech.Utils, but with a ID of InitechUtil2 or something. NuGet.org also changed similarly.
We may consider changing the behavior for Chocolatey feeds.
Hi @marc-ledent_9164 , sorry on the slow reply, I wasn't so familiar with OpenShift so I wanted to research a little.
First, I think your Service.MessengerEndpoint should be tcp://*:4242, because you don't know which node will be active. It might be buildmaster-0, but it might not.
What also isn't clear, do you need to "open" or otherwise map port 4242? I'm thinking the service messager is working on the node that can connect to itself, but the nodes aren't communicating over the internal network.
Cheers,
Alana
Hi @marc-ledent_9164 ,
BuildMaster doesn't use a reports directory... I'm guessing someone configured that to store reports generated from custom queries using something like SQL REporting or something? Or maybe PDFS/screenshots for auditing?
Cheers,
Alana
Hello @bkohler_1524,
Because packages are immutable, you need to decide the future-compatibility at the time you create the package. This is where Semantic Versioning dependencies can really help.
Following the rules of SemVer, you can say that "ProductA 1.0.0 has a dependency on ProductB (1.1.0-2.0.0]", what that means is that you can use every version of ProductB from 1.1.10 up to (but not including) 2.0.0.
This allows you to make as many minor versions of ProductB (such as 1.200.0) before "breaking" compatibility.
If these version numbers are business/marketing-driven, and it's impossible to change their mind on versioning, you can always maintain an internal version number - this is what many products (such as Microsoft) does.
Hi @jon-benson ,
We don't have a tutorial for that I'm afraid :(
However, you're on the right track -- it does involve using the Advanced Properties on the User Directory to specify the domain controller host, credentials, and so on.
Since you mentioned you have / will purchased ProGet, let me reach out to my colleague Gene, he is our Customer Advocacy Manager and often sets-up appointments to help with onboarding/configuration for new users. This could be a chance to get some assistance on that.
Thanks,
Alana
Hi @jon-benson
Sorry for the mix-up / confusion. I'll try to clarify a few points.
LDAP/AD integration is only available on paid editions of ProGet (Basic, Enterprise), and you can integrate with Azure AD using LDAP. This requires you to type in your Azure AD username/password in the ProGet login page.
SAML/Single-Sing-On is only available in ProGet Enterprise edition. This allows you to sign-in to ProGet without typing in your Azure AD username/password. Also, this is no longer a preview/beta feature (the docs were just incorrect; I've now updated).
For ProGet v2022 (aka 6.1), we are developing an improved Security Management user interface. It's basically the same, but just easier to work with. It's available as a preview feature, and that preview is enabled by default on new installations. We just added SAML support to this new interface, so on 6.0.12 you should be able to use it.
hope this helps clarify,
Alana
Hi @nuno-ildefonso_8876 ,
Thanks for that information; this will be fixed in the next maintenance release (BM-3788), later this week (on Friday).
Hi @pariv_0352 ,
Are you able to see the results of the "DockerGarbageCollection" job?
This is actually what's responsible for deleting those images, and it runs nightly by default.
Let me share the code to it; if you can understand the database structure already, then hopefully it will help you to identify why it's not workikng, and what you might be able to look at in the logs to help troubleshoot:
[ScheduledTaskProperties(
ScheduledTaskTypes.DockerGarbageCollection,
"Deletes unreferenced Docker blobs.")]
public sealed class DockerGarbageCollectionTask : ScheduledTaskBase
{
public override Task ExecuteAsync(ScheduledTaskContext context) => this.GarbageCollectAsync();
private async Task GarbageCollectAsync()
{
using var db = new DB.Context();
this.PercentComplete = 0;
var usedBlobs = new HashSet<DockerDigest>();
this.LogDebug("Gathering list of all rooted blobs...");
foreach (var image in db.DockerImages_GetImages(Feed_Id: null))
{
if (image.ContainerConfigBlob_Id.HasValue)
usedBlobs.Add(image.ContainerConfigBlobDigest);
DockerManifest manifest;
try
{
manifest = new DockerManifest(image.ManifestJson_Bytes);
}
catch (Exception ex)
{
this.LogError($"Image {image.Image_Digest} has invalid manifest: {ex.Message}");
continue;
}
// "Fat images" do not have blobs as layers.
if (manifest.Layers == null)
continue;
foreach (var l in manifest.Layers)
usedBlobs.Add(l.Digest);
}
this.LogDebug($"Found total of {usedBlobs.Count} rooted blobs; finding unreferenced blobs...");
var allBlobs = (await db.DockerBlobs_GetBlobsAsync(Feed_Id: null))
.Where(b => b.Feed_Id == null)
.Select(b => DockerDigest.Parse(b.Blob_Digest));
var unreferencedBlobs = allBlobs
.Where(d => !usedBlobs.Contains(d))
.ToList();
this.LogDebug($"Found {unreferencedBlobs.Count} unreferenced blobs.");
using var fileSystem = new DirectoryFileSystem(ProGetConfig.Storage.DockerBlobStorageLibrary);
for (int i = 0; i < unreferencedBlobs.Count; i++)
{
this.PercentComplete = (i + 1) * 100 / unreferencedBlobs.Count;
var digest = unreferencedBlobs[i];
this.LogInformation($"Deleting blob {digest}...");
if (!ProGetConfig.Feeds.RetentionDryRun)
{
await db.DockerBlobs_DeleteBlobAsync(Feed_Id: null, Blob_Digest: digest.ToString());
await fileSystem.DeleteDockerBlobAsync(digest);
}
}
}
}Hi @NUt ,
LDAP / Active Directory integration is a paid feature, and should not be available in ProGet Free Edition. We don't intended to change that in our next version (v2022).
If you're able to log-in or use it, then it's most certainly a bug of the new Security Management preview features. Please don't rely on that, because it will probably not work in a newer version :)
Cheers,
Alana
@mcascone I don't think it's new, but it's just used to specify the dependencies field in the manifest file; I'm thinking, perhaps, it might be similar/identical to the consumes field you added?
https://docs.inedo.com/docs/upack-universal-packages-manifest
The only thing Inedo tools use it for today is just displaying information in ProGet, on the dependencies tab. That may / may not be helpful.
@ales-bahnik_2824 thanks; so that tells us the call is making it to ProGet OK, and there are no permissions issues. If there were permissions problems, you'd get an error like you did above.
Not sure what to do from here. I tried very quickly to reproduce, and the package deletes fine
PS C:\Users\atripp> Invoke-RestMethod -Method Delete -Uri "http://proget.localhost/nuget/nugets/package/Newtonsoft.Json/13.0.1" -Headers @{"X-NuGet-ApiKey"="058fca3993cde88d771b142b876913a0a126f16b"}
The delete will not produce an error if the package doesn't exist. So maybe it's already deleted.
Otherwise I'm not sure how else to debug. The method the API and Web page call are exact the same.
hi @ales-bahnik_2824 , what happens if you provide an incorrect API key? Do you get an error message?
@kichikawa_2913 the best way to troubleshoot this would be with using a tool like Fiddler to compare/contrast the request/responses from NuGet and ProGet. At this point, we're not sure why your ProGet is behaving different than our ProGet, or NuGet.org. If you can share a .saz archive, we can try to look as well.
The registrations-gz/selenium.webdriver.chromedriver/index.json is basically just asking for a list of versions. It's dynamically generated, That may not have the version NuGet wants/expects? It's really hard to guess...
@kichikawa_2913 said in Package not found restoring from ProGet but works from nuget.org:
I wasn't seeing anything in the logs before but this is what I get now. There's an initial query that looks to return a 401 then a second query after the LDAP lookup that returns a 200.
That's considered an "authentication challenge"; in the first request, Visual Studio doesn't send any credentials (and the 401 is returned), so then he retries again with credentials (hence the 200). So that isn't so unusual of a pattern, and seems to be operating fine.
What we're lookign for is which requests in Visual Studio are giving results with no packages, etc. It should be obvious from the URL, which package is being requested from proget.
Hi @kichikawa_2913,
Unfortunately I'm not able to reproduce this, and that packages are downloading just fine through ProGet, so it's a bit of guesswork as to where the issue could be.
Can you find errors on the ProGet-side around the time this is happening?
Can you isolate which specific requests are failing in ProGet? The easiest way to do this is with Fiddler (visual Studio should automatically connect to it) - and you can even share the failed requests as a .saz file so we can review as well...
Thanks,
Alana
Hi @ab-korneev_0401 ,
The npm API is very complicated behind the scenes, and third-party repositories often have bugs that cause odd behaviors when used via connector. Unfortunately it's really hard to say or figure out what's happening with out debugging.
If you can provide us with reproduction instructions like,
Then we can attach a debugger to ProGet and see where it's failing.
If http://upstream.address is sensitive or requires authentication, you can email details to support at inedo dot com. Make sure to put [QA-797] in the title, and tell us when you respond so we can fish out the address.
Cheers,
Alana
Hi @Russell-Kahler_4399 ,
It looks like there was an error running the upgrade scripts to v4.7. That would have been a long time ago, and it's hard to guess what it could be, and it'd require a bit of analysis to figure out.
I'd recommend migrating to a new instance; you can use Feed Importers in the latest version to pull all the content from your existing instance.
It's possible to bypass the errors with inedosql resolve-error --all but I really wouldn't recommend it, since it could lead to more problems.
Cheers,
Alana
@mcascone the drop path isn't really designed for this, and I wouldn't recommend that approach; it would have the effect of "overwriting" the packages, which may set new publish dates, and cause server-side metadata (like package counts) to reset
Also - if we change the path without changing anything else, will proget just start keeping new packages in the new path, and still be able to access the old path?
No; you'd get "package file not found errors", since the disk path is always constructed from those values.
@mcascone said in Proget: retention policy for branches in a package:
This means mybranch and mybranch2 can be reduced to mybranch2.
Sorry but wouldn't this be the reverse:
*mybranch*will match*mybranch2*?
What I mean to say.... because it's an AND conditional, the *mybranch* is effectively ignored. Everything that matches *mybranch2* will also match *mybranch*, but the opposite isn't true. E.g. mybranch1 won't match both conditions.
@mcascone said in Proget: retention policy for branches in a package:
in this feed, delete matches of 'mybranch', except the latest 3 versions of those matches, which would only impact the versions matching mybranch and leave all other non-matches untouched;
Correct. And do note that you can set retention policies to run in dry mode, where nothing is deleted, to verify it's the behavior you want.
Hi @mcascone ,
Good question; the documentation isn't very clear. How do you feel about this, I just updated the docs :)
If you want to change the directories your packages are stored, you'll also need move/copy the contents from the current location to the new location. We generally recommend:
You can also keep ProGet online the entire time; this will just cause a number of "package file not found" errors if anyone tries to download the package before the transfer is complete.
Depending on how many packages files you have, transferring may require a significant amount of time; you may not wish ProGet to be offline or for users to experience errors during the process. In this case, we recommend first mirroring the files using a tool like robocopy /MIR a few times (just in case packages were uploaded during the initial copy), and then changing the settings in ProGet.
Hi @mcascone!
To the "simple" question about the retention behavior, each retention rule starts by building a list of all packages. It loops over every package, and removes items from the list based on criteria you select. The packages not removed from the list are deleted. This ultimately has the effect of having everything be an "AND" in a single rule. This means *mybranch* and *mybranch2* can be reduced to*mybranch2*.
The rules run one after another. So the second rule would start with a new list, and elimate items based on what you checked off.
To the more complex question... why not just let the "dev" packages packages get messy? You can use a time- and/or usage-based rule. That might simplify things a lot. You can enable differential storage on Windows, which will reduce real space consumtion by like 90% or more.
Or maybe use a different feed? Just throwing ideas out :)
Cheers,
Alana
@can-oezkan_5440 thanks for letting us know what the issue was :)
Hopefully this is a trivial thing to fix in our code; we'll take a look and let you know!
@jan-primozic_9264 thanks for posting the update!
Please let us know if you can see a place for us to improve documentation :)
Hi @mcascone
Group names are optional with universal packages.
For example, we don't use them in this feed:
https://proget.inedo.com/feeds/BuildMasterTemplates
Unfortunately I'm not totally sure where the issue is, or how to troubleshoot the Jenkins plugin; it was created by the community, but I think it's possible to submit a pull request if there's an issue?
Perhaps thsi should check for a groupName, and not append property if null?
Are there any errors on the ProGet side?
Thanks,
Alana
Hi @p-boeren_9744 ,
The documentation isn't very clear
I had to look this up myself in the code. If you set the allowed property, then a global rule is also created. Therefore, the following should work instead:
{
"licenseId": "package://@progress/kendo-react-grid/5.0.1/",
"title": "package://@progress/kendo-react-grid/5.0.1/",
"urls": [
"package://@progress/kendo-react-grid/5.0.1/package/LICENSE.md"
],
"allowedFeeds": ["NpmLicenseTest"]
}
Cheers,
Alana
Hi @bryan-ellis_2367 ,
I'm not an Azure DevOps expert, but last I checked, it's not possible to add NuGet package sources other than it's own ADO Packages product or the public repositories. That may just refer to "upstream sources", but I'm not totally sure.
However, if you want to use the ADO Pipeline's built-in NuGet commands to publish packages, I guess you can set up a service connection using this?
Not totally sure -- but please let us know what you find :)
Cheers,
Alana
Hi @moriah-morgan_0490 ,
Glad to see the environment un-scoping worked! It's definitely possible to get environment-scoping of credentials to work... but it'd probably be best to confirm what you're looking to accomplish.
The main purpose of the environment-scoping is to enable limited management access to Otter. For example, users can edit/maintain all configuration except for production servers. Take a look at Multiple Environments per Server to see how the behavior works.
I assume that you're running the Inedo Agent? We're still learning the exact privileges ourselves 
We've seen systems restricted in very unexpected ways (and only in the field of course, never or own environments), and they don't give any logical error messages. But we'd love to help you get this working, so we can document it.
Here's what we know so far:
Let us know what you find :)
Hi @mcascone,
We don't have that functionality in ProGet, but it should be pretty easy to do with a pair of Invoke-WebRequest PowerShell commands :)
You could probably parse/scrape the HTML and download / upload in bulk as well.
Please share the script if you end up doing that, it might be useful for other usecases as well!
Cheers,
Alana
Hi @moriah-morgan_0490 ,
If you've created a Secure Credential (Username & Password) under Admin > Secure Credentials that's named TestCred2 then the OtterScript you presented should work. There are several challenges with getting impersonation to work Windows in general (as I'm sure you've learned as a Windows admin), so we don't recommend using it unless you have to.
In your Otter configuration, my guess is that there is a problem with environment scoping; that can get a bit tricky - I would just set it to use (Any) environment for now.
As for "Secure" part of "Secure Credentials" - the password fields are stored as encrypted data, and you can only decrypt them with the Encryption Key (stored separately). They're also held in memory as securely as possible, and are "handed" to PowerShell as a PSCredential object.
The "Allow encrypted properties to be accessed" prevents OtterScript from exposing the passwords using the $CredentialProperty() function. For example, this would cause an error unless that was checked:
Log-Information $CredentialProperty(TestCred2, Password);
However, PowerShell has no such restriction surrounding credentials. For example, you can always just do this:
$credtest = Get-Credential
Write-Host $credtest.GetNetworkCredential().Password
Microsoft's recommended way to handle this is to use Windows Authentication.
Our recommendation is to generally avoid passing important credentials to scripts (API keys, etc. seem fine)... but if you must, use a change process to ensure that you aren't running scripts that dump passwords like that.
Hope that helps,
Alana