Best posts made by atripp

Hello; I've updated the documentation to clarify this, but it's available starting in ProGet 5.2.9. So, you'll need to upgrade to enable it :)

Thanks for the update! I've noted this in the docs, and linked to this discussion :)

https://github.com/Inedo/inedo-docs/commit/d24087911584bbda833314084a58c2ae1ff41c39

Hello,

That API will only delete package metadata from the database, not from disk. It's mostly intended for internal use only, and probably shouldn't be exposed to the API. In any case, we don't store the @ with internally, so if you change @myscope to myscope it should work.

Note that the NPM doesn't provide a way to delete packages, and we never implemented it. There hasn't been any demand for it to date, as people don't really delete packages programmatically - but you're definitely welcome to submit a feature request and help us understand why it'd be a value (like, the workflow you use that requires deleting packages, etc).

Alana

Hello, for sure!

It's pretty easy; just don't give the Anonymous user any access to your feeds, and then authentication will always be required, either when browsing the ProGet application or using the API (such as Install-Module).

When you use the Register-PSRepository command, you can the Credential option to specify a credential.

This credential can be the name/password of a user inside of ProGet (let's say, Admin:Admin), or it can be username of api with a password of an api key you've configured (so, api:my-secret-key).

Hello;

The Native API is for low, system-level functions, and it's "all or nothing". If you give someone access to Native API, you are effectively making them an administrator, as they can also change permissions and grant admin privileges. So, I don't think you want this. Instead, you'll want to use the Debian API endpoint that we implement.

It's a third-party API format

In order to support third-party package formats types like NuGet, npm, etc., ProGet implements a variety of third-party APIs. We only provide minimal documentation for these APIs, as they are generally either already documented elsewhere. However, you can generally find the basics by searching for specific things you'd like to do with the API, such as "how to search for packages using the NuGet API" or "how to publish an npm package using the API".

So in this case, I recommend to search "how to view and download apt packages".

I'm not very familiar with PyPi packages, but I know there are some oddities with - and _, and that they are sometimes supposed to be treated the same, and sometimes not. We don't totally understand all the rules, to be honest (even after reading PEP503 specifications).

In this case, the package is actually websocket_client, not websocket-client.

See: https://pypi.org/project/websocket_client/

When you search for websocket_client in ProGet, it shows up, as expected.

In ProGet 5.3, we plan to have a couple tabs on each Tag (i.e. container image) that would provide this info: Metadata (will be a key/value pair of a bunch of stuff), and Layers will show details about each of these layers.

That might help, but otherwise, we have retention policies which are designed to clean up old and unused images.We'll also have a way to detect which images are actually being used :)

As @apxltd mentioned, we've got a whole bunch planned for ProGet 5.3.

I've logged this to our internal project document, and if it's easy to implement in ProGet 5.2 (I can't imagine it wouldn't be), we'll log it as a bug and ship in a maintence release.

Do note, this is not an IMAGE description, it's a REPOSITORY (i.e. a collection of images with the same name, like MyCoolContainerApp) description; so this means the description will be there on all images/tags in the repository.

You're right, I guess that's showing the "layers" instead of the "tags"; I think it should be showing container registries separately (they're not really feeds), but that's how it's represented behind the scenes now.

Anyways we are working on ProGet 5.3 now; there's a whole bunch of container improvements coming, so I've noted this on our internal project document, to make sure we get a better display for container registries.

We've got some major container improvements coming in ProGet 5.3, and will revamp our product; hopefully we'll be able to present this pretty soon!

I think, once you see what we have planned, you'll want to change/improve your workflows to simplify things, and this may not even be necessary... anyways, stay tuned.

This is was addressed in PG-1795, which was released in 5.3.9 just recently!

You can reset the Admin user account password by stopping the ProGet Windows service, then running ProGet.Service.exe and selecting the "ResetAdminPassword" option. Make sure to restart the Windows service when finished.

Hello;

That "error" message is to be expected; that's what ProGet returns when you visit to /v2 with a web browser. But the error sounds like a sort of SSL/certificate configuration problem.

After searching for "tls: no renegotiation", here is an article I found about that might help: https://www.digicert.com/news/2011-06-03-ssl-renego/

Hi @scott-wright_8356,

I looked into this further; for Maven feeds, the Connector Health Check simply queries the downloaded index. If there is no downloaded index, then it reports back as healthy. If there's an error with the index, it reports an error. However, most maven repositories don't have a downloadable index.

Anyway the health check is not very useful, and we plan to revisit the functionality in the future, since there's really no way to determine if ProGet can download an artifact unless you know the exact name.

Thanks,
Alana

Hi all, just an update! We will be shipping a potential fix in PG-1783, which adds a new checkbox in advanced settings (unchecked by default):

Close Database Connections Early
EXPERIMENTAL (for debugging/testing only) - As of ProGet 5.3, database connections are left open during the lifecycle of a NuGet API request as a means to reduce overhead; however, this may be causing ProGet to run out of available connections to SQL Server. Set this value true to open/close database connections as needed on NuGet feeds.

We'll update when this is shipped --- but if we can get some folks to verify that this works better (we can't repro, at all ), then we will likely make it the default. Hopefully this will do it. Seems better than raising connection pool limits

If so, then the savings in connection open/close overhead don't seem to make it particularly worthwhile. This "keep open" technique made a ton of differences elsewhere in our software, but since a NuGet API requests may yield a ton of other network requests (via connectors) and block, the pool may be getting drained too quickly...

just a theory, as I mentioned. Anyway hope this helps, stay tuned!!

Thanks!

Hi @Justinvolved ,

Oooh - I thought we removed those flags/notices from BuildMaster 2022

Looks like we only removed the code that clears them, and almost all of the code that sets them (with the exception of the CEIP page).

Anyway, I've vanquished the code for good via BM-3839.

In the mean time, please run sql this command to clear the notices:

DELETE [Configuration] WHERE [Key_Name] IN ('Service.DelayExecutionsWhenRestartRequested','Service.RestartRequested', 'Web.RestartRequested')

Cheers,
Alana

ProGet has a really cool feature called Semantic Versioning for Docker Container Images, and it works pretty much how you described, but it also enforces that containers have a valid semantic version tag. For example, latest will always refer to the highest, stable-version of a container, and 4 will be the latest stable of 4.x.y

As for "generic archives", check out Universal Packages - they're really powerful, and are like "nuget/maven" but for your own applications and components, and extensible.

ProGet will use the v3 (JSON-LD) API when possible, but not all v2 (ODATA) queries can be "translated" to the v3 (JSON-LD) API, and thus they need to be forwarded to the v2 (ODATA) end-point.

ODATA is a general-purpose querying API (like SQL but for the web), so clients do a lot with it (sorting, etc.). The v3 (JSON-LD) API is much more limited. So I recommend you to update the clients.

@atripp said in Creating PowerShell repository, protecting pull/download by API key:

this credential can be the name/password of a user inside of ProGet

Or a user that's configured in your Active Directory, assuming you have enabled that integration.

Hi @mcascone, I'm afraid it's not possible.

nuget or dotnet nuget tools only understand the NuGet Package Format, and the NuGet API, so there's no way for the tools to use different formats or different APIs...

@scroak_6473 we could definitely try a screen share, but in a case like this (where we have no idea what's wrong), it's mostly digging in the code and trying to think of things to try to get a clue for more information. Currently, I'm at a loss... because the error you have shouldn't be happening, but it clearly is.

So now, I had a new idea. I would like to eliminate Docker from equation, as it handles the
"api" username slightly differently than everywhere else. Plus you can do this all in your browser.

Can you try to visit a restricted (i.e. not anonymous view) NuGet endpoint using the "api" user name, and a password?

For example, it should look like /nuget/NuGetLibraries/v3/index.json, and then your browser should prompt for a Username/Password.

Depending on the result of this, we will explore different code paths, and then might need to add some more debugging codes.

Best,
Alana

Hello;

That search syntax is really only supported by NuGet v3 API, I think; so, ProGet simply forwards on the query to that API, and returns the results.

But regardless, connector filters need to be applied after the remote feed returns results, because connector filter logic can be more complex that what is supported by the various feed APIs (you can allow Microsoft.* and Inedo.* for example).

More advanced connector filter options are definitely something we've considered, and we'd love to do things like "version: 3.*" for example. But, it's a lot more complicated under the hood, and probably isn't even feasible given the nature of feeds.

Alana

Hello; you should see a checkbox, like this.

However, due to a bug (now fixed via PG-1885) the checkbox wasn't being displayed due to a license validation problem.

It will be fixed in the next maintence release, scheduled for later today

@joshuagilman_1054 I don't really know PowerShell myself (today I learned you can do classes )... but behind-the-scenes it's .NET, and that means we can use nullable value types.

I tried [int?] (nullable shortcut syntax) and [Nullable<int>] (generics shortcut syntax) but PowerShell isn't so happy with either. So the long way it is...

[System.Nullable``1[[System.Int32]]] $myInt = 0
echo "myInt is $myInt "
$myInt = $null
echo "myInt is $myInt "
$myInt = 1000
echo "myInt is $myInt "

That should do the trick for you, and is close to our JSON Model anyways.

Hi @Stephen-Schaff ,

I'm not sure about the API, but ProGet implements the Chart Repository API, and from a look at that, you should be able to just access index.yaml at the API Endpoint URL

Here is some information about the format of that file: https://helm.sh/docs/topics/chart_repository/#the-index-file

Can you give that a try and let us know what you find out? Cheers :)

Cheers,
Alana

Hi @sejhemming_5842

Is there an accepted Code of Practice for managing prerelease stuff?

Yes, the general rules to follow are these:

• Packages are immutable; do not delete/republish packages as part of your normal workflow
• Use Prerelease Packages & Repackaging to take tested/validated prerelease packages to stable packages
• Use Package Promotion to move packages across feeds
• Use Retention Rules to automatically cleanup unused prerelease packages

Do people publish prerelease stuff to a different feed and only post the released stuff to the more public feed? Or is it just expected that once the release is out that the prerelease builds are simply removed from the feed?

Yes to both

On our ProGet Instance, we use both patterns.

• Extensions (plugins) have two feeds: Extensions (Stable) and PrereleaseExtensions; the reason is that we use CI (build automation), and publish a new package on every commit, and these could be really unstable - we don't want anyone using a prerelease extension unless we explicitly point them to it
• Our NuGet libraries have one feed (NuGetLibraries) that has both prerelease and release packages; these are not intended for anyone other than Inedo Engineers, and we have policies, practices, and training in place to make prerelease packages are shipped appropriately

My advice for deciding which pattern to follow would be looking at the consumers of your feeds/packages (i.e. who uses your packages vs who publishes your packages).

Using a single feed that has release and prerelease packages requires more training for developers. If one of your developers accidently uses a prerelease package and commits that, then it's going to cause problems. Even if you catch it before production, it will waste time and resources.

@Stephen-Schaff said in API to apply an Alternate Tag to Docker Container Image:

Here is the PowerShell function that does both the promotion and the tagging (incase anyone ever needs something like this). Kind of a "do it yourself" repackaging. (Might be nice to have the Repackaging API support Docker container images someday).

Thanks for sharing this, I've added it to our Semantic Versioning for Containers docs page, I hope that's okay :)

And yes I agree, it woudl be nice to make this an easier API call

@Stephen-Schaff you got it! We'll try to get PG-1996 added in the next maintenance release. Should just be a pretty straight-forward loop :)

@kichikawa_2913 glad it's working now, you seem to have found the issue.

FYI, you can also configure that value from Advanced Settings page under Admin.

Hi @Justinvolved ,

The first thing I would try is enabling verbose logging; this is on the advanced tab of the operation, or you can set with Verbose: true parameter.

That will give a file-by-file comparison, and show you what's being copied and not. You will see texts like:

Copying C:\apxltd\artifact\Inedo.DependencyScan.dll to C:\apxltd\artifact2\Inedo.DependencyScan.dll...
Inedo.DependencyScan.dll already exists in C:\apxltd\artifact2.
Source timestamp: 11/12/2022 8:26:38 AM, Target timestamp: 11/12/2022 8:26:38 AM
Source size: 54272, Target size: 54272
Size and timestamp are the same; skipping Inedo.DependencyScan.dll...

Hopefully that will help trace this a little bit better.

Cheers,
Alana

Hello,

The NuGet Client only uses the the -ApiKey you on certain requests to the -Source, specifically the PUT of the package file. Before doing that, it will query the package source using a GET, and in this case, the API Key isn't sent.

The easiest solution is to allow anonymous view access to your feed. That simplifies configuration for all users of your feed.

Alternatively, you will need to add credentials to the PackageSourceCredentials in the NuGet.config file . Note this file exists in several places, please see Config file locations and uses

Cheers,
Alana

Hi @Justinvolved ,

Hundreds of configuration files in one application -- definitely too much. I'd probably seek a different solution if that's the case. But having one or a few per application is okay.

There is an Http-Post Operation that you can probably use to make API calls. Conceptually it's similar to the PowerShell Invoke-WebRequest method.

I'm not sure what the configuration file would look like (array? map variable?), but there's also a variable function, ($Filecontents()), that could read a file, and an $Eval() function that can can convert text into variables.

That said... it might be a bit challenging to do all this in OtterScript. It's not really designed for this. You may be better off writing a global PowerShell script that can process input from a configuration file that you deploy to the working directory. That would also be easier to test as well.

Cheers,
Alana

Hi @mcascone ,

I think that was a feature that got lost in 5.3, with the new UI.

Easy enough to bring back, I think, so we'll add it via PG-2134

Cheers,
Alana

@shaun-d-scott_2657 Inedo products do not use log4j, so it's not an issue for ProGet :)

See more information here: https://blog.inedo.com/log4shell-high-severity-vulnerabilities

Well, that's an interesting way to specify an embedded license file. I don't know if that's a convention or specification, but that seems to be a new way of handling it. It's kind of documented now, which is good: https://docs.npmjs.com/cli/v8/configuring-npm/package-json#license

Anyways, we already handle this for NuGet packages using a URL convention like this:

• No license - packageid://Aspose.Words/21.9.0
• File license -package://Aspose.Words/21.9.0/License\Aspose_End-User-License-Agreement.txt

Those will be defaulted in the fields if the package specifies no license or a file license.

Not sure if it works now for npm packages, but it'd be relatively easy to adoption that convention, and then suggest it when the "license" field starts with "SEE LICENSE IN"...

Anyways we'll investigate this and update in a day or two.

Cheers,
Alana

Hi @cronventis ,

Great find - that seems to explain what we're seeing: containerd reports on containers differently than dockerd. So, we'll just search for container images based on configurationblob_digest OR image_digest

This change was trivial, and will be in the next maintenance release (or available as a prerelease upon request) as PG-2081 - scheduled release date is Feb 11.

Cheers,
Alana

Hello @OtterFanboy ,

Thanks for the bug report! We'll fix it via OT-483.

That tab only displays if "Configuration drift:" is set to something other than "None", but there should be a "no roles" type of message instead of a crash.

Cheers,
Alana

@OtterFanboy got it, thanks!

So this particular change is a bit more involved unfortunately. For example, if you clicked on "Windows Server", you wouldn't find any drifted servers - which would be confusing, because the environment would show as drift.

I'll note this as something to think about for v2023

Hello @OtterFanBoy123 ,

I was able to reproduce this; it's a UI regression BuildMaster 2022's new OtterScript editor. This won't impact deployment, the script, or anything like that.

I've logged this as BM-3817 and we'll try to get this fixed in the next maintenance release :)

Cheers,
Alana

Hi @chris_5235,

Unfortunately I'm not really sure what's happening, but I suspect it has to do with an HTTPS redirect?? I would look there next.

As you mentioned in the OP, the package can be download from the endpoint, so long as you provide the credentials. So, I guess, PowerShell isn't sending credentials after a redirect, perhaps?

One thing that you might want to do, is use a tool like Fiddler to capture the request/responses. That will let you verify that the issue is indeed PowerShell not passing credentials to ProGet. You can look at the authentication header.

Things to look at:

• make sure ProGet is sending an https download link for the package file in the FindPackagesById() request
• X-Forwarded-Proto is set to be https
• If you must, set BaseUrl under advanced settings

Cheers,
Alana

Hi @NUt ,

LDAP / Active Directory integration is a paid feature, and should not be available in ProGet Free Edition. We don't intended to change that in our next version (v2022).

If you're able to log-in or use it, then it's most certainly a bug of the new Security Management preview features. Please don't rely on that, because it will probably not work in a newer version :)

Cheers,
Alana

Hi @karsten-meichsner_9039 ,

This error is happening at the SSL-level, which is managed by the operating system (platform). There's nothing you can configure in ProGet to resolve this, so you'll need to make some changes to the operating system.

Can you tell me, are you using ProGet for Windows, or on Linux (Docker)? In either case, you'll need to trust the root certificate, but the ways to do this are a bit different.

In Windows, the easiest way is to install the certificate using the UI (Certificates-Current User > Trusted Root Certification Authorities into Certificates (Local Computer) > Trusted Root Certification Authorities). You can also verify that the installation was successful by trying to navigate to a URL in your S3 bucket.

In Linux/Docker, it's a bit trickier; in general you'll want to copy the .crt into /usr/local/share/ca-certificates and then run /usr/sbin/update-ca-certificates. There's a few ways to do this, but one common way is to build a docker image on top of our image. You can also SSH into the running container and handle this as well.

Cheers,
Alana

@kichikawa_2913 we don't have a typical set-up per se, really

A lot of set-ups are very simple, since the team is new to CI/CD, and they just have the BuildMaster server do everything.

Others have dozens of build agents configured in a pool to service various teams, and use lots of shared configuration.

Most are somewhere in the middle -- since they started simple, and gradually explored over time. That's generally what i'd start with -- tackle one pattern/team, then move on to the next and improve over time.

Hi @karsten-meichsner_9039,

I'm glad you could solve the issue! I'm not very familiar with Ceph, but it sounds like it works with the AWS API, which we use to talk to AWS. I added a brief notes to the docs about Ceph/RGW usage, but if you have suggestion to improve, please let me know :)

As for the "Use server-side encryption" option, the checkbox sets the option on the API Request (e.g. CopyObjectRequest) to use AWS Server-side Encryption. ProGet does not encrypt the data.

Cheers,
Alana

Hi @Justinvolved ,

Ah, thanks for clarifying that! Yes... they should be Testing to follow everything else. Pipeline templates are a brand-new feature :)

This was a trivial change, and will be updated in next version as BM-3805

Cheers,
Alana

Hi @torgabor_4445 ,

I believe that, when creating the npm connector, you can select "Authentication: Bearer", and then enter email and your token for the username.

Let us know if that works!

Cheers,
Alana

Hi @Justinvolved ,

In this case, I would recommend staging the artifact files first, and then transferring them. Here is the OtterScript you can use for that:

Deploy-Artifact;

Transfer-Files
(
    ToDirectory: c:\websites\the-website-root,
    Exclude: Images\**
);

Cheers,
Alana

Thanks @kichikawa_2913 ! We'll add this to our 2023 roadmap review, and we can at least spend sometime reasearching / learning more about it then.

Hi @doejohn_7742 ,

The underlying error message "Invalid object name 'dbo.RpmPackages' is implying that there's something pretty wrong with the database; in this case, a missing table (RpmPackages).

If that's the case, it wouldn't be easy to fix or troubleshoot unfortunately, and would require SQL expertise, etc. We don't have any general advise for this, but we may be able to help some paid users depending on the issue (it's quite time consuming as you can imagine).

HOWEVER -- it could also be something really simple, like your username (i.e. set in the connection string when you upgrade) is in the wrong schema (needs to be dbo schema), so I would check that too. Our script is supposed to detct that, but sometimes it doesn't work.

Cheers,
Alana

@Stephen-Schaff thanks for the bug report, I verified that this may happen depending on permission of user, and which feeds they can/can't use --- but it seems an easy enough fix that we can do via PG-1894 (targeted to next release) - the packages can't be viewed upon clicking, but it's a sub-optimal experience for showing packages they can't see