Posts made by Stephen.Schaff

We use the ProGet API to promote docker images from a PreRelease feed to a Release feed. This API takes a version property in the request body that is the Tag of the container image.

Today we ran into an issue where we needed to promote a container image, but it did not promote. Further investigation found that the container image was already in the Release feed, but it did not have the tag that we were trying to promote. We assume that ProGet saw that the container was already in the target feed and so ignored the request.

We tested this using the UI and found that it also ignored our promotion request.

We were able to work around this issue by using the UI to manually add an additional tag to the container image.

However, we use tags in our process to refer to container images. Even though we know that the image is actually uniquely defined by its digest, the tag is very important to us.

I would like to requests that when a Docker Image promotion request arrives to ProGet, if it sees that the container image is already in the target feed, that it looks at the tag that was used in the request, and if the tag is not on the container image in the target feed, it adds the tag to the image.

So it would follow this logic structure:

1. A container image promotion requests arrives to ProGet (via the UI or the API)
2. ProGet checks if the container image already exists in the target feed. If not, normal promotion logic occurs.
3. If yes, then ProGet checks if the tag used to request the container image promotion is on the container in the target feed.
4. If not, then ProGet adds the tag to the container in the target feed.

If this is not acceptable, then I would suggest that the request to do the promotion API respond with a 403 - Already Exists response. That way we can know that the promotion was of something that was already there and know to go add the tag.

@atripp

I got this working. Here is my PowerShell script to get a Docker Token from ProGet:

function GetDockerToken() {

    param (
        [string] $packageName = $(throw "-packageName is required.  This is the namespace and image name.  For example: library/my-container-image"),
        [string] $feed = $(throw "-feed is required"),
        [string] $actionToAuthorize = $(throw "-action is required.  This is the docker action to be authorized (pull, push, delete, etc)"),
        [string] $apiKey = $(throw "-apiKey is required"),       
        [string] $progetBaseUrl = $(throw "-progetBaseUrl is required. "),
	[string] $service	
    )


	if ($service -eq "") {
          # This expects that $progetBaseUrl is prepended with "https://"  If you are using "http://" then change 8 to 7 below.
	  $service = $progetBaseUrl.SubString(8,$progetBaseUrl.Length-8)
	}
	
	$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f "api","$apiKey")))
	$response = Invoke-WebRequest -Uri "$progetBaseUrl/v2/_auth?service=$service&scope=repository`:$feed/$packageName`:$actionToAuthorize" -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)} 
	if ($response.StatusDescription -eq "OK") {
		$token = ($response.Content | ConvertFrom-Json).token
		$token
	}
}

This can then be used like this:

$pullToken = GetDockerToken -feed $toFeed -packageName $packageName -actionToAuthorize "pull" -apiKey $apiKey -progetBaseUrl $progetBaseUrl
$manifest = Invoke-WebRequest -Uri "$progetBaseUrl/v2/$fromFeed/$packageName/manifests/$fromVersion" -Method GET -Headers @{Authorization=("Bearer {0}" -f $pullToken)}

Since this using the API Key for calls like this is no longer supported in ProGet 6, you may want update your documentation here: https://docs.inedo.com/docs/proget-docker-semantic-versioning

This page has a script I wrote a while back to repackage a container image that uses API Keys on the ProGet Docker API. I posted an update to the script using Docker Tokens on the original thread that caused its creation: https://forums.inedo.com/topic/3255/api-to-apply-an-alternate-tag-to-docker-container-image/4

If you like, you can update the documentation with the updated script that is compatible with the removal of API Keys functionality on the ProGet 6 Docker API.

Version 6 of ProGet removes the ability to make calls to the docker API via a ProGet API Key.

As such the code posted above no longer works. I updated it to work using the Docker Token based Authentication system. I am posting it here in case someone comes along this post and wanted to use this code:

function RepackageContainerImage() {

    param (
        [string] $packageName = $(throw "-packageName is required.  This is the namespace/image-name"),
        [string] $fromFeed = $(throw "-fromFeed is required"),
        [string] $toFeed = $(throw "-toFeed is required"),
        [string] $fromVersion = $(throw "-fromVersion is required.  This is the current tag on the image in the fromFeed."),
        [string] $toVersion = $(throw "-toVersion is required.  This is the tag to be applied after the image is promoted."),
        [string] $comments = "Promoted by automation",
        [string] $apiKey = $(throw "-apiKey is required"),
        [string] $progetBaseUrl = $(throw "-progetBaseUrl is required")
    )
    
	# Promote the Container Image
    $postBody = @{
        packageName="$packageName";
        groupName="";
        version="$fromVersion";
        fromFeed="$fromFeed";
        toFeed="$toFeed";
        comments="$comments"
    }         
    $promoteResponse = Invoke-WebRequest -Uri "$progetBaseUrl/api/promotions/promote" -Method POST -Body $postBody -Headers @{"X-ApiKey"="$apiKey"}
    
    # Retag the container image by downloading the manifest and then re-uploading it as the new version
    $pullToken = GetDockerToken -feed $toFeed -packageName $packageName -actionToAuthorize "pull" -apiKey $apiKey -progetBaseUrl $progetBaseUrl
    $manifest = Invoke-WebRequest -Uri "$progetBaseUrl/v2/$fromFeed/$packageName/manifests/$fromVersion" -Method GET -Headers @{Authorization=("Bearer {0}" -f $pullToken)}
    	
    $pushToken = GetDockerToken -feed $toFeed -packageName $packageName -actionToAuthorize "push" -apiKey $apiKey -progetBaseUrl $progetBaseUrl
    Invoke-WebRequest -Uri "$progetBaseUrl/v2/$toFeed/$packageName/manifests/$toVersion" -Method PUT -Body $manifest.ToString() -Headers @{Authorization=("Bearer {0}" -f $pushToken)}
}


function GetDockerToken() {

    param (
        [string] $packageName = $(throw "-packageName is required.  This is the namespace and image name.  For example: library/my-container-image"),
        [string] $feed = $(throw "-feed is required"),
        [string] $actionToAuthorize = $(throw "-action is required.  This is the docker action to be authorized (pull, push, delete, etc)"),
        [string] $apiKey = $(throw "-apiKey is required"),
        [string] $progetBaseUrl = $(throw "-progetBaseUrl is required"),
	[string] $service	
    )

	if ($service -eq "") {
	  $service = $progetBaseUrl.SubString(8,$progetBaseUrl.Length-8)
	}
	
	$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f "api","$apiKey")))
	$response = Invoke-WebRequest -Uri "$progetBaseUrl/v2/_auth?service=$service&scope=repository`:$feed/$packageName`:$actionToAuthorize" -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)} 
	if ($response.StatusDescription -eq "OK") {
		$token = ($response.Content | ConvertFrom-Json).token
		$token
	}
}

@atripp do you have any docs on how to do this token authentication with proget? I just need a way to curl or otherwise do a GET to see if the container is there already.

We recently upgraded from ProGet 5.3.38 to 6.0.6. It worked fine for a few days, but then part of our automation tried to use an API Key to check if a container exists in an repository exists. This check was working fine before we upgraded, but now is failing.

Here is the command that was run (in PowerShell):

$manifestResponse = Invoke-WebRequest -Uri "https://myProGetServer.net/v2/MyFeed/PackageLibraryHere/PackageNameHere/manifests/1.0.0" -Method GET -Headers @{"X-ApiKey"="___API_KEY_HERE__";"accept"="application/vnd.docker.distribution.manifest.v2+json"}

Now when this runs it returns:

{"errors":[{"code":"UNAUTHORIZED","message":"Anonymous is not permitted to perform the Feeds_ViewFeed task for the current scope.","detail":[{"Type":"repository","Name":"MyFeed/PackageLibraryHere/PackageNameHere","Action":"pull"}]}]}

Prior to upgrading this API Key had a user that it ran as. After upgrading to version 6, it seems to have been auto-converted to a "Personal" API Key. It still has the correct user on it post upgrade and the API Key seems to be unchanged (though I can't really verify that).

To see if it was specific to this user, I created a Feed API Key (assigned to my feed). When I tried the call with that API Key (which looks longer than the old ones), it still failed with the same "Anonymous is not permitted" error.

I want to reiterate that this was tested and was all working fine before we upgraded to version 6.

How can I get this call to check if a Container is already uploaded working correctly again?

Say I have a Helm feed with 20 different charts in it.

If I setup that feed like below (delete all but the latest 100 versions):

Will that keep the newest 100 helm chart version in the feed (up to 100 in total)? Or will it keep the newest 100 versions in each chart (up to 2,000 charts in total)?

I had to get into my permissions for ProGet again today. As I am doing so, I end up wondering why I have some permissions set the way they are. I then have to try to figure it out again (and hope I don't miss anything).

I would like to suggest that you allow adding a comment/note next to permission entries. This will allow explanations of permissions for future maintainers.

Basically it would hold comments like:

• Don't add normal users to to this feed, it contains sensitive information
• This restriction is needed to keep the "general use" api key user out of this feed.
• This feed's department requests to be contacted before new users are added to the feed.

This is not a high priority need, but would be useful (at least to me).

I created a new feed and then I added the publish permission for that feed to an existing user. Then I had to wait 30 minutes for it to take effect.

I am running 5.3.39 (Build 2). I only run a single ProGet server instance (no HA cluster).

Again, this is not a really big issue. Waiting does cause it to start working, but I would suggest firing an update to your cache when a user's permissions are changed.

I hit this again today. While waiting 30 minutes does work, it is a bit frustrating.

If not not a lot of work, I would suggest adding a button somewhere to force the update for a given user.

@atripp - That worked!

A bit odd to me that removing the Container Image and re-adding did not work, but adding a tag did.

Either way, I have a work around for when this happens in the future.

Thank you!

@rhessinger,

How many tags does the build/ops-dotnet-6.0 have?

The build/ops-dotnet-6.0 repository has only one container image it it and it has only the 1.0.1 tag.

Does the base-prerelease registry also have strict versioning enabled?

Yes it does. And the latest tag is working there.

Do you have your Sonatype vulnerability source attempting to scan your Docker registry?

We are configured to use OSS Index as a scanning source.

I removed any scanning, deleted the container image and repository and then re-promoted the container. As you indicated it would, this did not fix the problem. (The latest tag is still not applied to the container.)

Any ideas on how I can get the latest tag to be auto applied?

I have a docker feed. It is called base. (Because it holds my base images.)

I have this feed setup to use SemVer 2:

And for most of my repositories in that feed, it works just fine. Here is an example:

However, for one of my repositories, it is not working:

And if I try to add the latest tag, it will not work, because of the SemVer 2 restrictions!:

So it knows it should be doing SemVer 2, but it is not applying the latest tag.

Note: I tried to just pull the latest tag directly, to ensure that it was not just a UI issue. It gave me an error:

But if I put in the actual tag for the only container image that is in that repository, it works:

What can I do to get this repository to apply the latest tag?

NOTE: incase it is needed, we are running version 5.3.39 (Build 2)

Update:
I tried deleting the container images in the build/ops-dotnet-6.0 repository, and then deleting the repository itself.

I then promoted the container image from my base-prerelease feed. (Note that in that repository, the latest tag is correctly applied.) When I promoted it, the build/ops-dotnet-6.0 repository was created again, with the container image in it (tagged with 1.0.1). But the latest tag was not applied.

So deleting and recreating does not seem to fix the problem...

Update 2:

Digging a bit deeper, I noticed this message in the logs:

 Logged: 12/14/2021 8:51:00 PM
 Level: Warning
 Category: Web
 Message: Could not update vulnerabilities on demand for package base:build/ops-dotnet-6.0:<redacted>: Object reference not set to an instance of an object.
 Details: System.NullReferenceException: Object reference not set to an instance of an object.
at Inedo.Extensions.Sonatype.VulnerabilitySources.OSSIndexVulnerabilitySource.<>c.<GetCoordinateChunks>b__15_0(IVulnerabilityPackage p)
at System.Linq.Enumerable.WhereSelectArrayIterator`2.MoveNext()
at System.Linq.Enumerable.Sum(IEnumerable`1 source)
at Inedo.Extensions.Sonatype.VulnerabilitySources.OSSIndexVulnerabilitySource.<GetCoordinateChunks>d__15.MoveNext()
at Inedo.Extensions.Sonatype.VulnerabilitySources.OSSIndexVulnerabilitySource.<GetVulnerabilitiesAsync>d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Inedo.ProGet.Feeds.Docker.DockerFeed.<>c__DisplayClass30_0.<<GetVulnerabilitiesForContainerAsync>b__1>d.MoveNext()

Maybe the vulnerability scan failing is causing it to fail to set the latest tag? If so, how can I fix this failure?

Currently, when I load a package in ProGet it looks like this:

When I load it at NuGet.org it looks like this:

The full description is on the main page for the package on NuGet.org. ProGet shows part of the description, but it is truncated. I know I can drill into a version an get the full description, but I was wondering if I can get that on the Homepage for the package (so I can match the functionality of NuGet.org in ProGet.)

(I ask this because we are considering locking down NuGet.org at my company and forcing all package acquisition to go through ProGet.)

Kubernetes has an issue where if you use the latest tag, it sometimes will not pull the image, even when it has changed. So I am trying to write a script that will get a different, more unique tag, for use in my helm charts.

The standard Docker REST API does not have anything that can do in in a reasonable number of calls. (You have to get a list of ALL the tags, and then iterate through them one at a time looking to see which ones have a digest that matches the "latest" tag.)

I am wondering if ProGet has something better built into it. Does ProGet have a way to get the other tags on a Docker Image?

Basically, I am looking for the list of tags found on this page:

Except that I would start with the latest tag and end up with the 1.0.9 tag as the result.

API Keys have the ability to emulate a user. This allows restricting the feeds that an API Key can connect to (very useful).

I find I am submitting a lot of tickets to my System Admins to get network users made, just so I can make an API Key have limited permissions.

In reality the user does not need to exist anywhere except logically in ProGet. The connection uses the API Key, which then uses the user's permissions in ProGet to restrict their permissions. The user's password is never used and the user is never logged in.

I would like to request that a way be made that I could restrict an API Key to specific feeds without needing to create a user in my domain.

One possible way of doing this is to make an option for the name of the Key to be a "user" on the permissions page. That way it can have permission restrictions.

NOTE: I think this would be easier if I was not using active directory, as I could just make the user in Proget. But with active directory as my user provider, I cannot just add a user.

That works too.

Basically, I need a way to get rid of all versions of one chart.

My docker repositories have a button in the top corner that looks like this:

The "Delete Repository" option allows me to delete the repository in one action.

There is no similar button for a Helm repository. This leaves me with the tedious task of having to delete each an every chart version one at a time. While a viable workaround, it is frustrating and slow for a repository that has had a lot of versions.

Please consider adding a "Delete Repository" type feature to the Helm side of things.

As an additional note, after a chart is deleted, ProGet loads a 404 page. Instead, I would recommend redirecting to the list of versions for the Helm chart repository, since that is likely the page they were using before deleting the chart.

It is weird, but after some time, it just started working... While concerning, I am no longer blocked in my work.

I am getting an error when I try to push a Docker container image to one of my repositories. It says:

denied: my-user@mydomain.net is not permitted to perform the Feeds_AddPackage task for the current scope.

Here is a screenshot incase it is needed:

When I go to test privileges for that user, at first it says "No privileges for the specified context". Then it adds the permissions (though it still shows the "No privileges for the specified context" message as well. Here is a screenshot:

If I load it again, then the "No privileges for the specified context" is not there the second time.

I am not sure what to do to get my user to have the needed permissions. It is already setup with the "Publish Packages" permission (as the above test shows).

What can I do to get this user to be able to push docker images?

Turns out the promotion history is on the container image in the source feed as well. So as long as one of the feeds does not get a release version number, you can still get at the promotion history.

Here are the steps to reproduce this issue:

1. Create a docker image in FeedA with a prerelease version number (ie 1.8.0-build.15)
2. Promote the image to FeedB, adding a comment when prompted.
3. Open the image (to the tag level) in FeedB and go to the history tab. Note your comment is there.
4. Add an Alternate Tag to the image (using the prelease tag as the source) enter a release version number (ie 1.8.0)
5. Try to find your comment again. It cannot be shown.

I think this is because if you try to go to the pre-release tag, it automatically re-directs you to the release tag. In general, this is a good idea. But sometimes the history is useful to see. It would be nice if there were someway to see this history information.

I found a way to do this by downloading the image manifest and then uploading it again as a different tag.

Here is the PowerShell function that does both the promotion and the tagging (incase anyone ever needs something like this). Kind of a "do it yourself" repackaging. (Might be nice to have the Repackaging API support Docker container images someday).

function RepackageContainerImage() {

  param (
      [string] $packageName = $(throw "-packageName is required.  This is the namespace/image-name"),
      [string] $fromFeed = $(throw "-fromFeed is required"),
      [string] $toFeed = $(throw "-toFeed is required"),
      [string] $fromVersion = $(throw "-fromVersion is required.  This is the current tag on the image in the fromFeed."),
      [string] $toVersion = $(throw "-toVersion is required.  This is the tag to be applied after the image is promoted."),
      [string] $comments = "Promoted by automation",
      [string] $apiKey = $(throw "-apiKey is required"),
      [string] $progetBaseUrl = $(throw "-progetBaseUrl is required")
  )

  $postBody = @{
      packageName="$packageName";
      groupName="";
      version="$fromVersion";
      fromFeed="$fromFeed";
      toFeed="$toFeed";
      comments="$comments"
  } 
  # Promote the Container Image
  Invoke-WebRequest -Uri "$progetBaseUrl/api/promotions/promote" -Method POST -Body $postBody -Headers @{"X-ApiKey"="$apiKey"}

  # Retag the container image by downloading the manifest and then re-uploading it as the new version
  $manifest = Invoke-WebRequest -Uri "$progetBaseUrl/v2/$toFeed/$packageName/manifests/$fromVersion" -Method GET `    
        -Headers @{"X-ApiKey"="$apiKey";"accept"="application/vnd.docker.distribution.manifest.v2+json"}    
  Invoke-WebRequest -Uri "$progetBaseUrl/v2/$toFeed/$packageName/manifests/$toVersion" -Method PUT -Body $manifest.ToString() `
       -Headers @{"X-ApiKey"="$apiKey";"content-type"="application/vnd.docker.distribution.manifest.v2+json"}
}

My Docker container images flow goes like this:

1. Devs create the image with a Version Number like this: 1.5.0-build.62
2. Ops promotes the image to Test (using Octopus Deploy calling the Proget Package Promotion API)

As part of the second step, I want set the container image to a release version. So in my example above, it would get the alternate tag of 1.5.0.

But I need to have this done when Ops promotes the release to Test in Octopus Deploy. I have not been able to find documentation on how to apply an alternate tag to a docker container image via API (It is very clear how to do it via the UI). I considered using the Repackage API for it, but the UI does not offer Repackage for Docker container images, so I assume the API is also not setup for it.

How can I apply an alternate tag to a container image via an API?

@NanciCalo,

Thank you for your prompt help! We have contacted our purchasing department to get a rush order on the new license. ("Rush" means that I hope to have it in a week or so.)

I was able to put in a trial license and everything seems to be working fine. It says it is good until the 27 of June, which should be more than enough time to get a new license purchased and installed.

Thank you again for keeping us running!

I logged on to ProGet just now and saw this banner:

Apparently my license key will expire in 5 days!

What happens if my slow moving finance department can't get a new key purchased in 5 days?

Do my features revert to the free version of ProGet?

My company has the Basic version. My biggest worry with losing "Basic" features is the active directory integration. No auto-builds will be able to authenticate with ProGet if that drops out.

If the features do revert to free mode, is there a "grace period" to allow for slow moving enterprises to get their purchase ducks in a row?

(As a side note, I would propose a feature request to make this notification pop up 30 days prior to expiration.)

That does indeed have the info. Thank you!

Is there a way, via API, to get a list of versions for a given helm chart?

The helm command line tool can do it with this command:

helm search repo proget/my-helm-chart --versions --version ^v1.0

That will return all the versions that start with v1.0.

I need to do this in my auto build system. I could use the above example, but for this command to work, I have to save to disk a proget repository first (via helm repo add). I have found that builds that rely on persisting data to disk are brittle, to say nothing about the fact that the API key is now saved on disk for others to get at.

I would like some way to get a listing of versions without having to persist anything to disk. Is there a way of doing that?

Do both of your feeds have the same storage type? Under Manage Feed > Storage?

I misspoke when I said I was using two feeds. It was all in the same feed, I was just trying to make a new docker repository in the same feed. (Basically, I wanted to change the name of my container image by doing a docker push with it having a different name. But it was all to the same ProGet feed.

Here are the storage properties of that feed:

I dug into this more and eventually found this error:

Image_Digest must be unique across the containing feed.

It would appear that the fact that I had this same image in another feed made it fail when I tried to put it into a new feed with a new image name and tag.

When I went to my source code and added a whitespace to a line and rebuilt the docker image it worked fine.

I would like to suggest that this could be better. At the least, it would be nice if this error was more discoverable. I had to turn IIS logging (using this guide). Once I did that, the logs said that it was a status code 500 with an ErrorCode of "The operation completed successfully. (0x0)". (Not very helpful :)

Eventually I was able to dig through the lower level logs till I found this:

{"errors":[{"code":"UNKNOWN","message":"50000161TR__DockerImages__ValidateUniqueDigest19`Image_Digest must be unique across the containing feed.","detail":[]}]}

Ideally this would show up in the Administration page's Diagnostic Center. Failing that, it would be nice if it was in the event log for the host machine.

I also have to wonder why this limitation exists. I thought docker could have the same image in different repositories any number of times.

I have been able to successfully push docker images for a while now. Most of the images I was pushing were all to the same feed's docker repository. Today I needed to push to a new docker repository in the same feed.

When I did that I got this error:

received unexpected HTTP status: 500 Internal Server Error

I went back to an existing repository and tried deleting a tag and repushing it. That worked fine.

Something is going wrong when I use a new repository name.

I went to the administration page in ProGet and looked in the Diagnostic Center and there are no logs during the time that the error occurred.

How can I get additional details on this error so I can go about diagnosing it?

Here is what my attempt to push the docker image looked like:

[builderuser@SWDGNUBUILD01N ~]$ docker login proget.mydomain.net
Username: proget-admin
Password:
WARNING! Your password will be stored unencrypted in /home/builderuser/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
[builderuser@SWDGNUBUILD01N ~]$ docker push proget.mydomain.net/application-prerelease/runtime/my-application-service:1.9.0-build.147
The push refers to repository [proget.mydomain.net/application-prerelease/runtime/my-application-service]
d1969e3e3bf7: Layer already exists
d79cf2683c2b: Layer already exists
fdce96d81153: Layer already exists
379b6cbd8951: Layer already exists
5c65967e47cc: Layer already exists
b49c0ab385b6: Layer already exists
bd13a0d30d03: Layer already exists
49a130b8b29d: Layer already exists
68f8345afd82: Layer already exists
4466146b6e2c: Layer already exists
8b9bd48d3c62: Layer already exists
fe5e09df9bed: Layer already exists
86248bc7c05e: Layer already exists
093496e1b286: Layer already exists
139aa1a59c8a: Layer already exists
f3bc884425c5: Layer already exists
6db0849b3e99: Layer already exists
800afb4d883b: Layer already exists
cb42413394c4: Layer already exists
received unexpected HTTP status: 500 Internal Server Error

Also, this was run from a Linux machine (in case that matters).

Not having a user for an API Key gives a lot of access. While I am the admin of my ProGet instance, there are others that have admin access and don't realize the hole they are opening up with an API Key that is not restrained by any user permissions (because they don't enter a user for the key).

I would like to suggest an Advanced Setting that, when toggled on, would force any new API Keys to select a valid user for the API Key to run as.

I always cringe a bit when I go to the API Keys page. All the keys are just shown for easy use. While nice from an ease of use point of view, it does not align with what I have seen in other products' API Key features.

I would like to offer this suggestion:

When an API Key is generated, clearly state that this is the ONLY time that key will be shown. From then on, it is differentiated only by its description.

When you go to store the API Key, you don't actually store it. You put it through a one way hash and then store the hashed value.

Then when an application calls Proget using the API key, you put it through the same hash and if the newly hashed value matches the stored hashed value, then access is granted.

This makes a security breach far less likely because:

1. You are not storing the API Keys. Any attack on the ProGet database does not breach any keys.
2. The keys are not re-shown in the UI, so screen scrapers or over the shoulder lookers are less likely to get the key.

(Reasoning: Password storing is a business that already has many products. Most companies have a system in place to keep their passwords safe. API Keys are basically usernames and passwords in one, and should be stored in a Keystore system.)

I think you can just add a file to the chart at the root level. But I am not a helm expert, so take my opinion with a grain of salt.

I am going to be doing some helm testing soon(ish). I will try to add in a random file and see if Helm/Kubernetes just ignores it when I do an install.

I am attempting to use the promotion API to promote a Docker Container Image from one feed to another. When I do that I get this error:

System.NotImplementedException: The method or operation is not implemented.
at Inedo.ProGet.Feeds.Docker.DockerFeed.<Inedo-ProGet-Feeds-ISyncFeed-GetSyncPackageAsync>d__68.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Inedo.ProGet.WebApplication.SimpleHandlers.Api.PackagePromotion.PackagePromotionApiHandler.<ProcessRequestAsync>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Inedo.ProGet.WebApplication.SimpleHandlers.Api.ProGetApiKeySecuredHandler.<ProcessRequestInternalAsync>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Inedo.Web.Handlers.Api.ApiKeySecuredHandler.<ProcessRequestInternalAsync>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Inedo.Web.Handlers.Api.StandardApiHandler.<ProcessRequestAsync>d__1.MoveNext()

I am calling it from postman like this:

URL: https://myProGet.mydomain.net/api/promotions/promote
HTTP Method: POST
Request Body:

{
  "packageName": "dockerNamespace/myContainerImageName",
  "version": "2.5.1-build.162",
  "fromFeed": "images-prerelease",
  "toFeed": "images",
  "packageType": "docker"
}

Content Type: JSON
HTTP Headers:

• X-ApiKey: myapikeyhere

What do I need to change to get my request to work?

I am currently building out my CI/CD pipeline for Kubernetes. One of the things I am currently doing is making a process to repackage a prerelease Helm Chart as a release version and promote it to a release feed. To accomplish that, I am doing this:

1. Download a Helm Chart
2. Update its version numbers (a Helm Chart has 2 version numbers), and Image Tag (which for me is the same as the version number.) Basically, I am converting the version numbers from pre-release (ie 2.0.3-build.12) to release version numbers (ie 2.0.3)
3. Upload the chart into a new feed.

As I was working on that, I noticed an Repackaging API. Looking into it, it seems it is very similar to what I am doing, but it is intended for NuGet and universal feeds only.

This is not a "Feature Request" (because even if you made it, I will not need it by the time you get done). But I thought I would throw it out there as a possible suggestion to add value to ProGet. (The only real concern I could see is if the packages were signed, then you would need access to the Key to re-sign them.)

@rhessinger

You may also want to remove this message:

The big red banner is what got me looking into the license types in the first place. (I did not want my users to see the big red banner and I could see no way to suppress it except for getting a license assigned.)

OK. That aligns with what I was seeing.

Thanks for explaining it to me!

I have two main docker feeds setup. One called "application-prerelease" and one called "application".

My initial builds go into the "application-prerelease" feed, then the good ones get promoted to the "application" feed.

I am setting up integration to this with a third party tool and it wants the URL of the Feed.

I tried giving it https://myproget.mydomain.net/application-prerelease/ but it said:

Feed endpoint https://myproget.mydomain.net/application-prerelease/ does not appear to expose a valid Docker API. The url was tested against the v1 (https://myproget.mydomain.net/application-prerelease/v1) and v2 (https://myproget.mydomain.net/application-prerelease/v2) endpoints and both failed. Please check the uri and the feed provider, and try again. Read http://g.octopushq.com/DockerRegistries for more details.

I also tried using the base url of my proget server (https://myproget.mydomain.net) and it did not have any errors. But that url has ALL of the container images from all my feeds (application, application-prerelease, and others). (Which makes sense, because it is the root URL.)

How can I get a URL that will only have the containers of a single feed in it?

OK, thanks!

I am not set on it being in the LICENSE file. If there is another way that Proget would like to get this information, let me know and I will do it that way.

When I look at my helm chart metadata in ProGet it says ? not specified for the License.

Where in the Helm Chart is it looking for the License?

The Helm docs say to create a file called LICENSE. I did that and put the text "All Rights Reserved" into it. I then when to ProGet and added a License called "All Rights Reserved". But it still says ? not specified.

Is ProGet looking somewhere else for the license declaration? If so, where?

Wahoooo!

We were able to successfully swap user directories this morning!

They key was changing the Managed Pipeline Mode setting for ProGet's App Pool in IIS. While set to Integrated (what we had been using) it would not work. Once I set it to Classic, I was able to finagle it to work. I had to fiddle with the Integrated Login setting in ProGet and the Anonymous Authentication setting in IIS, but it worked out!

As a side note, the debug/integrated-auth page still says Username not found and Principal not found, (it is the same as what I pasted in above when it was not working). Just thought I would let you know in case you are interested.

I am very excited to get moving forward on my project and stick with ProGet! Thank you for your hard work helping me get this working!

Stephen

@apxltd

We tried to swap User Directories again. (Since we did it on Friday morning, 5.3.25 had not dropped yet. So we only upgraded to 5.3.24, not 5.3.25 that had the feature you showed above. I hope to try again on Friday to try that feature out.)

We first upgraded to 5.3.24. Then we set LoadUserProfile=true. Then we swapped the User Directory. Then we rebooted the server.

It once again did not find the user.

We went to the debug/integrated-auth URL and this is what it showed:

IntegratedAuthEnabled:   False
LOGON_USER:              MY-NETBIOS\1234
HttpContext.User.Identity.Name: MY-NETBIOS\1234
HttpContext.User.Identity.IsAuthenticated:      True
WebUserContext.UserInfo.Name:   Anonymous
---------
Current User Directory:
Queries the current domain, global catalog for trusted domains, or a specific li
---------
Domain:         MY-NETBIOS
Id:             1234
---------
LOGON_USER parsed as: 1234@mydomain.net
Username not found.
Additional messages:
 - Trying to a Users search for principal "1234@mydomain.net"
 - Search string is "(&(|(objectCategory=user)(objectCategory=msDS-GroupManagedServiceAccount))(sAMAccountName=1234))"...
 - Domain alias "mydomain.net" will be used.
 - Searching domain mydomain.net...
 - Trying to a Users search for principal "MY-NETBIOS\1234"...
 - Search string is "(&(|(objectCategory=user)(objectCategory=msDS-GroupManagedServiceAccount))(sAMAccountName=MY-NETBIOS\5C1234))"...
 - No domain specified, searching through aliases.
 - Searching domain mydomain.net...
 - Principal not found.

NOTE: I have replaced the text for my NetBios with MY-NETBIOS and my domain with mydomain.net and the user name with 1234 to de-identify this data.

I plugged the first query into a powershell command and it worked perfectly:

C:\src> Get-ADUser -Credential My-ProGetADAccount -LDAPFilter '(&(|(objectCategory=user)(objectCategory=msDS-GroupManagedServiceAccount))(sAMAccountName=1234))' | Format-Table Name,SamAccountName -A

Name           SamAccountName
----           --------------
Doe, John      1234

(Again, I replaced anything identifying when pasting here.)

The first query it tried should have worked. I am confused why it did not. I am hopeful you have some advice on what else we could try.

Is there a process for us to get back to version 5.3.15? While I want the features of the later versions, we need permissions working again.

Is there a way we could rollback to version 5.3.15 from 5.3.21?

Another thing to try is visiting /debug/integrated-auth in your instance

I am not sure what this is, but I am willing to try out anything. Do you have any instructions on how to do this?

The software/code is identical, so if this is happening then it means something is different in your production environment.

I only have one environment for this. It is all production. We have one Active Directory setup at my company. I used the same one for ProGet and for the tool.

Though it may be important to note, the "Test Tool" has the same functionality as the test tool embedded into ProGet. That test tool works fine (both before and after the change of the User Directory). It is ProGet itself that cannot find the user. The Test Tool can always find the user (both the one integrated into ProGet and the one that is run from Visual Studio.)

this abrasive communication style causes only unnecessary stress to the people you interact with

I apologize that this came across as abrasive. Basically I am trying to communicate that I am suck. I first reported my problem over a month ago. My project is dependent on being able to setup permissions. And I cannot find a way to do that. And the debug tools all show that everything is working great. Up till when I make the switch. Then the user cannot be found.

I cannot test this out in a non-production environment because we have not purchased a second license for testing. And the free version does not include the very thing I need to test (AD Integration). So I have very short downtime windows to try to troubleshoot this.

Again, I am sorry my communication was abrasive. I was trying to say that I can't leave it like this. I have to have working permissions. I don't want that to be with another product. But permissions are required.

<sigh>

We tried again today. I set the NETBIOS mapping and it still failed....

I am not sure what to do now.

We are stuck. I can't update user permissions because of a bug in the Legacy LDAP setup I use. And I can't switch to the newer Active Directory LDAP setting because of... I don't know why. And after I make a change, I can't change back (I have to restore the server and database from a back up to get it working again).

I tried using your Active Directory debug tool and it the settings work perfectly there. The same settings fail in ProGet.

As much as I really like ProGet (both price and features), I am going to have to spend some time today to go shopping for a product that can integrate with my Active Directory system. :(

I can probably convince my Operations Team to give it one more go if you have any suggestions.

@apxltd

We do high availability on some of our stuff. And it may be that we move to it for ProGet as we start doing more with containers.

The value of this feature idea isn't so clear

It is possible that you are underestimating the value that enterprises place on governance of their processes.

As a note, I have done some more research into this feature request, and it will not be possible based off the container image alone. Multistage Dockerfiles to not pass any part of the Dockerfile that was used from a previous stage on to the later stages (aside from any files copied).

I will have to find some sort of process change for this part of our governance.

@apxltd said in Feature Request: Limit Container Images to those that are inherited from another feed:

Is this Dockerfile typically stored in source control, alongside the application source code? In a world of cookie-cutter microservices and microapps, it seems these Dockerfiles should be nothing more than the absolute basics on top of a base specific image?
Separately, but related... I just read up on multi-stage builds and the Builder pattern vs. Multi-stage builds in Docker, but I'm struggling to see how this all adds up to real-world use inside of organizations?
Is this really a pattern that is emerging?

This all goes together as one is the cause of the other. In many ways, Microservices are and should be cookie cutter. But in many ways they cannot be (and still have reasonable dev practices).

Here is an example:

Consider a medium sized company that has 3 microservices. They use .Net Core 3.1.12 for their auto build. They setup their build servers and they get things working great.

A month goes past and they need a new micro service. They go to develop, it but .Net Core has updated to a newer version. They are now faced with a choice.

1. They update the build server to the newer version.
2. They make a new build server.
3. They develop with the older version of .Net Core

All three options have issues.:

1. Updating the build server means that the 3 microservices are also getting upgraded. Something that may not be in their prioritized list of work at this time. (We almost never do this option.)

2. Making a new build server gets expensive as all most dependencies are always putting out new versions. (We have quite a few build servers because of this.)

3. Using older tech stifles innovation and keeps features and bug fixed that were in later versions of dependencies out of the final product.

We trade off between numbers 2 and 3, as number 1 is too risky for my company.

Multistage Dockerfiles help fix this issue. Each build is done in its own sandbox. The initial 3 microservices can keep using .Net Core 3.1.12 (or a company approved descendent image). They do this because their dockerfile has not changed.

But the new micro service can user the latest version of the .Net Core SDK Container (or a company approved decendant image) in its docker file. And, because of Docker, they can all run on the same build server.

Giving development teams not only control over their own base image, but over the tools to compile/build their code? This just seems to add another layer to what ought to be a very simple process, and more opportunities for supply-chain attacks.

As my company's IT Software Architect this is something I worry about a lot! Hence the reason for my request that started this chain. I need to know that all the containers in the multi stage build file were containers that came from an approved ProGet source. (Not just the final one.)

We use mostly Microsoft technologies at my company. So if some developer goes and downloads a PHP or Ruby container and uses it to build an containerized application, I want them blocked as soon as they try to upload the image to ProGet. This allows me to enforce our technology chain.

How much more would you pay (or encourage your organization to pay) for ProGet if this feature were to be there?

The costs to a company of allowing developers to "just use what works best for them" are big. Management wants to be able to move developers between teams. And as product dependencies get old and need to be replaced or upgraded, each migration plan takes time to come up with. If you have to be making custom migration plans for .Net, Angular, React, VueJs, Ruby, PHP etc as each has a new major version it becomes very expensive.

But developers really like to use the new shiny stuff. And many will just do it if they can. And Multistage Dockerfiles make it even easier because they don't have to ask for anything to be installed on the build server anymore. If you are not really careful, you can get it trouble fast.

I am still planning out how I will enforce our tech stack as we move to containers in earnest. It could be a parsing of the Dockerfile, or it could be a proget hook.

Still, if there were a solution that just worked, it would be worth money to my company. I don't get approve purchases, but I do recommend them a lot. (For example, I just got another ProGet license approved as a Sandbox instance).

My company starts to balk after you get over a few thousand dollars unless the value is really big. For example, the enterprise version is too expensive for just HA features (from my company's point of view). If this container feature was in the Basic version and it went up around 30% I don't think my company would complain.

But small companies just don't need a feature like this. (Not sure of you have small companies in your customer demographics).

And in general you have to be careful about price hikes. ProGet has a nice position in the market right now. It has great features and is much cheaper than its competitors. You could lose some market share if you raise your prices too high.