Posts made by rhessinger

Hi @aneequaye_1868,

I apologize, I was speaking from the mapped folder on the Docker host. Inside the container, it is still /var/proget/packages. Does this still happen on the latest version of ProGet? We did make some changes to how the headers are returned in the latest version, especially for feeds that require authentication.

Thanks,
Rich

Hi @viceice,

I was just about to link you to another forums post about X-Forward-* headers, but I realize that post was with you. Just out of curiosity. Are you using a different URL for your docker feed than your nuget feed? Also in your docker client, do you have your URL registered as an insecure registry?

Thanks,
Rich

Hi @viceice,

Thanks for the information. Let me dig into this a bit further and let me work on recreating it. Hang tight!

Thanks,
Rich

Hello @viceice,

What version of ProGet are you running? There was a bug that was fixed on ProGet 5.3.15 that corrected the headers in the docker login process when using the proget image. Also, please make sure that you set the Web.BaseUrl to the https reverse proxy URL in the Administration -> Advanced Settings section.

Thanks,
Rich

Hi @viceice,

How many docker feeds do you have configured on the fresh ProGet instance? I have seen one other user who has seen a similar issue, but it only happened on the one feed and manually creating the folder has resolved it. Any other docker feeds that were created did not seem to have the issue. Also, how did create the folder? Did you create it from within the ProGet container?

Thanks,
Rich

Hi @atillio-menegoni_1334,

If you navigate to that artifact version in the UI and verify that cdd-application-5.7.0.5.zip exists in the list of files? If it does, can you click the file and see if it downloads manually for you?

Thanks,
Rich

Hi @nuno-guerreiro-rosa_9280,

Do you have any other applications connecting to your SQL enterprise instance? Do you see any timeouts in those applications? Would you be able to run a docker stats when these timeouts are occuring? Could you also up your timeout in ProGet's SQL Server connection string?

Also, what version of SQL server are you running?

Thanks,
Rich

Hi @nuno-guerreiro-rosa_9280,

Sorry for the late notice but the release of 5.3.14 was pushed back to later today. I'm sorry if this caused inconvieninces for you.

Thanks,
Rich

Hi @srbauti_9412,

The best solution to this would be to create a new NuGet feed that uses a self-connector to connect back to your existing NuGet feeds. You can then use the connector filters to handle filtering by your package name format.

Thanks,
Rich

Hi @aneequaye_1868 & @informatique_1703,

I may have found a potential cause for this. Can you verify a feed folder exists in your /proget-packages/? You should see proget-packages/.docker/common and/or /proget-packages/.docker/F<Feed ID> (ex: proget-packages/.docker/F1).

Thanks,
Rich

Hi @philipp-grathwohl_1687,

Currently, the only way to set the description would be to do it using the UI or the Native API. In order to set it using the Native API, you would need to do the following:

First, Get the Docker Repository ID by making an HTTP GET request to <Base URL>/api/json/DockerImages_GetRepositoryByName?key=<API Key>&Feed_Id=<Feed_ID>&Repository_Name=<Repository Name>
This will return a list of images (should only be 1 in the list) and you will need to get the DockerRepository_Id from each item.

Then, you will need to HTTP POST a JSON object to <Base URL>/api/json/DockerImages_CreateOrUpdateRepository.
The JSON object you would post would be:

{
	"API_Key" : "<API Key>",
	"Feed_Id": "<Feed Id>",
	"DockerRepository_Id": "<ID from previous request>",
	"Repository_Name": "<Repository Name>",
	"RepositoryIcon_Url": "<Icon_URL>",
	"ShortDescription_Text": "<Short Description>",
	"FullDescription_Text": "<Full description (Readme.md)>"
}

Please note that you will need to populate all of the values. For example, if you do not populate the RepositoryIcon_Url, the value will be cleared.

Thanks,
Rich

Hi @toni-wenzel_6045,

Currently, the only way to set the description would be to do it using the UI or the Native API. In order to set it using the Native API, you would need to do the following:

First, Get the Docker Repository ID by making a GET request to <Base URL>/api/json/DockerImages_GetRepositoryByName?key=<API Key>&Feed_Id=<Feed_ID>&Repository_Name=<Repository Name>
This will return a list of images (should only be 1 in the list) and you will need to get the DockerRepository_Id from each item.

Then, you will need to POST a JSON object to <Base URL>/api/json/DockerImages_CreateOrUpdateRepository.
The JSON object you would post would be:

{
	"API_Key" : "<API Key>",
	"Feed_Id": "<Feed Id>",
	"DockerRepository_Id": "<ID from previous request>",
	"Repository_Name": "<Repository Name>",
	"RepositoryIcon_Url": "<Icon_URL>",
	"ShortDescription_Text": "<Short Description>",
	"FullDescription_Text": "<Full description (Readme.md)>"
}

Please note that you will need to populate all of the values. For example, if you do not populate the RepositoryIcon_Url, the value will be cleared.

Thanks,
Rich

Hi @csyy321_2677,

What version of CENTOS is your Docker host running on? Can you also tell me what version of docker is installed?

Thanks,
Rich

Hi @nicolas-morissette_6285,

No problem! If you are still having issues after you reach out, please feel free to reply back to this topic.

Thanks,
Rich

Hi @nicolas-morissette_6285,

I believe Azure DevOps NuGet feeds depricated alternate credentials back in March and they will only support PATs moving forward. Here is an article on the deprication Azure DevOps Will No Longer Support Alternate Credentials.

I know you were sepcifying your Azure AD username and password, but I believe Microsoft treated those as an alternate method because the normal Azure AD login requires OAuth based logins moving forward. The PAT was their solution for adding a way to authenticate without requiring a user to intervene in the login process.

Thanks,
Rich

Hi @aneequaye_1868,

Would you be able to run an ls -l to check your folder permissions? I have seen some recent issues that linux reports back some weird errors when a folder doesn't have the proper read/write/delete permissions.

Thanks,
Rich

Hi @msimkin_1572,

What feed types are you using? Some of our feed types have a built-in process to verify this for you.

Thanks,
Rich

Hi @informatique_1703,

Thanks for clarifying that for me. The issue did exist in 5.2.32, but the scenario to recreate it was different. Now that I understand that the blob is not in the database (I apologize for missing that before), I don't think the clear cache issue caused this. Just to verify, when you searched for the blob digest in the database, did you limit it only to that FeedId? Or did you just search for any blobs with that digest? Could you send me the SQL query you used? Also, can you check to see if this feed is configured to use shared storage? You can find that by navigating to the Manage Feed page and selecting the Storage & Retention tab, then click the Configure link to the right of Blob Storage.

Thanks,
Rich

Hi @informatique_1703,

I'm sorry for the confusion. If the clear cache was ran in 5.3.12 or lower, that may have removed the shared blob from your storage location, but left the link in the database. Upgrading to 5.3.13 removes the possibility of deleting a blob that is still in use from your storage location. Can you verify if 2622b3cbec4c3f908fde9a413e48eca0145887f5c0719a4384d2e862978270b0 exists in your storage location? You can also see by going to an image that has that blob layer and looking at the 'Layers` tab. If it has a red exclimation mark next to the layer, then it was most likely removed from the storage location.

Thanks,
Rich

Hi @viceice,

That error is safe to ignore. It is currently a known bug and we are looking to fix that in an upcoming version of ProGet. The ticket tracking the fix for the log message is PG-1841.

Long story short, the ProGet service correctly detected that the product wasn't activated, and then logged that message. But it was doing it every time it accessed license information, which is on every connector health check, replication run, etc.
Activation happens automatically as soon as someone visits the Web application, and re-activation is required after upgrading certain versions.

Thanks,
Rich

Hi @informatique_1703,

I think I see the issue here. There was a bug, PG-1832, in the clear docker connector cache that was patched in ProGet 5.3.13. The issue that was happening is that the clear cache would fail when a blob was shared between a chached image and a local image. The side affect is that it would delete the blob from the storage location, but it fail to remove the reference from the database. This causes our API to think the blob exists but then fails to find it on the disk.

This is why a new feed will work because you are pushing the new layer up that doesn't already exist in the feed. If you copy the blob named 2622b3cbec4c3f908fde9a413e48eca0145887f5c0719a4384d2e862978270b0 from the new feed storage location to the old feed storage location, then most likely the push will work. You can find the storage location by checking the Storage.PackagesRootPath, Storage.DockerRepositoryLibrary, and Storage.DockerBlobStorageLibrary in Administration -> Advanced Settings. You may also need to check in the Manage Feed settings for a registry level storage or if your registry is using Shared Storage.

Thanks,
Rich

Hi @nuno-guerreiro-rosa_9280,

The docker login issue has been scheduled to release in ProGet 5.3.14 on October 30, 2020. I will let you know if anything changes.

Thanks,
Rich

Hi @informatique_1703,

Are you using any docker connectors on this feed? If so, have you cleared the cache on these recently? Also, if you create a new feed, then push the image, does that error happen?

Thanks,
Rich

Hi @informatique_1703,

Would you be able to switch to the progetmono image and see if you still have this issue? I would like to rule out .NET Core as a cause.

Thanks,
Rich

Hi @nuno-guerreiro-rosa_9280,

Thank you for all the extra information. I believe I have found the culprit, but I will need to work with the team internally to determine the proper fix. It looks like the call to /v2/ that tells docker it needs to authenticate is not returning the proper headers. This is a .NET Core only issue.

The reason that Docker is the only feed affected now is because of how the docker client/api works. It basically walks through the process of an API looking for the API to return very specific errors and headers. That is how the client knows what to do next. In the case of auth, it will first hit /v2/ and look for a 401 unauthroized and some specific headers to tell it to redirect to /v2/_auth and to pass an Authorize header. That is not happening. I will update when we have scheduled this for release.

Thanks,
Rich

Hi @informatique_1703,

Currently, the only way to transfer all docker images automatically between feeds is by using feed replication, which requires an enterprise license.

One manual option is to promote the image from your old feed to your new feed. You would need to do that on each image manually though.

Is your ProGet instacnce installed on Windows or are you using a Docker image based version of ProGet?

Thanks,
Rich

Hi @philippe-camelio_3885,

Just to verify, you are running the command line as an administrator correct? Can you also please verify the user you are trying to run the service as has the correct password and is not locked out on the domain?

Thanks,
Rich

Hi @informatique_1703,

Does this happen for all images or just the one image? Do you see any errors in the console from docker?

Thanks,
Rich

Hi @informatique_1703,

Is this the error from ProGet or from the Docker client. Do you see any other errors in the ProGet diagnostics center when you try to push an image?

Thanks,
Rich

Hi @viceice,

Thanks for confirming that for me. I ran into a few issues on my test server with SQL 2014 and TLS1.2 support. You may want to make sure that TLS 1.2 is enabled on that VM. SQL Server 2014 and Windows Server 2012 R2 both had some unique issues with teh TLS 1.2 conversion (both needing updates to be able to enable them) . Although this did not cause an issue with hanging, it did cause my authentication to fail. You may want to check that.

Thanks,
Rich

Hi Phillipe,

I created an issue on GitHub, #8, to track adding this to the extension. I'll reply back once there is a CI version released of the extension.

Thanks,
Rich

Hi @viceice,

I'm having trouble recreating this issue. I tested this by setting up a SQL Server 2014 instance with updates installed to bring it to version 12.0.6372.1. I have tried both the proget and the progetmono images at version 5.3.12. Both of them boot up and run all the scripts completely fine. I have even tested using your exact docker command and only changing out the SQL connection string. So far I cannot get it to hang. Is there anything else unique about your network configuration? Do you have just a standard install of SQL Server? Is there any proxy or VPN that requests are routed through to connect to your SQL Server?

Thanks,
Rich

Hi @p-bruch_5023,

Thanks for bringing this to our attention. I have created a ticket, PG-1838, to track a fix for this. It will be realeased this Friday as part of ProGet 5.3.13.

Thanks,
Rich

Hi @nuno-guerreiro-rosa_9280,

The .NET Core version is definitely more effecient than the mono version. One of the biggest issues with mono is how it manages web requests. Mono reimplemented its web client and connection pool from scratch due to the complexity and internal dependencies of the .NET Framework. This is probably part of the reason you see the mono version tax the server a bit. As for NuGet querying all 40 packages in ProGet, we don't have much control over that. NuGet queries all sources in the list until it finds one. It will do custom sources first then fallback to NuGet.org last. There is an interesting ticket on NuGet's GitHub page discussing the priority (https://github.com/NuGet/Home/issues/3676).

Thanks,
Rich

Hi @nuno-guerreiro-rosa_9280,

I was finally able to recreate the issue. We created a ticket, ILIB-98, to track the issue. It will be released on Friday with ProGet 5.3.13.

Thanks,
Rich

Hi @nuno-guerreiro-rosa_9280,

No problem! Gald it helped!

Thanks,
Rich

Hi @nuno-guerreiro-rosa_9280,

You can reduce the logging by adding some environment variable parameters to your docker start command. The environment variable paramters you will need to add are:

-e 'Logging__LogLevel__Default=Warning'
-e 'Logging__LogLevel__Microsoft=Warning'
-e 'Logging__LogLevel__Microsoft.Hosting.Lifetime=Warning'

You can also set them to Error if oyu wnt even less logging.

Thanks,
Rich

Hi @nuno-guerreiro-rosa_9280,

Have you tried uploading a new NuGet package and tried downloading it? I would just like to rule out the directory and file renaming. I'm running the core version locally and I'm not currently able to recreate the authentication issues in my tests.

Thanks,
Rich

Hi @viceice,

Nothing is sticking out to me that there is an issue. Your hardware looks fine (without knowing everything else running on the server/kube pods).

It looks like it is hanging when trying to load the current database schema version. The code for both mono and .net core does not differ for this component. Is there anything that is different from your progetmono and progetcore start scripts in kubernetes? I would put some enphasis around the networking piece in kubernetes.

Thanks,
Rich

Hi @barnaby-williams_8489,

This is just a manual copy of the files.

Thanks,
Rich

Hi @nuno-guerreiro-rosa_9280,

If you upload a new package, can you download it then? That would be the best way to verify the casing change did not affect the authentication.

Also, what user directory are you using? Are you using the built-in user directory, Active Directory, or LDAP Legacy?

Thanks,
Rich

Hi @jim-pg_1173,

I was able to identify the issue. I created a ticket to track the fix, PG-1831, this will be released in the next version of ProGet. ProGet 5.3.13 will be released this Friday, Oct 9th, 2020. I will reply back if anything changes.

Thanks,
Rich

Hi @jim-pg_1173,

Thanks for posting this here. The GitHub ticket is very helpful for this. Let me do some investigating and I'll get back to you about this.

Thanks,
Rich

Hi @nuno-guerreiro-rosa_9280,

I tested the scenario where I setup anonymous to view only and a user to download only and it successfully authenticated and downloaded the package for me. Would you be able to share the custom tasks you are using for the anonymous user and the user that has download only?

Here is what I setup:

View Only

Download Only

If you give the user View & Download rights, does it work?

Thanks,
Rich

Hi @viceice,

There shouldn't be any issue with your SQL Server version, but I will do some testing on that version to see if I can recreate the issue. SQL Server 2014 is technically deprecated by Microsoft, but we are not using any functionality that wouldn't work with it. I'll let you know if I find anything on that front.

As for the 100% CPU usage, I'm going to reach out to my colleague internally to see if there are any extra debug settings that can be enabled, but I do not think there is. Do you have the docker host throttling what hardware can be used with that container? If so, what are the restrictions for it? Also, are you able to share what hardware specs are for your Docker host?

Thanks,
Rich

Hi @viceice,

I'm having trouble recreating this issue. Could you try using the IP address as the SQL Server data source instead of db02? There isn't any difference in the startup between the dotnet core version and the mono (outside of the image base that is). If you let it run, do you get any timeout or cannot find server error?

Thanks,
Rich

Hi @viceice,

Can you tell me how many feeds you have and what types those feeds are? Also, does your ProGet database user have db_owner on the ProGet database or just the ProGetUser_Role?

Thaks,
Rich

Hi @barnaby-williams_8489,

Here is a knowledge base article for migrating feeds. Although the majority of this is probably not helpful, the very last section talks about how to migrate a Maven feed. You could probably use that to migrate any Maven feeds to Blob storage. It may be easier for Maven to:

1. Create a new feed
2. Migrate your packages over using the KB article
3. Remove your old feed
4. Rename your new feed to the old feed's name

Pulled from our Feed docs

Renaming a feed will also change its API Endpoint URL. As of ProGet 5.2.19, you can create "Alternate Names" for a feeds by going to Manage Feed > Rename, then clicking the Set Alternate Names link in the warning dialog. Alternate feed names essentially provide multiple endpoint URLs for a feed, and are useful when renaming feeds to keep backwards compatibility with old names.

That should cover all the feeds except Docker. Do you have any docker feeds configured? Is there any other feeds you are having trouble migrating?

Thanks,
Rich

Hi @barnaby-williams_8489,

Please see our documentation for cloud storage. There is a subsection for migrating a feed to cloud storage.

Thanks,
Rich

Hi @nkerifacclaud_6931,

Thanks for testing this out! I will get it released as a production version today!

Thanks,
Rich