Posts made by rhessinger

Hi @shiv03_9800,

After further research on this issue, it looks like PowerShell remoting via WSMan on Linux is not currently a supported feature from Microsoft. Even looking at the latest PowerShell Core SDK, they still have not added support for it as of yet. There is a new feature in PowerShell Core 7 that adds support for PSRemoting over SSH, but it includes a limited subset of commands. It would actually be better to just connect directly to the server using SSH in these cases. Especially since you would have to configure Window's built-in SSH server anyways.

The best option for connecting to Window's servers from Otter on Docker is to use the Inedo Agent. If you cannot use an Inedo Agent, then using SSH directly would be the fallback.

We will be making some changes, OT-430, to better address this issue in the UI and to bring awareness to the lack of support for PowerShell agent-less servers when using Docker. We will also be updating the documentation to reflect this.

Please let me know if you have any questions.

Thanks,
Rich

Hi @shiv03_9800,

After a bit more research, this looks like a problem with the deprecated omi library that Microsoft was using for PowerShell WSMan on Mac and Linux. We are still working through the problem, but I wanted to give you a quick update. I will provide an update when I have more information.

Thanks,
Rich

Hi @shiv03_9800,

I just wanted to let you know that we have received the logs and we are currently looking into the issue. I will let you know when we have more information.

Thanks,
Rich

This looks to be an issue when common blob storage is not enabled for a Docker registry. I have created a ticket, PG-2009, to track this fix. It is expected to be released in ProGet 5.3.39 which is scheduled to release on October 8th, 2021. I will post back here if anything changes.

There was also an issue, PG-2008, that was fixed 5.3.38 that would sometimes return a 500 error to Calair when trying to download the layer. PG-2008 seems to only affect ProGet running on Linux.

Thanks,
Rich

Hi @scusson_9923,

Could you please generate a temporary API Key and try wget with https://<proget_server>/api/docker-blobs/download/sha256%3Af033c4f65cdbf0bfa21d5543e56c0c41645eca4d893494bb4f0661b0f19ccc79?API_Key=<API_KEY> from the container? Can you also try that from your browser (it should try to download the file)?

It is throwing me off that you are getting a 404 for all the layer download requests. It sounds like either ProGet cannot find the layer, which should show an error in the log, or that Clair is calling to the wrong server for the download the layer.

I apologize for all the back and forth with this. This is the first time we have experienced this with Clair and I'm still trying to determine which component has the issue.

We are currently running Clair on our ProGet.inedo.com and it doesn't seem to have this issue. I'm also not able to recreate this locally, which makes this a bit more difficult.

Thanks,
Rich

Hi @scusson_9923,

What happens if you try to wget https://<proget_server>/api/docker-blobs/download/sha256%3Af033c4f65cdbf0bfa21d5543e56c0c41645eca4d893494bb4f0661b0f19ccc79 from the Clair container? Does that also return a 404 error? Just to confirm, all the requests in the Vulnerability log are warnings still correct?

Thanks,
Rich

Hi @scusson_9923,

Progress! It looks like we are past the SSL issue now. Can you check the diagnostics center in ProGet and see if there are any errors in there now?

Thanks,
Rich

Hi @scusson_9923,

I was just researching this a bit and it looks like they may have added a toggle to disable SSL checks in Clair when downloading docker layers. Can you try adding -insecure-tls to your docker run statement for Clair?

Thanks,
Rich

Hi @scusson_9923,

Please give me a little bit of time to work through this. If I have learned anything about Docker, it is that certificates are handled differently on every image. I need to do some digging to find out what is needed to make this work. I don't think HTTPS is a lost cause, we just need to figure out how Clair needs to handle these certs.

Thanks,
Rich

Hi @scusson_9923,

That is definitely the issue. It looks like the best way is to add your self-signed cert to the ca and add a docker mount to that (-v /path/to/quay/cert/ca.crt:/etc/pki/ca-trust/source/anchors/ca.crt). You may be able to do it with the Clair config also, but I could not find anything easily for that.

Thanks,
Rich

Hi @scusson_9923,

When you ping the server from the Clair image, are you pinging the server or the value that is in Web.BaseUrl? You should be using the value within Web.BaseUrl since that is the connection it is using. Also, are you using a port other than 443 for your ProGet cluster for the Clair connection?

Also, on your Clair configuration, do you have anything set for API Authorization Header? If so, could you try to remove that and see if that fixes your issue?

Lastly, does your Docker registry allow anonymous to pull your images? If not, could you temporarily allow anonymous access to that registry and give it a try? This will allow us to see if it is an issue with our automatic key creation logic for Docker images.

That would be the starting point I think to troubleshoot this. If that doesn't resolve the issue, then the next step would be to do some custom PowerShell calls to do a direct test with Clair.

I'm sorry for all the back and forth with this, but there definitely seems to be something blocking the connection, so now we just need to see which portion of the system is blocking it.

Thanks,
Rich

Hi @scusson_9923,

Looks like I misspoke earlier, the Clair integration will never return warnings unless an error happened while pulling an image. Would you be able to send a copy of the Vulnerability Scan logs to support@inedo.com with a subject of [QA-664] Clair Logs, so I can review the logs?

Thanks,
Rich

Hi @scusson_9923,

Two other things I forgot to ask.

Is your ProGet server accessible from your clair server? How the integration works, is ProGet sends a list of images to scan to Clair, which Clair then downloads the image layers from ProGet and scans them. Then ProGet calls back to clair to get the results.

Do you have your Web.BaseUrl set in the Advanced Settings? Because this scan runs from the ProGet Service, the Web.BaseUrl needs to be set so we know what URL to send to Clair to download the image layer.

Thanks,
Rich

Hi @scusson_9923,

Thanks for answering those. Let me do some checks in our lab and check back. The integration will always yield some warnings because not everything comes back in the expected layer format (configuration layers is a good example of this), but Clair randomly makes changes to their providers and their API and I want to make sure the test cases still work as expected.

Thanks.
Rich

Hi @scusson_9923,

I'm sorry to bombard you with some questions, but I think this will be a good way to start.

• What is your current version of ProGet?
• What is your current version of the Clair extension?
• What version of Clair are you running?
• Is this something new that has recently started happening?
• Does it give you that warning for all layers or just some specific layers?
• Would it be possible to share any sort of example image from a third-party registry that has this issue?

If you can answer those for me, that should give us a good start to resolve this issue.

Thanks,
Rich

Hi @brett-polivka,

Is only the health check failing? Are you still able to pull or search for images?

Also, do you see any errors in the Diagnostics center?

Thanks,
Rich

Hi @RobIII,

We received the email, please give us a little bit of time to review this and we will get back to you soon!

Thanks,
Rich

Hi @paul-reeves_6112,

No problem! Always happy to help.

Thanks,
Rich

Hi @paul-reeves_6112,

That's ok. It went through our internal testing and everything looked good so we pushed it to production with the last release. Thanks for verifying this for us!

Thanks,
Rich

Hi @paul-reeves_6112,

Thanks! Always happy to help. Please let us know if you find anything else!

Thanks,
Rich

Hi @harald-somnes-hanssen_2204,

Thanks for sending this over to us. It is very helpful! We will definitely be discussing this further as a team!

Thanks,
Rich

Hi @rbenfield_1885,

It looks like you are using the integrated webserver. It looks like this error is happening when it is attempting to register the port on Windows. Please make sure that the user is an administrative user and that the port that you are using for ProGet is not already in use.

Can you share the error that you are getting when you are trying to upgrade to ProGet 5.3?

Thanks,
Rich

Hi @rbenfield_1885,

Are you able to share the error you are seeing in the event viewer after upgrading to 5.2.32? Is that InedoHub error when you upgraded to 5.2.32 or when you tried to upgrade to 5.3.34?

Thanks,
Rich

Hi @paul-reeves_6112,

I think I have found the issue. I have created a pre-release extension, InedoCore 1.12.3-RC.1, that includes the fix. Could you please try to install this version and see if it fixes the issue for you?

You can follow our manual extension installation guide to either manually install the extension or to change BuildMaster to use our pre-release extension feed.

Thanks,
Rich

Hi @paul-reeves_6112,

Thanks for following up. We will take a look at this and see what is happening. Hang tight!

Thanks,
Rich

Hi @pumin_0299 & @jndornbach_8182 ,

This issue has been resolved and will release later this week in ProGet 5.3.34.

Thanks,
Rich

Hi @maxim_mazurok,

I'm glad to hear that repulling the package fixed this. Do you have an anti-virus installed on your ProGet server? Is it possible to check that to see if the package was quarantined?

Thanks,
Rich

Hi @maxim_mazurok,

Is this a cached package from a connector or did you upload this package directly? Has this version always been a problem or is this new? I think the best option may be to reupload this package or delete it and repull it from your connector (if you are using a connector).

The error you are seeing can happen when a file is also a 0 byte tgz file as well. My guess is that either the file failed to download/upload to ProGet or the zip file contents were quarantined from your local anti-virus.

Thanks,
Rich

Hi @paul-reeves_6112,

Thanks for taking a look. I corrected the color and improved the spacing in BuildMaster 7.0.7.

Thanks,
Rich

Hi @paul-reeves_6112,

Good catch! We will be fixing this as part of BM-3725 and it will release in BuildMaster 7.0.7.

Thanks,
Rich

Hi @internalit_7155,

After updating your license in ProGet, did you restart the web server (or application pool if using IIS)? If not, could you please try to restart your web server (or application pool if using IIS)?

Thanks,
Rich

Hi @liuhan907_8630,

I don't believe the health check is the reason for the error. Is it only gcr.io that is having this issue? Does this happen on all images for the gcr? Are you able to pull the image directly from gcr?

Also, has this image been pulled before? If so, there could be a bad image cached. You can delete the cached image by hovering over the button in the upper write and click Delete Cached Image and try again.

Thanks,
Rich

Hi @internalit_7155,

I know you have attempted the upgrade already, but in the future you can go to https://my.inedo.com/downloads and click the Upgrade Guidance & Change Notes button. That will allow you to enter in the versions you are upgrading from and to and view the upgrade path. Here is the result of that for your upgrade, ProGet 3.7.6 to 5.3.33. The upgrade path for this is to first upgrade to ProGet 5.2.32 then to 5.3.33.

Looking through your error, I have found this to be a known issue when upgrading a very old version of the database to our new tools directly. We actually recently fixed this bug in inedosql.

The best way to resolve this is to restore your database back to the 3.7.6 version and use the traditional installer to upgrade to ProGet 5.2.32 first. This will also allow you to log in and convert your NuGet feed to the new format. Once you upgrade to that version and convert your NuGet feed, then upgrade to ProGet 5.3.33. For ProGet 5.3.33, I would recommend using Inedo Hub to perform the upgrade. We have deprecated the traditional installer and it will no longer be generated in ProGet 6+.

Hope this helps!

Thanks,
Rich

Hi @paul-reeves_6112,

Thanks for bringing this up to us. I have fixed the styling and it will be updated in Build Master 7.0.6 that is expected to be released later this week.

Thanks,
Rich

Hi @paul-reeves_6112,

Thanks for checking this for me. I was able to get this resolved and it will be released later this week in BuildMaster 7.0.6.

Thanks,
Rich

Hi @paul-reeves_6112,

Thanks for bringing this up to us. Let me see what I can do about that. Quick question though, when you shrink the window, is it still 4 and 5 that have the issue or does it start shifting the problem down?

Thanks,
Rich

Hello @paul-reeves_6112,

I just pushed it to RC now. Can you please give it another try?

Thanks,
Rich

Hi @kichikawa_2913,

Thanks for the information. I have updated our Docker Troubleshooting Guide to include details about root-less containers. We are also discussing internally if we want to change the default port to be > 1024 going forward.

Thanks,
Rich

Hi @kichikawa_2913,

That's great to hear! Just to make sure, you left in the --expose=8080 correct?

For the LDAPS thing, that makes complete sense. The SSL certs are at a root level and there is nothing we can do to change that. But I will note that there is an undocumented feature I added to bypass certificate validation for LDAPS. If you navigate to Administration -> Change User Directory -> Advanced -> Active Directory (NEW) and then select Use LDAPS and Bypass LDAPS Certificate Validation, that will allow you to use LDAPS and it just bypasses any certificate errors in the process.

Thanks,
Rich

Hi @kichikawa_2913,

I have an idea on how to accomplish this. It looks like Docker allows you to expose ports in the run command. Here is what I'm thinking should work. The first thing to do is to map a volume to /usr/share/Inedo/SharedConfig. In my example, I'll map to SharedConfig.

So here would be the steps to try:

1. Create the config file using port 8080

echo '<?xml version="1.0" encoding="utf-8"?><InedoAppConfig><ConnectionString Type="SqlServer">'"`$SQL_CONNECTION_STRING"'</ConnectionString><WebServer Enabled="true" Urls="http://*:8080/"/></InedoAppConfig>' > SharedConfig/ProGet.config

2. run the container using the following command

podman run -d --userns=keep-id -v proget-packages:/var/proget/packages -v  `SharedConfig:/usr/share/Inedo/SharedConfig` -v /etc/pki/ca-trust/source/anchors:/usr/local/share/ca-certificates:ro --expose=8080 -p 8080:8080 --name=proget -e ASPNETCORE_URLS='http://+:8080' -e SQL_CONNECTION_STRING='Server=SERVERNAME;Database=ProGet;User ID=USERNAME;Password=PASSWORD' -e TZ='America/New_York' -i -t proget.inedo.com/productimages/inedo/proget:5.3.32 /bin/bash

Can you please give that a try?

Thanks,
Rich

Hi @kichikawa_2913,

Thanks for following up with that information. I was doing a little more research on podman this morning and it looks like this is a problem with the internal port. I have actually found quite a bit of people who claim that port 80 inside a container will not work on a root-less image, but looking at the podman document, their example uses port 80 inside the internal container. Let me look if there is an easy way to change that port binding on Linux and I'll get back to you shortly.

While I'm checking on that, can you try something for me? It looks like podman also needs a protocol to bind the port to. Could you try the following and let me know if that works?

podman run -d --userns=keep-id -v proget-packages:/var/proget/packages -v /etc/pki/ca-trust/source/anchors:/usr/local/share/ca-certificates:ro -p 8080:80/tcp --name=proget -e SQL_CONNECTION_STRING='Server=SERVERNAME;Database=ProGet;User ID=USERNAME;Password=PASSWORD' -e TZ='America/New_York' -i -t proget.inedo.com/productimages/inedo/proget:5.3.32 /bin/bash

Thanks,
Rich

Hi @kichikawa_2913,

I'm not very familiar with Podman root-less containers. Does it expose all ports that are internal to the container externally?

I think the issue that may be happening is the changing of the internal port which we have configured to be port 80. I do not think the ASPNETCORE_URLS will actually change the port that the site runs on internally because we are actually hosting the kestrel server within our service so we tell it to use port 80. Have you tried running the command as this:

podman run -d --userns=keep-id -v proget-packages:/var/proget/packages -v /etc/pki/ca-trust/source/anchors:/usr/local/share/ca-certificates:ro -p 8080:80 --name=proget -e SQL_CONNECTION_STRING='Server=SERVERNAME;Database=ProGet;User ID=USERNAME;Password=PASSWORD' -e TZ='America/New_York' -i -t proget.inedo.com/productimages/inedo/proget:5.3.32 /bin/bash

After a bit of research, it looks like you will get that error when you are trying to bind to a host port that is already in use. So make sure nothing else is using port 8080 on the host.

The messenger endpoint URL is a URL that is only used internally in the container. You should not run into any conflicts with the default port that is used since it is not exposed externally. If you want to test using a different port, you can connect to your SQL database and run the following stored procedure. Make sure to update to the port you want to try.

EXEC [Configuration_SetValue] 'Service.MessengerEndpoint', 'tcp://127.0.0.1:6001'

Please let me know what you find!

Thanks,
Rich

Hi @Jana-Rebmann_7461,

Unfortunately, NPM does not include any parameters you can pass to uncache the install results. You can try delete %appdata%\npm-cache before each run and see if that helps. I have also seen that this can be caused by the NPM verifying the hash with the one stored in the package-lock.json as well. But that would only break if an anti-virus or something deleted a file out of the zip.

Have you tried upgrading your NPM client recently? How long has this issue existed?

Thanks,
Rich

Hi @Jana-Rebmann_7461,

Thanks! I received the log and took a look. The only errors I see before EPERM error is an error saying warn tar ENOENT: no such file or directory, open for the core-js module. What I think I happening is the node_modules directory is becoming corrupted, which is a pretty common occurrence. Typically you just run rm -rf node_modules && npm install and that fixes it.

As for the EPERM error, this really looks like something is locking the files. Do you have any sort of anti-virus or anything watching that directory? Can you try ignoring that directory and testing again to see if you get the error still? Does this happen every time or just sporadically?

I know you mentioned that this didn't happen when switching to npmjs.org, but this could also be an issue with the npm cache and how it links to the installed files. When switching repositories, npm will create a different cache folder and link from a different location. I think it happens to just be a coincidence and you will most likely hit a similar issue after using npmjs.org for a bit.

Thanks,
Rich

Hi @Jana-Rebmann_7461,

Typically when you see an EPERM error that is caused by the OS. You can find quite a bit about it on Google and Stack Overflow. Typically users experience that issue on windows that have their drives indexed. Although, I want to believe that the npm client is your issue, I have a feeling that something is happening first and then the EPERM error is happening on the cleanup of the file. Especially since you are saying that npmjs.org does not have the problem. I noticed at the end of the error, it says "A complete log of the run exists at", would you be able to send that over to us?

If you don't feel comfortable attaching that to this post, you can send it to support@inedo.com and put [Support QA-609] in the subject. Just please let me know that you emailed it so I can keep an eye out for the email.

Thanks,
Rich

Hi @hwittenborn,

Could you please share a screenshot of what you are seeing? When I click on the package version, I see a Delete Package button when I hover over the Download Package multi-button. See below:

Thanks,
Rich

Hi @Fred and @hwittenborn,

You can actually configure docker to use insecure HTTP registries. As it states in our documentation, you can register a host and port as an insecure registry which will then tell your docker client to use HTTP instead of HTTPS. A good way to rule out the ProGet container would be to configure your Docker daemon to use insecure registries pointing to the HTTP port of your ProGet container and try to push it that way. For example:

If you have your ProGet container running HTTP on port 80 and host proget.domain.local, add this to your Docker daemon (or settings in Docker Desktop on Windows and Mac):

{
  "registry-mirrors": [],
  "insecure-registries": [
    "proget.domain.local:80"
  ],
  "debug": false,
  "experimental": false
}

Then if your repository name would be: proget.domain.local:80/my/imagename and your push command would look like:

docker push proget.domain.local:80/my/imagename:tagname

That will then push the image over HTTP vs HTTPS.

Is this only an issue in Visual Studio? Have you tried to push your image using the command line?

Thanks,
Rich

Hi @hwittenborn,

Thanks for the extra information. It is very much appreciated!

Are you saying the file fails to get added if you save it under the filename {feed-name}.pub?

I actually used it as a gpg file extension, but honestly, your solution much better. The file extension doesn't actually matter that much. Once it is added using apt-key, it just extract to contents and stores them in their keyring. You can see this by running sudo apt-key list.

I was more stating that in order for it to work for me on Ubuntu 20.04, I had to have everything named as the lowercase feed name in order for it to work for me. Although, I'm not sure if the lower case part matters. For example, for a feed name defaultdebian, I ran this:

wget -qO - http://proget.localhost/debian-feeds/defaultdebian.pub | sudo apt-key add -
echo "deb http://proget.localhost/ defaultdebian main" | sudo tee /etc/apt/sources.list.d/defaultdebian.list

sudo apt update

As a test I changed echo "deb http://proget.localhost/ defaultdebian main" | sudo tee /etc/apt/sources.list.d/defaultdebian.list to be echo "deb http://proget.localhost/ defaultdebian main" | sudo tee /etc/apt/sources.list.d/proget-defaultdebian.list and that actually wouldn't work for me. This caused a bunch of warnings when I ran sudo apt update. For some reason this only mattered in newer versions of apt though.

All in all, I will update the documentation to correct your command (adding the missing '-' ) and to include the updates to the naming. Hopefully, this will be clearer going forward for other users.

Thanks for all of your help!

Thanks,
Rich

Hi @hwittenborn,

I recently ran into similar problems in the latest version of apt on Ubuntu. This didn't actually seem to be an issue in the older versions of apt. I determined that naming is very important when using Debian and this is how I was able to get this to work:

wget -O "{feed-name}.gpg" http://{proget-server}/debian-feeds/{feed-name}.pub && sudo apt-key add "{feed-name}.gpg"
echo "deb http://{proget-server}/ {feed-name} {component-name}" | sudo tee /etc/apt/sources.list.d/{feed-name}.list

Basically, I had to add download the pub as a {feed-name}.gpg then add that key to apt. Then when registering it in the sources.list.d I had to name it {feed-name}.list. In my experience, if they are not all named the same way, when you try to run sudo apt update you will get a bunch of warnings and the packages will never actually show up as available to install.

I'm not sure if this is the same thing you were seeing, but I was waiting on a couple of customers to confirm that running the commands this way it all worked before updating the documentation.

My guess is that using wget -qO - http://{proget-server}/debian-feeds/{feed-name}.pub | sudo apt-key add -, it added the key with the name of {feed-name}.gpg or whatever extensions apt converts it to. When you ran echo "deb http://{proget-server}/ {feed-name} {component-name}" | sudo tee /etc/apt/sources.list.d/{proget-deb}.list what did your use of {proget-deb}.list? Was it your feed name?

Thanks,
Rich

Hi @Fred,

The message I'm pulling out of this error is The plain HTTP request was sent to HTTPS port. This indicates either the docker client is trying to push a non-SSL request to an SSL port (like HTTP://proget.com:443 where 443 is bound to SSL) or you have a bad forward of the host and port in your NGINX file. I recently did some testing on this and this was the NGINX file that I tested and worked: https://docs.inedo.com/docs/https-support-on-linux

Thanks,
Rich