Posts made by rhessinger

Hi @kichikawa_2913,

I have an idea on how to accomplish this. It looks like Docker allows you to expose ports in the run command. Here is what I'm thinking should work. The first thing to do is to map a volume to /usr/share/Inedo/SharedConfig. In my example, I'll map to SharedConfig.

So here would be the steps to try:

1. Create the config file using port 8080

echo '<?xml version="1.0" encoding="utf-8"?><InedoAppConfig><ConnectionString Type="SqlServer">'"`$SQL_CONNECTION_STRING"'</ConnectionString><WebServer Enabled="true" Urls="http://*:8080/"/></InedoAppConfig>' > SharedConfig/ProGet.config

2. run the container using the following command

podman run -d --userns=keep-id -v proget-packages:/var/proget/packages -v  `SharedConfig:/usr/share/Inedo/SharedConfig` -v /etc/pki/ca-trust/source/anchors:/usr/local/share/ca-certificates:ro --expose=8080 -p 8080:8080 --name=proget -e ASPNETCORE_URLS='http://+:8080' -e SQL_CONNECTION_STRING='Server=SERVERNAME;Database=ProGet;User ID=USERNAME;Password=PASSWORD' -e TZ='America/New_York' -i -t proget.inedo.com/productimages/inedo/proget:5.3.32 /bin/bash

Can you please give that a try?

Thanks,
Rich

Hi @kichikawa_2913,

Thanks for following up with that information. I was doing a little more research on podman this morning and it looks like this is a problem with the internal port. I have actually found quite a bit of people who claim that port 80 inside a container will not work on a root-less image, but looking at the podman document, their example uses port 80 inside the internal container. Let me look if there is an easy way to change that port binding on Linux and I'll get back to you shortly.

While I'm checking on that, can you try something for me? It looks like podman also needs a protocol to bind the port to. Could you try the following and let me know if that works?

podman run -d --userns=keep-id -v proget-packages:/var/proget/packages -v /etc/pki/ca-trust/source/anchors:/usr/local/share/ca-certificates:ro -p 8080:80/tcp --name=proget -e SQL_CONNECTION_STRING='Server=SERVERNAME;Database=ProGet;User ID=USERNAME;Password=PASSWORD' -e TZ='America/New_York' -i -t proget.inedo.com/productimages/inedo/proget:5.3.32 /bin/bash

Thanks,
Rich

Hi @kichikawa_2913,

I'm not very familiar with Podman root-less containers. Does it expose all ports that are internal to the container externally?

I think the issue that may be happening is the changing of the internal port which we have configured to be port 80. I do not think the ASPNETCORE_URLS will actually change the port that the site runs on internally because we are actually hosting the kestrel server within our service so we tell it to use port 80. Have you tried running the command as this:

podman run -d --userns=keep-id -v proget-packages:/var/proget/packages -v /etc/pki/ca-trust/source/anchors:/usr/local/share/ca-certificates:ro -p 8080:80 --name=proget -e SQL_CONNECTION_STRING='Server=SERVERNAME;Database=ProGet;User ID=USERNAME;Password=PASSWORD' -e TZ='America/New_York' -i -t proget.inedo.com/productimages/inedo/proget:5.3.32 /bin/bash

After a bit of research, it looks like you will get that error when you are trying to bind to a host port that is already in use. So make sure nothing else is using port 8080 on the host.

The messenger endpoint URL is a URL that is only used internally in the container. You should not run into any conflicts with the default port that is used since it is not exposed externally. If you want to test using a different port, you can connect to your SQL database and run the following stored procedure. Make sure to update to the port you want to try.

EXEC [Configuration_SetValue] 'Service.MessengerEndpoint', 'tcp://127.0.0.1:6001'

Please let me know what you find!

Thanks,
Rich

Hi @Jana-Rebmann_7461,

Unfortunately, NPM does not include any parameters you can pass to uncache the install results. You can try delete %appdata%\npm-cache before each run and see if that helps. I have also seen that this can be caused by the NPM verifying the hash with the one stored in the package-lock.json as well. But that would only break if an anti-virus or something deleted a file out of the zip.

Have you tried upgrading your NPM client recently? How long has this issue existed?

Thanks,
Rich

Hi @Jana-Rebmann_7461,

Thanks! I received the log and took a look. The only errors I see before EPERM error is an error saying warn tar ENOENT: no such file or directory, open for the core-js module. What I think I happening is the node_modules directory is becoming corrupted, which is a pretty common occurrence. Typically you just run rm -rf node_modules && npm install and that fixes it.

As for the EPERM error, this really looks like something is locking the files. Do you have any sort of anti-virus or anything watching that directory? Can you try ignoring that directory and testing again to see if you get the error still? Does this happen every time or just sporadically?

I know you mentioned that this didn't happen when switching to npmjs.org, but this could also be an issue with the npm cache and how it links to the installed files. When switching repositories, npm will create a different cache folder and link from a different location. I think it happens to just be a coincidence and you will most likely hit a similar issue after using npmjs.org for a bit.

Thanks,
Rich

Hi @Jana-Rebmann_7461,

Typically when you see an EPERM error that is caused by the OS. You can find quite a bit about it on Google and Stack Overflow. Typically users experience that issue on windows that have their drives indexed. Although, I want to believe that the npm client is your issue, I have a feeling that something is happening first and then the EPERM error is happening on the cleanup of the file. Especially since you are saying that npmjs.org does not have the problem. I noticed at the end of the error, it says "A complete log of the run exists at", would you be able to send that over to us?

If you don't feel comfortable attaching that to this post, you can send it to support@inedo.com and put [Support QA-609] in the subject. Just please let me know that you emailed it so I can keep an eye out for the email.

Thanks,
Rich

Hi @hwittenborn,

Could you please share a screenshot of what you are seeing? When I click on the package version, I see a Delete Package button when I hover over the Download Package multi-button. See below:

Thanks,
Rich

Hi @Fred and @hwittenborn,

You can actually configure docker to use insecure HTTP registries. As it states in our documentation, you can register a host and port as an insecure registry which will then tell your docker client to use HTTP instead of HTTPS. A good way to rule out the ProGet container would be to configure your Docker daemon to use insecure registries pointing to the HTTP port of your ProGet container and try to push it that way. For example:

If you have your ProGet container running HTTP on port 80 and host proget.domain.local, add this to your Docker daemon (or settings in Docker Desktop on Windows and Mac):

{
  "registry-mirrors": [],
  "insecure-registries": [
    "proget.domain.local:80"
  ],
  "debug": false,
  "experimental": false
}

Then if your repository name would be: proget.domain.local:80/my/imagename and your push command would look like:

docker push proget.domain.local:80/my/imagename:tagname

That will then push the image over HTTP vs HTTPS.

Is this only an issue in Visual Studio? Have you tried to push your image using the command line?

Thanks,
Rich

Hi @hwittenborn,

Thanks for the extra information. It is very much appreciated!

Are you saying the file fails to get added if you save it under the filename {feed-name}.pub?

I actually used it as a gpg file extension, but honestly, your solution much better. The file extension doesn't actually matter that much. Once it is added using apt-key, it just extract to contents and stores them in their keyring. You can see this by running sudo apt-key list.

I was more stating that in order for it to work for me on Ubuntu 20.04, I had to have everything named as the lowercase feed name in order for it to work for me. Although, I'm not sure if the lower case part matters. For example, for a feed name defaultdebian, I ran this:

wget -qO - http://proget.localhost/debian-feeds/defaultdebian.pub | sudo apt-key add -
echo "deb http://proget.localhost/ defaultdebian main" | sudo tee /etc/apt/sources.list.d/defaultdebian.list

sudo apt update

As a test I changed echo "deb http://proget.localhost/ defaultdebian main" | sudo tee /etc/apt/sources.list.d/defaultdebian.list to be echo "deb http://proget.localhost/ defaultdebian main" | sudo tee /etc/apt/sources.list.d/proget-defaultdebian.list and that actually wouldn't work for me. This caused a bunch of warnings when I ran sudo apt update. For some reason this only mattered in newer versions of apt though.

All in all, I will update the documentation to correct your command (adding the missing '-' ) and to include the updates to the naming. Hopefully, this will be clearer going forward for other users.

Thanks for all of your help!

Thanks,
Rich

Hi @hwittenborn,

I recently ran into similar problems in the latest version of apt on Ubuntu. This didn't actually seem to be an issue in the older versions of apt. I determined that naming is very important when using Debian and this is how I was able to get this to work:

wget -O "{feed-name}.gpg" http://{proget-server}/debian-feeds/{feed-name}.pub && sudo apt-key add "{feed-name}.gpg"
echo "deb http://{proget-server}/ {feed-name} {component-name}" | sudo tee /etc/apt/sources.list.d/{feed-name}.list

Basically, I had to add download the pub as a {feed-name}.gpg then add that key to apt. Then when registering it in the sources.list.d I had to name it {feed-name}.list. In my experience, if they are not all named the same way, when you try to run sudo apt update you will get a bunch of warnings and the packages will never actually show up as available to install.

I'm not sure if this is the same thing you were seeing, but I was waiting on a couple of customers to confirm that running the commands this way it all worked before updating the documentation.

My guess is that using wget -qO - http://{proget-server}/debian-feeds/{feed-name}.pub | sudo apt-key add -, it added the key with the name of {feed-name}.gpg or whatever extensions apt converts it to. When you ran echo "deb http://{proget-server}/ {feed-name} {component-name}" | sudo tee /etc/apt/sources.list.d/{proget-deb}.list what did your use of {proget-deb}.list? Was it your feed name?

Thanks,
Rich

Hi @Fred,

The message I'm pulling out of this error is The plain HTTP request was sent to HTTPS port. This indicates either the docker client is trying to push a non-SSL request to an SSL port (like HTTP://proget.com:443 where 443 is bound to SSL) or you have a bad forward of the host and port in your NGINX file. I recently did some testing on this and this was the NGINX file that I tested and worked: https://docs.inedo.com/docs/https-support-on-linux

Thanks,
Rich

Hi @hwittenborn,

Thanks for the additional information. We talked about this in our team meeting this week and we decided we are going to clean up when restarts are needed and we are hoping to give better messaging to the user when a container needs to be restarted. Some of the restart requirements are leftover from previous requirements that have since changed. So hopefully we can make this process a bit easier in the future.

Thanks,
Rich

Hi @hwittenborn,

I took a deeper look into this and it looks to work as expected. When it comes to Docker containers, we can't automatically restart the website like we can on Windows. So anytime that a setting is modified in the Advanced Settings or an extension is installed or removed, the docker container for ProGet will need to be restarted. In older versions of the image, we attempted to display a message to restart the container, but that implementation did not always work as designed so it was removed. I have forwarded this over to our products team to review any possible solutions that we could make in ProGet 6 to make this more obvious. I'm also working on ways to update our documentation to make this a bit clearer.

Thanks,
Rich

Hi @hwittenborn,

Let me do some testing on my side and I'll let you know what I find. What OS is your Docker engine running on?

Thanks,
Rich

Hi @hwittenborn,

Am I correct to say that the 502 errors are not with the NGINX proxy in front of it as well? Is this only when you are uninstalling the extensions? If not, what other settings are you changing when you see this issue?

Thanks,
Rich

Hi @Stephen-Schaff,

In newer versions of ProGet, we improved our performance by caching our user's privileges. This was especially necessary when using Active Directory based user directories. This can cause issues similar to this one where it can take some time before the privilege is updated. This is especially noticeable in ProGet HA clusters. The caching timeout is configurable in the Administration -> Advanced Settings by altering the Web.PrivilegeCacheExpiration value. By default, we set it to 30 minutes. There are areas that should refresh the cache on save, but those will only work in single instances of ProGet and I'm guessing it did not get triggered in this case.

Hope this helps!

Thanks,
Rich

Hi @Fred,

I think I may have found the issue. Can you include proxy_set_header Host $http_host; in your location node and see if that fixes your issue?

Thanks,
Rich

Hi @Fred,

It definitely can be settings in your Nginx settings, but nothing is jumping out at me. I am by no means an Nginx expert, but everything looks normal. I know we have quite a few users using Nginx with ProGet, so we know this is a working combination.

Just to get the easy Docker nuances out of the way first, I just want to verify that your certificate is not a self-signed certificate or generated by an internal certificate authority (there are things you need to set in the Docker client to get that to work). Also are you able to push images to ProGet using the command line? If not, are you able to send over the output of the CLI?

One other thing to try is to set the Web.BaseUrl in Administration -> Advanced Settings to your HTTPS URL (ex: https://proget.xxx.com).

Thanks,
Rich

Hi @Fred,

Does the HTTPS url (https://proget.xxx.com) work in your browser as well?

Thanks,
Rich

Hi @kichikawa_2913,

Sorry, this is actually expected. When we release our products, they include the extensions that were released at the time of the product release. In this case, I released this version of InedoCore after we did the product release.

If you look at our documentation for upgrading your docker image, the command includes --volumes-from=proget-old. This will auto migrate the previous volumes created from the previous version of ProGet and that will keep the updated extension (as long as the previous extensions is newer than the included extension version).

Also, in Administration -> Advanced Settings, you can change Extensions.ExtensionsPath to a mapped path and that will also do the same thing (if the version in this directory is newer than the included) and give you easier access to the extension files.

Thanks,
Rich

Hi @kichikawa_2913,

I'm sorry for not including that in my previous comment. You are correct 1.10.7 is the new version of the InedoCore extension that includes these fixes for LDAP and LDAPS on Docker.

Thanks,
Rich

Hi @kichikawa_2913,

I just released this extension to production now. Please let me know if you don't see an official release.

Thanks,
Rich

Hi @kichikawa_2913,

Did those users still experience the long initial login? Or did that go away?

Thanks,
Rich

Hi @kichikawa_2913,

You should just be able to remove the custom port and uncheck LDAPS then restart your container and that should remove LDAPS from your AD instance.

Thanks,
Rich

Hi @kichikawa_2913,

Thanks for the information and I'm glad you can log in now. Let me dig in and see what I can find on these Anti-CSRF errors. Normally these happen because a reverse proxy is not properly forwarding headers. Do you have a reverse proxy (like Nginx or apache) sitting in front of this container? Also, do you know if once they get logged in if ProGet seems to run fine, or does each page request take a while to load?

Would you be willing to test this without LDAPS so we can see if it is an LDAPS issue or not?

Thanks,
Rich

Hi @kichikawa_2913,

I'm going to attempt to recreate your error on my system as well and see if I can find this issue. LDAPS on Docker has not been a popular option with our customers so far due to the complexity in managing the AD certificates. I really appreciate your patience in working through this with us.

While I try to recreate this, could you try using an incognito browser and see if you are able to load and login? Also, can you try to restart your container again? Also, can you please verify the LDAPS connection is still working with the openssl command again?

Thanks,
Rich

Hi @kichikawa_2913,

I think I have identified the issue. I have just pushed another version of InedoCore, version 1.10.7-CI.2 . Could you update and give that a try? I also added an option to bypass the LDAPS certificate verification. It is something that I would only use while testing. The solution you have with adding your certificates as valid certs is a more secure solution. One last thing to make sure you set is the Domain Controller Host. It can just be set to your domain (ex: domain.network using your steps from above). Linux/Docker does not seem to translate domain URLs the same way windows does.

Thanks,
Rich

Hi @kichikawa_2913,

Unfortunately, these logs do not show any errors (outside of a couple of resource images failing to load). Do you see any errors in your Diagnostics Center in the administration page?

As for the port, upon further inspection, it looks like the port handling was removed when we upgraded ProGet to support .NET 5 on Docker. It used to parse it off the Domain Controller host address, I have now added it as a separate field that you can set. If you can upgrade your Inedo Core extension to 1.10.6-CI.13, that will give you the option to enter a custom port number. Could you please give that a try and see if that works?

Thanks,
Rich

Hi @kichikawa_2913,

Did you see any sort of error when using LDAPS? I don't see one in your previous logs message. Also, are you using the standard port (636) for LDAPS?

Thanks,
Rich

Hi @coskun_0070,

Thank you for your patience. I know how frustrating this can be, especially for something as simple as an installation. As Alana stated, this is the first time we have really seen this issue and your architecture is a very typical setup for our users. Normally the only thing you need to do is specify your SQL Connection string to your SQL Server and this install without issue.

Typically when installing using the Inedo Hub, if you have the connection string using Integrated Authentication it will use the current user's account to connect to the database and any file access needed while installing. Once it is installed, it will then use Network Service by default, but you can change the account the service and IIS run as. If you specify a SQL auth based username and password in the connection string to your SQL server in Inedo Hub, Inedo Hub will then use that account to connect to SQL server, the current user for any file access needed during the installation, and then it will default the Service and Web App to run as Network Service.

With all that being said, I don't think it is the Network Service account that is having issues because that will only be used after the installation. This almost sounds like the .NET Framework failed to install some needed components to connect to SQL Server. That would explain why installing SQL Express fixed the installation issue for you. Would it be possible to install SQL Server Express on your server then try to install ProGet using Inedo Hub with a connection string using sa pointing to your SQL Server instance on the other machine?

One last thing to try is to temporarily turn off Windows Firewall on your server. I'm wondering if Inedo Hub got blocked trying to communicate to SQL Server and that is causing this issue as well.

Thanks again for your patience and for working with us on this issue.

Thanks,
Rich

Hi @kichikawa_2913,

That's great! Thanks for giving me an update. LDAPS is definitely more of an advanced configuration option currently. The hardest piece of LDAPS is getting and keeping the certificate valid and up to date in the Docker container. It depends on how you have your AD server setup, but they typically regenerate their certs and distribute them automatically. We actually use a third-party library Novell.Directory.Ldap.NETStandard for our AD connection in Docker due to the LDAPS support missing from the built-in one for .NET 5.

As I stated before, creating a new image using the ProGet image as your base image tends to be the easiest way to add certificates to your container. Our image is built on top of dotnet/aspnet:5.0.5 (Debian 10 based). There are a handful of ways to do this.

• Using ProGet as a base image, add the domain cert file(s) in the DockerFile
• Using ProGet as a base image, add a mapping to the certificate folder in your DockerFile and manage that on your host.
• use docker exec to add the certificates after the image has started
• etc...

Stack Overflow can be helpful in adding certificates to your containers for your AD setup. Right now we have not determined a standard way of setting this up since each instance we have dealt with seems to be configured differently, but we always look for feedback in this configuration process.

Thanks,
Rich

Hi @philippe-camelio_3885,

I have just built a new CI version of the [Scripting Extension 1.10.3-CI.3] (https://proget.inedo.com/feeds/PrereleaseExtensions/inedox/Scripting/1.10.3-CI.3). This version includes the Minimum Version on Ensure PS Module and adds a new Ensure-PsRepository operation. Please take a look and let me know if you have any issues.

Thanks,
Rich

Hi @kichikawa_2913,

I have a build of the InedoCore extension 1.10.6-CI.11 that should resolve this issue. I have tested this in our lab and it has corrected the issue for me. Could you please try to upgrade this extension to see if this fixes your issue? The easiest way to install the pre-release extension on Docker is to follow the instruction listed in our pre-release extension guide.

Thanks,
Rich

Hi @kichikawa_2913,

We have identified the issue and are working on a solution. The good news is that this is an issue with our InedoCore extensions, so once we correct this, you will be able to just update that extension. I will let you know as soon as we have something for you to try (it should be tomorrow).

Thanks,
Rich

Hi @coskun_0070,

Yeah, that rules out an internet connection. We actually have a decent amount of logging during the process, but for whatever reason, this error is happening before we really start anything on the install process. As Alana stated, this is most likely happening during the initial package extraction. I'm going to talk with our installer team and see if we can add better error logging in those early stages of the install process. That should help with this process in the future.

Thanks,
Rich

Hi @bozho_2851,

You may have to restart your ProGet webserver (IIS or Integrated Web Server) to clear this message. You shouldn't need to but the violation may still be cached and restarting the webserver may correct this for you.

Thanks,
Rich

Hi @coskun_0070,

Glad that the traditional installer worked for you. I agree using the Inedo Hub is the preferred option since the traditional installer will be removed in the future.

Could you try to generate an offline installer using the InedoHub? That should at least tell us if it is a problem with the hub connecting to the internet or if it is a problem extracting the files locally. Also does your server have any sort of third party firewall it is connecting through? If so, it might be worthwhile to verify the connections are not getting blocked as well.

Thanks,
Rich

Hi @bozho_2851,

Does your ProGet instance sit behind a proxy, like nginx? We have seen this issue with self connectors when connecting to itself via the proxy address. If so, you can try to use the hostname of the server or 127.0.0.1 in your URL in place of the FQDN.

Thanks,
Rich

Hi @kichikawa_2913,

Thanks for providing this additional information. Please give me a bit of time to review this and get back to you. I was able to recreate this issue on the ProGet image, but I need to find the root cause of this first.

Thanks,
Rich

Hi @kichikawa_2913,

We don't have much experience with Podman, we typically use the Docker engine, but not having networks should be OK as long as you can access the database server from the container.

I think the first issue is that you will need to use the username as just user instead of user@domain.network.

The next piece is LDAPS. One thing that is tricky is that your domain has to have a valid certificate. Self-signed (or domain generated) certificates don't seem to work in most cases. We have had some success from customers who have registered the certificates inside the container. Typically you would add the certificate by creating a new docker image based on the ProGet image.

Hope this helps!

Thanks,
Rich

Hi @coskun_0070,

Are you using any sort of proxy on your server?

Thanks,
Rich

Hi @kichikawa_2913,

I have forwarded this over to our licensing team, they should be reaching out to you shortly.

Thanks,
Rich

Hi @philippe-camelio_3885,

Thanks for the information. I'll let you know when I get the minimum version added and you can give it a try.

Thanks,
Rich

Hi @flavio-campana_0936,

BuildMaster 7 currently supports SDK 1.12. When running in Docker, it supports a minimum SDK of 1.9. On Windows, it supports a minimum SDK of 1.0.

Thanks,
Rich

Hi @coskun_0070,

Do you see any errors in your windows event viewer?

Thanks,
Rich

Hi @flavio-campana_0936,

Thanks for bringing this up with us. We have fixed this issue in 7.0.0-rc.59 image. I will also be updating the documentation shortly for BUILDMASTER_SQL_CONNECTION_STRING.

Thanks,
Rich

Hi @philippe-camelio_3885,

Thanks for the information. I'll look at adding a new operation Ensure-PsRepository which at least should help with adding the repo for installing modules. I'll also add MinimumVersion to the Ensure-PsModule operation as well. I would expect it to take either version or minimum version but not both. Do you see any issues with that solution?

Thanks,
Rich

Hi @philippe-camelio_3885,

Thanks for bringing this to our attention. I have created a ticket, OT-414, because I believe this is a problem with the UI, not the underlying Git raft. If this ends up being an issue with the extension, I'll update this post and let you know.

Thanks,
Rich

Hi @philippe-camelio_3885,

What it looks like is happening is that the repository registered to the DSC module is not reachable via the Install-Module command. The Ensure-PsModule operation runs the Install-Module PowerShell command. I think the best way to test this is to run Get-PSRepository and see if your repository is listed in that operation.

It shouldn't be too difficult to use PSEnsure to make a script to add the repository to be used with Ensure-PSModule.

I do have a question though. Is there a reason you are using MinimumVersion over Version?

Thanks,
Rich

Hi @Stephen-Schaff,

I was finally able to recreate this error and I believe I fixed it in PG-1948. This is set to be released on Friday in ProGet 5.3.28. If you would like to apply the fix now, I have attached a SQL script to PG-1948 and you can run it against your ProGet database. If you run this script, it will not affect future upgrades of ProGet.

Thanks,
Rich