HTTPS with self hosted ProGet and internal web server

tkolb

In turn of the upcoming HTTPS-Everywhere at Nuget we currently face the situation that our build will become instable due to "nuget restore".

That currently produces the NU1803-Warning with the latest command line when using our internally hosted ProGet-Instance. We already levitate all warnings in CI to errors so this will break our CI if the command line is updated.

Other entries here regarding HTTPS are rather old and reference documentation not available anymore.

Are there plans for (easy) HTTPS support in the internal webserver? Applying a self hosted certificate and deploying it to all build servers and developer machines seems over the top for me right now.

We might opt-out for this warning and the upcoming error in 11/2023 for the time being but nevertheless at least in 11/2024 this will be the issue.

--
UPDATE: the following was added to the question on 2023/28/02

ProGet Integrated Web Server Now Supports HTTPS

Starting in ProGet 2022.19, we have added the ability to configure your integrated web server's HTTP/S bindings and a certificate directly in ProGet. See Configuring HTTPS on the Integrated Web Server to learn more.

rhessinger

Hi @tkolb_7784,

If you are hosting on a windows machine, the easiest solution right now is to migrate your server to use IIS and then add an SSL binding to your site. If you do not want to purchase a new certificate and the self-signed certificate is too much work, you can use Let's Encrypt and configure it via winacme.

If you do not want to use IIS, then you will need to use a reverse proxy to handle SSL connections. Any reverse proxy can be used and a pretty simple one to configure is stunnel. Most reverse proxies can also be used with Let's Encrypt.

If you are hosting via Linux (Docker), then you will need to use a reverse proxy to handle SSL connections. We have a documentation page for different Linux-based reverse proxies including an example for setting up NGINX. These reverse proxies also support Let's Encrypt also.

Please let me know if you have any questions.

Thanks,
Rich

apxltd

Hi @tkolb_7784

I wanted to provide an official answer to this:

Are there plans for (easy) HTTPS support in the internal webserver? Applying a self hosted certificate and deploying it to all build servers and developer machines seems over the top for me right now.

Yes. Not exactly sure how yet, but we would like the integrated webserver to support this as easily as possible.

jw

We are also interested in this support, since we really don't want to go down the IIS route.

Yes. Not exactly sure how yet, but we would like the integrated webserver to support this as easily as possible.

From what I can tell ProGet seems to be an ASP.NET Core 6.0 application and the underlying (internal) webserver is probably Kestrel. We faced a similar requirement with our own product running the same stack.

A first easy way to support this would be to expose more of the relevant Kestrel config: https://learn.microsoft.com/en-us/aspnet/core/fundamentals/servers/kestrel/endpoints?view=aspnetcore-6.0#replace-the-default-certificate-from-configuration-1

We are particularly interested in the HttpsInlineCertStore option, to be able to use certificates directly from the Windows cert store.

So maybe extending the ProGet.config with the relevant parts could be a good first step, before any more fancy features like UI integration or self-signed cert generation are implemented.

<InedoAppConfig>
  <WebServer Enabled="true" Urls="http://*:8624/">
     <endpoint config goes here />
  </WebServer>
</InedoAppConfig>

apxltd

Hi @lm ,

Thanks for the feedback - and you are correct; it is ASP.NET 6.

Starting with the configuration file is a great idea. The Inedo Hub provides a UI to that anyways.

The documentation for our configuration file is here:
https://docs.inedo.com/docs/installation-configuration-files

Obviously it's pretty easy to just wire up values (and we'd be happy to do that), but my only hesitation is documentation and cross-platform compatibility.

Brainstorming a bit but... from the UI, we will just support one URL I think. I don't know why you'd want multiple...

The docs you linked from Microsoft are confusing, and it's not really clear what fields to use or why. I guess, in general, you'd select a file (which InedoHub would generate?) or the cert store (that's Windows only, right?).

And I guess what, when you generate a cert yourself, it's a pfx or something? I always forget the differences....

How would you suggest we modify that element to support the different options they have?

  <WebServer Enabled="true/false" Urls="web server listen URLs" />

What would you want to put in, or suppose other (advanced) users would want to put in?

Cheers,
Alex

jw

Obviously it's pretty easy to just wire up values (and we'd be happy to do that), but my only hesitation is documentation and cross-platform compatibility.

Cross-platform should not be an issue, since this is one of the reasons Microsoft created Kestrel in the first place. Of course the Windows cert store part of the configuration does not make sense in a Linux context, but that should be pretty much the only limitation.

Brainstorming a bit but... from the UI, we will just support one URL I think. I don't know why you'd want multiple...

From what I understand at least two URLs are needed to support HTTP and HTTPS side by side, but that could be abstracted away by the UI.

The docs you linked from Microsoft are confusing, and it's not really clear what fields to use or why. I guess, in general, you'd select a file (which InedoHub would generate?) or the cert store (that's Windows only, right?).

There are basically four sections that are of interest:

"Http" => Plain HTTP (Similar to Urls but just one host per endpoint)
"HttpsInlineCertFile" => PFX file based (Most likely be used by Windows users that don't want to deal with the cert store)
"HttpsInlineCertAndKeyFile" => .crt+pem file based (Most likely used by Linux users)
"HttpsInlineCertStore" => Windows cert store integration, only works on Windows.

I think the other parts can be ignored.

And I guess what, when you generate a cert yourself, it's a pfx or something? I always forget the differences....

PFX is just one possible certificate format, it is possible to generate all types of self-signed certificates. I believe the majority of business/enterprise users will probably use a cert they get from their IT department from the company-wide PKI and then they just need different options to plug that into ProGet.
Letting the user installing ProGet generate a certificate can be done, but it will show up as insecure (red, broken lock, etc.) in all browsers and also package managers will probably complain about it being untrusted. So I'm not 100% sure that implementing this time well spent, since there are a million guides on the internet how to generate a cert.

How would you suggest we modify that element to support the different options they have?

I would probably switch from the Urls parameter to a list of endpoints. This way the HTTP+HTTPS scenario can be supported as well as the the 4 options above.
The JSON from the example could simply be translated to XML 1:1.

<WebServer Enabled="true/false">
  <Kestrel>
    <Endpoint>
      <HttpsInlineCertStore>
        <Url>https://192.168.0.1:443</Url>
        <Certificate>
          <Subject>my.company.domain.local</Subject>
          <Store>My</Store>
          <Location>LocalMachine</Location
          <AllowInvalid>false</AllowInvalid>
        </Certificate>
      </HttpsInlineCertStore>
    </Endpoint>
  </Kestrel>
</WebServer>

It should be possible, using the Microsoft.Extensions.Configuration.Xml, to read the config file into an IConfiguration object and feed the relevant portion into Kestrel, this is pretty much what we currently do. This way we don't have to reinvent the wheel with HTTPS configuration and can offer all the flexibility that Kestrel offers right out of the box.

new HostBuilder()
            .ConfigureWebHostDefaults(webBuilder => webBuilder
                .UseConfiguration(configuration)
                .UseKestrel((context, serverOptions) =>
                {
                    serverOptions.Configure(context.Configuration.GetSection("Kestrel"));
                })

apxltd

@lm excellent, thank you for the help!

OK - we'll wire these up via PG-2223, which may come in the next or following maintenance release. In theory it's trivial and low risk.

Legacy / HTTP-only Format (still works):
<WebServer Enabled="true/false" Urls="web server listen URLs" />

New / HTTPS-inclused Format:

<WebServer Enabled="true/false">
  <Endpoint 		  
      Url="{http|https}://host.name:1000/">
      CertFile=".pfx, .pem, .crt"
      KeyFile=".key"
      Password=""
      Subject=""
      Store=""
      Location=""
      AllowInvalid=""
     />
  </Endpoint>
</WebServer>

This lets us document an "Endpoint" using simple table of key/value pairs, as well as simplify error messages and the examples that people will just copy/paste:

Plain HTTP
PFX-file based
.crt+pem file based
Windows cert store integration, only works on Windows.

Oh --- and good point about self-signed; if anything we should use that ACME library to get a LetsEncrypt cert or something. At least in theory. A few other products seem to be doing that now as well. No idea how well that would work - but for evaluation purposes I think it's pretty important. Especially if you don't know how to configure HTTPS otherwise.

jw

Sounds like a plan.

Looking forward to trying it out.

jw

I saw this has been implemented a while ago, so thank you for that.

Some feedback:

HTTPS Binding to a Hostname

Essentially worked like a charm.

Maybe a minor thing for the documentation that could help people in the future:

I opened the certificate and blindly copied the first thing that looked like a thumbprint hash (Yea I know, reading is hard...). But in case of windows that is the Serial number and not the thumbprint, which of course totally won't work.

So a little hint the documentation could be helpful, e.g.

cerhash is the thumbprint of you SSL certificate
=>
cerhash is the thumbprint of you SSL certificate NOT the serial number

HTTPS Binding to a Port (Experimental)

Also worked nicely, here also just a minor documentation nitpick:

In the line
"For example, if you wanted to bind to port 4403 for NETWORK SERVICE, you'd run this command:"

The port is probably a typo? At leas the next lines say something different.

Also first it says
netsh http add urlacl url=https://*:443/ user="NETWORK SERVICE"

but then a few lines down it says

<WebServer Enabled="true" Urls="https://*:8625;http://*:8624" />

A couple lines down the 2nd line shows up again and even further down it shows up again with yet different ports.

For this configuration to work properly these settings must match up.

Dan_Woolf

Hi @lm,

Thanks for the notes on our documentation for HTTPS. We are actually working on providing better guidance on our preferred method for configuring SSL with ProGet. Also, we are in the process of implementing a way to configure this in the UI from within ProGet as well. So our documentation for this will be changing in multiple ways, but I'll at least get the typos fixed that you found. Thanks!

Thanks,
Dan

tkolb

I read all the answers here but because I'm not into ASP I'm not sure what I now have to to to make this HTTPS solution working at my site.

Can someone provide instructions on how to do that?

jw

There is updated documentation with step-by-step instructions here:

https://docs.inedo.com/docs/installation-windows-https-support#configuring-https-on-the-integrated-web-server

tkolb

@lm said in HTTPS with self hosted ProGet and internal web server:

There is updated documentation with step-by-step instructions here:

Thank you.

I was confused by the section "IIS & Web Hosting on Windows". I only searched in "ProGet". It should also be mentioned to open the other port as well which should be pretty obvious but who knows.

I now only need someone here to provide a certificate.