ProGet Docker built in database fails after restart

[tames_0545]

Hello,

I set up ProGet using Docker and nginx as a reverse proxy. When I start the container, everything works. But if I remove the container and start it again, it complains that it cannot connect to the database.

This is a clean install using the latest version of the container image.

# docker logs -f proget
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /usr/local/proget
ProGet version is 25.0.5.16.
Current DB schema version is unknown.
Cannot connect to database; will retry in 1 second... Full error: Failed to connect to 127.0.0.1:5728
Attempting to use shared configuration for ProGet.
Shared configuration file not found at /etc/inedo/ProGet.config.
info: Inedo.Web.BackgroundTaskQueueService[0]
      Background Task Queue is starting.
warn: Microsoft.AspNetCore.Hosting.Diagnostics[15]
      Overriding HTTP_PORTS '8080' and HTTPS_PORTS ''. Binding to values defined by URLS instead 'http://*:80'.
warn: Microsoft.AspNetCore.Server.Kestrel[0]
      Overriding address(es) 'http://*:80'. Binding to endpoints defined via IConfiguration and/or UseKestrel() instead.
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /usr/local/proget
info: Microsoft.Hosting.Lifetime[14]
      Now listening on: http://[::]:80
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /usr/local/proget
Cannot connect to database; will retry in 1 second... Full error: Failed to connect to 127.0.0.1:5728
Cannot connect to database; will retry in 1 second... Full error: Failed to connect to 127.0.0.1:5728
Cannot connect to database; will retry in 1 second... Full error: Failed to connect to 127.0.0.1:5728
Cannot connect to database; will retry in 1 second... Full error: Failed to connect to 127.0.0.1:5728
Cannot connect to database; will retry in 1 second... Full error: Failed to connect to 127.0.0.1:5728

Here is my start command:

#!/bin/bash

/usr/bin/docker run -d --name=proget --restart=unless-stopped \
        -v /opt/proget/packages:/var/proget/packages \
        -v /opt/proget/database:/var/proget/database \
        -v /opt/proget/backups:/var/proget/backups \
        -v /opt/proget/encryption.key:/var/proget/encryption.key \
        -v /opt/proget/connection_string.txt:/var/proget/connection_string.txt \
        -p 8624:80 \
        -e TZ='Asia/Qatar' \
        -e PROGET_ENCRYPTION_KEY_FILE='/var/proget/encryption.key' \
        -e PROGET_POSTGRES_CONNECTION_STRING_FILE='/var/proget/connection_string.txt' \
        proget.inedo.com/productimages/inedo/proget:latest

From the instructions it seems you only need the "PROGET_POSTGRES_CONNECTION_STRING_FILE" if you are using an external database, but the web interface complains about not having the connection string if I don't put it.

Am I missing something?

[tames_0545]

I removed the connection string to try again. This time things are fine if I stop, remove, and run again. However on reboot, it goes back to not working again.

The web interface shows the following:

[atripp]

Hi @tames_0545 ,

A connection string should not be specified.

I haven't seen this behavior before; the container runs postgres on start to listen to port 5728 and I can't imagine how that would fail without an error message.

Perhaps there's some kind of issue with /opt/proget/database and that's causing the postgres process to crash; I would start here.

The only other thing I can imagine is some kind of security configuration that is interfering with the running container.

I would keep it as simple as possible. Once you get it working, you can try adding in the encryption key. Unless you absolutely need to use a Docker secret (those are complex and don't really do anything), just specify the encryption key with an environment variable

-e PROGET_ENCRYPTION_KEY='37D27A670394F7D82CE57F1F07D69747'

Cheers,
Alana

[tames_0545]

I have removed the encryption key and connection string. I removed the proget directories and started again.

Things ran well until reboot. After reboot, the web was available but it started throwing the following message:

info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/1.1 GET http://proget.example.com/feeds - - -
fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HNE94OV39U9F", Request id "0HNE94OV39U9F:00000001": An unhandled exception was thrown by the application.
      Inedo.ProGet.Web.Security.UserNotFoundException: Exception of type 'Inedo.ProGet.Web.Security.UserNotFoundException' was thrown.
         at Inedo.ProGet.WebApplication.ProGetHttpModule.AuthorizeRequestAsync(AhHttpApplication app) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E576295\Src\src\ProGet\WebApplication\ProGetHttpModule.cs:line 264
         at Inedo.Web.InedoHttpModule.Inedo.Web.IAhWebModule.AuthorizeRequestAsync(AhHttpApplication app)
         at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context)
         at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context)
         at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context)
         at Inedo.Web.AhWebHost.<>c.<<Configure>b__22_0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/1.1 GET http://proget.example.com/feeds - 500 0 - 3.7097ms

Eventually the web server stopped responding.

It seems completely unrelated to what I was seeing before. However, it shows something is not stable.

[atripp]

Hi @tames_0545 ,

Sorry, no idea -- there is definitely something wrong with your environment but aside from the advice I gave earlier, I don't have any other troubleshooting steps. You may want to try a different environment / Docker host.

As I mentioned, it could something wrong with your database mount (/opt/proget/database) -- maybe that is getting deleted? Or it could be your security configuration that is interfering with the running container. We have seen some hyper-aggressive security tools that make things so secure nothing runs.

Thanks,
Alana

[atripp]

@tames_0545 I just thought of something, perhaps try a different versions of ProGet, instead of latest -- like maybe 25.0.3, 25.0.4, etc, perhaps into ProGet 2024 as well

[tames_0545]

I figured it out. The container is changing the /var/proget/database directory ownership from root to the internal postgres user. I had created the directories using Puppet, which was putting them back to root as the owner.

Thanks for the hints.

[dean-houston]

@tames_0545 thanks for letting us know! We'll see if we can improve troubleshooting on this in the future as well!