HTTP 500 When pushing docker image

wechselberg.nisboerge_3629

Hi,

when pushing one image to the ProGet Server we get a HTTP 500 error.

16:00:45  + docker push vl-dev-repo.ki.lan/sst-coco-oci-prod/sub-coco-cli:test
16:00:45  The push refers to repository [vl-dev-repo.ki.lan/sst-coco-oci-prod/sub-coco-cli]
16:00:45  5476da93a45d: Preparing
16:00:45  94cff3c0f1ca: Preparing
16:00:45  26e17999f588: Preparing
16:00:45  3668ef704aa2: Preparing
16:00:45  8248b221e3e1: Preparing
16:00:45  9e3250a6c86c: Preparing
16:00:45  1d76ae469fc3: Preparing
16:00:45  0168265636e1: Preparing
16:00:45  53cd28cbdad5: Preparing
16:00:45  43d0adb5e44f: Preparing
16:00:45  cca10bba424a: Preparing
16:00:45  d3fd88d75ece: Preparing
16:00:45  d157eb985080: Preparing
16:00:45  974a466e1a0a: Preparing
16:00:45  3819aa7bbe8b: Preparing
16:00:45  4c13ea2c0b02: Preparing
16:00:45  2ed6a19677f5: Preparing
16:00:45  d2f7abddd607: Preparing
16:00:45  53babe930602: Preparing
16:00:45  9e3250a6c86c: Waiting
16:00:45  1d76ae469fc3: Waiting
16:00:45  0168265636e1: Waiting
16:00:45  53cd28cbdad5: Waiting
16:00:45  43d0adb5e44f: Waiting
16:00:45  cca10bba424a: Waiting
16:00:45  2ed6a19677f5: Waiting
16:00:45  d3fd88d75ece: Waiting
16:00:45  d2f7abddd607: Waiting
16:00:45  53babe930602: Waiting
16:00:45  d157eb985080: Waiting
16:00:45  974a466e1a0a: Waiting
16:00:45  3819aa7bbe8b: Waiting
16:00:45  4c13ea2c0b02: Waiting
16:00:45  3668ef704aa2: Layer already exists
16:00:45  26e17999f588: Layer already exists
16:00:45  8248b221e3e1: Layer already exists
16:00:45  1d76ae469fc3: Layer already exists
16:00:45  9e3250a6c86c: Layer already exists
16:00:45  0168265636e1: Layer already exists
16:00:45  53cd28cbdad5: Layer already exists
16:00:45  43d0adb5e44f: Layer already exists
16:00:45  cca10bba424a: Layer already exists
16:00:45  d3fd88d75ece: Layer already exists
16:00:45  d157eb985080: Layer already exists
16:00:45  974a466e1a0a: Layer already exists
16:00:45  94cff3c0f1ca: Pushed
16:00:45  3819aa7bbe8b: Layer already exists
16:00:45  4c13ea2c0b02: Layer already exists
16:00:45  d2f7abddd607: Layer already exists
16:00:45  2ed6a19677f5: Layer already exists
16:00:57  53babe930602: Retrying in 5 seconds
[...]
16:01:20  53babe930602: Retrying in 1 second
16:01:30  5476da93a45d: Pushed
16:01:32  53babe930602: Retrying in 15 seconds
[...]
16:02:13  53babe930602: Retrying in 1 second
16:02:23  received unexpected HTTP status: 500 Internal Server Error

In the container logs we can see:

info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/1.1 PATCH http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/32b375c9-b883-4836-8aa7-49fc1f849ada - 202 0 - 8424.9972ms
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/1.1 PUT http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/32b375c9-b883-4836-8aa7-49fc1f849ada?digest=sha256%3A10af00adc39beb81a045806ad9605cea669cf8deb097de90d0aa1bdc21ea1342 - - 0
A 500 error occurred in sst-coco-oci-prod: Nullable object must have a value.
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/1.1 PUT http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/32b375c9-b883-4836-8aa7-49fc1f849ada?digest=sha256%3A10af00adc39beb81a045806ad9605cea669cf8deb097de90d0aa1bdc21ea1342 - 500 1091 application/json 38.1785ms

We had a similar issue in the past, which seemed to be fixed with version 25.0.10, but it now reappeared.
I updated to 25.0.12 now, but the issue existed in 25.0.10 as well as in 25.0.12.

atripp

Hi @wechselberg-nisboerge_3629 ,

This is definitely a strange error; are you using PostgreSQL by chance?

I'm seeing 53babe930602: Retrying... a few times. Is this consistently happening with this layer? Is there anything special about it (big, small, etc)?

Thanks,
Alana

wechselberg.nisboerge_3629

Hi,

I am using a setup that is pretty much consistent with your quickstart.
Linux Host and ProGet running in Container. I think it is Postgres in there, yes.

Regarding the layer: I will check with the team. (I am just the guy in IT ;) )

Regards,
Nis

EDIT:
So ... this is the whole Dockerfile that is built and uploaded here:

FROM vl-dev-repo.ki.lan/sst-devops-oci-prod/sub-coco-nodejs:latest

WORKDIR /cococli

RUN wget https://vl-jenkins-ctl.ki.lan/job/Coco/job/Coco/job/main/lastSuccessfulBuild/artifact/cococli-linux.tar.gz && \
    tar xzf cococli-linux.tar.gz && \
    rm -rf cococli-linux.tar.gz && \
    ln -sf /cococli/cococli /usr/bin/cococli

The vl-dev-repo is the ProGet and the base image is in ProGet just fine.

atripp

@wechselberg-nisboerge_3629 thanks for comfirming!

Any chance you can get more entries from the container log? It'd be really helpful to see more requests going back/forth. This is just such a strange behavior given the seeming simplicity of your image

Also, there should be an option, in ProGet 2025.12, to enable web logging (Admin > HTTPS Logging); it's a brand new feature, but it writes logs to a log file.

wechselberg.nisboerge_3629

I tried activating the HTTP Request logging, but I get another HTTP 500 here ...

Running Execution Dispatcher...
Checking for executions to dispatch...
Execution Dispatcher completed.
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/1.1 POST http://vl-dev-repo.ki.lan/administration/web-server/http-settings/configure - application/x-www-form-urlencoded 328
Attempting to update shared configuration for ProGet.
Attempting to use shared configuration for ProGet.
Shared configuration file not found at /etc/inedo/ProGet.config.
An error occurred in the web application: Cannot load shared configuration for ProGet.
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/1.1 POST http://vl-dev-repo.ki.lan/administration/web-server/http-settings/configure - 500 2521 text/html;+charset=UTF-8 43.0276ms

(Obviously out of scope/unrelated here, but wanted to let you know about that)

What kind of logs are helpful from the container log?

atripp

@wechselberg-nisboerge_3629 the main thing I'm looking for is the HTTP access logs - we have 1.5 Entires before (PATCH-finish, PUT-Start, PUT-finish), so seeing more would be really helpful.

What's odd is seeing the "retrying..."