Welcome to the Inedo Forums! Check out the Forums Guide for help getting started.
If you are experiencing any issues with the forum software, please visit the Contact Form on our website and let us know!
HTTP 500 When pushing docker image
-
Hi,
when pushing one image to the ProGet Server we get a HTTP 500 error.
16:00:45 + docker push vl-dev-repo.ki.lan/sst-coco-oci-prod/sub-coco-cli:test 16:00:45 The push refers to repository [vl-dev-repo.ki.lan/sst-coco-oci-prod/sub-coco-cli] 16:00:45 5476da93a45d: Preparing 16:00:45 94cff3c0f1ca: Preparing 16:00:45 26e17999f588: Preparing 16:00:45 3668ef704aa2: Preparing 16:00:45 8248b221e3e1: Preparing 16:00:45 9e3250a6c86c: Preparing 16:00:45 1d76ae469fc3: Preparing 16:00:45 0168265636e1: Preparing 16:00:45 53cd28cbdad5: Preparing 16:00:45 43d0adb5e44f: Preparing 16:00:45 cca10bba424a: Preparing 16:00:45 d3fd88d75ece: Preparing 16:00:45 d157eb985080: Preparing 16:00:45 974a466e1a0a: Preparing 16:00:45 3819aa7bbe8b: Preparing 16:00:45 4c13ea2c0b02: Preparing 16:00:45 2ed6a19677f5: Preparing 16:00:45 d2f7abddd607: Preparing 16:00:45 53babe930602: Preparing 16:00:45 9e3250a6c86c: Waiting 16:00:45 1d76ae469fc3: Waiting 16:00:45 0168265636e1: Waiting 16:00:45 53cd28cbdad5: Waiting 16:00:45 43d0adb5e44f: Waiting 16:00:45 cca10bba424a: Waiting 16:00:45 2ed6a19677f5: Waiting 16:00:45 d3fd88d75ece: Waiting 16:00:45 d2f7abddd607: Waiting 16:00:45 53babe930602: Waiting 16:00:45 d157eb985080: Waiting 16:00:45 974a466e1a0a: Waiting 16:00:45 3819aa7bbe8b: Waiting 16:00:45 4c13ea2c0b02: Waiting 16:00:45 3668ef704aa2: Layer already exists 16:00:45 26e17999f588: Layer already exists 16:00:45 8248b221e3e1: Layer already exists 16:00:45 1d76ae469fc3: Layer already exists 16:00:45 9e3250a6c86c: Layer already exists 16:00:45 0168265636e1: Layer already exists 16:00:45 53cd28cbdad5: Layer already exists 16:00:45 43d0adb5e44f: Layer already exists 16:00:45 cca10bba424a: Layer already exists 16:00:45 d3fd88d75ece: Layer already exists 16:00:45 d157eb985080: Layer already exists 16:00:45 974a466e1a0a: Layer already exists 16:00:45 94cff3c0f1ca: Pushed 16:00:45 3819aa7bbe8b: Layer already exists 16:00:45 4c13ea2c0b02: Layer already exists 16:00:45 d2f7abddd607: Layer already exists 16:00:45 2ed6a19677f5: Layer already exists 16:00:57 53babe930602: Retrying in 5 seconds [...] 16:01:20 53babe930602: Retrying in 1 second 16:01:30 5476da93a45d: Pushed 16:01:32 53babe930602: Retrying in 15 seconds [...] 16:02:13 53babe930602: Retrying in 1 second 16:02:23 received unexpected HTTP status: 500 Internal Server Error
In the container logs we can see:
info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 PATCH http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/32b375c9-b883-4836-8aa7-49fc1f849ada - 202 0 - 8424.9972ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 PUT http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/32b375c9-b883-4836-8aa7-49fc1f849ada?digest=sha256%3A10af00adc39beb81a045806ad9605cea669cf8deb097de90d0aa1bdc21ea1342 - - 0 A 500 error occurred in sst-coco-oci-prod: Nullable object must have a value. info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 PUT http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/32b375c9-b883-4836-8aa7-49fc1f849ada?digest=sha256%3A10af00adc39beb81a045806ad9605cea669cf8deb097de90d0aa1bdc21ea1342 - 500 1091 application/json 38.1785ms
We had a similar issue in the past, which seemed to be fixed with version 25.0.10, but it now reappeared.
I updated to 25.0.12 now, but the issue existed in 25.0.10 as well as in 25.0.12.
-
Hi @wechselberg-nisboerge_3629 ,
This is definitely a strange error; are you using PostgreSQL by chance?
I'm seeing
53babe930602: Retrying...
a few times. Is this consistently happening with this layer? Is there anything special about it (big, small, etc)?Thanks,
Alana
-
Hi,
I am using a setup that is pretty much consistent with your quickstart.
Linux Host and ProGet running in Container. I think it is Postgres in there, yes.Regarding the layer: I will check with the team. (I am just the guy in IT ;) )
Regards,
NisEDIT:
So ... this is the whole Dockerfile that is built and uploaded here:FROM vl-dev-repo.ki.lan/sst-devops-oci-prod/sub-coco-nodejs:latest WORKDIR /cococli RUN wget https://vl-jenkins-ctl.ki.lan/job/Coco/job/Coco/job/main/lastSuccessfulBuild/artifact/cococli-linux.tar.gz && \ tar xzf cococli-linux.tar.gz && \ rm -rf cococli-linux.tar.gz && \ ln -sf /cococli/cococli /usr/bin/cococli
The vl-dev-repo is the ProGet and the base image is in ProGet just fine.
-
@wechselberg-nisboerge_3629 thanks for comfirming!
Any chance you can get more entries from the container log? It'd be really helpful to see more requests going back/forth. This is just such a strange behavior given the seeming simplicity of your image
Also, there should be an option, in ProGet 2025.12, to enable web logging (Admin > HTTPS Logging); it's a brand new feature, but it writes logs to a log file.
-
I tried activating the HTTP Request logging, but I get another HTTP 500 here ...
Running Execution Dispatcher... Checking for executions to dispatch... Execution Dispatcher completed. info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 POST http://vl-dev-repo.ki.lan/administration/web-server/http-settings/configure - application/x-www-form-urlencoded 328 Attempting to update shared configuration for ProGet. Attempting to use shared configuration for ProGet. Shared configuration file not found at /etc/inedo/ProGet.config. An error occurred in the web application: Cannot load shared configuration for ProGet. info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 POST http://vl-dev-repo.ki.lan/administration/web-server/http-settings/configure - 500 2521 text/html;+charset=UTF-8 43.0276ms
(Obviously out of scope/unrelated here, but wanted to let you know about that)
What kind of logs are helpful from the container log?
-
@wechselberg-nisboerge_3629 the main thing I'm looking for is the HTTP access logs - we have 1.5 Entires before (PATCH-finish, PUT-Start, PUT-finish), so seeing more would be really helpful.
What's odd is seeing the "retrying..."
-
There you go ... this should be the complete log of pushing that image.
Execution Dispatcher completed. info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 GET http://vl-dev-repo.ki.lan/v2/ - - - info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 GET http://vl-dev-repo.ki.lan/v2/ - 401 145 application/json 2.4781ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 GET http://vl-dev-repo.ki.lan/v2/_auth?account=deployment&client_id=docker&offline_token=true&service=vl-dev-repo.ki.lan - - - info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 GET http://vl-dev-repo.ki.lan/v2/_auth?account=deployment&client_id=docker&offline_token=true&service=vl-dev-repo.ki.lan - 200 369 application/json 32.8540ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 GET http://vl-dev-repo.ki.lan/v2/ - - - info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 GET http://vl-dev-repo.ki.lan/v2/ - 200 47 text/plain 3.3824ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 GET http://vl-dev-repo.ki.lan/v2/ - - - info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 GET http://vl-dev-repo.ki.lan/v2/ - 401 145 application/json 0.7435ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 GET http://vl-dev-repo.ki.lan/v2/_auth?account=deployment&scope=repository%3Asst-coco-oci-prod%2Fsub-coco-cli%3Apush%2Cpull&service=vl-dev-repo.ki.lan - - - info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 GET http://vl-dev-repo.ki.lan/v2/_auth?account=deployment&scope=repository%3Asst-coco-oci-prod%2Fsub-coco-cli%3Apush%2Cpull&service=vl-dev-repo.ki.lan - 200 369 application/json 34.7176ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 POST http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/ - - 0 info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 POST http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/ - - 0 info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 POST http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/ - - 0 info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 POST http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/ - - 0 info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 POST http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/ - - 0 info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 POST http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/ - 202 0 - 22.3339ms info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 POST http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/ - 202 0 - 20.6079ms info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 POST http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/ - 202 0 - 31.6916ms info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 POST http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/ - 202 0 - 20.2714ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 PATCH http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/ed7f4055-d456-4767-a568-8bcb9e4dae22 - - - info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 PATCH http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/73fb2122-6cdd-4de1-a22e-a3fb26ba98e8 - - - info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 PATCH http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/f81b6fbd-ad19-4961-87b0-a222f8d65577 - - - info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 POST http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/ - 202 0 - 53.3528ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 PATCH http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/645157d8-8c16-402f-a968-c2778b39a6ed - - - info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 PATCH http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/f81b6fbd-ad19-4961-87b0-a222f8d65577 - 202 0 - 11.0440ms info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 PATCH http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/645157d8-8c16-402f-a968-c2778b39a6ed - 202 0 - 10.1261ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 PATCH http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/7c2c3611-2a39-4f48-8e8a-7bdf09899bae - - - info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 PATCH http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/7c2c3611-2a39-4f48-8e8a-7bdf09899bae - 202 0 - 8.5752ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 PUT http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/645157d8-8c16-402f-a968-c2778b39a6ed?digest=sha256%3Acb8670ad004ff5f0321e07da367e229a4808cff21969b0dd8c9447c32931c05e - - 0 info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 PUT http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/f81b6fbd-ad19-4961-87b0-a222f8d65577?digest=sha256%3Ac10a323c75f48a3f7f63938c250c034414d4fbf111a68c52197b9026821f46fd - - 0 info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 PUT http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/7c2c3611-2a39-4f48-8e8a-7bdf09899bae?digest=sha256%3A5a16507c304ed7ba2f1dec16bc412897126c6fc69ecbcee5fb8ed8457e0bddfd - - 0 Attempting to open blob cb8670ad004f (id=10128)... Successfully opened blob cb8670ad004f (id=10128) as .tar.gz file. Starting scan... Attempting to open blob c10a323c75f4 (id=10129)... Successfully opened blob c10a323c75f4 (id=10129) as .tar.gz file. Starting scan... Attempting to open blob 5a16507c304e (id=10130)... Successfully opened blob 5a16507c304e (id=10130) as .tar.gz file. Starting scan... info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 PUT http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/7c2c3611-2a39-4f48-8e8a-7bdf09899bae?digest=sha256%3A5a16507c304ed7ba2f1dec16bc412897126c6fc69ecbcee5fb8ed8457e0bddfd - 201 0 - 60.3794ms info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 PUT http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/645157d8-8c16-402f-a968-c2778b39a6ed?digest=sha256%3Acb8670ad004ff5f0321e07da367e229a4808cff21969b0dd8c9447c32931c05e - 201 0 - 75.1990ms info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 PUT http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/f81b6fbd-ad19-4961-87b0-a222f8d65577?digest=sha256%3Ac10a323c75f48a3f7f63938c250c034414d4fbf111a68c52197b9026821f46fd - 201 0 - 66.9033ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:5a16507c304ed7ba2f1dec16bc412897126c6fc69ecbcee5fb8ed8457e0bddfd - - - info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:c10a323c75f48a3f7f63938c250c034414d4fbf111a68c52197b9026821f46fd - - - The blob at http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:5a16507c304ed7ba2f1dec16bc412897126c6fc69ecbcee5fb8ed8457e0bddfd does not have a media type associated with it. This may be fixed by the feed cleanup task for sst-coco-oci-prod. The blob at http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:c10a323c75f48a3f7f63938c250c034414d4fbf111a68c52197b9026821f46fd does not have a media type associated with it. This may be fixed by the feed cleanup task for sst-coco-oci-prod. info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:5a16507c304ed7ba2f1dec16bc412897126c6fc69ecbcee5fb8ed8457e0bddfd - 200 265 application/octet-stream 22.6717ms info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:c10a323c75f48a3f7f63938c250c034414d4fbf111a68c52197b9026821f46fd - 200 205 application/octet-stream 7.3281ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:cb8670ad004ff5f0321e07da367e229a4808cff21969b0dd8c9447c32931c05e - - - The blob at http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:cb8670ad004ff5f0321e07da367e229a4808cff21969b0dd8c9447c32931c05e does not have a media type associated with it. This may be fixed by the feed cleanup task for sst-coco-oci-prod. info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:cb8670ad004ff5f0321e07da367e229a4808cff21969b0dd8c9447c32931c05e - 200 96 application/octet-stream 4.9245ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:a81f7bd97d92549642734fdf878f3ecc7b89ecd72aae22204f6c24c9fd083704 - - - info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:a81f7bd97d92549642734fdf878f3ecc7b89ecd72aae22204f6c24c9fd083704 - 200 10000654 application/vnd.docker.image.rootfs.diff.tar.gzip 5.1725ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:54b03df36d446a1163f86210f518ca113c233a084cfe8f5a9f3a9c8ad566c8cb - - - info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:bfa3d3e4cf7f9dd27246d40ceed05dc4d28bfeeb80324ea51bac2d401d5a13c4 - - - info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:54b03df36d446a1163f86210f518ca113c233a084cfe8f5a9f3a9c8ad566c8cb - 200 3430 application/vnd.docker.image.rootfs.diff.tar.gzip 5.5336ms info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:bfa3d3e4cf7f9dd27246d40ceed05dc4d28bfeeb80324ea51bac2d401d5a13c4 - 200 10372462 application/vnd.docker.image.rootfs.diff.tar.gzip 6.9847ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:e332af79be02415dcaf296fb7e257c3e43d851a3f51cd52f78a770d10e103cd5 - - - info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:e332af79be02415dcaf296fb7e257c3e43d851a3f51cd52f78a770d10e103cd5 - 200 15175983 application/vnd.docker.image.rootfs.diff.tar.gzip 6.8531ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:9157c33488042e7d75b36a4eb15ae02627cacd714c4c7212259c8c1352561121 - - - info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:0649e381bf819b1e0f8e86dfbc29c873931462b256cc6414b27c8d67fb4699cb - - - info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:9157c33488042e7d75b36a4eb15ae02627cacd714c4c7212259c8c1352561121 - 200 127149 application/vnd.docker.image.rootfs.diff.tar.gzip 5.0431ms info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:0649e381bf819b1e0f8e86dfbc29c873931462b256cc6414b27c8d67fb4699cb - 200 2071 application/vnd.docker.image.rootfs.diff.tar.gzip 5.5343ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:14cd97ab0673a50082bf7f2d053208a526479b6e0c13549d808e815d61ba9700 - - - info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:14cd97ab0673a50082bf7f2d053208a526479b6e0c13549d808e815d61ba9700 - 200 1771 application/vnd.docker.image.rootfs.diff.tar.gzip 5.6797ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:7337869f5d805b82847d243a3197299987bad8712bf1265d420e26d60862b859 - - - info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:788f4e8602eea4b058237b3c20c834b3997f2fac3c573666c2a0c384f2e99419 - - - info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:7337869f5d805b82847d243a3197299987bad8712bf1265d420e26d60862b859 - 200 1686 application/vnd.docker.image.rootfs.diff.tar.gzip 7.0375ms info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:788f4e8602eea4b058237b3c20c834b3997f2fac3c573666c2a0c384f2e99419 - 200 260 application/vnd.docker.image.rootfs.diff.tar.gzip 6.3954ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:775a325bc2aa2aa75e855af154839b397d850e14c927204905e3f46f718ffb58 - - - info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:775a325bc2aa2aa75e855af154839b397d850e14c927204905e3f46f718ffb58 - 200 446 application/vnd.docker.image.rootfs.diff.tar.gzip 7.6769ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:00f4ac723d07c76a3aa3f6744387791188f5d413214d80addff94a5732b559b9 - - - info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:00f4ac723d07c76a3aa3f6744387791188f5d413214d80addff94a5732b559b9 - 200 1250736 application/vnd.docker.image.rootfs.diff.tar.gzip 4.9155ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:8dcb664baf87e9bb932e2663beb0221ce0375a0b2d9b59c0ca8d771801e3aafb - - - info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:8dcb664baf87e9bb932e2663beb0221ce0375a0b2d9b59c0ca8d771801e3aafb - 200 55780645 application/vnd.docker.image.rootfs.diff.tar.gzip 6.2385ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:28432cc975ed52ac2085f9db91a57f9696424f6cff7703086464fd1a8220b63e - - - info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:353e14e5cc47664fba714a7da288001d90427c705494847ac773f5cc08199451 - - - info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:28432cc975ed52ac2085f9db91a57f9696424f6cff7703086464fd1a8220b63e - 200 3326 application/vnd.docker.image.rootfs.diff.tar.gzip 5.2463ms info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:353e14e5cc47664fba714a7da288001d90427c705494847ac773f5cc08199451 - 200 211352566 application/vnd.docker.image.rootfs.diff.tar.gzip 4.4519ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:255774e0027b72d2327719e78dbad5ad8c9cf446d055e45be7fc149418470bae - - - info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:255774e0027b72d2327719e78dbad5ad8c9cf446d055e45be7fc149418470bae - 200 64396484 application/vnd.docker.image.rootfs.diff.tar.gzip 5.6129ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:7cd785773db44407e20a679ce5439222e505475eed5b99f1910eb2cda51729ab - - - info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:091eb8249475f42de217265c501e0186f0a3ea7490ef7f51458c30db91fb3cac - - - info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:7cd785773db44407e20a679ce5439222e505475eed5b99f1910eb2cda51729ab - 404 158 application/json 4.9299ms info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:091eb8249475f42de217265c501e0186f0a3ea7490ef7f51458c30db91fb3cac - 200 24011136 application/vnd.docker.image.rootfs.diff.tar.gzip 4.5069ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 POST http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/ - - 0 info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 POST http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/ - 202 0 - 6.9731ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 PATCH http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/bc45b2de-d284-44d7-9707-197c69acc199 - - - Running Multipart Upload Cleanup... Multipart Upload Cleanup completed. Running Execution Dispatcher... Checking for executions to dispatch... Execution Dispatcher completed. Running Connector Cache Check... Getting list of connector responses... Connector Cache Check completed. Running Docker Upload Cleanup... Docker Upload Cleanup completed. info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 PATCH http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/bc45b2de-d284-44d7-9707-197c69acc199 - 202 0 - 11030.2968ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 PUT http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/bc45b2de-d284-44d7-9707-197c69acc199?digest=sha256%3A10af00adc39beb81a045806ad9605cea669cf8deb097de90d0aa1bdc21ea1342 - - 0 A 500 error occurred in sst-coco-oci-prod: Nullable object must have a value. info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 PUT http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/bc45b2de-d284-44d7-9707-197c69acc199?digest=sha256%3A10af00adc39beb81a045806ad9605cea669cf8deb097de90d0aa1bdc21ea1342 - 500 1054 application/json 28.3463ms Running Node Message Cleanup... Purging processed node messages and messages older than 10/20/2025 11:14:51... No messages to purge. Node Message Cleanup completed. info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 POST http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/ - - 0 info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 POST http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/ - 202 0 - 12.3959ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 PATCH http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/85b1b98a-4fe4-4b96-b94e-c5601f0479cf - - - Running Feed Replication... Feed Replication completed. Running Drop Path Monitor... Getting list of feeds with a drop path... No feeds with a drop path. Drop Path Monitor completed. Running Execution Dispatcher... Checking for executions to dispatch... Execution Dispatcher completed. info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 PATCH http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/73fb2122-6cdd-4de1-a22e-a3fb26ba98e8 - 202 0 - 24188.0247ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 PUT http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/73fb2122-6cdd-4de1-a22e-a3fb26ba98e8?digest=sha256%3Af4da01d28713fb9c6cca713ab630ad93131d2b25b47d954b0b13e49c1837cee4 - - 0 Created execution (id=6144) to scan uploaded blob (id=10131). info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 PUT http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/73fb2122-6cdd-4de1-a22e-a3fb26ba98e8?digest=sha256%3Af4da01d28713fb9c6cca713ab630ad93131d2b25b47d954b0b13e49c1837cee4 - 201 0 - 32.2901ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:f4da01d28713fb9c6cca713ab630ad93131d2b25b47d954b0b13e49c1837cee4 - - - The blob at http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:f4da01d28713fb9c6cca713ab630ad93131d2b25b47d954b0b13e49c1837cee4 does not have a media type associated with it. This may be fixed by the feed cleanup task for sst-coco-oci-prod. info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:f4da01d28713fb9c6cca713ab630ad93131d2b25b47d954b0b13e49c1837cee4 - 200 320734315 application/octet-stream 6.2105ms Running Service Heartbeat... Service Heartbeat completed. info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 PATCH http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/85b1b98a-4fe4-4b96-b94e-c5601f0479cf - 202 0 - 10991.9175ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 PUT http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/85b1b98a-4fe4-4b96-b94e-c5601f0479cf?digest=sha256%3A10af00adc39beb81a045806ad9605cea669cf8deb097de90d0aa1bdc21ea1342 - - 0 A 500 error occurred in sst-coco-oci-prod: Nullable object must have a value. info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 PUT http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/85b1b98a-4fe4-4b96-b94e-c5601f0479cf?digest=sha256%3A10af00adc39beb81a045806ad9605cea669cf8deb097de90d0aa1bdc21ea1342 - 500 1054 application/json 47.9726ms Running Execution Dispatcher... Checking for executions to dispatch... Execution Dispatcher completed. info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 POST http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/ - - 0 info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 POST http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/ - 202 0 - 14.0346ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 PATCH http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/ce08b054-2649-4aa6-84e9-39bf25f44de5 - - - info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 PATCH http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/ed7f4055-d456-4767-a568-8bcb9e4dae22 - 202 0 - 46200.1078ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 PUT http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/ed7f4055-d456-4767-a568-8bcb9e4dae22?digest=sha256%3Adb022c43a487cf61b704c7ae6bde238cc12230ce863706b38c1c075d0ad33f1f - - 0 Created execution (id=6145) to scan uploaded blob (id=10132). info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 PUT http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/ed7f4055-d456-4767-a568-8bcb9e4dae22?digest=sha256%3Adb022c43a487cf61b704c7ae6bde238cc12230ce863706b38c1c075d0ad33f1f - 201 0 - 26.2197ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:db022c43a487cf61b704c7ae6bde238cc12230ce863706b38c1c075d0ad33f1f - - - The blob at http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:db022c43a487cf61b704c7ae6bde238cc12230ce863706b38c1c075d0ad33f1f does not have a media type associated with it. This may be fixed by the feed cleanup task for sst-coco-oci-prod. info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 HEAD http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/sha256:db022c43a487cf61b704c7ae6bde238cc12230ce863706b38c1c075d0ad33f1f - 200 322376036 application/octet-stream 7.3476ms info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 PATCH http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/ce08b054-2649-4aa6-84e9-39bf25f44de5 - 202 0 - 10843.0526ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 PUT http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/ce08b054-2649-4aa6-84e9-39bf25f44de5?digest=sha256%3A10af00adc39beb81a045806ad9605cea669cf8deb097de90d0aa1bdc21ea1342 - - 0 A 500 error occurred in sst-coco-oci-prod: Nullable object must have a value. info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 PUT http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/ce08b054-2649-4aa6-84e9-39bf25f44de5?digest=sha256%3A10af00adc39beb81a045806ad9605cea669cf8deb097de90d0aa1bdc21ea1342 - 500 1054 application/json 41.3446ms Running Execution Dispatcher... Checking for executions to dispatch... Execution Dispatcher completed. info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 POST http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/ - - 0 info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 POST http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/ - 202 0 - 16.6565ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 PATCH http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/cdf84034-5bb6-460a-b3f9-1a904adb6482 - - - Running Execution Dispatcher... Checking for executions to dispatch... Execution Dispatcher completed. info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 PATCH http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/cdf84034-5bb6-460a-b3f9-1a904adb6482 - 202 0 - 10810.0643ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 PUT http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/cdf84034-5bb6-460a-b3f9-1a904adb6482?digest=sha256%3A10af00adc39beb81a045806ad9605cea669cf8deb097de90d0aa1bdc21ea1342 - - 0 A 500 error occurred in sst-coco-oci-prod: Nullable object must have a value. info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 PUT http://vl-dev-repo.ki.lan/v2/sst-coco-oci-prod/sub-coco-cli/blobs/uploads/cdf84034-5bb6-460a-b3f9-1a904adb6482?digest=sha256%3A10af00adc39beb81a045806ad9605cea669cf8deb097de90d0aa1bdc21ea1342 - 500 1054 application/json 38.5128ms Running Feed Replication... Feed Replication completed. Running Drop Path Monitor... Getting list of feeds with a drop path... No feeds with a drop path. Drop Path Monitor completed.