Forums

scusson_9923

Hi,

Just to update this issue. I ended up having to remove, clean up, and reinstall IIS\ProGet. After that, the default ProGet instance loaded successfully. Thanks for all of your input.

Scott

scusson_9923

Hi Alana,

Yes. Local connection was successful. I am in the process of setting up 2 web instances. I was able to resolve this issue after I noticed that a couple of SQL Server services on the DB server were installed as disabled by default (browser\agent). However, I am having issues when setting up the second instance. When I try to load the ProGet website, I get a "HTTP Error 503. The service is unavailable." message in the browser and in the Event Viewer I see: "The Module DLL C:\Windows\system32\inetsrv\aspnetcore.dll failed to load. The data is the error." This machine may be in a bad state as this seems to be a red herring. I'm tempted to have this machine rebuilt unless you have additional pointers :)

Thanks!
Scott

scusson_9923

Hi,

I am trying to connect a local 2022.11 IIS ProGet instance to a remote MSSQL express ProGet DB. The remote ProGet DB, local website, App Pool, and service are all running as the same user. That user is also the owner of the ProGet DB. I can successfully connect to the remote ProGet DB with SS Management Studio. I use the connection string 'Data Source=<DB_Server>\Inedo;Initial Catalog=ProGet;Integrated Security=True' when trying to load the local ProGet website and I get the error 'HTTP Error 500.30 - ASP.NET Core app failed to start' with no entries in the Event Viewer.

Thoughts?

Thanks,
Scott

scusson_9923

Thank you. I will do that.

scusson_9923

Hi,

I reset the cache on all nodes and updated to 2022.8 (planned update). I have a user that continues to exhibit this behavior. Is there something he can try on his end?

Thanks,
Scott

scusson_9923

Hi,

Thank for you for the quick reply! I'll try that.

Scott

scusson_9923

@NanciCalo said in User with permissions Publish permissions is denied:

"Invalidate Cache button" as something for us to try to get in :)

Hi,

We recently started getting this and the "Clear Cache" option does not seem to resolve our issue.  Any other options (other than setting Web.PrivilegeCacheExpiration)?

Thanks,
Scott

scusson_9923

Good news! Thanks!

scusson_9923

Hi,
We are looking to enable SAML in the near future and wondering where this issue stands? We need to keep our apikey access unchanged and allow for local logins.

Thanks,
Scott

scusson_9923

@rhessinger Hi Rich,

Works perfectly now.

Thanks!

Scott

scusson_9923

@scusson_9923Hi Rich,

if I just use -X I get the return:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">

<HTML><HEAD><TITLE>Length Required</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Length Required</h2>
<hr><p>HTTP Error 411. The request must be chunked or have a content length.</p>
</BODY></HTML>

We plan on adding this in an automated build scenario, so there is a good chance we'll be using this regularly.

Thanks,
Scott

scusson_9923

@rhessinger Hi Rich,

Thanks! Adding the /dir/ worked. I still get the "curl: (6) Could not resolve host: POST" error on Windows. Linux is "curl: (6) Could not resolve host: POST; Unknown error". Can your online documentation be updated with the full command?

curl -d -X POST https://<ProGet_server>/endpoints/<asset_dir>/dir/<new_dir>/ --user <usename>:<password>

Thanks again,
Scott

scusson_9923

Hi,

A little more info. Looking at the IIS log, I see:

2021-12-03 19:39:41 <ProGet_ClusterIP>7 POST /endpoints/<asset_dir>/<new_dir>/ - 443 - <Machine_IP>> curl/7.55.1 - 200 0 0 78

So we know it hits the ProGet server and returns success from IIS. Just very confused on "curl: (6) Could not resolve host: POST" from curl.

Thanks,
Scott

scusson_9923

@Dan_Woolf Hi Dan,

When I add -X, I get:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">
<HTML><HEAD><TITLE>Length Required</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Length Required</h2>
<hr><p>HTTP Error 411. The request must be chunked or have a content length.</p>
</BODY></HTML>

when I use -d -X, I get:

curl: (6) Could not resolve host: POST

If it helps, our environment is load balanced with 3 web nodes.

Thanks,
Scott

scusson_9923

Hi,

I am trying to follow the section "Create Directory Endpoint" on https://docs.inedo.com/docs/proget-reference-api-asset-directories-api and I am unable to create a subfolder with the following command:

curl POST https://<ProGet_server>/endpoints/<asset_dir>/<new_dir>/ --user <user>:<password>

I get "curl: (6) Could not resolve host: POST"

Can the online documentation be updated to give a full example instead of "POST .../dir/«path»"

Thanks,
Scott

scusson_9923

Hi Rich,

I created an API key with full access and still get wget: server returned error: HTTP/1.1 403 Forbidden.

When I place the same link in my local browser, I get:
The specified API key does not permit access to the Docker Blob Reader API.

In the Access Key Log:
/api/docker-blobs/download/sha256:f033c4f65cdbf0bfa21d5543e56c0c41645eca4d893494bb4f0661b0f19ccc79?API_Key=<apikey>
Response code: 403

As a test, I used wget to download a file from an Assets directory inside the Clair container successfully.

Thanks,
Scott

scusson_9923

Hi,

I get 'wget: server returned error: HTTP/1.1 403 Forbidden'

When I try curl with a ProGet user\password, I get 'An API key is required to download Docker blobs.'

I saw the API key in the previous logs (container without -insecure-tls), but not now.

Thanks,
Scott

scusson_9923

Baby steps! I still see no entries in the diagnostics center regarding vulnerabilities.

Thanks,
Scott

scusson_9923

Hi Rich,

Still is failing, different reason:

{"Event":"could not download layer: expected 2XX","Level":"warning","Location":"driver.go:136","Time":"2021-09-20 19:01:26.736879","status code":404}
{"Event":"failed to extract data from path","Level":"error","Location":"worker.go:122","Time":"2021-09-20 19:01:26.737319","error":"could not find layer","layer":"sha256:f033c4f65cdbf0bfa21d5543e56c0c41645eca4d893494bb4f0661b0f19ccc79","path":"https://<proget_server>/api/docker-blobs/download/sha256%3Af033c4f65cdbf0bfa21d5543e56c0c41645eca4d893494bb4f0661b0f19ccc79"}
{"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2021-09-20 19:01:26.737385","elapsed time":166008805,"method":"POST","remote addr":"<proget_web_node>:59778","request uri":"/v1/layers","status":"400"}

Thanks,
Scott

scusson_9923

Hi Dan,

I have updated Clair to 2.1.7. wget appears to ping ProGet successfully (curl as well), however, the vulnerability downloader fails to authenticate:

Inside Clair container:

curl -I https://<proget_server>
HTTP/2 200
cache-control: private
content-length: 23531
content-type: text/html; charset=UTF-8
server: Microsoft-IIS/10.0
x-aspnet-version: 4.0.30319
x-proget-version: 5.3.36.4
x-powered-by: ASP.NET
date: Mon, 20 Sep 2021 17:42:08 GMT

wget --spider https://<proget_server>
Connecting to <proget_server> (<proget_cluster_IP>:443)
remote file exists
/tmp # read escape sequence

On build machine

docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3a8a5c58b8cc quay.io/coreos/clair:v2.1.7 "/usr/bin/dumb-init …" 11 minutes ago Up 6 minutes zealous_jemison
e794f6f91d08 postgres:9.6 "docker-entrypoint.s…" 10 days ago Up 10 days 0.0.0.0:5432->5432/tcp, :::5432->5432/tcp charming_gould
[root@optjenkinscent09 ~]# docker attach 3a8a5c58b8cc
{"Event":"could not download layer","Level":"warning","Location":"driver.go:130","Time":"2021-09-20 17:43:52.834425","error":"Get https://<proget_server>/api/docker-blobs/download/sha256%3A80369df487363e56aea88d4d41b61f1607fc2ec198e9327cfde36a5346c71bf2?API_Key=0E-6E-66-D2-5B-CE-CB-78-1A-C0-62-1B-E9-50-72-59-54-50-90-41-43-AB-F7-55-B5: x509: certificate signed by unknown authority"}
{"Event":"failed to extract data from path","Level":"error","Location":"worker.go:122","Time":"2021-09-20 17:43:52.834515","error":"could not find layer","layer":"sha256:80369df487363e56aea88d4d41b61f1607fc2ec198e9327cfde36a5346c71bf2","path":"https://<proget_server>/api/docker-blobs/download/sha256%3A80369df487363e56aea88d4d41b61f1607fc2ec198e9327cfde36a5346c71bf2"}
{"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2021-09-20 17:43:52.834572","elapsed time":91078190,"method":"POST","remote addr":"<proget_web_node>:59389","request uri":"/v1/layers","status":"400"}

Maybe there is a process in the Clair container that can't detect our trusted CA cert.

Hope this helps.

Thanks,
Scott

scusson_9923

Hi,

Adding a little more info. I installed curl in the clair container and it comes back with success when wget does not.

curl -I https://<proget_server>
HTTP/2 200
cache-control: private
content-length: 22803
content-type: text/html; charset=UTF-8
server: Microsoft-IIS/10.0
x-aspnet-version: 4.0.30319
x-proget-version: 5.3.36.4
x-powered-by: ASP.NET
date: Mon, 20 Sep 2021 15:10:11 GMT

Thanks,
Scott

scusson_9923

Hi Dan,

The Diagnostic Center does not contain any warnings or errors regarding vulnerabilities.

Thanks,
Scott

scusson_9923

No problem Rich and thanks for looking into this. I think we will both learn something in the end. This is not a high priority at the moment, fyi.

Thanks,
Scott

scusson_9923

Hi Rich,

Unfortunately, I am not having any success. I have installed our trusted cert in the clair container and wget still fails ssl verification. I see the cert in /etc/ssl/certs ca-cert-USWILCA01v.pem -> /usr/local/share/ca-certificates/USWILCA01v.crt This trusted cert is installed on a mixture of Win\Linux environments (vms, local, and docker images) without issue.

Any other advice? Is this a futile attempt over https? :)

Thanks,
Scott

scusson_9923

Hi Rich,

The Web.BaseUrl can be pinged from the Clair image. Our ProGet cluster is 443, but the exposed ports of the Clair container 6060 and 6061 (health check). Our Clair API URL where Clair is running is http://<Clair_Host_IP>:6060. Since our cluster is 443, do we have to install our trusted cert inside the Clair container? When I run wget from inside the Clair container (which is running apline linux):

wget <Web.BaseUrl>
Connecting to <Web.BaseUrl> (<Web.BaseUrl_IP>:443)
ssl_client: <Web.BaseUrl>: certificate verification failed: unable to get local issuer certificate
wget: error getting response: Connection reset by peer

API Authorization Header is null
We do allow anonymous pulls from ProGet

Thanks,
Scott

scusson_9923

Hi Rich,

I guess clair had an issue pulling all layers since I do not have a scan log to provide. Is there a way to verify a valid connection from the clair server other than getting a valid ping to the ProGet cluster? Or vice versa?

Thanks,
Scott

scusson_9923

I also want to mention that this is in a NLB high availability environment which contains 3 web nodes.

scusson_9923

Hi,

Yes. ProGet can be pinged from the clair server and Web.BaseUrl is set correctly.

Thanks,
Scott

scusson_9923

Hi Rich,

What is your current version of ProGet: 5.3.36
What is your current version of the Clair extension: 1.9.0
What version of Clair are you running: 2.1.2
Is this something new that has recently started happening? Yes. Just setup the integration last week
Does it give you that warning for all layers or just some specific layers? There are ~69K WARN entries in the job output so I am thinking all layers. But without more log info from ProGet, I can't be 100% sure
Would it be possible to share any sort of example image from a third-party registry that has this issue? Sorry, no.

scusson_9923

Hi,

We have setup Clair integration with ProGet and connections appears ok. However, the only log entries from running the VulnerabilityDownloader job are WARN: BadRequest for layer sha256:<Layer_ID>. What can I do to diagnose these warnings?

Thanks,
Scott

scusson_9923

Just for reference, I was able to get the correct digest of a tag and delete it with this PowerShell script...

$image = "some_image"
$some_image_tag = "some_tag"

get digest

$curlReturn = curl -I HEAD https://<proget_repo>/v2/<container>/library/$image/manifests/$some_image_tag
foreach ($line in $curlReturn)
{
if ($line -match "Docker-Content-Digest: (.+)")
{
$digest = $Matches[1]
Write-Host "$image digest is $digest"
}
}

delete tag digest

$URI = "https://<proget_repo>/v2/<container>/library/$image/manifests/$digest"
Write-Host "Deleting tag $some_image_tag from image $image"
Invoke-WebRequest -Uri $URI -Method Delete -Headers @{"X-ApiKey"="<proget_api_key>"; }

scusson_9923

Thanks for the reply!

scusson_9923

By the way, do you have preliminary release notes for 5.3 that we can look at?

Thanks,
Scott

scusson_9923

Hi,

I am trying to obtain the correct digest of a particular docker tag and have seen two methods, with different outputs (see below), to accomplish this.  Which is correct?  From your documentation, I need to get the digest associated with 'Docker-Content-Digest', which is option 2.  However, documentation doesn't explicitly mention using curl (unless it's assumed?).

place 'https://<proget_repo>/v2/<container>/library/<image_name>/manifests/<tag>' in a web browser. Alternatively, use Powershell's Invoke-WebRequest
type 'curl -v -H Accept: "application/vnd.docker.distribution.manifest.v2+json" -X HEAD https://<proget_repo>/v2/<container>/library/<image_name>/manifests/<tag>' on the command line

Option 1 output:
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 4536,
"digest": "sha256:d8622010e224b5c7dc541c0e8843b993fd5c5afe422ed1309aac207bd06885a7"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 22524609,
"digest": "sha256:804555ee037604c40de144f9f8da0d826d38db82f15d74cded32790fe279a8f6"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 17692725,
"digest": "sha256:970251047358aea56ba6db6975b14ff12470b75de0c2477f4445240ddd727fd4"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 2978257,
"digest": "sha256:f3d4c41a4fd13f35c0b46f19a4e27845f4695163cc7174d908ff84836bbc2f5a"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 62145592,
"digest": "sha256:bd391c46585f9f8d84992bbaa9087189148c1601968eaaf097d5b3ed60840e5e"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 126,
"digest": "sha256:1674b9537e7545c5665f935cab34e05e09971a88dab2b86fdf00e516089708f5"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 25187509,
"digest": "sha256:51ecaa038642c6c35b4be45cf823e06d5f3f50f8f3f5e9cfb52c6b5151a57dc5"
}
]
}

Option 2 output:

Warning: Setting custom HTTP method to HEAD with -X/--request may not work the
Warning: way you want. Consider using -I/--head instead.

Could not resolve host: application
Closing connection 0
curl: (6) Could not resolve host: application
Warning: Setting custom HTTP method to HEAD with -X/--request may not work the
Warning: way you want. Consider using -I/--head instead.
Trying 10.2.0.87...
TCP_NODELAY set
Connected to <proget_repo> port 443 (#1)
schannel: SSL/TLS connection with <proget_repo> port 443 (step 1/3)
schannel: checking server certificate revocation
schannel: sending initial handshake data: sending 190 bytes...
schannel: sent initial handshake data: sent 190 bytes
schannel: SSL/TLS connection with <proget_repo> port 443 (step 2/3)
schannel: encrypted data got 1842
schannel: encrypted data buffer: offset 1842 length 4096
schannel: sending next handshake data: sending 158 bytes...
schannel: SSL/TLS connection with <proget_repo> port 443 (step 2/3)
schannel: encrypted data got 51
schannel: encrypted data buffer: offset 51 length 4096
schannel: SSL/TLS handshake complete
schannel: SSL/TLS connection with <proget_repo> port 443 (step 3/3)
schannel: stored credential handle in session cache

HEAD /v2/<container>/library/<image>/manifests/<tag> HTTP/1.1
Host: <proget_repo>
User-Agent: curl/7.55.1

schannel: client wants to read 102400 bytes
schannel: encdata_buffer resized 103424
schannel: encrypted data buffer: offset 0 length 103424
schannel: encrypted data got 483
schannel: encrypted data buffer: offset 483 length 103424
schannel: decrypted data length: 454
schannel: decrypted data added: 454
schannel: decrypted data cached: offset 454 length 102400
schannel: encrypted data buffer: offset 0 length 103424
schannel: decrypted data buffer: offset 454 length 102400
schannel: schannel_recv cleanup
schannel: decrypted data returned 454
schannel: decrypted data buffer: offset 0 length 102400
< HTTP/1.1 200 OK
< Cache-Control: public, max-age=300
< Content-Length: 1583
< Content-Type: application/vnd.docker.distribution.manifest.v2+json
< Vary: Authorization
< Server: Microsoft-IIS/10.0
< X-AspNet-Version: 4.0.30319
< X-ProGet-Version: 5.2.23.7
< Docker-Distribution-API-Version: registry/2.0
< Docker-Content-Digest: sha256:219db0d935303e3d454231d6146ee56d658c8399f10de95648cdd91379139ce5
< X-Powered-By: ASP.NET
< Date: Tue, 11 Feb 2020 19:25:03 GMT
<
schannel: client wants to read 1583 bytes
schannel: encrypted data buffer: offset 0 length 103424
schannel: Curl_read_plain returned CURLE_RECV_ERROR
schannel: encrypted data buffer: offset 0 length 103424
schannel: encrypted data buffer: offset 0 length 103424
schannel: decrypted data buffer: offset 0 length 102400
schannel: schannel_recv cleanup
Closing connection 1
schannel: shutting down SSL/TLS connection with <proget_repo> port 443
Send failure: Connection was reset
schannel: failed to send close msg: Failed sending data to the peer (bytes written: -1)
schannel: clear security context handle

Do I run option 1: 'DELETE /v2/<container>/library/<image>/manifestsd8622010e224b5c7dc541c0e8843b993fd5c5afe422ed1309aac207bd06885a7'

or option 2 (Docker-Content-Digest)

'DELETE /v2/<container>/library/<image>/manifests/219db0d935303e3d454231d6146ee56d658c8399f10de95648cdd91379139ce5'

Thanks,
Scott

scusson_9923

Thanks for the update. I am looking forward to 5.3! :)

scusson_9923

Hi and thank you for the reply! From what I have read, no Delete tag API exists. At least not yet. It appears that it may be in a future docker release.

https://github.com/docker/distribution/pull/2169

What I am looking for is what is outlined on this page:

https://serverfault.com/questions/872332/easy-way-to-remove-a-tag-from-a-private-docker-registry/872336

ideally: DELETE http://<docker_repo>/v2/<image>/<tag>

But in the meantime, it would be great if we could script how ProGet accomplished this in the UI. Although, ProGet's terminology is delete package\image to delete. Maybe it can be delete image\tag to delete? Also, assuming if you delete the last tag, the entire image is deleted as well.

Please correct me if I am wrong :)

Scott

scusson_9923

Hi,

Can anyone from Inedo chime in?

Thanks,
Scott

scusson_9923

Hi,

Since deleting docker tags is not supported via the docker registry API HTTP DELETE (yet), I was hoping we could script the process ProGet UI runs to remove tags. https://<proget_server>/feeds/<container_folder>/library%2F<image>->"Delete Package"-><tag>

Is this possible? I do not want to rely on retention rules for our docker images.

Thanks,
Scott

scusson_9923

@gdivis Sure thing. Just let me know when something is available for me to test.

scusson_9923

Will ProGet support 'helm push' at some point?

ex. helm push mychart-1.2.3.tgz https://<proget-server>/helm/<helm-feed>/

Posts made by scusson_9923

get digest

delete tag digest