Hello Alana,

We want to add to our current functionality and make it backward compatible. We have an archive job running in Jenkins which then uploads a config file to Otter and triggers a job. That Otter job runs remotely using Otter\PowerShell scripts with that config file and an input. What we want to do is have a related archive job with its own config file use the same Otter\PowerShell scripts. The issue is, we cannot guarantee that both config files will exist. It would help greatly if we can conditionally get an asset.

Thanks,
Scott

Hi,

I am trying to conditionally run Get-Asset in an Otter script based on the existence of a script in the default raft. I am using curl and regardless if the script is there or not, I always get a positive return. Example (foo.config does not exist):

curl.exe --head "https://<server_url>/0x44/Otter.WebApplication/Inedo.Otter.WebApplication.Pages.Scripts.ListScriptsPage/DownloadAsset?id=Default::BinaryFile::<path_to_script>/foo.config"
HTTP/1.1 302 Found

How can I either conditionally run Get-Asset or use curl to check for a script in the raft?

Thanks,
Scott

Hi Alana,

This seems to work for what I am trying to accomplish.

Thanks!

Scott

Hi Alana,

What I am trying to do is capture the result (success or fail) of this call:

InedoCore::Exec
(
    FileName: pwsh,
    WorkingDirectory: C:\,
    Arguments: test.ps1,
    ErrorOutputLogLevel: Error,
    OutputLogLevel: Information,
    SuccessExitCode: == 0
);

My understanding is that currently InedoCore::Exec does not have a return value. I was hoping for something like:

$ret = InedoCore::Exec
(
 ...
);

if $ret == "0"
{
    // do operations for success
}
else
{
   // do operations for failure
}

Thanks!
Scott

Any updates on this?

Thanks!
Scott

@dean-houston Hi Dean,

I did try setting the return of my PowerShell test script to exit -1 which fails the InedoCore::Exec call as expected. I just need to capture that failure and test on it.

Thanks,
Scott

@dean-houston Hi Dean,

I am having a difficult time understanding the workflow here.

As an experiment, I run a ps1 which sets $Fail to "true" as an Ad Hoc job:

InedoCore::Exec
(
    FileName: pwsh,
    WorkingDirectory: C:\,
    Arguments: test.ps1,
    ErrorOutputLogLevel: Error,
    OutputLogLevel: Information,
    SuccessExitCode: == 0
);

if $Fail == "true"
{
    InedoCore::Send-Email
    (
        ...
    );
}

I get this: Could not resolve variable $Fail.

Do you have a working example? I must be missing something.

Also, what is the purpose of SuccessExitCode if you cannot test on it?

Thanks!
Scott

Hi Alana,

Are you planning on adding a return value for an Exec operation?  In my job, I call an otter script which calls pwsh via Exec and would like to send an email based on its return (success\failure).

Thanks!
Scott

Hi Dean,

Setting RaftItemType_Code = 7 in this instance worked for me. Appreciate the list of valid types. Thanks for all of your help!

Scott

Hi Dean,

Here is what I'm running:

Invoke-WebRequest -Method Post -Uri "https://<otter_server>/api/json/Rafts_CreateOrUpdateRaftItem" -Body @{
API_Key = "<apikey>"
Raft_Id = 1
RaftItemType_Code = 4
RaftItem_Name = "JobConfigs/test.yml"
ModifiedOn_Date = Get-Date
ModifiedBy_User_Name = "scusson"
Content_Bytes = [System.Convert]::ToBase64String([IO.File]::ReadAllBytes("D:\test.yml"))
}

If I add the text file (test.yml) thru the UI, it gets added as binary.

Thanks,
Scott

Hi,

This functionality is fixed in 2024.4. However, the script gets created as type "Unknown". How can this be set and what types are valid?

Thanks,
Scott

@dean-houston Thanks!

Hi Alana,

I updated my instance to 2024.2 and still get the same error.

Thanks,
Scott

Hi,

Thanks for the example. However, I still get the error 'Invalid value for "Content_Bytes": Invalid cast from 'System.String' to 'Inedo.Data.VarBinaryInput'' with this body:

API_Key = "xxxxx"
Raft_Id = 1
RaftItemType_Code = 4
RaftItem_Name = "JobConfigs/test.yml"
ModifiedOn_Date = Get-Date
ModifiedBy_User_Name = "scusson"
Content_Bytes = [System.Convert]::ToBase64String([IO.File]::ReadAllBytes("D:\foo\bar\test.yml"))

It appears that the .yml is being detected as binary?

Scott

Hi,
I'm on 2024.1. In this instance, I use Add Script->Upload Scripts & Assets because I already have the file on disk. I will use the "Create a Blank Script" method as a workaround.

Also, can you provide an example of uploading a file from disk?

Thanks!
Scott

Hi,
When I try your example, I get the error 'Invalid value for "Content_Bytes": Invalid cast from 'System.String' to 'Inedo.Data.VarBinaryInput''.

Also, when I add a text file under Scripts, in my case .yml, it gets added as binary. I can't see a way to change this.

Thanks,
Scott

@atripp Hi Alana. Thanks! I'll try it out.

Hi,

Since Otter does not have a native api to upload a script, can we get a working example for Rafts_CreateOrUpdateRaftItem to perform this action to a subfolder? Our goal is to have an automated job push a script upon success.

Thanks,
Scott

Hi,

Just to update this issue. I ended up having to remove, clean up, and reinstall IIS\ProGet. After that, the default ProGet instance loaded successfully. Thanks for all of your input.

Scott

Hi Alana,

Yes. Local connection was successful. I am in the process of setting up 2 web instances. I was able to resolve this issue after I noticed that a couple of SQL Server services on the DB server were installed as disabled by default (browser\agent). However, I am having issues when setting up the second instance. When I try to load the ProGet website, I get a "HTTP Error 503. The service is unavailable." message in the browser and in the Event Viewer I see: "The Module DLL C:\Windows\system32\inetsrv\aspnetcore.dll failed to load. The data is the error." This machine may be in a bad state as this seems to be a red herring. I'm tempted to have this machine rebuilt unless you have additional pointers :)

Thanks!
Scott

Hi,

I am trying to connect a local 2022.11 IIS ProGet instance to a remote MSSQL express ProGet DB. The remote ProGet DB, local website, App Pool, and service are all running as the same user. That user is also the owner of the ProGet DB. I can successfully connect to the remote ProGet DB with SS Management Studio. I use the connection string 'Data Source=<DB_Server>\Inedo;Initial Catalog=ProGet;Integrated Security=True' when trying to load the local ProGet website and I get the error 'HTTP Error 500.30 - ASP.NET Core app failed to start' with no entries in the Event Viewer.

Thoughts?

Thanks,
Scott

Thank you. I will do that.

Hi,

I reset the cache on all nodes and updated to 2022.8 (planned update). I have a user that continues to exhibit this behavior. Is there something he can try on his end?

Thanks,
Scott

Hi,

Thank for you for the quick reply! I'll try that.

Scott

@NanciCalo said in User with permissions Publish permissions is denied:

"Invalidate Cache button" as something for us to try to get in :)

Hi,

We recently started getting this and the "Clear Cache" option does not seem to resolve our issue.  Any other options (other than setting Web.PrivilegeCacheExpiration)?

Thanks,
Scott

Good news! Thanks!

Hi,
We are looking to enable SAML in the near future and wondering where this issue stands? We need to keep our apikey access unchanged and allow for local logins.

Thanks,
Scott

@rhessinger Hi Rich,

Works perfectly now.

Thanks!

Scott

@scusson_9923Hi Rich,

if I just use -X I get the return:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">

<HTML><HEAD><TITLE>Length Required</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Length Required</h2>
<hr><p>HTTP Error 411. The request must be chunked or have a content length.</p>
</BODY></HTML>

We plan on adding this in an automated build scenario, so there is a good chance we'll be using this regularly.

Thanks,
Scott

@rhessinger Hi Rich,

Thanks! Adding the /dir/ worked. I still get the "curl: (6) Could not resolve host: POST" error on Windows. Linux is "curl: (6) Could not resolve host: POST; Unknown error". Can your online documentation be updated with the full command?

curl -d -X POST https://<ProGet_server>/endpoints/<asset_dir>/dir/<new_dir>/ --user <usename>:<password>

Thanks again,
Scott

Hi,

A little more info. Looking at the IIS log, I see:

2021-12-03 19:39:41 <ProGet_ClusterIP>7 POST /endpoints/<asset_dir>/<new_dir>/ - 443 - <Machine_IP>> curl/7.55.1 - 200 0 0 78

So we know it hits the ProGet server and returns success from IIS. Just very confused on "curl: (6) Could not resolve host: POST" from curl.

Thanks,
Scott

@Dan_Woolf Hi Dan,

When I add -X, I get:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">
<HTML><HEAD><TITLE>Length Required</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Length Required</h2>
<hr><p>HTTP Error 411. The request must be chunked or have a content length.</p>
</BODY></HTML>

when I use -d -X, I get:

curl: (6) Could not resolve host: POST

If it helps, our environment is load balanced with 3 web nodes.

Thanks,
Scott

Hi,

I am trying to follow the section "Create Directory Endpoint" on https://docs.inedo.com/docs/proget-reference-api-asset-directories-api and I am unable to create a subfolder with the following command:

curl POST https://<ProGet_server>/endpoints/<asset_dir>/<new_dir>/ --user <user>:<password>

I get "curl: (6) Could not resolve host: POST"

Can the online documentation be updated to give a full example instead of "POST .../dir/«path»"

Thanks,
Scott

Hi Rich,

I created an API key with full access and still get wget: server returned error: HTTP/1.1 403 Forbidden.

When I place the same link in my local browser, I get:
The specified API key does not permit access to the Docker Blob Reader API.

In the Access Key Log:
/api/docker-blobs/download/sha256:f033c4f65cdbf0bfa21d5543e56c0c41645eca4d893494bb4f0661b0f19ccc79?API_Key=<apikey>
Response code: 403

As a test, I used wget to download a file from an Assets directory inside the Clair container successfully.

Thanks,
Scott

Hi,

I get 'wget: server returned error: HTTP/1.1 403 Forbidden'

When I try curl with a ProGet user\password, I get 'An API key is required to download Docker blobs.'

I saw the API key in the previous logs (container without -insecure-tls), but not now.

Thanks,
Scott

Baby steps! I still see no entries in the diagnostics center regarding vulnerabilities.

Thanks,
Scott

Hi Rich,

Still is failing, different reason:

{"Event":"could not download layer: expected 2XX","Level":"warning","Location":"driver.go:136","Time":"2021-09-20 19:01:26.736879","status code":404}
{"Event":"failed to extract data from path","Level":"error","Location":"worker.go:122","Time":"2021-09-20 19:01:26.737319","error":"could not find layer","layer":"sha256:f033c4f65cdbf0bfa21d5543e56c0c41645eca4d893494bb4f0661b0f19ccc79","path":"https://<proget_server>/api/docker-blobs/download/sha256%3Af033c4f65cdbf0bfa21d5543e56c0c41645eca4d893494bb4f0661b0f19ccc79"}
{"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2021-09-20 19:01:26.737385","elapsed time":166008805,"method":"POST","remote addr":"<proget_web_node>:59778","request uri":"/v1/layers","status":"400"}

Thanks,
Scott

Hi Dan,

I have updated Clair to 2.1.7. wget appears to ping ProGet successfully (curl as well), however, the vulnerability downloader fails to authenticate:

Inside Clair container:

curl -I https://<proget_server>
HTTP/2 200
cache-control: private
content-length: 23531
content-type: text/html; charset=UTF-8
server: Microsoft-IIS/10.0
x-aspnet-version: 4.0.30319
x-proget-version: 5.3.36.4
x-powered-by: ASP.NET
date: Mon, 20 Sep 2021 17:42:08 GMT

wget --spider https://<proget_server>
Connecting to <proget_server> (<proget_cluster_IP>:443)
remote file exists
/tmp # read escape sequence

On build machine

docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3a8a5c58b8cc quay.io/coreos/clair:v2.1.7 "/usr/bin/dumb-init …" 11 minutes ago Up 6 minutes zealous_jemison
e794f6f91d08 postgres:9.6 "docker-entrypoint.s…" 10 days ago Up 10 days 0.0.0.0:5432->5432/tcp, :::5432->5432/tcp charming_gould
[root@optjenkinscent09 ~]# docker attach 3a8a5c58b8cc
{"Event":"could not download layer","Level":"warning","Location":"driver.go:130","Time":"2021-09-20 17:43:52.834425","error":"Get https://<proget_server>/api/docker-blobs/download/sha256%3A80369df487363e56aea88d4d41b61f1607fc2ec198e9327cfde36a5346c71bf2?API_Key=0E-6E-66-D2-5B-CE-CB-78-1A-C0-62-1B-E9-50-72-59-54-50-90-41-43-AB-F7-55-B5: x509: certificate signed by unknown authority"}
{"Event":"failed to extract data from path","Level":"error","Location":"worker.go:122","Time":"2021-09-20 17:43:52.834515","error":"could not find layer","layer":"sha256:80369df487363e56aea88d4d41b61f1607fc2ec198e9327cfde36a5346c71bf2","path":"https://<proget_server>/api/docker-blobs/download/sha256%3A80369df487363e56aea88d4d41b61f1607fc2ec198e9327cfde36a5346c71bf2"}
{"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2021-09-20 17:43:52.834572","elapsed time":91078190,"method":"POST","remote addr":"<proget_web_node>:59389","request uri":"/v1/layers","status":"400"}

Maybe there is a process in the Clair container that can't detect our trusted CA cert.

Hope this helps.

Thanks,
Scott

Hi,

Adding a little more info. I installed curl in the clair container and it comes back with success when wget does not.

curl -I https://<proget_server>
HTTP/2 200
cache-control: private
content-length: 22803
content-type: text/html; charset=UTF-8
server: Microsoft-IIS/10.0
x-aspnet-version: 4.0.30319
x-proget-version: 5.3.36.4
x-powered-by: ASP.NET
date: Mon, 20 Sep 2021 15:10:11 GMT

Thanks,
Scott

Hi Dan,

The Diagnostic Center does not contain any warnings or errors regarding vulnerabilities.

Thanks,
Scott

No problem Rich and thanks for looking into this. I think we will both learn something in the end. This is not a high priority at the moment, fyi.

Thanks,
Scott

Hi Rich,

Unfortunately, I am not having any success. I have installed our trusted cert in the clair container and wget still fails ssl verification. I see the cert in /etc/ssl/certs ca-cert-USWILCA01v.pem -> /usr/local/share/ca-certificates/USWILCA01v.crt This trusted cert is installed on a mixture of Win\Linux environments (vms, local, and docker images) without issue.

Any other advice? Is this a futile attempt over https? :)

Thanks,
Scott

Hi Rich,

The Web.BaseUrl can be pinged from the Clair image. Our ProGet cluster is 443, but the exposed ports of the Clair container 6060 and 6061 (health check). Our Clair API URL where Clair is running is http://<Clair_Host_IP>:6060. Since our cluster is 443, do we have to install our trusted cert inside the Clair container? When I run wget from inside the Clair container (which is running apline linux):

wget <Web.BaseUrl>
Connecting to <Web.BaseUrl> (<Web.BaseUrl_IP>:443)
ssl_client: <Web.BaseUrl>: certificate verification failed: unable to get local issuer certificate
wget: error getting response: Connection reset by peer

API Authorization Header is null
We do allow anonymous pulls from ProGet

Thanks,
Scott

Hi Rich,

I guess clair had an issue pulling all layers since I do not have a scan log to provide. Is there a way to verify a valid connection from the clair server other than getting a valid ping to the ProGet cluster? Or vice versa?

Thanks,
Scott

I also want to mention that this is in a NLB high availability environment which contains 3 web nodes.

Hi,

Yes. ProGet can be pinged from the clair server and Web.BaseUrl is set correctly.

Thanks,
Scott

Hi Rich,

What is your current version of ProGet: 5.3.36
What is your current version of the Clair extension: 1.9.0
What version of Clair are you running: 2.1.2
Is this something new that has recently started happening? Yes. Just setup the integration last week
Does it give you that warning for all layers or just some specific layers? There are ~69K WARN entries in the job output so I am thinking all layers. But without more log info from ProGet, I can't be 100% sure
Would it be possible to share any sort of example image from a third-party registry that has this issue? Sorry, no.

Hi,

We have setup Clair integration with ProGet and connections appears ok. However, the only log entries from running the VulnerabilityDownloader job are WARN: BadRequest for layer sha256:<Layer_ID>. What can I do to diagnose these warnings?

Thanks,
Scott

Just for reference, I was able to get the correct digest of a tag and delete it with this PowerShell script...

$image = "some_image"
$some_image_tag = "some_tag"

get digest

$curlReturn = curl -I HEAD https://<proget_repo>/v2/<container>/library/$image/manifests/$some_image_tag
foreach ($line in $curlReturn)
{
if ($line -match "Docker-Content-Digest: (.+)")
{
$digest = $Matches[1]
Write-Host "$image digest is $digest"
}
}

delete tag digest

$URI = "https://<proget_repo>/v2/<container>/library/$image/manifests/$digest"
Write-Host "Deleting tag $some_image_tag from image $image"
Invoke-WebRequest -Uri $URI -Method Delete -Headers @{"X-ApiKey"="<proget_api_key>"; }

Thanks for the reply!