Posts made by atripp

Hi @Jon,

There haven't been any major changes recently. What version did you upgrade from?

Here are the recent changes to Otter:
https://my.inedo.com/downloads/issues?Product=Otter

Assuming you were on v2022 before, I would suggest to rollback, and see if it helps.

Cheers,
Alana

Hi @ssuenaga_1020 ,

Thanks for the detailed report; this seems easy enough to fix, so we'll get this addressed in the next maintenance release (2023.20) via PG-2499.

Cheers,
Alana

Hi @e-rotteveel_1850,

Thanks for reporting this; we'll get this fixed via PG-2498 in ProGet 2023.20 (in two weeks).

Cheers,
Alana

Hi @philippe-camelio_3885 ,

We tried to reproduce this in the latest version of Otter, but are not having any luck :(

Here is my Job Template; note that you can get the JSON for job template under Admin > Raft Repositories > Browse.

{
  "Description": "",
  "JobConfiguration": {
    "ScriptId": "Default::Script::z-z-test/test-hello.otter",
    "ServerTargeting": 1,
    "ServerNames": [
      "LOCALHOST"
    ],
    "ServerRoleNames": [],
    "EnvironmentNames": [],
    "Variables": {},
    "Arguments": ""
  },
  "TemplateUsage": 1,
  "JobVariables": [
    {
      "Name": "params",
      "Description": "",
      "InitialValue": "%(env:item1)",
      "Type": "List",
      "Usage": "Input",
      "ListValues": [
        "%(env:item1)",
        "%(env:item2)",
        "%(env:item3)"
      ],
      "ListRestrict": true
    }
  ]
}

Here is my OtterScript. I wanted to verfify that env would be written, which it was.

foreach $i in @MapKeys(%params)
{
    Log-Debug Hello $i;
}

It's possible it was fixed by something else since the version you used, but I don't know.

Might be worth upgrading; if you still have the issue, can you send your JSON of the Job TEmplate?

Thanks,
Alana

Hi @mness_8576,

Unfortunately it's not going to be possible/practical to troubleshoot issues inside of ProGet when there are possibly problematic gateway appliances in the way. In general I can't imagine how there would be issues if there was only ProGet.

We know there's at least one HTTP gateway server between the build server and ProGet (which explains the 504 error), but there may also be a gateway between ProGet and nuget.org.

So I would first find out where these gateways are, and bypass them. If that's not possible, then you'll have to see what's happening on the gateway side. There are typically logs issued, etc.

We've seen everything from content filters to cache servers to DDoS protectors cause problems here. It's really difficult to guess.

Hope that helps,
Alana

@lucas-almeida_8120 it's hard to say; it's likely due to some kind of operating system configuration or issue that's preventing the service from restarting. We haven't seen this happen very often, and restarting the web is not a common activity.

It's probably not worth spending time investigating or trying to resolve, especially given the age of the server. You'd probably have to look at the Windows service logs to find what the issue might be.

Hi @mness_8576,

A 504 would be coming from some kind of intermittent network equipment (gateway), most typically a proxy server or SSL Offloading, etc. It's not issued by ProGet or IIS.

We've see this quite a bit - and usually the gateway has a timeout configured. Once reached, it terminates the connection, giving a 504 to the client and a "disconnect" to the server. The only way to fix this is to find that intermediate device and disable the timeout.

Best,
Alana

Hi @jw,

Thanks, good points! I'll try to explain the inner-workings a little more.

I would ask first is why ProGet does need the full package in its cache for the SCA tracking to work correctly.

This gets a little tricky and confusing; SCA checks for deprecation, vulnerabilities, and licenses.

• Vulnerabilities only need the Package Id + Version, so no package is needed
• License is generally stored in the package manifest file (e.g. nuspec file)
• Deprecation is server-side metadata, which means ProGet's value may be different than the connector's value

So the design thought was, if only vulnerabilities were detected it would be weird. Hence, missing package issue only. Maybe this assumption is incorrect.

Maybe a parameter to SBOM uploading/analyzing could be introduced that enables downloading the required metadata or packages (depending on what is really needed) before analysis is started

This unfortunately gets pretty complicated...

• ProGet will only store metadata for local/cached packages
• Package caching is an optional setting (not all feeds use it)
• There is no mechanism for automatically pull or cache packages

Designing a mechanism inside of ProGet to automatically download would not trivial at all for this use case. A PowerShell script to handle this on a one-off basis would probably be much easier.

FYI -- we are considering some kind of job that automatically download certain, trusted packages. This is in the context of "package approval workflows" and basically allowing something like AWSSDK (which frequently publishes new versions) to just always promote a new version. That's not exactly related to SCA/SBOM however.

Alana

Hi @jw ,

Thanks for the feedback; we definitely have plans to review/improve our SCA/SBOM processes, so my answers will be questions ---- please don't hesitate to give us additional feedback in the aggregate when you have a chance :)

As for this particular case, we've seen this too - and just found that downloading the package caused the issue to go away, and that was simple enough since there weren't too many packages (just lots of annoying clicking).

I don't think we want to have ProGet automatically download/populate the feed with packages, but perhaps a better way to handle this is some kind of rule or ignore setting for missing packages??

And it'd be nice if ProGet automatically shipped with a list of these too. I really can't think of any reason why anyone would ever want a notice of "Microsoft.Extensions.Localization" being missing. Can you?

This is probaly something unqiue to NuGet as opposed to other package managers.

Thanks,
Alana

Hi @carl-westman_8110 ,

That's a bug within the Azure SDK, and it looks familiar. I think I've seen it before, and we upgraded or worked-around it a while ago.

What version of ProGet is this? I would upload to latest version if you're not there.

Otherwise, can you try to "play around" and try different feed types (NuGet for example) and different Azure buckets/configurations? Given that this is an Azurer SDK bug, it's difficult to guess what exactly is causing the problem.

Thanks,
Alana

Hi @mvanweel_2810 ,

I'm happy to point you to methods! Sorry that I don't have example scripts, but I hope this helps...

[1] This would require using the Native API method, specifically ApiKeys_CreateOrUpdateApiKey; we don't document the format of ApiKey_Configuration, but it's pretty basic XML that you can figure out by creating other API Keys in the UI and looking at the value of the data

[2] It looks like we don't expose these currently in any API; I can give you a "cheat code" to expose these in the Native API, so they will show up on /reference/api:

update __StoredProcInfo set Internal_Indicator = 'N' where StoredProc_Name like 'FeedGroups%'

Please let me know if this API works for you, and I will mark them as public in a future maintenance release.

[3] "Pulling to ProGet" is the same thing as downloading and then uploading that package. That's effectively what the UI does. So, you can just do that with two calls to the Common Package API.

Please feel free to share your example scripts :) This is an area we really want to improve in our docs.

Cheers,
Alana

Hi @w-repinski_1472 ,

Even with heavy usage, SQL Server Express will be fine; performance issues from heavy usage will come at the network level, and that's solved by configuring ProGet as a High Availability & Load Balanced Cluster.

We've seen a few 10GB+ ProGet databases in the field, and 95% of the space is taken up with years of individual package download records stored in PackageDownloads table. Those are obviously okay to purge (or not enable at all).

Otherwise, to move to a different SQL Server, just restore a database backup to the new server and update the connection string in ProGet.

Best,
Alana

Thanks for clarifying @philippe-camelio_3885

There was a big change to Job Templates in 2022, so it's not surprising it's a regression.

We'll get this via OT-496, hopefully in the same next maintenance release as well.

@philippe-camelio_3885 thanks! I've logged this as OT-495, and we'll try to get it fixed in the next mainteance release (scheduled Sept 15)

I added this to our Otter 2023 list of items to review/consider :)

Please stay tuned

Hi @philippe-camelio_3885 ,

I didn't try to reproduce this or investigate, but there is clearly some kind of problem parsing that variable value because it's a map expression, and that's crashing the job processor (i.e. script isn't even being run).

I don't think we considered this would be used for maps, but just user-friendly variables instead. Are you intending to use this as a user-selectable map??

Just as an idea, does prefixing it with the grave apostrophe escape symbol work? Like

`%(env:item1)
`%(env:item2)
`%(env:item3)

With that we can explore/consider how to fix.

Thanks,
Alana

Thanks for confirming @philippe-camelio_3885

The "Server Checker" and "Check Configuration" are two different functions, so that kind of makes sense. The "Server Checker" should be triggered immediately after adding or modifying agent details (like IP, etc), but it clearly isn't here.

One more question - are you adding these servers via the API?

I just want to make sure we can find/repro/fix this :)

@philippe-camelio_3885 thanks for clarifying.

Next time, can you try running the Server Checker task and seeing if that clears the problem within a short time?

A configuration execution will wait until a server is in a specific status (the Server Checker puts it in that status), so I wonder if it's not being triggered. I didn't look further but confirming it twould help us isolate the issue.

Hi @philippe-camelio_3885 ,

Sorry this must have fell through the cracks...

Few questions.

• How soon after you add the server do you run a configuration run?
• When you say "reset otter", do you mean the Otter Service?
• Are you able to cancel those executions? Like, if you navigate to them?
• Does running the Server Checker task (on the Service Page, same page that shows Execution Dispatcher) cause them to kick off? What output do you see when you run that task, if not?

Thanks,
Alana

Hi @en_1446 ,

Not sure what version of ProGet you're using, but this seems to be on the recent versions I've seen, so I would recommend to upgrade :)

Cheers,
Alana

Hi @k_2363 ,

What version of ProGet are you using?

Another question: Is there any ETA on the API development?

An API for the built-in User/Groups are not really a priority and there's basically no demand for it. Everyone just uses LDAP instead.

Can you help us understand how/why you're using this API?

Thanks,
Alana

Hi @kichikawa_2913,

You can ignore the Age field; it's just used internally by Visual Studio, along with the Id. Not sure why, but it's a strange value in PortablePDB files.

Here's more on what Age means if you're dying to read PE-COFF specifications

https://github.com/dotnet/corefx/blob/master/src/System.Reflection.Metadata/specs/PE-COFF.md#codeview-debug-directory-entry-type-2

Best,
Alana

Hmm that's really peculiar; we haven't seen an issue like this, so I'm not sure how to troubleshoot.

I don't think this is operation-specific, so I wonder if you could try some other tests on this server. Perhaps do something simple, like shexec echo hello world; ? And perhaps see if it's an issue on other servers?

Thanks,
Alana

Thanks for clarifying! So it sounds like the issue is, the Git::Checkout-Code operation is freezing on a remote server (connected via SSH).

I don't think there's anything you're doing wrong, but there's clearly something going on. It's hard to guess what; typically an input prompt is about the only thing that will cause a freeze like that -- but I don't know how that can happen with that operation.

From here, i would check out the sshd logs, and see what's going on between the servers. You should see the last ssh command ran, and maybe that will give some kind of clue as to a problem

Hi @philippe-camelio_3885 ,

Hmmm, good question; 8min: 20seconds is quite a long time. So I guess, there's a timeout going on? Maybe it's a prompt for input that's happening? I dunno why that would be the case.

There's a lot of logs missing, so we can't really see where it's failing. Does it work if you're checking out to a Windows server (maybe localhost?)

I thought if the application is linked to the git repo, it is transparent and I don't need to add the infor i n the checkout function.

The build is associated with a Commit/Branch/Repository. The operation will effectively default as follows:

    Git::Checkout-Code
    (
        From: $Repository,
        BranchOrCommit: $Commit
    );

$Repository will just be the name of your connection, and before running on the remote server, the url/username/password will be extracted from the connection.

Thanks,
Alana

Hi @v-makkenze_6348 ,

Thanks so much for narrowing that down! Thanks to that, I was able to find the issue and fix it.

This will be fixed in ProGet 2023.16 (we will release today or tomorrow), but I recommend to patch this.

To patch, download the SQL Script attached to PG-2466 and then run it against your ProGet database.

Cheers,
Alana

Hi @v-makkenze_6348 ,

Unfortunately this can be a really tricky issue to identify, as you'll need to figure out specifically what's causing these problems. It's often unrelated to ProGet, and could be caused by anything from Windows updates to low disk space.

I'll try to ask a few questions and give some tips on how to narrow things down.

The best place to start, what version did you upgrade from? If it was an earlier version of ProGet 2023, I would rollback; that will let you identify if it is in fact related to an upgrade.

The next thing I would try is disabling the ProGet Service; this is separate from the ProGet Web Service. The regular ProGet Service doesn't need to run for the Web Service (Web UI) to function.

If the problem goes away, then I restart the service but disable scheduled jobs (Admin > Scheduled Jobs) like feed clean up, vulnerability download, package analyzer, etc.

If the problem goes away, then I would try to find out which service specifically is causing problems.

You can also look at the execution logs (Admin > execution logs), and if something is taking a really long time, then that could be an indication of a problem

Thanks,
Alana

Hi @philippe-camelio_3885 ,

If the Encryption Key is okay, then you shouldn't have a problem viewing the "Secure Credentials" page; that has encrypted values. I would also expect a different error (some "invalid padding" or something) if it was a bad encryption key.

I couldn't find the error "Invalid signature for supplied public key, or bad username/public key combination" in our codebase, which means it's coming from a library we're using. In this case, libssh2.

And if that's the case, it usually usually means the problem is on the server (i.e. linux server you're connecting to); and also that someone else might have the same problem.

Here's what I found on this page about debugging SSH:

This error can be quite misleading. You'll see this if your server wanted two forms of authentication and you've only provided one.

Hopefully that helps. You may find other help by searching that same error. And if you discover, please let us know what it is - so another future engineer can also discover the secret way to fix it ;)

Thanks,
Alana

Hi @philippe-camelio_3885,

The first thing that comes to mind is that the encryption key wasn't moved/set correctly on the new instance; https://docs.inedo.com/docs/installation-linux-supported-environment-variables

If this is the case, then i think you would also get errors browsing some pages that have encrypted data.

Thanks,
Alana

Hi @ivan-magdolen_6846,

It looks like you're on the right track with finding the XML for that key.

Since you're automating the installation, and already have DBO access to the ProGet database, I would suggest just directly adding it to the database using that stored procedure. You can also add/edit some values (like license key) if you want to the Configuration table.

Alternatively I guess you could try Admin:Admin as the API key, since that account will be created by default. I'm not sure if it will work with the native API however.

Best,
Alana

Hi @ccordova_8628 ,

We have several customers running ProGet as K8 cluster in Production, and we've even added some "special" Kubernetes-only features like Upgrading the Database Only (Optional)... but we do not have an official Helm chart we can provide.

This is because, unlike Windows/IIS and Docker/NGIX, we aren't very experienced at troubleshooting broken K8 clusters.

If you have someone on staff who is a K8 expert, and you already maintain a K8 cluster for other applications, then you should be fine using K8 and creating a helm chart.

Hopefully it should be relatively easy to configure following our Linux documentation:

• https://docs.inedo.com/docs/installation-linux-docker-guide
• https://docs.inedo.com/docs/installation-upgrading-docker-containers

Here is some community discussion on the matter:

• https://forums.inedo.com/topic/3140/proget-manual-database-upgrade-docker-kubernetes

If you end up creating a helm chart, please do share, and we can consider it a community-provided chart :)

Best,
Alana

Hi @4391728_4499 ,

Since you've disabled "Anonymous" access to "View & Download Packages", then NuGet will also need to authenticate to the feed using Basic authentication (username/password) to view and push packages.

You can do this with the username api and the password of your API Key.

You'll need to use nuget add source to configure the username/password:
https://learn.microsoft.com/en-us/dotnet/core/tools/dotnet-nuget-add-source

Best,
Alana

Hi @nathan-wilcox_0355 ,

Thanks for clarifying; so basically I think you you'll need to parse those version numbers instead of trying to rely on on publish dates. I think this will require some kind of custom script to capture the deletion logic you need....

You may want to consider using Universal Packages, which does let you keep the last X versions of a particular package. You could then use pre-release versioning as well, which typically is what something like a CI server would create.

Hope that helps,
Alana

Hi @nathan-wilcox_0355 ,

Asset directory policies don't consider the creation date; instead, we would recommend setting a policy like "keep files downloaded in last 90 days".

That will delete everything that hasn't been downloaded in the last 90 days.

Cheers,
Alana

It's hard to say, because we haven't created our 2024 product roadmap yet :)

You can see when past versions were released if you are curious:
https://inedo.com/products/roadmap

And I'm sure you noticed but looks like this was released :)

Hi @w-repinski_1472 ,

Thank you for the suggestion!

We are considering developing our own container scanning solution, potentially in ProGet 2024, similar to ProGet Vulnerability Central (PGVC) but for containers.

But in the mean time, you may be able to add this as a VulnerabilitySource similar to Clair:
https://github.com/inedo/inedox-clair

Cheers,
Alana

Thanks so much @shayde, on a quick glance the code looks good :)

From here we can do the easy part early next week - internal reviewed, merge, build, test, deploy!

Hi @Justinvolved ,

Ah yes, getting all this modeling done sensibly is a challenge, and documenting it is a whole new pain

The main issue we're facing is that you can't create a Site in IIS without a binding; the API will simply reject it and error. This means that if you use IIS::Ensure-Site to create a site, but don't specify a binding, it will error. However IIS::Ensure-Site can update a site no problem.

This is why we originally created the Bindings property. However, it's a but challenging to use, and exhibits the behavior you describe: it "ensures" that list matches whatever is in the Site.

Our current way of thinking is this:

IIS::Ensure-Site MySite
(
    AppPool: MyPool,
    Path: C:\Websites\MySite,
    BindingProtocol: http,
    BindingHostName: app.local,
    BindingPort: 80
);

IIS::Ensure-SiteBinding
(
    Site: MySite,
    Protocol: https,
   ... ssl properties ...
);

Our "new" way of thinking is that it might make sense to allow IIS::Ensure-Site to have two sets of binding properties.

IIS::Ensure-Site MySite
(
    AppPool: MyPool,
    Path: C:\Websites\MySite,
    HttpHostName: app.local,
    HttpBindingPort: 80,
    HttpsBindingPort: 443,
    HttpsCertificateOrWhatever...
);

This seems to align with how most people want to set up a site in IIS (i.e. two bindings).

Definitely open to your feedback

Cheers,
Alana

@sebastian thanks for confirming!

I've added this as something to fix via PG-2441 and targeted it as 2013.14 (next Friday), but it's a lower-priority issue so it will may get "bumped" to the next or following depending on other issues

Hi @Justinvolved ,

What properties are you setting?

If you run Ensure-Site with the Bindings property, it will:

• update the properties of the bindings specified if needed
• delete the bindings not specified
• add the ones that don't exist

Note that you can specify a list of bindings in that property, so you could do this:

IIS::Ensure-Site test
(
    Path: E:\wwwroot\test,
    AppPool: testPool,
    Bindings: @(
        %(IPAddress: *, Port: 80, HostName: test.domain.local, Protocol: http),
        %(IPAddress: *, Port: 443, HostName: test.domain.local, Protocol: https),
     )
);

Thanks,
Alana

Hi @e-rotteveel_1850 ,

Thanks for sharing the packages for this (and that other CONDA issue); my wild-guess is that it's related to your package metadata. But we'll use your packages, attach a debugger and find out :)

Please give us a few days to investigate/resolve this, and hopefully it'll be a very easy fix.

@sebastian thanks, that's what I was hoping you could test :)

Can you check something else: can you actually download the package by the URL directly (i.e. using what the API would do?)

It should work, because I think this is just the UI incorrectly "double checking" against PGVC records. Not sure If I can explain it well, but maybe you can understand code better...

When you download a package, PGVC is first queried and then vulnerability records are added. Then, those added vulnerability records are checked against download rules.

The records are not added while browsing a package, which is why we perform that second check. However, that second check should first cross-reference PGVC records against vulnerability records...

Anyway, I wanted to confirm that was the issue - and then if so, we'll take a stab at fixing this.

Thanks so much @martijn_9956; after researching the matter further, it seems that the MERGE can be a pretty buggy statement, and its behavior seems to vary based on the operating system, SQL Server SKU, patch version, and probably the phase of the moon.

We will rewrite this procedure to use a more straight-forward UPDATE/INSERT/DELETE (like you did) via PG-2437, which will ship in next maintenance release.

@shenghong-pan_2297 that's a really old version of ProGet

I looked here to see all the changes:
https://my.inedo.com/downloads/issues?Product=ProGet&FromVersion=5.2.8&Count=all

I CTRL-F'd for API and found this:
PG-1594 5.2.13 FIX: Feed Management API fails to handle feed type codes correctly

I guess that's it?

Anyway I would upgrade and hopefully the issue goes away. Note that we're 4 major versions later (5.3 > 6.0 > 2022 > 2023), so you're pretty far behind ;)

Thanks so much for the update @martijn_9956; even 2 minutes that is a surprising amount of time to take for what should be a really basic insert/update

But I'm glad you could repro it to the proc; we can also take a stab at playing around with the procedure as well if you us the script you're running.

And just to confirm -- you were seeing that @p2 temporary was only around 1658 rows, right? That's what I would expected based on the log messages. I know we tested this with 10k+ rows at least.

Hi @sebastian ,

That's really strange... I can't reproduce this, and I can't think why it would behave this way. But the logic is kinda complex. I don't think it has to do with a PGVC vs OSS vulnerability though.

I did reproduce another bug...

However, it's related to the "block" global rule it seems:

When I change that to "allow" it works fine. I didn't experiment further, b/c I'd like to repro your specific bug and fix that as well.

Any other input on how to repro would be appreciated; maybe try re-assisng to something else.

Does it work if you override the block at the package level? You may have to pull the package to do that first.

Thanks,
Alana

@martijn_9956 thanks, let us know what you find! That is really slow for Executions_AddLogEntry; it's just an insert I think.

Can you check... what is the database recovery model for ProGet configured to? It should be SIMPLE. Not sure if that would make a difference or not, just a thought...

Hi @caterina

Sorry that issue fell off our radar; we're not great at keeping track of GitHub issues.

We haven't noticed this issue in our testing/environment yet, but I'm guessing that's b/c we haven't gone to v9?

Looking at the code, I guess it seems easy enough to support? Just a matter of iterating packages/dependencies instead of dependencies perhaps??

https://github.com/Inedo/pgscan/blob/master/Inedo.DependencyScan/NpmDependencyScanner.cs#L23

The hardest part for us is getting this tested/verified. We don't have a "problem project" ourselves yet, so we have to repro, study, fix, test, etc.

Any help here would be appreciated :)

Thanks,
Alana

Hi @martijn_9956 ,

We've seen this happen in the field a few times, and it seems to be very specific to SQL Server version, the hardware, or something like that. It's a bit of a mystery, because when we import a database backup, the migration happens really quickly. We even tried same version of SQL Server. Another user restored the database to another SQL Server, and it worked fine.

In ProGet 2023.13 (which you're using), we increased the feed migration connection timeout to 1000 seconds (up from 30 seconds), so the fact this this is still happening is totally bizarre. I wonder if you could help troubleshoot, by seeing what's happening on the SQL Server side?

Based on the log messages, the timeout is happening on when executing FeedPackages_UpdatePackageData; this procedure inputs a Table-valued parameter with 1658 rows (based on your data). Here is the C# code that invokes the query:

await this.Db.FeedPackages_UpdatePackageDataAsync(
    Feed_Id: this.FeedId,
    Packages_Table: latestVersions.Select(
        p =>
        new TableTypes.FeedPackageEntry
        {
            PackageName_Id = p.Key,
            Latest_Package_Version = p.Value.LatestVersion?.ToNormalizedUniqueString(),
            LatestStable_Package_Version = p.Value.LatestStableVersion?.ToNormalizedUniqueString(),
            Total_Download_Count = p.Value.TotalDownloads
        }),
    DeleteMissing_Indicator: false
).ConfigureAwait(false);

You can peek in SQL Server to see the code, but it's really just doing a straight-forward "upsert" into the FeedPackages table.

If you attach SQL Profiler, you should be able to see exactly what's going on. The only rough idea we have is that there's something "wrong" with the way we're doing the upsert in FeedPackages_UpdatePackageDataAsync and some version of the query analyzer is tripping over it (but not reporting a deadlock?)

Any insight would be appreciated, this one's a mystery for now

Thanks,
Alana