Posts made by stevedennis

@hwittenborn it's generally in C:\ProgramData\Inedo\SharedConfig\ProGet.config; here's information about where to find the configuration file:
https://docs.inedo.com/docs/installation-configuration-files

It's just sample code; you would need to write a C# program (or whatever language you'd like) that follows that same logic to decrypt the content stored in SecretKeys using AES128

I'm not entirely how SecretKeys are persisted, but I think either base64 or hex literals

Hi @pariv_0352,

Thanks for clarifying; looking closer, ProGet requires that X-Forwarded-Host is simply a hostname. You're right, there is no "standard" for this, but that's what ProGet does for the time being.. and if the input is invalid, then you get the error you'll see.

I would change your reverse-proxy header configuration to:

• X-Forwarded-Host: www.testdomain.com
• X-Forwarded-Port: 82

Hope that helps,
Steve

If you're looking for nuget.org-specific metadata, I recommend querying nuget.org directly; of course if you need to work-around internet access issues, you could configure a special feed/connector with no caching.

But if you're looking for latest version of a package, the registration API is your best choice. That's what Visual Studio (NuGet client) does for every package and dependency, every time a restore happens.

Hi @jw,

While we added the published-by column to the database, it seems that it's not being populated properly in all cases; we'll get it fixed by PG-2413 .

Cheers,
Steve

Hi @aivanov_3749 ,

I'm afraid the reply is the same :(

ProGet 2023 is effectively a new database architecture entirely, and while we tested every possible scenario we could imagine (as well as dozens of customer databases), some regressions are to be expected. It's also possible that there was a bug or edge case in the old retention rules, and the packages that should have been deleted weren't/

Upgrading to ProGet 2023 will automatically disable all retention rules on all feeds, and you'll be prompted to attempt a dry run before re-enabling them. The best way to troubleshoot retention rules deleting unexpected packages is to use the "dry run" feature. This will let you tweak the rules, and find which setting is behaving unexpectedly.

If you can let us know specifics or provide those execution logs, we will definitely do our best to identify the underlying cause.

Thanks,
Steve

Hi @rosario-digiovanni_1930 ,

In an ideal environment, when a user is logged into a domain-joined Windows workstation, then Visual Studio or Edge/Chrome should never prompt the user when WIA is enabled. This applies to ProGet, or any other site/webapp that uses WIA.

However, there are many things that can go wrong, and cause WIA to break. Even something as simple as an out-of-sync clock on a workstation. We've written some docs that try to explain how WIA works and give some tips on how to troubleshoot the issue:
https://docs.inedo.com/docs/various-ldap-troubleshooting#integrated-authentication-not-working

My personal opinion is that WIA was designed for a time before password managers and when everyone worked in an office without VPN. You may find it just not worthwhile to use.

NOTE: you can still use your domain credentials (i.e. Active Directory / LDAP), but users will just be required to enter them into ProGet. They can use an API key inside of Visual Studio.

Cheers,
Steve

@v-makkenze_6348 said in Reporting & Software Composition Analysis (SCA) shows many unresolved Issues:

I repackaged the Owin package but didn't relalize that that would break all my builds as the dll's are now in a 1.0.0 folder where all the project files expect them in the 1.0 folder.
I guess this would work if the projects are in sdk project format but most of them are not.

Unfortunately, a consequence of those quirky versions. Hopefully it won't be too bad to update those projects/references with a bit of search/replace :)

Hi @sebastian,

could you tell me what you mean by disabling the SCA feature in the Feed Features? I don't think I've seen this option in the Feed Features

This is new to ProGet 2023, and you can find it under the Manage Feed page:

Also: there are at least two mechanisms in ProGet to block/allow package downloads: license filters and package filters (in the feed's connector settings). What happens when you combine those filters? Is a package always blocked when it is blocked by one mechanism and allowed by the other? What happens if we'd set the default license filter rule to "Block downloads by default" and allow packages like Microsoft.* in the Nuget connector? Could Microsoft.* packages without a known license be downloaded or would they be blocked?

A package can be blocked due to vulnerabilities, licenses, connector filters, or package filters rules (i.e. white source). Any one of those will block a download, so I think in your case "Microsoft.* packages without a known license" would be blocked.

This can be overridden at a package level, FYI:

Cheers,
Steve

Hi @v-makkenze_6348 ,

Unfortunately it's a bit difficult to troubleshoot what happened with the information provided...

The best way to troubleshoot retention rules deleting unexpected packages is to use the "dry run" feature. This will let you tweak the rules, and find which setting is behaving unexpectedly.

FYI, retention rules do not consider package statistics ("download history".. i.e. records of individual downloads) but instead use "last download date" and "download count" (metadata fields on the package version). If you delete a package, and then re-add it, the "download count" would effectively reset to zero, but the "download history" records would still remain.

Hope that helps,
Steve

@guyk thanks for letting us know; unfortunately this would require a substantial change to the way we handle authentication on connectors; if we see more of this down the line we'll definitely consider it further :)

@sebastian said in Reporting & Software Composition Analysis (SCA) shows many unresolved Issues:

I just noticed that the fix seems to be offered only for "short" versions (i.e. 1.0 to 1.0.0) but not for "long" versions (i.e. 1.0.0.0 to 1.0.0). Is this intended? I think that in cases where the last version part is 0, long versions could be auto-fixed the same way as short version.

A four-part version is not considered a "quirky version" (it's still supported by NuGet), but for some reason the NuGet client/API will occasionally drop the last 0 (e.g. 1.0.0.0 -> 1.0.0), but not always (e.g. 2.1.0.0 isn't dropped?). So we didn't bother with figuring out the rules when displaying that helper-dialog.

[1] Packages with packageid:// type licenses are still reported as "Unknown License". According to PG-2381 this should have been fixed in 2023.7, but it seems that the problem still persists. When I look at the package's page, the (manually applied) license is displayed correctly, but the SCA report still does not recognize it.

Can you create a new thread/ticket for this, with some specific repro instructions/packages (or attach an SBOM so we can very easily recreate it)? This could could be related to PG-2405, but we'd want to see some specific examples of packages to test.

[2] We have a certain license type which is allowed in some feeds and blocked in other feeds. We do this to make sure that packages with that license are downloaded from the "correct" feed. This has worked fine so far. However, starting with ProGet 2023, all packages with that specific license show up as issues in our SCA reports. How can we get rid of that? Manually resolving those issues is not an option, as we are talking about ~100 affected packages on a project with daily builds.

This was actually how ProGet 2022 was supposed to work: if a package download would be blocked in at least one feed, then an issue will be created. The reason for this, pgscan (or an SBOM ) won't know/specify the feed the package is being used from.

The solution we have is to disable the "SCA feature" on the Feed Features. Would that work? We're open to other ideas, but you can see the problem we have... which feed should the analysis use? Etc.

// FYI: might be worth opening a new topic for this one, since it's a different issue as well

Thanks @sebastian, that's pretty much it :)

The underlying issue is that Visual Studio (NuGet) is referencing 1.0.0 while the actual package uses a quirky version 1.0. ProGet does not fully support quirky versions, and the SCA Feature will not try to resolve those differences.

If you have a "quirky version" of a NuGet package, ProGet 2023 will prompt you to fix it:

In the case of the above, I just created a blank NuGet feed and downloaded "Owin 1.0.0". Then Owin 1.0 appeared in the feed. Anyway, once you fix the quirky versions its should work fine.

Cheers,
Steve

Hi @dionc_5568 ,

Great question; I've updated the documentation as follows:

Usage (CLI/tool)

Execute pgscan with the identify command. For example, to generate an SBOM and submit the dependencies of v1.0.0 the MyLibrary project to ProGet:

_
pgscan identify --input=MyLibrary.csproj --proget-url=https://proget.local --version=1.0.0
_

Note that the identify command requires ProGet 2022 and later. If you're using ProGet 6.0, you'll need to use the now-deprecated publish command; see the old version of this README to learn how.

Hopefully that makes it clear. But yes, please just use identify. It uses a different API that's much slower and will be removed in later versions of ProGet.

Hi @espen-blikstad_6547,

Please check the Web.BaseUrl property under advanced settings; when that is set, the X-Forwarded headers will not be used.

Thanks,
Steve

@guyk there were some changes to containers with the implicit library changes in ProGet 2023, however we didn't encounter this error when testing

can you confirm that this works in ProGet 2022, but not 2022? What architecture are you using, sometimes it's related to so-called "fat manifests"?

Hi @jimbobmcgee,

I haven't seen this option in too many tools that use SSH; and actually it's the first time I've even heard of this as an option. But I'm not an SSH expert by anymeans, so no idea if this is common.

We use libssh2, and I have no idea if it's technically possible. We use the libssh2_userauth_password_ex(IntPtr session, IntPtr username, uint username_len, IntPtr password, uint password_len, IntPtr passwd_change_cb) method to authenticate.

Unless it's absolutely trivial to change (like a simple flag on libssh2 or something), it probably doesn't make sense for us to invest in this feature... unless it came from a from a paid user trying to solve a specific problem/usecase that we could work together on.

SSH is already difficult to support/maintain, so this would add more complexity to testing, debugging, documentation, etc... and we've got enough of that already heh

Cheers,
Steve

Hi @markus-karthaus_8928 ,

I haven't seen that error before.

It's coming from the .NET cryptography library, and when searching for the error message ("The key contents do not contain a PEM, the content is malformed, or the key does not match the certificate."), I'm not getting any hints on specific "gotchas" or ways to resolve the error.

This is where the error is coming from:
https://learn.microsoft.com/en-us/dotnet/api/system.security.cryptography.x509certificates.x509certificate2.createfromencryptedpem?view=net-6.0

So I think the issue must be that the PEM is "invalid" - at least according to the library we're using. I wish I had more information on that, but perhaps you can try a different way to generate it, or try a different way to configure HTTPS.

Cheers,
Steve

We don't maintain the Jenkins extension, but I wonder if you entered an API Key? It looks blank on that screen...

I'm really not sure what the error means otherwise ...

Hi @sebastian ,

[1] That definitely doesn't sound right; that didn't happen when we tested, so we'll have to check that out, it could be a bug...

[2] 274/566 seems awfully high; several do not have scores, but since we have to compute the score ourselves with equations like these, it's very possible that the underlying data isn't formatted perfect or there's a bug somewhere -- can you share the examples you found so we can investigate?

[3] This is expected; it seems that many (or most) vulnerabilities in the database do not have a conspicuous CVE number (perhaps they're not CVEs??), and in those cases, the descriptions are very thorough... it's a huge dataset so we're still learning what's in it.

Cheers,
Steve

Hi @rie_6529 ,

It's hard to say; one possibility is that your server is overloaded. If you have multiple build servers, multiple connectors, etc., then it's like a denial of service attack. Under Admin > Advanced Settings, there is a setting called Web.ConcurrentRequestLimit; I would configure that to 500.

Next, I would investigate SQL Server, and see where/what queries are taking a ton of time.

With the new indexing system in ProGet 2023, it's possible we missed a SQL index or something. We tested with absolutely massive datasets, but it's hard to say.

Cheers,
Steve

Hi @reincarnator247_4909 , please submit a ticket for this with as many details as you can (types of error messages, configuration, etc.), so we can properly review and give advice. It's very case-by-case.

Hi @rie_6529 ,

Did you allow the data migration to complete?

That can take a significant amount of time if you have a lot of packages (which you likely do).

That's the only thing that would cause an issue like that which I can think of. Here's the full upgrade notes:
https://docs.inedo.com/docs/proget-upgrade-2023

Steve

Hi @sebastian,

Thanks for clarifying that :)

[1] auto-assess should work just fine; that would be our recommendation anyway

[2] this is what we thought too, but there were just so few that this would have worked on that we gave up

So in that case, you can just enable PGVC, enabled download blocking on the feed, and then you'll get all the new PGVC vulnerabilities added to the system after running a Vulnerability Download scheduld job.

If you delete the OSSIndex source, then all the vulnerabilities/assessments will be deleted.

There was a very long-standing bug where ProGet wouldn't update or delete a vulnerability if it was updated/deleted at the source. We fixed that in 2023.

Now I can't say for certain if that's what happened here... but we noticed that some similar erroneous vulnerabilities -- like a vulnerability with a mangled title or some other data entry problem -- disappeared after a nightly scan.

FYI - expiration dates won't delete the assessments, it'll just consider them invalid

Hope that helps!

Hi @jw ,

The version number is a bit buried in the logs I believe, but it sounds like things are working now... and it's too much of a guessing game to figure out what might have happened now.

Cheers,
Steve

Hi @sebastian ,

We actually used your set to analyze this - and we just couldn't come up with a solution.

The PGVC found more total vulnerabilities than OSS Index did, but without doing some really complex code or machine learning something, we couldn't figure out an simple way to reconcile the two datasets.

Many of the OSS Index vulnerabilities didn't list a CVE number, and the titles and descriptions were different - but it seemed like they were talking about the same problem in the same package.

We gave up after that. Open to ideas for sure!

Thanks,
Steve

Hi @cshipley_6136 ,

Since it sounds like there's some logs/sensitive info.. I just submitted a ticket on your behalf (EDO-9257), so we'll work to troubleshoot from there!

Cheers,
Steve

Hi @martin-helgesen_8100 ,

Nothing new I'm afraid; you're the second user who's inquired about this so far.

We do have several customers using Terraform and ProGet, either using universal packages (like @jeff-miles_5073 suggested) or using Asset Directories. We asked them a while back if they had interest for a "proper" Private Registry, and the response was that even if it was available... they likely wouldn't use it because it would involve changing their system, and they saw no benefit to a feed.

We know basically nothing about Terraform and have done no research into the costs/complexity of implementing a registry. Do other products/vendors support this?

The first "red flag" on my end is this:

Terraform can use versioned modules from any service that implements the registry API. The Terraform open source project does not provide a server implementation, but we welcome community members to create their own private registries by following the published protocol.

In our experience, documentation is almost always outdated and inaccurate. So this means lots of reverse-engineering of API protocols and lots of debugging.

Not saying it's impossible, but we'd definitely need community support to make it happen. Check out how the rpm feeds came to be - if it's something you can prototype in Asset Directories (for example), that could go a long way in making it a first-class feed in ProGet :)

Cheers,
Steve

Hi @msimkin_1572 , good idea! I just added a small note at the bottom, where we specify manifest. To be honest, I didn't even know that was possible

Hi @itsoftware_2704 ,

We don't have instructions specific for Docker containers, but you basically just need to run the ProGet.Service.exe resetadminpassword command on the container. And then restart the container.

Here is more information on what this does:;
https://docs.inedo.com/docs/installation-security-ldap-active-directory

I'm not very well versed in Docker, so I don't know the exact way to run that command - but in the interest of giving you an answer right away, I wanted to share this. Hoping you know how to though :)

Feel free to share how you ran the command, in case someone else runs into this thread!

Cheers,
Steve

Hi @richard-carpenter_9650 ,

Thanks for confirming that; that is really strange. I just tried it msyefl and got the same result.

The technical answer is that PowerShell gallery is sorting the feed alphabetically, and is only returning the top 100 entries. Of those entries, all but two are unlisted. So that's all you see.

Not a great UX, so hopefully it goes away (i.e. they return the default sort order by downloads), or maybe we can sort differently ourselves. We're not too keen to change this, since it's the "very sensitive" legacy NuGet v2 API, but perhaps in v2023 we can alter this.

Cheers,
Steve

Hi @richard-carpenter_9650 ,

When you add a connector to your feed, it will display the packages from those connectors. There's no way to "remove" these packages, since they're on the PowerShell Gallery. But you can always add "connector filters" to show a subset of packages (by name).

The PowerShell recently changed the API sort order to show packages alphabetically (instead of by download count), which is why you see those packages first.

Cheers,
Steve

Hi @jwest_6990

What Windows server version is Otter installed on? That might help us narrow this down.

We have seen that error if RSAT is not enabled:
https://learn.microsoft.com/en-us/troubleshoot/windows-server/system-management-components/remote-server-administration-tools

Cheers,
Steve

Hi @e-rotteveel_1850 ,

Thanks for all the details here! Very helpful, especially since we know very little about Conda.

In our code, we have a WriteChannelData and a WriteRepoDataAsync method, which write out these files on demand using the Newtonsoft.Json library for this.

So, I just specified a StringEscapeHandling of EscapeNonAscii, which will escape all properties. I don't think that will be a problem.

The change is PG-2295, and it will ship in next maintenance release (Friday, Mar 10). If you'd like to try in a prerelease, just let me know and I can promote our CI-build so you can use it sooner.

Cheers,
Steve

Hi @priyanka-m_4184 ,

That message usually indicates a network problem; are you seeing any issues on the ProGet side of things?

Can you try pushing a package with PowerShell directly instead of using upack.exe? That way you can see an error more clearly, hopefully.

Cheers,
Steve

Hi @OtterFanboy ,

I was able to identify/fix this as BM-3818; looks like the issue was purging a deployed build from a deployed release causes the overview page to crash.

You can work-around this by finding that release on the main "Releases" tab and purging it too.

Cheers,
Steve

@OtterFanboy forgot to mention, we did plan to build a first-class Gitea integration, so you could browse your repositories (just like GitHub etc.), but didn't get it in time. It's still on our list!

Hi @OtterFanboy ,

Thanks for all the details :)

Looks like this is a regression/bug with browsing Generic Git repositories. That's a brand-new 2022 feature, and we're still working out kinks with it. We'll investigate and get it fixed ASAP!

Stay tuned :)

Cheers,
Steve

Hi @testintergraph_2317,

We haven't specifically tested w/ OpenLDAP (I think), but the integration does work with other providers. One compatibility issue seems to that sAMAccountName isn't the Username property on all LDAP servers.

However, in v4 of the LDAP provider, you can now customize these queries. Have you tried this yet? https://docs.inedo.com/docs/en/various-ldap-v4-advanced

In any case, we'd be happy to work with you on getting OpenLDAP working. Just let us know what specific issues you're having, and we can patching, etc.

Of course, you're welcome to try modifying the code yourself... and if you want to try I would start by forking the InedoCore extension and then using a custom build of that (just use a version number higher than published). Make sure to delete that custom version once we accept pullrequest or publish version with identical changes.

If you want a totally custom extension, just make one witha different name.

We don't have instructions for custom extensions with Docker, but it follows the same process as Windows (just restart container instead of AppPool/Service): https://docs.inedo.com/docs/proget-administration-extensions#manual-installation

Cheers,
Steve

Hi @testintergraph_2317 ,

I'm afraid that sample code is ancient and won't work in any modern version of ProGet...

Can you let me know how you found that repository? I've just now archived the repository.

Can you share the use case for implementing a custom directory? This is very rare requirement...

In any case, you can look at the user directories we implement here: https://github.com/Inedo/inedox-inedocore/tree/master/InedoCore/InedoExtension/UserDirectories

Cheers,
Steve

Hi @joacim-svensson_8194 ,

That should have worked, but it's of course possible there's a bug.

Can you confirm steps?

Is this basically what you did?

1. Create Two feeds (Feed1 and Feed2), download a vulnerable package in each feed
2. Create Two Vuln Sources (OssIndex1, OssIndex2), associate each to each feed
3. Run the "Vuln Downloader" Job, and see two identical vulnerabilities added to ProGet
4. Assess the vulnerabilities differently (1 = Block, 2= Ignore)
5. Package should Blocked on Feed1, and allowed on Feed2

I want to make sure we're following the steps you did, so we can test this.

Cheers,
Steve

Hi @osnibjunior ,

Oh I see -- I must have missed the part, "it seems it manages to list the package that is in Azure Artifact but fails to download when requested resulting in a 404."

Unfortunately it's hard to say what's wrong; Azure DevOps package registries are pretty buggy, and don't always follow the NuGet API specs.... we've worked-around a few of their glitches, but maybe they added one again?

Anyway, if you can provide us with (1) the Connector URL and PAT token and (2) a package that you can't download, we can attach a debugger and see if it's something we can identify/fix.

You can email that to support at inedo dot com, but please enter [QA-1021] in the subject so we can find the email. And please let us know when you send the email, as we don't regularly monitor that box.

Cheers,
Steve

Hi @smiller-circ_2844 ,

Product activation is required. If you're unable to keep the same MachineID, then you'll need to get an activation code from my.inedo.com.

While it's not officially supported and isn't documented, you could "listen" to the activation request that ProGet makes by using a proxy server like Fiddler, and make the same API request with your key. You would then need to add that value to the database.

I don't want to give specific instructions on how to do this, because like I mentioned it's not officially supported -- but it's not "too difficult" to do, if you're familiar with the above.

Cheers,
Steve

Hi @osnibjunior ,

Did you follow our AzureDevOps Migration tutorial?

Based on the information you've entered in the connector, I don't think it will work because your Connector URL looks incorrect (it should look something like https://pkgs.dev.azure.com/kharnagy/kharnagy/_packaging/internal-nuget/nuget/v3/index.json).

I'm not sure about the username/password but I understand you need to create a PAT and use that as well. I think it doesn't matter where you put the PAT (username or password field).

There's nothing in v2022 that should have impacted this

Cheers,
Steve

Hi @OtterFanboy ,

Under the hood, Otter runs PowerShell scripts through a custom PowerShell host that runs within the agent's process. While this generally leads to performance improvements, it can lead to memory leaks or worse if PowerShell scripts are created improperly. It sounds like this is the case here -- especially if your scripts are prompting for input, not terminating, etc.

This is where Process Isolation can come in. Instead of using the agent's process to run the PowerShell script, you can use OtterScript's with isolation feature. This creates a new process that runs the script.

Here is an example of an OtterScript plan that runs the same PowerShell script in different processes, even though it is the same script and runs on the same agent:

for server targetsv
{
    with isolation
    {
        PsExec >>Write-Host "Process ID: $pid">>;
    }
    with isolation
    {
        PsExec >>Write-Host "Process ID: $pid">>;
    }
}

Note that you can also specify timeouts with with timeout=1000 as well.

Cheers,
Steve

hi @cshipley_6136,

We chatted a bit about this as a team today.

We believe that the application failed during initialization (i.e. the first web request after starting the service); when this happens, the error message will be returned until you restart the application (container/service).

This is by design; there's no way to guess where or why the initialization failed, and if we "crash" the application (and trigger a restart), it can end up in a endless loop and make diagnosing really difficult.

In this case, the error is network related (server name not found). That error can be transient when you have network problems, or permanent if it's a typo in the connection string.

Since restarting the application (container) fixed it, then it was a transient network issue that happened during initialization.

Hope this helps,
Steve

Hi @joacim-svensson_8194 ,

A vulnerability is tied to a vulnerability source (you can see the source name on the vulnerability page), and a vulnerability source is tied to one or more feeds.

So when you create a second vulnerability source, then you'll see two sets of vulnerabilities and be able to assess each vulnerability differently.

Cheers,
Steve

Hi @cshipley_6136 ,

Based on this, it looks like there are some network problems between ProGet and your SQL Server:

A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections.

It's really easy to reproduce this error (just make a typo in the server name) but when it's intermittently happening, it means the problem is happening within the network, between the two servers.

Unfortunately you won't be able to find any useful information on the SQL Server side, since the connection cannot be established. And the same is true on the ProGet side, for the same reason.

This is definitely something to bring up with your Network/Operations team. It could be anything from DNS issues to router configuration to a bad/failing switch.

Best,
Steve

Hi @cshipley_6136

Can you give more details about the error messages you were seeing? You mention it's a login error, like a bad SQL password or account or something?

The health check (/health) runs the Feeds_GetFeeds procedure... so it should have given a 500 error / crashed if the database couldn't be connected.

We'll try to reproduce the failed failure, and get it fixed asap.

Cheers,
Steve

Hi @tyler-engelhardt_8011 ,

I looked at the docs briefly, but I'm really not familiar enough Ansible to know what could be the issue, or how they try to run Chocolatey behind-the-scenes. It's most certainly a configuration issue in your Ansible script though.

So far as I can tell, Chocolatey is not installed on the server, so Ansible is trying to look for a file called install.ps1 on the ProGet server (which obviously wouldn't be there) to install Chocolatey. That file is located here: https://community.chocolatey.org/install.ps1

I would try contacting the Ansiblesupport team or play around with the parameters like bootstrap_script. Please let us know what you find.

Cheers,
Steve