Posts made by stevedennis

Hi @v-makkenze_6348 ,

Unfortunately it's a bit difficult to troubleshoot what happened with the information provided...

The best way to troubleshoot retention rules deleting unexpected packages is to use the "dry run" feature. This will let you tweak the rules, and find which setting is behaving unexpectedly.

FYI, retention rules do not consider package statistics ("download history".. i.e. records of individual downloads) but instead use "last download date" and "download count" (metadata fields on the package version). If you delete a package, and then re-add it, the "download count" would effectively reset to zero, but the "download history" records would still remain.

Hope that helps,
Steve

@guyk thanks for letting us know; unfortunately this would require a substantial change to the way we handle authentication on connectors; if we see more of this down the line we'll definitely consider it further :)

@sebastian said in Reporting & Software Composition Analysis (SCA) shows many unresolved Issues:

I just noticed that the fix seems to be offered only for "short" versions (i.e. 1.0 to 1.0.0) but not for "long" versions (i.e. 1.0.0.0 to 1.0.0). Is this intended? I think that in cases where the last version part is 0, long versions could be auto-fixed the same way as short version.

A four-part version is not considered a "quirky version" (it's still supported by NuGet), but for some reason the NuGet client/API will occasionally drop the last 0 (e.g. 1.0.0.0 -> 1.0.0), but not always (e.g. 2.1.0.0 isn't dropped?). So we didn't bother with figuring out the rules when displaying that helper-dialog.

[1] Packages with packageid:// type licenses are still reported as "Unknown License". According to PG-2381 this should have been fixed in 2023.7, but it seems that the problem still persists. When I look at the package's page, the (manually applied) license is displayed correctly, but the SCA report still does not recognize it.

Can you create a new thread/ticket for this, with some specific repro instructions/packages (or attach an SBOM so we can very easily recreate it)? This could could be related to PG-2405, but we'd want to see some specific examples of packages to test.

[2] We have a certain license type which is allowed in some feeds and blocked in other feeds. We do this to make sure that packages with that license are downloaded from the "correct" feed. This has worked fine so far. However, starting with ProGet 2023, all packages with that specific license show up as issues in our SCA reports. How can we get rid of that? Manually resolving those issues is not an option, as we are talking about ~100 affected packages on a project with daily builds.

This was actually how ProGet 2022 was supposed to work: if a package download would be blocked in at least one feed, then an issue will be created. The reason for this, pgscan (or an SBOM ) won't know/specify the feed the package is being used from.

The solution we have is to disable the "SCA feature" on the Feed Features. Would that work? We're open to other ideas, but you can see the problem we have... which feed should the analysis use? Etc.

// FYI: might be worth opening a new topic for this one, since it's a different issue as well

Thanks @sebastian, that's pretty much it :)

The underlying issue is that Visual Studio (NuGet) is referencing 1.0.0 while the actual package uses a quirky version 1.0. ProGet does not fully support quirky versions, and the SCA Feature will not try to resolve those differences.

If you have a "quirky version" of a NuGet package, ProGet 2023 will prompt you to fix it:

In the case of the above, I just created a blank NuGet feed and downloaded "Owin 1.0.0". Then Owin 1.0 appeared in the feed. Anyway, once you fix the quirky versions its should work fine.

Cheers,
Steve

Hi @dionc_5568 ,

Great question; I've updated the documentation as follows:

Usage (CLI/tool)

Execute pgscan with the identify command. For example, to generate an SBOM and submit the dependencies of v1.0.0 the MyLibrary project to ProGet:

_
pgscan identify --input=MyLibrary.csproj --proget-url=https://proget.local --version=1.0.0
_

Note that the identify command requires ProGet 2022 and later. If you're using ProGet 6.0, you'll need to use the now-deprecated publish command; see the old version of this README to learn how.

Hopefully that makes it clear. But yes, please just use identify. It uses a different API that's much slower and will be removed in later versions of ProGet.

Hi @espen-blikstad_6547,

Please check the Web.BaseUrl property under advanced settings; when that is set, the X-Forwarded headers will not be used.

Thanks,
Steve

@guyk there were some changes to containers with the implicit library changes in ProGet 2023, however we didn't encounter this error when testing

can you confirm that this works in ProGet 2022, but not 2022? What architecture are you using, sometimes it's related to so-called "fat manifests"?

Hi @jimbobmcgee,

I haven't seen this option in too many tools that use SSH; and actually it's the first time I've even heard of this as an option. But I'm not an SSH expert by anymeans, so no idea if this is common.

We use libssh2, and I have no idea if it's technically possible. We use the libssh2_userauth_password_ex(IntPtr session, IntPtr username, uint username_len, IntPtr password, uint password_len, IntPtr passwd_change_cb) method to authenticate.

Unless it's absolutely trivial to change (like a simple flag on libssh2 or something), it probably doesn't make sense for us to invest in this feature... unless it came from a from a paid user trying to solve a specific problem/usecase that we could work together on.

SSH is already difficult to support/maintain, so this would add more complexity to testing, debugging, documentation, etc... and we've got enough of that already heh

Cheers,
Steve

Hi @markus-karthaus_8928 ,

I haven't seen that error before.

It's coming from the .NET cryptography library, and when searching for the error message ("The key contents do not contain a PEM, the content is malformed, or the key does not match the certificate."), I'm not getting any hints on specific "gotchas" or ways to resolve the error.

This is where the error is coming from:
https://learn.microsoft.com/en-us/dotnet/api/system.security.cryptography.x509certificates.x509certificate2.createfromencryptedpem?view=net-6.0

So I think the issue must be that the PEM is "invalid" - at least according to the library we're using. I wish I had more information on that, but perhaps you can try a different way to generate it, or try a different way to configure HTTPS.

Cheers,
Steve

We don't maintain the Jenkins extension, but I wonder if you entered an API Key? It looks blank on that screen...

I'm really not sure what the error means otherwise ...

Hi @sebastian ,

[1] That definitely doesn't sound right; that didn't happen when we tested, so we'll have to check that out, it could be a bug...

[2] 274/566 seems awfully high; several do not have scores, but since we have to compute the score ourselves with equations like these, it's very possible that the underlying data isn't formatted perfect or there's a bug somewhere -- can you share the examples you found so we can investigate?

[3] This is expected; it seems that many (or most) vulnerabilities in the database do not have a conspicuous CVE number (perhaps they're not CVEs??), and in those cases, the descriptions are very thorough... it's a huge dataset so we're still learning what's in it.

Cheers,
Steve

Hi @rie_6529 ,

It's hard to say; one possibility is that your server is overloaded. If you have multiple build servers, multiple connectors, etc., then it's like a denial of service attack. Under Admin > Advanced Settings, there is a setting called Web.ConcurrentRequestLimit; I would configure that to 500.

Next, I would investigate SQL Server, and see where/what queries are taking a ton of time.

With the new indexing system in ProGet 2023, it's possible we missed a SQL index or something. We tested with absolutely massive datasets, but it's hard to say.

Cheers,
Steve

Hi @reincarnator247_4909 , please submit a ticket for this with as many details as you can (types of error messages, configuration, etc.), so we can properly review and give advice. It's very case-by-case.

Hi @rie_6529 ,

Did you allow the data migration to complete?

That can take a significant amount of time if you have a lot of packages (which you likely do).

That's the only thing that would cause an issue like that which I can think of. Here's the full upgrade notes:
https://docs.inedo.com/docs/proget-upgrade-2023

Steve

Hi @sebastian,

Thanks for clarifying that :)

[1] auto-assess should work just fine; that would be our recommendation anyway

[2] this is what we thought too, but there were just so few that this would have worked on that we gave up

So in that case, you can just enable PGVC, enabled download blocking on the feed, and then you'll get all the new PGVC vulnerabilities added to the system after running a Vulnerability Download scheduld job.

If you delete the OSSIndex source, then all the vulnerabilities/assessments will be deleted.

There was a very long-standing bug where ProGet wouldn't update or delete a vulnerability if it was updated/deleted at the source. We fixed that in 2023.

Now I can't say for certain if that's what happened here... but we noticed that some similar erroneous vulnerabilities -- like a vulnerability with a mangled title or some other data entry problem -- disappeared after a nightly scan.

FYI - expiration dates won't delete the assessments, it'll just consider them invalid

Hope that helps!

Hi @jw ,

The version number is a bit buried in the logs I believe, but it sounds like things are working now... and it's too much of a guessing game to figure out what might have happened now.

Cheers,
Steve

Hi @sebastian ,

We actually used your set to analyze this - and we just couldn't come up with a solution.

The PGVC found more total vulnerabilities than OSS Index did, but without doing some really complex code or machine learning something, we couldn't figure out an simple way to reconcile the two datasets.

Many of the OSS Index vulnerabilities didn't list a CVE number, and the titles and descriptions were different - but it seemed like they were talking about the same problem in the same package.

We gave up after that. Open to ideas for sure!

Thanks,
Steve

Hi @cshipley_6136 ,

Since it sounds like there's some logs/sensitive info.. I just submitted a ticket on your behalf (EDO-9257), so we'll work to troubleshoot from there!

Cheers,
Steve

Hi @martin-helgesen_8100 ,

Nothing new I'm afraid; you're the second user who's inquired about this so far.

We do have several customers using Terraform and ProGet, either using universal packages (like @jeff-miles_5073 suggested) or using Asset Directories. We asked them a while back if they had interest for a "proper" Private Registry, and the response was that even if it was available... they likely wouldn't use it because it would involve changing their system, and they saw no benefit to a feed.

We know basically nothing about Terraform and have done no research into the costs/complexity of implementing a registry. Do other products/vendors support this?

The first "red flag" on my end is this:

Terraform can use versioned modules from any service that implements the registry API. The Terraform open source project does not provide a server implementation, but we welcome community members to create their own private registries by following the published protocol.

In our experience, documentation is almost always outdated and inaccurate. So this means lots of reverse-engineering of API protocols and lots of debugging.

Not saying it's impossible, but we'd definitely need community support to make it happen. Check out how the rpm feeds came to be - if it's something you can prototype in Asset Directories (for example), that could go a long way in making it a first-class feed in ProGet :)

Cheers,
Steve

Hi @msimkin_1572 , good idea! I just added a small note at the bottom, where we specify manifest. To be honest, I didn't even know that was possible

Hi @itsoftware_2704 ,

We don't have instructions specific for Docker containers, but you basically just need to run the ProGet.Service.exe resetadminpassword command on the container. And then restart the container.

Here is more information on what this does:;
https://docs.inedo.com/docs/installation-security-ldap-active-directory

I'm not very well versed in Docker, so I don't know the exact way to run that command - but in the interest of giving you an answer right away, I wanted to share this. Hoping you know how to though :)

Feel free to share how you ran the command, in case someone else runs into this thread!

Cheers,
Steve

Hi @richard-carpenter_9650 ,

Thanks for confirming that; that is really strange. I just tried it msyefl and got the same result.

The technical answer is that PowerShell gallery is sorting the feed alphabetically, and is only returning the top 100 entries. Of those entries, all but two are unlisted. So that's all you see.

Not a great UX, so hopefully it goes away (i.e. they return the default sort order by downloads), or maybe we can sort differently ourselves. We're not too keen to change this, since it's the "very sensitive" legacy NuGet v2 API, but perhaps in v2023 we can alter this.

Cheers,
Steve

Hi @richard-carpenter_9650 ,

When you add a connector to your feed, it will display the packages from those connectors. There's no way to "remove" these packages, since they're on the PowerShell Gallery. But you can always add "connector filters" to show a subset of packages (by name).

The PowerShell recently changed the API sort order to show packages alphabetically (instead of by download count), which is why you see those packages first.

Cheers,
Steve

Hi @jwest_6990

What Windows server version is Otter installed on? That might help us narrow this down.

We have seen that error if RSAT is not enabled:
https://learn.microsoft.com/en-us/troubleshoot/windows-server/system-management-components/remote-server-administration-tools

Cheers,
Steve

Hi @e-rotteveel_1850 ,

Thanks for all the details here! Very helpful, especially since we know very little about Conda.

In our code, we have a WriteChannelData and a WriteRepoDataAsync method, which write out these files on demand using the Newtonsoft.Json library for this.

So, I just specified a StringEscapeHandling of EscapeNonAscii, which will escape all properties. I don't think that will be a problem.

The change is PG-2295, and it will ship in next maintenance release (Friday, Mar 10). If you'd like to try in a prerelease, just let me know and I can promote our CI-build so you can use it sooner.

Cheers,
Steve

Hi @priyanka-m_4184 ,

That message usually indicates a network problem; are you seeing any issues on the ProGet side of things?

Can you try pushing a package with PowerShell directly instead of using upack.exe? That way you can see an error more clearly, hopefully.

Cheers,
Steve

Hi @OtterFanboy ,

I was able to identify/fix this as BM-3818; looks like the issue was purging a deployed build from a deployed release causes the overview page to crash.

You can work-around this by finding that release on the main "Releases" tab and purging it too.

Cheers,
Steve

@OtterFanboy forgot to mention, we did plan to build a first-class Gitea integration, so you could browse your repositories (just like GitHub etc.), but didn't get it in time. It's still on our list!

Hi @OtterFanboy ,

Thanks for all the details :)

Looks like this is a regression/bug with browsing Generic Git repositories. That's a brand-new 2022 feature, and we're still working out kinks with it. We'll investigate and get it fixed ASAP!

Stay tuned :)

Cheers,
Steve

Hi @testintergraph_2317,

We haven't specifically tested w/ OpenLDAP (I think), but the integration does work with other providers. One compatibility issue seems to that sAMAccountName isn't the Username property on all LDAP servers.

However, in v4 of the LDAP provider, you can now customize these queries. Have you tried this yet? https://docs.inedo.com/docs/en/various-ldap-v4-advanced

In any case, we'd be happy to work with you on getting OpenLDAP working. Just let us know what specific issues you're having, and we can patching, etc.

Of course, you're welcome to try modifying the code yourself... and if you want to try I would start by forking the InedoCore extension and then using a custom build of that (just use a version number higher than published). Make sure to delete that custom version once we accept pullrequest or publish version with identical changes.

If you want a totally custom extension, just make one witha different name.

We don't have instructions for custom extensions with Docker, but it follows the same process as Windows (just restart container instead of AppPool/Service): https://docs.inedo.com/docs/proget-administration-extensions#manual-installation

Cheers,
Steve

Hi @testintergraph_2317 ,

I'm afraid that sample code is ancient and won't work in any modern version of ProGet...

Can you let me know how you found that repository? I've just now archived the repository.

Can you share the use case for implementing a custom directory? This is very rare requirement...

In any case, you can look at the user directories we implement here: https://github.com/Inedo/inedox-inedocore/tree/master/InedoCore/InedoExtension/UserDirectories

Cheers,
Steve

Hi @joacim-svensson_8194 ,

That should have worked, but it's of course possible there's a bug.

Can you confirm steps?

Is this basically what you did?

1. Create Two feeds (Feed1 and Feed2), download a vulnerable package in each feed
2. Create Two Vuln Sources (OssIndex1, OssIndex2), associate each to each feed
3. Run the "Vuln Downloader" Job, and see two identical vulnerabilities added to ProGet
4. Assess the vulnerabilities differently (1 = Block, 2= Ignore)
5. Package should Blocked on Feed1, and allowed on Feed2

I want to make sure we're following the steps you did, so we can test this.

Cheers,
Steve

Hi @osnibjunior ,

Oh I see -- I must have missed the part, "it seems it manages to list the package that is in Azure Artifact but fails to download when requested resulting in a 404."

Unfortunately it's hard to say what's wrong; Azure DevOps package registries are pretty buggy, and don't always follow the NuGet API specs.... we've worked-around a few of their glitches, but maybe they added one again?

Anyway, if you can provide us with (1) the Connector URL and PAT token and (2) a package that you can't download, we can attach a debugger and see if it's something we can identify/fix.

You can email that to support at inedo dot com, but please enter [QA-1021] in the subject so we can find the email. And please let us know when you send the email, as we don't regularly monitor that box.

Cheers,
Steve

Hi @smiller-circ_2844 ,

Product activation is required. If you're unable to keep the same MachineID, then you'll need to get an activation code from my.inedo.com.

While it's not officially supported and isn't documented, you could "listen" to the activation request that ProGet makes by using a proxy server like Fiddler, and make the same API request with your key. You would then need to add that value to the database.

I don't want to give specific instructions on how to do this, because like I mentioned it's not officially supported -- but it's not "too difficult" to do, if you're familiar with the above.

Cheers,
Steve

Hi @osnibjunior ,

Did you follow our AzureDevOps Migration tutorial?

Based on the information you've entered in the connector, I don't think it will work because your Connector URL looks incorrect (it should look something like https://pkgs.dev.azure.com/kharnagy/kharnagy/_packaging/internal-nuget/nuget/v3/index.json).

I'm not sure about the username/password but I understand you need to create a PAT and use that as well. I think it doesn't matter where you put the PAT (username or password field).

There's nothing in v2022 that should have impacted this

Cheers,
Steve

Hi @OtterFanboy ,

Under the hood, Otter runs PowerShell scripts through a custom PowerShell host that runs within the agent's process. While this generally leads to performance improvements, it can lead to memory leaks or worse if PowerShell scripts are created improperly. It sounds like this is the case here -- especially if your scripts are prompting for input, not terminating, etc.

This is where Process Isolation can come in. Instead of using the agent's process to run the PowerShell script, you can use OtterScript's with isolation feature. This creates a new process that runs the script.

Here is an example of an OtterScript plan that runs the same PowerShell script in different processes, even though it is the same script and runs on the same agent:

for server targetsv
{
    with isolation
    {
        PsExec >>Write-Host "Process ID: $pid">>;
    }
    with isolation
    {
        PsExec >>Write-Host "Process ID: $pid">>;
    }
}

Note that you can also specify timeouts with with timeout=1000 as well.

Cheers,
Steve

hi @cshipley_6136,

We chatted a bit about this as a team today.

We believe that the application failed during initialization (i.e. the first web request after starting the service); when this happens, the error message will be returned until you restart the application (container/service).

This is by design; there's no way to guess where or why the initialization failed, and if we "crash" the application (and trigger a restart), it can end up in a endless loop and make diagnosing really difficult.

In this case, the error is network related (server name not found). That error can be transient when you have network problems, or permanent if it's a typo in the connection string.

Since restarting the application (container) fixed it, then it was a transient network issue that happened during initialization.

Hope this helps,
Steve

Hi @joacim-svensson_8194 ,

A vulnerability is tied to a vulnerability source (you can see the source name on the vulnerability page), and a vulnerability source is tied to one or more feeds.

So when you create a second vulnerability source, then you'll see two sets of vulnerabilities and be able to assess each vulnerability differently.

Cheers,
Steve

Hi @cshipley_6136 ,

Based on this, it looks like there are some network problems between ProGet and your SQL Server:

A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections.

It's really easy to reproduce this error (just make a typo in the server name) but when it's intermittently happening, it means the problem is happening within the network, between the two servers.

Unfortunately you won't be able to find any useful information on the SQL Server side, since the connection cannot be established. And the same is true on the ProGet side, for the same reason.

This is definitely something to bring up with your Network/Operations team. It could be anything from DNS issues to router configuration to a bad/failing switch.

Best,
Steve

Hi @cshipley_6136

Can you give more details about the error messages you were seeing? You mention it's a login error, like a bad SQL password or account or something?

The health check (/health) runs the Feeds_GetFeeds procedure... so it should have given a 500 error / crashed if the database couldn't be connected.

We'll try to reproduce the failed failure, and get it fixed asap.

Cheers,
Steve

Hi @tyler-engelhardt_8011 ,

I looked at the docs briefly, but I'm really not familiar enough Ansible to know what could be the issue, or how they try to run Chocolatey behind-the-scenes. It's most certainly a configuration issue in your Ansible script though.

So far as I can tell, Chocolatey is not installed on the server, so Ansible is trying to look for a file called install.ps1 on the ProGet server (which obviously wouldn't be there) to install Chocolatey. That file is located here: https://community.chocolatey.org/install.ps1

I would try contacting the Ansiblesupport team or play around with the parameters like bootstrap_script. Please let us know what you find.

Cheers,
Steve

Hi @lm ,

We don't track that information on the package history page at this time; you'll have to find it under Admin > Events.

It's definitely something we can consider with/after v2023, once we've centralized information about packages in the database. I'll make a note of that!

Cheers,
Steve

Hi @lm,

Unfortunately it's not trivial to get a reliable URL for the package within the webhook module/code now, so it's not something that's easy to add on our end. The URL you have works for NuGet, but for other formats (Ruby, Debian, etc.) you need to include more information I believe.

After v2023, where we plan to centralize a lot of this, it might be easier to reconsider this. But for now, what you have is the best/easiest solution.

Cheers,
Steve

Hi @lm ,

Can you give a little more insight into the use case for this? Why would you want to proxy these symbols?

There doesn't seem to be much gained by caching/proxying symbols, since the symbols basically contain a pointer to a public GitHub repository (or a public source server with the older format of PDB), which wouldn't be practical to cache/proxy.

Cheers,
Steve

Hi @lm ,

We've considered this over the years, but realized it'd be more engineering effort that just changing cookie duration; we'd want to track login sessions on the server side, allow them to be invalidated, etc.

It's possible, but it seems the "Remember me" function on most services (GitLab, etc.) seems to be very forgetful anyways (for security purposes), and with Integrated Windows Authentication and browsers managing passwords, it seems not so valuable for the effort.

Cheers,

Steve

Hi @paul_6112,

There is no built-in blocking to BuildMaster/Inedo agent, and each execution (deployment) uses a separate temporary directory, so there's no conflict on "our" side of things.

However, if you wanted blocking of sorts, you could create resource pools if you'd like to do that, or use lock statements.

Cheers,
Steve

Hi Scott,

Glad you could resolve it; definitely a strange error, I would have expected a better one than just 500.30.

How did you install ProGet? Using the Inedo Hub?

"The Module DLL C:\Windows\system32\inetsrv\aspnetcore.dll failed to load. The data is the error" -- this sounds like a sort of internal error, caused by the .NET6 hosting bundle not being installed, or not installed properly.

https://learn.microsoft.com/en-us/aspnet/core/host-and-deploy/iis/hosting-bundle

The Inedo Hub will install this when you install/upgrade to v2022.

Thanks,
Steve

Hi @mcascone,

I'm not sure if we'll be able to help too much with analyzing your Wireshark traffic to be honest; we're not really experts at debugging TCP/SSL-level things, and we'd probably need to spend a half a day re-learning how it all works before we could tell you what it might even be.

There's likely some obscure Windows setting or some network setting that your network ops team would be much more familiar with. They're a lot more familiar than we are :)

As for the upack bug, I just posted an issue now -- https://github.com/Inedo/upack/issues/72

We don't have a regular release schedule for upack, so I can't give an ETA.... hopefully it's relatively easy to work-around for the timebeing?

Cheers,
Steve

Hi @mcascone ,

Just looking at the code real quick, I suspect we have a bug where it writes out the wrong files name for the new package:
https://github.com/Inedo/upack/blob/master/src/upack/Repack.cs#L120

That's probably an easy fix, which we can do as part of this Q&A item. I'll wait to hear back about this one.

As for the error, "The underlying connection was closed: An unexpected error occurred on a send.", that sounds like it's HTTPS related. Could you attach Fiddler, or something like that, to find out what's happening under the hood? We may be able to error message to better report it if so.

Cheers,
Steve