Posts made by stevedennis

@OtterFanboy forgot to mention, we did plan to build a first-class Gitea integration, so you could browse your repositories (just like GitHub etc.), but didn't get it in time. It's still on our list!

Hi @OtterFanboy ,

Thanks for all the details :)

Looks like this is a regression/bug with browsing Generic Git repositories. That's a brand-new 2022 feature, and we're still working out kinks with it. We'll investigate and get it fixed ASAP!

Stay tuned :)

Cheers,
Steve

Hi @testintergraph_2317,

We haven't specifically tested w/ OpenLDAP (I think), but the integration does work with other providers. One compatibility issue seems to that sAMAccountName isn't the Username property on all LDAP servers.

However, in v4 of the LDAP provider, you can now customize these queries. Have you tried this yet? https://docs.inedo.com/docs/en/various-ldap-v4-advanced

In any case, we'd be happy to work with you on getting OpenLDAP working. Just let us know what specific issues you're having, and we can patching, etc.

Of course, you're welcome to try modifying the code yourself... and if you want to try I would start by forking the InedoCore extension and then using a custom build of that (just use a version number higher than published). Make sure to delete that custom version once we accept pullrequest or publish version with identical changes.

If you want a totally custom extension, just make one witha different name.

We don't have instructions for custom extensions with Docker, but it follows the same process as Windows (just restart container instead of AppPool/Service): https://docs.inedo.com/docs/proget-administration-extensions#manual-installation

Cheers,
Steve

Hi @testintergraph_2317 ,

I'm afraid that sample code is ancient and won't work in any modern version of ProGet...

Can you let me know how you found that repository? I've just now archived the repository.

Can you share the use case for implementing a custom directory? This is very rare requirement...

In any case, you can look at the user directories we implement here: https://github.com/Inedo/inedox-inedocore/tree/master/InedoCore/InedoExtension/UserDirectories

Cheers,
Steve

Hi @joacim-svensson_8194 ,

That should have worked, but it's of course possible there's a bug.

Can you confirm steps?

Is this basically what you did?

1. Create Two feeds (Feed1 and Feed2), download a vulnerable package in each feed
2. Create Two Vuln Sources (OssIndex1, OssIndex2), associate each to each feed
3. Run the "Vuln Downloader" Job, and see two identical vulnerabilities added to ProGet
4. Assess the vulnerabilities differently (1 = Block, 2= Ignore)
5. Package should Blocked on Feed1, and allowed on Feed2

I want to make sure we're following the steps you did, so we can test this.

Cheers,
Steve

Hi @osnibjunior ,

Oh I see -- I must have missed the part, "it seems it manages to list the package that is in Azure Artifact but fails to download when requested resulting in a 404."

Unfortunately it's hard to say what's wrong; Azure DevOps package registries are pretty buggy, and don't always follow the NuGet API specs.... we've worked-around a few of their glitches, but maybe they added one again?

Anyway, if you can provide us with (1) the Connector URL and PAT token and (2) a package that you can't download, we can attach a debugger and see if it's something we can identify/fix.

You can email that to support at inedo dot com, but please enter [QA-1021] in the subject so we can find the email. And please let us know when you send the email, as we don't regularly monitor that box.

Cheers,
Steve

Hi @smiller-circ_2844 ,

Product activation is required. If you're unable to keep the same MachineID, then you'll need to get an activation code from my.inedo.com.

While it's not officially supported and isn't documented, you could "listen" to the activation request that ProGet makes by using a proxy server like Fiddler, and make the same API request with your key. You would then need to add that value to the database.

I don't want to give specific instructions on how to do this, because like I mentioned it's not officially supported -- but it's not "too difficult" to do, if you're familiar with the above.

Cheers,
Steve

Hi @osnibjunior ,

Did you follow our AzureDevOps Migration tutorial?

Based on the information you've entered in the connector, I don't think it will work because your Connector URL looks incorrect (it should look something like https://pkgs.dev.azure.com/kharnagy/kharnagy/_packaging/internal-nuget/nuget/v3/index.json).

I'm not sure about the username/password but I understand you need to create a PAT and use that as well. I think it doesn't matter where you put the PAT (username or password field).

There's nothing in v2022 that should have impacted this

Cheers,
Steve

Hi @OtterFanboy ,

Under the hood, Otter runs PowerShell scripts through a custom PowerShell host that runs within the agent's process. While this generally leads to performance improvements, it can lead to memory leaks or worse if PowerShell scripts are created improperly. It sounds like this is the case here -- especially if your scripts are prompting for input, not terminating, etc.

This is where Process Isolation can come in. Instead of using the agent's process to run the PowerShell script, you can use OtterScript's with isolation feature. This creates a new process that runs the script.

Here is an example of an OtterScript plan that runs the same PowerShell script in different processes, even though it is the same script and runs on the same agent:

for server targetsv
{
    with isolation
    {
        PsExec >>Write-Host "Process ID: $pid">>;
    }
    with isolation
    {
        PsExec >>Write-Host "Process ID: $pid">>;
    }
}

Note that you can also specify timeouts with with timeout=1000 as well.

Cheers,
Steve

hi @cshipley_6136,

We chatted a bit about this as a team today.

We believe that the application failed during initialization (i.e. the first web request after starting the service); when this happens, the error message will be returned until you restart the application (container/service).

This is by design; there's no way to guess where or why the initialization failed, and if we "crash" the application (and trigger a restart), it can end up in a endless loop and make diagnosing really difficult.

In this case, the error is network related (server name not found). That error can be transient when you have network problems, or permanent if it's a typo in the connection string.

Since restarting the application (container) fixed it, then it was a transient network issue that happened during initialization.

Hope this helps,
Steve

Hi @joacim-svensson_8194 ,

A vulnerability is tied to a vulnerability source (you can see the source name on the vulnerability page), and a vulnerability source is tied to one or more feeds.

So when you create a second vulnerability source, then you'll see two sets of vulnerabilities and be able to assess each vulnerability differently.

Cheers,
Steve

Hi @cshipley_6136 ,

Based on this, it looks like there are some network problems between ProGet and your SQL Server:

A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections.

It's really easy to reproduce this error (just make a typo in the server name) but when it's intermittently happening, it means the problem is happening within the network, between the two servers.

Unfortunately you won't be able to find any useful information on the SQL Server side, since the connection cannot be established. And the same is true on the ProGet side, for the same reason.

This is definitely something to bring up with your Network/Operations team. It could be anything from DNS issues to router configuration to a bad/failing switch.

Best,
Steve

Hi @cshipley_6136

Can you give more details about the error messages you were seeing? You mention it's a login error, like a bad SQL password or account or something?

The health check (/health) runs the Feeds_GetFeeds procedure... so it should have given a 500 error / crashed if the database couldn't be connected.

We'll try to reproduce the failed failure, and get it fixed asap.

Cheers,
Steve

Hi @tyler-engelhardt_8011 ,

I looked at the docs briefly, but I'm really not familiar enough Ansible to know what could be the issue, or how they try to run Chocolatey behind-the-scenes. It's most certainly a configuration issue in your Ansible script though.

So far as I can tell, Chocolatey is not installed on the server, so Ansible is trying to look for a file called install.ps1 on the ProGet server (which obviously wouldn't be there) to install Chocolatey. That file is located here: https://community.chocolatey.org/install.ps1

I would try contacting the Ansiblesupport team or play around with the parameters like bootstrap_script. Please let us know what you find.

Cheers,
Steve

Hi @lm ,

We don't track that information on the package history page at this time; you'll have to find it under Admin > Events.

It's definitely something we can consider with/after v2023, once we've centralized information about packages in the database. I'll make a note of that!

Cheers,
Steve

Hi @lm,

Unfortunately it's not trivial to get a reliable URL for the package within the webhook module/code now, so it's not something that's easy to add on our end. The URL you have works for NuGet, but for other formats (Ruby, Debian, etc.) you need to include more information I believe.

After v2023, where we plan to centralize a lot of this, it might be easier to reconsider this. But for now, what you have is the best/easiest solution.

Cheers,
Steve

Hi @lm ,

Can you give a little more insight into the use case for this? Why would you want to proxy these symbols?

There doesn't seem to be much gained by caching/proxying symbols, since the symbols basically contain a pointer to a public GitHub repository (or a public source server with the older format of PDB), which wouldn't be practical to cache/proxy.

Cheers,
Steve

Hi @lm ,

We've considered this over the years, but realized it'd be more engineering effort that just changing cookie duration; we'd want to track login sessions on the server side, allow them to be invalidated, etc.

It's possible, but it seems the "Remember me" function on most services (GitLab, etc.) seems to be very forgetful anyways (for security purposes), and with Integrated Windows Authentication and browsers managing passwords, it seems not so valuable for the effort.

Cheers,

Steve

Hi @paul_6112,

There is no built-in blocking to BuildMaster/Inedo agent, and each execution (deployment) uses a separate temporary directory, so there's no conflict on "our" side of things.

However, if you wanted blocking of sorts, you could create resource pools if you'd like to do that, or use lock statements.

Cheers,
Steve

Hi Scott,

Glad you could resolve it; definitely a strange error, I would have expected a better one than just 500.30.

How did you install ProGet? Using the Inedo Hub?

"The Module DLL C:\Windows\system32\inetsrv\aspnetcore.dll failed to load. The data is the error" -- this sounds like a sort of internal error, caused by the .NET6 hosting bundle not being installed, or not installed properly.

https://learn.microsoft.com/en-us/aspnet/core/host-and-deploy/iis/hosting-bundle

The Inedo Hub will install this when you install/upgrade to v2022.

Thanks,
Steve

Hi @mcascone,

I'm not sure if we'll be able to help too much with analyzing your Wireshark traffic to be honest; we're not really experts at debugging TCP/SSL-level things, and we'd probably need to spend a half a day re-learning how it all works before we could tell you what it might even be.

There's likely some obscure Windows setting or some network setting that your network ops team would be much more familiar with. They're a lot more familiar than we are :)

As for the upack bug, I just posted an issue now -- https://github.com/Inedo/upack/issues/72

We don't have a regular release schedule for upack, so I can't give an ETA.... hopefully it's relatively easy to work-around for the timebeing?

Cheers,
Steve

Hi @mcascone ,

Just looking at the code real quick, I suspect we have a bug where it writes out the wrong files name for the new package:
https://github.com/Inedo/upack/blob/master/src/upack/Repack.cs#L120

That's probably an easy fix, which we can do as part of this Q&A item. I'll wait to hear back about this one.

As for the error, "The underlying connection was closed: An unexpected error occurred on a send.", that sounds like it's HTTPS related. Could you attach Fiddler, or something like that, to find out what's happening under the hood? We may be able to error message to better report it if so.

Cheers,
Steve

Hi @fkarlsson_2176 ,

Linux servers need to be restarted?

Yes - it's a bit tricky to do the reboot in OtterScript, due to the connection being lost like that. That operation has some special handling for that.

Anyways - we'll add support for this via BM-3793 -- I think that never made its way over from Otter. We hope to ship that in the October 21 maintenance release, but it might be delayed if it's complex or sisues during testing.

Cheers,
Steve

@jim-borden_4965 that doesn't sound right; can you create a new forum post (don't want to clutter this anymore) with some more details/etc, and then we can reply/investigate. Thanks

Hi @scusson_9923, this was implemented in ProGet 2022 :)

Hi @karsten-meichsner_9039,

I haven't seen that error before, but based on the text ("ProxyAuthenticationRequired for layer"), I think that Clair is trying to download an external layer?

Some container image manifests (especially Windows, but not entirely) will point to a URL outside of the registry. This is often done for licensing reasons. What this means is that, Clair (or the docker client) downloads the layers from a url instead of ProGet.

I'm not familiar enough with container scanners (Clair) to know how they search for vulnerabilities; I believe it's done by looking at the packages installed on the system. Log4j is not a package installed on the system (I think), but a library used in some applications.

Cheers,
Steve

Hi @gurdip-sira_1271,

Just to let you know, Otter v2022 has been released; from here it'd be best to start a new thread if you have a specific issue -- we'll be happy to help!

Steve

@nkumari_3548 you'll likely want to check with your network team for help on this one

ProGet is a standard web application, and will listen for all traffic on the port you've configured. If you want to access ProGet with proget.my-company.corp, then you'll need to have someone configure appropriate domain/DNS records, etc. You may also need firewall settings, and a certificate for HTTPS.

https://proget.inedo.com/ is our public instance of ProGet. It's not intended for direct access by users, but by tools like docker or Inedo Hub. You can't login to it.

@sebastian that's awesome, great you could figure it out!

The Package_Versions field is supposed to be some kind of range specifier (e.g. something like [3.4.4-3.4.4.8), but I don't know format offhand). However, we've also never seen it in the wild in any dataset. It's always just a CSV of versions.

Regarding licensing, that information is not really in the database. It's parsed from the manifest file (e.g. .nuspec) in the front-end. That file is stored in database, but it's practical to use in SQL. We talked about building a kind of job that would normalize that into a PackageLicenses table, and then allow custom reporting (or show how a consumer is using it).

At some point, we'd love to get a copy of your data (database backup if possible) so we can see some real-world consumers/consumption and build some pages from it. We do have some sort of idea in PRoGet how to make this look/work better, but seeing real data would be helpful. That development won't start until Q2, so maybe we'll reach out in a month or so and ask :)

@sebastian great, let us know what you find!

When it comes to reporting/reading data, no problem directly querying the tables. We definitely support that, and some folks have quite advanced reports that tie together various systems.

I'd just go directly to SQL Server for that , and do SELECT on the appropriate views/tables. Let us know what questions / issues you have!

Hi @sebastian, thanks for the feedback!

These both could be pretty complex (specially the API) and would thus end up as strategic, roadmap-level items (as opposed to some minor enhancements); we'd already have ProGet v2022 mostly planned out, so it would be a bit before we can consider it.

How's your relational database / SQL skills? If you're comfortable exploring the data in there, it might give you some insight / ideas into what we can do as a low-risk, minor-enhancement that would add a lot of value. Once you're able to see what is possible with the existing data, then it might be a view we can do.

We don't have a "guide" on how to query the tables, but we can assist if you have questions. The database columns should be familiar, but tables of interest: Vulnerabilities, FeedVulnerabilitySources (links feeds + vulnerability sources), and PackageDependents (package consumers).

Thanks,
Steve

Hi @sebastian ,

Your current workflow sounds like it's the best approach for now, albeit cumbersome; there's also the possibility of directly querying the database, and generating a kind of report with your own tooling.

This is of course something we can consider improving, but it's hard to guess where without knowing more details and having real world datasets. So definitely an opportunity for a feature request / collaboration.

Let us know as you solidify your processes and learn what data you find valuable!

Thanks,
Steve

HI @sebastian,

There are three different package records:

• Downloads ("Record individual downloads for advanced statistics")
• Deployments ("Record where packages have been deployed")
• Usage ("Record where packages are being used") - i.e. package consumers

You can add these records for remote packages (or packages that don't even exist on the feed yet), and none of these records should be purged when you delete a package (manually or via retention). If you delete a package, and then add it back, those "old records" will reappear.

The "Download Count" is part of the "server-side package metadata", along with Listed/Unlisted Status, Tags, etc. This is one reason why "Download Count" and "Number of Downloads" can vary.

hope that helps,
Steve

@can-oezkan_5440 just an update, we plan to ship this in the next maintence release in ProGet as PG-2111

@kichikawa_2913 I'm wondering if this might be a regression with the preview feature, but I can't imagine how. I have one other idea, too...

I used the "test privileges" function and it shows that the group has View/Download permissions.

Can you clarify this? The "test privileges" should only work with a username, not a group name. Could share what happens when you:

1. Have a specific user navigate to a package in a NuGet feed, and then try to download it from UI? Is there a specific message body you see? (outside of 403)
2. Enter that same username in the "test privileges" with that particular feed? What are all the permissions you see?

After doing those, the last thing I would try is to revert to the 6.0 behavior, and see if the problem still occurs. AT least that will tell us where to look....

Hi @claes-hermansson_6271,

Great! Our recommend three-feed workflow (unapproved, approved, internal) is similar, and keeps the third-party packages in the first two feeds. This way, you can scan for vulnerabilities much more easily.

As I understand it though, to get full coverage the pgscan tool needs to be installed on every build server, and the pgscan publish... command needs to be implemented in every build?

This is correct. Dependency resolution is complex and often nondeterministic, so it can only really happen at build-time. Hopefully you can templatize pretty easily :)

Cheers,
Steve

Hi @kichikawa_2913 ,

The NuGet client's behavior is based on NuGet.org, where no authentication is ever required to view/download packages. As such, it doesn't pass the API key when doing those queries; instead, you can use a username of api and the password of your api key.

Based on the issue though, it sounds like ProGet is unable resolve the groups; I would use the "test privileges" function on the Tasks page to verify this. Thatw ill show you if the username can download packages or not.

The most common reason that groups aren't resolving is that the member is not directly in the group (i.e. they're in a group which is a member of the group), and you don't have recursive groups enabled; do note that this is really slow on some domains.

Cheers,
Steve

Hi @claes-hermansson_6271 ,

Good question; it sounds like you're creating a Package approval workflow, and are on the right track to thinking about "package governance".

Your observations about licenses and vulnerabilities are correct, and you could simply apply the same vulnerability source to each feed, if you want to use that feature. But i generally suggest putting it in the approved-packages feed (using our workflow above).

A couple important things to note.

All packages have a license (even "no license" is a license), and a package with an unacceptable license (i.e. one you explicitly block) is always unacceptable. There are really no exceptions to this (maybe the CEO or legal officer someone could override this rule?), and if someone accidently uses an unacceptable license, it presents an immediate legal risk that you should remediate.

Very few packages have vulnerabilities, and those that do are usually acceptable to use. Most vulnerabilities won't impact your application, and those that do are usually easy to work-around aren't severe enough to warrant action. Severe vulnerabilities (i.e. where remediation is needed, like log4shell) will likely come up once or twice a year for you... if that.

When a severe vulnerability is discovered (i.e. one that you wish to block), tracking across feeds isn't really that big of a problem, compared to discovering which applications consumed that package. What I would recommend is investing in configuring package consumers (see section #2 of the article).

Cheers,
Steve

Thanks for the update, @janne-aho_4082

1153 packages yields about 2000 quests to ProGet, and I'm guessing you have a connector to npmjs.org, that will probably yield another 1000-2000 requests outbound (e.g. "what is latest version of del for example). It's possible that some of those proxied requests are timing out, and that's why you're seeing 404s. Hard to say.

Can you try using 6.0.9.rc.3 +1.13.1-rc.10? This will have both the LDAP User caching (it wasn't fully implemented in 6.0.8) plus the "memberOf" property caching.

The next step, if you enabled CEIP, we can track down your sessions and find what's taking so long.

Hi @paul_6112 ,

The general philosophy that BuildMaster was designed around is this:

• A "Release" is an intended set of changes to production
• A "Build" is an attempt at implementing those changes
• When a "Build" makes it to production (i.e. the final stage in the pipeline), then the changes were applied to production, thus the "Release" has occurred
• If you want to change production again, then you create a new "Release"

This means that, after build of 1.1 is successfully deployed to production (and no rollbacks are needed, etc.), it cannot be released again. You'd need to do 1.2, or so on.

However, in practice users want to deploy 1.1 build 1234 to production, and then 1.1 build 5678 to production.

So with this, we have a few options:

1. Create the releases 1.1.1234 and 1.1.5678, and then have build 1 (sounds like you're doing this)
2. Create Release 1.1.0 Build 1234, Release 1.1.1 Build 5678, and use a Release Name of 1.1 (this overrides the display in nearly all places)
3. Edit the pipeline, and uncheck "Mark the release and build as deployed once it reaches the final stage."; this will give you more control over changing release status
4. Don't use releases at all (set Release Usage to None on advanced settings), and have the build number be 1.1.1234, 1.1.5678

Definitely open to some other ideas as well. A few users have had to "workaround" this design philsophy, and we'd rather just support it!

Hi @janne-aho_4082 ,

That version will work on 6.0.8 as well! Please let us know, we're eager to get this improved.

Since there's a lot going on, I want to share a summary...

In some maintenance release of v5, we updated Active Directory libraries. Apparently a side effect for your domain was that querying the "memberOf" property of a LDAP user object is really slow. Who knows why.... it makes no sense. But that's LDAP. Anyways, I guess it must not have been noticed from the front-end? Hard to say.

In v6, we redid API authentication. However, we didn't use the LDAP User Cache. We fixed this mostly in 6.0.8, and hopefully fully in 6.0.9-rc.3. That seems like it helped some, but it must be still slow querying the "memberOf" property.

We figured out a way to cache the "memberOf" property, and then applied that to InedoCore-1.13.1-rc.10. So hopefully with the user object cached and memberOf property cached, this should be much faster.

Cheers,
Steve

Hi @janne-aho_4082 ,

No problem, and great you can try this so quickly; PG-2096 is available in 6.0.9-rc.3, and you can install this via the Inedo Hub; https://docs.inedo.com/docs/desktophub-overview#prerelease-product-versions

Cheers,
Steve

Hi @nmorissette_3673,

Unless you blurred out /feeds in your screenshot, the URLS are different:

• Server: /feeds/nuget/Puma.Security.Rules/versions
• Workstation: /nuget/Puma.Security.Rules/versions

Otherwise, I can't help but wonder if there's some rewrite rule, proxy server, or something that's interfering between your workstation and the server

Cheers,
Steve

Hi @martin-noack_4528 ,

This isn't currently supported, in part because it wouldn't be possible to do with cloud storage providers. It also doesn't seem to happen that often, so we didn't implement it.

Is this an operation you'd do often? We can consider adding it, though it wouldn't work for cloud storage because, for whatever reason, the only way to "rename" a folder is by moving each item individually (which can take a long time).

Cheers,
Steve

Hi @claudio_9251 ,

This is most certainly another bug/quirk with Telerik's proprietary feed, and them not following the clear NuGet API specifications

If it's easy we'll consider trying to work-around, but otherwise they really need to be following the API specifications. Or they should migrate away from their own server, like how Infragistics did ;)

Anyway, if you can share the specific HTTP request that is return an unexpected result (it's probably some GET query that returns 404?), then we can try that against the same credentials you sent earlier, and attach a debugger to see what's going on, and if it's an easy fix.

You can find this request by attaching Fiddler, then doing a side-by-side comparison in Visual Studio.

Cheers,
Steve

Hi @benjamin-soddy_9591,

No problem "resurrecting" topics! We definitely want to hear from users about feedback/feature requests.

We still haven't had anyone else ask for deprecation since this request, but I wonder if there's a better solution to solving your challenges than this feature. It sounds like you want to increase governance of your NuGet Packages, potentially with some sort of compliance in mind.

The dotnet list package --vulnerable is probably not what you want for your organization; NuGet's Built-in Vulnerability Scanning is really limited, in part because it only reports on a fraction of known package vulnerabilities (164 as of today). It also won't block packages that you deem problematic, unlike ProGet's feature.

The same is true with dotnet list package --outdated -- it's probably not what you want, because it relies on developers to have to know (1) to run the command, and (2) know what to do if there's an outdated dependency.

There are better ways to manage third-party packages (see How to Create a Package Approval Workflow for NuGet), and you'd better served knowing who's consuming outdated packages (see Use Package Consumers to Track Dependencies

Just some thoughts; like I said, we haven't had any demand for this feature, but these are proven solutions for improving governance of packages as organizations grow/expand their NuGet usage like you are.

Cheers,
Steve

Hi @claudio_9251

I believe that Telerik has a buggy/quirky API implementation, and sends invalid/broken metadata. It happens to work in Visual Studio, and I guess an older ProGet.

Anyways, if you can send us some credentials / instructions to connect to your Telerik feed, we will connect and attach a debugger, and trace the bad metadata.

You can email those to support at inedo dot com -- just reference [QA-743] in the subject. And please let us know when you send them email, because we don't monitor that address.

Cheers,
Steve

Hi @mcascone !

Interesting idea . just brainstorming here :)

Can you share the workflow you're thinking? Like... why do you want to know that a package was repackaged/promoted?

For example...

• I already know that a -rc.xyz version has been repackaged, because we only creat -ci.xyz versions.
• I already know that a package has been promoted, because that's the only way these packages are in a feed

... so this information isn't so helpful to me. Unless a mistake was made, only promoted packages are only in certain feeds, and all packages go through same workflow (repackaging).

Thoughts off the top of my head...

• Promotion records exist in the database, so perhaps easier to show
• Repackaging records are inside the package (I think), so maybe harder to show

Cheers,
Steve

@Michael-s_5143 have you installed using the Inedo Hub?

You can change this in the configuration tab. Otherwise, this is in the Installation Configuration Files.