Hi @mcascone,

I'm not sure if we'll be able to help too much with analyzing your Wireshark traffic to be honest; we're not really experts at debugging TCP/SSL-level things, and we'd probably need to spend a half a day re-learning how it all works before we could tell you what it might even be.

There's likely some obscure Windows setting or some network setting that your network ops team would be much more familiar with. They're a lot more familiar than we are :)

As for the upack bug, I just posted an issue now -- https://github.com/Inedo/upack/issues/72

We don't have a regular release schedule for upack, so I can't give an ETA.... hopefully it's relatively easy to work-around for the timebeing?

Cheers,
Steve

Hi @mcascone ,

Just looking at the code real quick, I suspect we have a bug where it writes out the wrong files name for the new package:
https://github.com/Inedo/upack/blob/master/src/upack/Repack.cs#L120

That's probably an easy fix, which we can do as part of this Q&A item. I'll wait to hear back about this one.

As for the error, "The underlying connection was closed: An unexpected error occurred on a send.", that sounds like it's HTTPS related. Could you attach Fiddler, or something like that, to find out what's happening under the hood? We may be able to error message to better report it if so.

Cheers,
Steve

Hi @fkarlsson_2176 ,

Linux servers need to be restarted?

Yes - it's a bit tricky to do the reboot in OtterScript, due to the connection being lost like that. That operation has some special handling for that.

Anyways - we'll add support for this via BM-3793 -- I think that never made its way over from Otter. We hope to ship that in the October 21 maintenance release, but it might be delayed if it's complex or sisues during testing.

Cheers,
Steve

@jim-borden_4965 that doesn't sound right; can you create a new forum post (don't want to clutter this anymore) with some more details/etc, and then we can reply/investigate. Thanks

Hi @scusson_9923, this was implemented in ProGet 2022 :)

Hi @karsten-meichsner_9039,

I haven't seen that error before, but based on the text ("ProxyAuthenticationRequired for layer"), I think that Clair is trying to download an external layer?

Some container image manifests (especially Windows, but not entirely) will point to a URL outside of the registry. This is often done for licensing reasons. What this means is that, Clair (or the docker client) downloads the layers from a url instead of ProGet.

I'm not familiar enough with container scanners (Clair) to know how they search for vulnerabilities; I believe it's done by looking at the packages installed on the system. Log4j is not a package installed on the system (I think), but a library used in some applications.

Cheers,
Steve

Hi @gurdip-sira_1271,

Just to let you know, Otter v2022 has been released; from here it'd be best to start a new thread if you have a specific issue -- we'll be happy to help!

Steve

@nkumari_3548 you'll likely want to check with your network team for help on this one

ProGet is a standard web application, and will listen for all traffic on the port you've configured. If you want to access ProGet with proget.my-company.corp, then you'll need to have someone configure appropriate domain/DNS records, etc. You may also need firewall settings, and a certificate for HTTPS.

https://proget.inedo.com/ is our public instance of ProGet. It's not intended for direct access by users, but by tools like docker or Inedo Hub. You can't login to it.

@sebastian that's awesome, great you could figure it out!

The Package_Versions field is supposed to be some kind of range specifier (e.g. something like [3.4.4-3.4.4.8), but I don't know format offhand). However, we've also never seen it in the wild in any dataset. It's always just a CSV of versions.

Regarding licensing, that information is not really in the database. It's parsed from the manifest file (e.g. .nuspec) in the front-end. That file is stored in database, but it's practical to use in SQL. We talked about building a kind of job that would normalize that into a PackageLicenses table, and then allow custom reporting (or show how a consumer is using it).

At some point, we'd love to get a copy of your data (database backup if possible) so we can see some real-world consumers/consumption and build some pages from it. We do have some sort of idea in PRoGet how to make this look/work better, but seeing real data would be helpful. That development won't start until Q2, so maybe we'll reach out in a month or so and ask :)

@sebastian great, let us know what you find!

When it comes to reporting/reading data, no problem directly querying the tables. We definitely support that, and some folks have quite advanced reports that tie together various systems.

I'd just go directly to SQL Server for that , and do SELECT on the appropriate views/tables. Let us know what questions / issues you have!

Hi @sebastian, thanks for the feedback!

These both could be pretty complex (specially the API) and would thus end up as strategic, roadmap-level items (as opposed to some minor enhancements); we'd already have ProGet v2022 mostly planned out, so it would be a bit before we can consider it.

How's your relational database / SQL skills? If you're comfortable exploring the data in there, it might give you some insight / ideas into what we can do as a low-risk, minor-enhancement that would add a lot of value. Once you're able to see what is possible with the existing data, then it might be a view we can do.

We don't have a "guide" on how to query the tables, but we can assist if you have questions. The database columns should be familiar, but tables of interest: Vulnerabilities, FeedVulnerabilitySources (links feeds + vulnerability sources), and PackageDependents (package consumers).

Thanks,
Steve

Hi @sebastian ,

Your current workflow sounds like it's the best approach for now, albeit cumbersome; there's also the possibility of directly querying the database, and generating a kind of report with your own tooling.

This is of course something we can consider improving, but it's hard to guess where without knowing more details and having real world datasets. So definitely an opportunity for a feature request / collaboration.

Let us know as you solidify your processes and learn what data you find valuable!

Thanks,
Steve

HI @sebastian,

There are three different package records:

• Downloads ("Record individual downloads for advanced statistics")
• Deployments ("Record where packages have been deployed")
• Usage ("Record where packages are being used") - i.e. package consumers

You can add these records for remote packages (or packages that don't even exist on the feed yet), and none of these records should be purged when you delete a package (manually or via retention). If you delete a package, and then add it back, those "old records" will reappear.

The "Download Count" is part of the "server-side package metadata", along with Listed/Unlisted Status, Tags, etc. This is one reason why "Download Count" and "Number of Downloads" can vary.

hope that helps,
Steve

@can-oezkan_5440 just an update, we plan to ship this in the next maintence release in ProGet as PG-2111

@kichikawa_2913 I'm wondering if this might be a regression with the preview feature, but I can't imagine how. I have one other idea, too...

I used the "test privileges" function and it shows that the group has View/Download permissions.

Can you clarify this? The "test privileges" should only work with a username, not a group name. Could share what happens when you:

1. Have a specific user navigate to a package in a NuGet feed, and then try to download it from UI? Is there a specific message body you see? (outside of 403)
2. Enter that same username in the "test privileges" with that particular feed? What are all the permissions you see?

After doing those, the last thing I would try is to revert to the 6.0 behavior, and see if the problem still occurs. AT least that will tell us where to look....

Hi @claes-hermansson_6271,

Great! Our recommend three-feed workflow (unapproved, approved, internal) is similar, and keeps the third-party packages in the first two feeds. This way, you can scan for vulnerabilities much more easily.

As I understand it though, to get full coverage the pgscan tool needs to be installed on every build server, and the pgscan publish... command needs to be implemented in every build?

This is correct. Dependency resolution is complex and often nondeterministic, so it can only really happen at build-time. Hopefully you can templatize pretty easily :)

Cheers,
Steve

Hi @kichikawa_2913 ,

The NuGet client's behavior is based on NuGet.org, where no authentication is ever required to view/download packages. As such, it doesn't pass the API key when doing those queries; instead, you can use a username of api and the password of your api key.

Based on the issue though, it sounds like ProGet is unable resolve the groups; I would use the "test privileges" function on the Tasks page to verify this. Thatw ill show you if the username can download packages or not.

The most common reason that groups aren't resolving is that the member is not directly in the group (i.e. they're in a group which is a member of the group), and you don't have recursive groups enabled; do note that this is really slow on some domains.

Cheers,
Steve

Hi @claes-hermansson_6271 ,

Good question; it sounds like you're creating a Package approval workflow, and are on the right track to thinking about "package governance".

Your observations about licenses and vulnerabilities are correct, and you could simply apply the same vulnerability source to each feed, if you want to use that feature. But i generally suggest putting it in the approved-packages feed (using our workflow above).

A couple important things to note.

All packages have a license (even "no license" is a license), and a package with an unacceptable license (i.e. one you explicitly block) is always unacceptable. There are really no exceptions to this (maybe the CEO or legal officer someone could override this rule?), and if someone accidently uses an unacceptable license, it presents an immediate legal risk that you should remediate.

Very few packages have vulnerabilities, and those that do are usually acceptable to use. Most vulnerabilities won't impact your application, and those that do are usually easy to work-around aren't severe enough to warrant action. Severe vulnerabilities (i.e. where remediation is needed, like log4shell) will likely come up once or twice a year for you... if that.

When a severe vulnerability is discovered (i.e. one that you wish to block), tracking across feeds isn't really that big of a problem, compared to discovering which applications consumed that package. What I would recommend is investing in configuring package consumers (see section #2 of the article).

Cheers,
Steve

Thanks for the update, @janne-aho_4082

1153 packages yields about 2000 quests to ProGet, and I'm guessing you have a connector to npmjs.org, that will probably yield another 1000-2000 requests outbound (e.g. "what is latest version of del for example). It's possible that some of those proxied requests are timing out, and that's why you're seeing 404s. Hard to say.

Can you try using 6.0.9.rc.3 +1.13.1-rc.10? This will have both the LDAP User caching (it wasn't fully implemented in 6.0.8) plus the "memberOf" property caching.

The next step, if you enabled CEIP, we can track down your sessions and find what's taking so long.

Hi @paul_6112 ,

The general philosophy that BuildMaster was designed around is this:

• A "Release" is an intended set of changes to production
• A "Build" is an attempt at implementing those changes
• When a "Build" makes it to production (i.e. the final stage in the pipeline), then the changes were applied to production, thus the "Release" has occurred
• If you want to change production again, then you create a new "Release"

This means that, after build of 1.1 is successfully deployed to production (and no rollbacks are needed, etc.), it cannot be released again. You'd need to do 1.2, or so on.

However, in practice users want to deploy 1.1 build 1234 to production, and then 1.1 build 5678 to production.

So with this, we have a few options:

1. Create the releases 1.1.1234 and 1.1.5678, and then have build 1 (sounds like you're doing this)
2. Create Release 1.1.0 Build 1234, Release 1.1.1 Build 5678, and use a Release Name of 1.1 (this overrides the display in nearly all places)
3. Edit the pipeline, and uncheck "Mark the release and build as deployed once it reaches the final stage."; this will give you more control over changing release status
4. Don't use releases at all (set Release Usage to None on advanced settings), and have the build number be 1.1.1234, 1.1.5678

Definitely open to some other ideas as well. A few users have had to "workaround" this design philsophy, and we'd rather just support it!