Hi @sebastian,

As @gdivis mentioned, it will be a bit of an undertaking to fill-in the missing data (in particular the ID/Scores) using the NVD datasets. We are also exploring aggregating some additional datasets as well, in particular ones for system packages (Debian, RPM), so we can incorporate container image scanning directly in ProGet.

It seems our userbase (and the community at large) is slowly starting to "grok" vulnerabilities, but it's quite a ways off.

We've got some other work ahead of this, so I'm going to say perhaps late June / early July is when we can consider picking up on this again. Looking forward to any feedback you have in the mean time :)

Thanks,
Alex

I'm curious to know too if anyone else is interested in this.

FYI: we were intentionally "lazy" when we came across these, because the expressions got pretty complex (in the specs) and it seemed more suitable for a human to determine what WITH and this and that meant.

ProGet already supports multiple licenses per packages (they are treated as an OR), but we thought it might be unintuitive to only support OR, so we just left it as is.

Thanks for the additional thoughts @sebastian!

I agree... it's not totally infeasible from a technical standpoint, but it's still pretty tricky. Just to comment on a technical thing, FYI...

Now, to be able to serve the package to the user, Proget has to download it first, right?

Actually, ProGet "streams" content from connectors. This means that, when a user requests a package from ProGet (and that package is on a remote connector), ProGet will then request a package from the connector. As the file is being downloaded, ProGet will send the same data back to the user and optionally write that data to disk. If we didn't do this, ProGet would be basically unusually slow.

A ZIP archive (what package files use) use a tail index, which means you have to read it backwards from End of File. So it's not possible to read an embedded file unless we've downloaded the entire package.

There are a few other "gotchas" we'd need to consider, even for cached/local packages. For example, we can't open/seek the package file just to know the license and if the package should be blocked - especially when it comes to cloud storage (for the same reason - tail indexing). So, we would obviously need to store package license file info in the database too... but then we'd need a way to deal with existing packages on disk that don't yet have that info.

We may also want to add some sort of heuristic analysis of license text, even if it's simple as a basic distance check. Personally I think that's a bad idea to rely on... but other products do, and the reality is most users would just skim a license anyway.

This all becomes a lot easier after v2023 with centralized data and a package analyzer that can background scan all these, but still not trivial. And then there's the real hard part... the UI and documentation

We definitely don't want to hack something in like packageid:// and package::// -- those have been a total pain and plus, I hate the design

Anyway -- just wanted to give more technical insight into why ProGet behaves like this, and why I'm hesitant to jump on the "reading license file" approach without adding somethign that's a lot more valuable than what we have now.

Hi @sebastian

Thanks for the idea -- yes, I think it's an interesting approach!

We explored it a while ago, and this was where we ended up...

1. It's even more confusing to use than packageid://, so we'd need to find a better UI solution

2. We'd want to store the full license text as well, so it'd be easy to confirm the contents

3. This is all a nontrivial engineering effort

4. We're not sure how many packages this would impact and how much value / time savings it would represent

5. None of this would even work for remote packages, which is by far what most users find confusing and have issues with

6. It would probably require less engineering effort to scan/query all packages on NuGet and make a "database" of package licenses using a little human intelligence

7. It would require even less effort to just ask package authors to specify license codes, and then eventually the problem will go away on its own probably

And then we gave up because there were more priority things to address ;)

Cheers,
Alex

@sebastian said in Package license definition:

That being said, having a new entity "package owner" or "publisher" or something like that and being able to filter for that entity could be a cool new feature. This could also be used in the SBOM reporting feature (like: 40 packages come from Microsoft, 20 from vendor A, 7 from vendor B).

We considered doing "something" with this metadata ages ago, but found that a lot of packages (including npm, etc.) have multiple owners/authors. On the top page of nuget.org, just 3/20 seem to have a single author

In general, the human-driven "package approval workflow" seems to be the best bet. Maybe p[ainful at first, but "not too bad" in the long-run

@sebastian @pmsensi

Thanks for added insights! This doesn't seem as simple as I had hoped...

The current solution in ProGet now (i.e. packageid:// and package:// urls that can be associated with license codes) feels hacky, but seems we don't have many options.

Well, one thing that might work.... submitting a pull request to those packages to add the MIT license code to their project file? It's probably just an oversight on their part...

@sebastian said in Package license definition:

What's driving me nuts at the moment is that Microsoft seems to be using embedded license info files for quite a number of their packages. Assigning licenses to those is going to take a while...

Oh really? They even have their own accepted SPDX code... whyyy Microsoft

In the past, we thought of adding a kind of wildcard URL for licenses, like a "package://Microsoft.*" => "MSPL" would basically associate all packages with that prefix that don't otherwise have a SPDX code, or an explicit license.

Wonder if that would help here?

Hey @mrbill and all,

We've since had a handful of "mentions" for it on the presales side, but it was more like a "wishlist" than anything else. There's probably some opportunity however - so I'm leaning towards re-evaluating for v2023.

Big thing will be partners who can show us how to use R -- I added this to our internal list for the v2023 review :)

Cheers,

Alex

@lm excellent, thank you for the help!

OK - we'll wire these up via PG-2223, which may come in the next or following maintenance release. In theory it's trivial and low risk.

Legacy / HTTP-only Format (still works):
<WebServer Enabled="true/false" Urls="web server listen URLs" />

New / HTTPS-inclused Format:

<WebServer Enabled="true/false">
  <Endpoint 		  
      Url="{http|https}://host.name:1000/">
      CertFile=".pfx, .pem, .crt"
      KeyFile=".key"
      Password=""
      Subject=""
      Store=""
      Location=""
      AllowInvalid=""
     />
  </Endpoint>
</WebServer>

This lets us document an "Endpoint" using simple table of key/value pairs, as well as simplify error messages and the examples that people will just copy/paste:

• Plain HTTP
• PFX-file based
• .crt+pem file based
• Windows cert store integration, only works on Windows.

Oh --- and good point about self-signed; if anything we should use that ACME library to get a LetsEncrypt cert or something. At least in theory. A few other products seem to be doing that now as well. No idea how well that would work - but for evaluation purposes I think it's pretty important. Especially if you don't know how to configure HTTPS otherwise.

Hi @lm ,

Thanks for the feedback - and you are correct; it is ASP.NET 6.

Starting with the configuration file is a great idea. The Inedo Hub provides a UI to that anyways.

The documentation for our configuration file is here:
https://docs.inedo.com/docs/installation-configuration-files

Obviously it's pretty easy to just wire up values (and we'd be happy to do that), but my only hesitation is documentation and cross-platform compatibility.

Brainstorming a bit but... from the UI, we will just support one URL I think. I don't know why you'd want multiple...

The docs you linked from Microsoft are confusing, and it's not really clear what fields to use or why. I guess, in general, you'd select a file (which InedoHub would generate?) or the cert store (that's Windows only, right?).

And I guess what, when you generate a cert yourself, it's a pfx or something? I always forget the differences....

How would you suggest we modify that element to support the different options they have?

  <WebServer Enabled="true/false" Urls="web server listen URLs" />

What would you want to put in, or suppose other (advanced) users would want to put in?

Cheers,
Alex

Hi @tkolb_7784

I wanted to provide an official answer to this:

Are there plans for (easy) HTTPS support in the internal webserver? Applying a self hosted certificate and deploying it to all build servers and developer machines seems over the top for me right now.

Yes. Not exactly sure how yet, but we would like the integrated webserver to support this as easily as possible.

Thanks for the updates/info!

Hopefully some other folks will find this and express some interest -- I'll update if I hear anything from other users, or there's any marketing interest.

By the way, this is different than Android Package Manager APKs. Someone asked about that once because they were curious if ProGet could be like a private app store for mobile apps.

Hi @shfunke_1795 , @jrottmann_6111

Thanks, I just added this to our documentation page!

I didn't look too deeply, but I found some initial documentation:
https://wiki.alpinelinux.org/wiki/Package_management

It seems this is "like Yum/RPM but for Alpine Linux"? None of us here use Alpine Linux, so there's a pretty big learning curve to get started. Any help here would be appreciated, and definitely move this along :)

Is this related to "APK" that Android uses?

Is the "API" mostly like basic file downloads, based on an index file? Are you able to "hack" or do a PoC using ProGet Asset Directories?

@jblaine_9526 Thanks for clarifying!

I think SAML integration is one of those "write-once" things, where we completely forget about how it works every time

I'm not so familiar with SAML behind the scenes... do you know how "SAML group claims" work? For example...

• Is it something that comes back in the XML response, or does it require a separate request?
• What do the "group claims" look like? Like a list of human-readable group names?

And them most importantly... what should ProGet do with such claims upon receipt? Treat the user as if they're in the group (kind of like LDAP groups), and allow permissions to be assigned against that group (like LDAp, but without searching)?

Hi @jeff-miles_5073 ,

Thanks for the first inquiry; I just updated the documentation! These must be "relatively new", and I know we've had a few customers using universal packages for this.

We have some Terraform integrations on our roadmap for 2023, so this definitely something we'll look into on our own as well.

Cheers,
Alex

We've been getting all sorts of outage/clear notices from our main datacenter since yesterday, so something's definitely up. Maybe a switch or something.

I flipped to our DR instance of ProGet until we get it figured out :)

Conda feeds are now available!

https://docs.inedo.com/docs/proget-feeds-python-conda

cc/ @stewart-ridgway_3626 @ben-hegarty_7893

I'm afraid it didn't make the roadmap for Q4. There's no documented API that we saw, and reverse engineering and then supporting the inevitable quirks takes a lot of effort -- and there still doesn't seem to be a market opportunity for us here I'm afraid (no search volume, etc.)...

This SO Question seems to indicate that using Git repositories is a popular way to reference packages...

https://stackoverflow.com/questions/54143695/how-to-use-my-dart-packages-private-and-not-show-on-pub-dart-lang#54143758

As I mentioned, our usage of this library is minimal, and we do not use it in a manner that would impact product security (i.e. "passing HTML from untrusted sources - even after sanitizing it - to one of jQuery's DOM manipulation methods (i.e. .html(), .append(), and others). So please consider this a "false positive".

The only benefit to our customers/users is to appease security teams that don't really understand how security or vulnerabilities work

However, that's actually a benefit in and of itself -- can you let me know what security scanner you use? What security processes are being used such that a security team is even looking at third-party vendor tools like ours? How do they learn or understand "false positives", etc? Understanding that would help us.

What I don't want to do is play the game of "introduce new bugs/glitches by constantly upgrading javascript libraries just to appease clueless security teams" -- so learning how they scan would really help me decide.

Thanks!

hey @harald-somnes-hanssen_2204,

I'd like to add something like this to ProGet v6. How has progress been w/ your end users? Do they ask about this?

Anyway, here's my rough idea...

I don't know if I like those words, "Feed Usage Instructions", but we have "Usage Instructions" on the Package overview page, so there's that.

Clicking on that would provide instructions similar to npm's, or the Visual Studio by default.

HOWEVER, I'm really thinking that you should be able to customize that page! Maybe you have specific technical instructions you want to add, etc.

LMK your thoughts, curious to know what you think, especially after it's been a while.