Posts made by stevedennis

Hi @johnsen_7555 ,

Ah ha, thanks for clarifying that!

This is the expected behavior, and the reason is a bit complex.

Unlike most package repositories, the PyPI Repository API (which a ProGet feed implements) does not provide any licensing information about packages. It's just a very basic listing of names and versions, which means that there is no license information (or description, author, etc). All of that is embedded in the package files.

However, pypi.org has a special API that ProGet queries to provide more information about a package hosted on pypi.org. This way, description and license information can be displayed on remote packages. But this API is only for pypi.org, and the pip client doesn't use it.

When you connect to another feed in ProGet, the regular API is used. And since the PyPi Repository API doesn't provide package metadata, this information isn't available. It's on our long-term roadmap to use a special API / method for ProGet->ProGet connections, but that's a ways off and requires a lot of internal refactoring.

That said, the workflow we support to accomplish what you want is as follows:
https://blog.inedo.com/python/pypi-approval-workflow/

Thanks,
Steve

Thanks for clarifying @johnsen_7555.

I'm struggling a bit to see what kind of configuration might cause this issue or reproduce the issue. Is your python-accessible feed connected directly to PyPi.org?

If you go to re-analyze the package, you should get a really long set of debug logs (no need to send them). But after you do that, can you try the download again?

Hi @johnsen_7555 ,

Sounds like you're building a sort of Python Package Approval Workflow, which is great to see.

If the user doesn't have permission to download the file, I would expect a 401 (if anonymous) or 403 if authenticated.

A 400 error is a bad request. It could be coming from ProGet, as ProGet will occasionally throw that message when there is unexpected input. But it could also be coming from an intermediate server that's processing the request before forwarding to ProGet.

In this case, I believe pip is simply just performing a GET on the URL in the error message:

.../download/numpy/2.0.0/numpy-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl#sha256=6d7696c615765091cc5093f76fd1fa069870304beaccfd58b5dcc69e55ef49c1

I'm not 100% sure that's what pip is doing, but why don't doing a curl -v against that URL and see if you also get a 400?

If so, then you should get an error in the message body from curl. ProGet will write this out to the response stream.

If not, then you'll need to capture the traffic and see what the difference is. Maybe it's a header that's different? I'm not sure what would cause ProGet to yield a 400.

Let us know what you find,
Steve

Hi @scott-wright_8356 ,

Once connector caching is enabled, the error pattern is not used, so we only have this warning. I added a small change via PG-2726 which will add the connector name. This will appear in the next maintenance release (2024.9), scheduled for this week.

Removing connector caching should reveal the connector name, so maybe that helps you identify it until then

Thanks,
Steve

@jw thanks for clarifying!

The pipelines in ProGet are not really meant to track status. The main reason for the build stages is to control automatic archival, issue creation, and notifications. For example, your stable releases might stay in a "Released" stage indefinitely.

We also had (in the preview feature) threre build statues: Active, Archived, Released. We don't use Released currently, but definitely something we may bring back. You don't want to delete a Released build, but you would probably want to delete an Archived build.

Anyway, I'd check out the pgutil builds promote command - that way you can keep your "archival rules" in ProGet.

Hi @jw,

We'd love to learn more - why not?

We envisioned that there would be lots and lots of builds in the Build stage (i.e. created by a CI server), and the ones released might got to a stage like Production.

Thanks

@jw this was not addressed; the "build pipelines" are fairly primitive ProGet 2024, and we plan to review it as a whole as we get more feedback from users

If you click on "all builds" that might be the view you are expecting

Hi @jw ,

We'll address that via PG-2718 by displaying a message on the Bulk Edit/Promote Pages if the user lacks permission to delete or promote the selected packages.

Thanks,
Steve

Hi @fabrice-mejean_6049 , @brett-polivka

Thanks for the feedback!

With ProGet 2023 out of the way (which made developing new feeds a lot simpler), and the addition of the Cargo Registry Web API, this is something we're much more open to implementing.

We want a user partner, someone who is already using Rust/Cargo and would be able to work with testing/developing this for us. Currently there hasn't been any market demand from prospects or other users.... and for your team, seems like a "nice to have" more than anything.

Thanks,
Steve

Great!

I don't believe that there is any public reporting of VISX package vulnerabilities, so this is not data that would be available in ProGet either

Hi @arkady-karasin_6391 ,

Thanks for sharing the error; basically this error means that you're connecting to an invalid feed/URL. You can only connect to a VISX feed, which is an ATOM-based format.

Note: You cannot mirror the "official" Visual Studio Gallery URL, as that is technically not a VISX gallery. It uses a different and undocumented format, and Visual Studio will not recognize that format for anything except official gallery.

Connectors in VISX aren't very useful, and really are only useful for connecting to other ProGet instances.

Thanks,
Steve

Hi @Scati,

It looks like there isn't a way to do this currently; we'll fix this via PG-2716 in an upcoming maintenance release. Maybe this week's, but more likely two weeks from now.

In ProGet 2024, debian should create a "Debian" feed, and debianlegacy will create a "Debian (Legacy) Feed. We'll also add a feedGroup property.

Thanks,
Steve

Hi @daniels,

This is technically possible, but it's not something that we support out-of-the-box I'm afraid.

The closest way to accomplish something like this would be:

• Use the Otter API to create a server entry with the appropriate roles (similar to what you'd do in the UI)
• Install/run the agent on the computer which will connect to Otter
• Use the Otter API to trigger a job/ remediation
• Optionally, remove the server from Otter and Stop/uninstall the Agent

So it's not easy from the user perspective. However. if Otter and this approach looks like it will work for you, then it's something we can definitely explore together as a user/customer, and build out a use case / case study / etc. That'd be best to start a conversation w/ someone in our customer / sales team on that.

Otherwise, there really hasn't been enough demand for this particular use case, and it's hard enough to market Otter's other use cases, let alone develop build new ones ;)

FYI - note that you can also run the Inedo Agent in outgoing mode:
https://docs.inedo.com/docs/inedoagent-overview#communication-modes

Thanks,
Steve

Hi @arkady-karasin_6391 ,

Under Admin > Diagnostic Center, can you locate the error message and share the full stack trace?

Thanks,
Steve

@parthu-reddy but let us know if you'd like to use a patch/prerelease. We can ship it shortly after 2025.6 is released, and it'd be the only fix likely for the version

Hi @parthu-reddy,

That's generally our plan, and we'll fix it via PG-2702. It's currently scheduled for the 2025.7, on June 14.

Thanks,
Steve

Hi @parthu-reddy ,

It looks like the version field on Maven artifacts is limited to 50 characters, and 2.14.7-sbt-a1b0ffbb8f64bb820f4f84a0c07a0c0964507493 is 51 characters. This field has been limited since "day one" (very many years now), so it's unrelated to the upgrade.

The only workaround is to use a smaller version number (or download the artifact and re-add it without that long hash or something).

Unfortunately this isn't an easy change, as that is used in a primary key. We will have to research and let you know.

Thanks,
Steve

@chris-blyth_5143 this means that the database and application code our out of sync somehow. That's unusual and is often a result of restoring the database but not application code, doing manual installation, etc.

I would just uninstall everything, make sure all the components are gone (except the config file of course), then reinstall and point to the same database. The database poitns to the file share, so it should work upon installation again

In that case please just upgrade and it should be resolved :)

Hi @sebastian ,

This was fixed in ProGet 2024 via PG-2630 (FIX: Dual License Packages should show as compliant if one or more licenses are compliant). It was a bug in the implementation of policies, so it wouldn't work in ProGet 2023 either.

So this should get fixed once you upgrade :)

Thanks,
Steve

Hi @Darren-Gipson_6156, sounds like you've found the right tools to configure the integration.

Here's some more information about the Advanced settings:
https://docs.inedo.com/docs/various-ldap-v4-advanced

The DOMAIN\username situation is a little complex. The DOMAIN is considered a NetBios Alias, and needs to be mapped to a domain to search (like domain.com). Then an LDAP query is constructed babsed on that. So in otherwords, you can't search directly for DOMAIN\username in a search like that.

Try adding a Netbios Alias mapping in the advanced setings, like DOMAIN=domain.com; that might allow you to log-in.

Hi @jw ,

FYI - We just wanted to clarify what "inconclusive" meant - this was a "late" change on our end, and we realized the documentation wasn't very clear. Here is how we describe it now:

Inconclusive Analysis

A build package (and thus a build as a whole) can be have an "inconclusive" compliance status. This will occur when two conditions are met:

1. A rule would cause the build package to be Noncompliant, such as Undetected Licenses = Noncompliant or Deprecated = Noncompliant
2. The package is not cached or otherwise pulled to ProGet, which means ProGet doesn't have enough information about the package to perform an analysis because the package is

You can resolve this by pulling or downloading (i.e. caching) the package in a feed in ProGet, or not defining rules that require server-based metadata. For example, vulnerability-based rules can be checked without the package, but deprecation or license detection cannot.

The analysis message is incorrect however, it should be "Package is Warn because of Package Status is unknown, No license detected."

Thanks,
Steve

Hi @sebastian ,

Thanks for all of the details, this is indeed a regression. We'll get this fixed via PG-2679 in teh upcoming maintenance release , ideally later this week (Friday).

Thanks,
Steve

@daniel-scati looks like there was a redirect problem, but this is the method to try:
https://docs.inedo.com/docs/proget-api-packages-query-latest

So basically this:
GET /api/packages/MyDebianFeed/latest?name=pacomalarmprocessor

Hi @pawel-ostrowski_5669 ,

ProGet dynamically generates these indexes based on an aggregation of locally stored packages and connector results on each request, so caching doesn't make a lot of sense.

npmjs.org, on the other hand, needs to only update indexes when a new version is uploaded, so the cache duration can be a long time.

Thanks,
Steve

Hi @pbinnell_2355 ,

It looks like you have Windows Integrated Authentication enabled. Curl does not support this, but with powerShell you would need to add -UseDefaultCredentials

Thanks,
Steve

Hi @jw ,

I haven't investigated this yet, but I assume that the results are the same in the UI? That's all just pulling data from the database, so I would presume so.

Could you find the relavent parts of the analysis logs? That helps us debug much easier.

Thanks,
Steve

Hi @pawel-ostrowski_5669 ,

ProGet does not set cache headers for npm requests, so this behavior is expected.

Thanks,
Steve

@jw thanks for clarifying! We'll get the error fixed, but these would not show up in the export, since they are not a build package then

Hi @jw,

[1] Based on the stack trace, I think the issue is that one of the SBOM documents you uploaded has a Component with a null/missing Purl field. Obviously this should error, but that's what the error must be looking at the code. If you can confirm it, that'd be great.

[2] ProGet is considered the "source of truth", so a new SBOM document will be generated based on the build packages. That SBOM will then be augmented with some information in the original SBOM(s), such as component "Pedigree", "Description ", etc.

[3] Thanks, we'll try to play with CSS to improve this down the line.

Thanks,
Steve

@artur-wisniowski_4029 thanks for the troubleshooting! We'll investigate/fix this via PG-2671; we're going to target this Friday's maintenance release

Hi @rick-kramer_9238,

The issue sounds like it's related to LDAP configuration (i.e. slow queries to your LDAP/AD server), but it's hard to say. This wouldn't behave any differently in IIS for IWS.

The first thing I would try is disabling LDAP / AD. And Windows Integrated Authentication (if you have it enabled). If the server is still slow, then I would try http:// instead of https://

Once you've identified where the slowness is coming from, we can address it. The most common issue with LDAP is recursive/nested group searches - especially when there are like thousands of groups and everyone's a member of something.

I would "play" with your LDAP settings and try to isolate why it's so slow.

Thanks,
Steve

Ultimately this is going to involve training for your developers. Just like instituting a code review process will be new and uncomfortable at first, a package review process will be the same. Developers will not like it and they will complain.

However, 99% of the time, developers will be fine using the approved feed. 1% of the time (when they want to test a new package or upgrade), then will use the unapproved feed. They'll just need to learn how to switch package sources (it's a drop-down in Visual Studio) and then learn not commit commit these package references.

My advise is to make it incumbent upon developers to not commit code/configuration that depends on unapproved packages. If they do, then it will "break the build" because the packages aren't available. This is an expected behavior - it would be like if a developer decided to upgrade to .NET9-beta.

"Don't break the build" is a common mantra on development teams, and it means developers can't commit code changes that will do that. Just extend that to using unapproved packages.

Hi @jw ,

We added a compliance property via PG-2658 in the next maintenance release.

It basically shows what's in the database (which is also what the page in the UI does):

writer.WritePropertyName("compliance");
writer.WriteStartObject();
writer.WriteString("result", Domains.PackageAnalysisResults.GetName(package.Result_Code));
if (package.Detail_Text is not null)
    writer.WriteString("detail", package.Detail_Text);
if (package.Analysis_Date.HasValue)
    writer.WriteString("date", package.Analysis_Date.Value);
writer.WriteEndObject();

I think you can rely on result=Inconclusive meaning the package isn't in ProGet. That's all we use the status for now, but in the future it might be used for something else. A result=Error means that our code crashed and you shouldn't ever see that.

We'll definitely considering doing something other than a single result string down the line, but for now this was the easiest :)

Thanks,
Steve

Hi @pbinnell_2355 ,

The normal workflow for a two-feed package approval is to generally have developers use the approved feed but allow them to use the unapproved feed when they need to use a new package or version.

However, this shouldn't be their default development style. If they want to use packages that aren't approved, they can request approval for the package(s) they want to use.

This obviously slows down development, but so does code review. And it's a tradeoff in general.

You can use bulk promotion if you'd like. Go to the PAckages page, then select "Cached" from the type of package, then select the packages you wish to promote.

Hope that helps,
Steve

@rick-edwards_9161 that is correct, these will only be developed for ProGet 2024

Hi @pbinnell_2355 ,

It sounds like you've built a kind of package approval process?

https://blog.inedo.com/nuget/package-approval-workflow/

If that's the case, you'll need to promote the packages developers need to feed "B" or ask the developers to not use unapproved packages.

Thanks,
Steve

Hi @rick-edwards_9161 ,

There is a corresponding API, but we haven't documented it yet.

For now, you have to "reverse engineer" the code (ProGetClient.cs):

    public async IAsyncEnumerable<VulnerabilityInfo> AuditPackagesForVulnerabilitiesAsync(IReadOnlyList<PackageVersionIdentifier> packages, [EnumeratorCancellation] CancellationToken cancellationToken = default)
    {
        ArgumentNullException.ThrowIfNull(packages);

        using var response = await this.http.PostAsJsonAsync("api/sca/audit-package-vulns", packages, ProGetApiJsonContext.Default.IReadOnlyListPackageVersionIdentifier, cancellationToken).ConfigureAwait(false);
        await CheckResponseAsync(response, cancellationToken).ConfigureAwait(false);

        using var stream = await response.Content.ReadAsStreamAsync(cancellationToken).ConfigureAwait(false);
        await foreach (var v in JsonSerializer.DeserializeAsyncEnumerable(stream, ProGetApiJsonContext.Default.VulnerabilityInfo, cancellationToken).ConfigureAwait(false))
            yield return v!;
    }

We do plan to document all this in the coming weeks/months.

Thanks,
Steve

Hi @dan-brown_0128 ,

Sorry that we your reply. It must have been closed by mistake on our dashboard or something.

You're correct - this does require a database cleanup. As it so happens, we do have a "duplicates clean-up" script available, but it's intended for ProGet 2024. It's extraordinarily complicated, as you can see:

https://gist.github.com/apxltd/351d328023c1c32852c30c335952fabb

If you're able to send us a copy of your database (we can share a secure link for you in EDO-10419, just let us know), then we can review and think of the best plan to fix it. That will be either running the script in ProGet 2023 or upgrading then running the script.

Let us know your thoughts

Thank you,
Steve

@jw we'll definitely keep this in mind, it doesn't look trivial based on our usage of that marked library

Personally, I always try to keep the Diagnostic Center clean and empty, so when new issues show up I can easily spot and address them. Sifting through messages that are basically spam, without being able to filter or ignore them costs me more of my time that I would like to invest for monitoring.

We do not recommend using the Diagnostic Center for proactive monitoring. It's only intended as a tool for troubleshooting things like connector or 500 errors that you / end-users encounter.

There are a lot of non-problem errors and warnings logged that aren't worth time even looking at.

Thanks for clarifying @jw !

I understand how this can be annoying, but I don't think we want to change the 404 not logging error for this issue in particular. Open to ideas if you have them.

As an FYI..

• I think we use the marked library to turn text into Markdown, like this: el.innerHTML = marked(el.textContent);
• some users definitely use absolute urls within readmes to navigate and show content/images within ProGet
• some users may use relative urls for links, but I can't imagine that ever working for images

If there's a way to supress relative urls in the marked library somehow, we could probably add it if you know how!

@jw thanks for the bug report! This will be fixed via PG-2650 in the next maintenance release. It should say "License Id" instead - and it will then redirect to the Edit page after, which is almost as convenient as using the same dialog but won't require rewriting the page ;)

Hi @jw,

This is expected behavior, as 404 errors are logged for non-API requests, and that's a relative URL. Is README.assets a kind of documented standard? It might just intended for GitHub?

Thanks,
Steve

Hey @rick-edwards_9161 ,

Yes - this would be easiest to do with the pgutil vulns audit command, which we're still working on documenting.

Description:
  List vulnerabilities associated with a package or project file

Usage:
  pgutil vulns audit [options]

Options:
  --input=<input>      Project to audit for vulnerable packages
  --package=<package>  Name of package to audit for vulnerabilities
  --type=<type>        Type of package to audit for vulnerabilities
                       Valid values: apk, deb, maven, nuget, conda, cran, helm, npm, pypi, rpm, gem
  --version=<version>  Version of package to audit for vulnerabilities
  -?, --help           Show help and usage information

See Getting started with pgutil to learn more.

Hi @rick-kramer_9238 ,

I wouldn't recommend upgrading to ProGet 2024 solely for that reason, as it's a major release and if there were issues, it'd be easier to isolate Integrated Web Server vs Regression.

Here are the upgrade notes:
https://docs.inedo.com/docs/proget-upgrade-2024

That said, if you need to rollback to ProGet 2023, you can do so without restoring the database by simply using the Inedo Hub. While there are database schema changes, they are all backwards-compatible with ProGet 2023, which means you can safely rollback your ProGet installation if there's a showstopper bug, and then upgrade later.

However, you should backup your database as an extra precaution anyway.

Thank you,
Steve

Hi @rmusick_7875,

I'm afraid API endpoints are not customizable and we do not support doing "reverse proxy" or otherwise rewriting the URLs. Hopefully this will be a good chance to make the endpoint-url more easy to configure/change - this will be important, as you may wish to move to multiple feeds, etc.

Good luck,
Steve

Hi @greg-swiderski_0221 ,

It looks like there is an "Object reference not set to an instance of an object" error that's occurring while trying to connect:

2024-04-30 09:52:41,341 8256 [WARN ] - Unable to connect to source 'http://localhost:8624/nuget/approved-choco/':
 Object reference not set to an instance of an object.

That error is presumably occurring from the Chocolatey client (choco), and unfortunately there's no way to know what it means. Most likely, it's an "error reporting an error" message, but it's hard to say.

You could use some HTTP monitoring software (Fiddler Classic, Wireshark), and see if ProGet is returning an error of some kind.... but even if so, choco should report that error.

I would check with the chocolatey team on this one.

Thanks,
Steve

@philippe-camelio_3885 I just cloned the issue for Otter (OT-509), so should be an easy fix. Maybe this will hlep w/ data sync issues as well!

Hi @jw ,

Thanks - we modified the script and tested it against your database backup:
https://gist.github.com/apxltd/351d328023c1c32852c30c335952fabb

That said, your duplicate data should not cause any problems. The only feed packages this seems to modify in your database are Microsoft.NetCore.App.Runtime.win-x86-8.0.0 and Microsoft.NetCore.App.Runtime.win-x64-8.0.0. Both are cached.

That said, given that it had a bug, we probably won't put this script in 2024.2; we'd much rather "share the script on a case-by-case" basis to fix problems, until we're confident they will be resolved. Then decide how to have users repair the data.

Thank you,
Steve