Posts made by gdivis

Hi @it4it_9320,

That's about the size I'd expect for a connector to that index. So far, I can't get it to grow much beyond that - I'm wondering if this was caused by the transaction rollback after the constraint error happening over and over. In any case, we'll add an explicit VACCUUM after major updates that ought to prevent the index from expanding again.

-Greg

Hi @it4it_9320,

I've confirmed that the constraint violation issue will be fixed in this Friday's release of ProGet 2024.21.

Regarding the other issue about growing indexes, which remote repository are you seeing this with? Is it the same one that exhibited the constraint error? In all of my testing, the sqlite index is usually in the range of 2-10MB - it's nothing more than the contents of the Packages index file.

-Greg

Hi @Scati

Working with package qualifiers and purls is a little confusing with the API because they are already in a URL encoded query string format and you have to encode them again to be passed in through the API url. If you url encode what you are passing in to purl= then it should work:

https://serverip/api/packages/unstable-lin/delete?purl=pkg%3Adeb%2Fscatinat%407.2.2.0-1%3Farch%3Damd64%26component%3Dscati%26distro%3Djammy

The complexity of dealing with these is the reason we added all of the extra arguments and handling in pgutil to build these qualifiers out more easily.

Hope this helps!
-Greg

Hi Caterina,

pgutil does have the capability to do this if you use the builds sbom command. Apparently this command is hidden from the command list for some reason, but if you run pgutil builds sbom --help it should still show the options for it.

Does this help?
-Greg

Thanks @russell_8876! I've gone ahead and released pgutil 1.1.6 with a fix for this. Give it a try and let us know if it's still not working.

-Greg

Hi @caterina,

You are not overlooking anything, but we did :). We're still in the process of finalizing the commandsets for builds and projects - what we have in there now is something of a placeholder. I'll make sure we get that flag in the finalized version. We plan to have them done by early next week at the latest.

Thanks!
-Greg

Hi @Darren-Gipson_6156,

It looks like we hadn't published the latest version of inedoxpack to nuget.org. I've now published it manually, so if you run dotnet tool update inedo.extensionpackager, it should update to v1.0.7, which will be able to build the extension.

Sorry about that!

Hi @daniel-scati,

I've logged (and fixed) the duplicate versions issue as PG-2686, and that will be included in today's release (ProGet 2024.4).

Regarding the delete command, I've gotten it working using this command:

pgutil packages delete --source=scati --feed=stable --package="pacomalarmprocessor" --version=7.2.1.0-1 --qualifier="arch=amd64&component=scati&distro=bionic"

The only change I made was to put in the full version string instead of just 7.2.1.

Agreed on it being confusing that it succeeds even if the package wasn't found. That was one of those things that made sense in specification (why should I care if the package I want to delete doesn't exist?) but in practice it just hides an error. I think we'll eventually change this API to return an indication that something was actually deleted.

Hope this helps!

Hi @philippe-camelio_3885,

I've tried to reproduce this and I think the issue is that your variable is using JSON array notation instead of otterscript. So, it should be:

@(%(Nom:"Carbon",Version:"2.15.1"),...

It seems to work for me when I format it that way. Does this help?

Hi @artur-wisniowski_4029,

We'll get this fixed, but it's unlikely to be in this week's release. ProGet hashs the complete package file for every package uploaded to it, while the APK spec says to hash only over a certain tar segment of the package. We've been returning ProGet's hash of the package file as the checksum and this is incorrect as you've noted.

We'll post here again when we have a fix date, but I'd expect it will be either next week or the week after.

Thanks!

Hi @pbinnell_2355,

Looks like the example code was incorrect. I've updated it. When a dotnet tool is installed globally, you run it by just running the tool name directly, so pgutil instead of dotnet pgutil

Hope this help!
-Greg

Great to hear! Thanks for the followup

Hi @dan-brown_0128,

I've done some more research on this, and per the spec we definitely should not be url encoding anything in the Filename field: https://wiki.debian.org/DebianRepository/Format#Filename

I've filed this as PG-2591. It's an easy change, so we'll include it in this week's release on Friday. It still is strange that I couldn't reproduce this, but hopefully this will resolve the issue for you and anyone else that is seeing this.

Good find!

I did my testing against unixodbc-common_2.3.11-2+deb12u1_all.deb, and oddly everything worked- I wonder if this behavior varies in different versions of apt?

Anyway, we should change ProGet to match the Debian index behavior, so I will file this as a bug.

Hi @dan-brown_0128 , @stevedennis

I had that thought too, and I made sure caching was enabled when I tried to reproduce this. I have some more ideas for trying to recreate this today - hopefully with more luck this time.

Hi @mlorenzschleipen_5169,

Just wanted to let you know that May 1 was a typo - the release date is actually March 1, so a lot sooner :)

-Greg

Hi @dan-brown_0128,

I've tried to reproduce this both inside of Docker and on a clean bookworm vm, and haven't had any luck. It works for me whether the package has been pulled to ProGet or not, no matter how many times I try. Unfortunately I don't have any ideas what might be happening here.

-Greg

Hi @dan-brown_0128,

That is odd! We'll see if we can reproduce it and let you know what we find.

-Greg

Hi @dan-brown_0128,

I would try increasing the timeout period in the connector settings to see if that helps. Are you running ProGet on Windows or Docker/Linux?

Hi @daniel-scati,

Thanks for reporting these. We've fixed the documentation issues, and will have the integration instructions corrected in the next release (v2023.30 as PG-2585).

Regarding the apt-key deprecation, we were aware of that when redoing Debian for v2023.22, but at the time it didn't look like there was any kind of consensus for how to handle trusting a third-party repository, and nearly all other repos we looked at still only gave instructions for using apt-key. We're happy to reevaluate this if there is a better solution available.

-Greg

Hi @caterina,

Microsoft is confusing on this subject - they refer to the old format as legacy sometimes (https://learn.microsoft.com/en-us/nuget/create-packages/symbol-packages) but other times they just imply it's not preferred. Our documentation and the software itself is usually based on the language they used at the time we drafted the feature. I suspect we are just leaning more into the "legacy/deprecated" terminology than they are right now.

In any case, we have no plans to remove support for the symbol package format or anything. Is the problem that we are missing the "Download with Symbols" link now? I can file that as a bug if that's the case.

Hope this helps!
-Greg

Have you had any success deleting these after retagging? We haven't been able to reproduce this so far.

Hi @gunmaden,

Thanks for reporting this! I've filed it for investigation as PG-2519.

Hi @gunmaden,

Right now there is no way to delete a Maven package using an API, but I can put that in as a feature request.

Regarding Python, you have to know the package file name to delete it, and pass it in as part of the qualifer, for example: qualifier=filename%3Dmypackage.1.0.0.tgz

For Debian, the url query string should look like this:

name={name}&version={version}&group={component}&qualifier=arch%3Damd64

Does this help?

-Greg

Sure. I've logged that change as PG-2508. It should be easy enough to make that URL configurable in ProGet.

Hi @itops_6398,

Thank you for the bug report. I've logged this as PG-2507, and we will likely have it fixed for the next release of ProGet 2023.20, scheduled for October 13. If the fix or testing turns out to be more than expected, it may get deferred to the following release, but I don't expect that to be the case here.

Hi @philippe-camelio_3885,

We've reproduced this, and it is a regression in v2023. We will have it fixed in this Friday's release of BuildMaster v2023.2. It is logged as BM-3893.

Thanks!

Hi @zarniak-j_0637 and @philippe-camelio_3885

We should finally have this issue resolved in the latest Inedo Hub (v1.3.19). If it still doesn't work in that version, post another reply here and we'll look into it again.

Thanks!
-Greg

Thanks! Merged and released.

Hi @e-rotteveel_1850

Thanks for the bug report! We will have this fixed in v2023.14 to be released on Friday. It's logged as PG-2446, and looks like a regression from some of the internal consolidation we did for ProGet 2023.

-Greg

Hi @e-rotteveel_1850

Thanks again for the packages and detailed repro steps. I was able to reproduce this and we'll have a fix (PG-2445) in this Friday's release (v2023.14). It appears to have been a regression introduced when we added support for parsing package constrains information from connectors.

-Greg

Hi,

This is a regression in v23 that we have logged as PG-2389. It will be fixed in v2023.8, which is scheduled for release on Friday (June 16).

Thanks!

Hi @MF-60085,

Thanks for the data! We've found that the root cause this time is due to some duplicate rows in the original table, which for some reason was not created with any uniqueness constraints. We'll get this fixed in a prerelease version within a day or two.

We have a prerelease version available with this fix (PG-2354) if you would like to try it. To install on Windows using Inedo Hub, click the [config] link in the bottom and paste in https://proget.inedo.com/upack/PrereleaseProducts/ - you should then be able to upgrade to v23.0.5-rc.4. If you're running in Docker on Linux, just use tag 23.0.5-ci.4.

Hi @jeff-peirson_4344,

Thanks for the detailed steps. I've been able to reproduce this and we should have a fix for it shortly.

As you've seen, we've had a few regression in npm feeds for this release. Historically, npm was the first non-NuGet feed we added to ProGet, and we decided the major version release was a good opportunity to do some badly needed refactoring. Unfortunately, regressions are a result, but we are now able to iterate much more quickly.

Would you be willing to try a prerelease version?

Hi @sebastian ,

Regarding [1], I can't reproduce this behavior. When I add OSS Index I see vulnerabilities from it and also from PGVC, though this is only on the latest v2023 build, so maybe this works differently on another version?

For [2], you are correct. There are a lot more of these in the dataset than we initially thought. We plan to resolve this by filling in the missing data from NVD, though I can't give an estimate on when we will have that (maybe @apxltd can give you a rough idea of our schedule).

Finally, for [3], I think we will end up storing a CVE for most of these to help populate missing CVSS scores if for no other reason. I think this would be part of updates we do to address [2].

-Greg

I believe this has already been fixed as part of PG-2343 in v2023.3, which we are releasing today. In addition to being slow, the NuGet queries responsible for handling latest versions could return a lot more results than necessary. It didn't lead to incorrect behavior (except with counts) as we still validate and filter results in the frontend, but did cause problems like this.

This is now fixed in 2023.3, which is going to be released later today. Thank you!

@rie_6529

We've identified issues in the queries used for fetching the latest versions of NuGet packages in a feed that are causing these problems. We will include a fix (PG-2343) in tomorrow's release of ProGet 2023.3.

Thanks for the report and investigation!

Hi,

We've identified the issue causing the .snupkg push to fail (PG-2340). A fix will be included in tonight's hotfix release (v2023.2).

We haven't been able to reproduce issues with feed re-indexing deleting packages. I agree that that is unsettling. Did you have the "delete missing packages" option selected for the re-index?

-Greg

Hi,

Thanks for the bug report. This was caused by a change in the way we're handling npm package scopes. We have fixed it as PG-2338 for ProGet 2023.2, and we plan to release it later today.

Note that you will need to rename your storage folder back after upgrading - the @ is supposed to be there.

-Greg

Hi,

I thought we had already fixed this, but I was thinking of a fix we applied for RPM package uploads. It's a regression in a core platform library that has bitten us more than once. Fortunately, the fix here is exactly the same (logged as PG-2307). We'll get the fix in v2022.25 for sure. Originally we had planned to release on Mar 24, but we may move it up to this Friday instead to get this resolved more quickly.

-Greg

It turns out the problem was in the code that adds the package record to the database was using the raw version string instead of the normalized version, which causes a lot of problems as you can see. We've fixed that in PG-2278, and this fix will be in ProGet 2022.20 to be released this Friday. Unfortunately it won't fix any invalid records that have already been created. Time permitting, we will try to get a job in that can be run to fix nonnormallized versions in the database, but if it doesn't make it in this release you'll need to manually remove or update that row in the SQL database. Let us know if you need any assistance with this.

Hi Justin,

Looking at the error, I'm wondering if the directory is actually too long for git, and it's reporting the wrong error message. You could try changing the agent temp directory in the %PROGRAMDATA%\Inedo\SharedConfig\InedoAgent.config file by adding a <BuildMasterRootPath> element to the configuration file with a shorter path like C:\Temp.

However, if you just need to run on the local server, there's no reason not to just use the local agent instead - it can do everything the Inedo agent can do.

-Greg

I've logged this as an issue in the upack repo, and we should have this included in an updated version within a day or two.

Thanks!

@jeff-peirson_4344 - I just ran a test and verified the behavior. ProGet does the following on startup to determine the connection string (in this order):

1. look for SQL_CONNECTION_STRING environment variable (this was the old documented name)
2. look for PROGET_SQL_CONNECTION_STRING environment variable
3. look for PROGET_SQL_CONNECTION_STRING_FILE variable

The first of those with a non-empty and non-whitespace-only value is used. The connection string file requires only read access, and must be plain text containing only the connection string with no additional quoting or escaping. Empty lines are ignored - the first non-empty line is used as the connection string, and any trailing newlines are ignored.

Our Dockerfile does specify a default PROGET_SQL_CONNECTION_STRING, so you may need to override that to be empty. Perhaps it would make more sense if we checked for the file first - I'll discuss that with the team.

Hope this helps!

Sorry for the delay - this is logged as PG-2195 and will be fixed in tomorrow's releases of ProGet 2022.7 and 6.0.20.

Hi @pariv_0352

We've updated ProGet 2022.6 to be more permissive and allow the . in non-semver2 NuGet versions. v2022.6 will be released on Friday.

Thank you!

Hi @brett-polivka,

We've published v2.0.1 of the Azure extension for ProGet, which should resolve this issue. You can install it from the Admin->Extensions page.

Let us know if you are still having the issue after updating the extension.

Thank you!
-Greg

Hi,

This is a regression - pdbstr.exe is used to provide source server support for legacy symbol files. I've logged the issue as PG-2159 and it will be fixed in ProGet 2.0.2, scheduled for release this Friday.

Thank you!