ProGet somehow breaks signed NuGet packages

inok_spb

Hello!

We have a problem with NuGet feed in ProGet.
We use it to store public NuGet packages from nuget.org. The only thing that we do with that packages is that we download them from nuget.org and upload them to ProGet.

Initially, they downloaded from ProGet correctly, but after some time, for an unknown reason, some packages cannot pass validation with the following error:

C:\Nuget\nuget.exe verify -All .\Microsoft.AspNetCore.Http.Features.2.1.1.nupkg

Verifying Microsoft.AspNetCore.Http.Features.2.1.1
...\Microsoft.AspNetCore.Http.Features.2.1.1.nupkg

Finished with 1 errors and 0 warnings.
NU3005: The package signature file entry is invalid. The central directory header field 'compression method' has an invalid value (8).

It turns out that in such broken NuGet packages, the signature file is compressed with Deflate method. In valid NuGet packages, the signature file is not actually compressed.

I also described the issue with NuGet packages here: https://github.com/NuGet/Home/issues/11861
It seems that nuget.exe itself should support signatures compressed in such a way since they are perfectly valid after decompression. However, they are not supported now.

So the question is, how is it possible that ProGet after some time repackages NuGet packages and compresses signatures with Deflate method?

The only idea I have is that, because we enabled Symbol Server, ProGet somehow lazily processes NuGet packages to extract metadata or so, and updates packages, breaking signature. Maybe, it happens when someone tries to access symbols of the specific package.
Nevertheless, we disabled "Strip symbol files", "Strip source code" and "Remove signature file" for the Symbol server, so it seems that there is no reason for ProGet to overwrite packages.

What can we do?

atripp

Hi @borisov_1556 ,

Thanks for the detailed information.

At first, ProGet does not modify NuGet package files after you upload it.

If you enabled any of the "Strip" options, then ProGet will stream a modified archive file during download; this will have a different hash and be a different file. However, this wouldn't alter how existing items are stored in the archive file.... those would be streamed as-is.

ProGet also has a "repackage" feature, which will create a new package altogether. But I doubt you're using this, and you shouldn't use this with public / third-party packages, it's used for CI/CD pipelines.

I'm not sure what's going on, but I would investigate how you're uploading the files? Maybe you're re-signing them? But otherwise, if you just download a package file from nuget.org, then upllad it to proget, then download it again... it's going to be identical file.

Let us know what you find!

Cheers,
Alana

inok_spb

Hi @atripp ,

Thank you for the fast response!

No, it's definitely not the "Repackage" feature, we don't use it. Also, we don't re-sign packages, the promotion from nuget.org to ProGet is just a pair of native nuget restore (to also collect all dependencies of the package) and nuget push of collected .nupkg's.
So after uploading to ProGet, packages downloaded from nuget.org and ProGet are actually identical.

You say that ProGet doesn't modify files after uploading, only during it. But can ProGet strip and therefore modify packages after uploading in case of triggering re-indexing in a "symbol server" settings? If it's so, then maybe one of the stripping options was enabled some time ago and re-indexing was triggered... It explains how already uploaded packages could be broken.

Also, you say:

However, this wouldn't alter how existing items are stored in the archive file.... those would be streamed as-is.

Just in case: do you mean that not only the content of a file, but also the compression type of every file in the .nupkg will be preserved as-is, and ProGet handles that explicitly?

Because, if the new package is just a 'new' zip archive with all files that weren't filtered out by stripping features, and the compression type of ZIP entries is not preserved explicitly on a per-file basis, then in the end we will get a package with a signature file that is binary identical to the original after decompression, but in the archive, it's compressed using Deflate method, just like all other files inside .nupkg file.

Thank you!

atripp

Hi @borisov_1556

You say that ProGet doesn't modify files after uploading, only during it. But can ProGet strip and therefore modify packages after uploading in case of triggering re-indexing in a "symbol server" settings?

When you enable any of the "strip" options, ProGet will modify the file during the download process, when it's requested by a client (web browers, nuget.exe, etc). The zip-file is basically rewritten on-the-fly when sending it to the client requesting it. This modified file is not saved to disk or persisted inside of ProGet.

The symbol serving reindexing is unrelated.

do you mean that not only the content of a file, but also the compression type of every file in the .nupkg will be preserved as-is, and ProGet handles that explicitly?

Correct. ProGet effectively just "deletes" the "stripped" files entries from the archive file while it's being streamed. The existing file entries are not modified.

A file entry in a zip archive contains the compression type. So to "change" an entry's compression, you would need to "delete" it from the archive, and then "add" it back to the archive.

So, the point is... something else must be modifying your zip file. ProGet is not. You can verify this by:

download file from NuGet.org, check the hash of the file
uplaod file to ProGet, check the hash of the file in the package store
download file from ProGet, check the hash of the file on disk

MAybe nuget.exe is doing a modifcation? It's really hard to guess, but i would continue the investigation by finding what could be modifying the file.

Cheers,
Alana

inok_spb

Actually, in our case, broken files are stored in the ProGet storage on a disk, I checked them during the investigation.
So it's not an issue that happens on the fly during the download process. And it's also not an issue that happens during package uploading, since after uploading, packages are valid and identical to the original ones on downloading and in the feed's storage.

Maybe, those packages were broken in the feed's storage a relatively long time ago for some other reason (a bug in earlier versions of ProGet? a bug in our infrastructure?). And the issue starts to be observable only now because of caches or something like that...

At this point, thanks to your answers, it seems that ProGet is not the one to blame.

So, I ran a script yesterday that found all the broken packages in the feed, downloaded them from nuget.org, and republished them to ProGet. It was about 100 packages overall.

Now I'm sure that every package in our feed is valid. I'll keep monitoring the feed, and if new broken packages arise in the feed, I'll continue the investigation and let you know if ProGet is somehow related to the issue.

Thanks a lot, @atripp!