Posts made by atripp

Hi @fabrice-mejean,

Using packages.lock.json seemed to make the most sense to us too, but ultimately we decided not to use it for a few reasons.

First and foremost, none of the other .NET SBOM-generators seemed to use the packages.lock.json file. That's usually a sign that there's a "good reason" for us not to either.

From our perspective, pgutil builds scan is intended to be used in a CI environment, where dotnet build is run anyway and the assets file is already present. We don't have a use-case for an alternative workflow, where a build is not actually run.

In addition, packages.lock.json are still pretty niche and not widely used. You have to "go out of your way" to use it, and <PackageReference .../ > is by far the most common approach. It might be worth monitoring Issue #658 at CycloneDX/cyclonedx-dotnet to see anyone picks it up there.

Technically it's not all that complex to do, but it adds complexity and confusion... especially since most users will not be familiar with the differences between the lock and asset file. So it's not a good fit for pgutil builds scan.

HOWEVER, you could probably write ask ChatGPT to write a trivial PowerShell script that "Transforms" a lock file into a minimal SBOM document, and tweak it for what you want in ProGet. That same script could just upload the file to ProGet, or use pgutil as well.

Thanks,
Alana

Hi @layfield_8963 ,

Thanks that makes sense -- since you've already got a personal repo going, I think it makes sense to stick with that for now. If other users are interested, we can explore it.

We publish pgutil pretty regularly, so we'd need to automate updating that repository and that's just one more script t write and one more thing to break later :)

Thanks,
Alana

Hi @it_9582 ,

This is a known issue / UI quirk with Conan packages, and hopefully should only impact that one page in the UI.

To be honest I don't quite get the issue, but it has something to do with the fact that a Conan package is actually "set of packages that share a name and version". Each package in the set can define its own license file.

The particular page was never really designed for "package sets" so the display is a little weird. It's a nontrivial effort to fix and would obviously impact all other package types, so it's not a priority at the moment.

We would love to redo the UI at some point, s I think it'd mkae sense to do then.

Thanks,
Alana

Hi @Sigve-opedal_6476 ,

Thanks for clarifying; I should have mentioned that I know basically nothing about rpm except how a repository works

A gpgkey is just a file, right? I think you were on the right track with using an asset directory. I guess you would configure things like this, for an aset directory called gpg in the docker/gpg folder:

gpgkey=https://myproget.mycompany.com/endpoints/gpg/content/docker/gpg

There's really no "timestamp" on web-based files, the browser/protocol just does a GET and the bytes are returned. I can't imagine rpm is looking at a modified cache header.

Thanks,
Alana

Hi @aristo_4359 ,

I researched this a little further, and I'm afraid Conan Packages cannot be imported from Artifactory at this time; they do not behave like other repositories in Artifactory, which means it's a nontrivial effort to figure out how to get these imported using a different/alternative Artifactory API.

We will clarify this in upcoming ProGet release via PG-3122.

Thanks,
Alana

Hi @Sigve-opedal_6476,

I'm not really sure what you mean by this request.

An RPM repository is basically just a "dumb file server" with like repodata.xml and a bunch of index.tar.gz files. The rpm client downloads this files and does all the gpg stuff.

ProGet feeds implement an RPM repository and generate these indexes on the fly... but to the client, it seems like it's just downloading static files.

Thanks,
Alana

Hi @layfield_8963 ,

About the only knowledge of "homebrew" I have is that it's some kind of thing on Mac, perhaps like apt or chocolatey. I think we'd normally be all about hosting a "cask" or "keg", but I don't think that's what you're asking

It doesn't make sense for us to try supporting an ecosystem we know so little about. That's the same reason we never did a Chocolatey package on our own, but our friend @steviecoaster from Chocolatey created/maintains the proget package at chocolatey.org, and that has worked out just fine.

Anyway if you just need "something simple" from us like accepting a (simple) pull request or editing a build/deploy script, we might be able to do that. But otherwise it won't make sense for us to invest in learning about brew and supporting it.

Cheers,
Alana

Hi @tyler_5201

The underlying error is that there is no connection string, as you noticed.

The connection string is stored in a file (/var/proget/database/.pgsqlconn) that should be accessible to the container. I haven't tested it, but I guess if file is missing or deleted, then I suppose you might run into these issues.

It should be created on startup of a new container, however. So it's kind of weird. I think you'll want to "play" with it a bit, since there's clearly something going on with your permissions I'm thinking.

Note that the connection string can also be specified as an environment variable, but I don't think that applies here since you're trying to configure the embedded datbase:
https://docs.inedo.com/docs/installation/linux/docker-guide#supported-environment-variables

Thanks,
Alana

@udi-moshe_0021 sounds like it was a temporary outage on npmjs.org or perhaps even your proxy server. I wouldn't worry about it if it's working now since it's not something you could really control anyway

Hi @carl-westman_8110 ,

The error message means that the database wasn't updated as per normal during the start-up process. It's hard to guess why, as we have special handling for that.

It's likely that restarting the service would have fixed it, but downgrading and then upgrading would also force an upgrade as well. Unfortunately it's hard to say at this point.

Upgrading to 2025.10 should be fine.

Thanks,
Alana

Hi @jfullmer_7346,

It's nothing you did.

The underlying issue is that a bug in ProGet allowed WinSCP (ID=563) to be added to the PackageNameIds table; that should never have happened since nuget is case insensitive. We've since fixed that bug.

However, once you have a duplicate name, weird things happen since querying for "give me the PackageID for nuget://winscp" returns two results instead of one. So now when you query "give me the VersionID for (ID=563)-v6.5.3", a new entry is created.

This has been a very long-standing issue, but there aren't any major consequences to these "weird things" except casing in the UI and the health check now fails.

But we're on it :)

Thanks,
Alana

Hi @jfullmer_7346 ,

I haven't had a chance to look into more details, but thanks for providing the results of the query.

FYI - the PackageNameIds and PackageVersionIds are designed as a kind of "permanent, read-only record" -- once added they are not deleted or modified. Even if all packages are deleted (i.e. FeedPackageVersions). This is why this the "duplicate name" is such a headache to deal with.

That said, on a quick glance, we can see exactly where the error is coming from: there are duplicate versions (i.e. (ID=563)-v6.5.3 and (ID=562)-v6.5.3). So, when we try to deduplicate (ID=563) and (ID=562) (i.e. winscp and WinSCP), we get the error as expected.

What's not expected is that those versions were not de-duplicated in the earlier pass. My guess is that it's related to winscp being in one feed and WinSCP being in the other -- we tried to be conservative, and keep the de-duplication to packages related to the feed.

I'm thinking we just change that logic to "all packages of the feed type". Anyway, please stay tuned. We'll try to get it in the next maintencne release.

Thanks,
Alana

@jfullmer_7346 thanks for giving it a shot, we'll take a closer look!

The "good news" is that the error message is a "sanity check" failure, so now have an idea of what's causing the error:

    -- Sanity Check (ensure there are no duplicate versions)
    IF EXISTS (
        SELECT * 
         FROM "PackageVersionIds" PV_D,
              "PackageVersionIds" PV_C
        WHERE PV_D."PackageName_Id" = "@Duplicate_PackageName_Id"
          AND PV_C."PackageName_Id" = "@Canonical_PackageName_Id"
          AND PV_D."Package_Version" = PV_C."Package_Version"
          AND (   (PV_D."Qualifier_Text" IS NULL AND PV_C."Qualifier_Text" IS NULL)
               OR (PV_D."Qualifier_Text" = PV_C."Qualifier_Text") )
       ) THEN RAISE EXCEPTION 'Cannot deduplicate given nameid'; RETURN; END IF;

In this case, it's saying that there are "duplicate versions" remaining (i.e. WinSCP-1.0.0 and winscp-1.0.0). Those should have been de-duplicated earlier. I wonder if the PackageVersionIds_GetDuplicates() function is not returning the right results.

I'm not sure what your experience w/ PostgreSQL is, but are you able to query the embedded database? If not, that's fine... it's not meant to be easy to query.

Also, should the integrity check be taking 30 minutes?

Maybe. The integrity check needs to verify file hashes, so that involves opening and streaming through all the files on disk. So when you have a lot of large packages, then it's gonna take a while.

Hi @bohdan-cech_2403,

ProGet can handle most invalid Maven version numbers, but Borca-SR2 is really invalid and isn't currently supported. Our "name vs version" parsing requires that versions start with numbers, artifacts start with letters. This has been a Maven rule for 20+ years now.

It's non trivial and quite risky to change our parsing logic so it's not something we're keen on doing in a maintenance release. This scenario seems to be very rare and impact ancient artifacts and a few cases where authors didn't use Maven to deploy the artifacts.

Thanks,
Alana

@uli_2533 thanks for the additional info, great find in the source code too!!

On our end, I was looking at the PUT code, where a 301 kind of made sense. I think it must have been some kind of regression on the GET request? Not sure why it didn't get noticed before, but it's a trivial fix.

PG-3108 will be in the next maintenance release (Sep 19)... or if you want to try it now, it's in inedo/proget:25.0.10-ci.9 Docker image.

Thanks,
Alana

Hi @bohdan-cech_2403 ,

I'm not sure if that's the issue...

Returning a 201 has been the behavior for as long as we've had the feed (even the old version of the feed). The official Maven client does not seem to complain or cause any error in our testing, and no other user reported it as a problem.

Any idea why it's happening "all of a sudden" for you? Is there a new version of Maven or something?

FYI the PUT uploads for hash files are ignored and a 201 is always returned.

Thanks,
Alana

@jorgen-nilsson_1299 said in Machine ID changes after restart:

What is the Machine ID based on and how can I trouble shoot this? Any way to set a static Machine ID?

The Machine ID is based on the CPU Vendor ID, Machine Name (host name on Docker), and OS Version.

Hopefully @felfert gave some advice on how to make sure those don't change,

@pariv_0352 the code is not fixed in 25.0.9

However, inedo/proget:25.0.10-ci.5 will have the new code that should prevent this error

@jfullmer_7346 thanks! As an FYI...

• Names and Versions are centrally indexed, and were intended to be "write-only" by design
• NuGet does not have case-sensitive names, but some earlier bugs allowed duplicate names to be created
• we are adding Duplicate Names (e.g. winscp and WinSCP) and Duplicate Versions (e.g. winscp-4.0.0 and WinSCP-4.0.0 checking to feed integrity
• When re-indexing a feed, you'll get an option to de-duplicate names/versions, which will fix it across all feeds
• Pulling or Publishing a package will update the casing of a centrally-indexed name
• Also we're getting rid of the concept of "Noncannonical Names" altogether, since we've discovered many NuGet packages "change casing" at some version

Hi @pmsensi ,

Thanks for the heads-up; looks like there was a replication issue with one of our edge nodes. It should be there now

Thanks,
Alana

Thanks @felfert it was an easy add, so look out for PG-3106 in the next maintenance release!

Hi @mmaharjan_0067

The "unexpected argument" running without quotes is expected, but it works fine when I run with quotes. I'm afraid I can't reproduce this.

I would check under Admin > Diagnostic Center to see if anythings logged. Alternatively, you may need to query the API by doing something like this:

curl http://server:8624/endpoints/Test/metadata/metadata-test/v1/1%20-%20Normal.txt

Thanks,
Alana

Hi @bohdan-cech_2403 ,

Thanks for sharing that; I can confirm receipt, reproduction, and fixing (PG-3105). If you'd like to try it, you can get the fix in 25.0.10-ci.2 - otherwise it'll be in the next maintenance release (next Friday).

@wechselberg-nisboerge_3629 FYI I entered the url/username/password into ProGet, but I got the message "Failed to find any registries." So, I tried with curl and I got this:

$> curl "https://artifactory.REDACTED.com/artifactory/api/repositories?type=local" --user support
Enter host password for user 'support':
{
  "errors" : [ {
    "status" : 401,
    "message" : "Artifactory configured to accept only encrypted passwords but received a clear text password, getting the encrypted password can be done via the WebUI."
  } ]
}

I have no idea what explains the different behavior. Anyway, I logged into the portal on that URL and generated some kind of key, and it let me connect.

Thanks,
Alana

Hi @felfert,

So far as I can tell, the IP isn't currently logged in these messages... I can see how that would be helpful.

I can certainly do that (which would then show the X-Forwarded when available), but I wanted to make sure I'm looking in the right place. Because I don't see IP info now.

Thanks,
Alana

Hi @felfert ,

As an update, we'd are planning on this pattern instead of row/table locking (PG-3104). It gives us a lot more control and makes it a lot easier to avoid deadlocks.

I still can't reproduce the issue, but I see no reason this won't work.

CREATE OR REPLACE FUNCTION "DockerBlobs_CreateOrUpdateBlob"
(
    "@Feed_Id" INT,
    "@Blob_Digest" VARCHAR(128),
    "@Blob_Size" BIGINT,
    "@MediaType_Name" VARCHAR(255) = NULL,
    "@Cached_Indicator" BOOLEAN = NULL,
    "@Download_Count" INT = NULL,
    "@DockerBlob_Id" INT = NULL
)
RETURNS INT
LANGUAGE plpgsql
AS $$
BEGIN

	-- avoid race condition when two procs call at exact same time
	PERFORM PG_ADVISORY_XACT_LOCK(HASHTEXT(CONCAT_WS('DockerBlobs_CreateOrUpdateBlob', "@Feed_Id", LOWER("@Blob_Digest"))));

    SELECT "DockerBlob_Id"
      INTO "@DockerBlob_Id"
      FROM "DockerBlobs"
     WHERE ("Feed_Id" = "@Feed_Id" OR ("Feed_Id" IS NULL AND "@Feed_Id" IS NULL))
       AND "Blob_Digest" = "@Blob_Digest";

    WITH updated AS
    (
        UPDATE "DockerBlobs"
           SET "Blob_Size" = "@Blob_Size",
               "MediaType_Name" = COALESCE("@MediaType_Name", "MediaType_Name"),
               "Cached_Indicator" = COALESCE("@Cached_Indicator", "Cached_Indicator")
         WHERE ("Feed_Id" = "@Feed_Id" OR ("Feed_Id" IS NULL AND "@Feed_Id" IS NULL)) 
           AND "Blob_Digest" = "@Blob_Digest"
        RETURNING *
    )        
    INSERT INTO "DockerBlobs"
    (
        "Feed_Id",
        "Blob_Digest",
        "Download_Count",
        "Blob_Size",
        "MediaType_Name",
        "Cached_Indicator"
    )
    SELECT
        "@Feed_Id",
        "@Blob_Digest",
        COALESCE("@Download_Count", 0),
        "@Blob_Size",
        "@MediaType_Name",
        COALESCE("@Cached_Indicator", 'N')
    WHERE NOT EXISTS (SELECT * FROM updated)
    RETURNING "DockerBlob_Id" INTO "@DockerBlob_Id";

    RETURN "@DockerBlob_Id";

END $$;

@felfert amazing!! That script will come in handy when we need to help users patch their instance; we can also try to add something that allows you to patch via the UI as well!!

@felfert thanks for confirming!!

FYI tThe fix has not been applied yet to code yet, but you can patch the stored procedure (painfully) as a workaround for now. We will try to find a better solution. The only thing I can imagine happening is that the PUT is happening immediately after the PATCH finishes, but before the client receives a 200 response. I have no idea though.

We'll figure something out, now that we know where it is thanks to your help!!

Hi @parthu-reddy,

ProGet 2025 supports existing Maven classic feeds; you should be able to migrate just as you were in ProGet 2024.

Thanks,
Alana

Hi @parthu-reddy ,

Since these are network-level errors, you would need to use a tool like Wireshark or another packet analyzer to troubleshoot these kind of connectivity failures.

Thanks,
Alana

Did that, verified that the function actually has changed and did another test. Unfortunately this did not help, error was exactly the same like in my above wireshark dump.
Or does one have to "compile" the function somehow after replacing? (I never dealt with SQL functions before and in general have very limited SQL knowledge.)

Ah that's a shame! We're kind of new to "patching" functions like this in postgresql, but I think that should have worked to change the code. And also the code change should have worked.

If you don't mind trying one other patch, where we select out the Blob_Id again in the end.

CREATE OR REPLACE FUNCTION "DockerBlobs_CreateOrUpdateBlob"
(
    "@Feed_Id" INT,
    "@Blob_Digest" VARCHAR(128),
    "@Blob_Size" BIGINT,
    "@MediaType_Name" VARCHAR(255) = NULL,
    "@Cached_Indicator" BOOLEAN = NULL,
    "@Download_Count" INT = NULL,
    "@DockerBlob_Id" INT = NULL
)
RETURNS INT
LANGUAGE plpgsql
AS $$
BEGIN

    SELECT "DockerBlob_Id"
      INTO "@DockerBlob_Id"
      FROM "DockerBlobs"
     WHERE ("Feed_Id" = "@Feed_Id" OR ("Feed_Id" IS NULL AND "@Feed_Id" IS NULL))
       AND "Blob_Digest" = "@Blob_Digest"
   FOR UPDATE;

    WITH updated AS
    (
        UPDATE "DockerBlobs"
           SET "Blob_Size" = "@Blob_Size",
               "MediaType_Name" = COALESCE("@MediaType_Name", "MediaType_Name"),
               "Cached_Indicator" = COALESCE("@Cached_Indicator", "Cached_Indicator")
         WHERE ("Feed_Id" = "@Feed_Id" OR ("Feed_Id" IS NULL AND "@Feed_Id" IS NULL)) 
           AND "Blob_Digest" = "@Blob_Digest"
        RETURNING *
    )        
    INSERT INTO "DockerBlobs"
    (
        "Feed_Id",
        "Blob_Digest",
        "Download_Count",
        "Blob_Size",
        "MediaType_Name",
        "Cached_Indicator"
    )
    SELECT
        "@Feed_Id",
        "@Blob_Digest",
        COALESCE("@Download_Count", 0),
        "@Blob_Size",
        "@MediaType_Name",
        COALESCE("@Cached_Indicator", 'N')
    WHERE NOT EXISTS (SELECT * FROM updated)
    RETURNING "DockerBlob_Id" INTO "@DockerBlob_Id";

    SELECT "DockerBlob_Id"
      INTO "@DockerBlob_Id"
      FROM "DockerBlobs"
     WHERE ("Feed_Id" = "@Feed_Id" OR ("Feed_Id" IS NULL AND "@Feed_Id" IS NULL))
       AND "Blob_Digest" = "@Blob_Digest"

    RETURN "@DockerBlob_Id";

END $$;

If this doesn't do the trick, I think we need to look a lot closer.

Hi @inedo_1308 ,

I forgot how it worked in the preview migration, but the connection string is stored in the database directory (/var/proget/database/.pgsqlconn).

Thanks,
Alana

@inedo_1308 sounds good!

The code would almost certainly be the same, since it hasn't been updated since we did the PostgreSQL version of the script.

So, I think it's a race condition, though I don't know how it would happen. However, if it's a race condition, then it should be solved with an UPDLOCK (or whatever) in PostgreSQL.

1. SELECT finds no matching blob in the database (thus DockerBlob_Id is null)
2. ... small delay ...
3. UPDATE finds the matching blob because it was added (thus a row gets added to insert)
4. INSERT does run because there is a row in inserted
5. A NULL DockerBlob_Id is returned

If you're able to patch the procedure, could you add FOR UPDATE as follows? We are still relatively to PostgreSQL so I don't know if this the right way to do it in this case.

I think a second SELECT could also work, but I dunno.

CREATE OR REPLACE FUNCTION "DockerBlobs_CreateOrUpdateBlob"
(
    "@Feed_Id" INT,
    "@Blob_Digest" VARCHAR(128),
    "@Blob_Size" BIGINT,
    "@MediaType_Name" VARCHAR(255) = NULL,
    "@Cached_Indicator" BOOLEAN = NULL,
    "@Download_Count" INT = NULL,
    "@DockerBlob_Id" INT = NULL
)
RETURNS INT
LANGUAGE plpgsql
AS $$
BEGIN

    SELECT "DockerBlob_Id"
      INTO "@DockerBlob_Id"
      FROM "DockerBlobs"
     WHERE ("Feed_Id" = "@Feed_Id" OR ("Feed_Id" IS NULL AND "@Feed_Id" IS NULL))
       AND "Blob_Digest" = "@Blob_Digest"
   FOR UPDATE;

    WITH updated AS
    (
        UPDATE "DockerBlobs"
           SET "Blob_Size" = "@Blob_Size",
               "MediaType_Name" = COALESCE("@MediaType_Name", "MediaType_Name"),
               "Cached_Indicator" = COALESCE("@Cached_Indicator", "Cached_Indicator")
         WHERE ("Feed_Id" = "@Feed_Id" OR ("Feed_Id" IS NULL AND "@Feed_Id" IS NULL)) 
           AND "Blob_Digest" = "@Blob_Digest"
        RETURNING *
    )        
    INSERT INTO "DockerBlobs"
    (
        "Feed_Id",
        "Blob_Digest",
        "Download_Count",
        "Blob_Size",
        "MediaType_Name",
        "Cached_Indicator"
    )
    SELECT
        "@Feed_Id",
        "@Blob_Digest",
        COALESCE("@Download_Count", 0),
        "@Blob_Size",
        "@MediaType_Name",
        COALESCE("@Cached_Indicator", 'N')
    WHERE NOT EXISTS (SELECT * FROM updated)
    RETURNING "DockerBlob_Id" INTO "@DockerBlob_Id";

    RETURN "@DockerBlob_Id";

END $$;

Hmmm the only possibility I can see is that DockerBlobs_CreateOrUpdateBlob is returning NULL , which is failing conversion to int dockerBlobId. That's the only nullable converstion on that line.

There's gotta be some kind of bug with this postresql procedure. Maybe a race condition??

CREATE OR REPLACE FUNCTION "DockerBlobs_CreateOrUpdateBlob"
(
    "@Feed_Id" INT,
    "@Blob_Digest" VARCHAR(128),
    "@Blob_Size" BIGINT,
    "@MediaType_Name" VARCHAR(255) = NULL,
    "@Cached_Indicator" BOOLEAN = NULL,
    "@Download_Count" INT = NULL,
    "@DockerBlob_Id" INT = NULL
)
RETURNS INT
LANGUAGE plpgsql
AS $$
BEGIN

    SELECT "DockerBlob_Id"
      INTO "@DockerBlob_Id"
      FROM "DockerBlobs"
     WHERE ("Feed_Id" = "@Feed_Id" OR ("Feed_Id" IS NULL AND "@Feed_Id" IS NULL))
       AND "Blob_Digest" = "@Blob_Digest";

    WITH updated AS
    (
        UPDATE "DockerBlobs"
           SET "Blob_Size" = "@Blob_Size",
               "MediaType_Name" = COALESCE("@MediaType_Name", "MediaType_Name"),
               "Cached_Indicator" = COALESCE("@Cached_Indicator", "Cached_Indicator")
         WHERE ("Feed_Id" = "@Feed_Id" OR ("Feed_Id" IS NULL AND "@Feed_Id" IS NULL)) 
           AND "Blob_Digest" = "@Blob_Digest"
        RETURNING *
    )        
    INSERT INTO "DockerBlobs"
    (
        "Feed_Id",
        "Blob_Digest",
        "Download_Count",
        "Blob_Size",
        "MediaType_Name",
        "Cached_Indicator"
    )
    SELECT
        "@Feed_Id",
        "@Blob_Digest",
        COALESCE("@Download_Count", 0),
        "@Blob_Size",
        "@MediaType_Name",
        COALESCE("@Cached_Indicator", 'N')
    WHERE NOT EXISTS (SELECT * FROM updated)
    RETURNING "DockerBlob_Id" INTO "@DockerBlob_Id";

    RETURN "@DockerBlob_Id";

END $$;

Anyway we'll study that another day.. at least we think we know specifically where the issue is.

In case you're curious, here is #381

.... and now to figure out what could possibly be null in that specific area

@inedo_1308 finally!!! nice find :)

@inedo_1308 said in proget 500 Internal server error when pushing to a proget docker feed:

That change looks wrong to me, because (error.StatusCode == 500) is more specific/restrictive than (error.StatusCode >= 500 || context.Response.HeadersWritten)
In other words: It logs less than before.

Good spot / good find -- though we never actually raise anything except 500 anyway, so i thought it would be fine

public static DockerException Unknown(string message) => new DockerException(500, "UNKNOWN", message);

Anyway wriiting the detail to that array will hpefully be caught.

@inedo_1308 thanks for continuing to help us figure this out

Do you mind trying inedo/proget:25.0.9-ci.7?

I'm thinking it's some kind of middleware bug (our code? .NET code? who knows), and I can't see why the logging code I added didn't log that in diagnostic center.

Whatever the case, we can see the error JSON is being written: {"errors":[{"code":"UNKNOWN","message":"Nullable object must have a value.","detail":[]}]} ... so I just added the stack trace to detail element.

FYI, the code:

catch (Exception ex)
{
    WriteError(context, DockerException.Unknown(ex.Message), feed, w => w.WriteValue(ex.StackTrace)); // I added the final argument
}

....

private static void WriteError(AhHttpContext context, DockerException error, DockerFeed? feed, Action<JsonTextWriter>? writeDetail = null)
{
    /// code from before that should have worked
    if (error.StatusCode == 500)
        WUtil.LogFeedException(error.StatusCode, feed, context, error);

    if (!context.Response.HeadersWritten)
    {
        context.Response.Clear();
        context.Response.StatusCode = error.StatusCode;
        context.Response.ContentType = "application/json";

        using var jsonWriter = new JsonTextWriter(context.Response.Output);
        jsonWriter.WriteStartObject();
        jsonWriter.WritePropertyName("errors");
        jsonWriter.WriteStartArray();

        jsonWriter.WriteStartObject();
        jsonWriter.WritePropertyName("code");
        jsonWriter.WriteValue(error.ErrorCode);
        jsonWriter.WritePropertyName("message");
        jsonWriter.WriteValue(error.Message);
        jsonWriter.WritePropertyName("detail");
        jsonWriter.WriteStartArray();
        writeDetail?.Invoke(jsonWriter);
        jsonWriter.WriteEndArray();
        jsonWriter.WriteEndObject();

        jsonWriter.WriteEndArray();
        jsonWriter.WriteEndObject();
    }
}

Hi @james-woods_8996 ,

I looked into this a little more, and it turns out that our cloud storage providers do in fact support chunking -- but the outgoing replication code is not taking advantage of that. We would like to fix that in ProGet 2026.

However, in the meantime, we can fix the incoming replication code (i.e. what's throwing the error) pretty easily via PG-3102 - hopefully we'll get that in the upcoming maintenance release.

Thanks,
Alana

Hi @jfullmer_7346,

unfortunately it looks like the fix didn't work and made things slightly worse

We are reverting that change in ProGet 2025.9 and will try a new approach via PG-3100 - hopefully in ProGet 2025.10

Hi @inedo_1308

Just tested this 25.0.9-ci.6. Unfortunately, there is neither a stacktrace shown in GUI nor in stdout/stderr on the docker container that runs proget itself :-(

Sorry but just to confirm, you looked in the Admin > Diagnostic Center?

Basically, I just changed the code from this...

if (error.StatusCode >= 500 || context.Response.HeadersWritten)
   WUtil.LogFeedException(error.StatusCode, feed, context, error);

...to this...

if (error.StatusCode == 500)
   WUtil.LogFeedException(error.StatusCode, feed, context, error);

I really don't see how that wouldn't work to log it. I guess, next thing I could try is to write the stack trace in the detail.

Thanks,
Alana

I just changed some of the logging code via PG-3101, which will probably cause the exception to be logged.

It just finished building the image inedo/proget:25.0.9-ci.6 -- would someone mind giving that a try? Once we we get the stack trace it'll probably be one of those type of errors

@thomas_3037 @inedo_1308 @wechselberg-nisboerge_3629

Thanks!!!!

Thanks for the find @thomas_3037 !

What a strange scenario... I have no idea what that parameter does (aside from what's written in the docs). Do you know if that has to do with how the layers are compressed?

I think this definitely points to some kind of middleware bug. But still aren't sure how to better error report. We'll update a bit later!

Thanks,
Alana

Hi @james-woods_8996,

That's unexpected. Large files are supposed to be chunked into 50mb segments.

However, hooking over the code, chunking requires a seekable storage (i.e. random access) on the incoming server (i.e. proget-au.wtg.zone). Are you using cloud storage by chance?

If so, cloud providers do not currently support random access, so this is somewhat expected. I'm not sure if we could add that support, but we could likely work-around it on the Incoming code (i.e. what produces the logs) as well.

Open to ideas - not sure how important this is to get working or if it's just something you happened to notice -- let us know.

If you are NOT using cloud storage, can you temporarily enable verbose replication under Admin > Advanced Settings, and share the results up until that error?

Thanks,
Alana

@rickard-hagelin_7259 not a good surprise

Definitely open to ideas, but the general use case for these commands is like:

$> pgutil upack install --package=prod-tool --version=2.0.6 --feed=tools --target=/var/prod-tool

$> pgutil upack remove --package=prod-tool

Do you think that clarifying that remove deletes all the files would have helped?

Hi @ashleycanham ,

I see the issue... we'll get this fixed via PG-3099 in the upcoming maintenance release (Friday). This is a regerssion in ProGet 2025, so you could rollback and it should work fine.

Thanks,
Alana

Hi @wechselberg-nisboerge_3629 ,

Sure we can fix the username/password at the same time...

That said, if we don't hear back from @bohdan-cech_2403 then perhaps you can help by setting up a trial Artifactory cloud instance that ProGet fails to import from?

We are neither Maven nor Artifactory experts (or users for that matter), so it's very likely we uploaded things "wrong" or are using such "basic" test files.

Thanks,
Alana

Hi @inedo_1308,

Another user has reported this via ticket EDO-12089 (tagging for my reference), and we're currently analyzing their PCAP file.

We cannot reproduce it at all and have absolutely no idea where the error could be coming from -- the current theory is that it's a middleware issue by the way podman did requests (which was different than Docker client),.... but since you mentioned you can repro it on Docker client, then that's probably not it.

Anyway, if you're able to identify a version where this regressed that would be really helpful.

The reason nothing is logged is because it's explicitly suppressed in the code, but I don't know why. We may have to change that so we can see this message logged.

Please stay tuned.

Thanks,
Alana

Hi @bohdan-cech_2403 ,

In that case, the issue is in the importer -- maybe something with the API, it's really hard to say. This is a relatively new importer tool and we could only do so much testing.

Can you provide us with credentials to try this and attach a debugger?

You can share those to the earlier ticket or email to support at inedo dot com -- but we don't monitor the email, so just let us know when you do that.

Thanks,
Alana

Hi @tyler_5201 ,

We'll address this via PG-3098 in an upcoming maintenance release; basically we will add a permission check before chown on Docker/Linux for PostgreSQL. If the permissions are there, then chown won't be run.

Thanks,
Alana