Posts made by atripp

Hi @inedo_1308 ,

I forgot how it worked in the preview migration, but the connection string is stored in the database directory (/var/proget/database/.pgsqlconn).

Thanks,
Alana

@inedo_1308 sounds good!

The code would almost certainly be the same, since it hasn't been updated since we did the PostgreSQL version of the script.

So, I think it's a race condition, though I don't know how it would happen. However, if it's a race condition, then it should be solved with an UPDLOCK (or whatever) in PostgreSQL.

1. SELECT finds no matching blob in the database (thus DockerBlob_Id is null)
2. ... small delay ...
3. UPDATE finds the matching blob because it was added (thus a row gets added to insert)
4. INSERT does run because there is a row in inserted
5. A NULL DockerBlob_Id is returned

If you're able to patch the procedure, could you add FOR UPDATE as follows? We are still relatively to PostgreSQL so I don't know if this the right way to do it in this case.

I think a second SELECT could also work, but I dunno.

CREATE OR REPLACE FUNCTION "DockerBlobs_CreateOrUpdateBlob"
(
    "@Feed_Id" INT,
    "@Blob_Digest" VARCHAR(128),
    "@Blob_Size" BIGINT,
    "@MediaType_Name" VARCHAR(255) = NULL,
    "@Cached_Indicator" BOOLEAN = NULL,
    "@Download_Count" INT = NULL,
    "@DockerBlob_Id" INT = NULL
)
RETURNS INT
LANGUAGE plpgsql
AS $$
BEGIN

    SELECT "DockerBlob_Id"
      INTO "@DockerBlob_Id"
      FROM "DockerBlobs"
     WHERE ("Feed_Id" = "@Feed_Id" OR ("Feed_Id" IS NULL AND "@Feed_Id" IS NULL))
       AND "Blob_Digest" = "@Blob_Digest"
   FOR UPDATE;

    WITH updated AS
    (
        UPDATE "DockerBlobs"
           SET "Blob_Size" = "@Blob_Size",
               "MediaType_Name" = COALESCE("@MediaType_Name", "MediaType_Name"),
               "Cached_Indicator" = COALESCE("@Cached_Indicator", "Cached_Indicator")
         WHERE ("Feed_Id" = "@Feed_Id" OR ("Feed_Id" IS NULL AND "@Feed_Id" IS NULL)) 
           AND "Blob_Digest" = "@Blob_Digest"
        RETURNING *
    )        
    INSERT INTO "DockerBlobs"
    (
        "Feed_Id",
        "Blob_Digest",
        "Download_Count",
        "Blob_Size",
        "MediaType_Name",
        "Cached_Indicator"
    )
    SELECT
        "@Feed_Id",
        "@Blob_Digest",
        COALESCE("@Download_Count", 0),
        "@Blob_Size",
        "@MediaType_Name",
        COALESCE("@Cached_Indicator", 'N')
    WHERE NOT EXISTS (SELECT * FROM updated)
    RETURNING "DockerBlob_Id" INTO "@DockerBlob_Id";

    RETURN "@DockerBlob_Id";

END $$;

Hmmm the only possibility I can see is that DockerBlobs_CreateOrUpdateBlob is returning NULL , which is failing conversion to int dockerBlobId. That's the only nullable converstion on that line.

There's gotta be some kind of bug with this postresql procedure. Maybe a race condition??

CREATE OR REPLACE FUNCTION "DockerBlobs_CreateOrUpdateBlob"
(
    "@Feed_Id" INT,
    "@Blob_Digest" VARCHAR(128),
    "@Blob_Size" BIGINT,
    "@MediaType_Name" VARCHAR(255) = NULL,
    "@Cached_Indicator" BOOLEAN = NULL,
    "@Download_Count" INT = NULL,
    "@DockerBlob_Id" INT = NULL
)
RETURNS INT
LANGUAGE plpgsql
AS $$
BEGIN

    SELECT "DockerBlob_Id"
      INTO "@DockerBlob_Id"
      FROM "DockerBlobs"
     WHERE ("Feed_Id" = "@Feed_Id" OR ("Feed_Id" IS NULL AND "@Feed_Id" IS NULL))
       AND "Blob_Digest" = "@Blob_Digest";

    WITH updated AS
    (
        UPDATE "DockerBlobs"
           SET "Blob_Size" = "@Blob_Size",
               "MediaType_Name" = COALESCE("@MediaType_Name", "MediaType_Name"),
               "Cached_Indicator" = COALESCE("@Cached_Indicator", "Cached_Indicator")
         WHERE ("Feed_Id" = "@Feed_Id" OR ("Feed_Id" IS NULL AND "@Feed_Id" IS NULL)) 
           AND "Blob_Digest" = "@Blob_Digest"
        RETURNING *
    )        
    INSERT INTO "DockerBlobs"
    (
        "Feed_Id",
        "Blob_Digest",
        "Download_Count",
        "Blob_Size",
        "MediaType_Name",
        "Cached_Indicator"
    )
    SELECT
        "@Feed_Id",
        "@Blob_Digest",
        COALESCE("@Download_Count", 0),
        "@Blob_Size",
        "@MediaType_Name",
        COALESCE("@Cached_Indicator", 'N')
    WHERE NOT EXISTS (SELECT * FROM updated)
    RETURNING "DockerBlob_Id" INTO "@DockerBlob_Id";

    RETURN "@DockerBlob_Id";

END $$;

Anyway we'll study that another day.. at least we think we know specifically where the issue is.

In case you're curious, here is #381

.... and now to figure out what could possibly be null in that specific area

@inedo_1308 finally!!! nice find :)

@inedo_1308 said in proget 500 Internal server error when pushing to a proget docker feed:

That change looks wrong to me, because (error.StatusCode == 500) is more specific/restrictive than (error.StatusCode >= 500 || context.Response.HeadersWritten)
In other words: It logs less than before.

Good spot / good find -- though we never actually raise anything except 500 anyway, so i thought it would be fine

public static DockerException Unknown(string message) => new DockerException(500, "UNKNOWN", message);

Anyway wriiting the detail to that array will hpefully be caught.

@inedo_1308 thanks for continuing to help us figure this out

Do you mind trying inedo/proget:25.0.9-ci.7?

I'm thinking it's some kind of middleware bug (our code? .NET code? who knows), and I can't see why the logging code I added didn't log that in diagnostic center.

Whatever the case, we can see the error JSON is being written: {"errors":[{"code":"UNKNOWN","message":"Nullable object must have a value.","detail":[]}]} ... so I just added the stack trace to detail element.

FYI, the code:

catch (Exception ex)
{
    WriteError(context, DockerException.Unknown(ex.Message), feed, w => w.WriteValue(ex.StackTrace)); // I added the final argument
}

....

private static void WriteError(AhHttpContext context, DockerException error, DockerFeed? feed, Action<JsonTextWriter>? writeDetail = null)
{
    /// code from before that should have worked
    if (error.StatusCode == 500)
        WUtil.LogFeedException(error.StatusCode, feed, context, error);

    if (!context.Response.HeadersWritten)
    {
        context.Response.Clear();
        context.Response.StatusCode = error.StatusCode;
        context.Response.ContentType = "application/json";

        using var jsonWriter = new JsonTextWriter(context.Response.Output);
        jsonWriter.WriteStartObject();
        jsonWriter.WritePropertyName("errors");
        jsonWriter.WriteStartArray();

        jsonWriter.WriteStartObject();
        jsonWriter.WritePropertyName("code");
        jsonWriter.WriteValue(error.ErrorCode);
        jsonWriter.WritePropertyName("message");
        jsonWriter.WriteValue(error.Message);
        jsonWriter.WritePropertyName("detail");
        jsonWriter.WriteStartArray();
        writeDetail?.Invoke(jsonWriter);
        jsonWriter.WriteEndArray();
        jsonWriter.WriteEndObject();

        jsonWriter.WriteEndArray();
        jsonWriter.WriteEndObject();
    }
}

Hi @james-woods_8996 ,

I looked into this a little more, and it turns out that our cloud storage providers do in fact support chunking -- but the outgoing replication code is not taking advantage of that. We would like to fix that in ProGet 2026.

However, in the meantime, we can fix the incoming replication code (i.e. what's throwing the error) pretty easily via PG-3102 - hopefully we'll get that in the upcoming maintenance release.

Thanks,
Alana

Hi @jfullmer_7346,

unfortunately it looks like the fix didn't work and made things slightly worse

We are reverting that change in ProGet 2025.9 and will try a new approach via PG-3100 - hopefully in ProGet 2025.10

Hi @inedo_1308

Just tested this 25.0.9-ci.6. Unfortunately, there is neither a stacktrace shown in GUI nor in stdout/stderr on the docker container that runs proget itself :-(

Sorry but just to confirm, you looked in the Admin > Diagnostic Center?

Basically, I just changed the code from this...

if (error.StatusCode >= 500 || context.Response.HeadersWritten)
   WUtil.LogFeedException(error.StatusCode, feed, context, error);

...to this...

if (error.StatusCode == 500)
   WUtil.LogFeedException(error.StatusCode, feed, context, error);

I really don't see how that wouldn't work to log it. I guess, next thing I could try is to write the stack trace in the detail.

Thanks,
Alana

I just changed some of the logging code via PG-3101, which will probably cause the exception to be logged.

It just finished building the image inedo/proget:25.0.9-ci.6 -- would someone mind giving that a try? Once we we get the stack trace it'll probably be one of those type of errors

@thomas_3037 @inedo_1308 @wechselberg-nisboerge_3629

Thanks!!!!

Thanks for the find @thomas_3037 !

What a strange scenario... I have no idea what that parameter does (aside from what's written in the docs). Do you know if that has to do with how the layers are compressed?

I think this definitely points to some kind of middleware bug. But still aren't sure how to better error report. We'll update a bit later!

Thanks,
Alana

Hi @james-woods_8996,

That's unexpected. Large files are supposed to be chunked into 50mb segments.

However, hooking over the code, chunking requires a seekable storage (i.e. random access) on the incoming server (i.e. proget-au.wtg.zone). Are you using cloud storage by chance?

If so, cloud providers do not currently support random access, so this is somewhat expected. I'm not sure if we could add that support, but we could likely work-around it on the Incoming code (i.e. what produces the logs) as well.

Open to ideas - not sure how important this is to get working or if it's just something you happened to notice -- let us know.

If you are NOT using cloud storage, can you temporarily enable verbose replication under Admin > Advanced Settings, and share the results up until that error?

Thanks,
Alana

@rickard-hagelin_7259 not a good surprise

Definitely open to ideas, but the general use case for these commands is like:

$> pgutil upack install --package=prod-tool --version=2.0.6 --feed=tools --target=/var/prod-tool

$> pgutil upack remove --package=prod-tool

Do you think that clarifying that remove deletes all the files would have helped?

Hi @ashleycanham ,

I see the issue... we'll get this fixed via PG-3099 in the upcoming maintenance release (Friday). This is a regerssion in ProGet 2025, so you could rollback and it should work fine.

Thanks,
Alana

Hi @wechselberg-nisboerge_3629 ,

Sure we can fix the username/password at the same time...

That said, if we don't hear back from @bohdan-cech_2403 then perhaps you can help by setting up a trial Artifactory cloud instance that ProGet fails to import from?

We are neither Maven nor Artifactory experts (or users for that matter), so it's very likely we uploaded things "wrong" or are using such "basic" test files.

Thanks,
Alana

Hi @inedo_1308,

Another user has reported this via ticket EDO-12089 (tagging for my reference), and we're currently analyzing their PCAP file.

We cannot reproduce it at all and have absolutely no idea where the error could be coming from -- the current theory is that it's a middleware issue by the way podman did requests (which was different than Docker client),.... but since you mentioned you can repro it on Docker client, then that's probably not it.

Anyway, if you're able to identify a version where this regressed that would be really helpful.

The reason nothing is logged is because it's explicitly suppressed in the code, but I don't know why. We may have to change that so we can see this message logged.

Please stay tuned.

Thanks,
Alana

Hi @bohdan-cech_2403 ,

In that case, the issue is in the importer -- maybe something with the API, it's really hard to say. This is a relatively new importer tool and we could only do so much testing.

Can you provide us with credentials to try this and attach a debugger?

You can share those to the earlier ticket or email to support at inedo dot com -- but we don't monitor the email, so just let us know when you do that.

Thanks,
Alana

Hi @tyler_5201 ,

We'll address this via PG-3098 in an upcoming maintenance release; basically we will add a permission check before chown on Docker/Linux for PostgreSQL. If the permissions are there, then chown won't be run.

Thanks,
Alana

Hi @james-woods_8996,

It sounds like there's a problem decrypting authentication tickets. This is almost certainly due to not having the same encryption key on both servers.

Check out https://docs.inedo.com/docs/installation/linux/docker-guide#supported-environment-variables to learn more, but it's an environment variable like this:

-e PROGET_ENCRYPTION_KEY='37D27A670394F7D82CE57F1F07D69747'

It's a 32-char hex string. You can just generate a GUID and delete the "-" to have one.

Thanks,
Alana

Hi @bohdan-cech_2403,

I wonder if there's some kind of issues with the artifacts. Can you try uploading some of those using the UI to ProGet?

You can start with the POM file, then upload the JAR.

That will hlep us identify what the issue might be.

Thanks,
Alana

hi @power_pille @pmsensi ,

Just as an update, we will revert the change via PG-3097 in the next maintenance release.

We will solve duplicate package names another day -- but we'll do it the "hard way", which is to update all the related tables when it occurs.

Thanks,
Alana

Hi @tyler_5201

Thanks for the additional info; so it's not that this is an actual "network drive" just (i.e. far from the server), it just happens to be how the configuration works.

I don't think we can/should try to detect the owner; do you if chown will fail if the user is already postgres? If so, then we can wrap this with a try/catch and throw a more helpful error.

Otherwise, we can try/catch and give a warning. But we want to make sure the user can see it when the database fails to load.

Thanks,
Alana

Hi @james-woods_8996 ,

Docker feeds utilize the multi-part upload part of the S3 API, which you can see being used in the BeginResumableUploadAsync method of our file system abstraction.

If they do not implement multi-part uploads, then you will not be able to use it with ProGet. Other feeds rely on that method as well.

I would check w/ the vendor to see what the issue could be - it's possible they simply don't support it.

Thanks,
Alana

Hi @parthu-reddy ,

This behavior is expected; these have a status of "unlisted", which you can see on NuGet.org if you look in the API or type in the URL directly:

• https://www.nuget.org/packages/System.ComponentModel.Composition/2010.2.11
• https://www.nuget.org/packages/System.ComponentModel.Composition/2010.2.11.1

ProGet does not hide unlisted packages by design, but shows them with an unlisted icon instead.

Thanks,
Alana

@power_pille @pmsensi thanks for helping to investigate this

What I can say is this is most definitely a regression in ProGet 2025.8 caused by PG-3047 -- I'm not sure if we want to revert that change or fix it. But rolling back to ProGet 2024.7 will definitely address the issue.

As for casing, it's all really strange to honest - some package ecosystems (rpm) are case sensitive, and different packages that differ by casing. Most are not insane like that.

However, due to a bug from a while back, "bad data" snuck into the PackageNameIds table from a vulnerability database update -- and that's why NuGet has those duplicates. Clearing them "blindly" from databases in the field isn't easy, but the feed reindex was a step in that direction.

Anyway we're going to continue researching how to address this

Hi @yaakov-smith_7984 ,

It looks like we forgot to ship the updated extension in 2025.8, so it's not "included" when you install ProGet. You should be able to update under Admin > Extensions, but unless you make the extensions directory (under Admin > Advacned Configuration) mapped to the docker host it will not "survive" a container reboot.

We could get you a prerelease container version if you'd like to try it as included, just let us know

Thanks,
Alana

@misael-esperanzate_5668 thanks for the reproduction details!

I was ablet to reproduce this very easily and fixed it via PG-3091 - it was an ordering problem

This will be in the upcoming maintenance release this Friday

Hi @tyler_5201

I'm not really familiar with all this topic, but why would you want to put the database file on an NFS? The PostgreSQL docs on clustering seem to warn against this.

That said, we added the chown command during early testing because PostgreSQL was failing with something like this:

The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The only way to get to work was:

1. Log into the pod via: $kubectl exec -n -i -t -- bash -il
2. Change the directory to: $cd /var/opt
3. Get the UID of postgres-User: $id -u postgres
4. Recursively change owner of the folder database: chown -R 101:101 database/

Hence, why we added the chown.

I'm not sure the best approach here -- we could try/catch, but I think the postgres user still needs to own the files?

Cheers,
Alana

Hi @kc_2466 ,

Understood, that was the same reason given before.

Our products' AD/LDAP integration works different than JIRA/TeamCity (live lookup vs sync) so it's just not that simple.

We're not willing to add complexity to work-around "dumb" organizational policies -- what you described is the whole point of AD/LDAP (centrally administered access).

It's possible to delegate group administration in AD, but if they don't want to do that... I guess they are okay with you just creating less-secure built-in users until they get around to completing their tickets ;)

Thanks,
Alana

Hi @thomas_3037 ,

ProGet does not support these, as they are not "real" Debian repositories:

These are used only by the Debian installer and are not installed on a Debian system under normal circumstances.

Unlikely we would add support for them, given the highly-limited use case.

Thanks,
Alana

@tames_0545 I just thought of something, perhaps try a different versions of ProGet, instead of latest -- like maybe 25.0.3, 25.0.4, etc, perhaps into ProGet 2024 as well

Hi @tames_0545 ,

Sorry, no idea -- there is definitely something wrong with your environment but aside from the advice I gave earlier, I don't have any other troubleshooting steps. You may want to try a different environment / Docker host.

As I mentioned, it could something wrong with your database mount (/opt/proget/database) -- maybe that is getting deleted? Or it could be your security configuration that is interfering with the running container. We have seen some hyper-aggressive security tools that make things so secure nothing runs.

Thanks,
Alana

Hi @tames_0545 ,

A connection string should not be specified.

I haven't seen this behavior before; the container runs postgres on start to listen to port 5728 and I can't imagine how that would fail without an error message.

Perhaps there's some kind of issue with /opt/proget/database and that's causing the postgres process to crash; I would start here.

The only other thing I can imagine is some kind of security configuration that is interfering with the running container.

I would keep it as simple as possible. Once you get it working, you can try adding in the encryption key. Unless you absolutely need to use a Docker secret (those are complex and don't really do anything), just specify the encryption key with an environment variable

-e PROGET_ENCRYPTION_KEY='37D27A670394F7D82CE57F1F07D69747'

Cheers,
Alana

Hi @kc_2466,

Thanks for the suggestion; we've had this request in the past but don't believe it's a good fit.

It ought to be trivial to add a group to the AD/LDAP server, so it doesn't make sense to add a "quasi-group" feature on the ProGet end. That would increase complexity and could introduce unbehavior that is challenging to support and troubleshoot.

We haven't seen other products doing something like this, likely for these reasons.

Cheers,
Alana

Hi @kc_2466 , nice catch! We'll get it fixed via PG-3060 in the next maintenance release!

Hi @m-lee_3921 ,

Thanks for the heads-up; we'll get this fixed via PG-3059 in the next maintenance release (scheduled for later today).

And you're right -- its basically the same issue. SQL Server silently truncates string values whereas PostgreSQL does not, hence these errors.

We decided to not auto-truncate, since it's better to know about these cases than not. In this case, it's a IPv6 address that was longer than the 20-character limit. We plan to refactor API key logging at some point, so we'll just continue to truncate v6 IPs.

Thanks,
Alana

Hi @carsten_1879 ,

Can you try out buildmaster:25.0.0-ci.23 container image?

I would do a new/fresh installation - new applications created in it should use the new Git library. It seems to work, but would like to see if it works for you as well.

Thanks,
Alana

@it_6176 thanks for sharing!

You're right, we did not consider this usecase when designing Debian connectors. Our "default mode" in ProGet is "dynamic indexes", but the quirkiness of the Debian repository "API" and obscene size of operating-system libraries make that impractical when there is a lot of feed activity (i.e. packages being added to the feed via caching).

We are going to redesign this at some point, and allow for some kind of "periodic indexing" for some connectors. That would be run as a job, as opposed to triggered by a user's request.

As for the database.... we've had one other user report that those SQLLite databases "blew up" as well, and our conclusion is that "something" crashed at the wrong time. Probably, due to the resource overusage? It has something to do with the WAL Files, but we're not totally sure.

I think we just need some kind of regular clean-up for these "huge" connectors so the database doesn't get so huge.

Hi @caterina ,

I agree and there doesn't seem to be any good usecases or official guidance from npm tags command, but we do not recommend using them: https://blog.inedo.com/npm/smarter-npm-versioning-with-semver/

From the api/technical standpoint, npm tags are a little weird -- a package themselves doesn't contain tags. Instead, the registry has a key/value tag store. By convention, the store has keys like latest and values like 4.8.1, but it seems to allow anything? At least from data we've seen on npmjs.org

In any case, the npm client seems to set a latest tag, so we'll just support that use case then.

FYI here is the logic/code I added:

        var tags = await this.GetPackageTagsAsync(purl.Name, false, cancellationToken: cancellationToken);
        if (tags.TryGetValue("latest", out var latestVersion) && latestVersion == PackageVersion<SemanticVersion2>.Parse(purl.Version))
            await DB.Npm_SetTagVersionAsync(toFeed.FeedId, purl.Name.Id, "latest", purl.Version, DateTime.UtcNow);

It sets the latest tag on the other feed if the current feed's latest tag points to the package you are promoting.

Thanks,
Alana

Hi @caterina ,

This behavior is expected as only the package itself is promoted, not repository metadata.

I'm not an npm developer, but one concern I have is that the latest tag doesn't always refer to the most recent version of a package? It's a little confusing to be honest, but I read that developers will sometimes use latest to "pin" what they feel is the most stable version?

I don't understand why this is a practice in npm, but I think this is probably an exception. So, we will change the behavior via PG-3057 such that the "latest" tag is set on the new feed when promoting latest-tagged npm package.

This will be in the next maintenance release of ProGet 2025.

Cheers,
Alana

Hi @phopkins_6694

Looks like this is a regression in ProGet 2025; we'll get it fixed in the next maintenance releaese, scheduled for this Friday via PG-3056

Cheers,
Alana

Hi @kc_2466 ,

Can you share how / why you're create Universal Packages in the UI? And perhaps how you're using upacks in general?

We added them as a way to introduce the format to new users, but didn't really anticipate it being used beyond that.

Thanks,
Alana

Hi @kc_2466 ,

We'll Add support for LC_TIME environment variable & user setting via PG-3054 in an upcoming maintenance release -- p[robably not this week's, but in the next or following.

Thanks,
Alana

Hi @carsten_1879 ,

We should have a build of BuildMaster 2025 later this week that you can try out that includes the new library. We'll let you know!

Thanks,
Alana

Hi @imm0rtalsupp0rt ,

Thanks - I was able to reproduce this issue. It's some long-standing behavior / bug, and not a trivial fix.

We will see about fixing it in the upcoming maintenance release (PG-3048), but it's lower priority and will likely get pushed back a few releases.

Cheers,

Alana

Nice idea! That was pretty trivial to add in there, and will be in the next maintenance release via PG-3038:

Hi @misael-esperanzate_5668 ,

I'm not really sure, but I'll explain how things work so it might help troubleshoot.

First, the Maven API does not provide a file listing. While you (as a user) can often "see" via an HTML page (like this listing at jboss.org), it's simply not available for listing in the API. The only required file in an artifact is the .pom file, so when you "pull" an artifact to ProGet, that's all you'll get.

Next, the "remote" icons next to the files indicate that they were cached (i.e. added to the feed) via a connector. That means the files were successfully written to disk, recorded in the database... but now they are gone. Hence, why you keep getting the "file not found" message.

The most likely culprit for this is something deleting the files from a packagestore. We often see security tools doing that for "safety" reasons, since they are .jar files that may be dangerous I guess.

Overall, the maven API is a very simple series of GETs. So perhaps, you can just experiment with this? First, start with a brand new feed and a connector.

Then, run the command:;

curl http://my.server.local:8624/maven2/my-new-feed/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar --output hamcrest-core-1.3.jar

You should see the file download via CURL. Afterwards, you should see the artifact as a cached package (Top Navigation > Packages > Select Cached). You should also see that .jar file on disk, written to the package store location (Manage Feed > Storage).

If you don't see a .jar file downloaded to that location on disk, then it means something is "blocking" the file being written. If it's there, then it means something is deleting it after.

Thanks,
Alana

Hi @rel_0477 ,

Sounds like this is a pretty specific edge case. Can you provide a reproduction case so we can take a look?

Thanks,
Alana