Best posts made by atripp

Hi @kichikawa_2913 ,

PG-2290 was the regression fix, and it was included in ProGet 2022.22

Cheers,
Alana

Hi @rob-leadbeater_2457 ,

It sounds like you'll need to grant yourself (UserB) "sysadmin" access. You should probably also grant the local Administrators group access as well while you're at it :)

The easiest way to do this is with the SQL Server management tools, since it's all gui-driven. That's a free download/install from Microsoft, and I would recommend installing it on the server anyways. But it's also possible to do it with the command line/scripts.

Microsoft has some guidance on how to do this;
https://learn.microsoft.com/en-us/sql/database-engine/configure-windows/connect-to-sql-server-when-system-administrators-are-locked-out?view=sql-server-ver16

Once you've given yourself sysadmin, you should have no issues.

Alana

Hi @mahdy-merry_6333 ,

It's possible, and you can do this by creating a second feed that effectively proxies the first feed and uses disk storage. You could use a Connector on the two feeds or REplication.

Cheers,
Alana

Hi @kichikawa_2913,

You can ignore the Age field; it's just used internally by Visual Studio, along with the Id. Not sure why, but it's a strange value in PortablePDB files.

Here's more on what Age means if you're dying to read PE-COFF specifications

https://github.com/dotnet/corefx/blob/master/src/System.Reflection.Metadata/specs/PE-COFF.md#codeview-debug-directory-entry-type-2

Best,
Alana

Hi @jim-borden_4965 ,

It looks like that isn't supported; but it should be relatively easy :)

I added a change (PG-2305) that we'll try to get in the next release (scheduled Mar 24).

Cheers,
Alana

Hi @h-morgenthaler_3015, we are currently investigating this. ,It sounds like an issue we fixed in ProGet 2022 already, but maybe it didn't get merged over. Please stay tuned.

Hi @scott-wright_8356 ,

Thanks; it sounds like the underlying issue is indeed Google Android connector you added. I don't have an information or idea on why that might be the case... some third-party repos (Maven, PyPi, NuGet, etc.) are just really slow and buggy.

ProGet is basically acting like a proxy server here, so "garbage in, garbage out" applies - if the connector is slow and error-prone, then you will experience slowness and errors.

Sometimes third-party repos will create packages that don't follow the specifications but just happen to work in certain versions of clients. That could be what's happening here. In cases like that, we can really only address those on a case-by-case basis, and we'd need to know exactly how to reproduce the error and investigate it later.

Best,
Alana

Thanks so much @martijn_9956; after researching the matter further, it seems that the MERGE can be a pretty buggy statement, and its behavior seems to vary based on the operating system, SQL Server SKU, patch version, and probably the phase of the moon.

We will rewrite this procedure to use a more straight-forward UPDATE/INSERT/DELETE (like you did) via PG-2437, which will ship in next maintenance release.

Hi @ssuenaga_1020 ,

Thanks for the detailed report; this seems easy enough to fix, so we'll get this addressed in the next maintenance release (2023.20) via PG-2499.

Cheers,
Alana

Hi @Justinvolved ,

You should be able to see some kind of error message on Windows Event Logs, but my guess is that the account doesn't have access to the BuildMaster SQL Server database .. you'll need to grant that.

Cheers,
Alana

And I'm sure you noticed but looks like this was released :)

Hi @mharen,

You can try to enable Windows Integrated Authentication in ProGet, and then use two sites in IIS (once with WIA, one without), but that will only work for NuGet packages. Other client (npm, docker) do not support WIA.

Otherwise, you can take a "DMZ" approach, and configure two instances of ProGet - one for Devs, one for CI systems - then use a connector. The two-instance approach also has the benefit of reducing load, though not quite as much as a load balancing configurating.

Cheers,
Alana

@itops_6398 glad that worked! I noticed same quirk in the UI and fixed it :)

Probably why we/no one noticed before... editing the connector just happened to fix it once the value was saved.

Hi @sbaeurle ,

That's definitely strange; there's really nothing in ProGet that would cause intermittent 403s.

If you aren't doing so already, I would suggest switching to using API Keys in ProGet instead of LDAP. Make sure they are not "Personal Keys" but Feed keys. That will eliminate any potential issue with LDAP queries returning inconsistent results.

Otherwise, you'd almost need to capture those bad requests and see how they're different. ProGet does not have a means of capturing HTTP traffic, so you'd have to use something like Fiddler or Wireshark.

Cheers,
Alana

Added to our PRoGet 2024 roadmap to ivnestigate/explore - it might be easier this time!

Hi @PhilipWhite ,

You're correct, this is a regression in the validation code...

if (!value.All(c => (char.IsLetterOrDigit(c) && char.IsLower(c)) || c == '.' || c == '_' || c == '-' || c == '/'))
    return new ValidationResults(false, $"Docker Repository names may only contain lowercase letters, digits, periods, underscores, slashes, or dashes.");

I guess we recently added char.IsLower because UpperCase characters caused all sorts of problems.... easy fix, and easy work around.

1. Just enter blah in the Repository name field, Click Save
2. Click "view all"
3. Click on the Docker repository in that view, edit

We'll get this fixed for teh next maintenance release via BM-3932

Cheers,
Alana

Hi @m-karing_2439 ,

Unfortunately, this repository doesn't following PyPI repository conventions.

For example:

• No way to see latest packages (should be https://download.pytorch.org/whl/cu118/rss/updates.xml)
• Does not return anything for /simple listing (e.g. https://download.pytorch.org/whl/cu118/simple)
• Does not support JSON-based API to retrieve package metatadata

I guess it "happens to work" in pip , or perhaps they install a plugin that allows it to work... but ProGet is designed around PyPi repository specs. which is why a non-confirming download site won't work very well.
I'll add a note to investigate this further, but for the time being you should just download the wheels and upload them to a feed in ProGet.

Cheers,
Alana

Hi @forbzie22_0253 ,

ProGet Free edition may be used in commercial use. There is no limit on the number of instances, but a ProGet Free instance may not connect/network to another ProGet instance -- they need to be stand-alone instances.

ProGet Free instances can be connected to NuGet.org, just not other instances of ProGet.

Best,
Alana

Hi @yogeshshines_9136 ,

This should be resolved by downloading the latest Inedo Hub from https://my.inedo.com/downloads

Thanks,
Alana

Hi @scott-wright_8356,

It's a bit hard to figure out how to help or troubleshoot with the information here, but a few key points:

• A "healthy" Maven connector doesn't really mean much; just that the URL can be accessed by the ProGet server.
• A Maven repository (like the one you are linking to) is just a "web file system" with url/folder based conventions
• A Maven "package" consists of a .pom file and zero or more other files (.jar); think of it liked an unzipped package that you can add/delete files from
• This is why you can't import only .jar files - you need the .pom file (which serves as the package manifest)
• Sometime marvin "packages" contain malformed .pom files that happen to work in some versions of the client, and ProGet will sometimes error on reading those; maybe that's what happening here

To help on this, we'll need to know specifically where the issue is and have instructions on how to reproduce it on a new feed/instance of ProGet.

First, I would start by trying to download the .pom file and .jar files from Google (like that gradle one you found), then upload them to ProGet from the Web UI. If that doesn't work, then we know the POM file is probably bad, and we can investigate it.

Next, I would try to pull the pom/jar from a connector. If that doesn't work (but uploading it does), then maybe the "API" for that repository is incorrect. And we can investigate it.

Best,
Alana

Hi @Justinvolved ,

This is not uncommon with PowerShell; in general the processes will be closed, but there are enough scenarios with long-running/hanging PowerShell scripts where those processes will report termination but the process will not be terminated. So they can stick around

I'd recommend using the Inedo Agent instead, even if it's on the same sever. That gives better process isolation compared to the local agent, and you shouldn't see this behavior for too long. The local agent runs in-process.

Cheers,
Alana

Hi @scott-wright_8356,

When installing ProGet with the Inedo Hub, you can specify the SQL Server and Database Name. If it's an existing ProGet database, then it will be upgraded/downgraded as needed. I'm don't know what the errors were, but typically the installing users has a lack of db_owner permissions.

To upgrade ProGet in an HA set-up, you can just use the Inedo Hub to upgrade each node. Usually it takes 1-2 minutes per node, so you can do it manually pretty easily. There shouldn't be any errors.

Best,
Alana

Hi @scott-wright_8356,

This is usually due to a networking error; it may also be resolved by restarting. It's also just something you can ignore. The service messanger isn't very important in ProGet.

Best,
Alana

Hi @scott-wright_8356 ,

This error means that "something" that is taking an exclusive lock handle on files within your SMBShare. Unfortunately, the operating system does not provide information about what that "something" is, but it's typically an anti-virus, search-indexing, or back-up tool.

ProGet uses shared read handles, but obviously needs a lock handle when writing. There are a few race conditions where this could occur, but they are extremely rare and require multiple requests on different servers in a cluster that attempt to add the same package at the same time.

However those are very rare, so I suspect "something else" is locking those files. Usually it's an anti-virus tool - since it "looks like" a zip file, it's being opened? You'd need to use a tool to monitor openarting system handles, like sysinternals procmon.

Best,
Alana

Hi @scott-wright_8356 ,

I'm not sure, but based on this URL:
https://maven.google.com/web/index.html#android.arch.core:common:1.1.1

I can see that the POM for android.arch.core.common 1.1.1 is located here:
https://dl.google.com/android/maven2/android/arch/core/common/1.1.1/common-1.1.1.pom

So the base URL would be:
https://dl.google.com/android/maven2

That should work, since maven works by convention. You will not see a listing of artifacts unfortunately, since Maven does not provide an index. You just need to know the articats by name, then append maven-metadata.xml.

As expected, the metadata file for that artifact is here:
https://dl.google.com/android/maven2/android/arch/core/common/maven-metadata.xml

Cheers,
Alana

Hi @scott-wright_8356 ,

Thanks; it's hard to say what the issue is, and unfortunately we don't really have any good troubleshooting tools here. That error is coming from within SQL Lite, and we can't reproduce it when we do a SMB share.

I wonder if you can follow some of the same troubleshooting steps? Specifcally like using ProcMon or mounting the network drive with SysInternals?

We just don't really have any means to troubleshoot this, so we'd need to go back to the drawing board to figure out why it doesn't work for you. You're welcome to keep exploring C#, but let me share the code that's closer to ProGet.

Here is the code that ProGet is running. It's kind of similar, but a little different. Note we are using System.Data.SQLite.Core-1.0.118 and .NET6.

private static SQLiteConnection OpenOrCreateDatabase(string fileName, out Dictionary<string, string> config)
{
    var str = new SQLiteConnectionStringBuilder
    {
        DataSource = fileName,
        FailIfMissing = false
    }.ToString();

    if (FileEx.Exists(fileName))
    {
        var conn = new SQLiteConnection(str);
        try
        {
            conn.Open();

            config = ReadConfig(conn);
            if (config.TryGetValue("SchemaVersion", out var versionStr))
            {
                int version = int.Parse(versionStr);
                if (version == 1)
                {
                    using var cmd2 = new SQLiteCommand(GetScript("UpdateSchemaToV2"), conn);
                    cmd2.ExecuteNonQuery();
                    return conn;
                }
                else if (version == 2)
                {
                    return conn;
                }
            }
        }
        catch
        {
        }

        conn?.Dispose();
        FileEx.Delete(fileName);
    }
    else
    {
        DirectoryEx.Create(PathEx.GetDirectoryName(fileName));
    }

    var conn2 = new SQLiteConnection(str);
    conn2.Open();

    using var cmd = new SQLiteCommand(GetScript("CreateSchema"), conn2);
    cmd.ExecuteNonQuery();

    cmd.CommandText = GetScript("UpdateSchemaToV2");
    cmd.ExecuteNonQuery();

    config = ReadConfig(conn2);

    return conn2;
}
private static Dictionary<string, string> ReadConfig(SQLiteConnection conn)
{
    using var cmd = new SQLiteCommand("SELECT Key_Name, Value_Text FROM OtherData", conn);
    using var reader = cmd.ExecuteReader();

    var res = new Dictionary<string, string>();

    while (reader.Read())
    {
        var key = reader.GetString(0);
        var value = reader.IsDBNull(1) ? null : reader.GetString(1);
        res[key] = value;
    }

    return res;
}

Here is the CreateSchema script:

CREATE TABLE OtherData
(
    Key_Name TEXT PRIMARY KEY NOT NULL,
    Value_Text TEXT
);

INSERT INTO OtherData (Key_Name, Value_Text) VALUES ('SchemaVersion', '1');

CREATE TABLE FetchInfo
(
    Subdir_Name TEXT PRIMARY KEY NOT NULL,
    Modified_Date INTEGER,
    Fetch_Date INTEGER NOT NULL
);

CREATE TABLE ChannelData
(
    Package_Name TEXT PRIMARY KEY NOT NULL,
    Flags_Value INTEGER NOT NULL,
    Description_Text TEXT,
    DevUrl_Text TEXT,
    DocSourceUrl_Text TEXT,
    DocUrl_Text TEXT,
    Home_Text TEXT,
    Icon_Hash TEXT,
    Icon_Url TEXT,
    License_Text TEXT,
    RunExports_Json BLOB,
    SourceGitUrl_Text TEXT,
    SourceUrl_Text TEXT,
    Summary_Text TEXT,
    Timestamp_Value INTEGER,
    Version_Text TEXT NOT NULL,
    Subdirs_Csv TEXT NOT NULL
);

CREATE TABLE RepoData
(
    Package_Name TEXT NOT NULL,
    Subdir_Name TEXT NOT NULL,
    Version_Text TEXT NOT NULL,
    Build_Text TEXT NOT NULL,
    ArchiveType_Code INTEGER NOT NULL,
    Build_Number INTEGER NOT NULL,
    License_Text TEXT,
    LicenseFamily_Text TEXT,
    MD5_Hash BLOB,
    SHA256_Hash BLOB,
    Package_Size INTEGER NOT NULL,
    Timestamp_Value INTEGER,
    Dependencies_Json BLOB,
    AppEntry_Text TEXT,
    AppType_Text TEXT,
    Type_Text TEXT,
    Summary_Text TEXT,
    Icon_Hash BLOB,

    PRIMARY KEY (Package_Name, Version_Text, Subdir_Name, Build_Text, ArchiveType_Code)
);

Here is the UpdateSchemaToV2 script:

UPDATE OtherData
   SET Value_Text = '2'
 WHERE Key_Name = 'SchemaVersion';

ALTER TABLE RepoData
  ADD Constraints_Json BLOB;

From there, it's just a bunch of inserts into those tables. The "unable to open database file" error would probably be occurring in the middle of those, but it's hard to say.

The race condition occurs when multiple threads run OpenOrCreateDatabase at same time. It's rare as I mentioned.

Alana

Hi @scott-wright_8356 ,

To set a package to blocked, first pull the package to ProGet, then use "Set Package Status". You can then set "Download allowed:" option.

do you intend to allow users to modify the vulnerability database

We do not intend to allow users to modify the database. There are several reasons for this.

1. We could not come up with a suitable use case considering the alternative options
2. It's not something other security products allow
3. It creates a usage/support problem because it's easy to "forget" that the data was edited and then there's a strange behavior

Cheers,
Alana

Hi @akillen_5633 ,

Sorry on the slow response, but we're kinda stuck on this one; for some reason, the query to list packages is timing out. That's really unexpected, especially in 2023.31.

If it was related to heavy load, I would expect a lot more symptoms - like API calls, etc timing out. That doesn't seem to be the case.

The only thing that we can think of is that something is wrong with SQL Server, in so much as it needs some maintenance.:

• run exec sp_updatestats
• rebuild indexes (see https://docs.inedo.com/docs/installation-sql-server#advanced-maintenance)

We've also seen some old/unpatched versions of SQL Server cause strange performance problems due to some obscure bugs in SQL Server. So that's another thing to consider.

If that doesn't do the trick, you may need to dig in a bit deeper to figure out what's going on, and see what SQL Server is doing behind-the-scenes. Ultimately this is just executing the Packages_GetPackages stored procedure, which is basically just doing a straight-forward select on FeedPackageVersions_ExtendedWithMaven view.

Best,
Alana

Hi @chris-cantrell_1211 ,

We do not support nor recommend this type of configuration for a couple reasons.

First, it doesn't change the attack surface, since the APIs are already the "weakest link". The Web Site use cryptographically-secured authentication tickets with anti-CRSF protection. The API just requires an API key to access.

Second, it's confusing to end-users who are trying to troubleshoot why some urls aren't accessible. The API may provide them with a link (for example to a vulnerability in a package), and then it will give some kind of error because the page is blocked. This causes everyone a headache.

Obviously you can use a lot of tools to block/allow access to URLs, just not using ProGet.

Cheers,
Alana