@jimbobmcgee thanks for the detailed analysis! This is on my list, but haven't had time to properly reproduce and test this.... but as you noted, the code fix looks so easy.
I just made a quick change (OT-513) and prerelease (2024.2-rc.1). Can you give it a shot?
https://docs.inedo.com/docs/installation/windows/howto-install-prerelease-product-versions
Hi @ghollosy_9163,
I'm afraid this will not be an easy problem to troubleshoot; there were some major changes in ProGet 5.3 with regards to Docker feeds, so it's possible that there was a data migration problem during the upgrade process.
Ultimately it looks like your container configuration files aren't there. I don't know why, but I'll try to point you to where/how they are stored.
Docker blobs are stored on disk, and you'll see files named by their hash. Based your screenshot, it'd be a file like bf3c18259..... These files are also indexed in the database, in the DockerBlobs table.
If the files aren't anywhere on disk, then check backups. It's possible they got accidently deleted, and you can just add them back.
If the files are missing on disk, then they are probably gone for ever. If they are not in the database, then perhaps something happened during the migration? I really don't know. It's possible to insert them in the database, but we don't have an automatic way to do this - basically you'd need to write a script.
-- Dean
hi @jaehyung-shin_8059 ,
If you click on the latest tag, do you see any information about the packages in the container?
Container images are scanned on upload if they are small or queued to be scanned in the background if they are large. There are sometimes errors scanning containers, and if so, these errors will be logged... typically under Admin > Executions.
-- Dean
Hi @jaehyung-shin_8059 ,
From what you described, it sounds like you'd like to have a single tag (like :v3) resolve to different architectures?
This is called a multi-platform image, and you'll need to study/follow the Docker guidance on how to create one: https://docs.docker.com/build/building/multi-platform/
Long story short, this is not something that you can do by simply pushing two images to the same tag like you're doing; it's quite a bit more complex, and you have to build the images in a very specific manner.
But once you do that, ProGet does indeed support them (they are called "fat manifests") and you will see a note/warning when viewing an image for one.
-- Dean
@parthu-reddy yes, but we do recommend migrating...
There are some known bugs/quirks with Maven (Classic) feeds and we will not fix them.
We will likely not migrate them to PostgresSQL, which means you won't be able to migrate to PostgreSQL in ProGet 2025+.
So it's likely that ProGet 2027 will not support them.
Hi @parthu-reddy ,
Huh... that's interesting... but not surprising. The Maven (New) feeds assume POM files are valid XML, but I guess Marvin supports invalid XML for POM files 
I fixed this via PG-2859 which add Add Support for Invalid POM files on Maven (New) Feeds. You can access it via 2024.23-rc.1 on the prerelease feed (InedoHub), or the 24.0.23-rc.1 Docker image.
Hopefully this will work-around these migration issues.
-- Dean
Hi @parthu-reddy ,
This error means that the previously listed pom file (ant-1.6.5.pom) is somehow corrupted. It cannot be loaded as XML due to the error.
I assume it's this artifact:
https://repo1.maven.org/maven2/ant/ant/1.6.5/
That one is fine at the source. I would just remove it from your old/classic feed, since I assume it's just something from maven central. And it can then be downloaded again.
-- Dean
Hi @parthu-reddy,
For a major upgrade, we recommend to schedule a maintenance window and then stop traffic from the load balancer. Then, upgrade the servers.
For a maintenance release, it's fine to just upgrade them one at a time. It only take as few minutes to do that. There may be a few errors with the database/code being out of sync, but it likely will not cause any problems.
-- Dean
@parthu-reddy this is a somewhat expected error, and it's something we'll improve in a future version of ProGet, but in all a cases it will cause some kind of problem.
The problem is that you added a new feed/connector (in this case, a Maven (New)) that is unknown to previous versions of ProGet. So when you try to load that unknown type in older versions of ProGet, you get this error.
If you deleted the connector before downgrading, it would have worked. So, I would upgrade, delete, then you can downgrade again.
-- Dean
@sneh-patel_0294 I don't think so, its only displayed on that page...
Hi @sneh-patel_0294 ,
We've never seen that before nor had anyone else report it. The only thing I could guess is network communication on port 33237. I guess I would try restarting the web server? I would try it on different servers?
It's just really weird behavior, but that's the onnly thing I could guess.
You can access a dialog to disable the Automatic Failover, which would also disable the network communication via /administration/cluster/configure
Changing the license key would also disable it. These are the only ways I can think to test if it's "something" blocking/delaying/interfering the 33237 communication. We've seen some firewalls do that.
-- Dean
Unfortunately the PowerShell gallery is pretty buggy/slow/glitchy these days. We don't think it's actively maintained by Microsoft anymore, as their attention is shifted to "next gen" (PSGet 3.0).
Connectors to the gallery don't work as well as they should, and you may find using an approval workflow (i.e. pulling them proactively, promoting them to other feeds, etc) is the most reliable. Many queries to PowerShellGallery.org (especially a FindPackagesById()) will simply timeout, which will cause ProGet to log an error (Admin > Diagnostic Center) and return a 404 in the API.
There are also versioning and API quirks that you should be aware of. They apply to some packages (mostly older, but some newer, sometimes). It's not easy for us to work-around these without a major development effort, and most of our users have learned how to deal with these annoyances.
However, this should all work in PSGet3.0. Now they are about 5 years behind on the release... BUT with Copilot's help, perhaps 2025 just might be the year they get it fixed!
-- Dean
Hi @sneh-patel_0294 ,
That page mostly displays some items stuff from the database (specifically ClusterNodes_GetNodes) and attempts to do some network communication on port 33237; there's likely some kind of firewall/trap that's preventing communication on that port, and it will timeout after awhile.
You should be able to visit that page from any browser, as opposed to localhost. That page is mostly just useful in making sure the load-balancer is configured correctly.
-- Dean
Great news, thanks for the update!
Hi @sneh-patel_0294 ,
That error message is coming from the operating system; it doesn't necessarily mean a permissions issue.
Does it happen every time for every package, consistently?
If that's the case, then it's certainly some kind of permission configuration. The user running the ProGet Web Service (or IIS App pool) may not have the appropriate permissions to the folder.... or it could be something related to network access? I don't really know.
The operating system is opaque with the error message, and you might have to use a tool like procmon to see exactly what's going on. That will show you what programs/processes request file handles.
If this is sporadic, then it means the file is locked. It's possible for ProGet to lock the file, but it's unlikely and would require basically two processes trying to write to the same file at the same time. We've only seen that with misconfigured build servers that publish same build twice.
More likely the file locking is coming from like backup, index scanning, or malware that's masquerading as "security software". Procmon will also advise this, if you can catch it.
-- Dean
Hi @forbzie22_0253 ,
Tags are a field in the nuspec file, which is embedded within the NuGet package:
https://learn.microsoft.com/en-us/nuget/reference/nuspec#tags
As such, we do not recommend using tags in NuGet, because they are "permanent" and you can't "untag" packages. It makes it hard to think of a tagging system that will be useful for the long-term: https://blog.inedo.com/nuget/best-practices-internal-nuget-packages/
-- Dean
Hi @arkady-karasin_6391 ,
A 403 error means Not Authorized; since those are publicly-available repositories, my guess is that you have a Proxy or Content Filter that's blocking that URL, and the ProGet Server is getting a 403 response from that intermediate server.
-- Dean
@joel-shuman_8427 thanks for the heads up!
I just updated it
https://github.com/Inedo/inedo-docs/blob/master/CONTRIBUTING.md
HI @parthu-reddy ,
Thanks for the feedback; I update the docs to mention that the feeds/permissiosn would also need to be updated as well.
We hope to handle an in-place migration in a future version, but didn't want to delay shipping the feed. Please let us know if you have any issues/feedback.
Note we are adding several improvements to the Maven (New) feeds in ProGet 2024.15:
-- Dean
Hi @parthu-reddy ,
I'm afraid we don't have enough information to help with this; it appears to be an error the tool you're using (packet), and there's not enough information in the screenshot to see what error packet is encountering. It just says "Packaged failed with could not download..."
We aren't familiar enough with packet to know how to follow their Stack Trace.
If you can't find a clear error message, I would use an HTTP Proxy tool like Fiddler Classic to inspect the traffic that packet is making, and see if you can spot an issue there.
I would also try downgrading packet , as it's very possible there's a regression in the tool.
-- Dean
Hi @kichikawa_2913,
We see multiple connectors pretty often, and it rarely presents a problem.
The main downside comes in the overhead of aggregation; for some queries like "list all package versions", each connector will need to be queried and have the results aggregated. So it could cause performance issues for for very high-traffic feeds - at least that's what we see on the support side of things.
However, if you plan on using a package-approval workflow, then it won't be a problem, as your approved-npm feed wouldn't have any connectors.
Hope that gives some insight,
Dean
ProGet will download nightly updates of the vulnerability database from an inedo.com server.
If you're running in a totally air-gapped environment, then you obviously won't get these updates. However, each version of ProGet includes an up-to-date database, so upgrading will get you the updates.
--Dean
Hi @steviecoaster ,
The Native API can be a little finicky, especially since you can invoke with JSON, forum-encoded values, querystring, and I think even XML. But it sounds like you're on the right track.
Let me share the C# code that ProGet uses to set the password:
using (var rfc2898 = new Rfc2898DeriveBytes(password ?? string.Empty, 10, 10000, HashAlgorithmName.SHA1))
{
var bytes = rfc2898.GetBytes(20);
DB.Users_SetPassword(userName, bytes, rfc2898.Salt);
}
... it looks a little different than the code you're using, so hopefully that will help!
-- Dean
Hi @russell_8876,
Thanks for all the additional details!
Actually, this is definitely a bug in pgutil, since it's sending a negative offset to ProGet, and then ProGet is responding with a 400 error ("Missing or invalid "offset" parameter.)
Asset uploaf capability is just a few weeks old, and I'm not sure if we tested with such a large file. Anyway, we'll get it looked at and let you know once fixed; just a small tweak needed here I think:
-- Dean
Here is documentation on how to enable HTTPS in our products on Windows and Linux:
-- Dean
Hi @steviecoaster,
The Native API isn't hacky, just harder to used. Here is the documentation on the Native API:
https://docs.inedo.com/docs/proget/reference-api/proget-api-http#native-api-endpoints
We don't have any articles/guidance on how to call the Native API beyond what's there.
The Users_* procs have not changed in years and are very safe to use. There are a few forums posts here and there with "hints" on work with the User_ procs, like this:
https://forums.inedo.com/topic/4198/reset-proget-admin-password-via-api/2
-- Dean
Hi @steviecoaster ,
I agree it'd be nice, but it's also not trivial as you noticed. I believe it's possible to configure security with the Native API, but obviously not as easy. So I would explore that, it's probably close to what you wnat.
A first-class Built-in Users/Groups API has been requested over the years (and is now something that makes sense with pgutil) - however if we made it, the API would be paid-editions only. We haven't had any interest from paid users in such a feature, as they generally use LDAP or don't mind non-API configuration.
-- Dean
@steviecoaster great, thanks for sharing!
I added this to our documentation (https://github.com/Inedo/inedo-docs/pull/253), but apparently I don't have ability to merge PRs in that repo so it'll go live sometime later I'm sure
@steviecoaster glad you were able to figure it out :)
Easy typo and we should consider a validator on that "Update SSL Certificate" page as well to save a headache like this!
Hi @v-makkenze_6348 ,
If you download (i.e. cache) the package, then you shouldn't see the compliance issue anymore. The reason is that ProGet does not have information about the package unless it's cached/local, or if you're viewing it on the package overview page.
When ProGet runs a build analysis (first screenshot), it only uses local/cached package data. This is for performance reasons, as users will have 100's of builds with 1000's of packages in each build, and that much traffic to each connector is problematic.
However, we are working on building a "remote metadata cache" that will fetch this data in a more performant manner.
-- Dean
Hi @steviecoaster ,
Sorry about giving the bad advice there -- I did not realize that the offline installer does not include the hub.exe program. It looks like the Offline Installer Creation Process must strip that from the offline installer we provide.
As you discovered, the advice from before wouldn't really work. Without doing a deep dive in the code, I don't know how to make it work. This isn't a use case we designed for, and I'd hate to send you down a wild goose chase.
How about just using our standard silent installation approach, which I shared before:
# create working directories
mkdir C:\InedoHub
cd C:\InedoHub
# download and extract file to working directory
Invoke-WebRequest "https://proget.inedo.com/upack/Products/download/InedoReleases/DesktopHub?contentOnly=zip&latest" -OutFile C:\InedoHub\InedoHub.zip
Expand-Archive -Path InedoHub.zip -DestinationPath C:\InedoHub
# perform silent installation
hub.exe install ProGet:5.2.3 --ConnectionString="Data Source=localhost; Integrated Security=True;"
This will basically install the desired version and it's likely "good enough" for the time being.
-- Dean
@MY_9476 I'm afraid not; what you're seeing is a diff of the OtterScript, which is what's stored in the database. The issue is that the "serialization to code" order must have changed between major versions.
We try to not have that happen, but fortunately this only happens the first time you edit such a script. Next time, only your changes will be preserved.
Hi @jw,
There's no problem deleting the deprecated licenses and adding their SPDX identifiers to the new licenses. When you delete a license, it will remove the association with packages.
However, the compliance analysis scheduled job will reassociate them. This runs nightly, or you can manually run it. Also, if you visit the package page or download the package, it should associate with the new license.
We'll definitely reconsider the design/approach if there's more demand, but we need to really make sure there's a value to the user -- there's a relatively high cost to change things and then there's a chance of regression/bugs, which is really frustrating to users.
-- Dean
Hi @jw ,
[1] is somewhat expected behavior, as some older versions of ProGet allowed non-normalized URLs to be added; newer versions should not allow this. We do not plan to "clean-up" the data at this time, but if you're "brave" you could could like do it with some API/DB calls
[2] This is probably some quirk related to older data, but you can probably just cut the items to clipboard, save the license, the edit again, and it should work-around the quirk
FYI, the URLs that we have in our database for LGPL-3.0-only are: as follows
gnu.org/licenses/lgpl+gpl-3.0.txt
gnu.org/licenses/lgpl-3.0-standalone.html
opensource.org/licenses/LGPL-3.0
I'm guessing you have the www from an older version
-- Dean
Hi @MY_9476 ,
I'm afraid this isn't technically feasible.
When you edit in Visual Mode, the statement is basically "serialized" to OtterScript, and the property order is determined by however the operation's code (i.e. C#) is laid out. Fortunately this is really consistent, and really only is a problem when you go back-and-forth.
But in this case, there must have been some change to the C# / order of properties (refactoring perhaps?) so it's inconsistent across versions of Otter? If you edit it again, the order will be the same.
-- Dean
Hi @steviecoaster,
We're currently working on the pgutil feeds commands, and expected create and update to be finished in a week or so. In the mean time, I just published pgutil-1.1.2, so please check that out - it will have a basic version of pgutil feeds create.
As for the PowerShell command, you'll need to pass the JSON structure that's documented in that article. You would want to use something like ConvertTo-Json to convert a Hash table to JSON.
-- Dean
Hi @steviecoaster,
If you want to use the offline installer in a script (which I think is best for a Chocolatey script), then just unzip it after downloading. I know it's an .exe file, but you can just unzip it like any other zip file (e.g. with Expand-Archive or something in PowerShell).
Once you've done that, you can just run the hub.exe per the silent installation instructions.
Alternatively, you can write a script like this, which will instruct hub.exe to download a version:
# create working directories
mkdir C:\InedoHub
cd C:\InedoHub
# download and extract file to working directory
Invoke-WebRequest "https://proget.inedo.com/upack/Products/download/InedoReleases/DesktopHub?contentOnly=zip&latest" -OutFile C:\InedoHub\InedoHub.zip
Expand-Archive -Path InedoHub.zip -DestinationPath C:\InedoHub
# perform silent installation
hub.exe install ProGet:5.2.3 --ConnectionString="Data Source=localhost; Integrated Security=True;"
The only downside to this approach is that it's not really version controlled (i.e. it's always using latest Inedo Hub) and the package can't be realizably internalized, since hub.exe will download files from the internet.
-- Dean
Hi @cstekelenburg_4169 ,
If you created a connector with the proper URL, username, and password, then you've set it up correctly.
Please note, you will not be able to list or search remote packages in the ProGet UI. This is a limitation, since ADO is rudimentary and does not support listing/searching.
Instead, you will need to use the NuGet API (Visual Studio, Nuget.exe) to pull packages. Once package has been pulled through ProGet, you will "see it" in the UI since it's been cached.
I'm not trying to be pedantic with terminology, but please note it's not a "sync" -- it's more of a "proxy" or "pass through". When you make a request via the NuGet API, that API request is forwarded to your connector (I.e. Azure DevOps).
-- Dean
Azure Artifacts does not support the NuGet Search API, so this means you won't see packages on the feed/search page in ProGet. This is expected behavior with rudimentary NuGet repositories that don't support listing/searching.
In the past, I'm guessing that you either:
-- Dean
Hi @cstekelenburg_4169 ,
ProGet does not "sync" remote repositories, but instead displays and caches results through a "connector" you configure. If there are any connector errors, they will be logged when you do a query.
Azure artifacts do not support searching or listing packages, so you won't "see" a package unless you type in the exact package name.
-- Dean
Hi @johnsen_7555 ,
Thanks for sharing the additional details; I was able to reproduce this, and its a regression/bug.
We fixed this via PG-2723 (FIX: Noncompliant Pypi packages can be downloaded when blocking is enabled on trial licenses), which is going to be shipped in the next maintenance release.
However, in the mean time, you can try a patch/prerelease version that contains the fix; since you mentioned Docker, the tag to pull would be 24.0.9-ci.1.
Hopefully that will solve the issue!
-- Dean
Hi @johnsen_7555 ,
Thanks for sharing all of the configuration details; can you navigate to one of the GPL packages that you're trying to install, and see what it says on the package page within ProGet?
On that page, you can also ReAnalyze the package (using the dropdown button ) and get a log of which policies/rules were applied.
-- Dean
Thank you so much for the kind words, and it sounds like you're on the right track for setup/configuration!
ProGet is designed for the scenario you described (i.e. a multiple instances that point to the same database/files), and we call it a server cluster:
https://docs.inedo.com/docs/installation-high-availability-load-balancing
The above instructions are for Windows, but the same principle would apply for Azure Web Apps, Kubernetes, etc. We just don't have documentation or the ability to support the underlying platform. What I mean by that, if the "scale out" feature in Azure doesn't do what it's supposed to, you would need to contact Azure support.
All that said, do note that a ProGet clustered installation requires a ProGet Enterprise license:
https://docs.inedo.com/docs/proget-administration-license
Hope that helps,
-- Dean
Hi @scott-wright_8356 ,
That procedure is ancient and hasn't caused problems before. I suspect the issue might be the size of the data in that able - you can see what that looks like on the connector cache management page.
You could also try to optimize the query like so:
ALTER PROCEDURE [dbo].[Connectors_GetCachedResponse]
(
@Connector_Id INT,
@Request_Hash BINARY(32)
)
AS BEGIN
SELECT * INTO #ConnectorResponses
FROM [ConnectorResponses] WITH (READUNCOMMITTED)
WHERE [Connector_Id] = @Connector_Id
AND [Request_Hash] = @Request_Hash
UPDATE [ConnectorResponses]
SET [LastUsed_Date] = GETUTCDATE(),
[Request_Count] = CR.[Request_Count] + 1
FROM [ConnectorResponses] CR
JOIN #ConnectorResponses _CR ON CR.[Connector_Id] = _CR.[Connector_Id]
SELECT * FROM #ConnectorResponses
END
Let us know if that works, we can update the code accordingly if so.
-- Dean
Troubleshooting the database can be a pain, but there are a lot of tools that can help. Here is a nice guide from Microsoft, which includes a script that shows you how to use the sys.dm_exec_query_stats and sys.Dm_exec_sql_text views:
https://learn.microsoft.com/en-us/troubleshoot/sql/database-engine/performance/troubleshoot-slow-running-queries
Just make sure to remove WHERE t.text like '<Your Query>%' from the sample, if you want to see all queries. Some red flags on that query will a disproportional total_elapsed_time , etc.
That said, there are two different jobs:
A "Server Checker" is run frequently and should be very fast. There is very little database activity (just Servers_GetServers to get all servers, then Servers_UpdateServerStatus per server).
A "Routine Configuration" is obviously more complicated, and may involve a lot more database activity.
-- Dean
Hi @scott-wright_8356 ,
Thanks for sharing a lot of the details; it's not really clear what objects the locks are on, can you look at the XML view of the deadlock report to find out?
Since you're "already there", you're welcome to try doing what we would do --- just add WITH (READUNCOMMITTED) to the problematic queries (i.e. the ones where object locks are occurring on).
For example, like this: SELECT * INTO #PgvdPackageNames FROM [PgvdPackageNames_Extended] WITH (READUNCOMMITTED) WHERE [PackageName_Id] = @PackageName_Id
As an FYI, this should not be an issue in ProGet 2024, since package analysis is cached for a short while. Also, our internal development practices (going forward) are to READUNCOMMITTED in "hot" queries this anyway.
-- Dean
hi @andreas-unverdorben_1551 ,
The timeout message you shared is for a NuGet feed - so the issue isn't the 727 packages that you're installing (which often results in 1400-2100 requests being fired off simultaneously) , but the NuGet build(s) that are also going on at the same time and hammering the server.
The server cluster will definitely help with this peak load.
FYI --- that error message you shared is is from NuGet v2 API, so something is still calling it.
Cheers,
Dean
Hi @andreas-unverdorben_1551 ,
What's happening here is your server is being overloaded due to lots and lots of traffic -- effectively you are doing a Denial of Service attack on you server. As I mentioned in npm install slow on proxy feed, ProGet is not a static file server and there is a lot of processing required for each request.
The best way to handle this is to run a load-balanced ProGet server cluster.
Alternatively, you will need to reduce or throttle traffic.
Best,
Steve
Hi @andreas-unverdorben_1551 ,
npmjs.org primarily serves static content and runs on massive server farms running in Microsoft's datacenters.
Your ProGet server is much less powerful and does not serve static content. Not only is every request is dynamic (authentication, authorization, vulnerability checking, license checking, etc), but most requests (such as "what is the latest version of package X") need to be forwarded to npmjs.org and aggregated with local data.
So, a much less powerful server doing a lot more processing is going to be a little slower ;)
Running ProGet in a server cluster will certainly help.
Cheers,
Dean
In this case, can you open a ticket and send us that tar.gz file (attach on the ticket)?
From there we will load it in a debug version of ProGet and identify where the problem is occurring. Not something that will be easy to spot by inspecting a file :)
Thanks,
Dean