Do you have anything else defined on the other tabs? That could have an impact if so...
Otherwise, best way to troubleshoot this would be to look at the retention logs. That should give an idea of what's happening in the policy. And then if you can find a specific example of an untagged image, that would definitely help to track it down...
Cheers,
Alana
Hi @mharen,
You can try to enable Windows Integrated Authentication in ProGet, and then use two sites in IIS (once with WIA, one without), but that will only work for NuGet packages. Other client (npm, docker) do not support WIA.
Otherwise, you can take a "DMZ" approach, and configure two instances of ProGet - one for Devs, one for CI systems - then use a connector. The two-instance approach also has the benefit of reducing load, though not quite as much as a load balancing configurating.
Cheers,
Alana
Hi @jw ,
Thanks for reporting these!
We'll get the fixed in hopefully the next upcoming maintenance release via PG-2520 and PG-2521
Cheers,
Alana
Hi @lucas-almeida_8120 ,
This error implies that there's some kind of database corruption or other error that happened while updating the database a while ago. But it's hard to say; this is the first time I've seen this error.
Unfortunately, unless I were to analyze a backup of your database and try to reproduce the installation/upgrade, I won't be able to offer any guidance or scripts that could repair the state of things.
From here, I would recommend rolling back to whatever version was working, then starting a new ProGet server and importing your packages.
Best,
Alana
To accomplish this, I would recommend writing a script that does the following:
hub.exe to perform silent installationInstalling SQL Server in an automated manner is not trivial, and you will need to research the best way to do that in your environment. Here is an article I found when searching for that: SQL Server Unattended Installation with PowerShell.
AS for steps #2 and #3, here is an example script from our Silent/Automated Installation Guide that you can adapt as you need:
# create working directories
mkdir C:\InedoHub
cd C:\InedoHub
# download and extract file to working directory
Invoke-WebRequest "https://proget.inedo.com/upack/Products/download/InedoReleases/DesktopHub?contentOnly=zip&latest" -OutFile C:\InedoHub\InedoHub.zip
Expand-Archive -Path InedoHub.zip -DestinationPath C:\InedoHub
# perform silent installation
hub.exe install ProGet:5.2.3 --ConnectionString="Data Source=localhost; Integrated Security=True;"
Best,
Alana
Hi @jw ,
With the way things are "wired up" today, there are some edge cases when this will not show up right away. This is primarily for performance reasons, and it's something we absolutely plan to address in ProGet 2024.
The "trick" is that the PackageAnalyzer job needs to be run to do some back-end linking in the database; this is typically done on a nightly basis (there is a scheduled job for this), and in practice it's rarely something you'll spot outside of testing.
For example, after that job runs... if you were to delete then recreate TestProject 1.0.0 by pushing an SBOM, it should show the vulnerability.
Thanks,
Alana
Hi @rob-leadbeater_2457 ,
Good timing! This is something we plan to ship in the next or following maintenance release.
Actually, we had to completely reimplement Debian/APT feeds from scratch, so there will be a "Debian" and a "Debian (Legacy)" feed type, and a way to migrate from the legacy to the standard feed types.
I added a note to update this post once we have it ready to ship.
Cheers,
Alana
Hi @haralambop_0645 ,
A 403 error means that the user does not have the required permission to perform the action. So you'll need to make sure that feeduser is authorized to push packages to the feed, which you can do under Admin > Tasks.
Cheers,
Alana
Hi @gunmaden_7628 ,
The Common Packages API does not work with Maven feeds, since they are not a package-based feed. Otherwise... it's hard to say what the issue is with Python or Debian, but most likely it's missing a parameter that identifies which file to delete?
If you can share more details about the calls you are making and the packages we may be able to help.
Cheers,
Alana
@haralambop_0645 oh see - in this case, you'll need to add a source with the username api and password of your key using dotnet nuget add source
dotnet nuget add source https://myProGetServer.local/nuget/myFeed/v3/index.json -n myFeed -u api -p myApiKey --store-password-in-clear-text
This is because NuGet does not send an api key during list, it uses the name/password in the source file.
Hi @haralambop_0645 ,
Can you share more information about the usage scenario? Are you trying to use Docker images, for example?
Thanks,
Alana
You may have also seen this message but...
What Windows server version is Otter installed on? That might help us narrow this down.
We have seen that error if RSAT is not enabled:
https://learn.microsoft.com/en-us/troubleshoot/windows-server/system-management-components/remote-server-administration-tools
Cheers,
Alana
Thanks for the update @mness_8576; based on this, it sounds like there's definitely a Gateway problem, and that the Gateway is misconfigured / misreporting some error condition from ProGet due to a quirky package request. That's a pretty common thing we've seen as well.
Hi @itops_6398 ,
On a remote package (i.e. the ones with the radio icon), the download count is whatever is presented by the remote feed. On python packages, some have negative numbers.
Once you pull or cache (i.e. download) the package , the count will start at 0 and ProGet will keep track of the download count.
Cheers,
Alana
Hi @itops_6398 ,
It sounds like you're on the right track!
From here, you should create a group on your AD domain forest called "ProGet Administrators" (or whatever). Basically what you described in [2].
Then, on the "Tasks" tab, just add the appropriate permission (i.e. admin) for that group. You should see the "ProGet Administrators" appear in the dialog when you type a few characters.
After that, it should work as you'd like.
Behind the scenes, ProGet will attempt to query the user directories you've configured to discover the groups that the user belongs to. On the Testing tool (in the drop down on the security page), you can try to list a particular user's groups if it's not working.
Hope that helps!
Cheers,
Alana
@itops_6398 glad that worked! I noticed same quirk in the UI and fixed it :)
Probably why we/no one noticed before... editing the connector just happened to fix it once the value was saved.
Hi @itops_6398 ,
It doesn't work for me either, and when I looked closer I realized what the issue was 
PyPi.org doesn't support searching, and the "New Feed Wizard" doesn't check of the "Use Exact Match" on the connector by default. So to work-around this issue, simply edit the connector, make sure box is checked (on the "Advanced" tab), then hit save.
After you do that, you can find numpy. But you won't be able to "search" packages unfortunately due to API limitations.
I forgot about that. We'll fix the Wizard via PG-2512 in the next maintenance release, but the work-around will do the trick.
Cheers,
Alana
Hi @avoisin_5738 ,
Thanks; so in this case, it sounds like there is something wrong with the zip file that our compression library does not like. We of course do not write our own zip library, but we can investigate. We have never had this problem in over 10 years... so it must be a something very strange. But we will look at it.
From here, can you create a test .npukg that we can use to reproduce? And also, can you send us the script/code that you use to create the custom package?
If it's a small file, then please email it to support at inedo dot com, with [QA-1267] in the subject (so that we can find it). If it's a huge file, then let's find a nother way to transmit it to us.
Thanks,
Alana
Hi @admin_9486 ,
We did not design the internal storage to be used/managed outside of ProGet... so it's not a scenario we really considered. Does re-indexing add the files that are missing?? That might be an unintentional function.
I guess, the first thing that comes to mind is using a drop-folder for new files.
Can you help us understand the use case? Why does it become updated externally, but also need to be updated in ProGet? It's just not one we imagined... so understanding will help give a appropriate solution
Thanks,
Alana
Hi @itops_6398 ,
That's definitely not right; searching for "numpy" should return results basically the same as this: https://pypi.org/search/?q=numpy
However, I just tried it, and I'm not seeing any packages come up. This used to work, so I think there must be some outage or issue with pypis' search api.
Let's give this a day and see if it resolves itself, then we can investigate further.
Thanks,
Alana
Hi @admin_9486 ,
That URL is for a Web Page within ProGet, and is not an API Endpoint; so what you're trying to do will not work and is not supported.
Also, re-indexing is not something you should ever need to do unless the file or database store has become corrupted some how. So even if it were possible to "script" such a thing, it wouldn't make sense.
Cheer
Thanks for the additional feedback; this is definitely something we're revisiting for ProGet 2024. As you described, a lot of the pieces are there - they just aren't wired together very well in some cases. We're working on redesigning this to be more cohesive.
A lot of the existing behavior has to do with performance. For example, /packages/from-purl?pUrl=... is a bit of a "hack" that we used. It's the navigation URL, then then redirects you to the proper URL. This is because finding the URL of a package can be expensive, especially when there are 1000's of packages in a SBOM and multiple candidate feeds for each package. Doing big joins across multiple feeds on the FeedPackages is something we need to very carefully do.
On the same performance note, "Pulling from the source" can be expensive -- especially when you aggregate multiple feeds into one using connectors. So we need to be really careful before doing that.
But like I said, this is something we are redesigning in ProGet 2024; you can think of ProGet SCA features as a "2.0" right now (I guess "1.0" was the old package consumers feature?), and hopefully "3.0" will be a lot closer to getting it right.
Cheers,
Alana
Hi @avoisin_5738 ,
How did you create this .nupkg file?
If you rename the .nupkg file to a .zip file, will it open a zip file in Windows or with a tool like 7zip?
If you can open it as a zip file on your workstation, then please test if you can import the file into ProGet using this method: https://docs.inedo.com/docs/proget-bulk-import-with-droppath
Thanks,
Alana
Hi @MaxCascone ,
No problem; I deleted the "alternate" accounts that were auto-created, and then I changed your forum email to the new one.
Cheers,
Alana
There's no limitation about that in ProGet... the * is something the NuGet client resolves.
Based on the error, it looks like there's a problem with Visual Studio communicating to ProGet. No further information is provided; maybe this is related to ProGet having an issue communicating to Nuget.org through your connector. You'd need to dig into that a bit further.
I'd also make sure to use the V3 API Endpoint. The query that's shown in the error is the v2/ODATA endpoint.
Cheers,
Alana
@philippe-camelio_3885 oh wow, great find.
Hmmm..... so I guess that must be some kind of UI bug with not trimming the newlines 
variable.ListValues = variable.Type == VariableTemplateType.List ? txtListValues.Value?.Split(new[] { Environment.NewLine }, StringSplitOptions.RemoveEmptyEntries) : null;
I'm sure this should be fixed eleswhere too, but for now I'll just add | StringSplitOptions.TrimEntries and I guess that will at least fix the UI bug
Hi @Jon,
There haven't been any major changes recently. What version did you upgrade from?
Here are the recent changes to Otter:
https://my.inedo.com/downloads/issues?Product=Otter
Assuming you were on v2022 before, I would suggest to rollback, and see if it helps.
Cheers,
Alana
Hi @ssuenaga_1020 ,
Thanks for the detailed report; this seems easy enough to fix, so we'll get this addressed in the next maintenance release (2023.20) via PG-2499.
Cheers,
Alana
Thanks for reporting this; we'll get this fixed via PG-2498 in ProGet 2023.20 (in two weeks).
Cheers,
Alana
We tried to reproduce this in the latest version of Otter, but are not having any luck :(
Here is my Job Template; note that you can get the JSON for job template under Admin > Raft Repositories > Browse.
{
"Description": "",
"JobConfiguration": {
"ScriptId": "Default::Script::z-z-test/test-hello.otter",
"ServerTargeting": 1,
"ServerNames": [
"LOCALHOST"
],
"ServerRoleNames": [],
"EnvironmentNames": [],
"Variables": {},
"Arguments": ""
},
"TemplateUsage": 1,
"JobVariables": [
{
"Name": "params",
"Description": "",
"InitialValue": "%(env:item1)",
"Type": "List",
"Usage": "Input",
"ListValues": [
"%(env:item1)",
"%(env:item2)",
"%(env:item3)"
],
"ListRestrict": true
}
]
}
Here is my OtterScript. I wanted to verfify that env would be written, which it was.
foreach $i in @MapKeys(%params)
{
Log-Debug Hello $i;
}
It's possible it was fixed by something else since the version you used, but I don't know.
Might be worth upgrading; if you still have the issue, can you send your JSON of the Job TEmplate?
Thanks,
Alana
Hi @mness_8576,
Unfortunately it's not going to be possible/practical to troubleshoot issues inside of ProGet when there are possibly problematic gateway appliances in the way. In general I can't imagine how there would be issues if there was only ProGet.
We know there's at least one HTTP gateway server between the build server and ProGet (which explains the 504 error), but there may also be a gateway between ProGet and nuget.org.
So I would first find out where these gateways are, and bypass them. If that's not possible, then you'll have to see what's happening on the gateway side. There are typically logs issued, etc.
We've seen everything from content filters to cache servers to DDoS protectors cause problems here. It's really difficult to guess.
Hope that helps,
Alana
@lucas-almeida_8120 it's hard to say; it's likely due to some kind of operating system configuration or issue that's preventing the service from restarting. We haven't seen this happen very often, and restarting the web is not a common activity.
It's probably not worth spending time investigating or trying to resolve, especially given the age of the server. You'd probably have to look at the Windows service logs to find what the issue might be.
Hi @mness_8576,
A 504 would be coming from some kind of intermittent network equipment (gateway), most typically a proxy server or SSL Offloading, etc. It's not issued by ProGet or IIS.
We've see this quite a bit - and usually the gateway has a timeout configured. Once reached, it terminates the connection, giving a 504 to the client and a "disconnect" to the server. The only way to fix this is to find that intermediate device and disable the timeout.
Best,
Alana
Hi @jw,
Thanks, good points! I'll try to explain the inner-workings a little more.
I would ask first is why ProGet does need the full package in its cache for the SCA tracking to work correctly.
This gets a little tricky and confusing; SCA checks for deprecation, vulnerabilities, and licenses.
So the design thought was, if only vulnerabilities were detected it would be weird. Hence, missing package issue only. Maybe this assumption is incorrect.
Maybe a parameter to SBOM uploading/analyzing could be introduced that enables downloading the required metadata or packages (depending on what is really needed) before analysis is started
This unfortunately gets pretty complicated...
Designing a mechanism inside of ProGet to automatically download would not trivial at all for this use case. A PowerShell script to handle this on a one-off basis would probably be much easier.
FYI -- we are considering some kind of job that automatically download certain, trusted packages. This is in the context of "package approval workflows" and basically allowing something like AWSSDK (which frequently publishes new versions) to just always promote a new version. That's not exactly related to SCA/SBOM however.
Alana
Hi @jw ,
Thanks for the feedback; we definitely have plans to review/improve our SCA/SBOM processes, so my answers will be questions ---- please don't hesitate to give us additional feedback in the aggregate when you have a chance :)
As for this particular case, we've seen this too - and just found that downloading the package caused the issue to go away, and that was simple enough since there weren't too many packages (just lots of annoying clicking).
I don't think we want to have ProGet automatically download/populate the feed with packages, but perhaps a better way to handle this is some kind of rule or ignore setting for missing packages??
And it'd be nice if ProGet automatically shipped with a list of these too. I really can't think of any reason why anyone would ever want a notice of "Microsoft.Extensions.Localization" being missing. Can you?
This is probaly something unqiue to NuGet as opposed to other package managers.
Thanks,
Alana
Hi @carl-westman_8110 ,
That's a bug within the Azure SDK, and it looks familiar. I think I've seen it before, and we upgraded or worked-around it a while ago.
What version of ProGet is this? I would upload to latest version if you're not there.
Otherwise, can you try to "play around" and try different feed types (NuGet for example) and different Azure buckets/configurations? Given that this is an Azurer SDK bug, it's difficult to guess what exactly is causing the problem.
Thanks,
Alana
Hi @mvanweel_2810 ,
I'm happy to point you to methods! Sorry that I don't have example scripts, but I hope this helps...
[1] This would require using the Native API method, specifically ApiKeys_CreateOrUpdateApiKey; we don't document the format of ApiKey_Configuration, but it's pretty basic XML that you can figure out by creating other API Keys in the UI and looking at the value of the data
[2] It looks like we don't expose these currently in any API; I can give you a "cheat code" to expose these in the Native API, so they will show up on /reference/api:
update __StoredProcInfo set Internal_Indicator = 'N' where StoredProc_Name like 'FeedGroups%'
Please let me know if this API works for you, and I will mark them as public in a future maintenance release.
[3] "Pulling to ProGet" is the same thing as downloading and then uploading that package. That's effectively what the UI does. So, you can just do that with two calls to the Common Package API.
Please feel free to share your example scripts :) This is an area we really want to improve in our docs.
Cheers,
Alana
Hi @w-repinski_1472 ,
Even with heavy usage, SQL Server Express will be fine; performance issues from heavy usage will come at the network level, and that's solved by configuring ProGet as a High Availability & Load Balanced Cluster.
We've seen a few 10GB+ ProGet databases in the field, and 95% of the space is taken up with years of individual package download records stored in PackageDownloads table. Those are obviously okay to purge (or not enable at all).
Otherwise, to move to a different SQL Server, just restore a database backup to the new server and update the connection string in ProGet.
Best,
Alana
Thanks for clarifying @philippe-camelio_3885
There was a big change to Job Templates in 2022, so it's not surprising it's a regression.
We'll get this via OT-496, hopefully in the same next maintenance release as well.
@philippe-camelio_3885 thanks! I've logged this as OT-495, and we'll try to get it fixed in the next mainteance release (scheduled Sept 15)
I added this to our Otter 2023 list of items to review/consider :)
Please stay tuned
I didn't try to reproduce this or investigate, but there is clearly some kind of problem parsing that variable value because it's a map expression, and that's crashing the job processor (i.e. script isn't even being run).
I don't think we considered this would be used for maps, but just user-friendly variables instead. Are you intending to use this as a user-selectable map??
Just as an idea, does prefixing it with the grave apostrophe escape symbol work? Like
`%(env:item1)
`%(env:item2)
`%(env:item3)
With that we can explore/consider how to fix.
Thanks,
Alana
Thanks for confirming @philippe-camelio_3885
The "Server Checker" and "Check Configuration" are two different functions, so that kind of makes sense. The "Server Checker" should be triggered immediately after adding or modifying agent details (like IP, etc), but it clearly isn't here.
One more question - are you adding these servers via the API?
I just want to make sure we can find/repro/fix this :)
@philippe-camelio_3885 thanks for clarifying.
Next time, can you try running the Server Checker task and seeing if that clears the problem within a short time?
A configuration execution will wait until a server is in a specific status (the Server Checker puts it in that status), so I wonder if it's not being triggered. I didn't look further but confirming it twould help us isolate the issue.
Sorry this must have fell through the cracks...
Few questions.
Server Checker task (on the Service Page, same page that shows Execution Dispatcher) cause them to kick off? What output do you see when you run that task, if not?Thanks,
Alana
Hi @en_1446 ,
Not sure what version of ProGet you're using, but this seems to be on the recent versions I've seen, so I would recommend to upgrade :)
Cheers,
Alana
Hi @k_2363 ,
What version of ProGet are you using?
Another question: Is there any ETA on the API development?
An API for the built-in User/Groups are not really a priority and there's basically no demand for it. Everyone just uses LDAP instead.
Can you help us understand how/why you're using this API?
Thanks,
Alana
Hi @kichikawa_2913,
You can ignore the Age field; it's just used internally by Visual Studio, along with the Id. Not sure why, but it's a strange value in PortablePDB files.
Here's more on what Age means if you're dying to read PE-COFF specifications 
Best,
Alana
Hmm that's really peculiar; we haven't seen an issue like this, so I'm not sure how to troubleshoot.
I don't think this is operation-specific, so I wonder if you could try some other tests on this server. Perhaps do something simple, like shexec echo hello world; ? And perhaps see if it's an issue on other servers?
Thanks,
Alana
Thanks for clarifying! So it sounds like the issue is, the Git::Checkout-Code operation is freezing on a remote server (connected via SSH).
I don't think there's anything you're doing wrong, but there's clearly something going on. It's hard to guess what; typically an input prompt is about the only thing that will cause a freeze like that -- but I don't know how that can happen with that operation.
From here, i would check out the sshd logs, and see what's going on between the servers. You should see the last ssh command ran, and maybe that will give some kind of clue as to a problem