Posts made by apxltd

Hi @nicolas-morissette_6285 ,

Can you help me understand the use case here of ProGet and Azure DevOps Packages? Basically I'm a bit hesitant to invest in continued work-arounds for bugs/quirks in ADO Packages.

The ADO Packages team has made it very clear they aren't interested in spec compliance (whether NuGet or otherwise), and that they won't support interoperability with any other tool aside from Azure DevOps. That makes sense because they want you to only and entirely use Azure DevOps.

We've already work around one quirk, and I have little confidence that fixing this quirk will actually solve the problem (i.e. enable the use case you're trying to build) and that it will stay fixed. In a few weeks/months, when they ship some new functionality, your hybrid-ADO Packages usecase may break again.

For example, when ADO-Packages decided to block "private upstream feeds" (I guess that's what it's called?), it disabled the use-case of having ADO-Packages pull from ProGet servers, or vendor's feeds, or anywhere other than NuGet.org or ADO-Packages feeds. Fortunately it's easy enough to work-around in ADO.

Bottom line, using ADO Packages this way may prove to be very fragile, and you may be better served by not using ADO Packages altogether. A lot of other users have made that decision, and ADO works fine (better IMO) without using their Packages feature... and it's highly likely it will continue to indefinitely.

Alex

Hard to say, but there were some bugs with NuGet.org, where some packages would not show up when queried through the v2 API, but would come with v3. I wonder if it's related to you using a v3 API connector, but a v2 API call to ProGet? It should work.

You can append #v2 to a nuget.org connector's API to force it to use the v2 api instead of the v3.

There's no limit set by ProGet to the size of a feed, and we have customers with terrabytes of packages, so it should be okay.

Hello;

Kaspersky is totally wrong. This is a 100% false positive. Can you share this thread with them?

LoupeViewer.exe is a tool that, among other things, inspects MSIL of live processes for debugging purposes, and it's embedded in our installer. It seems their (or whatever provider they use) heuristics algorithm saw an ".exe that embeds an .exe that uses MSIL inspection hooks like this", and flagged it as this signature.

It shouldn't take more than a few seconds to verify that LoupeViewer.exe is not a trojan, and it'd be easy to inspect the disassembly.

Microsoft and several other providers have already removed whitelisted this in response to other customers submitting this.

Our installer is safe. Rest assured that if the package is signed by Inedo, you can be absolutely certain that no "viruses," malware, Trojans, or other malicious code were injected into our installers.

Please see this article for more information: https://inedo.com/support/kb/1113/false-positive

Hi Stijn,

The PowerShell client is indicating that there's an error downloading that file, and that when it tries to open the package (which is a zip file), it's corrupt.

I don't know what the error is, but if you follow that URL in your browser, you should receive a .nupkg file. If you can open it up as a zip file, and it opens ok, then the error might have to do with a proxy server, or something, that's corrupting the fileon the way from ProGet to PowerShell?

Hello;

We are aware of the vulnerabilities.

Our usage of this library is minimal, and we do not use it in a manner that would impact product security (i.e. "passing HTML from untrusted sources - even after sanitizing it - to one of jQuery's DOM manipulation methods (i.e. .html(), .append(), and others). So please consider this a "false positive".

Upgrading the library to 3.5.0 would add no value to ProGet users, but it would provide significant risk of UI regressions and risk of introducing unknown security vulnerabilities, that haven't yet been reported.

Aaand, published :)

Appreciate verifying & helping us finding this bug.

Just a quick in-progress preview of what we've got coming; feedback welcome!

UPDATE: Conan support is here!

https://docs.inedo.com/docs/proget/feeds/conan

Are you actually using Conan?

I've talked with a lot of users over the years about C/C++ development, and the emergence of Conan, and after a careful evaluation, not a single user opted to not use it due to the complexity it adds. Some instead switched to NuGet (which supports C++), and others just use UPack / Universal Packages.

So far as I can tell (I'm not a C/C++ developer), the big challenge is that Conan "wants to be" a cross-platform open source hub (like the NuGet.org of C/C++), but C/C++ isn't cross-platform, and there's no real standardization in how you configure C/C++ project, so the Conan platform has to jump through a lot of hoops to make it work.

~~From my understanding, you need to make "recipes" that are like a kind of installation script, and these are written in Python (??), and executed using Bash. So, on Windows, that many more steps to follow. ~~

Another problem I heard was development environments vs target environments. Like how you can code on Windows, but your target is embedded something, so it adds a whole new layer of complexity. I also remember something about, building from source vs libraries, and how it was a lot of work to get it to work the way you want.

Our customers had zero interest in taking open source packages from the Conan's public repository; it was only for their own, internal-use packages.

Again... this is all second-hand for me, so, if you have more insight please add it.

@olivier we're mostly still looking for insight/answers into the questions @gdivis had asked, so if you can help on this we could at least move the ball forward

@gdivis said in Support for R and CRAN:

1. ProGet typically needs to understand versioning rules for packages (to show which is the latest, to do retention policies, etc). From what I've gathered, CRAN uses 2 or more integers separated by dots. Does anyone know if that's required universally, or if it's just a convention?
2. CRAN doesn't appear to have a server-side repository API, instead just exposing a list of packages using HTML. Is my understanding correct here?
3. Submitting a package to CRAN looks like it's only done manually using a form here. Is that right, or is there some other automated mechanism that is used?

Thanks for the additional info... I definitely appreciate the desire for consistency in package management.

we have to manually copy down the packages

Why is this? Is there a sort of internet restriction policy in your build environment?

What do you mean by "drop in value"... is it like, "we already have ProGet, and it would save X amount of time to not have to create/manage a separate, private Conda repository"?

I spent a bit of time looking into CONDA, and here's my take.

According to the documentation, "in its default configuration, conda can install and manage the over 7,500 packages at repo.anaconda.com that are built, reviewed, and maintained by Anaconda."

This leads me to believe that the primary use case for Conda is to gain access to the curated package repository for data scientists that use the Anaconda development platform.

One of the major value propositions of ProGet is that it gives you the ability to curate free and open-source packages... so I don't see how ProGet will help? I could be wrong... but I'd really want to see some market demand or more customer demand for this before we consider investing engineering resources.

While it seems that Conda could also be used for other cases (they say "it was created for Python programs, but it can package and distribute software for any language"), the documentation is almost entirely focused on the primary use case.

If you're using Conda in it's second usecase (as a sort of generic package system), ProGet already has Universal Packages built in. These were created for first-party application and components (first and foremost), so they will likely be much more suitable than the secondary case of Conda

Ah, you must be running this against a Linux-based server; I don't the error message is very clear, so I've already changed it in the code); hopefully this is more clear now:

To use Git-based operations on Linux/SSH-based servers, you must specify a path for the Git executable. This can be configured on the operation, or by creating a variable named $DefaultGitExePath, which you can scope to the server, role, etc.

@markcodyre_7146 thanks for the note Mark, I appreciate it! Glad it's working well, despite the challenging install - we've got a lot of great container features in the pipeline, so hopefully we'll learn more about Linux in the process ;)

Definitely sounds like a bug; we've get it logged.

By the way, we've got some real exciting container/registry features coming very soon in ProGet 5.3. Stay tuned!

Great solution, thanks for sharing @Zazcallabah

Our (paying) customer hardly ever configures ProGet in this manner, and we aren't exactly Linux experts so writing documentation on seemingly dozens of different ways on how to do this sort of thing is really tough for us.

I'd very much welcome your contribution to our documentation, it's on GitHub (just click "Propose an Edit" at the bottom): https://docs.inedo.com/docs/proget/installation/installation-guide/linux-docker

I want ProGet to be a great community tool, and hope you can help contribute to the Linux installation experience; thanks!

This is, unfortunately, a known versioning bug; see the "PowerShell Module Package Version Quirks" section in our PowerShell Feeds documentation.

I recommend you to escalate it and ask the folks at Microsoft to fix it; if enough people ask, I'm sure it will eventually be addressed.

Actually, I'm not wrong; ProGet is simply not designed for your use case. It is infinitely more stable in the use cases it was designed for. ProGet also can't bake you muffins; an "oven" is much more suited for this purpose.

There are lots of tools that solve different problems, and ProGet is not designed as a desktop-based, single-user "server" for personal use. We have no intention of building such a tool; our tools are designed for teams to work together, not power tools for single users.

Anyways, it looks like found the tool that you're looking for. Best of luck.

Hi PhilippeC;

Not sure I totally understand; can you give an OtterScript example, and then sort of describe how the behavior Should work?

Yes; you would use a Docker feed and a Connector for this. ProGet will cache the container images and metadata, so it should give you better reliability when Azure container services has an outage or they introduce a new "feature" that breaks things ;)

Get in touch direct,happy to work toether on POC/implementation

We're targeting Q1 for this feature, to integrate w/ on-prem container vulnerability scanning.

Working w/ long paths is tough, and requires specialized Win32 API calls. Maybe we missed one, maybe NuGet or VS missed one.

Is this an error on the ProGet server? I.e. does its how up in Admin > DIagnostic Center? Can you pull this package to ProGet service, and download it from WebUI?

We haven't heard any other complaints, so it leads me to think it's either your end (proxy?) or npm's end (blocking). We have no idea what their blocking mechanism could look like. For all we know, they could be doing agent-string filtering, and blocking requests from ProGet software, but not your web browser or npm client.

Best bet is to run ProGet through a proxy (like fiddler or wireshark), by going to Admin > Proxy. Then you can see exact traffic.

Of course, plan versioning is introduced in a later version, and Rafts (which allow for Git-based storage) is coming in 6.2.

I definitely understand the hesitance to upgrade, but worth noting ---- for once 6.2 comes around we'll offer some great migration tactics to let you pull applications from really old versions of BuildMaster. We'll also better support multi-instances of BuildMaster, so each group can upgrade as they'd like.

That said, the Plans database table will contain your OtterScript plans. Not sure if that's a good solution, but pulling from that table will give you latest versions.

ProGet does not support SQL Server 2005; sometimes the installer doesn't detect it, but this error will happen if you're using SQL 2005.

There are no additional logs; newer versions of ProGet may improve, but ultimately if this is a result of rate limiting there wouldn't be much to detect. I think the registry file it's downloading is massive (hundred of megs, gigabyte?), so if it was rate limited there'd be no way to detect it.

You could run PRoGet through a proxy (like fiddler or wireshark), by going to Admin > Proxy. That might be your best bet to see exact traffic.

Not sure which organization you're with, but you may have been blocked by npm. They recently announced they will be blocking large organizations; I'm not sure how to check this, because the only information is that blog article.... but it sounds like rate limiting to me?

I don't think it's a SQL problem, but from your screenshot it looks like you're using SQL 2005. You should definitely upgrade that. ProGet (or Microsoft) doesn't support that anymore, and hasn't for a couple years.

Hello;

There may have been a miscommunication somewhere; do you know specifically what you were told?

We recently added the Feeds Management API, but that's only to manage feeds (not packages).

I just updated the documentation, and it will be published soon.

You can delete (permanently remove) or unlist (hide from most search results) NuGet packages from your feed by navigating to the package page and clicking the corresponding Delete Package or Unlist Package button. These actions require the Feeds_DeletePackage or Feeds_UnlistPackage permission attribute, respectively.

To programmatically delete a package from your feed, you can use the NuGet CLI's delete command, or make a DELETE request via HTTP:

DELETE http://{proget-server}/nuget/{feed-name}/package/{ID}/{VERSION}`

Note that this behavior is different than NuGet.org's DELETE command, which unlists packages instead.

To programmatically unlist (or relist) a package, you can use the NuGetPackagesV2_SetListed method within the Native API.

Is that helpful?

The ExecutionDispatcher doesn't start right away after a service restart; it takes about 5 minutes after service start. This is a known issue (it's a hack/workaround), that we will fix ASAP; it has to do with the fact that we don't want to run an execution while the agents are updating.

InedoCore 1.5.1 should not have been available to be downloaded (we recently moved the inedo.com/den/feed to proget.inedo.com/upack/extensions). It's deleted from the feed.

You should be able to delete it, then reinstall 1.5.0 as per normal.

ProGet does not support the deprecate command, but you could accomplish the same thing by adding a Manual Vulnerability.

We can't seem to figure out what the deprecate command actually does, or how it's supposed to work. It's pretty poorly documented, and since there is no deprecated flag in npm package metadata, the client must be doing some strange things to check. Who knows...

Anyways, I'd stick with Manual Vulnerability, since at least that's documented a lot better and won't change from client version to client version.

The Execution Dispatcher (Admin > Manage Service) doesn't start immediately after service restart; this causes the same symptoms you're seeing now. You can manually trigger it on that page, or just wait.

It's a known issue, and just a (temporary) workaround to problem where executions start in the middle of agent upgrades. We've addressed it in Otter, but it's a bit complex to do in BuildMaster pre-6.2.

The homepage will attempt to connect to inedo.com to download latest news, and it will also connect to the database to provide a summary of feeds (dashboard). However, if either of those operations times out, then you should get a prompt error.

Maybe, for some reason, the request isn't even being routed to ProGet. You can connect ProGet to a proxy like Fiddler (Admin > Proxy), and then attach to SQL Server to see if anything is timing out..

All "names" are, unfortunately, limited to 50 characters; we thought this was "more than enough", and it's a common limitation in a lot of programming languages.

This is not something feasible to change for a lot of reasons (consistency across all products, database columns, UI validation, etc)...

Assuming that ProGet has access to inedo.com, ProGet will automatically activate. Otherwise, you will have to manually activate. Activation is required when the CPUID changes or the MAC address changes. So, the easiest thing to do is just to ensure those don't change; these are almost certainly options in the VM Host.

It's possible to programmatically active, but quite bothersome. You'd need to call our (undocumented) product registration endpoint, then add data to the ProGet database, then reset the web application. We don't really support this, and it's quite fragile (i.e. having someone else maintain this as a SOP), so it's best to just ensure the ProGet VM doesn't change and require re-activation.

Hi Chris,

You'll need to run the NuGetPackagesV2_SetListed method instead. In order to this, you'll need to mark it as non-internal; this can be done with a simple database query...

UPDATE [__StoredProcInfo] SET [Internal_Indicator] = 'N' WHERE [StoredProc_Name] = 'NuGetPackagesV2_SetListed'

This will be also be updated in 5.2.10 as PG-1549.

Here's the response that @jrasch posted to that ticket;

The All Versions tab should definitely still be there, and the only case I could see where it's hidden is for the feed type of "Docker" which does not use versions.

Can you try:

• Hard browser refresh e.g. CTRL + F5
• Verify that you are logged in and not browsing anonymously

Is that what you're looking for?

Basically it sounds like it was an issue of either not being logged in, or some old Javascript cached in your browser? It's hard to say, because you had responded in the ticket that it was resolved...

We recently moved our documentation; do you know where you found those 404s? We're also monitoring via Analytics Tools as well, but finding sooner is better :)

Anyways PSCall is good in Orchestration Plans (can be done in a Configuration Plan, under some conditions), and PSEnsure is best for Configuration Plans.

Explaining Ensure vs Execute can be tricky, but I'd recommend checking out our ebook, called Windows-first Guide to Infrastructure as Code and Continuous Configuration Automation :)

We definitely want to add these as first-class Operations in our Windows extensions, but in the mean time the best route is to use the PSDsc Operation to invoke the Windows Feature DSC Resource

PSDSC WindowsFeature
(
  Name: Web-Server,
  Ensure: present
);

Hope that helps!

Will help support a push command at some point? ;-)

We documented a few ways you can Publish Helm Charts, because there was no push command at the time. It doesn't seem there's now one... https://helm.sh/docs/helm/

Can you be more specific? I don't understand?

You most definitely do not want to "throw away" your package store or database...

There shouldn't be a problem doing this. Just makes sure the BaseUrl is configured properly in Advanced/All Settings under Admin.

FYI: it's on our roadmap to have BuildMaster (and Otter) work on a multi-node installation, so you could have multiple web and multiple service nodes (similar to ProGet).

This is why it works directly...

Remember that the NuGet.org not only runs on a massive web farm with dozens of load-balanced servers, but it's a static-file based index that's done mostly with CDN-based files.

Each request you make to ProGet, on the other hand, needs to be authenticated, checked for vulnerabilities, licenses, sent to connectors, etc.

The sockets are not getting exhausted, the async awaits are timing out. This is exactly what to expect in a connection overload situation, which will be common place in the way you're using ProGet.

ProGet is not designed nor supported as a desktop tool.

That's strange, but it sounds like request filtering, WebDav, or some other security feature (outside of ProGet)... ProGet doesn't restrict anything by extension or anything like that.

There's a few places to look, but i'd start here:

https://stackoverflow.com/questions/12828476/what-file-extensions-are-blocked-by-default-in-iis

Is that helpful?

Oh; yeah that'll definitely do it.

50 packages yields hundreds of requests to a NuGet feed (ProGet). Each request to ProGet is then forwarded to Nuget.org. Add to that Docker request routing, PostGres network connections... and all of this on a single machine, calling itself over network channels...

You're basically DoS-ing yourself ;)

Hi; do you have any specific recommendations? We are reevaluating our Docker strategy, perhaps to include things like Swarm or Kubernetes as distribution options.

Here's what we do now...

https://github.com/inedo/proget-docker

I'm just following up to see if you were ever able to work-past this, or if it's still an issue?

This is on our future roadmap; for now you'll need to install and manage as a normal windows application.

@philwaller5269_6322 what page did you find the broken link on?