Posts made by apxltd

Get in touch direct,happy to work toether on POC/implementation

We're targeting Q1 for this feature, to integrate w/ on-prem container vulnerability scanning.

Working w/ long paths is tough, and requires specialized Win32 API calls. Maybe we missed one, maybe NuGet or VS missed one.

Is this an error on the ProGet server? I.e. does its how up in Admin > DIagnostic Center? Can you pull this package to ProGet service, and download it from WebUI?

We haven't heard any other complaints, so it leads me to think it's either your end (proxy?) or npm's end (blocking). We have no idea what their blocking mechanism could look like. For all we know, they could be doing agent-string filtering, and blocking requests from ProGet software, but not your web browser or npm client.

Best bet is to run ProGet through a proxy (like fiddler or wireshark), by going to Admin > Proxy. Then you can see exact traffic.

Of course, plan versioning is introduced in a later version, and Rafts (which allow for Git-based storage) is coming in 6.2.

I definitely understand the hesitance to upgrade, but worth noting ---- for once 6.2 comes around we'll offer some great migration tactics to let you pull applications from really old versions of BuildMaster. We'll also better support multi-instances of BuildMaster, so each group can upgrade as they'd like.

That said, the Plans database table will contain your OtterScript plans. Not sure if that's a good solution, but pulling from that table will give you latest versions.

ProGet does not support SQL Server 2005; sometimes the installer doesn't detect it, but this error will happen if you're using SQL 2005.

There are no additional logs; newer versions of ProGet may improve, but ultimately if this is a result of rate limiting there wouldn't be much to detect. I think the registry file it's downloading is massive (hundred of megs, gigabyte?), so if it was rate limited there'd be no way to detect it.

You could run PRoGet through a proxy (like fiddler or wireshark), by going to Admin > Proxy. That might be your best bet to see exact traffic.

Not sure which organization you're with, but you may have been blocked by npm. They recently announced they will be blocking large organizations; I'm not sure how to check this, because the only information is that blog article.... but it sounds like rate limiting to me?

I don't think it's a SQL problem, but from your screenshot it looks like you're using SQL 2005. You should definitely upgrade that. ProGet (or Microsoft) doesn't support that anymore, and hasn't for a couple years.

Hello;

There may have been a miscommunication somewhere; do you know specifically what you were told?

We recently added the Feeds Management API, but that's only to manage feeds (not packages).

I just updated the documentation, and it will be published soon.

You can delete (permanently remove) or unlist (hide from most search results) NuGet packages from your feed by navigating to the package page and clicking the corresponding Delete Package or Unlist Package button. These actions require the Feeds_DeletePackage or Feeds_UnlistPackage permission attribute, respectively.

To programmatically delete a package from your feed, you can use the NuGet CLI's delete command, or make a DELETE request via HTTP:

DELETE http://{proget-server}/nuget/{feed-name}/package/{ID}/{VERSION}`

Note that this behavior is different than NuGet.org's DELETE command, which unlists packages instead.

To programmatically unlist (or relist) a package, you can use the NuGetPackagesV2_SetListed method within the Native API.

Is that helpful?

The ExecutionDispatcher doesn't start right away after a service restart; it takes about 5 minutes after service start. This is a known issue (it's a hack/workaround), that we will fix ASAP; it has to do with the fact that we don't want to run an execution while the agents are updating.

InedoCore 1.5.1 should not have been available to be downloaded (we recently moved the inedo.com/den/feed to proget.inedo.com/upack/extensions). It's deleted from the feed.

You should be able to delete it, then reinstall 1.5.0 as per normal.

ProGet does not support the deprecate command, but you could accomplish the same thing by adding a Manual Vulnerability.

We can't seem to figure out what the deprecate command actually does, or how it's supposed to work. It's pretty poorly documented, and since there is no deprecated flag in npm package metadata, the client must be doing some strange things to check. Who knows...

Anyways, I'd stick with Manual Vulnerability, since at least that's documented a lot better and won't change from client version to client version.

The Execution Dispatcher (Admin > Manage Service) doesn't start immediately after service restart; this causes the same symptoms you're seeing now. You can manually trigger it on that page, or just wait.

It's a known issue, and just a (temporary) workaround to problem where executions start in the middle of agent upgrades. We've addressed it in Otter, but it's a bit complex to do in BuildMaster pre-6.2.

The homepage will attempt to connect to inedo.com to download latest news, and it will also connect to the database to provide a summary of feeds (dashboard). However, if either of those operations times out, then you should get a prompt error.

Maybe, for some reason, the request isn't even being routed to ProGet. You can connect ProGet to a proxy like Fiddler (Admin > Proxy), and then attach to SQL Server to see if anything is timing out..

All "names" are, unfortunately, limited to 50 characters; we thought this was "more than enough", and it's a common limitation in a lot of programming languages.

This is not something feasible to change for a lot of reasons (consistency across all products, database columns, UI validation, etc)...

Assuming that ProGet has access to inedo.com, ProGet will automatically activate. Otherwise, you will have to manually activate. Activation is required when the CPUID changes or the MAC address changes. So, the easiest thing to do is just to ensure those don't change; these are almost certainly options in the VM Host.

It's possible to programmatically active, but quite bothersome. You'd need to call our (undocumented) product registration endpoint, then add data to the ProGet database, then reset the web application. We don't really support this, and it's quite fragile (i.e. having someone else maintain this as a SOP), so it's best to just ensure the ProGet VM doesn't change and require re-activation.

Hi Chris,

You'll need to run the NuGetPackagesV2_SetListed method instead. In order to this, you'll need to mark it as non-internal; this can be done with a simple database query...

UPDATE [__StoredProcInfo] SET [Internal_Indicator] = 'N' WHERE [StoredProc_Name] = 'NuGetPackagesV2_SetListed'

This will be also be updated in 5.2.10 as PG-1549.

Here's the response that @jrasch posted to that ticket;

The All Versions tab should definitely still be there, and the only case I could see where it's hidden is for the feed type of "Docker" which does not use versions.

Can you try:

• Hard browser refresh e.g. CTRL + F5
• Verify that you are logged in and not browsing anonymously

Is that what you're looking for?

Basically it sounds like it was an issue of either not being logged in, or some old Javascript cached in your browser? It's hard to say, because you had responded in the ticket that it was resolved...

We recently moved our documentation; do you know where you found those 404s? We're also monitoring via Analytics Tools as well, but finding sooner is better :)

Anyways PSCall is good in Orchestration Plans (can be done in a Configuration Plan, under some conditions), and PSEnsure is best for Configuration Plans.

Explaining Ensure vs Execute can be tricky, but I'd recommend checking out our ebook, called Windows-first Guide to Infrastructure as Code and Continuous Configuration Automation :)

We definitely want to add these as first-class Operations in our Windows extensions, but in the mean time the best route is to use the PSDsc Operation to invoke the Windows Feature DSC Resource

PSDSC WindowsFeature
(
  Name: Web-Server,
  Ensure: present
);

Hope that helps!

Will help support a push command at some point? ;-)

We documented a few ways you can Publish Helm Charts, because there was no push command at the time. It doesn't seem there's now one... https://helm.sh/docs/helm/

Can you be more specific? I don't understand?

You most definitely do not want to "throw away" your package store or database...

There shouldn't be a problem doing this. Just makes sure the BaseUrl is configured properly in Advanced/All Settings under Admin.

FYI: it's on our roadmap to have BuildMaster (and Otter) work on a multi-node installation, so you could have multiple web and multiple service nodes (similar to ProGet).

This is why it works directly...

Remember that the NuGet.org not only runs on a massive web farm with dozens of load-balanced servers, but it's a static-file based index that's done mostly with CDN-based files.

Each request you make to ProGet, on the other hand, needs to be authenticated, checked for vulnerabilities, licenses, sent to connectors, etc.

The sockets are not getting exhausted, the async awaits are timing out. This is exactly what to expect in a connection overload situation, which will be common place in the way you're using ProGet.

ProGet is not designed nor supported as a desktop tool.

That's strange, but it sounds like request filtering, WebDav, or some other security feature (outside of ProGet)... ProGet doesn't restrict anything by extension or anything like that.

There's a few places to look, but i'd start here:

https://stackoverflow.com/questions/12828476/what-file-extensions-are-blocked-by-default-in-iis

Is that helpful?

Oh; yeah that'll definitely do it.

50 packages yields hundreds of requests to a NuGet feed (ProGet). Each request to ProGet is then forwarded to Nuget.org. Add to that Docker request routing, PostGres network connections... and all of this on a single machine, calling itself over network channels...

You're basically DoS-ing yourself ;)

Hi; do you have any specific recommendations? We are reevaluating our Docker strategy, perhaps to include things like Swarm or Kubernetes as distribution options.

Here's what we do now...

https://github.com/inedo/proget-docker

I'm just following up to see if you were ever able to work-past this, or if it's still an issue?

This is on our future roadmap; for now you'll need to install and manage as a normal windows application.

@philwaller5269_6322 what page did you find the broken link on?

There was a regression for the PowerShell agents that caused this to behave like this in some cases, but it was fixed in Otter 2.2.5 (released today). I should be fixed

Let me know if this resolves your issue!

The first example you mentioned (i.e. the one to /api/management/feeds/create) is using the Feeds Management API; it looks ok to me on first glance... can you share the error message you got when invoking it? You should be able to see logged request/responses in the Admin > API Keys as well.

The second example you mentioned (Feeds_CreateFeed) is using the Native API, which we don't really recommend if there's an alternative available. It is basically a wrapper around stored procedures and the database. But in this case, it looks mostly correct, but the FeedType_Name is wrong; if you look at the Feeds table in the database, you'll see a universal feed is actually called ProGet in the database.

Anyways, please use /api/management/feeds because it's easier to use and won't change if we update the database or stored procs.

Quick update: there's a bug we identified with some WsMAn connections that are causing different errors, but it might be related. We're going to fix this in Otter 2.2.5, shipping Friday.

Otter 2.2.2 doesn't seem to exhibit this behavior.

Long story short, your workstation is overwhelming your server with network connections.

Remember that the NuGet.org not only runs on a massive web farm with dozens of load-balanced servers, but it's a static-file based index that's done mostly with CDN-based files.

Each request you make to ProGet, on the other hand, needs to be authenticated, checked for vulnerabilities, licenses, sent to connectors, etc. And I would be surprised if your server is more powerful than your workstation.

There are some features in ProGet Basic like metadata caching that will help, but ultimately when you scale to more developers you ought to invest in better server hardware and eventually load balancing. See https://blog.inedo.com/proget-free-to-proget-enterprise

This is because Get-Http is an execute-only operation, which means it will only run if configuration changed.

To force execute-only operations to run in configuration plans, you need to specify the execution directive to be always execute, as follows...

with executionPolicy=always 
{
   ...
}

Hopefully we can better document this in the future; it's buried in the formal specification.

@knitvijay_7631 said in During build getting error (unable to clone from github):

Clone failed: unknown certificate check failure

I did a quick search on this message, and there's lots of advice on how to get this working. The problem is coming from Git, and BuildMaster is just reporting the problem. I think your best bet will just be to use HTTPS instead of SSH. It's a lot easier to configure...

BUt here's a post that seems to be quite popular that gives lots of tips and tricks on resolving this..

Example: https://stackoverflow.com/questions/3777075/ssl-certificate-rejected-trying-to-access-github-over-https-behind-firewall

That's not surprising; as I mentioned, the problem lies with your configuration. Either you're using the wrong name and password in Otter, or the WSMan endpoint on the remote server isn't enabled.

I recommend you to use to Inedo Agent.

We hope to include this ability in a future release, but it's a bit more complicated to get the details worked out. For now, just uninstall, then reinstall (pick version you want) will work.

Here is the underlying error message:

Can not connect to Windows servers with WSMan endpoint. Try to use credentials FQDN and Netbios, result - exception.

Basically, this means that your username/password is not being accepted. It should be something like DOMAIN\username or username@domain.local

It could also be that WSMan isn't configured to allow these connections; this can be controlled at the domain. I recommend you use the Inedo agent, it's a lot easier to set-up and get working

@valeon fantastic, thanks so much! This will really help us explore; it doesn't look "too bad", and is "somewhat similar" to how Debian manages it's packages.

We'll try to start hacking around with a POC in the coming weeks, hopefully; i'll update when I can!

Just as an update, we will be doing this:

https://inedo.myjetbrains.com/youtrack/issue/PG-1555

"Proxy npm audit requests to npmjs.org (experimental)"

Unfortunately, npm audit is a totally undocumented endpoint and based on past experiences, npm's API frequently changes is nontrivial to reverse engineer. Moreover, npm, Inc does not permit or support third-party access to the API that's used by npm audit.

When they change that underlying API (whether to enforce the no third-parties rule, or to do something from the client), ProGet will once again be broken (or worse, provide incomplete/incorrect results). At least now, you know that this is the only supported way to handle it...

Do you have an npm enterprise license? This might be something to work with through their support channel.... they don't have a partner program at this time, so getting permission or insight into how we can access this API is difficult.

ProGet does not have the concept of "public" or "private" feeds, you can instead grant the "Anonymous" user certain permissions, including viewing and publishing packages.

ProGet Free Edition does not allow feed-level permissions, only system-level permissions. So you can grant "Anonymous" whatever rights you want, just not at a feed-level.

this is solved but i'm still replying

Hi there.

This is another test of the test

If you set Warn, will it automatically advance?

One idea as well... how about also setting a build variable using Set-ReleaseVariable called IsValidBuild, and then using a Variable Value Promotion Requirement (IsValidBuild = true)?

hi all, thanks for the interest/comments; I decided to write-up a page that details this on the docs.

http://inedo.com/support/documentation/proget/feeds/other-types

I'm hoping we can use this public thread to maintain the discussion on technical detail; otherwise it'll get stuck in my email, or somewhere else, and we can get everyone to chime in this way.

That said, @M-W if you've got any insight into how R/CRAN works please do share :)

The Postgres container has had a lot of performance problems at scale, and neither our customers nor engineers could figure it out. A regular instance was fine, but our customers wanted containers.

But in the long term, maintaining two separate code bases doesn't make sense. And now that Sql Server is available, it makes sense to do it.

Hi, sorry on the slow reply!

I think you've got a good undersatnding of the situation, but a couple of comments:

• do not share the Temporary files (ServiceTempPath, WebTempPath); these should be kept on the same server as the BuildMaster web/service app. These are only used during the runtime of those applications.

• on the back up server, make sure both the Web Application and Service are set to DISABLED, or configured with a bad username/password such that they cannot be started easily; having two identical BuildMaster instances pointing to the same database and the same agents will cause that multi-master problem you dont' want to deal with :)

• we have had some customers put our products in a Windows Container, but largely the support on the Microsoft side isn't so great, and it's more trouble than it's worth; we are moving towards DotNetCore so we can have BuildMaster run on Linux (and linux containers)

Does that make sense? Let me know how it goes.

And by the way, I'd love to document this better... would you be interested in helping me with this, especially once you have it running in your set up? I think it would really help the community :)