Posts made by atripp

Hi Nik;

There's nothing in ProGet that would yield this behavior, and it's not behavior that others experience, so I think it must be something between ProGet and your workstation. Maybe proxy server, request filtering, etc.?

Just to check, you are using chunked uploads with Docker, right? It's been the default for years, but i guess it can still be disabled. The same intermediate server may be stripping out ProGet's support of chunks? I don't know -- https://github.com/docker/distribution/blob/master/docs/spec/api.md#chunked-upload

We've also had reports of strange errors happening on random blobs from one or two customers. They discovered it was the Request Filtering feature enabled in IIS. The error was different, and it had to do with blocking query strings with :’s in them. But maybe it's this?

hope this helps

Otter uses Inedo.SDK 1.6, where as BuildMaster is on Inedo.SDK 1.7. The major addition to 1.7 is SecureResources and SecureCredentials, which replace ResourceCredentials. However, it's still backwards compatible.

The current (1.7) version of GitLab::Get-Source has two changes:

• Changed Credentials property to From (but keeps Credentials as an alias)
• Added From as the default property, so it will be used as argument right after the operation name

This keeps it backwards compatible.

We will add SDK 1.7 to Otter later this year, but in the mean time you can use the Otter version code and create a ResourceCredential in BuildMaster of the same name.

For this case, you're right -- the Instance ID would be the way to go. On the new BuildMaster server, you can set the Agent.InstanceId field (any string is fine, "bm62" would work) , so this way, both BuildMaster instances won't collide.

We are using our "Customer Success Program" to provide upgrade assistance for BuildMaster 6.2, so the best way to get started on this would be to talk to your account representative.

You can also submit a ticket and put 6.2 UPGRADE in the "How can we help" section, and we'll route that to the right people :)

Indexing the symbol server wouldn't make a difference. I would try clearing your connector cache.

It's very unlikely that this is a regression in 5.3; this is a very common use case, nothing strange about the package, and I can't reproduce it. So it's likely configuration or network related.

To rollback, please follow the restore instructions: https://docs.inedo.com/docs/proget/installation/backing-up

I think this is related to the foreign layer. This is a known issue, it's actually not an "error", but "by design" of Docker.

https://medium.com/@amlys/solution-docker-push-fails-with-skipping-foreign-layer-a119e03bdf0d

I think this will fix it.

Hello;

That "error" message is to be expected; that's what ProGet returns when you visit to /v2 with a web browser. But the error sounds like a sort of SSL/certificate configuration problem.

After searching for "tls: no renegotiation", here is an article I found about that might help: https://www.digicert.com/news/2011-06-03-ssl-renego/

Hi Nik, ProGet does indeed support pushing Windows containers.

What sort of errors are you receiving?

Oh I see; in this case, please remove your credentials from the connector. They're not required for NuGet.org; ProGet does not push to connectors, only download files.

It's a common mistake and we will try to make it clear in the UI.

First, upgrade to BuildMaster 6.1 (see KB#1163 Upgrading to BuildMaster 6.1; this is a relatively low risk, as BuildMaster 6.1 still has all of the legacy features included. There are several "bridge features" to help get you to 6.2

Then, consider a upgrade or migration strategy to BuildMaster 6.2 (see KB#1766 Upgrading to BuildMaster 6.2). Depending on your legacy feature usage, you may want to do a direct upgrade, or you may want to migrate.

And lastly, please keep in mind this note from the docs on BuildMaster v6.2 Migration:

Because this upgrade is more involved than previous upgrades, special support for it has been included with paid licenses. If you are interested in learning more, use the Submit Ticket Form and put 6.2 UPGRADE in the "How can we help" section to let us know when you are going to upgrade. We also recommend you include the Legacy Features Dashboard Report Logs in the body of the ticket.

Hello; I'm not able to reproduce this.

I can successfully download Microsoft.EntityFrameworkCore 3.1.3 without an issue from ProGet 5.3.0-m5.

Can you navigate to the package in your instance, and hit the Download button?

Maybe it was a temporary NuGet outage?

Hello; i don't know how easy it is to implement, but I put I added this for an internal assessment; https://inedo.myjetbrains.com/youtrack/issue/BM-3595

We'll try to evaluate in the coming weeks. Stay tuned!

@philippe-camelio_3885 I responded to this before realizing it was deleted; so I restored the post and moved it to my username. hope that's ok :)

The SHExec operation does not consider the error code for success; it appears to be relatively easy to change the code to do so (SHUtil::ExecuteScriptAsync returns an error code), but I think (I could be wrong) exit codes from shell scripts aren't so consistent?

In any case, it appears if a script writes to the error log (stream=2) then it will fail.

Thanks for the update! I've noted this in the docs, and linked to this discussion :)

https://github.com/Inedo/inedo-docs/commit/d24087911584bbda833314084a58c2ae1ff41c39

Hello
Howto raise an error in Buildmaster for a failed SHEXEC ?

I have an ansible playbook which failed (RC=2) but the Pipeline step is OK while it should be in error

I can't figure out how to manage this.

Here is the log:

...
INFO : 2020-05-03 13:46:43Z - fatal: [127.0.0.1]: FAILED! => {"changed": false, "msg": "Source template/site.conf.j2 not found"}
INFO : 2020-05-03 13:46:43Z - 	to retry, use: --limit @/tmp/bm/extranet-monespace/ansible/main.retry
INFO : 2020-05-03 13:46:43Z - PLAY RECAP *********************************************************************
INFO : 2020-05-03 13:46:43Z - 127.0.0.1                  : ok=4    changed=2    unreachable=0    failed=1   
INFO : 2020-05-03 13:46:43Z - Exit code is: 2
DEBUG: 2020-05-03 13:46:43Z - Script completed.
DEBUG: 2020-05-03 13:46:43Z - Deleting temporary script file (/tmp/otter/scripts/d2f7283d77c94cfb89151ca2e3004212)...
DEBUG: 2020-05-03 13:46:43Z - Temporary file deleted.
DEBUG: 2020-05-03 13:46:43Z - Cleaning up...
DEBUG: 2020-05-03 13:46:43Z - Deleting /tmp/otter/_E174799 on VM111021...
DEBUG: 2020-05-03 13:46:43Z - /tmp/otter/_E174799 on VM111021 deleted.
DEBUG: 2020-05-03 13:46:43Z - Cleanup complete.

Any ideas ?

What do you think about it? We use GitLab for our internal projects, but would like to understand the choices between GOGS/Gitea and the "big players" like GitHub/GitLab.

I know we are very interested in helping organizations use as many different source control platforms as possible, and GOGS seems pretty interesting! So keep us in the loop, as far as how your integration goes with GOGS, it might be worth us building a specific integration. I've also heard of Gitlea, but really not sure about their popularity in organizations, or if it's worth investing/investigating how to build better/first-class integrations to our products

Aww, that's too bad. Unfortunately Repository Webhooks aren't exposed in the SDK at this point, so it's not nearly as easy to try building/testing it on your own. We'll definitely consider adding this in a future SDK release however.

Thanks @kvasudevan_8753 - that definitely will do it :)

@nik-reiman_6009 package interaction is done through the package-specific, third-party apis; we only provide minimal documentation for these APIs, as they are generally either already documented elsewhere. We simply implement those API.

So in this case, if you search for how to delete a nuget package from the api, the first result is from Microsoft which details how to Push and Delete packages from a NuGet API

What I mean by "republish" it is to republish it to your ProGet feed. This local package will then "mask" the connector package, and ought to behave fine.

In this case, since it's just one package behaving like this, and there's such an easy work around (download package, edit metadata, republish), it's really not worth the investment to even investigate how easy/difficult it would be to fix. If it becomes more widespread, we'll investigate.

This has to be done on NuGet and PowerShell feeds for some old and quirky packages (like ones that have a 1.01.1 and 1.1.1 version). It may be trivial, but it could introduce the exact regression you described (people using build metadata in their package.json), or other (unknown) regressions, and it's just not worth the risk to our users.

The Webhooks are GitLab and GitHub specific, but if Gogs mocks/uses the GitHub pattern it should work.

Thanks, that makes sense where it comes from.

Since this is the only package with a problem, I wonder if you could just "republish" it with the build-metadata number missing?

We're a bit hesitant to make change that could impact this. For all we know, this works in older versions of the npm client, or newer versions of the npm client, or the yarn client, etc., and users are relying on this behavior.

Hello;

Sounds like there's a lot to unpack...

First, we've got some major changes in ProGet 5.3 that should help out quite a bit, but I've included a lot of resources in this post to consider as well. Consider that, as a private registry, ProGet is designed to host "private images" that you manage (which can be build from "public images"), and it's really not designed as a "registry mirror/cache". These are two different usecases.

The authentication issues, that might be different problem and might be worth a new thread?

Repository Mirroring in ProGet

From But here is our current thoughts/documentation on repository mirroring in Proget:

There is the ability to "mirror" hub.docker.org, but this feature was really only designed to enable Docker, Inc. to host public registries in sensitive regions like China (so you can mirror to registry.docker-cn.com instead).
Unfortunately, the Docker client will only use the configured "mirror" for building and pulling; images are always pushed to hub.docker.org.
Docker, Inc. does not seem to be interested in enabling this, and we aren't particularly keen on forking the Docker client to enable this. Thus, it seems there's no sense in ProGet supporting this anytime soon

Base Image Pattern

On Dockerhub, there are a lot of problematic images that cause a new breed of "worked on my Docker" problems in companies. openjdk:11 is a great example of this problem. The connector "problems" in ProGet are a symptom of the problem, but we strongly recommend you to not use "third-party" container image directly, but instead follow our "base image" development pattern.

First, and foremost, openjdk:11 currently references what's called a "fat" manifest, which is a sort of "pointer" to other manifests that will be used depending on the run-time environment of Docker. There are currently four different environment-specific images that this "fat" manifest points to, so when you build an image on top of it (e.g. FROM openjdk:11), your image will be "locked" to whatever environment configuration the build machine (or even workstation) had at the time the command was run. This may be different than the machine you run the image on.

Even more problematic, however, is that 11 is a time-sensitive tag, and what 11 refers to keeps changing. Today, it points to four different versions of four different images (as of 7 days ago), but tomorrow... it could point to 3 totally different images. Who knows? It's a repository maintained by a third party, and 11 means what they say it means, at their discretion.

The Dockerhub is meant to be the "authority" on this repository, and this is where connectors are a challenge. ProGet caches certain informations about these images, but when the meaning of these things change, it defeats the purpose of caching. If you disable caching, it should work... but then you've already , but what's the point then? You're downloading huge image files constantly then.

So instead, here's how we recommend you develop a practice around third-party base images.

docker pull openjdk:11
docker tag openjdk:11 proget.mycompany.com/dev-images/openjdk-linux64:11.0.6-rc.1
docker push proget.mycompany.com/devimages/mycomp/openjdk-linux64:11.0.6-rc.1

Even better would be to use a Dockerfile (e.g. FROM openjdk:11) that lets you customize the image as needed, but this a good start. Then, after you/developers are happy with that version of that image, then you can "re-tag" from 11.0.6-rc.1 to 11.0.6. In ProGet 5.3, these operations are made easier, and "virtual" tags are automatically generated. But the important thing is that you treat "tags" as immutable, and provide consistent images for your users.

Note that, the repository name is mycomp/openjdk-linux64. It's important to not use only openjdk-linux64, since that name is identical to _/openjdk-linux64 and library/openjdk-linux64. The library/empty/_ namespace are intended only for "Dockerhub-certified" images

Docker is supposed to solve "works on my machine", but unless you treat tags as immutable, you'll keep getting "works on my Docker when I ran it," and that's even harder to debug. ProGet is "exposing" the underlying problems with these machine-specific, time-specific tags, but I guess it's better to discover it early on, then in Production.

Hello, you can definitely delete packages form the API!

But please note, from the API Docs page:

API for Third-party Packages and Feed Types

In order to support third-party package formats types like NuGet, npm, etc., ProGet implements a variety of third-party APIs. We only provide minimal documentation for these APIs, as they are generally either already documented elsewhere. However, you can generally find the basics by searching for specific things you'd like to do with the API, such as "how to search for packages using the NuGet API" or "how to publish an npm package using the API".

hi Ryan,

Can you try Active Directory (New)? The Active Directory with Multiple Domains will be deprecated in 5.3, and the code behind the scenes is totally different anyways. So, it might behave differently.

Regardless, based on what you wrote, it seems to be a problem of NETBIOS mapping.

Basically, the LOGON_USER string contains DOMAIN\username, and ProGet needs map DOMAIN to the actual domain. Sometimes the automatic querying doesn't work (permissions, not configured on domain, etc), so you can specify it in an advance property in the Active Directory (New) provider.

So please try this if it doesn't work once switching to Active Directory (New).

Hmm, it's strange. Is this the only package and version that's acting like this? The "+" implies a sort of build-metadata in SemVer, but i'm not sure if npm supports this?

Anyways the npmjs.org version doesn't seem to have the text 4.12.20 anywhere, at all; any idea where that's coming from?

OK; at this point, I think it's going to best to schedule some one-one-one review time with our engineering team.

I will start the process on this end, but can you please open a ticket at here - https://my.inedo.com/tickets/new

You can write "6.2 UPGRADE" in the "How can we help", and include any other emails you'd like us to be in touch with in the form as well. Then we will work and get stuff scheduled!

Hello;

Given the problems with the existing server, here is my recommend:

1. Install BuildMaster 6.1 on a new server.
2. Verify it works
3. Stop WebApp and Service
4. Restore 5.8 Database on BuildMaster 6.1 server
5. Manually upgrade newly-restored 5.8 database using the 6.1 version of manual package (use the dbupdater command from the manual install)

Manual install guide for the database: https://docs.inedo.com/docs/buildmaster/installation-and-maintenance/installation-guide/manual

Backup and Restore instructions are here: https://docs.inedo.com/docs/buildmaster/installation-and-maintenance/backing-up

Glad to see you made your own templates!

So, there are two types of templates:

• Setup Templates, which are the JSON-based templates that give you access to quick settings

• Application Templates, which contain plans, pipelines... and setup templates

I just want to confirm, when you say "Apply Template", do you mean the latter (Application Template)?

If you can let me know about the error, such as a stack trace or log file, I can investigate!

That error message looks like it's the PipelinePage, not the Extensions page...

I think you've got a severe configuration problem on the server still. Between multiple databases, and "impossible" errors, I think you should try this...

Start with a totally new server.
Install BuildMaster 6.1, fresh installation
ensure it works (run the tutorial application)
Restore your database to that server.

From there, go to Admin > Extensions, and install what extensions you need.

All of the steps look correct!

I assume that, global::Proget BuildMasterTemplate contains a domain username.password with access to the feed?

The first thing I would try is to (temporarily) change global::Proget BuildMasterTemplate to your name/password. If that doesn't work, then grant Anonymous access to the feed.

If it still doesn't work, then I think the problem is that Integrated Authentication is enabled on IIS. Even though it's not enabled in ProGet, IIS will still force authentication.

I'm sorry, I misunderstood, I thought your instance was working. Now that I see older messages, I see not.

OK ...

What happens when you visit /applications/4? The error message you've shared appears to be from /all-applications.

Great, you've got 6.1 running.

So, BuildMaster v6.2 removes all legacy features, and the upgrade is blocked until you run the legacy feature detector and it says no features are detected. See the BuildMaster 6.2 Upgrade Notes.

I'm concerned about the state of your server. How about making a new server for BuildMaster 6.2, and then migrating your applications as per the guidance in the kb article? Then, you won't have more server problems in the future.

As for that script, I'm a little concerned with it. If it failed to run, then it means the BUILDNUMBER and RELEASENUMBER columns in the database may be too short (new length is 50 characters, old length was ... I forgot... but shorter). It's not a huge problem and would likely go undetected. Just woudl give an error if you made a build number that was between the old length and 50 characters.

Anyways, if you follow the plan of making a new BuildMaster 6.2 server, then migrating your applications per that article, then you won't have to worry about it.

It sure is! They use the same directory structure, and each product ignores what the other product doesn't use.

FYI: in our future vision, we want to make UPack-based rafts, so you can deploy a raft to a PRoGet instance, and download/manage versioned, read-only versions of it.

Can you try this instead? select top 10 executed_date, Script_Guid, Script_Name from __BuildMaster__DbSchemaChanges2 order by Executed_Date DESC

Can you try to restart the web application (in IIS, or restart the Integrated Web Server), then visit the home page? I think something might be cached... that error doesnt' make much sense.

How about browsing to /applications/4/ (since I saw Application with ID = 4 in the database screenshot).

So, if I understand correctly, using Email for username and PAT for password causes a problem. I'm guessing, this is happening during a Bearer token request.

Would you be able to run ProGet through a proxy server, so you can see the requests that are being made, and reproduce those requests? Alternatively, could you use the code I provided to request a Bearer token?

It's possible that you have an intermediate proxy server that's generating that 400 error, or it's a bug on Microsoft's end. The Microsoft documentation could also be wrong. But let's try to see if we can work-around or get information to Microsoft to fix it.

Can you try...

dbupdater.exe Update . /conn="Data Source=WIN-JG
8E2BQKINK\BUILDMASTER;Initial Catalog=BuildMaster;Integrated Security=SSPI;" /init=yes

Looking at dbupdater, the arguments are dbupdater update <script-path> <connection-string>, so I think the "." will include the current path.

Hmm... that's odd. I don't know. Could be a regression in the Legacy BuildMaster agent, or an unrelated problem.

Can you just upgrade it to the Inedo Agent?

https://inedo.com/support/kb/1039/comparison-of-buildmaster-agents

Can you try running inedosql instead of dbupdate?

Seems, an old link; here it is https://my.inedo.com/buildmaster/versions

Thanks; based on this, there's only one application? Is that correct? Seems like it might be off...

Anyways the database definitely wasn't upgraded. I'm guessing, during upgrade, it was pointed to a different database? Hard to guess...

You can manually upgrade the database:
https://docs.inedo.com/docs/buildmaster/installation-and-maintenance/installation-guide/manual

There may have been an error as well (look in the __BuildMAster__DbSchema2 table).

Hello; thanks to your package, we were able to reproduce this, and it's going to fixed as PG-1695 in the next maintenance release. Thanks!

Hello, these still function in the same manner, so I think it's configuration related.

Can you check to make sure the agent's temp directory is...

1. set to an actual filesystem location and not something accidentally entered into the field
2. owned/writable by the user the SSH agent logs in as
3. (if it's in /tmp or something like that) the mount settings don't forbid executing files

Thanks

FYI; this has been released already as a main feed type, no longer a pre-release :)

I'm going to lock the thread from here to make it easier to submit new issues if anyone has them

Something's definitely not right. I'm guessing, some manual adjustments were done on the server, perhaps moving around some files or something, at some point. It's probably not in a good state, and it's a bit strange to see two sQL servers installed, anyways.

From here, your best bet is to figure out what state your BuildMaster server is actually in: BuildMaster is a standard ASP.NET application and Windows Service, and you can ensure it's configuration is correct here:

• https://docs.inedo.com/docs/buildmaster/installation-and-maintenance/installation-guide/manual

The Agents installed on the remote servers are probably Inedo Agents, and if not, they should be:

• https://docs.inedo.com/docs/inedoagent/overview

Your best bet might be to rollback your server at this point.

thanks!! Very interesting to see.

Oh I see, then this actually just seems like a bug to me :)

I thought the problem was that BuildMaster didn't have the feature of "Dependent Roles". Anyways, I've logged the bug as BM-3588, and we'll try to fix this in an upcoming maintenance release.

In your example, TEST should have GSRole, SRole, and FRole in BuildMaster.

The sync should still work... are you getting an error? If you go from Otter -> BuildMaster, then it just won't copy over the role dependency relationships, but you should still have the roles and role variables, etc.

My guess is that you also have SQL Server 2005 installed? It's possible to have multiple instances of SQL Server installed, and an instance named INEDO is installed by default w/ the installer. you'll want to upgrade that instance.

BuildMaster doesn't support role dependencies; that's an Otter feature only.