Posts made by atripp

I'm sorry for the frustrations, this sometimes causes problems on some environments due to some really old bugs in Windows, and/or some really complicated Active Directory configuration.

There three things to unpack here.

COM ERROR. That error message "Creating an instance of the COM component with CLSID {080D0D78-F421-11D0-A36E-00C04FB950DC} from the IClassFactory failed due to the following error: 800401e4 Invalid syntax (Exception from HRESULT: 0x800401E4 (MK_E_SYNTAX))" is one such Windows bug (it's related to COM+ activation in Microsoft's AD libraries), and it's been around for almost twenty years now.

It just happens on some servers, for no good reason, and it goes away after a reboot. If you navigate to the Advanced Settings of your IIS App Pool and set the "Load User Profile" option to True, it will almost never happen again.

LOCKED OUT. As for getting locked out of ProGet, it's unclear to me; did you run the resetadminpassword command via ProGet.Service.exe? That will update the database, but the Web application must be restarted to read the new authentication options. The most sure way to restart the web application is to go into IIS and restart the application pool. If you ran the resetadminpassword command, then I think the web app wasn't restarted. It's easy to miss one or both steps during a "system is down" panic...

USER NOT FOUND. The message, "there was an error attempting to load the user MYDOMAIN\ME" almost alwasy means that you have Integrated Windows Authentication enabled, and ProGet was unable to map MYDOMAIN to a domain. You can just disable Windows Auth (via IIS), and then you can log-in with a Name/Password if you have this happen.

MYDOMAIN is a NETBIOS alias, and if ProGet doesn't have permission to query the forest for domains (to unmap the alias), then you'll need to specify this in the Advanced settings of the Active Directory (LDAP).

The message is strange, but there's just a bug in the ProGet error message, I think, where it's using the Description of the directory instead of the name of the directory.

@Joshua_1353 sorry about that, I missed it.

As you can see from the flurry of commits I just made to your repository, I'm simply not experiencing the problem. I tried this with both a Secure Credential, and without.

Here's what I discovered during my testing

• the Git Raft Editor isn't very clear about "keeping the same password" if it's unchanged, but that's what happens
• GitHub doesn't seem to look at the username? I entered invalid usernames and it worked fine
• the PAT permission required seems to be "public_repo

Unless you have something that's "doing something" between Otter and GitHub, I think that:

• you made a typo or something in your PAT; you can't have any spaces, it must be exactly a PAT, so make sure to edit and save it
• your PAT doesn't have right permission

From here, about the onloy thing you could do is attach a proxy server like Fiddler between Otter and GitHub, and see specifically what traffic is flowing through. But like I said I think it's one of the above to

@nuno-guerreiro-rosa_9280 the API you referenced is the "NuGet v2 API", and we simply follow the specification when we implement their API.

In the same way, we follow the Docker API specification, which does not provide a way to query a docker image in the way you'd like. I think you have a good point, but it's not our specification or decision, but Docker's

There's not much of a demand for a ProGet-proprietary API for packages at this time, but if you can help us understand the value of this, then we can better considering it - https://inedo.com/open/feature-requests

Hello,

This is a bug, in that PSVerify shouldn't show up on the statement list. Instead, your scripts should, in a manner like this, with a Script Execution mode:

Then, they'll look in Otter like this:

PSVerify VerifyFreeSpace
(
    MinimumFreeGB: 24,
    DriveToVerify: C
);

Here's how this script works behind the scenes...

<# 
.DESCRIPTION
Verifies that an hdars server has the required and expected configuration

.AHDESIREDVALUE 
True

.AHCURRENTVALUE 
$AmpleFreeSpace

.PARAMETER DriveToVerify
The drive letter (such as "C") for what to verify

.PARAMETER MinimumFreeGB
The minimum free space required on the drive, in gigabytes


#>
param ([int]$MinimumFreeGB = 20, [string]$DriveToVerify = "C")

$ActualFreeGB = [int]((Get-PSDrive $DriveToVerify).Free / (1024 * 1024 * 1024))
$AmpleFreeSpace = $ActualFreeGB -gt $MinimumFreeGB

And for reference, the additional help parameter descriptions...

.AHDESIREDVALUE
This is what you wish the configuration value to be. When not specified, the value of $true is assumed.

.AHCURRENTVALUE
This is the actual value of the configuration. When not specified, script's output is used.

.AHCONFIGKEY
This is the "configuration key" used by the script, which is a string that uniquely identifies configuration on a server. It's like a file on disk (a file is uniquely identified by its name), or the name of an IIS Application pool (an application pool is unqiuely identified by its name). Optional. When not specified, the name of the script is used.

.AHVALUEDRIFTED
This is an indicator as to whether the value is considered drifted. When not specified, it's a string comparison of the desired and current value.

.AHEXECMODE
This is either "Collect" or "Configure", and is only used on PSEnsure operations; it will be ignored (or set to Collect, depending on what's easier to code) on PSVerify. Using a PSEnsure without a .AHEXECMODE will cause an error.

The Additional Help items can be specified as a value or a variable; variables will simply start with a $.

hello;

I assume you're referring to the Docker feeds Docker API's implementation?

https://docs.docker.com/registry/spec/api/#listing-image-tags

The docs don't give a great clue, but it does say "ordered lexically", which it seems ProGet's API follows. We can't really add parameters, and changing behavior might break things that otherwise expect this order.

But I do agree the order doesn't' make much sense; we just followed the API. If you can find evidence find how other implementers handle ordering differently (DockerHub, etc.), we can consider to change it.

Cheers,
Alana

Hello;

Please check the BaseUrl property under Admin > Advanced. That will override the X-Forwarded-Host value.

Cheers,
Alana

@scroak_6473 FYI, this has been fixed in our library (ILIB-115) and we plan to upgrade to that library version via PG-1902 in the next maintence release -- so this should be fixed in 5.3.24, scheduled for the Friday the 5th

If you'd like a pre-release let us know, and we can let you know when it's merged in so you can use it sooner.

Hi @Stephen-Schaff,

Thanks for the request!

As you noted, testing/sandbox environments are really important... not just for configuration changes and upgrades, but also for training and exploring features.

But setting up the right test/sandbox environment -- one that delivers value -- can get complicated, because there's a lot of infrastructure to duplicate, which is significantly more expensive than just the licensing fee of ProGet. For example, when you have multiple instances of ProGet Enterprise replicating, some in HA, some not in HA, how and what do you test? And so on.

For ProGet Enterprise customers, we do our best to help develop a test/sandbox environment that works from a value-standpoint. Whether that means mixing less-expensive ProGet Basic instances in, or creating smaller clusters with, say, two nodes instead of four.

For ProGet Basic customers, there's just a single instance and single server to consider. As such, we've found that there isn't a lot of middle ground in terms of value between using ProGet Free as a test/sandbox environment, and purchasing another ProGet Basic license.

What features/configurations specifically would you want to test in ProGet Basic that you couldn't in ProGet Free?

@scroak_6473 thanks again for all the stack traces and help

Good news, we finally were able to track this down. It seems to impact .NET5 only (i.e. ProGet for Linux), and involves using impersonation with a somewhat "faulty" LDAP setup (like yours, with those expected referral errors).

There's an easy fix, and it involves a small library change on our end. We will get it in the next maintence release of ProGet, but I don't have the issue numbers off hand yet.

@Joshua_1353 sorry, I thought it was closed/resolved already

Did you enter main in the Raft Configuration's branch setting? The default branch name in a Git repository is is master, but GitHub now uses main as the default branch, which means you have to explicitly specify it in many tools.

If you don't specify main, then you'll get this error.

Basically, this involves denying or removing the "Overwrite Package" permission.

Under the Security and Access Controls Documentation, we have a fairly detailed guide on Creating and Customizing Tasks which might help go into more detail!

We definitely want to make this as easy as possible, without having to read through lots of documentation, so if you have recommendations on how/where to improve, please share it to us.

Oh and we also gladly accept docs pull requests ;)

That's great thanks for sharing!! We want to build a content backlog, and will add this to it -- at least a way to download from jenkins in this manner. nice find :)

@scroak_6473 sorry but this one is just really tough and we didn't get the chance to dig in deeper, we're still trying to find time as a team to figure it out

Is SA9350001 also a group?

The change that @rhessinger mentioned only impacted the "new" active directory, though there may have been a change in ProGet that happened. Do you remember what you upgraded from? It'd be easier to find on this long list ... https://my.inedo.com/proget/versions

Otherwise, can you elaborate on this?

I looked at the "non legacy" one and I cannot see how to make it work. It seems to not have the needed settings (to map user names and email addresses).

We hadn't found anyone using custom field mappings, and only a small handful of other products even support the ability to use custom fields (Windows doesn't), so we didn't add it...

So I wonder if it would "just work" after all

Hi @mcascone, I'm afraid it's not possible.

nuget or dotnet nuget tools only understand the NuGet Package Format, and the NuGet API, so there's no way for the tools to use different formats or different APIs...

@nuno-guerreiro-rosa_9280 very glad we could fix it! It's definitely a .NET5 bug, and in a totally unexpected place

@jn_7742 unfortunately, Microsoft has also abandoned Source Server in favor of SourceLink, and they will not add Source Server support to Portable PDB files

Technically, ProGet's Source Server works on Linux, but since ProGet can't inject its Source Server urls into the .pdb files, it's impractical to use. I'll let you guess which 20+ year old program is required to inject Source Server metadata into Microsoft PDB files

Bottom line, you'll need to upgrade your processes to use SourceLink

Unfortunately this is just an invalid package... someone really needs to tell the developers at Microsoft that the developers at Microsoft really want packages to be in SemVer2 format, per the Microsoft documentation

https://docs.microsoft.com/en-us/nuget/concepts/package-versioning

Microsoft's documentation seems to no longer describe what Microsoft's once documented "legacy versioning" used to be, but we captured it in our documentation before Microsoft removed it.

Bottom line, the "." is an invalid character in a non-SemVer2 (i.e. four-part version number) pre-release tag. Hence, why it's getting a 400 error. It used to do that on NuGet.org, too. I guess it doesn't now. We probably won't change this, since it kind of works already., and since it's only one package.

But if you email the Microsoft team responsible for that package, they probably will just change their versioning going forward. IT'd be nice if Microsoft documented how their non-SemVer2 packages are supposed to work.

Hello;

The best place to look to troubleshoot this would be the Symbol and Source Server documentation.

Symbol server does work in Linux, but just not with symbols in the "Microsoft PDB format" (which might be what you're referring to). It's fine for Portable PDB (which is the "new" format).

Unfortunately Microsoft seems to have totally abandoned the legacy (Microsoft PDB) format, and has not nor will not open-source the code or specifications required to read those files.

The only supported way is using PDBSTR.EXE, which is essentially a 20+ year old program that modern tools (including ProGet) embed. It can't run on Linux, only Windows.

We decided not to do some hacks (like use WINE) to run it, because there's less and less legacy usage each year.

Best,
Alana

Whoops, my bad!

@brett-polivka thanks so much for the help, this was a baffling .NET5 issue, but so glad we could identify and fix it. This will definitely go in the maintence release as well, of course.

@viceice thanks! We will add a command via PG-1897 to handle this, it seems quite straight forward...

Stay tuned :)

@brett-polivka thanks again for all the help and insights!

I talked this over with the engineering team, and we just decided to remove the ParallelEnumerable query.

Would you be able to try a pre-release container?

inedo/proget:5.3.23-ci.4

@viceice sorry on the slow reply, but I presented this at our engineering meeting and our product manager thinks it's a great idea. But, he asked me to write the documentation on it so they know exactly what changes need to be made, and how it will be used...

So I wonder if you can help?

https://docs.inedo.com/docs/proget/installation/installation-guide/linux-docker#upgrading

Basically I think we should add another heading like this.

Upgrading the Database (Optional)

The ProGet container will automatically upgrade the database when starting up; this upgrade might take a few minutes, which may appear to cause delays to automated probes (??) like Kubernetes.

You can run the following command to instruct the ProGet container to upgrade the database, and then exit.

docker run ???? docker exec ???

I'm just stuck at how to document the new command to run.

From a code/C# standpoint, we can simply just add a option to ProGet.Service.exe to upgrade the database and exit. And then, we won't upgrade the database if the version matches.

@brett-polivka fantastic, thanks for the additional research. This will make it so much easier to debug from here.

Do most of the stack traces look like this? I.e. coming from Inedo.Web.PageFree.PageFreeHandler.GetHttpHandler?

If so, the issue is happening in a totally unexpected place (basically routing a URL). This is just using a relatively simple query with Linq's ParallelEnumerable library methods. Under-the-hood (as the stack trace shows), that seems to leverage operating-system level mutex support... which is likely why it's problematic only on your specific hardware/environment.

We saw a ton of Mono-related bugs early-on with this (some even would SEGFAULT on certain hosts), largely because we tend to find the "corner-cases" with optimizations, like working-around Activator.CreateInstance slowness using using System.Reflection.Emit directly.

Anyways, I'm going to have to involve our engineering team for ideas, because at this point we have to guess how to work-around this.

@entro_4370 said in Support for R and CRAN:

Hi! Any news on R-support in ProGet? There seems to be quite a demand! As a Data Engineer in a large government that bases its package management on ProGet i feel that there is a clear gap here...

I'm afraid it's still not on our roadmap, based on our research. Here's how we evaluate this sort of things.

At first, what's the market demand outside of existing users? We didn't find much opportunity to attract data engineers (or their bosses, who would buy software) who aren't already using ProGet (or a competing product) to use a private repository instead of CRAN directly. It's just not big discussion of R/CRAN community, unlike other package types.

And then, we surveyed/asked users. But we phrase it like this: how much more would you pay for ProGet it if it had this feature, because this is an honest assessment of "how much more value would this feature bring to you" (which is what we want to decide).

Unfortunately everyone we asked said, "it's a nice to have, but we would actually pay X% more if ProGet had feature X instead."

Anyways still something we want to do, but we want to make features that bring most value to most people at first...

of course if you have insights on this please let us know! Cheers.

@scroak_6473 this is very helpful actually, thank you!

There might be a relation to the LdapReferalException here, so we're going to do some more research and try to suggest what to try next. It might involve some new code (and potentially an upgrade).

Please stay tuned...

Thanks for the update @brett-polivka

The ProGet code (application layer) is effectively identical between those images; the main differences is are the underlying container image and .NET5 vs Mono. We had some initial bugs with .NET5, but is the first we've heard of Mono vs DotNetCore problems of this nature out of a lot of users.

Unfortunately, we really don't have any means to diagnose this further at this point. We're not even sure what code we could put in place to do network / operating-system level debugging. And worse, we can't reproduce this in our containers, even on Azure host.

We're open to ideas here, but I think your best bet would be go to as basic of an installation as possible; just a simple container with a simple database, and then add the pieces back until you figure out which thing is causing problems.

If you've upgraded, then the problem is most certainly in your Azure/cluster configuration. I don't know, maybe it's some default configuration setting or something in your cluster that's causing strange network/routing problems... a total guess, because I don't know how to advice to kubernets cluster diagnostic. But that's where to look.

As you said, 100 simultaneous queries are really easy to handle -- the micro-servers (like 512MB RAM, including SQL Server) we use in our testing labs can do that no problem.

I'm afraid there's nothing in ProGet that can really help diagnose this... it would just tell you what you already know -- there are a lot of incoming/waiting connections and the database connections are timing out. All of that's happening at the operating system level, not in the application level of our codes.

@brett-polivka the standard version of ProGet can indeed handle a single project, and in fact many projects from many users at the same time -- it's the most performant private server on the market, by far, in fact.

But a single server/computer/container can only handle so many network connections, and what you're experiencing is the network stack being overloaded

It could be related to the virtualization or container settings, etc. It could be authentication across networks, which shouldn't happen in Azure but it might. But it's not really related to the database, or database performance at all. This is the first place you'll see stack overload, since database connections have short timeouts, compared to connectors like nuget.org.

There is an optimization in 5.3.22 (PG-1891) that might help in scenarios like yours (Linux connected to slow-ish database), so I would try to upgrade.

Otherwise investigate why the network is being overloaded.

@Stephen-Schaff is SA-RestrictedAuto also a group? It's possible in AD to have a User and Group have the same name.

There is (or was) an issue in our products where adding privileges will favor the Group when both a User and a Group have the same name. I'm not sure if it's been resolved, but @rhessinger might be able to confirm it.

The issue is related to the UI more than anything, and it can be worked-around with a "database hack", by changing the PrincipalType_Code from G to U in the Privileges table for the appropriate entry.

Unfortunately, this this behavior is to be expected with "dotnet restore"; it's massively parallel, and will issue as many parallel requests as the build agent allows. Typically, this is more than the server can handle. The end result of this is that the "network stack" gets overloaded. This is why the server is unreachable.

The reason is that, each request to a ProGet feed can potentially open another requests to each connector configured (maybe NuGet.org, maybe others), and if you are doing a lot of requests at once, you'll get a lot of network activity queuing up. SQL Server also running on the network, so those just get added to the queue, and eventually you run out of connection.

One way to solve this is by reducing network traffic (removing connectors to nuget.org, restricting the build agent if possible, etc.), but the best bet is to move to load-balancing with ProGet Enterprise - see this article to learn more: https://blog.inedo.com/proget-free-to-proget-enterprise

Another option is make sure you're not using --no-cache when you use dotnet restore command. NuGet typically will create a local cache of downloaded packages which would help alleviate some of the load on the ProGet server. Passing --no-cache will bypass that local cache and it will cause it to always pull from the server.

Another thing that might help is using the --disable-parallel option in dotnet restore. That will prevent restoring multiple projects in parallel, which will also reduce the load on ProGet.

Fortunately and unfortunately, NuGet does a lot of parallel options that can saturate a NuGet server. When you are restoring a lot of simultaneous builds and solutions with a large number of projects, it can really affect the performance of a server.

This is ultimately where load balancing will come in.

@Stephen-Schaff what was the restriction that you added? The "Publish Packages" and "View & Download Packages" tasks are made up of a collection of attributes, and those are what's tested. So you'd want to Restrict both of those.

Restrictions override grants, but more a more-specific grant will override a less-specific deny. But in any case, a Grant at the system level, with a Deny at the feed level should accomplish what you're doing.

If you can share a screenshot that might help us see it as well

@scroak_6473 we could definitely try a screen share, but in a case like this (where we have no idea what's wrong), it's mostly digging in the code and trying to think of things to try to get a clue for more information. Currently, I'm at a loss... because the error you have shouldn't be happening, but it clearly is.

So now, I had a new idea. I would like to eliminate Docker from equation, as it handles the
"api" username slightly differently than everywhere else. Plus you can do this all in your browser.

Can you try to visit a restricted (i.e. not anonymous view) NuGet endpoint using the "api" user name, and a password?

For example, it should look like /nuget/NuGetLibraries/v3/index.json, and then your browser should prompt for a Username/Password.

Depending on the result of this, we will explore different code paths, and then might need to add some more debugging codes.

Best,
Alana

Hi Simon,

That is strange, it's basically your browser "hiding" the underlying error. Sometimes that happens if the response body is too short.... which could happen if the server got in some really bizare state.

I could find the logs you sent, but they were very random and they also don't make sense; it's random ASP.NET errors, but we can't see the full situation. In general the 500 errors should be logged in ProGet > Admin; this will provide a stack trace as to what errors are happening.

If you can't get to the admin page, then something is really wrong with the server. I would try restrating your container.

Alana

@scroak_6473 is it the exact same message? Basically, the "api" user not found in the directory?

@Stephen-Schaff thanks for the bug report, I verified that this may happen depending on permission of user, and which feeds they can/can't use --- but it seems an easy enough fix that we can do via PG-1894 (targeted to next release) - the packages can't be viewed upon clicking, but it's a sub-optimal experience for showing packages they can't see

@philippe-camelio_3885 oh i see; you mean, capture the output of a process or script execution into a variable or something.

Definitely something to consider as an enhancement, I think. That wouldnt' be too bad (though the variable could get huge, and rutnime variables raen't realy designed for large amounts of text like that)

Thanks for the update; and the upcoming fixes will certainly make it so that purging is much more efficient on the manual execution side, just in case there's an "Explosion" of executions like this.

So right now, my concern is that it's "logging every sync" (once per hour), due to a sort of bug or something. Can you check what's getting logged in Infrastructure Sync? You should be able to see this under Admin > Executions, and see if you can spot a pattern?

No rush. The infra sync executions should clearly show a change history of what was updated on the infra side.

@philippe-camelio_3885 said in OTTER - Capture Logs from block execution and assign to variables ?:

The ANSIBLE::BM-Playbook module returns the log:

By this, I assume you mean, writes to the Otter execution log, either via Log-Information or an execute process? You might have to do this via PSExec, write the logs to text, and parse it out that way using a regular expression...

At this time, there's no way to read entries from the log during a live execution.

Best,
Alana

Thanks @scroak_6473; I found the email, and can see a lot of information from what you sent.

I can clearly see the identical api challenge/response, and the different behaviors from ProGet.

Unfortunately, I'm not able to reproduce the scenario on this end, using our own instance and a domain impersonated account. But I think that's because this issue may have already been fixed with PG-1859; would you be able to upgrade to 5.3.22 to confirm?

@Joshua_1353 I also got this message from GitHub;

Basic authentication using a password to Git is deprecated and will soon no longer work. Visit https://github.blog/2020-12-15-token-authentication-requirements-for-git-operations/ for more information around suggested workarounds and removal dates.

I wonder if it's disabled on your account, already

@Joshua_1353 Thanks, I was able to access it without a problem, and added a test script.

But I made a mistake at first, and I kept the default branch name of "master" in the Raft dialog. On your repository, the master branch is named "main".

What is the configuration inside of Otter? If it says "master" then there will be a sort of error.

I'm not familiar with what apt-cacher is, but I'm guessing it is a kind of proxy/caching for APT packages?

I'll note that connectors are not supported for Debian feeds at this time; only packages you publish yourself to that feed.

@mathieu-belanger_6065 said in Connection reset while downloading npm packages:

I am curious, would there be an impact on performance when "piping" connectors together? For example, internal feed A has a connector to internal feed B, which has a connector to internal feed C, which has a connector to npmjs.org?

Connectors are accessed over HTTP. So assuming you have a "chain" like A --> B --> C --> npm.js, (i.e. different 3 feeds and 3 different connectors), each request may yield 3 additional requests.

So when your browser asks feed A for package typescript@3.7.4, then following will happen.

1. If the package is cached or local, the file is streamed to the browser
2. Each connector (just B, in this case) is queried over HTTP for typescript@3.7.4
3. The first connector that returns a response, the response body is streamed to the browser

Each connector follows the same logic. When ProGet (via a request to feed A) asks feed B for that package, the same logic is followed:

1. If the package is cached or local, the file is streamed to the browser
2. Each connector (just C, in this case) is queried over HTTP for typescript@3.7.4
3. The first connector that returns a response, the response body is streamed to the browser

Continuing the pipe, when ProGet (via a request to feed B via a request to feed A) asks feed C for that package, the same logic is followed:

1. If the package is cached or local, the file is streamed to the browser
2. Each connector (just nuget.org, in this case) is queried over HTTP for typescript@3.7.4
3. The first connector that returns a response, the response body is streamed to the browser

This is why caching is important, but also why chaining may not be a good solution for high-trafficked npm developer libraries like typescript. The npm client basically does a DoS by requesting hundreds of packages at once. Same is true with nuget.exe as well.

@arozanski_1087 I don't think repackaging in the UI is the best for this in theory it should work, but we we never designed or tested it for this scenario; it's more to change pre-release versions.

In theory having both versions (1.0, 1.0.0) in the feed should work; I would just delete all versions, then upload from disk, then edit the file on disk (1.0 to 1.0.0), then upload the other.

We heard of one other user doing that.

If you can find "the trick" please do share, because certainly you won't be the last person using this ancient, broken package ;)

Hi Simon, you can send to support at inedo dot com. Please include [QA-473] in the subject, so we can find it easily :)

@arozanski_1087 ah, yes. Owin. It's most definitely a problem package

https://www.nuget.org/packages/Owin

There's not much we can do about this, when you have a connector to NuGet.org on the same feed. The reason is this.

• Depending on which NuGet.org API call you use (v2, v2+Semver, v3), the package will be reported from NuGet.org as either 1.0 or 1.0.0.
• Depending on which NuGet Client you use, and in which context, will first request "1.0.0", then request "1.0" if that fails. Or it might do it in the reverse order, depending on what's in dependency file.
• The .nuspec is 1.0, as you noticed

NuGet stopped supporting this back in 2016, but unfortunately... the developers are using an 8+ year old package.

If you don't use a NuGet.org connector on the feed, you can simply follow those manual repackaging steps I mentioned, and create your own Owin 1.0.0. The clients will still be able to get it, but may issue a 1.0 request first.

The two api endpoints I can think of are:

• /v2/_catalog returns all repository names ("container" names)
• /v2/<repository-name>/tags/list returns all tags within specified repository

Some more details are here: https://docs.docker.com/registry/spec/api/

That URL is a bit strange, and I wouldn't expect it to work.

What is /endpoints/Chocolatey/content/chocolatey.license.xml? If you visit in your browser, what happens?

Ah, glad we can see the message now.

Okay - so that error message is coming from within libgit2, and it basically just means there's an authentication error. The most common reason is that the username/password credentials for GitHub are invalid, but the git server (github in this case) won't give the reason -- it could also can also mean your account is locked, you're using a "username" instead of a "token", don't have access to the branch, etc.

If it works on your local computer, but not BuildMaster, then it means there's "something" in either your local repository or global git configuration that's allowing it. Usually, this is stored git credentials, or even a plug-in that's allowing it.

That being said, it could also be related to the credentials changes in Otter v3; the problem is we can't reproduce it here. Could we trouble you to do this.

1. Create a repository on public GitHub.com (i.e. not a private repo, or on a private github instance)
2. Try that, verify it works
3. If it doesn't, give me access whatatripp, so then we can test your repository

With this, we can be looking/working on the same repository, and at least figure out where the problem lies.