Buildmaster fresh install / Unable to clone repository

carsten_1879

Used docker image: buildmaster:latest

When trying to create a new application and choose azure devops repository i am able to browse projects and repositories, afterwards an error occurs:

Unable to clone repository: could not set credentials: could not open libssl

Username and personal access token setted, also ssl verification turned off, but still no luck.

Any chances to resolve this issue?

atripp

Hi @carsten_1879 ,

Can you try using buildmaster:24.0.5 instead of latest? There was a change in the Git library, and I'm wondering if this is a kind of regression on some platforms.

Thanks ,
Alana

carsten_1879

Tried exactly that version, but the same error shows up. Wondering that you've never fixed this in the product. Seems to be not widely used, right?

stevedennis

Hi @carsten_1879,

The underlying error message ("could not open libssl") is related to your specific environment; it's an error occurring at the operating-system level where it's failing to load a dependency that is already installed (in Docker/Linux version).

We made a change in BuildMaster 2024.06 that may have had an impact, but since you tried 2024.05 we can rule that change out.

So at this point, we need to try to figure out why it isn't working for other users (including us during testing).

Can you tell us a bit about your environment? Like the host? K8 vs Docker? Do you have any special security software running?

Thanks,
Steve

carsten_1879

@stevedennis said in Buildmaster fresh install / Unable to clone repository:

libssl

Thanks for answering quickly. First i had this issue testing buildmaster on a default docker installation running on ubuntu 24.04.2 LTS with docker 27.5.1. I had to put the topic aside for some time and now coming back to evaluate buildmaster. As i now have a kubernetes cluster available my first idea was to simply switch over to kubernetes and try it out again, but there the same issue arrived. Still the same error message. Buildmaster is now running on MicroK8s 1.32.3, default install, no additional security software installed on that host. The strange thing here is when adding a new connection to azure devops and entering the instance url, username and personal access token, i am able to browse the projects and repositories in the buildmaster setup dialog. I mean, i can see all the projects and repositories which are available at the azure devops server, so the connection should be already established. I've checked multiple times the personal access token and also created a new one, but still no luck. Also tried to set the minimum log level of buildmaster to 0 which seems to be debug, but no usable output is shown. Tell me if you need more informations

stevedennis

Hi @carsten_1879,

The mystery continues.

The strange thing here is when adding a new connection to azure devops and entering the instance url, username and personal access token, i am able to browse the projects and repositories in the buildmaster setup dialog.

Behind the scenes, the project/repository browsing are handled using the ADO API, which is called from .NET code, which eventually uses the openssl library installed on the operating system. Clearly, this is working... and I think your credentials, etc. are fine.

HOWEVER, the cloning is handled using a totally different library called libgit2. It uses libssl behind the scenes, and clearly this is not working.

Unfortunately, libgit2/libssl are a lot less forgiving with SSL issues than .NET and a lot more cryptic. So it's possible that the "could not open libssl" isn't even the actual error message.

The real mystery here is why is this not working for you but it works fine for us? Especially since we're both using the same container image?

One guess I have at this time is a proxy server is "reencoding" the SSL stream using an algorithm libssl doesn't understand? And then, this error is thrown? Or, perhaps it's a locale thing? Just some wild guesses.

Could you try this with like, GitHub perhaps? Or maybe you could try this on a public server that we could access, like a Amazon LightSail server?

FYI - on our end, we have written our own Git library from scratch using .NET that will not have these problems. It's almost ready.

Thanks,
Steve

carsten_1879

Hello. As you suggested i tried it with github. No issues there. But as you said it could be a problem with SSL i reconfigured the azure devops server to allow also http and disabled the redirect to https. Just to check if the issue then still exists. And the result: Even if you use pure http for the azure devops server you get the ssl error, which is very strange. Also tested this with another azure devops server at another site. Same problem there. Just to mention: The url i am using in buildmaster directly goes to the azure devops server, so there is no reverse proxy or something like this in between. Maybe you can reproduce the issue if i give you the azure devops url of my site. It's reachable for public. I created a test user and repository for you and created a personal access token, so you are able to reproduce the error at your site. Just give me a note where i can send you the credential informations so you can test it

stevedennis

Hi @carsten_1879 ,

Huh weird... so GitHub works, but your ADO Server doesn't. I bet it's related to some kind of TLS or something?? We may be able to see it with Wireshark or something.

Anyway if you can give provide us with credentials that would be great. We can then test on our new GitLibrary and see if it works.... maybe the solution is just to get you that instead.

Just email the info to support at inedo dot com, and include [QA-2678] in the subject line. But let us know when you send it, because we don't monitor that box.

Thanks,
Steve

carsten_1879

@stevedennis said in Buildmaster fresh install / Unable to clone repository:

[QA-2678]

Yes, perfect. I have sended the email with the subject [QA-2678] Additional informations
a few seconds ago... Hope to hear from you

stevedennis

@carsten_1879 excellent, thanks! Please give us a little time to review this.

We will also test it with the new Git library, which is available in buildmaster:24.0.8-ci.9 container image -- you have to go to Admin > Advanced Settings and enable it.

If you get a chance to try it yourself, please let us know. We are currently testing it ourselves on our main server.

carsten_1879

For sure. I tested it directly. Unfortunately this produces a 400 BadRequest error. So for first it seems to have no ssl problems. Too bad. One additional thing to mention which confused me a little bit: You allow to configure to access the azure devops via a PAT. If you use a PAT, then a username is obsolete, as the azure devops documentation says you should use a Basic Authorization to access the server which should be a base 64 encoded string create from "USERNAME:PAT" where USERNAME should be left empty. But in your UI the Username is a required field, which then for me doesn't make sense. Also to mention: If you ignore this and pass the username along with the pat, a git clone for example with this basic authorization also fails. Maybe it could be the case you did that wrong. But this would be strange, as you told me you don't have these issues. Or maybe you're testing against the cloud AzureDevopsServices. We are talking here about a on premise Azure Devops Server installation. Maybe their authorization mechs differs... Just some thoughts...

stevedennis

@carsten_1879 thanks for the update

We were able to reproduce this using your credentials, and It's definitely an "error reporting an error" on Linux. On Windows it's clearer ("too many redirects or authentication replays").

We do not test on ADO Server, but we quite a few customers users with it - who knows why it works for them but not you. Anyway, please bear with us as we figure it out.

stevedennis

Hi @carsten_1879 , just to give another update.

The issue appears to be localized to your ADO Server, perhaps it's the version or locale. The authentication header doesn't seem to make any difference.

Whatever the case is, the library we use, libgit2, simply cannot clone from it -- for whatever reason your ADOS is returning a 400 with some German error message when doing what should be a totally fine request. We did not test with git.exe.

Our new, proprietary library also does not work, but for a different reason - it looks like ADOS is using an ancient format (like from 2010??) for one of the reposes, so we don't support that yet. At least we think.

We'll need too spend a bit more time researching this, so it'll get pushed into the following weeks.

Thanks,
Steve

atripp

Hi @carsten_1879 ,

We should have a build of BuildMaster 2025 later this week that you can try out that includes the new library. We'll let you know!

Thanks,
Alana

atripp

Hi @carsten_1879 ,

Can you try out buildmaster:25.0.0-ci.23 container image?

I would do a new/fresh installation - new applications created in it should use the new Git library. It seems to work, but would like to see if it works for you as well.

Thanks,
Alana

carsten_1879

@atripp I was full of hope. Bute error messages are getting stranger than before.
Now we have: Unable to clone repository: HashDelta is not supported unless a repository is available.
:-(

carsten_1879

@atripp One additionally thing to mention. I tested this also with the test repository i given to you. This repository i was able to clone, but not others.

What i have found out: There is a bug in your implementation, which leads to the error given in my last reply:

If the repository name contains a dot (.) for example a repository with name: image.kubernetes-test

You'll get this error. So i think url or content encoding is not done the right way. I tried to rename one repository from: image.kubernetes-test to image-kubernetes-test and voila... That worked. So, maybe you can fix this too, so your implementation is usable for all valid repository names

gdivis

Hi @carsten_1879,

Thanks for trying it out. It is new, and has not been tested as much against older repositories. That said, we've reproduced this error and it's now fixed in the latest build - buildmaster:25.0.0-ci.28 if you would like to try it again. It's caused by older repos that don't support offset deltas being transmitted during a clone or fetch. Small repos that we used for testing ADOS so far were not large enough to have deltas at all, so those worked. So far as I can tell, the repository name containing a dot doesn't have any effect.

Thanks!
-Greg